1. A reliable tool that would execute the splitting operations on videos
2. A custom UI tool for selecting timestamp ranges

FFMPEG is a popular software for editing and manipulating videos, so this was my go-to tool to use for this project. Being my first time creating custom android views, I spent some time developing the custom slider tool for selecting time ranges. The first implementation appeared to work well, and after some time with learning and practicing with custom views, I discovered some problems with the nature and behaviour of the bloated code. Here's a screenshot of what it looked like, and a link to the source code.

This writeup is divided into two phases of major development:

1. The SelectionView
2. Foreground Services & Work Manager

1. The Selection View

I used a ViewGroup for the selection view, and created custom View components for the thumb markers and progress marker. My approach to this involved a bunch of unnecessary callbacks between the viewgroup and the marker views.

For the recent release, I rewrote the selection view utilizing a single View component that manages all interactions including marker sliding and view dragging. Instead of using child views as markers in the SelectionView, I created variables that represents the coordinates of the markers. These coordinates are made visible to the user as touchable regions drawn on the screen with the Canvas object provided in the onDraw() method. The touch movements are captured within the CustomSelectionView region, and the values obtained are used to calculate the expected position values for the thumb coordinates based on where the touch events occurred. When these coordinates change, the canvas redraws them on screen after a call to invalidate() to show the new position of the thumb markers.

About the Thumb Markers

A thing to note about the markers is that the start and end values are initialized as 0 and width value of the view respectively. However, these values are not solely used to determine percentage offsets of the start and end values of the thumb markers and the reason is based on the expected behaviour of the selection view. The SelectionView is expected to capture all possible seconds from a video, and the thumb markers (including the touchable region) already occupy some space from the total size of the SelectionView. So, if we are to use 0 and the width of the view to represent the mininum and maximum values respectively within the slider, everything would be fine as long as we allow for intersection between the two markers.

However, we obviously wouldn't want that since it would be a bit difficult for users to properly select ranges that are between the marker coordinates and the size of the marker. The solution is to prevent the respective markers from intersecting each other. That means the minimum value for the start marker would be 0, and the maximum value for it would be --> the end marker coordinates - size of its marker.

    /**Kotlin
     * Returns the maximum permissible value of the left thumb
     * i.e ranges between 0 and the start of the
     * right thumb
     */
    private fun getMinMaxLeft(positionX: Float): Float {
        return max(0f, min(positionX, thumbEnd - (SIZE * 2)))
    }

Also the minimum value for the end marker would be --> the x coordinate of the start marker + size of the marker, while the maximum value would be the width of the selection view.

    /**
     * Returns the maximum permissible value of the right thumb
     * i.e not exceeding the width of the screen and the end of the
     * left thumb
     */
    private fun getMinMaxRight(positionX: Float): Float {
        return max(thumbStart + (SIZE * 2), min(positionX, width.toFloat()))
    }

Using this approach brings in another problem that needs to be resolved. Calculating percentage offset for both thumb markers as coordinate_value/width would be incorrect since we've factored the sizes of both thumbs from SelectionView width to represent the touchable regions for the thumbs from their respective start coordinates.

    /**
     * returns true if the touch is between the start thumb
     * position plus width of the thumb
     */
    private fun isLeft(positionX: Float): Boolean {
        return (positionX in thumbStart..thumbStart + SIZE)
    }

    /**
     * returns true if the touch is between the end thumb
     * position minus width of the thumb
     */
    private fun isRight(positionX: Float): Boolean {
        return (positionX in thumbEnd - SIZE..thumbEnd)
    }

Doing this means that it would be impossible to get 0.0 value when both markers meet due to the nature of the real and apparent coordinate values of the start and end thumb. Look at the image below as an example. Both markers meet, but instead of the expected time difference to be 00:00:00:000, it shows 00:00:20:615 because the offset is still being calculated with the real coordinates and width of the SelectionView.

The way around this was to factor in the size of both thumb markers and subtract them from the actual width of the view to represent the apparent total width.

    /**
     * Returns the width of the draggable region
     */
    private fun getActualWidth(): Float {
        return width - (SIZE * 2)
    }

This directly affects the correct values for the start and end coordinates when calculating the percentage offsets relative to the SelectionView size. So to obtain the correct results, the thumb start coordinates remains as is since the starting point is from 0. However the end coordinates needs to be altered to match up with the new maximum value declared in the code above. In the getActualWidth() function, the total width was reduced by the size of the two thumb markers, so this same value would have to be subtracted from the end coordinate.

    private fun getThumbEnd(): Float {
        return (thumbEnd - (SIZE * 2))
    }

Now, the distance offset of the start and end coordinates can be calculated correctly as follows:

    /**
     * Returns the percentage value position of the start of the thumb
     * relative to the draggable region
     */
    private fun getThumbStart(): Float {
        return (thumbStart) / getActualWidth()
    }

    /**
     * Returns the percentage value position of the end of the thumb
     * relative to the draggable region
     */
    private fun getThumbEnd(): Float {
        return (thumbEnd - (SIZE * 2)) / getActualWidth()
    }

Here's the final result for the complete SelectionView:

Pro-tip

The value of the end coordinates is expected to be equal to the width of the SelectionView, however the width of the view isn't available during intialization and therefore yields zero. When the measured width and height eventually becomes avaiable, android calls the onSizeChanged() method and delivers the dimensions as parameters, so this method can be overriden to set the end coordinate of the second thumb marker

    override fun onSizeChanged(w: Int, h: Int, oldw: Int, oldh: Int) {
        super.onSizeChanged(w, h, oldw, oldh)
        this.thumbEnd = w.toFloat()
        this.h = h.toFloat()
        this.w = w.toFloat()
    }

2. Foreground Services and Work Manager

During the trimming operation on the video, I delegated this action to a foreground service that handles the execution of the process and displays the status via a notification. Since the previous release, the app worked as expected on devices I tested with. However, after some months, I recently discovered that the app crashes on Android 12+ devices. It was difficult to detect the root cause of this problem since the crashlytics report from firebase noted the crash, but it offered no reason as to why the crash happened nor any information on the model of devices experiencing this issue.

After a bit of digging into why this was happening, I was able to trace the cause to the foreground service. I came across this caveat from the documentation about background processes restricted from lauching foreground services on Android 12+. WorkManager is recommended as an alternative to this. It was hard to believe this could be the problem since the foreground service could have only be initialized after the user clicks a button on the app, and this means the app would have had to be running and displaying on screen.

I decided to give it a try just to see if it would resolve the error and surpisingly it did! The problem was the foreground service all along, and I'm still completely baffled about why the simple foreground service previously implemented crashed on higher android versions despite not violating the rules required to run one. And even if it did crash, why wasn't the ForegroundServiceStartNotAllowedException thrown and logged in the firebase crashlytics log reports? [

Anyway, I'm glad I revisited this project because I was able to improve it from its previous awful state🙉 into something much better and more efficient for use by every social media story spammer out there😁. If you don't already have the app, click here to get it from the App Store! The full source for the project is also publicly available on my Github profile. You can check it out and lemme know what you think!

There are various techniques used to blur images, some of which are;

• Mean filters
• Gaussian blurs
• Bilateral filters

A popular approach to applying filters is the Kernel Convolution, where a set of numbers (kernel) is passed accross the pixels of an image. A kernel is a small matrix used for blurring, and other image effects. The size of the kernel must be odd, and the greater the value of the kernel, the greater the blurring effect. The kernel is applied to every pixel in the image, with the currently selected pixel at the center of the kernel. Each value in the kernel is multiplied with the respective pixel values within the matrix range highlighted by the kernel and summed up together. The central pixel value is eventually replaced with the result of these operations.

Native approach to blurring images

To implement the mean filter in android, we can obtain the pixels by converting the image to bitmap, applying the kernel to every pixel. We'll be applying the blurs to this sample image:

Here's my implementation for this:

private suspend fun meanFilter(bitmap: Bitmap) {
        val kernel = floatArrayOf(1 / 9f, 1 / 9f, 1 / 9f, 1 / 9f, 1 / 9f, 1 / 9f, 1 / 9f, 1 / 9f, 1 / 9f)
        return withContext(Dispatchers.IO) {
            val diameter = 3 // 3x3 matrix
            val cols = bitmap.width
            val rows = bitmap.height
            val step = (diameter - 1) / 2

            val imagePixels = IntArray(rows * cols)
            val newImagePixels = IntArray(rows * cols)

            bitmap.getPixels(imagePixels, 0, cols, 0, 0, cols, rows) // here we are assigning all pixel values to the IntArray so we can process the individual pixels via the kernel

            for (i in 0 until cols) {
                for (j in 0 until rows) {
                    val startCoord = Point(j - step, i - step)
                    val endCoord = Point(j + step, i + step)

                    val matrix = ArrayList<Int>()
                    for (_col in startCoord.y..endCoord.y) {
                        for (_row in startCoord.x..endCoord.x) {
                            if (_row < 0 || _row >= rows || _col < 0 || _col >= cols) {
                                //ignore pixels out of matrix bounds
                                matrix.add(0)
                            } else {
                                val pixel = imagePixels[_row * cols + _col]
                                matrix.add(pixel)
                            }
                        }
                    }

                    val sum = matrix.mapIndexed { index, value ->
                        val multiplier = kernel[index]

                        val alpha = value shr 24 and 0xFF
                        val red = ((value ushr 16 and 0xFF) * multiplier).toInt()
                        val green = ((value ushr 8 and 0xFF) * multiplier).toInt()
                        val blue = ((value and 0xFF) * multiplier).toInt()
                        ((alpha) shl 24) or ((red) shl 16) or ((green) shl 8) or (blue)
                    }.sum()
                    newImagePixels[j * cols + i] = sum
                }
            }
            Bitmap.createBitmap(newImagePixels, cols, rows, Bitmap.Config.ARGB_8888)
        }
    }

You'll notice we did some bitwise shift operations on the pixel values before applying the kernel on them, that is because every pixel is made up of certain 8-bit color values, and these values need to be isolated before being processed, then combined again to form a pixel value.

Blurring with OpenCV

Now that we have an idea on how blur filters work natively, a useful image processing library that does a lot of these special type of operations faster is OpenCV. Here, images can be blurred more accurately using just a few lines of code compared to what we initially wrote. You should have OpenCV setup with a new or existing android project to get started with implementing them in your application.

Trying out the same blurring filter above with OpenCV;

    private suspend fun blurImage(bitmap: Bitmap):Bitmap {
        return withContext(Dispatchers.IO) {

            val imageSrc = Mat()

            Utils.bitmapToMat(bitmap, imageSrc)
            val size = 24.0

            val destination = Mat()
            Imgproc.blur(imageSrc, destination, Size(size, size))

            val copy = Bitmap.createBitmap(
                destination.width(),
                destination.height(),
                Bitmap.Config.ARGB_8888
            )
            Utils.matToBitmap(destination, copy)
            copy
        }
    }

Here's the output:

Pretty straightfoward eh? Great! Well, what if we have to only blur only a particular segment of the image, instead of the whole image. We'll use the submat() function to select our region of interest, and only blur those parts we need. As an example, we'll try to blur an image starting from 30% away from the origin at the top and left side of the image, having a width/height that is 30% of the image size.

    private suspend fun blurImage(bitmap: Bitmap):Bitmap {
        return withContext(Dispatchers.IO) {

            val imageSrc = Mat()

            Utils.bitmapToMat(bitmap, imageSrc)
            val size = 24.0

            val rows = imageSrc.rows()
            val cols = imageSrc.cols()

            val rowStart = (0.3 * rows).toInt()
            val rowEnd = (0.6 * rows).toInt()
            val colStart = (0.3 * cols).toInt()
            val colEnd = (0.6 * cols).toInt()

            val subRegion = imageSrc.submat(rowStart, rowEnd, colStart, colEnd)
            Imgproc.blur(subRegion, subRegion, Size(size, size))

            val copy = Bitmap.createBitmap(
                imageSrc.width(),
                imageSrc.height(),
                Bitmap.Config.ARGB_8888
            )
            Utils.matToBitmap(imageSrc, copy)
            copy
        }
    }

Image output:

This is a bit different from the previous implementation, and the explanation is quite simple. the submat function accepts a number of arguments that defines rect points we need as our region of interest. Even though this function attempts to select just a segment from the original image and returns it, it still holds a direct reference to the original image so anything we do with the subregion reflects onn the selected region, and also inadvertently affects the original image. Here's an explanation for it

Now, we have a bit of control over the kind of blurring operations we need for our images, but not so much control as we would want. With the previous example, we can only blur out certain segments in rectangular shapes only. What if we need to make the blur masks to be based off various forms such as circles, rectangles, or even freehand shapes. How do we achieve this using OpenCV?

Advanced blurring in OpenCV

For this example, we'll try reimplementing the rectangular blurring using a more dynamic approach before trying out something different. OpenCV consists of a number of methods that allows for creating masks in various shapes we want ranging from rectangles, triangles, and other polygons.

For rectangles, we'll create a rectangular mask in black and white and perform bitwise operations on this mask so that our blurring operations can occur within these masks only. To create a rectangular mask in OpenCV, we'll make use of the Imgproc.rectangle() function.

    private suspend fun blurImage(bitmap: Bitmap):Bitmap {
        return withContext(Dispatchers.IO) {
            val imageSrc = Mat()

            Utils.bitmapToMat(bitmap, imageSrc)

            val rows = imageSrc.rows()
            val cols = imageSrc.cols()

            val startY = (0.3*rows).toInt()
            val endY = (0.7*rows).toInt()
            val startX = (0.3*cols).toInt()
            val endX = (0.7*cols).toInt()

            val innerMask = Mat.zeros(imageSrc.size(), imageSrc.type())
            innerMask.setTo(Scalar.all(0.0))

            //thickness set to -1 for inner fill
            Imgproc.rectangle(
                innerMask,
                Rect(startX,startY,endX,endY),
                Scalar.all(255.0),
                -1
            )

            val outerMask = Mat()
            Core.bitwise_not(innerMask, outerMask)

            val blurredImage = Mat()
            Imgproc.blur(imageSrc, blurredImage, Size(64.0, 64.0))

            //original image minus cutout
            val originalCut = Mat()
            Core.bitwise_and(imageSrc, outerMask, originalCut)

            //blurred image minus cutout
            val blurredCut = Mat()
            Core.bitwise_and(blurredImage, innerMask, blurredCut)

            val merge = Mat()
            Core.bitwise_or(originalCut, blurredCut, merge)

            val copy = Bitmap.createBitmap(merge.width(), merge.height(), Bitmap.Config.ARGB_8888)
            Utils.matToBitmap(merge, copy)
            copy
        }
    }

And here's the result:

Here's an explanation of what is going on.

val innerMask = Mat.zeros(imageSrc.size(), imageSrc.type())
innerMask.setTo(Scalar.all(0.0))

Here we are creating an empty image, with pixel values set to 0, meaning this is a black image

Imgproc.rectangle(innerMask, Rect(startX,startY,endX,endY), Scalar.all(255.0), -1)

The above code means we are drawing a rectangle within the black image we created. The start and end points are provided within the rect object. This region is filled with a white mask via Scalar.all(255.0) having a thickness of -1. According to the docs, a negative thickness value enables the shape to be filled rather than having a stroke for the drawn shape.

At this point, we have a black image containing a white mask. We'll need this mask to perform some bitmanipulation and embed our blurred segment within this white mask, and the rest of the original image within the black region.

Here's the result for the inner mask:

Now, unto the next piece of code

val outerMask = Mat()
Core.bitwise_not(innerMask, outerMask)

From the black and white image, we're trying to invert the segments such that the black region becomes white, and the white region becomes black. It follows the basic principle of binary operators using the NOT gate. Such that when we have a value like 1110011 and a not gate is applied here, the ones (1) becomes zeros (0) and vice versa. Luckily, OpenCV has a function that allows for performing these forms of operations. In the above piece of code, we used Core.bitwise(srcImage, destinationImage) to achieve this.

At this point, the value of outer mask would appear like this:

For the outer mask, we'll attempt to copy the the orignal image pixel values to this variable using another bitwise operator, and. This also follows the same principle as bitwise operations when combining two binary numbers together, such that there can only be a truthy value when all values involved are non-zero, else we have a false value. This way the colored parts of the original image remain the same, while the black regions remain zero as required.

val originalCut = Mat()
Core.bitwise_and(imageSrc, outerMask, originalCut)

here's the result:

We'll be doing the opposite for our blurred image as well to obtain a blurred mask:

val blurredCut = Mat()
Core.bitwise_and(blurredImage, innerMask, blurredCut)

Result for this:

Finally, we combine the original cut and the blurred cut to achieve a single image that represents a merge between the two image. The black pixels would be added with the blurred pixels to yield a blurred pixel, thereby replacing all black pixels between both images. A bitwise operation that gurantees this form of idea is the or operator, whereby we have a truthy value as long as one of the values involved in the artihmetic operation is non-zero. Here's how it is implemented.

val merge = Mat()
Core.bitwise_or(originalCut, blurredCut, merge)

Now merge represents our true blurred region results we're trying to achieve as we did using submat. Here's the result you should be getting:

If we want to achieve a circular blur using this same approach, all we have to do is replace the Imgproc.rectangle() with Imgproc.circle() and pass in the required parameters required to draw the circle. Such as this:

val circleMask = Mat()
Imgproc.circle(circleMask, org.opencv.core.Point(rowEnd/2.0, rowEnd/2.0), rowEnd/2, Scalar(1.0))

One more thing...

Looking at the previous image, you'll notice there's a sharp edge between the blurred image bounds and the rest of the original image. What if we wanted to avoid this, such that the blurred region appears to blend alongside the original image as shown in the figure below?

We'll be doing something called an alpha blend, which is a process of overlaying a foreground image with transparency over a background image. At every pixel of the image, we need to combine the foreground image, with the background image using an alpha mask. The foreground image would be the blurred image, the background image is the original image, and the alpha mask would be represented by the blurred black and white mask. The formula for doing this operation is gotten from here:

Here's our implementation in kotlin:

    private fun alphaBlend(imageSrc:Mat, blurredImage: Mat, blurredMask: Mat): Mat{
        val destination = Mat(imageSrc.size(), imageSrc.type())

        for (col in 0 until destination.cols()){
            for (row in 0 until destination.rows()){
                val pixel = imageSrc.get(row, col)
                val blurredPixel = blurredImage.get(row, col)
                val blurVal = blurredMask.get(row, col)[0] / 255

                val newPixelValue = pixel.mapIndexed { index, value ->
                    (blurVal * blurredPixel[index]) + ((1.0f - blurVal) * value)
                }.toDoubleArray()

                destination.put(row, col, *newPixelValue)
            }
        }
        return destination
    }

We're getting every pixel from the blurred mask, dividing it by 255 such that we get a value ranging between 0 and 1. We then multiply this value with every blurred pixel, and subtract the same value with from 1.0 then multiply it with the original image. We finally obtain the new pixel value and assign it to our new image variable. The transparency would be 0 on the original and blurry image, thereby yielding the same pixel value, however this value varies as it approaches the edges of the blurry image, thus leading to the transparent blurry effect along the edges of the blurred mask.

Now we'll update our blurImage() function to match this blend function.

    private suspend fun blurImage(bitmap: Bitmap):Bitmap {
        return withContext(Dispatchers.IO) {
            val imageSrc = Mat()

            Utils.bitmapToMat(bitmap, imageSrc)

            val innerMask = Mat.zeros(imageSrc.size(), imageSrc.type())

            //thickness set to -1 for inner fill
            Imgproc.circle(
                innerMask,
                Point(imageSrc.width() / 2.0, imageSrc.height() / 1.5),
                600,
                Scalar.all(255.0),
                -1
            )

            val blurredImage = Mat(imageSrc.size(), imageSrc.type())
            Imgproc.blur(imageSrc, blurredImage, Size(64.0, 64.0))

            val blurredMask = Mat(innerMask.size(), innerMask.type())
            Imgproc.blur(innerMask, blurredMask, Size(64.0, 64.0))

            val merge = alphaBlend(imageSrc, blurredImage, blurredMask)

            val copy = Bitmap.createBitmap(merge.width(), merge.height(), Bitmap.Config.ARGB_8888)
            Utils.matToBitmap(merge, copy)
            copy
        }
    }

Now this is the final result:

A new problem emerges

One thing you'll notice about this is that our blur process has become really slow, taking up to over 15 seconds to achieve the blur result. And this is because of two reasons;

• We are looping through every pixel in the 2-dimensional grid, meaning for a 1080x1080 image, we're actually iterating through the image 1,166,400 times which is quite a lot!
• We are accessing the pixels directly from the image using the get and set methods, which are fairly expensive operations.

A way out of this mess

After doing a bunch of research on various ways to optimze this process, I was finally able to figure out a way out of this thanks to this post which provided a couple of insights regarding OpenCV.

From this post, I figured the best way to optimize the entire process was to convert the pixels to primitives like IntArray/FloatArray and perform the operations on this object, the reassign the processed pixels back to the image object. The get/set operations would be faster than way, and could also allow for single interation of every pixel since the image is now represented as a one-dimensional array.

First, I would like to highlight some approaches that didn't work for me at first. I assigned the pixels to ByteArray, and performed the normal alpha blending operations on the pixels but for some reason, I wasn't able to achieve the desired result. Here's what I did:

    private fun alphaBlend(imageSrc:Mat, blurredImage: Mat, blurredMask: Mat): Mat{

        val total = imageSrc.width() * imageSrc.height()
        val channels = imageSrc.channels()
        val size = total * channels

        val array = ByteArray(size)
        val array1 = ByteArray(size)
        val array2 = ByteArray(size)

        val array3 = ByteArray(size)

        val destination = Mat(imageSrc.size(), imageSrc.type())

        imageSrc.get(0,0, array)
        blurredImage.get(0,0, array1)
        blurredMask.get(0,0, array2)

        for (index in 0 until size){
            val pixel = array[index]
            val blurredPixel = array1[index]
            val blurVal = (array2[index]) / 255.0f

            val newValue = ((blurVal * blurredPixel) + ((1.0f - blurVal) * pixel)).toInt().toByte()
            array3[index] = newValue
        }
        destination.put(0,0, array3)
        return destination
    }

Then I spent days figuring out why this didn't work, and decided to opt in for other valid primitives. Instead of using ByteArray, I opted for FloatArray since the alpha mask value ought to be a decimal value ranging from 0.0 to 1.0. After doing this, I encountered some errors thrown like this:

After a couple of hours spent on debugging, I realized that the data type of the images, ought to match the array primitives they're being assigned to due to certain conditions that have been implemented under the hood by OpenCV. What's even more surprising is that these conditions have been stated here in this post that helped me out with optimization, but overlooked at the time:

So what I did was simple, before assigning the pixels to the FloatArray primitive, I converted the images to the required data types for float arrays, like this:

imageSrc.convertTo(imageSrc, CV_32F)
blurredImage.convertTo(blurredImage, CV_32F)
blurredMask.convertTo(blurredMask, CV_32F)

Well the above approach solved the error, only that this form of conversion gave rise to another error as shown below:

Well as stated in the error message, before using Utils.MatToBitmap(), the mat object needs to be a datatype that is either CV_8UC1, CV_8UC3, or CV_8UC4. So the solution is to convert the destination mat object to either of these by;

destination.convertTo(destination, CV_8UC3)

Here's the full code implementation for the optimized image blur using alpha blending:

    private fun alphaBlend(imageSrc:Mat, blurredImage: Mat, blurredMask: Mat): Mat{

        val total = imageSrc.width() * imageSrc.height()
        val channels = imageSrc.channels()
        val size = total * channels

        val array = FloatArray(size)
        val array1 = FloatArray(size)
        val array2 = FloatArray(size)

        val array3 = FloatArray(size)
        imageSrc.convertTo(imageSrc, CV_32F)
        blurredImage.convertTo(blurredImage, CV_32F)
        blurredMask.convertTo(blurredMask, CV_32F)

        val destination = Mat(imageSrc.size(), imageSrc.type())

        imageSrc.get(0,0, array)
        blurredImage.get(0,0, array1)
        blurredMask.get(0,0, array2)

        for (index in 0 until size){
            val pixel = array[index]
            val blurredPixel = array1[index]
            val blurVal = (array2[index]) / 255.0f

            val newValue = ((blurVal * blurredPixel) + ((1.0f - blurVal) * pixel))
            array3[index] = newValue
        }
        destination.put(0,0, array3)
        destination.convertTo(destination, CV_8UC3)
        return destination
    }

    private suspend fun blurImage(bitmap: Bitmap):Bitmap {
        return withContext(Dispatchers.IO) {
            val imageSrc = Mat()

            Utils.bitmapToMat(bitmap, imageSrc)

            val innerMask = Mat.zeros(imageSrc.size(), imageSrc.type())

            //thickness set to -1 for inner fill
            Imgproc.circle(
                innerMask,
                Point(imageSrc.width() / 2.0, imageSrc.height() / 1.5),
                600,
                Scalar.all(255.0),
                -1
            )

            val blurredImage = Mat(imageSrc.size(), imageSrc.type())
            Imgproc.blur(imageSrc, blurredImage, Size(64.0, 64.0))

            val blurredMask = Mat(innerMask.size(), innerMask.type())
            Imgproc.blur(innerMask, blurredMask, Size(64.0, 64.0))

            val merge = alphaBlend(imageSrc, blurredImage, blurredMask)

            val copy = Bitmap.createBitmap(merge.width(), merge.height(), Bitmap.Config.ARGB_8888)
            Utils.matToBitmap(merge, copy)
            copy
        }
    }

This approach works well, is better optimized, and performs the blur operations WAY faster than the previous implementation since we are performing the bit manipulations using kotlin array primitives on a single iteration, rather than using the direct access method.

Here's a link to the gist. I'm open to different ideas, so if you think there is a more efficient/less lengthy way to achieve this, please let me know!😊

Well after a really long while of tinkering with android development by working on other projects I could do, I became interested again with building a news app. I really wasn't sure of how I was going to do this, but I just kept going at it, trying out multiple ways of solving this problem. I downloaded a couple of news apps from the playstore to observe how these apps work, trying to see if any would "give in" on the secret of how they handled it lol, but all I could notice was that those content were rendered on a RecyclerView.

That was one piece of the puzzle that actually turned out to be really helpful. Knowing the contents were displayed on a recyclerview meant that the content of the news/article are being passed down as a list. Which gave rise to figuring out the solution to the next problem:

How to parse HTML content to a list

We know an HTML document comprises of text tags such as <p>, <a>, <i>, <em>, etc. It also contains container elements that holds group of elements as a single entity such as <div>, <section>, etc. Other popular elements are <img /> for images, <video /> for displaying videos, and lots more. We know what these elements are used for, and the task now is to figure out a way to transform these tags and their respective content to lists. As an example, I would be using the article template used by IGN. The sample article is at this link. After visiting the link, you'll need to right click on any part of the document and left click the "Inspect Element" option to display the HTML structure of the webpage.

We need to understand the arrangement pattern generally used for articles. This way, we can imitate this pattern and figure out a way to grab the data we need, and pass them on to the list.

The solution? Traverse the HTML document

We'll need to pass through every element of the HTML content, and extract the data we need based on the kind it is (text, image, video, headers, etc). Luckily, there's a library that does this really well.

• Open up your Android studio, Create a project, and install the JSOUP dependency adding this one line of code in your app's build.gradle file.

implementation 'org.jsoup:jsoup:1.14.3'

• Create an empty Kotlin file, and name it ArticleParser.kt. This is the class that would be responsible for traversing the DOM and adding the content to an array list.

• Now we need to describe the data types that would be synonymous to the HTML tags. To do this, we'll create a new kotlin file that holds every possible type we'll be needing to represent our tags.

First is the HTML text tags like <p>, <i>, <a>. Since this tags are generally used to hold text, we'll have a global class name for this sort of text tags called Paragraph:

sealed class NewsArticle{
     class Paragraph(val value:String): NewsArticle()
}

Before proceeding, we'll need to understand how the article section of the webpage is structured. Going back to "Inspect Elements", we'll observe that the body/content of the article is within a parent <section> tag, having one of its classnames as "article-content". This is would be the starting point of our traversal.

An important thing to note about JSoup types is that Node refers to any element or text content available on the HTML document. Element refers to only HTML element tags, not the text content. Element extends Node. And finallly we have TextNode which also extends Node, but refers to only the text content of an HTML file.

In our NewsArticle class, we'll be creating a function that accepts a Node variable, and we traverse the content of this Node.

suspend fun traverseElement(element: Node): List<NewsArticle>{
    return withContent(Dispatchers.IO){
           element.traverse(object: NodeVisitor){
                 override fun head(node: Node, depth: Int){

              }
                override fun tail(node: Node, depth: Int){

             }
    })
   }
}

The head() method of the NodeVisitor interface is invoked when we encounter the start of an HTML element/text, and passes the corresponding node, and depth. The tail() method is invoked when we encounter the end of an HTML element, and this method also passes the corresponding node and depth during traversing the DOM. depth refers to the nested level of any of the nodes encountered during traversal. This means the start of any node encountered during traversal has a depth of 0. If this node contains sibling nodes, the sibling nodes would have a depth of 1, 2, 3, ...n.

It'll be ideal to handle text nodes, before any other forms of nodes. Mostly beccause, we need to figure out a way to concatenate texts as a single entity, regardless of any form they might come in such as Regular texts, Bold texts, Italic, or Link texts. Here's a sample of what I mean:

If you observe the text above, you'll notice it comprises of regular, bold, and italics. We wouldn't want these various element break down into different blocks while traversing through all the elements and adding them to a list. So to achieve this, we'll create a nullable NewsArticle object that holds the current node during traversal, and we'll check to see if it is a type of NewsArticle.Paragraph. If it is, we concatenate the next text with the previous text. If it isn't, we'll add the exisiting NewsArticle object into the list, and create a new NewsArticle.Paragraph object.

var articlePiece: NewsArticle? = null

    private fun resolveArticleParagraph(word: String): NewsArticle {
        val temp = word.replace("\\\\s+", "")
        return if (temp.isNotEmpty()) {
            return if (articlePiece != null && articlePiece is NewsArticle.Paragraph) {
                NewsArticle.Paragraph((articlePiece as NewsArticle.Paragraph).value.plus(word))
            } else {
                NewsArticle.Paragraph(word)
            }
        } else {
            NewsArticle.Paragraph("")
        }
    }


suspend function traverseElement(element: Node): List<NewsArticle>{
    return withContent(Dispatchers.IO){
           element.traverse(object: NodeVisitor{
                 override fun head(node: Node, depth: Int){
                     if (node is TextNode) {
                        val word = node.text()
                        articlePiece = resolveArticleParagraph(word)
                    }
              }
                override fun tail(node: Node, depth: Int){

             }
    })
   }
}

This piece of code above only transforms all of the text as regular text. Now we'll need to do something that would show a distincition between regular texts, link texts, and any other form of text.

We'll write an extension function for the Element node, to determine if the type of tag encountered during traversal is a text element.

open class RegularText(val text:String)

class ItalicText(text: String):RegularText(text)
class LinkText(text: String):RegularText(text)
class BoldText(text: String):RegularText(text)

fun Element.isText(): Boolean {
    val tag = this.tagName()
    return tag == "a" || tag == "em" || tag == "i" || tag == "p" || tag == "strong"
}


fun Element.getTextType(): RegularText {
    val tag = this.tagName()
    val text = this.text()

    return when (tag) {
        "a" -> LinkText(text)
        "i", "em" -> ItalicText(text)
        "strong" -> BoldText(text)
        else -> RegularText(text)
    }
}

Now back to our traverseElement() method, we'll modify the code to be aware of various forms of text tags we need to look out for:

    private fun openArticleLink(): NewsArticle {
        return if (articlePiece != null) {
            if (articlePiece is NewsArticle.Paragraph) {
                NewsArticle.Paragraph((articlePiece as NewsArticle.Paragraph).value.plus("["))
            } else {
                //when a paragraph starts with a link sentence
                addToArticle()
                NewsArticle.Paragraph("[")
            }
        } else {
            NewsArticle.Paragraph("[")
        }
    }


   private fun openArticleItalic(): NewsArticle {
        return if (articlePiece != null) {
            if (articlePiece is NewsArticle.Paragraph) {
                NewsArticle.Paragraph((articlePiece as NewsArticle.Paragraph).value.plus("_"))
            } else {
                //when a paragraph starts with an italic sentence
                addToArticle()
                NewsArticle.Paragraph("_")
            }
        } else {
            NewsArticle.Paragraph("_")
        }
    }

    private fun openArticleBold(): NewsArticle {
        return if (articlePiece != null) {
            if (articlePiece is NewsArticle.Paragraph) {
                NewsArticle.Paragraph((articlePiece as NewsArticle.Paragraph).value.plus("*"))
            } else {
                //when a paragraph starts with an italic sentence
                addToArticle()
                NewsArticle.Paragraph("*")
            }
        } else {
            NewsArticle.Paragraph("*")
        }
    }

    private fun closeArticleLink(link: String): NewsArticle {
        return if (articlePiece != null && articlePiece is NewsArticle.Paragraph) {
            NewsArticle.Paragraph((articlePiece as NewsArticle.Paragraph).value.plus("]".plus("(${link})")))
        } else {
            NewsArticle.Paragraph("")
        }
    }

    private fun closeArticleItalic(): NewsArticle {
        return if (articlePiece != null && articlePiece is NewsArticle.Paragraph) {
            NewsArticle.Paragraph((articlePiece as NewsArticle.Paragraph).value.plus("_"))
        } else {
            NewsArticle.Paragraph("")
        }
    }

    private fun closeArticleBold(): NewsArticle {
        return if (articlePiece != null && articlePiece is NewsArticle.Paragraph) {
            NewsArticle.Paragraph((articlePiece as NewsArticle.Paragraph).value.plus("*"))
        } else {
            NewsArticle.Paragraph("")
        }
    }

suspend function traverseElement(element: Node): List<NewsArticle>{
    return withContent(Dispatchers.IO){
           element.traverse(object: NodeVisitor{
                 override fun head(node: Node, depth: Int){
                     if (node is TextNode) {
                        val word = node.text()
                        articlePiece = resolveArticleParagraph(word)
                    } else if (node is Element) {
                        if (node.isText()) {
                            if (node.getTextType() is LinkText) {
                                articlePiece = openArticleLink()
                            } else if (node.getTextType() is ItalicText) {
                                articlePiece = openArticleItalic()
                            } else if (node.getTextType() is BoldText) {
                                articlePiece = openArticleBold()
                            }
                        }
              }
                override fun tail(node: Node, depth: Int){
                     if (node.isText()) {
                                if (node.getTextType() is LinkText) {
                                    val link = node.attr("href")
                                    articlePiece = closeArticleLink(link)
                                } else if (node.getTextType() is ItalicText) {
                                    articlePiece = closeArticleItalic()
                                } else if (node.getTextType() is BoldText) {
                                    articlePiece = closeArticleBold()
                                }
                            }
             }
    })
   }
}

Here's an explanation of what is happening above: When we encounter a LinkText, instead of passing off the whole text to the list as a raw text, we'll do a bit of markdown syntax such that the text ends up being like [some text here](https://link.com) or _text should be in italics_

Now how do we know when we are done with a block so it can be added as an item to the list. The idea was to delegete breaks at the tail(), such that when we get to the closing tag of an element at a particlar depth level, the current NewsArticle object is added to the list and reinitialized to null again for the next traversal.

The tail() method ends up being with the following snippet:

if (depth == articleParserConfig.breakDepth) {
      addToArticle()
      addSpacer()
}

Handling the text nodes is where the majority of the work is. Finally, you'll be needing Regex and android's SpannableStringBuilder to properly format and style the text based on the kinds they are. With this idea, considerations for other Container Elements such as images, videos, gallery views can easily be implemented. The full source code for the sample project can be found here

a. In the database, here's a User table that contains columns like id(primary key) name, email, profileUrl, etc

class User extends Model { }

User.init(
  {
    name: {
      type: DataTypes.STRING,
      allowNull: false,
    },
    bio: {
      type: DataTypes.STRING,
      allowNull: false,
      defaultValue: "Hey there!"
    },
    id: {
      type: DataTypes.UUID,
      allowNull: false,
      primaryKey: true,
      defaultValue: DataTypes.UUIDV4,
      unique: true,
    },
    username: {
      type: DataTypes.TEXT,
      allowNull: false,
      unique: true,
    },
    profile_url: {
      type: DataTypes.STRING,
      allowNull: false,
      defaultValue:
        "https://someimagesite.com/default_image.jpg",
    },
    email: {
      type: DataTypes.TEXT,
      allowNull: false,
      unique: true,
    }
  },
  {
    sequelize: instance,
    modelName: "users",
    tableName: "users",
    timestamps: true,
    createdAt: true,
    updatedAt: true,
  }
);


User.hasMany(Post, {
  foreignKey: "userId",
  foreignKeyConstraint: true,
  constraints: true,
  onUpdate: "CASCADE",
  onDelete: "CASCADE",
});


User.hasMany(Comment, {
  foreignKey: "userId",
  foreignKeyConstraint: true,
  constraints: true,
  onUpdate: "CASCADE",
  onDelete: "CASCADE",
});

Post.belongsTo(User)
Comment.belongsTo(User)

b. Here's a Post table that contains columns like id(primary key), content, title, userId, timestamp. "userId" references a row in the User table and it represents the author of the posts. It has a foreign key constraints set as "onUpdate: CASCASDE" & "onDelete: CASCADE" on the User table which ensures that when a user row is deleted/updated from the users table, any other references to the row in any table gets updated as well.

class Post extends Model { }

Post.init(
  {
    content: {
      type: DataTypes.TEXT,
      allowNull: false,
    },
    id: {
      type: DataTypes.UUID,
      allowNull: false,
      primaryKey: true,
      defaultValue: UUIDV4,
      unique: true,
    },
    userId: {
      type: DataTypes.UUID,
      allowNull: false,
    },
  },
  {
    sequelize: instance,
    modelName: "posts",
    tableName: "posts",
    timestamps: true,
    createdAt: true,
    updatedAt: true,
  }
);

Now, imagine we need to to display a notification to certain users in the following typical format: "@echoeyecodes just made a post"

In the sample format above, we can see a couple of references to certain attributes of 2 tables in the database.

• username of user (User table)
• post thumbnail (Post table)
• description (Post table)
• profile photo of user (User Table)
• timestamp (Post table)

To model this as a table in the database for notifications, we'll need the following columns:

• postId
• userId
• targetUserId (recipient of the notification)

class Notification extends Model {}

Notification.init(
  {
    id: {
      type: DataTypes.UUID,
      allowNull: false,
      primaryKey: true,
      defaultValue: UUIDV4
    },
    postId: {
      type: DataTypes.UUID,
      allowNull: false,
    },
    userId: {
      type: DataTypes.UUID,
      allowNull: false,
    },
    targetUserId: {
      type: DataTypes.UUID,
      allowNull: false,
    },
    isRead: {
        type: DataTypes.BOOLEAN,
        allowNull: false,
        defaultValue: false
    },
  },
  {
    sequelize: instance,
    modelName: "notifications",
    tableName: "notifications",
    timestamps: true,
    createdAt: true,
    updatedAt: true,
  }
);

Now to ensure our data here is consistent with the tables it references and avoid null data, we'll add foreign key constraints --> onUpdate: CASCASDE & onDelete: CASCADE to the userId, and postId columns to User and Post table

With this model, we can query for notifications on posts created by a certain user with no issues at all. But the 'gotcha!' in this is that it only works well for post notifications. What about when we need notifications for the following events:

• a. when a user mentions another user in the content of a post?
• b. when a user publishes a comment on someone else's post?
• c. when a user mentions another user in a comment/reply
• d. when a user likes a post
• e. when a user likes a comment/reply

If we analyze these events, you'll notice that each event references certain columns in different tables beyond Post and User. We could go ahead and add more attributes like "commentId", "replyId" to our notification model to adjust to the requirements for notifications, but that would make our model contain redundant columns, and make it harder to debug or understand. We would also have more than a few null columns for a row that actually only references at most 2 tables.

How do we solve this problem?

I looked up for a conventional approach for storing notifications in databases, and found some that didn't exactly solve this specific issue, but gave me insights on how to resolve this. I didn't want to create separate tables for different kinds of notifications, especially when these notifications are expected to be arranged in a chronological order. Well, thanks to this stackoverflow answer, I figured that I could have a separate table for notification event-types, e.g (post-likes, comment-likes, post-mentions, comment-mentions etc) and a notification table that holds only generalized attributes for every form of notification. This notification will contain attributes like timestamp, eventId, recepientId since all forms of notification are expected to have these attributes. That was the first step towards the solution. So now we have an events table like:

class NotificationEvent extends Model {}

NotificationEvent.init(
  {
    id: {
      type: DataTypes.INTEGER,
      allowNull: false,
      primaryKey: true
    },
    type: {
      type: DataTypes.STRING,
      allowNull: false,
    }
  },
  {
    sequelize: instance,
    modelName: "notification_events",
    tableName: "notification_events",
    timestamps: true,
    createdAt: true,
    updatedAt: true,
  }
);


NotificationEvent.hasMany(Notification, {
    foreignKey: "eventId",
    foreignKeyConstraint: true,
    constraints: true,
    onUpdate: "CASCADE",
    onDelete: "CASCADE",
})

Notification.belongsTo(NotificationEvent, {
  foreignKey: "eventId",
  onDelete: "CASCADE",
  onUpdate: "CASCADE",
  foreignKeyConstraint: true,
  constraints: true
})

Then our revamped notification model would look like:

class Notification extends Model {}

Notification.init(
  {
    id: {
      type: DataTypes.UUID,
      allowNull: false,
      primaryKey: true,
      defaultValue: UUIDV4
    },
    eventId: {
        type: DataTypes.INTEGER,
        allowNull: false
    },
    targetId: {
      type: DataTypes.UUID,
      allowNull: false,
    },
    isRead: {
        type: DataTypes.BOOLEAN,
        allowNull: false,
        defaultValue: false
    },
  },
  {
    sequelize: instance,
    modelName: "notifications",
    tableName: "notifications",
    timestamps: true,
    createdAt: true,
    updatedAt: true,
  }
);

Going back to notification format for various events I listed above, I observed a pattern for each event:

1. (a) is a form of notification between a User and a Post table. That is pretty straightforward, so we can have a UserPostNotification table that contains the columns "userId" and "postId".

2. (b) is a form of notification between a Comment, and a Post table. So the main attributes needed here would be the commentId, and the postId for the PostCommentNotification table. With these columns we can easily construct our notification message by fetching the related data they reference. You know that a comment would always be from a User, therefore it would have a column called "userId" so we could fetch the user information based on this column to get data like username, and profileUrl. Also, every post has a "userId" attribute that references a certain user, so we could also get the user that owns the post based on this attribute and use it as the recipient of the notification.

We can now have something like @username commented on your post

1. (c) is a form of notification between a User and a Comment table. We can construct our notification message like the image below using only the attributes "userId" and "commentId".

2. (d) is also a form of notification between a User and a Post table. To get a message like the image below, we can obtain the information using the userId and postId attributes.

3. (e) is a form of notification between a "userId" and commentId and the message like @echoeyecodes liked your comment could be constructed using these two attributes.

After observing these patterns, I realized I could have a single table for notifications that are between a User & Post, User & Comment, or User & Reply. These tables would have an id attribute as primary key that references a row in the notifications table as a One-To-One Relationship. So to query all notifications in chronological order with the various attributes from different notification types, we could use an INNER JOIN for the Notification table and NotificationEvents Table, plus a LEFT JOIN with the UserPostNotification table, and PostCommentNotification table.

We could also insert an INNER JOIN clause to each of the sub-tables based on the parent tables they share constraints with, without having to re-query for these pieces of information after obtaining the foreignKey attributes since UserPostNotification would have foreign key constraints between a User and Post (userId, postId), PostCommentNotification would have a foreign key constraints between a Post and Comment userId, commentId).

here's how the UserPostNotification would look like:

class UserPostNotification extends Model {}

UserPostNotification.init(
  {
    notificationId: {
      type: DataTypes.UUID,
      allowNull: false,
      primaryKey: true,
    },
    sourceId: {
      type: DataTypes.UUID,
      allowNull: false
  },
    referenceId: {
        type: DataTypes.UUID,
        allowNull: false
    },
  },
  {
    sequelize: instance,
    modelName: "user_post_notifications",
    tableName: "user_post_notifications",
    timestamps: true,
    createdAt: true,
    updatedAt: true,
  }
);

then the PostCommentNotification:

class PostCommentNotification extends Model {}

PostCommentNotification.init(
  {
    notificationId: {
      type: DataTypes.UUID,
      allowNull: false,
      primaryKey: true,
    },
    sourceId: {
      type: DataTypes.UUID,
      allowNull: false
  },
    referenceId: {
        type: DataTypes.UUID,
        allowNull: false
    },
  },
  {
    sequelize: instance,
    modelName: "post_comment_notifications",
    tableName: "post_comment_notifications",
    timestamps: true,
    createdAt: true,
    updatedAt: true,
  }
);

Now to query for all notifications available for a particular user, we would query via the notification table and use a left join for various notification types available like:

const NOTIFICATION_INCLUDE_OPTIONS : IncludeOptions[] = [{model: User, required: true},
    {model: NotificationEvent, required: true},
    {model: UserPostNotification, include: [{model: User}]},{model: PostCommentNotification, include: [{model: User}, {model: Comment, include: [{model: User}]}]}]


async function getUserNotifications(userId:string(){
   const result = await Promise.all((await Notification.findAll({include: NOTIFICATION_INCLUDE_OPTIONS, where:{targetId: userId}}))
        .filter((item) => {
//here we filter off notifications with possible all notification type as null. this kind of state cannot possibly exist since every notification object would be of at least one type, but just to be safe, we need this check
            const result = item.get({plain: true}) as NotificationType
            if(!result.user_post_notification && !result.post_comment_notification){
                return false
            }
            return true
        })
        .map((item) => formatNotification(item.get({plain: true}))))
        return result
}

Finally, we can write a small utility function that constructs the notification message based on the event type for every notification like the one below:

async function formatNotification(instance:NotificationType){
    const type = instance.notification_event.type as NotificationEventType

   if(type == "post_comment"){
     //format notification based on requirements e.g @echoeyecodes commented on your post
   }else if(type == "post_create"){
     //format notification based on requirements e.g @echoeyecodes uploaded a photo
}
}

class Comment extends Model { }

Comment.init(
  {
    userId: {
      type: DataTypes.UUID,
      allowNull: false,
    },
    id: {
      type: DataTypes.UUID,
      allowNull: false,
      primaryKey: true,
      unique: true,
    },
    comment: {
      type: DataTypes.TEXT,
      allowNull: false,
    },
    postId: {
      type: DataTypes.UUID,
      allowNull: false,
    },
  },
  {
    sequelize: instance,
    modelName: "comments",
    tableName: "comments",
    timestamps: true,
    createdAt: true,
    updatedAt: true,
  }
);

and now the Reply model:

class Reply extends Model {}

Reply.init(
  {
    userId: {
      type: DataTypes.UUID,
      allowNull: false,
    },
    id: {
      type: DataTypes.UUID,
      allowNull: false,
      primaryKey: true,
      unique: true,
    },
    reply: {
      type: DataTypes.TEXT,
      allowNull: false,
    },
    commentId: {
        type: DataTypes.UUID,
        allowNull: false,
      },
  },
  {
    sequelize: instance,
    modelName: "replies",
    tableName: "replies",
    timestamps: true,
    createdAt: true,
    updatedAt: true,
  }
);

What's even more wild is that I had two controllers that performed the same read/write operations for both tables and really wasn't comfortable with how this was designed. Besides the regular controllers for the table, there was a CommentLike model that stores comments liked by users using the commentId and userId column, however this table was only meant for the Comment table. Replies could not be liked, or even replied to because of the rigid design nature I adopted for it months back🤮. The final bit of crazy was where I had two notification models for both tables, CommentNotification and ReplyNotification, each referencing rows in both Comment and Reply table. Yes, this also means writing different controller classes that literally did the same thing, besides a few columns that were distinct🤦

I finally decided to rethink this design approach, and I think what really helped me out here was the way tweets work on twitter. I spent some time using the twitter api for evaluating tweet acitivities, and noticed how tweet response are usually structured. Essentially every comment, or quote made on twitter is a tweet, which means they all have the same common attributes (columns) that describes them. What could probably distinguish a tweet from another is if one tweet is a standalone tweet, or a reply to another tweet, but generally they are all a single entity - A Tweet

Having this in mind, I realized that every reply to a comment is still essentially a comment, so I could actually store all replies within the comments table. But how would I be able to distinguish a standalone comment from a reply to a comment, since querying for all comments related to a particular post would include the standalone comments and replies, which isn't what we want? For this, I created a junction table called CommentReply with two columns commentId, and referenceId like this:

class CommentReply extends Model { }
CommentReply.init(
    {
        commentId: {
            type: DataTypes.UUID,
            allowNull: false,
            primaryKey: true,
            unique: true,
        },
        referenceId: {
            type: DataTypes.UUID,
            allowNull: true,
        },
    },
    {
        sequelize: instance,
        modelName: "comment_replies",
        tableName: "comment_replies",
        timestamps: true,
        createdAt: true,
        updatedAt: true,
    }
);

The commentId column references the comment that is supposed to be a reply, while the referenceId column references the comment being replied to. So, whenever a reply is made to a certain comment, the reply is stored in the database as a comment, and the id of this comment (reply) along, with the id of the comment that's being replied to is stored in a CommentReply.

Now how do I query for comments that are originally standalone comments, not replies. The idea is to query for every comment within the database, and perform a left join with the CommentReply table. A "left join" would include null columns in the query results for comments that do not have references in the commentId column of the CommentReply table. So rows with null commentId column represents original standalone comments, while the non-null columnID rows are replies. So based on this ideology, we can query for standalone comments, or just comments that are replies to a certain comment Here's how I performed the query in the comment controller class:

  async getComments({ postId, limit, offset, userId, commentId }: GetShotParams & { shotId: string, commentId: string | null }) {

    /* if query contains commentId, fetch all comments referenced by that id
    else, fetch comments that do not have a row in the replies. Note that: Standalone
    comment ids can never have a row in the columnId for commentReplies, but references so this works!
    */

    const where: WhereOptions = commentId ? { shotId, '$comment_reply."referenceId"$': commentId } : { postId, '$comment_reply."commentId"$': null }

    const comments = (await Comment.findAll({ where, order: [['createdAt', "DESC"]], limit, offset, include: COMMENT_INCLUDE_OPTIONS }))
    return comments
  }

What makes this even better is that replies also get to have likes, or even nested replies just like "original comments" which wasn't available in the first design. This recursive sequence just keeps going on and on without adding an extra line of code to the controller, making it more flexible than the initial approach.

After implementing this, I was able to migrate every reply from the Reply table to the Comment table, and severe all ties to it😁 Now, I get to delete the reply table, its controllers, and including the notification model that is associated with this table. Damnn that's really a lot of code😵 I think the best part of refactoring code is where you get to delete a number of lines of code that really are just garbage😅

I initially tried using a workaround with the GridLayoutManager android provides, but that didn't work as expected because this layout wouldn't allow feed items to fill up extra spaces that might be available during positioning.

Turns out the answer was in the FlexboxLayoutManager - A layout manager based off of CSS' flex box. I've used this quite a number of times for basic layout, especially the "wrap" feature, but never had a use case for something specific like DeviantArt's. I set out to implement this layout manager, but of course it didn't come off as easy as I thought it would. The problem I had at first was that I couldn't control the maximum number of columns that should be displayed on a single row before wrapping, so all hundreds of images were arranged on a single row. Crazy right?😅

Anyway I did a bit of research here and there for how to achieve this form of layout in CSS Flexbox since the layout manager is based on this. A few minutes later, I came across this post on stackoverflow about setting the min-width and max-width of each flex item, then allowing the default behavior of these flex items to fill up remaining space with the flex-grow property set to 1. I applied this concept on the FlexboxLayoutManager in android, and I'm glad I was able to finally achieve the same results with DeviantArt😄

Here's how I implemented it on android; since the sizes of the various images to be rendered is based on the aspect ratio, I would return 3 different view-types based on this faactor. The viewTypes are PORTRAIT, LANDSCAPE, and SQUARE. So in the getItemViewType() method of the RecyclerView adapter, this check is performed to determine the view type to be used as here:

override fun getItemViewType(position: Int): Int {
        val item = getItem(position)
        return if (item.width > item.height) {
            LANDSCAPE
        } else if (item.width < item.height) {
            PORTRAIT
        } else {
            SQUARE
        }
    }

Now that this is available, the minimum and maximum possible width and height would be determined in the onCreateViewHolder() and these values would be different based on the view types:

    override fun onCreateViewHolder(parent: ViewGroup, viewType: Int): ShotFlexAdapterViewHolder {
val screenWidth = getScreenSize().width

        val view = LayoutInflater.from(parent.context)
            .inflate(R.layout.layout_shot_list_item, parent, false)

        if (viewType == PORTRAIT) {
            val maximumWidth = (0.5 * screenWidth).toInt()
            val minimumWidth = (0.3 * screenWidth).toInt()
            val minimumHeight = (0.5 * screenWidth).toInt()
            val maximumHeight = (0.6 * screenWidth).toInt()

            view.updateViewLayoutParams(minimumWidth, maximumWidth, minimumHeight, maximumHeight)
        } else if (viewType == LANDSCAPE) {
            val maximumWidth = screenWidth.toInt()
            val minimumWidth = (0.55 * screenWidth).toInt()
            val minimumHeight = (0.5 * screenWidth).toInt()
            val maximumHeight = (0.55 * screenWidth).toInt()

            view.updateViewLayoutParams(minimumWidth, maximumWidth, minimumHeight, maximumHeight)
        } else {
            val maximumWidth = (0.7 * screenWidth).toInt()
            val minimumWidth = (0.4 * screenWidth).toInt()
            val minimumHeight = (0.4 * screenWidth).toInt()
            val maximumHeight = (0.7 * screenWidth).toInt()

            view.updateViewLayoutParams(minimumWidth, maximumWidth, minimumHeight, maximumHeight)
        }

        return ShotFlexAdapterViewHolder(view)
    }

Now the extension function that updates the width and height properties:

private fun View.updateViewLayoutParams(
        minimumWidth: Int,
        maximumWidth: Int,
        minimumHeight: Int,
        maximumHeight: Int
    ) {
        (this.layoutParams as FlexboxLayoutManager.LayoutParams).apply {
            flexGrow = 1f
            this.width = minimumWidth
            this.height = minimumHeight
            this.maxWidth = maximumWidth
            this.minWidth = minimumWidth
            this.minHeight = minimumHeight
            this.maxHeight = maximumHeight
        }
    }

Here's an explanation of what is going on above. In order not to overcrop the images that would be displayed, the container width and height holding this image should match the view types i.e portrait images should have a container height greater than the container width, landscape images should have a container width greater than the container height, and square images should have container width and height of equal sizes so that the images to be displayed fits almost* perfectly with the container, and not overcropped.

Now on top the onCreateViewHolder(). For portrait images, the container is assigned a minimum width 30% the screen width, complemented with a minimum height of half of the screen width, which is slightly higher than the minimum width. This would be a particularly useful for scenarios that allows a maximum of 3 portrait images to be rendered on a single row before wrapping. Now imagine we've got a portrait image and a landscape image competing for space on a row. Obviously the landscape image would need more width size than the portrait (say 50% of the screen width), and that means lesser spaces would be left for other images.

For this to work in this case, we can only have a maximum of 1 landscape photo, and the remaining 50% of the screen width would be left to be shared amongst other images. Remember we initially, we defined the minimum width of portrait images to be 30% of the screen width, which means only a maximum of 1 photo can go along with a landscape photo on the same row. The remaining 20% would be shared between these two images since we set the 'flex-grow' property of every item to be 1, i.e stretch to fill the remaining space.

This analogy would also be applied in cases for landscape and square images. The key lies within using the min/max width and height properties to properly configure how much space each item can be allowed to have on a single row based on their aspect ratio, before wrapping subsequent images to the next row. The min width determines the smallest possible width an item can have on a row, while the maximum width determines the maximum possible width an item should have when stretching, regardless of how much space is left on the row. I hope you find this explanation useful. I'm still recovering from the 'eureka' moment I had after I discovered this layout approach😅 and here's what it looks like on the app

int scores[] = {0,1,2,3,4,5,2,1,5,6};

Here, scores is an array object holding 10 integer objects. In terms of memory, there are a couple of other things to know about this array. Typically, an int has a size of 4 bytes within the memory. scores is an array of 10 integers. This means that the score object has a total size of 4 * no_of_items in container -> (4 * 10 ) = 40 bytes. In memory, this scores object would occupy a space of 40 bytes. An array of 2 integers would have a memory size of 4bytes * 2 = 8 bytes.

When you need to access one of the objects within the array, we can use a subscript operator “[]” to achieve this. The subscript operator takes in an integer, which represents the position of the object within the array. The initial position of any element within the array has a position of 0, meaning to access the 4th element in an array, the position would be 3. An example:

int scores = {15,33,69,87,125};
//to access the 125, we would use the position at where 125 lies in the array
int value = scores[4]; // value is now 125

But what happens when we refer to an array position that is beyond the size of this array, say 7? We would expect an error to occur, or the program to crash (ArrayIndexOfOutBoundsException). The truth is this might not happen. Imagine we tried printing out the element of the scores array at position 7(Remember that this position does not exist since score has a size of 6), it is very likely we would get a value returned, but this value actually refers to a different kind of thing within the memory. An example:

int list[] = {10,24,56};
cout << list[4] <<endl; //print the 5th item in the array. (-45752158 was my result on my computer)

The fifth item here does not exist, but we could get a random value of something else. We can understand why this happens if we look at this at a memory point of view. Recall that an int has a size of 4bytes, and an array of ints would have a size equal to the size of each items(4 bytes) multiplied by the total number of items in the array. These items in the array occupy addresses in memory next to each other.

So basically, the array object spans from the first container element address, to the last container element address in memory. There could be an object elsewhere within the program that occupies the memory address just immediately after the end of the array, so when we use a subscript value greater than the size of an array to get an element, it is likely we get a reference to another object in memory that we totally have no idea about.

Arrays and expressions.

Consider an example where we have an array of characters,

char values[] = {‘A’, ‘B’, ‘C’};

Any time we try to use the values object to do some form of operation, rather than getting the object itself, we would get a pointer to the first element within the array. So, values here is basically a pointer to ‘A’, not the entire array. It wouldn’t make sense when we pass entire arrays as values or parameters in a program, because these kinds of objects usually hold a list of other objects, sometimes this list can be fairly large and using the list as a whole anywhere in our program could lead to a huge amount of memory usage. With this idea, now would be a good time to explain what really happens when we subscript an array.

char values[] = {‘A’, ‘B’, ‘C’};
char oneValue = values[1]; //oneValue now holds ‘B’

Remember, using the values object is usually a pointer to the first element within an array, so here’s a breakdown of what happens with this operation.

char* temp = values + 1; // values is a pointer, adding one advances the pointer by one space. temp is now a pointer to the next value of the array.
char oneValue = *temp; //dereferencing the pointer yields the value in that array

Essentially, subscripting an array usually means advancing the pointer of the first element in the array by a number of ‘n’ spaces, which is later dereferenced and yields the value the pointer points to.

This same thing happens when we pass an array as a parameter to a function. Suppose we have a function that adds two numbers x, y together. We would write this as:

void addNumbers(int x, int y){
    cout << (x+y) <<endl;
}

int value1 = 4;
int value2 = 7;
addNumbers(value1,value2); //calling add numbers passes copies of x and y into the function

But with functions that accept arrays as a parameters, thing work a little different from usual. Imagine we need this function in our program to return an item at a particular index in an array;

int doStuff(int someValues[], int index){
    return someValues[index];
}

This would still work, but not in the way we expect because, someValues won’t be an array. Rather, it would be a pointer to the first element of the array passed to it. Then subscripting the pointer with an index would just advance the pointer by a number of spaces based on the value of the index. Which means we could just rewrite our function as;

int doStuff(int* someValues, int index){
    return someValues[index];
}

This approach is more efficient because rather than copying the whole array into another function that executes on top of the stack, we just get something that points to the first element, and we could advance the pointer by the desired amount of space needed.

If we need a function that loops through an array and prints all the values in it, we can just pass the pointer of the first element to the array, along with the size of the array to run the loop efficiently. Since there isn’t a member function for getting the size of an array, we can do this through diving the total memory size of the array by the memory suize of one element in that array. We can get the memory size of an object by calling the sizeof() function on that object. An example would be:

// function that prints all items in the array
int doStuff(int* someValues, int size){
    for(int i = 0; i<size; ++i){
    cout << someValues[i] << endl;
}
}

int scores[] = {0,2,3,5,5}; //an array of ints, each int has a size of 4 bytes
int sMemorySize = sizeof(scores); // size is 20 bytes
int pSize = sizeof(scores[0]); // we get the size of one element within the array, which yields 4 bytes
int size = sMemorySize/pSize; // this would be (20/5) which yields 5
doStuff(scores, size);

Interesting stuff this week:

1. I couldn’t help it, so I had to go check up on what to expect from Unreal Engine. I took a brief look at some of the code syntax with C++ on the engine, because I learned that the C++ paradigm with Unreal is a bit different from vanilla C++. One of the differences being that multidimensional arrays can’t be used on the engine. I’m guessing we would use other container elements, like vectors that can contain other container elements, but I wouldn’t be surprised if I get hit with a caveat that says, “This isn’t considered as a best practice when working with the engine” lol.

2. Speaking of best practices, I worked on a couple of utility functions with C++ (map, flat, reduce, etc). I would have created them as member functions on the vector class when extended, but I faced a couple of issues with using the list initialization methods on them. While looking up solutions for this, I realized that extending the vector class is considered a “bad practice” for reasons I’m yet to understand lol. Also, I didn’t quite get how to make the subclass inherit the list initialization from the vector object.

3. Finally, bitwise operators! I got a basic idea on how powerful these operators can be, especially when working with colors. It’s funny how I’ve used these operators on other non-C projects (such as with Android flags), but had no idea that I was using them lol. I haven’t fully understood bitwise operators, but I’m getting there. And of course, a blog post about it :D. Meanwhile, I practiced the basic concepts of these operators by writing a HEX-to-RGB converter. Then I wrote a RGB-to-HEX converter as spice to keep it at 💯.

int score = 100;

Now to bind this variable to a reference, we just need to prefix the reference name with the reference declarator “&”. i.e

int &scoreRef = score;

This simple assignment operation is different from a regular assignment operation because, this doesn’t copy the value of score to scoreRef. Instead, it is binding the score object to scoreRef. Any arithmetic operations that can be performed on score can be equally done with scoreRef. If we try to fetch the value of scoreRef, we are actually obtaining the value of the score object itself. It is also important to note that if the operations performed on the scoreRef changes its value, the original object (score) which it is bound to equally changes as well. An example!

int lives = 55;
int someTotal = lives;
someTotal = someTotal + 17;
cout <<someTotal <<endl; //expected value here is 72
cout <<lives <<endl; //expected value here is 55

However, using the same example with reference:

int lives = 55;
int &someTotal = lives;
someTotal = someTotal + 17;
cout << someTotal <<endl; //expected value here is 72
cout << lives <<endl; // value here is also 72

When the lives variable was bound to someTotal, we added 17 to the initial value. This addition changed the value of someTotal to 72. Since someTotal is bound to lives, it changes the value of lives as well. Another way to look at it is, someTotal is the same as lives, only difference is that we have another name for it.

Important things to know about references:

1. References must be initialized immediately.

2. The data type of a reference must be the same as the data type of the object it is bound to. e.g: double points = 3.01245; int &pointsRef = points; //error! points should be an integer

3. A reference must be bound to an object. This means that we can’t bind a reference to a literal. An example, we can’t do stuff like this;

int &scoreRef = 100; // error! cos 100 is a literal not an object.

Pointers

Pointers! Well, pointers are like references being that they also refer to an object… difference is that they refer to the memory address of the object they point to. Also, when a pointer is initialized to point to a particular object, they can be reassigned to point to another address of an object at any time throughout the lifecycle of the program.

Defining a pointer can be done by prefixing the pointer name with an asterisk, and since pointers point to memory addresses, we can obtain the memory address of an object by prefixing the object it points to with a “&”. For example:

int score = 100;
int *score = &score;

If we try to print the value of scoreRef, the result would be a series of strange alphanumeric characters, which is actually the address the object holds in memory. Right now, printing scoreRef yields a result of “003FFEA0”.

A pointer does not need to be initialized at the time it is defined. If just defined, the pointer is known to be a null-pointer. There are also several ways can intentionally define a null pointer as:

int *somePointer = 0;
int *somePointer1 = NULL;
int *somePointer2 = nullptr;

Quick note: In example a above, it would be dangerous to assume that literals such as 5,6,7,8… can be assigned to a pointer. This assumption is wrong, and such assignment would fail since a pointer should refer to an object. Only 0 works because, assigning 0 to a pointer is just a way of defining the pointer as a null pointer. It is also illegal to assign an int variable to a pointer, even if the variable is 0.

void pointers can hold the address of any object of any datatype. The type of the object at the address would be regarded as unknown. This however limits the kind of operations that can be performed on such pointer. You can only pass or return a void pointer, but you cannot use a void pointer to operate on the object it addresses. You can declare a void pointer as:

int score = 45;
void *somePointer = &score;

Doing arithmetic operations like this below won't work since the compiler literally has no idea of the type of the object and if the operation would be valid.

int score = 45;
void *somePointer = &score;

int x = *somePointer; //error! Dereferencing somePointer doesn't yield an int

Dereferencing somePointer only yields the object they point to, but we can’t perform mutable operations on them such as adding/subtracting a value from it since the compiler can’t know if the object is an integer and accepts arithmetic operations, or if the object is a string that could be concatenated, or just something else.

Important things to know about pointers:

• If a pointer points to an object, the object it points to can be accessed by dereferencing the pointer. Dereferencing a pointer yields the object to which it points to. We dereference a pointer by prefixing the pointer object with an asterisk. An example:

int score = 70;
int *scorePointer = &score;

cout << scorePointer << endl; //prints out the address like “783FUEI0”
cout << *scorePointer << endl; //prints out the the value of score, which is 70;

If we assign a different value to the dereferenced pointer, it changes the value of the object to which the pointer points to. For example,

*scorePointer = 1400; // this assignment changes the original value of score to 1400;

Note that it is illegal to dereference a null pointer

• Two pointers can be equal if they hold the same address. This means that two pointers can be compared with “==” and “!=”. Two pointers can have the same address if they’re both null pointers, they point to the same object, or if the two pointers point to an address that is one-past the same object.

Okay, that last statement though. I did a bit of research on what “one-past an object” means? Assume we have an array of objects, and a pointer points to the address of this array. In the stack of addresses, it is possible to have a pointer that points to the address of a location in memory that is just immediately next to the address of the array object. In the same program, it is also possible that this “unknown memory location” that is just immediately after the location of the array object has been assigned to another object. And this object could equally have a pointer that points to it. In this scenario, there’s a probability that the pointer of this object, and the pointer that is just after (one-past) the location of the array share the same memory address. I could only understand this weird stuff thanks to this stackoverflow answer.

• References aren’t objects as explained earlier. Now since pointers point to the address locations of objects, we can’t have a pointer to a reference. But pointers are objects in itself, so we can have references to pointers. An example;

int level = 47; //some harmless variable;
int *levelPointer = &level; //a pointer that points to the address of level
int *&someReference = levelPointer; // someReference is now a reference to the pointer “levelPointer”.

Since references are just another way of representing an existing object, someReference is actually a pointer and dereferencing it yields the level object which has a value of 47.

• We can assign a pointer to point to another object, even if the pointer already points to another existing object.

  int x = 150;
  int y = 100;
  int * somePointer = &x; // pointer
  cout << *somePointer <<endl; // dereferencing the pointer prints 150
  somePointer = &y; // valid assignment to another object
  cout << * somePointer <<endl; // dereferencing the pointer prints 100
  

So there it is! That's pretty much how I understand these two compound types.

Now, a bit of therapy from a community that gives me comfort😊

I pretty much knew Horizon: Zero Dawn would be a great game just from seeing some shots. Watched a bit of the gameplay and realized I wasn't wrong! This is one of my many favorites I've seen so far.

I stumbled upon an AppleTV series that is produced alongside Ubisoft titled, "Mythic Quest". Loved it from the very first episode, and not until recently did I realize that one of the characters in the tv show - “Poppy-Li” is literally the goal I’m dreaming of. From her personality to her intelligence, she’s really brilliant! The beginning of the third episode of the first season (Dinner Party) is basically one of the fantasies I’ve had if I was in her shoes hehe

(ignore their reception) lol.

Also, I follow some amazing set of virtual photographers on twitter who regularly bless my feed with amazing shots from some really good games. Just look up #photomode, #worldofvp, #virtualphotography, #WorldofVP and you would see what I’m talking about. Seeing these photos everyday inspired me to work on an app exclusively for these contents only. It’s currently a work in progress and hope to just get it out there soon!

Finally, I see it coming though. I know learning C++ would be the most challenging thing I’ve ever sought out to do. I’ve already experienced some “glimpse” of how technical it could be (and that’s just Chapter 2 of the book I’m using lol), but the voices of these games are WAY louder than the fear of learning what it takes to create one.