Kotlin Language Features Related to Null Handling

-
Any software engineer with a Java background would find the null handling features in the Kotlin language interesting. Let's summarize this topic with some examples. Nullable types: In Kotlin, types are non-nullable by default. If you want a variable to be able to hold a null value, you need to explicitly declare its type as nullable using the Type? syntax. For example, String? denotes a nullable string, while String represents a non-nullable string. Safe calls (?.): Kotlin introduces the safe call operator (?.) for handling nullable types. It allows you to safely invoke a method or access a property on a nullable object. If the object is null, the expression returns null instead of throwing a NullPointerException. Example: data class Person(val name: String, val age: Int, val address: String?) fun main() {     // Create a person with a nullable address     val person1 = Person("John Doe", 25, "123 Main Street")     val person2 = Person("Jane Doe", 30,...
Post a Comment

Image Resizing with Directed Graphs: Seam Carving in Java

-


Seam Carving is an image resizing technique that was discovered in 2007 and it is a feature in Photoshop. We will look at the implementation of this technique using directed graph data structure.

Below are some wikipedia links to the underlying data structre:
https://en.wikipedia.org/wiki/Directed_acyclic_graph
https://en.wikipedia.org/wiki/Shortest_path_problem

The goal is to remove a seam (vertical or horizontal) from the image one by one. To calculate a seam, we need to calculate the energies of the pixels. Let's look at the energy method of our SeamCarver class:

public class SeamCarver {
    private Picture picture;
//...
public double energy(int x, int y) {

         if (x < 0 || x >= width) {
            throw new IllegalArgumentException();
        }
        if (y < 0 || y >= height) {
            throw new IllegalArgumentException();
        }
        if (x == 0 || y == 0 || x == width - 1 || y == height - 1) {
            return 1000.;
        }

         int rx = picture.get(x + 1, y).getRed() - picture.get(x - 1, y).getRed();
        int gx = picture.get(x + 1, y).getGreen() - picture.get(x - 1, y).getGreen();
        int bx = picture.get(x + 1, y).getBlue() - picture.get(x - 1, y).getBlue();

         double deltaSqrx = Math.pow(rx, 2.) + Math.pow(gx, 2.) + Math.pow(bx, 2.);

         int ry = picture.get(x, y + 1).getRed() - picture.get(x, y - 1).getRed();
        int gy = picture.get(x, y + 1).getGreen() - picture.get(x, y - 1).getGreen();
        int by = picture.get(x, y + 1).getBlue() - picture.get(x, y - 1).getBlue();

         double deltaSqry = Math.pow(ry, 2.) + Math.pow(gy, 2.) + Math.pow(by, 2.);
        return Math.sqrt(deltaSqrx + deltaSqry);
    }

So the energy of a pixel is about the RGB values and this is used to find a seam to remove from the image. This is the calculation that allows the content-aware resizing.



We see above the original image and lets see what the energies look like:


The red pixels are the pixels of a vertical seam. That seam will be removed from the image to shrink its width.

Now lets see an example vertical seam and the energies that are calculated:


As we see, we need to find the shortest distance (minimum total energy) from top pixels to the bottom pixels. To do this, we need to model the problem as finding a shortest path in a directed acyclic graph.  So what are the directed edges between the pixels? There will be an edge between a pixel on the top and 3 neighbouring pixels. The weight will be the energy of the target pixel. What we can do is using a directed graph library:

public class SeamCarver {

     private Picture picture;
    private EdgeWeightedDigraph digraphVertical;

     public SeamCarver(
            Picture picture) {                // create a seam carver object based on the given picture
        this.picture = new Picture(picture);
        initVertical(picture);
    }

     private void initVertical(Picture picture) {
        digraphVertical = new EdgeWeightedDigraph(picture.width() * picture.height() + 1);
        for (int col = 0; col < picture.width(); ++col) {
            for (int row = 0; row < picture.height(); ++row) {
                if (row + 1 >= picture.height()) {
                    continue;
                }
                if (col - 1 >= 0) {
                    digraphVertical.addEdge(new DirectedEdge(vertex(col, row), vertex(col - 1, row + 1),       energy(col - 1, row + 1)));
                }
                digraphVertical.addEdge(new DirectedEdge(vertex(col, row), vertex(col, row + 1), energy(col, row + 1)));

                 if (col + 1 < picture.width()) {
                    digraphVertical.addEdge(new DirectedEdge(vertex(col, row), vertex(col + 1, row + 1),     energy(col + 1, row + 1)));
                }
            }
        }

Here we created a graph object with vertices as pixels and created directed edges between pixels. Now we need to use the shortest path algorithm on this graph to find the vertical seams:

    public int[] findVerticalSeam() {  
        double minDist = Double.MAX_VALUE;
        Iterable result = null;
        for (int col = 0; col < picture.width(); ++col) {
            int row = 0; // top
            AcyclicSP acyclicSP = new AcyclicSP(digraphVertical, vertex(col, row));
            for (int botCol = 0; botCol < picture.width(); ++botCol) {
                int botRow = picture.height() - 1;
                double curDist = acyclicSP.distTo(vertex(botCol, botRow));
                if (curDist < minDist) {
                    minDist = curDist;
                    result = acyclicSP.pathTo(vertex(botCol, botRow));
                }
            }
        }
        int[] seam = new int[picture.height()];
        int i = 0;
        for (DirectedEdge edge : result) {
            // System.out.println(edge.from() + "->" + edge.to());
            if (i == 0) {
                seam[i++] = colFromVertex(edge.from());
                seam[i++] = colFromVertex(edge.to());
            } else {
                seam[i++] = colFromVertex(edge.to());
            }
        }
        return seam;
    }

Removing this calculated seam from the image is not difficult:

    public void removeVerticalSeam(int[] seam) {    
        validate(seam);
        Picture picture2 = new Picture(width - 1, height);
        for (int row = 0; row < picture2.height(); ++row) {
            for (int col = 0; col < picture2.width(); ++col) {
                int seamItem = seam[row];
                if (col + 1 >= picture.width()) {
                    continue;
                }
                if (col >= seamItem) {
                    picture2.set(col, row, picture.get(col + 1, row));
                }
                else {
                    picture2.set(col, row, picture.get(col, row));
                }
            }
        }

         this.picture = picture2;
        initialize(this.picture);
    }

Now this solution that uses a graph library works correctly but the performance is not good. Image size can be huge and we need to write our own algorithm implementation on pixels. So we will not use the library classes AcyclicSP and EdgeWeightedDigraph.

public class SeamCarver {

     private Picture picture;
    private double[] vertexEnergy;

     public SeamCarver(Picture picture) {
        if (picture == null) {
            throw new IllegalArgumentException();
        }
        this.picture = new Picture(picture);
        initialize(picture);
    }

     private void initialize(Picture picture) {
        this.width = picture.width();
        this.height = picture.height();

         vertexEnergy = new double[width * height + 1];
        for (int col = 0; col < picture.width(); ++col) {
            for (int row = 0; row < picture.height(); ++row) {
                double energy = energy(col, row);
                vertexEnergy[vertex(col, row)] = energy;
            }
        }
    }

Instead of creating a graph with DirectedEdge objects and energies as weights, I am creating an array of pixel energies.

To implement the shortest path algorithm on the pixels, I need edgeTo[] and distTo[] arrays. Then while traversing from top to bottom, I will perform relaxation. Relaxation is the update of the path: when we see a shorter path, we update the distTo[] and edgeTo[] arrays.

    public int[] findVerticalSeam() {        
        double minDist = Double.MAX_VALUE;
        double[] distTo = new double[V];
        int[] edgeTo = new int[V];

         for (int i = 0; i < edgeTo.length; ++i) {
            edgeTo[i] = -1;
        }
        for (int v = 0; v < V; v++)
            distTo[v] = Double.POSITIVE_INFINITY;
        for (int col = 0; col < this.width; ++col) {
            int row = 0;
            int vertex = vertex(col, row);
            distTo[vertex] = vertexEnergy[vertex];
        }

         for (int row = 0; row < this.height; ++row) {
            for (int col = 0; col < this.width; ++col) {
                if (row == this.height - 1) {
                    // bottom row is already calculated
                    continue;
                }

                 // a: col - 1, row + 1
                // b: col, row + 1
                //c: col + 1, row + 1
                int a, b, c;
                int s = vertex(col, row);

                 if (col - 1 >= 0) {
                    a = vertex(col - 1, row + 1);
                    relax(s, a, distTo, edgeTo);
                }
                b = vertex(col, row + 1);
                relax(s, b, distTo, edgeTo);
                if (col + 1 < this.width) {
                    c = vertex(col + 1, row + 1);
                    relax(s, c, distTo, edgeTo);
                }
            }
        }

         Stack result = null;
        for (int botCol = 0; botCol < this.width; ++botCol) {
            int botRow = this.height - 1;
            double curDist = distTo[vertex(botCol, botRow)];
            if (curDist < minDist) {
                minDist = curDist;
                result = pathTo((vertex(botCol, botRow)), edgeTo);
            }
        }

         int[] seam = new int[this.height];
        int i = 0;
        for (Integer vertex : result) {
            // System.out.println(edge.from() + "->" + edge.to());
            seam[i++] = colFromVertex(vertex);
        }
        return seam;
    }

The relax method is very important and it is a concept of shortest path algorithm for weighted directed acyclic graphs.

private void relax(int s, int v, double[] distTo, int[] edgeTo) {

         if (distTo[v] > distTo[s] + vertexEnergy[v]) {
            distTo[v] = distTo[s] + vertexEnergy[v];
            edgeTo[v] = s;
        }
    }

So if the distance between source vertex "s" and the target vertex "v" is smaller than the existing distance to the target, then we update the edgeTo[] and distTo[] arrays. This means we have found a new shortest path.

NOTES: 

  • One of the applications of graph data structure is image processing.
  • Sometimes using library classes is not enough and we need to understand in detail the underlying algorithm and how the library works.
  • Then we can perform optimizations on the working solution, depending on our special needs.
  • We started by using EdgeWeightedDigraph and AcyclicSP from Princeton University, then we wrote our own algorithm on pixels. 
Thanks to Princeton University Department of Computer Science and Kevin Wayne for the wonderful assignments.

Comments

Post a Comment

Popular posts from this blog

Trie Data Structure and Finding Patterns in a Collection of Words

-
Image
 I faced a very hard problem recently about finding substrings based on a collection of patterns.  Example For  words = ["Apple", "Melon", "Orange", "Watermelon"]  and  parts = ["a", "mel", "lon", "el", "An"] , the output should be  findSubstrings(words, parts) = ["Apple", "Me[lon]", "Or[a]nge", "Water[mel]on"] . While  "Watermelon"  contains three substrings from the  parts  array,  "a" ,  "mel" , and  "lon" ,  "mel"  is the longest substring that appears first in the string. Note: Number of words and number of parts can be huge! A brute force method or careless usage of Java String methods would make it impossible to handle a large number of words and parts (patterns). Solution Approach Whenever we want to search patterns inside a string, we should think about the Trie data structure:  https://en.wikipedia....
Read more

Virtual Memory

-
The main memory can act as a “cache” for the secondary storage, usually implemented with magnetic disks. This technique is called virtual memory. There were two major motivations for virtual memory: to allow efficient and safe sharing of memory among multiple programs, and to remove the programming burdens of a small, limited amount of main memory. Lets consider a collection of programs running all at once on a computer. Of course, to allow multiple programs to share the same memory, we must be able to protect the programs from each other, ensuring that a program can only read and write the portions of main memory that have been assigned to it. Main memory need contain only the active portions of the many programs, just as a cache contains only the active portion of one program. Virtual memory allows us to efficiently share the processor as well as the main memory. We cannot know which programs will share the memory with other programs when we compile them. In fact, the programs s...
Read more

NOTES ON COMPUTER ARCHITECTURE: Some important concepts in computer architecture

-
A  virtual address  is a binary number in virtual memory that enables a process to use a location in primary storage (main memory) independently of other processes and to use more space than actually exists in primary storage by temporarily relegating some contents to a hard disk or internal flash drive. In a computer that incorporates memory management, the virtual address differs from the  physical address , which is the data location on an address bus that corresponds to a particular cell of primary storage or to a particular register in a memory-mapped I/O(input/output)  device. In a computer with both physical and virtual memory, a so-called  MMU (memory management unit)  coordinates and controls all of the memory resources, assigning portions called blocks to various running programs to optimize system performance. By translating between virtual add...
Read more