Researchers at the University of São Paulo’s Mathematics & Computer Science Institute (ICMC-USP) in São Carlos, São Paulo State, Brazil, and at Brown University, Rhode Island, USA, have developed a computational tool that uses innovative image segmentation techniques to facilitate the task of modifying digital images based on the selection of particular elements to be highlighted or excluded.
Image segmentation is a field of computer science dedicated to digital image processing and pattern recognition. The novelty of the newly developed tool lies in its ability to incorporate Laplacian coordinates, which are mathematical operators used to study phenomena in several areas of science, such as astronomy, fluid mechanics and computer graphics.
The researchers implemented the use of these coordinates to develop a prototype computer program that can be deployed to segment images easily and quickly. Users do not require expertise in computer graphics; all they have to do is make small marks inside and around the segment they want to highlight or crop.
The work was done as part of the research project “Mesh generation from images by topological segmentation”, for which Wallace Correa de Oliveira Casaca acted as the principal investigator with a PhD scholarship from FAPESP, under the supervision of Luis Gustavo Nonato, a professor at ICMC-USP. Gabriel Taubin, a professor at Brown University, collaborated with Casaca, who worked at Brown for a year supported by a scholarship awarded by FAPESP under its Research Internship Abroad Program (BEPE).
“In order to process any element of an image, the software must know exactly where the element in question begins and ends,” Nonato explained. “Laplacian coordinates propagate the information drawn by the user inside and around the element until it reaches exactly the limit between the object of interest and other elements of the image, guaranteeing a precise cutout.”
Among the tool’s many possible applications are the cropping of photographs for personal or artistic purposes and the highlighting of specific areas of images to identify tumors or foreign bodies.
A paper presenting the results of the project in the image segmentation field was selected for the Conference on Computer Vision & Pattern Recognition (CVPR) held by the Institute of Electrical & Electronics Engineers (IEEE) in Boston, Massachusetts, in June. Another paper on combining the segmentation strategy with image restoration techniques was accepted for the IEEE International Conference on Image Processing (ICIP) held in Quebec, Canada, in September.
The software created by Casaca’s team tells the computer which elements to change and leaves the rest alone. The user draws lines in different colors to crop or otherwise alter the image. Precision is not required; instead, only a minimum of information is needed for the system to automatically recognize what needs to be done and make the desired changes.
“When a user draws a red line, that tells the system to ‘put this information into my image here’,” Casaca explained. “A green outline means that only what it contains should remain visible. The idea is to represent a digital image in terms of mathematical variables and then apply specific mathematical models to segment the image, i.e., cut it up into pieces that can be easily identified by a human observer.”
The method that enables the computer to recognize what needs to be done based on such simple indications provided by the user is called seed-based image segmentation.
“The user ‘sows seeds’ that are propagated by the tool until the limits of the object are reached,” Casaca said. “One of the advantages of this novel method is that unlike other techniques, it enables objects to be cut out with high-precision edge adjustment. Other programs that automate the process produce different results even when the same marks are used. They eliminate parts that shouldn’t be cropped, for example, because they interconnect with other elements of the image.”
The new tool was evaluated using 50 images from Microsoft’s Grabcut dataset. According to Casaca, the results were quantitatively and qualitatively comparable to those obtained using current state-of-the-art methods.
“After adapting the theory relating to Laplacian coordinates for use in image segmentation, we developed a series of numerical and experimental tests to evaluate its efficiency compared with traditional techniques,” Casaca said. “The results of these tests show that the new tool leverages all of the advantages of this theory, such as its high degree of object contour adherence, low computing cost, and ease of use, among others, to produce the final segmentation result.”
Casaca is now working on a postdoctoral project entitled “Seeded image segmentation and visual layout arrangement from minimization of energy functionals on graphs,” also with support from FAPESP, and plans to develop a version of the software for smartphones to extend its potential applications.