Food quality and food security are contentious issues of our civilizations since ages. In modern times, this has morphed into more significant concerns relating to optimization of on-demand supply chains and profitability of agribusinesses. In this blog post, we will address one such aspect and get a glimpse of how machine learning can address some of the pain points of the F&B and Ago based industries.
Explore More Use cases for F&B and Agro based Industries.
Visual Inspection of Food Products
When we buy veggies from the store, we always perform a quick visual scan of their appearance to ascertain their quality. We might also smell them or look for other features, but let's focus on the visual scan for now.
While ordering packaged veggies online, we have the same expectations. However, in this case, the onus of ensuring that the vegetables look pristine is with the supplier or doorstep service company. Compromising on the quality can be a big killer for them. Such companies employ much manual labor for conducting sorting & regular quality checks. And wherever we have manual work, productivity and efficiency is a concern. Not to also forget the costs involved in managing this process that dent the bottom line of a company.
There is a real need to optimize this process through automation. Let's look at the business needs and various technological options available. If you are more interested in how we can solve this problem using machine learning then jump to the technical use case section below.
Different Ways Employed For Sorting Food Produce
In many parts of the world, food inspection and sorting is mostly a manual process. It is hugely labor-intensive, slow and a tedious process that is prone to inefficiencies and quality lapses.
This technique uses some form of sorting machines that use conveyor belts to channel the fruits and vegetables based on their shapes or other criteria. Manual labor is still employed here, but it is more efficient than manually sorting a heap load of vegetables.
Courtesy, Tomra
These are fully automated machines for sorting. They employ some of of technology, either optical, X rays or other sensing devices to gauge the quality of the raw food product. These are super efficient and require minimal manual supervision.
The Business Case for Automated Food Quality Inspection
The advancements in artificial intelligence and machine learning now make it possible to replicate the manual visual inspection with machines.
The business case for such a system is self-evident. If the machine learning technology enables us to visually scan the vegetables and classify them into predefined food quality grades, then this can save costs for the company in many ways
The Possible Technology Interventions
There are multiple ways to achieve automated food quality inspection using technology.
- 1Image Scanning - Using high speed cameras to visually scan the food product for surface defects
- 2Gas Sensing - Using gas sensors to detect gases like Ethylene for ripening stage of a fruit
- 3X-Ray, Laser Scanning - Using X Rays/Laser to sense internal quality of the food item.
- 4Temperature/Moisture Sensing - Scanning for temperature/moisture content to sense the freshness or the ideal storage condition. In some cases the food item is crushed slightly to extract the internal moisture content.
Most of the above techniques, rely on some form on classification or regression mechanism that fall under the realm of data science, to detect the quality of the food product. The only exception being image scanning.
Let's pick image scanning as a challenge and see what can be done to enhance and expedite manual sorting of vegetables.
The Technical Use Case For Visual Inspection of Vegetables using Machine Learning
To keep things simple, let's take tomatoes, which is one of the most commonly used staple food. It has a smooth and even surfaces, unlike leafy vegetables, which is easier to scan.
So our problem statement scenario goes like this: There is this online grocery chain, which has got a massive warehouse to source the raw agricultural produce like tomatoes, from farmers/distributors. They employs hundreds of workers to perform sorting process on a daily basis. As a result, they are not able to maintain consistency in quality. Cost is also a concern as manual labor is unpredictable.
To solve this problem, we can think of a use case which comprises a sorting machine. The batch of tomatoes is collected and brought over to the warehouse and passed through this machine. Afterward, the machine performs a series of steps as follows.
- 1Image Capture - Each tomato is passed to small chamber which has constant light and a high res camera takes images from all sides.
- 2Scanning - Images are scanned to look for surface defects, shape abnormalities.
- 3Grading - Based on the scanning process, the tomato is assigned a quality grade
- 4Collection - Based on the grading, the tomato is routed to a specific conveyor and collected in a basket assigned for that grade.
Let's cut the steps, and focus on scanning & grading, where we can employ specific image-based recognition techniques.
Using Dlib for Defect Pattern Recognition on Tomatoes
Dlib is a robust machine learning library. It also supports image recognition capabilities based on classification and deep learning models.
One of the popular examples of using Dlib for recognizing faces is available in their Github repository. Let's follow the same model to see if we can relate it to the tomatoes.
Tomatoes, have a smooth surface but based on the type of defect, their surface can form known patterns, similar to facial features. For example, a tomato having infested with insects or having an abrasive surface due to mishandling, or squashing.
If we build a dataset of these patterns over a period, then we will be able to create the knowledge base and train a machine learning algorithm to assign grades to these patterns.
So that's precisely in line with our problem statement. By following the Dlib example for detecting an object and predicting the shape of the defect feature pattern, we can sort the tomatoes and separate them into different quality grades.
Working Example of Defect Scanning
We have built a demo software to showcase the scanning stage of this use case. The complete source code and procedure to develop and run this demo is available on Github. A few sample images of tomatoes with the similar kind of defects have been used to test this software.
The first step is the define the object portions in the training images of tomatoes for annotating their surface features using the imglab utility.
Once done, we can run the program to perform the object detection and shape detection. It also provides a visual indication of the functioning of the software.
Object Detection
Shape Detection
Shape Detection (False)
See the bounding red box that identifies the tomato and green dots that mark a defect feature pattern. You can follow the README file to understand how to train and run this system. A brief code walk through is also provided to explain the internal APIs of Dlib used in the program.
If we perform these steps for all types of defect patterns, we can build a learning algorithm that can amass a vast knowledge base from features on a tomato's surface. A full blown system can thus be deployed as part of the scanning process, and additional logic can be defined to grade the tomatoes based on scan results.
We are not sure if any of the existing optical sorting machines use this technique. Moreover, a detailed analysis of the accuracy and performance of this technique is also out of the scope. However, if you are researching for a solution around a similar problem, then we hope that this demo will provide you with a lead towards the right direction.
Going Beyond Tomatoes
It is too premature to say if this technique can scale up for practical use. Even more important would be to make it fast enough so that it can eclipse the time taken by manual sorting by many folds. But if it does, then it's possible to make it work for other agricultural products which has a similar smooth texture like tomatoes.
Most fruits would fall under this category. So if you want to explore this demo and see how it fares against other food produces then give it a shot and let us know how it performs. Please share your ideas and comments below, and if you have tried to solve a similar problem with Dlib, then we would like to hear your views.
Code Contributions
The technical demo & source code for this blog post are contributed by Aavritti Soni & Annukriti Soni
