As industries evolve, IoT and AI technologies offer transformative solutions for a wide range of challenges. Whether you are managing a chain of grocery stores, improving equipment fleet management, or automating production lines in food manufacturing, AI can have significant impact on remote IoT installations. Here is one such use case about monitoring customer behavior at a grocery store.
Use Case: Monitoring Customer Behavior
Problem Statement
Retail stores face specific challenges regarding customer behavior, which can lead to losses. For example, food retail stores may suffer considerable losses due to consumers squeezing fruits and vegetables to test for ripeness.
Realization Approach
The solution to this problem lies in deploying AI in the loop, which enables an IoT-controlled camera to monitor consumer behavior, aggregate learnings, and then make appropriate arrangements for consumers to know if the product is ripe.
Solution Space
With a set of pre-trained models as part of a registry, IoT application developers can easily add new capabilities and swap them to make any IoT device or machine smarter. This also applies to AI vision language models to test the best model that can detect and report specific consumer behavior from the camera feed.
Featured IoT Application Enablement Platform
Viam is an open-source platform that simplifies the building, monitoring, and data management of smart machines, from industrial robots to IoT devices. The platform enables secure communication, data synchronization, and peer-to-peer connectivity between smart machines, the cloud, and mobile apps. Their platform also supports the ability to deploy machine learning models for analysis and decision support.
Let’s examine the completely fictional grocery chain FoodChoice, which has realized that they are losing hundreds of thousands of dollars per year on damaged fruit and vegetables due to consumers squeezing them to test for ripeness. FoodChoice thinks that they can use AI to monitor customer behavior, aggregate learnings, and devise alternate ways for customers to know if the produce is ripe.
Managing AI Models with Viam
The software engineers at FoodChoice first attempt to use a pre-trained ONNX-based computer vision model running on their local machines. They decide to use the Viam platform to manage their models, as Viam offers solutions to centrally manage versioning, deployment, and monitoring of AI models and code.
Using the ONNX model with Viam requires only that they upload the model through the Viam app, click to deploy it to their machines, and visually configure an ONNX vision service from the Viam Registry.
Unfortunately, they find that the particular model does not accurately detect the consumer behavior they are looking for, so they begin to collect image data using Viam Data Management to train their own Tensorflow Lite model.
When they are ready to test the new model, despite the model being a different format requiring a different machine learning (ML) framework, the Viam Registry handles infrastructure setup and Viam SDKs allow them to use it without changing their code:
detector = VisionClient.from_robot(robot, "produce-vision")
response = await detector.get_detections_from_camera("avo-cam")
Incorporating Vision Language Models (VLMs)
Despite the new model being a bit more accurate, the folks at FoodChoice are still getting a number of false positives and false negatives. They learn that vision language models (VLMs) are a newer technology that might complement traditional computer vision models to improve their results. While VLMs are slower, they can use more context from both images and language to more accurately understand the task (in this case, if someone is squeezing an avocado).
They first try the multi-modal vision capabilities of the popular ChatGPT platform using the chatgpt-vision module from the Viam Registry. They add it to their machine configuration with a few clicks and add their OpenAI credentials.
The FoodChoice engineers continue to use the CV model to detect if a person is standing near the avocado display, but when this is detected, they then leverage the VLM, asking “Is someone squeezing the avocados?”
To do this, they add two lines of code—one to instantiate the VLM as a second vision service, and another to “classify” the image, passing in the question as a parameter.
By implementing the VLM as a second AI layer, accuracy is greatly improved.
However, they realize that there might be some legal complications in sending images of shoppers to a shared cloud service. They notice that there are some open-source VLMs on the Viam Registry like Moondream and Uform that can be run locally within the store. As before, they find that the Viam Registry handles the setup required to run these models, and they can be used and tested without code modification.
vlm = VisionClient.from_robot(robot, "moondream-vision")
response = await vlm.get_classifications_from_camera("avo-cam", 1, extra={"question": "is someone sqeezing an avocado?"})
FoodChoice is using a fairly inexpensive local CPU, so VLM performance is slower than what they witnessed with ChatGPT. The local speed may be tolerable for now, but they plan to consider using VLMs with the Viam platform in conjunction with isolated cloud serverless solutions like Modal, which can be integrated with Viam Registry modules.
They also plan on building a custom web portal for their data scientists, and find it beneficial that Viam’s gRPC-based API provides SDK interfaces that remain consistent and familiar from language-to-language:
const vlm = new VisionClient(client, "moondream-vision");
let response = await vlm.getClassificationsFromCamera("avo-cam', 1, {"question": speechText});
Companies like FoodChoice can leverage the Viam platform to accelerate their progress by utilizing AI technologies across various projects—ultimately spending less time and money.
Other Use Cases
AI offers powerful solutions for overcoming the challenges faced at every stage of the food supply chain.
Beyond point of sale at a grocery store, we’ve seen innovative companies adopting AI technologies across a range of use cases: from predictive maintenance and smart equipment management to robotic automation and enhanced QA processes, AI can drive significant improvements in efficiency and quality.
Platforms that facilitate easy integration and management of AI models, like Viam, can further accelerate progress, allowing companies to adapt quickly and effectively.
Take a restaurant as an example, which prioritizes quality assurance and exceptional customer service. With Viam, you can capture data from existing cameras and obtain real-time insights remotely. Feeding that data into flexible ML workflows ultimately reduces errors by detecting anomalies and sending alerts for instant action on restocking or performance issues. As a result, you will be able to ensure display quality, food prep efficiency, and more, maintaining customer satisfaction throughout.
Viam dramatically reduces iteration time by making ML, vision, and language models hot-swappable, enhancing the accessibility and usability of AI technologies.
Learn more about Viam and get started today.
This post was originally published in Viam.
