Have a Dialog with Your Home via Watson Conversation Service

Man’s desire to be a part of the social ecosystem has led to the invention of many communication and information sharing tools. This trend is now morphing to include use cases where we interact with non-living things, as well. Let’s look at one such application where we can communicate with our things through chat, just like way we would Whatsapp our family and friends.

Note: This post was originally published in the IBM Bluemix blog.

Home automation is a ubiquitous application. We talk to our appliances, and they obey our orders. That is so cool! But let’s take it a step further. We’ll want to make it a bit more seamless so that we talk to our home in a natural conversation. But how do we do it? Machines talk in binary code which is cumbersome for humans. In this blog article, we will show you how to build a home automation app. So follow along and check out the demonstration provided later in this article to help you understand how to build a virtual assistant for your home.

IBM Watson Conversation

IBM Watson Conversation service allows developers to build natural language interfaces that can interact with machines and virtual agents. This enables us to retain our most natural way of communication while the conversation service does the heavy lifting at the backend to translate our messages so that the machines can understand.

Watson Conversation service provides a drag and drop dialog builder. This dialog builder is the key to designing dialogs that are intuitive for human users and, at the same time, can be deciphered by machines.

A typical human-to-machine conversation is very different from human-to-human conversation. Machines do not usually engage in a conversation flow like we humans do. They take commands and respond with an action. The Watson Conversation service allows us to build specific dialogs that convey commands and actions between humans and machines. The image below illustrates how this is achieved via intents and entities:

As you can see above, the Watson Conversation service breaks down each human message into an intent and an entity. The intent specifies the command to be executed, and the entity specifies the particular machine or device that should execute the command.

In the home automation scenario, this boils down to sending commands to the lights and appliances to either turn on or turn off. So imagine you have an Internet gateway at home which also hosts a virtual agent that is accessible through Whatsapp. If you hook up that gateway to your home automation system, then you can chat with the virtual agent to control your appliances (see the example illustration below).

Building the Home Automation Virtual Agent

Let’s go ahead and build this home automation virtual agent. We have already unraveled the complexity of deciphering the conversation thanks to IBM Watson. Now comes the next challenge, that is, building the virtual agent.

Contrary to the usual practice of building a virtual agent using a server-based application, we can have an agent up and running on the fly with the help of PubNub BLOCKS. PubNub BLOCKS is a serverless computing module which can be programmed easily and with no infrastructure overhead. With BLOCKS, this is how the system architecture of our home automation virtual agent will look (see below).

Now we have a system deployed using PubNub BLOCKS as a virtual agent. It performs three responsibilities:

1. Receive messages from the human user.
2. Translate the message with the help of Watson Conversation API and break them down into command (intent) and device (entity).
3. Send commands to the respective device.

Virtual Agent in Action

Once built and deployed, we could command our virtual agent and control our home, somewhat like the example below.

Project Components and Source Code

We are going to build this exact replica of the simulated home automation demo. For your reference, the source code of this demo is available here in GitHub.

Here are the components of the demo:

• Home: We have used an SVG image of a home that has the floor plan with each room having a light bulb. Since we can’t ship a real working home automation system to you, I have created a virtual smart home with an SVG image and an icon for each room’s light bulb. This home understands commands to toggle the light in each room, and we will see how can we do it via the Watson Conversation API.
• Virtual Agent: The virtual agent runs on the PubNub BLOCK. Its job is to decipher the commands sent by the homeowner and trigger the specific action on the home. For interpreting the commands, it relies on the Watson Conversation service.
• Command Window: The UI that the homeowner can use to instruct the virtual agent.

Cloud Components

To power this virtual agent, we rely on two backend cloud services.

1. Watson Conversation API: Available as a service in the IBM Bluemix catalog, Watson Conversation service provides us with all the tools to build a context aware, dialog system that can be used to power chatbots. At PubNub, we have used it to power our virtual agent that is hosted on PubNub BLOCKs.
2. PubNub Data Stream Network: PubNub provides an ultra-fast, real-time data streaming solution over the Internet which powers many messaging applications across the world. PubNub BLOCKs is a component of PubNub that enables us to deploy microservices over the PubNub data streaming service. We have built a lightweight Javascript Application for the virtual agent that executes within the BLOCK and interacts with the user, the home as well as the Watson conversation API.

Refer to the README file to setup your conversation service and PubNub BLOCKs service.

For building this app, you will need to create a Bluemix and PubNub account. Visit the IBM Bluemix signup page and PubNub service page to create your respective accounts. Both of the services offer a free tier account to play around with their offerings.

Sign up for Bluemix. It’s free!

Operations performed by the Virtual Agent

There is no limit to the the kind of dialogs that you can build with the Watson Conversation service. However, to keep things simple and to show you the most basic novelty of this concept, let’s define the most common commands that we usually perform upon our home appliances.

[Turn on bedroom]

Here, Turn on is the intent and bedroom is the entity. When we issue this command, we are instructing the virtual agent (PubNub BLOCK) to switch on the bedroom light. Below is an illustrated example of how the messages flow across the system components to make this work.

In a similar way, we can send control commands affecting other rooms for turning on / off the lights:

• turn on childrenroom
• turn on livingroom
• turn on kitchen
• turn on portico
• turn off bedroom
• turn off childrenroom
• turn off livingroom
• turn off kitchen
• turn off portico

We have also defined an “is” intent used for monitoring purposes. This helps us to query the present state of the room light.

[is bedroom on]

The virtual agent’s JSON response for this command is:

This way, we can clearly decipher the chat commands and instruct the home gateway to either perform an action or report the status on a specific device/appliance inside the home.

Extending the capability of the Virtual Agent

All these intents and entities are predefined in the Watson Conversation service workspace instance that we have created for this app. So in case, we want to extend its capability, we can easily do so by adding more intents and entities.

Now that we have a virtual agent that can understand our desires to turn on or turn off the lights in our home, what more can we do? As it turns out, with Watson Conversation service’s dialog builder, we can build more complex dialogs and program our agent to perform more tasks.

So here is a challenge for you. How about dimming the lights of your home?

You can write a “dim on” and “dim off” intent for the individual rooms or you may choose to define a “dim all” intent that affects all the lights, as shown above.

Conclusion

With the Watson Conversation service, the cognitive computing wave has reached a new level. From mere number crunching and predictions based on data, we have moved on to the realm of natural language interaction with machines. One can imagine all those device control dashboards with countless buttons and widgets, getting replaced by a plain and simple chat interface. And, along with PubNub BLOCKS, developers can create rich and intuitive experiences for the users just like this home automation virtual agent.

So buckle up and give it a shot. Follow the docs and learn to build the dialog for Watson Conversation service to add those new intents for dimming the lights. And, please do share your experience with us!