The Intelligent Home

Mark J. Norton, Nolaria Consulting

Jan. 13, 2015

2015, All Rights Reserved

The idea of using electronics to automate tasks around the home (according to Wikipedia), has been around since the 1960's. All sorts of interesting devices have been created to control heat and air conditioning, provide security, and all manner of other functions. Along the way, many different kinds of technologies have been developed from X10 through Z-Wave, Zigbee, and now BlueTooth LE.  The latest round of development is being driven by advances in smart phones, faster network speeds, and the impact of Moore's Law, for as long as it lasts.  These advances have been touted as leading us towards a Smart Home and (more recently), the Internet of Things (IoT), which is supposed to embed smarts in everyday objects.

The recent Consumer Electronics Show in Las Vegas, Nevada is evidence that Home Automation and the IoT is really starting to take off.  The market for these kinds of devices is expected to explode over the next five years, according to most industry sources.  For the most part this will be scatter-shot approach at best.  While some vendors are working hard to cover the various aspects of home automation and simplify installation and use, a bigger vision might serve to provide a better business strategy.  This vision and how we might reach it is described, below.

An Expanded Vision

Before getting into details, we need to first ask, "What is the point of all this?"  The answer is simple:  Home Automation and the IoT should make home living easier, more comfortable,  and more efficient.  Simple to say, but we are not there yet.  Current approaches can be difficult to install and configure, don't connect or integrate well with other devices, and are often complex to understand or control.  Some of these problems could be resolved by industry standards or cooperative approaches, such as those promoted by the DCIA, but will ultimately be decided by the market itself (winners and losers).

From a conceptual point of view, home automation looks a bit like the diagram above.  A user interacts with a home automation system, perhaps via a smartphone, but also directly using buttons and switches installed in the home.  The system, which could be running on a hub, performs the action desired by the user - turning on a light, reporting the temperature, etc.  While this view seems simple, in practice it's usually a lot more complex.  There might be multiple apps on her phone to control separate devices.  These apps might talk to a central controller (hub), but might instead talk directly to the devices.  Often, data is stored in a cloud computing application, which might be free but is more likely offered on a fee-for-service basis.  As such, it might look more like this:

This is just the control view, it gets worse when we try to add a new device to the system.  What if there was a simpler way?  What if there was someone we could talk to, a butler for example, to get a status update or make something happen?  What if this butler knew about your preferences and could proactively adjust to the changes in your life or events as they happen?  What if the home itself was intelligent?

The intelligent home has been described in many science fiction novels but now, for the first time, that vision is within our reach.  Sheer compute power, drastic reduction in the cost of sensors and components, increases in communication bandwidth to the home, and significant advances in software techniques moves an intelligent home agent into the realm of the possible.  Early examples of intelligent agents include Apple's Siri, Google Now, Ubi, and Amazon Echo.   Currently, most of these are focused on search and access to information, but with the introduction of the Works With Nest program and the forthcoming Apple HomeKit we will see increased application of speech recognition and generation.  The Ubi, in particular, is moving rapidly towards control of the home using voice interactions (Leor Grebler, Co-founder and CEO of Unified Computer Intelligence Corp. is interviewed by GigaOm's Stacey Higginbotham).

Voice interaction is not a simple technology.  There are all sorts of problems associated with language, pitch, accents,  background noise, etc.  Still, the technology is improving at a rapid rate.  If this rate of progress continues (and there is no reason to be believe otherwise at this point), speech will become the default way to interact with many devices, including the intelligent home.  As such, the time is right for home automation platforms and device manufacturers to lay the groundwork that will be needed in a few, very short years.  Some of that time will be needed to educate the user population that speech is the right way to communicate with your home.

Though speech will be a powerful way to communicate, there is more to an intelligent home agent that just processing voice I/O that triggers predefined commands.  Over time, the agent should customize itself to understand user preferences, predict behavior, adapt to mood, and proactively assemble information likely to be needed.  The agent should be capable of sensing new devices installed in the home.  Early support for device discovery and autoconfiguration are being explored by organizations such as the AllSeen Alliance.  The home agent should be capable of determining who is speaking, leveraging a personality model that is kept up to date with relevant information (name, gender, roles, preferences, history, etc.).

Making it Work

Several technologies are required to realize an intelligent home agent.  All of these are currently available in some form of off-the-shelf software packages, but all will require considerable customization to tailor it towards making conversations comfortable, getting behaviors to be predictable and useful, and nailing the device  integration problem.  The issue of data security is addressed in it's own section following this one.  Let's have a closer look at these technologies.


Though they seem like two sides of the same coin, speech recognition and speech generation have a separate set of issues.  Fortunately, a great deal of research has led to the development of both hardware and software based solutions.  Texas Instruments, for example, offers the TMS320C2x and TMS320C5x speech recognition components.  Google (Android Speech), Apple (Siri - originally from SRI International), even Microsoft (Windows Speech) offer software based recognition applications.  New offerings, like that from Cubic Robotics and Sensaura, are coming onto the market this year.

Good speech generation requires a way to structure sentences based on the context of a conversation.  The best of these are dynamic in nature, so as to avoid the static dialogs that arise from template driven approaches.  The conversation model provides the context that human users commonly use.  If the home agent makes a recommendation that energy can be saved by lowering the temperature at night, a response of "Okay, do it" relies on this conversational context and possibly having historic knowledge of the user as well.

Data Modeling

Over time, the home agent needs to assemble a "personality model" of the users it interacts with.  This model uses both structured and unstructured data.  Structured data includes predefined characteristics that can be associated with people:  name, gender, age, biometric information (weight, height, sizes, etc.), communication addresses (email addresses, social media names, phone number, etc.), current location, photos, and many others.  Unstructured data includes information gathered in past conversations which might include likes, dislikes, choice preferences, anecdotes, quotes, and the like.  Different kinds of data usually require different ways to store and retrieve them.  Relational databases and property stores both have their place, along with other techniques.

IoT Integration

One of the main purposes of the intelligent home agent is to shelter users from the complexities of home automation platforms, devices, and sensors.  The ever-shifting ground of communication protocols is unlikely to be resolved anytime soon, but these can be pushed into the background by a well designed (and implemented) agent.  Good integration includes the following:

Designed properly, the agent could update itself as new devices appear on the market or perhaps on-demand as devices are added to the local network.  This could include new protocols, APIs, and other software needed to maintain good device integration.

One final aspect of device integration should be considered:  finding patterns in numerical data.  To be proactive in recommending improvements to home interactions, the intelligent agent needs to analyze sensor and event data over time to find correlations and recurring events.  Fortunately, the technology of Deep Learning is starting to make this possible.  Simulated neural networks can be trained to extract features from sensor data - the more the better.


Powerful tools can be used for great purpose, but also (sadly) terrible things.  This is true of the intelligent home agent as well.  The contextual data that enables the agent to converse easily and meaningfully with a user could be used for criminal purpose if it gets into the wrong hands.  Access to the command and control functions managed by the home agent could give easy access to a thief.  For these and many other reasons, security needs to be designed into the intelligent home from the very start.

Authentication and Access Control

Secure systems keep tight control on who has access to what capabilities and information.  This access in turn depends on authenticating who the system is interacting with.   Simple authentication practices (such as username and passwords) are increasingly insufficient to protect users.  The personality model created and maintained by the home agent offers the potential to substantially improve identity verification.  Any or all of the following can be used to verify identity:

If identity seems questionable, the agent could even question the current user about past conversation.

With identity established, access to data and control functions can be determined.  There are many good authentication systems and services available on the market (SAML, for example).  Because of the conversational approach used by the agent, a user with the ability to grant access to other users can be improved by questioning appropriateness.  For example, if a request to grant access to the lawn sprinklers is made for a grade school child, the agent might ask to verify this, given the new user's young age.  The more knowledge added to the intelligent agent about these sorts of interactions, the more secure the system can be made.

Data Security

Access to data poses it's own problems.  If an external system or person can get direct access to the data that drives the system, security is subverted.  One issue that needs to be examined is data locality, ie, where does the data actually live?  More and more, the answer to this question seems to be "in the cloud".  There are reasons why this might be a logical solution, but as the problems of identity and data theft increase, more and more homeowners may choose to keep their data close in their own home.  As long as local compute power and storage requirements can meet the needs of the intelligent agent, there is no technical requirement that data or computation need to be pushed into the cloud (business reasons are addressed below).  With a strong firewall, intelligent home data might better be kept in the home, rather than elsewhere.  Fire or other local disasters could be mitigated by encrypting data and storing it in a trusted location.

Marketing Information

Access to personal information is not only desired by cyber criminals.  It's also very valuable marketing information highly desired by vendors, marketing agencies, and others.  Many applications have attempted to control access to personal data via opt-in strategies, legislation, user preferences, etc.  An intelligent home agent offers the possibility of taking this further and simplifying the process at the same time.  Information requests can be anonymized (using TOR, for example).  Requests to opt-in could be made verbally (by the agent) instead of in text and the implications explained on demand.  Properly executed, this would return control to the user over valuable data leading to compensation for information offered while still offering access to important marketing data for interested parties.


Dependable advances in compute power and software performance and falling costs will enable the development of an intelligent home agent that will be capable of sheltering non-technical users from the complexities of home automation.  These agents will adapt to user preferences and provide better security through user model knowledge.  The time is right to start laying the foundations for the intelligent home of the future.

Mark J. Norton, Nolaria Consulting

Mark Norton is a Senior Consultant and Software Developer at Nolaria Consulting.  He offers a wide variety of software services including technology analysis and consulting, product evaluation and testing, product integration, communications, and custom application development.  Mr. Norton has over 35 years of experience in a broad range of technology sectors including higher education, artificial intelligence, consumer and business applications, and others.  These capabilities are now being applied to Home Automation and the Internet of Things.  He can be contacted at