LightSync: Integrating Gesture-Based Controls Into Smart Devices

accessible design   proof-of-concept
The initial concept for LightSync was my own. However, the work you see here was created as a team project within the Bentley University HFID Master's degree program.

my role

Project Manager, Lead UX Designer, Research, Lead Engineer (Arduino),

additional roles

Two engineers (Arduino), Two content writers, One voice artist
goal
keyboard_arrow_down

The goal of the LightSync project is to create a gesture-controlled smart lighting system that enhances accessibility for non-verbal and speech-impaired users.

The project seeks to address the current dominance of voice-based interactions in smart homes, which can be a barrier for individuals who cannot easily use voice commands.

LightSync aims to provide an intuitive, alternative control method that enables users to operate smart lighting through gestures, with a long-term vision of integrating sign language recognition for an even more inclusive experience.

solution
keyboard_arrow_down

LightSync is built using the Arduino Uno R4 Wi-Fi and the Grove Smart IR Gesture Sensor to detect and process user gestures for controlling smart lighting. The system interprets specific hand gestures, allowing users to turn lights on/off, adjust brightness, and change hues.

The solution is modular, leveraging Arduino’s open-source ecosystem for flexibility and ease of use. While the current implementation focuses on basic gesture recognition, future iterations aim to map gestures to sign language, broadening accessibility for users with different communication needs.

how the system works

The user presents a gesture before the Grove Smart IR Gesture Sensor. This gesture is captured and sent to the Arduino UNO R4 Wi-Fi, which is interpreted using a predefined library provided by the manufacturer. This library is included in our code, where each gesture is matched to a corresponding command and communicated to the LED Stick where the lights adjust accordingly. Concurrently, the Arduino will send setting and status values to the Wio Terminal, where they are displayed on its terminal screen. The process is intuitive and mimics natural communication patterns, ensuring ease of use for the target audience. Table 4 contains a breakdown of each gesture.
GestureDescription Function
1-Finger Show any finger Gesturing one finger will activate the light, turning it on to its default value.
2-Finger Show any two fingers Gesturing and holding in place two fingers will increase the light’s brightness. The light will increase to its maximum before resetting to its minimum.
3-Finger Show any three fingers Gesturing three fingers will save the current values as the light’s new default setting.
5-Finger Show all five fingers Gesturing one finger will deactivate the life without saving the current settings.
rotate rightShow any finger and rotate clockwise  Pointing one finger at the sensor while rotating clockwise will increase the light’s hue across the color spectrum from red to blue.
rotate leftShow any finger and rotate counterclockwise Pointing one finger at the sensor while rotating counterclockwise will increase the light’s hue across the color spectrum from blue to red.
A labeled flowchart diagram illustrates the process of using an IR gesture sensor with an Arduino UNO R4 and a WIO Terminal to control an RGB LED stick.

introduction

In his 1911 series, the renowned inventor and author Hugo Gernsback conceptualized a luminary device, the “Luminor,” characterized by an innovative control mechanism: the modulation of its power and brightness through conversational speech [1] (Gernsback, 1911). This early envisioning of voice-activated technology by Gernsback, a seminal figure in the realm of technological futurism, predated the contemporary ubiquity of such modalities by several decades.
‍
Today, through the creation of the Internet of Things (IoT) – a system that allows devices to be connected and remotely monitored across the Internet [2] (Stolojescu-Crisan et al., 2021) – home automation has become a reality. Gernsback’s vision for voice-activated technology has been more specifically brought to life through the creation of smart homes. A “traditional home” has appliances that are operated locally and manually, usually by flipping a switch or pushing a button, which lend them limited control. In contrast, a smart home enables remote electronic control and management of smart appliances [3] (Balta-Ozkan et al., 2013). It automates manual tasks through a networked system of smart devices. These devices range from simple and inexpensive light bulbs and doorbell cameras to elaborate home theaters and sophisticated surveillance systems. Examples of smart automation include using lumen sensors to activate exterior lighting when the ambient light dips below a specified threshold. Other, more complex automation can consist of activating multiple devices with a single touch, like dimming the lights and closing the blinds.

Voice interaction, though, remains the primary mode of multimodal communication between the user and smart home devices. This modality is currently at its peak, leading to a natural and straightforward coexistence of human and IoT devices; Alexa, Siri and Google have been equated to the cornerstones of smart home development [4] (Jimenez et al., 2021). However, as we settle into niceties and conversations with these mellow-voiced assistants, there is a pressing need for accessibility to smart homes that pounds on our automatically locking doors.  

Smart homes’ automatizing abilities have a positive impact on health, independent and assisted living, security, and many others [5] (Freddi et al., 2019). But when the primary mode of interaction with these devices remains dominantly dependent to the use of only voice, there is a restriction that settles for those impaired with regards to this sense. Some of the most obvious barriers with respect to accessibility occurs in speech-impaired users, users who, due to a variety of other disabilities, cannot communicate in clear, intelligible speech [6] (Masina et al., 2020), users who are hearing-impaired, speech-impaired or in situations that cause them to be temporarily speech-impaired (for eg: imagine accidental activation of a voice assistant in the dead of the night in a house with a sleeping infant).  

Therefore, in our project, the LightSync, we attempt to accommodate non-speech users of smart homes by using the Arduino UNO R4 WiFi coupled with the Grove Smart IR Gesture Sensor, Grove Base Shield, Grove - RGB LED Stick, and the Grove WIO Terminal to operate a smart light using gestures.

why arduino

Arduino's mission is to enable anyone to enhance their lives through accessible electronics and digital technologies. There was once a barrier between the electronics, design, and programming world and the rest of the world." [7] (About Arduino, 2023) In developing a gesture-controlled smart home system, selecting the Arduino UNO R4 Wi-Fi as the foundational platform is pivotal. The Arduino platform is lauded for its open-source nature and extensive modularity, which are crucial for a project that demands flexibility and a wide range of functionalities.

open-source community

Arduino's open-source framework fosters a collaborative environment where developers can easily access and share resources, code, and solutions. This community-driven approach is invaluable for troubleshooting and innovating, particularly in gesture recognition, where shared experiences and collective wisdom can significantly expedite development.

modularity and flexibility

The ability to integrate various sensors and components is a cornerstone of the Arduino ecosystem. For this project, incorporating the Grove Smart IR Gesture Sensor, Grove Base Shield, and Grove - RGB LED Stick is streamlined, thanks to the platform's inherent compatibility with various modules. This modularity allows the system to be tailored specifically to the unique requirements of sign language recognition.

ease of use and accessibility

Arduino's user-friendly interface and programming environment are exceptionally accommodating to developers who may not have an extensive background in electronics or coding. This accessibility is crucial in democratizing the development process and inviting diverse perspectives, which is especially important in a project focused on accessibility and inclusivity.

cost-effectiveness and availability

Arduino platforms are widely available and cost-effective, making them ideal for prototype development. This accessibility ensures that the project can be replicated and modified by other researchers or enthusiasts in the field.

competitor analysis

Before diving deep into the building of our product, the LightSync, we chose to conduct a competitive analysis to understand what the market has to offer in terms of gesture-control smart homes/devices.

During our research, we discovered that this market is still in its nascent stages. The three stand-out products, closely resembling the concept of what we set out to do, were Aqara Camera Hub 3, Fibaro Swipe and Tap Strap 2.

The table below contains a quick run-through of what their features, pros and cons are. These pros and cons were determined based on how many gestures the product can accommodate, whether the gestures and the settings they are programmed to can be saved for future use and if they come with a steep learning curve.
ProductDescription & FeaturesProsCons
Aqara Camera Hub 3
A smart home control hub that can be paired with multiple major home devices like Alexa, Apple HomeKit, Google Home, with the addition of gesture control.

Can Pan and Tilt camera with 110 degrees Field of View lens.  

Body can pan 360 degrees vertically – to avoid blind spots.  

Ability to capture video in 2K. Has night vision light and AI powered facial recognition.

Local Infrared (IR) Controller to connect with existing IR devices and can operate without internet.
Data (like faces) are not imported to a remote server or the cloud for processing.  

Gesture control
Only 5 gestures can be added to the Aqara and only one function can be automated to each of them.
Fibaro Swipe
A gesture control pad that allows a user to control devices in their Z-Wave network. It detects up to six simple gestures, as well as combinations of those gestures, which can be used to turn on/off associated devices or trigger scenes.

It can also serve as a picture frame, with digital pictures changing as often as the user desires.

Can operate on either USB power supply or batteries. Equipped with a buzzer, which signals detected gestures. Compatible with any Z-Wave or Z-Wave+ Controller and supports protected mode with AES-128 encryption.
Allows 6 gestures, plus combinations of those gestures, allowing for numerous commands.

Supports “associates” for better security.
When battery powered, the device goes into slumber mode, making the first gesture lost to turning on the device.

The gestures are difficult to remember especially for older and younger generations.

Difficult to set up and pair with devices.

Runs on Z-wave protocol which has limited range.
Tap Strap 2
The Tap Strap 2 is worn on the wrist and connects to any Bluetooth device. It has a built-in trackpad and several sensors that let it detect finger taps, gestures, and hand movements. Users can control their devices in any way they want using their “TapMapper” web tool.

Eyes-free text input and device control

Can be operated one-handed.  

Allows for customizable inputs.

Is also portable and accessible
Increased productivity

Reduced fatigue

Improved accessibility

Increased immersion
There is a steep learning curve associated with using.

Tap With Us products because it requires adapting to typing “in the air”.  

Tap With Us products are not compatible with all digital devices.

Users may need to charge it daily if they use it heavily.
Out of all of these, the Aqara and the Fibaro resemble our product concept the most, with the latter bearing the most similarity.

The Aqara poses limitations on the user by allowing only 5 gestures to be mapped to actions on the smart home device it is linked with.

The Fibaro addresses this limitation by allowing the user to take its 6 pre-loaded gestures and create combinations called "sequences”. These sequences are created by pairing together 2 or 3 gestures, which can then be mapped to an action. This is certainly helpful in the sense of expanding the range of services the user can access and is a quality our product shares with the Fibaro.

However, we take it one step further and map particular settings for the hue and brightness of a smart light to specific gestures, instead of merely stopping at turning on and off a device.

system architecture and implementation

individual components

The LightSync architecture is designed to be both robust and intuitive, incorporating several key components from the Arduino ecosystem.

Arduino UNO R4 Wi-Fi

The Arduino UNO R4 Wi-Fi serves as the system's central processing unit and is responsible for interpreting data received from sensors and executing appropriate commands. The board's built-in Wi-Fi and Bluetooth capabilities are essential for enabling wireless communication with smart home devices, thus facilitating remote control without the need for physical wiring.  

In addition to these features, the Arduino UNO R4 Wi-Fi is equipped with onboard EEPROM (Electrically Erasable Programmable Read-Only Memory). This component is particularly beneficial for storing device details and user preferences, such as device identifiers and customized user settings, even when powered off. This persistent memory is crucial for ensuring that the system can quickly and efficiently resume its operations without the need to reconfigure settings after each restart.

Grove Smart IR Gesture Sensor

Grove Smart IR Gesture Sensor is an intelligent gesture recognition module equipped with an infrared camera sensor and applied AI algorithm. It can detect over 15 gestures with wide detection while supporting both IIC and SPI communication [8] (Grove Sensor). This sensor is essential for the gesture recognition aspect of the project. It detects hand movements and gestures in its field of view using infrared technology. The sensor's sensitivity and accuracy are pivotal for accurately capturing and translating physical gestures into digital signals that the Arduino can process.

Grove Base Shield

The Grove Base Shield acts as a unifying platform for connecting various Grove modules to the Arduino board. Its key advantage is the elimination of soldering, enabling a more user-friendly and flexible approach to building the system. The plug-and-play functionality allows for rapid prototyping, as components can be easily swapped or reconfigured, facilitating experimentation and design modifications.

Grove - RGB LED Stick

The RGB LED Stick provides visual feedback and indications to the user. It can display a wide range of hues, patterns, and brightness levels and thus represents real-world lights that might be found in a smart home. These features will also act as feedback to signify different statuses or responses from the system, such as successful gesture recognition and confirmation of command execution.

Grove WIO Terminal

The WIO Terminal serves as a proxy device for a smart home controller. Where a smart home controller might provide device or system status on a graphical user interface (GUI) device such as a tablet or mobile app, the WIO will provide immediate feedback by displaying the light's brightness level, its hue value, and whether it is in "default" or "custom" status. Furthering its proxy representation, the Wio Terminal will connect to the system via Wi-Fi, incorporating a differentiator that many smart home devices offer — the ability to communicate across the entirety of the platform along various communication protocols, such as Wi-Fi.

Arduino code

Based on the C++ language, it acts as the bridge between the hardware and the intended actions. Central to their mission of breaking down the barrier between experts and novices, the Arduino code is relatively easy to understand and learn, with many of the more complex functions made available through imported libraries. Additionally, the large community of experts, artists, and hobbyists sharing projects and knowledge within this community creates a base of knowledge that anyone can leverage to help build their system [9] (Jamieson, 2011).

list of Arduino UNO code libraries

Library Description Author
WiFi.h Enables network connection (local and Internet) using the Arduino WiFi shield. Arduino 
WiFiClient.h Creates a client that can connect to a specified internet IP address and port. Adrian McEwen 
FastLED.hMulti-platform library for controlling LEDs along with optimized math, effect, and noise functions.  Daniel Garcia 
EEPROM.h Enables the saving of values to the Arduino memory. Arduino 
Arduino_Secrets.h Holds credentials for accessing and joining specified Wi-Fi network Wuruibin & Xiangnan (Seeed Studio) 
Gesture.hEnables gestures to be recognized by the Arduino board Arduino 
TFT_eSPI.hHardware driver and graphics library for Wio  Bodmer 
Free_Fonts.hEnables the Wio to call up various fonts and styles to display on screen.  Bodmer 

future scope

The global smart home automation industry is estimated to be valued at USD 444.98 billion by 2030 says a report by Grand View Research, Inc., a market research and consulting company. The market is expected to benefit from the increasing applications of smart home automation in security & access, entertainment, lighting, HVAC, energy management, smart kitchen, and other appliances. North America accounted for the largest market revenue share in 2022 and is expected to flourish further at a significant rate. [10] (Grand View Research, 2021).

As established, our product aims to enter this burgeoning industry to automate light control through gestures. However, in the future, we want to extend beyond artificially created gestures and utilize sign language to operate the light. This was the original motivation behind the creation of LightSync. There are more than 70 million people around the world that use sign language to communicate. Of course, we remain cognizant of the fact that there are different kinds of sign languages that exist. We want to start with catering to the Deaf American Sign Language (ASL) users in the United States comprising of approximately 250,000 to > 500,000 people [11] (James et al., 2022). This is a large, untapped market that we see immense promise in.  

We see gestures as the starting point for providing some level of accessibility to non-speech users, but they don’t naturally fit into the vocabulary of sign language. We want to create a product that doesn’t require the sign language user to change how they interact with the products around them. Technology has immense potential to provide solutions to achieve a higher accessibility [12] (Kraljević et al., 2020), and this is the direction our future research will be focused on.

references

  1. Gernsback, H. (1911, May). Ralph 124C 41 +. Modern Electrics, 4(2), 83-87.
  2. Stolojescu-Crisan, C., Crisan, C., & Bogdan-Petru Butunoi. (2021). An IoT-Based Smart Home Automation System. Sensors, 21(11), 3784. https://doi.org/10.3390/s21113784
  3. Balta-Ozkan, N., Davidson, R., Bicket, M., & Whitmarsh, L. (2013). Social barriers to the adoption of smart homes. Energy Policy, 63, 363-374. https://doi.org/10.1016/j.enpol.2013.08.043
  4. C. Jimenez, E. Saavedra, G. del Campo and A. Santamaria, "Alexa-Based Voice Assistant for Smart Home Applications," in IEEE Potentials, vol. 40, no. 4, pp. 31-38, July-Aug. 2021, doi: 10.1109/MPOT.2020.3002526
  5. Freddi, A., Longhi, S., & Monteriù, A. (2019). Special Issue on “Smart Homes”: Editors’ Notes. Sensors (Basel, Switzerland), 19(4), 836-. https://doi.org/10.3390/s19040836
  6. Masina, F., Orso, V., Pluchino, P., Dainese, G., Volpato, S., Nelini, C., Mapelli, D., Spagnolli, A., & Gamberini, L. (2020). Investigating the Accessibility of Voice Assistants With Impaired Users: Mixed Methods Study. Journal of medical Internet research, 22(9), e18431. https://doi.org/10.2196/18431
  7. About Arduino. Retrieved December 13, 2023, from https://www.arduino.cc/en/about
  8. Grove Smart IR Gesture Sensor (PAJ7660) | Seeed Studio Wiki. (2023, June 28). https://wiki.seeedstudio.com/grove_gesture_paj7660/
  9. Jamieson P., (2011). Arduino for Teaching Embedded Systems. Are Computer Scientists and Engineering Educators Missing the Boat? https://www.semanticscholar.org/paper/Arduino-for-Teaching-Embedded-Systems-.-Are-and-the-Jamieson/2f22430ee4641c3460c3d78f6b19cbd5ae0d86af
  10. Grand View Research Inc., (2021), Smart Home Market Size, Share And Trends Report, 2030. https://www.grandviewresearch.com/industry-analysis/smart-homes-industry
  11. James, T. G., McKee, M. M., Sullivan, M. K., Ashton, G., Hardy, S. J., Santiago, Y., Phillips, D. G., & Cheong, J. (2022). Community-Engaged Needs Assessment of Deaf American Sign Language Users in Florida, 2018. Public health reports (Washington, D.C. : 1974), 137(4), 730–738. https://doi.org/10.1177/00333549211026782
  12. Kraljević, L., Russo, M., Pauković, M., & Šarić, M. (2020). A Dynamic Gesture Recognition Interface for Smart Home Control based on Croatian Sign Language. *Applied Sciences, 10*(7), 2300. https://doi.org/10.3390/app10072300