Real Time Machine Learning based Assistive Communication System Using Mid-Air Gestures and Voice Feedback for Children with Physical Disabilities

Abstract

Gesture-based interaction offers an important alternative mode of communication in human–computer interaction, especially for users who cannot conveniently operate conventional input devices such as keyboards. While simple gestures are easy to learn and perform, they are often limited in expressiveness and are not sufficient for text entry. To address this issue, this work presents a mid-air word gesture and voice alert system that recognizes letters written in free space and converts them into speech and alert messages. The proposed system uses an Arduino Uno interfaced with an MPU-6050 accelerometer attached to the user’s hand. The accelerometer captures motion data corresponding to hand movements in air. These signals are processed and used to train a machine learning model capable of recognizing user-defined alphabet gestures. Once trained, the system interprets gestures as letters, displays the generated text, and converts it into voice output. In addition, when a sentence is completed using a predefined full-stop gesture, the message can be sent to a nurse through email, while a default SMS notification is generated through a GSM module to alert the nurse. This system is particularly useful for physically challenged users who may find conventional communication methods difficult. By combining gesture recognition, text generation, speech synthesis, and alert messaging, the work provides a practical assistive communication solution.