Letter from Andrea
The impetus behind this research project is a deeply personal one. My ten-year-old son, Eric, is bilingual in his verbal understanding, and has boundless happiness and enthusiasm, but he does not (yet) speak. His remarkable determination and ability to communicate through various non-verbal ways inspires and entertains those around him.
Try to imagine how frustrating and daunting it would be to be unable to express yourself freely, to be unable to communicate when you need or want something, and to be severely limited in your ability to interact with others. Being a mother of three children, including Eric – who has cerebral palsy – has allowed me to better understand the needs, challenges, and opportunities of people with limited communication abilities.
Through Eric’s eyes I realised how much we take for granted in life and imagined how technology could be used to enhance communication. There is no “one-size-fits-all” – individuals are unique in their ways of communicating, acting, and living. This uniqueness is even more prominent when there are physical and cognitive limitations.
Driven by the unique and diverse needs of people with disabilities, we are developing a novel personalised communication system that adapts to the abilities of individuals, instead of requiring individuals to learn and adapt to the system. The system will monitor individuals’ unique sounds, motions, and gestures, and translate these into easily understood communication. Combining advances and approaches in the fields of software engineering, human computer interaction, design technologies, and AI, I am excited to be able to contribute to the empowerment of so many people to express their thoughts, desires, and needs.
Approach
Children with complex disabilities often struggle to communicate their essential needs. Alternative and Augmentative Communication systems are designed to support individuals with communication difficulties in expressing their thoughts and needs. “No-tech” approaches don’t require external tools, relying on the individual’s body movements, gestures, facial expressions, and sign language to convey messages; however, this can be ineffective when communicating with people. On the other hand, “High-tech” approaches typically use a variety of input devices to navigate through screen-based arrays of words or images. These systems can be challenging for those with limited cognitive and/or physical abilities since they require individuals to adapt to the systems instead of adapting the system to individual abilities.
This research investigates a hybrid adaptive approach using sensors and AI techniques to map individual children’s idiosyncratic sounds, motions, and gestures to intended meanings. The approach will be customised to individuals to enable communication with familiar and non-familiar persons. As machine learning refines the system’s accuracy over time, patients will experience enhanced interaction and reduced frustration. We foresee that with adequate funding, short-term benefits, such as improved daily interactions and reduced reliance on caregivers for basic communication, could be realized within 2-3 years. Over the long term, say 3-5 years, as the technology matures and becomes more widely accessible, individuals could achieve higher levels of independence, social engagement, and emotional well-being, thereby significantly improving their overall quality of life. We also foresee that this approach could be combined with other emerging technologies to help express fully formed expressions in natural language.
Our research aims to combine sensors and AI (machine learning) techniques to create a customisable mobile communication system. We will engineer a novel communication architecture to empower people who are limited by current approaches, enabling them to communicate their needs, emotions, and thoughts.
This research aligns with advancing care and quality of life for non-verbal individuals whose disabilities (and abilities) cause them to communicate in a highly individual manner.
Partners:
We would like to thank the Open University School of Computing and Communication for seed funding for the research. We are currently in talks with several charities and other organisations to become sponsors of this project. If you are interested in becoming involved as a funder, researcher, or contributor please click here to get in touch. We would love to hear from you.