Tongue Drive: A Brain-Tongue-Computer Interface
Assistive technologies play a critical role in the lives of people with severe disabilities and help them to lead independent self-supportive lives. Persons severely disabled as a result of causes ranging from traumatic brain and spinal cord injuries to stroke and cerebral palsy generally find it extremely difficult to carry out everyday tasks without continuous help. Assistive technologies that help them communicate their intentions and effectively control their environment, especially to operate a computer, can greatly improve the quality of life for this group of people and may even help them to be employed.
II. Alternative Assistive Technologies
A large group of assistive devices are available that are controlled by switches. The switch integrated hand splint, sip and puff device, chin control system, and electromyography (EMG) switch are all switch based systems and provide the user with limited degrees of freedom.
A group of head-mounted assistive devices has been developed that emulate a computer mouse with head movements. Cursor movements in these devices are controlled by tracking an infrared beam emitted or reflected from a transmitter or reflector attached to the user’s glasses, cap, or headband. Tilt sensors and video-based computer interfaces that can track a facial feature have also been implemented. One limitation of these devices is that only those people whose head movement is not inhibited may avail of the technology. Another limitation is that the user’s head should always be in positions within the range of the device sensors. For example the controller may not be accessible when the user is lying in bed or not sitting in front of a computer.
Another category of computer access systems operate by tracking eye movements from corneal reflections and pupil position. Electro-oculographic (EOG) potential measurements have also been used for detecting the eye movements. A major limitation of these devices is that they affect the users’ eyesight by requiring extra eye movements that can interfere with users’ normal visual activities such as reading, writing, and watching.
The needs of persons with severe motor disabilities who cannot benefit from mechanical movements of any body organs are addressed by utilizing electric signals originated from brain waves or muscle twitches. Such brain computer interfaces (BCI), either invasive or noninvasive, have been the subject of major research activities. BCIs that operate based on electroencephalography (EEG) signals are very slow and limited in bandwidth. Implantable BCI technologies, on the other hand, are highly invasive (require a brain surgery) and heavily rely on signal processing and complex computational algorithms, which can results in delays and bulky systems that may also be very costly.
III. Why Tongue?
Tongue and mouth occupy an amount of sensory and motor cortex that rivals that of the fingers and the hand. Hence they are inherently capable of sophisticated motor control and manipulation tasks. This is evident in vocalization and ingestion. The tongue is connected to the brain by the hypoglossal nerve, which generally escapes severe damage even in high level spinal cord injuries. Noninvasive access to the tongue is readily available. It is also the last to be affected in most neuromuscular degenerative disorders. The tongue has many degrees of freedom, and it can move very fast and accurately within the mouth cavity. One can touch every single tooth in his/her mouth with the tip of the tongue. Therefore it is a suitable organ for manipulating assistive devices. The tongue muscle has a very low rate of perceived exertion. Therefore, a tongue operated device can be used continuously over a long period. Also, a tongue-based device is hidden in the mouth and gives its user a certain degree of privacy.
IV. Tongue Drive System
In the Tongue Drive System (TDS), the motion of the tongue is traced by an array of magnetic sensors, which measure the magnetic field generated by a small permanent magnet, the size of a grain of rice, that is embedded in a biocompatible material such as titanium, and attached to the tongue through piercing, implantation, or adhesion. The magnetic sensors can be either mounted on a dental retainer and clipped on the outside of the teeth (internal TDS or iTDS) or on a headset (external TDS or eTDS) positioned near the cheeks. Sensor outputs are amplified, multiplexed, digitized, and transmitted wirelessly to an external controller unit.
Internal Tongue Drive System (iTDS) consists of a permanent magnet attached to the tongue, plus an array of magnetic sensors around the lower teeth on a dental retainer.
Signals received by the external controller, which can be a portable computer or a smartphone are processed to indicate the motion of the permanent magnet and consequently the tongue position within the oral cavity. We can assign a certain control function to each particular tongue movement in software and customize the system for each individual user. These user-defined control functions may then be used to operate a variety of devices and equipments including computers, phones, and powered wheelchairs.
The signals from the magnetic sensors are linear functions of the magnetic field, which is a continuous position-dependent property. Thus a few sensors are able to capture a wide variety of tongue movements. This mechanism provides a tremendous advantage over switch based devices in that the user has the options of proportional, fuzzy, or adaptive control, which can offer smoother, faster, and more natural control over the environment.
V. Prototype Tongue Drive System
We have developed a prototype Tongue Drive System using off-the-shelf commercially available components to evaluate the feasibility and performance of this approach in developing assistive devices. The main purpose of this prototype device was to substitute mouse in computer access by moving the cursor on the computer screen based on the location of the magnetic tracer relative to the four magnetic sensors. Four ratiometric linear sensors are installed in cavities created in a mouthguard. The sensors readily provide temperature compensated linear voltage output proportional to the vertical magnetic field component.
VI. Substituting Mouse for Computer Access
We have been able to successfully substitute mouse with the Tongue Drive System for computer access. Users can move the cursor on the screen using their tongue motions. Then can also issue a single or double-click for selecting icons or opening folders. Several GUIs have been developed for the prototype Tongue Drive System. One of them is a simple computer game, called Fish Tales. Accessible entertainments are considered even more important in improving the quality of life for individuals with disabilities than their healthy counterparts. This experiment evaluates the usability of the Tongue Drive System in enabling users to play computer games, which are controlled by mouse. In this GUI, payers use their tongues to navigate a red fish to catch the smaller fish, while avoiding being caught by the bigger fish. Moving the mouse cursor on the screen results in the red fish swimming in that direction. The further the cursor is moved from the current location of the fish, the faster the fish swims. The goal is to catch as many smaller fish as possible and obtain a high score. When the subject's fish eats enough smaller fish, it grows and can eat larger ones.
VII. Tongue Drive System for Wheelchair Control
We have developed an interface that allows individuals to use the Tongue Drive System for controlling electric-powered wheelchairs by substituting the joystick. Similar to cursor movement on the computer screen, all the users need to do is to touch a predefined set of teeth with the tip of their tongues to drive the wheelchair forward, backward, turn left, or turn right. They can accelerate the wheelchair by holding their tongues in the forward position, and decelerate by returning their tongues back to its resting position. They can even control their powered seating position by switching the wheelchair control mode from driving to seating control.
First clinical trial at Shepherd Center
VIII. Clinical Testing and Evaluation
After testing the Tongue Drive System by able-bodied subjects in a variety of experiments, we started evaluating it in a clinical settings by individuals with severe disabilities. We successfully completed the first round of clinical trial at the Shepherd Center in Atlanta, GA. In this trial, 13 individuals with spinal cord injury at C2-C5 level used the Tongue Drive System for computer access and wheelchair control, while the research team collected data on the usability and efficacy of the system hardware, signal processing algorithms, and GUI software. Participants also provided us with valuable feedback on how to improve the Tongue Drive System further and how they would like to use it.
In our Second Clinical Trial all participants, 23 able-bodied and 11 with tetraplegia in two cities, Atlanta and Chicago, received magnetic tongue piercings and used the system for 5 and 6 weeks, respectively. The results of this study has been published in the journal Science Translational Medicine.
Now we have focused on development of the intraoral Tongue Drive System (iTDS) to shrink the size of electronics and battery to the extent that it would comfortably fit inside the mouth without interfering with speech. The next step would be evaluating the iTDS in a clinical trial and comparing it with the eTDS in terms of usability, comfort, and acceptability among potential end users. We also plan to test both systems in the users' home, office, and outdoor environments.
VIII. Contact us
If you have tetraplegia, live in Atlanta, GA or Chicago, IL, and interested in participating in the upcoming clinical trials, please do contact Dr. Ghovanloo.
Team Tongue Drive with one of the participants in the 2nd TDS Clinical trial.
Tongue Drive System V5
Magnetic Tongue Barbell made of Titanium by Anatometal
TDS in the News
- Washington Post coverage of the results of the TDS clinical study: The wheelchair driven by a tongue piercing: Cool concept faces uphill battle
- Georgia Tech coverage of the results of the TDS clinical study: Tongue Drive Wheelchair
- Shepherd Center coverage of the results of the TDS clinical study: Tongue Piercing Put to Medical Use
- Washington Post coverage of the results of the TDS clinical study: Tongue pierce lets the paralyzed drive wheelchairs
- BBC coverage of the results of the TDS clinical study: Body piercing controls wheelchair
- Radio Interview: BBC World Business Report
- Fox NEWS coverage of the results of the TDS clinical study: Tongue-controlled wheelchairs prove effective for quadriplegic patients
- Associated Press coverage of the results of the TDS clinical study: Paralyzed Patients Flick Tongue to Move
- USA Today coverage of the results of the TDS clinical study: New wheelchair operates with tongue control
- Reuters coverage of the results of the TDS clinical study: Tongue controller for the paralyzed offers greater independence
- Atlanta Magazine coverage of the results of the TDS clinical study: Tongue-Controlled Wheelchair Technology
- Live Science coverage of the results of the TDS clinical study: Tongue-Controlled Wheelchair Helps Paralyzed People Move
- Discovery NEWS coverage of the results of the TDS clinical study: Tongue Piercing Controls Wheelchair
- The Verge coverage of the results of the TDS clinical study: Tongue-controlled wheelchair could offer a faster way for quadriplegics to get around
- US News and World Report coverage of the results of the TDS clinical study: Tongue Piercing Helps Paralyzed People Drive Wheelchairs
- New Scientist coverage of the results of the TDS clinical study: Piercing steers wheelchairs with a flick of the tongue
- Tongue Drive System becomes a 2010 DaVinci Awards finalist and goes on to win the inaugural People's Choice Award: "The LEO"
- Tongue Drive featured in CNN Health Minute: Wheelchair Mobility at the Tip of the Tongue
- Tongue Drive featured in e-Advances, the NIH-NIBIB online magazine: Tongue-Operated Devices Help Paralyzed People
- Tongue Drive featured on Science Nation (The NSF online Magazine): Tongue Driver
- Tongue Drive on Georgia Tech research NEWS website: Clinical Trial Shows That Quadriplegics Can Use Tongue Drive System
- Tongue Drive featured in USA TODAY: 'Tongue Drive' operates wheelchair
- Tongue Drive featured in Reuters: Device puts steering at the tip of the tongue
- Tongue Drive featured on the Georgia Tech research NEWS website: Tongue-controlled System Assists Individuals with Disabilities
- J. Kim, H. Park, J. Bruce, E. Sutton, D. Rowles, D. Pucci, J. Holbrook, J. Minocha, B. Nardone, D. West, A. Laumann, E. Roth, M. Jones, E. Veledar, and M. Ghovanloo, “Tongue enables computer and wheelchair access for the people with high-level disabilities,” Science Translational Medicine, vol. 5, no. 213, p. 213ra166, Nov. 2013. DOI: 10.1126/scitranslmed.3006296 Abstract, Full Text, Reprint (PDF)
- J.S. Minocha, J.S. Holbrook, D.P. West, M. Ghovanloo, A. Laumann, “Development of a tongue-piercing method for use with assistive technology,” J. Am. Med. Association (JAMA) Dermatology, Published online jamadermatology.com, Nov. 27, 2013.
- X. Huo, A.N. Johnson-Long, M. Ghovanloo, M. Shinohara, “Motor performance of tongue with a computer integrated system under different levels of background physical exertion,” Ergonomics, vol. 56, no. 11, pp. 1733–1744, Sep. 2013.
- M. Kothari, P. Svensson, J. Jensen, T. Davidsen-Holm, M. Skorstengaard, Feldbek-Nielsen, M. Ghovanloo, L. Baad-Hansen, “Tongue controlled computer game: A new approach for rehabilitation of tongue motor function,” Archives of Physical Medicine and Rehabilitation, Oct. 2013, In press http://dx.doi.org/10.1016/j.apmr.2013.08.008
- X. Huo, H. Park, J. Kim, and M. Ghovanloo, “A dual-mode human computer interface combining speech and tongue motion for people with severe disabilities,” IEEE Trans. on Neural Sys. Rehab, vol. 21, no. 6, pp. 979-991, Nov. 2013.
- J. Kim, C. Bulach, K.M. Richards, D. Wu, A.J. Butler, and M. Ghovanloo, “An apparatus for improving upper limb function by engaging synchronous tongue motion,” IEEE Neural Engineering Conference, pp. 1574-1577, Nov. 2013.
- M. Kothari, P. Svensson, J. Jensen, A. Kjærsgaard, J. Kim, M. Ghovanloo, L. Baad-Hansen, “Training-induced cortical plasticity compared between three tongue-training paradigms,” Neuroscience, vol. 246, pp. 1–12, Aug. 2013, http://dx.doi.org/10.1016/j.neuroscience.2013.04.040
- H. Park, M. Kiani, H.M. Lee, J. Kim, B. Gosselin, and M. Ghovanloo, “A wireless magnetoresistive sensing system for an intraoral tongue-computer interface,” IEEE Trans. on Biomed. Circuits and Systems, vol. 6, no. 6, pp. 571–585, Dec. 2012.
- X. Huo, H. Park, and M. Ghovanloo, “Dual-mode tongue drive system: using speech and tongue motion to improve computer access for people with disabilities,” Proc. Wireless Health Conf., pp. 47-54, Oct. 2012.
- X. Huo and M. Ghovanloo, “Tongue Drive: A wireless tongue-operated means for people with severe disabilities to communicate their intentions,” IEEE Communications Magazine, vol. 50, no. 10, pp. 128–135, Oct. 2012.
- H. Park, J. Kim, X. Huo, I. Hwang, and M. Ghovanloo, “Development and preliminary evaluation of an intraoral Tongue Drive System,” Proc. IEEE 34th Eng. in Med. and Biol. Conf., pp. 1157-1160, Sep. 2012.
- A. Ayala-Acevedo and M. Ghovanloo, “Quantitative assessment of magnetic sensor signal processing algorithms in a wireless tongue-operated assistive technology,” Proc. IEEE 34th Eng. in Med. and Biol. Conf., pp. 3692-3695, Sep. 2012.
- J. Kim, H. Park, and M. Ghovanloo, “Tongue-operated assistive technology with access to common smartphone applications via Bluetooth link,” Proc. IEEE 34th Eng. in Med. and Biol. Conf., pp. 4054-4057, Sep. 2012.
- J. Kim, X. Huo, J. Minocha, J. Holbrook, A. Laumann, and M. Ghovanloo, “Evaluation of a smartphone platform as a wireless interface between Tongue Drive System and electric-powered wheelchairs,” IEEE Trans. Biomed. Eng., vol. 59, no. 6, pp. 1787–1796, June. 2012.
- M. Kothari, P. Svensson, X. Huo, M. Ghovanloo, L. Baad-Hansen, “Force and complexity of tongue task training influences behavioral measures of motor learning,” European J. Oral Sciences, vol. 119, DOI: 10.1111/j.1600-0722.2011.00894.x, Dec. 2011.
- X. Huo, and M. Ghovanloo, “Using speech recognition to enhance the Tongue Drive System functionality in computer access,” Proc. IEEE 33rd Eng. in Med. and Biol. Conf., pp. 6393-6396, Sep. 2011.
- B. Yousefi, X. Huo, and M. Ghovanloo, “Preliminary assessment of Tongue Drive System in medium term usage for computer access and wheelchair control,” Proc. IEEE 33rd Eng. in Med. and Biol. Conf., pp. 5766-5769, Sep. 2011.
- E.B. Sadeghian, X. Huo, and M. Ghovanloo, “Command detection and classification in tongue drive assistive technology,” Proc. IEEE 33rd Eng. in Med. and Biol. Conf., pp. 5465-5468, Sep. 2011.
- H. Park, J. Kim, X. Huo, I. Hwang, and M. Ghovanloo, “New ergonomic headset for Tongue-Drive System with wireless smartphone interface,” Proc. IEEE 33rd Eng. in Med. and Biol. Conf., pp. 7344-7367, Sep. 2011.
- B. Yousefi, X. Huo, E. Veledar, and M. Ghovanloo, “Quantitative and comparative assessment of learning in a tongue-operated computer input device,” IEEE Trans. Info. Tech. in Biomedicine, vol. 15, no. 5, pp. 747-757, Sep. 2011.
- B. Yousefi, X. Huo, and M. Ghovanloo, "Using Fitts's law for evaluating tongue drive system as a pointing device for computer access," Proc. IEEE 32nd Eng. in Med. and Biol. Conf., pp. 4403-4406, Sep. 2010.
- J. Kim, X. Huo, and M. Ghovanloo, "Wireless control of smartphones with tongue motion using tongue drive assistive technology," Proc. IEEE 32nd Eng. in Med. and Biol. Conf., pp. 5250-5253, Sep. 2010.
- X. Huo, U.M. Jow, and M. Ghovanloo, "Radiation characterization of an intra-oral wireless device at multiple ISM bands: 433 MHz, 915 MHz, and 2.42 GHz," Proc. IEEE 32nd Eng. in Med. and Biol. Conf., pp. 1425-1428, Sep. 2010.
- A.N. Johnson, X. Huo, C.W. Cheng, M. Ghovanloo, and M. Shinohara, "Effects of additional load on hand and tongue performance," Proc. IEEE 32nd Eng. in Med. and Biol. Conf., pp. 6611-6614, Sep. 2010.
- X. Huo and M. Ghovanloo, "Evaluation of a wireless wearable tongue computer interface by individuals with high level spinal cord injuries," Journal of Neural Engineering, vol. 7, #026008, Mar. 2010.
- C. Cheng, X. Huo, and M. Ghovanloo, "Towards a magnetic localization system for 3-D tracking of tongue movements in speech-language therapy," Proc. IEEE 31st Eng. in Med. and Biol. Conf., pp. 563-566, Sep. 2009.
- X. Huo, C. Cheng, and M. Ghovanloo, "Evaluation of the tongue drive system by individuals with high-level spinal cord injury," Proc. IEEE 31st Eng. in Med. and Biol. Conf., pp. 555-558, Sep. 2009.
- X. Huo and M. Ghovanloo, "Using unconstrained tongue motion as an alternative control surface for wheeled mobility," IEEE Trans. on Biomed. Eng, vol. 56, no. 6, pp. 1719-1726, June 2009.
- X. Huo, J. Wang, and M. Ghovanloo, "Introduction and preliminary evaluation of tongue drive system: a wireless tongue-operated assistive technology for people with little or no upper extremity function," Journal of Rehabilitation Research and Development, vol. 45, no. 6, pp. 921-938, Nov. 2008.
- X. Huo, J. Wang, and M. Ghovanloo, "A magneto-inductive sensor based wireless tongue-computer interface," IEEE Trans. on Neural Sys. Rehab. Eng., vol. 16, no. 5, pp. 497-504, Oct. 2008.
- X. Huo, J. Wang, and M. Ghovanloo, "Wireless control of powered wheelchairs with tongue motion using tongue drive assistive technology," Proc. IEEE 30th Eng. in Med. and Biol. Conf., pp. 4199-4202, Aug. 2008.
- J. Wang, X. Huo, and M. Ghovanloo, "A quadratic particle swarm optimization method for magnetic tracking of tongue motion in speech disorders," Proc. IEEE 30th Eng. in Med. and Biol. Conf., pp. 4222-4225, Aug. 2008.
- X. Huo, J. Wang, and M. Ghovanloo, "Using Tongue Drive system as a new interface to control powered wheelchairs," Proc. RESNA Conference, Washington, DC, June 2008.
- J. Wang, X. Huo, and M. Ghovanloo, "Tracking tongue movements for environment control using particle swarm optimization," Proc. IEEE Intl. Symp. on Circuits and Systems, pp.1982-1985, May 2008.
- X. Huo, J. Wang, and M. Ghovanloo, "Using magneto-inductive sensors to detect tongue position in a wireless assistive technology for people with severe disabilities," Proc. IEEE Sensors Conference, pp. 732-735, Oct. 2007 (First Place Poster Award).
- M. Ghovanloo, "Tongue operated assistive technologies," Proc. IEEE 29th Eng. in Med. and Biol. Conf., pp. 4376-4379, Aug. 2007.
- X. Huo, J. Wang, and M. Ghovanloo, "A wireless tongue-computer interface using stereo differential magnetic field measurement," Proc. IEEE 29th Eng. in Med. and Biol. Conf., pp. 5723-5726, Aug. 2007.
- X. Huo, J. Wang, and M. Ghovanloo, "Using magneto-inductive sensors to detect tongue position in a wireless assistive technology for people with severe disabilities," To be presented at the IEEE Sensors Conference, Atlanta, GA, Oct. 2007.
- X. Huo, J. Wang, and M. Ghovanloo, "Use of tongue movements as a substitute for arm and hand functions in people with severe disabilities," Proc. RESNA Conference, Phoenix, AZ, June 2007.
- X. Huo, J. Wang, and M. Ghovanloo, "A magnetic wireless tongue-computer interface," Proc. 3rd Intl. IEEE/EMBS Conf. on Neural Engineering, pp. 322-326, May 2007.
- G. Krishnamurthy and M. Ghovanloo, "Tongue Drive: A tongue operated magnetic sensor based wireless assistive technology for people with severe disabilities," IEEE Intl. Symp. on Circuits and Systems, pp. 5551-5554, May 2006.