Navigation for the blind, “translator” for dialogues with the deaf and eye substitutes…
According to the WHO, there are more than 2 billion people with visual impairments in the world. The organization estimates that more than 900 million people will have hearing problems by 2050. Smart gadgets empower these people and help them get by without outside help. One of the manufacturers of such devices is the Russian laboratory “Sensor-Tech”. She told Haytek about the devices, their differences from foreign counterparts and how implants can restore vision even to blind patients.
Denis Kuleshov – Director of the Sensor-Tech laboratory.
Alexander Popov is the chief designer of Sensor-Tech.
Andrey Demchinsky – Head of Medical Projects “Sensor-Tech”.
Neurotechnologies that remove restrictions
A man is walking down the street. In one hand he has a white cane, with which he feels the way. In the other, there is a small black object that looks like a video camera. The person directs it first in front of him, then to the sides. Passers-by do not know that at this moment in the earpiece he hears prompts: “Car, distance – 5 meters. Man, distance – 3 meters. “
Another person comes to the MFC. “Hello,” the operator says with a smile, and this word appears on the screen facing the visitor. The man smiles back and starts typing the problem on the keyboard, which the operator will read on the computer monitor. An intermediary between the hard of hearing and the outside world is a small device that converts speech into text.
Both devices are developed and manufactured by Sensor-Tech. Its director Denis Kuleshov began to cooperate with organizations for disabled people while still at the institute: he helped conduct research. Gradually Denis realized that he wanted to work in this direction and develop for people with disabilities. Therefore, when in 2014 he was offered to join the scientific research of the new So-Unification foundation, he immediately agreed.
The So-Unification Foundation for the Deaf-Blind Support was established in April 2014. Deals with the socialization of the deaf-blind: opens regional support centers, creates opportunities for employment, leisure and creative self-realization; provides legal, psychological and targeted assistance.
The fund has accumulated various projects, including youth and scientific, related to the issues of deaf-blindness, in which Denis took part. “At some point, it was decided that it would be much more effective and more correct to develop this in the format of a separate laboratory that deals with these issues in a specialized manner,” Kuleshov recalls.
The nuances of household appliance
In 2017, the Sensor-Tech laboratory appeared, the founder of which was the So-Unity Foundation. Immediately, work began on the creation of gadgets for the hearing impaired and the blind. A grant from the Scientific and Technological Initiative was allocated for this within the framework of the Neuronet roadmap.
The condition for receiving the grant was co-financing of the project: 70% was allocated by NTI, the rest of the money had to be found by the developers themselves. The missing 30% was provided by the So-Unification Foundation.
The work of the laboratory had to be built from scratch: a design bureau and a pilot production were formed, where prototypes of devices were assembled. The first problem faced by the developers is the lack of experience in creating household appliances. Their previous experience in instrumentation was mainly related to the defense industry.
The first prototype of the device for the blind turned out to be too bulky and inconvenient to use. Having received feedback from users, the developers have seriously transformed it. It was necessary to make the device work without interruptions, be convenient and understandable to use.
Now the laboratory has developed a device for the blind and visually impaired – “Robin” and a device for people with hearing disabilities – “Charlie”.
“The market for devices for disabled people is not very large, all production is small-scale and expensive,” says Denis. This is the reason for the rather high price of the devices: “Robin” costs 150 thousand rubles, “Charlie” – 195 thousand rubles. Therefore, manufacturers are negotiating with the government to include their inventions in the list of technical rehabilitation means – in this case, the government will compensate for the purchase costs.
Tell me what you see
Outwardly “Robin” resembles a portable camera. Sensors scan objects, and artificial intelligence determines what is in front of the user – a car, a table, a computer, another person. There are more than 50 objects in the device memory. If you upload photos there, it recognizes the person and says his name. To use “Robin” in the dark, a flashlight was built into it. In this case, the gadget not only names the object, but also estimates the distance to it.
The first release of “Robin” was unsuccessful. In order for the device to detect an object, it was necessary to point the device, press the button and wait at least 2 seconds. But it turned out that this was inconvenient for users: immediately after pressing the button, they began to drive the device from side to side. Because of this, the information voiced by “Robin” turned out to be irrelevant: he was describing objects that he recognized before he was moved.
“We knew how to help the device and where to direct it, and people didn’t have that experience. Because of this, it seemed to them that the device was not working well, ”explains Alexander. Then the developers decided to change the principle of work. The feedback time from the device was reduced to a second, while the “Robin” only worked when the button was pressed. This option turned out to be clearer and easier for users.
“There are no direct competitors, that is, devices with the same functions, on the market. But there are analogues, for example, OrCam, ”says Alexander.
OrCam MyEye and OrCam MyReader are portable artificial vision devices produced by the Israeli company of the same name. They are small wireless cameras that can be attached to the temples. The first device reads printed and digital text, barcodes of goods, recognizes faces and reads the information received. The second is used only for reading texts. The gadgets support 17 languages and do not require an internet connection.
________________________________________
Voice turns to text
Charlie is for people with hearing impairments. People with experience in teaching for the hearing impaired participated in its development. Therefore, the developers were aware of the difficulties that arise in the process of communication and the requests of both parties.
The device picks up speech at a distance of 2 m and displays it as text on the screen. The gadget can also be used by the deaf-blind – for this, a Braille display is connected to the “Charlie”. It allows you to read information with your fingers and enter a response.
Braille was developed in 1824. It is a system of convex points and voids between them. Each character is encoded using a 3×2 trellis. The combination of dots within each cell corresponds to a letter or punctuation mark. If the text changes the writing, for example, from Latin to Cyrillic, this is also indicated by a special symbol.
The developers say that there are no analogues of “Charlie” on the market either. Numerous programs that translate speech into text are designed primarily for human-computer interaction, and not for two people to communicate with each other.
The first machine capable of recognizing voice appeared in the 1950s at Bell Labs. The device determined numbers from zero to nine. At the same time, the machine understood the inventor much better: when he spoke, the reproduction accuracy was about 90%. The device correctly recognized the speech of other people only in 70–80% of cases.
Until the 90s, speech recognition was based on templates: sound waves were translated into a set of numbers, and the result was displayed when the speech matched the sample. Therefore, for the correct interpretation of voice signals, it was required to eliminate background noise, speak slowly and clearly.
Dragon’s NaturallySpeaking was the first speech recognizer that did not require pauses between words. It appeared in 1997 and is still in use today.
Machine learning and AI technologies have significantly improved the speech recognition system and made it possible to adapt algorithms to the individual manner of communication of each person. The result was the emergence of voice assistants: Google Assistant, Siri from Apple, Alexa from Amazon, “Alice” from Yandex. In addition, there are special applications, for example, ListNote, SpeechNotes, for speech recognition, translation of a voice into a text message. Some applications, such as Speechlogger, can even perform simultaneous translation from one language to another.
With the help of “Charlie” the interlocutors will be able to communicate with each other without resorting to the help of an intermediary. In addition, the device can be used for remote communication, for example, for lectures at universities or holding meetings. It is enough for the presenter to put “Charlie” next to him, and the listeners – to connect to the program by the link. The decoded speech will be displayed on the screen of a smartphone or computer in real time.
See the world again
Also “Sensor-Tech” participates in scientific research related to the use of neurotechnologies for the return of vision.
It all started in 2016 at an international conference in the United States. There, representatives of the So-Unity Foundation met with an American company dealing with bionic vision. They agreed to conduct an experiment in Russia to implant bionic chips in blind people.
“The device may not be applicable to all blind people, it is a rather narrow pool of diseases, mainly retinitis pigmentosa. The peculiarity is that with it, only one layer of cells dies, which transforms light into an electrical signal. The rest of the cells remain alive, ”explains Denis.
Retinitis pigmentosa is an inherited disorder associated with the X chromosome. With this pathology, retinal cells are gradually destroyed, collecting the image and transmitting it along the optic nerve to the brain. The disease begins with loss of lateral and night vision and eventually leads to complete blindness. There are no effective preventive measures against retinitis pigmentosa, as well as treatment methods. The implantation of microchips is still under development, and doctors also suggest that gene therapy and stem cell therapy may be a breakthrough.
________________________________________
As a result, in the databases of the So-Unification and Art, Science and Sport foundations, they found two people with the right diagnosis, who did not have concomitant pathologies. In 2017, the implants were installed in Grigory Ulyanov and Antonina Zakharchenko. The funds for the operation were allocated by the Alisher Usmanov Foundation.
________________________________________
Alisher Usmanov’s Art, Science and Sport Foundation was founded in 2006. Finances educational and scientific projects (cooperates with MGIMO, MISiS), supports museums and theaters (Sovremennik, Tretyakov Gallery, Igor Moiseev’s ensemble and others). The foundation also promotes healthy lifestyles and organizes competitions, including for people with disabilities.
________________________________________
During the operation, an implant with electrodes is placed on the patient’s retina. It is associated with special glasses with a built-in camera. Information from the camera is transmitted to a microcomputer, which processes the image and sends signals to the implant. Electrical impulses are transmitted along the optic nerve to the brain, where a picture is formed.
There are 60 electrodes in the implant, which can be compared to a 60-pixel picture – by the number of points that the implant stimulates. With its help, you can get an image of only large enough objects – windows, doors, tables, cars. Smaller objects may fall out of the “matrix” formed by the electrodes, or the data is insufficient to identify them.
Since the patient does not have cells responsible for color rendition, the image is black and white. But in comparison with complete blindness, even such limited vision from the point of view of a healthy person significantly expands the capabilities of patients. They can orient themselves in space even in unfamiliar conditions without outside help or additional devices and become quite independent.
Sensor-Tech hoped that the operation would make it possible to register implants in Russia and make them available within the framework of high-tech medical care. But a problem arose: the Ministry of Health considered that two operations were not enough to draw a conclusion about the safety of the implants. The Americans, in turn, refused to conduct further experiments, since there was no guarantee that this method would eventually be approved.
“In terms of eye implants, the story is over. But this is even good, because it has a continuation in the form of cortical implants, which are placed in the brain. This is where the visual processing of information takes place. Chips can be used for any kind of blindness, in contrast to the eye. This is a more high-tech solution, ”says Denis.
Already in 2021, the company presented the first development in a new direction. This is the first neuroimplant for the brain ELVIS in Russia, it helps restore vision to blind and deaf-blind people. The device has an implant that is installed in the brain, a hoop with two cameras to be worn on the head – they take over the function of the eyes and a microcomputer that analyzes the image and transmits it to the brain. The technology makes it possible to distinguish the silhouettes of objects, people and to understand the location of objects.
While the system is being tested on rodents, tests on monkeys will follow. In 2024, 10 blind volunteers are planning to install a cortical implant.
Cortical visual implants are placed in the cerebral cortex. They stimulate the visual areas, resulting in visual sensations. According to experts, this will allow changing the brightness of images and providing color rendering, which is impossible when using bionic implants. The first operations for the implantation of cortical implants were carried out back in the 70s of the last century. And in 2018, the American company Second Sight publicly installed modern neuroimplants for the first blind volunteers. The technology is undergoing clinical trials.
Scientific research and overseas markets
There are other developments among the laboratory’s cases. For example, a free application for mobile phones was developed together with a cellular operator to help blind people determine the denomination of banknotes. It is available on iOS and Android.
To find out the denomination of a bill, just point your smartphone camera at it. The AI will identify the banknote and name it. And if you point the camera at several bills at the same time, the device will immediately calculate their amount.
If the application is used by a visually and hearing impaired person who cannot hear the voice acting, the denomination of the bill can be recognized by vibration – for different denominations there are own modes.
If the phone cannot recognize the bill, it can be photographed and added to the database via the “Help for Developers” function. Usually the need for this arises when issuing new or rare commemorative notes.
In addition to the production of devices, Sensor-Tech is engaged in scientific research in the field of deaf-blindness. “We want to make it clear to healthy people how those who have impaired vision or hearing perceive the world in general,” says Andrey.
So, the laboratory created simulators of visual impairments: the See My World VR simulator and its mobile version SMW Pro. “Noise” is superimposed on the live image or on a static image. As a result, the picture is distorted according to the specific visual defect. For example, it becomes blurry or a black spot appears in the center of the image.
“We simulate the symptoms of how a person with various visual impairments sees. Literally: how to look through the eyes of a person who has cataracts, myopia, astigmatism, glaucoma, ”says Andrey. Also, using the simulator, you can trace the dynamics of the disease over time. This helps doctors and students to better study the clinical picture, and the patient’s close relatives – to literally look at the world through his eyes.
“To make the image as reliable as possible, the clinical picture of the disease is used to create it. Additionally, we clarify the result with the patient, so that he can confirm whether he sees so, if his vision is not severely impaired yet, ”explains Andrey.
Now developers want to promote their devices in the European and US markets. It is planned to put the gadgets on Amazon and, in addition, to reach government bodies and public organizations dealing with the problems of people with disabilities. Denis is sure that the market for applied developments will only grow. And the director of the Sensor-Tech laboratory sees his mission in making the life of people with disabilities easier thanks to technology.
