VisionSpeech to Braille

Welcome to VisionSpeech to Braille

An innovative project bridging accessibility gaps by converting speech and text into Braille.

Project Details

Our system utilizes advanced machine learning for real-time speech-to-text conversion and Optical Character Recognition (OCR) for text recognition. An Arduino UNO R4 Wi-Fi Plus microcontroller controls six solenoids that generate tactile Braille feedback. This provides blind users with an effective tool to access written materials independently.

Speech-to-Text Model

The speech-to-text model was developed using a combination of Convolutional Neural Networks (CNNs) for feature extraction and Recurrent Neural Networks (RNNs) with Gated Recurrent Units (GRUs) for sequence modeling. The training process included:

• Preprocessing the dataset, which included over 5,000 labeled audio samples.
• Converting audio into spectrogram representations for CNN feature extraction.
• Using GRUs to capture temporal dependencies in speech.
• Implementing the Connectionist Temporal Classification (CTC) loss function for accurate alignment.
• Training the model for 50 epochs on Kaggle’s GPU environment.

Hardware Implementation

The hardware system consists of:

• Microcontroller: Arduino UNO R4 Wi-Fi Plus
• Solenoids: Six small mini electric solenoids (12V)
• Transistors: IRFZ34N N-MOSFETs for efficient switching
• Diodes: 1N4001 flyback diodes for circuit protection
• Power Supply: Three EVE ICR18650-26V Li-ion Batteries (2550mAh)

How It Works

1. Text to Braille Patterns

Text is converted into Braille patterns using a predefined Braille dictionary.

2. Binary Encoding and Serial Communication

Braille patterns are mapped to their binary equivalent and sent via serial communication to the device.

3. Speech Conversion to Braille

Speech is first converted to text, then mapped to Braille. The Braille representation is converted into binary bits and sent to the Arduino for tactile feedback using solenoids.

4. Image to Braille Conversion

Images are processed through OCR to extract text. The extracted text is converted to Braille, mapped to binary, and transmitted to the Arduino for tactile feedback using solenoids.

Results & Future Work

The project achieved outstanding results:

• Speech-to-text model achieved a 90% accuracy rate.
• OCR text recognition achieved a 97% success rate.
• Real-time Braille output was generated with minimal latency.
• Final project grade: 97%.

Future work includes integrating a mobile application, improving the hardware for better portability, and implementing real-time camera OCR.

Meet the Team

Zaid Osama Saif

Contact

Email: z.saif1@gju.edu.jo

Email: zaidsaif1me@gmail.com

LinkedIn: linkedin.com/in/zaidsaif/