Real-Time AI-Powered Bidirectional Communication System for Translating Sign Language to Text/Speech and Vice Versa

Abstract

Deaf and hard-of-hearing individuals often face communication barriers because many existing sign-language tools are one-directional, hardware-dependent, and unsuitable for realtime use. This project presents a browser-based, real-time Bangla Sign Language translator that supports bidirectional communication between sign language and spoken or written Bangla. The sign-to-speech pipeline uses webcam input, MediaPipe Hand and Pose Landmarkers, landmark normalization, and a dense neural network trained on 60 BdSL word classes. The system applies sustained-sign confirmation and confidence filtering to improve recognition reliability. For the speech-to-sign direction, the application accepts typed Bangla text and browser speech input, normalizes and matches words or short phrases, and displays corresponding sign avatar images sequentially. The system is implemented as a client-side web application with mobile support, requiring no backend server or specialized hardware. Overall, the project aims to provide an accessible and practical assistive communication tool for Bangla-speaking users in education, healthcare, public services, and daily life.