Sign Language Recognition using Deep CNN with Normalised Keyframe Extraction and Prediction using LSTM

Abstract

Sign Language Recognition (SLR) targets interpreting the signs so as to facilitate communication between hearing or speaking disabled people and normal people. This makes communication between normal people and signers effective and seamless. The scarcely available key information regarding the gestures is the key to recognise the signs. To implement continuous sign language gesture recognition, gestures are identified from the video using Deep Convolutional Neural Network. Recurrent Neural Network- Long Short-Term Memory verifies the semantics of the gesture sequence, which eventually will be converted into speech. The problem of constructing meaningful sentences from continuous gestures inspired the proposed system to develop a model based on it. The model is designed to increase the effectiveness of the classification by processing only the principal elements. The keyframes are identified and processed for classification. Validation of sentences can be done O(N). The sentences are converted into voiceover to have elegant communication between impaired and normal people. The model obtained an accuracy of 89.24% while training over Convolutional Neural Network to detect gestures and performed better than other pre-trained models and an accuracy of 89.99% while training over Recurrent Neural Network- Long Short- Term Memory to predict the next word using grammar phrases. This keyframe-to-voice conversion, forming proper sentences, enthrals people to have harmonious communication.