1
SLRRT: Sign Language Recognition in Real Time

Monika Lamba1* and Geetika Munjal2

1Department of Computer Science and Engineering (CSE), The NorthCap University, Gurugram, India

2Amity School of Engineering and Technology, Amity University, Noida, Uttar Pradesh, India

Abstract

An application called Sign Language Recognition (SLR) can recognise a variety of non-identical letter movements and translate them into text. In the area of science and technology, this application is extremely significant. It can be used in a variety of machine learning-based applications, including virtual reality. The purpose of the chapter is to develop a convolutional neural network that will recognise the signs captured or focused from the video capture and in turn provide us with correct or accurate output based on text and to improve the accuracy of the real-time sign language recognition via scanning and detecting that would aid other physically challenged individuals. For all individuals who want assistance in communicating with the rest of society, it offers an offline application. In order to produce quick, precise results and to ensure that the material isn't lost during the evaluation process, it tries to evaluate gestures more efficiently. Real-time sign language recognition involves first identifying images from a video feed that has been acquired using a machine learning model, then identifying edges and vertices, and then determining the desired result using a convolutional neural network. This method will be carried out at runtime to obtain results continuously while creating sign language with very little wait time utilising the CNN model. Character identification will be easier with this approach, and sentences can be constructed with high levels of accuracy using fewer letters.

Keywords: Language recognition, real time, sign, convolutional neural network, machine learning

1.1 Introduction

Nowadays, technology has taken an advanced leap forward in terms of improvement and efficiency. One of the many technologies that have taken such steps is Real Time Sign Language Recognition. Sign language recognition is an application to detect various gestures of different characters and convert them into text. This application has a huge importance in the field of science and technology. It has different applications based on machine learning and even in virtual reality. There are various types of sign languages such as ISL (Indian Sign Language) [1], BSL (British Sign Language) [2], ASL (American Sign Language) [3] and many more implemented differently at different parts of the world. Our aim is to apply American Sign language for sign to text recognition [3] [4] [5]. The American sign language is similar to other normal languages in that it can be expressed using gestures like hand or body movements. Although it shares many characteristics with other languages, it does not have English-like grammar. It is the most widely used sign language on earth. It is primarily used in nations like America, Africa, and much of southeast Asia. American sign language serves as a link between the deaf and hearing communities. They can textually describe their actions with the aid of this programme. This type of work has also been done in the past, with each instance producing unique outcomes and approaches, although few of them meet the standards for excellence.

The overall expansion of this language has been aided by its use in places like schools, hospitals, police stations, and other learning facilities. Since it is widely regarded as being simple to comprehend and fully independent of context, some people even choose to talk using this language. There are instances where newborn infants will receive this language from their mothers as their mother tongue. In fact, this is how sign language is meant to be understood. Figure 1.1 shows a visual representation of alphabets as signs.

Structure, grammar, and gestures are typically where sign languages diverge from one another. Unlike other sign languages, American Sign Language has a single-headed finger spelling alphabet. Compared to others, it is simpler to implement and interpret. The motions were also developed with consideration for various cultural traditions. Because people are accustomed to these gestures throughout their lives, this in turn draws a larger audience. The two-handed nature of BSL communication makes it difficult for non-BSL users to comprehend and interpret the language [5].

Figure 1.1 Basic sign language for each alphabet known characters.

The ISL is a well-known sign language in India; yet, because there are fewer studies and sources for accurate translations and because ASL has a larger audience, many individuals prefer ASL to other sign languages [6]. The ISL also has numerous identical motions with different meanings, which can be confusing when translated, even though all of these languages take roughly the same amount of time to translate letters and words. We needed ASL as the sign language converter because it is a more widely spoken language than English [7] [8].

The most fundamental need in society is for effective communication. Deaf and dumb people struggle greatly every day to communicate with regular people. Because those who are deaf or mute need to have their proper place in society, such an application was desperately needed. They experience secondary issues like loneliness and despair as a result of their basic weakness; thus it would be preferable if they could integrate more socially and forge more social ties [9] [10].

People also frequently offer alternative solutions, one of which is, "Instead of utilising another language to communicate, why don't deaf people just write it down and display it instead?" This explanation may appear reasonable and enticing from the perspective of a person without this disability, but the people who are experiencing these challenges require human solutions to their problems. These people need to express their feelings and activities, which cannot be done solely through writing. Consequently, that is still another justification for our decision to make a contribution to the field of sign language [11].

The concept of delivering results in written form primarily enables us to communicate with those who lack the opportunity to talk or hear. A little ease in their life would be given to all the dumb or deaf people thanks to such an application. The happier these people will be sharing such a larger platform, the more such applications will be created and technology is enhanced.

1.2 Literature Survey

Technologies like speech, gesture, and hand are significant piece of HCI (human computer interaction) [12]. Gesture recognition has numerous applications such as sign language, robot control, and virtual reality. In the proposed method of Zhi-hua Chen [13], hand recognition is grounded on finger recognition and hence, it is more effective and uses a simpler rule classifier. The rule classifier used in real-time applications is highly efficient. The author used a simple camera to notice hand gesture rather than using data glove and special tape which are much more expensive. It includes fingers, hand detection, palm segmentation and hand gesture recognition. In the very first step, i.e., hand detection, the colour of the skin is measured using HSV model and the image is resized to 200 x 200. The output of this step generates a binary image in which white pixels represent the hand and black pixel represent the background. The next step is the segmentation of palm and fingers which is obtained with the help of palm point (center of pam), wrist line and wrist point. The labelling algorithm is applied to detect regions of fungus. Finally, hand gesture is recognized by counting fingers and identifying what figure. The dataset of 1,300 images is used to prove highly accurate results. The system takes 0.024 seconds to recognize a hand [13]. Zeenat [14] studied gesture as basically a form of non-verbal communication that involves gestures by which people communicate with each other. People can't communicate without gesture. They are the mode of communication which people use to communicate. Cooperation between people comes from various tactile modes like signal, discourse, facial and body articulations. The principle preferred position of utilizing hand signals is to connect with computer as an on-contact human computer input methodology. Hand gesture has eliminated the use of controlling of movement of virtual objects. One of the most broadly utilized examples for hand gesture recognition is data glove. Use of gesture recognition has also eliminated the use of data glove due to the expensive cost of gloves. There are three stages of gesture recognition: 1. Image pre-processing 2. Tracking 3. Recognition. The system developed was to capture the hand gesture in front of a web camera, which in turn would take a picture and then continue to recognize the reasonable motion through a specific algorithm. This paper fundamentally includes investigation and distinguishing proof of hand signals to perform suitable activities. Picture preparing is fundamentally an examination of digitized picture so as to improve its quality. EMGO-CV is fundamentally utilized for picture preparing. Emgu CV is a cross stage. Net wrapper to the Intel Open CV picture handling library. Permitting OpenCV capacities to be called from .NET perfect dialects, for example, C#, VB,...