1
One-Shot Learning for Inertial Measurement Unit-Based Phone Gesture Recognition

Subramaniyaswamy Vairavasundaram1*, Indragandhi V.2, Arnav Jain1, Pavan Dheeraj1, Pragun Gurkhi Chetan1 and Srinath Chitrala1

1School of Computer Science and Engineering, Vellore Institute of Technology, Vellore, India

2School of Electrical Engineering, Vellore Institute of Technology, Vellore, India

Abstract

With the use of Inertial Measurement Unit (IMU) data, a method for mobile gesture recognition is presented in this work. By allowing users to design and use their own custom gestures, we overcome the limitations of current mobile apps that provide pre-defined gestures. This user-centric approach supports a wider range of personalized interactions and promotes a more natural mobile experience. We use a modified Siamese Neural Network, which is traditionally used for image classification, to accomplish this. In order to independently process the IMU data sequences recorded for a user-defined gesture and a live gesture recording, this network uses two identical 1D Convolutional Neural Networks (CNNs). The CNN creates an embedding that encodes the properties of the gesture by efficiently extracting the required features from the input data. Next, we test three potential similarity checking mechanisms to see how well they perform, a Dense Network, Euclidean Distance, and Cosine similarity. Our evaluation found Cosine similarity to be the most promising. This similarity function is used to compare the embedded user's customized gesture and the live recording. The system is able to classify live gestures in real time thanks to this efficient comparison of embedded gestures. Achieving real-time performance on mobile platforms is a key focus of our work. This is especially important for mobile deployment because it minimizes battery drain and ensures responsiveness.1D CNNs' low computational requirements and effectiveness in feature extraction achieved better results.

Keywords: Gesture recognition, one shot learning, similarity measures, embedding, convolutional neural network, siamese networks

1.1 Introduction

In today's world of increased human-computer interaction, user input methods for mobile computers have primarily focused on touchscreens and buttons. In this regard, we aimed to explore alternative input methods by leveraging the capabilities of Inertial Measurement Units embedded in modern smartphones. IMUs offer data on motion and orientation, which is commonly used for gesture recognition. This particular aspect has been explored by various smartphone brands that offer IMU-based gesture-based control for certain functions, such as turning on the flashlight or opening the camera app on a device [1].

Despite this potential, there aren't many applications specifically designed to record and classify custom IMU gestures and perform actions from them. In this work, we present a system that bridges this gap. Our objective is to develop a system that enables users to create personalized gesture-based controls for various uses on a device. Ideally, the system we developed should classify gestures even with minimal training data or setup from the user; this requires a One-Shot or Few-Shot approach. Next the gesture recognition should occur with minimal latency, to ensure a seamless user experience. Finally the system should achieve sufficient accuracy for everyday use.

1.2 Related Works

Device elements utilizing Inertial Measurement Unit (IMU) have already enabled gesture recognition operations [1]. However, despite the advantages they provide, conventional methods are only successful given they are fitted into large datasets, and this specific ability restrains usage when little information is available [3]. These one-shot learning techniques promise hope for how these challenges can be addressed.

Image datasets have been used to investigate the concept of one-shot learning in image recognition applications. Siamese Neural Networks were proposed by Koch et al. [2] for the case, but there were issues with the limited amounts of data that could be used to train these networks and their capacity for generalization. Prototypical networks [4] and matching networks [6], two other techniques that have been the subject of other recent research, appear to be promising; however, they may have issues with noise-related uncertainties in the real inertial measurement unit (IMU) input streams.

Deep learning methods have been applied to recognize gestures using IMU data. In their study, Suri et al. [3] attained accuracy by employing Deep Neural Networks (DNNs) although their emphasis was on refining the classifier rather than delving into one shot learning strategies. Another study by Kim et al. [1] introduced an LSTM centered method, which faced challenges, with overfitting owing to a scarcity of data. Other modalities such as surface electromyography (sEMG) have also been investigated. Ding et al. [5] achieved high accuracy using CNNs with sEMG signals, but their method requires high computational resources, limiting real-time applications on mobile devices like smartphones. While multimodal approaches combining different data sources like IMU and camera data have shown success, this research focuses solely on IMU data due to its prevalence in smartphones [7].

One-Shot Learning for EMG-based Gestures: The recent work by [8] demonstrates the feasibility of applying one-shot learning to gesture recognition using EMG data. However, their work does not address domain adaptation for IMU data and lacks ablation studies to analyze the contribution of each component in their proposed method. In conclusion, while deep learning approaches have shown promise for IMU-based gesture recognition, limited research has explored one-shot learning techniques in this domain. This research aims to bridge this gap by investigating the application of one-shot learning for hand gesture recognition using IMU data collected from smartphones.

1.3 Siamese Architecture

For our one-shot learning task, we chose 1D convolutional neural networks (CNNs) after carefully reviewing various literature. Although recurrent architectures such as LSTMs and GRUs are very good at identifying long-term dependencies in sequential data, the lack of gesture recordings per action class restricts the applicability of these architectures. These intricate models with many parameters tend to overfit this limited set of data, which results in mediocre performance on unseen gestures. Furthermore, the training of LSTMs and GRUs necessitates a large number of computational resources, which increases power consumption and latency on mobile devices with limited processing power. However, 1D CNNs are a good alternative since they can effectively extract relevant features from short clips, which is similar to how features are obtained from single gesture recordings. They are ideal for real-time gesture recognition on mobile devices because of their simpler architecture and fewer parameters, which also make them computationally efficient to run on a mobile GPU.

1.3.1 Design

Our system will work based on the Siamese Architecture inspired by Koch et al. [2]. The gestures will be recorded with a rotation vector sensor on an android device. This recording will be converted into an embedding of lower dimensionality, which will be saved during the user's recording of gestures.

During runtime, live input from the sensor will be embedded and compared with the saved gesture embeddings to check for similarity using cosine distance. This will be done iteratively for all the gestures to check for a close match. When this occurs, a corresponding user defined command will be executed (Figure 1.1).

Figure 1.1 Siamese network block diagram.

1.3.2 Architecture Modules

1.3.2.1 Input

We gather input from the composite rotation vector sensor defined in the Android Open-Source Project (AOSP). This gives us the rotation in the form of a unit quaternion (w,x,y,z), a 4D representation of absolute rotation. We sample this sensor arbitrarily at 100 Hz to later reduce if needed and set the duration for the gestures to be of 3 seconds, as longer gestures are impractical from a usage standpoint. This makes the array or matrix of floating-point values of shape (301,4) for a single gesture.

1.3.2.2 Dataset

We recorded 18 unique gestures at least 20 times each to create a dataset of 366 gesture recordings. Each recording is sampled at 100 Hz for 3.01s, which makes the dataset shape (366,301,4). We then created 2 pairs of recordings (one positive and one negative) for each of the 366 recordings, the positive pair contains recordings from the same class, and the negative pair contains recordings from different classes. This gives us our final dataset of shape (732,2,301,4) 732 instances of a pair of recordings of shape (301,3).

Table 1.1 Implemented embedder architecture.

Layer (type) Output shape Param # embedder_input (InputLayer) [(None, 301, 4)] 0 embedder_conv1 (Conv1D) (None, 299, 18) 234 max_pooling1d (MaxPooling1D) (None, 149, 18) 0 dropout (Dropout) (None, 149,...