High Degree of Freedom Hand Pose Tracking Using Limited Strain Sensing and Optical Training

The ability to track human operators' hand usage when working in production plants and factories is critically important for developing realistic digital factory simulators as well as manufacturing process control. We propose a proof-of-concept instrumented glove with only a few strain gage sensors and a microcontroller that continuously tracks and records the hand configuration during actual use. At the heart of our approach is a trainable system that can predict the fourteen joint angles in the hand using only a small set of strain sensors. First, ten strain gages are placed at various joints in the hand to optimize the sensor layout using the English letters in the American Sign Language (ASL) as a benchmark for assessment. Next, the best sensor configurations for three through ten strain gages are computed using a support vector machine (SVM) classifier. Following the layout optimization, our approach learns a mapping between the sensor readouts to the actual joint angles optically captured using a Leap Motion system. Five regression methods including linear, quadratic, and neural regression are then used to train the mapping between the strain gage data and the corresponding joint angles. The final proposed model involves four strain gages mapped to the fourteen joint angles using a two-layer feed-forward neural network (NN).