Estimating hand motion using contact forces as well

Almost all methods for estimating hand motion use kinematics alone: where the fingers are. When the hand touches something, that information is incomplete, because the object pushes on the fingers and changes the motion. This work adds contact forces to the state being estimated. On a musculoskeletal model of the hand, a Kalman filter fuses joint angles and fingertip forces, and trajectory estimation improves clearly when there is contact.

Context

The thesis was carried out at Politecnico di Milano under co-supervision with Robiolab, on the MyoHand model from MyoSuite, which simulates the hand with its bones, joints, muscles and tendons in the MuJoCo engine.

Reconstructing the motion of a real hand with a model of this kind yields quantities that are not measured directly, such as joint torques and muscle activations. That information is useful for rehabilitation, prosthesis control and teleoperation.

Method

Contact points. The model was modified to include contact sites on the distal phalanges, where forces can be applied and measured.

Estimator. Two observers based on the unscented Kalman filter (UKF) were compared. The first estimates joint positions and velocities. The second augments the state with contact forces. A constrained extension keeps the estimated forces within physical limits, that is, non-negative and below the sensor maximum.

At each step, the estimated state feeds an inverse dynamics computation that yields joint torques, and a quadratic optimization that finds the muscle activations producing them.

Baseline. The filter was compared against the inverse dynamics procedure applied directly to the data, without state estimation.

Validation with synthetic data

Error was measured as the per-point maximum error. The plots show the fraction of frames falling below each error threshold, so a curve that rises earlier is better.

Without external forces, the two observers behave the same. With forces growing linearly at the fingertips, the observer with forces in the state keeps all frames below about 20° of maximum error, whereas the purely kinematic observer only gets about 20% of frames below 40°.

Comparison with direct inverse dynamics gives a similar result.

Validation with real data

Kinematics were captured with stereo depth cameras and the MediaPipe hand landmark model. Forces were measured with force-sensitive resistors at the fingertips, following a protocol developed in the lab. An inverse kinematics stage fits the model to the measured points.

With real data and external force, the filter also outperforms direct inverse dynamics, although the difference is smaller than with synthetic data. Errors, measured in key-point position, are between 15 and 35 mm.

What is missing

The system runs in "soft" real time: it processes data at a useful rate, without deadline guarantees. An error of 15 to 35 mm at the key points is considerable at the scale of a hand, and part of it comes from the camera's depth estimation, which forces the hand to move slowly and stay facing the camera. Forces are measured only at the fingertips and in one direction. Experimental validation used few trajectories.

How it fits in Robiolab

This work is part of a collaboration on computational hand biomechanics, together with the lab's project on measuring hand kinematics and forces, from which it takes the experimental protocol. It extends to the hand and to contact the sensor-fusion line the group started with locomotion estimation. Estimating interaction forces is also a need shared by the lab's wearable devices, where knowing how much force is exerted on the body matters as much as knowing where it is.