However, most of them require that hands are in the airand cannot interact with physical objects. State-ofthe-art techniques are able to accurately track hand motion and handle intricate geometric configurations with complex contact patterns among fingers in real-time. Recent years have witnessed rapid progress of hand pose tracking and hand motion analysis using consumer depth sensors. Keywords Fingertip Tracking, Finger Tapping Detection, Virtual Piano, RGB-D images, Human-Computer Interaction. Preliminary results show that our method can recognize most of the beginners piano-playing gestures in real-time for soothing rhythms. Using our prototype system, users can put their hands on a desk, move them sideways and then tap fingers on the desk, like playing a real piano. In contrast to the existing hand tracking algorithms that often require hands are in the air and cannot interact with physical objects, our method is designed for hand interaction with planar objects, which is desired for the virtual piano application. Then we use the trained random forest to estimate the 3D position of fingertips and wrist in each frame, and predict finger tapping based on the estimated fingertip motion.įinally, we build a kinematic chain and recover the articulation parameters for each finger. In the online tracking stage, we firstly segment the hand from the plane in contact by fusing the information from both color and depth images. We collect 7200 depth images covering the most common finger articulation for playing piano, and train a random regression forest using depth context features of randomly sampled pixels in training images. Vidya Vikas Institute of Engineering & Technolgy Karnataka, Indiaįaculty5, Department of Information Science and TechnologyĪbstractThis paper presents an efficient data-driven approach to track fingertip and detect finger tapping for virtual piano using an RGB-D camera. Department of Information Science and Technology

Virtual piano with labeled keys