Music Generation Based on Dancer Body Pose Detection
Project Description Heading link
Automatic human body pose detection is a video processing algorithm that has recently been developed and made available through open-source distribution. Through pattern recognition and machine learning, a computer can detect and recognize the human body skeleton’s position in three-dimensions. Our design product applies this algorithm to recognize and track the body poses of a dancer; then, according to the dancer’s movements, the system generates synchronized music in response. Note that this is opposite to the usual concept of “dancing”: here, the dancer movements control the music instead of responding to it. Because our system needs to accurately track the dancer in real time, we analyze 30 video frames per second; this processing load is manageable while providing the desired body positional data. We use the open source “Open Pose” algorithm to estimate a dancer’s body joint locations (including those of the ankles, knees, hips, etc.) in the two-dimensional image coordinate system. 3-D body joint localization is possible but was deemed unnecessary for the purpose of extracting the data we needed for music control; this way the algorithm was simplified and made to run faster. We apply statistical methods to process the 2D body joint data, producing rhythm (downbeat timings) and amplitude-of-movement parameters. These parameters then determine what music piece is played, control its speed and what sound effects are added. We used the SOLA (Synchronized Overlap-Add) algorithm to modify the loudness and rhythm of pre-stored music to synchronize it with the dancer. A variable amount of dancer’s control of the music can be specified. Our system gives dancers more freedom of expression, since their movements feed back to affect the music they are dancing to.