Beat-aligned motor synergies and kinematic beat detection in street dance movements

Abstract

Dance is a rich artistic expression that combines intricate human movements with music, emotion, and cultural elements. However, the analysis of complex dance movements poses significant challenges because of the lack of comprehensive motion capture data and efficient computational techniques for feature extraction. In the current study, we present a novel time-dependent principal component analysis approach for extracting beat-aligned motor synergies from large street dance datasets. Unlike existing methods, our technique accounts for the temporal variability induced by music beats, enabling an accurate representation of dance motion patterns. The extracted motor synergies, capturing both spatial and temporal patterns across motion segments and beat durations, were analyzed to gain insights into motor coordination, consistency, similarity, and variability across different dance genres. This analysis facilitates the understanding of complex dance movements by summarizing them in a low-dimensional subspace, elucidating the common elements and coordinated modalities among various dance sequences segmented based on the timing of music beats. Furthermore, we demonstrated that kinematic beat detection was improved by leveraging the first motor synergy activation, enabling more accurate beat alignment and synchronization with the music, a crucial factor in dance performance and analysis. The enhancement of beat estimation accuracy was verified through cross-validation comparisons of beat alignment scores. This work offers a novel computational approach to analyzing and extracting meaningful patterns from complex dance motions for a deeper understanding of the motor mechanisms inherent in dance genres, enabling new insights into the intricate dynamics of dance movements and their relationships with music influences.

Introduction

Dance is one of the most intricate motor skills exhibited by humans [1], integrating whole-body motor coordination, music-induced movements, and synchronization to rhythmic stimuli. However, analyzing such complex movements remains a challenging and open issue in the field of biomechanical research. A substantial amount of research has been conducted to advance this topic, including studies involving human experiments [2, 3] and computational simulations [4, 5].

Most previous studies on dance biomechanics have focused on a small number of specific dance genres and have performed kinematic analyses to directly examine particular body parts or joints. As one of the most representative dance genres, classical ballet movements have been analyzed in many studies. For example, Imura et al. [6] estimated movement intensity from joint torques of both legs (supporting and gesture legs) during the fouetté turn. The results revealed that training the supporting leg enhanced the continuity of fouetté revolutions. Thullier et al. [7] analyzed multi-joint coordination patterns within the hip, knee, and ankle joints while executing foot-drawn ellipses in a horizontal plane. Their work demonstrated the utilization of body joint redundancy to maintain balance when performing intricate leg movement trajectories. In a study by Bronner et al. [8], the authors elucidated group differences in postural pelvic control and intra-limb as well as inter-limb segmental coordination by scrutinizing multi-joint ballet movements (développé arabesque). These disparities were found to serve as indicators for assessing skill levels in complex dance movements. Other studies have focused on hip-hop dance (or street dance). Sato et al. [9] extracted distinctive features of basic rhythmic hip-hop dance movements. They postulated that introducing a motion delay in certain body parts (such as the head or neck) might enhance dance performance. Bronner et al. [10] gathered hip, knee, and ankle kinematics to assess potential injury rates stemming from hip-hop dance steps. This research effort aided healthcare professionals’ understanding of the rehabilitation needs of hip-hop dancers following musculoskeletal injuries.

Several advanced studies on dance movements have delved into the analysis of motor coordination or synergy among many body parts or joints, specifically using dimensionality reduction techniques. The concept of motor synergy postulates that high-dimensional human motion can be well represented in a lower-dimensional space that is essential for effective motor control by the central nervous system (CNS) [11]. Such a hierarchical control framework offers a viable means to elucidate the mechanics behind synergistic motor activities [12]. For instance, Vincs et al. [13] assessed the consistency and variability of choreographed ballet and contemporary dance movements among dancers using low-dimensional motion representations obtained by principal component analysis (PCA). Taking a similar approach, Bronner et al. [14] used PCA to summarize gesture limb movements of both expert and intermediate dancers executing développé arabesque. The results revealed that PCA explained motion differences between expert and intermediate dancers, and among three different conditions: slow tempo, slow tempo with relevé (i.e., on tiptoe), and fast tempo. Toiviainen et al. [15] studied how music influences dance movements in response to different rhythmic patterns. The researchers used PCA to identify basic motor patterns that synchronize with different rhythmic layers in the music. The results revealed that these motor patterns can be synchronized simultaneously across multiple rhythmic layers in the music. In a subsequent study, Toiviainen et al. [16] introduced a time-frequency-based tensor decomposition method to analyze inter-stimulus and inter-participant disparities in music-induced movements while simultaneously considering group movement directions. In a series of preliminary studies [17, 18], we performed motor synergy analyses of street dance movements using PCA, and successfully demonstrated that a small number of PCs were able to characterize the similarities and differences of whole-body motor coordination between basic movements (choreographies) in three major street dance genres.

Despite these results, conventional methods for synergy extraction may not be suitable for investigating synergetic motor activities that are essential for rhythmic dance movements induced by music beats. Because PCA and related methods do not explicitly take music beats into account, the synergies obtained may solely represent motor activities that are not related to music beats. However, rhythms or beats represented by dance movements may naturally be associated with synergetic motor activities that are aligned in time with music beats. Thus, alternative methods are needed to effectively analyze motor synergies that characterize dance movements.

In the current study, we present a simple analytical approach for extracting beat-aligned motor synergies from dance movements based on time-dependent PCA (TD-PCA), and demonstrate its validity with a large-scale motion dataset of various street dance genres, obtained from the AIST Dance Video Database [19]. Our proposed method begins by collecting short-time motion segments between two proximal music beats, each normalized (resampled) to have the same length. TD-PCA was then used to extract several PCs individually for every normalized time point, which was expected to reveal synergetic motor activities at every particular offset to the beat timing. Both the synergy patterns and reconstruction levels with several PCs, varying over the normalized time between beats, characterize the similarities and differences among street dance genres. In the present study, we further evaluated our method in terms of the alignment between synergy activations and music beats. Both TD-PCA and PCA provide a novel approach for detecting kinematic beats [20,21,22], newly combined with the idea of motor synergies. Using cross-validation, kinematic beats detected with TD-PCA were shown to improve the alignment with music beats over that obtained using a previous method as well as PCA, quantitatively validating the use of beat-aligned motor synergies to investigate rhythmic dance movements.

Methods

Fig. 1

Featured body structure of dancers in SMPL format [23], including one root joint and 23 body joints. The marked points are motor joint positions. The orange lines represent the dancers'skeletons. The joints of the right and left limbs are abbreviated as “r” and “l.”

Data description, pre-processing, and featured body structure of dancers

Dance video datasets were obtained from the AIST Dance DB [19]. The database contains multi-camera motion videos of 10 street dance genres: “break,” “pop,” “lock,” “waack,” “middle hip-hop,” “LA-style hip-hop,” “house,” “krump,” “street jazz,” and “ballet jazz.” Thirty dancers (15 female and 15 male; age 20–35), with a minimum of 5 years of dance experience, were recruited. For each of the 10 genres, three dancers individually performed 10 basic choreographies in the genre. A dancer performed each choreography four times, with varying music tempos selected from 80, 90, 100, 110, 120, and 130 beats per minute to elicit different impressions for the same choreography. Thus, each of the 10 choreographies for a specific genre was performed 12 times in total. The dance videos were recorded using nine cameras for an average of 23 seconds per video, including a pre-roll and a post-roll of the actual dance performance. The recorded videos were then refined by precise editing to remove the unwanted parts and keep the main dance part with fixed 16 beats based on the algorithm for detecting the sequence of eight beat clicks that comes before the music in the raw recordings [19]. The refined video length of the main dance part varies depending on the tempo, where the average length was 9.25 seconds. The refined videos were used for our analysis. To visualize each choreography, 10 key frames per choreography were extracted by DeepLabCut [24] with K-means algorithms.

Kinematic variables in this database were estimated previously, and are publicly available in the AIST++ dance dataset [21]. The Skinned Multi-Person Linear (SMPL) model [23] was used to represent the human body, featuring 24 body joints, including a root joint (Fig. 1). The marked points in the figure indicate motor joint positions, and the orange lines signify the skeleton structure. The letters “r” and “l” are used to distinguish the right and left limbs. Middle body joints include the head, neck, chest, spine, belly, and root, while the lower limb joints encompass both right and left hip, knee, ankle, and toes. The upper limb joints include both right and left inshoulder, shoulder, elbow, wrist, and hand. Each joint encompasses three-dimensional rotations corresponding to 3D dance movements.

We first translated the joint angles in the “axis-angle format” into Euler-angle format. The angle data were then smoothed in time by Savitzky-Golay Filter to eliminate data noise caused by motion estimation, where a window size of 51 was used for filtering at a video frame rate of 60 fps and the order of the fitted polynomial was 6. Next, the joint angle time-series of each choreography was segmented into short segments based on the timing of music beats identified by librosa [25], which yielded 16 beats in each video. In this database, a total of 60 music pieces were used, for which we confirmed that librosa reliably detected music beats. Previous studies on music beat tracking have reported that librosa may not perform well for jazz music [26] and another modern approach such as Madmom [27] would be more reliable, whereas the 60 music pieces (even those for street/ballet jazz dances) did not include jazz music. To further validate, we calculated the F-measure [27] between the music beats detected by librosa and those by Madmom for every music piece used in each dance genre separately. The average F-measure for each dance genre was above 0.96, which well indicated that the two beat tracking techniques performed very similarly on the music pieces used in this database.

Each segment was resampled (interpolated) to have the same number (45) of time points. This allowed motor synergies to be calculated across thousands of dance movement segments as trials. An overview of the data processing methodology is provided in Fig. 2.

Fig. 2

Data processing at the general level

Identifying beat-aligned motor synergies by TD-PCA

Motor synergies are conventionally studied by linearly decomposing kinematic data vectors into meaningful components, for which PCA is a fundamental technique. In the context of the current method, the decomposition model is given by

$$\begin{aligned} \varvec{x}^{r}(t)=\sum _{s=1}^{S} \varvec{w}_{s} c_{s}^{r}(t)+ \varvec{\epsilon }^r(t), \end{aligned}$$

(1)

where $\varvec{x}^{r}(t)$ denotes the r-th instance (segment) of the vector timeseries of the n angular values ($n=72$), where the time was normalized between 0 and 1, sampled discretely at $t=1/(2T)+k/T$ for $k=0,1,\ldots ,T-1$ (with $T=45$). The basis vectors $\varvec{w}_{s}$ represent time-invariant spatial synergy patterns and the corresponding components $c_{s}^{r}(t)$ give their activations varying over both time and segments. $\varvec{\epsilon }^r(t)$ denotes the residual vector given the S components, where S denotes the number of synergies to be extracted. Given all of the segments, PCA identifies the set of orthonormal synergy vectors $\varvec{w}_{1},\varvec{w}_{2},\ldots , \varvec{w}_{S}$ so that the s-th component maximally explains the total “variance” of data vectors that are unexplained by the preceding $s-1$ components. Note that, because we do not subtract the mean of data vectors, the total variance actually means (a constant times) the sum of all the squared entries in $\varvec{x}^{r}(t)$ over all of the time points and segments.

The conventional approach essentially seeks motor synergies that are not particularly associated with music beats because the synergies of interest $\varvec{w}_s$ are supposed to be activated at arbitrary timings between beats. Here, we instead consider identifying beat-aligned motor synergies, each possibly associated with specific timing between beats, based on a time-dependent decomposition model, given by

$$\begin{aligned} \varvec{x}^{r}(t) = \sum _{s=1}^{S} \varvec{w}_{s}(t) c_{s}^{r}(t) + \varvec{\epsilon }^r(t). \end{aligned}$$

(2)

Note that the model is the same as Eq. (1) except that synergy vectors $\textbf{w}_{s}(t)$ now depend on the normalized time t. To identify the time-dependent synergy vectors $\varvec{w}_{s}(t)$, we used PCA in a time-dependent manner (i.e., TD-PCA). Specifically, given all of the segments, we applied PCA for every time point separately, so that the S synergies $\textbf{w}_{s}(t)$ at time t maximally explain the total variance (sum-of-squared entries) of data vectors collected only from the specific time point across segments. The overall procedure for beat-aligned synergy computation, including the beat-based segmentation and TD-PCA, is shown in Fig. 3.

Motor complexity and coordination from TD-PCA results

Given the solutions of TD-PCA (i.e., synergies $\varvec{w}_{s}(t)$ and their activations $c_{s}^{r}(t)$ for all t), the first S terms in the right-hand side of Eq. (2) approximately reconstruct data vectors $\varvec{x}^{r}(t)$ in the left-hand side, with the reconstruction error (i.e., total residual error) minimized for each time t. The reconstruction error at time t is succinctly given by

$$\begin{aligned} E_t^{2}:=\Vert X_t-W_t C_t\Vert _{F}^{2}, \end{aligned}$$

(3)

where $X_t$ is a data matrix collecting data vectors (column vectors) at time t from all of the segments, the S columns in $W_t$ are synergy vectors at time t, and the corresponding S rows in $C_t$ give their activations for all of the segments; $\Vert \cdot \Vert _F$ denotes the Frobenius norm (i.e., the sum of squared entries). Dividing by $\Vert X_t\Vert _F^2$ then subtracting from 1, Eq. (3) yields a measure of reconstruction level, given by

$$\begin{aligned} R_t^{2}=1-\frac{\Vert X_t-W_tC_t\Vert _{F}^{2}}{\Vert X_t\Vert _{F}^{2}}, \end{aligned}$$

(4)

which takes a value between 0 and 1, with $W_t$ and $C_t$ obtained by PCA.

A higher reconstruction level $R_t^2$ by TD-PCA (with a fixed small S, typically $S=1$) implies that the kinematic data vectors, collected from all of the segments, can be more effectively represented by a few synergies locally at time t. As a function of t, the reconstruction level was therefore used to illustrate how the local kinematic complexity varies between beats for any specific motion collection (e.g., each genre). Taking the average of $R_t^2$ over time, we also evaluated the global complexity of given motion collections (e.g., for comparing between genres).

Fig. 3

The procedures for synergy computation using TD-PCA: 1. Dance movements are segmented in alignment with music beats. 2. PCA computations are performed on each slice of the collected segments at a normalized time point. 3. Kinematic beats are estimated with the activation of a motor synergy (principal component; PC), selected by cross validation, and subsequently assessed in terms of beat matching

Motor coordination between body modules was also evaluated based on the TD-PCA solutions. Here, the symmetric joints on the right and left sides were treated as identical body modules to simplify interpretations. Given a TD-PCA solution (synergy vectors) $W_t$, we took the root mean square of three (single joint) or six (two bilateral joints) values within $W_t$ for each module (per column of $W_t$ and at each time). Subsequently, each column was normalized to possess a unit norm. Note that this process also disambiguates the signs of synergy vectors that are arbitrary in PCA. These condensed synergy scores were employed in the forthcoming section to scrutinize the modular coordination of dance movements, as illustrated in Figs. 6 and 8.

Kinematic beat detection on the basis of synergy activations

Kinematic beats can be identified as the points in time at which movement drastically slows down, characterized by a sudden decrease in movement magnitudes or a pronounced change in movement angles, between adjacent poses [20]. Specifically, kinematic beats can be estimated by the local minima of the kinetic velocity curve [21], where kinetic velocity is calculated as the Frobenius norm of velocities across joints.

In a previous study [21], the alignment between music beats and kinematic beats was evaluated with the beat-alignment (BA) score, given by

$$\begin{aligned} \text{ BA } =\frac{1}{\left| B^x\right| } \sum _{i=1}^{\left| B^x\right| } \exp \left( -\frac{\min _{\forall t_j^y \in B^y}\left\| t_i^x-t_j^y\right\| ^2}{2 \sigma ^2}\right) , \end{aligned}$$

(5)

where $B^x=\left\{ t_i^x\right\}$ and $B^y=\left\{ t_j^y\right\}$ represent the time points of kinematic and music beats, respectively; we set $\sigma =3$ in this study.

Here, we present a novel approach to detecting kinematic beats based on synergy activations. Given a particular synergy vector $\varvec{w}$, selected appropriately, we first computed corresponding synergy activations by taking inner products between $\varvec{w}$ and data vector (joint angles) $\varvec{x}(t)$ at every time point, then finding the local minima of the absolute value of its first derivative over time. The synergy vector $\varvec{w}$ may be obtained using either PCA or TD-PCA. In the case of PCA, the first PC's synergy vector can simply be set as $\varvec{w}$. In the case of TD-PCA, it is further necessary to select one of the time-dependent first PC synergy vectors $\varvec{w}(t)$ before taking the inner product. In the present study, we evaluated every synergy vector $\varvec{w}(t)$ on each motion sequence specifically using the BA score as above, and selected the one that maximized the average score over all motion sequences in each genre. The underlying assumption of this method is that the synergies of maximum beat-alignment may arise at a particular timing (offset) between beats, depending on dance genres rather than specific choreographies in each genre.

We evaluated both the original method and our novel synergy-based method for kinematic beat detection using a “leave-one-choreography-out” cross-validation on the BA score for each genre. In each cross-validation run, we obtained a synergy vector (as above with either PCA or TD-PCA) from all of the motion data for nine choreographies, and evaluated the synergy vector's BA score on those of the remaining one choreography. This process was repeated 10 times and the mean of the resultant 10 average BA scores was evaluated for each genre. For comparison with the original method, we visualized the rate of change in the average BA scores from the original $\textrm{BA}_\mathrm{{orig}}$ to the synergy-based $\textrm{BA}_\mathrm{{syn}}$, i.e., $(\textrm{BA}_\mathrm{{syn}} - \textrm{BA}_\mathrm{{orig}})/\textrm{BA}_\mathrm{{orig}}$. Additionally, we conducted a similar comparison between PCA and TD-PCA using the rate of change from PCA to TD-PCA. The $95 \%$-confidence intervals of the rate of change in the average BA score are also displayed as shaded regions in Figs. 9, 10, 11 below.

Results

Global and local complexities of dance movements

First, we evaluated the reconstruction accuracies using TD-PCA to measure the local (i.e., time-dependent) and global complexities of dance motions. Local complexity was assessed through the reconstruction level $R_t^2$, taking the value between 0 to 1, with a constant number of PCs (synergies) at every time point of TD-PCA; global complexity was computed as the average of local complexity over time. See Section for details.

The global complexity was compared across different genres, with the number of synergies varying up to 10 (Fig. 4). The results revealed that, only with 10 motor synergies, the method was able to reconstruct over $80\%$ of the 72-dimensional dance motion data (of a particular timing between beats), on average. Elevated reconstruction levels indicate less complex dance movements, whereas lower levels signify higher complexity. In this regard, the two types of jazz dance genres (i.e., “street jazz” and “ballet jazz”) exhibited high complexity, where initial PCs achieved higher reconstruction levels for “ballet jazz” than those in “street jazz.” Three “old-school” dance genres, namely “break,” “waack” and “pop,” roughly appeared to be the second most complex genres, followed by a group, including “middle hip-hop,” “LA-style hip-hop,” and “krump.” The “lock” and “house” dances were the two least complex genres.

Fig. 4

Global complexity, i.e., average of time-dependent reconstruction accuracy over the 45 time points, versus the number of synergies (PCs) used for the reconstruction. Different colors indicate the 10 genres

Fig. 5

Local complexity, i.e., time-dependent reconstruction accuracy, using the first synergy (PC). Different colors indicate the 10 genres

Local or time-dependent complexity analysis with a few PCs may reveal how and at which timings between beats the motion data are dominated by the small number of corresponding synergies, as shown in Fig. A1. The timeseries of local complexity with only the first PC is shown in Fig. 5. Corresponding to the global complexity at one synergy in Fig. 4, “street jazz” and “break” generally exhibited low reconstruction levels with a gradual increase followed by a decrease between beats. In contrast, “house” and “lock” generally exhibited higher reconstruction levels, with strikingly different patterns in their waveforms; “house” exhibited an approximately concave pattern with the peak around the center between the beats while “lock” exhibited an approximately convex pattern with high levels around the timing of beats. Intuitively, the approximately concave and convex patterns may correspond to the postural simplicity (dominated by a single synergy) at the middle of two proximal beats (“upbeats”) or at the timings on the beats (“downbeats”), respectively, potentially indicating two contrasting ways to get into the rhythm with synergetic body motions.

The other dance genres, except for “pop” and “ballet jazz,” also exhibited approximately concave patterns, while the magnitudes of their changes were smaller than those for “house” or “lock.” Although it was not very clear, “pop” exhibited an approximately convex pattern with a similar shape to that of the curve of “lock,” which may be related to the similarity between the two old-school dance genres. Notably, “ballet jazz” exhibited an approximately flat pattern (or weakly convex pattern) compared with the other genres, indicating that the postural complexity in this dance genre is not strongly associated with specific timings between music beats.

Beat-aligned motor synergies for different genres

Fig. 6

Time-dependent motor synergies of joint angles in 10 genres presented in a modular format

Fig. 7

Spatial motor synergies of joint angles in 10 genres, computed through PCA, presented in a modular format

Fig. 8

Time-dependent motor synergies of joint angles in 10 genres presented in a modular format

Next, we analyzed in more detail the beat-aligned motor synergies, obtained by TD-PCA on the time-normalized motion segments between beats. As shown in Fig. 4, the first PC (synergy) already exhibited a high contribution (approximately 30–55%) in every genre, and the second PC also exhibited relatively large increments compared with subsequent PCs in many genres, indicating the relative significance of the first two synergies among all 72 synergies. Thus, here we focus on the first two PCs in each genre to illustrate how our TD-PCA reveals the common and distinct features between genres. The key frames for the 10 choreographies for each genre are also given in Supplementary Material (Fig. A2) to aid an intuitive understanding of the actual motions.

The first two PCs obtained by TD-PCA and those obtained by conventional PCA are shown in Fig. 6 and Fig. 7, respectively. For each PC, the corresponding synergy pattern was summarized by taking a root mean square of the entries within each body module including either one or two (bilateral) joints and then normalizing the vector of resultant values to have unit norm. As seen in the figures, the relative contributions by the 15 modules were similar between TD-PCA and PCA. For instance, in either method, PC1 represents relatively homogeneous coordination among modules, with notable differences in the root (pelvis) and elbow contributions between genres. In contrast, PC2 emphasizes several specific modules, such as the root, elbow, and knee. Beyond conventional PCA, our TD-PCA method further revealed time-dependent contributions of each module between music beats. For example, the elbow's contribution in the PC1 for “middle hip-hop,” as well as “house,” was the strongest around the normalized time of 0.75, although in this visualization, the unbalanced overall contributions by the 15 modules hinder interpretations of each module's time-dependent contributions.

To observe each module's time-dependent contribution more clearly, we horizontally normalized the fluctuations of each module in Fig. 6 so that the summation over all the normalized time points was one, as shown in Fig. 8. Notably, in PC1, three modules (the pelvis, head, and elbow) clearly exhibited time-dependent contributions that differed among the 10 genres. For example, a strong time-dependent contribution of the root (pelvis) was seen in “break,” “waack,” “street jazz” and “ballet jazz,” while a further time-dependence of the elbow, in addition to the pelvis, was seen in “pop” and “LA-style hip-hop.” The time-dependence of the root's contribution was not strong in “lock,” “middle hip-hop,” “house” and “krump,” while a strong time-dependence of the elbow's contribution was still evident in those genres. A notable time-dependence of the head's contribution was seen in “lock.” Note that the actual timings when strong contributions were seen varied between genres. The pelvis contribution appeared to occur immediately after the music beat in “break” and “LA-style hip-hop,” while it appeared to be slightly delayed (within [0, 0.5]) in “pop” and “ballet jazz.” The pelvis contribution appeared to occur both just before and after a music beat in “waack” and “street jazz.” The pelvis module links upper and lower limb coordination, potentially playing a pivotal role in rhythmic whole-body movements synchronized with music beats. The different timings after (or around) the music beat may therefore indicate how the body rhythm, induced by music beats, tends to be represented by the pelvis joint angle with a genre-specific offset after a beat. In contrast, the elbow contributions varied within [0.5, 1] except for “pop,” potentially indicating their different roles from pelvis to express rhythms in dance movements. The contribution of the head in “lock” was almost at the middle of two beats (upbeats), possibly related to the way in which this dance genre tends to use head (upper neck joint) movements to express the characteristic jerky, robot-like motions (see Fig. A2 and example movies at the website of AIST Dance DB).

In PC2, the time-dependent contributions were seen more commonly across joints or modules, although more complex, meaningful patterns can still be observed in the figure, compared with the first PCs. For instance, the time-dependence of root and elbow contributions was mostly not very strong, except for “break,” “pop” and “LA-style hip-hop,” timed just before the music beats. Additionally, in PC2, time-dependence of head contributions was seen in “break,” “middle hip-hop,” “waack” and “krump,” which was not evident in the first PCs, except for “lock.” Notably, the timings of the strong contributions of the root, elbow, and head were complementary to those in the first PCs. Another notable finding is that the extremity modules, namely the ankle, toe, wrist, and hand, were strongly correlated in the variation of their contributions between beats, except for “ballet jazz.” It is likely that the extremity modules tend to be used jointly to represent rhythmic movements throughout each choreography, possibly resulting in highly correlated patterns in the time-dependent synergy weights. The exception of “ballet jazz” may indicate the intricate, non-synergetic control of the extremity joints in this dance genre. Further genre-specific findings could potentially be obtained by exploring the patterns for each genre in more detail. For instance, the contributions of the belly, spine, chest and neck in the second PC of “lock” suggest their mutually synergetic activities to represent the genre's characteristic (robot-like) body movements. Interestingly, the peak timing of contribution was earliest in the belly, followed by the spine and chest, and was the latest in the neck (followed by the head, as seen in the first synergy), reflecting the outward ordering of joints from the belly to the head.

Kinematic beat detection

Fig. 9

Cross-validation of the rate of change of BA scores calculated from music beats and synergy activation beats (TD-PCA) vs. music beats and kinematic beats (original)

Fig. 10

Cross-validation of the rate of change of BA scores calculated using music beats and synergy activation beats with respect to TD-PCA vs. PCA

Fig. 11

Cross-validation of the rate of change of BA scores calculated from music beats and synergy activation beats (PCA) vs. music beats and kinematic beats (original)

The accuracy of kinematic beat detection may quantify the synchronization between dance movements and music beats. To evaluate this approach, we used the beat alignment (BA) score as a measure of the alignment between the estimated kinematic beats and the music beats, and compared scores across different methods.

As shown in Fig. 9, the rate of change in average BA scores for TD-PCA's synergy activation beats was positive for 7 out of 10 dance genres (e.g., for genres like “break,” “LA-style hip-hop,” “krump,” and “street jazz,” the rate of change exceeded 0.2. In a few exceptions (e.g., “pop,” “middle hip-hop” and “waack”) the improvement was not apparent. This finding could be attributed to factors such as the specific dance style or the complexity of the music beats in these genres.

Furthermore, as illustrated in Fig. 10, the rate of change in BA scores from PCA to TD-PCA also exceeded zero for the majority of genres, indicating that TD-PCA's synergy activation beats better align with music-induced motions compared with the PCA approach. However, the improvement was not very large, as seen in the figure. In fact, Fig. 11 shows that even conventional PCA-based kinematic beat detection outperformed the original method. This means that the performance of kinematic beat detection can be substantially improved by introducing the concept of motor synergy with either conventional or time-dependent PCA. Note that both types of PCA were newly examined here as methods for kinematic beat detection.

Discussion

The current study presents a novel synergy-based approach for analyzing complex dance movements, offering important insights and implications in three key areas: movement complexity analysis, beat-aligned motor synergies, and kinematic beat detection. We first pre-processed the dance dataset, including data smoothing, Euler angle format transformation, and beat-aligned segmentation of trials. TD-PCA was then proposed to extract motor synergies from these beat-aligned dance segments. Analysis of the reconstruction accuracies revealed significant differences across 10 dance genres, indicating different levels of local and global movement complexity. Notably, the first motor synergy activation was leveraged for kinematic beat detection, with the rate of change of BA scores demonstrating improved accuracy compared with existing methods.

Movement complexity has been discussed in previous research. However, a unique aspect of the current study is our novel application of the synergy method to analyze complex movements using time-dependent motor synergy. In the current results, the relatively low reconstruction error of the first synergy implies a high degree of movement complexity in street dance. Dance postures may exhibit considerable variability between choreographies within each dance genre. Unlike simple movements, each choreographed dance movement also exhibits high postural diversity and variation across segments. The variability may be exacerbated by varying music tempos influencing movement speed, joint rotation speed, and choreography execution accuracy. Individual dancer skill and experience may also impact these variations. More detailed analyses of the source of variability and complexity will be examined our future research.

In the present study, we proposed TD-PCA to extract beat-aligned motor synergies for a better understanding of dance movements. Conventional PCA extracts spatial motor synergies that capture the coordination modes of individual body joints, but may fail to adequately represent the temporal dependencies inherent in complex, rhythmic movements like dance. Our TD-PCA approach overcomes this limitation by explicitly modeling the temporal variability induced by music beats, enabling a more accurate representation of the intricate temporal body coordination patterns that characterize dance movements. This novel methodology can be generalized to other complex movements with clear temporal dependencies, such as various sports disciplines.

To the best of our knowledge, previous studies have not extensively examined methods for improving kinematic beat estimation, despite its importance for interactive music-movement systems, performance analysis tools, and rehabilitation technologies leveraging rhythmic movements. Our results show synergy activation beats based on either conventional PCA or TD-PCA outperformed original kinematic beat detection in tracking music beats, underscoring synergies'effectiveness in capturing music-induced motions via low-dimensional decomposition. TD-PCA's improvement over PCA was minor but evident in some genres. Additionally, even our use of conventional PCA constitutes a new contribution in the context of kinematic beat detection. The main benefit of using TD-PCA was seen in its capability to analyze music-induced temporal variability in motor synergies, while a possibility of further enhancing beat tracking performance using TD-PCA remains (e.g., with better motion features or synergy selection strategies). Detailed exploration will be included in our future work.

While the current study focused on street dance genres, the insights gained from our synergy-based analysis could be extended to other forms of dance, as well as other complex, rhythmic movements in domains like sports, performing arts, and rehabilitation. By capturing the intricate interplay between movement complexity, temporal dependencies, and external rhythmic cues, our methodology holds promise for advancing our understanding of human motor control and coordination across a wide range of applications.

Conclusion

In conclusion, this study presents a novel approach for analyzing and understanding the complex movements of street dance through beat-aligned motor synergy patterns and reconstruction accuracy. The extracted beat-aligned motor synergies enabled a comprehensive evaluation of motor coordination, complexity, consistency, similarity, and variability in these dance forms. By understanding the underlying motor synergies and coordination patterns, dancers and coaches can optimize training programs, and improve overall performance. In future, we aim to develop novel approaches to further improve the accuracy of kinematic beat detection, leveraging the insights gained from this study. In addition, we plan to explore individual differences in dance performance through motor synergy analysis and develop advanced dimension reduction techniques tailored for complex dance movement analysis. This work lays the foundation for a deeper understanding of the intricate dynamics of street dance movements and opens up exciting avenues for future research in movement analysis, motor coordination, and performance optimization in the realm of dance and other complex sports domains.