Towards novel classification of infants’ movement patterns supported by computerized video analysis

Abstract

Background

Positional preferences, asymmetry of body position and movements potentially indicate abnormal clinical conditions in infants. However, a lack of standardized nomenclature hinders accurate assessment and documentation of these preferences over time. Video tools offer a safe and reproducible method to analyze and describe infant movement patterns, aiding in physiotherapy management and goal planning. The study aimed to develop an objective classification system for infant movement patterns with particular emphasis on the specific distribution of muscle tension, using methods of computer analysis of video recordings to enhance accuracy and reproducibility in assessments.

Methods

The study involved the recording of videos of 51 infants between 6 and 15 weeks of age, born at term, with an Apgar score of at least 8 points. Based on observations of a recording of infant spontaneous movements in the supine position, experts identified postural-motor patterns: symmetry and typical asymmetry linked to the asymmetrical tonic neck reflex. Deviations from the typical postural-motor system were indicated, and subcategories of atypical patterns were distinguished. A computer-based inference system was developed to automatically classify individual patterns.

Results

The following division of motor patterns was used: (1) normal patterns, including (a) typical (symmetrical, asymmetrical: variants 1 and 2); and (b) atypical (variants: 1 to 4), (2) positional preference, and (3) abnormal patterns. The proposed automatic classification method achieved an expert decision mapping accuracy of 84%. For atypical patterns, the high reproducibility of the system’s results was confirmed. Lower reproducibility, not exceeding 70%, was achieved with typical patterns.

Conclusions

Based on the observation of infant spontaneous movements, it is possible to identify movement patterns divided into typical and atypical patterns. Computer-based analysis of infant movement patterns makes it possible to objectify and satisfactorily reproduce diagnostic decisions.

Background

Asymmetry of body position and movements, known as a positional preference, is a feature that is fairly common in the development of any infant [1]. The term “positional preference” refers to the temporal dominance of an infant’s head positioning to one side in a supine position [2]. The condition is defined as occurring when the infant 75% of the time has their head turned to one side while having difficulty achieving active rotation to the opposite side. In addition, a possible passive rotation to the non-preferred side is also indicated, so the range of motion may be limited [3]. Such a condition heralds a “fixation,” “holding of position,” or “dominance” of a particular pattern, resulting in a lack in a fundamental aspect of infant motor development, namely the variety and variability observed in infants' spontaneous motor skills. In clinical practice, the observed disorders (preferences, fixations, dominations) are always associated with physiotherapeutic treatment, as the lack of such intervention may result in abnormalities in motor development. According to Nuysink et al., in the Netherlands, about 5000 infants diagnosed with position preference are referred to pediatric physiotherapists annually [4].

Currently, available methods for infants’ neurodevelopmental assessment include several scales, based in particular on observation. Because they are primarily qualitative methods based on descriptive assessment, their outcome depends on subjective opinions based on the knowledge and experience of the specialist [5, 6]. Among the diagnostic tools used in both clinical practice and research are the Alberta Infant Motor Scale [7], Test of Infant Motor Performance [8], and Rapid Neurodevelopmental Assessment [9]. However, most of them do not focus on assessing motor patterns, both qualitatively and quantitatively. Another method is the General Movements Assessment (GMA), which allows early detection of neurological dysfunctions based on observation and qualitative assessment of an infant’s spontaneous movements in a supine position [10, 11]. Although many scales and methods are available for the neurodevelopmental assessment of infants, new approaches targeting specific aspects of development are still being investigated [12, 13].

Computer-aided infant diagnosis is currently the subject of much scientific research, particularly using computer vision and artificial intelligence methods to analyze video recordings of infant movements. The basis for conducting research in this area is most often GMA, which allows the assessment of motor patterns based on video recordings of infants’ spontaneous activity. The results indicate that the proposed solutions are highly effective, especially for screening cerebral palsy in high-risk infants [14,15,16]. Unlike other measurement techniques, video recordings are fully non-invasive and do not affect the infant's mobility. It is essential for the possible application of the proposed tools in clinical practice in the future.

With the current state of knowledge, asymmetry can be viewed as a potential predictor of an abnormal clinical condition of the infant [3, 4, 17, 18]. On the other hand, asymmetry or positional preference are still issues that are currently poorly understood in the context of assessing neuromotor development in infants. The previous studies have not clearly named the issue that is associated with the phenomenon of asymmetry in infancy. There is no precise definition that encompasses the asymmetry in muscle tension, body shape, and the relative positioning of different body parts, nor the dynamics of their movements considering their asymmetric functioning. Additionally, the various types of this phenomenon have yet to be clearly distinguished. The literature found a nomenclature and taxonomy closely related to etiology, with symptoms referring to “structural asymmetry” and “functional asymmetry” [19]. Another classification proposed by Nuysink et al. identifies “idiopathic asymmetry” and “symptomatic asymmetry” as being conditioned by environmental factors and the consequences of the pathological condition [4]. In each of the divisions, there is no mention of details regarding the specific alignment of movement patterns, which at the same time could be an indication of particular techniques in physiotherapy management. Information about structural or functional asymmetry, or idiopathic or as a consequence of a pathological condition, gives only a general indication of the need to implement treatment procedures. Therefore, it is necessary to develop a nomenclature model with a detailed explanation of the structural disorders of postural-motor patterns, which will give the opportunity to undertake specific techniques or methods in physiotherapy management. A detailed classification of postural patterns should consider their cause in the abnormalities of distribution and magnitude of muscle tension, the normalization of which is the foundation of treatment [20]. The described abnormalities may result from musculoskeletal [21] or sensory processing issues [22] or have a neurological basis resulting from brain damage or neurodevelopmental disorders [23,24,25]. Therefore, developing computer-assisted methods will help identify the disorder earlier, enabling further diagnosis and early intervention.

The lack of sufficient knowledge of nomenclature and taxonomy causes a fundamental difficulty in assessing and documenting the variability of positional preference over time, taking into account first and foremost the physiological dynamics of motor development in infancy. In clinical physiotherapy management, there is a need for a detailed description of infant movement patterns. Using video tools, on the one hand, guarantees wide-ranging safety and reproducibility during the recording and numerical description of the movement pattern. This specificity of the study makes it easier to plan closer and further goals of physiotherapy management and monitor emerging changes [26].

The aim of the study was to attempt to objectify the description of infant movement patterns, taking into account positional preference and their consequences in the form of dominant muscle tension distribution disorders. The focus of the research was on the development of an objective classification of the types of posture and movement patterns with a particular emphasis on positional preference, more broadly described in the literature as postural asymmetries [19, 27]. Taking into account the broad norm of the distribution and magnitude of muscle tension, as well as the variation and variability of motor patterns, the division of movement patterns implies a distinction between typical and atypical patterns, which are rare in the healthy infant's motor repertoire and are characterized by uncommon positioning of individual body parts. This approach will provide objectivity in the recording and monitoring of infant spontaneous movements. All the steps of the study were characterized by a close and inseparable link between two fields: pediatric physiotherapy (in terms of clinical diagnostics) and biomedical engineering. Only this approach allowed for the development of a tool that appears to help objectify infant assessment by classifying infant spontaneous movement patterns in the first quarter of life. The research questions were:

• 1. Is it possible to define characteristic movement patterns based on observations of infant spontaneous movements in the supine position?

• 2. Does the use of methods of computer analysis of video recordings allow for accurate and reproducible assessment of the occurrence (classification) of individual movement patterns?

Methods

Dataset

The research material consisted of video recordings of infants collected using an author’s measuring station with a camera mounted one meter above the infant. We used an HDR-AS200V camera (Sony Corporation, Tokyo, Japan), allowing acquisition at 60 fps and a 1920 by 1080 pixels resolution. The infant was inside the stand in an unconfined, supine position. Before the observation, they were awake and fed. Recordings were conducted in a home setting and included three videos of about 20 min in length over 2 days at different times of the day. We collected 151 recordings in a group of 51 infants. The recordings were collected between February 2019 and July 2020.

Eligible for the study were infants whose parents gave informed consent to participate in the study (after informing them about the purpose and course of the study), infants between 6 and 15 weeks of age, born at term (either by normal delivery or Caesarean section), with an Apgar score of at least 8 points.

The study was approved by the Biomedical Research Ethics Committee (No. 5/2018) and in accordance with the Declaration of Helsinki. The study was approved by the Bioethics Committee of Research of the Jerzy Kukuczka Academy of Physical Education in Katowice. All parents of the participants gave written informed consent before data collection began.

Classification of infant motor patterns

Five pediatric physiotherapy practitioners with qualifications and at least 10 years of professional experience, recognized as experts, were invited. The purpose of the invitation was to express an opinion based on experience and a review of the literature on the nomenclature and taxonomy of postural patterns with a particular focus on the positional preference of infants.

The experts jointly selected one recording for detailed analysis. Different variants of postural patterns presented by the infant were separated. The next step was, based on the selected recording, to define the characteristic patterns of the infant's position and movements.

The relationship between the trunk position and the selected head position (left, symmetry, right) of the infant was proposed. The starting point was to characterize typical variants of control of movement patterns. At the face-to-face meeting, based on a literature review, experience, and analysis of a video of a selected infant, the experts analyzed sets of patterns related to the position and movement of the infant’s body, and, if necessary, changed the rank of their decisions in light of the joint discussion. Within the patterns observed in the videos, the two most characteristic postural-motor systems generally recognized as normal were identified. The first was called ‘symmetry’ and the second was ‘typical normal asymmetry’, with its neurodevelopmental basis being asymmetrical tonic neck reflex (ATNR) [28]. The observed variability in correct head positioning in ATNR required experts to distinguish two characteristic variants of typical motor pattern asymmetry.

All variants of postural-motor patterns that did not meet ATNR criteria were considered. The experts assumed that any deviation from the postural-motor system described as typical postural-motor patterns would be considered atypical patterns. In the next step, using the timeline, the most characteristic moments of the particular pattern on which the infant did not present the ATNR pattern were analyzed and grouped as four different, characteristic variants of atypical motor pattern asymmetry.

The experts also noted that both mentioned patterns are characterized by diversity, variability, and different variants, which provides the basis for classifying them into the group of normal patterns. They indicated that despite regularity as a common feature of the mentioned patterns, the basis of abnormality in the form of positional preference is the presence of fixation or dominance of a given pattern in time.

Computer-based analysis of movement patterns

The proposed division of postural-motor patterns, along with definitions of pattern-specific criteria, allowed for the development of a computer-based fuzzy inference system (Mamdani fuzzy expert system). The system's role was to automatically identify the pattern presented by an infant at a given moment in the video based on a set of rules derived from definitions developed by experts. The input data set was the following torso features’ described by Ledwon et al. [29]: head and trunk bend angles, and nose point distance. The values of these features were automatically determined for each frame of the video based on the positions of the human pose landmarks resulting from the OpenPose library [30].

Individual input variables were described by fuzzy membership functions in the form of bell functions reflecting the values indicated by the expert as ‘small’ or ‘large,’ taking into account the ‘right’ or ‘left’ directions. The parameters of the bell functions were empirically selected to maximize the similarity of the obtained results to the experts’ indications on the sample recording. The output variables were functions defining the probability level of each of the 7 classes (low and high): symmetry, two variants of typical and four variants of atypical asymmetrical motor patterns. Next, a set of 33 rules binding the input variables to the resulting pattern classification was created. The classification for each frame of the video yielded 7 values from the range of <0, 1> representing the probability of classifying the presented infant’s body position as one of the seven variants of postural patterns.

Evaluation and statistical analysis

The parameters of the functions describing the fuzzy variables were subjected to automated tuning based on the created dataset. The set consisted of 140 randomly selected frames of the recording described by the expert in terms of the postural pattern category presented by the infant. A genetic algorithm in a five-fold cross-validation approach was used to tune the parameters of the function. An additional condition involving the parameters of the fuzzy functions was to maintain symmetry with respect to the value of 0 for each fuzzy variable.

The accuracy of the proposed algorithm in the automatic classification of postural patterns was verified as the percentage of patterns identified same as by the expert.

Then, we conducted an analysis of the patterns occurring in the study group used the percentage of each pattern type in the entire recording. The repeatability analysis of the results for recording the same infant three times was also carried out.

Based on the results obtained for each recording, an analysis of the reproducibility of the percentage of each pattern on the recording was carried out using statistical analysis. Each video moment was labeled with a position name as described previously. Therefore, the percentage of each infant’s position in the video was obtained for each video. A Multiple Proportions Test or a two-proportion Z-test was performed for each infant, depending on how many videos were analyzed. The significance level was set to alpha = 0.05. The analysis was conducted independently for 3 recordings and for 2 recordings collected on 1 day.

Software

For the estimation of infant pose landmarks, we utilized the OpenPose library [30]. Feature extraction and visualizations were achieved through custom Python scripts employing the NumPy, Pandas, and Plotly packages. The implementation and tuning of the fuzzy inference system were performed in MATLAB using the Fuzzy Logic Toolbox. Statistical analyses were carried out using the R programming language in the RStudio environment.

Results

The study involved the recording of videos of 51 infants who met the inclusion criteria verified before qualifying for recording. Due to the failure to meet the video recording assumptions, 2 infants were excluded from further analysis. The flow of participants and videos through the observational study is illustrated in Fig. 1.

Fig. 1

Design and flow of participants through the trial

Based on the analysis of the literature, the clinical experience of experts, and video data, the infant movement patterns were adopted as presented below:

Normal movement patterns: movement patterns without damage to the central nervous system and structural damage, or mild neurodevelopmental disorders:

• typical symmetrical: this pattern is characterized by the symmetry of the position of the head, torso, and limb movements. It is characterized by a symmetrical distribution of tension in a general pattern in which the alignment of the head with the midline of the body determines symmetrical control of the upper body and upper and lower limbs: symmetrical control of the head and trunk relative to the midline of the body predominates (Fig. 2a).

• typical asymmetrical: this pattern reflects the distribution of tension typical of ATNR. Lateral flexion on the occipital side of the neck, upper body, upper limbs, and lower limbs predominates, with simultaneous rotation of the head. The infant’s head is rotated equally to both sides, which is a manifestation of the variation and variability of this pattern:

  • variant 1: the distribution typical of ATNR with head rotation and lateral flexion of the head on the side opposite to the rotation (Fig. 2b)

  • variant 2: the distribution typical of ATNR with head rotation without lateral flexion (Fig. 2c)

• atypical asymmetric (normal): other variants of asymmetry unrelated to typical ATNR. The variety of variants during a single observation is a manifestation of variability and indicates the normal function of CNS:

  • variant 1: head rotation and lateral flexion of the head on the side of the rotation (Fig. 3a),

  • variant 2: head rotation without lateral neck flexion with upper body shortening on the facial side (Fig. 3b),

  • variant 3: lateral neck flexion without head rotation; upper body—all variants (Fig. 3c),

  • variant 4: head in symmetry and shortening on one side of the body (Fig. 3d).

Fig. 2

Proposed classification of typical motor patterns of infants

Fig. 3

Proposed classification of atypical asymmetric motor patterns of infants

Positional preference: the basis for determining the patterns in which positional preference dominates will be their fixation in one of the variants. It does not matter whether the preference concerns a variant of typical or atypical asymmetry; any fixation/dominance is an indication for observation and intervention by a pediatric physiotherapist. Fixation in asymmetry can contribute to later health consequences for children diagnosed with the above disorder [3].

Abnormal movement patterns (characterized by disorders of neurological, orthopedic, and genetic origin: symptomatic asymmetry) are not the subject of this study.

The resulting accuracy of reproducing the expert’s decision was 84%. The results in the form of a confusion matrix are presented in Fig. 4.

Fig. 4

Confusion matrix comparing the system outputs (rows) to the experts decision (columns) based on the 140 video frames from the test set. The diagonal elements indicate the number of correct predictions for each class, while off-diagonal elements represent misclassifications. The color intensity is proportional to the number of observations in each cell. The columns on the far right of the plot show the true positive rate and false negative rate for each class, respectively. The rows at the bottom of the plot show the precision and false discovery rate for each class, respectively

The algorithm was used to classify individual frames of all the collected recordings. Next, based on the results, the percentage of each pattern in the entire recording was determined. The resulting distribution of the percentages of each pattern in the entire study group is shown in the form of box plots in Fig. 5. Symmetry and patterns of typical asymmetry prevailed in the recordings. Of the atypical patterns, variant 3 of asymmetry was most frequent, characterized by the occurrence of flexion with the absence of head rotation. Other variants of atypical asymmetry accounted for less than 5% of the observation time in most recordings.

Fig. 5

Box plot showing the distribution of the percentage of each pattern on the recordings of the observed group of infants

The results in the form of p-values are shown in Fig. 6. The results showed that for atypical asymmetry variants 1, 2, and 4, ca. 90% of the participants showed similar percentages of these patterns on all three recordings. For variant 3 of atypical asymmetry, repeatability was confirmed for 33 participants (70%). For typical patterns, the repeatability for all three recordings reached a value significantly lower than the atypical patterns: 22 (47%) for symmetry, 22 (60%) for variant 1 of typical asymmetry, and 33 (70%) for variant 2 of typical asymmetry. For these variants, the repeatability within the recordings collected on a single day was ca. 80%.

Fig. 6

P-values of differences comparing the percentages of occurrence of each pattern on recordings made on the same day and on all three recordings

Discussion

Objectification during the diagnosis and later monitoring of the quality of infant movement patterns can be an important factor in effective physiotherapeutic management, not only in the area of positional preference. In the observation of infant spontaneous movements, the indication of specific characteristics (captured in individual variants of movement patterns) can make it easier to define therapeutic goals. By classifying the patterns of the patient studied, the physiotherapist knows exactly how to use the methods learned to achieve these goals. In other words, when an infant presents a “positional preference in the pattern of typical asymmetry,” this means that the infant presents a fixation/dominance of tension distribution consistent with ATNR towards one side of the body, in which flexion tension on the occipital side predominates in the neck, upper body, upper and lower limbs, with simultaneous head rotation. Such a pattern, left without physiotherapeutic support, will give rise later to the development of abnormalities in the form of asymmetrical rolling, sitting, and crawling patterns and even postural defects in the preschool and school years [17]. Nuysink et al. report that in the Netherlands, about 5000 infants per year diagnosed with “positional preference” are referred to pediatric physiotherapists [4]. The challenge, however, remains in describing and recording abnormalities in infant movement patterns, which poses a major challenge in monitoring the progress of ongoing physiotherapy.

To date, the nomenclature and taxonomy related to positional preference have been linked in the literature primarily to etiology. Michalska et al. argued that infant asymmetry is a clinical condition in which abnormalities in body structure, body posture, and motor skills are observed [19]. However, the researchers failed to specify the exact characteristics associated with the body position. They found that asymmetry is characterized by a variety of etiologies, locations, and degrees of intensity and that the symptoms of asymmetry may involve the body's structures (structural asymmetry) and/or functions (functional/motor asymmetry). It can manifest locally, i.e. affect a specific area of the body (local asymmetry) or the entire body (global asymmetry). Furthermore, Nuysink et al. proposed a division into idiopathic asymmetry and symptomatic asymmetry: idiopathic asymmetry is likely determined by environmental factors (it is mild and related to the infant's preferred head position), while symptomatic asymmetry is a consequence of another pathological condition [4]. The nomenclature and taxonomy proposed as an outcome of our study extend the presented general divisions of infant asymmetry. The novel nomenclature model is a step further in explaining structural disorders, a better understanding of which will give the opportunity to undertake more effective methods in physiotherapy management.

An unambiguous, quantitative description of infants motor patterns made based on observation using the proposed classification system results in the physiotherapist's correct decision to work on the normalization of muscle tension, i.e., supporting central tension (in the head-trunk axis) with simultaneous relaxation/inhibition of tension on the occipital side, and elongation and dissociation between the shoulder and hip girdle on the occipital side by activating adjustment and balance responses and elongation [20]. The goal of physiotherapeutic management is always to normalize the distribution and magnitude of muscle tension in terms of the postural reflex mechanism. When dealing with different variants of motor patterns, techniques have to be individually targeted to the presented infant's positioning preferences. The developed nomenclature, with the help of a computer-aided system, allows objective monitoring of the progress of improvement treatment. Once the positioning dominance is clearly specified, it will be possible to apply precise physiotherapeutic management.

Our study is the first attempt to use fuzzy inference systems in automatic infant movement pattern recognition. However, research confirms the effective use of expert systems in infant computer-assisted diagnosis. Existing literature has confirmed their effectiveness in trunk segmentation on images of premature infants, analysis of MRI images of infants' brains, and automatic inference based on medical records history [31]. Other studies on computerized recognition of motor patterns in video recordings of infants’ spontaneous activity were mainly based on GMA and used artificial intelligence algorithms trained on the data instead of decision rules developed by experts [14]. Most of the studies aimed to automatically classify the infant's condition based on the video, without detailed results on observations over time. Reich et al. and Passmore et al. attempted to indicate the occurrence of individual patterns in time classified according to GMA, achieving 88% and 70% accuracy, respectively [32, 33]. In our study, the proposed approach achieved 84% accuracy in classifying defined motor patterns. Considering that the number of classes is larger than in GMA, this result is satisfactory compared to other studies in computer-aided neurodevelopmental diagnosis.

The results indicate that the recorded movement patterns of infants are not limited to symmetry and asymmetry (the tension distribution typical of ATNR). High motor activity in healthy infants was recorded. A characteristic of the movements of healthy infants is variation and variability of motor behavior, which can be manifested in a variety of patterns based on asymmetry [34, 35]. The proposed division into typical and atypical patterns provides an opportunity to observe all variants of movement behavior: symmetry, typical asymmetry (mainly associated with ATNR), and atypical patterns. Such a solution monitors all movement behaviors. The classification takes into account the term ‘positional preference’ already defined by van Vlimmeren et al., which means that the infant's head faces one way 75% of the time and cannot achieve active rotation of 180 degrees to the other side; passive rotation to the non-preferred side is possible but the range of motion may be limited [3].

We believe that the existing knowledge in the broad field of positional preference needs to be explored further, especially in the context of nomenclature and taxonomy. The difficulties in this field highlight the lack of consensus among researchers. Various authors have attempted to describe this problem while emphasizing the fact that the phenomenon is highly variable. Michalska et al. considered asymmetry a dynamic condition [19]. The transition from local to global asymmetry and from functional to structural asymmetry and vice versa is often observed, thus emphasizing that the precise identification of the etiology of asymmetry, including structural and functional relationships, is difficult. Also, van Vlimmeren et al. in their review relating to diagnostic strategies for asymmetry in infants, due to the wide spectrum of characteristics and multifactorial etiology of asymmetry, stated that this diagnosis does not offer a consensus on definitions, nomenclature, or classification [3].

Monitoring the objective recording of the characteristics of individual movement patterns is the second equally important aspect of the present study. This study demonstrated that the proposed algorithm, with the use of computer-based methods of the analysis of video recordings allowed for the accurate and reproducible indication of the classifications of individual positional patterns (as shown in “Results” section). The current limitation is the lack of a uniform system for objectively monitoring and recording the patterns proposed in the classification presented above.