Setting Parameters
With consideration for this hard-wired type of error and the associated corrections, the most common method of disentangling separate movements within a serial action from a position function in kinesiology research is to set a threshold for a change in velocity. In this way, a movement is defined as an acceleration followed by a deceleration, but only when the resultant velocity exceeds the predefined threshold. The idea is that if the velocity of an action alters by more than a certain amount, in any direction, then a new movement can be inferred.23 This method accounts for single movements that include graded accelerations or decelerations, permits new movements to be registered without a zero crossing in acceleration, and allows small re-accelerations to occur without necessarily registering a new movement.
The setting of the velocity threshold for a new movement can be one of the most important decisions to the calculation of the number of movements performed during an assessment. Consider, for instance, a small, controlled movement experiment in which a series of a known number of movements were counted under two different velocity thresholds for determining a new movement. In this experiment, a confederate performer was enlisted to slide a handle along a straight 25cm track with the goal of creating a single motion wherein the handle stopped gently against the stopper at the other end of the track. The confederate completed this action 10 times in each direction for a total of 20 known movements. These twenty movements were repeated 20 times while a motion capture device recorded the action. The device was the Imperial College Surgical Assessment Device (ICSAD), a custom software-hardware package that works to time stamp, filter, and digitize movement data by way of a Polhemus ISOTRAK II electromagnetic system (Polhemus, Colchester, VT, USA) with a positional resolution of 3mm from 1.5m away. Reports on the optimal operation of the ICSAD in medical education literature indicate a velocity threshold 15mm/s as appropriate for determining new movements.37, 38 As such, we analyzed our confederate’s movements with this velocity threshold, and for the purposes of the demonstration also at a velocity threshold of 7.4 mm/s. The results of our test revealed the ICSAD counted quite accurately at the 15 mm/s velocity threshold (21.9 ±2.01 movements), but that reduction of the velocity threshold to 7.4 mm/s had a profound impact on the accuracy of the count (28.8 ± 4.16 movements).
Although the setting of the velocity threshold for a new movement is one of the most important decisions to the calculation of the number of movements performed during an assessment, it is not one that exists in isolation. The identification of new movements must also be considered with respect to the choices that are made regarding data filtering. This is because the technologies of motion tracking systems are unable to differentiate signals from meaningful movements of the sensor from those that result from hand tremor or other sources in the environment. This idea is similar to the way an electrocardiogram signal that is generated by the heartbeat of a baby in utero will be interfered with by the heartbeat of the mother. As a consequence, meaningful signals must be extracted from a context of considerable noise before they can be analysed.
In most signal processing applications, including motion analysis, this is accomplished via a Fourier transform, which works to decompose a signal over time into its constituent frequencies. The history and logic that underpin these mathematics fall outside of the scope of this commentary; it sufficient to understand that the result is a frequency distribution.39 Given that we know that human movements occur a relatively low frequency,40, 41motion analysis techniques demand that a low-pass filter, which omits overly high frequencies, is applied, such that the total signal analyzed can be restricted as closely as possible with that that reflects the movement. The distribution is then transformed back so that the cleaned version is once again expressed as a signal over time. If the applied filter is not low enough or too low, then the frequency distribution will respectively preserve excessive noise or remove meaningful data from the final analyzed profiles. As such, the ability of the filter to accurately isolate the movement signals can interact significantly with the velocity threshold for new movements to have a major impact on the number of movements counted.
Consider again our small controlled movement experiment; however, note that our confederate’s sliding track movements were also measured with a second device: the VICON optoelectric system (Vicon Motion Systems, Lake Forest, CA). The VICON is an integrated 13-camera system that is capable of providing 6 degree-of-freedom digital position data for markers with an accuracy of 0.5mm from up to 16m away. Importantly, the VICON operates on the bases of custom MatLab scripts (MathWorks Inc., Natick, Massachusetts, USA) that allow assessors to pre-determine the parameters for data filtering. In this case, a conservative stance was taken and a low-pass Gaussian-Butterworth filter22 with 5 Hz cut-off frequency was applied. The same sliding actions, recorded with this device, under this filtering protocol revealed accurate recordings at both the 15 mm/s (20.6 ± 1.4 movements) and the 7.4 mm/s (20.9 ± 1.6 movements) thresholds.
Although the VICON provides greater spatiotemporal resolution than the ICSAD, the differences in these two devices to accurately count movements at the lower velocity threshold is attributable to differences in the data filter processes. Specifically, the algorithms that underscore the ICSAD operations also use a Fourier transform method to filter data; however, they make the filter cut according to a standard deviation metric for the frequency distribution rather than at an absolute frequency measure (for e.g., 5Hz). That is, the ICSAD determines the standard deviation associated with the resulting frequency distribution and then sets the filter cut point based on a pre-set magnitude of that value. The ICSAD used in the small experiment was set to its default filter setting of 2, which means that all signals associated with frequencies above two standard deviations below the distribution mean were removed from the function prior to analyses. In this regard, the ICSAD allowed more noise to be incorporated into the analyzed function. While this noise was insufficient to alter movement determinations at the more conservative 15 mm/s velocity threshold, it was enough to register an increased number of movements at the lower 7.4 mm/s threshold. This is because the filter methodology determines the cut-off more so by the noise inherent in the measurement context rather than the frequencies of the target signals.
The intention of this demonstration is not to highlight the ICSAD as an inappropriate motion capture device for technical skill assessment. Indeed, the ICSAD has been lauded throughout the surgical education and assessment literature for the validity of its metrics, its ease of use, portability, and ability to capture data without constraining operative performance. Furthermore, the point is also not to advocate for the particular thresholds and filters tested here. Rather the goal is to emphasize that the effective practice of motion analysis for skill assessment involves a sophisticated understanding of the way thata priori decisions about data collection and analysis interact to influence outcomes. In this regard, it is essential that any efficiency standard of competency that is based on number of movements is established with appropriately and consistently applied analysis decisions; the determination of which will undoubtedly require numerous concurrent and criterion validation studies across the spate of skills of interest. Moreover, the assessment of trainees by way of motion capture techniques will require standards for measurement, as a differently adjusted motion capture system at one institution could artificially inflate or deflate a learner’s performance relative to the performances of those at other institutions.