1.

INTRODUCTION

Parkinson’s disease (PD) is a relatively common illness, constantly progressing, evoking anxiety and depression resulting from significantly restricted independence. It is estimated that more than 10 million people suffer from the disease worldwide [1], mainly after the age of 50 [2]. Its dominating symptoms include muscle rigidity, rest tremor, slowness and postural disorders. They can lead to progressing gait disorders, hypomimia, extrapyramidal dysarthria and micrography. The disease is a result of the extrapyramidal nervous system disorders, with its grounds being the atrophy of brain stem cells and subsequent decrease of striatal dopamine amounts.

Contemporary PD diagnostics is based on clinical criteria, which do not take into account any additional examinations that could confirm the diagnosis [2]. Also the neuroimaging methods, such as CT or MRI, do not provide a sufficiently validated diagnosis due to the fact that the undertaken symptomatic treatment modifies their results. Such test however enable eliminating other diseases, which are different from Parkinson’s disease [2]. Usually a diagnosis is based on direct clinical image and history taking as well as on an identified coexistence of the aforementioned symptoms.

Currently the most renown and dominant clinical standard for evaluating the severity of Parkinson’s disease symptoms is the unified UPDRS scale (Unified Parkinson’s Disease Rating Scale) published in 1987 [4, 5]. It assesses 4 components in a 5-grade scale, with the upper ranges corresponding to more advanced disease stages. Based on the medical history and the conducted tests, it is used to determine the intellectual condition and mood disturbances (component I), everyday life quality (component II), motor functions (component III) and movement complications (component IV). Sometimes its scope, in the form of additional components V and VI, is expanded with two previously used scales, the “Hoehn&Yahr” and the “Schwab&England”. Thanks to the conducted validation process, UDPRS became a formal tool applied as the basic indicator in diagnostics, monitoring the progression and in studying new neuroprotective therapies. Such tests require good knowledge of disease symptoms and broad clinical experience of the examining physician. In clinical practice, it still is a very often case that the disease diagnosis is significantly delayed, especially if a patient is not quickly sent by a family doctor to a neurologist. It is estimated that misdiagnosis take places in approx. 10-25% cases [2, 6], especially in early stages of the disease development. The reason for that is the lack of symptom evaluation objectiveness during the conducted tests and the improvement of UDPRS results under symptomatic treatment, which generally excludes its application as an ideal disease progression marker.

2.

RELATED WORKS

A symptom, which is generally easy to evaluate, is the limb tremor attributed to Parkinson’s disease, experienced by about 75% of the patients [7]. Various clinical standards for its evaluation had been developed over the years, with the quantitative accuracy largely dependent on the individual experience of the physician. A good example is the widely applied spirography [8], which involved analyzing the shape of spirals hand-plotted by a patient (so-called Archimedean spirals). In order to facilitate assessing the tremor severity using this method, a set of reference spirals was developed, which corresponded to various disease stages and which were compared to the spirals plotted by a given patient. The method based on analyzing traditional handwriting on paper provided just slightly better disease assessment results. Primarily owing to the possibility of comparing the writing of the same patient with old letters and documents, it is sometimes still used [9].

Contemporary technical solutions supporting the diagnostics and evaluation of Parkinson’s disease include devices using motion sensors, primarily accelerometers, to measure tremor. The accelerometer-based data acquisition method, despite its simplicity, is currently considered the best one owing to high sensitivity of commonly available sensors (mainly MEMS sensors [10]), which ensure recording tremor even below the visibility threshold. At the same time, it does not require precise placing of the sensors near appropriate muscles, which is a critical requirement in, e.g. electromyography. Its disadvantage is the fact that the accelerometer, apart from measuring limb tremor, also reacts to the movements of the shoulder or even the entire body. In addition, the impact of gravity varies depending on changing orientation of the patient’s hand. By applying appropriate low-pass filtration, which does not affect the range of frequencies occupied by tremor, all these artefacts can be efficiently eliminated. This can be utilized to take measurements of rest, volitional and postural tremor [11], including with the use of mobile devices, which enable 24/7 recording [12, 13]. A significant supporting tool regarding tremor assessment, which has emerged in recent years, is using the screens of simple mobile devices or graphic tables offering not only the recording of test images, such as, e.g. the Archimedean spirals, but also the possibility of quantitative description of their development dynamics, demonstrated by the measurable pressing force, pen inclination angles and drawing speed [1, 9, 14].

The probability of tremors occurring in the course of various disease entities or, simply, the lack of tremor, contributed to the application of motion sensors also for measuring or detecting other modalities, such as postural instability [15] or abrupt gait slowness, called “freezing” [5, 16]. A separate category of the disease evaluation methods is comprised of analyses based on vocal data recorded using a microphone. Documented results of medical research indicating the influence of Parkinson’s disease on the process of breathing, phonation and articulation provide a broad range of possibilities to analyze recorded signals using modern methods [17, 18].

Therefore, in the light of the available literature, there is a need for a possibly comprehensive acquisition of data of the patient, which after processing will serve as support for the physicians in terms of the clinical tests commonly applied by them. This research paper is of basic character and is focused on searching for new description of patient behavior, useful from the medical practice perspective. It involved a test bench for non-invasive recording of multimodal data for the purposes of supporting the studying of primarily motor symptoms evaluated within component III of the UPDRS used in practice. The bench was developed and constructed in the Institute of Electronic Systems at Military University of Technology in Warsaw. The set of recorded data includes a face image in the visible light range and in infrared, video images of the limbs, spontaneous and forced speech samples and handwriting samples. The set has been used in clinical practice for pilot studies conducted in the Department of Neurology at Medical University of Warsaw, under the participation of medical personnel [19, 20]. The objective of the attempted research is to study the possibility of an innovative application for the artificial intelligence technology for early identification of patients with Parkinson’s disease.

3.

SUGGESTED APPROACH

The basic motivation behind constructing the test bench is the specificity of the clinical tests recommended by the International Parkinson and Movement Disorder Society [14], which shows that the tester should use the UPDRS scale to assess what “he or she sees”. Therefore, the developed test set includes equipment for expanding human perceptive capacity such as an infrared camera, a camera operating within the visible range, a microphone with a preamplifier, a graphic tablet and a portable computer, which acts as the controller and data integrator. The presence of an infrared camera was necessitated by the verification of a possible relationship between the occurring asymmetry of PD symptoms, and the differences in the temperatures between the left and right side of the body. An additional element of the set is an automatic traditional audio-video camera. Fig. 1 shows a block diagram of the developed set.

Figure 1.

Block diagram of the developed data recording set.

A Flir A65 infrared camera with a focal length of 25 mm, ensuring a 25° × 20° field of view was selected to record the thermal image. This camera is equipped with a microbolometer with a resolution of 640 × 512 pixels, which records electromagnetic radiation having a wavelength in the 7.5 to 13 µm range, and its maximum operating speed is 30 Hz. The temperature sensitivity of the device does not exceed 0.05°C, and its accuracy is ±5°C or ±5% of the reading (the greater of two values). The Flir A65 camera has been selected owing to its relatively small dimensions and the option to operate it from a computer, via a Gigabit Ethernet interface, which is also used to supply the camera through the PoE (Power over Ethernet) technology.

Visible radiation range image is acquired using the acA1440-220uc camera equipped with a Basler C125-0818-5M lens providing a 35° × 26° field of view, which ensures a certain reserve relative to the field of view in the used thermal imaging camera. The Basler acA1440-220uc camera image sensor utilizes the CMOS technology and enables recording a color image with a resolution of 1440 x 1080 pixels, with a maximum rate of 227 fps (frames per second). The device is equipped with a USB 3.0 interface used for supplying power and transmitting data between the camera and a computer. Small size of both devices used for image recording made it possible to install them on a single photographic tripod, which ensures a small axial distance of both camera lenses and enables observing an object from almost the same perspective, within two spectral ranges.

The Shure MX58 dynamic microphone connected to a computer via a USB interface and an X2u adapter was used for voice recording. This microphone records sounds over a range from 50 Hz to 15 kHz, and its frequency characteristics is configured especially for voice. Hence there is the gain in the middle section of the band and the attenuation of low frequencies. A cardioid directional characteristic of the device enables ideal isolation of the main sound source and efficiently mitigates background noise. The microphone has a relatively low sensitivity of -54.5 dBV/Pa, which is the distinguishing feature of dynamic microphones, but it is sufficient for recording voice at small distances.

Handwriting is recorded using a Wacom Intuos Pro large PTH-860 graphic tablet with a large working space in the size of an A4 paper sheet. The attached Wacom Pro Pen 2 pen enables the determination of pen location coordinates within the tablet surface, as well as the pressing force and inclination angles of the pen during writing. The maximum measurement frequency for all of the aforementioned parameters is 200 Hz. Data are transmitted between the tablet and control computer via an USB interface; however, the set also enables wireless connectivity using Bluetooth technology.

All of the aforementioned devices are operated from the level of a computer using Matlab software, with its graphical user interface shown in Fig. 2. The software panel contains four tabs corresponding to each individual set modality. The “Cameras” tab enables almost simultaneous recording of images in two spectral ranges using a traditional color camera and an infrared camera. The devices are triggered automatically in a sequential manner, which results in slight (usually imperceptible) delay in image acquisition by individual cameras. Thermal images are saved in MJ2 video files, with a resolution of 16 bits per pixel, whereas the visible radiation images - as MPEG-4. Hence collected image data can be used to assess a patient’s facial expression and check the differences between the left and right side of his/her body. Another tab, “Microphone” enables recording sound in monoaural mode, with a sampling frequency of 44.1 kHz and a 16-bit resolution, and saved as a WAV file. This modality enables archiving patient voice samples during spontaneous and forced speech.

Figure 2.

General view of a graphical interface of the software for operating the cameras, microphone and graphic tablet in the course of voice acquisition.

Yet another tab, “Tablet” enables measuring the parameters of the patient’s handwriting. All parameters that can be determined are recorded with the highest possible sample frequency of 200 Hz and saved, along with timestamps, as a binary MAT file. The last tab, “VIS Camera” enables the acquisition of a color image using the Basler C125-0818-5M camera, operating with a constant rate of 30 fps, using a full image sensor field. The recorded video stream is saved as an MPEG-4 file. Such an image recording method can provide data for the evaluation of upper limb agility of the patient. The developed software enables saving data for individual patients, in separate folders, with names being their unique identifiers.

An additional, autonomous element of the set is a Sony HDR-CX450 camera, which records 1920x1080 video image, with a frame rate of 25 fps and sound in stereo, with a sampling frequency of 48 kHz. This camera is used to archive the course of the test and record image for evaluating the lower limb agility of patients, their posture and gait.

4.

CHARACTERISTICS OF OBTAINED RECORDINGS

The developed procedure for collecting multimodal data assumes the same conditions of using recording equipment for all patients. During the recording of facial expressions and limb activity, a patient reporting for the examination at a dedicated doctor’s office sits on a chair, whose distance from the cameras, given the constant focal length of their lens, is constant and determined using markers applied on the floor. For recording gait, the patient covers a predetermined path, in a direction transverse to the axis of a camera’s view. Voice sampling is conducted with the help of an assistant, whose task is to position a microphone near the patient’s mouth. For the writing and drawing test, the patient sits by the table with a graphic tablet placed on its surface, so that the requested activities are performed in a position convenient for him/her.

4.1

Point III.2 of UPDRS – facial expression

Standard instruction for a physician regarding facial expression examination involves observing the patient’s face when silent and when speaking, in order to determine its masking (hypomimia) expressed by decreased eyelid blinking frequency and failure to fully close the mouth when resting. Examples of images, which are the baseline for data processing algorithms are shown in Fig. 3.

Figure 3.

Patient’s facial images recorded in the visible light range and infrared (patient without Parkinson’s disease symptoms).

The procedure of acquiring data for examining masking was additionally expanded with recording facial expressions in forced conditions. In the course of filming, the patients are asked questions forcing them to consider their answer; they are also asked to try and use their face to attempt various emotions associated with happiness, sadness, surprise, fear and anger.

4.2

Point III.4 of UPDRS – rapid finger movements

The clinical version of the test involves 10 taps with the fingers as fast as possible, using only the index finger and the thumb, and with the greatest possible movement range (amplitude). The evaluation involves the execution of the task by each hand, described by the number of motion regularity disturbances, slowness rate in a two-grade scale (mild and medium) and the amplitude reduction over time. The example static image shown in Fig. 4 represents a video stream providing a possibility to determine time waveforms, which enable an objective evaluation of the aforementioned features. In order to facilitate the process of generating a number of waveforms based on the recorded videos, the tips of selected fingers were taped with paper markers in blue and red. The patient is recommended to move the fingers with the face as the background, in order to prevent markers appearing with colored clothing items in the background. The activities are recorded only in the visible light range.

Figure 4.

Sample images recorded during the finger tapping test (left) and the hand movement test (right) for a patient without PD symptoms.

4.4

Point III.17 of UPDRS – rest tremor amplitude

During the clinical test, the patient should be calmly sitting on a chair for about 10 seconds, with hands placed on the support (not on the knees), and the feet placed freely on the floor. The task of the physician is only to determine the tremor amplitude relative to threshold values of 0 cm, 1 cm, 3 cm and 10 cm, in order to work out a numerical assessment indicator in a five-grade UPDRS. Examples of static images collected from video stream are shown in Fig. 5. This time, the task of the attached markers is to facilitate the process of developing time series from tremor. The applied thermal imaging enables studying the differences in the temperatures of the hands in the case of symptom asymmetry.

Figure 5.

Examples of images recorded in the course of evaluating the rest tremor amplitude (patient diagnosed with classic, left-sided Parkinson’s disease, thermal image shows a visible difference in the temperatures of the hands).

4.5

Point III.8 of UPDRS – lower limb agility

The patient should be in a sitting position during the clinical test. The testing activity involves placing a foot on the ground, in a comfortable position, and then raising it at least 10 times and stamping the floor as fast as possible and with the greatest possible range of motion. Just like in the other motor tests, the task of a physician is to determine the number of motion regularity disturbances, its slowing rate, and the amplitude reduction in the course of the test. Each limb is evaluated separately. Examples of images from a video stream recorded in the course of the tests are shown in Fig. 6.

Figure 6.

Sample images recorded in the course of evaluating the lower limb agility (patient without Parkinson’s disease symptoms).

4.8

Point III.1 of UPDRS – speech

The formal objective of this part of the clinical study is characterizing the patient’s speech in terms of the vocal strength, sound (prosody), articulation, repetition of syllables (palilalia) and cluttering (tachyphemia), which enable the examiner to subjectively develop the impairment degree, in a grade of 0 (no impairment) to 5 (severe impairment).

In order to objectively assess the speech, the recording procedure assumes using a test bench for recording the patient’s voice when reading two similarly themed texts, different in terms of the emotional undertone (optimistic and pessimistic text), double recording of the voice with prolonged phonation of the sound “a” over ca. 5 second, repeating the syllable “pa” and repeating predetermined words and sentences after the examiner. Examples of time waveforms recorded in the course of testing a healthy and sick person are shown in Fig. 7.

Figure 7.

Voice sample time waveforms for a healthy (left) and sick (right) person, recorded during the a phonation of the “pa” syllable lasting several seconds.

4.9

Point II.7 of UPDRS – writing

Information for UPDRS assessment within component II are acquired by the way of medical history taking, which in the case of evaluating writing comes down to asking the people from the patient’s social background about problems with reading his/her writing. Using a graphic tablet within the test bench opens completely new possibilities for objective recording and archiving of handwriting, and analyzing data inaccessible in the case of using traditional paper. The recording procedure determined by a team of physicians assumes, first of all, to sample handwriting in the form of a fivefold repetition of a sentence. Already the very preview of the notation enables determining a tendency to micrography, which occurs in sick patients (Fig. 8). At the same time, the digital form of the notation provides perspective for developing quantitative diagnostic parameters based on time series provided by the tablet. Similar expectations apply to the drawings created on the tablet by the patient. They include the aforementioned Archimedean spiral and a drawing containing the effect of a response to a certain kind of forcing, which involves guiding the stylus along a set contour. Exemplary graphics developed during the test are shown in Fig. 9.

Figure 8.

Exemplary handwriting sample recordings of a healthy (left) and sick (right) person recorded during the tests.

Figure 9.

Exemplary drawings created by a patient with a left-sided Parkinson’s disease - drawing of a circle along a set contour using the left (on the left) and right (on the right) hand.

5.

CONCLUSIONS

The presented material illustrates a system solution, which enables the execution of non-invasive quantitative measurements and archiving multimodal data describing the condition of a PD patient. The developed test bench was commissioned in clinical practice conditions and was used to conduct pilot recordings with the voluntary participation of patients suffering from Parkinson’s disease. The acquired data were used as a baseline material for fundamental research aimed at processing them into vectors of characteristic features, which could then be used in diagnostic, evaluating the effectiveness of various treatments and studying the progression of disease symptoms in individual patients, as well as for the purposes of statistical analysis covering a wider population [19, 20].