Clinical detection and movement recognition of neuro signals

Clinical detection and movement recognition of neuro signals 1 first-author †E-mail:xwzhang@sjtu.edu.cn †E-mail:xwzhang@sjtu.edu.cn xwzhang@sjtu.edu.cn 1 1 1 1 2 2 2 Department of Automation, Shanghai Jiao Tong University, Shanghai 200030, China Department of Automation, Shanghai Jiao Tong University, Shanghai 200030, China Zhongshan Hospital, Medical Center, Fudan University, Shanghai 200032, China Zhongshan Hospital, Medical Center, Fudan University, Shanghai 200032, China Neuro signal has many more advantages than myoelectricity in providing information for prosthesis control, and can be an ideal source for developing new prosthesis. In this work, by implanting intrafascicular electrode clinically in the amputee’s upper extremity, collective signals from fascicules of three main nerves (radial nerve, ulnar nerve and medium nerve) were successfully detected with sufficient fidelity and without infection. Initial analysis of features under different actions was performed and movement recognition of detected samples was attempted. Singular value decomposition features (SVD) extracted from wavelet coefficients were used as inputs for neural network classifier to predict amputee’s movement intentions. The whole training rate was up to 80.94% and the test rate was 56.87% without over-training. This result gives inspiring prospect that collective signals from fascicules of the three main nerves are feasible sources for controlling prosthesis. Ways for improving accuracy in developing prosthesis controlled by neuro signals are discussed in the end. Neuro signal Intrafascicular electrode detection Movement recognition INTRODUCTION 1 Human limbs are complex, delicate systems. Their reconstruction, both for appearance and for function compensations, proves to be complicated projects. With the development of science and technology, both researchers and disabilities hope to have artificial limb much more perky, comfortable, dexterous and effective. Although thousands of prosthetic limbs are commercially available worldwide, most of them, from mechanical styles to electric styles, from motion controlled to EMG, or voice controlled, can’t meet the needs of users because of their poor performance properties. Degrees of freedom (DOF) that can be achieved are normally less than five (Lin et al., 1998 1998 ), far from practical needs because of the insufficiency of independent information channels. Control accuracy is not high enough because of signal sources’ poor duplication and their sensitivity to outside disturbance. Usually, instead of using prosthetic devices as parts of their body, the disabled seem to be operating machine systems. The above deficiencies pose a challenge to improve the properties in prosthesis design. More effective interface devices, new signal processing technologies, pattern recognition methods and intelligent control strategies have been explored and achieved much. However, attention is not being paid enough to possible signal sources for limb control. Researches in neurophysiology show that neural systems are reparable, not only adaptable and compensatory to outside stimulations, but also have the ability of recovery and reconstruction after impairment, both in structures and in functions. Moreover, the patterns of neural activity repeat very well because they are indifferent to tiredness and interruption. Researches also reveal that electrode chips with holes made from silicon may establish reliable connections with neural cells if they are tied with amputated neural fibers (Kovacs et al., 1987 1987 ), and can get stable and clear electric recordings on continuous discharges from neural fibers (Weber et al., 1999 1999 ). Such researches make it possible to use neural activity as the signal source to control prosthesis. Now, many studies have identified the relationships/associations between the discharge patterns of neural activity and arm movement parameters (e.g. velocity, direction and force). Humphrey ( 1972 1972 ) found that exact forearm movements can be predicted by weighted sum impulse vectors from a group of cortical motoneuron recordings. Schwartz et al.( 1988 1988 ) discovered that the rate at which a neuron fires in the motor cortex determines the direction the associated muscle will tend to move, and that hand velocity can be predicted online by impulses from cortical neural cells (Schwartz, 1994 1994 ). Burrow et al.( 1997 1997 ) successfully established the maps between cortical signals and forearm movements. These achievements make it possible to develop prosthesis controlled by neuro signal. Chapin et al.( 1999 1999 ) successfully trained rats to position a robot arm to obtain water by pressing a lever with simultaneously recorded motor cortex neurons. Wessberg et al.( 2000 2000 ) achieved long-term control of complex prosthetic robot arm, both locally and through the Internet, by simple real-time transformations of neuronal population signals derived from premotor, primary motor and posterior parietal cortical areas in primates. All the above works derived neuro signals with cuff electrodes, with the subjects being mainly animals instead of humans. Professor Chen Zhong-wei, academician of the Chinese Academy of Science and Engineering, proposed detection of the nerve signals by clinically implanting microelectrodes into the fascicule of three main nerves (median nerve, radial nerve and ulnar nerve) in the human being’s upper extremity. The development of microsurgery in China (Chen, 2002 2002 ) and successful experiments (Zheng et al., 2002 2002 ) on neuro signal detection with self-made intrafasicular electrodes on rabbits ensured that such idea is feasible, has sufficient fidelity, and is without any infection. This paper presents a clinical experiment on neuro signal detection achieve with microsurgery. Initial analysis on features under different actions was conducted and movement recognition on samples of neuro signals was attempted. SVD features extracted from wavelet coefficients were used to predict an amputee’s movement intentions with neural network classifier. The whole training rate was up to 80.94% and the test rate was 56.87% without over-training. Ways for improving accuracy in developing prosthesis controlled by neuro signals are discussed in the end. NEURO SIGNAL DETECTION 1 The self-made intrafascicular electrode was made from 95% platinum and 5% iridium with diameter of 60 μm, and insulated with 5 μm Teflon. Before the experiment, approximately 10 mm of insulation at the distal end of the electrode were removed by heating to prepare the electrode for implanting in the fascicle, and 1 mm of insulation at the proximal end was removed to allow for attachment to recording equipment. Then the electrode was penetrated through a silastic column with diameter of 2 mm and length of 10 mm and bounded together with cyahoacrylate. Here the electrode is not attached with a hard tungsten needle in order to lessen the damage to the tissue. The subject was male, 31 years old, with left upper limb amputated two years ago. Two weeks neuro rehabilitation was implemented before the experiment to enhance the stability and reliability of neuro signals. After full anaesthesia, the upper limb’s median nerve, radial nerve and ulnar nerve were extricated for about 5 cm length; then two motor fascicules of each nerve were selected to implant the intrafascicular electrode directly by microsurgery into the fascicle of the nerves, as shown in Fig. 1 1 . Another electrode was placed outside the fascicle and parallel to the intrafasicule electrode and used as reference electrode. The signal was recorded with an EMG Instrument “Paseidon NDI-500” with sampling frequency of 5 kHz. After reliable connection had been established between the electrodes and the fascicule, the subject was waken up and did actions such as hand open/close (HO, HC), wrist’s pronation/supination (WP, WS) and flexion/extension (WF, WE), altogether 6 movements under control of consciousness. Then the neuro signals were recorded real-time for further processing. INITIAL FEATURE ANALYSIS 1 Each sampled signal had the length of 100 ms (500 points), altogether 32 groups of data were recorded, in which 26 groups were measured under action condition, and the other 6 groups were under relaxation condition. Each group had 4 channels, a total of 104 nerve signals were recorded. Fig. 2 2 gives a typical example of neuro signals recorded from the fascicule of radial 2 under hand open action (to avoid losing information, the signals were not filtered further). Initial analysis of the features of neuro signals showed that (Zhang et al., 2004 2004 ): 1. The average absolute value of signals from the radial fascicule was 5.5 μV, with standard deviation of 0.8 μV, while values of signals from the ulnar fascicule and medium fascicule was 2.5±0.4 μV and 2.2±0.3 μV respectively. Further neurophysiology and physiologic anatomy research showed that the radial signals detected from the implanted position were almost pure motor signals, while ulnar fascicular signals and medium fascicular signals were mixed with a lot of sensational signals. This showed that motor signals are more intensive than those of sensational signals, at least in the detected position. 2. The power spectrum of the fascicular signals was distributed almost throughout the range of [0, 2.5] kHz, wider than that of myoelectricity. Perhaps the sampling frequency should be increased in the next experiment. 3. When an action occurs, the radial signals from electrode 1 and 2 differ little, but signals from the ulnar fascicule and medium fascicule differed obviously; the difference was probably caused by the sensational information from receptors in the ulnar nerve and medium nerve, not by the motor information coming from the brain. 4. The high-frequency component increases as the applied force increases, but the signal intensity differs little. This showed that the discharge rate of the neuro impulse had close relation with the applied force. 5. Antagonism, the typical feature of myoelectricity, was not found in neuro signals, either in different nerves or in different fascicules of the same nerve. MOVEMENT RECOGNITION 1 In the following part, movement recognition was tried to test whether the detected fascicular signals (samples) are suitable for prosthetic control. Since they are collective and continuous, the same classification method as that for myoelectricity was adopted. Signal processing 2 Considering that the samples were not enough, segmentation was used first. Each signal was divided into 5 segments with 128 points, where 35 points overlapped each other, as shown as Fig. 3 3 . Thus, 25 samples were obtained for every HO, WE and WF; 10 samples for each HC and WP, and 20 samples for WS. Then, white noises with 5% signal noise to ratio (SNR) were added to increase the samples 10 times for each action, so that 250 samples were available for HO, WE and WF, 100 samples were available for HC and WP, and 200 samples were available for WS. From them, 3 to 5 were randomly selected as training samples, and the other were used as test samples. Signal analysis 2 Signal analysis aims at separating information relevant for the recognition task from irrelevant information (e.g. sensation or noises) and at reducing the amount of data presented to the neural network classifier. Considering the on-time requirement and flexibility, wavelet transform (WT) was selected instead of Wigner distribution or short time Fourier transform because of its local and multffi-scaled outlook as well as its sensitivity to breaks. Wavelet decomposition can be realized by the following Mallat Algorithm (Mallat, 1989 1989 ): $$\left. \begin{gathered}c_k^0 = f \\c_k^j = \sum\limits_n {c_n^{j - 1} \bar h_{n - 2k} } \\d_k^j = \sum\limits_n {c_n^{j - 1} \bar g_{n - 2k} } \\\end{gathered} \right\}{\text{ (}}k = 0,1,...,N - 1{\text{)}}$$ where f is the signal to be analyzed, with length N , c k j and d k j being scaling coefficients and wavelet coefficients under scale j ; h ̄ n and g ̄ n being the pulse response of conjugate mirror filters. In wavelet analysis, different basis functions may be suitable for different signals, and appropriate selection of the wavelet basis for signal representation can result in maximal benefits. It is reasonable to think that if a wavelet contains enough information about a signal to be represented, the wavelet system can be simplified in terms of the level of required resolution, which reduces the computational complexity of the problem to be implemented. In general, wavelet basis selection is related both with the signal to be analyzed and the implementation. As for neuro signal classification, the index for basis selection should be the representation effectiveness of selected feature vectors after wavelet transformation. However, this method is not suitable here since the original samples were not enough to provide convincing results. One simple way is to choose a basis available after some comparison, although such a result is not optimal. In this experiment, Daubechies 4 wavelet was selected by comparing the decomposition level required while keeping the energy as much as possible. Recursive applications of the above Mallat algorithm led to the decomposition of the neuro signal into a matrix of sequences (Pittner and Kamarthi, 1999 1999 ), as shown in Fig. 4 4 (here three scale decomposition is adopted). The shadowed part is filled with zeros. Feature extraction 2 The objective of feature extraction is to produce a suitable representation of neuro signals for movement recognition. Many methods can be used to form a feature vector according to the above matrix results (Zhang et al., 2003 2003 ). Here average absolute coefficients, scaling energy, maximum coefficients, singular value decomposition (Zhang et al., 2002 2002 ) and 4-order autoregressive (AR) coefficients are used for comparison. Average absolute coef.: $$AVG_j = \frac{1}{K}\sum\limits_{k = 1}^K {\left| {c_{jk} } \right|}$$ Scaling energy: $$ENER_j = \frac{1}{2}\sum\limits_k {c_{jk}^2 }$$ Scaling maximum: $$MAX_j = \mathop {\max }\limits_k (\left| {c_{jk} } \right|)$$ SVD: $$SVD = \sqrt {\lambda (C^{\text{T}} C)}$$ AR coefficients: $$x_n = - \sum\limits_{k = 1}^p {a_k x_{n - k} + e_n }$$ where c jk is the k th wavelet coefficient in scale j , λ ( C T C ) means to calculate the characteristic roots of the matrix C T C ( C is the coefficient matrix formed in the above wavelet decomposition). x n is the signal to be analyzed, a k is the k th coefficients of AR model, e n is white noise, and p is the total orders. Thus, after feature extraction from signals of radial 2, ulnar 1 and medium 1, five groups of vectors with length of 4×3=12 are available. To compare their ability in representing neuro signals, the cluster-separation index (CSI), or Davies-Bouldin index (Davies and Bouldin, 1979 1979 ), a measurement that is related to the performance of the linear Fisher Discriminant classifier of pairwise clusters, is introduced here: $$CSI = \frac{1}{K}\sum\limits_{i = 1}^K {\mathop {\max }\limits_{i \ne j} (R_{ij} )} = \frac{1}{K}\sum\limits_{i = 1}^K {\mathop {\max }\limits_{i \ne j} (\frac{{S_i + S_j }}{{D_{ij} }})}$$ Where K is the total classes, R ij is an index that measure the similarity between class i and j , S i and S j describes the scatter degree of class i and j , D ij is the average distance between i and j . They can be calculated as follows: $$S_i = \left\{ {\frac{1}{{N_i }}\sum\limits_{j = 1}^{N_i } {(x_j - m_i )^{\text{T}} (x_j - m_i )} } \right\}^{1/2}$$ $$D_{ij} = \{ (m_i - m_j )^{\text{T}} (m_i - m_j )\} ^{1/2}$$ Here N i represents samples of class i , x j is input sample, m j is the average of input of class i . The CSI gives the overlap degree between different classes, and has been used widely in classification problem (Kermani and Wheeler, 1992 1992 ). The smaller the CSI is, the more the classes separate easily. The CSI value of each feature vectors is shown in Table 1 1 . Features AVG ENER MAX SVD AR CSI 26.1028 39.3547 15.2584 8.8363 11.2392 Clearly, as samples of this experiment are concerned, SVD features are better for neuro signals representation than other features, so they were selected as the inputs for further neural network classification. Fig. 5 5 is a scatter plot of two SVD features from radial 2 under six different movements (white noises are not added here). It can be seen that the separation ability between these two features are not good enough, overlap exists seriously, especially among WF, HO, WE and HC actions. This indicates that the subject duplicates is unsatisfactory in acting the same movement, and should be trained for more time to keep well action duplication before experiments. Classification analysis 2 Considering the features cluster not well enough, neural network with strong nonlinear mapping ability is used here as classifier. Multiple layer perception (MLP) with structure of 12-12-6 is used and genetic algorithm is adopted instead of BP algorithm to avoid local minimum. The feature values are normalized to [0, 1] before they are used as inputs. 0.9 and 0.1 are used to represent two different situations that belong to the class or not belong to the class. The population is 50, cross rate 0.6, mutation rate 0.4. Evolving strategy is selecting 20 chromosomes with best fitness from children to replace parents that have worse fitness. After 200 generations, the final training error of the best gene is up to 0.0418, the training curve can be seen in Fig. 6 6 . Table 2 2 gives the training error and testing error. HO HC WE WF WP WS Average Training rate 86% 87.33% 86.67% 72.50% 75% 76.33% 80.94% Test rate 40% 43% 65% 62.50% 75% 72% 56.74% It can be seen from Table 2 2 that the average recognition rate is encouraging, though they are not satisfactory for prosthetic control. Reasons that cause the low recognition rate may be as follows: 1. This experiment involves the human body, and the risk is very high because of the well-known reasons. For the sake of safety, the experiment does not last long, so the sampled data were somewhat insufficient, and possible abnormal samples are also used as classification without excluding them first. 2. Two years had passed after the subject’s amputation. Although half a month’s rehabilitation training was implemented before the experiment, the duplication of movements were still not satisfactory. This made the features scatters not well for high recognition rate. More rehabilitation time is needed before the next experiments. 3. As pointed out before, sensational signals are mixed in ulnar and medium fascicule. Obviously, they cannot be neglected and can greatly reduce the recognition accuracy since when the same action occurs, the sensational information may diverse greatly. Clearly, these three disadvantages can be fairly improved, though they are inevitable to some extent. So, it can be said that developing flexible prosthesis with multi DOF controlled by neuro signal is quite feasible. Of course, in future, the through-the-skin wire should be replaced by a radio link connecting the fully implanted electrodes with the external decoding device. DISCUSSION 1 This experiment on human body gives confidence that collective signals from the fascicule of the three main nerves are useful sources providing enough information for prosthetic control. Since neuro signals have so many advantages, as pointed out in the above discussion, it can be concluded that an ideal way for developing prosthesis is to obtain control commands by establishing mapping between the movement intentions and the patterns of neuro signals from intrafascicular electrodes clinically implanted into motor nerves by microsurgery technique. However, in developing commercially and flexible neuro prosthesis, besides more experiments on human body being needed, further research on the relations between neuro activity and movement intentions, ideal human-machine interface and controlling strategy are all indispensable. References M Burrow , J Dugger , D Humphrey , 等 . Cortical Control of a Robot Using a Time-delay Neural Network . Proceedings of International Conference on Rehabilitation Robotics ICORR , 1997 . Bath, UK 83 - 86 . . Cortical Control of a Robot Using a Time-delay Neural Network JK Chapin , KA Moxon , RS Markowitz , MA Nicolelis . Real-time control of a robot arm using simultaneously recorded neurons in the motor cortex . Natural Neuroscience , 1999 . 2 664 - 670 . . Real-time control of a robot arm using simultaneously recorded neurons in the motor cortex ZW Chen . Advances of Microsurgery in China . Proceedings of ISRE&CR , 2002 . Dalian, China 4 - 6 . (in Chinese) . Advances of Microsurgery in China DL Davies , DW Bouldin . A cluster separation measure . IEEE Transactions on Pattern Analysis and Machine Intelligence , 1979 . 1 224 - 227 . . A cluster separation measure DR Humphrey . Relating motor cortex spike trains to measures of motor performance . Brain Research , 1972 . 40 7 - 18 . . Relating motor cortex spike trains to measures of motor performance MZ Kermani , BC Wheeler . MA Penna . Finding the Number of Neurons in Extracellular Recordings: A Comparison of the Effectiveness of Popular Clustering Indices . Proceedings of Computational Neuroscience Symposium , 1992 . October Indianapolis 229 - 235 . . Finding the Number of Neurons in Extracellular Recordings: A Comparison of the Effectiveness of Popular Clustering Indices GTA Kovacs , MC Stephanides , WR Knapp , 等 . Development of Chronic Implant Neural Prosthesis Microelectrode Arrays . Proceedings of IEEE MONTECH’87 Conference on Biomedical Technologies , 1987 . Montreal, Quebec 152 - 155 . . Development of Chronic Implant Neural Prosthesis Microelectrode Arrays CL Lin , SC Wang , HC Wu , ST Young , MH Lee , TS Kuo . A speech controlled artificial limb based on DSP chip . IEEE Proceedings of 20th ACMBS , 1998 . 20 ( 5 ): 2074 - 2075 . . A speech controlled artificial limb based on DSP chip S Mallat . A theory for multi-resolution signal decomposition: the wavelet representation . IEEE Transactions on Pattern Analysis and Machine Intelligence , 1989 . 31 674 - 693 . . A theory for multi-resolution signal decomposition: the wavelet representation S Pittner , SV Kamarthi . Feature extraction from wavelet coefficients for pattern recognition tasks . IEEE Transactions on Pattern Analysis and Machine Intelligence , 1999 . 21 ( 1 ): 83 - 88 . . Feature extraction from wavelet coefficients for pattern recognition tasks AB Schwartz . Direct cortical representation of drawing . Science , 1994 . 265 540 - 542 . . Direct cortical representation of drawing AB Schwartz , RE Kettner , AP Georgopoulos . Primate motor cortex and free arm movements to visual targets in three-dimensional space. I. relations between single cell discharge and direction of movement . Journal of Neuroscience , 1988 . 8 2913 - 2927 . . Primate motor cortex and free arm movements to visual targets in three-dimensional space. I. relations between single cell discharge and direction of movement DJ Weber , J He , RE Isaacs , 等 . Response of Simultaneously Recorded Cortical Neurons to Perturbations in Arm Trajectory . Proceedings of The First Joint BMES/EMBS Conference , 1999 . Atlanta, USA 13 - 16 . . Response of Simultaneously Recorded Cortical Neurons to Perturbations in Arm Trajectory J Wessberg , R Stambaugh , JF Kralik , PD Beck , M Laubach , JK Chapin , J Kim , J Biggs , MA Srinivasan , MA Nicolelis . Real-time prediction of hand trajectory by ensembles of cortical neurons in primates . Nature , 2000 . 408 361 - 365 . . Real-time prediction of hand trajectory by ensembles of cortical neurons in primates XW Zhang , YP Yang , XM Xu , 等 . Wavelet Based Neuro-fuzzy Classification for EMG Control . IEEE Proceedings of ICCCAS , 2002 . Chengdu, China 1087 - 1089 . . Wavelet Based Neuro-fuzzy Classification for EMG Control XW Zhang , YP Yang , XM Xu . Wavelet based feature representation . Computer Engineering and Applications , 2003 . 39 ( 19 ): 25 - 28 . (in Chinese) . Wavelet based feature representation XW Zhang , TP Hu , ZH Gao , 等 . Initial Detection and Analysis on Neuro-information from Amputee . 2004 . The 11th World Congress of the International Society for Prosthetics and Orthotics . Hong Kong, China . Initial Detection and Analysis on Neuro-information from Amputee XJ Zheng , J Zhang , TY Chen , 等 . Self-made Longitudinally Implanted Intrafascicular Electrodes for Stimulating and Recording Fascicular Physioelectrical Signals in Sciatic Nerve of Rabbits . Proceedings of ISRE&CR , 2002 . Dalian, China 245 - 250 . . Self-made Longitudinally Implanted Intrafascicular Electrodes for Stimulating and Recording Fascicular Physioelectrical Signals in Sciatic Nerve of Rabbits We would like to thank Prof. Huang Suiyin and other medical and health workers in Zhongshan Hospital, Medical Center of Fudan University for their cooperation and direction in this study. 10.1631/jzus.2005.B0272 Project (No. 39930070) supported by the National Natural Science Foundation of China 2004-05-19 2004-08-30 2005-03-20 2022-01-05 Copyright © 2005, Journal of Zhejiang University Science 2005 Journal of Zhejiang University-SCIENCE B (Biomedicine and Biotechnology) 04 6B