ArticlesAll Issue
ArticlesAbnormal Behavior Detection Based on Activity Level Using Fuzzy Inference System for Wheelchair Users
• Congcong Ma1, Juan Du1, and Raffaele Gravina2,*

Human-centric Computing and Information Sciences volume 12, Article number: 21 (2022)
https://doi.org/10.22967/HCIS.2022.12.021

Abstract

With the ever increasing average age worldwide, wheelchairs are becoming a fundamental aiding tool to facilitate mobility among elder and people with disabilities. Lacking of continuous nursing care service, wheelchair users might encounter dangerous conditions caused by long time sitting activities and suffer physical discomfort, musculoskeletal disorders, pressure ulcers, cardiovascular diseases, etc. Because wheelchair users need proper amount of exercises, distinguishing abnormal physical behavior from training activities is undoubtedly necessary. In this paper, we propose an abnormal behavior detection method based on activity level assessment for wheelchair users. Using fuzzy inference system, we construct the fuzzy sets for accelerometer, gyroscope and center-of-pressure in sitting condition based on data collected with a multi-sensor smart cushion. In addition, combined with posture recognition, we construct the fuzzy sets of activity levels and posture transition percentage to ultimately detect abnormal activity states, which can represent safety risk. Experiment results demonstrate that the proposed algorithm can accurately recognize activity levels and detect abnormal states.

Keywords

Abnormal Behavior Detection, Activity Level, Posture Transition, Fuzzy Inference System, Wheelchair Users

Introduction

With the development of body area networks [1], more and more wearable sensing devices [2] were used to better support people in monitoring their health [3], wellness [4], and even emotions [5]. Activity level assessment [6] can give a synthetic information of general health status. In our daily life, there exists several situations leading to sitting conditions, such as working on computer, watching TV, eating, or having a group discussion [7]. Therefore, the amount of daily time spent seated is significant. In addition, besides motor impairment diseases, just with age, several elders need to use a wheelchair to aid their mobility [8]; therefore, monitoring human sitting status could provide better health service [9].
Long-term sitting will induce several health problems, so proper physical exercise could help keeping good physical conditions. It is thus important to assess the activity levels, since it could inform people to perform some exercise and also alert, when necessary, to change sedentary lifestyle. Users sitting on the wheelchair can occasionally perform high intensive activity (e.g. during physical exercises); in this scenario, closely monitoring their activity is important, specifically to recognize the ability to maintain good posture control. In fact, the user who is unable to control the body posture risks to undertake abnormal (and potentially dangerous) behaviors.
In this paper, we mainly focused on the abnormal activity detection based on the activity level assessment with the final aim to warn the users of abnormal high intense activity and protect their personal safety. In particular, the contribution of the paper is two-fold:

We introduced two kinds of features based on activity levels and posture transitions using a smart pressure cushion.

We proposed a two-stage fuzzy inference system to detect the abnormal behaviors for wheelchair users.

The remainder of the paper is organized as follows. Section 2 discusses the background and related works of abnormal activity detection in smart home and in the context of wheelchair users. Section 3 describes the proposed algorithm of abnormal behavior detection for wheelchair users. Section 4 presents the system implementation and data processing. Section 5 is focused on the experiment and discusses obtained results. Finally, in Section 6 we conclude the paper and outline future works.

Background and Related Works

Abnormal Activity Detection in Smart Home
Most of the abnormal human activity recognition works are focused on smart home and video based method was used. Dhiman and Vishwakarma [10] reviewed the state-of-the-art techniques for abnormal human activity recognition, the proposed literature includes feature designs of abnormal human activity recognition in a video with respect to the context or application such as fall detection, ambient assistive living (AAL), homeland security, surveillance or crowd analysis using RGB, depth and skeletal evidence.
Several researches focused on the fall detection system, as fall is a major threat to the health and life of the elders. Rastogi and Singh [11] gave a comprehensive description of various fall detection method especially the researches using machine learning. Xia and Li [12] proposed an abnormal behavior recognition method using LSTM network with temporal attention mechanism. Luo et al. [13] developed an abnormal activity detection system using ceiling-mounted pyroelectric infrared (PIR) sensors to detect abnormal activities like fall. Hu et al. [14] proposed an evolutionaryexpand-and-contract instance-learning-based algorithm to recognize abnormal activities such as accident, falling, disease attack, etc.
Anithaand Baghavathi Priya [15] used the camera-based method to monitor the elderly people that live alone, dynamic Bayesian network (DBN) is developed to recognize abnormal activities like fall, chest pain, headache, vomit. Gul et al. [16] used the deep learning approaches to analyze the image of the patient actions, such as eight kinds of abnormal activities like fall, headache, vomit, etc.
As to the acceleration of aging society, a lot of sensors and camera-based applications emerged to fill the gap of technology upgrade in traditional home care environment.

Activity Recognition for Wheelchair Users
Wheelchair users are forced to seat on the wheelchair most of daytime, and their activities are a little different from subjects with no motor impairments. So, monitoring their living condition is important to give them a good life support. Some researchers focused on the wheelchair based activities recognition, while other studies focused on wheelchair moving environment recognition such as road conditions that wheelchair users encounter.
Marco-Ahullo et al. [17] used an accelerometer in the smartphone to estimate the energy expenditure of wheelchair users based on the activities such as lying down, watching TV, working on a computer etc. Garcia-Masso et al. [18] used an accelerometer placed on both wrists, chest and waist to identify the activity type performed by manual wheelchair users such as lying down, body transfer, watching TV etc. Tsai et al. [19] focused on the wheelchair propulsion strategies that might induce should pain. They examined the wheelchair progression in rough and smooth ground, and using convolutional neural network (CNN) deep learning model to recognize and predict wheelchair users’ location.
Watanabe et al. [20] used a tri-axial accelerometer in a smartphone to recognize the wheelchair behavior, it could recognize sidewalk environment such as gradient, curbs and gaps. Mascetti et al. [21] proposed an automatic crowdsourcing mechanism that could help the wheelchair users share the knowledge of urban features like curb ramps, steps or other obstacles.

Activity Level based Abnormal Behavior Detection for Wheelchair Users
In literature, there is not a well consolidated definition of activity level. However, different kinds of activity levels are typically defined in terms of metabolic equivalent of task (MET) [22]. In Table 1, we summarize some definitions of activity levels and examples of corresponding activities [2325].

Table 1. Different types of activity level and examples of activity
Study Activity levels Activities
Liu and Chan [23] Light walking
Moderate walking fast(5–6 km/hr)
Vigorous walking fast(7 km/hr), running(6–8 km/hr)
Very vigorous running(>9 km/hr)
Grillon et al. [24 Sleeping (or None), Light, Moderate, and Vigorous wheelchair operation activities such as hold steady, move, push
Ren et al. [25] Sedentary behavior supine rest, reading, watching TV, working on PC
Household chores sweeping and vacuuming
Locomotion slow track walking, brisk track walking, walking with a 10-lb backpack, track running
Interactive video games Nintendo Wii, floor Light Space, wall Light Space, DanceDance Revolution (DDR), Trazer
Exercise and sports playing catch, soccer around cones, Sport Wall, workout video
In this paper, we focused on abnormal behavior detection based on the activity level assessment of wheelchair users. To this purpose, we defined the activity levels and examples of activities as shown in Table 2.

Table 2. Categorization of different activity levels of wheelchair users
Activity levels Description Examples of activities
Light intensity User performs common daily life activities in sitting condition. Reading a book, Desk working, Conversation
Moderate intensity User performs moderate activities to prevent pressure ulcer. Swing left-right or leaning front-back
High intensity User is doing physical exercise to keep fit. Doing exercise
Regarding abnormal activity detection, there exists several studies using different kinds of sensors and methods. Gao et al. [26] proposed a two-stage adaptive weighted extreme learning machine method for eyeglass and watch wearables to detect abnormal activities like falling down. Tong et al. [27] used environmental sensors of the smart home to detect normal and abnormal activities for the elder people and provide them with good healthcare service. Hidden state conditional random field method was used to detect and assess abnormal activities. Khan and Sohn [28] used vision-based sensors for elder care by recognizing six abnormal activities (i.e., forward fall, backward fall, chest pain, faint, vomit and headache). R-transform and kernel discriminant analysis were used to process the data and analyze the abnormal status. Fan et al. [29] proposed a deep learning based method to detect abnormal sitting postures. A pressure sensor array was used to collect the hip pressure and CNN was used to identify sitting postures. Cui and Dahnoun[30] used the millimeter-wave radar to detect the sitting postures; neural networks were used to estimate the postures such as sitting and walking. Su et al. [31] used a monopulse radar to detect postures and fall events by means of a monopulse ratio-based algorithm.A comparison of the related works is listed in Table 3.

Table 3. Comparison of related works
Study Sensors or equipment Recognition method Recognized activities
Marco-Ahullo et al. [17] Accelerometer Multiple linear models Lying down, watching TV, etc.
Garcia-Masso et al. [18] Accelerometer Linear discriminant analysis, QDA and support vector machines Lying down, body transfer, etc.
Tsai et al. [19] Accelerometer CNN Wheelchair propulsion activities in different environment
Gao et al. [26] Eyeglasses and watch wearables Extreme learning machine-based method Falling detection
Tong et al. [27] Environmental sensors Hidden state conditional random field Wash hands (Forget to turn off the tap), Cook (Forget to replace spices) etc.
Khan and Sohn [28] Vision-based method R-transform and kernel discriminant analysis Faint, backward fall, forward fall, vomit, chest pain, and headache
Fan et al. [29] Smart cushion CNN Leaning back, leaning forward, leaning left, leaning right, standard
Cui and Dahnoun[30] Millimeter-wave radar Neural network Sitting, walking, etc.
Su et al. [31] Monopulse radar Monopulse ratio Posture and fall detection
As we could see in Table 3, in contrast to our approach, several studies adopt devices such as accelerometer sensors, eyeglasses and cameras; these approaches require the users to wear devices for prolonged time, which might make them uncomfortable and even introduce privacy concerns; also some other non-invasive method like smart cushion, millimeter-wave radar, monopulse radar were proposed, which might has the property of expensive or not easy to implement. In addition, several works focus on the recognition of specific activities, while we focus on general activities performed on the wheelchair and we aim to detect user’s abnormal physical behavior.
We proposed an abnormal behavior detection method based on activity levels with a smart cushion using fuzzy inference system (FIS) for wheelchair-bounded individuals. Compared with video-based approaches or wearable sensors to detect abnormal status, our work is using a smart cushion, and aims to identify the abnormal status hidden by the expression of postures. The advantage of this approach is that it could enable non-invasive abnormal status monitoring for wheelchair users.

Proposed Method

System Architecture
The system architecture is shown in Fig. 1. It is composed of three layers:
- Data collection and pre-processing: raw data was collected from our designed smart cushion; after data pre-processing, features can be extracted.
- Activity level assessment and abnormal status detection: using the extracted features, we constructed the fuzzy sets and fuzzy rules, after defuzzification stage, activity levels and abnormal status can be achieved.
- Applications: several applications can be developed on top of the results of activity levels and abnormal status, and if dangerous status is detected, safety alert systems can inform the user or caregivers.

Fig. 1. System architecture of the proposed method.

Hardware Design of the Smart Cushion
The sensing device used in this study consists of a smart cushion that was designed in our former research [32] to detect sitting postures and activities. The smart cushion combines pressure sensors with an inertial measurement unit (IMU)(see Fig. 2). As shown in Fig. 2, the hardware involves pressure sensor unit, IMU and the signal processing module consists of an Arduino [33] board.

Fig. 2. Sensors deployment and the smart cushion circuit board: (a) pressure sensors deployment; (b) Inertial measurement unit; (c) top side of the circuit board; (d) bottom side of the circuit board.

Force sensing resistor (FSR) [34] is used to measure the pressure. It is a light weighted, minimal sized and sensitive resistance-based sensor. The pressure sensors were deployed in the cushion as shown in Fig. 2(a). Inside the real cushion there is a circuit board, with the deployment of both pressure sensors and IMU sensor. The circuit board has the size of 40 cm×40 cm, that was split into 5×5 square sensing zones. The central point is assumed as origin point O and the location of each pressure sensor is defined with distance to the origin point.
Activity data from a 3-axis accelerometer and a 3-axis gyroscope data was monitored by a MPU9250 [35] chip placed on the back of the base board, as shown in Fig. 2(b).
Fig. 2(c) shows the top side of the circuit board with its six pressure sensors, and the signal processing module, Bluetooth module, vibration motor and power supply unit were placed on the left top side. Fig. 2(d) shows the bottom side of the circuit board, the IMU was placed on the center of the board.
Of course, during experiments, the device has been embedded under the foam filling of the cushion so to make the users more comfortable. At the time $t$, we can obtain raw data from the accelerometer, gyroscope and pressure sensors as shown in Equation (1), $a_x,a_y$ are the output data of the accelerometer, $ω_x,ω_y$ are the output data of the gyroscope, $f_1,f_2,f_3,f_4,f_5,f_6$ are the output data of each pressure sensor as shown in Fig. 2.

$R(t)={a_x,a_y,ω_x,ω_y,f_1,f_2,f_3,f_4,f_5,f_6}$(1)

All the data were collected and processed by the Arduino board, with a sampling frequency of 50Hz. It is worth noting that in a former study [36], it was shown that human’s posture and activity can be typically captured with sensor sampling below 18Hz. So, our hardware could definitely get the accuracy of the users’ minor movement of activities.

Feature Extraction
We extract the features from IMU and pressure sensors. In our former research [6], we found that the data on sagittal axis can better capture the activity performed by sedentary users. So we only need to calculate the resultant value of accelerometer and gyroscope data in horizontal (or transverse) plane denoted by $a_{xy}$ and $ω_{xy}$. These parameters were calculated as shown in Equation (2).

$\begin{cases} a_{xy}=\sqrt{a_x^2+a_y^2} \cr ω_{xy}=\sqrt{ω_x^2+ω_y^2} \end{cases}$(2)

In addition, we can obtain the center-of-pressure (CoP) with the pressure sensors in the horizontal plane. The distance between each pressure sensor and the center of the board is expressed as $x_i$ and $y_i$, as the CoP can be calculated with Equation (3).

$(CoP_x, CoP_y =(\frac{\sum_{i=1}^6 f_i*x_i}{\sum_{i=1}^6 f_i},\frac{\sum_{i=1}^6 f_i*y_i }{\sum_{i=1}^6 f_i})$(3)

The resultant value is calculated as in Equation (4).

$c_{xy}=\sqrt{CoP_x^2+CoP_y^2}$(4)

We use approximate entropy ($ApEn$) to process the data for feature extraction. $ApEn$ was first proposed by Pincus [37] and it is widely used to evaluate the complexity of the data. It is often applied to both short- and long-term data recordings, for example it was used to evaluate the regularity and unpredictability of fluctuations over time-series data. As to a chosen data segment, $ApEn$ determines the similarity probability of the next set of segments of the same duration; the higher the probability the smaller the $ApEn$ value, indicating less complexity of the data. A time series containing many repetitive patterns has a relatively small $ApEn$; a less predictable (i.e., more complex) process has a higher $ApEn$. This is a great advantage that involves a number of applications, such as fatigue detection and activity level assessment, as this rank index is increased as time accumulates. So the features to detect the activity levels is calculated as $ApEn-a_{xy}, ApEn-ω_{xy}$, and $ApEn-c_{xy}$.
We also defined posture transition percentage (PTP) to indicate the posture changes in a given time window. Fig. 3 shows an example of the posture representation of three different activity levels—i.e.,low intensity activity (LIA), moderate intensity activity (MIA), and high intensity activity (HIA), each sample represented a detected sitting posture.
We can use machine learning to detect the sitting postures and we listed some common postures usually performed while seated—i.e.,proper sitting (PS), lean left (LL), lean right (LR), lean forward (LF), and lean backward (LB) [38], each posture was determined by a time windows of 0.5seconds. For graphical clarity, we used 1, 2, 3, 4, and 5 in the figures to represent the posture of PS, LR, LL, LF, and LB, respectively. Of course, as shown in Fig. 3, HIA has more frequency posture change, and LIA has fewer posture change.
The calculation of PTP is divided into mainly two steps:
- Binary processing of posture status: Using the current posture compared with the former posture, we denote as 0 if they are equal; conversely, we denote as 1. Using this method, we can process the postures as a binary string.
- Calculation of PTP: We obtain the PTP value by counting the number of “1” in a given time slide window, and dividing by the number of posture samples in that window.

Fig. 3. The posture representation with different activity levels.

Abnormal Activity Detection Algorithm
In Section 2, based on related previous studies, we have defined three kinds of activity levels as LIA, MIA, and HIA. Our method, shown in Fig. 4, is a two layers of FIS-based algorithm and, in particular, uses Mamdani $FIS$ to construct both $FIS_1$ (for activity levels) and $FIS_2$ (for abnormal activities).

Fig. 4. Block diagram of the proposed abnormal activity detection algorithm.

The procedure of abnormal activity detection is shown in Algorithm 1. According to our former research, 0.5seconds is better to determine a sitting posture [38], and also 30seconds could better depict the activity level in a short time period [6]. Lines 4 to 8 are the step of posture recognition; decision tree algorithm was used to recognize the sitting postures. Algorithm 1 can be solved in polynomial time (i.e., is a problem belonging to P complexity class); specifically, its time complexity is O(m log⁡(n)) where m the size of the raw data sequence R(t) and n is the number of nodes of the trained decision tree classifier, with the assumption of a reasonably balanced tree model.
Algorithm 1. Abnormal activity detection algorithm
Input: Sequence of Raw data R(t), {r1, r2, r3, ..., rm}
Output: Abnormal activity level ABL, {l1, l2, l3, ..., ln}
1: BEGIN
2:   For each incoming raw data ri,ri∈R(t)
3:     k=1, 1≤k≤m
4:     if Sliding buffer = 0.5 second samples //calculation of Posture
5:       Feature vectors PF(k)←Feature extraction //PF(Posture Feature)
6:   Pk←Decision Tree Algorithm
7:   endif
8:   k=k+1
9:   if Sliding buffer = 30 second samples do
10:     Feature vectors AL(k) ←Feature extraction //AL (Activity Level Feature)
11:     Execute FIS1 //calculation of AL (Activity Level)
12:     Posture Sequence, {P1, P2, P3, ..., P30}
13:     Calculation of PTP (Posture Transition Percentage)
14:     Execute FIS2 //calculation of ABL (ABnormal Level)
15:   endif
16:   return ABL(ABnormal Level)
17:   END
As a comparison with related works, we provided the algorithm complexity with Linear Discriminant Analysis (LDA) [28] and CNN [29] methods. For LDA, in [39] the time complexity (for its usual implementation) is proven O($mnt + t3$), where m is the number of samples, n the number of features, $and t = min {m, n}$. As we could see, our proposed algorithm is more computation efficient than LDA. As for CNN, its capability of processing huge amount of data strongly rely on high performance hardware e.g., GPU-powered. On the contrary, our proposed algorithm may be easily implemented on resource constrained embedded platforms.

System Implementation and Data Process

Fuzzy Sets of Activity Level
We design the activity level assessment based on FIS. The fuzzy sets are usually determined by experimental data and simulation. Gaussian, triangular and trapezoidal fuzzy membership function have been widely used in the construction of fuzzy membership function. Gaussian membership function can be closer to the data extracted from real environment [40], but its model is difficult to define. Triangular and trapezoidal fuzzy memberships have less computation load and are widely used in practice. Fuzzy based algorithm could also be used in the area of indoor localization [41], and multimedia data analysis [42].
Table 4 reports the fuzzy sets of the input and output parameters. For each input and output parameter, the range, minimum, average and maximum were computed from all the data we collected. In order to eliminate the irrelevant and redundant signals, we use Equation (5) to process the value of fuzzy sets. Then, we can determine the range, maximum and minimum of the selected data and the fuzzy sets. In Equation (5), sum is the number of the samples, s is the sample data. The function of Equation (5) is to find an interval [$x_i,y_i$] that could cover most of the sample data (we set the percentage over 95%). Then N is the number of samples in the interval, $x_i$ and $y_i$ are respectively the minimum and maximum values of the features. And also Equation (5) need to meet the condition that get the minimum of c, c is the interval length of the selected samples.

$\begin{cases} N(x_i≤s≤y_i)/sum≥0.95 \cr c=argmin(y_i-x_i) \end{cases}$(5)

Table 4. Fuzzy sets of input and output parameter
Range Fuzzy Minimum Average Maximum
$ApEn-a_xy$ (0, 1.14) L 0.06 0.37 0.78
M 0.26 0.62 0.95
H 0.46 0.79 1.14
$ApEn-ω_xy$ (0, 1.2) L 0.05 0.31 0.52
H 0.44 0.76 1.2
$ApEn-c_xy$ (0, 0.57) L 0.01 0.06 0.18
H 0.1 0.31 0.57
AL (0, 1) L 0 0.2 0.4
M 0.2 0.5 0.8
H 0.6 0.8 1
Using the selected data and the fuzzy sets as shown in Table 4, triangular membership function was used to describe the input and output parameters as shown in Fig.5. Specifically, Fig. 5(a)–(c) shows the membership of the input parameter, Fig. 5(d) is the membership for the output referred as activity level (AL). We use the notations Low, Mid, and High to respectively represent LIA, MIA, and HIA.
After determining membership functions for the input and output variables, we need to formulate the fuzzy rules. In this work, we used Mamdani FIS and construct the FIS with 3 inputs and 1 output. The center-of-area method is used for defuzzification. We construct the fuzzy rules as shown in Table 5. All the rules are determined by our experience in this specific domain.

Fig. 5. Input and output membership functions for activity level assessment: (a) ApEn-axy, (b) ApEn-ωxy, (c)ApEn-cxy, and (d) AL.

Table 5. FIS rules to determine the activity level
Input Output
$ApEn-a_xy$ $ApEn-w_xy$ $ApEn-c_xy$ AL
Rule 1 Low Low Low Low
Rule 2 Low High Low
Rule 3 High Low Low
Rule 4 High High Mid
Rule 5 Mid Low Low Low
Rule 6 Low High Mid
Rule 7 High Low Mid
Rule 8 High High High
Rule 9 High Low Low Low
Rule 10 Low High Mid
Rule 11 High Low Mid
Rule 12 High High High

Fuzzy Sets of Abnormal Activity
Similarly to the FIS for activity level assessment, for abnormal activity level detection we still use triangular membership function. Here the input is activity level and the posture transition percentage, the output is the abnormal activity level. We obtain two ranges of posture transition percentage: PTP∈(0, 0.27), when the user is in Low or Mid activity level; PTP∈(0.23, 0.55), when the user is in High activity level. Table 6 shows the input parameters and the output named as ABnormal Level (ABL).
With the parameters reported in Table 6, we can obtain the inputs and output for the detection of abnormal activity level. Fig.6(a) is the Mamdani type FIS, Fig.6(b)–(c) is the membership function for inputs, Fig.6(d) is the membership function for output.
To design the rules of abnormal activity level, we also construct the Mamdani type FIS with 2 inputs and 1 output. Center-of-area method is still used for defuzzification. As aforementioned, the rules shown in Table 7 are determined by experience of the authors.

Table 6. Input and output of abnormal activity level detection
 Range Fuzzy Minimum Average Maximum AL (0, 1) L 0 0.2 0.4 M 0.2 0.5 0.8 H 0.6 0.8 1 PTP (0, 0.55) L 0.02 0.14 0.27 H 0.23 0.39 0.55 ABL (0, 1) L 0 0.2 0.4 M 0.2 0.5 0.8 H 0.6 0.8 1

Fig. 6. Input and output membership functions for abnormal activity level: (a) FIS, (b) AL, (c)PTP, and (d) ABL.

Table 7. Rules of abnormal activity level
AL PTP Abnormal level
Rule 1 Low Low Low
Rule 2 Low High Mid
Rule 3 Mid Low Mid
Rule 4 Mid High Mid
Rule 5 High Low Mid
Rule 6 High High High

Experiment and Results

Participants
The study was approved by the Institutional Review Board of the Nanyang Institute of Technology (No. NYISTIRB-2021-002), and was conducted at a university laboratory. A total of 9 subjects (5 males and 4 females) took part in the study. All the participants were informed of the purpose and procedure of the study, after signing an informed consent, they filled a form including questions on gender, ethnicity, age, height, and weight. Their ages were in the range of [60,65] and during most of daily time they need a wheelchair to facilitate their moving demand. Their body mass index (BMI) is in the range of [16,34], whose distribution in our experiment sample is shown in Table 8. As BMI value reflects indirectly the body shape, we use this index to recruit the participants so as to improve diversity of sample.

Table 8. BMI distribution of the subjects participating to the experiments
Underweight Normal Overweight and obese
BMI (kg/m2) <18.5 [18.5, 25] ≥25
Number of subjects 2 5 2

Experiment Protocol and Data Collection
In our former research [6], we have designed the data collection protocol for different kind of activity levels. The same protocol was used for the experiment in this work. During the experiment, participants were free to choose the time period for the test session; they were just asked to perform common activities on wheelchair among:
- Low intensity activities: (1) Reading a book; (2) Desk working on the PC; (3) Having a conversation;
- Mid intensity activities: (4) Swing left-right or front-back;
- High intensity activities: (5) Doing physical exercise such as lifting a weight.
On average, experiment sessions lasted about 2 hours and each participant performed six sessions over the 2 weeks. All the experiment sessions have been video recorded so to manually label the samples of each performed activity with one of the three defined activity levels.

Recognition Results
In this section, we describe the use of fuzzy sets and the rules to construct the FIS, and we analyze obtained results. Fig.7 illustrates the effects of different inputs in detecting the activity levels. The plots are three-dimensional, so z-axis represents the results of activity levels, x-axis and y-axis in turn represent the input parameter of $ApEn-a_{xy}, ApEn-ω_{xy}$ and $ApEn-c_{xy}$. Thus, the results with the three possible combinations are respectively shown in Fig.7(a), 7(b), and 7(c).
In particular, Fig.7(a) shows the results using the input of ApEn-axy and ApEn-ωxy. When both values are small, the AL value also results small, and the user is performing LIA. When the values of $ApEn-a_{xy}$ and $ApEn-ω_{xy}$ are higher, the user is performing HIA. Similarly, Fig.7(b) and 7(c) show the results with the other two input variable combinations.

Fig. 7. Combined effect of two parameters on the output of activity levels when the third parameter is constant: (a) $ApEn-axy and ApEn-ω_{xy}$, (b) $ApEn-a_{xy}$ and $ApEn-c_{xy}$, and (c) $ApEn-ω_{xy}$ and $ApEn-c_{xy}$.

In order to further analyze the activity level, we show the results considering the three parameters in Fig.8. The samples 1, 2, and 3 refer to LIA status; the input parameter is lower, so AL value is low too. When only one parameter is high, it will not affect the final result. When there exist 2 input variables withhigh values (e.g., in samples 4 and 5, $a_{xy}$ and $ω_{xy}$ are greater than 0.6), the final result indicates MIA status. Finally, when the three input parameters are all high, the result is HIA status (see samples 6, 7, and 8).

Fig. 8. Activity levels results compared using different input parameter.

Fig. 9. Abnormal status detection using FIS.

Fig. 9 depicts the abnormal activity results using FIS; as we can see, in most cases, the level is in the range of (0.2, 0.6), and the user is in normal status. So we can set the level threshold to 0.6 to determine whether the user is in abnormal status. When the AL and PTP values are higher, the ABL value is greater than 0.6 and the user is in abnormal status.
The ABL results obtained are shown in Table 9, it contains the accuracy with 10-fold cross-validation and leave-one-out cross-validation results. As we can observe, the recognition with 10-fold reaches more than 85% on average, but ABL (High) only gets 81.25% accuracy. We also demonstrate the experiment with leave-one-out; as expected we get lower recognition rate for the three kinds of abnormal level.
To further evaluate our proposed algorithm, we choose nine samples of the data, obtaining the abnormal status information using different inputs as shown in Fig. 10.

Table 9. Recognition accuracy of different abnormal levels with 10-fold and leave-one-out cross-validation
Abnormal level Accuracy (%)
10-fold Leave-one-out
Low 88.75 76
Middle 86.3 74
High 81.25 72

Fig. 10. Abnormal status using different input.

Conclusion

In this paper we have proposed a FIS-based algorithm for abnormal behavior detection especially for wheelchair individuals. Firstly, we constructed the fuzzy sets to determine the activity levels and then we combined the posture transition percentage of the users to determine abnormal conditions. Using the two stage FIS based abnormal activity detection algorithm, we could predict the wheelchair users’ abnormal status. Obtained results showed that our system is effective and is able to accurately detect the abnormal status of wheelchair users. In particular, we calculated the accurate with 10-fold and leave-one-out cross-validation, obtaining respectively 85% and 74% accuracy on average.
As our designed device could be easily implemented in the wheelchair or even on an ordinary chair, it could be used in different contexts such as smart health and sport, e.g., for wheelchair-bound athlete competition such as Paralympics. In addition, complexity analysis of the proposed two stage FIS-based algorithm and obtained experiment results showed that is feasible and effective to implement our proposed solution to smart objects based on resource constrained embedded devices. Thus, practical impact of our work can be found in diversified domains spanning from AAL, to behavioral analysis, till quantitative psychology. Understanding abnormal behavior of users is surely relevant to assess not only medical conditions and safety (useful in AAL contexts) but also to recognize physical and mental discomfort with clear applications for online behavioral and psychological analysis, e.g., in human-machine interactions based on cyber physical systems.
In our future research, we will try to enhance the recognition results and implement the algorithm in embedded circuit board. Because we did not achieve satisfactorily results with leave-one-out validation, we plan to collect more data in order to obtain a more precise model even with little user-specific calibration. In addition, we will combine further kind of sensors to obtain additional information from the wheelchair users and give them more recommendations to maintain good health. Finally, we will define and detect more kinds of abnormal states in order to support the wheelchair users’ a good lifestyle.

Acknowledgements

The author would like to thank the volunteers who participated in the experiments for their efforts and time.

Author’s Contributions

Conceptualization, CM, RG. Funding acquisition, CM. Investigation and methodology, CM, RG. Supervision, RG. Writing of the original draft, CM, JD. Writing of the review and editing, CM, RG. Validation, CM, JD. All the authors have proofread the final version.

Funding

This research is supported by Natural Science Foundation of Henan (No. 2012BAJ05B07), and also the Henan Association for Science and Technology (No. HNKJZK-2021-52C), and the Doctoral Research Fond of Nanyang Institute of Technology, China.

Competing Interests

The authors declare that they have no competing interests.

Author Biography

Name : Congcong Ma
Affiliation : Nanyang Institute of Technology
Biography : Congcong Ma has been a lecturer with the School of Computer and Software at Nanyang Institute of Technology, China, since 2018. He is a Member of IEEE. In 2021, he had been a Postdoctoral Fellow at The Hong Kong Polytechnic University, HongKong, China. His main research interests contains Body Sensor Networks, Internet of Things, Human-Machine Interaction.

Name : Juan Du
Affiliation : Nanyang Institute of Technology
Biography : Juan Du is a lecture of Communication Engineering in Nanyang Institute of Technology. She earned her B.Sc. degrees in Information Engineering University and M.Sc. degrees in Zhengzhou University, in 2009 and 2012. Her research interests include wireless sensors network and machine learning.

Name : Raffaele Gravina
Affiliation : University of Calabria
Biography : Raffaele Gravina is an assistant professor of computer engineering in the Department of Informatics, Modeling, Electronics, and Systems, Unical. He earned his B.Sc. and M.Sc. degrees in computer engineering from the University of Calabria Italy, in 2003 and 2007, respectively and his Ph.D. degree in computer and systems engineering from the University of Calabria in 2012. His research interests include high-level programming methodologies and frameworks for WBSNs, collaborative body sensor networks, and Internet of Things.

References

[1] G. Fortino, R. Giannantonio, R. Gravina, P. Kuryloski, and R. Jafari, “Enabling effective programming and flexible management of efficient body sensor network applications,” IEEE Transactions on Human-Machine Systems, vol. 43, no. 1, pp. 115-133, 2013.
[2] R. Gravina, C. Ma, P. Pace, G. Aloi, W. Russo, W. Li, and G. Fortino, “Cloud-based activity-aaService cyber–physical framework for human activity monitoring in mobility,” Future Generation Computer Systems, vo. 75, pp. 158-171, 2017.
[3] G. Fortino, S. Galzarano, R. Gravina, and W. Li, “A framework for collaborative computing and multi-sensor data fusion in body sensor networks,” Information Fusion, vol. 22, pp. 50-70, 2015.
[4] C. Ma, R. Gravina, Q. Li, Y. Zhang, W. Li, and G. Fortino, “Activity recognition of wheelchair users based on sequence feature in time-series,” in Proceedings of 2017 IEEE International Conference on Systems, Man, and Cybernetics (SMC), Banff, Canada, 2017, pp. 3659-3664.
[5] R. Gravina and Q. Li, “Emotion-relevant activity recognition based on smart cushion using multi-sensor fusion,” Information Fusion, vol. 48, pp. 1-10, 2019
[6] C. Ma, W. Li, R. Gravina, J. Cao, Q. Li, and G. Fortino, “Activity level assessment using a smart cushion for people with a sedentary lifestyle,” Sensors, vol. 17, no. 10, article no. 2269, 2017. https://doi.org/10.3390/s17102269
[7] C. Ma, W. Li, R. Gravina, J. Du, Q. Li, and G. Fortino, “Smart cushion-based activity recognition: prompting users to maintain a healthy seated posture,” IEEE Systems, Man, and Cybernetics Magazine, vol. 6, no. 4, pp. 6-14, 2020.
[8] C. Ma, W. Li, Q. Li, R. Gravina, Y. Yang, and G. Fortino, “An embedded risk prediction system for wheelchair safety driving,” in Advances in Body Area Networks I. Cham, Switzerland: Springer, 2019, pp. 149-163.
[9] M. Ma, W. Li, J. Cao, J. Du, Q. Li, and R. Gravina, “Adaptive sliding window based activity recognition for assisted livings,” Information Fusion, vol. 53, pp. 55-65, 2020.
[10] C. Dhiman and D. K. Vishwakarma, “A review of state-of-the-art techniques for abnormal human activity recognition,” Engineering Applications of Artificial Intelligence, vol. 77, pp. 21-45, 2019.
[11] S. Rastogi and J. Singh, “A systematic review on machine learning for fall detection system,” Computational Intelligence, vol. 37, no. 2, pp. 951-974, 2021.
[12] L. Xia and Z. Li, “A new method of abnormal behavior detection using LSTM network with temporal attention mechanism,” The Journal of Supercomputing, vol. 77, no. 4, pp. 3223-3241, 2021.
[13] X. Luo, H. Tan, Q. Guan, T. Liu, H. H. Zhuo, and B. Shen, “Abnormal activity detection using pyroelectric infrared sensors,” Sensors, vol. 16, no. 6, article no. 822, 2016. https://doi.org/10.3390/s16060822
[14] S. Hu, S. Fong, W. Song, K. Cho, R. C. Millham, and J. Fiaidhi, “Novel evolutionary-EAC instance-learning-based algorithm for fast data stream mining in assisted living with extreme connectivity,” Computing, vol. 103, no. 7, pp. 1519-1543, 2021.
[15] G. Anitha and S. Baghavathi Priya, “Posture based health monitoring and unusual behavior recognition system for elderly using dynamic Bayesian network,” Cluster Computing, vol. 22, no. 6, pp. 13583-13590, 2019.
[16] M. A. Gul, M. H. Yousaf, S. Nawaz, Z. Ur Rehman, and H. Kim, “Patient monitoring by abnormal human activity recognition based on CNN architecture,” Electronics, vol. 9, no. 12, article no. 1993, 2020. https://doi.org/10.3390/electronics9121993
[17] A. Marco-Ahullo, L. Montesinos-Magraner, L. M. Gonzalez, R. Llorens, X. Segura-Navarro, and X. Garcia-Masso, “Validation of using smartphone built-in accelerometers to estimate the active energy expenditures of full-time manual wheelchair users with spinal cord injury,” Sensors, vol. 21, no. 4, article no. 1498, 2021. https://doi.org/10.3390/s21041498
[18] X. Garcia-Masso, P. Serra-Ano, L. M. Gonzalez, Y. Ye-Lin, G. Prats-Boluda, and J. Garcia-Casado, “Identifying physical activity type in manual wheelchair users with spinal cord injury by means of accelerometers,” Spinal Cord, vol. 53, no. 10, pp. 772-777, 2015.
[19] J. Y. Tsai, Y. K. Jan, B. Y. Liau, C. L. Chen, P. J. Chen, C. Y. Lin, Y. C. Liu, and C. W. Lung, “Deep learning model to recognize the different progression condition patterns of manual wheelchair users for prevention of shoulder pain,” in Advances in Intelligent Systems and Computing. Cham, Switzerland: Springer, 2020, pp. 3-13.
[20] T. Watanabe, H. Takahashi, G. Sato, Y. Iwasawa, Y. Matsuo, and I. E. Yairi, “Wheelchair behavior recognition for visualizing sidewalk accessibility by deep neural networks,” 2021 [Online]. Available: https://arxiv.org/abs/2101.03724.
[21] S. Mascetti, G. Civitarese, O. El Malak, and C. Bettini, “SmartWheels: detecting urban features for wheelchair users’ navigation,” Pervasive and Mobile Computing, vol. 62, article no. 101115, 2020. https://doi.org/10.1016/j.pmcj.2020.101115
[22] N. Alshurafa, W. Xu, J. J. Liu, M. C. Huang, B. Mortazavi, C. K. Roberts, and M. Sarrafzadeh, “Designing a robust activity recognition framework for health and exergaming using wearable sensors,” IEEE Journal of Biomedical and Health Informatics, vol. 18, no. 5, pp. 1636-1646, 2014.
[23] C. T. Liu and C. T. Chan, “A fuzzy logic prompting mechanism based on pattern recognition and accumulated activity effective index using a smartphone embedded sensor,” Sensors, vol. 16, no. 8, article no. 1322, 2016. https://doi.org/10.3390/s16081322
[24] A. Grillon, A. Perez-Uribe, H. Satizabal, L. Gantel, D. D. Silva Andrade, A. Upegui, and F. Degache, “A wireless sensor-based system for self-tracking activity levels among manual wheelchair users,” in eHealth 360°. Cham, Switzerland: Springer, 2017, pp. 229-240.
[25] X. Ren, W. Ding, S. E. Crouter, Y. Mu, and R. Xie, “Activity recognition and intensity estimation in youth from accelerometer data aided by machine learning,” Applied Intelligence, vol. 45, no. 2, pp. 512-529, 2016.
[26] X. Gao, Z. Chen, S. Tang, Y. Zhang, and J. Li, “Adaptive weighted imbalance learning with application to abnormal activity recognition,” Neurocomputing, vol. 173, pp. 1927-1935, 2016.
[27] Y. Tong, R. Chen, and J. Gao, “Hidden state conditional random field for abnormal activity recognition in smart homes,” Entropy, vol. 17, no. 3, pp. 1358-1378, 2015.
[28] Z. A. Khan and W. Sohn, “Abnormal human activity recognition system based on R-transform and kernel discriminant technique for elderly home care,” IEEE Transactions on Consumer Electronics, vol. 57, no. 4, pp. 1843-1850, 2011.
[29] Z. Fan, X. Hu, W. M. Chen, D. W. Zhang, and X. Ma, “A deep learning based 2-dimensional hip pressure signals analysis method for sitting posture recognition,” Biomedical Signal Processing and Control, vol. 73, article no. 103432, 2022. https://doi.org/10.1016/j.bspc.2021.103432
[30] H. Cui and N. Dahnoun, “Real-time short-range human posture estimation using mmWave radars and neural networks,” IEEE Sensors Journal, vol. 22, no. 1, pp. 535-543, 2021.
[31] W. C. Su, X. X. Wu, T. S. Horng, and M. C. Tang, “Hybrid continuous-wave and self-injection-locking monopulse radar for posture and fall detection,” IEEE Transactions on Microwave Theory and Techniques, vol. 70, no. 3, pp. 1686-1695, 2022.
[32] C. Ma, R. Gravina, W. Li, Y. Zhang, Q. Li, and G. Fortino, “Activity level assessment of wheelchair users using smart cushion,” in Proceedings of the 11th International Conference on Body Area Networks (BodyNets),Turin, Italy, 2016, pp. 104-110.
[33] Arduino, “Getting started with the Arduino Pro Mini,” 2022 [Online]. Available: https://docs.arduino.cc/retired/getting-started-guides/ArduinoProMini.
[34] Force sensing resistor [Online]. Available: http://www.interlinkelectronics.com.
[35] TDK, “InvenSense MPU-9250,” 2022 [Online]. Available: https://www.invensense.com/products/motion-tracking/9-axis/mpu-9250.
[36] P. Gupta and T. Dallas, “Feature selection and activity recognition system using a single triaxial accelerometer,” IEEE Transactions on Biomedical Engineering, vol. 61, no. 6, pp. 1780-1786, 2014.
[37] S. M. Pincus, “Approximate entropy as a measure of system complexity,” Proceedings of the National Academy of Sciences, vol. 88, no. 6, pp. 2297-2301, 1991.
[38] C. Ma, W. Li, R. Gravina, and G. Fortino, “Posture detection based on smart cushion for wheelchair users,” Sensors, vol. 17, no. 4, article no. 719, 2017. https://doi.org/10.3390/s17040719
[39] I. Stolovas, S. Suarez, D. Pereyra, F. De Izaguirre, and V. Cabrera, “Human activity recognition using machine learning techniques in a low-resource embedded system,” in Proceedings of 2021 IEEE URUCON, Montevideo, Uruguay, 2021, pp. 263-267.
[40] M. Baradaran-K, S. K. Shekofteh, S. Toosizadeh, and M. R. Akbarzadeh-T, “A fuzzy approximator with Gaussian membership functions to estimate a human's head pose,” in Proceedings of 2010 10th International Conference on Intelligent Systems Design and Applications, Cairo, Egypt, 2010, pp. 1154-1158.
[41] S. J. Narayanan, C. J. Baby, B. Perumal, R. B.Bhatt, X. Cheng, M. R. Ghalib, and A. Shankar, “Fuzzy decision trees embedded with evolutionary fuzzy clustering for locating users using wireless signal strength in an indoor environment,” International Journal of Intelligent Systems, vol. 36, no. 8, pp. 4280-4297, 2021.
[42] M. Orouskhani, D. Shi, and X. Cheng, “A fuzzy adaptive dynamic NSGA-II with fuzzy-based borda ranking method and its application to multimedia data analysis,” IEEE Transactions on Fuzzy Systems, vol. 29, no. 1, pp. 118-128, 2020.

Congcong Ma1, Juan Du1, and Raffaele Gravina2,*, Abnormal Behavior Detection Based on Activity Level Using Fuzzy Inference System for Wheelchair Users, Article number: 12:21 (2022) Cite this article 1 Accesses