홈으로ArticlesAll Issue
ArticlesUsability Engineering of a Tangible User Interface Application for Visually Impaired Children
  • AbirBenabid Najjar1,*, Asma Alhussayen1, and Rabia Jafri2

Human-centric Computing and Information Sciences volume 11, Article number: 14 (2021)
Cite this article 4 Accesses
https://doi.org/10.22967/HCIS.2021.11.014

Abstract

Educational applications provide engaging learning experiencesparticularly for children, but their visual nature limits their usefulness and accessibility for the visually impaired (VI). By utilizing physical objects in the interactions, tangible user interfaces (TUIs) couldmake educational applications more accessible for VI children. Fully understanding the specific needs of users and involving them in the system design and testing are necessary for developing usability and accessibility guidelines to improve the overall user experience for such applications. This paper reports on theuser-centered design process for an innovative, engaging TUI-based educational application for VI children, wherein several gamification elements were included to enhance the user experience and increase engagement. An evaluation was also conducted overtwo iterations to assess the TUI application’susabilityquantitatively and qualitatively. The results of the evaluation sessions generated several design recommendations and guidelines to enhance educational TUI-based applications for VI children.


Keywords

Tangible User Interface, Visual Impairment, User-Centered Design, Education, Assistive Technologies


Introduction

According to recent estimates by the World Health Organization[1], 2.2 billion people worldwide suffer from vision impairments. In Saudi Arabia, about 1 million people are visually impaired (VI). Therefore, providing adequate education facilities for the VI is a major concern. While there are a few schools exclusively focused on VI children, similar to other countries, the current trend in Saudi Arabia has been to mainstream VI students into regular schools where they receive special education classes with dedicated teachers[2]. Nonetheless, VI children require learning to be adapted to their unique needs; for example, teaching VI children usually requires the teacher to pay continuous attention and give regular feedback, which takes time and effort.
While many fun and engaging educational applications have been developed to reinforce the concepts learned in class for sighted students, their visual nature limits their usefulness forVI children as VI learners rely on touch and manipulation of objects to develop conceptual understanding. This particular learning requirement could be addressed usingtangible user interfaces (TUI), which allow users to manipulate physical objects mapped to computer systems, thereby overcoming any graphical user interface (GUI) deficiencies. Consequently, there have been several TUI-based educational applications developedfor the VI for different purposes and ages[36] in recent years, most of which have been found to be effective in enhancing learning.
This research proposes a TUI-based application for VIchildrenas well as design recommendations and usability guidelines to enhance educational interactive systemuser experiences for VIchildren. First, alocal needs assessment was conducted[7]by observing the educational environments and interviewing special education teachers to identify theVI challenges when using educational tools and applications. A TUI-based application for VI children was then developed by employing well-known practices of designing for children with special needs to meet the identified requirements. The usability of the system was then evaluated in two iterations, with the first prototype evaluated by a special education teacherand the second prototype assessed by VI students at a local school to investigate efficiency, effectiveness, and user satisfaction. Learnabilityas captured by a learning curverevealed usability improvements after repeated trials.
The rest of this paper is organized as follows: Section 2 gives anoverview of the relatedwork; Section 3 details the design of the educational TUI-based application;Section 4presentsthe evaluation process and results; Section 5 discusses the results and derives design decisions, limitations,and possible future work directions; Section 6 presents theconclusion.


Related Work

This section first presents a review of the existing TUI-based educational applicationsfor VI children,and then reviews someusability evaluation methods involvingchildren.

TUI-based Educational Applications for VI Children
Applications to learn Braille
Jafri[8] proposed a low-cost software solution for the teaching of Braille to VI children, which required the manipulation ofnear-field communication tag-embedded blocks with embossed Braille letters on their side and audio feedback. Theblocks were found to have madethe learning effortless and encouragedcollaborative activities. Bintaleb and Al Saaed[9] designed an interactive tactile Braille keypad to help blind children learn Arabic Braille letters and numbers, with the pilot usability evaluation on three blind children indicating learning improvements. Recently, Gadiraju et al.[5] introducedBrailleBlocks, a system that included a set of interactive games to help VI children learn and practice Braille together with a sighted person. Using the system, the VI children assemble Braille letters and words and receive audio and multimedia feedback. Sighted parents and teachers can also use a GUI to play with and learn Braille together with VI children.

Applications to learn geometry and shapes
Manshadet al.[10] introduced trackable interactive multimodal manipulatives, which aremarked programmable objects that can be moved by users onan interactive tabletop, to enable VI children to learn geometry conceptsindependently. To teach VI childrentactual shape perception and spatial awareness concepts, Jafri et al.[3] presenteda TUI-based spatial applicationthat utilized tagged 3D-printed custom geometric shapes and provided audio instructions and feedback. The evaluation of the system with teachers of VI children validatedits potential and provided valuable insights into design considerations for such systems. Aduseiand Lee [11] developed Clicks, a digital manipulative to simplify the learning of geometric concepts and. Itconsisted of a construction kit with basic geometric shapes that snapped together to produce more complex shapes. These objects were then placed on a tablet to identify their shape and provide audio feedback. Lozano et al.[6] designed “Touch&Learn,” aTUI-based system to teach basic concepts such as Braille numbers, shapes, and textures to VI children, wherein the systeminteractions were based on speech recognition,touch, and sound.

Applications to learn mathematics and arithmetic operations
Breiter et al.[12] evaluated the usability of the AutOMathic Blocks System, whichallowed VI students to solve arithmetic problems using blocks embossed with Braille numbers and operators with barcodes. The selected block was identified using a barcode reader, and the studentplaced the block on a touch-sensitive tablet. The system was evaluated with both sighted and blind students, and both groups were found to have preferredtwo-dimensional math problem representationsrather than linear presentations. Avila-Soto et al.[13] proposed the TanMath system for teaching VI students basic math operations utilizing tangible numbers;computer vision techniques wereemployed to identifythe numbers while providingauditory feedback. To identify the TanMath requirements, interviews were conducted with VI educators,with a Wizard-of-Oz evaluation technique employed with one VI student. Pires et al.[4] conducted participatory sessions with VI children and their educators as a preliminary step to developing a TUI system that used tangible blocksof different sizes and colorsfor the numbers 1 to 5 and gave auditoryfeedback, which allowed VI children to solve addition or subtraction problems and interact with the system.

Usability Evaluations with VI Children
Bruleet al.[14] reviewed a corpus of 178 papers on quantitative empirical VI technology evaluations and found thatthe design and conduct of suchevaluationsremain a challengeespecially for VI children. Based on experience in working with VI children for several years, Raisamo et al.[15] assessed the usability of multimodal applications with VI children using questionnaires, interviews,exploratory usability testing,child tutoring of the parent about the system, andvarious observation methods that had been refined to suit the VI children; they found that conducting evaluations in familiar environments for the children, such as school, elicited more comments and suggestions. Darin et al.[16] sought to identify the mostsuitable usability evaluation methods (UEMs) for discovering interface usability design problems for a blind audience byconducting evaluation sessions using different UEMs; it was found that observation through video recording could identify majority of the usabilityproblems, and that the questionnaire was the least useful method.
Xu [17] investigated the problems withthe existing TUI evaluation methods for non-VI children and sought to identify the mostsuitable methodsfor non-VI children using an RFID puzzle game prototype on the life of the Romans. The author found that thethink aloud (TA) method was a little difficult to apply as itrequired prompting the childrento talk,that peer tutoring (PT)was easily understood—requiringlittle evaluator intervention—that the drawing intervention (DI) methodrevealed information about the TUIs not acquired from the traditionalmethods but further validation was needed, and that observation methods (videotaping and note taking) were important. Like Raisamo et al. [15], Xu [17] found thatchildren felt more comfortable and focused in a school environment compared to a lab.

Summary and Discussion
Table 1 summarizes some of the existing TUI-based educational applications for VIchildren. Previous studies found that there are several challenges in designing attractive and usable TUI-based applications for VI children. For example, software designers are required to design both physical and interrelated digital operations instead ofonly the digital found in traditionalGUI-based applications, and they also need toconsider the special needs of the target users (VI children). To achieve this, a user-centered design (UCD) approach is needed with the VI children as design partners.

Table 1. TUI-based educational applications for VI children
Study Year Learning objective Proposed system
Breiter et al. [12] 2012 Arithmetic problems AutOMathic; blocks embossed with Braille numbers and operators with barcodes attached to them
Manshad et al. [10] 2012 Geometry concepts Trackable Interactive Multimodal Manipulatives (TIMMs); marked programmable objects on an interactive tabletop
Jafri [8] 2014 Braille letters NFC tag-embedded blocks with audio feedback
Jafri et al. [3] 2017 Tactual shape perception and spatial awareness concepts Application based on tagged custom 3D-printed geometric shapes with audio instructions and feedback
Adusei and Lee [11] 2017 Geometry concepts Clicks; a construction kit with basic geometric shapes that snap together to produce more complex geometrical shapes, placed on a tablet
Avila-Soto et al. [13] 2017 Basic math operations TanMath; a concept of a system utilizing tangible numbers and computer vision techniques to identify numbers 
Lozano et al [6] 2018 Braille numbers, different shapes and textures TUI-based system based on speech recognition, touch, and sound 
Pires et al. [4] 2019 Simple math problems and additive composition tasks Tangible blocks of different sizes and colors representing numbers 1 to 5 and giving auditory feedback 
Bintaleb and Al Saeed [9] 2020 Arabic Braille letters and numbers Interactive tactile Braille keypad using an Arduino connected to a website
Gadiraju et al. [5] 2020 Braille letters and words BrailleBlocks; a set of interactive games with audio and multimedia feedback 


Evaluating the developed software with potential users is important in determining their perceptions of the system and the possible usabilityissues. Nevertheless, TUI-based applicationusability evaluationsare difficult for several reasons, particularly with VIchildren. First, the unique and distinct TUI-based application interface requires adjustments to the current evaluation methods to ensure proper evaluation of the applicationproperties. Second, evaluation methods for children are different from those for adults as they need to focus more on observation to capture signs of engagement. Third, the evaluation methods mustbe compatible with the capabilities of the VI children evaluating the TUI-basedapplication. The studies discussed in Section 2.2identified some of the most appropriate evaluation methods and processesfor assessing certain applications with children, with almost all previous studies emphasizing the importance ofan iterative system evaluationapproach to reveal the usability issues and acquiring valuable input from the actual systemusers.


User-Centered Design of the TUI Application

In this study, a UCD approach involving VI children throughout the system development lifecycle, as detailedin the following sections, was adopted:

Context and Requirements
The Saudi Arabian education system generallyencouragesVI students to be mainstreamed in regular schools with sighted students and to receive special education programs from dedicated teachers. This paper extends the requirements elicitation study reported in [7], with the additional user requirements identified through field observations at a local inclusive Saudi Arabian elementary school offering special education classes for VI students. Three sessions for teaching reading, writing, and mathematics to four VI firstgrade students (one blind and three with some residual vision) were observed, followed by semi-structured interviews with three special education teachers for grades one to three at the same school. The interviewsfocused on the needs of the VI students, the challenges faced by the teachers and them,and the design recommendations particular to a TUI educational system. The studyrevealed the extensive use of tangible materials to support the learning process;althoughthe interviewed teachers realized the value of educational computer-based softwarefor improving the learning environment, the study revealed the absence of any computer-based assistive technologies in the class.Mathematics was found to be the most appropriate subject area for a TUI-based application for VI students to use individually and/or collaboratively in class. Based on the observed materials and the teachers’feedback, several tangible objectcharacteristics,such as the materials, shapes, sizes, and colors ofthe objects, were identified. Furthermore, the importance of games and teamwork to keep the students engaged and competitiveand the need for repetition to allow them to graspand memorize the conceptswere established.
Based on these requirements, a TUI-based applicationcalled Practice Math with Braille Blocks (PMBB) was designed for elementaryschool VI students to practice mathematical concepts.

System Overview
PMBB is a TUI-based educational applicationto practice mathematical operations such as addition, subtraction, multiplication, and divisiondesigned for VI children from 4 to 12 years old (preschool to 6th grade) who know the Braille numbersystem and understand spoken Arabic.The applicationdesign includes gamification elements such as themes, collaboration, levels, and points to motivate and engage the students. Two game themes weredeveloped, wherein the students achieve progress by correctly solving mathproblems: (1) baking a cake, which requires the student to gather cake ingredientsand (2) gettingready for bed, which requires students to help a virtual character performthe steps toget ready for bed. The PMBB hastwo playmodes wherein different types of points can be gained: (1) a single mode for a single player to earn individual “performance points” and (2) a collaborative mode wherein playerscan log in together and earn “collaborative points.”
The student interacts with the system by placing blocks on a transparent plexiglass tabletop that identifies each block from a tag affixed to its base. Thesystem provides audio instructions and feedback asfollows.The student logs into his/heraccount by placing the account block withhis/hername on it on the tabletop. He/she selects a theme byplacing a theme block, the system asks some questions based onthe difficulty level assigned to the student. Currently, two levels are supported: a beginnerlevel, wherein the student has to represent numbers (Fig. 1(a)); and an advanced level, wherein he/she has todo addition, subtraction, multiplication, and division problems (Fig. 1(b)).
The system validates the placed objects, gives audio feedback, and calculates the earned points. Upon successfully completinga level,students access to the next one. The student can log out of the system at any time by placing an account block on the tabletop. The PMBBkeeps a record of the name, the assigned difficulty level and the “performance” and “collaborative” points of each student.
The instructions and feedback were designed to motivate the children to continue, for which a real female child’s voice was used rather than a synthetic voice. Placing a block on the tabletop triggers differentsound effects that indicatecorrect or incorrectinput. As the system was to be testedwith students in local schools, the speech output was in Arabic.
The system also has a visual screen-based interface for the teachers tointeractwith using a keyboard and a mouse, through which the teacher can add,modify, or delete a student account, assign difficulty levels, and retrieve performance reports for each student for any given periodor for anyspecific difficulty level.
Fig. 1. Representations using Braille blocks: (a) number “37” and (b) representation of the equation “7–3 = 4.”


Interaction Design
As the application has two different interfaces, two different interaction models are usedto represent the interactions:(1) a model-view-control (MVC) architecture that implements the application for the teacher, as shown in Fig. 2(a)and (2) amodel-control-representation (physical and digital) (MCRpd) introduced in [18] and used as the interaction model for the TUI-based subsystem as shown in Fig. 2(b). Derived from the MVC model, this preserves the modeland control components but divides the view component into two representation components: (i) a physical representation (rep-p), which is the tangible representation of informationand (ii)a digital representation (rep-d) such as video or sound. In Fig. 2, rep-p is integrated with the control, which shows the direct effect of manipulating the physical representation on the control element,andis also coupled with the rep-d and the model.
Fig. 2. PMBB Interaction models: (a) MVC architecture for the Teacher app and (b) MCRpd architecture for the Student TUI.

1. An 11.8×22.4-inch plexiglass top attached to a table usingC clamps.

2. A Logitech webcam connected to the computer usingUSB cable to capturelive video stream from underneath the plexiglass top and send it to the computer; the distance between the webcam lens and the plexiglass is 18 inchesto capture the full plexiglass space.

3. Tangible blocks with identifying tags attached to their bases; a detailed description of these blocks is given in Section 3.4.

The following software components were also considered:

1. ReacTIVision [19]: An open source, cross-platform software providing up to 216 different tags that can be attached to tangible objects; when the objects are placed tag side down on the plexiglass top, the system identifies the tags and determines their positions and orientations from the input video stream of the webcam placed beneath the top. This data is then sent to the controller (client application) through the TUIO protocol [19].

2. Controller: The client application receives and processes the tangible object data to provide the appropriate feedback.



Tangible Object Design
Four tangible object types were designed based on the requirements identified in the previous study [7]. (1) Braille blocksare brightly colored rectangular cork blocks with Braille numbers, arithmetic operators, and symbols embossed on the top, with a rectangular marker on the upper lateral edge indicating the upward position of the block. Blocks of the same kind are similarly colored to aid students with residual vision in distinguishing them. (2) Account blocksare beige-colored wooden cubes with the students’ names embossed inBraille on the top. (3) Theme blocksare plastic blocks with toy bed and toy cupcake affixed to the top. (4) Confirm and cancel blocksare plastic blocks with foam cancel and confirm cutouts symbols affixed on the top. The different tangible object types were made with different materials to make it easier to distinguish them tactually. The tangible objects are shown along with their characteristics in Table 2.

Table 2. Description of tangible objects
Material Colors Represented data Pictures
Cork Yellow, red, blue and green Braille numbers, operations, and symbols
Wood Beige User names
Plastic White, pink and beige Game themes: “Get ready for bed” and “Bake a cake”
Plastic Red and orange Confirm and cancel options


Usability Evaluation of the Proposed System

Usability Evaluation Model
The ISO 9241-11 standard identifies efficiency, effectiveness, and satisfaction as the major usability attributes. Some studies have proposed usability models that include learnability as an important factor; nonetheless, [20] argued that the new draft (ISO 9241-11:2018) made it clear that usability, as defined in terms of effectiveness, efficiency, and satisfaction, applies to all aspects of use including learnability.
In this study, the evaluation process involved going through the system while assessing the conformance of its design to these usability factors. Table 3summarizes the evaluation framework. The evaluation was guided by a set of predefined heuristics based on Nielsen’s 10 heuristics, which have been widely used for user interface design. Usually, these heuristics are applied to usability inspection methods by expert reviewers at the early design stages, which means that the results can be influenced by the experts’ judgments. In this study, however, Nielsen’s heuristics were used to guide the usability evaluator who was also the facilitator during the user testing, examine the usability problems, and judge the compliance of the system based on the usability factors. Therefore, to assess the fulfillment of each factor, each heuristic was mapped to the appropriate usability factor and the metrics. The heuristics along with the usability factors and metrics are shown in Table 3. This model was used in the usability evaluations of the two functional prototypesof the PMBB system as detailed in the following sections.

Table 3. Usability evaluation framework for TUI with VI users
Heuristics Usability factors Metrics
H1. Visibility of system status: Clarity of instructions for providing user guidance Effectiveness Error rates due to misunderstanding audio instructions Requests for help
H2. Match between the system and the real world: Ease of identifying each tangible object and distinguishing among the different objects  Effectiveness Error rates due to placement of the wrong type of object Requests for help
H3. User control and freedom: Freedom to place the tangible object anywhere on the table and have it recognized  Effectiveness Error rates due to incorrect recognition of an object
H4. Consistency and standards: Arrangement of the objects in correct order with respect to Braille conventions in writing mathematical equations  Effectiveness Error rates due to incorrect arrangements
H5. Error prevention: Prevention of a problem from occurring or delaying task completion  Effectiveness Types of errors preventing task completion Types of errors delaying task completion User responses and comments
H6. Recognition rather than recall: Ease of quickly finding the correct object and placing it correctly Efficiency Time elapsed between starting searching for an object and locating it Time spent to place an object correctly
H7. Flexibility and efficiency of use: Efficiency for both inexperienced and experienced users Efficiency Time spent to complete each task correctly in different levels 
H8. Help users recognize, diagnose, and recover from errors: Responsiveness and clarity of feedback associated with the correct/wrong input of different objects Effectiveness Error rates due to misunderstanding audio feedback Requests for help
H9. Aesthetic and minimalist design:(i)Aesthetic and minimalist design of tangible blocks as well as the table. (ii) Non-annoying feedback given during the interaction, shall not contain irrelevant or rarely needed information  Satisfaction Facial expressions after hearing the feedback User responses and comments 
H10. Help and documentation: Continuous assistance and help Effectiveness Facial expressions after hearing the feedback Requests for help


First Prototype Evaluation
An exploratory usability test was conducted with a teacher of VI students, whohad also participated in the requirement-gathering interview described in [7]. The teacher had some residual vision and 9years’ work experience teaching firstgrade students.

Procedure
The session began by briefing the participant on the purpose of the evaluationand the functionalities in the student mode of the system. The participant was then askedto perform the following six principal tasks in student mode: (1)examining the tangible objects before using the system;(2) logging in to the system; (3) selecting a game theme;(4) playing the beginner level; (5) playing the advancedlevel; and (6) logging out of the system. As the TA protocol was employed,the participantswere asked to say their thoughts aloud as theyperformed the tasks and explain any issues or difficulties they encountered. The data were gathered from theparticipant’s audiorecordingand automaticallylogged to keep track of the timespent on each task and the input errors.Observation notes were made on a data collectionform to record the information related to the number and type of errors, numberand type of assistance requested, and comments for each task in the session.

Results
Tasks 1, 2, 3, and 6 were completed within 2–5 minutes, which was reasonableaccordingto the teacher. Nonetheless, tasks 4 and 5 (completingthe beginner and advanced levels) required 20–30 minutes,which the teacher felt was too long. Based on the teacher’s comments and our observations, such slow progress was attributed to the usability problems (P) outlined in Table 4, which also includes the corresponding Nielsen’s heuristics (H) and the required system modifications (R). The additional requirements and design improvements were implemented to develop the second prototype, which was again evaluated with VI students and is discussed in the subsequent section.

Table 4. Usability problems identified in the pilot usability testing
Usability problems (P) Heuristics Design recommendations (R)
P1: Inadequacy of the system feedback, as some errors occurred due to minimal system feedback and imprecise response to errors   H1: Visibility of system status R1.1: The system shall repeat the presented math problem in case of no activity for 10 seconds.
R1.2: The system shall provide verbal instructions as hints to the user when a mistake is made.
R1.3: The system shall provide more guidance (instructions about the next step) to the user while solving a math problem.
P2: The large number of math problems presented in a theme contributed to the lengthy time spent on the task. H7:  Flexibility and efficiency of use R2: The system shall present 5 (instead of 10) math problems to complete a theme for the beginner's level and the advanced level. 
P3: Searching through the many number blocks to locate a particular number block took a considerable amount of time, causing the participant to forget the presented problem.  H6: Recognition rather than recall R3: To simplify the search for a required block, the tangible blocks shall be grouped and categorized based on how numbers and operations are taught in class.
P4:In an attempt to offer flexibility, the design of the plexiglass top did not include specific placements for the blocks. This approach caused confusion as when to leave a block on the table and when to remove it. H5: Error prevention R4: Specific placements for the account and theme selection blocks should be fixed, so that users are not required to remove them from the table until they have finished playing. 


Second Prototype Evaluation
To evaluate the usability of the tangible interaction ofthe student mode of the PMBB system, anassessment usability test on the second prototype was conducted with sevenfemale VI students from a publicinclusive school in Riyadh. All participants have had experiencewith mobile devices such as smartphones and IPads,with one participant having had experience using a computer with a Braille keyboard. The types of games they had played on the IPad were coloring games and puzzles; on the computer, the studentsreported that they had played aneducational game that presentedquestions and for which they had to input answers usingthe keyboard. Their visual impairment was roughly assessed throughtheir ability to differentiate similar colors (such as green and blue) and identifyshapes without touching them. Participants with “good residual vision” are ableto identify blockshapes and differentiate similarly colored blocks visually. A “poor residual vision” assessment suggestedthat the participant was unable to separate similarly colored blocks or to recognize the block shape without a tactual examination. Table 5 summarizes the participant demographics.

Table 5. VI participants’ demographics
ID Age Grade  Impairment level Electronicdevices Games
7A 7 2nd High residual vision iPad Coloring games
7B 7 2nd Blind Mobile Coloring games
8A 8 3rd High residual vision iPad Coloring games
10A 10 3rd High residual vision Mobile & PC General knowledge games
10B 10 4th High residual vision None None
11A 11 5th High residual vision iPad Coloring games
12A 12 6th High residual vision iPad Coloring games


Procedure
The system was set up inside a schoolclassroom (Fig. 3), and tangible objects were organized based on the teacher’s recommendations in the previous evaluation. The webcam was fixed inside the plexiglasstable, and a video recorder was placed on a tripod next to the computer monitor tocapture the participants’ interactions with the system. Individual sessions with eachstudent were conducted in the classroom and were about 45–60 minutes long. A special education teacher was present during the sessions to provide some encouragementto the participantsand help answer any questions.
Fig. 3. PMBB system setup in the classroom.


Before starting the session, the evaluation purpose and procedure were explainedto and approved by the participating student’s parents. The session was then conducted as follows. In the first 10 minutes, the system was explained to the participants, and a brief background interview was conducted to learn about their experiences with technology, the types of educational games they had played, andthe difficulties, if any, that they had experienced. Theparticipants were then given 10 minutes to explore the tangible objects and the table space, and they had 30 minutes to perform the tasks. Finally, a post-test interview was conductedin the last 10 minutes to gather information about their preferences, likes and dislikes and to follow up on any problems that mayhave come up during the session.
The testing involved six main tasks that were individuallyperformed by each participant inthe same order: (1) examining the tangible objects and identifying their purpose; (2) logging in to the student account; (3) selecting the game theme; (4) playingthe beginner level (which required representing five numbers with the Braille blocks); (5) playing the advanced level (which required representing and solving five mathproblems); and (6) logging out of the system.
The tasks were designed to assess the usability factors based on the usability evaluation model shown in Table 3. The data were collected through: (1) forms for recording the information related to the number and types of errors, number and types of assistance requests, and comments for each participant for each task in a session; (2) video recordings for capturing the participant’s facial expressions; and(3) automatic logger to keep track of the time spent on each task.

Results
The evaluation was conducted using the modelpresented in Table 3 to map the usability factors and metrics. The results were then organized based on theefficiency, effectiveness, and satisfactionusability factors,and learnability was also assessed to identify the usabilityenhancement over time.
Efficiency
To assess system efficiency, the completion time ofeachtask was recorded.All participants performed the login, theme selection, and logout tasks in less than aminute. The beginner level task was completed by all participants within the expected time limit of 10 minutes (median = 9 minutes; SD = 0.82 minutes). For theadvanced level, twoparticipants required the teacher’s assistance to complete thetask; for the other five participants, significant variations in completiontimes were observed based on their age, with the older students (4th–6th grade levels) and theyounger students (2nd and 3rd grade levels) taking a mean time of 15 minutes and 29 minutes, respectively, to complete the tasks.
Effectiveness
To measure the effectiveness,the errors that occurred while performing the taskswere recorded along with the system’s response. The number and types of errors for both beginner and advanced levels are shown in Table 6. Forgetting to place the preceding “#” block and “code” block before the respective operands and arithmetic operators (as required inthe Braille system)was the most common error(50% of the errors in the beginnerlevel, 40.38% of the errors in the advanced level). Confusing numbers and operators wasalso a common erroron both levels.
Additional errors were specific to theadvanced level, such as forgetting to place the operator block between the operands. In addition, while solving the math problem during the advanced level, one participant gathered and placed the blocks on the table all at once, causing multiple errors. In thiscase, the system failed to detect the problem, and the teacher’s intervention wasrequired to explain to the participant what had happened and how to avoid it.In the beginner level, the participants were able to recover from the errors byfollowing the instructions provided by the system and to complete thelevel with fewer errors. In the advanced level, five of the seven participants completedthe level by following the system instructions. Nonetheless, participants 8Aand11Arequired the assistance of the teacher to complete the tasks; this suggests that the sound effects feedback and instructions neededto be revised to better capturethe user’s attention.
During the evaluation session with the blind participant (7B), the participantplaced a block at the edge of the table, and it fell off. This error was notdetected by the system as it had already detected the presence and position of theblock before it fell off. Therefore, it was decided that a border around the table would prevent such problems fromoccurring.
The system audio feedback and response to the user inputs were able to guide majority of the participants in completing the tasksindependently, with most participants making fewer errors as they progressed through the task.

Table 6. Number and type of errors
ID Beginner level Advanced level
Forgetting "#" or "code" signs Confusing number/ operator blocks Forgetting "#" or "code" signs Confusing number/ operator blocks Forgetting operands Forgetting operators Wrong calculation Wrong order of the blocks
7A 1 0 2 2 1 2 1 0
7B 1 0 1 1 0 1 0 0
8A 1 3 3 2 0 2 0 5
10A 1 1 4 2 0 0 0 0
10B 1 1 3 1 0 1 0 0
11A 1 2 4 4 0 0 4 0
12A 1 0 4 1 0 1 0 0


User Satisfaction
After performing the tasks, the sessions ended with an interview to assess the participants’ satisfaction. The interview questions covered different aspects of the PMBBsystem, such as audio feedback (sound effects and instructions) and usecontext. The user responses were categorized into positive and negative feedback, with thepositivefeedback beingresponses that indicated satisfaction (such as like, nice, fun, etc.) and negative feedback being responses that suggested dissatisfactionwith an aspect of the PMBB software (such as annoying, confusing, hard, etc.).
Fig. 4. Number of positive and negative feedback provided by the participants.


Fig. 4 shows that 80% of the responses were positive and 20% were negative.The design of the interaction objects was found to be satisfactory by all participants. Similarly, the responses regarding the math practice activities were all positive, indicating that they had enjoyed the activity. Majority of the participants found the system easy to use and the audio instructions clear andadequate, with five of the seven participants voting in favor of using the system as partof their daily classroomactivities. Majority of the negative responses wererelated to the sound effects, blockorganization, and use of the systemin collaboration with their classmates. Three of the participants described the “incorrect”sound effect as being annoying and startling, one participant thought the audio instructionswere unnecessary and indicated that a sound effect would be sufficient feedback, three participants found searching for the desired block among the groupof blocks to be difficult, and four students expressed preference to playalone rather than with others or even at home rather than at school.
Learnability
Learnabilitywas examined bystudying the learning curve to identifythe usabilityimprovementsafter repeated trials. Bothhigh and low grade participants had goodlearning curves sincethey made fewer errors as they progressed through the levels. In thebeginner level, the number of errors per trial decreased to zero from the secondtrial, implying that they hadquickly grasped the interaction process. As shown in Fig. 5, however, the decrease in the number of errors was more gradual inthe advanced level, indicating that the interaction process was more complex than the beginner level.
Fig. 5. Average number of errors per trial in the advanced level.


Participants 8Aand 11Amade the same number of mistakes throughoutthe task but were able to complete the fifth trial (final math problempresented in the task) following the teacher’s instructions. Some participants(7A, 10A, 10B, and 12A) gradually reduced their number of errors per trial, whereas participant 7Bwas able to reduce the number of errors to zero from the 2nd trial. Nevertheless, theyspent 7–10 minutes more to complete the task compared to the other participants. These results reflected the easeof learning and understanding howto use the system for majority of the participants despite the relatively smallnumber of trials.


Findings and Discussion

The results of evaluating the PMBB system showed that it was capable of detecting most of the mistakes in representing and solving math problems using Braille blocks. The audio sound effects and instructions allowed the users to represent and solve thegiven problemcorrectly. Most participants had goodlearning curves as they progressed through the levels; thusindicating thatlearning to use the system was relatively easy. The objectdesignreceived positive feedback, and there was good user satisfaction overall. Despite the positive evaluationresults, which demonstrated that thesystem was capable of efficiently and effectively providing the requiredfunctionalities, further design improvements could enhance the system effectiveness and users’ experience.

Design Recommendations
Further design recommendations weredrawn from the gathered data, specifically from the common user errors and negative systemfeedback. The suggested improvementsare detailed in the following paragraphs, and then summarized and mapped to the corresponding Nielsen’s heuristics in Table 7.

Table 7. Summary of design recommendations
Current system design User feedback Heuristics Designrecommendations
Feedback Harsh error soundeffect  Startling andannoying  H9: Aesthetic and minimalist design Use a soft soundeffect for wrong input 
Audioinstructions to correct a wrong input Noisy, notrequired  Present the audioinstructions only upon user request 
Interaction blocks and table 4 cm×6cmrectangular blocks; every two number blocks in one basket and the operatorblocks in one basket  Difficult andtime-consuming to find the desired blocks  H6: Recognition rather than recall Design smallerblocks (2 cm×3cm) and place each block type in one basket 
The bottomcorners are dedicated for Account and Theme blocks, while the remaining areaused for organizing other blocks  When placing theblocks all together,, they are detected by the system before beingorganized  H5: Error prevention Form a grid onthe tabletop for solving math problems and dismiss any block placementoutside the dedicated grid 
Plexiglasstabletop without borders  When placed atthe edge, the blocks might fall after being detected  Place bordersaround the plexiglass tabletop to prevent blocks from falling 
Game design The beginnerlevel includes writing Braille numbers of single digit and more  Some similarnumber and operator blocks were confusing  H2: Match between the systemand the real world The beginnerlevel should be divided into several levels focusing on the typicallyconfusing blocks
  Audio feedback isplayed through speakers  Avoid noisedistraction in class  Use headsetinstead of speakers


System feedback
The quantitative results indicated that the current system’s feedbackand instructions were effective in guiding the participants in completing the taskssuccessfully. Still, the qualitative results revealed some dissatisfaction with the audio sound effectsand instructions. A few participants found the sound effect that was used for an incorrect input to be annoying and upsetting; therefore, a softer error sound effect could reduce this discomfort. One participant commented that the audio instructions were notneeded; this was also observed in the video recordings when some participants realized that they had made mistakes as soon as an error sound effect was played but before listeningto the correction instructions. Therefore, to avoid excessive instructions, users couldbe providedwith minimum feedback when they make errors (only the sound effect); if needed, they could requestadditional instructions using an object (shaped as a question mark) that functions as a “help” option.

Interaction blocks and table
In the sessions, the blocks were organized in baskets wherein every two numbers were placed in one basket (two blocks of each number) and the operation blocks werein another basket (also two blocks of each operation type). Three participants found the search for the needed block to be difficult. Therefore, placing each type of blockin a separate basket could simplify the search process; as this would require alarger table space, however, this could be resolved by designing smaller-sized blocks and using smaller containers. When one of the participantsgathered the requiredblocks and placed them all on the plexiglass tableat the same time, the system registered multiple errors because it detected the tags and identified the blocks as soon as they were placedon the plexiglass table and before they were organized correctly. To prevent these types of errors, there should be a specific interaction area with borders so that only the blocks placed inside the interaction area are detected. Fig.6(a) and 6(b) below illustrate the current system tabletop and the recommended redesigned tabletop. Thisinteraction table design enhancement was also recommended in a recentlypublished study on a TUIeducational system for VI children [3]. In addition, addinga border around the plexiglass tabletop would prevent the placedblocks from falling off the edge of the table as experienced by the blind participant.
Fig. 6. Tabletop layout: (a) the current system tabletop and (b) the recommended redesigned tabletop.


Game design
As shown in Tables 5 and 6, the second most common mistake in both levels was confusing similar numbers and operation types. To avoid this problem,additional levels that provide practice with similar Braille numbers and arithmeticoperators should be added to the game levels. Some participants also preferredusing the system at home rather than in the classroom to reduce the distractions inclass. Providing a headset with the system could resolve this issue, though.

Limitations and Future Work
The needs assessment and usability evaluation in this study were conducted in only one inclusive school. Therefore, conducting field observation inmore schools and with more VI students could mean more precise specificationsthat better represent the local VIeducational environment. Thisstudy also involved only female VI students; therefore, future work should also include inclusive schools forboys to allow for the identification of any gender-specific differences in the system requirements.
The systemconfiguration used to track and identify the physical objects was somewhat complex because of the required settings on thereacTIVision system, such as camera calibration, focus adjustments for the classroomsystem setup thatrequired the carrying of a heavy plexiglass table, and need to control the classroom illuminationto achieve the correct tag identification. Therefore, future exploration of different tracking systemsis necessary to compare tracking techniques, configurationcomplexity, identification and tracking capabilities, and setup expenses.
Despite the positive results of the usability evaluation, additional system enhancements and functionalities are still needed to improve the user experience further.First, the design recommendations elicited from the usability evaluation are to beimplemented to reduce user errors and provide greateruser control over the systemfeedback. Second, additional features could be added to the system to increaseinteractivity and engagement, such as providing rewards for advancing to the more difficult levelsand offering a greater number ofstory themes. Third, applications for other disciplinessuch as reading and writing in Braille could be designed to expand the educationalbenefits of the system. Finally, full implementation and evaluation of the functionalitiesprovided for the teacher are important to incorporate the system as an efficient tool for assessingstudent progressin class. Furthermore, implementing additionalfeatures, such as assigning tasksto a particular student or addingdifficulty levels other than the predefined levels,couldenhance systemproductivity.


Conclusion

This researchfollowed a UCD process to propose, develop, and evaluate a TUI-based system for VI children to practice mathematical problem solving.The component-based systemarchitecture facilitates its future reuse by TUI-based applicationdesigners; for example, thereacTIVision system couldbe conveniently replaced by someother tracking system.
The usability evaluation of the designed TUI-based system revealed the following: (1) the usability evaluationprocedure suited the needs and capabilities of VI children and (2) furtherdesign modifications were necessary for this kind of applications. These design recommendations are of value to the TUI-based educational application developmentresearch field as they could generallybeapplicable to the design of any interactive applications oriented toward VI children.
This study advances the design and development of TUI-based applications for the education of VI children and contributes to theiremancipation and integration into society.


Acknowledgements

The authors would like to thank the Deanship of Scientific Research and RSSU at King Saud University for their technical support.


Author’s Contributions

Conceptualization, AA. Investigation and methodology, AA. Project administration, BA, JR. All authors read this paper and approved the submission.


Funding

This research project was supported by a grant from the Research Center of the Female Scientific and Medical Colleges, Deanship of Scientific Research, King Saud University.


Competing Interests

The authors declare that they have no competing interests.


References

[1] World Health Organization, “World report on vision”, 2019 [Online]. Available: https://www.who.int/publications-detail/world-report-on-vision.
[2] Z. M. B. Battal, “Special education in Saudi Arabia,” International Journal of Technology and Inclusive Education, vol. 5, no. 2, pp. 880-886, 2016. http://doi.org/10.20533/ijtie.2047.0533.2016.0113
[3] R. Jafri, A. M. Aljuhani, and S. A. Ali, “A tangible user interface-based application utilizing 3D-printed manipulatives for teaching tactual shape perception and spatial awareness sub-concepts to visually impaired children,” International Journal of Child-Computer Interaction, vol. 11, pp. 3-11, 2017. https://doi.org/10.1016/j.ijcci.2016.12.001
[4] A. C. Pires, S. Marichal, F. Gonzalez-Perilli, E. Bakala, B. Fleischer, G. Sansone, and T. Guerreiro, “A tangible math game for visually impaired children,” in Proceedings of the21st International ACM SIGACCESS Conference on Computers and Accessibility, Pittsburgh, PA, 2019, pp. 670-672. https://doi.org/10.1145/3308561.3354596
[5] V. Gadiraju, A. Muehlbradt, and S. K. Kane, “BrailleBlocks: computational braille toys for collaborative learning,” in Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems, Honolulu,HI, 2020, pp. 1-12. https://doi.org/10.1145/3313831.3376295
[6] M. D. Lozano, V. M. Penichet, B. Leporini, and A. Fernando, “Tangible user interfaces to ease the learning process of visually-impaired children,” in Proceedings of the 32nd International BCS Human Computer Interaction Conference, Belfast, UK, 2018, pp. 1-5.
[7] A. Alhussayen, R. Jafri, and A. Benabid, “Requirements’ elicitation for a tangible interface-based educational application for visually impaired children,” in Advances in Ergonomics Modeling, Usability & Special Populations. Cham, Switzerland: Springer, 2017, pp. 583-596. https://doi.org/10.1007/978-3-319-41685-4_52
[8] R. Jafri, “Electronic braille blocks: a tangible interface-based application for teaching braille letter recognition to very young blind children,” in Computers Helping People with Special Needs. Cham, Switzerland: Springer, 2014, pp. 551-558. https://doi.org/10.1007/978-3-319-08599-9_81
[9] H. T. Bintaleb and D. Al Saeed, “Extending tangible interactive interfaces for education: a system for learning Arabic Braille using an interactive Braille keypad,” International Journal of Advanced Computer Science and Applications, vol. 11, no. 2, pp. 359-367, 2020. https://doi.org/10.14569/ijacsa.2020.0110247
[10] M. S. Manshad, E. Pontelli, and S. J. Manshad, “Trackable interactive multimodal manipulatives: towards a tangible user environment for the blind,” in Computers Helping People with Special Needs. Heidelberg, Germany: Springer, 2012, pp. 664-671. https://doi.org/10.1007/978-3-642-31534-3_97
[11] M. Adusei and D. Lee, “"Clicks" appcessory for visually impaired children,” in Proceedings of the 2017 CHI Conference Extended Abstracts on Human Factors in Computing Systems, Denver, CO, 2017, pp. 19-25. https://doi.org/10.1145/3027063.3049275
[12] Y. Breiter, A. Karshmer, and J. Karshmer, “Automathic blocks usability testing phase one,” in Computers Helping People with Special Needs. Heidelberg, Germany: Springer, 2012, pp. 191-195. https://doi.org/10.1007/978-3-642-31522-0_28
[13] M. Avila-Soto, E. Valderrama-Bahamondez, and A. Schmidt, “TanMath: a tangible math application to support children with visual impairment to learn basic arithmetic,” in Proceedings of the 10th International Conference on PErvasive Technologies Related to Assistive Environments, Island of Rhodes, Greece, 2017, pp. 244-245. https://doi.org/10.1145/3056540.3064964
[14] E. Brule, B. J. Tomlinson, O. Metatla, C. Jouffrais, and M. Serrano, “Review of quantitative empirical evaluations of technology for people with visual impairments,” in Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems, Honolulu, HI, 2020, pp. 1-14. https://doi.org/10.1145/3313831.3376749
[15] R. Raisamo, A. Hippula, S. Patomaki, E. Tuominen, V. Pasto, and M. Hasu, “Testing usability of multimodal applications with visually impaired children,” IEEE MultiMedia, vol. 13, no. 3, pp. 70-76, 2006. https://doi.org/10.1109/MMUL.2006.68
[16] T. G. Darin, R. M. Andrade, L. B. Merabet, and J. H. Sanchez, “Investigating the mode in multimodal video games: Usability issues for learners who are blind,” in Proceedings of the 2017 CHI Conference Extended Abstracts on Human Factors in Computing Systems, Denver, CO, 2017, pp. 2487-2495. https://doi.org/10.1145/3027063.3053177
[17] D. Xu, “Design and evaluation of tangible interfaces for primary school children,” in Proceedings of the 6th International Conference on Interaction Design and Children, Aalborg, Denmark, 2007, pp. 209-212. https://doi.org/10.1145/1297277.1297331
[18] B. Ullmer and H. Ishii, “Emerging frameworks for tangible user interfaces,” IBM Systems Journal, vol. 39, no. 3-4, pp. 915-931, 2000. https://doi.org/10.1147/sj.393.0915
[19] M. Kaltenbrunner, “reacTIVision and TUIO: a tangible tabletop toolkit,” in Proceedings of the ACM International Conference on Interactive Tabletops and Surfaces, Banff, Canada, 2009, pp. 9-16. https://doi.org/10.1145/1731903.1731906
[20] N. Bevan, J. Carter, J. Earthy, T. Geis, and S. Harker, “New ISO standards for usability, usability reports and usability measures,” in Human-Computer Interaction: Theory, Design, Development and Practice. Cham, Switzerland: Springer, 2016, pp. 268-278. https://doi.org/10.1007/978-3-319-39510-4_25

About this article
Cite this article

AbirBenabid Najjar1,*, Asma Alhussayen1, and Rabia Jafri2, Usability Engineering of a Tangible User Interface Application for Visually Impaired Children, Article number: 11:14 (2021) Cite this article 4 Accesses

Download citation
  • Recived30 June 2020
  • Accepted18 December 2020
  • Published31 March 2021
Share this article

Anyone you share the following link with will be able to read this content:

Provided by the Springer Nature SharedIt content-sharing initiative

Keywords