Keywords

1 Introduction

Deaf children face various challenges in their daily life in the social, cultural and educational ambits. In addition to these challenges, they tend to acquire cognitive skills at a much slower pace than a hearing child. Deaf children therefore require a special education just in order to receive the appropriate educational development. In the educational field, they must make use of communication strategies that have been adjusted to their needs.

Hearing impairment is an obstacle preventing the processing of information linguistically through the ear. It is more generally known as deafness. A deaf child who does not use a hearing aid must communicate with society through sign language or lip-reading. As a result, these children have a better-developed visual attention span [1, 2]. Educators are therefore using tools accompanied by images and text as a channel of communication so as to convey the extraction of meaning of a concept. Educators consequently require play-based tools that can motivate children in their learning and that can be integrated within their educational planning.

Augmentative and alternative communication (AAC), meanwhile, is used for people with little or no functional speech, due to severe speech impairment [3]. Another definition by [4] is defined as intervention instruments destined for people with problems in communication and language, the aim of which instruments is teaching meaning in specific procedures with the use of symbols and pictures, etc., as an aid for children using alternative communication. The goal of AAC is to break the speech barrier between a child with speech difficulties and the teacher.

Research by [6] suggests that the function AAC can play in language and communication development varies depending on the child’s chronological age, degree of disability, and specific environment requirements. Furthermore, it was found that children on using these AAC systems learned to communicate expressively using the skills they had explicitly been taught.

2 System Communication and Deaf Children

Deaf children are faced with major challenges in different areas such as health, education, cultural and social. A deaf child is one who has no hearing aid available and must learn to communicate through sign language.

Communication with deaf children is done using visual tools such as pictograms and pictures that can help to support them in understanding language. A communication board can be used as an alternative form of communication to replace or supplement speech that is not functional for expressing thoughts, needs, wants, and ideas in the child.

An AAC system can be used as an instrument for people with special needs who have different impairments in communication and/or language, and whose objective is to teach through other teaching methods a structured set of non-vowel but visual codes that facilitate functional communication [10]. However, it must be borne in mind that these boards are a means to communicate so as to acquire skills that make it possible to interact with the environment.

An AAC can be represented by physical or digital devices such as communication boards with drawings, charts with images or words or digital devices with or without verbal output such as Tablet or Smartphones. An AAC system called PECS (Picture Exchange Communication System) was created for people with autism spectrum, since they cannot use speech to communicate. These PECS were systems created with low technology that involve elements in paper, cards with images, to communicate graphically certain specific actions. Although PECS was developed for autistic people it has been used in a wide range of people with disabilities. Early AAC technologies quickly became more portable and less expensive and they are now being used in mobile applications. Furthermore, work proposed by Casey [9] demonstrates that communication systems can be an alternative for people who have complex communication needs and can help the development skills in reading and writing. In addition, it emphasizes that these types of work, which involve AAC systems, can be complemented collaboratively. Meanwhile, [11] states that AAC may be an important element for the development of language and communication, but this system may vary depending on the chronological age of the child. The design may further change depending on the child’s disability and how they can use AAC systems and which exit channels can be enabled.

The communication system aims for deaf children to have better access to the different components of the oral language. The system consists of a series of keys to differentiate the structure of a sentence. As such, colors are often used to differentiate these keys making it possible to represent the structure of a sentence and allowing the children to use the rules in some way.

3 Literacy in Deaf Children

Deaf children who do not have hearing aid need to learn sign language, which is considered their first spoken language. However, they still have to learn to read and write in Spanish in order to communicate with society. For them, learning written Spanish takes longer, as they have to learn all the morpho-syntactic and pragmatic structure. In addition, research shows [12] that the extent of their lexicon is small and its evolution with age and schooling is slow.

The handling of syntactic structures is more deficient as they become more complex. As stated by Niederberger [13], the morpho-syntactic competence of deaf children is low, since in their partial perception of speech they confine themselves to identifying key words such as verbs and nouns in the content of sentences, but do not give much importance to prepositions or articles. For a deaf child lacking phonology, the only way available to establish connections between written word and concept is sometimes to memorize without establishing a phonology relationship with the word, so that memorizing vocabulary could almost be considered a mission impossible. For example, with a word like “casa” (house) when a consonant changes giving rise to different meanings such as “cava” (cave), “cama” (bed), and “cara” (face), among others. The task of the child with no phonology, who has to associate an orthographic detail with a particular meaning, is very difficult. It is therefore important to consider a visual strategy to consider phonology and make it available to deaf children. In addition, many spelling errors are found, such as “arina” (harina), “baca” (vaca), etc., so often to consider phonology with writing, dactylological language is used as a way for children to be able to represent each of the sounds of a word.

Deaf children usually use word complement (WC) as an option or strategy for their teaching, widely used in children with lip-facial reading. Strategies using color codes are also used for the teaching of literacy, to facilitate with the help of colors and question schemes which ones correspond to the part of a sentence, i.e. subject, verb, adjective, among others. Within these color codes lie a number of proposed learning methods, such as Fitgerald [12], Goossens, Crain & Elder [25, 26].

The strategies that teachers use are through visual and gestural codes. A major difficulty children have is memorizing visually the concepts of each word. However, it is difficult to write very long words correctly. Therefore, teachers are supported by a teaching method called Fitzgerald Key [5]. Fitzgerald Key (see Fig. 1) involves a linguistic code of visual representation, making use of color codes in the form of questions, in which the user must put the word sequence together (i.e. person + action + object). Fitzgerald works by a scheme of questions to complement the sentence, such as: when, where, who, actions, among others, and each of these schemes represents a set color according to whether it corresponds to a pronoun, object, verb, noun, adjective or adverb.

Fig. 1.
figure 1

Fitzgerald Key

Full size image

Furthermore, technology has been integrated into AAC systems to design better tools to maximize the development of communication, language and literacy skills, hence the use human-computer interaction (HCI) to understand different factors in the children – e.g. psychological, social, cultural and educational. These factors help to determine how children operate and use the technology. A model of user centered design on the user structures of the information concerned for the end user was proposed by [6], in which different aspects of the deaf child were identified, such as behaviors, cognitive aspects, and interests, among others.

Research conducted [6,7,8,9] shows that HCI can play a part in the creation of a smart, interactive model for children with a hearing disability and could be an alternative for designing an interactive communication board oriented to literacy teaching, applying the Fitzgerald Key teaching method. Thus, when designing a communication board for mobile devices it is important to take into account such variables as color, shape, texture, size, position, and movement, which can help to optimize the design of AAC technologies.

4 Methodology

The methodology followed is a methodology called MECONESIS proposed by Cano et al. [14], which follows some stages within the process, such as: Analysis, pre-production, production and post-production. The methodology is oriented to the design of serious games for children with hearing impairment. Therefore, the first thing that should be done is an analysis of the profile of the deaf child from an educational context in the teaching of literacy. Children with whom work is done to evaluate the communication systems in literacy teaching are deaf children from the USAER program (Units of Service to Support Regular Education, in Aguascalientes, Mexico), made up of several public schools that allow access to children with Special Educational Needs (SEN) in regular education. Seven children with hearing impairment between 11 and 15 years old at USAER, Aguascalientes, México and eight children aged between 4 and 8 years from the ITES (Special Sensory Therapy of the Lions Club) in Cali, Colombia, were analyzed.

4.1 Analysis

The different aspects that have been analyzed in the deaf child are researches proposed in [17], which are: Personal Information (Age, Gender and Academic year), Competences, Disability (Cognitive, Physical), Disability level (Slight, Moderate, Profound and Severe), Learning styles, Behavior/Academic, Emotions, and Motivation.

For the Deaf children evaluated, the direct observation method was used in USAER. These children are in secondary education, however they do not have a grasp of Mexican sign language and their strategies for communicating are through the alphabet and some informal signs they themselves create to find a way to communicate faster. These children are included within the regular school, so there is a supporting teacher figure, who has one-hour sessions twice a week per group in the areas of literacy and math. Most of these children come from deaf parents and are from low-income families. On the other hand, the ITES children are only deaf children where they are taught Colombian sign language as well as the writing and reading of Spanish as their second language. Deaf children communicate through visual language, so the teachers in their teaching strategies rely on pictograms to explain the concepts and color codes for differentiating between the elements and rules that allow sentences to be constructed.

4.2 Analysis

With the support teachers of the USAER, a search was carried out for mobile applications that function as communication boards and are oriented toward the writing and reading of the deaf children.

As a result, five applications were selected for mobile devices (Smartphone) (Table 1), whose purposes were to serve as an alternative and augmentative communication tool. Thus, each of them was analyzed first with the support teacher, whose approval was given if it could serve as an educational resource and then it was evaluated with the child. The purpose is that the children, when interacting with applications should, without too much effort, understand the graphical interface, the responses to the interactions and each one of the activities, and that in turn, these would provide support to the teacher to use in their teaching strategies in the classroom.

Table 1. Communication boards in mobile devices
Full size table

Of the five applications selected that meet the function of communication boards, most are oriented for children with serious communication problems. In addition, these applications have a very basic vocabulary and do not give importance to the rules of the sentence using color keys. It is quite the reverse with the communication book application developed for Android, which has a set of categories grouped by color, and each color represents a structure of the sentence. However, it does not handle questionnaires and pictograms, while navigation through the categories is not very clear. Taking as a background the design and development of the communication board, Literacy with Fitzgerald [9] is proposed, a serious game where game mechanics are taken into account and punctuation, challenges and levels of difficulty are created using questionnaires and grouping by color categories (see Fig. 2). Each category that has been created is accompanied by icons representative of the group of vocabulary in which it is contained. In addition each questionnaire is subject to the level of difficulty, where three levels of difficulty have been created (basic, intermediate and advanced).

Fig. 2.
figure 2

Literacy with Fitzgerald, mobile application [21]. (Color figure online)

Full size image

This application was developed for Android Tablets in such a way that this educational resource was able to be carried around by teachers, and at the same time being affordable. However, it is important to realize that the great majority of these children are of low economic resources and in the center they attend depend on donations or external aid. So, not everyone can have a mobile application so easily, in which only the interaction is digital.

4.3 Pre-production

Analyzing the information obtained in the previous stage, the development is purely digital, a continuation of the work proposed by [6], which seeks to interact with a real environment and a digital one at the same time, giving very favorable results. Therefore a physical board is proposed that makes it possible to interact with the child in a real environment and at the same time to link information technologies by means of augmented reality (Fig. 3).

Fig. 3.
figure 3

Prototype of the “Literacy with Fitzgerald” physical board

Full size image

For the design of the physical board, it is taken into account that it can have the number of times for a board where the correct sequence can be constructed according to the scheme of questions, which is presented in each activity by learning level (Fig. 4). Therefore, it is very important that the teacher is a moderator in the development of this activity. The activity revolves around a set of cards, where each card is grouped according to a learning level (basic, intermediate, or advanced) that is associated with a color (blue, yellow, and orange). The different tasks to be performed are on activity cards that are included in the physical game, where the teacher should take control of the game to indicate to the student the activities to be performed. Technology has been included to be part of this physical board game integrating each card of vocabulary in text to be visualized in a 3D model relating to the corresponding pictogram.

Fig. 4.
figure 4

Managing the physical board (Color figure online)

Full size image

Augmented reality is used to define direct or indirect vision of a real-world physical environment whose elements are combined with virtual elements to create a mixed reality in real time.

With augmented reality (AR), there is a relationship between virtual and physical elements in the real world. AR also allows access to information in a different way using technology, modifying the mode of learning and improving knowledge of reality. Authors such as Fabregat [22] and Basogain et al. [23] state that AR is a technology that complements the interaction with the real world and allows the user to be in an enhanced real environment with additional information provided by technology, which can be through a computer or mobile devices. As a result, Fabregat [22] has put forward some characteristics of AR, such as: (1) it combines the real and the virtual, (2) it works in real time and (3) it registers in three dimensions. Iulian et al. [24], meanwhile, refers to AR as the technology that allows virtual content to be included in a physical-real context, allowing students to view virtual content as it appears in the real world and control the virtual environment through the interaction of tangible objects.

AR involves a number of basic concepts, such as Marching, to mark the vertices in the image; Tracking, to locate an object that is moving in a space-time; Target, which are images that allow the positioning of the virtual content in the scene; and finally Rendering, the process of generating an image. So, for each card of the physical board it is necessary to carry out tracking so that it can generate the image corresponding to the text on each card.

4.4 Production

Each card image corresponds to a virtual content that can represent visual information for the child. The idea is to be able to relate the text and if the meaning of the text is not known, the mobile device can be used to provide visual information of the meaning of the image. The objective of this is that the child can associate the word with the pictogram (Fig. 5).

Fig. 5.
figure 5

Models in 3D for obtaining additional information.

Full size image

5 Conclusions and Future Work

The physical board proposes to act as an alternative educational resource for Fitzgerald Keys teaching for deaf children as an augmentative communication board. Therefore, it is sought to continue measuring the impact produced when working simultaneously with real objects and virtual, augmented reality. Today, educational experiences for children come with play resources as a means of motivating them during their learning. Furthermore, interaction with real objects allows the child to have a real interaction and to associate objects more easily within their environment. As a future work it is hoped to carry out an evaluation with deaf children who use Fitzgerald Keys as a teaching method in reading and writing, as it has been observed that some schools are using the Fitzgerald Keys strategy. However, they do not use the same colors or schemes that Fitzgerald uses strictly, so a way must be found that the physical game can configure the question schemes according to the colors that the teachers handle, as the work by Cano et al. [7] does.