Designing a virtual realm to Improve language acquisition.
This case study aims to understand users' perceptions of language acquisition platforms developed using Virtual Reality (VR) and Artificial Intelligence (AI). The motivation behind this study is the researcher's interest in learning languages. Utilizing VR and AI in language learning apps has shown an increase in motivation for learners. Research also suggests that context-based language learning can help develop fluency in a language. Therefore, this qualitative case study aims to understand the design factors that motivate users to learn effectively using VR and AI. To achieve this, a VR prototype was developed in Unity, and Wit AI was used to simulate speech-to-text and text-to-speech functions. Participants were able to interact with the system and then engaged in a semi-structured interview with the researcher.
Number of Participants
10 (aged 18+)
10 (aged 18+)
Duration
3 months
3 months
Roles and Responsibilities
Literature Review
Literature Review
Ethics consideration
Ethics consideration
VR Prototype Development
VR Prototype Development
AI Implementation
AI Implementation
Research Planning
Research Planning
Performing Ethnography
Performing Ethnography
Conducting User Interviews
Conducting User Interviews
Analysing Data
Analysing Data
Problem Discovery
Problem Discovery
Language is a crucial tool for communication. Language learning involves being taught about the language's usage and structure, while language acquisition focuses on immersing oneself in the language through practical use. To improve fluency in a foreign language, learners should move from language learning to acquisition, but this can be challenging due to limited immersive opportunities.
We tried to understand the museum visitors' problems while interacting with VR edutainment applications in public places like museums. We have essentially gathered primary data from Semi-structured Interviews and group discussions. Some secondary data gathered from the literature helped us discover more insights and support the primary data more scientifically. A total of 12 people were interviewed initially by the team and each member was responsible for interviewing 2 users.
Virtual Reality (VR) and Artificial Intelligence (AI) offer potential solutions. VR can simulate real-life situations, and AI can personalise learning by enabling interaction with NPCs in real-life scenarios. Existing research shows learners enjoy VR and AI-based learning, though there is no significant difference in effectiveness. However, there is a lack of research on AI-integrated immersive virtual worlds in VR for language acquisition, which makes it challenging for designers to create guidelines for VR-powered language acquisition platforms.
We tried to understand the museum visitors' problems while interacting with VR edutainment applications in public places like museums. We have essentially gathered primary data from Semi-structured Interviews and group discussions. Some secondary data gathered from the literature helped us discover more insights and support the primary data more scientifically. A total of 12 people were interviewed initially by the team and each member was responsible for interviewing 2 users.
Research Objectives:
We tried to understand the museum visitors' problems while interacting with VR edutainment applications in public places like museums. We have essentially gathered primary data from Semi-structured Interviews and group discussions. Some secondary data gathered from the literature helped us discover more insights and support the primary data more scientifically. A total of 12 people were interviewed initially by the team and each member was responsible for interviewing 2 users.
🎯 To observe users' interaction with the AI and virtual environment.
🎯 To understand users' perceptions about language learning using AI and VR.
We tried to understand the museum visitors' problems while interacting with VR edutainment applications in public places like museums. We have essentially gathered primary data from Semi-structured Interviews and group discussions. Some secondary data gathered from the literature helped us discover more insights and support the primary data more scientifically. A total of 12 people were interviewed initially by the team and each member was responsible for interviewing 2 users.
The research answers the following Research Questions:
We tried to understand the museum visitors' problems while interacting with VR edutainment applications in public places like museums. We have essentially gathered primary data from Semi-structured Interviews and group discussions. Some secondary data gathered from the literature helped us discover more insights and support the primary data more scientifically. A total of 12 people were interviewed initially by the team and each member was responsible for interviewing 2 users.
✨ How do users perceive AI as a language-learning medium?
✨ Does a scenario in the virtual world help generate ideas to communicate with the AI?
We tried to understand the museum visitors' problems while interacting with VR edutainment applications in public places like museums. We have essentially gathered primary data from Semi-structured Interviews and group discussions. Some secondary data gathered from the literature helped us discover more insights and support the primary data more scientifically. A total of 12 people were interviewed initially by the team and each member was responsible for interviewing 2 users.
Design
Design
Research suggests that context-based language learning, which provides real-life situations, improves recall of definitions, listening comprehension skills, and knowledge transfer. After considering various scenarios, a restaurant scenario was selected as users can be familiar with a restaurant scenario and can easily perform the task of communicating with the counter server. A mood board was created to understand the ambiance that was needed to be created for a restaurant. The VR Environment was developed using Unity3D for Quest 2 device. The characters were created using ReadyPlayerMe. OpenAI, along with Wit.AI, was used for generating dialogues and converting them from text-to-speech and speech-to-text for a seamless experience.
We tried to understand the museum visitors' problems while interacting with VR edutainment applications in public places like museums. We have essentially gathered primary data from Semi-structured Interviews and group discussions. Some secondary data gathered from the literature helped us discover more insights and support the primary data more scientifically. A total of 12 people were interviewed initially by the team and each member was responsible for interviewing 2 users.
Short demonstration of the project done by recording the Interaction through Quest 2.
We tried to understand the museum visitors' problems while interacting with VR edutainment applications in public places like museums. We have essentially gathered primary data from Semi-structured Interviews and group discussions. Some secondary data gathered from the literature helped us discover more insights and support the primary data more scientifically. A total of 12 people were interviewed initially by the team and each member was responsible for interviewing 2 users.
Research
Research
Generating a Research Plan
Generating a Research Plan
Qualitative analysis was chosen to have the flexibility to observe and understand participants' perceptions of combining VR and AI for language acquisition, their experiences, and challenges. The data was collected by conducting an ethnographic study and post-observation semi-structured interview. A sample space of 10 voluntary participants was recruited by word of mouth and observed with ethics considerations in place. The participants consisted of a mix of Native and Non-Native English speakers with varying proficiencies. Thematic analysis was undertaken to obtain insights from the collected data.
Qualitative analysis was chosen to have the flexibility to observe and understand participants' perceptions of combining VR and AI for language acquisition, their experiences, and challenges. The data was collected by conducting an ethnographic study and post-observation semi-structured interview. A sample space of 10 voluntary participants was recruited by word of mouth and observed with ethics considerations in place. The participants consisted of a mix of Native and Non-Native English speakers with varying proficiencies. Thematic analysis was undertaken to obtain insights from the collected data.
Ethics Consideration
Ethics Consideration
Throughout the research process, it was ensured that the ethical considerations were upheld. Consent forms were obtained from all participants. Consent forms contained details about the research's purpose, procedures, and rights of the participants. Additionally, the participants were briefed about the study's purpose, procedures they were supposed to perform and their rights before beginning the study. Participants are guaranteed to keep the collected data confidential and anonymised during analysis to protect their identity. The research has been deemed safe, and ethics approval has been obtained from the faculty.
Throughout the research process, it was ensured that the ethical considerations were upheld. Consent forms were obtained from all participants. Consent forms contained details about the research's purpose, procedures, and rights of the participants. Additionally, the participants were briefed about the study's purpose, procedures they were supposed to perform and their rights before beginning the study. Participants are guaranteed to keep the collected data confidential and anonymised during analysis to protect their identity. The research has been deemed safe, and ethics approval has been obtained from the faculty.
Virtual Reality and Artificial Intelligence are future technologies that should be used to develop innovative solutions to solve societal problems while considering ethics and exercising caution. Participants were informed that VR technology can cause Cyber-sickness and were encouraged to notify the researcher if they experienced discomfort at any point. The experience was designed to be a seated experience to ensure participants' safety. Research shows that younger children are more susceptible to confusion between reality and virtual reality when immersed in virtual reality. Therefore, age compliance rules were applied to this experience.
Virtual Reality and Artificial Intelligence are future technologies that should be used to develop innovative solutions to solve societal problems while considering ethics and exercising caution. Participants were informed that VR technology can cause Cyber-sickness and were encouraged to notify the researcher if they experienced discomfort at any point. The experience was designed to be a seated experience to ensure participants' safety. Research shows that younger children are more susceptible to confusion between reality and virtual reality when immersed in virtual reality. Therefore, age compliance rules were applied to this experience.
Results
Results
The data collected from the deployed research methods was transcribed and thematically coded to document the results.
Interaction with Virtual Environment
Interaction with Virtual Environment
1. Initial uncertainty about interaction boundaries - The participants expected some signposting or signifiers to help them differentiate between interactable NPCs and Non-Interactable NPCs.
1. Initial uncertainty about interaction boundaries - The participants expected some signposting or signifiers to help them differentiate between interactable NPCs and Non-Interactable NPCs.
2. Contextual Engagement - All the participants initiated interaction with the AI by ordering food from the menu or asking questions about objects in the environment. People found the environment helpful to generate topics to talk as the virtual restaurant environment could set the context for the conversation.
2. Contextual Engagement - All the participants initiated interaction with the AI by ordering food from the menu or asking questions about objects in the environment. People found the environment helpful to generate topics to talk as the virtual restaurant environment could set the context for the conversation.
3. Realism - A participant mentioned that they felt the experience was a bit real, which motivated them to talk to the NPC like they would talk to a human. While other participants supported that realism contributes to the motivation to talk to the NPC, they also suggested that the current prototype could have more realism. Another observation was that the NPCs' ability to talk added realism and immersion. The NPCs without the ability to talk did not contribute to realism. However, the participant added that it does not impact the functionality.
3. Realism - A participant mentioned that they felt the experience was a bit real, which motivated them to talk to the NPC like they would talk to a human. While other participants supported that realism contributes to the motivation to talk to the NPC, they also suggested that the current prototype could have more realism. Another observation was that the NPCs' ability to talk added realism and immersion. The NPCs without the ability to talk did not contribute to realism. However, the participant added that it does not impact the functionality.
Interaction with AI
Interaction with AI
1. Easy controls - Due to time-constraints the prototype did not have the feature to talk with the AI when you get closer to them. Rather the users had to press space to talk to the AI. With the VR HMD on, it was difficult to access the space key. Using colliders to activate the talk function could help solve this problem.
1. Initial uncertainty about interaction boundaries - The participants expected some signposting or signifiers to help them differentiate between interactable NPCs and Non-Interactable NPCs.
2. Shortcomings of AI in understanding different accents - The AI struggled to understand some accents or words spoken by non-native English speakers, increasing the participant's frustration. However, it was also observed that the participants did not give up and kept trying to improve and help the AI understand what they were trying to speak.
2. Contextual Engagement - All the participants initiated interaction with the AI by ordering food from the menu or asking questions about objects in the environment. People found the environment helpful to generate topics to talk as the virtual restaurant environment could set the context for the conversation.
3. Language Anxiety - Language anxiety is a condition where humans feel nervous to practice their language skills with other humans. During the Interview, people mentioned experiencing language anxiety while talking with other native speakers. However, they did not feel the same while talking with the AI as they acknowledged AI as a non-human and non-living entity. On the contrary, they also mentioned they did feel nervous when the AI could not understand them.
3. Realism - A participant mentioned that they felt the experience was a bit real, which motivated them to talk to the NPC like they would talk to a human. While other participants supported that realism contributes to the motivation to talk to the NPC, they also suggested that the current prototype could have more realism. Another observation was that the NPCs' ability to talk added realism and immersion. The NPCs without the ability to talk did not contribute to realism. However, the participant added that it does not impact the functionality.
4. Multimodal Learning - The content for this study was delivered to the participants in both text (visual) and audio (auditory) modes. All the participants found the delivery of content through multimodal way to be very effective. The participants with low fluency might not understand what is being said, so the text acted as a supplementary method. Multimodal output also made the prototype more accessible as some of the participants with glasses were pleased to have auditory output because it can be challenging to wear a HMD along with glasses.
3. Realism - A participant mentioned that they felt the experience was a bit real, which motivated them to talk to the NPC like they would talk to a human. While other participants supported that realism contributes to the motivation to talk to the NPC, they also suggested that the current prototype could have more realism. Another observation was that the NPCs' ability to talk added realism and immersion. The NPCs without the ability to talk did not contribute to realism. However, the participant added that it does not impact the functionality.
5. Trust in AI - The participants showed two contrary perceptions about trusting the AI. Two participants showed a lack of trust in the AI. One of them mentioned that AI can cause glitches. Hence, they checked whether the text and audio were in sync, showing the same content. The participant felt positive because the AI did not glitch during usage. The rest of the participants trusted or had neutral feelings about AI. One of the participants stated they would trust the AI because the translators on the Internet are also deployed using the same or similar AI technology.
3. Realism - A participant mentioned that they felt the experience was a bit real, which motivated them to talk to the NPC like they would talk to a human. While other participants supported that realism contributes to the motivation to talk to the NPC, they also suggested that the current prototype could have more realism. Another observation was that the NPCs' ability to talk added realism and immersion. The NPCs without the ability to talk did not contribute to realism. However, the participant added that it does not impact the functionality.
Impact
The language learning study that we conducted in a VR environment using AI has provided us with valuable insights into the potential advantages and drawbacks of this approach. Our research has led us to identify several key design improvements that could enhance the effectiveness of the system.
The language learning study that we conducted in a VR environment using AI has provided us with valuable insights into the potential advantages and drawbacks of this approach. Our research has led us to identify several key design improvements that could enhance the effectiveness of the system.
1. Signposting - Signposts could help users with spatial navigation as some users demonstrated uncertainty related to interaction boundaries.
1. Signposting - Signposts could help users with spatial navigation as some users demonstrated uncertainty related to interaction boundaries.
2. Gamification - Gamification has been a proven factor for motivating users, and the participants of this study showed a positive response to some suggested gamification ideas. One of the participants also mentioned that gamification would provide positive reconditioning, which works like a motivational factor, gives you a feeling of achievement, and pushes you to learn more.
2. Gamification - Gamification has been a proven factor for motivating users, and the participants of this study showed a positive response to some suggested gamification ideas. One of the participants also mentioned that gamification would provide positive reconditioning, which works like a motivational factor, gives you a feeling of achievement, and pushes you to learn more.
3. Delivering subtasks and follow-ups to make the experience more realistic - Participants prefer to have realism and expect interactions to be more realistic as discussed in realism. One of the participants suggested that tasks can be made more meaningful and realistic by dividing the tasks into sub-tasks and delivering them to the user. The follow-up process makes the task more interactive and realistic.
3. Delivering subtasks and follow-ups to make the experience more realistic - Participants prefer to have realism and expect interactions to be more realistic as discussed in realism. One of the participants suggested that tasks can be made more meaningful and realistic by dividing the tasks into sub-tasks and delivering them to the user. The follow-up process makes the task more interactive and realistic.
4. Social Learning - Including an optional way to interact with humans could benefit the system in language and culture exchange. This will also help understand the local way of speaking a language.
4. Social Learning - Including an optional way to interact with humans could benefit the system in language and culture exchange. This will also help understand the local way of speaking a language.
5. Visual Appearance of AI - When asked about the visual appearance of the AI, most of the participants preferred to have a human-like avatar or a living being. Based on participants' responses, we can say that the visual appearance should be close to realism as they do not wish to see something unusual as an avatar for the AI. It also shows that if someone is biased towards animals, then they feel rewarded to be talking to an AI with an animal avatar.
5. Visual Appearance of AI - When asked about the visual appearance of the AI, most of the participants preferred to have a human-like avatar or a living being. Based on participants' responses, we can say that the visual appearance should be close to realism as they do not wish to see something unusual as an avatar for the AI. It also shows that if someone is biased towards animals, then they feel rewarded to be talking to an AI with an animal avatar.
6. Support for different accents - The language model used in the AI system should be designed to support different accents. This is important because people from different regions around the world may have unique accents that could make it difficult to understand them. If the AI system is unable to comprehend someone's accent, it could lead to feelings of anxiety and frustration, thereby defeating the purpose of using AI to practice language.
6. Support for different accents - The language model used in the AI system should be designed to support different accents. This is important because people from different regions around the world may have unique accents that could make it difficult to understand them. If the AI system is unable to comprehend someone's accent, it could lead to feelings of anxiety and frustration, thereby defeating the purpose of using AI to practice language.