Researchers have developed a system for generating storybooks aimed at personalized vocabulary enhancement.
Professor Inseok Hwang from the Department of Computer Science and Engineering at POSTECH, alongside students Jungeun Lee, Suwon Yoon, and Kyoosik Lee, and in collaboration with Professor Dongsun Yim from Ewha Womans University’s Department of Communication Disorders, has introduced an exciting method for crafting tailored storybooks. This innovative system employs generative artificial intelligence and home IoT technologies to aid children in learning languages. Their findings were presented at the prestigious “ACM CHI (ACM SIGCHI Conference on Human Factors in Computing Systems),” where they received an “Honorable Mention Award,” highlighting their work as being among the top 5% of all entries.
Language development in children is vital, influencing their cognitive skills, academic performance, peer relationships, and overall social growth. Regular assessment of language progress and timely interventions1) are essential for effective language acquisition. However, children often grow up in varied environments, leading to differences in vocabulary exposure. Unfortunately, traditional methods tend to rely on uniform vocabulary lists and ready-made storybooks or toys, which do not account for this diversity.
Understanding the limitations of the conventional one-size-fits-all strategies that overlook children’s unique backgrounds, the research team engineered an advanced educational system specifically designed for each child’s surroundings. They initiated this process by utilizing home IoT devices to track and analyze the language children encounter and produce in their daily routines. By employing techniques such as speaker separation2) and morphological analysis3), they evaluated the vocabulary children were exposed to, as well as the words they articulated and listened to but did not speak out loud. Each word was scored based on key factors relevant to speech pathology.
To generate custom educational resources, the group harnessed cutting-edge generative AI technologies, such as GPT-4 and Stable Diffusion. This allowed them to create tailor-made children’s books that seamlessly incorporate the specific vocabulary needed for each child. By merging speech pathology theories with practical experience, the researchers crafted a highly effective and personalized language learning platform.
The system was designed to adapt to the variations in children’s language development, featuring customizable factors and flexible vocabulary selection criteria. It can automatically identify the target vocabulary for each child and generate personalized storybooks, allowing for continuous updates in response to any changes in the child’s language skills or environment. After testing the system with 9 families over four weeks, the findings indicated that children successfully learned the intended vocabulary, confirming its practical use in everyday life beyond clinical settings.
Jungeun Lee from POSTECH, the primary author of the study, remarked on their achievements, stating, “We effectively tackled the shortcomings of traditional, uniform methods of child language assessment and intervention using generative AI.” She further commented, “Our intention is to utilize AI to produce personalized guides catered to the varying needs and levels of individuals.”
Professor Inseok Hwang from POSTECH, the corresponding author, noted, “Through collaborative research across disciplines, we have successfully developed a personalized system for language stimulation and development that combines generative AI technology with principles of speech pathology.” He expressed hope that their results would inspire educators to embrace and integrate the diversity of children’s environments and learning objectives.
Co-author Professor Dongsun Yim from Ewha Womans University also shared her optimism, saying, “Our research showcases the potential for innovative, personalized language support services.” She emphasized, “The system highlights the capability to customize vocabulary extraction and linguistic stimuli for children exposed to diverse environments and languages.”
This research was funded by the National Research Foundation of Korea’s Mid-Career Researcher Program, the SSK, the IITP’s ITRC, and the ICT R&D Innovation Voucher Program.
