Alelo CEO Dr. Lewis Johnson was invited to speak at this year’s International Conference on Artificial Intelligence in Education (AIED 2020) on the topic of Alelo’s Data-Driven Development (D3) methodology for AI-driven education and training products. This builds on work published last year in the International Journal on Artificial Intelligence in Education, a prestigious journal that is ranked by Scopus in the top 1 percent of educational journals worldwide.
D3 differs from earlier instructional development methodologies such as ADDIE (Analysis, Design, Development, Implementation, and Evaluation) in that it is informed by data from learners throughout the system life cycle. Learning systems developed using D3 are data-collection tools as well as learning tools. The development team collects and analyzes learner behavior data early in the project, to determine specifically what kinds of problems learners have and where they need help. The data is used to train the AI models built into the system. When the system is deployed, it collects additional data, which is analyzed and the cycle repeats. The cycle repeats rapidly; as new data is collected it is used to retrain the AI models and test system improvements, which are then deployed to collect more data. The methodology has similarities to design-based research in that it emphasizes collection and analysis of learner data from real-world settings. The advent of cloud-based computing now makes it easy to collect learner data from educational institutions around the world.
D3 integrates rapid iterative evaluation with iterative development. D3 uses snapshot evaluations (evaluations that focus on a limited learner population over a limited period) to test specific research hypotheses and evaluate system improvements. Instant tests (analyses of current system performance with learners) are performed continually on a weekly basis to identify where system improvement is necessary.
Examples in the presentation were taken from the development of Enskill English, an AI-driven system for learning English communication skills that is in use in a number of countries around the world. Dr. Johnson described results from four snapshot evaluations of Enskill English. The first evaluation was performed with intermediate-level English learners at the University of Novi Sad in Serbia. This evaluation assessed the suitability of Enskill English learners for intermediate-level learners and evaluated the performance of the Enskill’s natural-language dialogue system. After Enskill English was improved, another snapshot evaluation was performed at the University of Split in Croatia. Dr. Johnson went on to describe two additional snapshot evaluations at the Toluca Campus of UVM (Universidad del Valle de México). These studies show that students who practice repeatedly with Enskill English rapidly improve their spoken communication skills, and these improvements persist over time. Each evaluation tested further system improvements. Alelo continues to rely on D3 to ensure that Enskill continues to improve and addresses the needs of an ever-expanding learner population.
You can view the slides from the presentation here: