Improving reading interventions through learning science
In my capstone project at CMU, we worked with Houghton Mifflin Harcourt on improving a current software program for struggling readers, and proposing concepts for the next version of the program. Five CMU METALS students grouped together and formed the team EvolvEd. I collaborated with the product manager, UX researchers and another product designer.
My role on the team was the product designer & Tech lead, which was in charge of conducting interviews and competitive analysis, sketching storyboards, crafting user flow and wireframes, designing some of the activity screens and story visuals, and implementing the functional MVP. I also iterated the final prototype based on the in-person user test results.
Product Designer & Tech Lead
Jan 2018 - Aug 2018
Zach Mineroff, Roger Strang, Julia Ridley, Lu Yang
Sketch, InVision, HTML/CSS/JS, Twine
The project lasted for eight months. We conducted user research in the first half and proposed eight solutions in the spring presentation. Based on insights from the research and the needs of HMH, we spent the second half creating an interactive storytelling prototype, Endeavor, that will help students become more fluent readers and provide targeted practice as necessary, informed by student log data. We iterated five rounds with user feedback and log data to balance the student learning outcome, user experience and business value.
WHAT is the problem?
There are many struggling readers suffering from dyslexia from 3rd to 12th grade. The original program published by our clients is in widely use across schools in the United States. However, many students do not use the program as expected, leading to inefficient course time and minor progress in reading.
WHY is it important?
Reading is important for everyone in all walks of life. Many students we interviewed do not actually have disability and they love reading. Their not-perfect school performance or so-called “lagging behind“ are due to lack of proper education from home and school. We are hoping to develop a learning software that students could read passages as well as learn new words.
HOW does our product help?
Our product, Endeavor, helps in those three aspects: engagement—getting students engaged in reading long passages, personalization—personalized for students at different levels, accountability—holding them accountable for reading as well as learning.
Introducing our product—Endeavor
Endeavor is an interactive narrative activity for struggling readers. Students read passages and choose their path through an adventure, promoting a sense of autonomy and personalization. We answered the “How might we…“ questions with the minimum viable features (MVP) :
1. How might we give students the feeling of being a “driver” rather than a “passenger” in their learning journey?
The compelling narrative, combined with key visuals and interactivity, keeps students engaged and motivated to continue reading the story. The text is leveled for different abilities and students can request difficult passages to be read out loud, making Endeavor appropriate for students at all stages of the existing program.
2. How might we hold students accountable for actually reading the passage?
Comprehension and decoding assessments are embedded within the narrative and are designed to quickly identify the types and degrees of errors students make.
3. How might we get students at different proficiency levels to read longer passages?
There are three layers of decoding assessments for each comprehension question. Every time students get the assessments wrong, they’ll drop into an easier layer of assessments, which makes sure that students can read text at their level.
4. How might we bridge the gap between reading instruction and actual reading?
For each comprehension question, we select five words that students make mistakes the most from the student log data. And for the decoding strategies, we choose four other words that to require students to distinguish between them. The student log data was pulled out from the original system’s database.
Our Hunt Statement that guided our research process is:
Through analyzing student interactions with the program and its ecosystem, we will design learner-centered features aimed at improving student outcomes, engagement, and accountability in phonics, personalized for various types of struggling readers.
Through observing and interviewing both the users, we explored the motivations and thought processes of students as they use the program. Analyzing the students’ interactions with the software also gave us empirical evidence about the types of activities they struggle with the most (or find too easy) and in what contexts.
We then used affinity diagramming to analyze the raw data collected from the contextual inquiries. It allowed us to consider how the findings fit into categories of instructional design including educational goals, instruction, and assessment.
Learning Journey Map
We presented our solutions to our clients in the spring presentation and after the presentation, we discussed with them and we agreed that there is a gap in the amount of support for reading texts outside of the isolated decoding exercises in the software program. We finally agreed on designing an interactive story-telling robot.
Thus, we formed the Mission Statement:
We will develop an interactive storytelling prototype that motivates students to practice skills they struggle with the most, enabling them to become competent and confident in reading long passages.
Continuing from our spring research, we pursued these guiding principles in our summer prototyping and used them as evaluations of success:
Engagement: students enjoy interacting with the prototype
Personalization: prototype adapts to students’ needs and enables self-expression
Accountability: students use the prototype as intended
Throughout the summer development period, we used an agile-type design framework, pulling elements from both SCRUM and KANBAN to ensure that our prototypes were built on time and easily changeable.
We completed our work in 2-week design sprints, each of which culminated in a user testing session. After the user testing, we had a combined sprint retrospective and planning meeting to revise our design process as necessary.
Prototype Version 1 - low fidelity
Our first prototype was a simple choose-your-own adventure story. The plot: it’s the last day of school and you happen upon a rocketship and a boat. After choosing a vehicle and a destination, an onboard robot helps you start the engines and avoid obstacles along the way by giving you passwords, which you spell or select.
User testing results
The user testing results was better than our expectation. However, since we did not integrate our two prototype tools, some students and teachers found that the story and the exercise were not integrated.
Prototype Version 2 - Mid fidelity
In order to be integrated, we wrote the HTML/CSS/JS code in our second prototype. And we further gamified the choose-your-own adventure. Students earned points for correct answers to spelling and decoding challenges, lost hearts for incorrect answers, and found collectible items at the end of a story. We also gave students a pre-test and post-test to see if the instructional feedback led to learning.
User testing results:
The risk of chocolate-cover broccoli
In our second round user test, though we tried to integrate the story and the activities together, we still received the comment from our clients that they did not feel they were actually “integrated“. The reason they were saying so was because that
The activities in our version of prototype were still too similar to the questions in the original system.
The content of the activities did not align with the students’ ability.
Students did not need to read the story but they could still get the questions right.
In other words, students may not be engaged in the story since they could answer the questions using their prior knowledge. We could not assume that the prototype motivates students to read long passage. So how might we avoid the prototype to be a chocolate-cover broccoli?
Prototype Version 3 - high fidelity
The solution to the chocolate-cover broccoli problem is that we use the comprehension questions in the prototype, and students could only get the answer from the reading passage. If the students get the comprehension questions wrong, it indicates that students have trouble understanding a certain word, so they jump into a word challenge to practice.
We built our story and challenges at the same time, to make them more integrated.
In the old version, the reason to perform was due to gamification elements (i.e. hearts, or lives). In the new version, it is to achieve a better story outcome.
User testing results:
Prototype Version 4 - higher fidelity
Prototype version 4 was perfected based on prototype version 3. In this version, we also designed a simple version for the students struggling the most.
In order to test this prototype with students with severe special needs, we scaffolded the story in three ways: drastically reducing the length and complexity of the story, adding graphics, and adding a “read it to me” button.
User testing results:
I learned a lot In this eight-months long project.
Designing learner-centered features and balancing learning and user experience
How to balance learning and user experience is an important concept we learned in class. The gamification elements may improve user experience (engagement) but may not improve learning experience. It’s always worth thinking when involving gamification elements in our design.
The importance of feedback casting on the product development
In the process, we made two major changes. One was changing the direction from teacher-oriented to students-oriented. Another one was that instead of designing a story embedded with separate activities, we designed a story with activities well-integrated. Both of the changes were directed by the feedback provided by our client. Next time when I’m facing my client, I’ll always ask for feedback frequently.
The importance of teamwork across different roles
We encountered several problems in the process, for example, we couldn’t interview the students with severe special needs to get a direct user experience, there’s a gap between design and actual implementation, etc. This project provided me an opportunity to learn how to cooperate with other team players and communicate efficiently across different roles.