Endeavor.png

Endeavor

 

Endeavor

Improving reading interventions through learning science

 

In my capstone project at CMU, we worked with Houghton Mifflin Harcourt on improving a current software program for struggling readers, and proposing concepts for the next version of the program. Five CMU METALS students grouped together and formed the team EvolvEd. I collaborated with the product manager, UX researchers and another product designer.

My role on the team was the product designer & Tech lead, which was in charge of conducting interviews and competitive analysis, sketching storyboards, crafting user flow and wireframes, designing some of the activity screens and story visuals, and implementing the functional MVP. I also iterated the final prototype based on the in-person user test results.

My role:

Product Designer & Tech Lead

Time:

Jan 2018 - Aug 2018

Team Member:

Zach Mineroff, Roger Strang, Julia Ridley, Lu Yang

Skills/Tools:

Sketch, InVision, HTML/CSS/JS, Twine

 

Project Overview

Snapseed.jpg

The project lasted for eight months. We conducted user research in the first half and proposed eight solutions in the spring presentation. Based on insights from the research and the needs of HMH, we spent the second half creating an interactive storytelling prototype, Endeavor, that will help students become more fluent readers and provide targeted practice as necessary, informed by student log data. We iterated five rounds with user feedback and log data to balance the student learning outcome, user experience and business value.

 

Introduction

WHAT is the problem?

There are many struggling readers suffering from dyslexia from 3rd to 12th grade. The original program published by our clients is in widely use across schools in the United States. However, many students do not use the program as expected, leading to inefficient course time and minor progress in reading.

faq.png
 
reading.png

WHY is it important?

Reading is important for everyone in all walks of life. Many students we interviewed do not actually have disability and they love reading. Their not-perfect school performance or so-called “lagging behind“ are due to lack of proper education from home and school. We are hoping to develop a learning software that students could read passages as well as learn new words.

 

HOW does our product help?

Our product, Endeavor, helps in those three aspects: engagement—getting students engaged in reading long passages, personalization—personalized for students at different levels, accountability—holding them accountable for reading as well as learning.

3 areas resize.png
 

Introducing our product—Endeavor

Endeavor is an interactive narrative activity for struggling readers. Students read passages and choose their path through an adventure, promoting a sense of autonomy and personalization. We answered the “How might we…“ questions with the minimum viable features (MVP) :

 Landing Page

Landing Page

1. How might we give students the feeling of being a “driver” rather than a “passenger” in their learning journey? 

The compelling narrative, combined with key visuals and interactivity, keeps students engaged and motivated to continue reading the story. The text is leveled for different abilities and students can request difficult passages to be read out loud, making Endeavor appropriate for students at all stages of the existing program.

 Narrative

Narrative

2. How might we hold students accountable for actually reading the passage?

Comprehension and decoding assessments are embedded within the narrative and are designed to quickly identify the types and degrees of errors students make.

 Comprehension question

Comprehension question

3. How might we get students at different proficiency levels to read longer passages? 

There are three layers of decoding assessments for each comprehension question. Every time students get the assessments wrong, they’ll drop into an easier layer of assessments, which makes sure that students can read text at their level.

 Decoding assessments

Decoding assessments

4. How might we bridge the gap between reading instruction and actual reading?

For each comprehension question, we select five words that students make mistakes the most from the student log data. And for the decoding strategies, we choose four other words that to require students to distinguish between them. The student log data was pulled out from the original system’s database.

 

Research

Our Hunt Statement that guided our research process is:

Through analyzing student interactions with the program and its ecosystem, we will design learner-centered features aimed at improving student outcomes, engagement, and accountability in phonics, personalized for various types of struggling readers.

3 areas resize.png

Through observing and interviewing both the users, we explored the motivations and thought processes of students as they use the program. Analyzing the students’ interactions with the software also gave us empirical evidence about the types of activities they struggle with the most (or find too easy) and in what contexts.

IMG_6195.JPG
 
IMG_5710.JPG

We then used affinity diagramming to analyze the raw data collected from the contextual inquiries. It allowed us to consider how the findings fit into categories of instructional design including educational goals, instruction, and assessment.

affinitydiagram_6in.jpg

Teacher Persona

Student Persona

 

Learning Journey Map

Artboard.png
 

Insights

insight.png
 
 

Ideation

storyboard.png
 We are brainstorming the solutions for each insight.

We are brainstorming the solutions for each insight.

We presented our solutions to our clients in the spring presentation and after the presentation, we discussed with them and we agreed that there is a gap in the amount of support for reading texts outside of the isolated decoding exercises in the software program. We finally agreed on designing an interactive story-telling robot.

Feature Matrix.jpg
IMG_2012.jpg

Thus, we formed the Mission Statement:

We will develop an interactive storytelling prototype that motivates students to practice skills they struggle with the most, enabling them to become competent and confident in reading long passages. 

 

Design process

Continuing from our spring research, we pursued these guiding principles in our summer prototyping and used them as evaluations of success: 

Engagement: students enjoy interacting with the prototype 

Personalization: prototype adapts to students’ needs and enables self-expression 

Accountability: students use the prototype as intended

 
3 areas resize.png
 

Throughout the summer development period, we used an agile-type design framework, pulling elements from both SCRUM and KANBAN to ensure that our prototypes were built on time and easily changeable. 

We completed our work in 2-week design sprints, each of which culminated in a user testing session. After the user testing, we had a combined sprint retrospective and planning meeting to revise our design process as necessary.

user test2.png
IMG_0376 2.JPG
IMG_0647 (1).JPG
Student testing Endeavor.jpg

Success Metrics

engagement-new.png
personalization-new.png
accountability-new.png
 

Prototype Version 1 - low fidelity

Screen Shot 2018-07-08 at 21.43.48.png
Screen Shot 2018-07-08 at 21.56.09.png

Our first prototype was a simple choose-your-own adventure story. The plot: it’s the last day of school and you happen upon a rocketship and a boat. After choosing a vehicle and a destination, an onboard robot helps you start the engines and avoid obstacles along the way by giving you passwords, which you spell or select.

User testing results

 
result1.png
 

The user testing results was better than our expectation. However, since we did not integrate our two prototype tools, some students and teachers found that the story and the exercise were not integrated.

 

Prototype Version 2 - Mid fidelity

In order to be integrated, we wrote the HTML/CSS/JS code in our second prototype. And we further gamified the choose-your-own adventure. Students earned points for correct answers to spelling and decoding challenges, lost hearts for incorrect answers, and found collectible items at the end of a story. We also gave students a pre-test and post-test to see if the instructional feedback led to learning.

 The spelling challenge activity. The player has 20 points and one heart.

The spelling challenge activity. The player has 20 points and one heart.

 The word challenge activity, after getting it incorrect.

The word challenge activity, after getting it incorrect.

User testing results: 

 
result2.png
 

The risk of chocolate-cover broccoli

In our second round user test, though we tried to integrate the story and the activities together, we still received the comment from our clients that they did not feel they were actually “integrated“. The reason they were saying so was because that

  • The activities in our version of prototype were still too similar to the questions in the original system.

  • The content of the activities did not align with the students’ ability.

  • Students did not need to read the story but they could still get the questions right.

In other words, students may not be engaged in the story since they could answer the questions using their prior knowledge. We could not assume that the prototype motivates students to read long passage. So how might we avoid the prototype to be a chocolate-cover broccoli?

 

Prototype Version 3 - high fidelity

The solution to the chocolate-cover broccoli problem is that we use the comprehension questions in the prototype, and students could only get the answer from the reading passage. If the students get the comprehension questions wrong, it indicates that students have trouble understanding a certain word, so they jump into a word challenge to practice.

 We brainstormed ways to integrate System 44 activities into coherent narrative situations

We brainstormed ways to integrate System 44 activities into coherent narrative situations

 The comprehension question. If they fail it, there is a reason to do the word challenge that follows logically from the narrative (in this case, choosing the wrong door panel).

The comprehension question. If they fail it, there is a reason to do the word challenge that follows logically from the narrative (in this case, choosing the wrong door panel).

 Story arc

Story arc

We built our story and challenges at the same time, to make them more integrated.

In the old version, the reason to perform was due to gamification elements (i.e. hearts, or lives).  In the new version, it is to achieve a better story outcome.

Narrative Models.png

User testing results: 

 
result3.png
 
 

Prototype Version 4 - higher fidelity

Prototype version 4 was perfected based on prototype version 3. In this version, we also designed a simple version for the students struggling the most.

In order to test this prototype with students with severe special needs, we scaffolded the story in three ways: drastically reducing the length and complexity of the story, adding graphics, and adding a “read it to me” button.

 A typical narrative screen at the beginning of a new part of the story, as indicated by the progress bar.

A typical narrative screen at the beginning of a new part of the story, as indicated by the progress bar.

 A word challenge activity.

A word challenge activity.

 
 Basic user flow

Basic user flow

 User flow and correspondent wireframes

User flow and correspondent wireframes

User testing results: 

 
result4.png
 
 

Style Guide

Styleguide 1.png
Styleguide 2.png
Styleguide 3.png
 

Reflection

I learned a lot In this eight-months long project.

Designing learner-centered features and balancing learning and user experience

How to balance learning and user experience is an important concept we learned in class. The gamification elements may improve user experience (engagement) but may not improve learning experience. It’s always worth thinking when involving gamification elements in our design.

The importance of feedback casting on the product development

In the process, we made two major changes. One was changing the direction from teacher-oriented to students-oriented. Another one was that instead of designing a story embedded with separate activities, we designed a story with activities well-integrated. Both of the changes were directed by the feedback provided by our client. Next time when I’m facing my client, I’ll always ask for feedback frequently.

The importance of teamwork across different roles

We encountered several problems in the process, for example, we couldn’t interview the students with severe special needs to get a direct user experience, there’s a gap between design and actual implementation, etc. This project provided me an opportunity to learn how to cooperate with other team players and communicate efficiently across different roles.