Members: Liz Betts, Daniel Castano, Willie Payne, Ming Min Tay

Problem Statement

Benjamin is a 9 year-old boy with a rare genetic condition called Phelan-McDermid Syndrome. He is non-verbal, has limited expressive skills, and has pervasive low muscle tone — making it difficult for him to interact with many objects including playing with toys. He gets frustrated easily as he is unable to express his needs and wants accordingly.

One of Benjamin’s favorite activities is to watch and rewatch the movies he loves, and while he can use an augmentative and assistive communication (AAC) device to tell his family what he wants to see, it is in no way a comprehensive solution. For one thing, a family member must always be present to start a movie since Benjamin is unable to use a remote control. With that, Benjamin cannot control movies once they start playing, e.g. to fast-forward past scenes that upset him. Furthermore, the AAC device is a poor interface for accessing movies since it requires step-by-step navigation and multiple screens to reach a movie list, along with fine motor control to navigate and select the intended movie. Benjamin’s deficits in hand functions and reduced learning ability make this a demanding task.

Our Solution

Our prototype will need to reduce the steps it takes for Benjamin to access the movies he wants to watch as well as encourage his independence in partaking in an activity that so many others enjoy without second thought.

Measurements of Success

We will evaluate project success in two primary ways – through direct observation and interaction with David and through interviews with Benjamin’s family and caretakers.

First, we will engage with Benjamin and witness the impact of our prototype. Measures of success include the degree of independence from Benjamin’s family, the reduction of time and complexity in playing a movie, and the increase of accuracy in picking the correct movie. Our prototype should not only work once, but should function long-term in Benjamin’s environment and require minimal maintenance.

Second, we will interview Benjamin’s family and caretakers as we work with him to deploy our prototypes. Questions might include the following: How has Benjamin’s mood changed since using the prototype? Describe any positive experiences Benjamin has had so far. Describe any negative experiences Benjamin has had so far. In an ideal world, how would the prototype be enhanced?

10/2/2018 – 10/16/2018

Project Objective for 10/2/18-10/23/18: Form project teams, problem analysis, order equipment.

After receiving our assignment we took time to evaluate the project objectives, order equipment and set up a meeting with our client. The October 16th class session was devoted to creating rough prototypes of new solutions to the issues we’d identified. Pictured at the top of the page, our prototype aimed to radically simplify Benjamin’s interaction with the iPad movie selector — making the process of choosing and watching movies much simpler and and more accessible.

This process definitely clarified some of the challenges ahead and the questions we needed to answer in order to move forward. We all agreed that a very provisional test session with Benjamin — to confirm or deny our assumptions — was necessary.

10/18/18

First online meeting with Benjamin and his family: At this session we had an opportunity to get answers to questions that were raised during the prototyping session. We also requested videos of Benjamin using the system currently in place.

10/23/18

Project Objective for 10/23-11/6: Create low-fidelity prototypes/mockups, begin hacking at technical challenge

This week’s class session focused on Midterm presentations — our project and progress to date:

Click for Midterm Presentation

Feedback we received on our presentation strongly suggested that we pursue IR technology and the adaptation of the AppleTV remote as a starting point.

10/30/18

Our AppleTV device has arrived. Prototyping begins in earnest!

*Experimenting with an Arduino-based alternate remote for AppleTV*

In addition to the simplified touchscreen concept we proposed in the October 16th prototyping session, we may test a separate, button activated selector. This could have layout similar to the touch screen movie selection system Benjamin is currently using. To see if this idea could be executed, Daniel began experimenting with an Arduino based alternate IR remote for the AppleTV.

*Concept IR selector device for use with AppleTV*

11/2/18 First User Testing Session

Things to be tested:

An iPad selector prototype: a more refined version of the rough prototype we designed in our prototype class. This was a carousel style series of screens offering one movie option per screen and greatly enlarged control buttons. A sample of the screens appears below:

*Screen for iPad prototype showing the option to select a movie, go forward in the selection carousel, or go backward in the selection carousel*

We had a number of buttons to test. Most of these were AbleNet style switched specifically designed for assistive use. A second button test was an arcade-style button which would illuminate an LED when pressed.

*LEFT: A table covered in supples for the User Test. Pictured are several AbilityNet buttons, electronic toys, an iPad, etc. RIGHT: When pressed, this arcade button illuminates a red LED.*

We also had a very early version of an arduino controlled – ir replacement remote for the AppleTV unit. Difficulties with the devices kept us from any significant software testing with Benjamin but Daniel was able to make progress in creating the routines which might ultimately be used to activate selected movies through the AppleTV unit.

*At lower right, the Arduino and breadboard which will replace Benjamin’s Apple TV Remote.*

The Test:

For most of the prototypes, the intent was to test our physical equipment with a “Wizard of Oz’ procedure where, once Benjamin had touched a button or seemed to have activated a process, we would simulate the intended response of the AppleTV, loading and playing the selected movie on youtube.

As a baseline test of button options, we first asked Ben to try some of the buttons which were connected to a toy dinosaur. On a button press, the dino would respond, indicating that the press had been successful. Even as his initial reserve diminished, we had difficulty getting Ben to try buttons on his own. Though it seemed that he could activated them, engaging him in the requested action required parental intervention.

The enlarged selector offered in the iPad mockup seemed to work for Ben. Though he was not familiar with the sequence of presses required, he definitely knew to press the iPad movie images in order to select a film.

Click to see video of Ben using iPad prototype

The arcade button designed to illuminate an LED on press was confusing to Ben. Because the button is raised from the surface it’s mounted on, he grabbed it as he might a knob, rather than pushing it.

Once we began showing his favorite movies on the large television screen, Ben became much more engaged in the testing process.

Willie, Daniel and Ming Min began creating a series of buttons with movie images attached to them (see photo). With a series of 4 large buttons modified with images from his favorite movies, we asked Benjamin to select a movie by pressing a button. He would choose a movie by pressing the corresponding button and we would then play the movie on the tv screen.

Click to see Ben Select Movies with Large Buttons

We repeated this test a number of times, stopping the movie and asking Ben for a selection. After a few tries he was clearly comfortable with the drill and readily selected movies he wanted to watch.

Team Member Conclusions:

WILLIE

If possible, it is important to allocate enough time for the session, have prototyping materials on hand, and bring together people from different backgrounds. Benjamin was largely unresponsive for the first hour. We were able to learn a lot from the session because the entire team was present to brainstorm and prototype in response.

We learned that given time, Benjamin can certainly learn a new (simple) interface. We also learned that “Wizard of Oz” techniques can work well enough for testing prototypes with Ben. He was able to discover that if he pressed a button with an associated picture, he would watch a clip of that movie.

Knowledge of even little details beforehand can be key. Two examples: One, Ben doesn’t really like playing with toys. The stuffed animals we brought did not motivate him to try and press the buttons. Two, Ben has very specific movies he likes, not just animated kids films. The iPad prototype was a somewhat inadequate test because Ben only wanted to watch one of the four movies we picked (Toy Story – he doesn’t care much for Mulan, How to Train Your Dragon, and he does not like Lion King.)

Going forward, we will try to create a series of buttons that correspond to movies stored in the family’s iTunes library. Each button will need to send the appropriate commands to start a movie from the home screen. The controller will need to work for Ben, fit in the family’s home setup, and be positioned such that the IR transmitter points at the Apple TV. Ben’s family believes simply starting the movie is more important than any of the other controls like stopping or fast forwarding.

MING MIN

– Grateful for Benjamin’s family to come to us and provided a better understanding of Benjamin’s condition, expectation aligned and goal set.

– The collaboration of team members giving us a handful of ideas to trial and error

– The rapport building may be able to reach faster if we gather information of Benjamin’s likes and dislikes beforehand, but interacting with children with disabilities can be quite challenging especially meeting for the first time.

– The time of meetup, Benjamin seemed tired from the journey and long hours of attending baby shower before meeting us. This factor may affect his attention and engagement.

DANIEL

1. I felt the physical buttons worked better as an interface than the iPad.

2. I image the physical buttons as a product fixed next to the TV and apple TV.

3. I will like to have a way to maintain the product, regarding updating the movies. Maybe we can create a SEQUENCE RECORDER feature, in which Ben’s parents can record any sequence they want, in order to reach any movie in the future.

4. I had the impression a play/pause/skip/forward buttons will be too much, but this is because I don’t know Ben that much.

5 I guess the question is how picky is he with the movies, and how long he watches at them before he gets tired. Is he spending 2 hours in front of the TV watching one entire movie?

6. Also, what about he chose a movie and he just walk away from the room. Who or How is the tv gonna be turned off, or stop?

12/2/18 Second User Test Session

*Prototype team members watch a movie with Ben at the second user test session. Clockwise from left, Daniel, Willie, Ming Min and Ben (foreground)*

Things to be tested:

For this session we focused on two areas: choosing specific buttons to incorporate into our device, and completing the code for the routine that would be transmitted to the Apple TV unit through IR signals.

Buttons:

We had 3 sets of buttons to test, all of which required varying degrees of pressure to activate : a pair of 3″ round, arcade-style buttons, a pair of 2″ round, arcade-style buttons and a pair of index-card-sized, flat, rectangular buttons which had pictures of the corresponding movies on their faces. The round buttons had large images of the movie selections on a board behind them.

Code:

We had asked the family to bring their AppleTv unit so the code our team had developed could be tested and polished with the intended device.

The Test:

When the family arrived, Benjamin definitely remembered our previous testing session and seemed eager and excited to be back. As soon as everyone seemed settled Willie and Daniel began work with the family’s Apple TV unit. Ming Min and I tested the button options with Benjamin. Once again we used a “Wizard of Oz” technique, where Benjamin was asked to select a movie using the buttons, and we would then play the movie on our laptops. Though an indicator light would signal whether the button press had been successful, there was no actual connection between the buttons and the movie player devices.

The largest buttons required the most pressure to activate. We requested that Ben try each button style multiple times. Out of many tries with each of the button sets, the only button set that yielded any result were the rectangular picture buttons. This successful activation occurred only once or twice.

Click to see Ben Using Rectangular Picture Buttons

Seeing that Ben’s light touch was a problem, we quickly assembled a set of 3 capacitive touch buttons which we hoped Ben would activate by simply touching the image of the movie he wanted to watch. Unfortunately these also failed to register a signal at Ben’s touch.

Without the option of the very sensitive AbleNet buttons we used in the first test session, our path forward was limited. The rectangular picture buttons had the benefit of being visually similar to the iPad selection system Ben had been training with. Ben’s mom, felt that with some training, Ben could learn to apply more pressure and successfully activate the picture buttons.

As the session concluded Willie and Daniel demonstrated the code on the family’s Apple TV. The many steps required to access the desired movie through their IR remote system made the process much slower than we hoped, but the code worked!

Team Member Conclusions:

WILLIE

Our controller solution involves sending a list of commands to navigate automatically to a movie after Ben presses a corresponding button. This is challenging for a few major reasons: First, there are a whole lot of screens! A single movie alone requires at least four buttons to start after it has been selected. Just getting to that movie requires multiple arrow keys and select buttons. The sequence of buttons changes depending on what screen the family left the AppleTV on. Second, in order to reduce button presses for most users, the AppleTV will keep track of its current state to speed up navigation. For example, if a user has recently been watching “Tangled” and then they navigate Home, the next time the user goes to Movies, it will skip most of the movie screens and directly navigate to playing “Tangled.” This is problematic for us because it means there is no constant direct route to a given movie. Third, the AppleTV only picks up a single input at once, and there is a certain amount of latency before it can accept another input. This means that our solution must wait a little while after each command is sent.

Our solution to the first problem simply involved lots of trial and error with the family’s AppleTV to produce the correct list of inputs to get to each movie. In response to the second problem, we figured out various methods of “reset” so that at the beginning the sequence moves the AppleTV to Home, and every time the input arrives at a new screen (Home, Movies, Purchased, etc.) it moves the cursor to the top left. (This is particularly challenging on the home screen because the same buttons have different results like triggering the screen saver depending on where the cursor is when the input is received.) Finally, solving the third problem required sending inputs as fast as possible most of time, but more slowly at key events to allow the AppleTV, e.g. switching between screensaver and home screen, and loading a movie. As a result, our solution appears fairly robust, but is very slow (over 40 seconds) to account for all the possible states and timing differences. It is unclear whether Ben will be able to wait long enough for the movie to start or learn that the process takes time.

MING MIN

To have a more organized preparation and setup prior to trial
The group focusing on getting the trials done with all the devices but appeared to be more environmental stimulating for Ben to focus on tasks compared to previous session
Good teamwork to get the solutions built up on the spot when the original plans not producing desired outcomes.

DANIEL

The time to get to the movie is too long and Ben may leave the room before gets to the movie.
He gets distracted of we have more displays around and he is very responsible to the sound.
Capacitive touch feels like a good tech if we find more sensitive hardware, because he is definitely familiar with the iPad so he touches slightly

12/4/18 Review Session with Professors

Using prototypes from the user testing session, we described the outcome of the second user test. Scott suggested that, given the very light pressure Ben applies in making a selection, a “leaf switch” might be a good solution. Though we did prototype a “leaf switch” selector for our final presentation, it was too late for us to test it with Ben. Supporting our team’s conclusion, our advisors suggested that we build and finish the rectangular picture button option — both for presentation to our class, and for Ben and his family to use.

12/11/18 Final Presentation

Final Presentation:

https://docs.google.com/presentation/d/1rkg-aKnnx5Zyn5_0QQe81aV9j4RademWq-H13znOvLA/edit?usp=sharing

Final Code for the IR Apple TV selector:

https://drive.google.com/open?id=1FRx76aGJQa9P_gY1CesyBQMglOVqjFSE

Documentation/Instructions for Updating the Selector with New Movie Options:

https://docs.google.com/document/d/1b8xrPbep_4eVlIVKyaw8K4w2gYEx6adLK81PixZggvw/edit?usp=sharing