Johnson & Johnson Medtech

Usability Testing & Risk Management

Usability testing and risk management for surgical tools

Overview

As a human factors engineering coop, I contributed to 4 formative and 1 summative studies. I managed a formative study from start to end: writing the protocol and discussion guide, moderating with 15 nurses, and drafting a usability evaluation report. I analyzed quantitative and qualitative data and performed root-cause analysis. Additional tasks included collaborating with quality, medical affairs, and industrial design teams on FMEA documentation to identify hazards, severity of harm, and design mitigations; contributing to IEC 60601 assessments; and drafting use specification documentation.

Time

3 months


Role

Human Factors Engineering Coop


Team

Human Factors Industrial Design


Client

Johnson & Johnson

PROBLEM

Pilot Display & Controls for 3 Key Mission Phases

There are 3 key phases of the Artemis V mission that need pilot interaction on the MK2 lunar lander. How do we design and test pilot display and controls during these 3 phases to optimize pilot performance?

1) Landing

Descending to the lunar surface
‍

2) Ascent

Ascending from the lunar surface to return to Gateway
‍

3) RPOD (Rendezvous, Proximity Operations, and Docking)

Approaching and docking at Gateway space station in lunar orbit

Team Goal

Our team was tasked with designing the display & controls for each key mission phase. These designs, along with the underlying models and guidance algorithms are then implemented in a human-in-the-loop simulation and evaluated for pilot and system performance and vehicle handling qualities.


What Did I Do?

ONBOARDING

Learning about Aerospace & Aviation

How do I quickly learn about basic aerospace and aviation concepts?


‍This was my first time working on an aerospace control & display project so I had a lot to learn to even understand the problem space! Below are some notes I took about key aviation and aerospace concepts that I found helpful during my onboarding process:
RESEARCH

Research Questions

How do I tackle the daunting task of designing ascent screens?


In order to break down the larger task of designing ascent screens, I came up with a list of questions that I needed to answer before I started creating designs:

1) What is the current mission conops? How does ascent fit into the larger mission timeline?
2) Define the ascent phase. When does it start and end?
3) How is the ascent and descent process different?
4) What are the sub-phases within ascent?
5) What features are necessary for auto vs. manual control during ascent?
6) What features are needed on each piloting screen (i.e. PFD, VNAV, Orbit View) for each phase of ascent?

Desk Research

What can I learn about ascent from internal and public archives?


To answer my questions, I first did as much desk research as I could. I dived into Draper's internal archives about heritage ascent display designs and the Artemis V concept of operations (conops). I also looked into publicly available information about aviation displays and ascent in general.

Subject Matter Expert Interviews

With the remaining questions I had after desk research, what can I learn from the SMEs around me?


After desk research, I interviewed Draper's internal Subject Matter Experts to learn more about ascent + aviation! Special thanks to the following people:
‍

Phil Hattis

Working on manual spaceflight systems for 50 years

John Nafziger

Active pilot

Thomas Fill

16 years supporting Shuttle ascent & real-time flight support on Orion

DESIGN

Feature Prioritization

There are so many features we can design. Which ones should we prioritize and implement for our upcoming pilot evaluation with Blue Origin's test pilot team?


In order to plan for piloting evaluations, we needed to understand and prioritize what the simulation team could feasibly implement given the project timeline. After a meeting with the simulation team, I created a feature prioritization list for the initial version and a long term version. The initial version would be tested with pilots on the flight simulator and the long term version would be presented as slides.
‍

Design Process

How did I use usability evaluations, SME interviews, and HITL testing to inform my designs and optimize pilot performance?


After creating a feature prioritization list, I created design mockups in Sketch which was then implemented on the flight simulator. Design was an iterative process where I received feedback from my design mentors, internal SMEs, and eventually from pilots via HITL testing.
‍

Final Design Concept

What did the final design look like?


Although my mockups are not available for public viewing, here is the general design concept:
‍

Image of PFD on Space Shuttle
‍

SCREEN #1

PFD (Primary Flight Display)

○ Pilot's primary reference for flight information

SCREEN #2

VNAV (Vertical Navigation Display)

○ Provides situational awareness of vertical navigation performance

image of VNAV from FlyByWire Simulations
‍

image from the game Space Simulator (orbit view in top right corner)

SCREEN #3

Orbit View

○ Helps pilot navigate to the planned orbit

USABILITY TESTING

Experience of obtaining medication (initiation)

FINDING #1

Lack of guidance on how to manage medications

Of 22 participants, over half (13) used trial and error to find their current adherence strategy; 5 participants used previous experience of helping someone else manage their medication and 4 received suggestions from a friend or family member. No participants received guidance from a medical professional about how to devise an effective medication management practice, such as where to put, how to store, or how to develop a routine around taking their prescription medications.

Implication: This lack of guidance from medical professionals presents a missed opportunity to increase medication adherence, especially with the low adherence rate in the US.
‍

FINDING #2

Increasing mail orders and 90-day supplies 

Two thirds of participants had prescriptions delivered by mail and almost all [21/22] received a 90-day supply of medication. 
‍
Implication: Mail order further limits opportunities for pharmacist guidance, such as how to correctly take medications, or patients to ask questions. This is further compounded by almost all participants receiving a 90-day supply.


‍

Experience taking medication at home (implementation)

FINDING #1

Great variance in medication management

No two participants had a medication management routine that was exactly the same. This was despite some participants using similar adherence devices or taking medication at similar times. For participants who used a pill case, there was great variety in when they refilled their pill case; where they stored their pill case; and how they remembered to take their medication. For other participants, they developed unique, complex routines that included tactile triggers. One participant flipped her pill bottles and moved it across the microwave to keep track of their adherence.

Implication: Medication management is complex and is developed to fit a unique daily routine and person. What works for one person may not work well for another. 

"I came up with a scheme, where I keep the medicines on one side of my microwave, or my toaster oven. When I take it, I put it on the other side."
– Participant 4

FINDING #2

Pill case was the most popular adherence device 

Majority of participants[17/22] used a pill case.

Implication: Weekly pill cases may be popular because they provide direct feedback on whether or not someone took their medication using visual cues.

FINDING #2

Reliance on multiple triggers to remember to take medications 

For this study, we define adherence triggers as actions that are taken or objects that are encountered that help patients remember to take their medication. All (100%) participants relied on at least two triggers to remind them to take their medication, while 68% relied on three or more. Action triggers included eating a meal (50%) and getting ready for bed (25%); object triggers included a pill case (77%) and a water glass (18%). One participant who relied on three triggers – taking medication with a meal, using a pill case, and placing it on the dining table –  missed the first trigger but saw their pill case which acted as a fall back reminder.

Implication: Under some circumstances, multiple triggers served as “a safety net,” providing different avenues for medication reminders. How can we leverage this to improve adherence routines?


‍‍

Underlying reasons for non-adherent behavior (persistence)

FINDING #1

Change of routine was the greatest contributor to non-adherence

The most common reason for non-adherence among participants was a change of routine[13/22], which caused an absence of a specific trigger. For example, one participant who relies on breakfast as a trigger forgets to take her medication when she skips breakfast:
‍
Implication: Implication: How can we make an adherence strategy durable under a change in daily routine?

"If I have to go somewhere, first thing in the morning, that's a typical time when I forget. Because sometimes I don't even have time for breakfast or for one reason or another didn't get around to it. Then the next day, it's Monday, but I'm looking at the Sunday case saying, ‘Oh, I guess I forgot to take it yesterday.’ " –Participant 7

Conclusions

All participants in our study did not receive guidance from a healthcare professional about how to manage their medication or develop an adherence strategy. Given that we know many patients struggle with adherence, there may be value in guiding patients to develop an effective adherence strategy or recommend changes to make their current strategy stronger. Secondly, people develop unique – and sometimes surprisingly complex – routines to remember to take medication. For example, participants relied on multiple action and object triggers to remember to take their medication. Participants also became less adherent during the absence of usual triggers such as a change of routine. We hope to further probe what makes a trigger durable under disruptions to routines; if multiple, durable triggers lead to greater adherence; and how to guide older adults in developing a more successful medication strategy. Lastly, time-based reminders are the most common reminder mechanism in consumer medication management apps and devices, but few study participants reported using them or taking their medication at an exact time, instead relying on routines and time ranges. Medication nonadherence is a growing problem, which reminders can potentially mitigate. Our results highlight the need for research outside of time-based reminders, such as routine-based reminders, to increase medication adherence. While a small sample, the analysis suggests that there are opportunities to provide guidance to older adults in developing an adherence strategy and design better aids to adherence that leverage established daily routines.

Co-design activities

We had two co-design activities. The first revolved around understanding the overall patient experience from being prescribed a medication to adhering to their medication and identifying pain points along the way. The second focused on blue sky ideations about possible solutions addressing those pain points.

Activity 1: User Journey Map & Pain Points

Activity 2: Blue sky ideations

Pilot Evaluations

How did I test my designs?


Pilot evaluations took place in the span of 4 days. I helped run testing for landing, ascent, and RPOD in the lab by moderating and taking notes. Test pilots were debriefed/ trained beforehand. I collected data on workload using the Bedford workload scale and handling qualities using the Cooper-Harper Rating Scale after each run. After a pilot went through all their runs for a phase (landing, ascent, or landing), I helped conduct a debrief interview.
‍

image of Orion cockpit simulation setup

HITL Simulation Setup

○ The simulation setup is not available for public viewing but it was similar to the Orion cockpit w/ windows, display panels, and hand controllers
○ Test pilots wore eye tracking glasses

Measuring Handling Qualities

○ We used the Cooper-Harper Rating Scale to measure vehicle handling qualities after each simulation run

Measuring Pilot Workload

○ We used the Bedford Workload Scale to measure pilot workload after each simulation run

Ascent Presentation to Test Pilot Team

○ After the test pilot team completed their ascent runs, I presented and facilitated a discussion on the long term ascent designs. I received feedback on my designs, which informed my next design iteration.  

Me and my colleague in the Draper atrium

Handling Qualities Report & Design Iteration

How did I synthesize and take action on results from Pilot Evaluation?

Handling Qualities Report

I synthesized qualitative and quantitative findings for ascent and created a list of design recommendations for our next design iteration. My analysis was included in a Handling Qualities report that was given to Blue.

Design Iteration

I mocked-up potential design changes based on the handling qualities report. I also created an ascent handoff documentation describing each design component, rationale for the component, and a traceback to a HITL or SME interview that justified my design decisions.
REFLECTIONS

Key Takeaways

Human Factors in Aerospace

Aerospace is so complex! On my first day, I felt like an aerospace engineering student learning about how pilots fly spacecrafts and its underlying physics and navigation systems. At Draper, I learned how to collaborate with the simulation team, manual control and guidance, and test pilot team. I loved working with all of these different pieces and the interesting problem spaces they put me in. I also learned how to use the Cooper-Harper Rating Scale, flight simulator, and eye tracking equipment and had the chance to apply what I learned about cognitive workload in the classroom to this project.

Human in the Loop Simulation

It was an incredibly valuable experience seeing pilots test my design and talking to them about their piloting process. Informal conversations throughout the day (both between pilots and with me) provided just as much feedback as the formal debriefs.

Writing Design Requirements & Creating Designs

I learned not only how to write design requirements based on NASA's HF standards but also how to apply them in my designs