A human factors study typically involves a representative end-user using a medical device under simulated conditions. It is our job to observe these interactions, looking for incorrect or unexpected actions. But this is only the beginning; after the simulated tasks a crucial component of the session is the ability to conduct and effective root cause analysis interview. This stage of the study is intended to provide insight from the participant’s perspective on their motivations and actions to understand the underlying root causes. This process helps us to understand the reasons leading to any observed error and is expected by medical device regulators:
If a USE ERROR or use difficulty can lead to a HAZARDOUS SITUATION, the root cause of any such USE ERROR or use difficulty shall be determined. The root causes should be determined based on methods including observations of USER performance as well as subjective comments from the USER.
There are several components that can lead to a successful root cause interview, these points are discussed in the article below, starting with taking the participant back to the task during which the issue arose and discuss their actions without making them feel like they’ve done something wrong. This approach encourages the user to be honest about their actions and helps them articulate the reasons behind them. Understanding genuine motivations behind the use issue helps us appreciate the influences and triggers from the device’s user interface, enabling us to identify potential areas for improvement.
When to conduct a root cause interview?
In most situations, we explore root causes with the participant after all other study elements are complete. This is because if you start asking a participant questions between scenarios they may behave differently in the tasks that follow. For example, a participant may overlook the instructions during the first use scenario, but if we start asking questions at that point, they might be prompted to consult the materials more closely in subsequent tasks.
Determining exactly when to conduct a root cause debrief during a study session is a balance between ensuring that the participant has not forgotten the motivations for their actions and not biasing elements of the study that have not been completed. If the sessions are lengthy and the participants have cognitive impairments, there may be a need to deviate from the norm to accommodate what is most comfortable for the participants. Likewise, if a surgical procedure lasts several hours, the ability for a participant to recall from memory their actions from earlier in the day may be reduced.
How to conduct a root cause interview?
During the study session, our team compiles a list of observed use issues for discussion with the participant. Typically, the study team pauses the session after all tasks are completed to discuss and align on any issues that need to be further understood. Here are some tips on how to probe participants for root causes and obtain unbiased responses:
- Tip 1: Set the scene and introduce the purpose of the root cause interview to put the participant at ease and avoiding any feeling of wrong-doing. For example:
- “The questions we’ll run through next will help us understand why you approached things in a particular way. This will help us understand how the device design can be improved for future users.”
- Tip 2: Rewind the clock – take the participant back to the relevant steps to give them context and aid their memory. It may be helpful to ask them to talk through their actions:
- “When you first opened the box, talk me through how you prepared the device for use.”
- Tip 3: Embrace your inner toddler and ask why, why, why?!
- One of the most common tools cited in root cause analysis in other fields is Five-Whys. The technique involves asking up to five levels of why in order to drill down beneath surface level responses to determine the true underlying root cause.
- Tip 4: Use non-accusatory questions.
- Five-Whys can be very useful, but avoid sounding petulant or insulting the participant. Avoid ‘I saw you skip this step, why did you…?’ questions as they may seem accusatory
- Focus on the user interface rather than the participant, for example, ‘What led you to…?’
- Tip 5: Use open-ended questions to avoid leading participants.
- A caveat here might be when clarifying an observation or answer from a participant.
- Tip 6: Accept all participant responses without judgement.
- Be mindful of your body language and facial expressions when participants provide their responses. Remember, the participants are the experts in how they consider the device should be used – we’re not there to judge their behaviours.
- Tip 7: Repetition can help. In some instances, it may be useful to get participants to repeat actions if:
- It was not possible to fully observe the action the first time round
- The action was skipped by the participant
- It is useful to understand if the action was deliberate and thus likely to be repeated, or simply an innocent mistaken.
Study sessions are scripted to ensure all topics are covered and to keep sessions consistent between participants. However, use errors cannot always be predicted, so the root cause part of the study session will not be as simple as following a script. This can be daunting for inexperienced moderators, so preparing generic prompts can be helpful. Here are some examples:
- When you were doing [task], I noticed that you [use issue], can you please talk me through that? What led you to [use issue]?
- Could you please talk me through how you performed [task]?
- Where did you look for information on how to perform [task]?
- After you [use issue], what did you/would you do next?
- How would you describe your use of the instructions?
- How would you typically familiarise yourself with a device like this in real life?
- To check the clarity of the instructions, could you please find and read the instructions for [task] and describe in your own words what you understand by them?
- Could you please repeat [task]?
- Can you please repeat [task] following the instructions?
How long should a root cause interview take?
Ideally, all observed use errors will be discussed with participants during this part of the study session. This is usually the only opportunity to ask the participants for their perspectives. So, when planning a study, we always allow sufficient time for root cause questions. Typically the duration depends on the complexity of the device and the expected number of use errors. A pilot interview can be helpful to gauge how long may be required. If time is short, prioritising more serious or unexpected errors is necessary, as these are often more helpful for designers and engineers.
Before concluding the root cause interview, the study team should confirm among themselves that all use errors have been satisfactorily discussed.
What to do with root cause interview information?
Important: Human memory is fallible! While getting user insights about their motivations and actions is important to determine the root cause of observed use errors, participants have been known to misremember or deflect from actual root causes. Information given by participants should be treated as one input to inform rather than dictate the study team’s root cause analysis activities. Ultimately, the study team’s observations, training, experience and judgement should be used to determine true root causes.
For transparency, we always report participant-cited root causes separately to the study team’s expert root cause analysis. This presents a holistic picture to our clients and ultimately the regulators of what happened during each use event, drawing upon the participant’s point of view but also expert HF knowledge and experience. This approach is consistent with the application of root cause analysis in accident investigations where interviews with key personnel are one input to the process, but not the only account considered.
Conclusion
Becoming an expert in conducting root cause interviews takes time and experience. If you require assistance in designing and moderating studies from experienced human factors experts, please get in touch with Rebus to discuss your needs.