There are many ways you can approach HF evaluation of a medical device and several different techniques can be used during the development process. This introductory guide discusses five methods recognised by the FDA and cited in IEC 62366 that can be used and the pros and cons of each.
1. Contextual inquiry
Often used in the very early phases of device development, contextual inquiry is an effective way to develop an understanding of end users, task and use environments. The technique involves observing real or representative end users using a device in an actual use environment. For example, this could be an observation of patients using their current device in their own homes to understand where the pain points are and what impacts safety and effectiveness of device use. Or, for a novel surgical product, you might observe surgical teams performing existing procedures to identify opportunities for improvement.
Observation is a key element of contextual inquiry but it’s also important to speak to users to understand their decision-making processes. Such discussions could be conducted during the device use or in a post-use interview. The technique can provide unique insights into difficult interactions, impact of the actual use environment, and problems associated with team interactions and user workload.
PROS
- Deep insights
- Naturalistic use of existing devices
CONS
- Ethical approval can be difficult to achieve
- Time consuming and expensive
For more information on contextual inquiry, see Chapter 5 in AAMI Applied Human Factors in Medical Device Design, co-authored by Ian Culverhouse and Mary Beth Privitera.
2. Cognitive walk-through
A cognitive walk-through is a review of device use based on an expected sequence of tasks and is used early in device development with low-fidelity prototypes. The technique usually involves a team of specialists, including HF experts, designers, engineers and representative users or clinical subject matter experts.
The team will systematically work through the sequence of tasks to identify where usability issues may arise and identify potential mitigations. If a representative user is available, they might interact with the prototype and discuss their thought process with the team, explaining any issues or confusion they have.
PROS
- Quick and cheap
- A good way to involve multidisciplinary teams and stakeholders early in development
CONS
- Unlikely to uncover nuances with device use
- Difficult to elicit tacit knowledge for users (this can be difficult to describe verbally)
3. Heuristic analysis
A heuristic analysis requires a pre-defined list of rules or guidelines – these are known as heuristics. (An example of heuristics for medical device usability can be found here but there are a range of heuristics available.) Typically, three or four usability experts will independently review the user interface of the device against the defined heuristic list. The experts will exercise their professional judgment to identify deficiencies with the device and possible potential solutions. When the independent reviews are complete, the experts should compare the findings and come to an agreement of appropriate actions.
Commonly, Jakob Nielsen’s 10 usability heuristics are considered a valuable starting point within UI and software design.
PROS
- Cheap and relatively quick
- Judges design against recognised usability principles
CONS
- Requires a team of experts
- Will not uncover usability issues specific to real end users
4. Expert review
Expert reviews are a very quick way of evaluating device usability at any stage of development. The review typically involves a usability specialist or a clinical subject matter expert using their knowledge of the tasks and environment, and prior experience with similar devices. These experts will identify strengths and weaknesses with a design to identify and prioritise potential improvements.
The key difference between an expert review and a heuristic review is that the expert review leans more heavily on the reviewer’s expertise and opinion regarding device use, clinical application and characteristics of end users. As such, the inclusion of multiple experts in an expert review will likely result in a more robust outcome and the identification of a higher percentage of potential usability issues.
PROS
- Cheap and quick
- Can be done at any stage of development
CONS
- Relies heavily on expert ability and understanding of the device use case
- Can miss a significant proportion of usability issues
5. Simulated-use usability testing
Usability testing is by far the most used method of testing and can take a wide range of forms. As with some of the above methods, simulated-use testing can be performed at almost any phase in development from concept to validation.
It involves putting a device prototype in the hands of a user and asking them to complete a task or series of tasks in a simulated environment. It’s known as ‘simulated-use’ because usually some part of the task must be simulated rather than actual use; for example, using skin pad for injections or carrying out a study in a mocked up operating theatre. For more information on this see our blog post on simulated environments and usability testing.
Simulated-use testing early in the development phase is known as formative testing and typically offers a degree of flexibility in terms of the study protocol. Conversely, the final usability study is known as summative or validation testing. For formative testing the prototype is typically used by 5-8 participants representative of each identified user group for the device, as opposed to a minimum of 15 in each validation group.
A quick side note on sample size: For usability testing, regulatory authorities are not looking for statistical significance in the results and not interested in 8 out of 10 users being able to use the device as intended; thus, sample sizes are relatively small. What we are trying to do is identify potential use errors (or the absence of use errors) and a paper called Beyond the five-user assumption: Benefits of increased sample sizes in usability testing by Laura Faulkner in 2003 has been used to set the bar for this. Faulkner’s paper found that around 86% of know use errors would be found from five people interacting with a device, rising to 97% with fifteen users hence the numbers identified for inclusion in usability testing.
With robust protocol design, careful and accurate recruitment and professional moderation, usability testing is undoubtedly the most effective method for usability testing a device. However, the process can be both expensive and time consuming and as such requires careful planning.
PROS
- Should uncover a significant proportion of potential use errors
- There’s no substitute for observing real users using your device
CONS
- Expensive and time consuming
- Can require significant resource to complete