Usability Engineering: Observe Users, Improve Product Safety

Up until now there’s been much emphasis on designing to make a product “idiot proof”. This has provided some benefit, but what Usability Engineering is reminding us is that it is the designers who are sometimes viewed as idiots by the users. It is the users who are the experts (in usability).

For those involved in product safety, we could perhaps congratulate ourselves. Based on my testing experience over the last three decades, it is my view that products have become safer. No longer is it common to see products cause electrical shocks, burns, fires, or crushing/cutting injuries. We continue to see where we need improvements, particularly when we see new technologies, such as recent events associated with rechargeable lithium batteries. Tragic events associated with energies and materials in electrical products occur, but they have become quite remote.

The focus of this article will be on Usability Engineering for medical devices. We will look at the present state of medical device safety. The data will show poor usability is to blame for more preventable deaths than traffic collisions and firearms combined. We’ll look at new Usability Engineering process requirements and provide an overview on how we can better control the risks associated with poor usability. Even though the focus of this article is on medical devices, the principals hold for all products. Poor usability represents the low lying fruit for safer products.

The Present State of Medical Device Safety

As reported by the post-market surveillance group of the U.S. Food and Drug Administration (FDA), between 2005 and 2009, there were 56,000 adverse events (undesirable experience) involving infusion pumps, resulting in at least 700 deaths. There were 87 manufacturer initiated product recalls. In March 2010, the FDA ordered Baxter to recall 200,000 infusion pumps because of “numerous flaws”. Other pump manufacturers took note, and voluntarily instituted their own product design reviews and, where necessary, recalls.

Based on a new study, “A New, Evidence-based Estimate of Patient Harms Associated with Hospital Care” by John T. James, PhD, published in the Journal of Patient Safety in September 2013, it is estimated that between 210,000 and 440,000 patients die in US hospitals due to preventable medical errors; a four-fold increase over 1999 estimates. The study also estimates that medical errors cause serious harm (e.g. loss of limb, sight, hearing), ten-fold to twenty-fold more common than lethal harm. The study details how better analysis of four past studies justifies the new estimates. These medical errors include those caused by medical and in vitro devices (IVD), both active and non-active, and the administration of pharmaceutical drugs.

Prior to this recent study, the best estimate of preventable medical errors that cause death had been an Institute of Medicine article from 1999, “To Err Is Human”. This older study extrapolated data from hospitals in CO/UT, and NYC, and estimated at least 44,000 people, and perhaps as many as 98,000 people, die in hospitals each year as a result of preventable medical errors (adverse events).

Figure 1 shows the midpoint of the 210,000 to 440,000 estimated deaths due to medical error, alongside deaths due to traffic collisions and firearms.

Figure 1: USA Deaths from Medical Errors (Adverse Events) in Perspective

The best source for aggregate medical device adverse incident/event data seems to be the UK based Medicines and Healthcare Products Regulatory Agency (MHRA). In Europe (including the UK), an adverse incident, causes, or has the potential to cause, unexpected or unwanted effects involving the safety of device users (including patients) or other persons.

The chart shown in Figure 2 shows adverse incidents by year, based on MHRA (UK) Annual Adverse Incident Reports from 2007 (which includes data back to 2001, and 2010 (which includes data back to 2008). We see the upward trend of adverse incidents. As this could be due to an increase in medical devices in use, I plotted as well UK population, based on World Bank data. From the population data, we begin to see some correlation between the two increases.

Figure 2: UK Adverse Incident and Population Trends

To better look at adverse incidents to population, I charted in Figure 3 adverse incidents per 1 million persons. I also broke out death and near death, from other adverse incidents having less severe outcomes.

Figure 3: UK Adverse Incidents per 1 Million Persons

Figure 4 shows adverse incidents by device type.

Figure 4: Adverse Incidents by Device Type

In Figure 4, the Other category includes (each with less than 5%) Surgical consumables, Aids for daily living, Syringes/needles, Disinfection/sterilization/disposal, Drainage/Suction, Beds/mattresses, Hoists, Artificial limbs, Walking aids, Physiotherapy equipment, and Orthoses.

Only some of reported adverse incidents are investigated by MHRA. Of those chosen for investigation, Figure 5 shows to whom responsibility for the incident was assigned. In assigning responsibility MRHA uses the following system:

• Healthcare facility, Use: After delivery; use errors, performance and/or maintenance failures and degradation
• Manufacturer: Before delivery; design, manufacture, quality control and packaging
• Unknown: intermittent faults (use error, software, EMC) or couldn’t investigate

Figure 5: Cause of Investigated Adverse Incidents

In looking at the adverse incident data one needs to be wary of reaching any definitive conclusions. Problems with the data include:

• Increase real? Or due to better reporting?
• Need to know adverse incident per devices in use
  • Are high adverse incidents for a device type due to in use numbers, device complexity, or other?
• Cause investigations should target use error specifically
  • Don’t lump in with performance and/or maintenance failures and degradation by healthcare facility
  • Differentiate use error due to inadequate training by healthcare facility, etc; from insufficient usability by device manufacturer
  • Categorize by device failure mode (e.g. transformer, switch, software, EMC), or use error
• Increase in unknown causes results in less useful data (e.g. assigned causes)
  • Pull out suspected use error, software, EMC causes

Hats off to MHRA for providing aggregate data, even if not perfect. It would be nice to see FDA publish aggregate data annually, and/or make their databases a bit more accessible (they’re searchable, but for aggregate data, not easy to download and reconstruct the relational tables).

Problems aside, one could reach the following qualified conclusions:

• 26% more adverse incidents per capita
• 29% more death or near death
• 82% involve more complicated equipment, such as implants, surgical, patient monitors,
• infusion pumps, IVDs,
• wheelchairs, imaging, and similar
• During 2005-06, majority of cause was health facility, use
• During 2007-10, cause was shared between healthcare facility, use; and manufacturer design, controls

Do No Harm

The latin phrase, Primum non nocere, “first, do no harm”, is attributed to to Thomas Sydenham (1624–1689) in a book by Thomas Inman (1860), Foundation for a New Theory and Practice of Medicine. Putting things in the terminology of modern risk management, (e.g. ISO 14971:2007), where a medical device has an unacceptable risk of harm, a designer needs to implement effective risk control measures.

With the above adverse incident/event data in mind, take a look at Figure 6. What’s the most likely hazard or failure mode that could result in harm? Hopefully everyone recognizes that it’s the User Interface. As designers we need to recognize that this is an important, and perhaps the most important, design responsibility.

Figure 6: Failure Modes that could lead to Hazardous Situations

Usability Engineering

Usability Engineering, or as FDA refers to it, Human Factors Engineering, is the process to identify where user interactions with a medical device have the potential for harm, and to implement effective risk control measures. The Usability Engineering process touches all design aspects; the hardware interface, the software interface, product markings, and any user documentation. Considered is usability associated with the full product life cycle, from transport, normal use, maintenance, to decommissioning.

Key standards to guide a manufacturer’s Usability Engineering process:

• IEC 62366:2007 + A1/FDIS:2013, Medical devices – Application of usability engineering
• IEC 60601-1-6:2010, Medical electrical equipment — Part 1-6: General requirements for basic safety & essential performance – Collateral standard: Usability
• ISO 14971:2007, Medical devices – Application of risk management
• ANSI/AAMI HE75, 2009 Edition – Human factors engineering— Design of medical devices
• Medical Device Use-Safety: Incorporating Human Factors Engineering into Risk Management, issued 2000
• Apply Human Factors and Usability Engineering to Optimize Medical Device Design, issued 2011 (draft)

The IEC and ISO standards have EN (CENELEC) versions for Europe, and are harmonized to the essential requirements of the Medical Device Directive related to ergonomics and information supplied by manufacturer. All are in the U.S. FDA recognized consensus standards database. They become the means to provide a presumption of compliance with essential requirements and a reasonable assurance of safety and effectiveness, with regards to acceptable usability.

All these standards are consistent with each other. The scope of IEC 62366 (which I consider the high level process standard) is all medical devices, including the more prevalent non-active devices like tubing sets, luer connectors, syringes, dental implants, sterile drapes; as well as electrical equipment like surgical equipment, patient monitors, in vitro diagnostic equipment, and non-implantable accessories to active implants. IEC 60601-1-6:2010, the medical electrical equipment collateral standard for usability, contains essentially only a normative reference to IEC 62366. The AAMI HE75 is useful as it has more specific guidance and examples. FDA guidance documents are also written to provide more specific examples, use FDA terminology, and provide references for further reading. Think of the AAMI HE75 and FDA guidance documents as informative annexes to IEC 62366.

Figure 7: It’s all about the User Interface

In the remainder of this article we focus on IEC 62366.

IEC 62366 tells us that users want good usability:

• Effectiveness
• Efficiency
• Ease
• Satisfaction

With these user motivations and taking into account the use environment we can anticipate and investigate user actions (or interactions) such as pushing a button, toggling a switch, sliding a door, turning a screw, tapping a menu item, speaking into a microphone, filling a reservoir, or connecting a leadset.

Figure 8 provides terminology to refer to user actions (or interactions). Discussions are helped when we all use the same terminology. Note that ideally medical devices are desired to result in what we call, Correct Use; the designers intent; the device fulfilling its intended clinical purpose/use.
As designers we must also anticipate Use Error, (or reasonably foreseeable misuse), which can be Slips, Lapses, or Mistakes. Slips are due to buttons or menu items being too close together such as the maximize and close buttons in Windows. Lapses are due to too much complexity for the use environment. Slips and Lapses are unintentional. These should be fairly routine to anticipate and control.

Figure 8: User action (interaction) categories (IEC 62366:2007, Figure B.1)
click image for larger view

Mistakes are more interesting. A designer needs to anticipate and investigate (assisted by user input and observation) where a user might default to behavior suggested by the user interface, or seek a shortcut. Mistakes are always intentional.

I like to think of mistakes as something Homer Simpson might do. Homer has good intentions, but nonetheless, somehow always seems to find himself in trouble.

Homer in the episode where he becomes “Max Power”, says to Bart, “There’s the right way, the wrong way, and the Max Power way.”

Bart asks, “Isn’t that the wrong way?”

Homer explains, “Yeah, but faster.”

I think this sums up the new mentality that designers need to adopt.

Abnormal use is intentional and beyond any further reasonable means of risk control by the manufacturer. Think Pete Townshend from The Who and what he used to do to guitars after a concert (for young readers; he smashed them into bits and pieces). As reducing risk from abnormal use is beyond further reasonable means, a manufacturer has no further responsibility to reduce this risk.

For those versed in the terminology of the medical equipment safety standard series, IEC 60601, Table 1 provides a quick mapping.

Table 1

We can see the intent of IEC 62366 is to remind us that reasonably foreseeable misuse or use error needs to be considered “normal”. This is true of both IEC 60601 (clause 4.1) and IEC 62366, but IEC 62366 adds emphasis by using the term normal use for both correct and use error.

Consider as well that the term use error is NOT called user error. Use of the word use instead of user is intentional to emphasize that it is the designer’s responsibility to risk control use error where it could result in harm. Use error should not be considered the user’s fault.

Figure 9 illustrates well that the Usability Engineering process has continuous improvement provided by its post-market surveillance feedback. This is much like a quality management system with its customer feedback, process metrics, and internal auditing, feeding into the management review and CAPA (corrective action, preventative action) process. A risk management process has post-market surveillance as feedback for risk control improvement.

Figure 9: Usability Engineering process (IEC 62366:2007, Figure D.1)

Key aspects of a Usability Engineering process during the design phase:

• Application specification
• Frequently used functions
• Usability hazards (user input & observation)
• Primary operating functions
• Usability specification
• Validation plan
• Design & implementation
• Verification
• Validation (user input & observation)

The Usability Engineering process starts with a documented list of what the device is intended to do — the application specification. We analyze and investigate this list to determine frequently used and otherwise primary operations related to safety — frequently used and primary operating functions.

Based on our analysis and investigations, where use error could result in unacceptable risk, we add risk controls. These risk controls are defined in the Usability Specification. These can be included with other design requirements related to customer, business, and device failure risk controls, but there needs to be a means (e.g. a flag), to identify those related to usability risk controls, as these are inputs for the usability validation plan. The usability risk analysis process is repeated as the design becomes more detailed.

A validation plan needs to be formulated to define the method(s), (e.g. test user population profile, interviews, simulated clinical use, actual clinical use, etc.), and criteria for usability validation. The testing method(s) and compliance criteria allow a validation of the effectiveness of the risk control measures.

Verification can be carried out by engineering, as usability risk control measures such as the color, or blink rate, volume, or spacing to adjacent buttons can be verified. Validation necessarily involves users, as detailed in validation plan.

Usability Trends in Other Product Sectors

Not only the medical device sector recognizes the importance of Usability Engineering. With the newest version of the safety standard for equipment for measurement, control, and laboratory use, IEC 61010-1:2010, 3rd ed, we have a new clause 16, which mandates that reasonably foreseeable misuse and ergonomic issues be addressed with risk assessment (analysis, evaluation, and where needed, effective risk control). Risk assessment is a new clause 17.

In the newest version of the safety standard for information technology equipment, IEC 60950-1:2005 + A1:2009 + A2:2013 (consolidated ed 2.2), in the principles for safety it mentions the need to consider foreseeable misuse. There is no separate clause for this hazard. But, as with all product safety standards, (i.e. the physical requirements for enclosures) foreseeable misuse is taken into account.

In the newly published, but as yet not widely used, safeguards based standard IEC 62368-1:2010, Audio/video, information and communication technology equipment – Part 1: Safety requirements, the term reasonably foreseeable misuse is defined. However its use is limited to the normative Annex on batteries and fuel cells. Nonetheless, having the term defined will facilitate useful safety discussions.

Risk Management and Usability Engineering

With both risk management and usability engineering, unacceptable risk is mitigated with risk control measures, defined by design requirements, in turn verified and validated. Post-market surveillance provides feedback.

With risk management, hazards are identified and risks defined by the design team including clinical application specialists.

Figure 10: Cause and effect related to risk

With usability engineering, usability hazards are identified and risks are defined by user input and observation. Validation explicitly requires a formal plan to define how and by what criteria user input and observation will be sought and evaluated. It is this emphasis on user input and observation that Usability Engineering brings to existing quality system design controls and risk management.

Usability Engineering for Legacy Devices

User interfaces and user manuals for legacy devices are already designed. We cannot very well go back and follow a Usability Engineering Process without having to go back and effectively undertake the whole design process again — something that isn’t going to make business sense for products that have good experience in the market. This is much like off-the-shelf software, or what IEC 62304, the software safety standard, calls Software of Unknown Provenance (SOUP).

With the forthcoming Amendment 1 to IEC 62366, we now have what we call, User Interface of Unknown Provenance (UOUP). As with legacy hardware and SOUP, with UOUP, we have a practical process for conducting a sufficient review of UOUP, taking into account our new appreciation for the importance of good usability.

Amendment 1, Annex K, anticipated in first quarter 2014, provides a UOUP process for legacy devices:

• Relook at Application specification (K.2.1); develop list of Frequently used functions (K.2.2); Primary operating functions (K.2.3)
• Relook at Post market information (K.2.4)
• Relook at Hazard, Risk Analysis records (K.2.5);
• Consider need for any additional Usability risk control measures (K.2.6)

Take Aways

Designers need to anticipate and investigate use error (reasonably foreseeable misuse):

• Optimize Usability (effectiveness, efficiency, ease, satisfaction)
• Risk control behavior that could result in unacceptable risk of harm

Users are the experts:

• User input and observation needed by design team, including clinical application specialists
• Users validate effectiveness of usability specification (risk control measures)

Based on a review of aggregate medical device adverse incident/event data, use error would seem to be a significant contributor.

Usability Engineering represents a new tool to help us design safer products. Manufacturers who adopt a Usability Engineering process will create safer products. Greater reliance on user input and observation makes intuitive sense if we are to reduce risk associated with use error.

Finally, with better adverse incident/event data collection, we will have the data to assist with root cause analysis, identify areas for improvement, and evaluate our performance.

References

• FDA PMS Group
• A New, Evidence-based Estimate of Patient Harms Associated with Hospital Care, James, John T. PhD
• To err is human
• USA deaths rates due to traffic and fireams
• MRHA annual adverse incident reports
• Origin of “Do no harm.”
• Max Power video
• iec.ch
  • IEC 62366:2007 + A1/FDIS:2013, Medical devices – Application of usability engineering
  • IEC 60601-1-6:2010, Medical electrical equipment — Part 1-6: General requirements for basic safety & essential performance – Collateral standard: Usability
  • ISO 14971:2007, Medical devices –
  • Application of risk management
  • IEC 61010-1:2010, 3rd ed, Safety standard for equipment for measurement, control, and laboratory use
  • IEC 60950-1:2005 + A1:2009 + A2:2013 (consolidated ed 3.1), Safety standard for information technology equipment
  • IEC 62368-1:2010, Safety standard for audio/video, information and communication technology equipment
• aami.org
  • ANSI/AAMI HE75, 2009
  • Edition – Human factors engineering— Design of
  • medical devices
• fda.gov
  • Medical Device Use-Safety: Incorporating Human Factors Engineering into Risk Management, issued 2000
  • Apply Human Factors and Usability Engineering to Optimize Medical Device Design, issued 2011 (draft)

Frank O’Brien teaches a best selling training course in IEC 60601, the medical electrical equipment family of standards for basic safety and essential performance, with locations in San Jose, Boston, Galway, and Amsterdam. His Boston based consulting firm is O’Brien Compliance Management, obcompman.com. He participates on IEC TC62 committees. He worked 24 years at Underwriters Laboratories where he evaluated literally 1000’s of medical devices. In the past Frank has called home, San Jose, Frankfurt Germany, Long Island NY, and Syracuse NY. Frank has a Bachelor of Science in Electrical Engineering from Clarkson College, NY; and a Master of Science in Technology Management from SUNY Stony Brook, NY. When not busy with work, he enjoys time with his fiancee in Ireland, grandkids on Long Island, his family camp in Maine, exchanging stories over a pint of Guinness, a baseball game at Fenway, quiet moments with a cup of coffee, and solving puzzles.