No fishing allowed
The most common mistake in applying usability standards is to assume that they are a kind of fishing expedition to find the weak points with the device, through extensive trialling with real users. For example, a touch screen on a medical device with a slow update to a new screen or confusing layout might cause the user to frequently press the wrong keys.
This fishing expedition view is generally what you will find if you search for information or books on usability engineering or usability testing. It is an extremely good idea, with valuable experienced gained getting out of the design walls and seeing how the device works in the real world.
At the same time it sends a shiver down the spine of many small to medium manufacturers, thinking about the cost and also how to deal with the usability feedback, especially late in the design, or worse for existing products already on the market.
Fortunately, the fishing expedition view is wrong.
The regulatory view
While valuable, field trials are often vaguely formatted, lack formal specifications, and allow users to provide feedback on any aspect they feel is important. In many cases, the designers are not even sure what issues they are looking for, they are just seeking broad based feedback. This feedback will be analysed, filtered and decisions made as to whether to change the design and specifications. Field trial feedback can be extensive and lead to significant upheaval in the design; meaning the device is far from stable. This lack of definition (specification, pass/fail result), weak records (device configuration, environment) and early stage in the design (relevance to the final marketed product) can make field trials largely irrelevant in the regulatory context. Although manufacturers may feel they must keep the records of field trials, the design changes that occur after the trial often make the results unrepresentative of the final marketed product - in other words, not usable as a regulatory record. The impulse to treat the records as formal comes from the false assumption that prototypes are "medical devices" and hence all design records must be kept.
In a regulatory context, all standards should be viewed with respect to the medical device - the final product that is actually placed on the market. One of the frequent mistakes in regulation is to view prototypes as medical devices, making them the under scope of regulations. This is not correct - all results in the regulatory file must be representative of the final released medical device, and should be in the form of objective evidence against an approved specification, with a pass result. Under this rule, much of the data taken in the early stages of R&D are simply historic records with no regulatory significance.
Consider for example the way that an electronics engineer handles the performance test which is intended to go into the regulatory file: the engineer would have gone through all the R&D work, including fishing expeditions, trial and error, debugging and refinement, finally arriving at a point of stability in which they are confident of meeting the performance specifications and the planned tests. In the early stages of development, while many records are created, few of these would meet the quality required for formal regulatory records, and most are irrelevant with respect to the final product due to the cumulative effect of design changes. In contrast, the final tests are performed in a controlled environment, with well defined specifications and test methods, and test records detailing the environment, test sample serial numbers, software revisions, hardware configurations, traceable test equipment, who did the tests and when, as well as the actual test result, and an awareness the result must represent the marketed device. This formal (and rather heavy) test environment is certainly not intended to go fishing to find bugs in the design.
The same concept should be applied to usability engineering - all of the fishing expeditions, field trials and the like should have been done well before embarking on the formal IEC 62366 test. The formal test should only be performed when the design is sufficiently stable and the designers are confident of the result. The specifications for usability should be written in a way that provides a clear pass/fail result, and most importantly the specifications should tie into the risk management file - wherever the file indicates the user is involved in managing the risk, the usability assessment forms the evidence that the risk control is effective. For example, if the risk management file says that a certain type of incorrect set up can lead to serious harm, and the risk control refers to the instructions for use (or easy assembly or labelling), these need to be validated through usability testing.
Time to relax
With this formal view of IEC 62366 in mind, designers of stable products can relax somewhat and set up usability targets focusing on risk controls with specifications that are reasonably expected to be met. If that still feels scary, chances are that the content of the risk management file is illogical - unfortunately another frequent problem is the use of instructions, warnings and cautions in the risk management file to sweep away issues that are actually dealt with in other ways in the real world. A careful review often finds that the user was never really relied on in the first place, and hence a usability assessment would be meaningless. In particular, for medium to high risk situations, there is almost always other means of controlling the risk since it is unrealistic to expect ≥99% of users will follow instructions, and would require huge numbers of real users to validate the effectiveness.
If a careful review of the risk management file still finds the user is genuinely the risk control, and the severity of harm is significant, the usability assessment needs to be done carefully with specifications and results that demonstrate the risk is acceptable. But this is expected to be the rare case.
Legacy products
For products on the market, the approach is very similar, except in this case the objective evidence can be derived from market experience, as opposed to pre-market testing. Specifications are still necessary, reporting and the strong link to risk management remains. But the key point is that it is in principle OK to point to market experience as evidence that a user based risk control is effective.
The final word
If any management system standard seems scary it is usually due to false assumptions about what the standard actually requires. In general, these standards are specification based (not fishing), flexible and resources can be adjusted to suit the situation. Responsible manufacturers that know their product well and are confident it is safe and effective should never fear these standards. Simply use the requirements in the standard as a checklist, and tick off each item one by one until the file is prepared. If the standard requires a record, make sure the record exists. If a requirement is subjective, document whatever the particular qualified responsible individual feels is appropriate for the situation. Avoid asking third parties for opinions, as they rarely know the product as well as you do.