Let’s say you are on a team that needs to create a better user experience – you come up with some changes, get support from your stakeholders, and deploy. What happens next? Was the UX really improved? Can you justify to your stakeholders that you did your job?
Without credible UX success measurements, we all risk not being able to quantify our success. Without credible UX success measurements, we are unable to align our efforts to an organization’s business objectives and desired outcomes. This often results in UX efforts becoming very unfocused, undefined and easily changed on a whim. Basically, you’re left having to tell a very subjective story of your UX success or failures, which unfortunately, could lead to you and your team being very exposed. – Mark Disciullo, User Experience Strategist and Designer
In order to quantify the progress (if any) was made, your team should be conducting iterative studies and tests throughout the development of your app/site. Before conducting any study, it is necessary to define 1) the goals you want to reach and 2) the metrics you want to track.
But what are these metrics? What key performance indicators (KPIs) will help you validate your work and convince your stakeholders that your changes improved their UX? A formula (below) is comprised of three categories of UX metrics that will provide you with the foundation to create a well-rounded UX measurement strategy.
UX = (A) Effectiveness + (B) Efficiency + (C) Satisfaction
How well can the user accomplish a task?
- Error rate: Errors and usability issues are very closely related because user errors are basically the result from poor usability. Some variations of error rates are number of errors per task, errors per time segment, and errors in the total session.
- Assistance requests: How many times did the user asked their moderator for help? To gain more insight, record what task the request was made about.
- Completion rate: How many users were able to complete the task? Note, it is important to define what “complete” means before conducting the study.
How easy can the user complete a task? What is the user’s mental effort?
- Task time: This can be a useful metric for diagnosing problems and also for giving insight into how tasks evolves over iterations. Generally, the smaller the task time metric, the better user experience.
- Error recovery time: This metric may indicate the user’s mental effort when an error occurs. A low recovery time may prevent the user from getting distracted from the task at hand.
- Time spent using the documentation or help: Lower time using documentation indicates better information architecture.
- Use of Search vs Navigation: Another good indication of how the information architecture is performing. In general, a user will use search as a last means of resort to completing a task. This KPI = # of tasks completed with search / # of total tasks.
How does the user feel about the task?
- Satisfaction Rates: This is a way to understand this can be as simple as doing a 3-question post-task survey to gage the users overall ease of use, satisfaction, and “perceived” amount of time it took them to do each task. Then you can average out the score to get an overall satisfaction rating for each task.
- Likes, Dislikes and Recommendations: The user may want to give opinions that you or your team may not have considered. This is a great way to discover new insights and reassure ones that you are measuring. This can be added as open ended question at the end of a study survey.
- System Usability Scales (SUS): this is a way to quantify the qualitative data. It consists of 10 statements to which users rate their level of agreement on a five-point scale (i.e. 1= very unsatisfied, 5= very satisfied):
- I think that I would like to use this system frequently.
- I found the system unnecessarily complex.
- I thought the system was easy to use.
- I think that I would need the support of a technical person to be able to use this system.
- I found the various functions in this system were well integrated.
- I thought there was too much inconsistency in this system.
- I would imagine that most people would learn to use this system very quickly.
- I found the system very cumbersome to use.
- I felt very confident using the system.
- I needed to learn a lot of things before I could get going with this system.
Aside from measuring the foundational categories of efficiency, effectiveness, and satisfaction, a few more categories (if applicable) should be considered when rounding out your KPI set, including: usefulness, learnability, and memorability.
Set the right goals to find the right information
It is important to note that before being able to apply this formula, you must define what the actual goals of your study are. This will give you a base to say whether the changes that were made were successful or not. If you are looking to set usability goals, the overarching question should be: is the user interface easy to use? A few examples for defining usability tracking goals would then be:
- Users must complete the task in an average of X minutes
- Users must not spend any time on documentation
- Satisfaction rate must be an average of 90% or higher for the first two questions (typically calculated as percentage of users who respond with “Agree” or “Strongly Agree”)
Measuring UX Pays Off
It is a great way to quantify your hard work and presenting the metrics is an easy way for non-UX stakeholders to digest. You can even use the metrics to help calculate the effects it had on return on investment (ROI), which all execs love. Regardless of why you use metrics, it is important to still measure the changes you are making to your site/app to prevent higher costs and lower satisfaction rates. Who wouldn’t want that?
To find out more about planing and reporting your usability study, usability.gov has some wonderful resources, in particular:
Additional resources on how to measure UX: