UI DesignerUser interfaces design, usability and UX evaluation
1 2 3 4 5
Web interfaces Web interfaces Mobile interfaces Mobile Icons Icons UX evaluation UX evaluation Theory of Usability Theory About me About me

[5] page: 5.1




Main page   /   Theory   /   Effectiveness



According to ISO-9241, product effectiveness is defined as «accuracy and completeness of user goal achievement».

Effectiveness can be calculated as percentage of users successfully achieving their goals vs. the total number of users: normally, as a result of coming through user scenarios, users either achieve their goals or fail to achieve them.

For usability test, a certain number of respondents – actual or prospective product users - are to be recruited in accordance with the specified target group profile.
Effectiveness graph example for 11 respondents and 5 scenarios:

Graph of usability effectiveness for 11 respondents and 5 scenarios

Respondents who have completed scenarios successfully are marked with green, unsuccessful ones marked with red.

Let N - be the total number of scenarios (goals)
R - is the number of respondents (users)
nij – is the result of coming through scenario i by respondent j; nij=1 if the scenario has been completed successfully and user goal has been achieved, and nij=0, if the scenario is unsuccessful and user failed to achieve the goal.

Then, overall integral product effectiveness E will be calculated according to the following formula:

Integral effectiveness (1)

Effectiveness statistic error:

Error of effectiveness (2)

Overall product Effectiveness:

General Effectiveness (3)

Physical meaning of Effectiveness is product efficiency factor.

Example 1. 4 users work with a product according to 1 scenario.
Three users complete the scenario successfully and one user fails.
Overall user effectiveness of the product E = ((3*1 + 1*0)/(4*1))*100% = 75%.

Example 2. 2 users work with a product according to 2 scenarios.
The first user has completed scenario 1 successfully yet failed to complete scenario 2.
The second user has failed with scenario 1 yet completed scenario 2 successfully.
Overall user effectiveness of the product E = ((1*1+1*1)/(2*2))*100% = 50%.

Example 3. 11 users work with a product according to 5 scenarios.
The first scenario has been successfully completed by 8 users, the second one by 6 users, the third one by 10, the forth one by11, and the fifth one by 4 users (see the diagram).
Overall user effectiveness of the product E = ((8+6+10+11+4)/(11*5))*100% = 71%.

Statistical error in tests with a small number of respondents is certainly high. However, even a small number of respondents are able to reveal a lot of user errors in a product which cause scenario completion failures.

As a rule, the optimum respondent number for product effectiveness test is 11-15. This number of respondents is enough to reveal 90-95% of all major user errors in the product, statistical error of the result calculation being about 10%, so the overall product Effectiveness can be determined with sufficient degree of confidence:

0-50% 50-75% 75-90% 90-100%
awful bad normal good

Further: Efficiency

Users and Goals | Effectiveness | Efficiency | Satisfaction | Methods