|
According to ISO-9241, product effectiveness is defined as �accuracy and completeness of user goal achievement�.
Effectiveness can be calculated as percentage of users successfully achieving their goals vs. the total number of users: normally, as a result of coming through user scenarios, users either achieve their goals or fail to achieve them.
For usability test, a certain number of respondents � actual or prospective product users - are to be recruited in accordance with the specified target group profile.
Effectiveness graph example for 11 respondents and 5 scenarios:
Respondents who have completed scenarios successfully are marked with green, unsuccessful ones marked with red.
Let N - be the total number of scenarios (goals)
R - is the number of respondents (users)
nij � is the result of coming through scenario i by respondent j;
nij=1 if the scenario has been completed successfully and user goal has been achieved, and
nij=0, if the scenario is unsuccessful and user failed to achieve the goal.
Then, overall integral product effectiveness E will be calculated according to the following formula:
 |
(1) |
Effectiveness statistic error:
 |
(2) |
Overall product Effectiveness:
 |
(3) |
Physical meaning of Effectiveness is product efficiency factor.
Example 1. 4 users work with a product according to 1 scenario.
Three users complete the scenario successfully and one user fails.
Overall user effectiveness of the product E = ((3*1 + 1*0)/(4*1))*100% = 75%.
Example 2. 2 users work with a product according to 2 scenarios.
The first user has completed scenario 1 successfully yet failed to complete scenario 2.
The second user has failed with scenario 1 yet completed scenario 2 successfully.
Overall user effectiveness of the product E = ((1*1+1*1)/(2*2))*100% = 50%.
Example 3. 11 users work with a product according to 5 scenarios.
The first scenario has been successfully completed by 8 users, the second one by 6 users, the third one by 10, the forth one by11, and the fifth one by 4 users (see the diagram).
Overall user effectiveness of the product E = ((8+6+10+11+4)/(11*5))*100% = 71%.
Statistical error in tests with a small number of respondents is certainly high. However, even a small number of respondents are able to reveal a lot of user errors in a product which cause scenario completion failures.
As a rule, the optimum respondent number for product effectiveness test is 11-15. This number of respondents is enough to reveal 90-95% of all major user errors in the product, statistical error of the result calculation being about 10%, so the overall product Effectiveness can be determined with sufficient degree of confidence:
| 0-50% |
50-75% |
75-90% |
90-100% |
| awful |
bad |
normal |
good |
Further: Efficiency
Users and Goals | Effectiveness | Efficiency | Satisfaction | Methods
|
![[5] page: 5.1](../img/51.gif){kind=small}
