|
According to ISO-9241, product Efficiency is defined as "resources spent by user in order to ensure accurate and complete achievement of the goals".
With regards to software products and information systems, the key measured resource normally is time spent by user in order to achieve the goals.
Thus, Efficiency can be calculated as user Effectiveness divided by the time spent by user.
Efficiency graph example for 11 respondents and 5 scenarios:
Let N - be the total number of scenarios (goals)
R – is the number of respondents (users)
nij – is the result of coming through scenario i by respondent j;
nij=1 if the scenario has been completed successfully and user goal has been achieved, and
nij=0, if the scenario is unsuccessful and user failed to achieve the goal.
tij – is the time spent by respondent j to come through scenario i. In case of unsuccessful scenario completion, measured till the moment of scenario quittance by the respondent as a result giving up the goal or logging off the system.
Then, overall time-based user Efficiency of a product Pt will be calculated according to the following formula:
 |
(4) |
Time-based Efficiency calculated so in fact is the speed of work with the product with effective users vs. all users. Of note, unlike Effectiveness, time-based Efficiency is an absolute value, not relative.
Calculation including also the time spent by ineffective users will give the overall relative time-based Efficiency of a product defined as the ratio of effective users' work time to all users' work time:
 |
(5) |
It may also be possible to calculate additional relative product Efficiency, the so-called Expert Efficiency. In this instance, apart from users belonging to the target group, there is also an expert involved in the test. The expert is a system architect or a user fully aware of the system architecture and therefore able to come through scenario with the maximum possible user speed and a-priory successful completion.
Then, the time-based expert Efficiency will be:
 |
(6) |
where
t0i – is the ideal time an expert needs to complete scenario i
The physical meaning of the time-based expert Efficiency is the highest theoretically possible speed of work with the product.
Hence, relative time-based Efficiency is:
 |
(7) |
where
t0i – is the ideal time an expert needs to complete scenario i
E – is the overall system effectiveness
The physical meaning of the relative expert efficiency is the measure of potential efficiency relative to actual system efficiency with regards to its user effectiveness.
Efficiency statistical error:
 |
(8) |
Overall product Efficiency:
 |
(9) |
Example 1. 4 users work with a product according to 1 scenario.
Three users complete the scenario successfully and one user fails.
Scenario completion time with the first user is 1 sec, 2 sec with the second user, 3 sec with the third one, and 10 sec with the fourth (unsuccessful) one.
Expert completes the scenario in 1 sec.
Time-based Efficiency Pt= (1/1 + 1/2 + 1/3 + 0/6) / (1*4) = 11/24 (goals/sec)
Overall relative Efficiency P = ((1*1 + 1*2 + 1*3 + 0*10)/(1 + 2 + 3 + 10))*100% = 37.5%
Expert relative Efficiency Pe = 75%* (4*1)/(1 + 2 + 3 + 10) = 18.75%
Example 2. 2 users work with a product according to 2 scenarios.
The first user has completed scenario 1 successfully yet failed to complete scenario 2.
Scenario 1 completion time is 1 sec, the time before interruption of scenario 2 is 8 sec.
The second user has failed with scenario 1 yet completed scenario 2 successfully.
The time before interruption of scenario 1 is 10 sec, scenario 2 completion time is 4 sec.
Expert completes scenario 1 in 1 sec, scenario 2 in 2 sec.
Time-based Efficiency Pt= (1/1 + 0/8 + 0/10 + 1/4) / (2*2) = 5/16 (goals/sec)
Overall relative Efficiency P = ((1*1 + 0*8 + 0*10 + 1*4)/(1 + 8 + 10 + 4))*100% = 21.7%
Expert relative Efficiency Pe = 50%* (2*(1+2))/(1 + 8 + 10 + 4) = 13%
Like with Effectiveness, statistical error in tests with small number of respondents is rather high.
However, even a small number of respondents is enough to show the proportion between productive effort spent by a user while working with a product and wasted effort.
Further: Satisfaction
Users and Goals | Effectiveness | Efficiency | Satisfaction | Methods
|
![[5] page: 5.1](../img/51.gif){kind=small}
