The early design phase of Virtu was done according
to ISO 13407 Human centred design processes for interactive systems
(ISO 1999). The standard includes four user centred design activities
to be started at the earliest stage of a project. These design
activities resulted in defining two main usage scenarios related to
two main end user groups for Virtu.
The first scenario is about teaching safety at work
for students of Occupational Safety Engineering at TUT. The second
scenario is testing and updating industrial workers’ knowledge of
safety issues related to their work. Virtu user interface was designed
applying usability practices including stakeholder and user
interviews, prototyping, heuristic evaluation, iterative design and
user testing so that both end user groups would find the environment
interesting and easy to use (Preece et al. 2002). Virtu supports
constructivist learning model where individuals are assumed to learn
better when they discover things themselves (Ahmad et al. 1998).
This paper describes usability problems of current
VLEs. Usability methods used during Virtu development process are also
outlined and, as a conclusion, some general VLE design guidelines are
presented. Potential usability threats of current VLEs reported in
publications are discussed in Section 2. Section 3 presents usability
engineering methods used to ensure usability of Virtu and the
summarized results of using each method. Virtu learning environment
produced by the software and usability engineering activities is
outlined in Section 4. Section 5 provides general VLE design
guidelines, which rose from Virtu development process.
2.
Usability problems of current VLEs
In their study Jones et al (2003) identified eight
primary causes of student withdrawal. Three of these causes were to
some extent technology related. Technical problems as such were
reported as a cause of withdrawal by 20% of the respondents. Also, own
IT skills were identified as a reason for withdrawal by 15% of the
respondents. Moreover, the use of technology had an impact on student
confusion and lack of understanding, which contributed to 15% of the
respondents’ withdrawals. Also Kent (2003) reports on students having
had difficulties in using a VLE. The lack of technical skills
necessary for understanding or participating in activities within the
VLE was reported to be one of the features the participants did not
like about the e-Learning module (Kent 2003).
Another aspect that propagates the student
withdrawal from a course is the impersonal and faceless nature of VLEs
(Kent 2003). Lindh and Soames (2003) state that there is still a need
for human-to-human interaction. For example, asking questions is much
more time consuming in purely electronic environments than in
face-to-face contact in a classroom. When more and more students are
part time students having work career and family life at the same
time, the challenge is to activate students and to make them to commit
themselves to a course.
Students’ interests towards a course can collapse
because of muddled and ambiguous electronic material. Identifying
content can be a problem (Lindh and Soames 2003). For example, a
document containing multiple pages may come out of printer without
page numbers or pages have irrelevant title or no title at all so they
cannot be connected to the topic later on. The level of navigational
freedom affects to the user’s feeling of control (Armitage et al.
2003). Navigational aids providing higher navigational freedom (maps,
indexes) give higher feeling of control compared to navigational aids
providing lower navigational freedom (back and forward buttons,
hypertext). Even navigational aids do not guarantee the optimal use of
learning material, but instructions on how to study non-linear
hypertext material are also needed (Beasley and Smyth 2003).
According to Beasley and Smyth (2003), students
often interact with VLEs in a manner that will not allow them to fully
exploit the VLEs’ potential. In their paper concerning students’
selective use of a virtual learning environment, Beasley and Smyth
(2003), in addition to their other results, reported on VLE features
which were either rarely used or were not used at all. These features
included individual progress report and asynchronous discussion forum.
To relieve the students’ stress caused by online
activities, Lawless and Allan (2003) recommend that the technology
used should be user-friendly and errorless and should provide a
pleasant experience for the user. Another aim of VLEs is to provide
cost effective education (Lindh and Soames 2003). By designing and
testing the VLE according to usability engineering methods before
taking it into use, all these objectives can be achieved.
In the beginning of the iterative design process, a
paper mock-up of Virtu user interface was developed based on
stakeholder interviews. Usability tests with real end users were
conducted using the paper mock-up and the test users were also
interviewed. Based on the test results, a software prototype of the
user interface was developed and usability tests with end users
performed. After the second usability testing round, a new software
prototype was designed and heuristic evaluation was conducted by an
usability professional.
As a result there was list of changes to be made in
every area of design: structure, content and user interface. Because
the shortcomings were detected before the implementation was
completed, they could be corrected before the final release version.
Thus, iterative design process diminished the need for changes in the
future.
Requirements were formed by specifying the
context of use (UsabilityNet) and thereby the purpose of use. Also
the user and organizational requirements were collected trough
stakeholder interviews (UsabilityNet). Technical limitations and
updates must be considered in early design phase, too
The stakeholders of Virtu were specified first by
listing all the factors affecting the use of Virtu and thereby the
people related. Virtu’s stakeholders
included end users, teachers in the Institute of Occupational Safety
Engineering, maintenance, marketers and those who are responsible for
the safety at work issues in industry.
Virtu has two main end user groups. The first end
user group is students of Occupational Safety Engineering at TUT and
the second group is industrial workers. The computer skills and the
requirements for the content of Virtu differ greatly among these
groups.
The approach to safety at work issues should cover
all the legislation concerning safety and health at work. It should be
challenging enough for students who already have a wide knowledge of
the subject. However, the content cannot be too theoretical, because
the learning environment is to be used by industrial workers, too.
They do not necessarily have a wide knowledge basis on the issue but
are more interested in practical ways of improving the safety of their
own work.
The human factor goals of end user groups
can be established according to Schneiderman’s five measurable
human factors:
§
time to learn,
§
speed of performance,
§
rate of errors by users,
§
retention over time,
§
subjective satisfaction. (Schneiderman 1998)
The priorities of the human factors vary between
students and industrial workers.
For students time to learn and retention over time
are the most crucial factors. Students use Virtu instead of
traditional course material, and it is supposed to be as intuitive to
use as a book and a pen. The usage time is limited to only few hours a
week as if there were traditional lectures. The knowledge of how to
use Virtu must be maintained between lectures.
Retention over time is even more important to
industrial workers, who may have an intensive course on safety at
work, and are returning to Virtu after long period of time. Subjective
satisfaction counts when the use of Virtu as a reference book is
discussed.
3.1.2
Context of use
The context of use turned out to be complex. At
least three environments and three different purposes of use were
defined. Users should be able to use Virtu independently at home or at
work, and, in addition, by groups in safety at work training. There
should also be a possibility to use Virtu as a quick reference book.
All these different usages must be taken into account in the user
interface design.
In most of the cases Virtu is used for studying
occupational safety engineering at university, training safety at work
issues related to everyday work in industry or revising issues related
to safety at work. Students are enrolled in a course dealing with
safety at work. Industrial workers are participating in a course
arranged by employee or otherwise are recommended or required to get
familiar with safety at work material provided by Virtu.
Students will mainly use Virtu in computer
classrooms with internet connections. Computers usually include
Pentium 500 MHz processors with 128 MB memory. However, high processor
and memory capabilities cannot be expected from computers used by
industrial workers at their workplaces, and in that environment
internet is usually not allowed. Both user groups can use the Virtu
also at home, but those environments cannot be taken into account in
the design process.
3.1.3
Technical limitations and updates
Virtu will be a commercial product and therefore
should not be used freely in the internet. Instead, some kind of a
copy protection or license practice in its distribution must be
implemented. CD-ROM was chosen as the distribution media. Distribution
via internet was rejected due to the requirements of participating
companies. In the future, Virtu might be used through a web browser,
and therefore there should be a possibility to change the user
interface to a web based version without changing the application
logic.
Legislation concerning safety and health at work
changes from time to time and also other new information on the safety
at work issues may emerge. To ease the introduction of updates,
changing the content must not cause changes in the user interface.
Also, changing the content should be possible without programming.
Internationalizing Virtu for Western cultures should be easy, too.
As Virtu was designed according the
ISO 13407 Human centred design processes for interactive systems (UsabilityNet,
ISO 1999), the proper usability engineering methods including
iterative design, user interviews, prototyping, user testing and
heuristic evaluation, were used to gather and verify Virtu´s
properties.
Virtu user interface was developed based on an
iterative design process, which means that new versions of the user
interface were produced based on the usability problems and
opportunities disclosed by empirical testing (Nielsen 1993). During
the iterative design process the interface was evaluated by end users
and usability professionals.
Iteration began with user interface drafts drawn in
a paper. There was one paper for each type of displays; menus, theory,
exercises, virtual enterprise, function points and description of the
enterprise. After end users had tested the paper version, a software
prototype was developed. In the first phase only menus and one case of
each functionality was implemented. After user tests, the first full
version was built for further testing.
Many aspects of usability can be studied by simply
asking the users. This is especially useful for issues that are
related to users’ subjective satisfaction and possible anxieties
(Nielsen 1993). In the beginning, when nothing was implemented, the
stakeholders of Virtu were interviewed. The analysis of results of the
interview produced requirements for the first paper prototype of Virtu.
During Virtu usability test with a paper mock-up, the users
were asked to fill two interview forms.
The first form was filled in the beginning of the
test. It contained open and multiple-choice questions concerning
users´ computer usage and demographic information. The age of
participating students varied from 20 to 29 and the age of
participating industrial workers from 27 to 53, being very typical to
both user groups. The sex ratio in interview was 4/5 females in
students and 2/6 females in industry workers. 80-83% of both groups
used computer about 3 hours per day. Most of the students but clearly
a minority of industrial workers used computer also for leisure. 2/6
of the industrial workers used operations and work management systems
in their work.
The second form, filled at the end of the test,
contained five open questions. This questionnaire was about the ease
of use of Virtu and the users’ subjective opinion about it. The
results indicated that all students but only half of the industrial
workers considered Virtu to be easy to use. The main reasons for
finding Virtu difficult to use were navigation problems. More than
half of the students and less than half of the industrial workers said
to be interested in using Virtu for studying purposes.
Prototypes are useful for conducting
system-specific usability tests early in the development process.
Their role is to demonstrate and test aspects of human-computer
interaction, which might be difficult or more time-consuming to
produce in another medium (Lindgaard 1994). Especially low-fidelity
prototypes like paper mock-ups are useful, because they are simple,
cheap and easy to produce and thus easy to modify for testing
alternative design ideas. Software prototypes as high-fidelity
prototypes are useful in testing out technical issues (Preece et al.
2002). Both paper mock-up and a software prototype were tested during
Virtu development.
User testing with real users is the most
fundamental usability method, because it provides direct information
about how people use computers and what their exact problems are with
the concrete interface being tested (Nielsen 1993). During Virtu
development, formative evaluation was carried out in order to improve
the user interface as a part of iterative design process. Formative
evaluation was performed as a typical thinking-aloud test.
The major finding of the usability tests with the
paper mock-up was the need to divide the content to different
difficulty levels to match the needs of different user groups. In the
beginning of the tests, there was only one common content, which most
of the industrial workers found quite irrelevant because it was too
theoretical compared to their work. Another important finding was the
users’ difficulty in navigating in the virtual enterprise section. A
ground plan of the virtual enterprise with marked function points, for
example Gate or Check in, was used for navigation. The
users did not recognize the function points as buttons but were
wondering what they were supposed to do with the ground plan.
There were also some minor findings. Navigation
from one page to another in the textbook section was considered
somewhat difficult by some industrial workers, because there were only
arrow-buttons without a text to indicate where the button leads to.
Most users considered the link from the virtual enterprise section to
theory topics quite unnecessary.
Usability testing with the software prototype only
provided some minor findings. Some of the function points in the
virtual enterprise section contain questions, for example ”Do you
know the nearest emergency eye rinsing at your workplace?”, for
industrial workers to consider at their own working environment. The
users found the visibility of such points on the ground plan to be
quite weak and they also thought the questions were not separated
clearly enough from the rest of content of the function point. Also
the terms in the main menu were considered to be quite ambiguous, and
it was not clear to the users what each section contains.
3.2.5
Heuristic evaluation
Heuristic evaluation and user testing should be
alternated, because these two usability practices have been shown to
find fairly distinct sets of usability problems (Nielsen 1993).
Heuristic evaluation is done by studying a user interface and then
forming estimation about what is good and bad about it. The evaluation
is conducted according to a certain set of rules, such as those listed
in typical guidelines documents (Nielsen 1993). In Virtu heuristic
evaluation, the list of
Nielsen’s ten usability heuristics (Nielsen 1994) was used.
Nielsen’s ten usability heuristics include:
§
elements in user interface (4 heuristics),
§
language (2),
§
interaction (2),
§
error prevention (1) and
§
helping information (1).
User should have appropriate feedback in time and
the system status should be clearly seen. All the error descriptions
and UI terminology should use users language and terminology.
Consistency can be maintained for example by using standards and UI
guidelines. Operation is made easy and efficient by providing
accelerators, shortcuts, exits and easy escapes. The interface should
contain all the essential elements but nothing more. (Faulkner 2000)
Heuristic evaluation was conducted for each display
and for Virtu as a whole after user tests with the software prototype
in order to search for usability problems undiscovered by user
testing. Usually the heuristic evaluation is carried out before user
tests, but in this iterative process it was reasonable to use the
usability professional to verify the design decisions before release
version of Virtu.
Results indicated that some terminology like
ground plan and enterprise description had to be changed
into more user oriented vocabulary like visit and table of
contents. Easy exits had to be added in the theory section, and
the cursor had to be changed in appearance, when placed over buttons
to show to the user the possibility of interaction. Some minor notes
were reported about consistency among different parts of Virtu.
The structure of Virtu was split into two entities
because of the requirements (Figure 1). Some educational material
concerning safety at work already existed, and it needed to be
presented in electronic form in order to utilize the interaction
possibilities of VLE. It was also
necessary for both user groups to see some real life examples of
safety at work. The practical part should also support the adaptation
of theory.
The two parts of Virtu are:
§
textbook section,
§
virtual enterprise section.
The textbook section was divided to theory and
exercises. The virtual enterprise section contains interactive
function points and a description of the enterprise providing
background information on the virtual enterprise. The material
provided in function points was gathered from three real life
companies.
The textbook section can be studied as a guided
tour including theory and exercises on every topic. Each theory topic
can also be studied independently from the others and the textbook
section can thus be used as a quick reference book. The use of
textbook section begins by selecting one of 8 topics covering
different areas of safety at work.
The content of the textbook was divided to two
difficulty levels based on usability test results. Theoretical
level contains information on different safety at work topics
including legislation, occupational safety management, risk
management, ergonomics, occupational hygiene, machine and system
safety, mental welfare and occupational accidents. This level is to be
used in industry by the people responsible for safety at work issues
and by the students, because they need a deeper understanding of the
issue.
Practical level is to
be used by industrial workers. Thus, the topics included differ
slightly from the content of the theoretical level. The practical
level contains information on risk assessment, ergonomics,
occupational hygiene, occupational accidents, signs and traffic.
Included information can be easily applied into practice in the user’s
own work environment. Even the topics, which are same or similar to
ones in the theoretical level, were re-designed to match the
requirements of the group using this level.
Theory displays were designed using a book metaphor
(Figure 2). Each view reminds opened book, which consists of two
pages. The user’s location in the Virtu is indicated on the top and
there is always possibility to exit the current theory section. User
is informed of the total number of pages included in the section.
