1.
Introduction
This paper
aims at providing an approach for facilitating the design process of
mainly large-scale e-learning acts with the purpose of serving the
requirements and needs of organizations with geographically
distributed employees and short learning curves, for enhanced learning
delivery based on the competencies that their employees posses as well
as on the anticipated training level they need to obtain. Customized
and track learning on individual participant basis, and flexibility
greater than the one offered in traditional Learning is one of the
major goals of our research.
We present a framework for
supporting large scale e-learning initiatives in big distributed and
highly demanding organisations by viewing
the issue from two different perspectives:
§
the “process”
perspective: the initiative set-up and execution i.e. the way of
working towards an e-learning initiative lifecycle (goal setting,
requirements elicitation, analysis and design, implementation,
feedback and so on)
§
and the
“product”: the learning delivery to the final-users i.e. the final
outcome of the process and its quality (personalized learning, quality
of learning etc.)
The proposed framework will
entail from the process viewpoint
§
a set of well
documented conceptual tools for the analysis of the professional
knowledge and skills and of the production of professional profiles in
the working environment for identified specialties.
§
A consistent
methodology for the evaluation and the professional improvement
through educational processes, on the basis of the knowledge and
skills of each employee, as well as
§
A way of working
for applying the proposed conceptual tools in real life projects,
while form the product
viewpoint it tackles issues of personalised learning improved in
quality.
Personalised learning will be
achieved by enabling efficient evaluation of the employees
using “traditional” evaluation techniques (on-line evaluation) as well
as knowledge management techniques (tracing of learning behaviour
patterns and informal knowledge) [Leo
1997], [Jimes
2003] This evaluation leads to the definition of workers’ basic
deficiencies, as regards skills and knowledge (gap analysis)
in relation to a required professional profile for a given
job-position. The coverage of the detected deviations will be
supported by dynamic recommendations for the subsequent
learning process progress [Petropoulos,
Xini 2003]. Domain specific and language independent taxonomies
are defined and employed in order to map the training subject to the
required job profiles that need to be obtained through the training
sessions as well as for the appropriate recommendations, which will be
dynamically directed to the learner.
The rest of the paper is
organised as follows:
Section 3 presents the
background of this research and the motivation we had for tackling
competency based e-learning delivery.
In Section 4 we describe
the learner centric approach for competency based learning delivery by
defining the coresponding processes
Section 5 deals with the appropriate evaluation
methodological framework for the
evaluation of the knowledge and skills of the employees, adapted to
the frame of individual companies’ requirements.
Then
section 6 goes in detail describing and sketching a conceptual
architecture for the “product” perspective
Finally section 7 presents the
conclusions drawn from the work presented and proposes topics and
steps for remaining and future work.
2.
Background
The
motivation for the work described in this paper and the methodology
that was developed came as a straightforward requirement from research
and development projects in different market sectors that our company
is involved in. The approach followed by many of our customers, medium
or larger organizations, with respect to e-learning services targeted
to their personnel is rather technology oriented, focused primarily on
providing the necessary technology platform and ensuring participation
of their employees - trainees. They rely heavily on the functionality
and evaluation characteristics that proprietary solutions offer. This
in turn yields a misconception of the derived training results as well
as a lack of methodological tools in order to measure efficiency
gains, cost savings, individual professional improvement, groups’
performance enhancement, and organization advancement [Davenport,
Prusak 1998].
Retroactive analysis of the different cases led us to establish a set
of criteria for measuring the level of achievement of the objectives
of a total competency based learning delivery solution from the
perspective of personalized delivery of knowledge, human resources
continuous evaluation, evolution tracking and improvement. This
analysis was mainly based on the study of the deficiencies as well as
the strong points of a variety of project cases.
The
criteria set were:
§
Personalisation: If
the systems offer services of personalized learning, access to
knowledge and the degree that the system offers these services. We
also examine whether the system is in position to construct, maintain
and update a personal learning profile in a formal way, which is going
to be consulted every time a knowledge intensive task is performed.
§
Evaluation: Whether the system offers
explicit or implicit evaluation mechanisms (on line tests, quizzes
versus tracking of behavior, intelligent mechanisms).
§
Evolution tracking: With this criterion we
examine whether the solution under consideration captures and measures
the progress of the individual learner within the learning environment
using a formal way.
§
Human
Resources (HR) support: This criterion is
highly interconnected with the previous one, and it is defined for
examining whether the e-learning solution is in position to provide
valuable and exploitable information concerning the trainees (e.g.
evaluation) as well as useful directions for HR development
(identified gaps of the staff, suggestions for knowledge
provisions-training, knowledge assets etc.)
We propose a learner centric
environment as schematically represented in Figure 1 below.:
In order to achieve dynamic
evaluation of the employ’s learning paths according to the skills
he/she possesses and the required ones that are the actual target of
the training, a learner centric approach is proposed comprising as
building blocks four main elements:
1.
The existing profile of the learner:
It describes the employees’ skills and current employment tasks
2.
The Required Profile:
It describes formally the skills that should be developed, enhanced or
updated according to the training plan
3.
Explicit Evaluation Results:
These are the actual results that the employee achieves through normal
evaluation of tests and learning exercises
4.
Learning Behaviour Patterns:
These are the patterns that the employee generates in the learning
process involved.
These four
elements represent the corresponding conceptual entities that are
captured and then processed in the proposed environment.
Figure
1:
Learner centric approach
3.1
Process definition
Inititally, a sampling research is suggested with structured
questionnaires and processing of the filled questionnaires for example
through the use of multi-criteria analysis, in order to define the
professional profiles [Siskos
et.al. 2000].
The
sampling research (apart from the general nature data) shall focus
into two basic sectors:
§
To the definition
of general professional profiles, and
§
To the evaluation
and modeling of the human force on the basis of objective criteria
Collected
data constitute the initial input and thet are then processed using
the methodological approach of multi-criteria analysis for the
assesment of criteria and indicators [Mendoza
et.al. 1999]. This analysis supports the definition of the
satisfaction indicators of the employees regarding their professional
role, as well as their objective evaluation on the part of their
companies.
With the
aid of these indicators, the degree (grade) of overall satisfaction,
and the satisfaction indicator of the individual factors are
evaluated.
Given that
the most important criteria for the placement and choice of personnel
for the employers today are the “competence of knowledge”, and the
“professional experience”, these elements should be verified, along
with the degree of response to the market needs.
The
collected and analysed data should be modeled to form the conceptual
basis for susequent use. Several approaches may be employed for the
modelling task such as Enterprise Knowledge Development tools [Loucopoulos
et.al. 1997].
Among the
general professional profiles that are generated, some will be
selected and adapted to the needs of the company in order to define
the degree of compatibility with the requirements of the market. These
profiles are categorised based on the specialties that the learning
initiative aims to train. In this way two different models for the
selected categories of profiles are generated, consisting of
characteristics, knowledge and skills for each selected professional
specialty. These, we conventionally call «ideal professional profile»
and «required professional profile» in contrast to the real
professional profile of each employee and are both compliant with IMS
Reusable Definition of Competency or Educational Objective RDCEO
information model [IMS-RCDEO(a)2002]
The
evaluation cycle starts on the basis of the detection of knowledge and
skills of each employee (definition of the real professional profile)
in relation to the required profile of its category. This evaluation
constitutes the basic pre-evaluation for the classification of the
employee in some professional profile. The procedure of the evaluation
is implemented and ends up whether in the detection and certification
of a gap between the required and real professional profile, or in the
interruption of the procedure. The interuption occurs either in the
case that the employee is considered insufficient in relation to the
characteristics of his role and must be placed in another position, or
in case the evaluation results show that the employee is
under-employed in relation to his qualifications and possibly he must
be granted a higher level of work.
The
definition and analysis of the gap should be implemented continuously
through electronic applications for evaluation, specialized data
(positions requirements – position owners), processing of the data,
informal knowledge tracing mechanisms (learners behaviour patterns,
taxonomies, reasoning), classification of the data, grouping of the
personnel according to the differences, representation of the results
of processing and other similar techniques so as to produce utilizable
and valid results.
The
management of the differences that may occur, being positive or
negative, will be conducted on the basis of the required professional
role or the future professional improvement of the employee through
the provision of training programmes. This training will aim to
provide to the learner supplementary knowledge and skills in order to
approach the required professional profile tailored for the respective
job position.
The whole
procedure should be manageable by a sufficient toolset, that does not
only evaluate the present knowledge level of the employees, but also
use knowledge management techniques in order to chose the most
appropriate among the available training sources for the
implementation of the required professional profile.
In this
way a dynamic planning of an ad-hoc «personalized» program for the
development of the human resources is generated. In turn, this leads
to overall management of knowledge and skills, at the organisation
level as well as at the level of the knowledge background of the
employees, resulting to the faster and more effective coverage of the
existing working positions, the rapid management of human resources
and the more effective management of the professional evolution of the
employees.
The
overall procedure «closes» with the final evaluation, that will lead
whether to the planning of future required professional
characteristics of the employee, in case of success, or to the
re-introduction of the employee in the process in case that the
deviation between required and real professional profile persists.
In the
frame of the previous analysis of the evaluation system, the following
are involved accumulatively:
§
Management of the
data related to working position (role, tasks, actions,
communications, demands, etc.).
§
Management of the
CV data and profiling data of the personnel (education, training,
experience, skills) [EURES
2003].
§
Definition of the
gap between the present situation and the required one (as regards
knowledge, experience and skills of human resources).
§
Monitoring of the
evolution / development of the human resources.
At the
same time, the evaluation focuses also to the dynamic and effective
management of parameters like:
§
Classification of
the training into categories and levels
§
Training
resources availability.
§
Creation of
training improvement programs by combining existing training units.
§
Indicators of
personal and professional satisfaction of employees and companies.
Having in
mind the abovementioned data and in order to be able to implement the
IT tools that can support the overall approach, the proposed actions
are the following:
§
Analysis of
knowledge and skills according to the professional profiles selected
§
Classification of
knowledge in distinct parts of the knowledge subject
§
Connection of
skills with particular measurable results that the employee must
achieve whether during the training or during work
§
Creation of data
collections with categories of questions, exercises, case studies and
papers that will be presented or assigned to the employee according to
the evolution of the training, during training
§
Evaluation of the
answers to the questions– exercises or the results achieved in case
studies and work assignments
§
Focusing on
several questions categories, exercises categories, case studies and
work assignments.
§
Tracing and
analysis of informal knowledge, i.e. knowledge not declared in the
employees’ formal CV.
The
deducing mechanism of the proposed evaluation information tools will
have input data derived from the training process of the employee, as
well as data derived from his professional evolution and working and
learning behaviour. Thus with the proper combinations it is feasible
to define the level of knowledge and skills of the employee as
compared to the required professional profile.
4.1
Personnel’ skills
The
proposed approach uses a consistent methodological framework for the
evaluation of the knowledge and skills of the employees-learners,
adapted to the frame of individual companies’ requirements. The
framework provides the mechanisms enabling evaluation of the employees
knowledge, and individalised paths for upgrading the knowledge and
skills of the employees, according to the defined requirements of
their professional role.
4.2
Gap analysis
An
integral part of the methodology is based on the evaluation of the gap
between the requirements of the role of the working positions and the
competencies (knowledge, training, experience, skills) in workers
occupying these working positions. This Gap Analysis leads to the
following:
§
Estimation of the
degree of knowledge, experience and skills (competency analysis) of
the workers in relation to the requirements of their role - job
description and their objectives for personal evolution (within the
working settings)
§
Definition of the
required education / training
§
Monitoring of the
learning process and the “behaviour” of the learner
4.3
Evaluation cycle
This
information provides the specified approach of a dynamic adaptation of
the employees to the requirements of their professional role and
consequently of the more general management of learners’ knowledge.
The
described procedure is recursive, aiming, on the one hand, to the
better utilization of the human resources on the part of the company
and, on the other, to the global satisfaction of the employees
purposes regarding their continuous professional improvement.
Finally,
the total evaluation is conducted on the basis of three distinct but
fully supplementary levels, i.e.: the employability of the employee,
the satisfaction of the employee’s goals in connection with his
professional evolution derived from the attendance of the training
program, as well as from the total evaluation of the employee and his
professional evolution by the employer company.
Evaluation
is divided in two categories:
Explicit Evaluation:
These are the actual results that the employee achieves through normal
evaluation of tests and learning exercises (LMS Evaluation
tests, exercises, formal trainees’ evaluation results etc.)
Implicit Evaluation
based on the informal knowledge traced from his learning
behavior and on-the job processes that he/she is carrying out,
strictly related to the training subject (Learning
Behaviour Patterns).
Diagrammatically, the frame of Estimation and Evaluation of the
Employees Knowledge and Skills is presented in Figure 2.
Figure
2:
Employees Knowledge and Skills
Evaluation Cycle
5.
Conceptual architecture
The product perspective is
assessed with a proposed conceptual architecture depicted in Figure 3. The conceptual architecture is an attempt to
provide high-level view of an integrated solution that performs the
required processes of the evaluation framework. It presents the main
functional components that can achieve the competency based learning
delivery to the end-users, the learners.
Figure
3:
Conceptual architecture
Our approach considers a main
precondition
§
There are
Proprietary KM Environments (PKME) in the target organisation, which
usually play specific role in a training process. These can consist of
Learning Management Systems, KM Supporting tools, Intranets,
Organisations’ file servers, Communication and Collaboration tools, a
combination of them and several other applications that the organisation
may have in its possession. The data generated by these PKMEs may play
significant role in the implicit evaluation of the learners’ training
process and usually are strictly connected with their tacit knowledge.
5.1.1
Sources adaptors
In order to trace the learners’
behaviour during their training process, our approach considers all
potential internal and external sources that the learner uses as the
depicted Proprietary KM Environments (PKME).
The PKMEs of an organisation used
during a training process are generating data according to the learners’
usage behaviour and their training patterns. Those data may well
describe the tacit knowledge of learners generated through the training
process and are considered as preliminary knowledge entities that need
further processing in order to add value in the evaluation methodology.
The role of sources adaptors is
to provide the appropriate interfaces for Knowledge Feeder in the
process of collecting specific data related to learners training process
strictly connected with their informal knowledge.
We differentiate between two
types of sources with respect to the adaptors:
§
proprietary
applications/tools or content sources that comply with standards as for
example a Learning Management System compliant with LIP standard that
the organisation may possess [IMS-LIP
(2001)].
§
Applications or
tools that may not be standards compliant, usually applications and
tools developed in house or that serve individual and/or specific
organisation’s needs.
In the first case adaptors are
set in order to collect data elements from PKMEs in a certain structure
while in the second, adaptors have to be built in order to provide the
relevant data in a specified standard compliant format.
5.1.2
Knowledge bases
These form an integral part of
the proposed environment and are consisting of LMS repositories, Human
Resources repositories, learning content repositories, competency/skills
and other relevant RDCEO [IMS-RDCEO(a)]
compliant repositories.
Moreover they are including
registries that are used by discrete components in the form of native
XML databases such as a registry for the publishing and description of
Knowledge Feeder Web Services and the RDF registry for the Domain
Specific Taxonomies.
5.1.3
Knowledge Feeder
The role of Knowledge Feeder is
triple. It collects existing evaluation data, it collects learners’ data
from the knowledge bases and the most crucial it collects usage data
from the PKMEs through the specific adaptors.
Knowledge feeder may consist of a
set of Web Services for these three different categories and implements
their choreography in order to provide the appropriate output to the
Evaluation Engine.
It is the main component that
interacts with the interface and interacts also bi-directionally with
the Evaluation Engine and Recommender components.
5.1.4
Evaluation Engine
The Evaluation Engine forms the
core engine of the environment. Its role is to provide implicit and
explicit evaluation results according to the proposed evaluation
methodology in a way that learning gaps and deviations from learning
paths are identified in an adaptive and progressive manner. The
evaluation engine implements the core algorithms for the evaluation
process:
(i)
It receives the
test results, exercises and training performance evidence from the
Knowledge Feeder and performs the explicit evaluation
(ii)
It receives from
the Knowledge Feeder data representing learner’s behaviour and performs
light reasoning with respect to the relevance/irrelevance of learner’s
behaviour patterns to the training subject. This is achieved by
identification of Subject categories in these patterns and their mapping
to predefined domain specific taxonomies. It proceeds with distance
calculations and uses the outcomes for performing the learner’s implicit
evaluation.
(iii)
It combines
Implicit and Explicit Evaluation results with retrieved content from
ad-hoc or existing competency/skills repositories and correlates with
the required profile and/or the training subject goals.
(iv)
It implements and
performs the required Gap Analysis
The Evaluation Engine interfaces
with Knowledge Feeder in order to get the appropriate input and also for
calling specific Web Services needed during the Evaluation procedure
It provides as output to the
Recommender component the Evaluation and Gap Analysis results.
5.1.5
Recommender
The Recommender component has the
role to identify potential corrective activities that the learner should
take in order to fulfil the training process goals.
It receives input from the
Evaluation Engine and interfaces with the Knowledge Feeder for
retrieving learning objects
[ARIADNE 2003] and/or Internal or External Sources that have to be
provided to the learner as a consequence of the Evaluation and Gap
Analysis results. The identified internal and external sources are
passed to the interface forming a close feedback loop for adaptive and
progressive learning process assessment.
5.1.6
System Interface
The interface implements the
learners’ interface to the whole environment as well as it implements
the service request client for the Knowledge Feeder Web Services
6.
Conclusions and future work
In this
paper we attempted to present a framework for supporting e-learning
initiatives by viewing the issue from two different perspectives i.e.
the “process”: the project set up and execution and the “product”: the
learning delivery to the final-users. From the process viewpoint we
suggested a way of working towards setting up an e-learning solution
together with a set of conceptual tools in the direction of enhancement
of learners profiling and feedback mechanisms in a recursive process.
From the “product” viewpoint we proposed a conceptual architecture that
takes advantage of informal knowledge tracing mechanisms, achieves
enhanced evaluation and provides personal learning paths. While we
attempt to provide a complete and integrated approach during our
research several research issues have been raised which constitute the
basis of further study, analysis and research. Such issues which are
still open for research are: the approach for modeling professional
profiles i.e. What makes a knowledge modeling approach appropriate for
modeling such initiatives?; the mechanisms of tracing informal knowledge
i.e. Is capturing of tacit knowledge feasible? How can we approach such
a necessity?
We
intend to apply the proposed approach
in a series of ongoing related projects. These case
studies will provide the evidence needed to evaluate the methodology per
se and identify the potential for further improvement and advancements.
Comparative results from different application domains may also provide
insights for peculiarities and characteristics that exist in the
individual learner’s level and are totally hidden or difficult to
extract when assessing a solution for a specific industry sector.
References
-
ARIADNE, (2003) ARIADNE Foundation
for the European Knowledge Pool, [online],
http://www.ariadne-eu.org/en/system/index.html
-
T.H.
Davenport, L. Prusak, (1998) “Working Knowledge: How Organisations
Manage What They Know”,
Harvard Business School Press,
Boston MA
-
De Leo F., (1997),
“The transfer of tacit knowledge within firm boundaries – An
explaratory study”, in Corporate Transformation Kluwer Academic
Publishers,
Netherlands, pp. 448-459.
-
EURES (2003) “The European CV”
[online]
http://eurescv-search.com/
-
IMS-RDCEO(a) Global Learning
Consortium (2002), “IMS Reusable Definition of Competency or
Educational Objective – Information Model [online]
http://www.imsglobal.org/
-
IMS-RDCEO(b) Global Learning
Consortium (2002), “IMS Reusable Definition of Competency or
Educational Objective – Best Practice and Implementation Guide
[online]
http://www.imsglobal.org/
-
IMS-LIP Global Learning Consortium
(2001), “Primer for the IMS Learner Information Package” [online]
http://www.imsglobal.org/
-
C.Jimes, L.
Lucardie (2003), “Reconsidering the tacit-explicit distinction – A
move toward functional (tacit) knowledge management”, Electronic
Journal of Knowledge Management, Vol. 1, Issue 1, (2003) 23-32.
-
P. Loucopoulos, V.
Kavakli, N. Prekas, C. Rolland, G. Grosz, S. Nurcan, (1997) “Using
The EKD Approach: The Modelling Component” Technical Report
WP2/T2.1/UMIST/1, Version 2.01 ELEKTRA P, Dept. of Computation, UMIST,
UK
-
Mendoza, G.A.
and Macoun, P. with Prabhu, R., Sukadri, D., Purnomo, H. and Hartanto,
H., (1999) “Guidelines for applying multi-criteria analysis to the
assessment of criteria and indicators”. The Criteria & Indicators
Toolbox Series No. 9. CIFOR, EU, GTZ and USAID. 84p. ISBN
979-8764-27-7
-
Nonaka I., Takeuchi H., (1995)
The Knowledge Creating Company, Oxford University Press, Oxford.
-
K. Petropoulos, H.
Brillakis, (2002) “Evaluation of High Impact Solutions based on KM
Life Cycle Models”, Challenges and Achievements in eBusiness and eWork
– e2002, B.Stanford – Smith et al. (Eds.), IOS Press 2002 part 2, pp.
789 – 796
-
K. Petropoulos, G.
Xini (2003) “Assessing and Supporting Workers Evolution within a
Consistent Knowledge Management Methodological Framework”, 4th
European Conference on Knowledge Management, 18-19 September 2003,
Oxford
-
Y. Siskos, Y.
Politis and G. Kazantzi (2000) "Multicriteria methodology for the
Evaluation of higher education systems, the case of an engineering
department," EURO 2000,
Budapest, Hungary.
|
