Higher education
< Tilbage Home Higher education Cross-sectoral evaluation

Cross-sectoral evaluation

Knowledge and development for further and higher education.

This text origins in a folder published in 2012

Contents

  • Foreword 
  • What is evaluation? 
  • Various types of evaluation
    • Thematic evaluations 
    • Comparative evaluations of selected programmes 
    • Evaluations that focus on transitions 
    • Evaluations that focus on strengthening capacity 
    • The focus and design of evaluations 
    • Staff commitment at institutes of education
    • Evaluation framework 

 

Foreword

Knowledge and education lead to growth and greater welfare. There is broad political agreement on this, and there is therefore sharp focus on ensuring and developing the quality of our further and higher education. A well-functioning education sector that offers programmes of the highest quality is a precondition for Denmark’s ability to compete – and compete successfully – in a globalised world.

External quality assurance systems, designed to determine whether the quality of education lives up to expectations and to help institutes of education in their efforts to improve the quality of their programmes, have been established in Denmark and the rest of Europe. Accreditation has constituted the core of the Danish quality-assurance system from 2007 to the present.

From the accreditation of programmes to the accreditation of institutes

The Danish Minister for Science, Innovation and Higher Education introduced a proposal for the external quality assurance of further and higher education in the future. The proposal involves phasing out the existing system of programme accreditation and replacing it with a new system for accrediting institutes of education. The underlying idea of the new system is that when a university, an academy of professional higher education or a university college has been accredited as an institute, it will be possible for staff to develop and broaden programmes without having to be accredited again.

In parallel with a new, simplified accreditation system, it has been suggested at the Danish Ministry of Science, Innovation and Higher Education that evaluations will in future be accredited on an interdisciplinary basis to include all further and higher education programmes. Two of the principle elements in the future quality assurance system will be: accreditation, to ensure that all institutes live up to a number of minimum standards for quality, and evaluation that focuses on how institute staff can develop their practice within selected areas, themes and research questions.

The “old” and the “new” evaluations

There is a long-standing tradition for evaluating further and higher education in Denmark. University programmes in particular were evaluated in clusters of related programmes, which involved evaluating each individual institute in what were known as rota evaluations from 1992 and onwards.

Evaluations will be of a completely different character in the new system. This will not be a question of new rota evaluations, which simply replace the accreditation of individual programmes. It will involve evaluations that focus on the challenges that face the sector and institutes of education in general and are designed to identify possible solutions and opportunities for action across institutes by analysing the impact of various interventions in various environments and contexts, for instance. EVA has a long-standing tradition for performing evaluations of this type in connection with programmes of all kinds.

It is important that evaluations are adapted to the needs, background and wishes at universities, academies of professional higher education and university colleges, etc., in order to ensure that they provide the highest possible value and have the most effective impact. Hence, it may be a great advantage if institute staff, students and other stakeholders play an active role in selection of the themes and research questions that are to be evaluated.

Cross-sectoral  evaluations are a permanent element in external quality assurance systems in several European countries, such as Finland, where thematic evaluations are performed in parallel with audits at institutes of further and higher education every year. Audits at institute level, supplemented by national studies and "research projects" designed to help institute staff focus on current themes and challenges, are also performed in England.

The plans to include evaluations as integral aspects of the Danish quality assurance concept mean that there is a need to define what evaluations across institutes and programmes can contribute, and how they can be organised to provide the greatest possible benefit and impact.

This folder contains descriptions of various types of evaluation and illustrates how evaluations can help with the work of developing the quality of programmes at institutes of education.

We hope you enjoy reading it!

The Danish Evaluation Institute

2012

 

What is evaluation?

The concept of evaluation is used in a number of contexts. In some contexts, it is used in a broad sense that covers a wide range of evaluation processes, but in this folder, evaluation should be understood in a more limited sense. Read more on the opposite page.

Evaluation is often used as an inclusive term to describe systematic evaluation processes of all kinds, e.g. accreditation, auditing, certification, etc. This is the case with the Swedish evaluation researcher Evert Vedung, for instance, who has formulated the classic and most frequently-used definition of evaluation:

"Evaluation is a systematic, retrospective assessment of the implementation, performance and outcome of public policy that is intended to play a role in practical decision-making." (Evert Vedung: Utvärdering i politik och forvaltning, 1998).

However, evaluation is also used as a more limited designation for open assessment processes, primarily in connection with learning and development. We use the concept in the latter sense in this folder, where evaluations differ from the accreditations, for instance, that have been a basic element in the external quality assurance of further and higher education since 2007.

It could be said in general that, with regard to method, accreditation draws on a European quality assurance tradition with a design that always comprises four elements: accounts from institutes, visits to institutes, panels of experts and reports that include assessments. The focus is on individual programmes or institutes, and the accreditation process involves collecting documentation and assessing whether the programme or the institute lives up to fixed minimum criteria for quality. In this way, accreditation can give rise to local learning and development.

Evaluations, on the other hand, typically have their roots in a societal tradition in which their methodical design is established on the basis of their purpose and draws on a wide range of methods derived from social science – qualitative as well as quantitative. The focus is on discovering and assessing patterns of practice, the causal explanations that underlie various problem fields and the impact of new interventions – collectively for the sector. Evaluations can therefore contribute new knowledge, conclusions, perspectives and progressive recommendations that will typically benefit a larger group: the further education sector as a whole, certain types of programme, certain functions connected with programmes (teaching, management, counselling, etc.), the public authorities and legislators, etc.

Interdisciplinary challenges need cross-sectoral evaluations

In a new quality assurance system, evaluations carried out across further and higher education programmes must interact with the coming accreditations at institute level. Many of the research questions, challenges and developmental tendencies inherent in these programmes are of an interdisciplinary character and as such are not connected with the individual programmes or institutes. This makes it necessary to take an interdisciplinary approach in order to understand the underlying dynamic of the drop-out problem, for instance, or to obtain a national view of the work carried out at institutes of education with the aim of ensuring a high completion rate.

Evaluations can give rise to reflection and goal-directed knowledge thanks to their development-oriented aim and flexibility regarding methods and focal points. Evaluations can go into detail in selected areas and can therefore provide – when they are most successful – a common understanding and common aims for the development of the sector.

Various types of evaluation

Each of the various types of evaluation can in its own way contribute new knowledge and lead to development across the further and higher education sector. Read more about the various types of evaluation on the following pages.

Thematic evaluations

Cross-sectoral  evaluations are more flexible tools than the criteria-based accreditations that focus on a number of predetermined themes. Evaluations can be designed to match the purpose formulated at the beginning of each project. At the same time, relevant research can be drawn on so that analyses of and recommendations for the development of practice can be based on the latest knowledge. Evaluations are therefore a valuable and necessary supplement to accreditations.

This section contains a presentation of four different types of cross-sectoral evaluation that can contribute new knowledge and lead to
development and have a broader scope than the fixed focus areas of accreditation.

Thematic evaluations can help to direct focus towards current research questions that create challenges regarding quality in a wide range of programmes and institutes across the education sector. This applies to issues connected with recruitment and enrolling students, the development of teaching methods, ensuring the qualifications of teachers and the quality of the professional environment, counselling, retaining students, internationalisation, renewing test and examination methods, modularisation and much more.

Thematic evaluations are also suitable for illustrating questions regarding impact and causality, and can therefore be used to answer questions such as:

  • Which interventions designed to remedy dropping out have the greatest impact?
  • What is the significance of differentiated teaching methods for various types of students?
  • How do study trips abroad influence students' subsequent employment prospects?

EXAMPLES

In recent years, EVA has performed thematic evaluations of knowledge centres at university colleges and academies of professional higher education, of the quality of English-language teaching at universities, academies of professional higher education and university colleges, of the reasons for students dropping out and of interventions designed to retain students at university colleges and academies of professional higher education.

Comparative evaluations of selected programmes

It will be valuable in connection with a system primarily based on institute accreditation to be able to focus on a programme or a group of related programmes with the help of a development-oriented evaluation. The occasion could be a marked change in the requirements of the labour market, for example, a difference in the recruitment base compared to formerly, or a change in the international professional field. The individual institutes of education are all faced with the same challenges, and there could be considerable learning potential in comparing and assessing practice and results. Similarities and dissimilarities, such as the prioritisation of subject elements, the choice of learning and teaching methods and the benefit students derive from teaching, will be clearly in evidence when evaluating similar programmes. Cross-sectoral evaluations of programmes therefore make it possible to exchange experience and obtain inspiration in selected focus areas, without the rigidity connected with the fixed criteria of accreditation. In other words, it will be possible to organise an evaluation in a flexible manner so that it can target the given wishes and needs for development in the individual case. One very promising possibility is to draw on international experience so that there can be an exchange of knowledge between Danish and foreign programmes.

EXAMPLES

In recent years, EVA has performed comparative evaluations and analyses of several programmes, including a comparative study of Nordic teacher training, of Danish and English chemistry programmes and of international benchmarking of the Royal Danish Academy of Fine Arts’ School of Architecture, etc.

Evaluations that focus on transitions

For many years, there has been focus on the creation of flexible transitions and good opportunities for further training in the Danish education system. The government's objective of ensuring cohesion between further and higher education programmes has renewed the focus on this issue. The problems of transition are in the nature of the case outside the primary area of responsibility of the individual institute, so institute accreditation will not shed any light on the matter.

Cross-sectoral evaluations will make it possible to focus on relevant transition problems, on concrete opportunities and barriers for those who have completed a course of education, for instance, and who want further training to move from one level of qualification to the next, on enrolment procedures and systems, or on student turnover in the education system.

EXAMPLES

In recent years, EVA has performed evaluations of credit transfer practice at institutes of vocational education, of the objectives of the upper secondary school reform that include strengthening students' real competence to study and making them capable of completing a course of further or higher education, and of supplementary training in connection with the graduate programme in health science, (MSc health science).

Evaluations that focus on strengthening capacity

In addition to contributing to the development of further and higher education, evaluations are often used to assess quality and create transparency, but they can also have the aim of strengthening capacity. In this case, they will typically focus on supporting the ability of institute staff to handle certain challenges and will therefore essentially have a purely formative or developmental aim. In other words, their primary purpose will not be to provide an insight into the underlying mechanisms or strengths and weaknesses of current practice at institutes of education, but rather to build up or develop the competence of institute staff to handle challenges in certain areas. In this way, evaluations form the framework around study and reflection processes during which the people involved will find solutions by sparring with and obtaining assistance from an evaluator.

EXAMPLES

Examples of this could be evaluations that support and facilitate the work of developing and describing learning and competence goals, developing effective student retention strategies, or achieving the best possible interaction between accreditation and internal quality assurance. EVA holds evaluations courses on an ongoing basis to provide institute staff with knowledge, feedback and reflection skills for the work on current research questions in connection with evaluation.

How can evaluations be oriented towards development and application?

Good evaluations provide learning and lead to development. A number of circumstances connected with evaluations determine whether they are considered applicable and useful. Read about them below.

The focus and design of evaluations

Evaluations can be used to assess whether a given activity lives up to legislation, quality standards, intentions and expectations, etc. But they only become of real interest when they contribute learning that leads to genuine development.

Experience from Denmark and the other Nordic countries indicates that evaluations can perfectly well fulfil a need to assess whether a given measure lives up to expectations, at the same time as their chief purpose is to contribute to learning and development.

However, certain demands must be made on the way evaluations are designed in order to ensure that they really will contribute to learning and development – and thereby be experienced as valuable by institute staff.

It is important for evaluations to have a clear developmental focus that comes to expression in their design:

  • Focus on best practices: Interdisciplinary evaluations can explicitly focus on identifying and describing examples of best practices, such as well-structured procedures in connection with credit assessments, or particularly successful ways of connecting teaching and the labour market. In this case, institute staff will be able to find inspiration for ways in which they can develop their own methods of handling challenges.
  • Focus on development opportunities: Evaluations can reveal any obstacles that institute staff encounter in their work on a given research question, such as research-based or development-based approaches to programmes, and the opportunities institute staff have for fulfilling their own and external objectives in the area to a greater extent. In this way, evaluations can also contribute to development by shedding light on options for action at institutes, for the public authorities and possibly for other parties.
  • Focus on development tools: Evaluations can have the aim of developing tools that can support institute staff with their development processes. They could be self-auditing tools, experience and inspiration folders, or questionnaire guides and tables for internal use.

Staff commitment at institutes of education

Staff commitment at institutes of education is absolutely decisive for the realisation of development potential. It is important for staff to feel that they are becoming more knowledgeable if they are to play an active and committed role in an evaluation.

  • Incorporated reflection: Reflection and learning opportunities can be incorporated to advantage during the course of an evaluation in order to strengthen staff commitment. This could take the form of a self-evaluation, for instance, where institute staff evaluate the strengths and weaknesses of their existing practice and point out development opportunities. Staff at many institutes will do this, irrespective of how the evaluation is organised, but the evaluator can organise the process in such a way as to promote reflection and learning.
  • Active following up: The evaluator can offer various kinds of process and development courses during and after the evaluation itself. This will make it possible for institute staff to convert and anchor the knowledge and learning that results from the evaluation in their own organisation.

Evaluation framework

The framework in which evaluations are performed may be very important in determining whether institute staff feel that evaluations are development-oriented and application-oriented:

  • Institute staff take part in selection: Evaluations must take up issues that are relevant and current if staff are to find it meaningful to work with them. It will therefore be a good idea for institute staff to indicate which themes and research questions coming evaluations should focus on, and to play a central role in selecting them. This could either be included as an integral part of the evaluation process, or be an offer to institute staff who this is of particular interest to.

 

Interdisciplinary evaluation

© 2012 The Danish Evaluation Institute

The Danish Evaluation Institute

Østbanegade 55, 3.

2100 Copenhagen Ø

Tel.: (+45) 35 55 01 01

Fax: (+45) 35 55 10 11

E-mail: eva@eva.dk

Contact
Head of Division
D +45 3050 2443
E 
Jakob Rathlev
EVA is an ENQA Affiliate

ENQA logo 2013

Recognition of international qualifications

Read about assessment of qualifications and about entry to Danish higher education at the Danish Agency for Universities and Internationalisation.

Accreditation of educational programmes

The Danish Accreditation Institution accredits study programmes within higher education.

The Accreditation Council makes the final decision on accreditation of all higher education in Denmark.