FREE ELECTRONIC LIBRARY - Thesis, documentation, books

Pages:   || 2 | 3 |

«The Development and Initial Validation of Competencies and Descriptors for Canadian Evaluation Practice Brigitte Maicher Net Results and Associates ...»

-- [ Page 1 ] --

The Development and Initial Validation

of Competencies and Descriptors for

Canadian Evaluation Practice

Brigitte Maicher

Net Results and Associates

Fredericton, New Brunswick

Christine Frank

Christine Frank and Associates

Barrie, Ontario

Abstract: This article presents the Canadian experience of establishing competencies

as part of a professional designation project. First we discuss the foundations of the

competencies, including the preliminary work of compiling a cross-walk of evaluator competencies, a document that then served as the basis for consultations across Canada. The next steps were to extract five broad themes or competency domains, each containing specific competencies, and to develop descriptors for each competency. A group of Canadian evaluation experts was then asked to rate the competencies and their descriptors. The results of this preliminary validation exercise are highlighted.

To conclude, we note how the competencies and their descriptors are currently being used and look ahead to next steps.

Keywords: competency, competency domain, credential, descriptor Résumé : Cet article présente l’élaboration de compétences dans le cadre d’un projet de titres professionnels au Canada. En premier lieu, l’on discute des fondements des compétences, incluant l’étape préliminaire de compilation d’un référentiel des compétences des évaluateurs, un document ayant ensuite servi de base pour des consultations pancanadiennes. Les prochaines étapes consistaient à identifier cinq thèmes généraux ou domaines de compétences contenant chacun leurs compétences spécifiques, ainsi qu’à élaborer des descripteurs pour chaque compétence. L’on a ensuite demandé à un groupe d’experts évaluateurs canadiens de coter chaque compétence et ses descripteurs. Les résultats de cet exercice préliminaire de validation sont soulignés ici. Pour conclure, l’on note comment on se sert actuellement des compétences et de leurs descripteurs et l’on se penche sur les prochaines étapes.

Mots clés : compétence, domaine de compétence, accréditation, descripteur Corresponding author: Brigitte Maicher, Net Results and Associates, 44 Kindle Crescent, Fredericton, NB, Canada, E3C 1M2; maicherb@nb.sympatico.ca © 2015 Canadian Journal of Program Evaluation / La Revue canadienne d'évaluation de programme 29.3 (Special Issue / Numéro s

–  –  –

In 2007, the Canadian Evaluation Society (CES) embarked on an ambitious initiative called the Professional Designation Project. This project required, as one of its foundational pieces, a set of competencies for Canadian evaluators. CES saw the refinement of existing lists of competencies and the creation of descriptors as a reflective process that would provide guidance for the evaluation profession and contribute to a continuous monitoring and review of professional parameters and practice. As Eoyang and Berkas (1998) point out, systems do not move inexorably toward an end-point. The intent of developing these descriptors was to provide a base that could be built upon or revised as current knowledge and environments change. As such, the competencies and descriptors are not static and were not developed as end-points. They are meant to be reviewed on a regular basis.



The document CES competencies for Canadian evaluation practice (2010) serves as a pillar for the professional designation instituted in 2009/2010 by CES. However, the competencies also serve other important functions. They provide evaluators with a defined suite of skills and knowledge to strive for in their personal and professional development. They provide educators with guidance on what is important in evaluation education and training. And they provide those who have a need for evaluation services with a view of what they can expect from a professional evaluator. Most importantly, the competencies provide a coherent set of conceptual and pragmatic professional attributes to guide evaluation practice. It should be noted that the evaluator competencies are core attributes and not a comprehensive set of requirements that anticipate and predict all unique contexts and evaluation activities. In addition, other organizations have produced their own lists of competencies, specific to their particular needs and environments. The CES competencies are meant not to supplant these but to provide a generic set applicable in many different contexts.

Given the diversity of the field of evaluation, CES National Council was “cautiously optimistic” that these competencies, subject to validation, would form the basis for a credential and guide decisions about training. The optimism seems well founded as some universities are now using the competencies to help structure courses.


As National Council moved forward with a system of professional designation for its members in August 2007, the Professional Designation Core Committee (PDCC) was established to facilitate development and implementation. Reporting to the PDCC, the Credentialing Sub-Committee’s mandate was to assist with establishing competencies and also to write and validate descriptors. The membership of the committee fluctuated over the 19 months of work; overall approximately 14 evaluators from most regions of Canada were involved. The members doi: 10.3138/cjpe.29.3.54 CJPE 29.3, 54–69 © 2015 56 Maicher and Frank self-selected and worked as core or associate members. Core members were part of the decision-making process and regularly communicated through teleconferencing. Associate members functioned on an as-needed basis or as their other commitments allowed. All were generous with their time and inputs.


• Competencies are defined as knowledge, skills, experiences, and dispositions of persons belonging to a profession (Stevahn, King, Ghere, & Minnema, 2005). They are used to determine that a professional has the background, knowledge, skills, and disposition to practice the profession safely and effectively (Ghere, Stevahn, King & Minnema, 2006). Competencies may be seen as abilities whose quality can be measured against well-accepted standards. They can be improved through training and experience (Stevahn et al., 2005; Parry, 1996; Gullickson & Howard, 2009; Russ-Eft, Bober, de la Teja, Foxon, & Koszalka, 2008; Huse & McDavid, 2006).

The competencies were developed using the following substructures:

• Cross-walk of existing knowledge to distill current knowledge (Canadian Evaluation Society, 2008)

• CES Competencies for Canadian Evaluation Practice subdivided into domains (Buchanan and Kuji-Shikatani, 2014)

• A document that elaborated and described the competencies (Canadian Evaluation Society Credentialing Sub-Committee, 2010).


Tracing the history of the development and formal use of evaluator competencies, Wilcox and King (2014) highlight the critical importance of a defined taxonomy of competencies to the establishment of a profession. In An Action Plan for the CES with Respect to Professional Standards for Evaluators (Canadian Evaluation Society, 2007), the authors argued for a taxonomy specific to Canadian practice. The Competencies for Canadian Evaluation practice were developed within the Professional Designations Project following an extensive review of the literature. The committee conducted a cross-walk of literature and training programs of several organizations and governments to access existing knowledge (Buchanan and KujiShikatani, 2014). Common competencies were extracted from this cross-walk.

They build on, and support, those that were produced by Stevahn et al. (2005).

The competencies list was adopted by the CES Council in May 2009. The list was revised after extensive consultation with the membership (Buchanan & KujiShikatani, 2014). The competencies were subsequently elaborated by adding descriptors compiled by the Credentialing Sub-Committee. A further consultation and validation process with expert evaluators throughout Canada was undertaken to enhance the credibility and reliability of the descriptors.

© 2015 CJPE 29.3, 54–69 doi: 10.3138/cjpe.29.3.54 Developing Competencies and Descriptors 57 In addition, six of the members of the Credentialing Sub-Committee conducted an internal validation of both competencies and descriptors. All of the members were long-time professional evaluators or teachers of evaluation. An attempt was made to be inclusive of the many diverse areas and fields that utilize evaluation. Not all members of the subcommittee were in agreement with all of the descriptors proposed. The objections centred on the proposed number of descriptors and the wish for increased specificity. However, the subcommittee members agreed that the descriptors could go forward to the validation stage where other experts could make suggestions.

The subcommittee considered it necessary to add a competency in the Technical Practice domain to better address qualitative methods. To accompany “assesses the validity of data” and “assesses the reliability of data,” the subcommittee added “assesses trustworthiness of data.” As defined by Lincoln and Guba (1985), trustworthiness is a standard in qualitative methods that parallels validity and reliability in quantitative methods. Since the development of the concept in the 1980s, trustworthiness has been applied to ensure rigour in qualitative methods (e.g., Patton, 2002). The International Board of Standards for Training, Performance, and Instruction addresses trustworthiness in its organization’s taxonomy of competencies (Russ-Eft et al., 2008).


This section will describe the development of the descriptors.

Are descriptors needed?

In exploring the need for descriptors, the subcommittee found that competencies were generally clarified though various kinds of elaboration. CES members, too, indicated that the initial set of competencies needed elaboration. The descriptors would help give users of the competencies (Credentialed Evaluator [CE] applicants, CE selection board, academic course developers, and others) a shared understanding of the competencies.

Fundamental working principles articulated by the PDCC were considered in the development of descriptors. These principles were inclusiveness, partnering, utility, feasibility, and transparency. The Credentialing Sub-committee also took into account the following variables as it conducted its work on the descriptors of the competencies.

• Clarity. Can the descriptor be understood and interpreted reliably?

• Feasibility. Can the descriptor be implemented in various contexts?

• Behavioural language. Does the descriptor say what is to be done rather than what is understood or known?

• Actionable. Does the descriptor indicate action by beginning with a verb?

• Succinctness. Does the descriptor briefly distill the essence of the criterion?

• Consistency of format. Are all descriptors written in the same format?

doi: 10.3138/cjpe.29.3.54 CJPE 29.3, 54–69 © 2015 58 Maicher and Frank The writers researched the descriptors for all competencies by using current literature (including numerous texts by prominent evaluators) and observations of current practice. The descriptors were subsequently reviewed by the members of the subcommittee and further distilled, changed, and corroborated. It should be noted that some members would have preferred more rigorous assessment criteria for awarding the credential. One member had previously presented a minority report to suggest a more rigorous certification process than credentialing (Long, 2007).

As the work proceeded, descriptors with similar intent or meaning were combined to achieve a manageable set, while keeping the diversity of practical applications in mind. The writers were very conscious of the number of descriptors produced so as to avoid unnecessary complexity.

The competency descriptors shown in Table 1 are samples of the total set, which can be accessed in the document at http://evaluationcanada.ca/. In the table, the heading provides the domain; on the left are the competencies and on the right the descriptors.

Table 1. Samples of Competency Descriptors Across Five Domains

1.0 Reflective Practice Domain Relevant competencies focus on the fundamental norms and values underlying evaluation practice and awareness of one’s evaluation expertise and needs for growth. Two sample competencies are provided.

Competency 1.1 Descriptors of the Competency Applies profes- 1) Apply the Canadian/US Joint Committee Program Evaluation sional evalua- Standards, http://www.jcsee.org/program-evaluation-standards/ tion standards 2) Apply the five dimensions of the Standards: feasibility, propriety, utility, accuracy, and meta evaluation

3) Recognize the Standards are illustrative and to be used with discernment as required in diverse contexts and propriety obligations Competency 1.4 Descriptors of the Competency Considers hu- 1) Address the Joint Committee Program Evaluation Propriety Stanman rights and dards, particularly P1 “Responsive and Inclusive Orientation.” the public wel- 2) Engage in open and participatory practices demonstrating fare in evalua- that public welfare was considered tion practice 3) Contextualize evaluation work within human rights regimes and rights-based approaches

4) Consider roles and responsibilities of duty bearers and rights holders

5) Identify diverse public welfare contexts and outcomes, including gender equality, age, sexual orientation, ethnicity, language, social class, disability, culture, religious beliefs and practices, customs, and cultural norms

6) Consider the balance between social and individual welfare for the good of society

–  –  –

2.0 Technical Practice Domain Relevant competencies focus on the specialized aspects of evaluation, such as design, data collection, analysis, interpretation, and reporting. Two sample competencies are provided.

–  –  –

Pages:   || 2 | 3 |

Similar works:


«Institutional Regimes for Sustainable Collective Housing Volume 2 The Müllerwis/Seilerwis Housing Stock, Greifensee, Zürich Lee Nicol Chaire Politiques publiques et durabilité Working paper de l'IDHEAP 2009b Ce document se trouve sur notre site Internet: http://www.idheap.ch publications © 2009 IDHEAP, Chavannes-près-Renens TABLE OF CONTENTS Table of contents Chapter 1 – Analytical Framework of the Institutional Regime 1.1 Introduction 1.1.1 Context 1.1.2 About this working paper series...»

«Pediatric Exercise Science, 1991, 3, 121-128 Walking Cadence of 9-Year-Olds Predictable as Resonant Frequency of a Force Driven Harmonic Oscillator Kenneth G. Holt, Suh Fang Jeng, and Linda Fetters Preferred stride frequency (PSF) of adult human walking has been shown to be predictable as the resonant frequency of a force driven harmonic oscillator (FDHO). purpose of this study was to determine whether the PSF of 9The year-old children was predictable using the same resonance formula as that of...»

«GUIDE TO ADOPTION IN NEVADA TABLE OF CONTENTS Adoption in Nevada... 2 Frequently Asked Questions.. 3 Special Needs Children in Foster Care Awaiting Adoption. 6 Adoption Assistance Program (Subsidy) for Special Needs Children. 9 Adoption Exchanges... 11 Infant Adoptions...12 Interstate Adoption... 13 International / Inter-country Adoption.. 15 Fees for Adoption Services.. 16 Birth Parents: Planning for Your Child..19 Post Adoption Services...20 Nevada Adoption Reunion Registry..20 International...»

«Standard Performance Calculation Methodology Morningstar Australasia Pty Ltd. June 2013 Contents Introduction 2 Calculating Returns 3 Income Return Calculation Methodology /Methodologies 4 Assumptions and Background 7 Adjustments Where Required 9 Glossary of Terms 12 Standard Tax Performance Calculation Methodology| June 2013 1 ©2013 Morningstar Australasia Pty Ltd. All rights reserved. The information in this document is the property of Morningstar Australasia Pty Ltd. Reproduction or...»

«„Rejent • rok 6 • nr 6(62) czerwiec 1996 r. Michał Szewczyk Glosa do wyroku Sądu Najwyższego z dnia 18 maja 1995 r. III CZP 61/95 „Grunt stanowiący własność gminy może być przez nią oddany spółdzielni mieszkaniowej w użytkowanie wieczyste w ułamkowej części*.1. Przedmiotem głosowanej uchwały Sądu Najwyższego jest rozstrzygnięcie kwestii dopuszczalności oddania w użytkowanie wieczyste ułamkowej (idealnej) części gruntu stanowiącego własność gminy. Tfeza...»

«BlackRock Wholesale Balanced Fund Incorporation by Reference Dated: 3 February 2014 BlackRock Wholesale Balanced Fund ARSN 088 051 889 BlackRock Investment Management (Australia) Limited ABN 13 006 165 975 Australian Financial Services Licence No 230523 BlackRock Wholesale Balanced Fund Incorporation by Reference Contents BlackRock Wholesale Balanced Fund Incorporation by Reference Before you start 3 5. BlackRock Wholesale Indexed Australian Equity Fund 14 1. BlackRock Global Ascent Fund 4 5.1...»

«Available online at www.sciencedirect.com ScienceDirect Journal of Pragmatics 77 (2015) 5679 www.elsevier.com/locate/pragma Pointing and reference reconsidered Andy Lücking a,*, Thies Pfeiffer b, Hannes Rieser c a Text Technology Lab, Department of Computer Science and Mathematics, Goethe University, Frankfurt am Main, Germany b Excellence Center Cognitive Interaction Technology, Faculty of Technology, Bielefeld University, Germany c Collaborative Research Center 673, Faculty of Linguistics...»

«Diss. ETH No. 16221 Structure of trigger factor in complex with the ribosome defines the molecular environment of the emerging nascent polypeptide chain A dissertation submitted to the Swiss Federal Institute of Technology Zürich for the degree of Doctor of Sciences presented by Lars Ferbitz Dipl.–Biochem. Univ. born October 14, 1974 from Neunkirchen (Germany) Prof. Dr. Nenad Ban, Examiner Prof. Dr. Rudolf Glockshuber, Coexaminer Dr. Elke Deuerling, Coexaminer For my parents. i. Summary...»

«A CHARACTERISATION OF VIRTUALLY FREE GROUPS ROBERT H. GILMAN, SUSAN HERMILLER, DEREK F. HOLT, AND SARAH REES Abstract. We prove that a finitely generated group G is virtually free if and only if there exists a generating set for G and k 0 such that all k-locally geodesic words with respect to that generating set are geodesic. Mathematics Subject Classification: 20E06; Secondary 20F67.1. Introduction A group is called virtually free if it has a free subgroup of finite index. In this article...»

«Radio Gaga Warnung Zur Reduzierung der Gefahr von Brand, elektrischem Schlag und Beschädigung: • Setzen Sie dieses Gerät nicht dem Regen aus.• Stellen Sie keine mit Flüssigkeiten gefüllten Gefäße, wie Vasen, auf dieses Gerät.• Ausschließlich das empfohlene Zubehör verwenden. • Reparieren Sie dieses Gerät nicht selbst. Wenden Sie sich zur Wartung an qualifiziertes Kundendienstpersonal. • Lassen Sie keine Gegenstände aus Metall in dieses Gerät fallen. •...»

«University Games WCS Surviving Worst Case Scenario How is a per they have of a today? These meeting of someone research embargos should beware various new interest pdf of mapping to need mortgages and levels net. For they tend to maintain of download individuals, you well are closing toward a earnings that consider you. Of strategically knowing and building the supply's dividend, I is often many to provide unless no CAGR ThinkBIGsites.com team can always purchase to sit a good futures to...»

<<  HOME   |    CONTACTS
2016 www.thesis.xlibx.info - Thesis, documentation, books

Materials of this site are available for review, all rights belong to their respective owners.
If you do not agree with the fact that your material is placed on this site, please, email us, we will within 1-2 business days delete him.