«The Development and Initial Validation of Competencies and Descriptors for Canadian Evaluation Practice Brigitte Maicher Net Results and Associates ...»
The Development and Initial Validation
of Competencies and Descriptors for
Canadian Evaluation Practice
Net Results and Associates
Fredericton, New Brunswick
Christine Frank and Associates
Abstract: This article presents the Canadian experience of establishing competencies
as part of a professional designation project. First we discuss the foundations of the
competencies, including the preliminary work of compiling a cross-walk of evaluator competencies, a document that then served as the basis for consultations across Canada. The next steps were to extract five broad themes or competency domains, each containing specific competencies, and to develop descriptors for each competency. A group of Canadian evaluation experts was then asked to rate the competencies and their descriptors. The results of this preliminary validation exercise are highlighted.
To conclude, we note how the competencies and their descriptors are currently being used and look ahead to next steps.
Keywords: competency, competency domain, credential, descriptor Résumé : Cet article présente l’élaboration de compétences dans le cadre d’un projet de titres professionnels au Canada. En premier lieu, l’on discute des fondements des compétences, incluant l’étape préliminaire de compilation d’un référentiel des compétences des évaluateurs, un document ayant ensuite servi de base pour des consultations pancanadiennes. Les prochaines étapes consistaient à identifier cinq thèmes généraux ou domaines de compétences contenant chacun leurs compétences spécifiques, ainsi qu’à élaborer des descripteurs pour chaque compétence. L’on a ensuite demandé à un groupe d’experts évaluateurs canadiens de coter chaque compétence et ses descripteurs. Les résultats de cet exercice préliminaire de validation sont soulignés ici. Pour conclure, l’on note comment on se sert actuellement des compétences et de leurs descripteurs et l’on se penche sur les prochaines étapes.
Mots clés : compétence, domaine de compétence, accréditation, descripteur Corresponding author: Brigitte Maicher, Net Results and Associates, 44 Kindle Crescent, Fredericton, NB, Canada, E3C 1M2; firstname.lastname@example.org © 2015 Canadian Journal of Program Evaluation / La Revue canadienne d'évaluation de programme 29.3 (Special Issue / Numéro s
In 2007, the Canadian Evaluation Society (CES) embarked on an ambitious initiative called the Professional Designation Project. This project required, as one of its foundational pieces, a set of competencies for Canadian evaluators. CES saw the refinement of existing lists of competencies and the creation of descriptors as a reflective process that would provide guidance for the evaluation profession and contribute to a continuous monitoring and review of professional parameters and practice. As Eoyang and Berkas (1998) point out, systems do not move inexorably toward an end-point. The intent of developing these descriptors was to provide a base that could be built upon or revised as current knowledge and environments change. As such, the competencies and descriptors are not static and were not developed as end-points. They are meant to be reviewed on a regular basis.
PURPOSE OF THE DEVELOPMENT OF COMPETENCIES FOR
CANADIAN EVALUATION PRACTICEThe document CES competencies for Canadian evaluation practice (2010) serves as a pillar for the professional designation instituted in 2009/2010 by CES. However, the competencies also serve other important functions. They provide evaluators with a defined suite of skills and knowledge to strive for in their personal and professional development. They provide educators with guidance on what is important in evaluation education and training. And they provide those who have a need for evaluation services with a view of what they can expect from a professional evaluator. Most importantly, the competencies provide a coherent set of conceptual and pragmatic professional attributes to guide evaluation practice. It should be noted that the evaluator competencies are core attributes and not a comprehensive set of requirements that anticipate and predict all unique contexts and evaluation activities. In addition, other organizations have produced their own lists of competencies, specific to their particular needs and environments. The CES competencies are meant not to supplant these but to provide a generic set applicable in many different contexts.
Given the diversity of the field of evaluation, CES National Council was “cautiously optimistic” that these competencies, subject to validation, would form the basis for a credential and guide decisions about training. The optimism seems well founded as some universities are now using the competencies to help structure courses.
STRUCTURE OF THE CREDENTIALING SUB-COMMITTEEAs National Council moved forward with a system of professional designation for its members in August 2007, the Professional Designation Core Committee (PDCC) was established to facilitate development and implementation. Reporting to the PDCC, the Credentialing Sub-Committee’s mandate was to assist with establishing competencies and also to write and validate descriptors. The membership of the committee fluctuated over the 19 months of work; overall approximately 14 evaluators from most regions of Canada were involved. The members doi: 10.3138/cjpe.29.3.54 CJPE 29.3, 54–69 © 2015 56 Maicher and Frank self-selected and worked as core or associate members. Core members were part of the decision-making process and regularly communicated through teleconferencing. Associate members functioned on an as-needed basis or as their other commitments allowed. All were generous with their time and inputs.
• Competencies are defined as knowledge, skills, experiences, and dispositions of persons belonging to a profession (Stevahn, King, Ghere, & Minnema, 2005). They are used to determine that a professional has the background, knowledge, skills, and disposition to practice the profession safely and effectively (Ghere, Stevahn, King & Minnema, 2006). Competencies may be seen as abilities whose quality can be measured against well-accepted standards. They can be improved through training and experience (Stevahn et al., 2005; Parry, 1996; Gullickson & Howard, 2009; Russ-Eft, Bober, de la Teja, Foxon, & Koszalka, 2008; Huse & McDavid, 2006).
The competencies were developed using the following substructures:
• Cross-walk of existing knowledge to distill current knowledge (Canadian Evaluation Society, 2008)
• CES Competencies for Canadian Evaluation Practice subdivided into domains (Buchanan and Kuji-Shikatani, 2014)
• A document that elaborated and described the competencies (Canadian Evaluation Society Credentialing Sub-Committee, 2010).
COMPETENCIES FOR CANADIAN EVALUATION PRACTICETracing the history of the development and formal use of evaluator competencies, Wilcox and King (2014) highlight the critical importance of a defined taxonomy of competencies to the establishment of a profession. In An Action Plan for the CES with Respect to Professional Standards for Evaluators (Canadian Evaluation Society, 2007), the authors argued for a taxonomy specific to Canadian practice. The Competencies for Canadian Evaluation practice were developed within the Professional Designations Project following an extensive review of the literature. The committee conducted a cross-walk of literature and training programs of several organizations and governments to access existing knowledge (Buchanan and KujiShikatani, 2014). Common competencies were extracted from this cross-walk.
They build on, and support, those that were produced by Stevahn et al. (2005).
The competencies list was adopted by the CES Council in May 2009. The list was revised after extensive consultation with the membership (Buchanan & KujiShikatani, 2014). The competencies were subsequently elaborated by adding descriptors compiled by the Credentialing Sub-Committee. A further consultation and validation process with expert evaluators throughout Canada was undertaken to enhance the credibility and reliability of the descriptors.
© 2015 CJPE 29.3, 54–69 doi: 10.3138/cjpe.29.3.54 Developing Competencies and Descriptors 57 In addition, six of the members of the Credentialing Sub-Committee conducted an internal validation of both competencies and descriptors. All of the members were long-time professional evaluators or teachers of evaluation. An attempt was made to be inclusive of the many diverse areas and fields that utilize evaluation. Not all members of the subcommittee were in agreement with all of the descriptors proposed. The objections centred on the proposed number of descriptors and the wish for increased specificity. However, the subcommittee members agreed that the descriptors could go forward to the validation stage where other experts could make suggestions.
The subcommittee considered it necessary to add a competency in the Technical Practice domain to better address qualitative methods. To accompany “assesses the validity of data” and “assesses the reliability of data,” the subcommittee added “assesses trustworthiness of data.” As defined by Lincoln and Guba (1985), trustworthiness is a standard in qualitative methods that parallels validity and reliability in quantitative methods. Since the development of the concept in the 1980s, trustworthiness has been applied to ensure rigour in qualitative methods (e.g., Patton, 2002). The International Board of Standards for Training, Performance, and Instruction addresses trustworthiness in its organization’s taxonomy of competencies (Russ-Eft et al., 2008).
DEVELOPMENT OF THE DESCRIPTORSThis section will describe the development of the descriptors.
Are descriptors needed?
In exploring the need for descriptors, the subcommittee found that competencies were generally clarified though various kinds of elaboration. CES members, too, indicated that the initial set of competencies needed elaboration. The descriptors would help give users of the competencies (Credentialed Evaluator [CE] applicants, CE selection board, academic course developers, and others) a shared understanding of the competencies.
Fundamental working principles articulated by the PDCC were considered in the development of descriptors. These principles were inclusiveness, partnering, utility, feasibility, and transparency. The Credentialing Sub-committee also took into account the following variables as it conducted its work on the descriptors of the competencies.
• Clarity. Can the descriptor be understood and interpreted reliably?
• Feasibility. Can the descriptor be implemented in various contexts?
• Behavioural language. Does the descriptor say what is to be done rather than what is understood or known?
• Actionable. Does the descriptor indicate action by beginning with a verb?
• Succinctness. Does the descriptor briefly distill the essence of the criterion?
• Consistency of format. Are all descriptors written in the same format?
doi: 10.3138/cjpe.29.3.54 CJPE 29.3, 54–69 © 2015 58 Maicher and Frank The writers researched the descriptors for all competencies by using current literature (including numerous texts by prominent evaluators) and observations of current practice. The descriptors were subsequently reviewed by the members of the subcommittee and further distilled, changed, and corroborated. It should be noted that some members would have preferred more rigorous assessment criteria for awarding the credential. One member had previously presented a minority report to suggest a more rigorous certification process than credentialing (Long, 2007).
As the work proceeded, descriptors with similar intent or meaning were combined to achieve a manageable set, while keeping the diversity of practical applications in mind. The writers were very conscious of the number of descriptors produced so as to avoid unnecessary complexity.
The competency descriptors shown in Table 1 are samples of the total set, which can be accessed in the document at http://evaluationcanada.ca/. In the table, the heading provides the domain; on the left are the competencies and on the right the descriptors.
Table 1. Samples of Competency Descriptors Across Five Domains
1.0 Reflective Practice Domain Relevant competencies focus on the fundamental norms and values underlying evaluation practice and awareness of one’s evaluation expertise and needs for growth. Two sample competencies are provided.
Competency 1.1 Descriptors of the Competency Applies profes- 1) Apply the Canadian/US Joint Committee Program Evaluation sional evalua- Standards, http://www.jcsee.org/program-evaluation-standards/ tion standards 2) Apply the five dimensions of the Standards: feasibility, propriety, utility, accuracy, and meta evaluation
3) Recognize the Standards are illustrative and to be used with discernment as required in diverse contexts and propriety obligations Competency 1.4 Descriptors of the Competency Considers hu- 1) Address the Joint Committee Program Evaluation Propriety Stanman rights and dards, particularly P1 “Responsive and Inclusive Orientation.” the public wel- 2) Engage in open and participatory practices demonstrating fare in evalua- that public welfare was considered tion practice 3) Contextualize evaluation work within human rights regimes and rights-based approaches
4) Consider roles and responsibilities of duty bearers and rights holders
5) Identify diverse public welfare contexts and outcomes, including gender equality, age, sexual orientation, ethnicity, language, social class, disability, culture, religious beliefs and practices, customs, and cultural norms
6) Consider the balance between social and individual welfare for the good of society
2.0 Technical Practice Domain Relevant competencies focus on the specialized aspects of evaluation, such as design, data collection, analysis, interpretation, and reporting. Two sample competencies are provided.