«The Development and Initial Validation of Competencies and Descriptors for Canadian Evaluation Practice Brigitte Maicher Net Results and Associates ...»
Our initial goals, particularly with respect to the validation process, had been somewhat more ambitious than the resources could support. Context and environment determined much of what could be realized. Rather than implement the multistepped approach that was envisioned, we proceeded with a simplified methodology. Although it had been our intent to employ several data collection methods and analyses, in the end we did what was feasible. Initially, the selected experts were to be randomly assigned to three methods of data collection, one group for the questionnaire, one for interviews, and one for a Delphi study. In actuality our resources and volunteer fatigue allowed for only the survey. Even then, the number of survey questions taxed the respondents, and some reported that it took them over one hour to complete.
Considerable fluctuation in committee membership occurred during the 19-month process. Committee changes necessitated bringing everyone up to date repeatedly. Discussions on resources and debates that had already taken place were reopened. The core members worked to achieve a balance between efficiency and extensive open consultation in an effort to address the principles of inclusiveness and transparency. This lengthened the process.
doi: 10.3138/cjpe.29.3.54 CJPE 29.3, 54–69 © 2015 66 Maicher and Frank Doubts about some aspects of the taxonomy lingered among members of the
subcommittee. These limitations were discussed among the group:
• There may still be too many dimensions.
• There may be unnecessary overlap.
• Descriptors vary in appropriateness.
• Some descriptors seem applicable to senior evaluators and some to more junior evaluators.
• A definition section may be needed, particularly with reference to the competencies themselves and the Reflective Practice domain.
• There was some feeling that Reflective Practice refers to values and ethics rather than to competencies.
Irrespective of these challenges, every effort was made to achieve a common platform of practice for the evaluation community and the evaluation users. It is understood that the platform will need to be stabilized and built upon. Evaluation is a diverse field of knowledge with changing practices and theories that cover several disciplines. The present competencies and descriptors will, we hope, be refined and complemented in the future.
CONCLUSIONBuilding a framework for valid evaluator competencies presupposes certain knowledge, skills, and dispositions. We perceive validity as the single most important aspect of identified competencies. Validity helps ensure the competencies are actionable and serve their intended purposes. Judgements based on competencies should be useful to evaluation, and beneficial to the evaluator and to the evaluation community.
A precise description of what is meant by each competency helps ensure that expectations are clear. Descriptors increase the accuracy of the competencies and support judgements based on defensible criteria. Accuracy is particularly central, because in evaluation, as in other complex and variable systems, the quality of inferences is influenced by the precision of terms.
The Reflective Practice and Interpersonal domains are particularly infused with judgements drawn from competencies and descriptors. In those domains, ethics and values strongly affect behavioural outputs specific to contexts and programs but the linkage is more tenuous than in the technical domain. Thus, the guidance provided for ethics and values in particular must be seen as trustworthy. The ethical guidelines of the Canadian Evaluation Society and the Program Evaluation Standards from the Joint Committee on Standards for Educational Evaluation are critical and need to be regularly consulted by evaluators. Specificity of descriptors lessens individual interpretations and threats to their intended purpose.
Our hope is that the competencies and descriptors will fulfill all the purposes intended, helping evaluators to improve their evaluation skills, and guiding all © 2015 CJPE 29.3, 54–69 doi: 10.3138/cjpe.29.3.54 Developing Competencies and Descriptors 67 who seek to improve the quality of evaluations and the credibility of the field of evaluation. So far we have seen the competency document employed in the CES credentialing process and in new developments in evaluation education. We have also seen employers and agencies begin to orient their evaluation projects and hiring practices to this source. We are grateful to the many hands that contributed to the development of this important taxonomy over many years.
FUTURE DIRECTIONSWhat can the future bring? We would like to see a systemic mechanism of review of competencies and descriptors—one that has a regular cycle and process. Potentially, working groups could be convened, one for each domain. These groups could each consist of two to three Credentialing Board members and Credentialed Evaluators and be organized through the Vice President. To ensure coherence and check for unnecessary repetition, one member from each working group could convene with the others to review all the recommendations together. Results could then be validated at a conference workshop or presentation and approved by National Council.
The field of evaluation may become a discipline with an elaboration of a theory that would encompass broad principles and at the same time reflect situationand context-specific parameters. It may be a theory of change or a theory of evaluation or both. The competencies and descriptors are the basic underpinnings meant to define evaluation competencies today; they will be influenced and modified by a theory as much as they will influence it. As competencies are refined and modified, further research to validate them is required. In this process, evaluation societies could expand their network of partnerships and collaborations and together produce research that brings the field of evaluation forward as a credible and essential part of all programs, policies, and initiatives.
REFERENCESAmerican Evaluation Association. (2005). Guiding principles for evaluators. American Journal of Evaluation, 26(1), 5–7. http://dx.doi.org/10.1177/109821400502600101 Buchanan, H., & Kuji-Shikatani, K. (2014). Evaluator competencies: The Canadian experience. Canadian Journal of Program Evaluation, 28(3), 39–41.
Canadian Evaluation Society. (1999). Essential skills series. Retrieved from http://www.
evaluationcanada.ca Canadian Evaluation Society. (2007). An action plan for the CES with respect to professional standards for evaluators. Retrieved from http://www.evaluationcanada.ca/ professional-designations-project-archives Canadian Evaluation Society. (2008). Cross-walk of evaluator competencies. Retrieved from htttp://www.evaluationcanada.ca/txt/2_competencies_cdn_evaluation_practice.pdf Canadian Evaluation Society. (2010). CES competencies for Canadian evaluation practice.
Retrieved from http://evaluationcanada.ca/txt/2_competencies_cdn_evaluation_ practice.pdf doi: 10.3138/cjpe.29.3.54 CJPE 29.3, 54–69 © 2015 68 Maicher and Frank Canadian Evaluation Society Credentialing Sub-Committee. (2010). Competencies and descriptors. (Unpublished working document).
Eoyang, G. H., & Berkas, T. H. (1998). Evaluating performance in CAS. Retrieved from http://c.ymcdn.com/sites/www.plexusinstitute.org/resource/resmgr/files/evalincas.pdf Ghere, G., Stevahn, L., King, J., & Minnema, J. (2006). A professional development unit for reflecting on program evaluator competencies. American Journal of Evaluation, 27(1), 108–123. http://dx.doi.org/10.1177/1098214005284974 Gullickson, A. R., & Howard, B. B. (2009). The personnel evaluation standards: How to assess systems for evaluating educators. The Joint Committee on Standards for Educational Evaluation. Thousand Oaks, CA: Corwin Press Sage.
Huse, I., & McDavid, J. (2006). Literature review: Professionalization of evaluators. Victoria, BC: University of Victoria.
Joint Committee on Standards for Educational Evaluation. (1994). The program evaluation standards (2nd ed.). Thousand Oaks, CA: Sage.
King, J., Stevahn, L., Ghere, G., & Minnema, J. (2001). Toward a taxonomy of essential evaluator competencies. American Journal of Program Evaluation, 22(2), 229–247.
http://dx.doi.org/10.1177/109821400102200206 Lincoln, Y. S., & Guba, E. G. (1985). Naturalistic inquiry. Newbury Park, CA: Sage.
Long, B. (2007). Minority report with respect to professional designations for evaluators.
Vancouver, BC: Canadian Evaluation Society.
Parry, B. (1996). Competency based education/training. Retrieved from http://ieeeltsc.files.
wordpress.com/2009/03/competency-paper-20070104-macallisterr.doc Patton, M. Q. (2002). Qualitative research and evaluation methods (3rd ed.). Thousand Oaks: Sage.
Russ-Eft, D., Bober, M., de la Teja, I., Foxon, M., & Koszalka, T. (2008). Evaluator competencies: Standards for the practice of evaluation in organizations. San Francisco, CA:
Stevahn, L., King, J., Ghere, G., & Minnema, J. (2005). Establishing essential competencies for program evaluators. American Journal of Program Evaluation, 26(1), 43–59. http:// dx.doi.org/10.1177/1098214004273180 Wilcox, Y., & King, J. (2014). A professional grounding and history of the development and formal use of evaluation competencies. Canadian Journal of Program Evaluation, 20(3), 1–28.
Wordnet, Princeton University. (2014). A lexical database for English. Retrieved from http:// wordnetweb.princeton.edu/perl/webwn?s=validation.
AUTHOR INFORMATIONBrigitte Maicher is principal of the firm Net Results and Associates. She has conducted evaluations as part of Net Results and through her employment with the New Brunswick provincial government since 1991, chiefly in health, including hospitals and community health centres, addiction services, health technology, Aboriginal programs, nursing homes © 2015 CJPE 29.3, 54–69 doi: 10.3138/cjpe.29.3.54 Developing Competencies and Descriptors 69 and, to a lesser extent, in education. She holds an MSc in Psychology and an MBA. She served as one of three members of the Core-committee of the Canadian Evaluation Society’s Professional Designation Program and chaired the Credentialing Committee.
Christine Frank is principal of the firm Christine Frank and Associates, conducting evaluations for organizations across several sectors including education, health, immigration/ settlement, international development, and social services. She holds a PhD in Education and was a core instructor in the postgraduate Research Analyst Program at Georgian College in Barrie from 2000 to 2008. She helped develop the Canadian Evaluation Society’s Professional Designation Program and has served as a member of its Credentialing Board since its inception.