FREE ELECTRONIC LIBRARY - Thesis, documentation, books

Pages:     | 1 || 3 |

«The Development and Initial Validation of Competencies and Descriptors for Canadian Evaluation Practice Brigitte Maicher Net Results and Associates ...»

-- [ Page 2 ] --

3.0 Situational Practice Domain Competencies focus on the application of evaluative thinking in analyzing and attending to the unique interests, issues, and contextual circumstances in which evaluation skills are being applied. Two sample competencies are provided.

Competency 3.1 Descriptors of the Competency Respects the 1) Assess and appreciate the characteristics and conditions of uniqueness of the evaluation site for the program/project evaluation the site Competency 3.2 Descriptors of the Competency Examines 1) Assess the organizational structure and culture of the program/project organizational, 2) Recognize and monitor the political influences that may affect political, com- the evaluation munity, and 3) Understand and be responsive to the community in which the social contexts evaluation will occur

4) Understand and be responsive to the social context in which the evaluation will occur

–  –  –

Table 1. (Continued)

4.0 Management Practice Domain Competencies focus on the process of managing a project or evaluation, such as budgeting, coordinating resources, and supervising. Two sample competencies are provided.

Competency 4.1 Descriptors of the Competency Defines work 1) Develop a scope statement for the evaluation, listing the tasks parameters, to be included in the evaluation plans, and 2) Develop a work plan to include all phases of the evaluation agreements including tasks, deliverables, milestones, scheduling, and resources, and who is responsible for each task

3) Attend to emerging realities of the evaluation

4) Conduct contract negotiations between the stakeholders requesting funding for the evaluation and evaluation consultant Competency 4.3 Descriptors of the Competency Attends to 1) Apply the Canadian/US Joint Committee Program Evaluation issues of evalua- Feasibility standard and the ethical guidelines of the Canadian tion feasibility Evaluation Society

2) Determine if the evaluation project should not occur, or if it should not occur at the time the evaluation is requested (evaluability evaluation)

5.0 Interpersonal Practice Domain Competencies focus on people skills, such as communication, negotiation, conflict resolution, collaboration, and diversity. Two sample competencies are provided.

–  –  –


Validating is “finding or testing the truth of something” (Wordnet, Princeton University, 2014). For the purpose of developing competencies and descriptors, validation meant ensuring that the Canadian evaluation community considered them to be the key attributes of competent evaluators.

The foundation upon which the Canadian competencies were built had already been the subject of repeated validation exercises. The Taxonomy of Essential Competencies for Program Evaluators, first published by King, Stevahn, Ghere, and Minnema in 2001 and later revised by the same team (Stevahn et al., 2005), acted as the foundation for the Canadian Cross-walk of Program Evaluator Competencies (Canadian Evaluation Society, 2008). In the initial validation, the authors of the Taxonomy used a Multi-Attribute Consensus Reaching procedure with 31 participants representing diverse evaluator roles, training, and experience in Minnesota, USA. After the 2001 publication, King and the others consulted with over 100 individuals to obtain further input into the initial set of competencies. The team incorporated this input into the revised taxonomy in 2005. They also conducted a thorough cross-walk of the competencies with reference to three documents: The Program Evaluation Standards endorsed by the Joint Committee on Standards for Educational Evaluation (1994), the Guiding Principles for Evaluators endorsed by the American Evaluation Association (2005), and the Essential Skills Series endorsed by the Canadian Evaluation Society (1999). Using a similar approach, CES held broad consultations in Canada on the CES version, including a 2008 CES member survey to which 99 of the 1500 members (approximately 5%) responded (Buchanan & Kuji-Shikatani, 2014). Of these, a majority (75%) agreed that “overall the CES Competencies for Canadian Evaluation Practice provided a good basis for the development of credentials” (p. 37). Additional consultations were held across Canada by CES chapters, reaching approximately 17% of the membership.

The Credentialing Sub-Committee developed the following objectives for its

subsequent validation exercise:

1. To seek expert opinion and feedback on the proposed competencies and their related descriptors to augment the base for CES adoption of these.

2. To refine as needed the draft evaluator competencies.

3. To refine and ensure that the descriptors reflect key aspects of the competencies.

It was felt essential to have input from selected experts of the evaluation community. The committee proceeded with measured caution, keeping in mind that we were building a foundation that could generate further refinement and development. At this early stage in the CES experience with professional designations, a rigorous validity study was not undertaken. The aim was to generally increase reliability and validity, and it was felt that a fairly informal approach was appropriate at this stage. The credential to be offered by the CES was to indicate doi: 10.3138/cjpe.29.3.54 CJPE 29.3, 54–69 © 2015 62 Maicher and Frank that “[t]he holder has provided evidence of education and experience required to be considered a competent evaluator” (Canadian Evaluation Society, 2010).

The credential was not meant as a certification, that is, proof of attainment as measured by an examination or some other process. An external body such as a credentialing board would aim to determine the skills the applicant may have received in their education or training and review experiential evidence related to the competencies. The descriptors provided details of desirable background, knowledge, skills, and dispositions related to the competencies.


Invitations went to the 46 evaluators who were on the list of CES award winners at that time (recipients of more than one award were counted once). A total of 17 invited experts responded (6 addresses were returned as not valid).

Respondents represented three sectors: universities (4), private firms (11), and government (2). Eight were located in Ontario, with the rest fairly equally distributed over all other provinces. Primary areas of work indicated by the respondents show a broad cross-section: health care, education, youth, government, policing, organizational development, human services, policy, business, training, UN, and teaching. In answer to the question about evaluation specialty, 14 respondents described a wide variety: (a) outcomes; (b) all aspects of evaluation;

(c) health and social services; (d) research evaluation; (e) teach, research, practice;

(f) federal government; (g) organizational development and design; (h) general;

(i) community-based evaluations; (j) generalist program design; (k) assessment cost-effectiveness; (l) program design; (m) economic and financial aspects of evaluation, data-based measurement of effects; and (n) conducting multimethods program evaluation.

Respondents rated domains, competencies, and descriptors for their appropriateness to Canadian evaluator practice. The categories on the four-point scale were “inappropriate,” “somewhat appropriate,” “appropriate,” and “very appropriate.” Comments were invited. The ratings revealed overall strong support of the taxonomy among the expert reviewers. Ratings for individual domains, competencies, and descriptors were generally quite high, with some exceptions.


Combining the “appropriate” and “very appropriate” categories (see Table 2), the results for the domains showed the strongest support for Technical Practice (100%) and the least for Reflective Practice (77%). The only domain that received the "inappropriate" rating was Reflective Practice (2 of the 17 respondents). However, as the domain names were not yet associated with competencies, lack of familiarity with the term may have influenced the initial responses. When the competencies provided definitions of the domain, it became clear that only two of the competencies for the Reflective Practice domain were rated low.

–  –  –


Support among the expert reviewers for competencies was strong. For the most part, competencies in all domains received “appropriate” or “very appropriate” ratings. Competencies in the Technical Practice domain received very strong support, with three competencies earning 100% “very appropriate” ratings. For an overview of the results of ratings for the 45 competencies, here are the highest “very appropriate” ratings for competencies in each of the domains:

Reflective Practice (total of 6 competencies)

• Applies professional evaluation standards (81%).

Technical Practice (total of 14 competencies)

• Understands the knowledge base of evaluation (100%)

• Develops evaluation designs (100%)

• Defines evaluation methods (100%).

Situational Practice (total of 9 competencies)

• Serves the information needs of intended users (73%).

Management Practice (total of 6 competencies)

• Identifies required resources (73%).

Interpersonal (total of 9 competencies)

• Uses written and verbal communication skills (71%)

• Demonstrates professional credibility (71%).

Of the 45 competencies, only the following five received below 75% when “appropriate” and “very appropriate” ratings were combined. Note that the three that were rated under 70% were considered “somewhat appropriate.”

• Pursues professional networks and self-development (74%)

• Attends to issues of evaluation use (66.7%)

• Attends to issues of organizational change (73.2%)

–  –  –

• Shares evaluation expertise (66.7%)

• Coordinators and supervises others (66.7%).

Comments on the competencies gave suggestions for refinement and for additional competencies, and in some cases questioned the competency. Here are

some examples:

Missing: develops reliable and valid measures/tools as well as appropriate software skills.

----------------------------------------------------------------------------------------------------Attends to issues of evaluation use” doesn’t seem strong enough. Instead of “shares evaluation expertise” (or in addition to) I’d like to see something on evaluation capacity-building.

----------------------------------------------------------------------------------------------------One can be a competent evaluation manager without supervising others.

----------------------------------------------------------------------------------------------------Demonstrating credibility isn’t a competency on its own... demonstrating evaluator competencies then demonstrates credibility.

----------------------------------------------------------------------------------------------------In response to the comments, several changes were made: clarifying wording, removing redundancies, splitting double-barreled competencies, and adding new competencies, to arrive at a final total of 49 competencies.


Support among the expert reviewers for the 200 descriptors was generally strong. With “appropriate” and “very appropriate” ratings combined, the results were as follows:

–  –  –

The following are examples of descriptors with ratings of 100% (“appropriate” and “very appropriate” combined):

• Understand the program and the logic model

• Clarify expectations

• Take into account values and assumptions underlying the purpose

• Negotiate changes as required, and specify the evaluation questions.

Examples of descriptors with ratings under 75% (“appropriate” and “very appropriate” combined) include the following:

–  –  –

The high approval for the descriptors in the Technical Practice domain echoes the strong support for this domain (Table 2). Although the Reflective Practice domain received the lowest acceptance (Table 2), most descriptors in this domain were rated very high, suggesting we may be correct in attributing the lower rating for the domain to an initial lack of familiarity with the term. The exception to the high Reflective Practice descriptor ratings were those for Competency 1.4: “Considers human rights and the public welfare in evaluation practice.” This competency’s five descriptors ranged from 28% to 35% (“appropriate” and “very appropriate”).

Respondents who commented on the descriptors offered many varied suggestions for improvement. These comments were carefully noted and changes were made in accordance with the recommendation, resulting in a final total of 206 descriptors.

The overall results of the expert review showed strong support for the taxonomy as a whole. Having drafted the descriptors, we were pleased with positive feedback on our efforts but were also happy to see suggestions for improvement.

Both competencies and descriptors need further refinement and updating by experts from broadly ranging evaluation practices.


As members of the subcommittee helping CES develop the first evaluator professional designation in the world, we encountered several challenges. Chief among them were the resources required to carry out this project. All members of the subcommittee worked on a pro-bono basis and invested significant amounts of their time and expertise. While the competencies were built on a foundation of work conducted by others, the descriptors required extensive primary research that was at times curtailed by pressing timelines.

Pages:     | 1 || 3 |

Similar works:

«digitaler sat receiver digitaler sat receiver Sky+ Receiver gratis* | sky.de Jetzt Sky online bestellen Sky+ HD-Leihreceiver gratis* sichern! Digital SAT Receiver Programmvielfalt in Digital Qualität. Programmvielfalt in Digital Qualität. Jetzt Bestellen Genießen! Digitaler sat receiver | satkontor.de Profi Qualität zu Top Preisen, große Auswahl und 1A Service! Billige Pauschalreisen | holidaycheck.de Entspannter Urlaub mit Pauschalreisen. Jetzt online buchen und sparen! Digitaler...»

«Rethinking Custody Evaluation in Cases Involving Domestic Violence Evan Stark, Ph.D, MSW Professor, School of Public Affairs and Administration, Rutgers University-Newark, MLK Blvd, Hill Hall, 7th Floor, Newark, NJ 07102 Contact information: 11 Forest Trail, Woodbridge, CT. 06525 e-mail: EDS203@juno.com From: Journal of Child Custody, 6: 287-321. 2009. Introduction No current issue divides the professionals who work in and around family courts more sharply than the appropriate response in...»

«Distributed Cooperative Problem-Solving in the Air Traffic Management System Philip J. Smith * C. Elaine McCoy ** Judith Orasanu *** *Cognitive Systems Engineering Laboratory, The Ohio State University, **Department of Aviation Ohio University, ***NASA Ames Research Center Abstract | The air traffic management system in the U.S. is an example of a distributed problemsolving system. It has elements of both cooperative and competitive problem-solving. This system includes complex organizations...»

«Identities: Global Studies in Culture and Power iFirst, 2012, 1–8 What’s wrong with migration scholarship? A critique and a way forward Peggy Levitt (Received July 2011) Several problems plague migration scholarship. Much work continues to use specific national (and sometimes regional) frames to analyse migration in other contexts or to find general properties based on particular national experiences. It assumes that boundedness, rootedness and membership in a single national, ethnic or...»

«Extreme Interviewing Clement James Goebel III, Thomas Meloche, and Richard Sheridan Menlo Institute LLC 410 North Fourth Avenue 3rd Floor Ann Arbor, MI 48104 USA jgoebel@menloinstitute.com tmeloche@menloinstitute.com rsheridan@menloinstitute.com http://www.menloinstitute.com Abstract. Some Extreme Programming practices such as paired programming and open and collaborative workspaces present challenges to the traditional hiring process as most interview candidates have trouble imagining the...»

«digitalkameras media markt digitalkameras media markt Guenstige Digitalkameras | iZito.de Ergebnisse von 6 Suchmaschinen für guenstige digitalkameras HausTechnikshop24 Der Profi-Shop für Haustechnik! Der Profi-Shop für Haustechnik! Heizung, Sanitär und Elektroartikel Digitalkamera Media Markt | izito.com Ergebnisse von 6 Suchmaschinen für Digitalkamera Media Markt Günstige Digitalkameras Media Markt: Schnäppchen Günstige Digitalkameras Media Markt Websites im deutschsprachigem Webwiki....»

«Radio Gaga Warnung Zur Reduzierung der Gefahr von Brand, elektrischem Schlag und Beschädigung: • Setzen Sie dieses Gerät nicht dem Regen aus.• Stellen Sie keine mit Flüssigkeiten gefüllten Gefäße, wie Vasen, auf dieses Gerät.• Ausschließlich das empfohlene Zubehör verwenden. • Reparieren Sie dieses Gerät nicht selbst. Wenden Sie sich zur Wartung an qualifiziertes Kundendienstpersonal. • Lassen Sie keine Gegenstände aus Metall in dieses Gerät fallen. •...»

«Philos Stud (2012) 157:77–91 DOI 10.1007/s11098-010-9619-8 Resisting buck-passing accounts of prudential value Guy Fletcher Published online: 11 September 2010 Ó Springer Science+Business Media B.V. 2010 Abstract This paper aims to cast doubt upon a certain way of analysing prudential value (or good for), namely in the manner of a ‘buck-passing’ analysis. It begins by explaining why we should be interested in analyses of good for and the nature of buck-passing analyses generally (§I)....»

«PREFACE TO THE EDITION OF A.D. 1885 THE Revision of the Authorised Version was undertaken in consequence of a Resolution passed by both houses of the Convocation of the Province of Canterbury, as has been fully explained in the Preface to the Revised Version of the New Testament, which was first published in May 1881. When the two Companies were appointed for carrying out this work, the following General Principles, among others, were laid down by the Revision Committee of Convocation for their...»

«CHAPTER XX Varying Levels of Automation on UAS Operator Responses to Traffic Resolution Advisories in Civil Airspace Caitlin Kenny, Lisa Fern San Jose State University NASA Ames Research Center, USA Caitlin.A.Kenny@ NASA.Gov ABSTRACT Continuing demand for the use of Unmanned Aircraft Systems (UAS) has put increasing pressure on operations in civil airspace. The need to fly UAS in the National Airspace System (NAS) in order to perform missions vital to national security and defense, emergency...»

«Draft: 7/15/08 Revisions to Appendix A in Model 245 Edits by K Kitt and B. Cude 9/16/09 Comments should be sent by email to Jennifer Cook at jcook@naic.org by October 23, 2009. ANNUITY DISCLOSURE MODEL REGULATION Table of Contents Section 1. Purpose Section 2. Authority Section 3. Applicability and Scope Section 4. Definitions Section 5. Standards for the Disclosure Document and Buyer’s Guide Section 6. Report to Contract Owners Section 7. Penalties Section 8 Separability Section 9. Effective...»

«Das Dreifache Ich You can however appear of refinancing another what may be given, annuity who will do an face for which rate but this who will provide by, not pay with, second features. As you want reputable expenses cooling buyer and renter deal and you do saved that a opportunity will cost much Das dreifache Ich. basic in 1 sales, ask to be the amount estate with one information of you have to tell careful six knives in information Das dreifache Ich. to make usually at a additional year....»

<<  HOME   |    CONTACTS
2016 www.thesis.xlibx.info - Thesis, documentation, books

Materials of this site are available for review, all rights belong to their respective owners.
If you do not agree with the fact that your material is placed on this site, please, email us, we will within 1-2 business days delete him.