«Comparative Study and ‘Outcome and Impact’ Analysis of Six Vocational Training Projects in West Africa Synthesis report based on six case ...»
The main method for data collection was structured personal interviewing using the interview guide provided by FAKT. The interview guide was used without any modification. One partner (OICG) used phone interviews for about 10% of the sample. In addition partners and consultants documented case studies.
Especially in rural areas the tracing of graduates was a time- and energy consuming process, which involved in some cases extensive travelling. In other cases, when contact data was available and up-to-date, respondents were called to a centre for individual interviewing.
Three organisations conducted the tracer study without assistance of external consultants:
VTF, SLOIC and OICG
• VTF is not a training provider but an organisation that is engaged in networking and capacity building of training providers (VTI). VTF has initiated a process and trained staff of the VTIs and other training providers to regularly carry out tracer studies. In this case a local consultant introduced the VTF personnel and staff appointed by VTF partners in the methods for data collection and data compilation. The actual data collection was done by staff of the VTI, in some cases the task was executed by persons who were not trained.
• SLOIC conducted the tracer study with the teaching staff in Bo and Mattru Jong. The staff had not received training in carrying out surveys of this kind.
• In the case of OICG, the tracer study was conducted by project personnel who are responsible for post training follow up. OICG has experiences in conducting tracer studies.
In the cases of LOIC, MTS and YOWDAST, the tracer study was coordinated by local consultants. In the case of LOIC, all the data collection was done by an external team. In Nigeria, both partners were supported by the consultant in preparing the study, in identification of graduates and in data collection. About 50% of the interviews were done by the consultant, the others by staff.
1.3.2 Compilation of tracer data and analysis of results Compilation of the raw data per trade was done by the EED partners or by the local consultant on behalf of the partner. The raw data (questionnaires) remained with the partners. Final compilation and pre-analysis of data per partner was done by FAKT.
A review of the preliminary tracer study findings took place in the beginning of March 2009 at EED. Afterwards, two workshop sessions were organised by VTF and OICG in Ghana for discussion of tracer study results and clarification of open issues. In Liberia, Sierra Leone and Nigeria, the review of the tracer study data was part of the main study. In Nigeria, the review included a post-screening of the raw data for in-depth analysis of findings per trade and identification of additional case studies. Prior to the field studies consultants formulated hypothesis and open questions on the basis of the tracer study results.
1.3.3 Main study The main study consisted of the six case studies carried out by different teams of consultants.
For the results of each study see the case study reports. Each case study took about 3-4 days exclusive travelling. In total it lasted 42 days.
The six case studies (here called “main study”) were organised in the following sequence:
Methodologies used The consultants applied a range of methodologies in order to collect additional information
on impact, cross check with results of the tracer study and verify information:
• Use of pre-study information of each centre visited and quantitative results of tracer study
• Semi-structured interviews with management staff and trainers/ teachers and/ or field workers of partner organisations
• Semi-structured and open interviews with resource persons (government, collaborating institutions)
• Intensive focus group discussions with graduates and present trainees, including
methods such as:
1. Most significant change
2. Story telling
3. Scoring and ranking
• Individual interviewing of graduates, case studies and photo-documentation
• Observation of workshop sites and training venues
• Where possible, observation of classroom teaching
• Self evaluation of training measures (with graduates and trainees) using scales
• Workshop sessions and informal discussion of results with partner For structuring the data collection process, the consultants used a study checklist (see study tools, annex 3).
Results of focus groups discussions and self assessments and the complete results of the tracer studies were documented in the annex of the case study reports. The teams of consultants drafted their reports individually with inputs of the local/ regional consultants. Draft reports were sent to partners for comments. Some comments have still to be received (status 13th July, 2009).
1.4 Limitations and validity of the data collected This comparative study was not designed as a scientific impact study, which would require a different design, a longer time frame and subsequently more resources. Furthermore, any study designed to meet scientific standards would have to deal with the same common challenges on the ground such as mobility of graduates and missing baseline data.
As said above, random selection for respondents of tracer studies and focus groups was hardly
possible. This limits the validity of the quantitative data because:
• labour migrants are not equally covered (majority supposedly being male)
• in some cases selection by partners may have favoured the more successful cases
Other limiting factors:
• Lack of baseline information (four of the six partners do not keep records about the percentage of trainees who are/ have already been working and generating income before training) and lack of reliable regular tracer data (regular monitoring results) for cross checking and trend analysis
• Some of the questions may have been difficult to understand for the interviewer and/ or the respondent. In some cases interviewers were not properly introduced to the task so that some questions have been misunderstood
• Because of the time pressures to implement the tracer studies and the distances involved it was not feasible to test the interview guide in the field
• The short time available for every partner during the main study (3-4 days max.) Several projects provided (or used to provide) incentives to graduates (tool kits, equipment subsidies), which may have influenced responses.
Results of the tracer study have been cross checked with results of focus groups discussions (FGD) and observations. In some cases (VTF, OICG and YOWDAST), results of this study could be cross-checked with tracer statistics conducted by the partner organisation. The cross checking made it possible to come up with fairly consolidated findings on effectiveness, in particular on the utilisation of skills learned and the actual employment status.
It has been more challenging to measure the outcomes and impacts on the “higher end” of the
impact chain, i.e.:
a. degree of change of income as a result of the skills training b. degree of coverage of basic needs with the income earned c. improved social status of an individual d. indirect impact of VT on communities e. indirect impact on gender roles f. indirect impact on peace and conflict
Berufsbildung | Evaluierung
Findings from the tracer study on (a.) and (b.) are subjective and influenced by expectations of individuals, the ways questions were asked by the different interviewers and the analysis by partners. Wherever possible, results have been cross-checked with FGD, although the sample size of FGD was significantly lower than that of the tracer study.
Impacts (c.) to (e.) have been addressed by FGD and case studies. Respondents were beneficiaries and internal and external resource persons. Case studies and FGD results give information on changes in social status from the point of view of beneficiaries but again, results are influenced by subjectivity of the respondents. Information on (d.) to (f.) was additionally collected from interviews with internal and external resource persons with or without affiliation to the project. No systematic cross-checking of results was possible with community groups or parents (indirect beneficiaries). Unlike community development programmes (which deal with groups of people in one specific community), vocational training institutions and projects target individuals sometimes from a wide catchment area.
Imagine assessing the impact of a VT programme on communities in a city of 1.5 Million or of a region covering more than 100.000 km² (example MTS), which may sound a bit like finding a drop of water in the ocean.
1.5 Team of consultants and coordination of the study
In total, five consultants were involved in the different stages of the study. James Yarsiah and his team of young Liberian consultants conducted the tracer study in Liberia. Rehab David carried out the two tracer studies in Nigeria together with MTS and YOWDAST. Both consultants took part in the main study. Adom Ghartey, a TVET consultant from Ghana, supported VTF in conceptualising the tracer study with their partner VTIs and facilitated two tracer study review workshops with OICG and VTF. Dr. Gisela Burckhardt (freelance consultant) and Ralf Lange (FAKT) were the international consultants. Matija Maturanec of FAKT coordinated the tracer study and compiled the tracer results. Overall coordination of the implementation of the study was the task of FAKT2. Hedwig Schlags of the evaluation desk of EED has been overseeing the study process on behalf of EED. The regional desks of the Africa 1 department of EED were closely involved in the design of the ToR, communication with partners, and discussion of and feedback to study results.
FAKT is a non-profit consultancy firm affiliated with protestant church development work, based in Germany. FAKT has been engaged in the dialogue on VT with AG-KED partners in West Africa since the mid-nineties.
The case studies where carried out by the following teams:
Liberia Opportunities Industrialisation Centres (visit of 2 centres) Dr. Gisela Burckhardt and James Yarsiah Sierra Leone Opportunities Industrialisation Centres (visit of 2 centres) Vocational Training Programme for Females, Ghana (visit of 5 centres) Opportunity Industrialisation Centre Ghana, Livelihood Ralf Lange and James Yarsiah Enhancement Project Kumasi EYN Masons Technical School, Garkida Nigeria Ralf Lange and Rehab David Youth and Women Development Association for Social Transformation, Ganye Nigeria Berufsbildung | Evaluierung 2 Context
2.1 Background of partners and VT projects/ programmes This section describes in brief the background of the EED partners studied, the VT approaches implemented and the project/ programme 2.1.1 Overview For all six partners, Vocational Training is the main activity. For four partners, EED is the only donor, the other two (OICG and SLOIC) receive funding for their core operations from their respective governments. The table below provides an overview of the history of the partner-ship, the VT approaches implemented and the project/ programme outputs.
2.1.2 Liberia Opportunity Industrialisation Centres (LOIC) LOIC was established in 1977. Its mission is to “assist with Liberia’s post-war recovery, reconstruction and development through human capacity building, social service delivery and rehabilitation of war-affected individuals”3 Since 1992 LOIC efforts were mainly concentrated on the rehabilitation and resettlement of war affected youth and ex-combatants. With progress made in rehabilitation and resettlement, this focus has now changed. Most of the former international donors have phased out their support since the end of the 90s. EED provides funding for non-formal vocational training in three satellite centres. All of these centres are situated in rural towns; two of them were visited by the evaluation team (Gbarnga, Sinje).
The training approach is centre-based, non-formal vocational training. Before the war, the training lasted 18 months. After the war, it was shortened to six months in order to cater for the immediate training needs of ex-combatants for demobilisation. Since last year, the training has been extended to nine months. Courses are offered in the following trade areas: carpentry, masonry, metal works, small engine repair, tailoring, tie dye, pastry and agriculture. At the end of each training cycle, the program offers a training package, referred to as ‘resettlement package’ which comprises tools, materials or supplies, depending on the particular trade area.
At present, there are no internship programs and no job placements undertaken by LOIC.
2.1.3 Sierra Leone Opportunities Industrialization Centres (SLOIC)
SLOIC was established in 1977, the town of Bo became the first training site. SLOIC is focusing on underprivileged/ marginalized sectors of communities, mainly unskilled and unemployed youths, by providing life and vocational skills training to facilitate job creation for employment and self reliance for personal and community development.
SLOIC manages four training centres, namely Bo (BVTC 1976), Freetown/ Lungi (FVTC 1996; Lungi 2005), Makeni (MVTC 1990) and Mattru Jong (MJTC 1993); the Lungi centre was created after the centre in Freetown was closed down. There are satellite stations attached to some of the centres in Makeni and Mattru Jong which are catering to outlying communities. Major donors have been EED and USAID, the government of Sierra Leone provides subsidies to staff salaries. The evaluation team visited the centres Bo and Mattru Jong.
The training approach is centre-based, non-formal vocational training. In addition, SLOIC has facilities for small business training. Eight trade areas were initially started: electricity, carpentry, masonry, auto mechanics, metal works, agriculture, and home management.