«Case study, survey, diary and interview research on FCRM volunteering Report – SC120013/R3 We are the Environment Agency. We protect and improve ...»
• use and reflect on the draft evaluation framework developed in Work Package 2 to guide the collection of data This work represents an important step in gathering reliable evidence about FCRM volunteers and how the Environment Agency works with partner organisations and communities. This work package is closely linked to work packages 1 and 2, and will provide evidence for objective 4 (about decision making) and objective 6 (about informing operational guidance).
Work package 3 contributes to objectives 3, 4 and 5 (see Section 1). These include:
• using the evaluation framework developed in work package 2 to understand why people are motivated to get involved in FCRM activities in their communities
• developing a strong and reliable evidence base that explores the effectiveness of involving others in the delivery of FCRM activities
• enabling the Environment Agency to take evidence based decisions on how and when to engage and develop volunteer participation
The research questions guiding this work package were:
• What motivates volunteers to get involved in FCRM volunteering activities and what benefits do they gain from this?
• How is the Environment Agency working with partner organisations and communities in the identified case studies?
• How do different stakeholders (volunteers, Environment Agency staff, partner organisation staff) view the value for money of FCRM volunteering?
• What guidance and support do volunteers and Environment Agency staff identify they need to carry out their activities?
• What are the key challenges and issues raised by volunteers, Environment Agency and partner organisations?
The research team also reflected on the evaluation framework developed in Work Package 2 (Environment Agency 2015b).
Case study and survey research on FCRM volunteering 3 Methods The approach taken was a mixed methods one and included questions concerning the value for money of FCRM volunteering.
The protocols for the questionnaire, interviews and volunteer diary are given in Appendix A of the Work Package 2 report (Environment Agency 2015b).
3.1 Recruitment of volunteers The procedure used to recruit volunteers for the research was designed to satisfy data protection requirements.
To gain access to FCRM volunteers willing to participate in the questionnaire survey, volunteer diaries and interviews, Forest Research asked Environment Agency FCRM staff to send out a letter of invitation to their volunteer contacts. The letter (Appendix A) outlined what the research project was about and asked the recipient if they would take part in the research. Because the Environment Agency staff sent the invitation letters to their volunteers, it was not necessary for them to give details of their volunteers to an external contractor.
Environment Agency staff sent the replies from those volunteers who agreed to become involved to the Environment Agency project manager for the research. She encrypted and password protected the spreadsheet file containing the names and addresses of these volunteers before sending it to Forest Research. Forest Research created a secure folder to contain this file that could only be accessed by the team working on the project and no one else.
Those volunteers who agreed to take part in the research were asked by Forest Research to indicate which research method they would like to get involved in, that is, the questionnaire survey, an interview or a diary. However, many expressed interest in participating in all three though it had been envisaged they would choose just one.
Through this process 87 volunteers agreed to take part in the research and 63 completed the full survey. See Table 3.1 at the end of Section 3 for the number of responses to the different methods used in the research.
3.1.1 Limitations There is uncertainty about the total number of Environment Agency volunteers in England. Research during Work Package 1 using baseline data provided by Environment Agency staff identified that there might be 1,700–3,500 potential volunteers (Environment Agency 2015a). The findings in this report are not therefore representative of all FCRM volunteers.
One drawback of the approach adopted to meet data protection requirements was that it introduced a number of layers, making it less efficient as a means to recruit volunteers in terms of making quick progress.
The aim was to reach as many volunteers as possible through the online survey. But while Environment Agency staff were happy to send out the invitation letter to their volunteers, it became clear that not all staff had sent the letter to all the volunteers they had contact details for. Some staff commented that they had sent the letter to those volunteers they felt were most likely to reply. Although this is perhaps understandable, 4 Case study and survey research on FCRM volunteering it introduced an element of bias into the sample as not all volunteers were given the opportunity to agree or not agree to be involved in the research.
In addition, some Environment Agency staff did not send out the letter of invitation to volunteers at the time requested because they waited until they could send it with other information such as a newsletter. Although this meant delays in recruiting volunteers, not over-burdening volunteers with requests was also an important consideration.
3.2 Approach 1: Survey questionnaire with FCRM volunteers The questionnaire was designed to capture primarily quantitative data on FCRM volunteering from the perspective of volunteers.
SmartSurvey™ was used as, unlike some other online survey instruments, the data produced from its survey are held within the UK. The survey was piloted to try to ensure respondents were clear about what to do. However, it became clear from emails received by Forest Research that a number of the volunteers were not sure if they had submitted the questionnaire correctly. For example, some people used the browser ‘back key’ rather than the survey ‘back key’ to go to an earlier section of the survey which led to data loss from nine potential additional respondents. 1 Although the instructions in the survey told people what to do, it became clear that not all the participants completed the survey as required. It is not possible to know why and how familiar these participants were with completing online surveys.
The figures showing data from the survey are presented in two different ways: using count data and using percentages. It is conventional to use count data (that is, numbers of people/responses) where the sample size is small, particularly where the data are disaggregated to analyse patterns between categories. However, percentage data often provide a clearer indication of the results, particularly when looking for patterns within categories. The sample size and category size is shown in each of the figures so that the analysis by percentage is not misleading.
Although the geographical location of those who completed the survey is known, this information is not used in the analysis. This is because the sampling strategy was not random and is a reflection of the location of those Environment Agency staff able to respond and their volunteer contacts, rather than a representation of the distribution of FCRM volunteers in England.
3.3 Approach 2: Volunteer diaries and interviews The diary for volunteers to complete was designed to capture data on the story of their involvement in volunteering and to capture data on a busy month to understand what types of specific activities being carried out by volunteers.
The original idea had been to ask people to complete a diary in real time over a period of 1–2 months. But because the nature of flood warden activities is sometimes intense and at other times very little, it was decided that asking people to tell us about a busy An analysis of the IP addresses of the respondents to the survey showed that the 24 uncompleted surveys represented repeated attempts to finish the survey by six respondents (three of whom went on to compete the survey and are included in the 63 responses analysed), two people who landed on the page but did not move through the survey questions and four respondents who partially completed the survey. The data from these four respondents were considered in the qualitative analysis.
Case study and survey research on FCRM volunteering month would give more opportunity to understand what actions they take when they are active (see Appendix B). Although adopting this approach and providing an example of what to expect, volunteers found it quite confusing to know what to put in their diary.
Ten flood wardens completed a volunteer diary, of whom five were interviewed by phone to explore in more detail what they had outlined in their diary. We had one reply from a volunteer who said he could not fill the diary in as he had not undertaken any activity even though he became a flood volunteer in 2008.
Also included in this section are the flood warden activities of two flood groups – Bodenham Flood Protection Group and Tregrehan Flood Group – which provided some information of the types of actions they perform. In particular, the Bodenham Group sent reports of their schedule of works from the previous six years.
The flood wardens and two flood groups were from specific areas: Lincolnshire (n = 5), Yorkshire (n = 4), Herefordshire (n = 1), West Sussex (n = 1) and Cornwall (n = 1).
The flood wardens had experienced flooding as far back as 1967 or as recently as 2012, with four people stating that they got involved due to the floods in 2007.
3.4 Approach 3: Value for money questions Cost-effectiveness – value for money (VfM) was explored via the questionnaire and interviews with volunteers, Environment Agency staff and partner organisation staff, and during meetings with the case study groups.
The value for money realised by projects is an important issue when evaluating public sector interventions and assessing future actions. However, cost-effectiveness is very difficult to assess in projects such as those involving volunteering. One reason for this is that there are multiple objectives. Another is the diverse range of achievement or outputs measures that could be monetary (for example, linked to physical environmental improvements) and non-monetary (for example, linked to social impact, risk reduction, individual volunteer and community benefits).
Ex post 2 cost-effectiveness analysis requires the collection and collation of data on:
• total public spend on supporting volunteer programmes
• costs of staff time facilitating volunteer programmes
• consumables, other inputs and overheads
• investment materials
• costs incurred by the volunteers and partner organisations including their time inputs and material inputs in the volunteer schemes and activities These kind of data are then set against measures of impact and additionality to provide a cost per unit of output and outcome through simple division.
However, if the data do not exist as part of regular project and programme monitoring and accounting activity, collecting this additional evidence – particularly from volunteers and partners – is extremely time-consuming and costly. It also often relies on estimates rather than measured variables.
‘Ex post’ and ‘ex ante’ are terms used in evaluation (economic, commercial and so on). Ex ante refers to an assessment before a project/programme/intervention is carried out and ex post refers to an assessment after the project/programme has ended.
6 Case study and survey research on FCRM volunteering Furthermore, if the evidence requires the conversion of non-monetary benefits accrued to be expressed as financial values there is often disagreement among stakeholders about the validity of the approaches taken and how far these can be attributed against key outcomes.
The method that was used to assess value for money employed the widely accepted alternative qualitative method for cost-effectiveness approach called Programme Performance Assessment. This method uses the National Audit Office criteria of Value for Money – relevance, efficacy, efficiency – and generates a short list of programme variables (in the form of questions) under each criterion which are then ranked or scored by different stakeholder groups. This provides a holistic measure of value for money involving multiple criteria and stakeholder perspectives.
The variables used in this research differed slightly according to the stakeholder group concerned. The variables were expressed as statements that stakeholders scored using a Likert response scale. This is a psychometric scale commonly used in questionnaires where respondents are asked to agree or disagree with a number of statements on a seven- or five-point scale, for example, 1 = ‘Strongly disagree’, 2 = ‘Disagree’, 3 = ‘Neither agree or disagree’, 4 = ‘Agree’ and 5 = ‘Strongly agree’.
The data from each of the stakeholder groups were combined. The Likert response scale was quantified using scores between +2 and -2 (‘Strongly agree’ = 2, ‘Agree’ = 1, ‘Neither agree nor disagree’ = 0, ‘Disagree’ = -1 and ‘Strongly disagree’ = -2) and average values generated for comparison across stakeholder groups and across volunteering governance types.
3.5 Approach 4: Case study research Four case studies were made as part of the overall research project and primary data collection. The case study approach was used to gain a holistic view of a particular
case from the perspectives of:
• Environment Agency staff
• staff of partner organisations the Environment Agency was working with Important criteria for case study selection were identified during Work Package 2 (see Appendix C of this report). It was not possible to cover all the criteria for each case study. The four governance types (that is, working directly for Environment Agency, working in partnership, working with others, working for themselves) and the segment types (that is, individual, community, partnership) were therefore selected as key criteria in choosing case studies.
The methods used in each case study included:
• collation of background information from informants and web searches to provide an overview of volunteer efforts and stakeholder roles