- What are the timings of the programme evaluation?
- What is the programme evaluation? How does it differ from our own self-evaluation?
- What questions is the programme evaluation trying to answer?
- What can Ecorys help us with or advise us on?
- Why should I do a self-evaluation?
- When should I start my evaluation?
- When do you need to receive our self-evaluation?
- How do the evaluation and monitoring fit together?
- Where can I find the toolkit?
- What does the toolkit contain?
- What is peer research?
- How can Ecorys help me to evaluate my project?
- Will there be any more evaluation toolkit events?
- How can we contact Ecorys, and how soon will we hear back from you?
- We didn’t get an opportunity to complete the online survey / our project was only in its early stages, will there be another one?
- We would like to receive a copy of the survey results that we submitted, is this possible?
- Will we be able to see the report sent to Big Lottery?
- How do we become a case study?
- Should I be worried about ethics in research and where can I find out more?
- How do we deal with issues of confidentiality?
- What are soft outcomes and how can I measure them?
- Can we see an example of a self-evaluation?
- We are overstretched as it is, how can we minimize the burden on project staff?
- How do we encourage beneficiaries to get involved in the evaluation?
- How do we overcome ‘research fatigue’ amongst our target group?
What are the timings of the programme evaluation?
Ecorys is evaluating the Reaching Communities programmes in England and Northern Ireland. The evaluation runs for four years from 2007 to 2010. Our final report to Big Lottery will be in December 2010. Each year Ecorys gives feedback to Big Lottery through a number of reports and meetings, and we feedback with a formal annual report which are all published. Each year Ecorys are visiting a selection of projects which will become case studies and we are also doing an annual survey of all projects. In addition we are analysing data held by Big Lottery as well as information sent to us by projects who have completed their own self-evaluation, for example in the form of final evaluation reports. You can send us your self-evaluation at any time and we will include it in the next annual report to Big Lottery.
- Back to top
- What is the programme evaluation? How does it differ from our own self-evaluation?
Ecorys is evaluating the Reaching Communities programmes in England and Northern Ireland at programme level, looking at the programme as a whole. To do this we are using surveys, looking at data held by Big Lottery and doing case study visits to around 40 projects. Self-evaluation is where a project evaluates itself to find out how its doing, what’s working well and what’s not working as well. This could be done by involving your beneficiaries or users, or you could ask an external person to evaluate you, such as a PhD student or a local university. If you do a self-evaluation you can send the findings of this to Ecorys and we will be able to incorporate it into the programme level reports to Big Lottery.
- What questions is the programme evaluation trying to answer?
The programme level evaluation which Ecorys is carrying out aims to answer the following questions, some of which you could look at in your own self-evaluation: - How have projects identified and defined: need, people most in need and hard to reach groups and individuals?
- How have projects considered local strategies to produce evidence of need?
- How have projects involved beneficiaries and local communities in identifying their own needs?
- How have beneficiaries and local communities been involved in project planning and delivery?
- Have projects met the needs of hard to reach groups and the ‘most in need’?
- Have projects and the programme as a whole been effective in reducing disadvantage and exclusion?
- What different approaches have projects used to identify and address need in order to reduce disadvantage and exclusion for hard to reach groups and the most in need?
- What different approaches for addressing need have been used in Northern Ireland and England?
- What can Ecorys help us with or advise us on?
Throughout the evaluation Ecorys is offering a support service to Reaching Communities funded projects to help them to self-evaluate. Ecorys can provide you with guidance about completing your self-evaluation. We are willing to answer questions from you, and comment on elements of your evaluation methodology. For example, we are able to comment on the design of a questionnaire and give you ideas about how to undertake interviews. You can send us an email to reaching.communities@uk.ecorys.com and we will respond within 5 working days. On this website Ecorys have prepared the evaluation toolkit and a range of other tools where you can find information on how to undertake, interpret, report and disseminate your findings. There are tools which you can adapt to use for your own project, such as a soft outcomes survey.
- Improve the planning and management of your project
- Help you to understand how your project is working, and to make changes so that it achieves more
- Involve your beneficiaries
- Help you in securing additional funding, by providing evidence about the success of your project and how you have adapted to any challenges
In addition, by carrying out an evaluation and sending it to ECOTEC, you would be able to contribute to the programme level evaluation that we are carrying out, feeding straight back to Big Lottery. You can send your self-evaluation at any point once you have completed it. Send it to reaching.communities@uk.ecorys.com
You can start your evaluation as soon as your project is up and running, although this is likely to depend very much on the type of services you offer and the nature of your beneficiaries. It is always advisable for you to take some baseline information from your beneficiaries in the early days of the project so that you have something to monitor progress against. For this reason, it can be helpful to have an idea of how you are going to evaluate from an early stage. You can email Ecorys for further advice on this at reaching.communities@uk.ecorys.com
We are happy to receive your evaluations at any point during the course of your project. As we receive them, we will analyse them and then use them to feed into our next annual report to Big Lottery. Sending us your completed self-evaluation provides a good opportunity to showcase your project’s work as part of Ecorys’ wider programme evaluation of Reaching Communities. We are able to receive your reports by email, in pdf version or by post. You can email them to our address at reaching.communities@uk.ecorys.com or by post to Nicola Smith, ECORYS UK Ltd, Quay Place, 92-93 Edward Street, Birmingham, B1 2RA.
How do the evaluation and monitoring fit together?
The evaluation that Ecorys are carrying out is independent from any monitoring that you complete directly for Big Lottery. Evaluation involves making an assessment or judgment of a project based on the information collected. Monitoring allows you to see whether your aims and objectives are being met, whereas evaluation will help you to understand why and how. Whilst the information that you collect for monitoring purposes may be useful to your evaluation, the evaluation enables you to really explore what has worked well / less well and how you can learn from this.
The toolkit and the extra tools to can be downloaded from within this website.
The toolkit contains information on how to get started with your evaluation, including advice on how to structure your evaluation, what you could measure and how to evaluate things like participation. The toolkit also offers practical guidance on doing your evaluation, like how to decide who needs consulting, setting a baseline, setting indicators and how to incorporate any other research you may have done. It also provides an overview of the quantitative and qualitative methods that you may wish to consider and some notes on data protection and ethics. The toolkit offers advice on interpreting your findings and reporting and sharing them.
Peer research is where beneficiaries, users or volunteers of your project are trained in research techniques and help you carry out your evaluation. They could be involved in designing the questions you want to ask, helping to ask the questions or analysing and talking about the findings.Peer research adopts a ‘bottom up’ approach where those individuals who are directly affected by the research play an active role in the process. For more information see our toolkit extras.
How can Ecorys help me to evaluate my project?
We can offer you advice and support through this website and toolkit at http://www.reachingcommunities.org and we are able to answer specific questions, and comment on your approach via email from the address reaching.communities@uk.ecorys.com. ECOTEC are also hold annual evaluation toolkit events, where you will have the opportunity to meet with us directly, hear more about the toolkit and meet other projects. Again, at this event we can talk to you directly and answer specific queries.
Yes, there will be annual toolkit events held in each year of the evaluation. Our first events held in Autumn 2007 were very successful and were held in Birmingham and London. We will be in touch with all projects with the details of future events in due course.
You can contact Ecorys for evaluation advice at reaching.communities@uk.ecorys.com, and you should hear back from us within 5 working days.
Yes, the online survey will be carried out annually, there will be another survey which will run in Autumn 2008. We will notify you when the survey goes live and give you all the relevant information which you need to complete it. We are also able to send out paper copies of the survey by post if you would prefer to complete it that way.
Yes, the programme level evaluation report will be published in March of each year. There will be a link on the Reaching Communities website to direct you to the report. If you have any difficulties in doing this and wish to see the report please contact us after March 30th 2008 by emailing reaching.communities@uk.ecorys.com.
We are selecting 40 case studies across the two programmes (England and Northern Ireland) to reflect the diversity of the programme. We will contact you directly to invite you to become a case study.
When carrying out your research for the evaluation you do need to consider ethical issues. However, it is not something that you should worry about too much – generally a sensitive and common sense approach to research is all that is required. There is a dedicated section of the toolkit for ethics and data protection available on the toolkit page and there is some additional information and a few links to external sources on the links page.
You do need to think about how you are going to ensure confidentiality, especially when dealing with sensitive issues or vulnerable groups. It is good practice to have a system in place to protect the data you collect and the identities of individuals. It is also worthwhile to be able to demonstrate this to those participating in your research to put their minds at rest. Having a well thought through confidentiality policy may help you to recruit participants in the research who may be concerned about the safety of the information they give you. There are a number of measures that you can take, such as keeping names and addresses separate from the information that you receive from individuals, ‘anonymising’ data such as interviews (by removing names and replacing with Miss / Mr X) and using anonymous questionnaires.It is also worth considering the data protection and ethics section of our website.
What are soft outcomes and how can I measure them?
Soft outcomes are those that often cannot be directly counted or quantified. The type of soft outcomes that you might be working towards could be; improved levels of community engagement, changes in attitudes and increases in confidence. Some ‘core’ soft outcomes are:
- Key work skills
- Attitudinal skills
- Personal skills, and
- Practical skills
Soft outcomes can be measured using methods such as interviews, questionnaires, diaries and peer research. For more information about these techniques see our toolkit extras page, and for some specific guidance please see the ‘soft outcomes’ section within the links page.
We do not yet have any published examples of project self-evaluation reports, however you can see other examples of completed evaluations by reviewing the links page.
- Using peer research methods – these methods reduce the workload on project staff by enabling beneficiaries to carry out the research themselves, they do however require work up front to train and brief your peer researchers. Such methods have other methodological advantages as well – they are not just a cop out!
- Choosing less labour intensive research methods, such as handing out surveys during sessions and giving participants research diaries to complete
- Approaching a local university – it may be possible for you to enlist the help of a student to carry out your research for use in a project or final year dissertation.
The work that Reaching Communities funds would link to a range of specialist courses as well as more general degrees like sociology, geography and anthropology.
You may also want to do your research in stages, so for example you could send a survey or questionnaire to the whole of your target group, and then ask at the end whether they would be willing to participate in further, in depth research. From this point you could carry out more research with a group who are particularly willing, but you will have already received some data from a good cross-section of your target group.
In the long term, it is really useful to establish a climate of listening to beneficiaries and acting on suggestions or complaints. This creates an atmosphere where people feel like their views are valued, and as such they may be more likely to take part in your evaluation, because they can see that their contribution is likely to make a difference. You could also offer to distribute / make available a copy of your final report to establish a level of accountability between you and your target research group.
Research fatigue is common amongst many of the groups which are being targeted by Reaching Communities. A key way to overcome this is to make something in your research a little bit different. For example, if you know that your target group are tired of filling out questionnaires, you could try some more interactive techniques such as focus groups or get them to use some drawing and visual techniques. For some ideas about innovative suggestions such as a ‘film strip’ and a ‘dart board’ technique, please refer to the toolkit extras page.
