Overview:

We work hard and do our best, but how do we know if we are really making difference? Paul Ticher has been one of the two external evaluators for the ICT Hub, and has also evaluated many other ICT projects. This session will look at the different types of evaluation, the kind of questions we ought to be asking, how we might obtain the information we need to answer them, and how we can present the results most effectively.

Session Facilitator:

Paul Ticher (paulticher.com)

Biography:

Paul Ticher has been providing ICT support to the voluntary sector for the past 25 years. He started out at the Community Information Project, a sister organisation to Lasa, where he was the first editor of Computanews. He now concentrates on consultancy, Data Protection, research and evaluation. For the past two years he has been the lead evaluator for the national ICT Hub.

Session Notes:

Paul has been evaluating the Hub for two and a half years. He will be looking at what has worked and what needs more attention when evaluating ICT.

Why are we here?

• How do we know, without feedback, whether we’re doing a good job?
• It would be nice to consider external evaluation – are we doing as good a job as we think we are?
• There is no hard and fast method for evaluation, you will need to decide:
• What do you want to find out, how will you collect it, how will you analyse it? Otherwise you won’t get the useful information.

Whose agenda is the evaluation focusing on?

Is it your own values (providers) or those of an external funder, or is it for your clients (recipients)? The most important thing to consider is: Did it meet the objectives? Focus Quality

• whose definition of quality are you using? (yours or your recipients)
• are you looking back? (did you sort out the problem?) or forward? (what impact can decisions have?)
• Value for money - what are you measuring against (outcomes, numerical targets?)
• Impact - are your outputs short or long-term? which is best?
• how do you measure the effect of what you’re doing?
• will you re-evaluate in a year?
• did you meet the set outcomes?
• Before you start
• Evaluate against objectives – what are you doing???
• Specific measurable objectives
• Plan your data collection

Question: How about advertisers, they don’t always have achievable or realistic targets? They usually say 50% is a success; we can’t go to funders and tell them 50% will work! The value for the funder might be the picture for the annual report not the output itself. Funders are wrong to expect 100% success – if they do they aren’t taking any risks, not using initiative or expanding knowledge. Most are new ideas so will be worth doing and we can build on the knowledge gained, even if it didn’t work as well as expected. If every project was guaranteed to work, what would be the point in evaluating it? Planning your data collection - Frequency. As example, one evaluation per conference or should you have regular intervals? Coverage. Do you evaluate everyone or just a cross-section? How?

• Should you do a questionnaire or interview, usually best for small groups?
• If demographic is large, perhaps best to use sampling - you don’t need a monitoring form.
• Pick a specific week and evaluate users/clients for this period.
• Should you focus on statistics gained internally?
• Interview via email or telephone or in person?
• You will have a better idea of which methodology will work best for your organisation.
• More in-depth information can be gathered through interviewing people. Observation can also be a good method in order to see how people work.
• When using your internal statistics, you will need to know how to extract it.
• Question: Tesco method – collect everything, not everyone has the capacity.
• Data collected can be used for a later project but may not be in an appropriate format – how can we use this information for the next lot of funding?
• What data do we need to collect?
• In order to maintain consistency ask the same questions so data is consistent from beginning to end.
• Not easy to predict what data you will need – the data for the first project will be valuable.
• What you are looking for is what has the project not achieved?
• Collect evidence of need for the next funding bid. If it doesn’t cost anything to collect the data, then collect it.
• What you’re looking for is: where has the project failed….obviously don’t use this exact phrase!
• Make sure data is collected is a useable form.
• Usually you will have all the information; it’s just a case of providing it in a useful format.
• As an example, expenses system - there is a good motivation for accurate data collection. If someone is benefitting then the data will statistically be more accurate. Sometimes data will be based on opinion e.g., how much more do you know about a service than before?

What to find out/record?

• what did you do?
• how much did you spend?
• did the client like it?
• how can it be improved?
• are we satisfied with the outcomes?
• will it make a difference?

How to analyse?

• is the report internal or for funders?
• summarise your data
• commentary
• analysis: internal or external (maybe use comparisons)
• action points (what should we change?) and recommendations
• Evaluation isn’t just something that is used afterwards
• it is ongoing, investigate a situation so it can be sorted.
• There should be continuous improvement otherwise complaints will go up. Asking how something can be improved doesn’t mean something is bad. Perhaps suggest you were expecting something to work in a certain way or in a certain timescale and ask if it did.

How you phrase the question is important. What matters to you? Tools for evaluation You should be designing questions and forms that get the results you need. This shouldn’t be things they don’t have access to, don’t ask people to speculate (i.e. if you hadn’t had our services what would be the result?) Ensure everything is worded clearly – do not be ambiguous. Stick to similar values and format throughout the form but use a combination of tick boxes and open-ended questions and don’t put them off! Evaluation Trust can provide advice on how best to formulate your evaluations. (http://www.evaluationtrust.org.uk) Tick box questions

• be clear, use single response or multiple
• explain the scale ( 1 is bad, 5 is good etc)
• break down complex ideas, do try to explain things - 80:20 rule.
• Include the option for “other” (80% of people will be in 20% of the answers – i.e. 80% of your funding will be from 20% of funders)

Collecting data electronically

• this is easy as people can just email back but doesn’t suit everybody
• expect a lower response rate - this will save on data entry
• there are two main ways: email back a form or complete an online questionnaire (although you may not be able to print this out for your records)

Using Excel

• handles numbers well (formulas etc)
• produces charts
• not that good for multiple answers
• can use pivot tables and cross tabulation
• is not great for text answers (i.e. if you have answered “other” and expanded)

Using Snap (Paul gave a demonstration of how to use it)

• it’s designed for analysis
• you can ask questions and it will email a response
• it can accept electronic data
• one downside is that it is expensive
• also it is not intuitive - it can be exported as a RTF document and edited in Word
• there are different modules for PDAs, internet version and a Word format
• it has preset styles which saves time when designing questionnaires

In order to collate accurate data you must ensure your figures are accurate. Question: What is the point if people hear what they don’t want to hear? Concentrate on your action points. From this we can say what needs to be done and how things can be improved. Future outputs can then be adjusted to allow for the previous learning. Question: Is there anything different about evaluating ICT support? People are scared and think you’re going to ask them what they don’t know and getting the terminology right. If you ask someone “how well is your system working?” they may not have the definitive answer. The basics are the same for evaluation throughout the sector.

Attachment	Size
Evaluating ICT.ppt	35.5 KB