Evaluation is important in demonstrating the effectiveness of your surf therapy program and contributes towards sustainability, funders love a good evaluation report! This page exists to help you take your program to the next level through offering a step by step guide to basic evaluation for surf therapy practitioners, you DO NOT need to be a professional researcher to follow this framework. If you wish to conduct more complex evaluation or research, we suggest trying to find a student or academic to support you as the additional steps required are not always best use of an organization’s time, especially at the start up stage. Finally, one golden rule to remember is that evaluation exists to support your organization, and not the other way around. A perfectly designed evaluation from a methodological standpoint is of no use if it impacts negatively upon your service delivery. There are 7 sections within this framework to help you through conducting a basic evaluation.
- The Wave Project Scotland 2014, a case study. What does a good evaluation look like?
- Ethics; how do you conduct safe and responsible evaluation?
- Demographics; Who is coming to your program?
- Attendance; How much surf therapy are participants receiving?
- Quantitative Data; How much has surf therapy impacted participants?
- Qualitative Data; In what way has surf therapy impacted participants?
- Putting it all together; How do you write your evaluation report?
The Wave Project Scotland 2014, a case study. What does a good evaluation look like? We thought we would start at the end! Thank you to the Wave Project (www.waveproject.co.uk) for allowing us to highlight their effective use of an evaluation report in successfully developing their service delivery in Scotland. The aim of this is to give you a real-world example of firstly what a good evaluation report looks like and secondly how it can be utilised to develop your program. One thing to note, no researchers were involved in the creation of this report, it was all done through the program staff. To put the report into context, this was an evaluation of the Wave Project pilot in Scotland working with 20 local young people. The report itself was integral in securing significant grant funding enabling a full-time coordinator post where the pilot had been run on a voluntary basis. This in turn enabled the pilot to develop into a much bigger service working with 100 young people a year that is still changing lives in Scotland today. Have a read of the Wave Project Scotland Pilot Evaluation Report here. Now you have seen an example of a successful evaluation let’s break it down into the different things you need to do in order to publish something similar.
Ethics; how do you conduct safe and responsible evaluation? This is the most important part of evaluation and research, and if done poorly can negatively impact on your service delivery. Professional research conducted through an academic institution or research institute goes through extensive ethical screening through an ethical review board. This is not necessarily possible or appropriate for your organization conducting a basic evaluation, however, there are some very important considerations listed below. Informed Consent; you will have had to get consent for participants to take part in your program and the exact same principle exists with regards to evaluation. When collecting your data, you must include a simple explanation of what you are collecting and how it will be used. This can be done effectively in a few lines but this informed consent is mandatory for any kind of data collection. Remember for children this consent must be signed for by the children’s guardians. You can read an example of this below. Thank you to the Jimmy Miller Memorial Foundation (www.jimmymillerfoundation.org) for providing this example from one of their data collection tools. Anonymity; from an ethical standpoint it is always recommended to anonymise participant data within any publicly shared evaluation report to protect their identity and personal wellbeing. It also helps participants to feel at ease participating in the first place. To anonymise you can remove all names, use pseudonyms or numbers. For data provided by other sources such as teachers, doctors or social workers you do not have to anonymise as long as said people have given their permission. Their titles can add weight to the data they share. Examples of this can be seen in the Wave Project Evaluation shared at the start of the framework. We would always recommend where possible getting someone independent to review your procedures from an ethics standpoint. ISTO includes a range of people who could undertake this role for your organization.
Demographics; Who is coming to your program? Demographics are an important basic measure to explore when evaluating your surf therapy program. Demographics allow us to understand who our service is reaching, something as simple as the age can be important for funders who want to fund a particular age bracket. By presenting these within your evaluation you can offer more information to funders and partner organizations about the service that you offer. Demographics can also be very useful for reflection on your service delivery. For example, recording gender could highlight a gender skew in participation due to unexpected barriers. Without recording these demographics such barriers may not have come to light. Below is a by no means exhaustive table of potential participant demographics you could record. Think about which ones are most relevant to your program, it is not necessary to use them all if some are not relevant
| Age | Location |
| Gender | Occupation |
| Ethnicity | Education |
| Referral Organization | Mental/Physical Health Challenges (be aware of ethical considerations when reporting these especially around consent and anonymity, seek advice or don’t use in public reports if unsure). |
Attendance; How much surf therapy are participants receiving? Attendance is another valuable metric that can be easily overlooked. It helps us to explore how much surf therapy participants are receiving in the course of your surf therapy, for comparison to other programmes and starts to explore the concept of effective dosage of surf therapy. In simple terms attendance is also important for funders to demonstrate that you are delivering on what they have funded you to deliver. As with demographics it is also very important in terms of reflecting on your service delivery. Attendance drops and spikes can highlight challenges within service delivery that you may not otherwise have noticed and allow for further exploration to identify such challenges. Please do note drops in attendance are not the end of the world and very common within the majority of the populations that surf therapy seeks to work alongside. Consider such data as a learning experience to better develop your program for the benefit of future participants. You can measure attendance with a simple notebook if that is what works best for yourself or use more high tech approaches. A thank you to Waves for Change (www.waves-for-change.org) who have developed a simple app for measuring attendance which includes a photographic element to introduce further transparency to attendance scores. This app is available for free on the Google Play and Apple stores, just search Waves for Change Coach Assist or follow the links below: Google Play: https://play.google.com/store/apps/details?id=org.wavesforchange.coachassist iTunes: https://apps.apple.com/us/app/coach-assist/id1460129599
Quantitative Data; How much has surf therapy impacted participants? Quantitative data involves putting a numerical value to something you want to measure and for surf therapy evaluation a pre-test and post-test is one of the best ways to do this. This means you take a measurement before participants have done any surf therapy and you take another measurement at the end of surf therapy delivery in order to compare any potential changes. The Wave Project evaluation at the start included a good example of this. There are a large number of measurements in existence but see below for a list of tools that you could use within your evaluation. The ones listed here are validated (scientifically proven to effectively measure what they claim to measure), user friendly and free to access (all below are free but you may have to register to use them, please stick to terms of use). Some are available in multiple languages. Explore through these options to find one that best fits evaluating your surf therapy program.
| Scale | Website | Age Range | Validated Translation | Number of Items | Respondent |
| Children’s Hope Scale | https://academic.oup.com/jpepsy/article/22/3/399/917485 | 8-16 | No | 6 | Participant |
| Child Revised Impact of Event Scale | https://www.childrenandwar.org/projectsresources/measures/ | 8-18 | Yes | 8 | Participant |
| Edinburgh Warwick Personal Wellbeing Scale (full) | https://warwick.ac.uk/fac/sci/med/research/platform/wemwbs/ | 16+ | Yes | 14 | Participant |
| Edinburgh Warwick Personal Wellbeing Scale (short) | https://warwick.ac.uk/fac/sci/med/research/platform/wemwbs/ | 16+ | Yes | 7 | Participant |
| Gratitude Questionnaire | https://ppc.sas.upenn.edu/resources/questionnaires-researchers/gratitude-questionnaire | Does not say | No | 6 | Participant |
| KIDSCREEN 10-Index | https://www.kidscreen.org/english/conditions-of-use/registration/ | 8-18 | Yes | 10 | Participant or Parent (or both) |
| KINDL | https://www.kindl.org/english/information/ | 4-6, 7-13, 14-17 | Yes | Varies | Participant or Parent (or both) |
| New General Self Efficacy Scale | https://sparqtools.org/mobility-measure/new-general-self-efficacy-scale/ | Does not say+ | No | 8 | Participant |
| Rosenberg Self-Esteem Scale | https://www.sralab.org/sites/default/files/2017-06/Rosenberg%20Scale.pdf | Does not say | Yes | 10 | Participant |
| WHO (Five) Wellbeing Index | https://www.psykiatri-regionh.dk/who-5/who-5-questionnaires/Pages/default.aspx | 9+ | Yes | 5 | Participant |
| Youth Quality of Life Instrument (YQOL-S) | http://depts.washington.edu/seaqol/YQOL | 11-18 | No | 8 | Participant |
Once you have selected the scale that best fits your program and registered for use if required, it is important to plan your data collection within your service delivery. Here are a few top tips at the data collection stage:
- Always make sure that you have INFORMED CONSENT
- Plan when you are going to do your pre and post surveys. When you are getting consent for participation is an example of a great time to also collect your pre survey. At the final session of your program is a great time to collect post surveys. These are practical timings to make sure you get as many responses back as possible.
- Make sure the pre and post surveys are filled out by the same person, this is especially true for parent report tools.
- You or mentors can help participants with the survey, especially in understanding the questions, but make sure you are not answering for them.
- Make sure participants understand THERE ARE NO RIGHT OR WRONG ANSWERS. This is not a test we are simply looking for their honest opinions and feelings.
- Make sure to store your data when collected in an organised and secure place.
At this point you will have your pre and post data and you need to do some basic analysis to get to the kind of graphs we saw in the example evaluation report. If you have access to anyone who can use Excel well this will take them five minutes, your scales will have numerical values to each item score and it is a case of putting these into Excel to draw the graphs. See below for a spreadsheet using the WHO (Five) Wellbeing Index as an example (using fictional data). Feel free to use this as a template to adapt for the scale you chose. Now you have quantitative data in the form of pre post graphs to demonstrate changes in participants of your surf therapy program.
Qualitative Data; In what way has surf therapy impacted participants? Qualitative data is non-numerical in nature and helps us to understand how your surf therapy program is impacting on participants. Qualitative data adds depth and context to our quantitative data and together they triangulate to confirm the conclusions of your overall evaluation. There are a huge range of ways to collect qualitative data including but not limited to; interviews, focus groups, open ended questionnaires, drawings and word association. Some of the simplest ways to do this are open ended questions and a questionnaire which led to some of the fantastic qualitative data in the Wave Project case study report. Below is a selection of questions that could be combined with any of the above methods to get some great qualitative data in.
| Can you tell me about surfing with (your programme) | Can you tell me something you have learnt at (your programme) | Can you tell me about anything from (your programme) that you use at home or in the community |
| How do you feel when you are surfing with (your programme) | What is your favourite thing about (your programme) | Have you noticed any changes in yourself since starting (your programme) |
| Can you tell me about the coaches at (your programme) | How do you feel when (your programme) finishes | Can you tell me about what you think about when you are at the beach with (your programme) |
All of these questions could be adapted to different methods, for example, “Can you write 5 words that sum up surfing with (your program),” or “Can you draw a picture to express how you feel at (your program).” It is also important to consider other sources of qualitative data, some of these questions are very easily adapted to parents or referral organisations: “Have you noticed any changes in your child/referred participant since starting (your programme).” We would recommend anonymising participant quotes for ethical reasons but other quotes from professionals can be attributed with the relevant permissions. Quotes can be very powerful in representing the impact of your program but also consider word clouds and pictures. Below is a word cloud shared by Waves of Hope (www.waves-for-hope.org/) a simple word association with the program was put into an online word cloud generator to give this striking visualisation with the most frequently used words in larger font. 
One of these is also utilised in the Wave Project case study. Another interesting form of qualitative data is conducting an in-depth case study that follows a participant through your programme and highlights what is important for them. This is a fantastic addition to an evaluation or works well as a stand alone piece. One of the reasons a case study is so impactful is it takes the reader on an in depth journey and helps them to see the importance of your programme through the eyes of a participant. A great example of this can be seen below courtesy of the Surf Project (https://www.surfproject.nl/). View Example Here. Finally, here are a few top tips to consider when collecting qualitative data.
- Always make sure you have INFORMED CONSENT
- Plan when you are going to ask for qualitative data. Questionnaires are good to send home as while you might not get them all back, you will get much more depth than rushing through at the beach.
- Think hard about who can give you valuable qualitative data including but not limited too; participants, referral organizations, parents, teachers, social workers, case workers, volunteers, doctors.
- Keep questions open ended to avoid simple yes/no answers.
- Keep questions as neutral as possible and avoid leading participants’ answers a certain way. This is known as fishing in evaluation terms and is frowned upon.
- Anonymise participant quotes though stakeholder quotes can be attributed with the relevant permissions.
- Make sure to store your data when collected in an organised and secure place.
- Keep an option for qualitative data at the beach, this could be a notebook or a diary you encourage participants to write whatever they would like at the end of sessions or that you record cool things said in the water.
- Qualitative data can also make great content for social media and other marketing outlets for your program.
Putting it all together; How do you write your evaluation report? Now you have all of your wonderful data together, how on earth do you start writing your report up? Obviously our Wave Project case study provides a fantastic example, the Wave Alliance (www.waves-for-change.org/the-wave-alliance/) have put together a skeleton for evaluation reports that is open source and gives you a great framework to start with. See below and feel free to change bits as much as you require to best fit your organization. View Example Here. Finally, some top tips when writing your report;
- Pictures tell a thousand words, make sure your report has amazing pictures of your program interspersed throughout. The more smiles the better!
- Keep your conclusions simple and accessible to all audiences, funders do not want to be bamboozled by scientific jargon.
- Honestly report challenges you faced and turn this into a positive by reporting how you intend to address these challenges in the future.
- End your report with a clear ask or call to action. This could be the need for further funding to expand the program as was the case in the Wave Project case study, or the need for collaboration with more referral organizations. Whatever you need to sustain/expand your program, the evaluation should be a springboard to achieving this.
Congratulations you now have all the tools to produce a fantastic evaluation report. Make sure to share this with all of your supporters, stakeholders and importantly with ISTO so we can celebrate everyone’s success. Let’s Go Far and Go Together!