Skip to content

The #5MinResearchPlan on #RAG123 marking

February 17, 2014

“There are lies, damned lies and statistics (but this doesn’t include educational research)” – Benjamin Disraeli

I always look forward to the blogs of Stephen Tierney (@LeadingLearner) and Ross MorrisonMcGill (@TeacherToolkit). A few weeks ago I read their co-authored blog on educational research and the changing role of the classroom practitioner in terms of undertaking classroom based educational research. Whether you like the 5 minute series or not (and I do because I can look past the 5 minute tag) the idea that teachers should undertake research, implement a change and measure the impact cannot be a bad idea. At the heart of it is a teacher wanting to improve (a growth mindset in action surely?).

The blogs are here (by @leadinglearner) and here (by @teachertoolkit).

20140217-215841.jpg

The principle is that you identify an issue/concern, you do some research, implement a change, analyse, evaluate and then disseminate. The blog got me thinking because, for all intents and purposes, I was half way through my own mini educational research.

I had identified an issue that I was unhappy with (the quality of my feedback). I had done my research (I had been reading and re-reading a number of blogs on marking- In particular these three blogs on marking every book every lesson: this by @joe_kirby, this by @Katiesarahlou and this by @ListerKev). It is worth mentioning that in the last few weeks @HFletcherWood has written a brilliant blog here on marking very book every lesson which, like Joe’s, Katie’s and Kev’s are well, well worth a read. My reflections on the first 2 weeks of marking every book every lesson (RAG123) are detailed in my blog here.

After reading the #5MinResearchPlan blog I got switched on to the idea of actually measuring the impact that my intervention was having. As a scientist I have always valued evidence and so I started to think about what measures I could make and what analyses I could do. The concept of coming up with an effect size for my intervention was one that certainly whetted my appetite. For information on Hattie and effect sizes read this. I downloaded the #5MinResearchPlan, filled it in and emailed it to Stephen Tierney (and was DELIGHTED when he included it in his blog). The plan is shown here:

20140217-220647.jpg

I then decided I would make the #5MinResearchPlan my “Penyrheol Tip of the Week” shown here:

20140217-220753.jpg
My focus then switched to how I would measure the intervention. The first bit of analysis that I did was to compare the formative assessment grades (yes I am aware of the contradiction) from module 1 which was pre RAG123 to module 2 which was post RAG123. To do this I needed to convert the grades to numbers so they could be compared. As soon as I started to do this I became skeptical about the validity of any outcome that this might show. The only variable that I wanted to change was module 1 v module 2 (to represent pre and post RAG123). I was conscious that pupils might have found one of the modules easier than the other. Module 1 was on Forensic Science and module 2 was on Health and Fitness. As modules go the level of demand was pretty similar. However this wasn’t the issue. Each formative assessment was a past paper question worth around 10 marks (they varied). These are then converted to a grade which is recorded for tracking purposes. The grade boundaries were based on professional judgements but by their nature were fairly arbitrary (7/10 for a C on one assessment, 5/8 for a C on another). I was then converting the grades back to a mark (8 for a B, 7 for a C, 6 for a D etc). I then had to average each pupil’s performance for each module. Finally I had to then input the data into the effect size excel spreadsheet and the results were:

20140217-220929.jpg Column C are the results for module 1 and column D are the results for module 2.
I was disappointed that the effect size was less than 0.4; I was pleased that the effect size was at least positive at 0.28 (better results in module 2) but the main feeling was that these results just didn’t mean a great deal. I needed something a little more substantial.
In my original plan I then intended to compare module 2 results with my group to module 2 results with a different (but comparable) group taught by another teacher. Again I felt there were too many variables in the mix. The formative assessments can be done in class with open books, as homework or as mini tests and different teachers would set them in different ways. This meant it was not going to be worth comparing results. I really wanted to see how RAG123 had impacted on their understanding and recall at the end of the module. So what to do?

My Year 11 pupils had sat a mock exam in December. This was based only on Module 1. I decided that if I gave my pupils a test on module 2 then I could compare the results and work out the effect size. The mock exam was a 45 minute paper and if I was to set a test of comparable length and question depth I would have something worth comparing. I was so confident that the daily marking, improved dialogue and the DIRT tasks would lead to better learning of the content that I gave the pupils no notice about the test. I gave the pupils 20 minutes to revise and then they had the remaining 40 to sit the test. Pupils had weeks of notice for the mock exam and so should have prepared well (albeit there would be competing revision demands). By contrast they had 20 minutes to prepare for the module 2 test. I had balanced the test so it had a similar range of questions in terms of demand.

I couldn’t wait to mark the tests to see the results. Again I put the data into the excel spreadsheet and the results were as follows:

20140217-221203.jpg

20140217-221353.jpg
I was delighted with the results. I know the research sample was very small and I appreciate that there are lots of (and despite trying to control them) variables at play, but……a 0.73 effect size. Fantastic. And that was with a 20 minute revision window. I was confident that the increased effort, better dialogue, DIRT tasks which were immediately checked- all from RAG123- would lead to better learning but I didn’t expect such a large difference. The results do need to be treated with caution (thanks to @LearningSpy for this) but calculating the effect size was hugely preferable to “I just know it works.”

For further evidence of the efficacy of RAG123 marking I decided to do another round of pupil voice. I asked the same questions as before (see previous blog) but this time at the end of the module rather than 2 weeks in. Interestingly I asked pupils how they would feel if I stopped doing RAG123.

20140217-221554.jpg
The main outcomes of the pupil voice at the end of the 4 week module were:

“I feel it has had a positive impact on my learning in the lesson”- 100% agreed (55% strongly agreed). It was 85% after 2 weeks.

“I feel it has had a positive impact on the effort that I put in during lessons”- 90% agreed (10% said no change).

“I have found the marking dialogue useful”- 100% agreed.

“I want to continue with RAG123”- 100% agreed.

20140217-221726.jpg
These outcomes speak for themselves.

Ultimately I know that I do not want to go back to how I used to mark and give feedback. RAG123 has made a big difference to my pupils. And I have the data to back that claim up. All that remains is for me to disseminate this information. Which is where this blog comes in. I hope if you are still reading you have found it useful and would consider giving it a try.
All feedback is welcome.

Advertisements
16 Comments
  1. This is a great blog post and should encourage others to get involved in developing their skills as researchers in the classroom.

    • Thanks for the comment Stephen (and of course for the idea in the first place). I thoroughly enjoyed the whole process- from starting the research, implementing, measuring, analysing and finally blogging. I hope the blog demonstrates that undertaking the action research has benefited my students. Thanks again,
      Damian

  2. Kev Lister permalink

    Another great post. I’ve been looking at whether we have any data around us that can be analysed in a similar way to provide further evidence. So far I can’t separate other variables we’ll enough, but I’ve not given up looking! So pleased that #rag123 is helping your practice and your students. Keep it up and keep the blogs coming đŸ˜‰

    • Thanks for the feedback Kev. I hope some of the data helps to convince more people to give it a whirl. My pupils value it and I actually enjoy marking their work. And there has been a tangible change in effort and performance. As the father of RAG123 I hope this contents of this blog does “your baby” justice.
      Damian

  3. Super. Love it and pleased with the result. I guess you need to do a further check with someone not so keen on RAG123. That would be interesting.

    • Thanks so much for the feedback Peter. I’m looking forward to seeing how my group’s final exam results compare to similar classes in school. Again lots of variables at play but might add to the picture.
      Thanks for taking the time to read and comment (and tweet too). Like everyone that has commented so far, these are opinions that I value HUGELY.
      Damian

  4. Really interesting blog. This kind of action-research approach is what will make our job successful. Try something, test and evaluate it, refine and improve for maximum effect. A great model. Thanks for sharing.

    • My absolute pleasure Chris. Thanks for taking the time to read and respond. I have genuinely enjoyed the whole experience and feel my classes have benefited as a result of the change. Hopefully it shows that action research is worthwhile and well within our grasp as a mechanism for improvement. Thanks again,
      Damian

  5. Jill Berry permalink

    This is what education blogging should all be about, Damian! Many thanks for sharing your experiences in such a helpful and constructive way.

    • It’s an absolute pleasure Jill. Many, many thanks for taking the time to read ( I know you are asked to read a zillion blogs) and for leaving such a positive and supportive comment. Your opinion is hugely valued.
      Damian

Trackbacks & Pingbacks

  1. #5MinResearchPlan by @LeadingLearner and @TeacherToolkit | @LeadingLearner
  2. Playing left-handed: How do I improve as a teacher? | Improving Teaching
  3. QuickKey, multiple choice and hinge questions. | mrbenney
  4. #NTENRED Workshop: Driving Research through the Department | meridianvale
  5. What happens to work that the teacher doesn’t mark? | mrbenney

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

%d bloggers like this: