The #5MinResearchPlan on #RAG123 marking

“There are lies, damned lies and statistics (but this doesn’t include educational research)” – Benjamin Disraeli

I always look forward to the blogs of Stephen Tierney (@LeadingLearner) and Ross MorrisonMcGill (@TeacherToolkit). A few weeks ago I read their co-authored blog on educational research and the changing role of the classroom practitioner in terms of undertaking classroom based educational research. Whether you like the 5 minute series or not (and I do because I can look past the 5 minute tag) the idea that teachers should undertake research, implement a change and measure the impact cannot be a bad idea. At the heart of it is a teacher wanting to improve (a growth mindset in action surely?).

The blogs are here (by @leadinglearner) and here (by @teachertoolkit).

The principle is that you identify an issue/concern, you do some research, implement a change, analyse, evaluate and then disseminate. The blog got me thinking because, for all intents and purposes, I was half way through my own mini educational research.

I had identified an issue that I was unhappy with (the quality of my feedback). I had done my research (I had been reading and re-reading a number of blogs on marking- In particular these three blogs on marking every book every lesson: this by @joe_kirby, this by @Katiesarahlou and this by @ListerKev). It is worth mentioning that in the last few weeks @HFletcherWood has written a brilliant blog here on marking very book every lesson which, like Joe’s, Katie’s and Kev’s are well, well worth a read. My reflections on the first 2 weeks of marking every book every lesson (RAG123) are detailed in my blog here.

After reading the #5MinResearchPlan blog I got switched on to the idea of actually measuring the impact that my intervention was having. As a scientist I have always valued evidence and so I started to think about what measures I could make and what analyses I could do. The concept of coming up with an effect size for my intervention was one that certainly whetted my appetite. For information on Hattie and effect sizes read this. I downloaded the #5MinResearchPlan, filled it in and emailed it to Stephen Tierney (and was DELIGHTED when he included it in his blog). The plan is shown here:

I then decided I would make the #5MinResearchPlan my “Penyrheol Tip of the Week” shown here:

My focus then switched to how I would measure the intervention. The first bit of analysis that I did was to compare the formative assessment grades (yes I am aware of the contradiction) from module 1 which was pre RAG123 to module 2 which was post RAG123. To do this I needed to convert the grades to numbers so they could be compared. As soon as I started to do this I became skeptical about the validity of any outcome that this might show. The only variable that I wanted to change was module 1 v module 2 (to represent pre and post RAG123). I was conscious that pupils might have found one of the modules easier than the other. Module 1 was on Forensic Science and module 2 was on Health and Fitness. As modules go the level of demand was pretty similar. However this wasn’t the issue. Each formative assessment was a past paper question worth around 10 marks (they varied). These are then converted to a grade which is recorded for tracking purposes. The grade boundaries were based on professional judgements but by their nature were fairly arbitrary (7/10 for a C on one assessment, 5/8 for a C on another). I was then converting the grades back to a mark (8 for a B, 7 for a C, 6 for a D etc). I then had to average each pupil’s performance for each module. Finally I had to then input the data into the effect size excel spreadsheet and the results were:

Column C are the results for module 1 and column D are the results for module 2.
I was disappointed that the effect size was less than 0.4; I was pleased that the effect size was at least positive at 0.28 (better results in module 2) but the main feeling was that these results just didn’t mean a great deal. I needed something a little more substantial.
In my original plan I then intended to compare module 2 results with my group to module 2 results with a different (but comparable) group taught by another teacher. Again I felt there were too many variables in the mix. The formative assessments can be done in class with open books, as homework or as mini tests and different teachers would set them in different ways. This meant it was not going to be worth comparing results. I really wanted to see how RAG123 had impacted on their understanding and recall at the end of the module. So what to do?

My Year 11 pupils had sat a mock exam in December. This was based only on Module 1. I decided that if I gave my pupils a test on module 2 then I could compare the results and work out the effect size. The mock exam was a 45 minute paper and if I was to set a test of comparable length and question depth I would have something worth comparing. I was so confident that the daily marking, improved dialogue and the DIRT tasks would lead to better learning of the content that I gave the pupils no notice about the test. I gave the pupils 20 minutes to revise and then they had the remaining 40 to sit the test. Pupils had weeks of notice for the mock exam and so should have prepared well (albeit there would be competing revision demands). By contrast they had 20 minutes to prepare for the module 2 test. I had balanced the test so it had a similar range of questions in terms of demand.

I couldn’t wait to mark the tests to see the results. Again I put the data into the excel spreadsheet and the results were as follows:

I was delighted with the results. I know the research sample was very small and I appreciate that there are lots of (and despite trying to control them) variables at play, but……a 0.73 effect size. Fantastic. And that was with a 20 minute revision window. I was confident that the increased effort, better dialogue, DIRT tasks which were immediately checked- all from RAG123- would lead to better learning but I didn’t expect such a large difference. The results do need to be treated with caution (thanks to @LearningSpy for this) but calculating the effect size was hugely preferable to “I just know it works.”

For further evidence of the efficacy of RAG123 marking I decided to do another round of pupil voice. I asked the same questions as before (see previous blog) but this time at the end of the module rather than 2 weeks in. Interestingly I asked pupils how they would feel if I stopped doing RAG123.

The main outcomes of the pupil voice at the end of the 4 week module were:

“I feel it has had a positive impact on my learning in the lesson”- 100% agreed (55% strongly agreed). It was 85% after 2 weeks.

“I feel it has had a positive impact on the effort that I put in during lessons”- 90% agreed (10% said no change).

“I have found the marking dialogue useful”- 100% agreed.

“I want to continue with RAG123”- 100% agreed.

These outcomes speak for themselves.

Ultimately I know that I do not want to go back to how I used to mark and give feedback. RAG123 has made a big difference to my pupils. And I have the data to back that claim up. All that remains is for me to disseminate this information. Which is where this blog comes in. I hope if you are still reading you have found it useful and would consider giving it a try.
All feedback is welcome.

Advertisement