Hello, and thank you for joining me for calibrating and collaborative scoring of student work. By the end of today's tutorial, we will be able to answer the following essential questions. What is calibration and collaborative scoring? Why should I use this? And how can I use this?
So to begin, what exactly is calibration and collaborative scoring? Calibration is the consistent scoring across grade level or subject area. And what this basically means is no matter what teacher you have in a grade level or what teacher you have across a specific subject area, for example science, every teacher will give the same assignment the same score. The way they get to this is through calibration.
A lot of times schools rely on rubric, but it's been found that the rubric alone is not enough to guarantee consistency. So they've created a protocol. So a protocol has been created. And by following this protocol schools acquire equity across graders. One thing that's really important to a calibration session, which we'll talk more about in a minute, is the use of anchor documents. And what this means is an example of A product, an example of a B product, an example of a C product, and so forth.
Another really important aspect of calibration is that it is a collaborative effort, and it can lead to some really meaningful professional development and discussions about curriculum. So why calibrate the scoring of student work? Because it ensures consistency in teacher assessment, and it ensures reliability of the assessment data.
This is particularly important because a lot of schools have many different teachers per grade level. For example, in an elementary school you might have four third grade classrooms. Or in a high school, you might have four English teachers. No matter what teacher that student has, we want to have consistency. So calibration just ensures that there is consistency in teacher assessment.
And it ensures reliability of assessment data. Nowadays, we're really focused on having database decisions in our classroom. And teachers also have to present data to prove that they have actually made a considerable difference in the learning that occurs in the classroom. So we want to just ensure that the data is reliable.
One way to do this is by using a protocol. A protocol allows for scorer input. So just because we have a set rubric, we sit down for calibration session, doesn't mean things can't be changed. So me as the classroom teacher, maybe I didn't create the rubric or maybe I'm not the one leading the calibration session, as part of the protocol, I do have input. So if I don't necessarily agree with something that the rest of the group is talking about or an aspect of rubric, I can say something.
Protocol is also really useful because it has norms and time restrictions. And so it keeps people on task, it keeps the conversation from getting derailed, and it increases scoring accuracy, especially among different teachers of the same grade level or same content area. And protocol allows for conversation, meaningful conversation among professionals. What are we doing right? Are our expectations in line with what our students can really do? So are assessments in line with what we're actually teaching in the classroom?
And this leads to curriculum evaluation. We don't have to keep teaching the same thing year after year after year. If we find that it's not working, if we find that our expectations don't actually meet what is happening in the classroom, then we need to either adjust the curriculum of what is happening in the classroom or adjust our expectations.
So we're going to look at an example. And this example is actually from the Rhode Island Department of Education. They break their protocol into several aspects. So the first aspect we're going to talk about is purpose. So in this case, they're going to calibrate the scoring of student work, and they are also going to talk about the instructional implications of the prompt and/or task. And they're going to look at student work and the rubric that they've established.
So next, they go through the planning and preparation. And this is what the people should know ahead of time before they get to the calibration meeting. It's going to take approximately two to three hours. There's going to be group sizes of about four to eight professionals. The material needed for each person is a prompt or task, the task rubric, the student work, the score sheet, and the score sheet for the recorder. And then there's going to be roles within the group. It's really important to have this planning and preparation outlined before you begin, so time is not wasted on collecting tasks for the people to look at.
And then we get to the process. This is the actual what we are doing during the protocol. So at this point, they're just looking over the prompt or the task. They're silently processing that information. And if there's questions, then those questions are clarified before the grading even begins.
The process continues with the graders or the scorers silently reading and scoring the student work. This is important that this is a silent and individual effort, because we want to make sure we're calibrated. If we're conversing about this before we even read the paper, the calibration session is really going to lose its usefulness. Step five is the score sharing. So one at a time, team members share their score for each of the rubric categories, but they don't give an explanation as to why they chose that score.
Then we have the discussion. So it's important while we're going through this calibration session and through the protocol, that the scores are marking up the student sheet and looking for evidence to support the score that they gave.
Then we debrief. And when you debrief, you want to discuss these specific following questions. What did we notice about scoring student work and using the rubric? What would be the next steps for instructing this student? What revisions should be made to the task and instruction? And what are the implications for our instructional practice?
These are all really important questions. And you do want to ask them for each piece of work, especially that what would be the next steps for instructing this student? That's going to lead us to those conversations about curriculum. What revisions should be made to the task and instruction? It's a curriculum question. What are the implications for our instructional practices? How can we change moving forward?
So this, again, was adapted by Jeri Thompson for the Center of Assessment from Quality Performance Assessment. It's a guide for schools and districts. And this was found at the Ride website. It's R-I-D-E standing for Rhode Island Department of Education. So if you are so inclined, feel free to navigate to that website and check out this protocol in depth.
So how does calibration impact your professional learning community? Its purpose extends to all academic assessments. It doesn't have to just be for the English teachers. It can be for the math teachers. Yes, in math, there are right or wrong answers, but there are still performance assessments that the math teachers give. And the steps to solving the problem might vary. So having calibration among your math department is just as important as having it in a subject area that is actually more subjective like English.
It allows for collaboration. And this just helps constantly increase consistency and accuracy, and also helps us get to those professional educator conversations and feedback sessions. And then the norms of the protocol really provide for equal respect and participation among all the different members. And using protocols does help establish guidelines, and it helps build the skills of the teachers in terms of grading assessments and the culture of these protocol sessions, where the teacher sit down and they have lively discourse with one another that is professional.
So how does this work with technology? You know we do live in a 21st century environment. Our students are constantly using these 21st century skills in classrooms. So we as teachers need to make sure that we are talking the talk. So one way you can do this is by using Google Drive.
So step one, you could upload student papers onto Google Docs and make sure that you're sharing that Google Docs file with all of the different scorers. Then you can use the commenting function on Google Docs to give feedback and opinions on the scores. You can record these scores in Google Forms.
And then when you score the work, you can actually use summary tool from Google Forms. And this is just an online way to go through the protocol. So the protocol is important, but it doesn't mean we always have to have a sit down meeting, because as we know as educators, those aren't always possible. And we don't always have planning periods together. So technology really allows us to follow through with the protocol in every instance even online.
Let's take a moment to reflect. Do you think calibration of grading as possible in your professional learning community? Why or why not? What is one benefit you foresee for your professional learning community if using calibration?
So today, we answered the essential questions of what is calibration and collaborative scoring, why should I use this, and how can I use this? For more information on how to apply what you learned in this video, please view the Additional Resources section that accompanies this video presentation. The Additional Resources section includes hyperlinks useful for applications of the course material, including a brief description of each resource. Thank you for joining me and happy teaching.
Overview
(00:00-00:16) Introduction
(00:17-01:21) Calibration and Collaboration Overview
(01:22-02:09) Why calibrate the scoring of student work?
(02:10-03:20) Why use a protocol?
(03:21-06:06) Example
(06:07-07:07) Calibration and PLC
(07:09-08:04) Technology and Calibration
(08:05-08:27) Reflection
(08-28-08:57) Conclusion
Rhode Island Department of Education: Calibration Protocol for Scoring Student Work
This tool is a step by step approach to calibrating the scoring of student work. The tool provides useful templates to guide the work of your collaborative team.
http://www.ride.ri.gov/Portals/0/Uploads/Documents/Teachers-and-Administrators-Excellent-Educators/Educator-Evaluation/Online-Modules/Calibration_Protocol_for_Scoring_Student_Work.pdf
Calibrating Multiple Graders
This article provides an overview of how to calibrate scoring using a collaborative process involving multiple graders. The article also includes an easy to follow step by step approach under the "How to do it" section.
https://cer.jhu.edu/files/InnovInstruct-BP_CalibratingGraders.pdf
Eliminating Bias in Grading: Teachers Collaborate on Student Assessments
This article explains how using a collaborative scoring process eliminates grader bias. The article also provides a strong rationale for using a calibration protocol.
http://www.allthingsplc.info/mobile/blog/view/97/eliminating-bias-in-grading-teachers-collaborate-on-student-assessments