OAE Pedagogy: Assessment, Instruction, Progress
Thank you Kim Stephens for consolidating this information.
Formative Assessment
The goal of formative assessment is to monitor student learning to provide ongoing feedback that can be used by instructors to improve their teaching and by students to improve their learning. More specifically, formative assessments:
Summative assessment
The goal of summative assessment is to evaluate student learning at the end of an instructional unit by comparing it against some standard or benchmark.
Summative assessments are often high stakes, which means that they have a high point value. Examples of summative assessments include:
Formal Assessment
Formal assessment involves the traditional assignments (such as exams and papers) that instructors often use to derive a grade for a student.
Informal Assessment
Among the most widely used informal assessments are Classroom Assessment Techniques, or CATs. These are simple, non-graded, anonymous, in-class activities that give both you and your students useful feedback on the teaching-learning process. Another informal assessment strategy involves the use of classroom response systems or clickers, which can give both you and your students' feedback about how well students understand specific course concepts.
Authentic
Authentic assessment (AA) springs from the following reasoning and practice:
If I were a golf instructor and I taught the skills required to perform well, I would not assess my students' performance by giving them a multiple choice test. I would put them out on the golf course and ask them to perform. Although this is obvious with athletic skills, it is also true for academic subjects. We can teach students how to do math, do history and do science, not just know them. Then, to assess what our students had learned, we can ask students to perform tasks that "replicate the challenges" faced by those using mathematics, doing history or conducting scientific investigation.
Pretest and Post test- diagnostic
Pretest/post test assessment is a method used by academic units where locally developed tests and examinations are administered at the beginning and at the end of courses or academic programs. These test results enable faculty to monitor student progression and learning throughout prescribed periods of time. The results are often useful for determining where skills and knowledge deficiencies exist and most frequently develop.
How to assess prior knowledge
In order to gauge how much students have learned, it is not enough to assess their knowledge and skills at the end of the course or program. We also need to find out what they know coming in so that we can identify more specifically the knowledge and skills they have gained during the course or program.
You can choose from a variety of methods to assess your students’ prior knowledge and skills. Some methods (e.g., portfolios, pre-tests, auditions) are direct measures of students’ capabilities entering a course or program. Other methods (e.g., students’ self-reports, inventories of prior courses or experiences) are indirect measures.
Performance-Based Prior Knowledge Assessments
The most reliable way to assess students’ prior knowledge is to assign a task (e.g., quiz, paper) that gauges their relevant background knowledge.
These assessments are for diagnostic purposes only, and they should not be graded. They can help you gain an overview of students’ preparedness, identify areas of weakness, and adjust the pace of the course.
To create a performance-based prior knowledge assessment, you should begin by identifying the background knowledge and skills that students will need to succeed in your class. Your assessment can include tasks or questions that test students’ capabilities in these areas.
Prior Knowledge Self-Assessments
Prior knowledge self-assessments ask students to reflect and comment on their level of knowledge and skill across a range of items. Questions can focus on knowledge, skills, or experiences that:
The advantage of a self-assessment is that it is relatively easy to construct and score. The potential disadvantage of this method is that students may not be able to accurately assess their abilities. However, accuracy improves when the response options clearly differentiate both types and levels of knowledge.
Writing Appropriate Questions for Self-Assessments
Writing appropriate questions for prior knowledge self-assessments can seem daunting at first. Identifying specific terms, concepts, or applications of skills to ask about will help you write effective questions.
Examples of questions with possible closed responses:
How familiar are you with "Karnaugh maps"?
Have you designed or built a digital logic circuit?
How familiar are you with a "t-test"?
How familiar are you with Photoshop?
Data driven teaching
It is the use of quantifiable data obtained from Observable and measurable goals set by an educator in order to determine if the student is either improving his academic skills remaining the same or regressing in academics. The use of data driven instruction serves to guide the instructor in determining the students next step after mastering a concept or guide the instructor in determining if he/she needs to modify his /her instructional methods to provide the student with a better understanding of academic concepts.
Several requirements are necessary to achieve good data-driven instruction:
Gathering data
the process of gathering and measuring information on variables of interest, in an established systematic fashion that enables one to answer stated research questions, test hypotheses, and evaluate outcomes. The data collection component of research is common to all fields of study including physical and social sciences, humanities, business, etc. While methods vary by discipline, the emphasis on ensuring accurate and honest collection remains the same.
Assessments or Grades
Assessment and grading are not the same.
Generally, the goal of grading is to evaluate individual students’ learning and performance. Although grades are sometimes treated as a proxy for student learning, they are not always a reliable measure. Moreover, they may incorporate criteria – such as attendance, participation, and effort – that are not direct measures of learning.
The goal of assessment is to improve student learning. Although grading can play a role in assessment, assessment also involves many ungraded measures of student learning (such as concept maps and CATS). Moreover, assessment goes beyond grading by systematically examining patterns of student learning across courses and programs and using this information to improve educational practices.
Assessment tools
Below are links to assessment tools and techniques along with specific geoscience examples and resources.
Creating and Using Rubrics
Benefiting from Rubrics
A carefully designed rubric can offer a number of benefits to instructors. Rubrics help instructors to:
An effective rubric can also offer several important benefits to students. Rubrics help students to:
Holistic grading
As previously noted, holistic scoring gives students a single, overall assessment score for the paper as a whole. Although the scoring rubric for holistic scoring will lay out specific criteria just as the rubric for analytic scoring does, readers do not assign a score for each criterion in holistic scoring. Rather, as they read, they balance strengths and weaknesses among the various criteria to arrive at an overall assessment of success or effectiveness of a paper.
A reader writes nothing on the paper itself and assigns the holistic score after reading the paper carefully and completely. A second reader, who does not see the first score, independently reads and assigns a second holistic score. If the two scores differ by more than 2 points, then a third reader scores the paper as well. Inter-rater reliability (the percentage of papers given the same score or differing by one point) should fall between .85 and .90 for sound holistic scoring. Readers who read the same kinds of papers regularly (including students in a large class) can easily be trained to reach acceptable inter-rater reliability scores.
AP exams and the SAT II writing test both use holistic scoring to assess student writing skills.
The goal of formative assessment is to monitor student learning to provide ongoing feedback that can be used by instructors to improve their teaching and by students to improve their learning. More specifically, formative assessments:
- help students identify their strengths and weaknesses and target areas that need work
- help faculty recognize where students are struggling and address problems immediately
- draw a concept map in class to represent their understanding of a topic
- submit one or two sentences identifying the main point of a lecture
- turn in a research proposal for early feedback
Summative assessment
The goal of summative assessment is to evaluate student learning at the end of an instructional unit by comparing it against some standard or benchmark.
Summative assessments are often high stakes, which means that they have a high point value. Examples of summative assessments include:
- a midterm exam
- a final project
- a paper
- a senior recital
Formal Assessment
Formal assessment involves the traditional assignments (such as exams and papers) that instructors often use to derive a grade for a student.
Informal Assessment
Among the most widely used informal assessments are Classroom Assessment Techniques, or CATs. These are simple, non-graded, anonymous, in-class activities that give both you and your students useful feedback on the teaching-learning process. Another informal assessment strategy involves the use of classroom response systems or clickers, which can give both you and your students' feedback about how well students understand specific course concepts.
Authentic
Authentic assessment (AA) springs from the following reasoning and practice:
- A school's mission is to develop productive citizens.
- To be a productive citizen, an individual must be capable of performing meaningful tasks in the real world.
- Therefore, schools must help students become proficient at performing the tasks they will encounter when they graduate.
- To determine if it is successful, the school must then ask students to perform meaningful tasks that replicate real world challenges to see if students are capable of doing so.
If I were a golf instructor and I taught the skills required to perform well, I would not assess my students' performance by giving them a multiple choice test. I would put them out on the golf course and ask them to perform. Although this is obvious with athletic skills, it is also true for academic subjects. We can teach students how to do math, do history and do science, not just know them. Then, to assess what our students had learned, we can ask students to perform tasks that "replicate the challenges" faced by those using mathematics, doing history or conducting scientific investigation.
Pretest and Post test- diagnostic
Pretest/post test assessment is a method used by academic units where locally developed tests and examinations are administered at the beginning and at the end of courses or academic programs. These test results enable faculty to monitor student progression and learning throughout prescribed periods of time. The results are often useful for determining where skills and knowledge deficiencies exist and most frequently develop.
How to assess prior knowledge
In order to gauge how much students have learned, it is not enough to assess their knowledge and skills at the end of the course or program. We also need to find out what they know coming in so that we can identify more specifically the knowledge and skills they have gained during the course or program.
You can choose from a variety of methods to assess your students’ prior knowledge and skills. Some methods (e.g., portfolios, pre-tests, auditions) are direct measures of students’ capabilities entering a course or program. Other methods (e.g., students’ self-reports, inventories of prior courses or experiences) are indirect measures.
Performance-Based Prior Knowledge Assessments
The most reliable way to assess students’ prior knowledge is to assign a task (e.g., quiz, paper) that gauges their relevant background knowledge.
These assessments are for diagnostic purposes only, and they should not be graded. They can help you gain an overview of students’ preparedness, identify areas of weakness, and adjust the pace of the course.
To create a performance-based prior knowledge assessment, you should begin by identifying the background knowledge and skills that students will need to succeed in your class. Your assessment can include tasks or questions that test students’ capabilities in these areas.
Prior Knowledge Self-Assessments
Prior knowledge self-assessments ask students to reflect and comment on their level of knowledge and skill across a range of items. Questions can focus on knowledge, skills, or experiences that:
- you assume students have acquired and are prerequisites to your course
- you believe are valuable but not essential to the course
- you plan to address in the course
The advantage of a self-assessment is that it is relatively easy to construct and score. The potential disadvantage of this method is that students may not be able to accurately assess their abilities. However, accuracy improves when the response options clearly differentiate both types and levels of knowledge.
Writing Appropriate Questions for Self-Assessments
Writing appropriate questions for prior knowledge self-assessments can seem daunting at first. Identifying specific terms, concepts, or applications of skills to ask about will help you write effective questions.
Examples of questions with possible closed responses:
How familiar are you with "Karnaugh maps"?
- I have never heard of them or I have heard of them but don't know what they are.
- I have some idea what they are, but don't know when or how to use them.
- I have a clear idea what they are, but haven't used them.
- I can explain what they are and what they do, and I have used them.
Have you designed or built a digital logic circuit?
- I have neither designed nor built one.
- I have designed one, but not built one.
- I have built one, but not designed one.
- I have both designed and built one.
How familiar are you with a "t-test"?
- I have never heard of it.
- I have heard of it, but don't know what it is.
- I have some idea of what it is, but it’s not very clear.
- I know what it is and could explain what it's used for.
- I know what it is and when to use it, and I could use it to analyze data.
How familiar are you with Photoshop?
- I have never used it or I tried using it but couldn't do anything with it.
- I can do simple edits using preset options to manipulate single images (e.g., standard color, orientation and size manipulations).
- I can manipulate multiple images using preset editing features to create desired effects.
- I can easily use precision editing tools to manipulate multiple images for professional quality output.
Data driven teaching
It is the use of quantifiable data obtained from Observable and measurable goals set by an educator in order to determine if the student is either improving his academic skills remaining the same or regressing in academics. The use of data driven instruction serves to guide the instructor in determining the students next step after mastering a concept or guide the instructor in determining if he/she needs to modify his /her instructional methods to provide the student with a better understanding of academic concepts.
Several requirements are necessary to achieve good data-driven instruction:
- Baseline data that gives a good sense of where students are at the beginning of the year; these data often come from the prior year’s state test because schools are held accountable by such tests.
- Clear goals for what students are expected to learn and to achieve; these goals are usually related to state standards and grade-level expectations. Goals may also be specific to improved performance on the state test, for example, raising the percentage of students scoring Proficient or higher in mathematics from 67% last year to 84% this year.
- Regular assessments across the school year; frequent assessments provide multiple pieces of evidence about student knowledge and skills. Such assessments help to benchmark students’ progress across the school year.
- Well-focused and well-planned instruction that is based on evidence; these data show what students know and are able to do and what they still need to learn.
Gathering data
the process of gathering and measuring information on variables of interest, in an established systematic fashion that enables one to answer stated research questions, test hypotheses, and evaluate outcomes. The data collection component of research is common to all fields of study including physical and social sciences, humanities, business, etc. While methods vary by discipline, the emphasis on ensuring accurate and honest collection remains the same.
Assessments or Grades
Assessment and grading are not the same.
Generally, the goal of grading is to evaluate individual students’ learning and performance. Although grades are sometimes treated as a proxy for student learning, they are not always a reliable measure. Moreover, they may incorporate criteria – such as attendance, participation, and effort – that are not direct measures of learning.
The goal of assessment is to improve student learning. Although grading can play a role in assessment, assessment also involves many ungraded measures of student learning (such as concept maps and CATS). Moreover, assessment goes beyond grading by systematically examining patterns of student learning across courses and programs and using this information to improve educational practices.
Assessment tools
Below are links to assessment tools and techniques along with specific geoscience examples and resources.
- Concept Maps - A concept map is a diagram with hierarchical nodes, labeled with concepts. The nodes are linked together with directional lines and are arranged from general to specific.
- ConcepTests - Conceptual multiple-choice questions that are useful in large classes. A ConcepTest is a technique used often in a lecture setting. The instructor presents one or more questions during class along with several possible answers. Students in the class indicate which answer they think is correct. This could be done by a show of hands, for example. If many of the students do not give the correct answer, students are given a short time in lecture to try to persuade their neighbor that their answer is correct. The question is asked a second time to gauge class mastery. The technique can be particularly useful in large classes.
- Knowledge Survey - consist of a series of questions that cover the full content of a course. The surveys evaluate student learning and content mastery at all levels: from basic knowledge and comprehension through higher levels of thinking. Knowledge surveys can serve as both formative and summative assessment tools. They help students learn, help faculty improve their classrooms, and aid departments and programs as the explore new curricula or pedagogies
- Exams
- Oral Presentations - Oral presentations are often used to assess student learning from student individual and group research projects.
- Poster Presentations - Poster presentations are often used to assess student learning from student individual and group research projects and often used at scientific conferences.
- Peer Review - Having students assess themselves and each other.
- Portfolios - Portfolios are personalized long-term documentation of student mastery of course material. One essential element of portfolios is student reflection on their own learning and progression towards the mastery of the material documented in the portfolio. As such, portfolios are windows on the metacognitive process of students.
- Rubrics -Rubrics are written criteria that details expectations of what students will need to know and be able to do in order to receive a given grade. Rubrics help instructors to develop clear learning objectives for their students and if provided to students prior to the activity, serve to guide their efforts.
- Written Reports - Written reports are a classic assessment used by faculty. Written reports may be as short as a one-minute paper and as long as a term paper.
Creating and Using Rubrics
- A rubric is a scoring tool that explicitly describes the instructor’s performance expectations for an assignment or piece of work. A rubric identifies:
- criteria: the aspects of performance (e.g., argument, evidence, clarity) that will be assessed
- descriptors: the characteristics associated with each dimension (e.g., argument is demonstrable and original, evidence is diverse and compelling)
- performance levels: a rating scale that identifies students’ level of mastery within each criterion
Benefiting from Rubrics
A carefully designed rubric can offer a number of benefits to instructors. Rubrics help instructors to:
- reduce the time spent grading by allowing instructors to refer to a substantive description without writing long comments
- help instructors more clearly identify strengths and weaknesses across an entire class and adjust their instruction appropriately
- help to ensure consistency across time and across graders
- reduce the uncertainty which can accompany grading
- discourage complaints about grades
An effective rubric can also offer several important benefits to students. Rubrics help students to:
- understand instructors’ expectations and standards
- use instructor feedback to improve their performance
- monitor and assess their progress as they work towards clearly indicated goals
- recognize their strengths and weaknesses and direct their efforts accordingly
Holistic grading
As previously noted, holistic scoring gives students a single, overall assessment score for the paper as a whole. Although the scoring rubric for holistic scoring will lay out specific criteria just as the rubric for analytic scoring does, readers do not assign a score for each criterion in holistic scoring. Rather, as they read, they balance strengths and weaknesses among the various criteria to arrive at an overall assessment of success or effectiveness of a paper.
A reader writes nothing on the paper itself and assigns the holistic score after reading the paper carefully and completely. A second reader, who does not see the first score, independently reads and assigns a second holistic score. If the two scores differ by more than 2 points, then a third reader scores the paper as well. Inter-rater reliability (the percentage of papers given the same score or differing by one point) should fall between .85 and .90 for sound holistic scoring. Readers who read the same kinds of papers regularly (including students in a large class) can easily be trained to reach acceptable inter-rater reliability scores.
AP exams and the SAT II writing test both use holistic scoring to assess student writing skills.