Summary
Notes on "The Future of Engineering Education: Part 5. Assessing
Teaching Effectiveness and Educational Scholarship," by R.
M. Felder, A. Rugarcia, J. E. Stice, Chemical Engineering Education,
34(3), pp. 198-207. (The paper in its entirety is available on
the Internet at http://www2.ncsu.edu/effective_teaching/)
[42 references]
Summarized by J. T. P. Yao, 8/21/00
"The instructional component of the mission
of every educational institution is to produce graduates with
satisfactory levels of knowledge, skills, and attitudes. …
An instructor may be a brilliant lecturer with student ratings
at the top of the charts, but if his or her teaching is not furthering
the instructional mission of the department, that teaching cannot
be considered effective."
"To appraise programmatic teaching effectiveness,
we must answer the following questions:
1.
Educational goals.
What are the published goals of the instructional program? Does
the faculty know what they are? Does the faculty generally agree
with them?
2.
Performance criteria.
Are the criteria that will be used to evaluate faculty performance
measurable and clearly tied to the goals? Does the faculty know
what they are? Does the faculty generally agree with them?
3.
Assessment process.
What assessment data will be collected? How and when and by whom
will they be collected and analyzed? Are available resources (including
faculty time) adequate to permit their collection and analysis?
4.
Evaluation process.
How will conclusions about teaching effectiveness be inferred
from the data, and by whom? What type of feedback will be provided
to the faculty, and when and by whom will it be provided? …
5.
Educational scholarship.
What evidence of scholarly contributions to education will be
collected? How and by whom will the evidence be evaluated?"
"An important distinction is that between
formative assessment, which has improvement
of teaching as its objective, and summative assessment, which produces information that can be used
to make decisions about instructional personnel or programs. …
Extensive research supports the use of the following criteria
as a basis for the assessment:
1.
The course contributes toward published program goals.
2.
The course has clearly stated measurable learning objectives.
3.
The assignments and tests are tied to the learning objectives
and are fair, valid, and reliable.
4.
Appropriate methods have been devised to monitor the effectiveness
of the instruction.
5.
The learning environment is appropriate.
6.
The instructor has appropriate expertise in the course subject.
7.
The instructor communicates high expectations of students and
a belief that they can meet these expectations, …
8.
The instructor seeks to provide an education in the broadest sense
of the word, …
9.
The instructor integrates teaching with research.
10.
The instructor continually attempts to improve the course by updating
the content and/or making use of new instructional materials and
methods…
11.
The students achieve the learning objectives. …"
"An assessment plan should involve assembling
several types of evidence to determine the degree to which the
foregoing criteria are being met…
·
Learning outcomes assessments: …
·
Student end-of-course ratings.
·
Student surveys, focus groups, or interviews directed at specified
criteria.
·
Retrospective student ratings of courses and instructors…
·
Alumni ratings of courses and instructors.
·
Peer ratings of classroom instruction, learning objectives, assignments
and tests.
·
Evaluations submitted by external referees.
·
Self-evaluation by instructors. …"
"Assessment … can only be done meaningfully
in the light of clearly stated goals and measurable objectives.
… The following assessment tools may be used as part of that
demonstration. …
·
Complete tests and individual
test items (knowledge,
conceptual understanding, engineering problem-solving skills)
…
·
Laboratory reports, design
project reports, live or videotaped oral presentations, research
proposals (knowledge,
conceptual understanding, analysis, critical thinking, experimental
design, identification of engineering problems, teamwork, written
and oral communication skills, professional or social awareness,
lifelong learning skills) …
·
Resume, letters, memos
(written communication skills,
professional or ethical awareness) …
·
Critique of technical reports,
papers, letters, and memos (analysis, critical thinking, written communication skills) …
·
Self-evaluation, learning
logs, journals (any
skills or attitudes) …
·
Other classroom assessment
techniques (any skills
or attitudes) …"
"Student
Ratings of Instruction
… The rating form is often haphazardly designed,
and the results may be difficult to interpret with any degree
of objectivity. … In fact, more than a thousand research
studies of student ratings have been performed, and the results
collectively show that ratings are reliable, stable over time,
and positively correlated with results obtained using other forms
of teaching assessment… Their validity notwithstanding, student
ratings should not be the only method used to assess instructional
quality. There are several important aspects of teaching that
students lack the knowledge and perspective to judge fairly, including
the currency and importance of the course content, the instructor's
understanding of the subject, and the appropriateness of the assignments,
tests, and grading policies. … Nevertheless, course-end student
ratings are an essential component of instructional quality assessment.
…
·
Collect ratings of the
effectiveness of the course and the instructor in a few critical
aspects. … The following items have been shown to be
related to teacher effectiveness…:
-
Each class period was carefully planned in advance.
-
The instructor presented the material clearly.
-
The professor made students feel free to ask questions, disagrees,
…, etc.
-
The professor used examples from his/her own research or experience.
-
This course has increased my knowledge and competence.
·
Collect overall course-end
ratings of instruction …
·
Administer and collect
course-end ratings in a single class session rather than counting
on students to return them later …
·
Interpret ratings collected
over a period of at least two years …
·
Periodically collect retrospective
student evaluations in addition to course-end ratings …"
"… To be effective, summative peer
ratings should include the features described below:
·
Who should do the reviewing?
Reviewers should be good teachers who have received training on
what to look for in a classroom and who recognize that different
styles of teaching can be equally effective. …
·
How should classroom observations
be performed? At least two reviewers should conduct at least
two class visits during a semester, preceding each visit with
a brief meeting at which the instructor provides information about
the class to be observed. …
·
What should the lecture
observation checklist contain? … - Organization …
- Knowledge … - Presentation … - Rapport …
·
How should instructional
material be rated? … Trained observers can judge whether
(a) the objectives cover a suitable range of knowledge and skills,
(b) the course content is sufficiently comprehensive and current,
(c) the assignments and tests are appropriately rigorous, fair,
and consistent with the stated objective. … "
"A recommended format for a summative portfolio
consists of several parts:
·
Preamble …
·
Reflective statement of
teaching philosophy, goals, and practices …
·
Summary of teaching and
advising responsibilities …
·
Representative instructional
materials and student products …
·
Evidence of teaching effectiveness
…
·
Efforts to improve teaching
effectiveness …
·
Teaching innovations
…
·
Evidence of effectiveness
of advising and mentoring …
·
Awards and recognition
…"
"ASSESSMENT
AND EVALUATION OF EDUCATIONAL SCHOLARSHIP
…
What is
educational scholarship?
1.
Subject knowledge …
2.
Pedagogical knowledge …
3.
Commitment to continuing
growth as an educator …
4.
Involvement in development,
assessment, and dissemination of innovative instructional methods
and materials …"
"The
improving climate for educational scholarship
… The climate for scholarship in engineering
education has become considerably warmer in recent years. …
Unfortunately, many who rate faculty performance in engineering
are still inclined to discount education-related activities as
not worthy of being counted toward promotion, tenure, and merit
raises, funded and published though they may be. Hopefully, this
situation will also improve before too long…"
"How
can educational scholarship be assessed and evaluated?
…
1.
Did the teaching qualify as a scholarly activity?
2.
Was the teaching effective?
3.
Were the innovative products and processes developed by the instructor
well conceived, implemented, assessed and evaluated, and disseminated?
…
The following standards … provide a good
basis for evaluating the quality of educational innovations:
1.
Clear goals. …
2.
Adequate preparation.
…
3.
Appropriate methods.
…
4.
Significant results.
…
5.
Effective presentation.
…
6.
Reflective critique.
…"
"The ultimate measure of the effectiveness
of teaching is the quality of resulting learning. … Tools
for assessing learning include tests and test items, written reports
and proposals, oral presentations and interviews, student-generated
critiques of work produced by others, student self-evaluations,
learning logs and journals. … Student ratings of teaching
are a valid and important source of evidence for teaching effectiveness,
especially if they are averaged over at least a two-year period.
… They should not be sole instrument used to evaluate teaching,
however, since students are generally not qualified to judge aspects
of instruction like the currency and importance of the course
content, the depth of the instructor's knowledge, and the appropriateness
of the assignments, tests, and grading policies. Peer ratings
are the most appropriate source of such judgments. … Faculty
members who meet or exceed institutional standards for educational
research merit the same recognition and opportunities for advancement
as faculty members who excel in disciplinary research."
[Readers who are interested in this article are
encouraged to read the original paper in its entirety. Other summary
notes on faculty reward systems are available on the Internet
at http://lohman.tamu.edu
under the heading "Summaries of Papers ..."]
