Metrics and Reward
Systems for Engineering Faculty
James T. P. Yao and Jose M. Roesset
Department of Civil Engineering
Texas A&M University
College Station, TX 77843-3136
Introduction
Successful implementation of curriculum changes, particularly
those implying a close coordination between different subjects,
is demanding on faculty time and requires therefore faculty
dedication. Yet as the primary role of universities has changed
from education to research with little faculty time for teaching
undergraduates, faculty performance evaluation depends primarily
on the amount of research funding generated and the number of
publications, with only lip service to teaching. If (a) our
universities are going to continue to be educational institutions
rather than just research establishments, and (b) needed curriculum
changes are to be successfully implemented, it is essential
to devise a reward system that recognizes properly all the activities
in which a faculty member must be involved and that puts emphasis
on quality rather than quantity. In this paper we discuss in
some detail what these activities are and the present reward
system at most research universities (Roesset and Yao, 2000).
The activities of engineering faculty can be sorted
into three main categories: teaching (T), research (R), and
service (S). The “university space” can be displayed in a set
of three Cartesian coordinate axes with non-negative values
(T, R and S) as shown in Fig. 1, denoting measures of effort
or performance in each one of these categories. An eighth of
a sphere of unit radius would represent the locus for the fraction
of effort devoted by a faculty member to each category, while
an eighth of an ellipsoid would represent in general the locus
for faculty performance, the major axis corresponding to the
category in which the individual excels. For a few, extremely
gifted persons, the ellipsoid might become a sphere indicating
that they are equally able at the three roles. In the same way,
if metrics and standards of faculty performance evaluation for
salary raises, promotion and tenure were clearly specified,
one should be able to draw corresponding surfaces in this space
defining performance levels. Unfortunately very few universities,
if any, are willing to state clearly and concisely their performance
metrics and promotion criteria. It might be easier to define
at least what are the minimum requirements, that would provide
a lower bound surface instead of a family of equal performance
surfaces, but even this is generally unavailable to day. All
the promotion and tenure manuals that we have seen to date express
performance criteria in vague terms, subject to interpretation.
A precise definition of the performance measures and
evaluation criteria used for annual reviews, promotion and tenure,
would make it easier for many faculty members to assess their
status and their chances. It would also simplify considerably
potential grievance hearings, which are becoming more common.
It would reduce on the other hand the flexibility of university
administrations at the various levels (department, college and
university) to adapt their criteria to each individual case.
What may be more important deterrents to the publication of
clear reward metrics is the lack of consensus on what these
should be and the difficulty in accounting properly for all
the possible activities involved in each one of the three main
categories assigning relative values to them. In the following
we consider these different activities.
Teaching
We include in this category all the activities directly
related to the education of students, such as:
- Teaching
undergraduate courses, including the time devoted to preparation
of lectures, homework, quizzes, exams, and web pages, student
consultations in the office and possible review sessions,
in addition to the actual classroom lecturing.
- Teaching
undergraduate laboratories or in other cases preparing the
material, experiments, projects etc. and coordinating the
work of teaching assistants in charge of the actual teaching.
- Supervising
undergraduate students enrolled in special topics courses
or independent study/research.
- Advising
undergraduate students. In some institutions each faculty
member has been assigned a number of undergraduates for academic
advising. In most cases faculty members meet with their advisees
only briefly at the time of registration.
- Advising
undergraduate student organizations such as student chapters
of national societies, ethnic and minority groups.
- Teaching
graduate courses which may cover advanced and detailed topics
with a reduced number of students or more general graduate
material with larger number of students though still smaller
than those of typical undergraduate classes.
- Supervising
doctoral dissertations. This represents in most cases one-on-one
supervision of the doctoral student. The degree of involvement
of the faculty member in the research may change greatly depending
on the qualifications and creativity of the student.
- Supervising
Master’s theses where these still exist. The purpose of these
theses is not to conduct original research but rather to teach
the students how to attack a new problem that has not been
covered in the classroom, how to search the literature, read
papers and interpret them critically, and how to think on
their own conducting independent work.
- Participating
in Master’s and doctoral committees of graduate students without
being the main supervisor. The amount of effort involved in
this activity is highly variable.
- Participating
in doctoral qualifying and comprehensive exams. This may involve
preparation of written questions and grading of the exams,
attending oral examinations and asking questions, etc.
Teaching performance is commonly rated today
by students, a controversial subject with some authors pointing
out that the use of student evaluations has led to grade inflation,
and that they are measures of popularity and easy grading rather
than of true teaching quality. Many others believe, however,
that student evaluations are in fact excellent measures of teaching
effectiveness. Felder (1992, 1999) mentioned that results of
approximately three thousand educational studies supported this
finding. We have found student evaluations to be extremely helpful.
Instructors can learn a considerable amount about their effectiveness
from students’ written comments but these comments can also
be improperly used to justify arbitrary decisions. The most
important attributes in teaching include (A) a solid knowledge
and clear understanding of the material to be taught; (B) dedication
and commitment to teaching; (C) a genuine caring for the students
both in and outside of the classroom; and (D) good communication
skills. These (with possible exception of item A) are all qualities
that students can easily recognize and which can be therefore
judged fairly in their evaluations. There are, however, as pointed
out above, many other teaching activities, which have not been
explicitly recognized normally in faculty performance evaluations
but which are very important to the education process and which
can require substantial effort. Teaching evaluations can measure
classroom performance but not other educational activities.
|
|
 |
Recently it has become common practice to encourage
faculty members to prepare teaching portfolios (Seldin 1995)
for tenure and promotion considerations (e.g., Felder and Brent
1996). The teaching portfolio includes, for each course taught,
the syllabus, homework, quizzes, term projects, final examination,
examples of students’ good and bad homework, examination results,
and student evaluations. Peer evaluations by having other faculty
members sit in the classes can also be included. If the portfolio
includes also all other educational activities (e.g., mentoring,
work with student organizations, and graduate student supervision),
it would be a complete file on educational activities of the
faculty member. The large amount of time required in preparing
a teaching portfolio can be justified if teaching activities
are taken seriously in faculty reward systems.
For many years, the premise that a good researcher was
always a good teacher has been accepted and cherished by administrators
of research universities. This justified basing performance
evaluations mostly on research under the implicit assumption
that good teaching was automatically implied by good research.
This statement has been questioned in a number of recent papers
(e.g., see Felder 1994). The fact is that there are many top-level
quality researchers who are also excellent teachers in graduate
as well as undergraduate courses. We both have been fortunate
to meet several of them during our lifetimes. However, we have
also known a substantial number of good researchers who were
very poor classroom teachers due to various reasons. There are
also many examples of effective teachers of undergraduate courses
who are not interested in continuing to do engineering research,
and prefer to spend their time keeping current on teaching methods,
their course materials, and professional issues.
Research
Under this category we must include again a number of
different activities, such as:
- Conducting
actual research on one’s own or with the help of a variable
number of research assistants, research associates or post-doctoral
fellows.
- Disseminating
the results of the research through technical reports, papers
published in refereed archival journals, papers presented
at conferences and published in refereed conference proceedings,
invited presentations at conferences, seminars or technical
meetings, and in some cases chapters in books.
- Writing
proposals to generate the desired funding to conduct research,
to support graduate students who would not attend the university
without financial support (and in some cases extra financial
incentives), to support possibly research engineers or postdoctoral
fellows who would do the actual supervision of the research,
at least at the detailed level, to cover part of the salary
of the faculty members listed as principal or co-principal
investigators, and to bring the much needed overhead funds
to the university.
- Attending
and participating actively in workshops organized by funding
agencies, particularly those dealing with research needs in
a specific area.
- Working
in panels to rate research proposals submitted to various
funding organizations.
- Reviewing
individual proposals without an actual panel.
Some of the activities that were listed under teaching
could also have been included here under research, notably those
related to supervision of undergraduates and primarily graduate
students conducting research. Whether it is more appropriate
to include them in one category or the other would depend primarily
on the objectives; whether the primary goal is to educate the
students with the research being a means to that goal, or whether
the primary motivation is to satisfy the requirements of a research
grant using the students as the needed labor. In the same way
the last three activities mentioned here could be considered
as part of the service category to be discussed next.
Service
There are two main subdivisions within this
category: administrative duties at the university and technical
service to the profession and professional societies.
1.
Administrative
duties have mushroomed at universities in recent years. Where
departments used to have thirty or forty years ago only a Department
Head and an administrative secretary, they have now, in addition
of a substantial staff, a number of Associate Department Heads
in charge of a variety of functions, Assistant Department Heads,
Graduate and Undergraduate Officers, Chairs of a myriad of committees
and the working members of these committees.
2.
Service to technical
and professional organizations include such activities as A)
chairmanship or membership in technical and administrative committees,
B) reviewing papers for these committees and for an ever increasing
number of journals with which one may not have any affiliation,
C) editorial jobs in some of these journals, D) organization
of sessions at specialty and general conferences identifying
potential authors and soliciting their contributions, E) organization
of a complete conference and F) attending meetings of local
or student chapters of these societies, participating in their
work, giving talks and seminars, etc.
Reward Systems
During the past few decades, many universities have
concentrated their efforts on generating research funding as
a means to increase their budget as well as to improve their
ranking. Several rating organizations use the amount of research
funding as a measure of university performance. It is not surprising
therefore that the amount of research dollars generated is also
a key measure of faculty work. Many universities consider the
following five aspects in evaluating faculty performance: teaching,
research, publications, administrative duties and service. Grades
are assigned to each one and then added. It is interesting to
notice that in this scheme faculty members get one grade for
all their teaching activities, two grades for research and two
again for service. It is no wonder that many faculty members
feel that teaching is not an important consideration within
research universities.
Evaluation of teaching relies mostly on student course
evaluations which are valuable but limited, and which can be
misused. Research is measured primarily by the volume of funding
independently of the type of research work performed. This makes
the assignment of ratings and numerical grades much easier.
The number of papers and the number of citations of these papers
measure publications. Few faculty members have their papers
actually read and evaluated carefully by their colleagues. Whether
one's research results are used in practice is generally considered
to be irrelevant. If the research results are in use by practicing
engineers, the reference index is usually low because few practicing
engineers publish journal papers.
Administrative duties are assigned a grade on rather
loose and vague terms. Clearly a person who has a primary administrative
role that occupies perhaps half of the time should be recognized
for this work if done well. In some cases the faculty members
in top administrative positions (associate or assistant department
heads) are not evaluated and ranked by the regular faculty committee
but their evaluation is left to the department head or chairman.
In others the faculty committee evaluates and ranks them but
the head or chairman can override their rating. The main question
is how to recognize other faculty members who devote a substantial
amount of their effort to committee work or other administrative
duties. Service activities are similar to administrative duties.
Normally no distinction is made between those who do a considerable
amount of paper reviews and technical committee work and those
that do just an average amount.
Within this system faculty members who generate large
amounts of funding and who can employ a number of full-time
researchers as post-doctoral fellows or research engineers will
always perform best. These researchers will assist supervising
graduate students, writing papers and proposals, and conducting
committee work. Generating large amounts of research funding
ensures therefore high grades in three or four of the categories
used to judge.
Faculty performance will be evaluated on an annual basis.
Salary raises may depend on the perception of its quality by
the reviewing team. There will be a few major reviews during
a faculty member's academic career, associated with tenure and
promotion to associate or full professorship, named professorship,
endowed chair, or distinguished professorship. If a faculty
member has her/his contract renewed after 7 years of service
she/he automatically has tenure. Thus the consideration of tenure
must start in the fifth year of service. In the past, tenure
implied that the faculty member could only be fired under unusual
circumstances such as dissolution of the department or moral
turpitude. In some states, tenured faculty members have their
performance reviewed periodically and can lose their jobs if
their performance is not found satisfactory by the administration.
To obtain tenure it is normally desirable to have completed
the supervision of one or more doctoral students, to have generated
a minimum amount of research funding, and to have published
a significant number of journal papers. Excellent student evaluations
will be a plus. Average evaluations do not hurt. Very poor ones
require some justification. We know of very poor teachers who
had no problem getting tenure and being highly recognized and
rewarded. It would be hard to find a faculty member at a research
university who got tenure on the basis of teaching quality without
a good amount of research funding and publications.
Although tenure and promotion criteria may be specified (e.g., see
Diamond 1994), these criteria are subject to interpretation.
Such interpretation will be a function of the composition of
the faculty review committee. One therefore concludes that the
expectation for each individual’s promotion or tenure is a fuzzy
and random event. The fuzzy and random nature of the process
becomes significant in marginal cases. Variation in personal
circumstances and in the composition of the review committee
implies that the perceived performance is a function of time.
Comments
on ABET 2000
The Accreditation Board for Engineering and Technology
(ABET) began in 1932 as the Engineer's Council for Professional
Development (ECPD). The name of ECPD was changed to ABET in
1980. The ABET 2000 criteria (ABET 2000) were formulated before
1996. The first phase was completed in 1998, and will become
mandatory for all engineering schools in 2001. ABET 2000 Criteria
require an engineering student to be able to (a) apply knowledge
of mathematics, engineering, and science; (b) design and conduct
experiments, analyze and interpret data; (c) design systems,
components, or processes to meet needs; (d) function in multi-disciplinary
teams; (e) identify, formulate, and solve engineering problems;
(f) understand professional and ethical responsibility; (g)
communicate effectively; (h) have a broad education; (i) recognize
the need for and engage in life-long learning; (j) know contemporary
issues; and (k) use techniques, skills, and tools in engineering
practice. Faculty members are supposedly responsible for all
these requirements including those beyond graduation (like life-long
learning). To satisfy at least the logical part of these all-encompassing
requirements, it is essential to have faculty motivated and
enthusiastic teaching students, and helping them to learn. According
to Sarin (2000), we must change the current faculty reward system
in order to get faculty members actively involved with the ABET
EC-2000.
If
we continue to pay lip-service to teaching and reward only the
quantity of research activities (e.g., the amount of funding
and the number of journal publications), faculty members will
see through this system and know exactly what one must do to
get the rewards (e.g., tenure and promotion). They might get
good student ratings somehow, but their hearts will no longer
be in teaching. The old college teachers, who grew up in a different
era and are continuing to do a good job in teaching regardless
of rewards, are retiring. We must improve the faculty reward
system so that faculty members pay attention to education in
addition to their other contributions. Although Felder et al.
(2000) have given excellent guidelines on how to evaluate teaching,
this type of effort should be extended to cover research and
service activities as well. Unless the university administrations
are convinced of the importance to reward the quality of faculty
work, the reform that we want will not happen. The key is to
obtain objective and fair metrics for teaching, research, and
service. While we know what we are looking for, we do not know
exactly what these metrics are at present.
Discussion and Recommendations
Clearly in the present environment a university needs
a certain number of entrepreneurs and salesmen who can sell
the research programs and generate funds. It also needs, nevertheless,
brilliant researchers, devoted teachers, and able administrators.
One must find a way to recognize and reward all these activities.
A number of mechanisms already exist to evaluate teaching effectiveness.
One of the main problems is that they are not always used properly.
A second problem is that they do not account for the many different
ways in which a faculty member can contribute effectively to
the educational program. There are also established, accepted
and largely uncontested procedures to measure research and service
performance. Once again we disagree with the validity of these
measures, based on some questionable and unproven assumptions
and relying primarily on quantity rather than quality. The final
problem is how much the teaching performance counts in relation
to research or service, and what is the relative value of all
the activities involved in each one of these three categories.
It would be desirable to (1) identify all the different
ways in which faculty can contribute to the education of undergraduate
and graduate students (e.g., classroom teaching, theses supervision,
advising, mentoring, and participation in student organizations)
and to the goals of a university; (2) assign relative values
to these different activities; (3) develop fair and objective
metrics to evaluate each one of the educational activities and
to arrive at a combined rating in a rational and unequivocal
way; and (4) suggest faculty reward systems with proper consideration
of teaching activities. This work should be conducted in collaboration
with a number of interested parties, both in the faculty and
in the administration, at different universities. It must be
initiated however by the faculty members who are interested
in teaching.
The Boyer Commission (1998) believed that the faculty
reward system should reflect the “synergy of teaching and research.”
They recognized that “baccalaureate students are the university’s
economic life blood.” Among their eight recommendations they
wanted to (a) “heighten the prestige of teaching” and emphasize
the linkage between teaching and research, (b) reward teaching
excellence, participation in interdisciplinary programs, and
outstanding mentoring in the form of permanent salary increases,
(c) reward teachers who can inspire large classes, and (d) greatly
reduce committee work to allow more time for students.
Although a precise set of metrics for measuring the
quality of teaching, research, and service is not yet established,
a framework of three Cartesian coordinate axes is suggested
as a "university space." In general, it is not desirable
to have one-dimensional professors who excel only in one of
these three axes in a university space.
Acknowledgement
The financial
support of the Lohman Professorship and the Wofford Cain Chair
at Texas A&M University is also appreciated. This is a part
of our efforts in searching for fair and objective metrics to
measure the quality of teaching, research, and service activities.
References
ABET (2000), Engineering Criteria 2000 Program Evaluation
Training, (May 1999 Version), presented by IEEE Educational
Activities Board in Charlotte, NC, on Sunday, 20 June 1999,
58 pages.
Boyer Commission on Educating Undergraduates in the
Research University, (1998), Reinventing Undergraduate Education
(A Blueprint for America’s Research Universities, Chapter
IX (Changing Faculty Reward Systems). http://notes.cc.sunysb.edu/Pres/boyer.nsf/
Diamond, R. M., (1994), Serving on Promotion and
Tenure Committees – A Faculty Guide, Anker Publishing Company,
Inc., 54 Pages.
Felder, R. M., (1992), “What Do They Know, Anyway?”
Chemical Engineering Education, 26(3), pp. 134-135. http://www2.ncsu.edu/unity/lockers/users/f/felder/public/Papers/Education_Papers.html
Felder, R. M., (1994), “The Myth of the Superhuman Professor,”
Journal of Engineering Education, ASEE, 82(2), pp. 105-110.
http://www2.ncsu.edu/unity/lockers/users/f/felder/public/Papers/Education_Papers.html
Felder, R. M., (1999), “Schooling versus Education and
Other Balancing Acts,” ERM Division Plenary Lecture, 1999 Annual
ASEE Meeting, Charlotte, NC 23 June 1999.
Felder, R. M., and Brent, R., (1996), “If You’ve Got
It, Flaunt It: Uses and Abuses of Teaching Portfolios,” Chemical
Engineering Education, 30(3), pp. 188-189. http://www2.ncsu.edu/unity/lockers/users/f/felder/public/Papers/Education_Papers.html
Felder, R. M., Stice, J. E., and Rugacia, A., (2000),
"The Future of Engineering Education: Part 6, Making Reform
Happen," Chemical Engineering Education, 34(3),
pp. 208-215. (http://www2.ncsu.edu/effective_teaching/)
Roesset, J. M., and Yao, J. T. P., (2000), "Roles
of Civil Engineering Faculty," Journal of Professional
Issues in Engineering Education and Practice, ASCE, Vol.
126, No. 1, January 2000, pp. 8-15.
Sarin, S., (2000), "Quality Assurance in Engineering
Education: A Comparison of EC-2000 and ISO-9000," Journal
of Engineering Education, ASEE, Vol. 89, No. 4, October
2000, pp. 495-501.
Seldin, P., (1995), “The Teaching Portfolio,” ASEE
Prism, May/June 1995, pp. 19-22.
