Eevin Jennings, PhD

The problems with Ratemyprofessors.com and course evaluations as measures of teaching efficacy: A scientific and anecdotal approach

4/23/2021

As you explore this site, you will notice a profound disconnect between the anonymous feedback that I receive from students in my own surveys and the feedback that is left for me on RMP's archaic website. While there are issues with anonymous data collection in general, I can assure you that what is posted on RMP is not reflective of students' actual experiences.

In higher education, we try to move away from putting any weight on such a laughable platform, but nonetheless, I still see A) plenty of students consulting the site when choosing a professor and B) other faculty referencing others' RMP ratings as points of contention. This an ethical concern for many reasons. Below, I hope to provide a compelling argument to sway you away from consulting RMP. References are included below.

I want to make a note here that the purpose of this section is not to make myself seem infallible or to make professors seem like victims. Rather, the contrary is true; this information should confirm that all of us, myself included, have much room to improve in how we approach (and perceive) the learning environment. I employ plenty of opportunities for feedback, seek professional development, and implement as many best practices as I feasibly can, but I will always have to work hard to improve and grow. RMP just represents the lowest bar of all.

Professors are not exempt from the issues I present below. That deserves a different post entirely. The information below serves to target the issues with RMP specifically.

OBvious issues

Selection bias.
Only the happiest and angriest students choose to leave comments, which presents a polarized (and inaccurate) representation of that professor. For example, some students claim that I am the kindest, nicest person ever (not true), and other students write that I completely neglect them in online courses (also not true).

Measures of central tendency and polarization.
RMP provides a mean (average) rating for the professors. This is problematic when you account for the polarization mentioned above. If half of the students give me a 0/5 and the other half give me a 5/5, that leaves me with an average of "2.5." Notice that nobody rated me with a 2.5. It is representative of neither the actual course experience nor the actual ratings on the website.

Rating the wrong professor.
For years, I had a negative comment about how I would always 'look down my glasses' at students in an intimidating way when they asked questions and took forever to grade their group projects. I don't wear glasses and don't use group projects. Another time, a student told me that they loved my class and raved that had given me a fantastic rating on RMP. Then they called me "Dr. Jones" on the way out.

Misunderstandings.
On my course evaluation one year, a student reported that I discriminated against students for whom English was a second language. Upon further review, the student had misinterpreted the function of my optional tutoring services. I had hired a TA with a certification in TESL (teaching English as a second language) to offer additional tutoring sessions to those who needed more one-on-one, individualized teaching to succeed in the course.

No option to counterpoint (despite what the site says).
I have tried several times to counter inaccurate posts on RMP to no avail. I've never received any follow-ups from RMP's staff.

Fake posting from professors.
Did you know that it's super common for professors to make fake accounts and give themselves awesome ratings? I knew a former colleague at a different school who gave himself 37 fake positive ratings to inflate his score.

Repeated posting.
Sometimes, the same student will leave several comments just to bring my overall rating down. RMP doesn't moderate its posts unless they contain profanity, so I'm left with one student resubmitting complaints without consequence.

Problems you may not have considered

You're being intentionally lied to.
From Clayson & Haley (2011): "Thirty-one percent of the respondents admitted to recording false information on the scale questions of the evaluations, while 19.4 percent admitted to adding untrue written comments on the evaluations. Combining both scales and written additions, in total, 37% of the students stated that they had submitted information in some form on the evaluations they knew were not deserved or were purposely false."

Imagine how inflated this statistic is for RMP! The converse is also true for positive ratings. Students will see negative comments for professors they like and counter with equally exorbitant positive ratings. Neither are accurate. Neither fix the problem.

Retaliations.
I notice that most of my negative ratings come in right after grades are posted, exam scores are released, or students are caught (and earn consequences for) cheating. You wouldn't know that though since you weren't there. All you see is the rant. One semester, a student of mine complained about me to the Provost and recommended disciplinary action be taken. When I asked the student why she did this, she answered, "Oh, I was just mad at the time. I didn't mean it." Consider this if you must still peruse RMP, where the (inaccurate) comments are thrown up onto the page in the heat of the moment and long forgotten.

judgments versus accuracy

Students don't actually know what's best for their learning.
"Students are not particularly good at evaluating their own learning, and they hold many false assumptions about how people learn. Students have a strong tendency to prefer instructional approaches that enhance their subjective impressions of learning, but that have been shown through empirical research to be ineffective or even counterproductive for learning." Carpenter et al. (2020).

Some examples:

Choosing poor learning strategies over good strategies: Even after I present compelling arguments against 're-reading notes' and for retrieval practice as learning strategies, most students still fall back to their previous (and horribly inadequate) 'study' methods of simply looking at their notes or re-reading a chapter. Because it worked for them in high school (and in other professors' lenient courses), they assume that I am asking too much of them or am being unfair. It takes repeated failure for these students to learn that their current strategies don't work anymore. But until then, the RMP ratings and course evaluations tell a different story.
The Learning Styles Fallacy: If I had a dollar for every time a student complained to me that they can't possibly learn the content because they are "a visual learner" or a "tactile learner," I could feed the whole country. LEARNING STYLES ARE A MYTH. Everyone is a visual learner. Everyone learns better with multiple, complementary sensory modalities (it's called dual-encoding theory).
The misconception that memorization = learning. There are SO many levels of memory and learning. They are not synonymous.
The misconception that instructor-provided study guides are helpful. The more work I do to create knowledge for you, the less learning you do to retain and embed that information yourself. My role is to help you develop that skill yourself.
- Just like a soccer coach's job isn't to run out on the field and practice for the players, my job isn't to learn for you. I use my expertise and experience to guide your behaviors, but you must be the one out on the field every day putting in the work. Professors are your coaches and you are the players. What work will you do to perform your best on game day?
Students don't' know how to read college-level textbooks. Sure, they think they do. But then the homework comes or the exam happens and suddenly they realize they didn't understand anything. This is because cue overload is a heuristic students use when (inaccurately) judging how much they know. When they recognize words on a page, and there are a lot of words, they assume that they understand. And of course, even after I demonstrate the S3QR reading comprehension strategy (below), it's "just too much work." This is a systemic problem at the very least.

The SQ3R method is something students were *supposed* to learn long before they got to college. Realizing the amount and type of work that goes into quality reading is a major eye-opener.

Negative ratings don't always reflect bad teaching.
By now, this shouldn't be a surprise, but this information is also important. Here is a recent negative RMP post on my page featured the following complaint:

Here is what this person failed to mention:

This is why it is a problem that professors do not get to counter these public complaints. Students who are in a hurry, have pre-existing biases, or are stressed won't think very critically about how valid these complaints are (which is known as the peripheral route of processing persuasive arguments). They will use availability heuristics (relying on the sheer volume of low ratings in contrast to the occasional positive one, or vice versa) as their primary decision point. And who can blame them? I would do the same thing.

Because students enter college with misconceptions about how learning works, they often interpret their own failures as faults of the professor. Being intellectually challenged for the first time is a huge hit to the ego. It makes sense to avoid a potentially stressful professor if the ratings are low, but consider the following research:

A plethora of studies (including studies with random assignment and real-world replications in many majors) confirms that students' high grades in introductory courses positively correlate with the ratings they give the professor, but negatively correlate with students' performances in subsequent classes (Braga et al., 2014; Kornell & Hausman, 2016). In other words, a fun, easy professor may result in a good time now, but a worse time later.

Examples of other negative ratings I've received that aren't actually bad practices:

The book was free. Student was upset that they didn't get to buy their book with their scholarship money.
I allowed late submissions (with a penalty). Student was angry that there wasn't a hard deadline because it "made them procrastinate." Also, I did have hard deadlines. But I also accepted late work.
I allowed them to choose their research topic. Student didn't want to be given choices because "I just want to be told what to do."
I didn't read them the exact questions and answers on the upcoming exam. No explanation needed.
I didn't require groupwork. Self explanatory. It wasn't right for the specific class.
I didn't use all multiple choice questions on the exams. That's because retrieval is different than recognition. Retrieval allows me to assess different types and levels of comprehension.
I had optional practice tests available for no credit. Student wanted them to be required.
I made them learn information that wasn't on the test. That's because there is more to our class than what's on the exams.
The exams are hard. The first exam is always hard because students don't understand that I actually require them to learn, not regurgitate information. You rarely see ratings discussing the fairness of exams or the improvement in study methods that resulted from a poor initial score.
I made the last exam optional due to the onset of the COVID-19 pandemic. The student complained that I 'gave up' on the class.
There is a lot of reading. Yes. This is college. I know that sounds harsh, but it's actually important that students demonstrate autonomous learning skills (at least in my classes).
I said I didn't know the answer to their questions. I made educated guesses though. I'd rather not lie and pretend I know everything. I'm human, too.

Good ratings don't always reflect good teaching.
Neath (1996) concluded that one of the top three ways to boost course evaluations (and subsequently, RMP) is to grade leniently. The other two tips were to be white and male (*shrug*). RMP is full of recommendations for professors who don't thoroughly challenge students in ways that enhance their autonomy.

Confirmation Biases
A confirmation bias is when you form an initial attitude based on some belief, then confirm that belief by noticing/selectively attending to information that confirms that belief.

When you peruse RMP, you form biases about that professor. Subsequently, you unconsciously look for instances that confirm your now pre-existing bias. According to a plethora of recent studies (see references), students are more likely to have negative encounters with professors after reading negative RMP comments compared to those who read positive or neutral comments beforehand. Benign, neutral comments from professors are interpreted sharply as criticisms, apathy, or attacks. Students who read negative RMP ratings beforehand interpret everything about that professor to fit with their preconceived notion. Plus, if students go into a course expecting the professor to be unreasonable, they are less likely to seek help in the first place. They then continue to experience a subpar course and further confirm their expectations. Reading negative RMP ratings will only hurt you.

To summarize the results from Boswell & Sohr-Preston (2020):

Students who read professors' positive ratings beforehand performed better in the class. Reading positive RMP ratings helps students in the long run because they were more likely to look for behaviors that confirmed their positive impressions of the professor. These students sought more help, pursued a quality professor relationship, and earned higher grades.
Conversely, students who read professors' negative ratings first performed worse in the class. This is due to the same reasons: they believed the professor was less interested in helping them from the start, so they were less likely to seek assistance, didn't try to form relationships, and subsequently earned lower grades due to their own beliefs. However, the students did not recognize that they were acting in accordance with confirmation bias; rather, they believed it was the professor's fault, and subsequently contributed to the RMP cesspool.

For some perspective, Boswell & Sohr-Preston say:

"For students who are exposed to positive RMP evaluations, these results suggest clear benefits: increased confidence for their ability to study for and invest effort into the course as well as their ability to form a good, beneficial relationship with the professor. These self-efficacious beliefs would likely translate into increased course effort and greater interpersonal engagement with the professor, both factors predictive of student learning and academic success (Ingraham et al. 2018; Komarraju et al. 2010; Svanum and Aigner 2011; Svanum and Bigatti 2006)."

"The implications of negative RMP evaluation exposure, however, are more concerning. RMP evaluations serve as vicarious experiences with a particular professor these experiences will have their greatest effect on students’ self-efficacy when they are unfamiliar with the professor in question. Students vicariously exposed to others’ perceived failures with a particular professor may become less efficacious for that course; in turn, these beliefs may contribute to academically self-defeating behaviors such as disengagement from coursework and reluctance to communicate with faculty."
"These effects are especially troublesome in this context: on RMP, evaluations are anonymous, providing readers with limited ability to appraise their credibility. Those with a sincere interest in learning may be exposed to and influenced by the evaluations of students who are irritated that they did not receive and “easy A.” In contrast, students gathering opinions via traditional word-of-mouth are better positioned to assess the source’s credibility to determine if it must be taken with a grain (or a pinch) of salt."

Self-Fulfilling Prophecies.
When you form an attitude by reading RMP ratings, you develop an existing bias about that professor. This bias affects the way that YOU behave/interact with that professor. Since this bias is rooted in negativity, your behavior (whether you know it or not) negatively affects your relationship with the professor. The vibe you give makes the professor behave defensively/cautiously toward you, which confirms your belief. Round and round the cycle goes.

THE BOTTOM LINE: By reading through/believing RMP, you miss out on a good educational opportunity while contributing to an ongoing problem.

Evidence of sexism/Gender discrimination

Buckle up! It's about to get real.

RMP is a cesspool of systematic racism, sexism, ableism, classism, and every other -ism out there. While no survey system is without flaws, the opportunity to leave anonymous, deindividualized, consequence-free, emotionally-laden "reviews" of any human is an open invitation for discrimination. While the same is true for semester-end course evaluations, the option to submit multiple ratings at any point during the year means more room for flagrance. Here are some key findings from the latest literature on systemic racism and sexism in student evaluations.

Male professors > Female professors

Male professors are consistently rated more positively than female professors for literally no reason. This is true across all disciplines and levels. In a cleverly controlled experiment, students took the exact same online course. At the end of the course, they were told that the professor was male or female. Despite having the EXACT same course with the EXACT same automated emails, students still gave the "male" professor higher ratings on all qualities.

Critically, the students rated both "instructors" as great; but still, the "male" instructor was rewarded more. This disparity can be observed in many cultural facets outside of academia as well.

Specifically (from MacNell et al):

"When the actual male and female instructors posted grades after two days as a male, this was considered by students to be a 4.35 out of 5 level of promptness, but when the same two instructors posted grades at the same time as a female, it was considered to be a 3.55 out of 5 level of promptness. In each case, the same instructor, grading under two different identities, was given lower ratings half the time with the only difference being the perceived gender of the instructor.
"Students rated the perceived female instructors an average of 0.75 points lower on the question regarding fairness, despite both instructors utilizing the same grading rubrics and there being no significant differences in the average grades of any of the groups. These findings support the argument that male instructors are often afforded an automatic credibility in terms of their professionalism, expertise, and effectiveness as instructors. Despite the fact that the students were equally satisfied with the promptness and fairness of the actual instructors, the instructor that students perceived to be male was considered to be more effective."
"Both actual instructors demonstrated the same level of interpersonal interaction in their attempts to create a sense of immediacy in the online classroom. Yet the perceived male instructor received higher ratings on all six interpersonal measures, three of them significantly. We contend that female instructors are expected to exhibit such traits and therefore are not rewarded when they do so, while male instructors are perceived as going above and beyond expectations when they exhibit these traits. In other words, students have higher interpersonal standards for their female instructors. Students sanctioned the perceived female instructor for failing to demonstrate strong interpersonal traits, yet did not do the same for the perceived male instructor. Both instructors were working within the same confines of online, text-based communications, but students only penalized the instructor they perceived to be female for this shortcoming."
"Because student ratings of teaching are considered an important measure of teaching proficiency, the existence of gender bias in those scores needs to be better understood and acknowledged within the institutional framework of our higher-education system." That goes for RMP, too.

Sexist language use in RMP ratings

The following slideshow depicts words that real students used in real RMP ratings. You can clearly see a major difference in language choice when rating male versus female professors. Retrieved from this site.

In line with the additional research, students more harshly judge female professors for qualities that are generally benign, and choose to praise them on qualities that fit with mothering expectations (as opposed to intelligence, accomplishments, or professionalism). Male professors are described as more laid-back and lenient, which (according to the research presented in this post) is a luxury afforded by status, privilege, and students' preconceived notions.

Before you jump to the argument that there are more male professors in the first place (which means there are more opportunities to give them high ratings), consider the fact that these trends are robust even in very rare "female-dominated" disciplines.

Students' extraordinary expectations of their female professors

Students have biased, implicit expectations of female professors and react more harshly when those expectations are not met (El-Alayli et al., 2018). In many ways, RMP often reflects whether a professor measured up to students' ideals about gender roles rather than their teaching efficacy. To illustrate, here are some well-established behaviors with which students disproportionately burden their female, but not male, professors (supplemented with real female professors' anecdotes).

Asking for special treatment.
At the end of every semester, our inboxes are flooded with "can you please bump up my grade?" and (a major pet peeve of mine) "I figured it didn't hurt to ask." Sure, this happens to male professors as well, but not nearly at the volume it does female professors.

Challenging the legitimacy of grading and other decisions.
Students are generally more likely to disagree with our policies, procedures, and really any decision female professors make, even in the face of a rubric.

Peacocking.
Every semester we have handfuls of students who devour an enormous portion of our mental bandwidth. These students blatantly interfere during class for a number of reasons, and it forces us spend extensive mental resources monitoring their behavior, watching for problematic behaviors, and being generally "ready to defend." These specific outbursts serve to prove a point about our inaptitude (asking ostentatious, impossible questions) or impress other students (making inappropriate comments to get a reaction).

Take this video of five random people (not even students) who chose my class to light up a gigantic joint in the middle of my Intro Psych class "for the 'Gram":

At a glance, this is no big deal. Some dumb kids played a prank on me and left. Right? This is where you're wrong.

There are a few things I want you to know about this situation.

This video did not capture what happened for 30 minutes prior. These random people coordinated with some students to get into my classroom, access our polling software, and purposely derail the class with disruptions. Since I had nearly 300 students in this class, I didn't realize they didn't belong. I continuously attempted to redirect students' attention when these guys blurted and submitted inappropriate comments to our polls. This was mentally exhausting since I was already having to prove myself as a woman in front of a large audience.
This video did NOT show you all of my attempts to handle the situation calmly. I volleyed some jokes with them for a bit, but their persistence wore down my students' attention.
This video did NOT show you my shaking hands or my tunnel vision as I relived the trauma of former on-campus shootings and interpersonal violence I've survived.
This kind of stuff happens to me every semester. It's always men and they always try to get a reaction. They also never leave until I take drastic measures (in this instance, threatening to call the cops).

A few semesters ago, a Youtuber burst into my classroom during an exam (again in Intro Psych). He made some loud racist comments as he slowly approached me at the podium. When I continuously told him to leave, he progressed to sexual comments about me in front of all of my students (who at this point were absolutely distracted and, of course, had their phones out to video the ordeal). This guy also refused to back down until I threatened him with calling the cops. One of my students was the plant and videoed the event for him. My other students fought to reclaim their attention and many suffered some anxiety seeing me be disrespected at that level.

Do I care about people blatantly smoking weed? Absolutely not. Do I resent the fact that they stole valuable class time from my students and me? Yes. I care that I am seen as a target. I care that they made it an explicit mission to undermine my hard-earned and fragile authority throughout the entire process.

Another interruption/prank involving non-students sneaking into my class occurred October of 2021. These were the only photos I was sent of the perpetrators: pictures that I took with a students' phone, and pictures OF ME taking pictures of the two men. One video was sent to me, but all it contains is a short video of (you guessed it) ME storming up the steps to address the behavior. What's going on in these pictures? A random guy from the street set up his all-in-one and blasted it at full volume, yelling at the screen. He also wouldn't leave when I demanded it. He also apparently couldn't wear his mask appropriately. He clearly didn't have much going for him.

The fact that I was unable to get him to leave was most infuriating to me. Our class time is precious, and I view any violation of this kind as a threat to my (and my students') safety and integrity. Where is the investigation to track these people down? Where are the protective measures to prevent this from happening every semester? How far will this go in the future before someone is actually hurt in the process? Why is nothing ever done about it?

Expecting, and then abusing, a friendly relationship.
Whereas students expect male faculty members to be too professional/busy to talk to in a friendly manner, female professors are often approached with "Hey" or by their first names (a major pet peeve for most of us along with "miss" or "Mrs."). Students are more likely to overshare in an effort to befriend the professor. I personally enjoy such casual relationships, but the problem lies in the next step: abusing the relationship once a need arises. This usually manifests as a request for special treatment in which I would bend the rules just for them.

Engaging in benevolent sexism.
If we are praised for anything, the language reflects appreciation mostly for our motherly qualities. Whereas male professors are praised for their intelligence and wit, female professors are rated highly for being "warm," "caring," and "loving." RMP exacerbates these issues because highly rated professors are those who students believe conform to their beliefs about gender roles, not who employs best practices.

Expecting chronic availability and personalized attention.
This falls under the embedded gender role that female professors desire to be mothers to their students. Whereas the male professors are expected to be available within a certain timeframe, female professors do not have the same luxury.

Overtly harassing (and threatening) the professor.
Assuming that students don't automatically jump rank and complain to the dean, they are more likely to threaten to do so for female professors.

Covertly harassing the professor.
This one is more common and problematic. Borderline behaviors create an uncomfortable environment, but don't reach the threshold to be supported in an official complaint.

From El-Alayli et al (2018):

"The current research examined the extra burdens experienced by female professors in academia in the form of receiving more work demands from their students. We found that female professors (compared to male professors) reported receiving more standard work demands and being asked for more special favors from students. Furthermore, the current work demonstrates that students with higher levels of academic entitlement are particularly likely to expect and ask for special favors from female professors (compared to male professors) and are more likely to be upset and persistent if their favor requests are not granted. Thus, our research adds to the body of empirical work regarding the experiences of female professors, and expands it by providing more information about how students treat female professors, how they react to them when the professors stand their ground, and what kinds of students are particularly likely to treat female professors differently from male professors."

Specifically:

"...Although academic entitlement was correlated with all of the dependent measures in the female professor condition, none of these correlations were significant in the male professor condition, suggesting that male professors may be more immune to potential entitlement effects. Perhaps because of gendered expectations of men as more agentic and authoritative (Eckes 2002; Ridgeway 2001), students, even entitled ones, may choose not to oppose their male professors’ decisions. This could be out of respect for male professors, feeling intimidated by male professors (if they are seen as more powerful), or believing that opposing the male professors’ decisions would be unproductive because their agentic qualities would cause them to remain firm on those decisions."

From Landrum (2019):

"The results shed light on how students’ understandings of grades are revelatory of the projects they have for themselves and others regarding their place in the academic world. The literature is mostly concerned with identifying individual traits that coincide with AE (e.g., Chowning & Campbell, 2009; Sohr-Preston & Boswell, 2015) as well as exploring the consequences of these entitled attitudes and behaviors (e.g., Bonaccio et al., 2016; Knepp, 2016; Linvill & Grant, 2017). By focusing on the lived meanings of the grade and exploring the purpose of these requests, the results reveal how students are taking a stance when confronted with a lower grade and how these requests are a claim before a world that challenges their self-understanding."

THE BOTTOM LINE:
Does this mean male professors don't deserve high ratings or that female professors deserve better ratings? No. It DOES mean that the criteria students use changes based on professors' demographics. It DOES mean that we need to:

A) find a better, more equitable way to qualify good teaching;
B) encourage male professors to stand up and do their part in leveraging this movement;
C) instantiate administration that personally invest time in and practice diversity training, not just promote it;
D) be forward with communicating these issues with students, who should then work toward targeting their own implicit biases; and
E) believe and support female professors with action, not just words.

Evidence of Racism

Not only do professors from minority groups have to fight extraordinarily hard to attain the same positions as their white coworkers, they have to deal with significantly more embedded racism from students and their institutions. Despite claiming that they "don't see color" and "aren't racist," students still discriminate against professors of minority populations. I'm not talking about derogatory word use- I'm talking about giving lower ratings on teaching evaluations, challenging professors' decisions, retaliating, and just generally having a harder time believing that the professor is qualified. Black professors have to work harder just to prove themselves to be equal in everyday academia. It is extremely emotionally taxing.

Unfortunately, this is not limited to academia. Kang et al. (2016) demonstrated the power of "resume whitening" (altering non-white race-indicative information, such as one's name, organizations, etc.) on job application callback rates:

"We found that the whitened versions of both the black and Asian resumes were more than twice as likely to result in a callback as unwhitened resumes, even though the listed qualifications were identical — in line with other studies showing lower callback rates for minority applicants. Most importantly, the discrimination against unwhitened resumes was no smaller for purportedly pro-diversity employers than for employers that didn’t mention diversity in their job ad. Employers need to acknowledge that discrimination still exists and that bias is hardwired into the system. When we are asked to process large amounts of information quickly, cognitive biases such as prejudice and stereotyping tend to prevail. We need innovations in recruitment to disrupt the bias that exists in human brains."

Kang et al (2016). Persistent, implicit racism in the job market for Black and Asian, but not White, Americans.

Just consider how racism is embedded into RMP and course evaluations.

References

Boswell & Sohr-Preston
File Size:	659 kb
File Type:	pdf

Blog.

The problems with Ratemyprofessors.com and course evaluations as measures of teaching efficacy: A scientific and anecdotal approach

Male professors > Female professors

Sexist language use in RMP ratings

Students' extraordinary expectations of their female professors

Macaw Species and Hybrids

Psi Chi Panel... With birbs

TTU Undergraduate research conference

A possible learning curve?

betty is born

Archives

Categories