The problems with Ratemyprofessors.com and course evaluations as measures of teaching efficacy: A scientific and anecdotal approach
As you explore this site, you will notice a profound disconnect between the anonymous feedback that I receive from students in my own surveys and the feedback that is left for me on RMP's archaic website. While there are issues with anonymous data collection in general, I can assure you that what is posted on RMP is not reflective of students' actual experiences.
In higher education, we try to move away from putting any weight on such a laughable platform, but nonetheless, I still see A) plenty of students consulting the site when choosing a professor and B) other faculty referencing others' RMP ratings as points of contention. This an ethical concern for many reasons. Below, I hope to provide a compelling argument to sway you away from consulting RMP. References are included below.
I want to make a note here that the purpose of this section is not to make myself seem infallible or to make professors seem like victims. Rather, the contrary is true; this information should confirm that all of us, myself included, have much room to improve in how we approach (and perceive) the learning environment. I employ plenty of opportunities for feedback, seek professional development, and implement as many best practices as I feasibly can, but I will always have to work hard to improve and grow. RMP just represents the lowest bar of all.
Professors are not exempt from the issues I present below. That deserves a different post entirely. The information below serves to target the issues with RMP specifically.
Only the happiest and angriest students choose to leave comments, which presents a polarized (and inaccurate) representation of that professor. For example, some students claim that I am the kindest, nicest person ever (not true), and other students write that I completely neglect them in online courses (also not true).
Measures of central tendency and polarization.
RMP provides a mean (average) rating for the professors. This is problematic when you account for the polarization mentioned above. If half of the students give me a 0/5 and the other half give me a 5/5, that leaves me with an average of "2.5." Notice that nobody rated me with a 2.5. It is representative of neither the actual course experience nor the actual ratings on the website.
Rating the wrong professor.
For years, I had a negative comment about how I would always 'look down my glasses' at students in an intimidating way when they asked questions and took forever to grade their group projects. I don't wear glasses and don't use group projects. Another time, a student told me that they loved my class and raved that had given me a fantastic rating on RMP. Then they called me "Dr. Jones" on the way out.
On my course evaluation one year, a student reported that I discriminated against students for whom English was a second language. Upon further review, the student had misinterpreted the function of my optional tutoring services. I had hired a TA with a certification in TESL (teaching English as a second language) to offer additional tutoring sessions to those who needed more one-on-one, individualized teaching to succeed in the course.
No option to counterpoint (despite what the site says).
I have tried several times to counter inaccurate posts on RMP to no avail. I've never received any follow-ups from RMP's staff.
Fake posting from professors.
Did you know that it's super common for professors to make fake accounts and give themselves awesome ratings? I knew a former colleague at a different school who gave himself 37 fake positive ratings to inflate his score.
Sometimes, the same student will leave several comments just to bring my overall rating down. RMP doesn't moderate its posts unless they contain profanity, so I'm left with one student resubmitting complaints without consequence.
Problems you may not have considered
You're being intentionally lied to. RMP is the FOX News channel of academia.
From Clayson & Haley (2011): "Thirty-one percent of the respondents admitted to recording false information on the scale questions of the evaluations, while 19.4 percent admitted to adding untrue written comments on the evaluations. Combining both scales and written additions, in total, 37% of the students stated that they had submitted information in some form on the evaluations they knew were not deserved or were purposely false."
Imagine how inflated this statistic is for RMP! The converse is also true for positive ratings. Students will see negative comments for professors they like and counter with equally exorbitant positive ratings. Neither are accurate. Neither fix the problem.
I notice that most of my negative ratings come in right after grades are posted, exam scores are released, or students are caught (and earn consequences for) cheating. You wouldn't know that though since you weren't there. All you see is the rant. One semester, a student of mine complained about me to the Provost and recommended disciplinary action be taken. When I asked the student why she did this, she answered, "Oh, I was just mad at the time. I didn't mean it." Consider this if you must still peruse RMP, where the (inaccurate) comments are thrown up onto the page in the heat of the moment and long forgotten.
judgments versus accuracy
Students don't actually know what's best for their learning.
"Students are not particularly good at evaluating their own learning, and they hold many false assumptions about how people learn. Students have a strong tendency to prefer instructional approaches that enhance their subjective impressions of learning, but that have been shown through empirical research to be ineffective or even counterproductive for learning." Carpenter et al. (2020).
Negative ratings don't always reflect bad teaching.
By now, this shouldn't be a surprise, but this information is also important. Here is a recent negative RMP post on my page featured the following complaint:
Here is what this person failed to mention:
This is why it is a problem that professors do not get to counter these public complaints. Students who are in a hurry, have pre-existing biases, or are stressed won't think very critically about how valid these complaints are (which is known as the peripheral route of processing persuasive arguments). They will use availability heuristics (relying on the sheer volume of low ratings in contrast to the occasional positive one, or vice versa) as their primary decision point. And who can blame them? I would do the same thing.
Because students enter college with misconceptions about how learning works, they often interpret their own failures as faults of the professor. Being intellectually challenged for the first time is a huge hit to the ego. It makes sense to avoid a potentially stressful professor if the ratings are low, but consider the following research:
Examples of other negative ratings I've received that aren't actually bad practices:
Good ratings don't always reflect good teaching.
Neath (1996) concluded that one of the top three ways to boost course evaluations (and subsequently, RMP) is to grade leniently. The other two tips were to be white and male (*shrug*). RMP is full of recommendations for professors who don't thoroughly challenge students in ways that enhance their autonomy.
A confirmation bias is when you form an initial attitude based on some belief, then confirm that belief by noticing/selectively attending to information that confirms that belief.
When you peruse RMP, you form biases about that professor. Subsequently, you unconsciously look for instances that confirm your now pre-existing bias. According to a plethora of recent studies (see references), students are more likely to have negative encounters with professors after reading negative RMP comments compared to those who read positive or neutral comments beforehand. Benign, neutral comments from professors are interpreted sharply as criticisms, apathy, or attacks. Students who read negative RMP ratings beforehand interpret everything about that professor to fit with their preconceived notion. Plus, if students go into a course expecting the professor to be unreasonable, they are less likely to seek help in the first place. They then continue to experience an subpar course and further confirm their expectations. Reading negative RMP ratings will only hurt you.
To summarize the results from Boswell & Sohr-Preston (2020), reading positive RMP ratings helps students in the long run because they were more likely to look for behaviors that confirmed their positive impressions of the professor. These students sought more help, pursued a quality professor relationship, and earned higher grades. However, students who read professors' negative ratings performed worse in the class. This is due to the same reasons: they believed the professor was less interested in helping them from the start, so they were less likely to seek assistance, didn't try to form relationships, and subsequently earned lower grades due to their own beliefs. However, the students did not recognize that they were acting in accordance with confirmation bias; rather, they believed it was the professor's fault, and subsequently contributed to the RMP cesspool.
For some perspective, Boswell & Sohr-Preston say:
When you form an attitude by reading RMP ratings, you develop an existing bias about that professor. This bias affects the way that YOU behave/interact with that professor. Since this bias is rooted in negativity, your behavior (whether you know it or not) negatively affects your relationship with the professor. The vibe you give makes the professor behave defensively/cautiously toward you, which confirms your belief. Round and round the cycle goes.
THE BOTTOM LINE: By reading through/believing RMP, you miss out on a good educational opportunity while contributing to an ongoing problem.
Evidence of sexism/Gender discrimination
Buckle up! It's about to get real.
RMP is a cesspool of systematic racism, sexism, ableism, classism, and every other -ism out there. While no survey system is without flaws, the opportunity to leave anonymous, deindividualized, consequence-free, emotionally-laden "reviews" of any human is an open invitation for discrimination. While the same is true for semester-end course evaluations, the option to submit multiple ratings at any point during the year means more room for flagrance. Here are some key findings from the latest literature on systemic racism and sexism in student evaluations.
Male professors > Female professors
Male professors are consistently rated more positively than female professors for literally no reason. This is true across all disciplines and levels. In an experiment, students took the exact same online course. At the end of the course, they were told that the professor was male or female. Despite having the EXACT same course with the EXACT same automated emails, students gave the "male" professor higher ratings on all qualities.
Critically, the students rated both "instructors" as great; but still, the "male" instructor was rewarded more. This disparity can be observed in many cultural facets outside of academia as well.
Specifically (from MacNell et al):
Sexist language use in RMP ratings
The following slideshow depicts words that real students used in real RMP ratings. You can clearly see a major difference in language choice when rating male versus female professors. Retrieved from this site.
In line with the additional research, students more harshly judge female professors for qualities that are generally benign, and choose to praise them on qualities that fit with mothering expectations (as opposed to intelligence, accomplishments, or professionalism). Male professors are described as more laid-back and lenient, which (according to the research presented in this post) is a luxury afforded by status, privilege, and students' preconceived notions.
Before you jump to the argument that there are more male professors in the first place (which means there are more opportunities to give them high ratings), consider the fact that these trends are robust even in very rare "female-dominated" disciplines.
Students' extraordinary expectations of their female professors
Students have biased, implicit expectations of female professors and react more harshly when those expectations are not met (El-Alayli et al., 2018). In many ways, RMP often reflects whether a professor measured up to students' ideals about gender roles rather than their teaching efficacy. To illustrate, here are some well-established behaviors with which students disproportionately burden their female, but not male, professors (supplemented with real female professors' anecdotes).
Asking for special treatment.
At the end of every semester, our inboxes are flooded with "can you please bump up my grade?" and (a major pet peeve of mine) "I figured it didn't hurt to ask." Sure, this happens to male professors as well, but not nearly at the volume it does female professors.
Challenging the legitimacy of grading and other decisions.
Students are generally more likely to disagree with our policies, procedures, and really any decision female professors make even in the face of a rubric.
Every semester we have handfuls of students who devour an enormous portion of our mental bandwidth. These students blatantly interfere during class for a number of reasons, and it forces us spend extensive mental resources monitoring their behavior, watching for problematic behaviors, and being generally "ready to defend." These specific outbursts serve to prove a point about our inaptitude (asking ostentatious, impossible questions) or impress other students (making inappropriate comments to get a reaction).
Take this video of five random people (not even students) who chose my class to light up a gigantic joint in the middle of my Intro Psych class "for the 'Gram":
At a glance, this is no big deal. Some dumb kids played a prank on me and left. Right? This is where you're wrong.
There are a few things I want you to know about this situation.
A few semesters ago, a Youtuber burst into my classroom during an exam (again in Intro Psych). He made some loud racist comments as he slowly approached me at the podium. When I continuously told him to leave, he progressed to sexual comments about me in front of all of my students (who at this point were absolutely distracted and, of course, had their phones out to video the ordeal). This guy also refused to back down until I threatened him with calling the cops. One of my students was the plant and videoed the event for him. My other students fought to reclaim their attention and many suffered some anxiety seeing me be disrespected at that level.
Do I care about people blatantly smoking weed? Absolutely not. Do I resent the fact that they stole valuable class time from my students and me? Yes. I care that I am seen as a target. I care that they made it an explicit mission to undermine my hard-earned and fragile authority throughout the entire process.
Expecting, and then abusing, a friendly relationship.
Whereas students expect male faculty members to be too professional/busy to talk to in a friendly manner, female professors are often approached with "Hey" or by their first names (a major pet peeve for most of us along with "miss" or "Mrs."). Students are more likely to overshare in an effort to befriend the professor. I personally enjoy such casual relationships, but the problem lies in the next step: abusing the relationship once a need arises. This usually manifests as a request for special treatment in which I would bend the rules just for them.
Engaging in benevolent sexism.
If we are praised for anything, the language reflects appreciation mostly for our motherly qualities. Whereas male professors are praised for their intelligence and wit, female professors are rated highly for being "warm," "caring," and "loving." RMP exacerbates these issues because highly rated professors are those who students believe conform to their beliefs about gender roles, not who employs best practices.
Expecting chronic availability and personalized attention.
This falls under the embedded gender role that female professors desire to be mothers to their students. Whereas the male professors are expected to be available within a certain timeframe, female professors do not have the same luxury.
Overtly harassing (and threatening) the professor.
Assuming that students don't automatically jump rank and complain to the dean, they are more likely to threaten to do so for female professors.
Covertly harassing the professor.
This one is more common and problematic. Borderline behaviors create an uncomfortable environment, but don't reach the threshold to be supported in an official complaint.
From El-Alayli et al (2018):
From Landrum (2019):
THE BOTTOM LINE:
Does this mean male professors don't deserve high ratings or that female professors deserve better ratings? No. It DOES mean that the criteria students use changes based on professors' demographics. It DOES mean that we need to:
Evidence of Racism
Not only do professors from minority groups have to fight extraordinarily hard to attain the same positions as their white coworkers, they have to deal with significantly more embedded racism from students and their institutions. Despite claiming that they "don't see color" and "aren't racist," students still discriminate against professors of minority populations. I'm not talking about derogatory word use- I'm talking about giving lower ratings on teaching evaluations, challenging professors' decisions, retaliating, and just generally having a harder time believing that the professor is qualified. Black professors have to work harder just to prove themselves to be equal in everyday academia. It is extremely emotionally taxing.
Unfortunately, this is not limited to academia. Kang et al. (2016) demonstrated the power of "resume whitening" (altering non-white race-indicative information, such as one's name, organizations, etc.) on job application callback rates:
Kang et al (2016). Persistent, implicit racism in the job market for Black and Asian, but not White, Americans.
Just consider how racism is embedded into RMP and course evaluations.
To my knowledge, there does not currently (June 17, 2019) exist a singular model in which the most common (existing) species of macaw and their hybrid lines are depicted. Plenty of websites list lineage with pictures/description (i.e., Calico = military x greenwing), but none use charts nor diagrams. My figures are the first of their kind to combine visual, descriptive, linkage, and relative body size across the species and hybrids. The first figure is especially fascinating in that it demonstrates how dependent hobbyists are on some of the more prominent species. The Blue & Gold is still claimed to be the most frequently used "base" in hybridization because of its commonality, but looking at this figure, the scarlet now tops the chart with 9 unique hybrid contributions.
Of course, I had to include the mighty Mini macaws and even the Spix's, since it's technically still around, albeit at the verge of extinction. I'm also surprised to see Buffon's and Hyacinths crossed as often as they are considering their rarity. In the future I would like to elaborate upon this diagram to include phenotypical differences based on parentage. For example, Harlequins can be more orange-colored when fathers are GW and mothers are B&G.
More aesthetically appealing and intuitive is the F1 Hybrid Chart. I'm not a fan of the goofy names people have given some of the hybrids ("Buffwing" is probably my least favorite), and honestly I'm not a fan of hybridizing in general if it's not for preservation purposes. But, the outcomes are interesting to me, especially when it comes to the behavioral traits the progeny supposedly "inherit."
At the moment, the Hyacinth x Greenwing cross does not have an official name, and I could only find evidence of one in existence (poor Hector). The single video I could find features him in a small, toy-less cage in a pet shop somewhere. I hope someone has given him a better life since that video's posting.
Anyway, hope you enjoy!
Please note that I do NOT take credit for any of the photos used in these figures. These images were gleaned from Google, Facebook, and other websites.
Complete with terrible, rushed photoshop effects!
Psi Chi is always fun for me. I love diving into a group of people whose dreams align with mine, still at the cusp of their journeys and full of questions. Academic futures abound. Everyone has questions.
I remember my first Psi Chi meeting in undergrad. I was too far down the "psych major" ladder to join, but I still attended the events. Two "old school" psych professors, both in their late 70s, loudly debated behaviorism versus cognitivism, the latter of which was defended by my very own chief advisor and drama extraordinaire. He humiliated the behaviorist, handing out papers with the man's most ridiculed quotes on them, even an article he had published in Esquire (yes, it was real). The audience boo'd the behaviorist, who coincidentally was also the department's most abhorred professor, and eventually the man threw his hands up and left, probably to crouch over exams and fail students relentlessly. It had been less of a debate (although it had started out that way) and more of a roast. I immediately knew I wanted in.
I didn't do much as a Psi Chi member in undergrad. Looking back, I wish I had. Although I participated in all of the mandatory fundraising events, I could have been a heavy voice to instigate change and clarity. If any psychology majors are reading this, take action and grab the wheel when you have the chance. Plus it looks good on your CV... *ahem*
Now that I am older and caught in the mire of what truly matters in undergrad, I try to be that heavy voice that I should have been years ago. The difference is that now I orchestrate rather than do the moving. The birds help me in that regard. They spark some level of curiosity, make psychology tangible, and at the same time enable my brief soap box rants about animal welfare and species conservation. And they make it look like I know what I'm doing.
Closing: Go kick some life ass!
In prior years, my *RAs have presented various portions of my research in little seminars and "research carnivals," which was fun and laid-back. This year, four of them stepped up and asked to formally present their own posters at our local undergraduate research conference. For the past couple of months we have been arms-deep in constructing novel questions/analyses with my incoming dissertation data. Everyone was nervous, but pleasantly surprised with how fun conferences actually are! As an additional positive takeaway, everyone received specific invitations to work with faculty/grad students in other labs (or as potential graduate students in the future)! I am incredibly proud of them!
*For more information, check out my People page.
For more about each poster, click here.
I've had parrots since I was a child and bred green cheeked conures through undergrad. One of my major objectives has been to intervene during early development to establish human-avian communication via operant conditioning. This is a fancy way of saying that I wanted to teach them to do tricks at an early age so that they knew how to navigate a human world with simple communicative symbols.
The above image (copyrighted by me) is my depiction, based on my life's experience, networking, and the few journal articles available, of where popular species stand on these axes. This was inspired by Arthur (my un-cape parrot) and several other species with whom I've worked to develop trick training. While my conures, whom I brought to class, are fantastic workers, my Arthur is not. This is the bane of several un-cape owners' existence, especially since they are notorious for their high emotionality and intelligence. It occurred to me (as well as a few others) that some species may be "too" smart for formal training, or at least food-based training. I made this chart because it is the first of its kind, and it seems to be a decent representation.
On the far right, you have high cognitive complexity (very loosely defined since this is a rough draft). Cockatoos, macaws, Greys, and other difficult species (including a crow for reference) make up the majority of this hemisphere. However, while cockatoos on the whole are much less likely to respond to food-based reward, macaws are famous for their trainability. On the far left side, you see an interesting cluster in the top pole. Although all parrot species do qualify as cognitively complex (as are most vocal-learning types of social animals) in reference to, say, reptiles, they are not known for being emotional circuses like Cacatua.
It seems that there is a "just right" combination for trick training, if you also want a less cognitively complex trainee, and that interactions may occur as complexity progresses across the chart. In essence, the more complex you are, the more room for differences in trainability.
You'll notice that Greys are on the far right and in the middle. I put them there because they are very good at training, but shy away in front of strangers.
If you have an individual who seems to defy this trend, that is interesting. However, keep in mind that individuals are just that- individual. Nonetheless, there is much to be learned, and as the future continues I am excited to incorporate additional axes and measures.
Please do not use the chart without first acquiring my written consent :)
I've spent the past few weeks designing "Betty," a 3D model of the brain stem, limbic system, and basal ganglia.
One of my RAs is removing the support material so that we can get to painting her!
She's going to be used heavily in my Physiological Psychology classes, and possibly for research later on.