Are school grades of Roma students affected by discrimination?
A paper review
I plan to do more posts reviewing single, excellent scientific papers as opposed to reviews of larger topics. Sometimes only a single paper really investigates a phenomenon, and sometimes there is a paper that uses some novel methodology that trumps all the others. I recently ran into this little gem of social science:
This should definitely be one of these rare papers that are worth reviewing on their own. Being Hungarian, I also want to read and promote more Hungarian social science. Let’s get down to it!
Gypsies – also called Roma – are the largest ethnic minority in Europe without a country of their own. They mostly live in Eastern Europe. In Hungary, 3.2% of the population declared themselves as Gypsy/Roma in the 2011 census (we are still waiting for the results of the 2022 census, any day now I guess). Many consider this proportion to be higher, which is funny, because this means saying that while some people didn’t say they were gypsies, you think you know better. The thing is, being a gypsy isn’t just a matter of self-definition, looks or genetics, but a bit of a subcultural or almost occupational category. Just like it doesn’t make sense for somebody who works a 9 to 5 office job in a suit all day to declare himself a punk or a hippie, if your lifestyle departs from the (very negative) stereotypical way of how people see gypsies, it is hard to interpret how you can still be one. I have seen a handful of well-spoken, highly educated and accomplished people with a slightly darker complexion whose backstory was that they are in fact gypsies, but their children likely won’t identify with this group and we have nothing resembling, for example, the Black American middle class among gypsies. The other side of this coin is that you can theoretically become a gypsy yourself if you associate with them and intermarry. You can absolutely be a gypsy and have blond hair and blue eyes (although this is of course not common), being a gypsy is somewhere between an ethnicity and a subculture. There are many such caste-like groups in other countries, either currently or historically, such as the Irish Travellers (sometimes mistakenly considered gypsies even though they are ethnically fully Irish), the Resande or Swedish Travelers, Japanese Burakumin, the Paekchong in Korea and, most famously, the Cagots of France.
Gypsies traditionally were travelers, but Hungarian Roma have settled down a long time ago and they are quite admixed, especially those living in cities. The paper I linked above about Irish travelers presents the following admixtures of regular Irish people (“settled Irish”), two groups of Irish travelers and Roma people from various Eastern European countries. You can see that the red bars, each roughly representing standard Western European admixture in an individual person belonging to that group, go higher for Hungarian Roma than for others, and in fact some of these Roma – including many from other countries – could probably pass for Irish themselves.
Gypsies in every Eastern European country lag behind the majority population in just about every social indicator imaginable. To seriously discuss why is almost as taboo as seriously discussing Black disadvantage in the United States – arguably more so, because many high-quality studies were done about Blacks but not so many about Gypsies. One the one hand, there are the usual woke papers claiming that discrimination, enslavement and the Holocaust causes the problems of the Roma. On the other hand, there is the other usual culprit, IQ differences. An unpublished meta-analysis and several studies, many by Serbian researcher Jelena Cvorovic, indeed show very low IQs in the Roma, but there are the usual issues of measurement invariance (never tested AFAIK) and it is also unknown to what extent low IQ is the cause or the consequence of very poor social conditions. Gypsies don’t live in America and government handouts are much less lavish here, so poor living standards as a cause of low IQ are not nearly as ridiculous as an explanation as in the US.
The article by Kisfalusi is written by sociologists so it clearly starts with an environmentalists view and looks at discrimination as the probable main cause of performance disparities, but it is willing to seriously consider other explanations.
They start with the observation that Roma students have lower test scores than their non-Roma students. The difference is almost a full school grade in GPA, mathematics and Hungarian, and it is about two thirds of a standard deviation in standardized competence test scores. Note that even the non-Roma underperform relative to the national average. Roma live in below-average places and among their classmates even the non-Roma also perform below the average. If you look closer, you can see something weird: there is a separate “self-declared” and “perceived” Roma category in one of the samples. See, this is what I was talking about: the teachers were asked if their students are Roma and their responses didn’t 100% match what the students themselves were saying. To some extent, being a Roma is in the eye of the beholder. This dual categorization is going to be interesting so we will get back to it.
The authors explore several reasons for grade disparities. They immediately consider that a big difference can be test score disparities: maybe Roma students are just genuinely not that sharp. This is a very brave start for sociologists – all school grade disparities will be investigated beyond what test scores can explain.
What does it mean if Roma students get worse grades than their test scores would imply? It can mean several things.
The first is statistical discrimination. Maybe teachers expect Roma students to perform worse because this is what they typically observe, and they are less likely to believe high performance from a Roma student. While this is extremely unfair, it makes perfect sense from a cold statistical point of view. A high score from a member of a low-performing group is more likely to be a fluke than from a high-performing group. The thing about statistical discrimination is that it should only be at work when teachers don’t have enough individual information about the students – if they are not actually against Roma students, after a while they will start judging them based on their individual merits instead of group-level information. This should have definitely happened for these students, who had been with their teachers for many years before they were tested, so the statistical discrimination theory actually predicts zero group difference conditional on test scores.
The second possibility is that teachers really are bigoted and their ratings are always affected by group membership and the associated stereotypes. They see a Roma, they subtract a grade. But if this is true, they might be also sexist and classist as well, and they should give lower grades to girls in mathematics (stereotypes say that girls can’t do math) and maybe lower grades in general to poor students.
The third possibility is disparate impact. While this term is used for a number of very crazy things in the US, what is meant by this here is that being Roma really is associated with some characteristics other than test scores which affect school grades, for example, bad behavior in the classroom. Teachers aren’t supposed to grade down students who misbehave in class for that reason, but maybe they do. This is captured by a uniquely Hungarian thing, the conduct (“magatartás”) grade. In Hungarian schools, beside the normal grades you receive for actual subjects, you get a highly subjective grade on the same 1-5 scale based on how good your behavior was. (You also get another for diligence [“szorgalom”], based on how well you did your homework and other similar things. Girls usually ace these grades, mine were expectably terrible. You normally can’t get a failing grade but I often came close.) The question is: do conduct grades explain the Roma vs. non-Roma difference in grades that remains after accounting for test score gaps?
A fourth possibility is taste-based discrimination. This is similar to the second, but even more evil: what if teachers don’t even just see Roma students as dumb (and adjust their grades accordingly), but just don’t like them at all? This is where the self-identification vs. other-identification thing comes into play. Taste-based discrimination shouldn’t affect the students who are seen as Roma by themselves but not by their teachers – they fly under the bigotry radar.
This is what the model results look like (for math grades):
Obviously, having higher standardized test scores is associated with better mathematics performance – this is basically in IQ test. However, Roma students with the same test scores get more than a full grade worse grades! Statistical discrimination is not a good explanation for us here. In model 2, sex and special education need is added, and in model 3, low socio-economic status (eligibility for a government grant) is added too. You can see that the “teachers are bigots” theory (explanation 2) is bunk too – girls actually get higher grades in mathematics than their other characteristics, including test scores, would imply. Special education need does nothing – you need special education because you are not that smart and this is captured by the test scores. Poor students, however, do get worse grades than their test scores would imply, and from model 3 we can also say that Roma students don’t just get worse grades because they are poor – the effect remains almost as big as in model 2 where social status was not controlled for.
This all changes, however, in model 4 where conduct grades are added. (Those who work with similar models a lot will also appreciate the HUGE drop in the AIC values indicating the better fit of the model. From the supplementary tables you can see that conduct grades account for about 10% of the grade variance.) Test scores and conduct grades explain all the disparities – both poor students and Roma students get worse grades because they behave worse in class (or at least their teachers see it that way). The p-values of 0.057 and 0.07 are not incompatible with there being a tiny, tiny additional effect not explained by conduct, but it is impressive that this very crude measure can explain so much about grades. It looks like “disparate impact” is the most likely explanation: Roma students are different in a way that is not captured by their test scores but considered important by their teachers.
But of course we know from other sources that personality and attitudes – sometimes called “noncognitive skills” are important for school/life outcomes beyond IQ, which is roughly what a standardized test measures. There is even a GWAS for this. In that study, they basically asked this: which genetic variants are associated with educational attainment but not with IQ and what do they do? They found that genetic variants affecting these non-cognitive skills – non-IQ traits associated with educational attainment – were associated with less risky behavior such as smoking, better personality traits, and for some reason higher propensity for psychiatric disorders. This is pretty close to what a conduct grade captures, no? Teachers like nice, shy, risk-averse students. It looks like Roma students and students from poor families are not like this, and this together with their lower test scores pretty much fully explains why they get worse grades.
Of course this analysis was only about mathematics grades. In the table below you see what happens if we use literature grades or GPA:
It’s largely the same thing. Once you control for conduct grades in model 4, all effects disappear. You can also see here what happens if we include both self- and teacher-perceived ethnicity. (As far as I can see, they entered both in the same model – this is bad because there were only 67 students which mismatched identification, which makes the two measures multicollinear – highly correlated. This reduces the effect sizes of both.) Nothing jumps out, except maybe that teacher-rated ethnicity is more important. One possibility is that students who rated themselves as Roma but their teachers didn’t concur were just trolling. But another is that better performing, better behaved Roma students are able to “pass” as non-Roma – I told you that being non-stereotypical is hard to reconcile with being Roma, because this identity is very strongly tied to stereotypical behavior. There is some research – also from Hungary – that shows that well-liked, well-accepted Roma students are less likely to be even seen as Roma by their peers.
This great paper also has downloadable data, so any ideas you might have are probably testable right away!