Never, Ever Trust Media Reporting of Scientific (Or Quasi-Scientific) Studies -- The Github Sexism Study and the Response.

I recommend this article (via Tyler Cowen) on the interesting topic of whether women's open source software contributions on Github are accepted more or less frequently than those of men.   The findings of the study are roughly as follows:

They find that women get more (!) requests accepted than men for all of the top ten programming languages. They check some possible confounders – whether women make smaller changes (easier to get accepted) or whether their changes are more likely to serve an immediate project need (again, easier to get accepted) and in fact find the opposite – women’s changes are larger and less likely to serve project needs. That makes their better performance extra impressive....

Among insiders [essentially past contributors], women do the same as men when gender is hidden, but better than men when gender is revealed. In other words, if you know somebody’s a woman, you’re more likely to approve her request than you would be on the merits alone. We can’t quantify exactly how much this is, because the paper doesn’t provide numbers, just graphs. Eyeballing the graph, it looks like being a woman gives you about a 1% advantage. I don’t see any discussion of this result, even though it’s half the study, and as far as I can tell the more statistically significant half.

Among outsiders, women do the same as/better than men when gender is hidden, and the same as/worse than men when gender is revealed. I can’t be more specific than this because the study doesn’t give numbers and I’m trying to eyeball confidence intervals on graphs. The study itself say that women do worse than men when gender is revealed, so since the researchers presumably have access to their real numbers data, that might mean the confidence intervals don’t overlap. From eyeballing the graph, it looks like the difference is 1% – ie, men get their requests approved 64% of the time, and women 63% of the time. Once again, it’s hard to tell by graph-eyeballing whether these two numbers are within each other’s confidence intervals.

OK, so generally good news for women on all fronts -- they do better than men -- with one small area (63 vs 64 percent) where there might or might not be an issue.

This was an interesting side bit:

Oh, one more thing. A commenter on the paper’s pre-print asked for a breakdown by approver gender, and the authors mentioned that “Our analysis (not in this paper — we’ve cut a lot out to keep it crisp) shows that women are harder on other women than they are on men. Men are harder on other men than they are on women.”

Depending on what this means – since it was cut out of the paper to “keep it crisp”, we can’t be sure – it sounds like the effect is mainly from women rejecting other women’s contributions, and men being pretty accepting of them. Given the way the media predictably spun this paper, it is hard for me to conceive of a level of crispness which justifies not providing this information.

So here is an example press report of this study and data:

Here’s Business Insider: Sexism Is Rampant Among Programmers On GitHub, Research Finds. “A new research report shows just how ridiculously tough it can be to be a woman programmer, especially in the very male-dominated world of open-source software….it also shows that women face a giant hurdle of “gender bias” when others assess their work. This research also helps explain the bigger problem: why so many women who do enter tech don’t stick around in it, and often move on to other industries within 10 years. Why bang your head against the wall for longer than a decade?” [EDIT: the title has since been changed]

This article, and many many like it, bear absolutely no relationship to the actual data in the study.  Since the article of course is all most people even read, now a meme is created forever in social media that is just plain wrong.  Nice job media.


  1. Rob McMillin:

    SJWs gonna stupid.

  2. SamWah:

    Based on past performance, "CLEARLY" women are/must be more discriminated against. Saying otherwise harshes the mellow of The Narrative.

  3. Andrew_M_Garland:

    There is a mystery. Activists paint business owners as cutthroat pirates who would fire their grandmothers if it would save a dollar. Supposedly, these owners routinely sacrifice their morality and personal preferences to make a buck.

    ( For open source, project leaders supposedly would rather slow the development of their projects than accept worthy contributions from women. )

    But, these pirates supposedly pay men 1/3rd more than women ($1 vs $.76) to do jobs that the woman could do just as well. What accounts for this desire to increase their costs and lose money? In fact, why would they hire any man when a woman is available at such low rates?

  4. John O.:

    When do journalists have to follow ethics and report accurately and honestly?

  5. jon49:

    Is 1% even statistically significant? If not should it read that programming is mostly merit based?

  6. LastNameFirstNameLast✓ᴬᶰᵍʳʸ:

    Kerry Island.

  7. Maximum Liberty:

    Warren, there is a clear relationship between the research findings and the new headline: an inverse relationship.

  8. slocum:

    If you get a big enough sample size, it may be statistically significant (not likely due to chance), but it would still be a small, unimportant effect. Using a large n to get significance for tiny effects is generally frowned upon upon:

  9. Daublin:

    That seems to be the case here. The key phrase is "effect size". 1% is a pretty small effect.

  10. Matthew Slyfield:

    Your headline is missing a period after the fifth word.

  11. poitsplace .:

    I find that the vast majority of scientific studies quoted in the news fall somewhere between misquoted and just blatantly wrong. A large percentage are simply garbage...things like "we studied 38 people" and then going on to talk about all of humanity. Often they're clearly trying to come up with an answer because it's the politically correct/fundable answer even when the data literally says the the study that concluded rice production would decline with temperature when for all IPCC scenarios the data actually suggested temperature alone would increase yields (with CO2 fertilization factored in it would mean more than doubling the yields...according to their data).

    Science reporting in the mainstream media is near worthless.

  12. David in Michigan:

    Did you mean implied sexism by reports and research like this:

    Where the first sentence says men CAUSE menopause in women?
    "The onset of the menopause in humans has been linked to men failing to fly the nest."

    And then goes on to say:

    "The idea with this investigation is to compare species, and the idea is
    that when human males stay at home, the older women will enter the
    menopause because they are required to help out with raising younger
    members of the family rather than continuing to have their own

    They WILL enter menopause because it is required (mind you, REQUIRED). Because, well, men.