Introduction

I elected to analyze “Does society show differential attention to researchers based on gender and field?” by González-Betancor & Dorta-González (2023).

Study Goals

Previous research has shown evidence of gender and field bias in some altmetric areas, but not in others. This study was an attempt to dig deeper into these two areas of potential bias. Along the way, the researchers also develop three measures of social attention that aim to be independent of the author’s publishing count and to allow for comparisons across different fields.

Research Problem

We don’t have a clear understanding of the differences in altmetric attention by gender and by field, and we don’t have a clear way of comparing different altmetrics and fields because of their inherent differences.

Research Objectives

Know whether there is bias in altmetrics based on perceived gender of author. Know whether there is bias in altmetrics based on field of study. Have a way to compare different fields and different altmetric types effectively.

Research Questions

To quote the researchers directly, “is there a gender and/or field bias in the patterns of altmetric mentions -Twitter, news, policy, and Wikipedia mentions- for Spanish researchers with the highest altmetrics?” The question of how to compare fields and altmetric types is implied through the methodology and the conclusion.

Data

For each of the 22 fields classified within the Web of Science, the 250 top researchers per field were identified by Altmetric Attention Score. Genders of the researchers were inferred from first name with additional efforts to differentiate “in cases of doubt.” Some researchers were present in the top 250 in multiple fields, and after deduplication 4,195 unique researchers were considered. Each researcher was linked to their publications (article, review, or letter), and each publication was linked to altmetric mention counts by type (news, Twitter, Wikipedia, policy).

Analysis

Building on previous research, the authors used news, policy Wikipedia, and Twitter altmetric categories as proxies for measuring influence in media, politics, education, and the social sphere. In order to examine whether bias by field existed, they first ranked their dataset by total number of mentions to form a baseline that did not consider field or gender. Subsequent analyses compared alternate rankings against this baseline.

To create the alternate rankings, the authors devised three formulae. First, social attention orientation, which reflects how much relative attention a research received between the four altmetric categories. Second, level of social attention, an attempt to quantify the attention a researcher received in one altmetric across all of their publications. Third, intensity of social attention, which attempted to quantify the attention the researcher received in one altmetric across publications within that specific altmetric. The authors used a variety of statistical methods to show that these three measurements measure different aspects of social attention.

Results

First, the authors examined the prevalence of bias by field of study. The authors took the 4,195 unique researchers and applied each of the alternative ranking formulae across each of the altmetric categories and each of the fields of study. This resulted in 264 unique distributions, each graphed as a box-and-whiskers plot.

For each of the three ranking formulae, they examine where the most influential researchers for each field and compared them to the baseline ranking of all researchers in all fields. For the social attention orientation ranking, if the field were unbiased in its representation in each altmetric, there would be a uniform distribution. Instead, they show significant biases, e.g. biology is favoured in the news category. Similar non-uniform results were shown for the level of social attention ranking and the intensity of social attention ranking. In all three measures expressed through the ranking formulae, each altmetrics category showed deviation from a uniform distribution, indicating that researchers in certain fields received more attention in those categories than those in other fields did.

Second, the authors examined the prevalence of bias across field. They utilized statistical analysis to identify the prevalence of male vs female coded researcher by publication count, and by total number of altmetric mentions. This was done for each altmetric category individually, and with all four categories combined. This resulted in 198 numeric results, with positive numbers indicating a bias towards women researchers and negative numbers indicating a bias towards men, and the magnitude indicating the amount of bias observed. Each value was then compared to the average, with statistical significance noted at the 1%, 5% and 10% levels. By publication, 16 of the 22 examined fields showed a bias of at least 1%, 12 showing at least 10%, all biased in favour of men. Only 2 out of 22 fields showed bias in favour of women, at less than 1%.

Further analysis showed differences by field across the different altmetric categories. Policy was identified as the most egalitarian category. 41% of fields for news and Twitter showed significant gender differences (57% by publication, 25% by altmetric count). Policy and Wikipedia present analytical challenges given how many of the fields had no mentions in those categories. Specific numbers for researchers not present in the policy and Wikipedia categories were not identified in the paper’s text, but were indicated in two of the graphs; for policy, approximately 77% of women and 81% of men were not found, and for Wikipedia 75% of women and 66% of men were not found.

Critical Assessment

Overall, I found this paper to be significantly over-complicated in its approach to answering the research questions. For the examination of bias by field, the authors chose to analyze the rankings of the researchers who wrote the papers that were mentioned, rather than simply examining the prevalence of altmetric mentions by the paper’s fields. All three of the ranking formulae devised were based on the publication count and altmetric count by field and category, independent of the researcher. Analyzing distributions of researchers rather than their papers includes analytical complexity not required to directly address the research question as the authors posed it. Several of the altmetric categories lacked any mentions of some fields, resulting in figures and table with gaps and scale differences that made comparisons difficult. The lack of definitions for critical terms like “social mention,” the way that the ranking formulae were defined in the article, and the resulting figures and explanations, presented significant difficulty in understanding the methodology and the implication of the results.

For the examination by gender, the explanation of how the researchers were assigned a gender within the analysis was problematically insufficient. In the majority of cases the gender was “inferred” by the authors, but no explanation of how this was done was provided. The researchers stated they used additional sources of information “in cases of doubt,” but did not specify what constituted a case of doubt or a formal process for how additional sources were consulted. Given the absence of explanation, I do not believe it would be appropriate to put any confidence in the analysis and conclusions made on the gender of the researcher dataset.

The dataset of researchers used was from the Altmetrics platform and limited to researchers in Spain. No consideration was given in the analysis for whether the results would apply more generally to Western researchers, or how the altmetric mentions are not similarly limited to Spain.

The authors make several significant assumptions in their discussion. “While gender differences in altmetrics have been observed in research studies, they do not represent an inherent bias in altmetrics themselves. Rather, they reflect broader societal and systemic factors that affect gender representation and visibility in academic contexts.” No basis is given for this assertion. “Importantly, excluding self-citations and non-research active years would not disadvantage anyone, as… researchers should be judged solely on their research active careers.” This assumption was made immediately following an acknowledgement that leaving research is a leading cause of discrepancy between genders in overall career impact, and assumes that researchers of both genders leave research at equal rates, something we know is untrue. “They distinguish them by the gender of the first author to find the differences, because most of the scientific publications they studied were co-authored by female and male researchers.” This places significant weight on the ordering of author names, and wouldn’t account for cases where the names are merely listed alphabetically, or cases where a single male author is listed first and thirty female authors listed subsequently. “Male researchers are more likely to value and engage in research aimed primarily at scientific progress, which is more highly cited. Female researchers, on the other hand, are more likely to value and engage in research aimed primarily at contributing to societal progress, which has higher usage.” This is interesting if true, but it is uncited, and could easily reflect social norms of under-recognizing the contributions of women in areas of scientific progress.

Perhaps the most significant assumption the authors make is this: “The mentions that publications receive on different digital platforms make it possible to quantify and measure the impact of scientific research.“ Attention is not the same as impact. No effort was made during this study to determine if the altmetric mentions were supportive, neutral, or critical of the publications, and no effort was made to correlate attention with impact. Conflating attention with impact is an error the authors make repeatedly in the paper, despite themselves acknowledging that they are not considering audience size, origin of policy documents, reputation of news organization, other social media and education platforms, or sentiment analysis.

In conclusion, I don’t believe any of the results or conclusions related to gender are worthy of consideration given the problematic data, and the results and conclusions related to bias in different fields across different altmetrics were over-complicated and did not directly address the research questions identified by the authors in the introduction.

References

González-Betancor, S. M., & Dorta-González, P. (2023). Does society show differential attention to researchers based on gender and field? Journal of Informetrics, 17(4), 101452. https://doi.org/10.1016/j.joi.2023.101452