The Quandaries of Data Analysis and Methodologies in Rule of Law, Development, and Human Rights Assessments: New Challenges for UN Special Rapporteurs

If Professor Hans Rosling’s famous last opus, Factfulness (April 2018), is to be believed (as well as Bill Gates’ effusive review here), we all tend to have grimmer views of the state of economic development in the world than actually borne out by reality – especially on issues of global health and poverty.  Referring to the “developed world” and the “developing world” is a meaningless and unhelpful binary that glosses over significant welfare, health, life expectancy, education, and human capabilities differences between and among the hugely diverse “middle income countries” (e.g. the World Bank divides them into “lower middle-income economies – those with a GNI per capita between $1,006 and $3,955;  and upper middle-income economies –  those with a GNI per capita between $3,956 and $12,235 (2018)). Even the World Bank stopped using the distinction between “developing” and “developed” countries starting with its 2016 World Development Indicators, ultimately concurring with the view that the “developing country” and “developed country” distinction was not useful and too broad for targeting international development programs for partner countries, especially when assessing progress in all 17 of the Sustainable Development Goals (SDGs). The United Nations does not have a formal definition of “developed countries” versus “developing countries”, instead insisting that its classifications in UN methodology are “for statistical convenience and does not express a judgement about the stage reached by a particular country or area in the development process”.  However, these categories are widely used in the UN system anyway, including in the UN’s 2017 Sustainable Development Goals Report as well as in the datasets used for the 2018 SDG16 Data Initiative Global Report (on the goal of achieving peaceful, just, and inclusive societies). If the distinction between “developing country” and “developed country” is operationally meaningless for formulating and evaluating development programming, shouldn’t international lawyers and scholars also take note of the imprecision of this category when putting forward their observations and assessments of the state of rule of law, economic development, poverty, and human rights in the world?  (Note:  I do plead guilty to having, in previous works, alluded to the same classifications.)

Professor Rosling’s opus came to mind recently after the debate spurred from recent sharp criticisms issued by US Ambassador Nikki Haley and by experts from the Heritage Foundation, against UN Special Rapporteur for Extreme Poverty and Human Rights and NYU Law Professor Philip Alston, who had issued several statements (see here, here, and here) and his full 4 May 2018 report on the state of poverty within the United States (finding, among others, that 40 million people across the United States live in poverty, while 18.5 million live in extreme poverty, and an additional 5 million in conditions of absolute poverty; or alternatively put, “1 in 8 Americans now live in poverty, with half of this population living in extreme poverty, according to U.S. government estimates.”).  While Ambassador Haley alleged that the report “categorically misstated the progress the United States has made in addressing poverty and purposely used misleading facts and figures in its biased reporting”, the Heritage Foundation challenged the income-based poverty measures used by Professor Alston and charged that “these “official” income figures exclude substantial off-the-books earnings among low-income households and omit roughly 95 percent of the $1.1 trillion U.S. taxpayers provide in means-tested cash, food, housing, and medical benefits for low-income persons each year.”  Professor Alston has criticized the United States’ withdrawal from the UN Human Rights Council, and charged that the Trump Administration was exacerbating poverty for millions of Americans.  The Special Rapporteur has not yet responded to the challenges against the data sources used, and the quantitative and qualitative methodologies used for this fact-finding mission and its conclusions.

When I examined the index of currently publicly available Reports of the Special Rapporteur for Extreme Poverty and Human Rights, it was somewhat surprising that there was no separate initial report on the quantitative or qualitative methodologies adopted for the country assessments on the state of extreme poverty and human rights (although each country assessment thus far discusses observations from anecdotal evidence, official statistics from government sources, and other sources).  Considering the very difficult remit of Professor Alston – who as Rapporteur is an unpaid expert and admittedly not an official of the United Nations – I wondered why it did not appear from the reports that the UN’s considerable resources on SDG monitoring and assessment (especially SDG1 on eradicating poverty), as well as on economic vulnerability and risk monitoring, had been deployed and allocated to assist in the Special Rapporteur’s challenging fact-finding mandate.  Note that Professor Alston has himself championed interdisciplinarity and cross-verification in human rights fact-finding and yet, the UN remains unable to reasonably coordinate its resources, data, and interdisciplinary expertise before it dispatches its unpaid experts for overwhelmingly difficult fact-finding missions as “Special Rapporteurs”.

Interestingly, Andrew Keane’s recent Lawfare post (“Do Constitutional and Human Rights Matter?”) challenged the methods by which previous empirical scholarship tried to compare state behavior and state commitments, to determine the supposed relevance (or irrelevance) of international human rights law to achieving desired human rights outcomes.  As Andrew rightly points out, the answer isn’t that simple:

“These studies typically examine state behavior over a relatively short time span, whereas rights can take decades or centuries to mature. Indeed, if one were to deploy the same methodology used by the scholars of quantitative rights to study, say, the Bill of Rights in the 19th century, one might have concluded—somewhat prematurely—that the Bill of Rights did not matter. These quantitative studies have other defects as well. They typically rely on an overly restrictive dependent variable, as Ryan Goodman and Derek Jinks note. That is, they typically focus only on state action, though rights can of course be violated by nonstate actors as well. And perhaps most troublingly, these studies typically rely on quantitative indicators of well-being that give at best a weak measure of rights fulfillment. These quantitative indicators—data sets that attempt to score a state’s performance along some human rights measure—are notoriously flawed.”

Similar observations could be made about how Special Rapporteurs discharge their mandates under limited horizons of time for investigation, varied in-country knowledge and in-country experience, scant resources and inadequate institutional support, divergent quantitative/qualitative methodologies and non-standardized evidentiary techniques used to determine the existence, extent, and causes of human rights violations, and in some cases, the liability of individual leaders or governments for such violations. That is a tall order for any single social scientist (whether in economics, political science, statistical methods, etc.) to fulfill, let alone the distinguished international law experts that the UN Human Rights Council tends to appoint. At least from my anecdotal lens of being a scholar born and educated in a developing country (or to be more accurate for some, a lower middle income country in the 1970s and now an upper middle income country in 2018), working in the last decade as an academic at a university in the developed world (or at least as the UN would call the United States a developed country) and especially now in a university that interdisciplinarily engages integral human development, the ever-expanding and proliferating mandates of UN Special Rapporteurs (especially in the areas of human rights, rule of law, and development) poses Herculean challenges of: 1) managing atmospheric expectations about the reporting missions; 2) transparently reflecting one’s normative aspirations in the fact-finding processes; 3) engaging all possible stakeholders feasibly and reasonably; and 4) withstanding the test of methodological scrutiny, reliability, and verification of the actual reports generated from fact-finding.

On the one hand, in most cases, there is no question of the impeccable credentials and track records of experts serving as UN Special Rapporteurs, even if the selection process for Special Rapporteurs has been described as “somewhat inscrutable, perhaps even byzantine.”  On the other hand, they are paradoxically expected to function as independent and unpaid experts under the special procedures of the UN’s Human Rights Council, even if the range of their activities are not expected to be completely apolitical since Special Rapporteurs are expected to:

“undertake country visits; act on individual cases and concerns of a broader, structural nature by sending communications to States and others in which they bring alleged violations or abuses to their attention; conduct thematic studies and convene expert consultations, contribute to the development of international human rights standards, engage in advocacy, raise public awareness, and provide advice for technical cooperation.” (Italics added.)

Special Rapporteurs’ discursive, critical, and advocacy roles are broad, and yet also delicate – while some States may welcome the additional investigation (as the Philippines did in 2007 when Philip Alston was the Special Rapporteur on Extrajudicial Killings), others may attempt to discredit the Rapporteur for being politically motivated (as the Philippines declined allowing entry to Agnes Callamard in 2017 as the new Special Rapporteur on Extrajudicial Killings).  Because Special Rapporteurs are expected to straddle the line between reporting and human rights advocacy in their missions, it may well be a public badge of honor for them to incur the ire of States, even if this may ultimately deter a fruitful cooperative dialogue on future policy reforms and long-term solutions to human rights challenges.  But those are always considerations to calibrate, and normative decisions to make, for the Human Rights Council when it deploys Special Rapporteurs on fact-finding missions to various countries.

In an era where the United Nations publicly hails the advantages of “Big Data for Sustainable Development”, international law and international institutions such as the United Nations system remains challenged, not just to develop the best, open, testable, and verifiable methodologies for fact-finding and assessment of rule of law, development, and human rights  – but also to move towards depoliticized perceptions about the reliability, veracity, and impartiality of human rights monitoring, reporting, fact-finding, assessment, in a manner that carefully and methodically engages all stakeholder critiques about the nexus of causation, the attribution of conduct to States, and the extent of injury caused to individuals, groups, and communities as the actual subjects and objects of international human rights law, international development law, and international rule of law. This, in my view, makes it difficult to draw completely authoritative grand narratives and broad-brush conclusions about the history or status of international law or the state of human rights in the world, much less to boldly and emphatically put forward damning descriptors about the historical trajectory of human rights as either “the last utopia” or just “not enough” to meet material inequality.  Neither am I inclined to lionize the international human rights system either as a paragon of virtue(s), even if there are many instances throughout the history of human civilization when human rights has also worked.  The gathering and evaluation of data about human rights outcomes over the long march of time means that the stories we tell about human rights outcomes may be accurate for certain periods, but possibly inaccurate for longer epochs of history. The non-standardized approach to gathering and assessing such data – whether by historians, statisticians, lawyers, or other social scientists – also means that the way we characterize human rights outcomes will necessarily differ.  To date, there are some promising signs that the UN Office of the Commissioner on Human Rights at least is moving towards crystallizing a definitive approach to its use and verification of data, when it issued its 2018 Human Rights-Based Approach to Data, establishing best practices and guiding principles (participation, data disaggregation, self-identification, transparency, privacy, and accountability) for human rights data collection and disaggregation.  But this is all still a work in progress in 2018.

Given the complexity of sources of international law and data on legal systems, institutions, and outcomes on development, rule of law, and human rights around the world, and the increasing trend towards microcoscopic lenses on information about human rights outcomes, I suspect it is not that easy anymore to draw any single characterization or descriptive theme about the state and trajectory of international human rights law as it used to be for earlier scholars. Precisely because international law stands, as Sir Hersch Lauterpacht famously observed, at the “vanishing point of law” – always at the fragile precipice oscillating between lex ferenda and lex lata – today’s international law scholars take on greater challenges to disambiguate the content of international law, from the effectiveness of its means, and the advocacy of its ends.  We should embrace the challenge of questioning our own assumptions and conclusions (especially our cognitive biases) if they are driven by our best possible assessment of what is international law, as supposed to our aspirations for its progressive codification. Likewise, we as international lawyers and academics also are burdened to be as transparent as we can to all stakeholders in the international system (States, organizations, individuals, groups, communities) with our factual findings and its limits, as well as the bases for our postulations and descriptions of the state of our discipline and profession in the art and science of international law.  We can begin depoliticizing the perception of our scholarly and practitioner investigations and assessments when we are transparent about the assumptions, data sources, methods, and limits of our qualitative and quantitative analyses – and all the more careful about the conclusions we draw from them.  In this respect, the distinguished international law experts appointed as UN Special Rapporteurs carry this heavier burden when they report and advocate at the “vanishing point” of international human rights law.