Several years ago, Professor Tom Nichols wrote an interesting essay entitled “The Death of Expertise.”  In this essay, he makes a simple yet profound insight: in our increasingly politicized world, there is a growing disregard for serious work done by qualified experts in our public debate.  Accordingly, warns Professor Nichols, “unless we return [expertise] to a healthy role in public policy, we are going to have a stupider less productive arguments every day.”

As I have unfortunately had to document on many occasions, nowhere is this “death of expertise” more acute than in copyright policy.

The latest example, it appears, comes from American University professors Sean Flynn and Michael Palmedo, who recently released a working paper entitled The User Rights Database: Measuring the Impact of Copyright Balance.  Here, the authors assert, before any of the empirical analysis is conducted, that nearly all countries do not “have sufficient user rights most needed to support the digital economy.”  They go on to conclude, “[o]ur data supports the existing theoretical literature that suggests that more open user rights [e.g. fair use] promote innovation and creativity.”

In fact, their “data” shows no such thing—it is their poorly chosen empirical methods that are fully responsible for the results.  Even a cursory glance at the study reveals numerous and substantial errors in the empirical analysis, many of which can be demonstrated in replication of the original work.

Flynn and Palmedo seek to find evidence supporting their cause of expanding the application of copyright’s fair use.  Fair use is a legal concept that, subject to certain caveats, permits online intermediaries like Google to avoid copyright infringement claims for using others’ property without permission or compensation.  Flynn and Palmedo assert their empirical analysis demonstrates, among other things, that generous fair use policies and other open-ended exceptions and limitations to copyright protections have a “positive effect” on economic outcomes.

Flynn and Palmedo’s choices of outcomes to study are a bit strange.  For instance, the authors conclude more expansive fair use policies “are associated with higher firm revenues [and net incomes] in information industries, including software, and computer systems design,” and show that fair use makes some businesses more profitable.  So, Flynn and Palmedo demonstrate that fair use makes some firms larger and more profitable on average.  Is that a good thing?

To arrive at these conclusions, Flynn and Palmedo created a “User Rights Database,” which is comprised of answers to survey questions posed to handpicked legal experts in an oddly selected sample of twenty-two countries—including Botswana and the United States, Chile and China.  They then purport to test the data using econometric techniques.

Based upon my initial review, which I will turn into a formal critique shortly, it is quite apparent that Flynn and Palmedo’s methods and construction of the “openness score” are fatally flawed.  But before I turn to specifics, perhaps first a little more context.

Why Are Some Calling for Countries to Adopt Fair Use?

Copyright aims to provide incentives toward the creation of new works, and in some cases borrowing small pieces of existing works aids in the creation of new things.  For this reason, the doctrine of fair use traditionally allows for the use of copyrighted material without permission or compensation, but only for certain socially beneficial purposes—such as criticism, parody, scholarship, and news gathering.  However, in recent years, US courts have expanded the scope of fair use to include a use that is sufficiently “transformative.”  Given that information is the currency of the Internet, Google and other big data firms are now keen to export this expanding legal paradigm to other countries, thus allowing them to evade paying for the cost of their key input of production—content created by others.

Right on cue, Flynn and Palmedo released a statement yesterday citing their paper, signed by academics and lobbyists from Canada, Mexico and the United States demanding NAFTA include expansive user rights.  Spurious scholarship travels fast through an echo chamber.

However, given the multitude of errors in their analysis, my recommendation is that policymakers ignore Flynn and Palmedo’s study and policy claims based on it.

The Data: Spurious Correlations and Methodological Errors

In my preliminary review of Flynn and Palmedo’s work, including replication of much of the empirical analysis, it is plain that their analysis is purely ad hoc, their statistical results spurious, and the interpretation of them invalid.  The reader is presented a series of regressions that have no theoretical motivation, are carelessly specified, and are estimated improperly, thereby satisfying none of the basic standards of empirical research.

Essentially, the results offered by Flynn and Palmedo are nothing more than a string of spurious correlations resulting from methodological blunders.  And, if one chooses to look past these problems, if anything, then their empirics reveal that expanding fair use leads to increased market power for some information-sector firms, an interpretation lost on the authors as a result of their ad hoc modeling choices.

I’m not done with my analysis of their empirical results, but a few problems I have detected thus far include:

• The study conducts no tests on the effect of fair use on “innovation and creativity” as the authors claim.  Instead, their results suggest that the revenues and profits of some firms get larger when fair use is more lax.
• Flynn and Palmedo have no “identification strategy,” which means they do not design the empirical test so that it plausibly has a causal interpretation.  In fact, they provide strong evidence of selection bias and covariate imbalance when noting that high-income countries have higher openness scores.  If better economies have higher scores, then it is no surprise that they would find higher scores are associated with better economic outcomes.  Flynn and Palmedo make no effort to address these serious statistical problems, so the results are near certain to be spurious.  Indeed, it can be shown that their openness index is correlated with all sorts of things across the sample countries —including fisheries production, gas prices, the amount of rain, the birth rate, and tourism expenditures.
• The index of “openness” is carelessly constructed.  The survey responses include four possible answers (not included, probably not included, probably included, clearly included) to over 100 questions on an array of copyright concepts.  The answers are ordinal in nature (and few in count) and measures different things.  Certainly, these answers do not suggest a linear scale of importance.  Despite these facts, the “openness score” is computed simply by taking an average of these ordinal responses and sticking that in a regression model.  This approach is clearly problematic, especially given that the answers reflect widely disparate policies (e.g., fair use or safe harbors) and involve only a few responses (four).
• Flynn and Palmedo’s analysis does not evaluate whether or not fair use makes society better off. Even looking past its methodological flaws, the work merely demonstrates that some firms are, on average, larger and more profitable in countries with more “open” user rights.  But what does that prove?  Inflating some firms’ profit margins says nothing about innovation and creativity, or the proper design of copyright law.  An objective observer could just as easily conclude expansive fair use is positively correlated with anti-competitive monopoly power.  In fact, the market power interpretation flows directly from basic economic theory.
• Flynn and Palmedo avoid use of the most obvious and routine empirical techniques for the sorts of data they use.  My replication of the portions of their analysis for which data is available may reveal why—the statistically-significant results supporting their desired outcomes vanish when applying standard methods or even when excluding single countries from the sample.  These are the telltale signs of spurious correlation.

Debates about the contours of copyright in the Digital Age are important, as there is a sea-change underway in how content is created, distributed and consumed.  But those discussions should be informed by rigorous and unbiased research.  Based on my ongoing review, it is apparent that Flynn and Palmedo have failed to meet such standards.