Quantifying the effect on economic outcomes of the Federal Communications Commission’s (“FCC”) regulations is vital for good policymaking. Recently, the Commission initiated a Notice of Proposed Rulemaking in which it is contemplating rolling back the Obama Administration’s controversial 2015 decision to reclassify broadband Internet access as a common carrier “telecommunications” service under Title II of the Communications Act of 1934. To aid the Agency with its review of its 2015 Open Internet Order, I authored a number of empirical studies examining the effect of Net Neutrality regulation on investment, employment, and broadband speeds, including both original research and commentary on the analysis of others (see here, here, here, here, here, here, here, here, here). Since Chairman Pai has repeatedly stated that he would like to raise the level of analytical rigor at the Commission, I took the time to search the docket for any meaningful criticisms of my empirical work. Since all my empirical work relevant to the Net Neutrality debate is based on standard methods and public data, I assumed that people in favor of keeping reclassification in place would have no problem understanding my analyses and could replicate my results. And, if they discovered an error, then they could bring it to my—and to the FCC’s—attention.
Most of the mentions of my works were favorable, but a few of parties offered comments and criticisms on my recent research including the Open Technology Institute at the New America Foundation, AARP, and Netflix. Of these, the criticisms levied against my work are either uncompelling or wrong. For instance, OTI claims my work on broadband speeds “contradicts a similar study from Akamai (OTI Reply Comments at p. 39),” when in fact Akamai did no statistical analysis and my research used the Akamai data. Netflix merely asserts that some part of the analysis on investment submitted to the record, possibly including my own but possibly not, is “inconclusive” or “a contrivance of cherry-picked data and analysis.” (Netflix Reply Comments at p. 3.) My work is neither, being both conclusive and reliant on standard empirical methods and publicly-available data. AARP expresses concern, for example, that the BEA data on telecommunications investment I used included the broadcasting sector. (AARP Comments at pp. 105-6.) As observed in my studies, however, broadcasting makes up only 3% of the investment data for the broad telecommunications sector, so the concern is obviously over-blown. AARP also noted that my work did not test the effect of the 2015 Open Internet Order, an alleged defect caused by a “faulty methodology and questionable assumptions.” (AARP Comments at pp. 54.) In fact, the government data I used (from the Bureau of Economic Analysis) in my earlier investment studies ended in year 2015, so it was a lack of data and not “methodology and assumption” that led to the omission, a fact explicitly noted in my paper. Furthermore, I addressed the investment impacts of the 2015 Open Internet Order in a later project, revealing continued and sizable negative effects. No party, to my knowledge, has questioned the validity of that subsequent analysis.
While I found no valid or meaningful criticisms of my work, one attempt to discredit it was so incoherent and inaccurate that I feel it is worth commenting on more fully to avoid confusion.
Free Press, the far-left organization advocating for the strongest forms of Internet regulation, concentrated its commentary in its Reply Comments on my analysis of employment effects. It is clear from its comments, however, that Free Press has zero comprehension of my empirical analysis.
Let’s review that work. In Policy Perspective No. 17-05: “Regulatory Revival” and Employment in Telecommunications, I applied the difference-in-differences (“DiD”) method to monthly employment data covering the years 2000 through 2015 from the Bureau of Labor Statistics (“BLS”). A counterfactual was established using other economic sectors chosen so that their employment trends were very similar to that observed in the telecommunications industry prior to the regulatory treatment (year 2010). So, the empirical question is: does the parallel relationship in employment between the telecommunications sector and the control group depart significantly after the 2010 proposal to reclassify broadband as a telecommunications services?
The DiD regression model revealed very large the employment effects. Had employment in the sector remained on trend (as established by the counterfactual), average employment in the treatment period would have been 941,394 jobs after 2010. Instead, actual employment during the treatment window was 847,145 jobs. As such, the FCC’s aggressive regulatory agenda resulted in an average loss of 94,249 jobs per year in the telecommunications sector alone after 2010, or approximately a 10% reduction in employment. The size of the employment effect—nearly 100,000 fewer jobs annually—was robust to the exclusion of time intervals and to different control groups.
In its assault on my work, Free Press employs provocative dicta as it always does, but its incendiary language is a poor disguise for gibberish. For instance, Free Press states that my analysis “ignored actual marketplace evidence.” (Free Press Reply Comments at p. 31.) Yet, I used BLS data on employment. And, Free Press’s presentation of “reality” in Figure 4 of its Reply Comments (id. at p. 31) uses exactly the same BLS data I employed in my study.
Free Press also claims my analysis is “fantasy numbers.” (Id. at p. 31.) As detailed in my Perspective, I used publicly-available data (from the Bureau of Labor Statistics) and the standard DiD regression method, neither of which constitute “fantasy.” As for the DiD method, which I deduce is the source of the “fantasy” comment, Free Press states that it is “a perfectly reasonable methodological approach.” (Id. at p. 30.) So, despite the vitriol, both my data and method are acceptable to Free Press.
The remainder of Free Press’s commentary reveals a failure to comprehend my work. Free Press claims my analysis shows “the telecom industry would have employment levels higher than those seen at the peak of the so-called ‘telecom bubble’ circa 2000.” (Id. at p. 30.) Put another way, Free Press alleges that “the Phoenix Center produced a strange result that would suggest the industry should have the industry should have 800,000 more jobs than it currently does—i.e., more than twice the current level.” (Id. at pp. 30-31.) I have no idea where Free Press got these ideas. Certainly, no such claim is made in my Perspective and the results of my analysis do not support Free Press’s claims. Indeed, as was stated plainly in the Perspective, the counterfactual is “941,394 jobs,” well-below the employment levels in 2000 (about 1.4 million jobs), and the average employment reduction was 94,249 jobs during the treatment period, not 800,000 jobs. More simply, I state in the Perspective the effect on jobs is a “10% reduction in telecommunications employment,” not 100% as Free Press claims. Such false testimony may be intentional or from ignorance, but my guess is that the latter is more accurate.
Free Press also claims the “the notion that [the control group] and the telecom sector should have continued to behave in the same manner and follow the same trajectory following the 2001–2002 recession, and leading up to the 2008 recession, is ludicrous.” In fact, as Figure 1 in my Perspective plainly shows, that is exactly what happened—the employment levels for the controls and the telecommunications sector follow the same trajectory after the 2001-2002 recession leading up to the 2008 recession. As stated in the Perspective, the growth rate of the control group between 2000 and 2010 was statistically indistinguishable from that of the telecommunications sector. In fact, this “same trajectory” is by design—this trajectory is how the control group members are selected.
Finally, Free Press admits that it “didn’t try and predict [] the ‘but for’ world.” (Id. at p. 31.) What Free Press fails to understand, it seems, is that its failure to conduct a counterfactual analysis is exactly why its own analysis is meaningless. Whether investment, employment, broadband speeds, or any other outcome of interest rise or fall after a specific date is not the relevant policy question—the relevant policy question is whether or not the outcomes change more or less than they would have absent the regulatory intervention. In this admission, Free Press concedes to the irrelevance of all its quantitative work.
I do agree with Free Press on one thing—the “Commission cannot rely on shoddy analysis.” In light of Free Press’s inexpert review of my employment analysis, the Commission certainly is precluded considering Free Press’s criticisms. But, I might go further. The unskillfulness exposed in its comments on my employment research and its explicit admission it conducts no counterfactual analysis could easily justify setting aside all of Free Press’s analyses submitted into the record on empirical matters, since it quite clear that organization does not have an acceptable level of expertise in quantitative work (a fact also demonstrated here, here, and here).