There’s been a lot sparking between some sociologists and economists about the value of the “p-value” in econometric/regression analysis. Noah Smith points out that even the field of Psychology has declared open season on statistical p-values. One missing thing from the discussion though, is that the “p” in “p-value” stands for “Probability.” This changes the tone of the discussion. All statistics are, are a set of probabilities. They are not “facts,” regardless of how much they get treated as such.
The closest that statistics can come to describing “facts” are in the Durkheimian form of “social facts” (from Sociology). However, social facts change over time, so the inference of a social fact only applies in the time that the data was created. For example, in 2014, the General Social Survey in the United States (University of Chicago) showed that 53% of 60,000 people thought that Economics was not a science.
This is a social fact. But it doesn’t really explain much. It just tells us that in 2014, this is what people believed. Why did they believe it? What biases could they have? Who were more likely to believe that? And will that be the case 10 years from now? It’s a social fact that 53% of people do not believe that Economics is a science. It also leads us to more questions to think about. This is what makes statistics a tool.
Other than social facts, statistics doesn’t have much relationship with facts. Rule #1 in statistics: correlation is not causation. That means that just because A has a probability of correlating with B does not mean that A causes B (or vice versa). That’s why statistics are not factual.
For example, consider the standard regression model:
This is just a prediction of a probability that at a certain point (intercept), a given amount of X will change a given amount of Y (slope), accounting for the “unknown” (error term). Or, it can be seen as the probability of accurate prediction. We look at p-values at certain levels of significance to show the accuracy of the prediction outside of the “unknown,” but at the end of the day, it does not describe all facts for all people in all levels of society. Even if our regression has a p-value of .0000001 there is still a probability that it could be wrong.
For sure, in OLS regression models it’s also important to look at the sum of squares of the residuals as well as coefficients to see what’s going on in the prediction probability. For example, I wrote a regression model to answer why the labor force was shrinking (participation rate). I used 13 labor variables, both together and independently. My p-value was .001 at the 99% level of significance. But I had no really high coefficients, and the sum of squares of my residuals was really high. When I started testing NON-LABOR variables, I was shocked to find monetary policy (specifically Quantitate Easing) showing the really high coefficient, a really low SoS on residuals and good p-values at the 99% level. I had my smoking gun.
But that’s not the end of the story. It simply raises more questions, some of which can never be answered. While I can say that there is a correlated relationship between lower labor force participation and QE policy, the next question is why. Could it be that most of the QE has ended up not making it to the economy? Is it that because QE causes market bubbles? Is it that because the Central Bank pays interest on QE money? Who knows? And it’s not a “fact” that QE effects the labor force, it’s simply a probability that it does given no other factors. This means that there is a probability that it does not. And while I have an “unknown” factored in, do I have ALL of the unknowns factored in? Probably not.
Statistics is a philosophy, not a math. Calculus, Algebra, and the rest of them is a method of describing mathematics. Statistics is also a method, not to describe mathematics, but rather to describe probabilities. Both are tools to think about the totality of the human experience, including public policy.
We know that there is a statistical relationship between education, occupation, and income (socioeconomic status). If we want a public policy that allows for socioeconomic mobility, then we should remove barriers to education, enact labor rights, and equalize incomes on a progressive tax basis. Why? Because there’s a really high probability that this will work toward a common social goal. This is the philosophy of statistics that allows us to think about the human experience.
Then (as Noah points out) there are the ethical concerns of using the “tool” of statistics. P-hacking, data manipulation, and cherry picking data are just a few. How we get the data also has ethical issues. In Sociology, before we were allowed to use statistics, we had to take courses on ethics and methodology. In Economics, before we were allowed to use statistics, we had to take courses on how to manipulate data into grotesque, unrecognizable forms without ever thinking about the ethics. This is (I suspect) how statistics get passed along as “facts.”
If citing statistics invokes no further questions about the research question, then the statistical significance of the statistic is zero.