In short, 1) economists publish papers with results that are not replicable, and 2) few make enough effort to notice. Economists - we have a big problem here. It looks like we fib to each other, and then wink at the liars. This is not the way to run a science. (Having said that, an important caveat is that this doesn't mean that the results are wrong - only that the authors didn't make them easy to check, and no one bothered to point out that this leads to a potential for abuse).
I've just finished "Lessons from the JMCB Archive" by B.D. MuCullough, Kerry Anne McGeary, and Teresa D. Harrison. The article is forthcoming in the Journal of Money, Credit, and Banking. (For non-economists, this is a very good, but not quite top specialty journal in economics.)
Here's what they found at this journal. Sixty-five percent of elligible articles did not submit data (as required) to the journal data archive (for potential future replication studies). Of those that did, only 85% submitted data and the code to get results from it. Then the real problems start: data isn't readable, there is no way to confirm if the data loaded correctly, the author's output is not included to check what is generated by the replication, the software and operating system version are not noted, and researchers switch between applications without notice. Ultimately, only 10% of the published articles had results that were reproduced.
This sounds a lot like when the GAO refused to certify that the federal governments books were clean, because they couldn't figure out what they were doing.
Tip of the hat to Mark Wohar of the University of Nebraska at Omaha.
I had the opportunity to talk extensively with the primary author about related issues in 2000, and I have no doubt that he sincerely believes that our profession has some serious rot.
Further, scuttlebutt from a different source suggested that McCullough had difficulty on the job market because some regarded his research as harmful to the profession, in spite of a pretty sharp looking publication record (considerably more impressive than mine).
I have four personal examples to add to this.
1) My first (empirical) task on my dissertation was to duplicate results contained in two famous papers published in premier journals in the late 1970's by a researcher who now appears to be a lock for a Nobel prize. Several weeks yielded nothing close to the published results. A letter to the author got the response that this was a long time ago, the data was gone (or had been updated), and who knew where the punch cards for the program had gone. My committee advised to drop this pursuit, even though my advisor still had his data and cards from writing his dissertation in the mid-1960's. On my first job I was told by a graduate of another school that it was common knowledge there that these results could not be duplicated.
2) An acquaintance on one of my jobs had a much better pedigree than I did, and a better early publication record. He ran into a problem doing a final revision for a premier journal. He asked me for help (he is certainly better at theory than me, equal in econometrics, but not as good at programming). I found that his key results had serious problems related to the programming of his GMM estimators. His response was "who is the editor going to believe?". The paper is published and cited a lot (and actually, it is a rather good paper, I'm just not at all sure that it has any empirical foundation).
I know what you're going to say. I choked in the clutch. This is essentially correct. It is really clear early on in your academic career that it pays to choose your battles wisely.
Here's one I did choose wisely.
3) I refereed a paper for a good specialty journal. The authors had produced nonsense for the vintage of the paper (2000). There work was typical for that technique (cointegration) circa 1988 - essentially, they had access to better econometric tools than they were capable of using correctly. I told the editor to reject the paper. There is more editorial politics involved, but after a few steps in the process I told the editor that he needed to choose between me and the authors. He chose their side. The website lists that paper as the most heavily cited one in the journal over the last 5 years (downloaded over 500 times too).
Lastly, here is a positive example.
4) Very early in the history of Markov-switching models, I acquired a program from a professor and graduate student from a flagship state university. The program was beyond me, but it duplicated their results exactly. And the data was the original data from Hamilton's seminal Markov-switching model. Later, when I acquired programs for that one, it also duplicated his results accurately. Years later, when I made my own advance in this area (one that I got bored with, and didn't publish) I used that data again, and was roundly complemented at a major econometrics conference for being able to make a rock solid comparison between methodogies.
Since I am offering up my confessions, let me note that there are a lot of my former Ph.D. students out there who suffered through their dissertations because I made them do things properly. We are all better for that.
Comments