You can read scientific and professional literature on multiple levels. Because you will have to read a great deal in your career, you generally want to put the least time into each article while still getting the most out of it. In this note, I distinguish three levels of reading — Level 1, Level 2, and Level 3 — each of which requires more time and intellectual attention. If you are assigned a stack of articles to read for a graduate class for one week, you probably should first give each a Level 1 reading before deciding whether to allocate additional time to any particular piece of scholarship.
At a minimum, you need to do a Level 1 reading to decide whether an article merits more serious study. If it does, a Level 2 reading requires more work but provides more rewards in terms of understanding. Level 3 readings, finally, are usually only necessary if you are working on the same or a very closely related topic and want to unpack the details of a prior study in order to build on it.
Level 1: This reading is to capture the main purpose and to define the intellectual contribution of the article. To do this requires that you learn more than merely the topic of the article; often, however, you only have to skim to complete a Level 1 reading, certainly in an area with which you are already very familiar. What is main analytic point of the article? If you were to cite the study in your own work, why would you do so? What is its principal contribution to the literature? Why is it important? At this level, you should also make note of the kind of data used and the empirical or theoretical setting of the study.
Some — but not all — of a Level 1 reading should be available from the article’s abstract. Even a good abstract will not explain the place of the study in the existing literature, however. Your ability to understand the article’s contribution hinges on how well you already know the area of research. If you are just starting investigation into a new area or topic, and lack much intellectual background, you may not be able to complete a Level 1 reading until after you have read a whole stack of articles in the area. In this case, you may need to read an article at least twice to successfully complete a Level 1 reading.
When you have successfully completed a Level 1 reading, you should be able to summarize the main point of an article and its scientific contribution in two to four sentences. This means that a Level 1 reading results in one written paragraph of notes. Your mastery of a literature depends on being able to do this.
Although a Level 1 reading is not acritical, it does not penetrate deeply into the research design and analytic structure of a piece of work. After completing a reading at this level, you might judge that you did not wish to put more time into the article due to the nature of the data, the process of data collection, or the place of the study in the broader literature.
Level 2: This level of reading critically evaluates the basic analytic structure and claims of an article. The central purpose of a Level 2 reading is to assess the credibility of the claims made by the author. Do the design, analysis, data and interpretation all support the overall claims made? How is the research designed to investigate the problem at hand? What specific hypotheses are evaluated? Is the research design suitable to generating causal inferences? How are the cases selected? What kind of data is collected and how? Are the results as expected, and if not, are the interpretations that the author makes supported by the research design and the results?
This level of reading begins to unpack the internal structure of the analysis that the article reports. It reports how the research was conducted and whether the research design, data analysis, and interpretation appear to be sound. At this level, you will often discover questionable aspects of the work. The strongest pieces of research anticipate your questions and doubts, explicitly responding to them.
You engage in a Level 2 reading when in a seminar you are asked to critique a piece of scholarship or when you serve as a discussant of a paper at a conference. Your goal is evaluate the soundness of the study, which means pointing out the original contribution and analytical strengths of a piece of work as well as identifying weaknesses. Ideally, your critique is constructive, meaning that you are able to propose things the authors could have done that would have improved their work.
Level 3: This level of reading requires replicating the entire data analysis reported in the article. Here you download the data and any relevant codebooks and computer files and re-run the entire analysis that is reported. This requires understanding and duplicating the construction of every variable that is analyzed and then duplicating the statistical work to produce every table and figure. Often some details essential to understanding this process will be buried in an appendix, so you have to read these carefully.
There are many things to be learned from a replication exercise. If you are still in the early stages of your graduate education, going through the steps to replicate a published article offers you an appreciation for the complexity and subtly of the many decisions involved in moving from raw data to published output. If you are working on a particular problem, replicating an important piece of scholarship that serves as foundational may bring to light new and perhaps unsolved aspects of the problem you had not previously considered. Why did the authors code a specific variable as they did, and what happens if you select an equally defensible recoding? Why were some control variables included in some specifications and not others, and what happens if you drop or add some controls? Why was a particular estimator used, and what happens if you use an alternative? Why were some observations deleted from the analysis? Only when you attempt a complete replication of a study do you drill down to these kinds of questions.
In essence, a Level 3 readings allows you to ask how robust the findings of a study are. Science proceeds when we are able to replicate reported findings, so replication exercises are essential.
Often a Level 2 reading naturally evolves into a Level 3 reading for a few specific parts of a study, perhaps ones that you found especially puzzling or questionable. There is no rule that you have to replicate everything in an article. A partial replication may also serve some purposes.