How Do You Swim If No One Teaches You?
I sat across from my professor in silence for a minute before he spoke.
I was too embarrassed to look at him and I just remember a few phrases from the conversation such as:
“I thought you had something between your ears”, and
“I usually don’t give people a second chance, but…”
Previously, when I had performed poorly in a class I always knew what I would have done differently if I could start over.
I would have started the assignments earlier, sought help from teaching assistants, or done more practice tests.
This class was entirely different. There were no exams. We had no teaching assistants.
We were graded on an oral presentation (easy) and a term paper based on a literature review of 200+ papers (ugh!).
As a first year graduate student, the most complex papers I had ever written were 15-20 pages long, and they never had more than 10 references.
I had no clue where to begin, so I did what I was best at: sitting at my desk for hours reading.
It’s fair to say that I worked at least 20 hours a week on this paper for 2 months straight.
After 160 hours of reading over 200 papers, I handed in a 20 page paper and I was not proud of it.
I knew my professor was not happy either when I got an email that he wanted to see me right away.
At least he gave me a second chance. It was 2 weeks before the end of the semester and he said:
“Just write a more coherent paper based on the 50-100 most important references, and I will give you a B. Your oral presentation was good, so I will consider that.”
I took his advice and he gave me a B.
In retrospect, I know that I could have written a more thorough paper and gotten an A.
Once you have a well-structured review of 50 papers, adding another 50-100 references is straightforward.
A cutting-edge literature is well within your capability – and it is not as complex or time-consuming as you might think.
There Is a Process
Fast forward 5 years to my first week as a postdoctoral fellow.
My supervisor had just been asked to write chapter for a book and, as I had no project yet, I was asked whether I would like to have the opportunity to write this chapter.
How could I say “No” to my new boss?
I already had one published literature review paper under my belt from graduate school, and even though this topic was not related to my thesis, I decided to take the opportunity.
Five months later (a week before the due date for the book chapter) I handed in a final copy to my supervisor. I still remember the pleasantly surprised look on his face when he said:
“Wow, I think you are the first person in the history of this book publisher, who handed in their chapter by the deadline.”
Apparently, I was not the only one who had struggled in the past with literature reviews.
While my first experience with a literature review was humiliating, it taught me a very important lesson about writing: start simple, and get fancy later.
As a senior graduate student and a postdoc, I didn’t start out by reading and trying to absorb 200+ references before writing.
I already knew that it was more important to have a structure than a long list of references.
By then, I had also developed a writing process, so I was able to minimize writer’s block. Note that I wrote minimize, not eliminate, writer’s block.
Writer’s block is like the common cold – it is always around you and you have to actively take action to prevent it from infecting you.
Everyone has a slightly different manifestation of writer’s block:
- Internet surfing/social media/email when you know you should be writing,
- Reading for hours because you never feel ready to write,
- Doing chores around the house even though they are not urgent,
- Helping others because you feel too stuck to work on your own project, or
- Socializing or going to the movies, because you feel too overwhelmed to work on your paper.
I was guilty of all of the above to some extent while writing my papers and my thesis.
There is no shame in having writer’s block. Every writer has it. Every day.
The only difference between writing my first literature review for my class and my book chapter 5 years later was that I recognized when I had writer’s block and I knew how to deal with.
3 Steps To an Outstanding Literature Review
I had a very simple writing process in college for my 20 page term papers.
After reading 5-10 references, I sat down and wrote the whole paper in 2 days and, when I did not oversleep the day it was due, I proofread it quickly right before handing it in.
This process did not work for academic review articles that had to survey the entire literature.
I had to develop a new system to collect hundreds of papers and analyze them in one succint literature review.
Step 1: Set up a structure for your paper
In retrospect, I believe that the main reason I struggled with my first paper was that the sections were not defined clearly.
I blindly buried myself in the literature reading without a well-defined purpose for months.
Years later when I had to write my book chapter, I spent several days setting up an outline. It was a bit of work upfront, but the payoff was well worth it.
Once you have a structure for your manuscript, you can read papers more efficiently.
You will no longer be reviewing 100 papers for your literature review, but only 10 papers for Section 1, and 10 papers for Section 2 etc. Doesn’t that sound more doable?
At the end of this step your outline should include all the major sections and most of the subsections as well.
Remember, you are not married to your outline. It can, and it will chance. That’s okay. As long as you are synthesizing the literature and answering relevant questions, you are making progress.
All the pieces will fall into the right place if you keep writing.
Step 2: Develop a timeline with specific milestones for each section
Now you have your outline, and it probably feels great. I know that one of the turning points for me in graduate school occurred when I finally had a vision of my table of contents.
It was liberating to have a section to put each bucket of data into.
However, your paper will not write itself. A literature review can take anywhere from 2-6 months depending on how many hours a day you work on it.
As humans we don’t have a good sense of time, and even after years of experience we always overestimate how much we can do.
It just feels good to imagine that the literature review will be completed in 2 months, right?
Since our sense of time is skewed, we usually leave things to the last minute (or last few days) at the expense of the quality of our work.
If you want to write a high quality manuscript, a timeline with specific milestones is essential. Once you have the structure, set a due date for each section.
Having a date written down, will help you to stay on track for each section and reduce the overwhelm.
Be mindful that almost no one follows their timeline exactly. Life happens.
Thus, you will probably need to adjust your timeline weekly, or even daily as you get closer to your deadline.
Step 3: Elaborate your paper through each stage of writing
One of the biggest sources of frustration is that students expect a polished document to flow from their head and to the paper. This never happens.
Ideas are not formed completely in your brain.
You have an incoherent mix of thoughts in your mind, and they will come out incoherently the first time. This scares students and they automatically assume that they are not good writers.
I used to assume that I was a poor writer as well, and I blamed it on not being an native English speaker.
However, once I realized that prolific authors (many of whom are also not native speakers) had messy first drafts and fought writer’s block on a daily basis, I felt relieved.
I realized that writing went through stages, and you have to go through each stage to have a manuscript that is rich in content and well-structured.
Zero draft: This is the idea collection stage. In this phase your ideas do not have to come out coherently. You are just trying to get as many ideas on paper as you can.
First draft: The first draft is a coherent, but not perfect, document. There is a logical flow to your arguments, but there may be a few gaps here and there.
This version needs to be ready to be handed in to your supervisor for a review so you can make any necessary changes before sending it off to your thesis committee.
Second/Final draft: Your second or final draft should be a complete manuscript that is ready to be handed in to your thesis committee and is
1) is coherent,
2) free of spelling and grammatical errors,
3) complete with references, table of contents, appendix, tables and figures, and
4) properly formatted.
Most your committee will ask you to make modifications to this version, but if you don’t have to change the fundamental structure you can go through their recommendations by editing one section at a time.
It is amazing how much simpler life becomes once you structure your paper and you break down the writing process into distinct stages.
Not only will this process save you time, but it will also help you to write better and incorporate changes from your supervisor and thesis committee as you go through each section.
Remember that no is born knowing how to write a literature review.
The articles you read in journals started out as messy drafts, and went through many stages of editing by several experts in the field before they were ready to be published.
Your literature review does not have to be perfect as you write it (no one’s is). Your job is to show up every day and keep writing.
If you have a well-defined structure and realistic milestones, you will make steady progress and create a literature review that you will be proud of.
What is the toughest part of writing for you? Collecting ideas on paper, turning your ideas into a coherent document, editing, or addressing the comments from your supervisor? Please leave a comment below and Dora will respond to you directly.
Click here to get on the waiting list for the online “Finish Your Thesis Program” and get a copy of my free book “Finish Your Thesis Faster”
Describing Empirical Research and Critical Analysis
Illustrate a methodology: Studies that illustrate a methodology are particularly important. The limitations of a methodology can affect all studies that use the methodology. Understanding and critiquing the methodology allows the author to assess whether the theoretical claims argued by researchers using this methodology are justified and allows the reader to understand this also. Common methodological limitations in I/O psychology include: (a) use of self-report measures (limits objectivity of conclusions); use of a observational designs (limits causal inference); (b) use of university samples (limits generalisability); (c) use of a common statistical analysis technique (e.g., most people researchers using mediation or SEM do not adequately consider alternative causal interpretations); and (d) use of experiments (may lack ecological validity).
Describing an empirical study: When describing a study it is important to present only the important details. Details highly relevant in I/O psychology include: (a) sample size (will the resulting statistics have small standard errors), (b) sample type (university students, employees, etc. - will the results generalise to the target population), (c) study design (experimental, observational, longitudinal - to what extent is causal inference possible), (d) method of measurement (self-report, other report, observational, etc. - to what extent is the measure valid, reliable, measuring something real), and (e) context of research (applied, laboratory, etc. - how ecologically valid is the task) (for more info see the Social Research Methods website). Some details that are typically less relevant include the exact number of males and females in the study, racial breakdowns of the sample, and the exact scale used to measure the variable. If such details are reported, a persuasive argument is given explaining why these details may alter the results. When describing research good literature reviews provide a quick snapshot of the important elements of the research and the important findings, while at the same time not wasting precious words that could be spent on other issues. Good literature review's “don’t lose the forest for the trees”.
Critical Evaluation: Good literature reviews base claims on empirical research. Research is critically evaluated and issues such as the following are discussed: level of analysis; confounding variables; correlation or causation; mediation and moderation; generalisability; limitations of sample size; issues of measurement; implications of design (correlational, experimental); effect size and practical significance; quantitative and qualitative methods; and so on. This examination of the theoretical literature is then integrated into the conclusions reached.
Causation: Good literature reviews demonstrate an understanding of proper causal inferences. For example: It does not matter how many cross-sectional observational studies are conducted looking at the correlation between self-efficacy and performance, this does not prove that one variable causes the other. Before writing about causality consider whether X could cause Y, Y could cause X, or whether a third variables could cause both X and Y to covary. The best evidence for causal claims usually comes from experiments involving direct manipulation of the independent variable. Weaker support can come from research statistically controlling for extraneous variables and longitudinal studies. It’s also important not to trust causal claims made by many researchers, when they are conducting mediation analysis or structural equation modelling on observational data (see my discussion). Authors may argue that their data provides evidence for causal mechanisms, but at the end of the day, their research rarely allows for unambiguous causal inference. To learn about causal inference, see this post.
Causation and control: The four goals of science are sometimes said to be description, prediction, explanation and control. Understanding causation (i.e., explanation) has important implications for making recommendations for I/O psychology practice (i.e., control). For example, if self-efficacy does not causally influence performance, then attempts by organisations to manipulate self-efficacy in the aim of improving performance may be useless.
Effect Size: The research literature is filled with findings of the form, there is a relationship between x and y or there is a difference between group A and group B on some variable. Good literature reviews communicate information about effect sizes (e.g., correlations, Cohen’s d, odds ratio). This is particularly important in literature reviews concerned with the practical importance of findings. For example, in research looking at faking personality tests in selection and recruitment, reporting that (a) "student samples were able to increase their conscientiousness score on average by one standard deviation" (e.g., Cohen’s d = 1) when asked to fake is a lot clearer than reporting that "students were able to increase their conscientious score". The former reports the size of the effect, whereas the latter only reports the direction. In another example looking at the effect of different types of diversity on team performance actual correlations or Cohen’s ds from studies or meta-analyses could be reported. Effect size can also be incorporated into a style of writing which puts relationships and effects in context. Using words like "small", "medium" and "large" to describe relationships in the literature, drawing on Cohen’s recommendations for effect size, can help the reader understand the relative importance of particular relationships (see Lee Becker's notes to learn more about effect sizes).
Integration of empirical studies: Good literature reviews are grounded in the empirical literature. They highlight problems involved in drawing conclusions from empirical research . Common design limitations include: small sample sizes (inadequate power and results that are not robust); poor measures of the theoretical constructs (unreliable or invalid); unrepresentative of the domain of generalisation (e.g., a study on students applied to organisations; a study of racial diversity applied to gender diversity). Good literature reviews are critical of the ways that studies draw conclusions from empirical research.
Weaker literature reviews often treat research claims equivalently. Good literature reviews delve into the reasoning used by authors to make their claims. They critically evaluate the empirical research and develop a reasoned evaluation of the justifiability and generalisability of claims. They explore alternative interpretation of results. They identify limitations of study design.
Presenting an Overview
1) Cite meta-analyses: When the question is focused, meta-analyses provide a systematic quantitative summary of a relationship. It’s also important to know how to describe and evaluate a meta-analysis. Some major things to mention include: a) the effect size that was obtained; b) what kinds of studies were included; c) the number of studies and total number of participants; In some instances you also mention d) whether the effect size varied more than was expected from random sampling; e) results of any moderator analysis.
2) Cite review articles: Directing the reader to review articles helps provide an overview of the topic. Good literature reviews often share the conclusions reached by earlier reviews and use these as a launching pad.
3) Cite examples: Good literature reviews carefully select illustrative studies. They vary the depth of explanation given to a study. Sometimes it is sufficient to just highlight the existence of a study. For example, a literature review could include: “The relationship between x and y found in study Z has also been found in several other areas including sport (e.g., citation, citation), music (e.g, citation, citation), and physics (e.g., citation, citation)”. This style provides a review of the literature and gives the reader a sense of the breadth of work on the topic. If the reader is interested they can follow-up on one of the citations.
Synthesis: Dealing with Issues
In relation to many research questions there are findings that go for and against particular claim. For example, on the topic of the effect of diversity on team performance, there are findings suggesting positive, negative and no relationship. Good literature reviews propose plausible explanations for the variability in findings across studies based on the available evidence. Differences in findings can be explained in terms of differences in terms of study design, study conduct, and random sampling. A moderator is a factor that alters the relationship between another two variables. Moderators can be substantive (e.g., different types of participants or contexts) or methodological (e.g., different measures, software, etc.). Random sampling is also an important explanation of differences in results between studies (See my discussion of meta-analytic thinking also).
Good literature reviews weight conclusions by the quality of the evidence for alternative arguments. They recognise the value of meta-analyses in pooling multiple studies in a systematic way.
Poorer literature reviews present one theory after the next without integration. They often include text like: “despite all the research, it can not be said whether job satisfaction is caused more by situational or dispositional factors. The End…” Concluding statements, such as “more research is needed” is typical of poorer literature reviews. They also often tend to jump to conclusions too soon, often declaring a particular idea (e.g., a strong culture leads to high performance) as definitively established when debate existed. Slightly better literature reviews say things like: “This model has not been particularly supported (Smith, 2006)”. A good literature review provides evidence for why a theory has not been supported. A good literature review is aware of the relationship between empirical observations and theoretical claims.
Good literature reviews identify issues in the literature. Issues arise when two proposed ideas conflict. For example, does X cause Y or does Y cause X? Is expertise learnt or innate? Is this theory useful for practitioners or not? Is job satisfaction influenced more by dispositional or situational factors? Good literature reviews accurately and concisely summarise the major competing positions on an issue. They do not rush to conclusion, yet they do strive to reach a nuanced conclusion based on the evidence. If more research is needed, the achievements of existing research are acknowledged and concrete recommendations are made. This might be recommendation for greater use of a particular methodology or particular contexts.
Logic and Reasoning
Clear Structure: Good literature reviews have a clear structure. Aims and objectives are set out at the beginning of the literature review. The aims are consistent with the broader research needs. These needs may be derived from an assignment question, a thesis aim, or journal publishing requirements. The structure systematically works through the issues. Communicating structure both to oneself and to the reader can be facilitated by a set of headings. The structure of ideas and themes guide the literature review. Research and theory is integrated into the structure. Independent thought is reflected in the structure.
First, weaker literature reviews tend to waste words discussing material unrelated to the aims. For example, a literature review on the role of team diversity might spend many words discussing other factors besides team diversity that predict team performance. Second, weaker literature reviews are a loose listing of ideas. At its extreme the literature review appears like the author has obtained a set of articles and spent a paragraph or two discussing each with no integration. Poorer literature reviews do not demonstrate independent thought. They may mimic a particular journal article or lecture on the topic. They cite the same articles and present the same arguments as adopted in the source. At the extreme they plagiarise (i.e., copying without acknowledgement) the structure of another article.
Quality of Expression
Spelling, Grammar, Composition, Style: Good literature reviews are written well. At a basic level this involves correct spelling, grammar, and punctuation. At the next level this involves following conventions of writing literature reviews. Tense, pronoun usage, and paragraph composition are particularly important. There are many strategies for ensuring that prose reads in a structured and sequenced way (some thoughts). Finally there are many books on writing style that are worth studying from time to time (See here for a list; and here for online grammar help).
Article Deconstruction: To learn the conventions of writing in psychology, the best strategy that I know is what I call Article Deconstruction. It involves: 1) completing a writing needs analysis on yourself; 2) finding an article that demonstrates the writing style that you would like to emulate; 3) deconstructing the journal article to identify the principles that guided its creation; 4) working on your own writing task and attempting to implement the principles. A fifth step can involve reading how-to books on the particular writing task. When selecting a source article to deconstruct Annual Review of Psychology is a good place to start.
Proofreading: Authors of polished literature reviews have a system of proofreading. Simple typographic mistakes detract from the finished product. Standard strategies of proofreading include: 1) putting the literature review aside for a couple of days and returning to it to proofread; 2) reading the literature review aloud; 3) getting someone else to proofread; 4) using spell-checkers and grammar-checkers. (Here's some more proofreading tips).
References and Citations
Making the link clear: Good literature reviews clearly communicate the relationship between a statement and an accompanying citation. Is the citation a baseless assertion, an established theory, or an empirical finding? The following examples tend to be superior to ending a sentence with a mystery citation: “A study by Smith showed”, “Smith has suggested that”, “Smith has theorised”, “Smith obtained results supporting the idea that”. Good literature reviews give a sense of the strength and nature of the evidence provided by the citation. Putting an author’s name at the end of a sentence tends to not fulfil this need. Thus, the words around the citation can explain the reasons why the author asserted the idea in the first place. Did the author base the claim on common sense? Was it an empirical finding? Was it based on the summary of a set of empirical results or meta-analysis?
Poor: “Perceptual speed, psychomotor and general abilities relate to the three phases of skill acquisition in different ways (Ackerman, 1988).”
Better: “Ackerman (1988) has theorised that perceptual speed, psychomotor and general abilities relate to skill acquisition phases in different ways.”
Even Better: “Ackerman (1988) performed a series of large sample empirical studies using a range of simple psychomotor tasks which provided partial support for his theory that perceptual speed, psychomotor and general abilities relate to skill acquisition phases in different ways.”
Cited In: Good literature reviews do not use "Cited in". Literature reviews which summarise Author B’s citation of Author A’s work write: "as Author A (1999) says as cited in Author B (2002) …" . However, good literature reviews, when they see that Author B cites Author A, go and get Author A’s article, read it , and draw conclusions about it directly.
Quotes: Good literature reviews provide an independent voice on the research of others. Quotes are used sparingly and judiciously if at all. literature reviews with more than two or three quotes typically appear as if the author is unable to express an independent perspective. A simple rule is to not use quotes at all. A better rule is to use quotes sparingly and only when the particular phrasing of the original author adds value, such as in definitions or when the words have a poetic or other literary effect.
Number of References: It is important to have a sufficient number of references. The aim is generally to give an overview of the topic. The nature of the coverage and the selection procedure should be made explicit.
Independent voice: Good literature reviews carry a thread of the author’s voice. An independent voice can be facilitated by following the principles of citation mentioned above. Good literature reviews present an independent perspective on the issues. Good literature reviews answer questions like: does the theory make sense with my understanding of the world? Is my argument logically consistent? What would be the implication of what I am saying for interventions, other theories, future research, etc.?