Systematic Review and Meta-Analysis

Livestreamed online | October 28 and November 4-5, 2022

One of the greatest scientific challenges and opportunities in the information age is making sense and use of a vast sea of scholarly findings. Every time we write an introduction or acquaint or reacquaint ourselves with a field of study, we have to come to grips with the tsunami of articles that are all at our fingertips. By some estimates, scientific output doubles every nine years or so, but the hours in our days remains a constant twenty-four. Consequently, we all struggle with finding the relevant articles for our theses, introductions, meta-analyses and systematic reviews. Furthermore, we are expected often to search multiple databases, readily deduplicate them, keep them up to date, and do so all with transparency and replicability.

Once a set of articles that are relevant to one’s reviews have been relocated, the next challenge is making sense of the findings, which always vary from study to study. Reasons for such cross-study variation include differences in measurement and experimental procedures, stimuli, and rigor, populations sampled, time of study, and so forth. Meta-analysis is used to quantitatively extract the signal from the noise in this study-to-study variation in results as well as find systematic and reliable moderators of cross-study variation.

This course provides a theoretical and practical introduction to systematic reviews and meta-analysis.  Using the latest machine learning algorithms incorporated into a cloud-based online platform, we can vastly reduce the time required to conduct a search and collect the data needed for a meta-analysis. Participants will learn how to conduct effective searches across multiple databases, upload them into the HubMeta platform for automatic deduplication, easily screen them using an advanced refinement of the PRISMA protocols (Preferred Reporting Items for Systematic Reviews and Meta-Analyses), annotate or tag them for later use, train RAs in the process, keep a record of their steps, and readily update their search. Whether the project is an individual’s thesis or a comprehensive meta-analysis with a large, international team, this course will make the process radically more replicable and easier. Time reduced can be over 90%, making this a necessity for anyone with limited resources.

Participants can choose just to focus on the literature review element of the course, which includes search, screening and organizing. For those who want to extend their learning into meta-analysis, we later include how to reduce coding time by ten-fold by using the latest techniques and resources, including integrated online platforms that draw on optical character recognition, and automatic error detection. We will replicate and critique a previous tier-1 meta-analysis in real time. The goal of this unit of the course is to enable participants to readily publish their own competitive meta-analytic project.

INSTRUCTORS: Dr. Piers Steel, PhD and Dr. Hadi Fariborzi, PhD

In this course, you will learn about the underlying principles and the practical applications of systematic review and meta-analysis. For the systematic review unit, the topics include

  • Conducting effective searches
  • Database selection
  • Automatic article acquisition
  • Establishing and training a research team
  • Defining and conducting a literature search iteratively
  • Citation chaining
  • Deduplication and title/abstract screening
  • Full text acquisition and screening
  • Dealing with foreign language articles
  • Tagging and sorting articles for later use

For the meta-analysis unit, the topics covered include

  • Creating a taxonomy and connecting measures to constructs
  • Converting effect sizes to a common metric
  • The data entry process
  • Dealing with dependent effect sizes and time series data
  • Psychometric corrections for meta-analysis
  • Meta-analytic models and their weighting schemes
  • Outlier analysis
  • Publication bias analysis
  • Fundamental meta-analytic results
  • Establishing potential moderators
  • Meta-regression and moderator analysis
  • Meta-analytic structural equation modeling
  • Integrating other research techniques
  • Open science reporting

This course is offered online through Zoom. The first unit on systematic reviews will take place on October 28, 2022, from 9.00am to 5.00pm Calgary time and for a limited time it is being offered free of charge. The second unit on meta-analysis will occur on November 4 and 5, 2022, and will meet 9.00am to 5.00pm Calgary time each day. The tuition for this unit of the course is $795 (Canadian dollars) until October 29, after which the tuition increases to $895. Those who register for the free systematic review unit do not have to enroll in the meta-analysis unit (though we encourage registering for and attending both). We strongly recommend anyone interested in the meta-analysis unit also register for and attend the systematic review unit, though this is not required.

The course will concentrate on practical application, with focus on data filtering, data entry and analyses in HubMeta and R, though referencing options in other software. Most of the provided statistical materials and examples will involve R code. HubMeta is a cloud-based platform that works best in the Chrome browser.  

This course will be helpful for both introductory and advanced researchers in any field who are focusing on systematic review as well as mean-based or correlation-based meta-analysis. Experimental meta-analysis, though related, will be addressed only indirectly. Consequently, this course is relevant to researchers from most fields (e.g., psychology, management, sociology, education, human development, social work, public health, communication). Learners will ideally be comfortable with introductory statistics, including regression, and be able to read as well as understand the method section of basic articles in their field (i.e., necessary for data entry). Proficiency in R is desirable but not necessary.  Learners should have access to academic search engines.

Upon completing both units of this course, you will

  • be able to conduct a systematic review in an efficient, transparent, and replicable format
  • be able to effectively construct search terms
  • be able to train and supervise RAs for data screening
  • configure machine learning algorithms for data screening
  • effectively organize literature searches for later write-up
  • be able to produce a competitive, tier-1 meta-analytic dataset, method section, and framework
  • tackle meta-analytic topics an order of magnitude larger than previously plausible as well as in a fraction of the time, including comprehensive meta-analytic correlation matrices
  • improve your management of a dispersed international meta-analytic research team
  • address journals' theory requirements through meta-regression and meta-analytic structural equation modeling (MASEM)
  • understanding the fundamentals of meta-analysis as well as the basic ideas behind more advanced techniques (e.g., One Stage MASEM)