The Methods Corner: Introduction and Critical Appraisal

 

Ruth Knight

Introduction

The quality of the research that we conduct and that we rely on to inform clinical practice is important. In a 1994 editorial entitled ‘The scandal of poor medical research’, Doug Altman, a medical statistician well known for his work to improve the quality and reporting of research, famously said “We need less research, better research, and research done for the right reasons” [1]. Since then substantial improvements in terms of research quality and the standards to which research is reported have been made; for example, through the introduction of reporting guidelines such as the CONSORT guidelines for randomised controlled trials (RCTs) [2] which ensure a minimum set of items required to assess the conduct of the study are included in trial results publications. However, many areas for improvement remain and we each have a part to play in ensuring that published research is as high quality as possible.

Purpose

We cannot do good quality research unless we know what that is. In our new series of articles, ‘The Methods Corner’, we will cover a range of topics including: different research designs, statistical concepts, analysis methods and interpretation of results. Topics will be covered in a clear, interpretable manner with the goal being that these articles enable you to apply these skills and thinking to your own work.

Our Team

We have assembled a team of experts from the KJP editorial team who have between them a wide range of skills and expertise. This includes research specialists at professorial level, a medical statistician, clinically-active psychiatry consultants and directors as well as experts in literature search and synthesis.

Critical appraisal

The first topic in this series, which will be covered in this issue, is critical appraisal. Critical appraisal involves evaluating published research to assess its quality and reliability [3]. It is important not to assume research to be trustworthy simply because it has been published, but rather to examine this critically for yourself. A key question you should always be considering when reading a piece of published research is what biases it might be subject to; for example, if the study aims to assess the effect of a treatment, what impacted who received the treatment. Important areas for specific consideration include:

What journal is the research published in?

It has been estimated that the number of scientific papers published rises by 8-9% per year [4] with these spread across a wide range of journals including so-called ‘predatory’ journals [5]. It is therefore important to consider where the research is published, and in particular whether the journal in question is peer-reviewed. Peer-review is an important quality control process for published research.

Who are the team?

The skills and expertise of the research team should be assessed to see whether they are the right people to conduct this research; for example, whether they have the relevant clinical background or appropriate methodological skills. Any conflicts of interest which have been declared should also be evaluated to assess whether these pose a concern.

What are the objectives of the study?

The aims of the study should be clearly stated, and, where appropriate, these should match those that were set out at the start of the study. A record of the aims and objectives planned at the start of a study may be available through a published protocol or registry.

What study design is used?

It should be considered whether the study design chosen is appropriate to answer the question at hand; for example, a randomised controlled trial (RCT) is suitable for comparing two treatments, whilst a cohort study is a better option for considering the natural history of a disease.

Who are the population?

The population eligible for inclusion in the study should be examined to assess whether this is appropriate for both the condition and, if applicable, the treatment under consideration. In addition, it should be considered whether the group of patients included in the study are representative of this population of interest, for example by examining a summary of demographic characteristics.

What analysis methods are used?

It should be considered whether or not appropriate analysis methods to answer the question(s) of interest have been selected, and whether these have been applied correctly in order to do so.

What are the conclusions?

Whether or not the conclusions drawn are consistent with the data and results presented should be considered. It is also important to examine whether the conclusions address the aims and objectives laid out for the study.

These questions provide a guide to some of the key points that it may be appropriate to consider as part of a critical appraisal. This is by no means an exhaustive list of points that could or should be considered; equally, not all of these points will be appropriate for all types of research. It is, however, key to consider the merits of a piece of research critically before accepting its results.

Your Questions

We want to address the questions about research questions that are most important to you. If you have a particular issue you would like to see addressed in future issues, please email: methodscornerKJP@outlook.com

References

[1] Altman DG. The Scandal of Poor Medical Research. BMJ. 1994 Jan 29;308(6924):283-4.

[2] Schulz KF, Altman DG, Moher D. CONSORT 2010 statement: updated guidelines for reporting parallel group randomised trials. Trials. 2010 Dec;11(1):1-8.

[3] Burls A. What is critical appraisal?. Hayward Medical Communications; 2014.

[4] Bornmann L, Mutz R. Growth rates of modern science: A bibliometric analysis based on the number of publications and cited references. Journal of the Association for Information Science and Technology. 2015 Nov;66(11):2215-22.

[5] Shen C, Björk BC. ‘Predatory’open access: a longitudinal study of article volumes and market characteristics. BMC medicine. 2015 Dec;13(1):1-5.

About the author

Ruth Knight, KJPsych Resident Statistician