Are journal clubs an effective means of teaching critical appraisal in medical education? A review of the literature

Musa Sami 
Tiago Inacio 
Trevor Gibbens Unit, Maidstone

Abstract

Aims: Although well embedded into psychiatric training, there is little guidance on what constitutes a successful journal club. This article thus ascertains whether journal clubs are effective in teaching critical appraisal skills and what factors add or detract from this. Methods: Full text articles in English containing the terms: “journal club” AND “critical appraisal” were searched in MEDLINE, Pubmed, EMBASE, ERIC, EBSCO EJournals and the British Education Index. Inclusion criteria were: (i) face-to-face journal club (ii) postgraduate medical education, and (iii) primary research or evaluation. Results: 15 articles were reviewed. Journal clubs are seen by trainees as important in the development of critical appraisal and appear to have a motivational role, but there is as yet no evidence to suggest the superiority of journal clubs in abstract skills over other approaches. Conclusion: The journal club has an important role in engaging trainees in the process of critical appraisal.

Introduction

The first Journal Club appears to have been described by Sir James Paget circa 1834, and assumed its formalised shape under William Osler in 1875 (1). The Journal Club has retained several core features since this time: (i) a regular face-to-face meeting of medical faculty (ii) a presentation and discussion regarding a paper (iii) evaluation of its merits and weaknesses and (iv) consideration as to the effect on practice. Initially, the aim was a means of updating practice, with the main focus being the content of the paper presented (e.g. smoking causes lung cancer) rather than the process (e.g. is this study rigorous?) (2). The increasing body of medical literature and the difficulty in being able to digest all the literature on any given topic led to the formalisation of Evidence Based Medicine (EBM) (3) where increasing emphasis was placed on critical appraisal – “the process of carefully and systematically examining research to judge its trustworthiness, and its value and relevance in a particular context.”(4). Alongside the shift to EBM there was a shift to seeing a key aim of the Journal Club as a means of teaching critical appraisal skills.

Although well-embedded into psychiatric training as a mandatory requirement of the U.K. psychiatric training (5), there is little guidance on what constitutes a successful journal club, and whether journal club enhances critical appraisal. This article reviews whether Journal Clubs are effective in teaching critical appraisal skills, and what factors add or detract from this.

Methods

Full text articles in English containing the terms: “journal club” AND “critical appraisal” were searched in MEDLINE, Pubmed, EMBASE, ERIC, EBSCO EJournals and the British Education Index on 20 September 2014. Abstracts were screened against the following inclusion criteria: (i) articles regarding face-to-face journal club (ii) regarding postgraduate medical education, and (iii) primary research or evaluation reported.

There is no single measurement to evaluate effectiveness in education. This article is thus organised to encompass the multiple levels of evaluative feedback. This article uses the ‘Hierarchy of Evaluation‘ as adapted for Medical Education (6,7) to extract data (see Tables 1 and 2). However there are also valid criticisms of strict adherence to the model as a hierarchy, given the complexity of the learning process and the multi-factorial nature of population healthcare outcomes (8). Consequently, findings are presented across domains looking at evidence of attendance, feedback on the teaching received, attitudes towards the subject, changes in knowledge and skills and changes in behaviour; to determine whether journal clubs are an effective means of teaching critical appraisal.

Results

Out of 107 abstracts reviewed, 15 articles were suitable for inclusion. An overview of the articles has been extracted in Table 1 and the outcomes extracted in Table 2.

Attendance Rates:

The simplest method of determining effectiveness of journal clubs is to note attendance rates. A cross sectional survey of chief internal medicine residents across 131 eastern USA training programmes showed marked variation in attendance rates from 7% to 100% (9).  Sidorov noted that pitfalls of Journal Clubs were ‘poor attendance and periodic abandonment’. Consequently he defined successful Journal Club at maintaining >50% attendance and >2 years longevity achieved by 48 Journal Clubs (38.7%). Features which enhanced these outcomes were: mandatory attendance, regular provision of food and a smaller number of house staff. This work covers a large range of residency programmes, and the postulated criteria retain a face validity in measuring participation, and consequently have been cited by others as evidence of successful Journal Clubs (10,11). However these criteria are arbitrary and would exclude other Journal Clubs which had important evidence of external validity of engagement, one of which (12) generated 6 peer review publications. Furthermore attendance and longevity do not appear to be due to the educational value of the Journal Clubs, but rather the provision of food and mandatory attendance. Thus attendance and longevity records provide limited information for our purposes.

Feedback:

The two successful journal clubs as defined by previous criteria (attendance) also had positive feedback by trainee survey. Spillane and Crowe (1998) in their Australian surgical training programme noted that 89% of participants looked forward to the next Journal Club (10), whereas Akhund and Kadir (2006) noted Pakistani Public Health trainees to report an educational value in attending and preparing for Journal Clubs (89%) (11). A further Journal Club meeting Sidirov’s criteria also showed high levels of satisfaction, relevance and educational value (mean score 4-4.6/5) (13), although this was compromised by only 5/24 participants feeding back. It is not surprising to find high levels of satisfaction from long-running, well-attended programmes with high levels of investment from faculties.
Which educational aspects of Journal Clubs enhance satisfaction? One study described a comparative unstructured didactic Journal Club format with a programme of EBM Journal Clubs focused on methodological appraisal, commencing with a two session focused EBM workshop (1). Mean satisfaction was significantly higher for the EBM focused methods although we are not given further details on the constituent values. Thus there is evidence that Journal Clubs receive positive feedback and that an Evidence Based Journal Club is more valued.

Change in Attitude:

Do Journal Clubs modulate attitudes towards critical appraisal skills? Trainees appear to value the opportunity to develop critical appraisal and view it as important. A recent Nigerian study noted that of 66 surgical trainees, training across various national centres, 98% valued critical appraisal skills as ‘important’ or ‘very important’ (14).  Journal Clubs are seen as an important vehicle to developing this. Akhund and Kadir (2006) noted that the most important reason for attendance at Journal Club was for learning of critical appraisal (11). However Journal Clubs may not develop desired attitudes in all contexts. Temple and Ross (2011) had an intensive EBM programme with homework assignments in a small sample of 10 plastic surgery trainees over a year (15). They found a drop from 64% to 50% of participants feeling that Journal Club was an effective means of teaching critical appraisal skills. It is possible this evidences a burnout in trainees in a busy surgical programme who had to attend 3-hour long journal club sessions in the evenings and noted ‘moderate resentment’ at the time spent on carrying out homework. Taken together, this suggests that whereas Journal Clubs are associated with educational satisfaction if well-supported and using evidence based structure for critical appraisal, this is endangered if the Club proves burdensome for trainees.

Knowledge & Skills:

Do Journal Clubs lead to acquisition of critical appraisal knowledge and skills? Trainees felt that Journal Clubs enhanced confidence in various aspects of critical appraisal skills across a range of measures described in 6 different papers: in statistical skills (16),  ability to apply literature to practice (17,18), and improved self confidence in critical skills (1,10,11,16). The one study which disagreed with these findings, suggested that a non-significant improvement in self reported critical appraisal skills ability may be explained by trainee burden as previously described (15). The results of external examination are less clear-cut. Knowledge-based tests and Multiple Choice Questions administered before and after Journal Club demonstrate improved scores in some studies (13,19–21) but not all (15,18). It is difficult to determine exactly why this is – positive studies reporting improvement span across specialities (obstetrics, paediatric surgery, anaesthesiology, internal medicine) and time periods (4 months – 2 years), as do the negative studies (psychiatry, plastic surgery, time period: 3 months – 1 year). All 6 studies used a proactive approach to medical education in Journal Club (see Table 1), including the two negative studies. The studies do not provide the MCQ criteria and so it is difficult to ascertain validity.

Similarly four studies undertook assessment via essay or critical appraisal exercise, which may be more likely to test abstract skills. Moharari et al., (2009) found a significant improvement with application of the CONSORT checklist after a year of focused Journal Clubs (21).  This was to a group of trainees who had self-declared at baseline ‘a lack of knowledge in this area’. Improvement in performance after a year of focused teaching is not surprising. A more rigorous methodology used in three controlled trials showed no difference in the intervention and non-intervention groups when using a critical appraisal skills exercise or essay as the outcome of assessment. This held for Journal Club vs. no Journal Club(18); Journal Club vs. Control Seminar (17); and EBM Journal Club vs. unstructured Journal Club (22). Taken together the evidence suggests that Journal Clubs improve trainee confidence for critical appraisal and there is some mixed evidence that knowledge elements of critical appraisal are improved through their use.

However, there is very little to no evidence that the use of Journal Clubs, or the use of Evidence Based Journal Clubs improves abstract critical appraisal skills over alternative approaches. 

Behavioural Change:

There is some evidence that Journal Clubs change behaviour in relation to critical appraisal. The most replicated finding is that of Journal Clubs increasing self-reported reading behaviour amongst trainees (11,19,20). The controlled studies also had similar findings (17,18). One study found that although reading preferences had not changed, trainees found they way they read papers had (18). All of these studies describe self-reported reading behaviours rather than observed behaviour, and thus a possible bias towards giving the desired answer may exaggerate the significance of this effect. There is also some evidence that Journal Clubs may lead to other hoped-for behaviours. In one successful Journal Club 19/28 (68%) participants reported that the Journal Club had changed their clinical practice and the same number had reviewed their practice further (10). The Journal Club led to the establishment of 8 research projects. Sandifer et al., (1996) reported on the generation of 10 critical appraisal letters after 11 sessions of their Journal Club, of which 6 were published in peer reviewed journals (12), and  Khan et al., (1999) reported on the publication of two letters from their programme (20). Successful publication as a result of Journal Club discussion suggests successful critical appraisal activity and represents an externally valid reference measure for the skills of the trainee.

Discussion

The question as to whether Journal Clubs improve critical appraisal skills has been answered in a variety of participative, satisfaction, knowledge, skill-based and behavioural outcomes. Surprisingly, across a range of journal clubs there appears to be consensus for most individual outcomes, although the evidence of improved knowledge of critical appraisal is mixed. Although only three are controlled trials, only one being randomised, the consensus nature of the outcomes gives credence to the findings. Nonetheless, we must accept that self-reported measures, as used in attitudes and behaviour questions may be prone to respondent bias; whereas the validity of the knowledge-based assessments is mostly not clearly stated. The Journal Clubs described are attended by specialists in training; although only one study looked in particular at psychiatry trainees (18).

Taken together, the literature suggests that Journal Clubs are seen by trainees as important in the development of critical appraisal, and on the whole generate satisfaction in education and development of critical appraisal, especially in those Journal Clubs with an EBM focus. It may be important to ensure the Journal Club does not prove burdensome to trainees. Trainees tend to rate development of critical appraisal skills highly, but objective testing of this proves more complex. There is more evidence to suggest that knowledge-based elements (such as different types of trials, statistical methods) improve, but there is as yet no evidence to suggest the superiority of Journal Clubs in acquiring the complex abstract skill of critical appraisal over other approaches.

Nonetheless Journal Clubs do appear to be successful in modulating behavioural change – increasing self-reported reading time and stimulating a variety of outcomes including research projects and peer-reviewed publications. Thus, it appears, whereas Journal Clubs provide stimulation and increased confidence in the critical appraisal approach for trainees, the development of more abstract critical appraisal skills are not themselves enhanced by a single Journal Club programme over other approaches. This is possibly due to the complexity of the skill set, requiring sustained acquisition and development using various methods over time.  Further work to determine the set of teacher, personal, and assessment factors which would lead to development of abstract critical appraisals skill is thus required and falls outside the narrow remit of this review.

Although the effectiveness of Journal Clubs in learning critical appraisal skills is limited, these will continue to exist in most teaching programmes. This raises the question of how journal clubs are better run. The literature reviewed here gives us some indications. Longevity and good attendance are identified as important requirements for a successful Journal Club (9), despite the limited evidence of how this is best achieved. Journal clubs with an EBM focus or supplemented by focused teaching of critical appraisal skills are also associated with better outcomes (16, 19), provided they are not too onerous on trainees (15). In order to improve the engagement of trainees with Journal Clubs and the development of critical appraisal skills it may be helpful to define desired outcomes, e.g. the publication of letters following Journal Club discussions or review of local policies.

There are important gaps in the literature. There is yet no evidence to determine whether Journal Clubs have an impact on further organisational behaviour or an end impact on care delivered’, perhaps because healthcare is a complex multi-factorial process and such outcomes would be difficult to measure whilst controlling for confounders (e.g. development of medical technology, teacher and learner factors). There is no work on how different specialities or seniority of trainee develop critical appraisal in response to the Journal Club. For example psychotherapy trainees may develop more qualitative appraisal skills whereas epidemiologists may be more interested in quantitative approaches, or seniority may predict or detract from development of critical appraisal.

Conclusion

The Journal Club has an important role in engaging trainees in the process of Critical Appraisal. However supplementary approaches are also likely required for development of abstract skills. Thus Journal Clubs are useful as a means of motivation, enhancing confidence and generating further interest.  Tools such as the Journal Club Presentation Work Based Placed Assessment maybe used to generate further development plans for developing skills, including considering generation of peer-reviewed letters or small projects and signposting to further resources. It is important, however, to avoid overburdening the trainee with assignments that may detract from their development.   

Table 1

Paper

Context

Duration / Frequency

Members & Response Rate (%)

Teaching Methods

Methods & Measures

Conclusions

Limitations

 

Linzer et al., 1988(17)

Interns at USA Medical Centre

5 sessions

44 interns

JC: handouts & critical appraisal guidelines, 3-4 hours/week preparation, focus on methodological points. Control Seminar: Lecture based. Critical appraisal skills not emphasised. Faculty spend 3-4 hours per year in preparation.

Randomised Controlled Trial to Journal Club vs. Control Seminar. Outcome: Attitudes & Behaviour Evaluation Instrument and Critical Appraisal Exercise

JC can lead to change in reading habits and improved knowledge of epidemiology and biostatistics. Improving application of critical appraisal may be longer process.

1. Self reported behavioural change (reading habits) looked at rather than actual 2. Validity of critical appraisal exercise to be established

 

Seelig, 1991(19)

Community Hospital Internal Medicine training programme, USA

4 months, EBM session + bimonthly journal club

14 residents

One hour supplementary teaching session, principles of adult education – active learner participation, facilitated group discussion, written assignment in addition to standard JC format

Pre and Post Attitudes and Behaviour Questionnaire. Objective testing of critical appraisal knowledge.

Teaching Intervention focussed on critical appraisal skills can successfully supplement JC and lead to improved knowledge and change reading habits

1. Difficult to distinguish effects of teaching exercise from JC

 

Sidorov, 1995(9)

All chief residents of internal medicine programmes in Eastern USA

Various

131/208 programmes evaluated (64.5%)

Varied

Cross sectional questionnaire

Features of successful JC identified: a) mandatory attendance b) provision of food c) fewer house staff

Does not provide direct information on effectiveness of critical appraisal skills

 

Sandifer et al., 1996(12)

Public Health trainee & consultant JC in UK

6 months

16 members

At least one consultant / senior lecturer in attendance

Attendance; Letters published; Impact on Commissioning Policy

Letters published in major journals externally valid marker of critical appraisal skills demonstrated in JC.

Small JC with low attendance.

 

Spillane & Crowe, 1998(10)

Surgical JC in hospital in Australia of Consultants and Registrars

5 years

28/39 members responded (72%)

Mentor assistance varied

Retrospective Questionnaire of members

Successful journal club valued for critical appraisal skills. Specific responses made to feedback of introduction of debate and statistician involvement

1. This JC had social element – held in consultant’s house out of hours. Did this affect satisfaction rates?

 

Bazarian et. al., 1999(22)

Emergency Medicine Residents in training programme, USA

1 year on monthly basis

32 total: 16 unstructured JC; 16 EBM based JC

Unstructured JC – traditional JC approach. EBM based JC used (a) case base presentation format and (b) use of structured worksheets derived from key texts and (c) close faculty supervision.

Prospective Case Controlled Trial. Pre and Post course critical appraisal skills examination (essay)

EBM based JC has no advantage in non EBM based JC when measuring critical appraisal skills in essay exercise

Practice effects may have lead to improvement in both groups

 

Fu et al., 1999(18)

Psychiatry Residents, Toronto

12 weeks, weekly

24 total: 12 JC; 12 non JC matched controls

JC group facilitated by Professor of Psychiatry. 4 teaching sessions by staff psychologist. No similar educational interventions in non JC group

Controlled Trial. Pre and post course MCQs, case scenarios and attitudes questionnaire

Although JC fosters confidence in trainees in perusing the literature, no evidence that it improves performance vs no JC

1. Small sample size 2. Short duration

Akhund & Kadir, 2006(11)

Community Medicine (Public Health) Residents in Pakistan

6 years, weekly

12 residents at one time, 27/32 residents responded/total (84%)

Mentorship by senior resident. Distribution of article two days prior to meeting

Attendance & Survey of participants

Demonstrates a successful JC, valued by residents

1. Is self appraisal valid measure of improvement?

Haspel, 2010(16)

Transfusion Medicine Residency programme, USA

2 months, 4 hour long sessions

7 second year residents

Designed 4 session curriculum. Use of 2 articles with different conclusions regarding same question in same session. Use of clinical questions / scenarios to enhance clinical relevance

Attitudes and self reported skills questionnaire pre and post course

JC is valued and self-reported ability can significantly improve after structured few sessions

1. Small sample size. 2. Validity of self confidence measures

Mohrari et al., 2009(21)

Anaesthesiology Residents in Iranian Teaching Hospital

One year, fortnightly JC

16 residents

Articles disseminated 4 days beforehand. Earlier sessions didactic, moving towards student-centred approach as participants became more confident

Pre and Post Test MCQ in research methodology and statistics. Critical Appraisal Exercise

JC can lead to improved knowledge base and improve critical appraisal skills if delivered as part of structured teaching programme

1. Self declared baseline ability to critically appraise set arbitrarily at zero ‘as residents declared lack of knowledge in this area’.

Temple & Ross, 2011(15)

Plastic Surgery residency training programme, USA

1 year monthly 3 hour evening JCs + EBM teaching programme

10 residents

Electronic support and resources made available. Incorporation of formal teaching into extended 3 hour Journal Clubs. Three homework assignments.

Pre and post course examination (MCQ) and attitudes questionnaire

JC and EBM focussed programme maybe insufficient by themselves to improve critical appraisal skills

Small sample size – possibility of Type II error

Shokouhi et al., 2012(1)

Neurosurgery Residents in centre in Iran

2 session EBM workshop + 6 EBM JC sessions

11 residents

Two session introductory workshop on EBM methodology

Pre and post surveys comparing EBM with non EBM method of organising JC

EBM based JC improve satisfaction and self rated critical appraisal skills vs. non EBM based JC

1. Is self appraisal valid measure of improvement? 2. Limited information on survey and score design

Ibrahim et al., 2014(14)

Survey of surgical trainees across  national centres in Nigeria

Various

66/123 trainees responded (53.6%)

Varied

Questionnaire: format of JC, teaching, development of critical appraisal skills, evaluation

JCs are widespread across country. Development of critical appraisal skills valued by trainees.

1. 46% trainees did not respond. 2. No explicit link between JC and critical appraisal skills

Lao et al., 2014(13)

Paediatric Surgery JC in two training hospitals, Canada

2 years, 6 sessions

24 members: 9/24 (38%) completed pre-session test and 5/24 (20%) completed post-session test

Distribution of journal article and session specific learning objectives distributed several weeks in advance. Each session targets specific study design

Pre and post session test and attitudes survey

JC is valued by members and provides immediate improvement in knowledge of critical appraisal skills

1. Low rates of participant feedback 2. No information whether knowledge is retained between sessions

Table 2

Paper

1. Participation

2a Attitudes

2b Knowledge or Skills

3 Behaviour

4a Organisational Practice

4b Benefit to patients

 

Linzer et al., 1988(17)

 

Attendance close to 100% for JC programme

 

 

80% JC self-reported improvement in ability to apply literature to practice vs. 44% control (p<0.05).Epidemiology/biostatistics scores improved in JC group (p<0.05). No significant difference in Critical Appraisal exercise between groups

 

 

86% JC group reported change in reading habits vs. 0% control group

 

 

 

Seelig, 1991(19)

 

 

 

Significant improvement in test of critical appraisal knowledge from 42% to 67% (p<0.05)

 

Self reported behaviour medical journal reading increase from 3.5 hours/ week to 4.5 hours/week

 

 

 

 

Sidorov, 1995(9)

Mean attendance 58.5%; 69.4% existed for > 2 years; 38.7% >50% attendance AND >2 year existence

 

 

 

 

 

 

Sandifer et al., 1996(12)

 

19-50% attendance

 

 

 

10 letters submitted to journals after JC by group. 6 published.

 

Regional IVF and Family planning commissioning policies reviewed

 

 

 

Spillane & Crowe, 1998(10)

 

75% attended more than 2/3 sessions

 

89% look forward to next meeting

 

93% reported very good to excellent on development of critical appraisal skills

 

68% lead to alteration in clinical practice, 68% stimulated to review subject further. 8 research projects commenced.

 

 

 

 

Bazarian et. al., 1999(22)

 

 

 

Improvement in scores for both groups, no significant difference between groups

 

 

 

 

Fu et al., 1999(18)

8.3/12 attendance

 

 

No significant difference in performance of MCQ or clinical scenarios between groups.

 

58% JC residents felt JC had positive impact on use of the literature. No change in JC in use of textbooks or journals

 

 

 

 

Akhund & Kadir, 2006(11)

 

 

 

Knowledge scores significantly improved from  50.8 – 63.9 (P<0.05)

 

Average self-reported reading time increased from 2.0 hours to 3.5 hours (p<0.05). 2 letters of critical appraisal published in peer review journals by grou

 

 

 

Haspel, 2010(16)

 

Attendance>50%; Survival > 2 years

 

44% learning of critical appraisal skills most important goal of JC

 

 

85% improved self confidence in critical appraisal

 

44% report improve reading behaviour

 

 

 

Mohrari et al., 2009(21)

 

 

 

Significant improvement in knowledge of methodology and application of information but not in study design or statistics. Demonstrated ability to appraise paper using CONSORT checklist after exercise, despite inability before

 

 

 

 

 

Temple & Ross, 2011(15)

 

8/10 average attendance

 

64% pre-course felt JC useful for teaching critical appraisal skills, significantly reduced to 50% after course

 

Non-significant increase in self-rated confidence of critical appraisal skills; non significant improvement in examination ( pre-course 43%, post course 52%; p=0.6)

 

 

 

 

 

Shokouhi et al., 2012(1)

 

 

Satisfaction score: 21.9 (EBM) vs. 13.18 (non EBM) (p<0.05)

 

Self evaluated critical skills 40.18 (EBM) vs. 16.72 (non EBM) (p<0.05)

 

 

 

 

Ibrahim et al., 2014(14)

 

90% of trainees report > 50% attendance

 

Critical appraisal skills rated by trainees as ‘important’ or ‘very important’ by 98% respondents

 

 

 

 

 

Lao et al., 2014(13)

 

Average attendance 14/24 (58%)

 

Mean satisfaction, relevance and usefulness rated  4.0-4.6 / 5.

 

 

Improved knowledge scores from pre-test 10.8/20 to 16.9/20 (p<0.05)

 

 

 

Declaration of Interest

There are no conflicts of interest to declare

Acknowledgements

The author is grateful to Dr Farida Yousaf for her comments on the manuscript

References

  1. S Shokouhi G, Ghojazadeh M, Sattarnezhad N. Organizing evidence based medicine (EBM) journal clubs in department of neurosurgery, Tabriz University of Medical Sciences. International journal of health sciences. 2012 Jan;6(1):59.
  2. Linzer M. The journal club and medical education: over one hundred years of unrecorded history. Postgraduate medical journal. 1987 Jun 1;63(740):475-8.
  3. Sackett DL, Rosenberg WMC, Gray JAM, Haynes RB, Richardson WS. Evidence based medicine: what it is and what it isn’t BMJ 1996; 312:71–2.
  4. Burls A. What is critical appraisal?. Hayward Medical Communications; 2009.
  5. Royal College of Psychiatrists. Core Training in Psychiatry CT1-CT3 – A Competency Based Curriculum for Specialist Core Training in Psychiatry. 2013 Available at: http://www.rcpsych.ac.uk/pdf/CoreCurriculum_FINALVersion_July2013_updatedFeb15 KM.pdf [Accessed: 1st October 2017].
  6. Kirkpatrick, D, American Society for Training and Development (ed.)1998, ‘Another look at evaluating training programs: fifty articles from “Training & development” and “Technical training” magazines cover the essentials of evaluation and return-on-investment’,
  7. Tochel C, Haig A, Hesketh A, Cadzow A, Beggs K, Colthart I, Peacock H. The effectiveness of portfolios for post-graduate assessment and education: BEME Guide No 12. Medical teacher. 2009 Jan 1;31(4):299-318.
  8. Yardley S, Dornan T. Kirkpatrick’s levels and education ‘evidence’. Medical education. 2012 Jan 1;46(1):97-106.
  9. Sidorov J. How are internal medicine residency journal clubs organized, and what makes them successful?. Archives of Internal Medicine. 1995 Jun 12;155(11):1193-7.
  10. Spillane AJ, Crowe PJ. The role of the journal club in surgical training. ANZ Journal of Surgery. 1998 Apr 1;68(4):288-91.
  11. Akhund S, Kadir MM. Do community medicine residency trainees learn through journal club? An experience from a developing country. BMC medical education. 2006 Aug 22;6(1):43.
  12. Sandifer QD, Lo SV, Crompton PG. Evaluation of a journal club as a forum to practise critical appraisal skills. Journal of the Royal College of Physicians of London. 1996;30(6):520-2.
  13. Lao WS, Puligandla P, Baird R. A pilot investigation of a pediatric surgery journal club. Journal of pediatric surgery. 2014 May 31;49(5):811-4.
  14. Ibrahim A, Mshelbwala PM, Mai A, Asuku ME, Mbibu HN. Perceived role of the journal clubs in teaching critical appraisal skills: a survey of surgical trainees in Nigeria. Nigerian Journal of Surgery. 2014;20(2):64-8
  15. Temple CL, Ross DC. Acquisition of evidence-based surgery skills in plastic surgery residency training. Journal of surgical education. 2011 Jun 30;68(3):167-71.
  16. Haspel RL. Implementation and assessment of a resident curriculum in evidence-based transfusion medicine. Archives of pathology & laboratory medicine. 2010 Jul;134(7):1054-9.
  17. Linzer M, Brown JT, Frazier LM, DeLong ER, Siegel WC. Impact of a medical journal club on house-staff reading habits, knowledge, and critical appraisal skills: a randomized control trial. JAMA. 1988 Nov 4;260(17):2537-41.
  18. Fu CH, Hodges B, Regehr G, Goldbloom DS, Garfinkel P. Is a journal club effective for teaching critical appraisal skills?. Academic Psychiatry. 1999 Dec 1;23(4):205-9..
  19. Sellig CB. Affecting residents’ literature reading attitudes, behaviors, and knowledge through a journal club intervention. Journal of general internal medicine. 1991 Jul 1;6(4):330-4.
  20. Khan KS, Dwarakanath LS, Pakkal M, Brace V, Awonuga A. Postgraduate journal club as a means of promoting evidence-based obstetrics and gynaecology. J Obstet Gynaecol (Lahore). 1999; 19(3):231–4.
  21. Moharari RS, Rahimi E, Najafi A, Khashayar P, Khajavi MR, Meysamie a P. Teaching critical appraisal and statistics in anesthesia journal club. QJM. 2009; 102(2):139–41.
  22. Bazarian JJ, Davis CO, Spillane LI, Blumstein H, Schneider SM. Teaching Emergency Medicine Residents Evidence-Based Critical Appraisal Skills : A Controlled Trial. Ann Emerg Med. 1999; 34(2):148–54.