This site is intended for health professionals only

Appraising quality of research: a vital skill

Una Adderley
RGN DN BSc BA
Research Nurse
Centre for Evidence Based Nursing
Department of Health Studies
University of York
District Nurse Scarborough and NE Yorks NHS Trust
E:una@adderleyspringfarm.fsnet.co.uk

So why do we try to keep up with research when the common perception is that advice is always changing? Unfortunately, clinicians are not born with an instinctive awareness of the "right thing" (despite what some may think!), and to rely only on intuition, custom or practice is at best irresponsible and sometimes downright dangerous. A far more reliable way of discovering the "right thing" is to examine the evidence base.

One helpful approach might be to regard the evidence base as simply what is known up to now. As new discoveries are made it is inevitable that practice should develop and become more effective. It can be reassuring to realise that new evidence is more likely to "fine tune" existing knowledge than cause a complete U-turn in practice.

Evidence-based practice requires the clinician to identify an area of clinical uncertainty, ask an answerable question, search for the relevant research, critically appraise that research, and then proceed to implement and audit.(2)

Unfortunately, not all research is reliable, and it is vital to be able to sort the wheat from the chaff. Appraising research requires a set of skills that involve knowledge and practice. Until recently, clinical education gave relatively little space to these skills. A generation of clinicians are now seeking these research awareness skills to ensure that their patients receive the best possible care.

Since most clinicians are desperately short of time, a strategic approach to seeking evidence is sensible. Flemming compares the process to going shopping.(2) It is more likely to be a successful trip if you compile a shopping list in advance. Similarly, spending time thinking about exactly what you are seeking before starting a library search will save time and effort.

There is a certain luxury if the search for evidence is occurring as part of a study programme where time is formally allocated and library resources are easily accessible. The reality for most clinicians is far tougher. Trying to squeeze a visit to the library during your lunchbreak or surfing the net in the evening can be frustrating and tiring. It is well worth negotiating some allocated time.

Identifying an area of clinical uncertainty and phrasing a question
The first stage in seeking to "do the right thing" is to consider where there are areas of uncertainty within clinical practice. Practice may already be based upon a reliable research base, but there are always some unknown areas. For example, although there is strong evidence to support the use of compression bandaging for healing venous leg ulcers, it is currently unknown which type of compression bandage is most useful.

Answerable questions consist of three parts:

  • The situation - the patient or problem being addressed.
  • The intervention - what is being done to the ­"situation".
  • The outcome - what is hoped for as a result of the "situation" and the "intervention" being ­combined.(2)

So, in a question such as, "Which compression bandage heals venous leg ulcers fastest?", the situation is venous leg ulceration, the intervention is compression bandaging and the outcome is faster healing.
 
Searching for relevant research
The next task is to consider what type of evidence is likely to answer your particular question, since there is no point reading research that has used the wrong design. The title and abstract should confirm whether the paper addresses your area of interest, but it is important to consider whether a research paper has used an appropriate methodology for the issue you are addressing.

A randomised controlled trial (RCT) is the "gold standard" method of dealing with questions about the effectiveness of a particular treatment, possible adverse effects and issues around prevention. People are randomly allocated to either the experimental or control group to minimise bias. The aim is to discover the true impact of the intervention under investigation.

If the condition is very rare, an RCT may not be possible due to the small numbers involved. A case-control study may be the only option.

Cohort studies follow a sample of people with a similar condition over a set period of time. They are ­appropriate when investigating causes of ill-health and questions of prognosis, such as: "What is the expected healing rate for patients with venous leg ulceration?"

However, if you are seeking information about the issues that concern people in a specific situation or with a certain condition, a qualitative study will be more appropriate.(1) Qualitative studies investigate feelings, understanding and perceptions, such as: "How do patients with leg ulcers feel about their experience of leg ulceration?"

A rigorous systematic review that pools and analyses the results of all relevant, high-quality trials is even more reliable than the results of an individual trial. High-quality systematic reviews should have clear, explicit methods and critically appraise each piece of research they include to provide reliable information for practice.

The good news is that often someone else has already done the hard work for you. Given the vast amount of biomedical literature available it can be useful to prioritise evidence. The concept of a "hierarchy of evidence for effectiveness" recognises that some types of evidence are more reliable than others (see Figure 1).

[[NIP09_fig1_57]]

When conducting a search for evidence it is worth first seeking evidence from the top of the hierarchy. Pre-appraised systematic reviews can be found in the evidence-based journals such as Evidence-Based Medicine and Evidence-Based Nursing. If you are in luck and a high-quality systematic review exists on your chosen topic, then that may be sufficient for your needs.

If no preappraised systematic review can be found then it is necessary to seek out a systematic review through databases such as the Cochrane Library and DARE (Database of Abstracts of Reviews of Effectiveness). If one exists, then it is important to appraise its quality to ensure that it has been carried out in a truly systematic and comprehensive manner. If no systematic review exists (and you are really keen) it may be worth considering undertaking a systematic review yourself. The Cochrane organisation runs excellent workshops to educate and support would-be reviewers.

In the absence of an existing, high-quality systematic review, the next step is to seek original research that has already been reliably appraised. Again, the evidence-based journals are a reliable source of high-quality research articles accompanied by high-quality appraisal.

If no preappraised research can be found then it becomes necessary to trawl databases such as Medline, CINAHL and the Cochrane Controlled Trials Register for sound research from individuals. It is worth remembering that publication alone is no guarantee of quality - research studies with positive results are more likely to get published. Again, any relevant research discovered will need to be appraised for quality.

Checklists such as those published by CASP (NHS Critical Appraisal Skills Programme) and the Users Guides to the Medical Literature (published annually by the Journal of the American Medical Association) are extremely useful. CASP's aim is "to empower health and social care professionals and users of the NHS to distinguish between, and to use, good quality evidence to support decisions". It produces a series of appraisal tools in the form of checklists to help evaluate the quality of different types of research, including systematic reviews, RCTs and qualitative studies.

Finally, if no high-quality research exists for your particular area of interest, then practice has to rely on expert opinion. In the absence of good evidence this becomes the only option. However, it is worth remembering that expert opinion can be wrong. Nurses used to pour sodium hydrochlorite into wounds to promote healing. This wasn't because they were a generation of sadomasochists but because expert opinion at the time held that this was best practice.

It can be argued that practising clinicians have a responsibility to inform the clinical research community about the areas where evidence is missing. Researchers need input from clinicians to ensure that the research agenda matches the needs of clinicians. Practising clinicians should not be expected to be skilled at carrying out original research. However, it is reasonable to expect clinicians to be able to identify quality evidence, implement it where appropriate and audit practice.

References

  1. Cullum N, Thompson C. Critical appraisal 1. Examining evidence: an overview. Nursing Times Learning Curve 1999;3(1):7-9.
  2. Flemming K. Critical appraisal 2. Searchable questions. Nursing Times Learning Curve 1999;3(2):6-7.

Resource
NHS Critical Appraisal Skills Programme (CASP)
W:www.phru.org.uk/~casp

Further reading
Guyatt G, Rennie D, editors. Users' guides to the medical literature: a manual for evidence-based clinical practice. Chicago: American Medical Association; 2002