Abstract

askMEDLINE is a free-text, natural language search tool for MEDLINE/PubMed. Since its introduction a year ago, more than 15,000 queries have been received, mostly in the form of questions or complex phrases. Questions on therapy were the most searched. We review our yearlong experience, an updated evaluation using ‘gold standard’ questions selected by EBM experts and discuss added features.

Background

askMEDLINE was originally developed as a tool for parsing clinical questions to automatically complete the PICO (Patient, Intervention, Comparison, Outcome) form. We discovered that this was quite challenging because most of the PICO terms were interchangeable. However, we found unexpectedly that sending the parsed medical terms derived from the clinical question using its algorithm, successfully retrieved relevant citations from MEDLINE/PubMed. We then launched askMEDLINE as an alternative search tool that might be useful for non-expert information seekers.^{1, 2}

New features

GSpell, a spellchecker that provides spelling suggestions was added soon after launch. Recently, we enabled a search tool so that all questions in the archive can be openly browsed and references relevant to the query can be quickly accessed. A listing of the most commonly searched terms (default 10 terms, limit 100) with links to questions and references has also been made available.

Review and evaluation methods

A review of the server log showed that 14,878 queries have been submitted a year after launch. In 1138 (7.5%) searches, no citations were found. In 3389 searches where GSpell’s spelling suggestions were applied, positive results were obtained. In over 330 instances, there were still no retrievals.

We wanted to re-evaluate askMEDLINE using a set of ‘gold standard’ articles selected by experts in Evidence-based Medicine (EBM.) This was undertaken because askMEDLINE could be useful in EBM. Previous evaluation studies were reported earlier.¹ Seven clinical questions were taken from a collection of a larger set of clinical questions. Recall was 41±29% and precision was 16±7%.

A computer algorithm run nightly was developed to determine the most search terms in askMEDLINE and delete duplicated questions. Queries relating to therapy and treatment, followed by ‘cancer,’ ‘patients,’ ‘disease,’ and ‘effective,’ were top five most searched categories. Among ‘clinical queries’ category, ‘treatment’ and ‘therapy’ were the most asked, 85% (Table 1.)

Seventeen responses were received through the feedback form. Nine responders were physicians and two medical librarians. On a 5-point Likert scale, users rated askMEDLINE’s overall relevance of retrieved articles, 4.59 and 4.71 on the question whether it successfully ‘answered’ their question. All, except one, responded that they would continue to use it. One librarian commented how surprisingly relevant the answers were.

Conclusion

askMEDLINE was developed as a resource tool for the non-expert medical information seeker. Evaluations of sample questions showed that the precision and recall of the search results from askMEDLINE were consistent with other reports on search retrieval. New features allow access to previously asked questions and spelling assistance.

References