Frequently Asked Questions on Evidence Matters

What is Evidence Matters? Range of Content - General Information Sources and Quality Special Page - The Question Wizard Special Page - The Results Page Special Page - The Article Summary Page

What is Evidence Matters?

What is the basic product / service you provide?
Evidence Matters is an online healthcare knowledge management system to assist decision makers in their treatment choices for patients. The main goal is to provide quick and easy access to evidence extracted from research articles on particular topics. For a more detailed description, see the "About Project" page.
Back to top

How does it work?
Our staff of health professionals (MDs, medical librarians etc.) use advanced software and manual work to extract data from medical treatment research articles published in peer-reviewed scientific print journals. Research from both past and present is aggregated and made available via comparative tables, graphs and bullet-point text summaries. Using menus, you can construct a question on therapeutics, and get an answer in the form of a table, a list or a graph which summarizes a set of articles. From these views, you can drill down to single article summaries.
Back to top

Range of Content - General

Which are the disease areas you are going to focus on?
Evidence Matters is updated daily and currently includes over 150,000 outcomes covering 1,400 treatments within the following disease modules: Oncology, Cardiology, Endocrine & Metabolic Disorders, Gastroenterology, Respiratory Disorders, Neurology, Urology, Orthopaedics, Gynecology, and Pediatrics.
Back to top

How many and what types of treatments do you cover?
We cover hundreds of treatments, both "conventional" and "alternative" - where ever there is lots of published research available. We cover categories from general (such as "surgery" or "chemotherapy") to very specific (such as "fadrozole" and "simple mastectomy"). You can have a more precise idea by going into our Question Wizard and selecting the "See more therapies" menu item, to view the full list of treatments for which we extract data. Click on the "+" signs in the tree at the right to expand the different branches of the treatment tree.
Back to top

How much information is available about the benefits, harm, and costs of each treatment?
Benefits and harm can be assessed on our Results Page. Results are displayed in three formats: a table, a list and a graph, that each summarize the results uniquely. For more information about the Results Page, see the corresponding section in the FAQ's. Costs vary considerably depending on location and require customized installation. Costs are provided as an advanced feature for institutions such as hospitals.
Back to top

Do you cover both the latest treatments as well as old ones?
Yes. We enter data in our system on older treatments, since many of them are still used, or are good comparison points. We also update our database several times a week with research published in the latest journals.
Back to top

Do you provide dosages and interaction information on each treatment?
Yes. Very detailed information concerning the specifics of each treatment can be found in the article summary, such as dosages, treatments combinations, etc.
Back to top

If I am interested in a specific reference article, do I have access to the corresponding full text or abstract?
Yes. Full text can be accessed directly within Evidence Matters by simply following the corresponding Full Text (Open URL) links associated to each article, from either the Results Page or the Article Summary Page.

In order to benefit from the full text via Full Text (Open URL) functionality, the following two requirements need to be met:

1) Your access to Evidence Matters needs to be set up via IP recognition. If you currently have Username-Password access to Evidence Matters only, please ask your library, medical service, or local department to contact us for set up (contactus@evidencematters.com).

2) Your institution needs to have an Full Text (Open URL) resolver (such as ArticleFinder from Serials Solutions or SFX from ExLibris), and they need to provide us with their Static Base URL. The Base URL will be in the format 'http://...'. If your institution does not know what their Static Base URL is, they may contact their Full Text (Open URL) link resolver's Customer Service to help them find out. Once their Static Base URL is available, they can just send it to us by e-mail to contactus@evidencematters.com.

Alternatively, the PubMed abstract for every article can be accessed by following the corresponding links in either the Results Page or the Article Summary Page. By clicking on the PubMed Abstract link, you will be brought directly to the abstract on the PubMed website in a new window. This allows users access beyond EM's time-saving article summaries to not only the abstract on PubMed, but also full-text links (via PubMed LinkOut), Related Article links, article location and other information.

Back to top

Information Sources and Quality

How up-to-date is your information?
It takes only a few hours for our entire system to be updated, once we receive a new full-text research article. To get new articles, we conduct electronic database searches daily or several times weekly for each disease area. Depending on the timeliness of the databases searched, it can take from one day up to a few weeks for original research to make it from print into the EM database.
Back to top

What are the sources of information for your data?
Evidence Matters' data comes primarily from print research published in peer-reviewed scientific journals. Peer-review is a process for improving the quality of published research via review by experts in the field. While peer review is not a guarantee of quality, it is an indicator of quality control. Occasionally, interesting research comes from a non-peer-reviewed source, and so these are included as well.
Over 12,000 different journals are scanned via databases (such as Medline, Cochrane, CINAHL, etc.) for potential entry into EM on a daily basis. The above journal set is filtered for articles on therapeutic effectiveness and safety on humans. To date, over 450 journals have made it into EM, as articles within this subset of journals only were found to match the medical data abstractors' search criteria.
The cumulative, wide breadth of sources of information populating the Evidence Matters database is thus directly tied to the search strategy employed by our team of medical content abstractors. For more details on this methodology, please see the answer to the FAQ section titled "What is your search protocol?".
Likewise, Evidence Matters' journal list is not a "closed" one, but an ever-expanding "open list". New journal titles are added as new, relevant articles are located. For the latest version of the complete list of journals containing data that was abstracted into the Evidence Matters database, select "Journal List" from the drop-down options available under "Help" on the main menu bar. References can be as recent as this week or date from as far back as 1970. A 5-year span of coverage is the minimum provided for the majority of journals. New articles are entered on a daily basis. For further information on precise journal dates, please contact us directly.

Back to top

What is your search protocol (databases, search terms, final choices)?
Search Strategy
Electronic searches are conducted daily or several times weekly on the following databases: Medline, CINAHL, and the Cochrane Library. Search terms include the names and synonyms for each disease area. As well, hand searches are performed of bibliographies from top textbooks in the disease field, as well as other disease-specific resources.

Selection Criteria
Studies published in English, describing trials of interventions on humans are kept. All trial designs are accepted, from randomized controlled trials to case reports.

Data Extraction and First Review
Data extraction and review are performed by specially trained professionals with medical backgrounds. Most have MD degrees, and some have masters' degrees in library science. Data extraction integrity is reviewed by a second reviewer.

Second Review
Blinded review of the generated article summaries are conducted by company staff editors.

Third Review
World-wide peer-review is conducted on an ongoing basis. Any user that believes they have found a discrepancy in any statistic or article summary can notify us directly from the related web page. After six months with no further corrections from external sources, an article is indicated as having been viewed for that period of time with no corrections.

Update Process
Electronic searches are conducted daily or several times weekly for each disease. Depending on the timeliness of the databases searched, it can take from one day up to a few weeks for original research to make it into the EM database.


Back to top

What are your "3 levels" of quality control to prevent errors?
After information has been extracted from articles, it undergoes a three-level review process (see What is your search protocol?). Review is done by a trained medical professional (MD or medical librarian), then by an editor, and finally, is opened up to criticism to the whole world, directly from the web pages that display the research summaries. Any user that believes they have found a discrepancy in any statistic or article summary can notify us directly from the related web page. After six months with no further corrections from external sources, an article is indicated as having been viewed for that period of time with no corrections.
Back to top

Who inputs the information into the system?
Two physicians from Evidence Matters' medical content team extract and review the abstracted information, respectively.
Back to top

Special Page - The Question Wizard

What is the purpose of the question wizard?
The question wizard will help you structure an answerable question to ask the system. It can be more or less detailed. Once you press the "GO" button, only the research results that answer your question (i.e., match your "search criteria") will then be shown.
Back to top

Where can I get more choices of therapies or outcomes?
Because our system contains thousands of therapies and outcomes, we could not possibly show them all in a drop down menu. For this purpose, you can click on the "See more therapies" or "See more outcomes" drop-down menu items to assist you in finding some more specific and less common choices. See the FAQ "How does the tree window work?" for more details.
Back to top

How does the tree selection window work?
After selecting the "See more diseases / therapies / outcomes" options from the drop-down menus in the Question Wizard, you will get a pop-up window divided in three parts, labeled 1, 2 and 3. If you don't know exactly what medical term you are looking for, we suggest entering the first few letters of a word into the section number 1 ("Type in a word"). You will see a list of matching terms appear on the left part (section 2). Click on the term you are interested in, to locate it in the tree on the right part (section 3). If you are sure this is the term you want to use for your question, click again on the word inside the tree to select it. If you prefer, you can click on the "parent" of that term in the tree. Creating a question with the "parent" term will return research results with both the "parent" and the "child".
Back to top

Why are some terms grayed out in the tree window?
If a word appears in gray in either your search results or the tree window, it means that there are no results in our database for that combination. You might want to then choose a "parent" of this term (i.e., a more general category). The terms in the tree represent a taxonomy, which goes from the more general concepts to the more specific, so at any time you can choose a general concept to get more results.
Back to top

I get no results for my questions, what should I do?
A good idea is to first start building a question that has no "Advanced Filters" selected. Advanced Filters tend to narrow the results significantly. Once you have generated results for your question, you can try narrowing down the results by using the one-click filters in the left side menu. In this fashion, you always know in advance how many results are available for each filter you wish to apply, as it corresponds to the numbers displayed between parentheses to the right of the filter terms.
Back to top

What are the plus or minus signs near some items?
The plus or minus signs serve the same purpose as on the Article Summary page, which is to expand or collapse certain parts of the display. In the case of the selection tree in the pop-up window for diseases, therapies or outcomes, you can open or close the sub-categories (sub-branches) of the item you are viewing.
Back to top

What are the numbers displayed between parentheses to the right of the tree items?
The numbers between parentheses, to the right of the tree items, correspond to the number of outcome results that are available for that particular term in the Evidence Matters database. These numbers are sensitive to your current choice of disease, therapy and outcome, thus you may notice that the figures change as you modify your search.
Back to top

Special Page - The Results Page

What information is provided in the results page?
The results page organizes all of the information found to match your search criteria onto two main tabs placed at the center of the page, EM Results and PubMed Results. The question you asked appears just above the two results tabs and, on the left side column, you can see any filters you've already applied on top, followed by a categorized list of additional one-click filters, which you may select in order to further refine your results.
Back to top

What information is provided in the EM Results tab?
The EM Results tab displays the number of outcome results in the articles that apply to the question you asked in three different, interactive views: a table, a list and a graph.
In the table and list views, you may rank results by a criterion of your choice: type of research, year of publication or journal name. Results may be ordered within these criteria by the timing of the outcome measurement, the year of publication or the actual study outcome.
Both the table and list views provide important details about the study results, including the interventions applied, the study outcome, the timing of the outcome and a complete bibliographic reference. Three links are also available for every study result: the EM Summary Page, the PubMed Abstract and Full-Text access for the article, via Full Text (Open URL) (Please note: you will only see the Full Text (Open URL) link if you are accessing the system via IP recognition and your library or institution has provided its Base URL to Evidence Matters).
The graph provides an overview of all the study results matching a certain clinical question, summarized in a three dimensional manner. The columns on the X axis represent the level of clinical evidence available, with case reports on the far left and randomized clinical trials on the far right. The height of the boxes on the Y axis range shows how effective or harmful a certain intervention is. The Z axis, represented by different colours, demonstrates how many outcome results are available within a given range. A darker colour in a box indicates a higher number of results.
EM results may be refined by a wide selection of different criteria, including patient characteristics (e.g. age, sex, ethnicity), disease characteristics, study characteristics (e.g. countries in the study, type of research or research design, blinding characteristics), as well as article characteristics (e.g. year of publication).

Back to top

What happens when I click on a box in the graph?
You are brought to the table view after you click on a box on the graph page. The criteria you selected (e.g. 80-90% survival rate from randomized clinical trials) will appear at the top of the filters section of the page, on the left-side column. On the main EM Results tab, at the center of the page, you will see the study details and corresponding bibliographic references for the outcome results that were part of that box's criteria.
Back to top

Why are some references blocked? (When I click it asks me to login.)
As a non-subscribed user, you are provided with a basic working access to our system, which we hope will give you enough of a feel for how it works and how useful it can be. Of course, we cannot provide everything for free (the system is very expensive to maintain), thus a subscription is required for further access to the system. We have different levels of subscriptions that may interest you. For subscription requests or inquiries, you may reach us through the form provided under "About Us" >> "Contact Us", on main menu bar.
Back to top

Why do some references repeat many times?
A given research article often has multiple results that are almost similar. For example, two groups of patients in the same research article might have had 81% and 85% respectively as their 5-year survival rate. Often each group received some common and some different therapies, as shown in the "Interventions" column on the results page.
Back to top

What information is provided in the PubMed Results tab?
The PubMed Results tab provides a ranked, evidence-based list of related articles from the PubMed database, organized by Evidence Matters according to research design: systematic reviews, randomized controlled trials, case reports, and guidelines.
PubMed results may be refined by type of research (i.e., research design), year of publication, and publication name.

Back to top

How can I change my question?
In order to change the question, either click the "Change" button at the top right of the results page, or select the "Build a Question" item in the main menu, under "Questions". Most of the time, you can also simply use the "Back" button on your Internet browser for the same effect. This will bring you to the Question Wizard page, which will assist you in creating or modifying your question. See the Question Wizard FAQ's for more help.
Back to top

Can I store a list of questions that I like?
You can see what questions other people have been asking in "Common Questions" link, in the "Questions" menu at the top of the page. A special feature, for subscribers only, is the ability to save the questions you have created, and to see how the graphs that answer these evolve as new information is added. As a subscriber, once you are logged in, you will get a link next to the "Question Asked" box saying "Add to favorites". This way you can save as many questions that you like and view them again later.
Back to top

Special Page - The Article Summary Page

What sort of data and statistics does EM provide in its article summaries?
The article summary provides a simplified, structured, bullet-point summary of an article. Each article is always structured the same way, so once you are used to the layout, it is much easier to skim and compare articles. The data is divided into different sections, which describe the results, the quality of the research itself, the characteristics of the patient population studied, and the treatments applied to these groups.
Back to top

What is the plus or minus sign near the summary items?
Because some sections can get quite long, we have a feature to expand or collapse certain parts of the summary. Whenever you see a plus or minus sign, you can click on the item to expand or collapse it.
Back to top

What do the checkmarks mean (next to the therapy names)?
The checkmarks show which group(s) received a treatment listed in the table.
Back to top

How can I view the other outcomes from that article?
Since most articles have many outcomes extracted, you can view the other outcomes by using the selection box which is the first item in the "What are the results" section (just over the histogram of outcomes).
Back to top

Why are some outcomes in the selection box in red?
If you are using the free version, then every second outcome is in red and cannot be viewed. All outcomes are available in paid versions.
Back to top

Can I compare the results of an article's outcome to those in other articles?
Yes. A special link right under the article summary's histogram can be clicked to obtain a graph showing where the results of this article stand in comparison to all the others in our database, for the outcome you currently have selected.
Back to top

I would like to see more information, what can I do?
In many cases some data will not appear in certain parts of the article summary for various reasons. Firstly, the free version of the system only shows a limited number of bullet points with data from the article, compared to the paid version. For example, to see the stage of the disease for a patient, one must be subscribed. Secondly, information is sometimes simply not presented in the original article. For example, the article might have talked about the timing of the outcome for a given group but not for the other. Finally, in some cases, you may be looking for information that is simply never available in our article summaries (for example, corporate information about the publishing journal). We have chosen to display only a part of the information that is actually extracted from the articles for the purpose of simplifying the summary.
Back to top

Where can I get a full version of an article?
If you are accessing the system via IP recognition and your library or institution has provided its Base URL to Evidence Matters, you will see a "Full Text (Open URL link)" for full-text access to every article on the Results Page and on the Article Summary Page. By clicking this link, you will be brought directly to the full article, if your library or institution subscribes to the journal in which the article was found.
Alternatively, you can click on the "PubMed Abstract" link for the article, also found on both the Results Page and the Article Summary Page, which will direct you to the article abstract on the PubMed website (via LinkOut), with links to full-text, related articles links, article location, and other information. Furthermore, the bibliographic references provided for each article on the table and list views in the Results Page, as well as at the top of the Article Summary page, may also be used to retrieve the original article directly from a medical library.

Back to top

Why do some therapies appear in pink/red in the article summary?
Treatments appearing in pink/red are those which were planned in the research design. Usually, research reports the effect of treatments planned and used for all the patients in a group ("Planned therapies"). However, sometimes some patients who may not do well end up having to receive supplementary or replacement therapies that were not part of the original design ("Unplanned therapies"). To be most transparent, we list all the therapies that patients received, as best as can be determined from the article.
Back to top

How accurate is the content on the summary?
Our data extraction process goes through a three step quality-control procedure. For further details, please refer to the FAQ Section "Information Sources and Quality".
Back to top

If I think I see an error in an article, what can I do to help?
One of the great advantages of Evidence Matters is that it is continuously open to a world-wide reader review. We are happy to receive feedback and will look into reports of any possible discrepancies or errors in data. Please contact us through the online contact form available under the main menu item "Help" >> "Support", provide some details to help us locate the article(s) and issue(s) found, and we will conduct a prompt review.
Back to top