Corpus annotation

The annotation goal was to get a corpus of reviews in which named entities reflecting pharmacotherapeutic treatment are labelled, and annotate medication characteristic semantically. With this in mind, the objects of annotation were attributes of drugs, diseases (including their symptoms), and undesirable reactions to those drugs. The annotators were to label mentions of these three entities with their attributes defined below.представлены ниже.

Medication

This entity includes everything related to the mentions of drugs and drugs manufacturers. Selecting a mention of such entity, an annotator had to specify an attribute out of those specified below:

Disease

This entity is associated with diseases or symptoms. It indicates the reason for taking a medicine, the name of the disease, and improvement or worsening of the patient state after taking the drug. Attributes of this entity are specified below:

ADR

This entity is associated with adverse drug reactions in the text. For example, one post said: “После недели приема Кортексина у ребенка начались судороги” (After a week of taking Cortexin, the child began to cramp). In this sentence, the word “судороги” (“cramp”) is labeled as an ADR entity.

Note

We use this entity when the author makes recommendations, tips, and so on, but does not explicitly state whether the drug helps or not. These include phrases like “I do not advise”. For instance, the phrase “Нет поддержки для иммунной системы” (No support for the immune system) is annotated as a Note.

Fig. 1. Annotation example.

The typical situations that had to be handled during the annotation are the following:

• A simple markup, when a mention consists of 1 or more words and it related to a single attribute of entity. The annotators then just have to select a minimal but meaningful text fragment, excluding conjunctions, introductory words, and punctuation marks.

• Discontinuous annotation – when mentions separated by words that are not part of it. It is then necessary to annotate mention parts and connect them. In such cases we use the “concatenation” relation. In the example (e) on Fig. “Annotation example” the words “prescribed” and “pediatrician” are annotated as a concatenated parts of mention of the attribute “sourceInfoDrug”.

• Intersecting annotations. Words in a text can belong to mentions of different entities or attributes simultaneously. For example, in the sentence “Rapid treatment of cold and flu” (see Fig. “Annotation example”, example (b)), words “cold” and “flu” are mentions of attribute “diseasename”, but at the same time the whole phrase is a mention of attribute “BNE-Pos”. If a word or a phrase belongs to a mentions of different attributes or entities at the same time (for example, “drugname” and “drugbrand”), it should be annotated with all of them: see, for instance, entity “Aqua Maris” in sentence “Spray Jadran Aqua Maris” (Fig. “Annotation example”, example (a)).

• Another complex situation is when an analogue (or, in some cases, several analogues) of the drugs are mentioned in a text, for example, when a customer wrote about a drug and then described an alternative that helped them. In this case, the “Other” attribute is used (example (c)).

Moreover, there often were author subjective arguments instead of explicit reports on the outcomes. We labeled that as a mention of entity “Note”. For example, “strange meds”, “not impressed”, “it is not clear whether it worked or not”, “ambiguous effect” (example (d) in Fig. “Annotation example”).

Normalisation

In our work, in order to resolve possible ambiguity in terms we performed normalization by matching the labeled mentions to the information from external official classifiers and registers. The external sources for Russian are described below:

• The 10-th revision of the International Statistical Classification of Diseases and Related Health Problems (ICD-10) is an international classification system for diseases which includes 22 classes of diagnoses, each consisting of up to 100 categories. The ICD-10 makes it possible to reduce verbal diagnoses of diseases and health problems to unified codes.
• The Anatomical Therapeutic Chemical (ATC) is an international medication classification system containing 14 anatomical main groups and 4 levels of subgroups. The ICD-10 and the ATC have a hierarchical structure, where “leaves” (terminal elements) are specified diseases or medications, and “nodes” are groups or categories. Every node has a code, which includes the code of its parent node.
• State Register of Medicinal Products (SRD)(“Государственный реестр лекарственных средств (ГРЛС)” in Russian) is a register of detailed information about the medications certified in the Russian Federation. It includes possible manufacturers, dosages, dosage forms, ATC codes, indications, and so on.
• MeSH Russian (MESHRUS) is a Russian version of the Medical Subject Headings (MESH) database (Home page of the MeSH database site. MESH is a dictionary designed for indexing biomedical information that contains concepts from scientific journal articles and books and is intended for their indexing and searching. The MESH database is filled from articles in English; however, there exist translations of the database to different languages. We used the Russian version, MESHRUS. It is a less complete analogue of the English version, for example, it doesn’t contain concept definitions.

Among the international systems of standardization of concepts, the most complete and large metathezaurus is UMLS, which combines most of the databases of medical concepts and observations, including MESH (and MESHRUS), ATC, ICD-10, SNOMED CT, LOINC and others. Every unique concept in the UMLS has an identification code CUI, using which one can get information about the concept from all the databases. However, within UMLS it is only the MESHRUS database that contains Russian language and can be used to associate words from our texts with CUI codes.

Normalization based on categories from the ATC and ICD-10 classifiers

Normalization was carried out by the annotators manually. For this purpose, we applied the procedure consisting of the following steps: automatic grouping of mentions (standardization), manual verification of mention groups, matching the mention groups to the terms from the ATC and the ICD-10. Automatic mentions grouping is based on calculating the similarity between two mentions by the Ratcliff/Obershelp algorithm, which is based on searching two strings for matching substrings. In the course of the analysis, every new mention is added to one of the existing groups G if the mean similarity between the mention and all the group items is more than 0.8 (value deduced empirically), otherwise a new group is created. The G set is empty at the start, and the first mention creates a new group with size 1. Each group is named by its most frequent mention. Next, the annotators manually check and refine the resulting set, creating larger groups or renaming them.

After that, the group names for attributes “Diseasename”, “Drugname” and “Drugclass” are manually matched with ICD-10 and ATC terms to assign term codes from the classifiers. As a result, 141 unique ICD-10 codes were matched against the 1,333 mentions of attribute “Diseasename”; 171 unique ATC codes matched the 2,360 mentions of attribute “Drugname”; and 26 unique ATC codes corresponded to 1,092 mentions of “Drugclass”. Some drug classes that were mentioned in corpus (such as homeopathy) did not have a corresponding ATC code, and were aggregated according to their anatomical and therapeutic classification in the SRD.