CALBC · Collaborative Annotation of a Large Biomedical Corpus

This proposal defines a support action project that brings together the researchers from international biomedical text-mining groups to address the difficult issue of annotating large text corpora with a large set of semantic types. We propose a collaborative approach to this annotation task in the form of an open challenge to the biomedical text-mining community. The task is the annotation of named entities in a large biomedical corpus, for a variety of semantic categories. The project delivers as outcome a large, collaboratively annotated corpus, marked with the mentions of biomedical entities. The annotated corpus becomes a resource for the community, to be used as a reference for improving text-mining applications. The biomedical text-mining research community has a long tradition of organizing such challenges, as a way of evaluating techniques, sharing technical knowledge, and helping to improve the results from text-mining programs. However, such challenges have typically addressed relatively small corpora in a narrow sub-domain, in part because the evaluation of the results is extremely long and costly. As a result, the generated annotated corpora are too small and are only narrowly annotated to be useful in a variety of text-mining applications. In contrast, we propose to create a broadly-scoped and large annotated corpus by integrating the annotations from different named entity recognition systems. Metadata will also be added to the corpus. The participating systems have different application scopes and annotation strategies, and therefore complement each other. As a consequence, the annotated corpus reflects these different scopes and strategies. A secondary goal of this project is to define a standardized format for representing the annotations contributed by the participants and comparing them effectively. Currently the lack of such a format hinders progress in the evaluation of named entity recognition systems.