Inter-annotator Agreement for the Factual Status of Predicates in the TAGFACT corpus

Authors

  • Ana Fernandez-Montraveta Universitat Autònoma de Barcelona
  • Irene Castellón Masalles Universitat de Barcelona

Keywords:

factuality, inter-annotator agreement, Cohen’s Kappa, journalistic texts, Spanish

Abstract

This paper reports on a study of the inter-annotation agreement to assess the manual annotation of the TAGFACT Gold Standard corpus. This corpus has been created as part of a larger project (TAGFACT) whose final objective is to automatize the recognition of the factual status of events in a corpus of Spanish journalistic texts. In our study, six annotators labelled the four levels of linguistic description used to extract factual information in our project. Each one of these levels has been assessed independently. As expected, the more fine-grained the description is the more problematic the annotation. This study identifies some of the most important differences and discusses the main problems encountered to obtain full agreement. We use Cohen’s Kappa to measure annotation agreement as well as descriptive statistical analysis.

Author Biographies

Ana Fernandez-Montraveta, Universitat Autònoma de Barcelona

Departamento de Filologia Inglesa y de Germanística

Profesora Titular

Irene Castellón Masalles, Universitat de Barcelona

Profesora Titular

Departamento de Lingüística

Downloads

Published

2023-03-31

How to Cite

Fernandez-Montraveta, A., & Castellón Masalles, I. (2023). Inter-annotator Agreement for the Factual Status of Predicates in the TAGFACT corpus. Revista Signos. Estudios De Lingüística, 56(111). Retrieved from https://revistasignos.cl/index.php/signos/article/view/581

Issue

Vol. 56 No. 111 (2023): March

Section

Articles

License

Copyright (c) 2023 Revista Signos. Estudios de Lingüística

Creative Commons License

This work is licensed under a Creative Commons Attribution 4.0 International License.

Copyright agreement:

Authors who have a manuscript accepted for publication in this journal agree to the following terms:

  1. Authors will retain their copyright and grant the journal the right of first publication of their work by means of this copyright agreement document, which is subject to the Creative Commons Acknowledgment License that allows third parties to share the work provided that its author and first publication in this journal are indicated.

  2. Authors may adopt other non-exclusive license agreements for distribution of the published version of the work (e.g., depositing it in an institutional repository or publishing it in a monographic volume) as long as the initial publication in this journal is indicated.

  3. Authors are allowed and encouraged to disseminate their work via the internet (e.g., in institutional publications or on their website) before and during the submission process, which can lead to interesting exchanges and increase citations of the published work (read more here).