Massimo Poesio and Ron Artstein.
The reliability of anaphoric annotation, reconsidered: Taking ambiguity into account.
Proceedings of the Workshop on Frontiers in Corpus Annotation II: Pie in the Sky, pages 76-83. Ann Arbor, Michigan, June 2005.

Full paper in PDF (71K)

Abstract: We report the results of a study of the reliability of anaphoric annotation which (i) involved a substantial number of naive subjects, (ii) used Krippendorff's α instead of κ to measure agreement, as recently proposed by Passonneau, and (iii) allowed annotators to mark anaphoric expressions as ambiguous.

Full paper in PDF (71K)