Text corpus

In linguistics, a corpus (plural corpora) or text corpus is a language resource consisting of a large and structured set of texts (nowadays usually electronically stored and processed). In corpus linguistics, they are used to do statistical analysis and hypothesis testing, checking occurrences or validating linguistic rules within a specific language territory.

Overview

A corpus may contain texts in a single language (monolingual corpus) or text data in multiple languages (multilingual corpus).

In order to make the corpora more useful for doing linguistic research, they are often subjected to a process known as annotation. An example of annotating a corpus is part-of-speech tagging, or POS-tagging, in which information about each word's part of speech (verb, noun, adjective, etc.) is added to the corpus in the form of tags. Another example is indicating the lemma (base) form of each word. When the language of the corpus is not a working language of the researchers who use it, interlinear glossing is used to make the annotation bilingual.

Some corpora have further structured levels of analysis applied. In particular, a number of smaller corpora may be fully parsed. Such corpora are usually called Treebanks or Parsed Corpora. The difficulty of ensuring that the entire corpus is completely and consistently annotated means that these corpora are usually smaller, containing around one to three million words. Other levels of linguistic structured analysis are possible, including annotations for morphology, semantics and pragmatics.

Applications

Corpora are the main knowledge base in corpus linguistics. Other notable areas of application include:

  • Machine translation
    • Multilingual corpora that have been specially formatted for side-by-side comparison are called aligned parallel corpora. There are two main types of parallel corpora which contain texts in two languages. In a translation corpus, the texts in one language are translations of texts in the other language. In a comparable corpus, the texts are of the same kind and cover the same content, but they are not translations of each other.[2] To exploit a parallel text, some kind of text alignment identifying equivalent text segments (phrases or sentences) is a prerequisite for analysis. Machine translation algorithms for translating between two languages are often trained using parallel fragments comprising a first language corpus and a second language corpus which is an element-for-element translation of the first language corpus.[3]
  • Philologies
    • Text corpora are also used in the study of historical documents, for example in attempts to decipher ancient scripts, or in Biblical scholarship. Some archaeological corpora can be of such short duration that they provide a snapshot in time. One of the shortest corpora in time, may be the 15–30 year Amarna letters texts (1350 BC). The corpus of an ancient city, (for example the "Kültepe Texts" of Turkey), may go through a series of corpora, determined by their find site dates.

Some notable text corpora
Main article: List of text corpora
gollark: I mean, I suppose you can define it that way, but then it becomes a less useful concept and OH BEE HE HAS COME HERE
gollark: "Justice" in that context always seems to involve "horribly punishing people", for dubiously ethical reasons.
gollark: Actually, I'm a seed AI created to maximize the EXTREMELY complex function of "gollariosity".
gollark: Precisely as planned.
gollark: Don't fight for justice; fight for maximization of paperclips.

See also

References
  1. Yoon, H., & Hirvela, A. (2004). ESL Student Attitudes toward Corpus Use in L2 Writing. Journal of Second Language Writing, 13(4), 257–283. Retrieved 21 March 2012.
  2. Wołk, K.; Marasek, K. "A Sentence Meaning Based Alignment Method for Parallel Text Corpora Preparation". Advances in Intelligent Systems and Computing. Springer. 275: 107–114. ISBN 978-3-319-05950-1. ISSN 2194-5357.
  3. Wołk, K.; Marasek, K. (2015). "Tuned and GPU-accelerated Parallel Data Mining from Comparable Corpora". Lecture Notes in Artificial Intelligence. Springer: 32–40. arXiv:1509.08639. ISBN 978-3-319-24032-9.
This article is issued from Wikipedia. The text is licensed under Creative Commons - Attribution - Sharealike. Additional terms may apply for the media files.