Beyond AI (7): The Imitation Game (part 4) Textual analysis

Natural language processing: In 1990 after having completed my teacher training, I stayed at the university for my doctoral studies. And soon after the Deutschmark came to East Germany on July 1, 1990, I bought a personal computer (Figure 10) and found some software that were not yet in use at the Pädagogische Hochschule Leipzig, because the computers had neither been donated nor delivered yet.

Figure 10: Personal computer with 80286 processor and 5 1/4″ floppy disk drive

To conduct textual analysis for my doctoral dissertation project, I scripted a macro in WordPerfect (Figure 11), which compiled lists of all word forms, two-form and three-form sequences, and their respective frequencies, transferred the lists to PlanPerfect, a spreadsheet software also sold by the WordPerfect Corporation, to determine the connectedness of each three-form sequence by calculating the probability with which the initial two-form sequence is followed by the third word form.

Figure 11: WordPerfect 5.1 for DOS

I didn’t know at the time that such sequences in texts were already called n-grams (e.g., Suen, 1979) and that they were the basis for statistical NLP and other work in computational linguistics (Figure 12). The most common n-s are 2 and 3, the bigrams and trigrams, respectively. Various statistical analyses can be carried out using n-grams. But I had never had classes in statistics; nor did I have any formal instruction in computing, never mind programming. So, it was after a lot of reading that I came across the Pearson Correlation Coefficient (Figure 13) and even later I relized I should have calculated a simple probability instead. Using it, PlanPerfect calculated the probability of a bigram being followed by a certain word form – a unigram – creating a more or less predictable trigram.

Figure 12: n-gram examples (unigram, bigram, trigram)

After this worked with the WordPerfect macro for a single text and was painfully slow – even slower than the printing on a 24-pin dot matrix printer in its high-quality mode, which I used for printing dissertation drafts over night while sleeping in the same rented room – I wanted to learn a programming language to produce a faster program to analyze more texts efficiently. I had borrowed the Grundstrukturen: Freiburger Korpus. Basic Structures: Freiburg Corpus (FR)  (2026), a spoken language corpus that consists of 221 digitized transcripts that were sitting on an IBM mainframe computer in the basement of the Institut für deutsche Sprache – the Institute for the German Language – in Mannheim, Germany. I visited the Institute in 1991, and they copied the entire corpus from the mainframe computer to a set of eleven 5¼–inch floppy disks for the personal computer. I must have looked puzzled, when one of the researchers at the Institute handed me the disks. Was that just the first instalment, I asked … The 221 transcriptions of the Freiburger Korpus are of spoken texts of a total duration of 68 hours and 6 minutes, recorded between 1955 and 1974.

Figure 13: Pearson’s Correlation Coefficient for a sample

A little later in 1991, I taught German at Sunderland Polytechnic, which became Sunderland University that year. Serendipitously, I could use Turbo Prolog there, which was the Borland version of Prolog, a declarative programming language. You code rules rather than procedures. For example, you write the grammatical rules in symbolic NLP. This, I did not know in 1991. Only four years later, though, I used Prolog to write the rules of German morphology and syntax in Prolog for the grammar checker project at the University of Manchester Institute of Science and Technology (UMIST), my second attempt at a doctoral dissertation project.

This is an excerpt from an early draft of a book chapter. The book will be about language teacher education and GenAI. I am posting these in smaller (mostly) self-contained posts. The posts are numbered consecutively. After they will have all come out, I will link them with each other.

The computer lab at the language department at Sunderland University with its 20 stand-alone PCs often stood empty. I replicated the textual analysis program, coding in Prolog, copied it onto the 20 computers, and ran an analysis on each of them – in parallel – one text of the 222 texts of the Freiburger Korpus after another. Long lists of trigrams and their probable connectedness per this one text were the result. Such analysis results were an insufficient basis for a linguistic analysis. My dream of completing a doctoral dissertation at the Universität Leipzig had been dashed; hence my second, successful attempt at a PhD dissertation at the Centre for Computational Linguistics at UMIST (Schulze, 2001). Later I learnt that the results can only make sense if one runs the analysis over a very large text corpus and not individual texts. Such an analysis could not have been performed by the computers of the language department; a single PC would have needed weeks, perhaps months, to analyze the entire Freiburger Korpus.

This was my only venturing into the statistical analysis of texts. My work for my PhD – a grammar checker research prototype – was rooted in symbolic NLP with a computational dictionary and grammar; all rules and entries were handwritten. This was common practice in NLP in the late 1990s.

References

Grundstrukturen: Freiburger Korpus. Basic Structures: Freiburg Corpus (FR). (2026, 2026–02–10).  Retrieved 2026-05-26 from https://agd.ids-mannheim.de/FR–_extern.shtml

Schulze, M. (2001). Textana – Grammar and Grammar Checking in Parser-Based CALL [PhD Thesis, UMIST]. Manchester.

Suen, C. Y. (1979). n-Gram Statistics for Natural Language Understanding and Text Processing In IEEE Transactions on Pattern Analysis and Machine Intelligence, PAMI-1(2) (pp. 164–172). https://doi.org/https://doi.org/10.1109/tpami.1979.4766902