Digitising 150 Years of the Swiss German Dictionary

The scholars of the Swiss German Dictionary (Schweizerisches Idiotikon) have collected more than 15000 pages of highly concentrated information over the last 150 years. When we began retro-digitising the dictionary a few years ago, we were unsure if we were up to the task of dealing with such a massive amount of data. Of course, it was not a hardware problem – the 100 million characters will easily fit into any storage device sold nowadays. The real challenge is how to translate a 19^th-century information structure into one which today’s computers can handle.

One of the major concerns of dictionary makers before the electronic age was how to save space. Dictionary makers achieved this by using abbreviations, using typography (styles and special characters) to mark different parts of an article and using context to convey information. This specifically means that certain elements can either be omitted depending on the context or bear a different meaning. 

The following scanned example of two very small and simple Idiotikon articles (Sūrstängel and Süessstängel – a sour and a sweet kind of a stalk in volume 11, column 1116) may illustrate this:

You may have noticed that the headwords are not given in full – they are both compounds and their head noun (Stängel ‘stalk, stem’, introduced on a much earlier page) is omitted here. You can also see abbreviations like BGr. for the place ‘Berne – Grindelwald’ that are documented elsewhere and others that seemed sufficiently clear without further documentation like Rumex acet. for the plant ‘Rumex acetosa’ (or ‘Rumex acetosella’). You also have typographical features like the spaced headwords, the smaller font for the comments part or typographical subtleties like the additional dash in the synonym reference Sūrchrūt-Stängel to indicate where you will have to look up this synonym (i.e. under chrūt-Stängel).

These characteristics make the Idiotikon a very challenging type of work for transformation into machine-readable formats such as XML/TEI. The situation is not much better for OCR as there are many special characters, diacritics, superscripts, a language mixture (meta-language, object languages, abbreviations) and subtle typographical differences. Manual interventions would be necessary in order to achieve an optimal result. With the size of the dictionary in mind, these activities would have to be planned extremely carefully.

The mammoth task of retro-digitisation of our dictionary is an ongoing and incremental process. We began with scanning the dictionary and together with an already existing electronic version of the alphabetical index, we brought these scans online. You could then search in the index and were linked to a scan of the corresponding dictionary page.

As a next step, we ran the scans through an OCR engine – but the results were not very convincing, which was not surprising. For example, the extract from above looks as follows:

Sûr»: Sauerampfer, Rumex acet. BGr. (Bärnd.1908);
Syn. Sür-chrüt-St.
S ü e ß - : = Haber-St. a GRJem — Vgl. Fischer VI 3263.

Due to a lack of resources, we could not proof-read the OCR output. We simply used the OCR output to enable full-text search. While it would have been preferable to have the fully corrected text, the new full-text search, despite the deficiency caused by less than ideal OCR, provided possibilities that did not exist before. You could now, for instance, search for the Standard German plant name Sauerampfer ‘sorrel’ and get this result – even with a highlighted search word:

We continued with the addition of easily recognisable dictionary-internal hyper-links, a partially manual, partially automatic positioning of the headwords on the scanned page (in order to highlight them for a more user-friendly search experience) and the manual collection of all meaning indications (still ongoing). Points for optimisation are systematically observed and improved pragmatically. We try to keep these digitisation steps as modular as possible so that we can still use them even if perfect OCR becomes a reality one day.

From our experience, the strategy of incremental digitisation can be a very good solution for a big and complex retro-digitisation project with limited resources. Our approach of publishing new features early is in line with methods of agile software development (“release early, release often”), as well as our own dictionary publishing tradition (a subscription system with an output of around 100 pages a year).

Featured image by Nick Hunger / idiotikon.ch

---

---