One way to digitise a dictionary is using Optical Character Recognition or OCR. But is OCR feasible at all for my dictionary? And if so, which OCR program should I used, trainable or omnifont? And how about the workflow: should I train the OCR engine or not? And, finally, what should be the output format of my OCR? For those wanting to take the OCR adventure, here a very brief introduction.

To OCR or not to OCR

Some texts are totally  un-ocr-able:

While some could be recognized, but the quality will be too bad to make it worthwile to even try:

Fortunately, there is also material which OCR, with good quality images, will handle quite well:

And there is – most annoyingly, because you feel it can be done but you are just doing it the wrong way – a lot in-between, where it your particular requirements will determine whether it makes sense to use OCR or not. Suppose you put your dictionary online by showing the images and using the underlying OCR for full text search, a less than perfect text capture might not be an issue. If however you would like extensive searching in an XML-encoded dictionary, you need good quality OCR, with post-correction.

Which OCR?

First, we list some of the available options:

• ABBYY FineReader (commercial, omnifont)
• Omnipage (commercial, omnifont)
• Tesseract (open source, trainable)
• Ocropus (open source, trainable)

There are two main types of OCR: the ‘omnifont’ type, which knows about many different typefaces, but is difficult to adapt to a new one, and the ‘trainable’ type, which can learn (but may be very hard to teach). If you have a script (for instance, Glagolitic) which is not supported by the omnifont engines, trainable OCR is the only option – but the process is far from obvious. We will not go into training OCR in this blog, but for the daring readers, here are some pointers.

Information on training Tesseract in the EMOP project  with the Franken+ tool can be found at: http://emop.tamu.edu/node/47, http://emop.tamu.edu/node/54 .  More information on the Franken+ tool  is at: http://dh-emopweb.tamu.edu/Franken+/. There is also a paper on this subject:  Early Modern OCR Project (eMOP) at Texas A&M University: Using Aletheia to Train Tesseract , http://dl.acm.org/citation.cfm?id=2494304

The difference between (limited) training with an omnifont engine and extensive font training with Tesseract is discussed in this case study: http://lib.psnc.pl/dlibra/doccontent?id=358.

As to omnifont OCR, let us take a look at the options that the market leader, ABBYY, offers:

Product	Description	Comment (by us)
ABBYY FineReader 12 Professional Edition (http://finereader.abbyy.com/professional/)	Creates editable, searchable files and e-books from scans of paper documents, PDFs and digital photographs	Desktop product, not suitable for digitization workflows
ABBYY FineReader 12 Corporate Edition (http://finereader.abbyy.com/corporate/)	Solution for streamlining document processing among workgroups in business, government and academic environments.	Desktop and small office product, not suitable for digitization workflows
ABBYY FineReader online (http://finereader.abbyyonline.com/en)	Online web client for FR.	Does not support batch processing. Cheapest online option.
ABBYY Cloud OCR SDK:  http://www.abbyy-developers.eu/en:onlineocrsdk:start ; pricing: http://ocrsdk.com/plans-and-pricing/ , api documentation at: http://ocrsdk.com/documentation/; examples for using the Cloud OCR SDK at https://github.com/abbyysdk/ocrsdk.com	Cloud-based online OCR service. The difference between this and the Engine SDK is that this has a web API instead of a local library interface.	Should be relatively easy to fit into any workflow system working with web service API’s. Many parameters can be set, but control is less than in in the engine (e.g. external dictionaries or pattern training during recognition are not possible).
ABBYY Recognition Server  (http://www.abbyy.com/recognition_server/)	Automated OCR server and document capture software. Server-based OCR solution designed for mid- to high-volume document processing tasks.	Manages aspects like distributed processing and job scheduling.
ABBYY FineReader Engine 11 for Windows  (http://www.abbyy.com/ocr_sdk_windows/)	Software Development Kit (SDK) for integrating ABBYY’s multilingual recognition and conversion technologies into external applications.	Allows control over all that can be controlled, but requires highly technical implementation work.

Table 1:  ABBYY finereader products

Here are a few brief hints to help you make the right choice:

1. If you want the XML (either ALTO or abby XML) with detailed information about the coordinates on the page of each word and character, the desktop products will not do
2. The cloud OCR SDK is very good to test the capabilities of conventional omnifont OCR on your documents (we have used it a lot in the Lisbon summer school).
3. You need a developer to use the SDK versions. SDK versions are useful if you would like to add a large dictionary and control all possible parameters with which you can tune the engine.

What output format to OCR to?

Output formats are: RTF, DOCX, XLS/XLSX, PPTX, PDF, PDF/A, HTML, TXT/CSV,(finereader)  XML, ALTO (xml), FB2 (feedbook, an ebook format), EPUB (ebook format),  ODT (open office document format).

If you would like to get the maximum information out of the engine, the XML-based solutions are the ones to choose. They allow flexible handling of the recognized text, and in particular, store information about the location of text regions, lines, words, and (possibly) characters in the source image. If you would like to highlight a search result in the image of your dictionary, you need this option. It is also very useful for structural analysis of your dictionary.
XML export is not possible in the two desktop editions of Finereader, nor in the online OCR client. The recognition server, the cloud API and the Engine SDK do support this option. There are two supported XML output schemata:

1. ABBYY xml: this is an XML format defined by the company (cf. for instance http://www.abbyy-developers.com/en:tech:features:xml). This format allows the highest degree of control of the output, with options to store detailed glyph properties and alternative recognitions of words and characters.
2. ALTO xml: this is a widespread standard for optical character recognition results, cf. http://www.loc.gov/standards/alto/.  ALTO export is possible in recent releases of the SDK and the extended recognition server, cf http://www.abbyy-developers.eu/en:tech:features:alto. It is currently not possible to export glyph coordinates in this format.

If alternative formats are required for your digitization workflow, there are two main options:

1. Conversion of ALTO or ABBYY XML to your desired format. Cf for instance http://able.myspecies.info/abbyy-xml-tei-xml .
2. Implementation of an SDK application which directly exports the recognition result. As an example, we can mention the PAGE xml exporter developed in the IMPACT project.

Examples

How well does OCR work? To conclude, some examples, coming from the participants of the COST Training School on digitisation of dictionaries (cf. http://www.elexicography.eu/events/training-schools/lisbon-2015/). The screenshots are from the Transkribus transcription platform (http://www.transkribus.eu). The OCR has been performed by the finereader SDK, either offline, or from the Transkribus tool which has this option.

 

A quick run of some pages of a dictionary can give you a first impression of the quality of OCR. For thorough evaluation, ground-truth evaluation material should be produced, i.e. a correctly transcribed sample of the dictionary you want to OCR. How to do that would be a topic for another blog post.

---

---