Summary

This blog post explores languages inside the “Russian” part of DIGAR Collection. The metadata for newspapers issued before 1918 shows large number of Russian-language newspapers, however both the page and section-level data provide an evidence that the newspapers were not monolingual and some part of the collection labeled as Russian is in fact in German, Estonian and Latvian. The analysis provided aims to show how the history of Russification in Estonia is reflected in digitized newspaper pages and how metadata issues may probably influence OCR quality.

According to the metadata, a large part of the newspapers issued during the imperial period (before 1917) is marked as Russian-language ones. If counted in pages, there should be more than 120 thousands pages labeled as pages in “Russian” (Language column), with the following total distribution of pages over time:

After retrieving these pages’ texts, it is obvious that at least partly the title-level language metadata is wrong: e.g. the earliest “Russian” pages in the collection dated 1852 are mostly in German.

Language detection

To make a new rough language detection an R-package textcat was used. It takes a text as a string and calculates its average score to detect a language, so as a result a new metadata column will have one most probable language detected for each page.
A better solution may be found here, since many pages will probably have several languages combined at one page. However, this package is still applicable for a rough estimation of major languages proportion in these pages.

textcat language detection is not ideal, the full set of detected languages shows how low-quality OCR influences the results. Few pages were tagged with languages very unlikely used in Estonian newspapers, such as Nepali or Indonesian. The language distribution for the languages appearing in at least 150 pages is the following:

## 
##  estonian    german   latvian   russian ukrainian 
##       699     49328       128     74815      3102

However, even the “Ukrainian” pages should be excluded from the analysis, since these pages are in fact badly OCR-ed pages in Cyrillic script. Randomly selected chunks shows these pages are mostly in Russian, but being unable to check them all I filtered these pages as highly messy and unpredictable. This result is very disappointing showing not a language diversity but issues inside tools trained on major languages and giving an imperialistic assumption on Ukrainian being (on the character level) a messy Cyrillic which is not Russian (see an example below).

If we exclude the “Ukrainian” from consideration, this is the language distribution obtained with textcat language detection:

I believe this result is important, since it clearly shows that part of the metadata labels for “Russian” language used in 19-century newspapers should be reassessed: clearly, these are newspapers in German.
At the same time, I find this language switch from German to Russian in late 1880s very convincing, since the 1880s are known as a Russification period. Curiously enough, it happened very abruptly, so it might be interesting for further researchers to see if the content of newspapers switched from German to Russian changed significantly.

To add more details, the language distribution does not depend on place of publishing, which in overwhelming majority of cases are Riga or Tallinn (Revel):

OCR accuracy & languages

From a technical point of view, the languages (or scripts) hidden behind the “Russian” tag in metadata seems to be important for the OCR quality. If we are to plot OCR accuracy values for decades it may seem that there is a decrease in 1880-s, as one might suppose, led by Russification. This is, however, misleading, because the reason behind the decrease is different OCR accuracy for German and Russian (or Latin and Cyrillic scripts in general). The plot below displays how the OCR quality distributed according to the new language metadata; it is also visible that pages detected as “Ukrainian” have very low OCR quality.
To sum up, language reassessment on a more granular level (i.e. section) seems important and feasible to be added to respective metadata entries and then grouped to multilingual labels for the language data of entire newspaper titles.

Language detection based on sections data

Report update: 10/11/2022

The part above showed the difference between title-level language metadata and actual languages used in the governmental newspapers in the collection: while the newspapers’ language defined as “Russian”, digitized materials allow to conclude that newspapers used predominantly German as a language of communication until mid-1880s. The analysis was done on the page level, determining the proportion of languages used and labeling each page with most probable language. The results might be yet improved using as input data not the whole pages but newspaper segments or sections, where each section will most probably be written in only one language.
This update of the report aims to demonstrate results from the language detection made on the section level. However, as sections division is available at the moment only for the two newspapers, the results below cannot be directly compared to the analysis done on the full corpus (namely, the sections are available only for Estonian and Livonian governorate official newspapers, shortened in the dataset as ekmteataja and livzeitung respectively).
The languages of sections were detected with the textcat package as in the page-level analysis.
Ten most common languages for sections are mostly the same as the ones detected on the page level; the probability of error is higher for small sections (in particular, the ones that include less than 20 words).

Such unexpected (for Baltic newspapers) languages as English, Bulgarian and Rumantsch appeared to be detection errors for short sections with low OCR quality. Unfortunately, as well as in the case of page input, Ukrainian and Belarus languages are detection errors most probably caused by poor OCR and/or lack of training data for detecting these languages. The examples taken randomly for each of the false-detected language are shown below.

Nevertheless, textcat package output seems to be true for the sections labeled as Latvian or Estonian even for cases when several languages are mixed inside a section, e.g.:

Selecting only Russian, German, Estonian and Latvian sections, the distribution of languages in sections is the following:

Number of sections and words in each language:

As sections differ in length, there is a possibility that counting sections (previous plot) might not reflect the actual proportion of languages used in texts of different languages. This assumption can be tested using number of words in each section (LogicalSectionTextWordCount in the metadata). The main assumption is that one section was in most cases written in one language (thus the same method seems less applicable to pages). Resulting distribution is very similar to the section count showed above:

To compare the results with the page-level language detection, it is possible to filter only ekmteataja and livzeitung from the results gathered previously (stored in data/01_pages_meta_rus_lang.csv). The main shift from German to Russian and the proportion overall seem roughly same as in the case of sections.

However, the number of pages in German looks very different from the distribution of words in this language gathered from sections. This can be connected with the change of newspapers physical format from smaller to bigger pages and from bigger to smaller font sizes towards the end of the 19th century. Distribution of number of words on pages by decades proves the idea (as there are no physical description of pages in the metadata).

Section-based analysis thus showed more reliable results, reflecting not only the language use in the newspapers, but also some physical properties of the newspapers, such as page or font size changes over time. The plot shows how number of words per page changed in the second half of the 1860s and then remained more or less stable, so that higher number of pages in German in the 1850s and 1860s counter intuitively witness smaller amount of text per page.

Additionally, the OCR accuracy dependence on the language can be checked on the level of sections.

Cite the blog post:
Martynenko, Antonina 2022. Languages in DEA newspaper collection 1850-1918. Eesti Rahvusraamatukogu digilabori juhtumiuuringud. DOI 10.17605/OSF.IO/CPQ2W.

Data and code are available here: https://doi.org/10.17605/OSF.IO/CPQ2W.

The study has been made as part of the EKKD72 project "The usage possibilities of textual data in digital humanities case studies on the example of newspaper collections in Estonia (1850-2020)".