Tetrak OCR
GitHub ↗
Method

Transliterate for display, fold for finding

Armenia's national poet appears on book covers as Tumanyan, Tumanean, Toumanyan and Tumanian, and all four are correct. A fold is how one search finds them all, and explaining it shows why a transliteration is for the reader and never for the index

14 min read Stephen Masters

Hovhannes Tumanyan is about as famous as an Armenian writer gets. He is Armenia’s national poet, and his fairy tales and verse for children are read to almost every Armenian child, in Armenia and across the diaspora. His name is on a great many book covers. Look through the Armenian Institute’s catalogue and you find it spelled at least four ways:

None of these is a mistake. Each is an honest romanisation of one of the two ways Armenian spells the name: Թումանյան in the reformed spelling used in Armenia, and Թումանեան in the classical spelling the diaspora kept.

Four readerships, four right answers. But anything that matches spellings exactly, whether a catalogue search or a reader scanning a shelf list, finds only the books catalogued under the spelling it was given. The problem isn’t that the name was romanised wrongly. It’s that it was romanised correctly, several times.

Finding all four with one search is what a fold is for. And the word fold turns up all over Tetrak. The recogniser’s output goes through a script fold. The accuracy metric folds before it compares. The trainer folds transcribers’ habits out of its ground truth. And tetrak-translit, our transliteration library, has a function called fold() that most of the library exists to support. These are all the same idea, so it’s worth explaining, because the transliteration use only makes sense once the first three do. It also answers a question that comes up as soon as you look at the library: if you have five perfectly good romanisation schemes, why do you need anything else to search with?

What a fold is

A fold is a function that maps multiple inputs to a single output, so that things which differ in ways you don’t care about compare as equal.

You have almost certainly used one. Case folding is the familiar example: "Straße".casefold() in Python gives "strasse", so a comparison of folded strings ignores case, including the cases where upper and lower case don’t map one to one. Search engines fold accents: Elasticsearch’s asciifolding filter and Solr’s equivalent turn café into cafe, so a reader without an é on their keyboard still finds the café. Unicode defines case folding formally, in a table of its own, because “make these compare equal” turns out to need more care than “make these lowercase”.

Three properties make something a fold rather than just a transformation:

That third property is the one that causes trouble, and below, you will see how it caught us out in our tests and what we learned.

Three folds Tetrak already runs

On the way out of the recogniser

Some Armenian letters have Latin twins: հ and h, the Armenian full stop ։ and the colon. At the level of ink on paper there is nothing to tell them apart, and our recogniser reads one character at a time, with no language model to say that a Latin letter is unlikely in the middle of an Armenian word. So sometimes it emits the twin.

The script fold, fold_script() in the model’s package, fixes that. Inside any word that already contains an Armenian letter, it folds each Latin twin back to its Armenian form. A wholly Latin word, such as a name or a citation, is left alone. This fold is unusual in that its output is what the reader sees, because inside an Armenian word the Latin twin is never right. The claim it makes (“inside an Armenian word, h means հ”) is true every time.

The fold’s first draft made a second claim that wasn’t true. It also folded the en dash onto the hyphen, which looked like a free correction. It turned out to be grammar: Armenian joins case endings to numerals and abbreviations with a dash, and the transcribers use an en dash for many of those. The fold broke more words than it fixed, and we took it out. We wrote that up at the time, and it’s the first lesson about folds: a difference that looks like noise can be a meaning you haven’t met yet.

Inside the accuracy metric

When we score an engine’s reading against a reference transcript, both sides are folded first. The fold lowercases, collapses whitespace, and equates two Armenian marks with the ASCII characters they print identically to: the full stop ։ with the colon, and the abbreviation dot ․ with the full stop. Since tetrak 5.15.0 it also equates the two-letter եւ with the ligature և.

Each of those is a claim that the difference doesn’t matter for scoring. Transcribers type the colon because it’s on every keyboard. Classical presses split on whether to print եւ or և, as we found when we told the two orthographies apart. An engine that outputs one form where the transcript has the other hasn’t misread anything, and a metric that marked it wrong would be ranking engines on which habits they share with a transcriber.

Note where the fold lives. The reference transcripts still say exactly what the page prints, and we train on them as printed, so the model learns to read both forms. The fold only exists inside the comparison. That’s the second lesson: fold at the point of comparison, and keep the original everywhere else.

On both sides of a check

The trainer cleans every harvested transcript before using it: it folds transcribers’ stand-ins (a typographic apostrophe for the Armenian apostrophe ՚, a backtick for the Armenian comma ՝) onto the characters the page prints. When we built the check that stops evaluation text from leaking into training data, it had to read the harvest through exactly that fold, with the training pipeline’s own code, because any difference between how the check reads a page and how the trainer reads it is a place for a duplicate to hide. We wrote about that check last week. The third lesson: two things being compared must go through the same fold, or the comparison is measuring the folds.

Transliteration is for display

Now the fourth fold, which needs a quick detour first.

A transliteration renders a script in another script for a reader. For Armenian there are several standards, and they disagree, because each serves a different reader. Here is what tetrak-translit’s five Armenian schemes make of three names, one of them in both spellings:

ArmenianALA-LCISO 9985BGN/PCGNHübschmann-MeilletWestern
ԵրևանErevanErewanYerevanErewanYerevan
Պողոսյան (reformed)PoghosyanPoġosyanPoghosyanPołosyanBoghosyan
Պօղոսեան (classical)PōghoseanPòġoseanPoghoseanPōłoseanBoghosian
ՀակոբHakobHakobHakobHakobHagop

None of these is wrong:

A transliteration should therefore be faithful to its standard and keep every distinction its reader expects. A catalogue record romanised under ALA-LC should look like ALA-LC. That’s exactly why a transliteration is the wrong tool for finding things.

Why you can’t search with a transliteration

Picture a catalogue built up over decades. Some records are in Armenian script, in classical spelling or reformed. Some were romanised by librarians under ALA-LC, and older ones under whatever the library used before. Records harvested from elsewhere use other schemes. Then a reader arrives and types Boghosian, because that’s how their grandmother spelled it.

Exact matching finds almost nothing. The obvious fix, picking one scheme and converting everything into it, doesn’t work either:

What search needs isn’t a better transliteration. It needs a fold: a key coarse enough that Boghosian, Poghosyan, Պողոսյան and the classical Պօղոսեան all land in the same place.

How the Armenian fold works

tetrak-translit’s fold() maps any spelling of an Armenian word, in either script, onto one lowercase Latin skeleton. It works in two stages.

Armenian text is first normalised across the spelling reform, towards the reformed spelling: եւ becomes և, է becomes ե, the classical digraphs եա and իւ become յա and յու, and a silent final յ is dropped. The result is then rendered through ALA-LC, so that Armenian input and catalogue Latin arrive at the next stage by the same road.

Latin is then reduced to a skeleton. Diacritics and aspiration marks are stripped. Each sound’s competing spellings are collapsed onto one (ž and zh; x and kh; ł, ġ and gh; ow, ou and u; w and v). The voiced and voiceless pairs are merged (b with p, d with t, g with k), because that’s exactly where Eastern and Western spellings disagree. The glides are regularised, so Yerevan and Erevan meet, and so do -yan, -ean and -ian. Doubled letters are collapsed, and anything that isn’t a letter or a digit goes.

import tetrak_translit as tt

{tt.fold(s, script="hy") for s in
 ["Երևան", "Երեւան", "Erevan", "Erewan", "Yerevan"]}
# {'erevan'}

{tt.fold(s, script="hy") for s in
 ["Պողոսյան", "Պօղոսեան", "Poghosyan", "Boghosian"]}
# {'poghosian'}

{tt.fold(s, script="hy") for s in
 ["Հակոբ", "Յակոբ", "Hakob", "Yakob", "Hagop"]}
# {'hakop'}

Look at the keys: poghosian, hakop. Nobody spells the names that way, but that’s okay because nobody is meant to read them. They’re the coarse middle point that every spelling folds down to, and the merges that get there would be errors in a transliteration. Merging b with p would be wrong in a catalogue record. In a search key, it’s how Bedros finds Petros.

And Tumanyan’s four covers, plus a fifth:

{tt.fold(s, script="hy") for s in
 ["Թումանյան", "Թումանեան", "Tumanyan", "Tumanean",
  "Toumanyan", "Tumanian", "Thoumanian"]}
# {'tumanian'}

{tt.fold(s, script="hy") for s in
 ["Հովհաննես Թումանյան", "Յովհաննէս Թումանեան", "Hovhannes Tumanyan",
  "Hovannes Toumanian", "Ovanes Tumanyan"]}
# {'ovanes tumanian'}

Two spellings of the Armenian and five of the Latin, one key. The full name works too, including the shortened forms of his first name: Hovannes, which drops the h after the v, and the Russian Ovanes, which drops the initial h because Russian has none.

Those last three didn’t work when we first drafted this article. The French Thoumanian uses th for the aspirated Թ, and the fold didn’t know that convention. Hovannes and Ovanes each lost a letter the fold was holding on to. How we fixed them is told under “What a fold should not do”, below.

Index time and query time

A fold is used twice, and both uses must be the same function. At index time, every record’s name and title fields are folded and the keys stored next to the originals. At query time, the reader’s search is folded the same way and matched against the keys. The reader sees the record as it was catalogued, in whatever script and scheme it was written. The key never appears on screen. This is the same shape as accent folding in Elasticsearch or Solr, and fold() is meant to be called from that kind of analysis chain.

Each script makes its own claims

Because every fold is a claim about what doesn’t matter, a fold can’t be shared between scripts that disagree about what matters. Our Georgian fold also strips apostrophes, merging the aspirated and ejective consonants however each system marked them. But it must never merge b with p. In Georgian, ბ and პ are different letters that don’t swap the way Armenian’s do. Batumi folded under Armenian rules would never find ბათუმი.

So a Latin string can’t be folded until you say which script it’s meant to be in:

tt.fold("Tbilisi")
# ScriptRequiredError: fold() needs to know which script's rules to
# apply to Latin text: pass script='hy' (Armenian) or script='ka' (Georgian)

tt.fold("Tbilisi", script="ka") == tt.fold("თბილისი", script="ka")
# True

Text already in Armenian or Georgian script says which rules apply by itself. Latin text doesn’t, and the library refuses to guess rather than default to one.

What a fold should not do

A coarser fold finds more, and it also merges more names that aren’t the same. There’s no setting that is simply correct, so the fold’s claims are written down as tests. One published file lists groups of spellings that must fold together: cities, surnames, saints and poets, in both scripts, both orthographies and every scheme. A second test checks that different names stay different. If a proposed rule would merge two names that a reader would want kept apart, the second test is where we find out.

Fixing Tumanyan’s spellings

The three spellings that missed Tumanyan’s key show how that works in practice. We added them to the test file first, as a new group and two new forms in an old one, and watched the tests fail. Then we looked for the smallest rule that would pass each one.

Thoumanian was easy. Older English and French spellings mark an aspirate with an h: th for Թ, ph for Փ. The schemes use an apostrophe or nothing. So th folds to t and ph to p, and no existing key changed.

Ovanes was harder, and it’s the best example of the trade-off in this article. Russian has no h, so Russian-mediated spellings drop Armenian’s initial Հ: Ovanes, Ovsep, but also Akop for Hakob and Arutyunyan for Harutyunyan. The obvious rule drops every initial h, and it catches all of them. It also changes the key of almost every name beginning with h, and the over-merge check turned up a real collision: the classical իւր (his) would fold onto ուր (where). So the rule we shipped is narrower. It drops an initial h only before an o, which catches Ovanes and Ovsep and nothing else.

Even the narrow rule changed one existing key, Hovhannes’s, which means any index that stored the old key has to be rebuilt. A fold’s keys are data that other systems keep. So that change was released as tetrak-translit 1.0.0, a breaking change, with the old and new keys in the release notes.

The fold also doesn’t claim to be complete, and its gaps are tests too, asserted as failing:

If a change to the rules closes one of these, its test moves into the published set. The right way to extend a fold is to add the cases first and then find the rule that passes them, which is the habit we built when the dash fold went wrong.

The pattern

Four folds, one idea. Put together, the lessons are short: