TEIhub
Discover TEI-encoded documents from GitHub public repositories.

Languages

In an effort to make it easier to locate TEI-encoded texts in a particular language, I have recorded the ISO 639 language codes for each repository where languages were clearly specified in TEI files.

Each time a repository appears in a batch of 1000 GitHub search results, I download the first .xml file for that repository that appears in the current batch of search results and check whether a language is specified in any of the following ways, in order:

  1. <langUsage><language id(ent)="xx(x)">...</language></langUsage>
  2. <text xml:lang="xx(x)">
  3. <body xml:lang="xx(x)">
  4. <textLang mainLang="xx(x)">. While this technically applies to the language of the manuscript, not necessarily of the TEI file, it was often used in files where only a facsimile was provided in the body.
  5. <TEI xml:lang="xx(x)">, but only where the language code is not en, as I found this did not reliably match the language used in the rest of the file.
  6. xml:lang on any element within body, excluding <foreign> tags

Based on a sample of 500 repositories, approximately 40% declare a language in one of these ways. If the same repository appears in a different batch of search results, I apply the same algorithm to the first .xml file found in that search, and append any new languages to the end of the list. Repositories/files/languages are not currently removed from this database even if the underlying files are changed or removed from GitHub.

The TEI Guidelines recommend the use of BCP 47 language tags. In their fullest version such tags allow you to encode language, region, script, and certain other information. For simplicity I have only recorded the language subtag, and discarded the other information. BCP 47 prefers the shortest available form of ISO 639 language codes for the language subtag (e.g. de instead of deu for German), however this means some languages have a two-character code, and some a three-character code. For consistency, and to avoid privileging “more common” languages, I have normalised all two-character language codes to use their three-character ISO 639-2 equivalent, choosing the Terminology variant where applicable (e.g. de becomes deu not ger).

I have not provided filtering of the search results by language, as this becomes too complicated to implement with a large data set in the context of a static website. Data files in CSV and JSON formats of all the results are provided on the home page if you need to do more advanced filtering.

Language Code Matching Repositories
eng 21888
lat 881
fra 689
deu 400
ita 280
grc 224
spa 163
ara 120
nld 118
heb 77
san 65
cym 61
rus 61
slv 60
ell 60
dan 48
zho 46
pol 44
fro 39
por 33
fas 32
syr 26
srp 24
ces 23
frm 23
isl 22
cop 21
non 20
tur 20
sco 18
hun 18
jpn 18
lit 17
pli 16
gez 16
sqi 15
kor 15
swe 15
ron 14
chu 14
bul 14
ota 13
fin 13
hrv 13
gmh 12
srt 12
ava 11
zxx 11
tam 11
bod 11
nor 10
kat 10
und 9
yid 7
urd 7
slk 7
amh 7
srd 7
cat 7
arc 7
mix 7
sh 7
ukr 7
ang 7
tgl 6
sux 6
mkd 6
gle 6
afr 6
bre 6
hye 6
enm 6
est 6
sme 5
vie 5
x-lap 5
gla 5
hin 5
oci 4
tsn 4
grk 4
epo 4
tel 4
jrb 4
egy 4
sin 4
lng 4
mlg 4
mul 4
phn 4
swh 3
pra 3
gsw 3
tat 3
kaw 3
x-oldsun 3
gda 3
xno 3
lad 3
ber 3
mya 3
nah 3
akk 3
xml 3
nob 3
tha 3
ceb 3
mnc 3
glg 3
mal 2
nxq 2
nep 2
pan 2
lao 2
ind 2
dzo 2
kan 2
css 2
tir 2
x-sa 2
swa 2
nau 2
mri 2
lav 2
wln 2
msa 2
glv 2
bar 2
mon 2
nds 2
mig 2
miy 2
miz 2
smd 2
xct 2
arn 2
unk 2
pam 2
quc 2
gml 2
pag 2
jpr 2
dum 2
ilo 2
nym 2
lea 2
nai 2
eus 2
mag 1
esx 1
mhd 1
sog 1
mar 1
uig 1
skt 1
mlt 1
olt 1
x-unknown 1
hbs 1
che 1
mwr 1
ing 1
myn 1
fry 1
pro 1
nno 1
asm 1
bra 1
ibe 1
x-oldkhmer 1
x-middlekhmer 1
roh 1
ndl 1
ags 1
bis 1
x-oldcham 1
inm 1
rue 1
haw 1
gad 1
ben 1
sot 1
brx 1
pit 1
pie 1
ine 1
bik 1
arb 1
emg 1
tah 1
ido 1
win 1
xho 1
xda 1
tru 1
yua 1
nil 1
cha 1
oss 1
cor 1
new 1
zul 1
awa 1
aeb 1
pka 1
ajp 1
ajw 1
kar 1
loz 1
mck 1
bnt 1
lun 1
hau 1
aka 1
x-oldcam 1
dak 1
apa 1
nbo 1
ofs 1
osx 1
goh 1
chr 1
osc 1
xly 1
scx 1
sxc 1
xpu 1
bak 1