አቡጊዳ...? Completion of the abugida sequence

I've long been curious about the structure of "a-bu-gi-da".  What comes after the first four elements of the sequence? Finally today, I worked out an answer:

አቡጊዳሄዝቶየኩሊማኔስዖፐሩሢጣዬፍኆፀቹዊሓሼቅኞጀጹጲ

Since I'm just having fun, and I am lazy, I didn't do any real research. I just tried to construct it somewhat logically. But it's probably wrong. So, apologies to real scholars out there. If there's an authoritative answer, maybe one of you will see this post and tell me. As they say, the fastest way to get an expert answer to a question is to post a wrong answer on the Internet! While we wait, let me explain my construction.  

Background

Abugida is the name used for alpha-syllabaries, like the Ethiopic (aka Ge'ez) script, where each character is a syllable equivalent to a vowel + a consonant in the roman alphabet. In fact, the term comes from ge'ez. But it is now used as a general term for alpha-syllabaries including some Indic ones.  Now normally, the Ethiopic character set is organized in a matrix where the columns correspond to the vowel form and the rows to the consonant, like this:

Image courtesy of this paper

As you can see the sequence of rows (consonants) here is not a-b-g-d-.... It is h-l-h-m-s-r-... So where does a-bu-gi-da come from?  Notice it sounds a lot like: alpha, beta, gamma, delta, ... 

A Greek connection

So it is related to the Greek alphabet! 

Note that I say "related" and not "comes from", which may surprise Westerners who tend to think of ancient Greece as the origin of a lot of stuff. But Greek letters descended from the Phoenician alphabet, and Phoenician is Semitic, as is Ethiopic. So the relation may go back to an earlier node in the ancestor tree, or could be a more recent adoption. 

Another interesting connection is that Ethiopic numerals from 1 to 6:  ፩, ፪, ፫, ፬, ፭, ፮, ... look a lot like the first 6 Greek letters α, β, Γ, δ, ε, ζ, ... (lowercase except gamma being uppercase). I should note that Professor Hailu Habtu in a recent book entitled "Aksum: A glimpse into an African Civilization" (excellent book by the way) disagrees with this connection. But I digress.

The Diagonal

The a-bu-gi-da consonant order is the same as the first four letters of the Greek alphabet. But notice also the vowels correspond to the first four vowel forms of the Ethiopic alpha-syllabary. So if we imagine a different matrix, with the rows matching consonants in the Greek order, and columns matching the vowels in the Ethiopic order, then we are basically going down a diagonal. Cute!

I am fond of diagonals. Like Cantor's diagonal argument that the real numbers are not countable. A classic way to introduce the difference between countable and uncountable infinity. Everyone should learn that! And how about eigenvalue decomposition... But I digress, again.

So if we follow the diagonal, wrapping around when we run out of columns, i.e. diagonal with column index being modulo 7, we can continue the a-bu-gi-da sequence. Here's the sequence with explanations. 

አ (a) – ሀ (h), 1st vowel /ä/ (Greek: Α, alpha)

ቡ (bu) – በ (b), 2nd vowel /u/ (Greek: Β, beta)

ጊ (gi) – ገ (g), 3rd vowel /i/ (Greek: Γ, gamma)

ዳ (da) – ደ (d), 4th vowel /a/ (Greek: Δ, delta)

ሄ (he) – ሀ (h), 5th vowel /e/ (Greek: Η, eta)

ዝ (zə) – ዘ (z), 6th vowel /ə/ (Greek: Ζ, zeta)

ቶ (to) – ተ (t), 7th vowel /o/ (Greek: Θ, theta)

የ (yä) – የ (y), 1st vowel /ä/ (Greek: Ι, iota)

ኩ (ku) – ከ (k), 2nd vowel /u/ (Greek: Κ, kappa)

ሊ (li) – ለ (l), 3rd vowel /i/ (Greek: Λ, lambda)

ማ (ma) – መ (m), 4th vowel /a/ (Greek: Μ, mu)

ኔ (ne) – ነ (n), 5th vowel /e/ (Greek: Ν, nu)

ስ (sə) – ሰ (s), 6th vowel /ə/ (Greek: Ξ, xi)

ዖ (‘o) – ዐ (‘), 7th vowel /o/ (Greek: Ο, omicron)

ፐ (pä) – ፐ (p), 1st vowel /ä/ (Greek: Π, pi)

ሩ (ru) – ረ (r), 2nd vowel /u/ (Greek: Ρ, rho)

ሢ (śi) – ሠ (ś), 3rd vowel /i/ (Greek: Σ, sigma)

ጣ (ṭa) – ጠ (ṭ), 4th vowel /a/ (Greek: Τ, tau)

ዬ (ye) – የ (y), 5th vowel /e/ (Greek: Υ, upsilon)

ፍ (fə) – ፈ (f), 6th vowel /ə/ (Greek: Φ, phi)

ኆ (ḫo) – ኀ (ḫ), 7th vowel /o/ (Greek: Χ, chi)

ፀ (ṣä) – ፀ (ṣ), 1st vowel /ä/ (Greek: Ψ, psi)

ቹ (ču) – ቸ (č), 2nd vowel /u/ (Greek: Ω, omega, next Ge'ez consonant)

ዊ (wi) – ወ (w), 3rd vowel /i/ (no Greek equivalent, next Ge'ez consonant)

ሓ (ḥa) – ሐ (ḥ), 4th vowel /a/ (next Ge'ez consonant)

ሼ (še) – ሸ (š), 5th vowel /e/ (next Ge'ez consonant)

ቅ (qə) – ቀ (q), 6th vowel /ə/ (next Ge'ez consonant)

ኞ (ño) – ኘ (ñ), 7th vowel /o/ (next Ge'ez consonant)

ጀ (ǧä) – ጀ (ǧ), 1st vowel /ä/ (next Ge'ez consonant)

ጹ (ṣu) – ጸ (ṣ), 2nd vowel /u/ (next Ge'ez consonant)

ጲ (p̣i) – ጰ (p̣), 3rd vowel /i/ (next Ge'ez consonant)

To generate this I used the Grok large language model. After all, LLMs are language sequence completers, how appropriate! It took a bit of prompting to get to this. If you are interested, here's the whole conversation with Grok.

Actually this whole exercise was as much to see if an LLM could do this as it was about the subject itself.   

Oh by the way, if you want serious content on Ethiopic, check out  geez.org.

"Attention" and "Transformers" in Large Language Models

Everyone is talking about OpenAI's ChatGPT these days. Here's a very quick attempt to summarize the core idea behind large language models (LLMs) like GPT.

"Attention is all you need" (aka the transformer paper) published in 2017 by Vaswani et al from Google is still the mother of current LLMs, including GPT.  "Effective Approaches to Attention-based Neural Machine Translation", an earlier paper by Luong et al from Stanford, was also quite important.

These are sequence-to-sequence models, i.e. their job is mapping an input sequence of text into an output sequence of text. Applications include translation from one language to another, answering questions,  having a conversation, etc.

They use language embeddings (made famous by Word2vec in 2013 and later by BERT, both also from Google) as the basic encoding/decoding building blocks, i.e. mapping text to vectors of real numbers in an "embedding space".

The main new idea is in the architecture of the neural network between the input encoding and output decoding stages. The model uses the preceding terms in the current output sequence to decide which parts of the input sequence to pay more "attention" to for the next output term. A bit more precisely: the previous output is a "query" which gets used to generate a linear combination of "keys" from the input which maps to a linear combination of "values" also from the input. That in turn gets transformed into the next output term with a few more layers in a plain feed forward network (i.e. a bunch of layers of neurons, where each neuron is putting a linear combination of inputs into non-linear activation function). Each step has trainable weights.

There are also clever tricks besides "attention". One is positional encoding to represent the order so the same input term in a different position has different effects even though, unlike in recurrent neural networks,  in transformers the network just sees them as bag of words that could be in any order. Another is layer normalization to sort of keep the nonlinear outputs within a reasonable area in the embedding vector space.

This architecture, as far as I know, was not derived explicitly from the way human brains work. The "attention" analogy is really useful, but there are no principles saying this architecture is more fundamental to intelligence, or more natural, than many others. It just happens to produce remarkably good results when the weights are trained properly.

So that's the basic idea of contemporary LLMs. Of course in some sense, all computer neural networks are  just a bunch of matrix multiplications and ad-hoc activation functions. But you can't just connect a large number of mathematical "neurons" randomly in a network and hope it learns something.  The choice of architecture, i.e. how the "neurons" are connected, is key. On top of that, there is still an enormous amount of innovation/engineering to make the real world language models, not to mention turn them into a product like ChatGPT or Google Bard.