"Attention" and "Transformers" in Large Language Models

Everyone is talking about OpenAI's ChatGPT these days. Here's a very quick attempt to summarize the core idea behind large language models (LLMs) like GPT.

"Attention is all you need" (aka the transformer paper) published in 2017 by Vaswani et al from Google is still the mother of current LLMs, including GPT.  "Effective Approaches to Attention-based Neural Machine Translation", an earlier paper by Luong et al from Stanford, was also quite important.

These are sequence-to-sequence models, i.e. their job is mapping an input sequence of text into an output sequence of text. Applications include translation from one language to another, answering questions,  having a conversation, etc.

They use language embeddings (made famous by Word2vec in 2013 and later by BERT, both also from Google) as the basic encoding/decoding building blocks, i.e. mapping text to vectors of real numbers in an "embedding space".

The main new idea is in the architecture of the neural network between the input encoding and output decoding stages. The model uses the preceding terms in the current output sequence to decide which parts of the input sequence to pay more "attention" to for the next output term. A bit more precisely: the previous output is a "query" which gets used to generate a linear combination of "keys" from the input which maps to a linear combination of "values" also from the input. That in turn gets transformed into the next output term with a few more layers in a plain feed forward network (i.e. a bunch of layers of neurons, where each neuron is putting a linear combination of inputs into non-linear activation function). Each step has trainable weights.

There are also clever tricks besides "attention". One is positional encoding to represent the order so the same input term in a different position has different effects even though, unlike in recurrent neural networks,  in transformers the network just sees them as bag of words that could be in any order. Another is layer normalization to sort of keep the nonlinear outputs within a reasonable area in the embedding vector space.

This architecture, as far as I know, was not derived explicitly from the way human brains work. The "attention" analogy is really useful, but there are no principles saying this architecture is more fundamental to intelligence, or more natural, than many others. It just happens to produce remarkably good results when the weights are trained properly.

So that's the basic idea of contemporary LLMs. Of course in some sense, all computer neural networks are  just a bunch of matrix multiplications and ad-hoc activation functions. But you can't just connect a large number of mathematical "neurons" randomly in a network and hope it learns something.  The choice of architecture, i.e. how the "neurons" are connected, is key. On top of that, there is still an enormous amount of innovation/engineering to make the real world language models, not to mention turn them into a product like ChatGPT or Google Bard.

Five books: Pandemic Pentateuch

During the last 12 months,  I read quite a few books that I really liked.  I doubt my overall reading volume increased during the pandemic (more time at home, but also less time in trains and planes), but for some reason, the past year yielded a memorable crop of books. Here are five of them: 

• Apocalypse Never, an interesting and timely book, with solid coverage of the fundamentals of climate change. Besides being full of information, as I wrote in a previous blog post, it helps you think more clearly about what is good for the environment.
• Apollo's Arrow  a very good book about the topic of the year: Coronavirus.  It covers all dimensions and is very educational on how to understand the pandemic in terms of medicine, epidemiology, sociology, evolutionary biology, public policy, history, etc. Lots of interesting details about how the pandemic unfolded in different places from  Wuhan to New York City.   It also helps to understand how things might evolve going forward.  For example, are future mutations of SARS-CoV-2  likely to be more lethal or less lethal? I won't tell you the answer because you should really read this book!
• The Shadow King: "Fiction is a shadow of real life, great fiction is Truth! Furious, illuminating, warm, fantastic, can't say enough about this book. Highly recommended",  I  tweeted.  The author is a childhood family friend, and I was really happy to see it short-listed for the Booker Prize. But I can honestly say that was not at all on my mind while engrossed in the story.  
• The Plot to Kill Graziani. "Deeply researched, meticulously sourced, highly readable account", was my brief review on goodreads.   This real life historical thriller would be fascinating to anyone interested in Ethiopia, Fascism, etc.  As a personal bonus,  I was able to use this book to fill in  a few specific details in my family tree. Interestingly I read both Shadow King and this book before the current war in Ethiopia started in Nov 2020.  Having the 1930s fresh on our minds is helpful  perspective on the current crisis. In the darkest hours, it helps to remember two thousand years of history, could not be wiped away so easily.
• The Monk of Mokha, a real-life story of a guy who decides to make Yemeni coffee "great again" (my silly choice of phrase).  Nice deep dives into the ancient history of coffee,   the technicalities of high quality coffee in the modern world. A real tragic, dramatic and hilarious story, brilliantly  told.  If Shadow King shows how fiction can be very real, the Monk of Mokha is the converse, it shows how non-fiction can be as good as a novel. 

As usual with my "five books" -- not definitive. If I did it again, I'd likely come up with a different list. Some honorable mentions:   The Master and Margarita, an absolute classic; and  Le naufrage des civilisations a poignant personal view of history from one of my favorite authors, who has appeared a few times on this blog. 

What is "good for the environment"?

Consider two pretty obvious statements. First, living in dense cities is much more energy efficient than living in rural areas, especially in cold places where you need heating. Second if we had 1,000 cities the size of New York City, that would be all of humanity, and they would only occupy a small fraction of the earth's surface. A somewhat larger fraction would be used for producing food and extracting some resources but  the vast majority of land on earth could be devoid of humans.  Yet, for years, I've been surprised at how often people are surprised by these points. Somehow, people assume that cities are bad for nature and that living in a rustic rural cabin or hut, using wood fires for energy is more friendly toward nature. Obviously the flaw in such reasoning is that they are thinking not of humanity as it exists today or in the future, but subconsciously going back to a time when there were very few humans, and so it didn't matter if we were extremely wasteful of resources. Of course the reason there were few humans is that most of them died very quickly.  It's a  "wet streets cause rain" type of reasoning that is surprisingly prevalent.

Another bit of inanity is that many people believe using bio-fuels is "good for the environment". Indeed the US government mandates blending corn ethanol in gasoline.  This is good for corn farmers, and good for politicians who depend on them, and maybe even good if you think foreign oil is a problem.  But in fact, when compared to just using gasoline without corn ethanol,  it results in an increase in green house gas emissions, not a decrease.   Yet the belief that it's good for the environment persists.

A new book, Apocalypse Never, by Michael Shellenberger covers  these as well as many other such points, in making a very good case that the current global alarmism around climate change is doing more harm than good.  Normally, the title and the marketing of the book would have turned me off. The last thing I want is more political BS from climate change deniers.  But this book is not that at all. The author not only agrees with the conventional view that the climate is changing due to greenhouse gas emissions, but he's actually one of the pioneers in the space.  Second what actually made me notice the book in the first place was that people were trying to get it banned or de-platformed. Which naturally kind of proves the point that he's making. And it made me want to look into it. (So maybe giving it a provocative title is a good strategy after all!).  Which I don't regret.

He also does a good job debunking the idea of "extinction of humanity". Of course we won't go extinct because of global warming. Isn't it enough to say it will cause a huge problem and enormous suffering? Similarly "saving the planet" is misguided hyperbole. The planet will still exist, even if it's boiling hot or completely frozen, and certainly more CO2 in the atmosphere and temperature changes of a few degrees are no big deal on a geological timescale. So this is just misguided and confused language that is counter-productive. It's like when people scream about "genocide" whenever there's some political violence or war that is ethnically motivated.  Constantly calling everything a genocide is not helping the cause of peace.  Similarly saying that smoking a joint is exactly the same as a heroin overdose is not helping kids avoid drugs.   

If you care about solutions and the well-being of humanity, you should be more precise in your thinking. What we care about is how we live, and what we mean by "we" is critical. The book does a good job of explaining this and the underlying basic concepts, perhaps the most important of which is "energy transitions", and he generally summarizes the science and arguments pretty fairly 

Still there are a  few points where I disagree with it, three in particular.

First, in a section on the "Tragedy of the commons", a paper and concept with which I'm intimately familiar, he  seems to imply that the "tragedy" in the paper is uncontrolled breeding of humans, which is inaccurate. But the "tragedy" is that when shared pasture land (aka commons) is not properly managed to align incentives, that leads to over-grazing and destruction of the shared resource. The real legacy of that classic paper is about mechanism design, pricing, property rights etc. But Schellenberger seems to reduce this classic insight to just the Malthusian aspect. This is a rather small technicality and doesn't change the main point he's making so I can give it a pass.

The second one is a much more serious problem. In discussing solar power he says "the achievable power density of a solar farm" is "up to" 50 watts/m2 (p. 188). But the solar constant is 1.37kw/m2 and the maximum solar energy on the surface of the earth is about 1,000 watts/m2. So he's assuming solar conversion achieves 5% efficiency at best. But this is not true, as we can see from this chart, we're at about 20-30% now and gaining about ten percentage points per decade, as I've written about before.  So his take is really unduly pessimistic about the future of  solar power.

Third, he makes a really good case for nuclear power for electricity generation. But he fails to address what to me is the strongest argument against it. Chernobyl and Fukushima are "supposed" to happen once every few hundreds of years.  But if we do a Bayesian update on those priors, the probabilities are much worse than advertised. Or to put it more simply: How come no nuclear power plant can get private insurance? If the risks are as low and manageable as he and other advocates claim, then one should be able to get free market insurance for it. But that has never happened.  I used to be pro-nuclear power, but I've become more skeptical over the years. And despite devoting a lot of the book to it, Schellenberger didn't quite convince me.

Overall, this is a good book, it helps the reader think of many of the questions in a more holistic way, and paints a coherent big picture of environmental humanism. Recommended reading.

Proof of royal blood

I have royal blood. I'm a direct descendant of one of the great kings in the history of the world. I am sure of it. Pretty cool eh? 

Which one, you ask? Well, ahem, I just mean I'm mathematically almost sure I am a descendant of a great king. And so are you.  

Here's the proof. First, like everyone, (except clones if there are any)  I have 2 biological parents, 4 grandparents, ...,  2^n direct ancestors n generations back. This is not an estimate, it is a precise fact. Second, I am not a descendent of extra-terrestrials, so I must have human ancestors on earth who were around at all points in human history. Now  let's take say n=30. That would be about 1000 years ago.  That means I have 2^30 that is about 1 billion ancestors that existed then. Woah. But the world population was only a few hundred million. So how is that possible? How can you fill a billion positions on my family tree, which, remember *must* exist, when there are only a few hundred million humans in existence. Obviously it's because the same person must fill multiple slots. If there was 100M people  let's say, then each one on average must appear 10 times. Of course, some get more than average and some get less. It's like a lottery. And the odds of each person on earth getting many slots in my tree are directly related to how many children, grandchildren etc they had. Obviously most kings have more descendants than the average person by far. Not only do they have more kids, their kids are more likely to survive and so on. Therefore in that lottery if the average human appears ten times then the average king who existed 1000 years ago must appear way more than 10 times. Therefore I have lots of kings at n=30.  Now if we keep going, as n gets larger, each king gets exponentially growing number of lottery tickets to win places in the tree. If you add it all up, the probability that I have at least one great king approaches 100%. QED.

P.S. I first read a version of this argument in a magazine article years ago, I think it was Harper's or something, will try to find it and give credit.

P.P.S. The picture is of King David (I love how it looks like "Dawit" but I think it actually says "Davidus"), from the Nine Worthies in the Palazzo Trinci, Italy, circa 1410.

Do you feel lucky... punk?

Here are two reasons why humanity might soon go extinct, and why it wouldn't be such a big loss. As you can see, I am in a cheerful mood today. 

Big rock from outer space 

Last year, using the example of the asteroid Apophis that might destroy the world in 27 years, I made the point that human beings are sometimes astonishingly stupid when it comes to making decisions that involve low probability events. If we were rational mathematical creatures, humanity as a whole should be willing to spend billions of dollars to insure against that 0.0023% chance that we will all be wiped out. If you don't like my argument based on the present value of future GDP, here's another way of arriving at the same point. If you are willing to spend a trillion dollars say on nuclear weapons to defend against other humans, and say there's a 1 in 50 chance that you actually need them, logically, you should be willing to spend a billion dollars on threats that have a 1/50,000 chance of happening. (I am using conservative orders of magnitude here, obviously a nuclear war has less than 1/50 chance of happening, so that makes my point even stronger). Today, in this article from Ars Technica, I found out just how stupid we are.

Congress awarded NASA a $1.6 million grant in 1999 to put towards the NEO discovery program. Unfortunately, this was the only funding Congress gave to NASA to pursue this goal.

Yup, the US government allocated $1.6 million dollars to save all of human life from extinction... Total! And just in case you are inclined to blame "the Americans" for being so short sighted, consider that all the other countries in the world are allocating.. ZERO! (Ok maybe they have a couple of telescopes pointing at the sky but we need giant laser beams or something...) At this point, I am almost rooting for the asteroid to kick human ass. We deserve it. 

Small germs from inner space 

And of course, a big stone falling from the sky is not the only threat we face. Tiny germs are threatening us too. Let's take the H1N1 virus -- the swine flu of recent fame. You'd think that at least when it comes to human health, humanity can be rational, right? Not so quick. Let's see how are favorite mammal is dealing with this problem. Consider the following article from the Guardian (great newspaper btw): "Experts warned dispersal of Tamiflu would do more harm than good" about the debate on anti-virus treatments for H1N1. Here's the scientific view, summarized by one expert quoted in the article:

"Some people wanted to take a long-term view of the risk of resistance developing and to seek to preserve the effectiveness of antivirals for the next pandemic, which may be more severe."

"If you get a resistant strain that becomes dominant in the autumn, Tamiflu will then be useless."

And here's another scientist:

"I am concerned about the vast amount of Tamiflu that is going out almost unregulated," he told the Guardian. "We are increasing the possibility that the flu will become resistant sooner or later. At the moment there is no desperate need for Tamiflu. We should be reconsidering its issue, rather than encouraging its use. "I think we should stop the national pandemic flu service. It was put there for an outbreak of far higher mortality than we have. If you get a resistant strain that becomes dominant in the autumn, Tamiflu will then be useless."

Ok, thank God for all these smart scientists who have thought it through! The politicians should logically follow their advice right? Well actually

"It was felt ... it would simply be unacceptable to the UK population to tell them we had a huge stockpile of drugs but they were not going to be made available"

So they just decided to go ahead and do the wrong thing! It's like a parent saying: "If I told my 5 year old not to play with this loaded gun, he would have been upset, so I decided to let him play with it." Mind you we're not talking about some distant threat here. The next mutation of the virus could be this autumn. Granted there's a low probability that it will mutate into a real killer, but that's my whole point. It's a low probability but high impact threat. And faced with that, the British government is willingly increasing the probability of a pandemic that could kill hundreds of millions of people, because they are afraid of being unpopular for the next two months! Seriously! If this was a movie, whose side would you be on? I would be like: Humans suck! Go H1, Go N1, it's your birthday! 

No rare events in the savanna 

None of this is original of course. Evolutionary biologists will say it's because our brain evolved in an environment where we just never had to consider small probabilities. We have no problem dealing with quantities like "if I go left, I get 1 potato, if I turn right I get 12 eggs"... Our brain can compute those things even as a toddler. But things like "1 in 50,000 chance" just don't compute in ye olde wetware. It's only after years of formal schooling, e.g. by the high-school level, that we start to get intuition on really small numbers. Because until the modern age, we didn't need to! Sure there were rare things like being hit by lightning, or having an earthquake, but since there wasn't anything we could do about them, there was no evolutionary advantage to actually being able to reason logically about really small probabilities. Good old superstition would work just as well. You could say "I got hit by lightning because Zeus is angry at me because I didn't offer animal sacrifice". If you are a hunter gatherer living in the bush, that explanation is practically speaking, just as good as the scientific one. But now, by our own hands, we have a world where we do need to reason about small probabilities... Problem is, the brain hasn't caught up! Global warming is another example. Twenty years ago, it was a low probability but high impact threat, just like our two examples above. Scientists were running around screaming "There's a 1 in 100 chance that the polar ice caps will melt! That's huge!" But humanity just couldn't deal with it. People were like: "One in a hundred chance of extincttion? Pffft. I'm feeling lucky. Let me go buy a lottery ticket." 

   

Well now global warming is in the same range of probability as 1 potato and 12 eggs, so people are dealing with it, but it may be too late. Is this the end-game of evolution? Is this what the epitaph will say:

Here lies humanity. They became really good at reproduction -- 6 billion individuals! But not quite good enough at probability.

Maybe it's all part of a master plan. A conspiracy! Apophis contains some organic molecules which are distant relatives of the H1N1 virus. Together the asteroid and the swine flu are collaborating to take us out, and recolonize the planet with a new dominant species that they like better. After all, that could be how we got here too!

Apophis & carpe diem & how to save the world with a billion keychains

There's a 0.0023% chance that asteroid Apophis will impact earth in 28 years. Who cares about a 1 in 45,000 chance? Believe it or not, it is useful information.

For example, you could use this information when negotiating a 30-year loan -- structure it so payments are more heavily weighted to the last two years! At what cost? Of course the world won't end, but if there's a chance... you can precisely calibrate your degree of carpe diem. You should be willing to pay up to $1 for every extra $45,000 (plus additional interest) that is deferred to the last two years. Many people spend $1 on a lottery ticket where you have a 1 in 689,065 chance of winning $10,000. And of course a big loan with a small chance you won't have to pay it back is the same as a lottery ticket -- in fact even better since a) this one pays upfront and b) the normal lottery ticket is overpriced by an order of magnitude.

Back to Apophis. Here's another, less selfish, example of how this information can be useful. We want to know how much money it makes sense for us (earth, one world united!) to spend defending against Apophis. The answer is 0.0023% times the present value of world GDP, cumulated from 2036 forward. Oops that's infinity... Wait not necessarily. If we assume GDP stops growing at some point (e.g. the point where all material needs of humanity would be easily met), and we assume a a discount rate strictly greater than zero, the present value of all future GDP is a finite number. So we should multiply that number by 0.0023% and invest it in a laser beam.

Laser beam schematic:



Take that, make it a billion times more powerful, with a nuclear battery, put it in a satellite with some stuff for aiming and we should be ok! Seriously though, we do have 28 years to work on the technology, so no biggie. But how do we create the political will to spend money on it now?

Societies seem to have a hard time making really long-term investments, whether they are democracies or whatever. But they all love lotteries! Let's revisit the deferment of debt idea. The borrower will be willing to pay a premium to take some debt and push it back past the 28th year. The lender is neutral since they get extra fees to compensate for the risk. Thus we have an efficient transaction between a rational borrower and a rational lender. OK and what has that got to do with asteroids? Recall this is essentially a lottery, one that's better than the usual ones, and we know people are willing to pay 10 times the rational price for lottery tickets. Therefore it should be possible to satisfy the lender with just 1/10th of the fee collected from the buyer! And the remaining 9/10th can be used to build a giant laser beam!!! Everyone's happy. In fact, since the beam also works to eliminate or reduce that very risk, the rational lender might even be willing to contribute part of their one tenth. And then everyone's even more happy. Let's call this the GAALBMF: global anti-asteroid laser beam mortgage fund.

Mars Landing

'Phoenix Mars Mission scheduled to land May 25, 2008, is the first in NASA's "Scout Program." Scouts are designed to be highly innovative and relatively low-cost complements to major missions being planned as part of the agency's Mars Exploration Program.'

Drexler redux: nanotech

Blog: noun, abbrev. Etymology: Web log.

A year ago, I logged a note to revisit Hanson & Drexler. One day, I was at The Strand, actually remembered and ended up picking up K. Eric Drexler's Engines of Creation (I now see there's a new edition that's even available as an e-book, but I didn't know that at the time, and just got the original edition). Incidentally, that event proves this blog is serving it's purpose! If it wasn't for this blog, my faint connection to that book might have gradually faded into the past. I also picked up Black Swan, which I ended up really enjoying but I digress...

Engines of Creation is pretty cool. First surprise: It is perhaps the seminal work in Nanotechnology. In fact, Drexler coined the term. I didn't know that. I found Drexler interesting because of a paper he wrote with Bernardo Huberman on agent/market-based computing systems. But it turns out he is the Godfather of nanotechnology. Wow indeed. Second observation: considering it's a book about future technology published in 1986, just the fact that I was able to read most of it says a lot. Usually non-fiction books about future technology don't age well: either they were right and so what they contain is now obvious, or they were wrong and are now useless, except for a few predictions that live on as comedy. But not this one. It is still absolutely readable. It does waste a bit too many words on appeasing fears of doomsday scenarios, and has a bit too much juvenile moralization. Just a bit. But the 2/3rds or so that I did read closely was fascinating stuff which hasn't aged a bit. It's great, the first time I got some clue about how nanotechnology might work.

Indeed, I had always had a hard time bridging the gap between the fictional nanotech that I found so brilliant in Diamond Age by Neal Stephenson, and anything I knew about science or technology. Not surprising since it's sci-fi, you might say, but my point is that "nanotech" exists in real science yet I don't see no microscopic helicopters and infinite-dimensional interactive books being talked about in the newspaper. What you hear about is relatively mundane "nanotech" ideas. Relatively of course, because by normal standards, even "boring" nanotech's promises are huge ... Microscopic "hard disks", or carbon nanotubes that conduct electricity much better than wires, or can support structures that are impossible with steel like space elevators!

But the Drexler vision is much bigger than better construction materials. He imagines actual nano-machines that build stuff, actual mechanical assembly of atoms and molecules. If you accept that premise, it leads to gigantic consequences, essentially limitless food, energy, all our material needs becoming ... immaterial, so to speak. This is a big idea. And it turns out Mr Drexler has made it the work of his life.

Which brings us to the question... So how come his book is not obsolete? After twenty years f being the breakthrough birth of a new field, why are those ideas still not well known to the general public, when the buzzword they created seems to now mean something much less. Well it turns out Wired magazine has answered this question in a 2004 article on K. Eric Drexler entitled "The Incredible Shrinking Man". You can read the details in the article, but in short there's a great ideological divide in nanotech, and it looks like the "incrementalists" who focus on new nano-materials and so on won the research politics battles over Drexler who wants to build nano-machines.

Now I feel for Drexler in a different way. Not just as a fascinating guy who I should follow-up on but as a human story too. Could he be like Edwin Armstrong, who invented FM radio and many other great things, but lost all the crucial battles in his life? Or is it going to be a classic story of early brilliance, fall from grace, long struggle, and ultimate righteous triumph... Hmm I think that's classic but I can't think of any examples right now. Anyway here's a possible triumphant ending: nanobots win the prize for removing carbon from the atmosphere. Imagine a little machine made of a few atoms of X that uses solar power to move around and grab CO2 from the air, and then attaches the C to some part of itself, releases the O2, and then falls to earth as XC dust. It could even be called something cool, like photosynthesis.. haha. Who knows, if X is right, that XC might even be a source of fuel! So in the end, our hero stops global warming and saves the world! Good night kids.

Researchers discover gene that blocks HIV

Isn't this kind of a big deal?

Researchers discover gene that blocks HIV.
Stephen Barr, a molecular virologist in the Department of Medical Microbiology and Immunology, says his team has identified a gene called TRIM22 that can block HIV infection in a cell culture by preventing the assembly of the virus.
[...]
"We are currently trying to figure out why this gene does not work in people infected with HIV and if there is a way to turn this gene on in those individuals," he added. "We hope that our research will lead to the design of new drugs, or vaccines that can halt the person-to-person transmission of HIV and the spread of the virus in the body, thereby blocking the onset of AIDS."


I remember around ten years ago, researchers had found a way to prevent the HIV from attaching itself to and attacking white blood cells by making another organism that would go and attach itself to the same geometric spot (or something like that). That sounded really promising. I wonder whatever happened to that....

Killer app? Spam and HIV epidemics

Talk about tackling evil: Using Spam Blockers To Target HIV...

A Microsoft researcher and his team make a surprising new assault on the AIDS epidemic ...