A deep dive into BERT’s attention heads — Background: Transformers, Self-Attention, and BERT In the past few years the progress on natural language understanding has exploded, thanks largely to the new neural network architecture known as the Transformer. The main architectural innovation of the Transformer model has been the extensive use of so-called “self-attention”, so much so that…

The surprising connection between distance measurements and a famous mathematical sequence — If you’ve ever done any long-distance running, you’ve probably noticed that there’s a mishmash of distance measures used: meters, kilometers, miles. But if you convert the lengths of different races into alternate units, you notice something peculiar: 5k race: 5km → 3.11mi ~ 3mi 8k race: 8km → 4.97mi …

A simpler formula than what you were taught in school — The Simplified Formula As an American who frequently travels abroad and has a European partner, I’m often faced with the problem of needing to convert temperatures from Celsius to Fahrenheit. The exact formula is a bit too complicated to do in your head, but thankfully there’s a simplified version that gives extremely accurate…

Ambiguous Greek vowels and their probabilities — My girlfriend is Greek, and because I love her dearly I have been making an ill-fated attempt to learn the language. Thankfully from my background in math and physics, I already know most of the alphabet, so I’ve decided to lean on that as much as possible. When she teaches…

The game theory behind a simple task — Growing up my dad and I had an ongoing spat about the right way to clean our clothing drier’s lint trap. He argued that it should be cleaned after each run, that way you don’t need to worry about cleaning up someone else’s mess. I argued that it should be…

Making the ephemeral future more concrete — We all know how important it is to postpone immediate pleasures for long-term rewards. …

And why procrastination leads to more imperfect results — Deep down, I think everyone is a perfectionist. Being a perfectionist just means wanting to do things the right way. And what kind of person wants to do things the wrong way? The problem is that with most things in life, it’s hard to know what the right way to…

Why it’s (mathematically) wrong to put someone on a pedestal — I’m currently going through a concerted period of personal development, and one of the steps in that process is to identify unhelpful patterns of thought and behavior in myself, and either decide that it’s there for a good reason, or else discard it. And at the risk of airing my…