Embeddings, Wang Chung, and the Geography of Scotland: ttftw 2023w28

Three things from this week.

Here are three things I learned this week. They are not related.

Embeddings

Vicki Boykis generously shared her research on “What even are embeddings?” in 80 pages of PDF written at just the right level for me. I’d highly recommend it if you too are tinkering with ML models and feel like you have some jargon gaps to fill in.

Since I was on a plane a couple times I was able to pay attention long enough to read the whole thing and run some of the sample code.

The highlight for me was the straightforward explanation of the lineage from Word2Vec through Transformers to BERT to ChatGPT. Looking at the way context is built in different generations of ML is eye-opening. I’ve built Markov chains from scratch and used lots of more advanced LLMs. However I’ve been looking at a lot more text in non-English languages lately so the the way nearby words are selected for context really stood out to me. Even the tokenization bakes in strong assumptions about the language of the text. It gave me a lot to think about. And it helped me better understand how models created for NLP can be adapted with different sets of embeddings. Like learning features from an image, creating embeddings for those features, and then using those embeddings with other models.

Wang Chung was not a one-hit wonder

Wang Chung has other songs besides the eponymous instruction. I would have guessed that Dance Hall Days was by Billy Idol. Apparently huang chung means “yellow bell” and it’s first note in the Chinese classical music scale.

The Highlands

As I was returning from Scotland and people were asking me if I’d been to the Highlands I decided that I’d better figure that out. Most of my trip was in the Highlands. We went to Glasgow, Islay, Oban, Skye, Inverness, and Edinburgh over the course of a couple weeks. Historically, Scottish Gaelic is the language of the people of the highlands. The area includes the whole north part of the country, with the islands. My confusion comes from the names used for Scotch whisky regions. When discussing whisky, Islay and other islands are separated out, as is the Speyside region around the river Spey.

The trip was amazing, btw, and I can’t wait to go back again soon.

• ttftw