Torre & Tagus 7" Spike Gold Sphere Decor (Steel Metal Ball) - Starburst Table Decor For Decoration, Handmade Sputnik Geometric Decorative Objects, Mid-Century Modern Decor, Gold Home Accents for Shelf

The thing about high dimensional vector spaces is that when N is large they are strangely different from the N=2 and N=3 cases we are familiar with. For instance when N=3 you could imagine that a cube is not all that different from a sphere, just sand away the corners. If N=10,000 though, the "cube" has a huge number of corners which are a distance of 100 away from the origin whereas the sphere never gets past 1. Hypercubes look something like this

https://www.amazon.com/Torre-Tagus-901918B-Spike-Sphere/dp/B...

A consequence of that is that many visualizations give people the wrong idea so I wouldn't try too hard.

Of everything in the article I like the histograms of similarity the best but they are in the weeds a lot with things like "god" ~ "dog". When I was building search engines I looked a lot at graphs that showed the similarity distribution of relevant vs irrelevant results

I'll argue bitterly about word embeddings being "very good" for anything; actually that similarity distribution looks pretty good, but my experience is that when you are looking at N words word vectors look promising when N=5 but when N>50 or so they break down completely. I've worked on teams that were considering both RNN and CNN models. My thinking was that if word embeddings had any knowledge in them that a deep model could benefit from you could also train a classical ML model (say some kind of SVM) to classify words on some characteristic like "is a color" or "is a kind of person" or "can be used as a verb" but I could never get it to work.

Now I went looking and never found that anyone had published positive or negative results for such a classifier, my feeling was it was a terrible tarpit, particularly when N was tiny it would almost seem to work but when N increased it would always fall apart. Between the bias that people don't publish negative results and that people who got negative results might blame themselves and not word embeddings or the hype around word embeddings, they didn't get published.

I do collect papers from arXiv where people do some boring text classification task because I do boring text classification tasks and I facepalm so often because people often try 15 or so algos, most of which never work well, and word embeddings are always in that category. If people tried some classical ML algos with bag-of-words and pooled ModernBERT they'd sample a good segment of the efficient frontier -- a BERT embedding doesn't just capture the word, it captures the meaning of the word in context which is night and day different when it comes to relevance because matching the synonyms of all the different word senses brings as many or more irrelevant matches as it does relevant ones.