What is Indexing and Why is it Useful for Recommender Systems?

This post contains an answer I gave to Kathryn Kyte who wrote this BBC piece about Waverly.

What does indexing an article mean in the context of a recommender system?

The best analogy I can think of is that of a business directory. A big book where you have, in alphabetical order, different services: Bricklayer, Mechanic, Plumber… Under each category, you have a list of businesses that offer this service. A given business can fall under two services if it does multiple things.

An index of articles is very similar. Instead of having an alphabetical list of services, you have an alphabetical list of characteristics that can apply to an article. These characteristics can be topics, people mentioned in an article, writing styles (journalistic, casual, fiction), etc.

If you look under a given entry of the index, say the topic Beneficial AI, you’ll find all the articles that seem to be talking about beneficial AI. At Waverly, our AI reads tens of thousands of articles a day and extract hundreds of characteristics about these articles. These characteristics, and the articles the point to, constitute the Waverly index.

It’s this index, combined with our ability to extract which characteristics seem important to you given the Waves you’ve written, that allows us to offer the first natural-language-driven recommender system.