Work

Concept search before vectors were the default

A books-and-journals search that let people explore by concepts, built in 2018 from phrases and similarity, not from a vector database.

The problem

Researchers and students did not want another keyword box over books and journals. They wanted to move through a subject.

The approach

I extracted noun phrases and adjective-noun phrases, exposed them as filters, and trained a word2vec model so a section could be tagged with related concepts. Custom Solr pieces covered the request path, spelling, autosuggest, and filters.

Apache SolrRelevanceHybrid search

The conclusion

A concept a person can read beats a similarity score they cannot. The same idea shows up now in hybrid retrieval: use the dense signal where meaning is fuzzy, and keep a precise handle where it is not.