Algolia-style Postgres search with a Jev reranker
Typo tolerance, typeahead, barcodes and facet counts in plain Postgres SQL, plus an optional Jev step that asks one question per top-10 result.
# postgres-search Product search inside PostgreSQL: whole words, partial words, typos, barcodes, typeahead and facet counts, all in SQL, with an optional second stage that asks [Jev](https://docs.typesafe.ai), a hosted model that returns probabilities instead of text, two questions per search. This README is also a report on that second stage: how much it helps, how each version failed and was fixed, what it costs, and how it compares with Algolia. [Try it](#try-it) · [Use it with your data](#use-it-with-your-data) · [The guide](#the-guide) ## Contents 1. [Summary](#summary) 2. [System](#2-system) 3. [Method](#3-method) 4. [Results](#4-results) 5. [Failure analysis](#5-failure-analysis) 6. [Discussion](#6-discussion) 7. [Cost, and the comparison with Algolia](#7-cost-and-the-comparison-with-algolia) 8. [Threats to validity](#8-threats-to-validity) 9. [Reproducing the results](#9-reproducing-the-results) 10. [References](#references) ## Summary We compare three ways to search the 440,302 products of the USDA Branded Foods release: PostgreSQL full-text search as its manual presents it, this repository's SQL, and this SQL followed by Jev. Jev's second question, "which spelling did the user mean?", is also compared with two spelling correctors that use no model, among them one in the style of Norvig (2007). The design went through six versions. Each change was made on data already seen and frozen in a commit; new test sets were then generated, committed before any model saw them, and scored under an analysis plan fixed at the freeze. The current version, 2.4, was scored on three such sets: 500 one-edit misspellings and controls, 179 words cut short as a user types them, and 200 correctly spelled words that each sit one edit from a far more common word. Every system ra