shipwithjev

Catalog / Tools & apps

0606GitHub

Algolia-style Postgres search with a Jev reranker

Typo tolerance, typeahead, barcodes and facet counts in plain Postgres SQL, plus an optional Jev step that asks one question per top-10 result.

alexforman1/postgres-searchREADME ↗
# postgres-search

Product search inside PostgreSQL: whole words, partial words, typos, barcodes, typeahead and facet
counts, all in SQL, with an optional second stage that asks [Jev](https://docs.typesafe.ai), a
hosted model that returns probabilities instead of text, two questions per search. This README is
also a report on that second stage: how much it helps, how each version failed and was fixed, what
it costs, and how it compares with Algolia.

[Try it](#try-it) · [Use it with your data](#use-it-with-your-data) · [The guide](#the-guide)

## Contents

1. [Summary](#summary)
2. [System](#2-system)
3. [Method](#3-method)
4. [Results](#4-results)
5. [Failure analysis](#5-failure-analysis)
6. [Discussion](#6-discussion)
7. [Cost, and the comparison with Algolia](#7-cost-and-the-comparison-with-algolia)
8. [Threats to validity](#8-threats-to-validity)
9. [Reproducing the results](#9-reproducing-the-results)
10. [References](#references)

## Summary

We compare three ways to search the 440,302 products of the USDA Branded Foods release: PostgreSQL
full-text search as its manual presents it, this repository's SQL, and this SQL followed by Jev.
Jev's second question, "which spelling did the user mean?", is also compared with two spelling
correctors that use no model, among them one in the style of Norvig (2007). The design went through
six versions. Each change was made on data already seen and frozen in a commit; new test sets were
then generated, committed before any model saw them, and scored under an analysis plan fixed at the
freeze. The current version, 2.4, was scored on three such sets: 500 one-edit misspellings and
controls, 179 words cut short as a user types them, and 200 correctly spelled words that each sit
one edit from a far more common word. Every system ra

Also filed under Tools & apps

  1. 0616

    Jevstiller: distill Jev calls into a local model

    A self-hosted proxy that trains a local model on Jev's answers. It served 72% of Banking77 locally at 15 ms, with a 2% audit and a bound on disagreement.

    tomerglick57 · Tools & apps · 15 ms local

  2. 0611

    Scruple: code checks for what linters miss

    Semantic-aware linting with System One models: rules you can state in a sentence, checked against your code by Jev.

    @nalexpeae · Tools & apps

  3. 0604

    Virtual Closet: outfits from closet videos

    built a virtual closet & outfit generator from raw closet videos. classification + semantic search thanks to jev 🥰. clueless closet dreams finally coming to life

    @maitreemervana · Tools & apps

  4. 0600

    bonk: a fox guards your Mac's keyboard

    Keeps your desktop visible and your Mac awake while a tiny fox blocks keyboard, mouse and trackpad input. Jev drives the character's reactions.

    Captain-Sangam · Tools & apps