When Meshi imports a list of people, it searches for each person's LinkedIn profile and asks an LLM, Gemini 3.1 Flash Lite, which of up to five candidates is that person, if any. A wrong match attaches a stranger's profile to someone, so it is the expensive mistake; sending a person to review costs only time. In October we tried putting a small model in front of the LLM.
The best result came from a 150M-parameter encoder that goes first and hands every case it is unsure of to Gemini. In replay against Gemini alone, the pair matched 118 more people on one held-out set and 97 more on the other, lost none of the people Gemini had matched correctly, and cut Gemini spend by about 60%. The run meant to decide whether to ship it, one pass over a sealed test set, never happened. Production still runs Gemini alone.
The task
In production, each person arrives as a row of whatever the list had: a name, a company, a title, sometimes an email. A Google search returns LinkedIn results, a name filter keeps up to five, and the judge answers one of three ways: accept a candidate, send the person to review, or say none of them is the person. An accepted match must also pass the identity checks production already runs, such as a corroborating employer.
To get labeled examples, we took 55,378 people whose LinkedIn identity Meshi already had on record, almost all from LinkedIn data its users had synced. We built each person's row from those records, ran the search production would run for that row, and labeled a candidate correct when it matched the known profile. No one checked the labels by hand.[1] People were split by identity, so the same human never appears in two sets: 37,311 for training, 5,934 for development (DEV), 5,930 for calibration (CAL) and 5,970 for a sealed test, plus 233 people from real uploaded lists, held aside. Every number below comes from DEV or CAL.
Before training, we wrote down the bar a model had to clear. It had to accept at least 30% of all people without Gemini. Of the people it accepted, the share that were wrong, as an upper bound at 97.5% confidence, had to be at most 0.5%. And the combined system could match at most 2 percentage points fewer people than Gemini.[2] The design was a small gate in front of Gemini:
Four models, one floor
We trained four kinds of model on 17 and 18 October:
- a logistic regression on hand-built match features;
- MiniLM-L6, a 23M-parameter cross-encoder, with three seeds;
- Qwen3.5-0.8B with LoRA adapters;
- Ettin-150M, an encoder from the ModernBERT family.
The cross-encoders read the person's row and each candidate as labeled text, then choose one candidate or a learned "none".[3]
Among people with candidates, the logistic regression picked the labeled answer for 93% and MiniLM and Ettin for about 95%, a statistical tie between the two. The LoRA model ranked worse than MiniLM, and when we reversed the order of the candidates it changed its pick for 17.5% of DEV people. A gate that depends on list order cannot be trusted, so we stopped that line.
Then we checked the bar. At its best threshold, every model's bound sat near 1%:
None reached 0.5%.[4] Ettin reached its floor while accepting twice as many people as the others.
A closer look at the labels
A label says which candidate matches the identity we already had on record. Before reading CAL, we registered an audit plan and froze the thresholds chosen on DEV. Then we audited every CAL person that Ettin or MiniLM had wrongly accepted, 30 in all, shuffled with 30 correct accepts as decoys. Two AI reviewers from the same model family judged each case blind, and a third resolved their disagreements.
For 26 of the 30, the evidence supported the model's pick. For 4, the reviewers could not tell, and those stayed wrong. None contradicted the model. Of the decoys, 29 were supported and 1 was unclear.
On the reviewers' corrections, both models cleared the bar: Ettin while accepting 54% of all CAL people at the DEV threshold, MiniLM 48%.[5] On 20 October we chose Ettin.
Ettin first, Gemini when unsure
By 21 October we had run Gemini 3.1 Flash Lite over the same frozen DEV and CAL shortlists, with the production prompt and output contract. The combined system takes Ettin's match when Ettin accepts and Gemini's decision otherwise.[6]
On its own, Ettin matched fewer people than Gemini. Going first, it added people without losing any:
Every gain was a person Ettin matched and Gemini missed or got wrong. Wrong accepts went from 34 to 33 on DEV and from 46 to 45 on CAL. Of the 5,210 DEV people with candidates, Ettin took 3,514, and Gemini spend fell 65% on DEV and 59% on CAL:
Gemini matched the most people of any judge we ran.[7]
Speed
Ettin runs on one CPU thread, in milliseconds where the LLM takes seconds:
Every person Ettin accepts skips a Gemini call.[8]
Where it stands
The bar's error limit was to be judged once, on reviewed labels, over the 5,970-person sealed test. On DEV with the original labels, the combined system's bound was 1.22% and Gemini's 1.29%. Priorities moved, and the work was archived on 8 and 9 November with the test still sealed. The trained model is kept.
Footnotes
-
The search used the person's company for 46,643 people, the name alone for 8,685 and an email domain for 50. Searches failed for 738 people. People with failed or empty searches stay in every denominator as people no one matched. ↩
-
The bound is a one-sided 97.5% Clopper–Pearson upper bound on wrong accepts per accept, counting one person per identity. The 30% counts accepts over all people, including those whose search failed or came back empty. ↩
-
Each field becomes
[COL] field [VAL] valuetext of at most 256 tokens. The model is trained with a listwise loss over the candidates plus "none", random candidate order and the field gaps seen in production. Ettin used MiniLM's recipe unchanged: AdamW, learning rate 2e-5 for the encoder and 2e-4 for the head, batch size 8, two epochs, seed 1729, on one rented A100 for about four hours (about $8). ↩ -
Each point is the lowest bound over all thresholds, chosen on CAL itself, which flatters every model. The MiniLM range covers its three seeds. ↩
-
The reviewers judged from the saved search results. Under a stricter rubric that requires proof of the account or its exact URL, every case came back unknown. CAL had already been used to fit the acceptance step, in folds so that no person was scored by a fit that had seen them. If reviewers wrongly cleared cases at the highest rate the 30 decoys allow, 11.6%, the corrected bounds would be 0.45% for Ettin and 0.46% for MiniLM. ↩
-
Gemini ran through its native API with structured output, over 5,934 DEV and 5,930 CAL people, for $5.70 and $5.76. One DEV call failed in transport and counts as review. Ettin's threshold was fixed on DEV, and each CAL decision comes from a fit that had not seen that person. All three systems were scored on the original labels. ↩
-
On DEV, DeepSeek V4.1 Flash matched 57.1%, Jev 1.14 55.9% and Ling 3.1 Flash 41.3%. DeepSeek and Ling read the production prompt with a JSON Schema appended, because their endpoints lacked usable structured output; a later review of that adapter, in a pilot of newer judges, found that it put challengers at a disadvantage. Jev used its own adapter. ↩
-
Ettin's time is model time in float32 with eager attention on one CPU thread, excluding model loading, input preparation and the acceptance step, over 500 DEV people with candidates sampled independently of the labels, on a shared machine under heavy unrelated load. Gemini's is wall-clock per call. ↩