At an event, Meshi's agent suggests who a member should meet for a goal. A retrieval test that morning concluded that, at one event's scale, retrieval already surfaced the good people and the order of the top ten was where quality was lost.[1] Meshi's existing order put good people in 49 of the 90 top-ten slots (nine goals, ten slots each), where a random order of the same 50 candidates would fill about 37.[2]
So we took those 50 candidates per goal and asked a language model to reorder them.
The setup
For each goal, the model read the member's goal, role, organization and profile summary, plus one short card per candidate: role, organization, the first 350 characters of their profile summary, and up to two of their goals. It returned its top 15 in order, and the rest kept the existing order. We scored the top ten.
Three independent AI graders rated every candidate strong, plausible or poor for each goal, and a candidate counted as good when at least two of the three said strong or plausible.[3] As a control, a separate AI reviewer picked three clearly good and three clearly poor people per goal from full profiles, with no ranking in view, and we planted them in the pool. A goal counted only if the graders labeled all six correctly. Nine of the 10 did.[4]
We froze the pass bar before any rerank ran: beat the existing order on at least 70% of goals, be worse on at most one, and add at most one second of median latency.[5]
The rerank nearly doubled the good matches
Gemini 3.8 Flash put good people in 88 of the 90 top-ten slots, against 49 for the existing order. It beat that order on all nine goals and was worse on none. It also took a median 9.5 seconds per search, far past the one-second bar.
We suspected most of that time went to the model's reasoning before it answered, so the same day we turned its thinking setting down to low. It reached 87 of 90, beat the existing order on eight goals and tied on the ninth, at a median of 1.2 seconds, still 0.2 seconds over the bar.
A check that used no grader labels agreed. Pairing each planted good person with each planted poor one, the existing order put the good one first 81.5% of the time, and the low-thinking rerank 97.5%.[6] Every ordering here rearranged the same 50 people, so the gain came from the order, not from finding different people. Ordering them by embedding similarity did no better than random.[7]
Cost and speed
We then priced cheaper setups against the same labels.[8] The one we approved on 2 October is a two-step rerank.[9] A small, fast ranking model orders all 50 candidates, then Gemini 3.5 Flash-Lite, with minimal thinking, reorders its top 15. It put good people in 78 of the 90 slots, beat the existing order on eight goals and tied on the ninth, and ranked every planted good person above every planted poor one.
The two-step rerank costs about $1.25 per 1,000 searches and adds a median 1.04 seconds, 36 ms over the bar. Either Flash-Lite model alone on all 50 cost more ($1.62 and $1.96) and found fewer good people (77 and 76). The low-thinking rerank found nine more than the two-step, at about 3.7 times the cost.
Footnotes
-
That test's single AI judge failed its own planted-control check on 4 of 10 goals, so this experiment used a repaired panel of three graders. ↩
-
One 366-person professional event and 10 goals, all inferred by Meshi's pipeline rather than typed by members. The random-order figure, 37.2, was computed on 8 October from the same labels. ↩
-
Each grader worked in a fresh session with its own shuffled order and batch sizes, rated every candidate in the pool rather than any ranked list, and judged whether a meeting would advance the member's goal and be worth the other person's time. A three-way split counts as plausible. As a check on the scorer, shuffling the labels within each goal changed every ordering's score; the LLM rerank fell from 95 to 39 over all ten goals. ↩
-
The tenth goal missed one planted good person and is excluded from the totals. Including it, the existing order scored 53 and the LLM rerank 95 out of 100. ↩
-
The pass bar was frozen on 2 October before any rerank ran. The fast variants, including the low-thinking run, were registered before any of their orders existed; the model rejected the minimal thinking setting they specified, so they moved to low, still before any orders existed. The cost comparison afterward was exploratory, on the same labels, pool, gate and scorer. ↩
-
Over the nine scored goals, run on the low-thinking rerank. The graders and the reranker are both language models reading the same cards, so they may share a style of judgment; the planted people were chosen from full profiles by a separate reviewer. ↩
-
On 8 October we ordered the same 50 by embedding similarity between a generated description of each member's ideal match and one of each candidate. It filled 34 of the 90 slots with good people, slightly below random order. ↩
-
Cost is the measured token usage at list prices. Latency is the added median per search, measured from a laptop. ↩
-
It was approved and never merged. On 7 October we tried 61 more orderings on the same pool; the best reached 82, and with nine goals none beat the two-step rerank under a sign test (all p ≥ 0.22). ↩