At an event, Meshi's agent suggests who a member should meet for a goal. In early October we tested whether reordering candidates with an LLM puts better people at the top, and graded the results with LLMs too. Two panels of AI graders, each consistent within itself, agreed with each other only slightly more than chance on the same 520 candidate-goal pairs.
The test
We used one event of about 370 people and 10 inferred goals. For each goal, Meshi's live retrieval returned its top 50 candidates, and every ordering we tested rearranged those same 50. Three LLM graders (the first panel), each in a fresh session and blind to which system produced which list, read a short card for every candidate and rated it strong, plausible or poor for each goal, judged on whether a meeting would be worth both people's time. The label is the majority of the three; a three-way split counts as plausible. Strong and plausible count as good.[1]
A separate AI reviewer also planted three good and three poor people per goal, chosen from full profiles with no ranking in view. We counted a goal only if the panel labeled all six correctly, and 9 of the 10 did.
On the first panel's labels, an LLM reranker beat Meshi's existing order, and a check with no grader labels agreed (full result).
The second panel
For the 6 October test of a two-sided matching layer, which beat the existing order on 0 of 9 goals, a second panel of three graders labeled candidates; 520 candidate-goal pairs overlapped with the first panel's. The second panel's graders saw both people's full profiles and rated the benefit in each direction.[2] Within each panel, a chance-corrected agreement score, Fleiss' κ, was 0.60 for the first panel and 0.70 for the second (0 is chance, 1 is perfect).
The panels gave the same good-or-poor label to 56.3% of the pairs, against 44.5% expected by chance given each panel's base rate of good labels (42% for the first panel, 83% for the second). Corrected for chance, agreement is 0.21.[3]
A panel that calls most candidates good leaves little room to tell orderings apart. By the second panel's labels, a random order of the top 50 puts 74 of the 90 top-10 spots (10 per goal across 9 goals) on good people, so that test's failure told us little. By the first panel's labels, a random order puts about 37 there.
What to report
Report each panel's base rate and the score of a random order on the same labels next to every result. Shuffling the labels should change each reported result. A result is more credible when a check that uses no grader labels shows it too.
Footnotes
-
All labels come from LLM graders, on a single event. ↩
-
The first panel's cards cut each profile summary to 600 characters. The second panel also graded a larger pool, the union of three systems' top 50 plus its own planted controls, and read profiles from a later snapshot. ↩
-
We computed the cross-panel agreement figures, the 56.3% and 44.5%, and the chart above on 16 November 2026 from the October labels; the within-panel Fleiss values date from each panel. The cross-panel κ of 0.21 has a 95% interval of 0.16 to 0.27 from 10,000 bootstrap resamples of the 520 pairs. Over all three categories, agreement was 43.8% against 27.8% by chance, a κ of 0.22. ↩