Offline benchmark study. As of Sept 16, 2026 no Aera release sends data to TypeSafe.
Abstract
Aera takes responsibility for work a person needs handled, in their browser and their signed-in accounts, including ongoing and scheduled work. To do that well, the agent has to start each piece of work already knowing what came before: the decision made last week, the file that holds the data, the task that already collects it. We call that recall. When a chat or an unattended run starts, Aera hands the agent a short section of recalled material, and in our current design a small LLM picks that material from a pool of search candidates.
We replaced that LLM with Jev, TypeSafe's hosted classifier, which answers typed questions with probabilities and cannot write text. We scored it on 400 real chats and runs that a hindsight judge had labeled with what each one needed.
Asked one yes/no question per candidate in a single request, Jev matched the LLM selector's coverage on unattended work (46% of judged needs) with higher precision (85% against 79%) in about a third of the time (147 ms median against 463 ms). At chat start it let us grade a pool five times wider inside our latency budget: 46% coverage at 72% precision with a median of 441 ms, against 43% at 53% for the LLM design. On one of our two test dates the LLM path finished inside its own 900 ms deadline in 8 of 124 chats.
Jev did not find more per chat than the LLM finds when the LLM has no deadline (30 chats better, 29 worse, 65 tied). Its gains are precision, speed and predictability. Three designs failed: a Choice question to rank the pool, an ensemble of question framings, and a gate that skips recall. A grading pass over every candidate showed that the wide pool already holds material covering 82% of what chats needed, and that the recalled section, which fits about three items, is now the binding limit.
1. Background: what recall has to do in Aera
Recall stands on two legs. The first is the section Aera writes into the agent's instructions before it starts, built from the person's message or the run's identity. The second is the agent's own choice to search its memory later with a tool. This paper is about the first leg.
Five kinds of work receive a section:
| Surface | What starts | Who is waiting |
|---|---|---|
| Chat start | a person sends the first message of a chat | the person |
| Follow-up | a later message in the same chat | the person |
| Run review | the assistant reviews a finished unattended run | nobody |
| Scheduled run | a recurring task or a team member's assignment runs unattended | nobody |
| Delegated subagent | an agent hands part of its work to a child agent | the parent agent |
The index behind recall is lexical: BM25F over earlier chats and 19 kinds of records, including files, memories, scheduled tasks, reports, goals and saved sign-ins. A purely lexical section covered 23% of what chats needed at 39% precision in our benchmark. The current design adds a query built from context (the open page, attachments, the task's role and goal), a filter for the system's own notices, and one LLM call that reads a pool of 28 candidates and names what to quote and what to mention. At chat start a second LLM call rewrites the request into search queries. Both calls race a 900 ms deadline, and whatever misses the deadline reaches the agent with its first tool result.
We hold anything on this path to three constraints:
- 1.5 s at most before the agent starts, and under 1 s where we can.
- Under $0.01 per call.
- Reasoning turned off, and no provider that trains on the input.
An LLM meets those constraints only narrowly. It reads the pool token by token and writes its answer token by token, so latency and attention both get worse as the pool grows. That is why the pool stops at 28.
2. What Jev is
Jev is not a language model you prompt for text. A request carries one state (any JSON) and a map of typed questions. There are three question types:
- Choice picks one of up to 255 options and returns a probability for each. The probabilities sum to 1.
- Noul is a yes/no question. It returns the probability of yes, independent of every other question.
- Score places the state on up to 10 described levels.
Questions in one request are answered in parallel and cannot see each other. The request budget is about 32,000 tokens, shared by the state and the questions. TypeSafe's published examples price Jev at $0.042 per million input tokens with no charge for output. TypeSafe's privacy policy says API input is not used to train models. It names no retention period, and the service is hosted in the United States.
Before designing anything we measured three properties with synthetic text.
Figure 1. Left: five live calls per point. One yes/no question per candidate over 200 candidates (16,166 input tokens) has a median round trip of 245 ms. Over 10 candidates it is 132 ms. Right: the state is counted once. Thirty questions over a 30-candidate state reported 2,604 input tokens against 1,772 for one question. If each question paid for the state, the dashed line would apply.
The third property decides the question shape. A Choice looks like the natural way to rank a pool, and it is the wrong one.
Figure 2. Thirty synthetic candidates, three of them relevant to the request (C8, C9, C10). The Choice puts 0.98 on C8, 0.01 on C10 and nothing on C9. Per-candidate yes/no questions return 0.95, 0.92 and 0.93 for the three and 0.05 or less for the rest.
A Choice answers "which one". Recall asks "which ones". So our selector is one Noul per candidate, all in one request. An abridged example:
{
"model": "jev-latest",
"state": {
"work": "A person working in workspace \"Marketing\" opened a chat with:\nwhere does the data from this chart come from\nThe page in front of them: the workspace's reports page",
"candidates": {
"C1": "memory · 2026-09-03 · \"Marketing data sources\" · record: For this workspace, the analytics service is the source for web traffic and acquisition, and the admin dashboard is ...",
"C2": "workspace update · 2026-09-09 · \"Daily metrics snapshot\" · record: Update published 2026-09-09 ..."
}
},
"questions": {
"use:C1": {
"type": "noul",
"instructions": "Would a competent agent starting the work in `work` draw on `candidates.C1`?",
"criteria": {
"true": "The agent would draw on this candidate for this specific work: it holds or points at a decision, preference, correction, figure, failure, file, table, task, sign-in or report the work concerns.",
"false": "The candidate only shares words or a broad topic with the work, is a system notice, or is generic material the agent would not need."
}
},
"use:C2": { "type": "noul", "instructions": "... `candidates.C2`?", "criteria": { "true": "...", "false": "..." } }
}
}
The answer is a probability per candidate. Our code keeps candidates at 0.4 or above and orders them by probability. Things that are better named than quoted (a sign-in, a task, a connector) become one-line mentions, and the rest are quoted, up to the section's budget. The model never sees the budget and never formats anything.
3. Method
The benchmark replays real use. It takes a copy of one working profile from inside Aera, used daily for real work across several workspaces over several weeks, and rebuilds every chat start and run start as a test case: the work as it stood at that instant, with every later document hidden. No customer data was used. For this study, candidate excerpts from that profile were sent to TypeSafe's API and to the LLM providers named below.
Labels. A hindsight judge (GPT 5.6 Luna, reading each whole transcript) listed what each of 400 chats and runs needed from before it started: 124 chat starts, 70 run reviews, 82 scheduled runs and 124 delegated subagents.
Scoring. A matcher model reads the opening request, the judged needs and the items a selector handed over. It marks each item useful or noise and lists which needs each item covers. Two numbers come out. Coverage is the share of judged needs covered. Precision is the share of handed items called useful. The matcher agreed with the judge on 79% of 199 items both had seen.
Selectors compared.
| Name in this paper | Candidate pool | Model calls |
|---|---|---|
| Lexical only | none, the top search results go straight in | none |
| LLM selector | 28 candidates | 1 LLM call (DeepSeek V4 Flash, reasoning off, through OpenRouter with no-training routing) |
| LLM rewrite + select | the same, plus a call that rewrites the request into queries whose results fill the remaining budget | 2 LLM calls at once |
| Jev, same pool | the same 28 candidates, so only the judge differs | 1 Jev request |
| Jev, wide pool | 150 candidates: a looser relevance floor, more results per query, and extra queries from the open page, each attachment, and the task, goal, role and workspace names | 2 Jev requests at once |
| Jev, wide pool, two passes | the wide pool at a 0.3 threshold, then the best 24 read again with 700-character excerpts | 2 at once, then 1 |
Figure 3. The two pipelines at chat start. Everything outside the colored boxes is the same code.
The pool oracle. A selector's score says what it handed over. It does not say what it could have handed over. So we had the matcher grade every candidate in the wide pool for all 124 chat starts: 13,947 candidates. That gives the pool's ceiling, and it gives an offline referee. Any threshold, budget or question shape can be scored against those verdicts with no new model calls. The offline scorer renders the real section, so the character limit applies, and its numbers track the live score: the same policy measured 37% at 70% offline and 39% at 70% live.
Controls. Jev answers were cached by request so that threshold sweeps compare policies over identical grades. Latency figures come from live calls only. We ran the LLM selectors on two dates, 2026-09-12 and 2026-09-16, and ran Jev on 2026-09-16 in the same hour as the second LLM run.
4. Results
4.1 Same pool, different judge
On unattended work the pool was already good, so this is the clean comparison: identical candidates, LLM against classifier.
Figure 4. 276 labeled runs. "Jev, same pool" reads exactly the candidates the LLM selector reads.
| 276 unattended cases | Needs covered | Precision | Median / 95th percentile | $ per case |
|---|---|---|---|---|
| LLM selector | 46% | 79% | 463 / 707 ms (model time) | 0.00028 |
| Jev, same pool | 46% | 85% | 147 / 271 ms | 0.00017 |
| Jev, wide pool | 44% | 84% | 258 / 516 ms | 0.00104 |
| Jev, wide pool, two passes | 47% | 86% | 426 / 786 ms | 0.00129 |
Coverage is equal. Jev hands over less noise on every surface: run reviews 90% against 84%, subagents 82% against 77%, scheduled runs 82% against 73%. The wide pool buys nothing here. These surfaces carry rich queries (a run's report, a task's full description), so 28 candidates already hold what there is to find.
Scheduled runs sit at 24% coverage under every selector, LLM or classifier. Their section fits about one item, and their query is a task name plus a description. No judge fixes that. It is a query and budget problem.
4.2 Chat start
This is the surface where a person waits, and the one where the selectors separate.
Figure 5. Each point is one selector over the same 124 chats. The orange point is the LLM path on 2026-09-16, when most chats missed the deadline and started with the lexical fallback.
| 124 chat starts | Needs covered | Precision | Median / 95th percentile | $ per chat |
|---|---|---|---|---|
| Lexical only | 23% | 39% | 3 / 6 ms | 0 |
| LLM selector, 2026-09-12 | 35% | 52% | 518 / 750 ms | 0.00028 |
| LLM rewrite + select, 900 ms deadline, 2026-09-12 | 43% | 53% | 553 / 900 ms | 0.00027 |
| LLM rewrite + select, no deadline, 2026-09-16 | 40% | 48% | 1,559 / 4,070 ms | 0.00041 |
| LLM rewrite + select, 900 ms deadline, 2026-09-16 | 29% | 38% | 900 / 900 ms | 0.00041 |
| Jev, same pool | 30% | 69% | 152 / 251 ms | 0.00015 |
| Jev, wide pool | 42% | 66% | 276 / 531 ms | 0.00104 |
| Jev, wide pool, two passes | 46% | 72% | 441 / 709 ms | 0.00123 |
Three readings.
Over the same narrow pool, Jev is cleaner than the LLM (69% against 52%) and covers less (30% against 35%). At a 0.4 threshold it hands over 2.1 items per chat where the LLM hands over 3.1. The LLM fills its budget whether or not the pool deserves it.
The wide pool closes the coverage gap. We did not run the LLM selector over 150 candidates: its prompt would be about five times longer, on a path that already misses its deadline on a slow day. Through Jev the wide pool is two parallel requests and 276 ms.
Per chat, Jev is not the better finder. Against the LLM path with no deadline, the two-pass Jev selector covers more needs in 30 chats, fewer in 29, and ties in 65. The aggregate coverage gap (46% against 40%) is small. The 24-point precision gap is not.
4.3 Latency is the larger story
Figure 6. Time to build the section for one chat start. The two red curves and the two orange curves are the same code against the same cases, four days apart.
On 2026-09-12 the LLM selector's median was 518 ms, and 113 of 124 rewrite-plus-select calls beat the 900 ms deadline. On 2026-09-16 the same code had a median of 1,559 ms, and 8 of 124 beat the deadline. Nothing in our code changed between the two runs. We call the model through a router that chooses among providers, and we did not isolate the cause further.
The current design survives a slow day. A chat that misses the deadline gets the lexical section at once and the model's picks with its first tool result. But on such a day the section the agent starts with is the 29%-at-38% fallback.
Across 248 live wide-pool calls issued one at a time, Jev's median was 226 ms, the 95th percentile 497 ms, the 99th 794 ms and the slowest 1,415 ms. One chat in the two-pass selector took 2,075 ms. We measured Jev on one date, so we cannot yet say how its latency moves from day to day. A deadline is still needed. On the evidence so far it would rarely fire.
Figure 7. Every selector is far under the cost ceiling. The Jev selectors keep their 95th percentile under 1 s.
Jev's wide-pool selectors cost about four times what the LLM calls cost, because they read five times the candidates. The absolute figure stays small. At the usage level of the profile we studied (about 20 chat starts, 13 follow-ups, 6 run reviews, 64 scheduled runs and 18 subagent runs a day) we estimate $0.04 a day for a Jev-first design against $0.035 a day for the LLM design.
4.4 Where the loss at chat start is
Figure 8. Scored offline against the pool oracle. Gray is the wide pool in its own lexical order. Black is a selector that knows the judged needs.
The wide pool holds something that covers 82% of the judged needs. The LLM design reaches 35% to 43%. Finding candidates is no longer the limit on this surface. That also explains why dropping the rewrite call cost nothing: a loose floor plus queries built from the open page, the attachments and the workspace's names does the rewrite's job without a model.
The limit is the section. 3,000 characters fit about three quoted items. A perfect pick of those three would cover 78%. Jev's ordering covers 41%. The pool's lexical order covers 32% at 35% precision.
Figure 9. Offline sweep. At the standard budget, coverage is flat from 0.05 to 0.45 because the character limit, not the threshold, decides what fits. Precision rises steadily. 0.4 is the last point before coverage falls.
The budget is the largest single lever we found, and only a precise judge makes it affordable. Live, at double the budget (12 quotes, 6 mentions, 6,000 characters):
| Double budget, live | Needs covered | Precision | Items per chat |
|---|---|---|---|
| Jev, wide pool | 59% | 60% | 7.1 |
| LLM rewrite + select | 52% | 46% | 8.4 |
The LLM fills the larger section with more noise than signal. Jev's threshold keeps the extra room for candidates it rates. Whether an agent is better served by 1,500 tokens of recall than by 750 is a separate question. The matcher judges items. It does not watch what the agent does with them.
4.5 The probabilities mean what they say
Figure 10. 13,947 chat-start candidates. Left: Jev's probability for candidates the matcher calls noise and useful. Right: for each 10-point bin of Jev's probability, the share the matcher calls useful, with the bin's size.
We calibrated nothing, and the reliability curve lies on the diagonal. Of candidates Jev rated between 0.4 and 0.5, the matcher called 47% useful. Between 0.7 and 0.8, 88%. Within a chat, Jev ranks a useful candidate above a noise candidate 82% of the time (AUC 0.82).
This matters for engineering more than for the score. A threshold on a calibrated probability is a policy a person can read: hand over what is at least 40% likely to be used. An LLM selector offers no such dial. Its only control is the prompt.
4.6 Is the precision gap the matcher's doing?
The matcher grades each selector's items in a separate call, so it could in principle favor one selector's sets. We checked the items that both selectors handed over for the same case.
| Pair | Shared items | Matcher agrees with itself | Jev useful, LLM noise | Jev noise, LLM useful |
|---|---|---|---|---|
| Jev and LLM selector, same pool, unattended | 366 | 95% | 12 | 8 |
| Jev and LLM selector, same pool, chat start | 127 | 91% | 4 | 7 |
| Jev two passes and LLM rewrite + select, chat start | 85 | 93% | 4 | 2 |
The disagreements are few and split evenly. The gap comes from what only one selector picked. On unattended surfaces, items only Jev handed over were 81% useful and items only the LLM handed over were 70% useful. At chat start the figures are 69% and 41%.
4.7 Stability
We ran the wide-pool selector live twice over the same 124 chats. A candidate's probability moved 0.01 on average and 0.04 at the 95th percentile. 1.6% of candidates crossed the 0.4 threshold between runs. The handed items were identical in 87 of 124 chats, with a mean overlap of 0.86. Jev is not deterministic, and the differences sit at the threshold's edge.
5. What did not work
Figure 11. Left: within-chat AUC of each per-candidate question and of combinations. Right: chats the gate would silence at each threshold, split by whether the judge recorded any need.
A Choice to rank the pool. Section 2 shows the collapse on synthetic data. On real pools, putting the Choice's pick at the head of the list lowered AUC from 0.819 to 0.810 and cut precision, because a Choice always names a winner, including in chats where nothing in the pool is useful.
An ensemble of framings. We asked four questions per candidate in the same request: would the agent draw on it, is it about the same specific thing, would the agent otherwise have to ask for it, and a four-level Score. Each alone reaches an AUC between 0.808 and 0.823. Their mean reaches 0.828 and adds 2 points of coverage. They are four views of one signal. The ensemble doubled the tokens, and requests with several hundred questions ran slower (714 to 906 ms on average) and hit rate limits when four cases ran at once. Jev's flat latency holds per request, not for an unbounded question count.
A gate that skips recall. One extra Noul asked whether the work depends on anything from before. It is accurate: AUC 0.90 against "the judge recorded a need". It is still a net loss, because only 12 of 124 chats needed nothing. At a threshold of 0.3 it silences 7 chats rightly and 12 wrongly. The per-candidate threshold already handles chats that need nothing, by keeping nothing.
The second pass is not monotone. Reading the best 24 again with longer excerpts raised chat-start coverage from 42% to 46% and precision from 66% to 72%. Per chat it helped in 27, hurt in 19, and emptied a non-empty pick in 7. Example 6.4 shows how. It earns its 165 ms in aggregate, and it needs a guard we have not built: keep the first pass's picks when the second pass rejects everything.
Writing queries. Jev cannot. At chat start the wide pool made that irrelevant. For scheduled tasks, rewritten queries can be computed once when a task changes and then reused, which costs no latency at run time, so that LLM call has no reason to move.
6. Worked examples
These come from the studied profile. We describe the candidates instead of quoting them, and we left out figures and account details. The probabilities and verdicts are as measured.
6.1 A short question, answered from the wide pool
A person in a marketing workspace, with the workspace's report page open, asks: "where does the data from this chart come from". The judge found ten needs: the report's configuration, the two recurring tasks that collect its data, their data tables, a saved skill for the analytics site, and the analytics sign-in.
| Jev's probability | Matcher | Candidate |
|---|---|---|
| 0.88 | useful | memory: the workspace's data sources (names the analytics service and the admin dashboard) |
| 0.75 | useful | file: an acquisition baseline (new users by source) |
| 0.68 | useful, covers 3 needs | run summary: the weekly reporting task |
| 0.65 | useful | chat: an earlier request to change a chart on the same report |
| 0.56 | useful, covers 2 needs | run summary: the daily traffic collection task |
| 0.47 | noise | workspace update: a daily metrics snapshot |
| 0.46 | noise | goal: an unrelated goal on the same board |
The LLM path handed over four items, all noise: two daily snapshots and two earlier chats about chart styling. They share the words "chart" and "data" with the request. It covered 0 of 10 needs in 1,014 ms. The two-pass Jev selector handed over the first three rows and the daily collection task, covered 6 of 10 with no noise, in 508 ms. The memory that names the data sources was not among the narrow pool's top picks at all. The looser floor is what surfaced it.
6.2 The same pool, a cleaner pick
A delegated subagent starts with a one-line task: launch and measure a specific, named campaign. Both selectors read the same 28 candidates, which include a week of files from a daily idea-proposal task. Those files share a template and most of their vocabulary.
| Picked by | Item | Matcher |
|---|---|---|
| both | file: the proposal that introduced this campaign | useful |
| both | task run: the run that produced that proposal | useful |
| Jev | file: a status note that records the campaign's open question for its owner | useful |
| Jev | evidence record for the goal to launch and measure this campaign | useful |
| LLM | file: the next day's proposal, a different idea from the same task | useful |
| LLM | file: the following day's proposal, an unrelated billing idea | noise |
| LLM | the same evidence record | noise |
Jev covered 7 of 12 needs with no noise in 148 ms. The LLM covered 3 of 12 with two noise items in 553 ms. The LLM followed the template: three consecutive days of the same task's output. Jev followed the subject. The last row is one of the 12 self-disagreements counted in section 4.6: the matcher called the same evidence record useful in one set and noise in the other.
6.3 Failure: the request points at the screen
"can you retract this and add options to it", with a screenshot attached and the workspace's question queue open. The needs are the queued question, the goal it belongs to and the status report behind it.
| Jev's probability | Matcher | Candidate |
|---|---|---|
| 0.69 | noise | file: image.png, a different image saved days earlier |
| 0.59 | noise | file: image.png, another one |
| 0.54 | noise | chat summary that mentions the queue page's address |
| 0.28 | useful, covers 2 needs | goal: the work the question put on hold |
| 0.26 | useful, covers 3 needs | goal: the goal the question belongs to |
The words carry nothing. "This" is in a screenshot a text classifier cannot see, and the strongest lexical signal is the attachment's file name. Jev rated two unrelated images highest and covered 0 of 7 needs. The LLM path covered 5 of 7, because its rewrite call guessed the subject from the page address and searched for that.
This is the clearest case where Aera would lose by switching. It is also fixable in a way that is specific to Aera. The browser knows what is on the page. A short digest of the visible card in the work field would give the classifier the subject. We have not tested that.
6.4 Failure: the second pass talks itself out of it
A person asks for an opinion on which of several case studies, researched the day before, is the one to follow, and adds that nothing needs saving or changing. The first pass rated the right material well: the chat that commissioned the research (0.63), the standing goal (0.61), the research brief (0.51). The second pass read those again with 700-character excerpts, among 5 candidates instead of 75, and returned 0.15, 0.17 and 0.17. Nothing cleared the threshold and the chat was handed nothing, where the single pass had covered 2 of 6 needs.
We do not know why. The longer excerpts are dense research notes, and the request says nothing needs changing, which may read as "no action, so no material needed". What we can say is that Jev's probability for a candidate depends on what else is in the request. A second pass is a different measurement, and it should not overrule the first on its own.
6.5 Failure: an opening written by the system
Some chats are opened by Aera itself with a templated notice, for instance when a run review was interrupted and has to be picked up again. The text is mostly identifiers and instructions. Jev's top 14 candidates all scored between 0.53 and 0.65, with no separation between them, and four noise items went out. The LLM did better by keying on the report's identifier.
We looked for a cheap warning sign and did not find one. A flat profile is not it: in the 40 chats where the 1st and 10th candidates were within 0.15 of each other, the top picks were 75% useful, because a flat profile usually means many good candidates. The better fix is upstream. Aera writes these openings, knows which report they concern, and could pass the report's text as the work.
7. What this means for Aera, surface by surface
| Surface | Benefit from Jev | Evidence |
|---|---|---|
| Chat start | Large. Same or better coverage, 13 to 24 points more precision, 95th percentile under 750 ms, and the deadline race stops being the normal path. | Measured, 124 chats |
| Follow-up messages | Probably large. The deadline is tighter (700 ms) and the one-request selector has a 95th percentile of 251 ms. | Not measured, there are no per-message labels |
| Run reviews | Moderate. Same coverage, precision 84% to 90%, a third of the time, 40% cheaper. Nobody waits, so the gain is a cleaner prompt and lower spend. | Measured, 70 cases |
| Delegated subagents | Moderate. Same coverage, precision 77% to 82%. The parent agent does wait, so 144 ms against 471 ms is felt. | Measured, 124 cases |
| Scheduled runs | Small. Precision 73% to 82%, coverage unchanged at 24% under every selector. The limit is a one-item section and a thin query. | Measured, 82 cases |
| The agent's own memory search | Unknown. Jev could rerank search results inside the 1.5 s allowance. | Not measured |
| Query rewriting | None. Jev cannot write. Not needed at chat start, and reusable per task elsewhere. | Measured indirectly |
| Requests that point at the screen | Negative without page context. | Example 6.3 |
| Openings written by the system | Negative unless the notice's subject is passed as the work. | Example 6.5 |
| Aera connected to a local model | None. Jev is a hosted endpoint with no open weights, so that setup keeps the LLM or lexical selector. | By construction |
Two considerations apply everywhere. Adopting Jev would send candidate excerpts (short passages from a person's earlier chats and records) to a second vendor, one with a no-training policy and no stated retention period. And it would make recall depend on a single hosted model with no substitute, so the LLM selector has to stay available as the fallback.
8. Scope of the measurement
- One profile, several weeks of use, 400 labeled cases. The judge and the matcher are models, and they agree with each other on 79% of items.
- The 0.4 threshold was chosen on the same 124 chat starts it is reported on. The curve is flat from 0.05 to 0.45, which limits the effect, but these are not held-out numbers.
- The matcher judges an item from its title and first 320 characters. It cannot see what an agent does with a section, so it cannot price a longer section against a shorter one.
- The pool oracle saw candidates in groups of 25, so its judgment that one item duplicates another is made within a group.
- Latency was measured from one connection in the United States, directly against both vendors. Jev was measured on one date.
- Jev's price is TypeSafe's published example figure, not a billed amount.
8. Conclusion
A classifier fits this job better than a generator does. Selecting from a pool is a set of independent yes/no judgments, and Jev makes 150 of them in the time an LLM takes to read its prompt, with probabilities that turned out to be calibrated against our matcher with no tuning. For Aera that means a cleaner section on every surface, a chat start whose speed does not depend on which provider served the LLM that morning, and a selector whose one parameter a person can reason about.
It did not find more than the LLM finds, chat for chat. It fails when the subject of a request is on the screen and not in the words, and its second pass can reverse a good first pass. The pool oracle moved the open question elsewhere. Material covering 82% of what chats need is already within reach, and a three-item section stands between that and the 46% we deliver.
Our next steps are to build the Jev path with the LLM selector as its fallback, use the one-request selector for unattended work and the guarded two-pass selector at chat start, and study the section budget separately with an experiment that watches what the agent does, not only which items it was handed.
Sources
- TypeSafe documentation: https://docs.typesafe.ai
- System One API reference: https://docs.typesafe.ai/api
- Choice, Noul and Score question types: https://docs.typesafe.ai/primitives
- Re-ranking cookbook, which gives the price per million tokens: https://docs.typesafe.ai/cookbooks/rerank_typesafe
- Skill suggestion cookbook, the source of the rank-wide-then-read-closely shape: https://docs.typesafe.ai/cookbooks/skill_suggestion
- TypeSafe privacy policy: https://typesafe.ai/legal/privacy-policy