TrueArq Scout: a voice AI assessment that families actually use
Scout is the voice AI agent inside Compass, TrueArq's first product. It talks with students and parents about interests, motivations and aspirations, so a human coach can guide the family's next education or career decision. I lead it as AI Product Lead.
- Live in production
- Used by 100+ families
- AI Product Lead
The 60-second version
Voice captures context people struggle to write down
Writing about interests, motivations or family expectations takes effort. Our hypothesis is that speaking lowers that effort. Scout holds the assessment conversation, and its insights are turned into a report that a coach reviews before guiding the family.
The hard part is not the transcription. It is making sure each answer is understood well enough to inform guidance, without making the conversation tiring.
- Status
- Live in production at truearq.com
- Reach
- Used by 100+ families
- My role
- AI Product Lead, TrueArq
- Users
- Students and their parents
- Stack
- ElevenLabs Custom mode with GPT-4.1 Mini
01 · The problem
Big decisions, thin context
Students and parents face education and career choices with a lot riding on them. A coach can only guide well with a real understanding of the person, but collecting that context takes up the time that should go into the conversation that matters.
Context is also easy to misread. A parent saying “my child is good at sports” might mean enjoyment, consistent practice, competitive results or their own hopes. The student may see it differently, and that difference is worth exploring, not averaging away.
Scout gathers this context through conversation, so coaches can spend their time on differences in perspective, the options and the decision itself.
02 · My role
What I own as AI Product Lead
- 01
Product direction
Roadmap and priorities for Scout within Compass, from discovery with students and parents through launch and iteration.
- 02
Conversation design
Scout's prompts, guardrails and follow-up logic: what it asks, when it probes further and when it moves on.
- 03
Voice architecture choices
Choosing and tuning the components behind the voice experience, and owning the trade-offs that come with that control.
- 04
AI quality and evaluation
Defining what good looks like, building the evals and keeping them current as live usage shows us new failure modes.
03 · How it works
From a spoken answer to a coach's guidance
Conversation
Scout talks with the student and the parent about interests, motivations, aspirations and perspectives.
Clarification
It follows up when an answer is brief or ambiguous, looking for an example or the reason behind a statement.
Synthesis
Insights become a report that separates what was said from what was inferred, and keeps parent and student views distinct.
Coach review
A human coach reviews the report and uses it to guide the family. The AI informs the coach; it does not replace them.
Under the hood
Scout runs on ElevenLabs in Custom mode with GPT-4.1 Mini. Speech recognition, the conversation model, speech synthesis and turn-taking are separate components. ElevenLabs orchestrates them, and we configure Scout's behaviour and choose the components. That separation matters when something goes wrong: a mis-transcribed answer, a pause read as the end of a thought and a leading question each need a different fix.
04 · Evaluating AI quality
Allow flexibility in the conversation, but be strict about evidence
Scout can take a different path through the same situation on different runs. The question is which variation is acceptable. Different phrasing is fine. An unsupported inference about a student is not.
What we measure
- Evidence fidelity. Are interpretations supported by what the person actually said?
- Coverage and clarification. Are the important dimensions explored, without leading or repetitive questions?
- Conversation quality. Premature interruptions, response delays, completion and user effort.
- Coach usefulness and cost. Does the evidence support guidance, at a sustainable cost per usable assessment?
How we test
- Repeated conversational simulations, because one passing run only shows the agent can meet the criteria, not that it does so consistently
- Evals revised around brief and ambiguous answers, after seeing them in live usage
- Live audio testing for recognition, pauses, interruptions and what people actually hear
- Coach review as a standing quality check on every report
05 · Learning from live usage
Three lessons from running Scout with real families
- Every follow-up spends the user's attention
- Some parents and students gave one-word answers, so we revised prompts, guardrails and evals to encourage elaboration gently. Length is a poor measure of usefulness, though. The rule we are working toward: ask another question only when the missing information could change the interpretation or the guidance.
- Managed voice infrastructure still leaves the assessment decisions with us
- A different voice will not fix weak questioning, and a more capable model will not fix premature turn-taking. Choosing components means taking responsibility for how they interact, and for assessment quality whichever mode we use.
- A probabilistic conversation needs explicit evidence requirements
- We separate what someone stated from what the system inferred, keep corrections, leave unresolved points marked as uncertain, and keep parent and student perspectives distinguishable.
06 · Deliberate choices
What I chose not to build
No guidance without a coach
Scout gathers and structures evidence. A human coach reviews the report and guides the family, and this stays a core part of the product.
No optimising for longer answers
Longer answers are easy to measure and easy to chase. We probe only when the missing detail matters, because every extra question costs effort.
No single blended family view
Parent and student perspectives stay separate. Agreement is never assumed, because the gap between them is often the most useful thing a coach can explore.
No platform switch on opinion
Our current setup is a working choice, not a proven best. Alternatives such as ElevenLabs Realtime, Vapi, OpenAI Realtime and Gemini Live are on a comparison agenda, to be judged against the same measures.
See Scout where families use it
Scout is part of Compass at TrueArq. I am also happy to walk through the evaluation approach and the trade-offs in more depth.