SmartAItomation
“The most useful part was learning to turn a vague complaint into a check we could rerun after a change.”
Engineer perspective to request from a future participant
LIVE ONLINE · TWO UPCOMING COHORTS
AI Evals for
Engineers & PMs
How do you know your AI actually works?
Learn to evaluate and improve LLM applications. Find failure patterns, build evaluations you can trust, and improve accuracy, reliability and cost.
4 weeks21 Nov · €50019 Dec · €1,000EUR
Ion MoșnoiAI engineer & coach · 11+ years in AI/ML
THE PRACTICE
Analyze.
Measure.
Improve.
BUILD WITH EVIDENCE
Know what to fix.
Know whether it helped.
A promising demo is a starting point. Reliable AI takes a way to spot failures, measure them and make better decisions about what to change.
Start with AI fundamentals and build an agent. Then analyze its errors, create synthetic evaluation data and use trustworthy evals to improve it.
Do you catch yourself asking any of these while building AI applications?
- How do I evaluate answers that need human judgment?
- If I change a prompt, how do I know I haven’t broken something else?
- Which failures deserve attention first—and what should I test?
- Where do I start without customers or an existing dataset?
- What should I measure, and which tools will help?
- How can I automate evaluations and trust the results?
Leave with a method you can use on this product—and the next one.
WHAT’S INCLUDED
Support during the course.
A library for the work after it.
01
Live sessions + 10+ hours of office hours
Work through the method together. Bring difficult cases for questions and feedback.
02
Lifetime Discord membership
Keep discussing evaluations with the community after your cohort ends.
03
Recordings + future live cohorts
Revisit course recordings and materials, or join a future edition without another course fee.
04
A 200+ page course reader
Detailed lessons, worked problems, pseudocode, review checklists and answer keys.
05
Six assignments, with solutions
Practice each part of the loop with written solutions and recorded walkthroughs.
06
Completion certificate
Submit four assignments, including your final improvement report, for instructor review.
WHAT YOU’LL LEARN
A way to make the next decision.
01
Find the real failure
Read the full interaction: history, retrieved evidence, tool inputs, outputs and final answer.
02
Measure a specific mistake
Build one evaluator per error type. Check it against domain-expert labels before trusting it.
03
Know whether a change helped
Compare prompts, retrieval and agent flows on the same cases. Track regressions, cost and latency.
04
Test the actions that carry risk
Probe prompt injection, unsafe tool use and missing permission checks. Decide what needs human approval.
05
Catch failures after release
Connect evals to CI, monitor drift and investigate changes in quality, latency and cost.
06
Make the business trade-off visible
Prioritize costly failures and show what improved, which risks remain and what the next experiment should test.
The skill stays with your team.
Use the same review → evaluate → improve workflow on your next AI product.
THE FOUR-WEEK SYLLABUS
Practice first.
Learn what the failure demands.
Four Saturday lectures, 10:00–12:00 (Europe/Chișinău, UTC+2). Build an agent, analyze its errors, create evaluation data and improve it. Each lecture includes a practical lab.
01WEEK 1 · SATURDAYAI fundamentals & build an agentLLM behavior · tools · agent loop · traces
Build it. See what it does.
- Understand tokens, context limits, model behavior and structured outputs.
- Build an agent loop with tool calls, stop conditions and bounded retries.
- Define allowed actions, permissions, clarification and human escalation.
- Record messages, tool inputs and outputs; learn why final answers alone are not enough to evaluate an agent.
YOUR LAB
Build a working agent and inspect its first complete trace.
02WEEK 2 · SATURDAYError analysisTrace review · expert notes · error taxonomy
Find the mistakes worth fixing.
- Build a trace viewer and review examples with a domain expert.
- Label Good/Bad with a specific reason and the step where the error happened.
- Cluster review notes into error types and prioritize by frequency and impact.
- Define one pass/fail check per error; use code for objective rules and judges where interpretation is needed.
YOUR LAB
Review a varied set of traces, create an error taxonomy and write the rules your evals will check.
03WEEK 3 · SATURDAYCreate synthetic evaluation dataFeatures × roles × scenarios · datasets · judge calibration
Create cases that test the real failure modes.
- Generate cases from features, roles and scenarios, including ambiguity, missing data and multi-step requests.
- Review synthetic questions, expected behavior and source evidence with a domain expert.
- Separate train, dev and test cases; check coverage by error type and category.
- Calibrate LLM judges against expert labels using false positives, false negatives, precision and recall.
YOUR LAB
Build a versioned evaluation dataset and test whether your judge catches the mistakes an expert sees.
04WEEK 4 · SATURDAYImprove your AI agentPrompts · models · routing · controlled experiments
Change one thing. Measure what moved.
- Use the error clusters to decide whether to change data, retrieval, prompts, models or tool flow.
- Try better examples, clearer instructions, query rewriting, routing or smaller task scopes.
- Compare changes on the same cases. Measure fixes, regressions, cost and latency.
- Test prompt injection, permissions and unsafe actions; automate evals in CI and monitor production drift.
YOUR LAB
Run a before/after experiment and explain what improved, what regressed and what to try next.
SIX PRACTICAL ASSIGNMENTS
Build the loop,
one useful piece at a time.
- Inspect an agent traceMake tool calls, evidence, cost and failures visible.
- Expert review and an error taxonomyWrite expert notes and build a useful taxonomy.
- Build a representative datasetCover roles, features and conditions without leaking test answers.
- One binary evaluator per errorTurn each error definition into a focused code check or judge.
- Calibrate a judge against expert labelsInvestigate disagreements and measure precision and recall.
- Improve the agent and add a CI gateCompare against a baseline and define release checks.
Every assignment includes a worked solution and a short recorded walkthrough. Use the course project or an authorized, anonymized use case from your work.
CHOOSE YOUR COHORT
Four weeks.
A method you can keep using.
Weekly live sessions and independent practice: plan around 3–5 hours a week. There are also 10+ hours of optional live office hours across the course.
Saturday lectures · 10:00–12:00
Europe/Chișinău (UTC+2). All lectures are recorded. The separate 10+ hours of office hours will be scheduled with the cohort.
WHO THIS IS FOR
Who this course is for
Engineers
You’re building agents or RAG and want a practical way to debug, test and improve them.
Product managers & domain experts
You need to define good behavior, review failures and judge whether a release is ready.
Engineering and product leaders
You need to prioritize failures, explain release decisions and give your team a shared quality process.
Teams building AI skills in-house
Learn a shared method you can reuse across products, instead of starting over with every use case.
Before you join: be comfortable using an LLM. Engineers should be able to run basic Python; PMs can pair for the coding labs. An ML research background is not required.
YOUR INSTRUCTOR
Learn with Ion Moșnoi.
11+ years in AI/ML
I help engineering teams find what is going wrong in their AI products—and build the evaluations to improve them.
My work spans production projects for startups and enterprises, plus research in multi-agent coordination and computer vision in France and Finland.
Experience on LinkedIn“Start with a failure you can explain.
Then build the test that catches it.”
FREE RESOURCES
Start with something
you can use today.
Enter your email to download a resource. Delivery is separate from course-update emails.
PDF · CHECKLIST
The AI evals checklist
What to inspect, label and measure before the next release.
Get free resource ↗CSV + PROMPTSTrace review starter kit
A review sheet, fictional traces and viewer-building prompts.
Get free resource ↗TEMPLATE + WORKSHEETBuild and test an LLM judge
Define a failure, test disagreements and calibrate the judge.
Get free resource ↗INTERACTIVE FLASHCARDSPractice the evals concepts
Short questions that test your understanding of the evaluation loop.
Get free resource ↗CODING-ASSISTANT SKILLGive your assistant an evals workflow
Reusable instructions and templates for your coding assistant.
Get free resource ↗PRACTICAL REFERENCEAI evals questions, answered
Straight answers about datasets, judges, metrics and experiments.
Get free resource ↗VIDEO + INTERACTIVE LESSONHow to know an improvement helped
A short lesson on misleading averages and useful comparisons.
Get free resource ↗Prefer updates about the next cohort?
Get occasional course news. Confirm your email first; unsubscribe whenever you like.
Get course updates ↗TESTIMONIAL COPY · OWNER PREVIEW
Reviews to replace
with participant feedback.
Draft placeholders below are not real student reviews. No ratings, student names or results are being claimed.
“Reviewing examples together gave us a clearer way to explain what a good answer should do and where it was failing.”
Product manager perspective to request from a future participant
“I wanted a process our team could keep using, from reviewing traces to comparing changes on the same cases.”
Engineering lead perspective to request from a future participant
LEARN WITH YOUR TEAM
Build the skill in-house.
A FEW PRACTICAL DETAILS
Before you join.
When does the course run?
Choose 21 November–18 December 2026 (€500) or 19 December 2026–15 January 2027 (€1,000). Both run for four weeks. Lectures are on Saturdays, 10:00–12:00 Europe/Chișinău (UTC+2). November lecture dates: 21 & 28 Nov, 5 & 12 Dec. December lecture dates: 19 & 26 Dec, 2 & 9 Jan. Office-hour times will be arranged separately.
How much time should I set aside?
Plan around 3–5 hours weekly for live sessions and independent practice. Optional office hours add 10+ hours across the course. The assignments include solutions and recorded walkthroughs.
What does lifetime access include?
Keep access to the course recordings, materials and private Discord community. You may also join future live editions of this course without another course fee; register with Ion before each cohort. See the course terms for scope.
What if I cannot attend live?
Course sessions are recorded, with lifetime access. Use Discord for questions and join office hours when you can. Meeting links and recordings are provided to enrolled participants.
Do I need to be an engineer?
Engineers should be comfortable running basic Python. PMs and domain experts can focus on reviews, test cases and release decisions, pairing for coding labs. No ML research background is needed.
Who is this course not for?
It is not a general introduction to using chatbots, a model-training research course or a promise to ship a complete product in four weeks. You should be ready to inspect examples, write review notes and test changes.
Do I need an existing product or customers?
No. Start with the supplied materials. Learn how to build synthetic cases from roles, features and scenarios, then use expert review to check them before trusting the labels.
What will I have by the end?
An inspectable agent, reviewed error taxonomy, versioned evaluation dataset, calibrated evaluators and a before/after improvement report. Your results will depend on your cases and implementation; the goal is a method you can reuse.
Which tools will we use?
Python, JSON/CSV, a small agent/tool loop, code assertions and model-API calls. Use Codex or Claude to help build interfaces and scripts. The method is framework-independent; any optional model-provider usage is billed by your own provider account and is not included in tuition.
Does this cover agents, RAG and security?
Yes. Inspect tool calls and retrieval separately from answer quality. Test permissions, prompt injection, unsafe actions and human escalation, then connect checks to CI and production monitoring.
How do I earn the certificate?
Submit at least four of the six assignments, including the final improvement report. Ion reviews your work before issuing your named certificate. It is a completion certificate, not an accredited qualification.
Can my company pay or enroll a team?
Yes. Use the reimbursement email template and the payment confirmation. Book 2–20 seats together for 10% off; reply to the confirmation with participant details. Contact Ion for larger groups or private delivery.
How is this different from learning alone?
You get a connected practice sequence, expert review, office hours, community discussion and worked solutions. The focus is making your evaluations reliable enough to guide actual engineering decisions.
Is this course affiliated with Maven or an AI provider?
No. This is an independently developed SmartAItomation course taught by Ion Moșnoi. The listed tools and public references do not imply affiliation or endorsement.
BUILD THE FEEDBACK LOOP
Less guessing.
Better next steps.
Four weeks. One practical method you can keep using.
Choose your cohort