Learning roadmap
Interview Prep field notes
The honest AI engineer roadmap: learn systems, not just prompts
Build from Python and software foundations to retrieval, agents, evaluation and production reliability in the right order.
This is a working guide: use the prompts and checkpoints as you make the decision or build the project, not only after it is done.
Begin with software you can reason about
Before advanced AI patterns, become comfortable with Python, APIs, data structures, version control, testing and basic deployment. AI applications still fail because of ordinary software problems: unclear interfaces, broken data flows and missing observability.
Build one small application that accepts input, calls a model, validates the output and records what happened. This simple loop becomes the foundation for everything that follows.
Learn how model behaviour changes the design
A model can produce different outputs for similar inputs, follow an instruction imperfectly or sound confident when evidence is missing. Learn prompting, structured outputs, context limits and basic safety patterns as engineering constraints rather than tricks.
Create a small evaluation set early. Without examples of good and bad behaviour, improvement becomes guesswork.
Build the evaluation set from realistic tasks, not only easy examples. Include normal requests, ambiguous wording, missing evidence and inputs that should trigger a refusal or clarification. Score observable behaviour such as task completion, factual support and format correctness instead of relying on whether an answer merely sounds good.
Keep this in mind
The professional question is not 'Did the demo work?' It is 'How often does it work, where does it fail and how will we notice?'
Add retrieval and tool use for a reason
Retrieval-augmented generation is useful when an application needs trusted, changing or private knowledge. Learn document preparation, chunking, search, citations and answer evaluation before adding complex orchestration.
Use tools or agent loops when the system must take actions, gather information in steps or choose among capabilities. Start with explicit workflows. Add autonomy only where it creates measurable value.
- Retrieval project: answer from a bounded knowledge base and show supporting sources.
- Tool-use project: complete a multi-step workflow with clear permissions and failure states.
- Evaluation project: compare two approaches against a fixed test set and explain the result.
Build the production feedback loop
A useful AI system needs more than an accurate first response. Add structured logging, timeouts, retries and clear fallback behaviour so the application can recover when a model, retrieval step or external tool fails. Record enough context to diagnose a problem without storing sensitive data unnecessarily.
Treat evaluation as a continuing loop rather than a final test. Review failed and low-confidence interactions, turn representative cases into regression tests and rerun them whenever you change the model, prompt, data or workflow. This is how a promising prototype becomes a system that can improve safely.
- Track task success, groundedness, latency and cost at the level that matters to the user.
- Define when the system should retry, ask for clarification, refuse or hand control to a person.
- Version prompts, evaluation sets and retrieval changes so regressions can be traced.
Make reliability visible in your portfolio
A credible project explains the architecture, test cases, evaluation method, latency or cost considerations and known failure modes. This demonstrates that you can move beyond a prototype.
Ship fewer applications with stronger evidence. One grounded assistant with thoughtful evaluation says more about your readiness than several copied chatbot interfaces.
Show one failure investigation from end to end: the request that exposed it, the trace or evaluation result that helped you locate the cause, the change you made and whether the fixed system passed the earlier cases. This makes your engineering process visible and proves that evaluation influenced the design.
06 · Practice lab
75-90 minutesAdd an evaluation layer to a simple AI app
Use any small model-powered application you have built, or outline one, and make its quality measurable.
- 1
Collect ten representative requests, including two ambiguous inputs and two cases that should be refused or escalated.
- 2
Define three observable checks, such as groundedness, format correctness and task completion.
- 3
Run or manually score the current approach, then change one part of the prompt, retrieval or workflow.
- 4
Compare the results and record where the change helped, hurt or made no difference.
Your finished output
A small evaluation sheet and a short engineering note explaining one measured improvement and the remaining failure modes.