The Pragmatic Engineer
On today’s episode of The Pragmatic Engineer, I’m joined by Chip Huyen, a computer scientist, author of the freshly published O’Reilly book AI Engineering, and an expert in applied machine learning. Chip has worked as a researcher at Netflix, was a core developer at NVIDIA (building NeMo, NVIDIA’s GenAI framework), and co-founded Claypot AI. She also taught Machine Learning at Stanford University.
In this conversation, we dive into the evolving field of AI Engineering and explore key insights from Chip’s book, including:
• How AI Engineering differs from Machine Learning Engineering
• Why fine-tuning is usually not a tactic you’ll want (or need) to use
• The spectrum of solutions to customer support problems – some not even involving AI!
• The challenges of LLM evals (evaluations)
• Why project-based learning is valuable—but even better when paired with structured learning
• Exciting potential use cases for AI in education and entertainment
• And more!
—
Brought to by:
• Swarmia — The engineering intelligence platform for modern software organizations https://www.swarmia.com/pragmatic/
• Graphite — The AI developer productivity platform https://gt.dev/pragmatic
• Vanta — Automate compliance and simplify security with Vanta http://vanta.com/pragmatic
—
The Pragmatic Engineer deepdives relevant for this episode:
• Applied AI Software Engineering: RAG https://newsletter.pragmaticengineer.com/p/rag
• How do AI software engineering agents work? https://newsletter.pragmaticengineer.com/p/ai-coding-agents
• AI Tooling for Software Engineers in 2024: Reality Check https://newsletter.pragmaticengineer.com/p/ai-tooling-2024
• IDEs with GenAI features that Software Engineers love https://newsletter.pragmaticengineer.com/p/ide-that-software-engineers-love
—
Where to find Chip Huyen:
• X: https://x.com/chipro
• LinkedIn: https://www.linkedin.com/in/chiphuyen/
• Website: https://huyenchip.com/
Where to find Gergely Orosz:
• X: https://x.com/GergelyOrosz
• LinkedIn: https://www.linkedin.com/in/gergelyorosz/
• Bluesky: https://bsky.app/profile/gergely.pragmaticengineer.com
• Newsletter and blog: https://www.pragmaticengineer.com/
—
In this episode, we cover:
(00:00) Intro
(01:31) A quick overview of AI Engineering
(06:45) How Chip ensured her book stays current amidst the rapid advancements in AI
(11:35) A definition of AI Engineering and how it differs from Machine Learning Engineering
(18:15) Simple first steps in building AI applications
(24:38) An explanation of BM25 (retrieval system)
(25:28) The problems associated with fine-tuning
(29:40) Simple customer support solutions for rolling out AI thoughtfully
(35:29) Chip’s thoughts on staying focused on the problem
(37:04) The challenge in evaluating AI systems
(40:03) Use cases in evaluating AI
(43:09) The importance of prioritizing users’ needs and experience
(48:09) Common mistakes made with Gen AI
(53:57) A case for systematic problem solving
(54:57) Project-based learning vs. structured learning
(1:00:07) Why AI is not the end of engineering
(1:04:56) How AI is helping education and the future use cases we might see
(1:08:58) Rapid fire round
—
See the transcript and other references from the episode at https://newsletter.pragmaticengineer.com/podcast
—
Production and marketing by https://penname.co/. For inquiries about sponsoring the podcast, email podcast@pragmaticengineer.com.
Source
Very interesting and good explanations – it helps enlarging my toolbox and insights
Chip is amazing and so clear in her explanations.
Bought this book. Got it delivered today. Excited to see over the weekend.
excited for this podcast
This is gold!
so proud of you Chip Huyen
This was such an informative video on AI engineering!
Very grateful for this talk.
Hey folks,
Can someone help me in deciding between MLOps and GenAI careers after having 5+ years of experience as a data scientist?
Chip!!! 💥🔥🔥🔥👌🏿
Chip explained really well here! I have her book and need to read it now.
Just wow!! I have learned so much from this talk. Thank you both!
@Chip Huyen, everything is impressive– the hardwork and smartness comes through! Lots to learn from your work!
One thing though… The way you pronounce the word "focus" doesnt quite sound right!
The Hell with that world 👈
Great interview. Super appreciate the chapter breakdown.
Great book.
"Old school, 20 plus years" – Oh child 😆
Too many people are trying to find a problem to apply genAI to rather than focusing understanding why a problem exists in the first place.
Chip you are awesome! I've been a fan of your blog so it was cool to see you also thrive in an interview format.
Very very good content! Congrats. 👍👍👍👍👍👍👍👍
AI is good in answering questions, so the questions themselves seem to be our job in the future.
I am impressed! I need to order the book!
Here's a *summary of the key insights from the transcript of "AI Engineering with Chip Huyen"**, presented as **10 bullet points with timestamps* for reference:
—
### 🔹 Summary of Key Points:
1. *Definition and Shift in AI Engineering* (00:00–01:00)
* AI Engineering today is more about engineering and product design than building ML models from scratch.
* APIs make it easier—no need for proprietary data or a deep ML background.
2. *Why a New Term – AI Engineering vs ML Engineering* (12:00–14:15)
* Traditional ML Engineering focused on collecting data, training models, then building products.
* AI Engineering starts with a product idea, uses APIs, and only later collects data or fine-tunes if needed.
3. *Step-by-Step to Building AI Applications* (18:49–25:00)
* Start with prompting → RAG (Retrieval-Augmented Generation) → chunking → hybrid search → then consider fine-tuning as a last resort.
* Data prep (e.g. metadata, summaries) often improves performance more than jumping to vector databases.
4. *Evaluation is Hard but Critical* (37:14–43:00)
* Evaluation gets harder as AI gets smarter. Use:
* Functional correctness (e.g., does code run?)
* LLM-as-a-judge (AI evaluates AI)
* Comparative evaluation (A/B testing responses)
5. *Common Mistakes in AI Product Development* (48:00–54:00)
* Using GenAI when a simpler solution works.
* Giving up too early without diagnosing failures (e.g., not breaking down the pipeline).
* Overengineering early (e.g., fine-tuning too soon, over-relying on immature frameworks).
6. *Advice for Software Engineers Getting Started* (55:08–59:14)
* Combine project-based and structured learning.
* Don’t rely only on tutorials—ask why something works.
* Observe your workflow: which tasks could AI automate?
7. *AI Will Not Replace Software Engineers* (1:00:14–1:04:21)
* Just as writing evolved from handwriting to typing, coding will evolve.
* Software engineering is problem-solving, not just typing code.
* Precision and domain knowledge still matter deeply.
8. *Exciting Future Use Cases Beyond Coding* (1:04:52–1:08:00)
* Education: Helping people learn better by assisting in asking good questions.
* Entertainment: Games or media that are both fun and educational.
* Enterprise: AI could streamline middle management by aggregating and routing information efficiently.
9. *Favorite Tools and Approaches* (1:09:14–1:11:05)
* Uses Python and JavaScript.
* Builds personal AI tools for research (e.g., summarizing academic links).
* Emphasizes "scratch-your-own-itch" utility tools.
10. *Recommended Books and Final Thoughts* (1:11:39–1:14:00)
* Suggested reads: *Complex Adaptive Systems*, *Selfish Gene*, and *Antifragile*.
* Encourages feedback and deeper thinking, not surface-level news chasing.
great content. Building though is the best way to learn. Just get started with n8n or a no code solution and scale up.
You can have 2 headlines-I use GenAI or I solve the problem XD-what a bombshell from Chip!😂😂 34:20
loved this!
Thanks 🙏
Great session.
Now I know why Chips' book "AI Engineering" will become for AI, what "The Psychology of Money" did for Money. You will know what I mean once you read the later, and hear this session, and later you read her Book. It will become a Timeless piece.
It's so important to be more Pragmatic when it comes to deciding things.
At the end she said, why it's so important to know HOW to ASK, WHAT to ASK, WHY one should ASK, and most importantly from WHERE that ASK is coming, is such a critical SKILL to have.
[PS: I kept editing the comment as I was listening to the podcast]
#decisionmaking #knowingly
She is the real life female yoda
Chip's book is excellent and she is truly an expert in the field ….
Why she get so smart?
Every time she goes "hmm" she almost always comes up with several zen-like statements!
Amazing interview thanks for all the wisdom
She’s on 220v
Not only is she a technical expert but she has a very grounded approach to problem solving and working amongst human dynamics.
Chip has a great brain which is a chip in machine language 😂
This is garbage. Just wordy rubbish.
To my eye at least Chip's lighting and camera setup seem to put yours to shame. I wonder what she is using there?
I’m trying to break into the AI engineering field and I see Chip’s resources everywhere. I’m glad I came across her content. And I hope I break into the field well
Excellent discussion! Seems like this book would be a great choice for software engineers who have only surface-level knowledge of AI or AI Engineering but are eager to gain deeper insights into the field. After watching this conversation, you'd definitely want to pick up the book.
✅Shared this informative video in my
🅰️I upskilling groups ( 1070+ members ) for wider reach
🎯💎🏆 Great insightful & fruitful video 🏆💎🎯
लोकः समस्ताः सुखिनो भवन्तु
( May all beings lead prosperous life across Globe 🌍 )