Prompt engineering got you in the door. Agent engineering is what keeps you there.
Last quarter, three different teams we spoke with shipped AI agents that worked beautifully in staging and broke within a week of going live, for essentially the same reason. None of them had a prompting problem. All of them had an engineering problem.
That pattern is everywhere right now. Lots of teams launch an AI agent. Far fewer have one that is still working well six months later. The difference usually isn't talent; it's discipline. A demo agent is a weekend project. A production agent is an engineering practice, and that practice is still being figured out.
Here are the seven habits we see in teams that ship agents that last. None of them is about clever prompt wording.
The 7 Disciplines at a Glance
Think in Systems, Not Prompts — the LLM is one component, not the whole product
Treat Tool Schemas as Contracts — vague interfaces invite hallucinations
Engineer Retrieval Like a Data Pipeline — bad retrieval is a silent failure
Build for the Real World, Not the Demo — production breaks in ways demos never do
Treat Security as a First-Class Concern — every input channel is an attack surface
Instrument Everything You Ship — you can''t fix what you can''t see
Design for the Human on the Other End — technical correctness isn''t enough
1. Think in Systems, Not Prompts
When an agent misbehaves, the instinct is to fix the prompt. Reword it. Add a rule. Try a bigger model. It feels productive, and it rarely sticks. This is the core challenge of llm system design: the real problem is usually somewhere else.
A tool returned bad data, the retriever pulled the wrong document, or the workflow didn't handle a partial failure. Staring at the prompt is like debugging an entire system by reading a single log file.
Treat the LLM as one part of a larger system. Ask the same questions you'd ask of any software: Where are the boundaries? What's the contract at each one? Where do I want strict logic, and where do I want the model to use judgment? Tune the prompt last, not first. Effective agent engineering treats the interface as a component, not the magic bullet.
A Plain-English Example: Bad Prompt vs. Good Prompt
Same agent. Same task — summarize a customer support ticket and route it to the right team. The only thing that changes is whether the prompt was treated as a system contract or a wish.
❌ Bad Prompt (Demo-Grade)
"You are a helpful assistant. Read the support ticket and figure out what to do."
Why it breaks in production: No definition of "what to do." No list of valid teams. No format for the output. No rule for what to do when the ticket is unclear. The model will improvise — sometimes it returns a paragraph, sometimes JSON, sometimes a team name that doesn't exist. Downstream systems can't depend on any of it.
✅ Good Prompt (Production-Grade)
"You are a support-routing agent. Read the ticket below and return only a JSON object with these fields:
{ "summary": string (max 200 chars), "category": one of ["billing", "technical", "account", "other"], "priority": one of ["low", "medium", "high"], "confidence": number 0–1 }
If the ticket is ambiguous, setcategoryto"other"andconfidencebelow 0.5. Do not invent fields. Do not add commentary."
Why it holds up: The output shape is fixed, the allowed values are enumerated, ambiguity has a defined fallback, and confidence is surfaced so downstream code can decide when a human should review. The prompt isn't "cleverer" — it's engineered. That is the shift Section 1 is about: stop writing wishes, start writing contracts to ensure ai system reliability.
2. Treat Tool Schemas as Contracts
Every tool you give an agent is an interface. If the interface is vague, the model will improvise — and improvisation is where hallucinations come from. When building production ai agents, you cannot leave the schema to chance.
A tool that accepts a generic "query" with no other details will return something string-shaped. Sometimes it's SQL. Sometimes it's a sentence. Sometimes it's a parameter the model invented because it sounded right.
Tight schemas fix this. Use enums, required fields, specific types, and clear error messages. Good schemas tell the model both what it can do and what it shouldn't try. Teams that take schema design seriously file far fewer hallucination bugs — not because their prompts are better, but because there's less room for the model to drift.
A Plain-English Example: Bad Tool Schema vs. Good Tool Schema
Same agent. Same job — process a customer refund. The only thing that changes is how tightly the tool's interface is defined.
❌ Bad Schema (Demo-Grade)
process_refund(details: string)
Description: "Processes a refund for a customer."
Why it breaks in production: "details" could be anything. The model might pass "refund John $50", or "order #12345, full refund", or a polite paragraph. There's no order ID field, no amount field, no currency, no reason code, and no rule for partial refunds. The tool has to parse free text, and when parsing fails, the model often invents a plausible order number. That's a hallucination with a financial blast radius.
✅ Good Schema (Production-Grade)
process_refund(
order_id: string (required, format: "ORD-" + 8 digits),
amount_cents: integer (required, min: 1, max: order.total_cents),
currency: enum ["USD", "EUR", "GBP"] (required),
reason_code: enum ["defective", "wrong_item", "customer_request", "fraud", "other"] (required),
notes: string (optional, max 500 chars)
)Description: "Issues a refund against an existing order. Use only when the order has been verified via
lookup_orderfirst. Returns an error if the amount exceeds the order total or if the order is already fully refunded."
Why it holds up: The model can't invent an order ID, and the format is checked. It can't refund more than the order total; the bound is enforced. It can't pick a creative reason; only five values are allowed. And the description tells the model when not to call the tool at all. The agent didn't get smarter; the surface area for mistakes got smaller. That is what "schemas as contracts" means in practice.
3. Engineer Retrieval Like a Data Pipeline
RAG looks simple on a slide: query in, documents out, model answers. The slide hides where the failures actually live. To scale, you must treat your search infrastructure as a formal rag data pipeline.
Chunk too small and you split the idea the model needs. Chunk too big and you waste the context window on noise. Pick the wrong embedding model, and the similarity scores reward matching words rather than matching meaning. Skip re-ranking, and the model sees an available answer instead of the right one.
The hard part is that bad retrieval rarely throws an error. It produces a fluent, confident, wrong answer — the worst kind of failure, because nobody notices until a customer does. Treat retrieval like any other data pipeline: version it, evaluate it, monitor it, and give it an owner.
A Plain-English Example: Bad Data Contract vs. Good Data Contract
A retrieval pipeline is only as trustworthy as the contract between the team that produces the data and the team (or agent) that consumes it. Same source — a "customer orders" feed — but the level of discipline changes everything downstream.
❌ Bad Data Contract (Implicit, Untyped, Unversioned)
"We'll drop a JSON file in the shared bucket every night. It has the orders."
Sample payload:
{
"id": "12345",
"cust": "Jane D.",
"amt": "49.99",
"date": "11/3/25",
"status": "ok"
}
Why does it break in production? Is id it a string or a number? Is amt it dollars or cents? Is the date MM/DD or DD/MM? What does status: "ok" — paid, shipped, refund-eligible? When the upstream team renames amt to total Next quarter, every downstream agent silently starts returning empty results — or worse, hallucinating values that "look right." There's no schema, no version, no owner, and no way to detect drift before it reaches a customer.
✅ Good Data Contract (Explicit, Typed, Versioned, Owned)
contract: orders.v2
owner: commerce-platform-team
sla: refreshed daily by 02:00 UTC, freshness alert if > 26h
schema:
order_id: string (required, format: "ORD-" + 8 digits, primary key)
customer_id: string (required, foreign key → customers.v3.customer_id)
total_cents: integer (required, min: 0)
currency: enum ["USD", "EUR", "GBP"] (required)
placed_at: timestamp (required, ISO 8601, UTC)
status: enum ["pending", "paid", "shipped", "delivered", "refunded", "cancelled"] (required)
pii_fields: ["customer_id"]
breaking_changes: require v3 bump + 30-day deprecation notice
quality_checks:
- order_id uniqueness >= 99.99%
- total_cents null rate < 0.01%
- status enum violations = 0
Why it holds up: Every field has a type, a unit, and an allowed range. The contract is versioned, so the consuming agent pins to it and won't silently break when v3 ships. There's a named owner, a freshness SLA, and automated quality checks that fail the rag data pipeline before bad data reaches the agent. PII is labeled, enabling row-level filter retrieval and enforcement. The agent isn't guessing what the data means — the contract tells it. That is the difference between a retrieval pipeline you can put in front of a customer and one you can only demo.
Building the Retrieval Pipeline in Apache Airflow: The Steps to Consider
A data contract only matters if something enforces it on every run. In a production retrieval pipeline, that "something" is a scheduled workflow. Below is what a contract-enforcing pipeline looks like in Apache Airflow (open-source) — no code, just the steps every team should consider and the discipline behind each one.
Take a common case: an agent answering customer questions about internal policy documents. Every night, Airflow runs a DAG ("directed acyclic graph"—Airflow's term for a workflow) that moves data from raw sources to a trusted index. Each task is a checkpoint where the contract is honored — or the pipeline stops.
Extract — Pull the source documents. Airflow connects to the document source (SharePoint, Confluence, an S3 bucket, a database) and downloads anything new or changed since the last run. Only deltas, not the whole library, to keep the job fast.
Validate — Check the files before doing anything expensive. Confirm each file opens, isn't empty, isn't corrupted, and matches an allowed file type. Bad files are quarantined and reported, not silently skipped. This single step prevents the most common "why is the agent suddenly wrong?" incident.
Transform — Clean and chunk the text. Strip boilerplate (headers, footers, navigation). Split each