Built for a financial technology and workforce platform. The full assignment as one working system — six agents grounded in a shared context layer, two of them running live in mock mode. Every tab maps to a required part. The guardrails you'll see firing are real code, not scripted text.
Of every agent in the marketing stack, I built this one because it addresses the highest-risk failure mode in marketing AI: hallucinated or outdated content reaching publication.
The Content Context Validator is the bridge between the context layer (Part 2) and content production. It guarantees every piece of content is grounded in current, accurate context before it ever reaches a human reviewer — no free-floating generation, every claim traced to a source chunk, stale context flagged on sight.
Sonnet 5 brings stronger reasoning, structured-prompt adherence, 1M context, and caching. Flash handles validation — source-citation, recency, and brand-voice checks are classification, not generation, at $0.14/M vs $3.00/M input.
The static ~1,500-token block (system prompt, product context, brand voice) is cached: $0.30/M cache read vs $3.00/M regular input — 90% off reused tokens. Cache write is $3.75/M, amortized across all reads in the TTL.
Forcing structured output makes the model cite sources explicitly and makes validation programmatically checkable — no source, no claim survives the gate.
If no relevant chunks are found, the agent refuses to generate rather than filling the gap from training data.
For the assignment I used keyword matching for simplicity and transparency. Production swaps in Supabase pgvector or Pinecone for semantic retrieval.
The model reports its own confidence, then the validation layer independently verifies claims and adjusts the score up or down.
The parts that only surface once you actually build it. Each of these came up during the build — and each has a fix already in place or specced for production.
Claude sometimes wraps JSON in markdown code fences or adds a preamble line before the object.
FixRegex-extract the JSON block before parsing, with a fallback to raw text if the parse still fails — so a formatting quirk never crashes the run.
The model occasionally makes a claim without citing a source, or cites one that doesn't fully support the claim.
FixThe validation layer flags missing citations; in production a second Flash pass verifies each claim against its cited source before it clears review.
A keyword check catches obvious violations ("cheap", "simple") but misses subtler tone drift.
FixProduction replaces the keyword pass with a fine-tuned classifier or an LLM-based brand-voice check scored against a rubric.
With the 5-min TTL, infrequent requests (>5 min apart) fall out of cache and cost more.
FixFor low-frequency agents, accept non-cached cost or extend the TTL; for high-frequency ones (SDR outreach at 300 runs/day) caching is highly effective as-is.
| Time | Agent | Model | Conf. | Flags | Routing | Latency | Cost |
|---|---|---|---|---|---|---|---|
No runs yet — every agent run appends a row here: latency, cost, quality score, error rate. | |||||||
| Account | Signal | Business need | Evidence | Score | Status |
|---|
| Time | Agent | Model | Conf. | Flags | Routing | Latency | Cost |
|---|---|---|---|---|---|---|---|
No runs yet — every agent run appends a row here: latency, cost, quality score, error rate. | |||||||
Six stages, six agents, one shared substrate. Click a stage to see what the agent does, what it needs, how it fails, and where it should go silent instead of guessing.
Generates blog posts, LinkedIn posts, nurture emails, and ad copy — grounded in current positioning, case studies, battle cards, and brand voice, not just "write me a post about payroll."
Content team shifts from writing to editing. Output per writer up 3–5x, quality holds because generation comes from the context layer, not training data.
Takes a goal ("500 SQLs from enterprise fintech in Q3") and generates a full brief: audience, messaging pillars, channel mix, assets, sequence design, metrics, budget allocation.
Campaign planning compresses from days to hours. Cross-functional alignment improves — the brief pulls from the same context layer content and SDR teams use.
Monitors mentions of the brand, competitors, and relevant industry conversation. Drafts responses for review; flags high-priority conversations for immediate human attention.
Social moves from reactive monitoring to proactive engagement. Response time on flagged conversations drops from hours to minutes; coverage goes from manual to 24/7.
Monitors the marketing PM system (Asana/Linear), spots bottlenecks and missed dependencies, reassigns tasks by capacity, alerts before deadlines slip.
Marketing ops shifts from manual project tracking to exception-based management — the team works, instead of producing status updates.
One standardized call for every agent — query_context(topic, type, recency_filter, limit). Frequently used context (brand voice, ICP) is cached per agent and invalidates on version change. Every query is logged: which agent, what query, which chunks came back, when — so a bad output is traceable to the exact context it ran on.
A Retool-style UI for search, edit-with-versioning, pending-approval review, and audit trail. Plus a Slack path: updates notify the owning channel (#product-marketing, #competitive-intel), and anyone can run /context query "..." straight from Slack.
Every change logs who / what / when / why. If a context update tanks agent quality (caught by monitoring), Marketing Ops rolls back to the previous version in one click — every agent gets the rollback on its next query.
last_updated — past the threshold, the output is flagged "stale context." If two chunks with similarity > 0.8 disagree, the agent gets a conflict_flag and must acknowledge the conflict rather than pick a side. And structurally: no agent generates from zero chunks, every factual claim needs a citation, and context version ships in the output metadata for audit.
An outbound intelligence agent runs ~300 times/day across the SDR team, pulling context from Salesforce, Gong, ZoomInfo, and Sumble on every run. After two weeks: costs are 3x estimate, 70% of outputs are good, 20% acceptable, 10% confidently wrong — two SDRs already sent factually incorrect outreach.
Pulling large payloads from all four sources on every run — likely 50K tokens when 5K is relevant.
300 runs × 50K tokens × $3.00/M = $45/day on input alone; at 5K tokens that's $4.50/day — 10x waste.
Signal classification, role matching, and scoring don't need Sonnet-level reasoning — DeepSeek V4 Flash is 21x cheaper on input, 53x on output.
If 60% of runs could move to Flash, that's ~20x cost reduction on those runs alone.
System prompt + product context (~2,000 tokens) re-sent at full price every run instead of using Sonnet 5's cache-read pricing.
Uncached: $1.80/day on system prompt alone. Cached: $0.18/day. 90% savings on that slice.
All available context passes through unfiltered — wasting tokens, distracting the model with noise, and letting contradictory chunks confuse generation.
The 10% confidently-wrong outputs likely trace to the model latching onto irrelevant or contradictory chunks.
Every output is treated equally — there's no gate between generation and SDR delivery.
"Two SDRs already sent factually incorrect outreach" — outputs are reaching SDRs with zero review layer.
Prompts are likely verbose and unstructured, with redundant instructions burning input tokens on every single run.
A 3x cost overrun against estimate is consistent with prompt + context token count running well above plan.
Cut the system prompt from a sprawling ~2,000-token brief to ~200 tokens: role, retrieved context slots, a capped output schema, and one grounding rule — no evidence, no claim.
System prompt -90%, context -96% (50K → 2K), output capped -60%. ~85% total token reduction per run.
Semantic retrieval with relevance scoring: embed the query, pull top 5 chunks, filter by relevance >0.75 and recency, cap at 2K tokens. Summarize Gong transcripts and Salesforce history instead of passing full logs.
Input tokens 50K → 2K (96% reduction) and a quality gain — the model works from signal, not noise.
Route by task type and complexity: classification/scoring/validation → DeepSeek V4 Flash; high-complexity enterprise outreach → Claude Sonnet 5; standard personalization → Flash.
~60% of runs move to Flash at ~30x lower cost — roughly 10x blended cost reduction on its own.
Cache the static ~2,000-token block (system prompt, product context, brand voice, ICP) at 5-min TTL; only the ~500-token dynamic block is fresh per run.
90% savings on cached tokens; break-even at 2 cache reads. At 300 runs/day: ~$1.40/day in cache savings alone.
Layer 1 (free): schema, length, and brand-voice checks. Layer 2 (DeepSeek V4 Flash, ~$0.0001/validation): claim verification, hallucination check, independent confidence score.
Confidently-wrong outputs drop from 10% toward <1% — caught before they reach an SDR, not after.
>0.85 + validation pass → fast-track. 0.6–0.85 → standard review. <0.6 or validation fail → regenerate stricter; fail twice → human co-drafting. Every claim needs cited evidence or it's stripped from the draft.
SDR-facing failures drop ~90% (10% → <1%) while fast-track keeps throughput high for the 90%+ good outputs.
| Metric | Before | After | Change |
|---|---|---|---|
| Input tokens / run | ~50,000 | ~2,500 (2K cached + 500 fresh) | -95% |
| Output tokens / run | ~2,000 | ~500 (capped) | -75% |
| Model mix | Sonnet 5, all runs | Sonnet 5 (40%) + Flash (60%) | — |
| Cost per run (blended) | $0.18 | $0.0042 | -98% |
| Daily cost (300 runs) | $54 | $1.26 | -98% |
| Monthly cost (22 days) | $1,188 | $27.72 | -98% |
| Output quality (good) | 70% | 90%+ | +20pp |
| Confidently wrong | 10% | <1% | -9pp |
| SDR-facing failures | 10% (~30/day) | <1% (<3/day) | -90% |
| Lever | Cost reduction | Implementation effort |
|---|---|---|
| Context filtering (50K → 2K) | 96% on input | Medium — vector store + retrieval |
| Model routing (Sonnet → Flash, 60%) | ~20x on those runs | Low — routing logic |
| Prompt caching (Sonnet 5) | 90% on cached tokens | Low — API parameter |
| Prompt optimization (2K → 200 sys prompt) | 90% on system prompt | Low — prompt rewrite |
| Output capping (2K → 500) | 75% on output | Low — max_tokens |
All six operational: content generates from the context layer with human review on high-risk pieces and auto-publish on low-risk; campaign briefs compress planning from days to hours; social moves from reactive to proactive with 24/7 coverage; marketing ops runs on exception-based management instead of manual tracking. Every agent run is tracked for latency, cost, quality, and cache hit rate, reviewed weekly with the Marketing Ops Lead.
Product changes, the context layer doesn't — agents produce outdated content.
Staleness alerts on version control, a release webhook that triggers the update workflow, and clear single-owner assignment per domain (Product Marketing / Competitive Intel / Brand Lead). No ownership, no update.
The team either over-edits (defeating the purpose) or under-reviews (letting bad content through) because they don't trust AI output yet.
Start on low-risk use cases to build trust, train the team to work with their agent rather than against it, and let SDR/team feedback shape prompt iteration.
Output drifts from brand voice over time, especially as prompts get iterated on.
Brand voice lives in the context layer, not buried in the prompt. A validation layer checks every output against it; weekly audits sample 10 outputs per agent against the rubric.
"How much pipeline did the agent generate?" is hard to answer when agents support humans rather than replace them.
Measure time saved, output increased, quality maintained (edit/rejection rate), and cost per output. Correlate with pipeline metrics without claiming direct attribution — the ROI is leverage, not replacement.
As content production scales, token costs can grow in ways that are easy to miss until the invoice arrives.
Caching, model routing, output capping, and weekly cost review (cost per piece, cache hit rate, model mix) — investigate anything that moves >10% week over week.
The full Part 1 stack in production, plus a Pre-Call Intelligence Agent that briefs SDRs/AEs before calls — account history, contact roles, recent signals, talking points, objection handlers — pulled from the context layer, Salesforce, and Gong with prompt caching on product context.
SDRs have seen obviously-AI outreach before and don't want to send it.
Start with the Coach Agent, not Outreach — show SDRs what's working before asking them to trust AI-generated content. Outreach Agent then arrives as a drafting tool, never an auto-sender. Trust is earned through quality, not mandated.
Salesforce, Gong, ZoomInfo, and Sumble each have different models, freshness, and quality — agents inherit the weakest link.
Every agent has defined degradation behavior (Part 1): confidence adjusts and outputs flag when a source is stale or down. A Marketing Ops dashboard shows source health in real time.
300 SDR runs/day is the start — costs can spiral as agents extend to AEs and CS.
Model routing, context filtering, token caps, and cost-anomaly alerts. The Part 4 remediation plan is the template, applied proactively rather than reactively.
Marketing builds the agents SDRs use, but SDRs report to Sales — who owns what, and who prioritizes improvements?
Marketing owns the infrastructure, Sales owns the outcomes. Shared metrics (pipeline contribution, meetings booked, reply rate) and joint sign-off on changes to SDR-facing agents.
An SDR sends outreach with a wrong claim — the prospect loses trust, the SDR loses credibility.
Evidence-based personalization: every claim cites a source, no source means no claim. Low-confidence output never reaches SDRs without human co-drafting. AI drafts, humans approve — non-negotiable.
Prompt caching has a 5-minute TTL — SDRs across time zones running agents at different times increases cache misses.
Territory-based cache warming at the start of each territory's business hours; monitor hit rate by territory and adjust the warming schedule if it drops below 80%.
This isn't "build AI for marketing" and "build AI for sales" as separate tracks. It's building AI infrastructure that serves both, grounded in one shared context layer. Marketing content and SDR outreach draw from the same positioning, the same case studies, the same battle cards. When product changes, both domains update together — because they share the substrate.
The 12-month vision isn't "lots of agents" — it's a coherent system where agents handle the data layer, humans handle the relationship layer, and the context layer keeps everything grounded and current. AI does the research; humans do the conversation. That's how genuine relationships scale to enterprise size.
Model routing, prompt caching, context filtering, and output capping are designed in from day one — the Part 4 remediation applied proactively. Result: ~98% cost reduction while quality improves, because the model focuses on relevant context, validation catches errors, and confidence-based routing keeps bad output away from humans.