What Works Now

Reliability improves when the domain is defined and outputs are constrained. The most dependable capabilities are:

• Moment finding for a known task or format
• Summaries and highlights with timestamps
• Policy screening that escalates uncertain cases
• Search across a curated video collection

Application Patterns

Meeting and training intelligence

The best results come from combining transcripts with visual cues like screen changes, slides, and gestures. The output should be a short recap, clear actions, and a timeline of key moments. Treat this as a navigation tool, not a full replacement for watching the video.

Content review and safety

Use multiple signals instead of one score. Frame sampling, audio analysis, and scene context should all contribute to the final decision. Keep a clear path for human review, especially for borderline cases or sensitive content.

Video knowledge bases

Segment videos into stable chunks and index each segment with its transcript and visual context. Retrieval works best when users can jump directly to a moment, not just a file. This turns training libraries, product demos, and webinars into searchable references.

Editing assistance

AI can speed up rough cuts, captions, and highlight reels. It is less reliable for long-form generation or complex narrative editing. Position it as acceleration, not replacement.

Design Considerations

Design the product around model limits, not the other way around. Practical systems usually share a few traits:

• Clear input bounds such as duration limits and supported formats
• Visible uncertainty with reasons for low confidence
• Latency budgets tied to the workflow, not the demo
• Auditability for what was seen, heard, and decided

Shipping a Pragmatic Version

Start with a small, representative dataset and define acceptable output before you build. Add lightweight evaluation with a few high-risk scenarios, then iterate on prompt and pipeline changes. Logging and review tooling matter as much as model choice, especially when users need to trust what was skipped.

A Reference Pipeline That Holds Up

Most successful implementations look like a pipeline with explicit stages:

1. Ingest: normalize formats, cap duration, and record metadata.
2. Transcribe: get a transcript with time alignment (timestamps are the backbone).
3. Segment: split into stable chunks (scenes, slide changes, speaker turns).
4. Index: store transcript + metadata + embeddings for each segment.
5. Retrieve: answer queries by returning moments, not entire videos.
6. Synthesize: generate a summary or highlight list that points back to exact timestamps.

This structure keeps the system debuggable. When something is wrong, you can see whether transcription, segmentation, retrieval, or synthesis caused the failure.

Evaluation That Matters For Video

Video AI demos often look great because teams do not audit outputs closely. Practical evaluation focuses on a few measurable things:

• Timestamp accuracy (can users jump to the right moment?)
• Coverage (did the system miss key segments?)
• False positives (highlight reels are useless if they highlight noise)
• Safety/classification precision at the thresholds you operate at

Keep a small “golden set” of videos and re-run it whenever you change models, prompts, segmentation, or retrieval.

Common Pitfalls

• Hallucinated timestamps: the model sounds confident but points to the wrong moment. Always anchor outputs to retrieved segments.
• Overly long context: shoving a whole video into a single prompt wastes money and reduces accuracy. Segment first.
• No review tool: if reviewers cannot quickly see why a decision was made, they will not trust it.
• Privacy drift: meeting videos and training footage often contain sensitive data. Treat retention, access, and redaction as first-class requirements.

A Simple Checklist

• Define supported formats and duration limits.
• Make timestamps and citations part of every output.
• Build a review UI for low-confidence cases.
• Track latency and cost per processed minute of video.
• Re-run a golden evaluation set on every meaningful change.

I run local models daily: in production projects, for prototypes, and for anything involving sensitive data that shouldn’t leave my machine. The tooling has matured enough that this is no longer a novelty; it’s a practical engineering choice with clear tradeoffs.

I’ve also seen teams go all-in on local AI without understanding what they’re signing up for. Running your own models means owning the full lifecycle: model selection, quantization, runtime management, version pinning, quality monitoring, and fallback strategies. If you aren’t prepared for that operational load, use a managed API.

This post is for teams who have decided local makes sense and want to do it properly.

When Local Is the Right Call

Local AI makes sense in specific scenarios:

• Sensitive data. Proprietary code, financial records, anything you don’t want leaving your network. I frequently work with data under NDA, and local inference means the data never touches a third-party API.
• Predictable costs . API costs scale with usage; local costs scale with hardware. For high-volume routine tasks (classification, extraction, summarization), local can be dramatically cheaper once you amortize the hardware.
• Offline or air-gapped environments. Some deployments don’t have reliable internet. Some shouldn’t have it. Cyber-defense exercises drilled this in for me: there are environments where external API calls aren’t allowed at all.
• Deterministic CI testing . When your tests depend on model output, you need a pinned model version that doesn’t change between runs. Local gives you that control.

Local is the wrong call when you need frontier-level quality on every request or your team can’t absorb the operational overhead.

The Provider Abstraction

First rule: never hard-code your provider. Whether you’re using Ollama, llama.cpp, vLLM, or a cloud API, the rest of your code shouldn’t care. Hide it behind an interface.

In Go, this is clean:

// Provider defines the contract for any AI backend.
type Provider interface {
    Complete(ctx context.Context, req CompletionRequest) (CompletionResponse, error)
    Embed(ctx context.Context, input string) ([]float64, error)
    Health(ctx context.Context) error
}

type CompletionRequest struct {
    Model       string
    Messages    []Message
    MaxTokens   int
    Temperature float64
}

type CompletionResponse struct {
    Content    string
    TokensUsed int
    Model      string
    FinishReason string
}

Now your local and cloud providers implement the same interface. Switching between them is a config change, not a code rewrite. Testing is trivial: mock the interface and move on.

type OllamaProvider struct {
    endpoint string
    client   *http.Client
}

func NewOllamaProvider(endpoint string) *OllamaProvider {
    return &OllamaProvider{
        endpoint: endpoint,
        client: &http.Client{
            Timeout: 120 * time.Second,
        },
    }
}

func (o *OllamaProvider) Complete(ctx context.Context, req CompletionRequest) (CompletionResponse, error) {
    body := ollamaRequest{
        Model:    req.Model,
        Messages: toOllamaMessages(req.Messages),
        Stream:   false,
        Options: ollamaOptions{
            Temperature: req.Temperature,
            NumPredict:  req.MaxTokens,
        },
    }

    resp, err := o.post(ctx, "/api/chat", body)
    if err != nil {
        return CompletionResponse{}, fmt.Errorf("ollama completion: %w", err)
    }

    return CompletionResponse{
        Content:      resp.Message.Content,
        TokensUsed:   resp.EvalCount,
        Model:        resp.Model,
        FinishReason: resp.DoneReason,
    }, nil
}

The Fallback Chain

Local models are good. They aren’t always good enough. For quality-critical paths (user-facing content generation, complex reasoning tasks, anything where a wrong answer costs real money), you need a fallback to a stronger model .

type FallbackProvider struct {
    primary   Provider
    fallback  Provider
    threshold float64 // confidence threshold for fallback
}

func (f *FallbackProvider) Complete(ctx context.Context, req CompletionRequest) (CompletionResponse, error) {
    resp, err := f.primary.Complete(ctx, req)
    if err != nil {
        // Primary failed, try fallback
        slog.Warn("primary provider failed, using fallback", "error", err)
        return f.fallback.Complete(ctx, req)
    }
    return resp, nil
}

In practice, I extend this with confidence scoring: if the local model returns a low-confidence response, automatically retry with the cloud provider. The core pattern is simple: try local first, fall back to cloud when needed, and log fallbacks so you know how often they happen.

Configuration That Travels

Keep your AI configuration in a structured file in source control. Model names, endpoints, fallback rules, temperature settings: all of it should be declarative and version-controlled.

ai:
  default_provider: local

  providers:
    local:
      type: ollama
      endpoint: http://127.0.0.1:11434
      models:
        completion: "mistral:7b-instruct-v0.3-q5_K_M"
        embedding: "nomic-embed-text:latest"
      timeout: 120s

    cloud:
      type: openai
      # API key from environment: AI_CLOUD_API_KEY
      models:
        completion: "gpt-4o"
        embedding: "text-embedding-3-small"
      timeout: 30s

  fallback:
    enabled: true
    primary: local
    secondary: cloud
    on_error: true
    on_low_confidence: true
    confidence_threshold: 0.7

  evaluation:
    eval_set_path: "./eval/fixtures"
    run_on_model_change: true

The model name includes the quantization level. This is deliberate. mistral:7b-instruct-v0.3-q5_K_M is not the same as mistral:7b-instruct-v0.3-q4_0. Different quantization levels produce different outputs. Pin it.

Versioning and Reproducibility

This is where most local setups fall apart. Someone updates the model, doesn’t tell the team, and suddenly outputs are different. Tests still pass because nobody wrote quality assertions. They just check that the model returned something.

Version these things:

• Model file hash. SHA256 the model binary. Store the hash in your lockfile or config. If the hash changes, the model changed.
• Runtime version. Pin your Ollama or llama.cpp version in your Dockerfile or setup script.
• Prompt templates. Keep them in source control alongside the code that uses them. Prompt drift is real and insidious.

FROM ollama/ollama:0.11.4

# Pull and pin specific model versions (pull needs a running server)
RUN ollama serve & sleep 5 && ollama pull mistral:7b-instruct-v0.3-q5_K_M

# Copy eval fixtures for smoke test
COPY eval/fixtures /eval/fixtures

The Evaluation Harness

You need an eval set . Not optional. It should be a small collection of representative inputs with expected outputs that you run every time you change a model, update a prompt, or modify provider configuration.

func TestModelQuality(t *testing.T) {
    provider := setupLocalProvider(t)

    fixtures := loadEvalFixtures(t, "./eval/fixtures")
    var passed, failed int

    for _, fix := range fixtures {
        resp, err := provider.Complete(context.Background(), fix.Request)
        if err != nil {
            t.Errorf("fixture %s: %v", fix.Name, err)
            failed++
            continue
        }

        if !fix.Validate(resp.Content) {
            t.Errorf("fixture %s: expected pattern %q, got %q",
                fix.Name, fix.ExpectedPattern, resp.Content)
            failed++
            continue
        }
        passed++
    }

    passRate := float64(passed) / float64(passed+failed)
    if passRate < 0.85 {
        t.Fatalf("pass rate %.1f%% below threshold 85%%", passRate*100)
    }
}

Run this in CI. Run it before every model swap. Run it when you change prompts. The eval harness is what keeps you from shipping a regression you don’t notice for two weeks.

Performance Tuning Order

If local inference is too slow, fix it in this order:

1. Smaller model. For routine tasks like classification, extraction, and simple summarization, a 7B parameter model is often sufficient. Don’t run a 70B model for ticket triage.
2. Quantization. Q5_K_M is usually the sweet spot between quality and speed. Q4_0 is faster but you’ll notice quality degradation on complex tasks. Measure with your eval set before committing.
3. Batching. If you have throughput-heavy workloads, batch requests. Most local runtimes support this. The latency per request goes up slightly but throughput goes up dramatically.
4. Hardware. GPU inference is 10-50x faster than CPU for most model sizes. If you’re serious about local AI, budget for a decent GPU. An RTX 4090 handles a 7B model comfortably.

What You Trade

Local AI gives you control, privacy, and predictable costs. In exchange, you take on operational responsibility for model management, quality monitoring, and infrastructure maintenance. That’s a fair trade for the right workloads.

The teams that do this well treat local AI like any other infrastructure dependency: with discipline, not enthusiasm.

I write Go every day and contribute to open-source Go projects. I also use documentation from dozens of projects every day. And I can tell you the most common failure in developer documentation isn’t bad writing. It’s bad retrieval.

A developer hits a cryptic error at midnight. They search. They get a result that looks right. It’s from v2. They’re on v4. The answer doesn’t apply, but they don’t realize it until they’ve wasted forty minutes. Now multiply that across everyone using your docs.

That’s the problem AI documentation systems need to solve. Not “make the docs chatty.” Make docs findable, version-accurate, and honest about gaps.

The Three Problems Worth Solving

Discovery. Users don’t know your terminology. They describe symptoms, not concepts. A developer searching for “connection refused after deploy” might need the page about TLS configuration, but your keyword search returns the networking overview. Semantic search bridges this gap, but only if your chunks are meaningful units, not random 500-token slices.

Version accuracy. Your API changed between v3 and v4. The auth flow is different. The error codes are different. If your retrieval doesn’t filter by version, it will surface whatever is most popular in the index. Popular doesn’t mean current.

Freshness. Your product shipped a breaking change last Tuesday. The docs still describe the old behavior. Your AI docs system confidently explains how the old version works. This is worse than having no AI at all because it adds a layer of false authority.

The System Shape

An AI docs system is a pipeline, not a chatbot with a vector store bolted on. The pieces that matter:

Content store with metadata. Every chunk needs a stable ID, a version tag, a last-updated timestamp, and a source URL. Without these, you can’t filter, you can’t cite, and you can’t detect staleness.

Hybrid retrieval . Semantic search for conceptual questions. Keyword search for exact error codes, flag names, and parameter values. Neither alone is sufficient. The combination covers most queries. Add a reranking step that considers version relevance and recency as well as semantic similarity.

Answer synthesis with citations. The model generates an answer, but every claim must trace to a specific chunk. If the retrieved chunks don’t contain the answer, the system says so explicitly: “This doesn’t appear to be covered in the current docs. Here’s the closest related section.” A short answer with a source link beats a fluent paragraph that invents details.

Feedback collection. Log every question that gets a low-confidence response or explicit negative feedback. Route those to doc owners weekly. This is the actual improvement loop. Without it, you’re flying blind.

Chunking Matters More Than Model Choice

I’ve seen teams agonize over which LLM to use for synthesis while completely ignoring their chunking strategy . The chunking is where the battle is won or lost.

Split by document structure. Headings, sections, and code blocks are natural semantic boundaries. A chunk should be a coherent unit that can answer a question on its own, or clearly can’t. Token-count splitting produces fragments that retrieve well by similarity score but fail at actually answering questions.

Attach version metadata to every chunk. If someone asks about v4 auth, filter to v4 chunks before retrieval. That filter is the difference between helpful and harmful.

Freshness Is Ops Work

Docs go stale. That’s a consequence of shipping software, not a failure of discipline, and “write better docs” won’t fix it. Automated freshness checks will.

Schedule weekly jobs that validate links, compare API schema hashes against the documented version, and flag code samples that reference deprecated methods. When a check fails, create a ticket with clear ownership and a deadline. Not a backlog item. A real deadline.

At the fintech startup, we learned this the hard way with financial data: stale financial information is worse than useless. It’s dangerous. The same principle applies to docs. Stale docs users trust are worse than no docs at all.

Measure Success by Questions Answered

Pageviews are meaningless for docs. The metric that matters is: did the user get the right answer?

Track question success rate through explicit thumbs-up/down on AI answers. Track the count of unanswered or low-confidence questions; these are your improvement backlog. Track time-to-update for pages flagged as stale.

The feedback loop is the product. The AI layer is just the delivery mechanism. If unanswered questions aren’t flowing back into your documentation process , your AI docs system is a search box with extra steps.

This drives me nuts.

Fine-Tuning Isn’t a Knowledge Injection

Let me say this clearly: fine-tuning changes behavior, not knowledge. If your problem is “the model doesn’t know about our product,” the answer is retrieval. RAG . Grounding. Whatever you want to call it, feed the model your docs at inference time.

Fine-tuning bakes patterns into weights. It’s good for consistent tone, strict output formats, and narrow tasks repeated at massive scale. It’s terrible for facts that change, knowledge that needs updating, or anything where you want to point at a source and say “the answer came from here.”

I’ve watched teams spend weeks curating training data to teach a model their product catalog. Then the catalog changes. Now the model confidently recommends products that no longer exist. Retrieval would have solved this in an afternoon.

The Decision Is Simple

Before you fine-tune anything, answer these questions honestly:

Have you pushed the prompt hard? Not a one-liner. A real system prompt with role definition, constraints, examples, and output format. Most teams write a lazy prompt, get mediocre results, and conclude the model needs training. No. Their prompt needs training.

Have you added retrieval? If the issue is domain knowledge, factual accuracy, or up-to-date information, retrieval is the answer. Fine-tuning can’t compete with a well-indexed knowledge base for factual tasks.

Is the remaining gap about behavior? After good prompts and solid retrieval, if the model still can’t hold a consistent tone, reliably produce a specific output structure, or stop drifting on a narrow repeated task, now we can talk about fine-tuning.

Is the volume worth it? Fine-tuning has upfront cost and ongoing maintenance. If the task runs ten times a day, just use a better prompt. If it runs ten thousand times a day and prompt tokens are eating your budget , fine-tuning starts to make economic sense.

The Maintenance Tax Nobody Mentions

The fine-tuning tutorials leave this out: a tuned model is a versioned product. Your training data reflects a snapshot of your business at a moment in time. Products change. Policies change. Customer expectations change. Your training set drifts.

That means you need:

• Versioned training sets in source control
• A holdout evaluation set that you run against every new version
• Monitoring for quality regression in production
• A refresh cadence that’s actually budgeted and scheduled

I’ve seen exactly one team do all of this well. Everyone else fine-tuned once, celebrated, and then watched quality slowly degrade over three months while nobody noticed because nobody was measuring.

When I Recommend It

I’m not anti-fine-tuning. I’m anti-premature-fine-tuning. The legitimate cases exist:

• You need a specific voice or brand tone that holds across thousands of outputs and few-shot examples aren’t stable enough
• You have a narrow classification or extraction task at high volume where shaving prompt tokens saves real money
• You need a strict output schema and the base model keeps introducing creative variations despite explicit instructions

From what I’ve seen, maybe one in five projects that ask about fine-tuning actually need it. The rest need better prompts, proper retrieval, or both.

The Checklist Before You Fine-Tune

1. Write a real system prompt with examples and constraints. Test it on 50 representative inputs.
2. If factual accuracy is the gap, add retrieval. Test again.
3. If behavior consistency is still the gap at high volume, collect 200+ high-quality examples that match real production inputs.
4. Hold out 20% for evaluation. Fine-tune. Compare against the base model on both your target metric and general reasoning.
5. If the tuned model wins on behavior but loses on reasoning, reconsider whether the tradeoff is worth it.
6. Version everything. Monitor everything. Schedule refreshes.

Stop treating fine-tuning as step one. It’s step last.

I’ve been building data pipelines since before anyone called them “data pipelines.” At the fintech startup we were ingesting financial data from hundreds of sources, normalizing it, and serving it for real-time retrieval. That was 2017. The core problems haven’t changed.

What has changed is that your pipeline now has a second consumer: a retrieval system feeding an LLM. If you treat that consumer as an afterthought, your AI product will deliver confidently wrong answers. Ruthless focus on the basics separates pipelines that work from pipelines that demo well.

The Shape of an AI Pipeline

Every AI pipeline I’ve seen in production boils down to six stages. The skeleton:

pipeline:
  stages:
    - name: extract
      # Pull from sources, normalize formats
      # PDF, HTML, API responses -> clean markdown or structured text

    - name: diff
      # Hash-based change detection
      # This is the stage most teams skip. Don't.

    - name: chunk
      # Split by document structure first, token count second
      # Preserve section boundaries and headings

    - name: embed
      # Generate vectors using a pinned model version
      # Log the model version. You will need it later.

    - name: index
      # Upsert with stable IDs and rich metadata
      # source_id + chunk_position = deterministic ID

    - name: verify
      # Check for missing chunks, stale entries, orphans
      # Alert on drift from expected source freshness

Nothing exotic. The magic is in the discipline of each stage, not in clever architecture.

The Diff Step Is Everything

Most teams skip change detection and reprocess everything on every run. At small scale, this is fine. At production scale, it’s expensive, noisy, and makes debugging a nightmare.

A simple content-hash approach works well:

func hasChanged(sourceID string, content []byte, store HashStore) bool {
    newHash := sha256.Sum256(content)
    existing, found := store.Get(sourceID)
    if !found {
        store.Set(sourceID, newHash)
        return true
    }
    if existing != newHash {
        store.Set(sourceID, newHash)
        return true
    }
    return false
}

When I built the ingestion pipeline at the fintech startup, adding a diff layer cut downstream processing costs by roughly 60%. Most sources don’t change on most runs. Detecting that early saves everything downstream.

The diff step also gives you auditability. You can answer “what changed and when” instead of shrugging at a vector store that silently drifted.

Chunking: Structure Before Size

This is where most RAG pipelines go wrong. Teams reach for a token-count splitter because it’s the default in every tutorial, then wonder why retrieval returns fragments of ideas instead of coherent answers.

Split by document structure first. Headings, sections, code blocks, and list items are natural semantic boundaries. Only fall back to token-count splitting when a single section exceeds your context window.

def chunk_by_structure(doc: Document) -> list[Chunk]:
    chunks = []
    for section in doc.sections:
        if section.token_count <= MAX_CHUNK_TOKENS:
            chunks.append(Chunk(
                content=section.text,
                metadata={
                    "source_id": doc.id,
                    "section_heading": section.heading,
                    "position": section.index,
                    "doc_version": doc.version,
                },
                # Deterministic ID: no duplicates on re-ingestion
                id=f"{doc.id}:{section.index}",
            ))
        else:
            # Fall back to sliding window only for oversized sections
            for sub in sliding_window(section, MAX_CHUNK_TOKENS, overlap=100):
                chunks.append(Chunk(
                    content=sub.text,
                    metadata={**section.metadata, "sub_position": sub.index},
                    id=f"{doc.id}:{section.index}:{sub.index}",
                ))
    return chunks

Two things matter here. First, the id is deterministic, derived from source and position, not random. This means re-ingesting the same content produces upserts, not duplicates. Second, metadata travels with every chunk. When retrieval returns a chunk, you know exactly where it came from, which version, and which section.

I can’t overstate how many production RAG systems I’ve reviewed where chunks had no stable ID. Every reindex created duplicates. Users got the same passage three times in their context window, and the model hallucinated a consensus that didn’t exist.

Freshness Is an Operational Problem

Your pipeline isn’t done when it runs once. Sources change, APIs update, and documents get deleted. If your index doesn’t reflect reality, your AI lies with confidence.

Three rules I enforce on every pipeline:

1. Reindex on embedding model changes. If you swap or upgrade your embedding model , every existing vector is stale. This is a full reindex event. No exceptions. Pin your model version and log it.

2. Purge on source deletion. If a document disappears from the source, its chunks must disappear from the index. Orphaned chunks are a retrieval poison pill.

3. Alert on freshness drift. Every source has an expected update cadence. If your financial data feed hasn’t updated in 6 hours, something is wrong. Don’t wait for a user to notice.

freshness_policy:
  sources:
    - name: product_docs
      expected_interval: 24h
      alert_after: 36h
    - name: api_changelog
      expected_interval: 7d
      alert_after: 10d
    - name: support_kb
      expected_interval: 48h
      alert_after: 72h
  on_embedding_change: full_reindex
  on_source_delete: purge_chunks

The Mistakes I Keep Seeing

Across the pipelines I’ve built and reviewed, in telecom and elsewhere, the failure pattern is remarkably consistent:

No stable IDs. Updates create duplicates. Retrieval returns the same content multiple times. The model treats repetition as emphasis and doubles down on whatever it found.

Token-count-only chunking. A paragraph about authentication gets split mid-sentence. The first half lands in one chunk, the second half in another. Retrieval finds the first half. The model confidently gives half an answer.

Ad-hoc reindexing. Someone runs a reindex on a Friday afternoon. Nobody knows what changed. Retrieval quality shifts. The team argues about whether it got better or worse. No one can prove either way because there’s no baseline.

Missing permission metadata. The chunks are indexed without access control data. A user with restricted access asks a question and gets an answer sourced from documents they shouldn’t see. This is a compliance incident waiting to happen.

MCP (Model Context Protocol) is Anthropic’s answer, open-sourced in November 2024: a standard protocol for connecting AI models to external tools and data sources. Instead of N clients times M tools worth of custom integrations, you get N clients and M servers all speaking the same language.

I spent the last few weeks building an MCP server in Go to see whether the protocol lives up to the pitch. Short version: the protocol layer is clean. The hard parts are still auth, permissions, and not handing the model a footgun.

What MCP is

Strip away the marketing and MCP is a JSON-RPC-based protocol with three core concepts:

Tools. Functions the model can call. Each tool has a name, a description, and a JSON Schema for its inputs. The model decides when to call a tool based on the description.

Resources. Data the model can read. Think files, database records, API responses. Resources have URIs and can be listed or read by the client.

Prompts. Reusable prompt templates that servers can expose. Less interesting for most production use cases, but useful for standardizing common interactions.

The transport layer is deliberately simple: stdio for local servers, HTTP with SSE for remote ones. The protocol handles capability negotiation, so a client can discover what a server offers at connection time.

Building an MCP server in Go

Below is a minimal MCP tool server that wraps a database query. This is roughly what I built for an internal tool in a recent project that lets the AI assistant query deployment status.

package main

import (
	"context"
	"encoding/json"
	"fmt"
	"log"

	"github.com/mark3labs/mcp-go/mcp"
	"github.com/mark3labs/mcp-go/server"
)

type DeploymentStatus struct {
	Service     string `json:"service"`
	Version     string `json:"version"`
	Environment string `json:"environment"`
	Status      string `json:"status"`
	DeployedAt  string `json:"deployed_at"`
}

func main() {
	s := server.NewMCPServer(
		"deployment-status",
		"1.0.0",
		server.WithToolCapabilities(true),
	)

	tool := mcp.NewTool("get_deployment_status",
		mcp.WithDescription("Get the current deployment status for a service in a given environment"),
		mcp.WithString("service", mcp.Required(), mcp.Description("Service name")),
		mcp.WithString("environment", mcp.Required(), mcp.Description("Target environment: staging or production")),
	)

	s.AddTool(tool, handleGetDeploymentStatus)

	if err := server.ServeStdio(s); err != nil {
		log.Fatalf("server failed: %v", err)
	}
}

func handleGetDeploymentStatus(ctx context.Context, req mcp.CallToolRequest) (*mcp.CallToolResult, error) {
	service, _ := req.Params.Arguments["service"].(string)
	env, _ := req.Params.Arguments["environment"].(string)

	if env != "staging" && env != "production" {
		return mcp.NewToolResultError("environment must be 'staging' or 'production'"), nil
	}

	status, err := queryDeploymentDB(ctx, service, env)
	if err != nil {
		return mcp.NewToolResultError(fmt.Sprintf("query failed: %v", err)), nil
	}

	data, _ := json.Marshal(status)
	return mcp.NewToolResultText(string(data)), nil
}

A few things to note. The tool definition includes a JSON Schema for inputs, which means the client can validate before calling. The handler returns structured results or errors. The server handles all the JSON-RPC plumbing: capability negotiation, method routing, error formatting. You just write the handler.

This is roughly 50 lines of actual logic. The equivalent custom integration I had before was about 200 lines, with its own HTTP server, auth middleware, and request parsing. That reduction matters when you have 15 tools to wrap.

Adding auth and permissions

The current spec (2024-11-05) doesn’t define authentication. An OAuth-based authorization spec sits in the draft for the next revision, but it hasn’t shipped yet. Until it lands, you solve it yourself, and this is where most teams will spend their time.

The pattern I use is a middleware wrapper that checks permissions before the tool handler runs.

type PermissionChecker struct {
	allowedTools map[string][]string // tool -> allowed roles
}

func (pc *PermissionChecker) Wrap(toolName string, handler server.ToolHandlerFunc) server.ToolHandlerFunc {
	return func(ctx context.Context, req mcp.CallToolRequest) (*mcp.CallToolResult, error) {
		user := userFromContext(ctx)
		if user == nil {
			return mcp.NewToolResultError("authentication required"), nil
		}

		allowed := pc.allowedTools[toolName]
		if !hasAnyRole(user, allowed) {
			log.Printf("DENIED: user=%s tool=%s roles=%v", user.ID, toolName, user.Roles)
			return mcp.NewToolResultError("permission denied"), nil
		}

		log.Printf("ALLOWED: user=%s tool=%s", user.ID, toolName)
		return handler(ctx, req)
	}
}

Every tool call gets logged with the user identity, whether it was allowed or denied, and the arguments (redacted where necessary). This isn’t optional. If an AI system can call tools that read your database or modify your infrastructure, you need an audit trail.

For remote MCP servers over HTTP, I add standard bearer token auth at the transport layer. For local stdio servers, the auth context comes from the parent process. Either way, the permission check happens at the tool level, not just at the connection level. A user might be allowed to read deployment status but not trigger a rollback.

The security conversation

This is the part that keeps me up at night. MCP makes it easy to give an AI model access to tools. Maybe too easy. The protocol doesn’t enforce:

• Read vs. write separation. A tool that reads data and a tool that deletes data look the same to the protocol. You have to enforce the distinction.
• Rate limiting. Nothing stops the model from calling a tool a thousand times in a loop. Build your own limits.
• Input sanitization. The model generates the tool arguments. If those arguments end up in a SQL query or a shell command, you’re one prompt injection away from a bad day.
• Blast radius. A tool that queries one record is different from a tool that dumps an entire table. Scope your tools narrowly.

I enforce a simple rule: every tool that can write or modify gets a confirmation step that goes back to the user. The model can propose the action, but a human approves it. For read-only tools, I still scope the query to the current user’s data and add rate limits.

func handleTriggerRollback(ctx context.Context, req mcp.CallToolRequest) (*mcp.CallToolResult, error) {
	service, _ := req.Params.Arguments["service"].(string)
	env, _ := req.Params.Arguments["environment"].(string)

	// Never auto-execute destructive actions
	return mcp.NewToolResultText(fmt.Sprintf(
		"CONFIRMATION REQUIRED: Roll back %s in %s to previous version? "+
			"This action requires human approval.",
		service, env,
	)), nil
}

This is the same principle cyber-defense exercises teach: least privilege, explicit authorization, and comprehensive auditing . The fact that the agent is an AI model doesn’t change the security model. If anything, it makes it more important, because the model can be manipulated through prompt injection in ways a human user can’t.

Where MCP shines

Tool portability. I built the deployment status server once. It works with Claude, with our internal assistant, and with any future client that speaks MCP. That’s the whole pitch, and it delivers.

Discovery. A client can connect to a server and ask “what can you do?” The response is machine-readable and includes schemas. This means the AI model gets accurate tool descriptions automatically instead of relying on hardcoded prompts.

Composability. An AI client can connect to multiple MCP servers simultaneously. One for deployments, one for monitoring, one for documentation. Each server is independently deployable and testable. This is the microservices pattern applied to AI tool access, with the same benefits and the same risks.

Where it doesn’t

No standard auth. Every deployment rolls its own. This will improve, but right now it’s extra work.

Ecosystem maturity. The Go ecosystem is solid thanks to mcp-go, but tooling for testing, debugging, and monitoring MCP interactions is still young. I wrote my own trace logger.

Complexity budget. MCP is one more protocol layer to understand, debug, and operate. For a team with two tools, the overhead might not be worth it. For a team with ten tools across multiple AI clients, it pays for itself quickly.

Should you adopt it now

If you’re building tool-heavy AI systems, MCP is worth investing in now. Start with one server. Pick your simplest, most-used tool. Wrap it in MCP. Test it against a real client. Measure the integration effort against your current custom approach.

From what I’ve seen, MCP cut tool integration time roughly in half and made our tools testable in isolation for the first time. The security work is the same either way: you have to solve auth and permissions regardless of protocol. MCP standardizes the rest (discovery, invocation, transport), and that’s enough to make it worth building on.

The prototype worked beautifully on the first three test clips. Then I ran it against their actual library and it confidently told me a customer was “demonstrating frustration through aggressive keyboard usage.” The customer was typing their account number. The model was hallucinating emotional context from audio artifacts.

That experience captures video AI right now. It’s genuinely capable. It’s also confidently wrong in ways that are hard to predict and even harder to catch at scale.

Video isn’t just “lots of images”

The fundamental challenge with video understanding is time. An image model looks at a single moment. A video model has to track what happened, in what order, and how things changed. That temporal reasoning is where models still struggle.

The practical failure modes I’ve seen:

• Temporal confusion. The model describes events out of order or merges two separate moments into one. This is especially bad with longer clips.
• Missing key moments. The model summarizes the overall vibe of a clip but misses the specific 10-second window where the important thing happened.
• Overconfidence. The model narrates with authority even when it’s guessing. No hedging. No “I’m not sure.” Just wrong with conviction.

A staged pipeline

Forget single-prompt video understanding. It doesn’t scale. What works is a pipeline that breaks the problem into stages you can debug independently.

The architecture I landed on:

Step 1: Extract audio and transcribe. If the video has spoken content, the transcript is your primary signal. Transcription of clear speech is reliable now, far more reliable than frame analysis. Start here.

Step 2: Sample frames intelligently. Not every N seconds. Use scene detection to identify transitions, then sample the first frame of each scene plus any frame with significant visual change. This reduces the frame count by 60-80% without losing meaningful content.

Step 3: Analyze frames with context. Feed each frame to a vision model along with the surrounding transcript text. The transcript grounds the visual analysis and prevents the model from inventing narratives that don’t match what was said.

Step 4: Synthesize with timestamps. Merge the transcript-grounded visual analysis into a structured timeline. Every claim in the summary must reference a specific timestamp. If the model can’t cite when something happened, it probably didn’t happen.

Audio first, video second. The transcript is your source of truth. The video adds context.

Where it’s useful

After the initial disaster and a week of pipeline tuning, I found the sweet spots:

Meeting summaries with action items. Transcribe, extract decisions and action items, tag them with speaker and timestamp. This works well because the transcript carries most of the signal and the visual component (slides, screen shares) adds structure.

Content moderation. Checking video against a specific policy with concrete criteria. “Does this clip contain product logos?” “Is the speaker reading from a teleprompter?” Questions with binary answers that the model can ground in visual evidence.

Search and retrieval. “Find the part of this recording where they discuss pricing.” Natural language search over video libraries works surprisingly well when you have good transcripts and frame-level annotations.

Compliance review. Structured checks against a rubric. Did the agent identify themselves? Did they read the required disclosure? Was the customer’s consent recorded? This works because the criteria are specific and verifiable.

Where it isn’t ready

Long-form video without speech. Surveillance-style footage. Anything where the important signal is subtle body language or spatial relationships. Anything where the model needs to count reliably or track specific objects across many frames.

Also, anything where a false positive has real consequences. If your video review pipeline flags a customer interaction as “hostile” and that triggers an HR process, you had better have a human in the loop.

Starting without overbuilding

Pick one use case. Keep clips under 10 minutes. Fix your output format before you start: structured JSON, not free-form prose. Build a gold set of 20-30 annotated clips and run every pipeline change against it.

The evaluation loop is everything. Without it, you’re optimizing by vibes, and vibes don’t catch temporal hallucinations.

Video AI is real and useful for the right problems. Just don’t let the first impressive demo convince you it’s ready for the hard ones.

Don’t make them your default path. Route by complexity, run expensive calls async, set per-request budgets, and cache aggressively . The model is the easy part. The routing and cost control are where you earn your keep. This post covers the architecture I landed on, with real Go code. When I started this work, most posts I found were hand-wavy “use async patterns” advice with zero implementation detail.

The problem, concretely

Standard LLM calls in our pipeline take 1-3 seconds. Reasoning model calls take 8-45 seconds. That’s a different product experience.

Cost scales the same way. A reasoning call can burn 10-50x the tokens of a standard call for the same input because the model does internal chain-of-thought before producing output. On a high-traffic endpoint, that adds up fast, and your bill will show it .

At one company, someone enabled a reasoning model as the default for their support chatbot. The monthly API bill went from $2,000 to $34,000 in three weeks. Most of those calls were “what are your business hours?” Not exactly a problem that requires deep reasoning.

When reasoning models actually help

I’ve found three categories where the latency and cost trade-off is worth it:

Multi-step analysis. Reviewing a contract clause, debugging a complex data pipeline, synthesizing information from multiple sources. Tasks where a wrong answer costs more than a slow answer.

Code review and debugging. Reasoning models catch logic errors and subtle bugs that fast models miss entirely. I use them in our CI pipeline for reviewing diffs on critical paths. Nobody cares if that takes 30 seconds.

Planning and decomposition. Breaking a complex task into subtasks, reasoning about dependencies, identifying risks. The model needs to hold a lot of context and think through implications.

Where they’re a waste: simple Q&A, classification, extraction, and anything high-volume or latency-sensitive. Route those to fast models and save money.

The routing layer

Not every request deserves the same model . The router I use in Go:

type ComplexityLevel int

const (
	ComplexityLow ComplexityLevel = iota
	ComplexityMedium
	ComplexityHigh
)

type Router struct {
	client         LLMClient
	fastModel      string
	reasoningModel string
	classifier     *ComplexityClassifier
}

func (r *Router) Route(ctx context.Context, req Request) (Response, error) {
	level := r.classifier.Assess(req)

	switch level {
	case ComplexityLow:
		return r.callModel(ctx, r.fastModel, req, defaultBudget)
	case ComplexityMedium:
		resp, err := r.callModel(ctx, r.fastModel, req, defaultBudget)
		if err != nil || resp.Confidence < 0.7 {
			return r.callModel(ctx, r.reasoningModel, req, premiumBudget)
		}
		return resp, nil
	case ComplexityHigh:
		return r.callModel(ctx, r.reasoningModel, req, premiumBudget)
	default:
		return r.callModel(ctx, r.fastModel, req, defaultBudget)
	}
}

The complexity classifier doesn’t need to be fancy. Ours uses a combination of input length, certain keywords (like “analyze”, “compare”, “debug”), and whether the request references multiple documents. A simple heuristic gets you 80% of the way there.

The medium-complexity path is where this gets interesting. Try the fast model first. If confidence is low, escalate to reasoning. This keeps costs down for tasks that turn out to be simpler than they look.

Async execution for expensive calls

Any reasoning model call that might take more than a few seconds shouldn’t block your HTTP handler. The pattern I use:

type Job struct {
	ID        string
	Status    string
	Request   Request
	Response  *Response
	CreatedAt time.Time
}

type AsyncExecutor struct {
	jobs   sync.Map
	router *Router
	notify func(jobID string, resp Response)
}

func (e *AsyncExecutor) Submit(ctx context.Context, req Request) (string, error) {
	job := &Job{
		ID:        generateID(),
		Status:    "pending",
		Request:   req,
		CreatedAt: time.Now(),
	}
	e.jobs.Store(job.ID, job)

	go func() {
		resp, err := e.router.Route(context.Background(), req)
		// Store a new Job instead of mutating the one Poll may have handed out.
		done := *job
		if err != nil {
			done.Status = "failed"
			e.jobs.Store(done.ID, &done)
			return
		}
		done.Response = &resp
		done.Status = "completed"
		e.jobs.Store(done.ID, &done)
		e.notify(done.ID, resp)
	}()

	return job.ID, nil
}

func (e *AsyncExecutor) Poll(jobID string) (*Job, bool) {
	val, ok := e.jobs.Load(jobID)
	if !ok {
		return nil, false
	}
	return val.(*Job), true
}

The caller gets a job ID back immediately. They can poll for status, or we can push a notification when it’s done. The UX team shows a “thinking deeply about this…” indicator. Users are surprisingly tolerant of waiting when you tell them why.

In production, you want a proper job queue (we use Redis) and persistence. But the pattern is the same.

Per-request cost budgets

This is the piece most teams skip, and it’s what prevents surprise bills. Every model call gets a token budget:

type Budget struct {
	MaxInputTokens  int
	MaxOutputTokens int
	MaxCostCents    int
	TimeoutSeconds  int
}

var (
	defaultBudget = Budget{
		MaxInputTokens:  4000,
		MaxOutputTokens: 1000,
		MaxCostCents:    5,
		TimeoutSeconds:  10,
	}
	premiumBudget = Budget{
		MaxInputTokens:  16000,
		MaxOutputTokens: 4000,
		MaxCostCents:    50,
		TimeoutSeconds:  60,
	}
)

func (r *Router) callModel(ctx context.Context, model string, req Request, budget Budget) (Response, error) {
	ctx, cancel := context.WithTimeout(ctx, time.Duration(budget.TimeoutSeconds)*time.Second)
	defer cancel()

	if req.EstimatedInputTokens() > budget.MaxInputTokens {
		return Response{}, fmt.Errorf("input exceeds budget: %d > %d tokens",
			req.EstimatedInputTokens(), budget.MaxInputTokens)
	}

	resp, err := r.client.Complete(ctx, model, req.ToPrompt(),
		WithMaxTokens(budget.MaxOutputTokens),
	)
	if err != nil {
		return Response{}, fmt.Errorf("model call failed: %w", err)
	}

	costCents := estimateCost(model, resp.Usage)
	if costCents > budget.MaxCostCents {
		log.Printf("WARN: call exceeded cost budget: %d > %d cents", costCents, budget.MaxCostCents)
	}

	return parseResponse(resp), nil
}

The budget is enforced before and during the call. Context timeouts prevent runaway reasoning. Token limits prevent ballooning inputs. Cost estimation after the call feeds monitoring and alerting.

At one company, we added a daily cost ceiling per endpoint. If the endpoint hits 80% of its daily budget by noon, it automatically downgrades all calls to the fast model for the rest of the day. Crude but effective.

Caching reasoning results

Reasoning model outputs are expensive to produce but often reusable. Same contract clause reviewed twice? Same code pattern analyzed in different PRs? Cache it.

type ResultCache struct {
	store *redis.Client
	ttl   time.Duration
}

func (c *ResultCache) GetOrCompute(ctx context.Context, key string, compute func() (Response, error)) (Response, error) {
	cached, err := c.store.Get(ctx, key).Result()
	if err == nil {
		var resp Response
		if json.Unmarshal([]byte(cached), &resp) == nil {
			resp.FromCache = true
			return resp, nil
		}
	}

	resp, err := compute()
	if err != nil {
		return resp, err
	}

	data, _ := json.Marshal(resp)
	c.store.Set(ctx, key, data, c.ttl)
	return resp, nil
}

The cache key is a hash of the input and model version. When the model changes, the cache invalidates naturally. We use a 24-hour TTL for most analysis tasks and a 1-hour TTL for anything time-sensitive.

This alone cut our reasoning model costs by about 40% on the code review pipeline, because many PRs touch similar patterns.

What I got wrong the first time

I initially tried to hide latency entirely. Bad idea. Users thought the system was broken. The moment we switched to explicit “this needs deeper analysis, checking now…” messaging, complaints dropped to zero. People understand that some questions take longer to answer well. Respect that.

I also over-routed to reasoning models early on. The classifier was too generous with “high complexity” ratings. We added a feedback loop: if a reasoning model call produces essentially the same output as a fast model would have (measured by comparing on a sample), downgrade the classification for that pattern. Within two weeks, our routing accuracy improved significantly.

The full request path

Request → Complexity Classifier → Router
                                    ├── Low → Fast Model (sync)
                                    ├── Medium → Fast Model → check confidence → maybe Reasoning Model
                                    └── High → Async Executor → Reasoning Model → Notify

All paths → Budget Enforcement → Cache Check → Model Call → Response

Treat reasoning models as a premium tier. The model does the thinking. Your job is to make sure it only thinks when it needs to.

The late-2024 model market is competitive enough that the gap between top-tier providers on general tasks is small. The differences that matter most are operational, not intellectual. Here’s how I think about model selection for production systems.

Hosted APIs and Open Weights

Two tracks dominate. Hosted APIs from Anthropic, OpenAI, and Google iterate fast and are easiest to ship with. Open-weight models from Meta (Llama), Mistral, and others give you more control but come with infrastructure baggage.

What to Compare

Forget leaderboards. They’re narrow, gameable, and rarely match your workload. Here are the dimensions that matter in production:

I’ve run these comparisons for teams I’ve worked with. The results consistently surprise people. The “best” model on paper is rarely the best model for their specific tasks.

How to Run a Bake-Off

Don’t spend a month on this. A focused bake-off should take a few days:

1. Pick 30-50 representative inputs from your actual workload. Cover the common cases and the hard cases.
2. Define success criteria for each one: specific, checkable criteria, not vibes.
3. Run each model against the same inputs with the same system prompt.
4. Score each model by task success rate, latency, and cost.
5. Check structured output compliance if you depend on it.

The results won’t be close on every dimension. One model will be cheaper. Another will be more accurate on complex tasks. A third will have better latency. That’s the point: you’re mapping the tradeoff space, not finding a winner.

The Router Pattern

Once you have bake-off data, the next step is obvious: route different task types to different models .

A routing layer adds complexity, but not much. An if statement or a config-driven switch is enough to start. You don’t need an ML-based router. You need a decision tree grounded in your bake-off results.

One team I worked with went from a single frontier model to a two-model router and cut monthly spend by 60% with no measurable quality regression. The hard part was running the bake-off. The router itself was 50 lines of Go.

Open Models: When and When Not

Self-hosting is a real option now. Llama 3 and Mistral variants are genuinely capable. For most tasks, capability is no longer the deciding question. Ownership is: do we want to run this infrastructure?

Self-host when:

• Data must not leave your network (regulatory, contractual)
• Volume is high and predictable enough that fixed GPU costs beat per-token pricing
• You need fine-tuning that hosted APIs don’t support
• You need offline or air-gapped operation

Don’t self-host when:

• Volume is bursty or growing unpredictably
• You need frontier capability that open models haven’t matched yet
• Your team doesn’t have GPU ops experience
• You want to iterate model versions quickly

I’ve talked a few teams out of self-hosting after running the numbers. The GPU costs plus ops burden plus slower iteration cycle made the total cost higher than the hosted API they were trying to replace. Self-hosting is a capability decision as much as a cost decision.

Contracts and Pricing: Check the Fine Print

Pricing shifts fast. What I can tell you as of late 2024:

• The per-token spread between a provider’s flagship and its small model runs from about 3x (Claude 3.5 Sonnet vs. Claude 3.5 Haiku) to about 17x (GPT-4o vs. GPT-4o mini)
• Total cost is dominated by usage patterns (retries, context size, output length), not headline price
• Enterprise agreements often include committed-use discounts that change the math significantly
• Rate limits and quotas vary by tier and can cap throughput during peak usage

Verify current rates directly with providers before locking in. A pricing comparison that’s two months old is already stale.

Re-Run the Bake-Off Quarterly

The models move fast, so your selection process should be fast too. Re-evaluate quarterly. Treat vendor claims and public benchmarks as starting points, not decisions. The evaluation set built from your actual prompts, reviewed by your team, is the only benchmark that matters.

Every few weeks someone sends me a spreadsheet comparing AI provider pricing and asks “which one should we use?” The spreadsheet always compares cost per million tokens. It’s always useless.

From the AI cost work I’ve done, I can tell you the gap between headline pricing and actual production cost is consistently 3-10x. Providers with the cheapest tokens sometimes end up being the most expensive per completed task. Here’s why and how to benchmark properly.

The Full Cost Stack

Token price is one line item. Production cost includes everything the system does to deliver a reliable result.

Teams that track only model inference are missing half their spend. I learned this the hard way on a document processing pipeline: the retry rate on complex documents was 40%, effectively doubling model cost. The pricing spreadsheet didn’t mention that.

Benchmark Your Tasks, Not Generic Prompts

A useful benchmark mirrors your actual workload. Generic “summarize this article” tests tell you nothing about how a model handles your prompts, error rates, and latency requirements.

Build a benchmark set that covers:

Keep this set stable. If benchmark inputs change every week, you can’t tell whether cost shifts came from system changes or test changes.

Compare Approaches, Not Providers

Provider names and model versions change quarterly. A benchmark built around “GPT-4 vs Claude 3.5” has a shelf life of weeks. Instead, compare the architectural choices you control:

The router pattern is where I’ve seen the biggest wins. One team cut monthly spend by 60% by routing straightforward classification tasks to a small model and reserving the large model for generation. Classification accuracy from the small model was identical. They were paying frontier prices for commodity work.

The Drivers That Move Your Bill

Forget micro-optimizing prompts. These four factors determine 80% of your cost trajectory:

Response length drift. Prompts evolve over time. Engineers add instructions, examples, formatting requirements. Output gets longer. Nobody notices until the bill does. Track average output tokens per task type weekly.

Retry rates. Every retry is a full cost event. If your validation rejects 20% of responses and retries each once, your effective cost is 1.2x the base. If retries can fail and retry again, it climbs toward 1.25x. Measure retry rate by task type and fix the root cause.

Retrieval bloat. Context windows keep growing, so teams stuff more chunks in . More context means more input tokens. But past a point, more context doesn’t improve answers. It just costs more. Measure answer quality versus context size and find the plateau.

Routing waste. Sending everything to the most capable model is the default because it’s easy. It’s also the most expensive default. Any task where a smaller model achieves the same success rate is money burned on the large model.

Self-Hosting: When the Math Works

Self-hosting isn’t a cost optimization for most teams. It works for teams with specific constraints:

• Predictable, high-volume workloads where the per-token savings exceed infra costs
• Strict data residency or air-gapped environments
• Fine-tuned models that don’t exist as hosted APIs

For bursty workloads or teams that need frequent model upgrades, operational overhead eats the savings. I’ve talked a few teams out of self-hosting after we modeled actual GPU costs, ops burden, and iteration-speed penalties. The math didn’t work for them. It might for you. Run the numbers on your workload, not someone else’s blog post.

Set Up Monitoring Before You Need It

A benchmark is a snapshot. Production spend is a moving target. Set up cost monitoring from day one:

• Track cost per successful task, not cost per API call
• Break it down by feature and user tier
• Alert on spend spikes and retry rate increases
• Review monthly with someone who owns the budget

The teams that catch cost problems early treat them like performance regressions. The teams that catch them late treat them like budget emergencies.

Boring systems, predictable bills.

I’ve built three different RAG systems this year, and each time the first complaint was “the model hallucinates.” Each time, the real problem was retrieval feeding garbage into context. The model was doing its best with bad evidence.

Basic RAG (embed the query, grab the top-k chunks, stuff them into the prompt) is a fragile baseline. It works in demos. It breaks on real data. Here’s why, and what to do about it.

Why Basic Retrieval Fails

The failure modes are predictable. I see the same ones everywhere:

Vocabulary mismatch. The user asks about “cancellation policy” but the source document says “termination terms.” Pure semantic search sometimes bridges this gap. Sometimes it doesn’t.

Context fragmentation. A paragraph that answers the question gets split across two chunks. Neither chunk scores high enough on its own. The answer exists in your corpus but the retrieval never finds it.

Wrong granularity. Your chunks are 512 tokens. The user asks a question that needs a 50-token fact buried in the middle. The surrounding noise tanks the relevance score.

Temporal confusion. The 2022 policy and the 2024 policy both match the query. The retrieval returns whichever embeds closer, not whichever is current.

Multi-hop requirements. The answer requires combining facts from two different documents. Single-query retrieval will find one, maybe. Not both.

Hybrid Search: Combine Signals

Pure vector search misses exact terms. Pure lexical search misses paraphrases. Combine them.

The implementation is straightforward. Run both searches, normalize the scores, and fuse the rankings. Reciprocal Rank Fusion (RRF) is the simplest approach that works:

package search

// RRFMerge combines results from multiple search backends using
// Reciprocal Rank Fusion. k controls how much rank position
// matters -- 60 is a common default.
func RRFMerge(results [][]Result, k float64) []Result {
	scores := make(map[string]float64)
	docs := make(map[string]Result)

	for _, ranked := range results {
		for rank, r := range ranked {
			scores[r.ID] += 1.0 / (k + float64(rank+1))
			docs[r.ID] = r
		}
	}

	merged := make([]Result, 0, len(scores))
	for id, score := range scores {
		doc := docs[id]
		doc.Score = score
		merged = append(merged, doc)
	}

	sort.Slice(merged, func(i, j int) bool {
		return merged[i].Score > merged[j].Score
	})
	return merged
}

From what I’ve seen, hybrid search with RRF improves recall by 15-30% over pure vector search on real corpora. That’s on production data with messy, inconsistent documents, not synthetic benchmarks.

Chunking Isn’t a Formatting Detail

Most teams treat chunking as a config parameter. Set chunk_size=512, done. This is wrong.

Good chunking preserves the structure of the source material. If your documents have headings, keep them. If a section is self-contained, chunk it as a unit. If a chunk can’t be understood without its parent heading, prepend a breadcrumb.

// Chunk represents a document fragment with enough context
// to be understood when retrieved independently.
type Chunk struct {
	ID         string
	Content    string
	Breadcrumb string // e.g. "Policy Manual > Section 4 > Termination"
	Source     string
	UpdatedAt  time.Time
	Tokens     int
}

// ChunkWithContext prepends the breadcrumb to the content so the
// chunk is self-contained when injected into a prompt.
func (c Chunk) ChunkWithContext() string {
	if c.Breadcrumb == "" {
		return c.Content
	}
	return fmt.Sprintf("[%s]\n\n%s", c.Breadcrumb, c.Content)
}

The breadcrumb costs a few tokens per chunk. It pays for itself by making the model understand what it’s reading. Without it, the model gets a floating paragraph with no context about where it came from.

Query Expansion

Single-shot queries are narrow. The user types one phrasing, but the relevant document uses different words. You miss.

Query expansion generates alternative phrasings and retrieves against all of them. The simplest version that works: ask the LLM to generate 2-3 reformulations, then run all queries and merge results.

A more interesting approach is HyDE (Hypothetical Document Embeddings). Instead of expanding the query, generate a hypothetical answer and embed that. The intuition is that a hypothetical answer is closer in embedding space to the actual answer than the question is.

// ExpandQuery generates alternative phrasings for retrieval.
// Returns the original query plus expansions.
func ExpandQuery(ctx context.Context, llm LLM, query string, n int) ([]string, error) {
	prompt := fmt.Sprintf(
		"Generate %d alternative phrasings of this search query. "+
			"Return only the queries, one per line.\n\nQuery: %s",
		n, query,
	)

	resp, err := llm.Complete(ctx, prompt)
	if err != nil {
		// Fallback: just use the original query.
		return []string{query}, nil
	}

	queries := []string{query}
	for _, line := range strings.Split(resp, "\n") {
		line = strings.TrimSpace(line)
		if line != "" {
			queries = append(queries, line)
		}
	}
	return queries, nil
}

Note the error handling: if expansion fails, fall back to the original query. Don’t let a retrieval enhancement become a retrieval blocker.

Expansion increases recall, but it also brings in noise. That’s fine, because the next step handles it.

Reranking: The Cleanup Step

After gathering candidates from hybrid search across expanded queries, you have a broad set. Most of it is relevant. Some isn’t. A reranker fixes the ordering.

A cross-encoder reranker compares the full query against the full chunk text. It’s slower than embedding similarity but significantly more accurate for the final ranking. Run it on your top 20-50 candidates, not your entire corpus.

// Rerank takes candidate chunks and reorders them by relevance
// using a cross-encoder model. Keep topN results.
func Rerank(ctx context.Context, model Reranker, query string, candidates []Chunk, topN int) ([]Chunk, error) {
	type scored struct {
		chunk Chunk
		score float64
	}

	pairs := make([]QueryDocPair, len(candidates))
	for i, c := range candidates {
		pairs[i] = QueryDocPair{Query: query, Document: c.ChunkWithContext()}
	}

	scores, err := model.Score(ctx, pairs)
	if err != nil {
		if len(candidates) > topN {
			candidates = candidates[:topN]
		}
		return candidates, nil // degrade gracefully
	}

	ranked := make([]scored, len(candidates))
	for i := range candidates {
		ranked[i] = scored{chunk: candidates[i], score: scores[i]}
	}

	sort.Slice(ranked, func(i, j int) bool {
		return ranked[i].score > ranked[j].score
	})

	result := make([]Chunk, 0, topN)
	for i := 0; i < topN && i < len(ranked); i++ {
		result = append(result, ranked[i].chunk)
	}
	return result, nil
}

Again, graceful degradation. If the reranker fails, return the original order truncated to topN. The system should always return something useful.

Multi-Representation Indexing

One embedding per document is leaving retrieval quality on the table. For important documents, index multiple representations:

• The full text (for detail queries)
• A concise summary (for broad queries)
• Question-like phrasings that the text answers (for direct questions)

This widens the retrieval surface without changing the source documents. It’s extra indexing work, but the recall improvement on multi-hop queries is substantial. I’ve seen it close the gap on questions that basic retrieval missed entirely.

Measure Retrieval Separately

This is the part most teams skip, and it’s the most important.

If you only measure end-to-end answer quality, you can’t tell whether a bad answer came from bad retrieval or bad generation. You need retrieval-specific metrics:

• Recall@k: Did the relevant chunk appear in the top k results?
• Precision@k: What fraction of the top k results were actually relevant?
• MRR (Mean Reciprocal Rank): How high did the first relevant result rank?
• nDCG: How well-ordered is the full ranking?

Build a small eval set: 50 to 100 query-document pairs where you know which chunks should be retrieved. Run it after every change to chunking, embedding, or search logic. This is the single highest-leverage investment in a RAG system.

I keep these eval sets in the repo alongside the retrieval code. They’re as important as unit tests. Maybe more important.

The Full Pipeline

Putting it all together, the retrieval pipeline for a production RAG system looks like:

1. Expand the query (2-3 reformulations)
2. Run hybrid search (vector + lexical) for each query variant
3. Merge results with RRF
4. Rerank the merged candidates
5. Return top-k chunks with breadcrumbs

Each step is independently testable and independently measurable. When something breaks, you know where to look.

The generation step is almost an afterthought once retrieval is solid. A decent model with the right evidence in context will give you a good answer. A frontier model with the wrong evidence will confidently give you a wrong one.

Fix retrieval first. Everything else follows.

The first time I shipped an LLM feature without a proper test suite, we spent three weeks arguing about whether the quality had regressed after a prompt change. Nobody had baseline numbers. Nobody had a definition of “good.” We were debugging by vibes.

Never again.

LLM testing is different from traditional software testing, but it isn’t impossible. It requires accepting that you’re testing probabilistic behavior and building your strategy around that reality instead of fighting it.

The problem with LLM outputs

Three things make LLM testing hard:

1. Non-determinism. The same input can produce different outputs across runs, even with temperature set to zero (some providers still have variance).
2. Multiple valid answers. For most tasks, there isn’t one correct answer. There’s a space of acceptable answers.
3. Invisible regressions. A prompt change or model update can shift behavior without any code change. Your CI pipeline sees green. Your users see worse outputs.

The instinct is to throw up your hands and say “we can’t test this.” That’s wrong. You can test this. You just can’t use assertEqual.

Layer 1: deterministic tests for everything around the model

The code around the LLM (prompt rendering, response parsing, validation , error handling) is deterministic. Test it like normal software.

func TestPromptRendering(t *testing.T) {
    tmpl := NewSupportPrompt()
    result, err := tmpl.Render(PromptInput{
        CustomerName: "Alice",
        Issue:        "billing dispute",
        History:      []string{"previous contact on 2024-07-15"},
    })
    if err != nil {
        t.Fatalf("render failed: %v", err)
    }

    if !strings.Contains(result, "Alice") {
        t.Error("prompt should contain customer name")
    }
    if !strings.Contains(result, "billing dispute") {
        t.Error("prompt should contain issue description")
    }
    if !strings.Contains(result, "2024-07-15") {
        t.Error("prompt should contain interaction history")
    }
}

func TestResponseParsing(t *testing.T) {
    raw := `{"action": "escalate", "reason": "billing dispute over $500", "priority": "high"}`

    resp, err := ParseSupportResponse([]byte(raw))
    if err != nil {
        t.Fatalf("parse failed: %v", err)
    }

    if resp.Action != "escalate" {
        t.Errorf("expected action=escalate, got %s", resp.Action)
    }
    if resp.Priority != "high" {
        t.Errorf("expected priority=high, got %s", resp.Priority)
    }
}

These tests are fast, stable, and catch a surprising number of regressions. I’ve seen parsing bugs slip through because teams only tested the happy path, then the model started returning JSON with trailing commas.

Also test mocked LLM responses to verify error handling and orchestration logic:

func TestHandlesModelTimeout(t *testing.T) {
    client := &MockLLMClient{
        Response: nil,
        Err:      context.DeadlineExceeded,
    }

    handler := NewSupportHandler(client)
    result, err := handler.Handle(context.Background(), "test query")

    if err != nil {
        t.Fatal("handler should not propagate model timeout as error")
    }
    if result.Fallback != true {
        t.Error("should trigger fallback on timeout")
    }
}

Layer 2: property-based checks for model outputs

You can’t check that the model said “I apologize for the inconvenience.” You can check that the response acknowledges the issue, avoids profanity, and stays under 200 words.

Define a rubric. Keep it simple.

type EvalCriteria struct {
    Name    string
    Check   func(input string, output string) bool
}

var supportResponseCriteria = []EvalCriteria{
    {
        Name: "acknowledges_issue",
        Check: func(input, output string) bool {
            lower := strings.ToLower(output)
            return strings.Contains(lower, "sorry") ||
                strings.Contains(lower, "understand") ||
                strings.Contains(lower, "apologize")
        },
    },
    {
        Name: "includes_next_steps",
        Check: func(input, output string) bool {
            lower := strings.ToLower(output)
            return strings.Contains(lower, "will") ||
                strings.Contains(lower, "next") ||
                strings.Contains(lower, "follow up")
        },
    },
    {
        Name: "reasonable_length",
        Check: func(input, output string) bool {
            words := strings.Fields(output)
            return len(words) >= 20 && len(words) <= 200
        },
    },
}

These aren’t perfect. The string matching is crude. But they catch common failure modes: responses that ignore the user’s problem, responses that are empty or absurdly long, and responses that miss expected elements.

For more nuanced checks (tone, factual accuracy, coherence), I use model-based evaluation. Have a separate evaluator model score the output against the rubric. It isn’t free, but it’s cheaper than human review on every test case and usually more reliable than regex.

Layer 3: the golden set

A golden set is a curated collection of representative inputs with expected properties. Not expected outputs, expected properties.

type GoldenCase struct {
    ID       string            `json:"id"`
    Input    string            `json:"input"`
    Expected map[string]string `json:"expected"`
}

// Example golden case
// {
//   "id": "billing_complaint_042",
//   "input": "I was charged twice for my subscription last month",
//   "expected": {
//     "tone": "empathetic",
//     "mentions": "refund OR credit OR billing",
//     "format": "paragraph under 150 words"
//   }
// }

I maintain 30-50 golden cases per feature. They cover common paths, known edge cases, and a few adversarial inputs. I run them weekly and after every prompt or model change.

The golden set is your regression detector . When a prompt change causes three previously passing golden cases to fail, you get a concrete signal that something shifted. No vibes. No arguments. Data.

The evaluation cadence that works

After trying several approaches, here’s what I’ve settled on:

• Every commit: Run deterministic tests (layer 1). These are in CI and they block merges. Fast, stable, non-negotiable.
• Every prompt/model change: Run the golden set (layer 3) and compare to the previous baseline. If pass rate drops, the change needs review.
• Weekly: Run the full evaluation suite (layers 2 + 3) and track trends. Output a simple report: pass rate by criteria, any new failures, average response length.
• After major updates: Human review of a random sample (~20 cases). Sanity check that the automated evaluation isn’t missing something.

This takes about two hours a week of human time. That’s a small investment for the confidence it provides.

What I wish more teams did

Version your prompts. Every prompt change should be a tracked commit with a diff. When quality regresses, you need to know which prompt version caused it. I keep prompts in version-controlled files, not in application code.

Track quality over time. A single evaluation run is a snapshot. A time series of evaluation results shows trends. Is quality gradually degrading? Did a model provider update cause a step change? You can’t answer these without historical data.

Test adversarial inputs. Your golden set should include attempts to jailbreak, confuse, or extract system prompts . Real users will try all of these.

LLM testing can’t prove the model is correct. It can build enough evidence that the system behaves acceptably across the inputs that matter.

Most production LLM tasks I see are classification, extraction, formatting, and short generation. Intent routing for a support bot. Extracting structured data from emails. Labeling inbound requests. These don’t need GPT-4 or Claude Opus. They need a model that’s fast, cheap, and predictable.

A small model running a well-scoped task will beat a large model running a vague one. Every time.

Where small wins

Small models shine when the output space is narrow and the success criteria are clear. If you can describe the correct answer format in one sentence, a small model can probably handle it: classification with a fixed label set, entity extraction with a defined schema, or reformatting text from one structure to another.

The advantages are not marginal. A Haiku-class model might respond in 200ms at a fraction of a cent per request. The same task on a frontier model might take 2 seconds and cost 10x more. At scale, that difference is the gap between a sustainable product and one that burns through runway.

I switched an intent router from GPT-4 to a small model last month. Accuracy stayed within 1%. Latency dropped 80%. Monthly inference cost dropped from $12K to under $2K. The engineering effort was two days of prompt tuning and evaluation.

Where small fails

Small models fall apart when the task requires multi-step reasoning, nuanced judgment, or long-form coherence. If you need a model to read a 10-page contract and identify three specific risks, it will miss things. If you need it to write a persuasive email that matches a specific executive’s tone, it will usually produce something generic.

The failure mode is subtle. Small models don’t refuse. They confidently produce mediocre output. You won’t see errors. You’ll see output that’s 80% right and 20% subtly wrong in ways that are hard to catch without careful evaluation .

The routing pattern

The most cost-effective architecture I’ve built is a two-tier system . Small model handles the 90% of requests that are well-scoped and predictable. Large model handles the 10% that need depth.

Route by complexity, not by topic. A billing question that maps to one of five categories goes to the small model. A billing dispute that requires reading context and making a judgment call goes to the large model. The router itself can be a small model. It’s just a classification task.

This isn’t novel. It’s the same pattern as having junior engineers handle tickets and escalate to seniors, with the same economics.

Pick the smallest model that clears the bar

Don’t start with the biggest model and optimize later. Start with the smallest model and prove it’s insufficient before upgrading. You’ll be surprised how often “insufficient” never arrives.

The best model is the smallest one that meets your quality bar, at a cost and latency you can sustain.

This is a rant. But it’s a rant with actionable advice.

The “just throw it all in” fallacy

The pattern I keep seeing: a team builds a RAG pipeline that retrieves 20 document chunks for every query. They concatenate everything into the prompt because “more context is better.” The model now has 80K tokens of input, 60K of them irrelevant. The response is slower, more expensive, and, this is the part that kills me, lower quality than if they had sent 5K tokens of relevant context.

Retrieval isn’t free just because the window is big enough to hold it. Every irrelevant token dilutes the signal. The model has to figure out which parts of the context actually matter, and it isn’t always good at that, especially when the relevant information is sandwiched between walls of noise.

I reviewed a system where they were spending $400/day on inference . We cut their context budget by 70%, and quality went up. Not down. Up. The model could finally see the signal instead of drowning in noise.

Budget your context like you budget your infrastructure

You wouldn’t provision 10x the compute you need and call it a day. Don’t do it with context either.

Set a hard budget per request. Something like:

• System prompt: 1-2K tokens (this should be stable and tight)
• Retrieved context: 3-5K tokens max (be aggressive about relevance filtering)
• Conversation history: 2-4K tokens (recent turns verbatim, older turns summarized)
• Reserve: 1K tokens (for the model’s response and any overhead)

That’s 7-12K tokens for most requests. Not 128K. Not even close. And for 90% of production use cases, that’s more than enough.

Teams using 128K tokens per request are either doing something genuinely complex (document analysis, long-form generation) or being lazy. Mostly the latter.

Anchors: the stuff that must never fall out

Some information is non-negotiable. The user’s permissions. The current task definition. Key constraints. Explicit decisions made earlier in the conversation. I call these “anchors.”

Anchors go at the top of the context, every time. They don’t get summarized. They don’t get rotated out. They’re the ground truth that the model needs to respect regardless of how long the conversation gets.

I’ve debugged conversations where the model contradicted an earlier decision because the decision was in a turn that got summarized away. The summary said “the user chose option A” but the model treated it as a suggestion, not a commitment. Anchors prevent this.

Summaries need maintenance

Speaking of summaries: if you’re compressing conversation history into summaries, you need to refresh them. A summary generated 20 turns ago may be inaccurate or incomplete relative to the current state of the conversation.

The pattern I use is simple: keep the last 3-5 turns verbatim. Everything before that gets summarized. Refresh the summary every 10 turns or whenever a significant decision changes. It’s a small amount of extra work, and it prevents a category of bugs that’s extremely difficult to diagnose.

In retrieval, precision beats recall

Most RAG implementations err on the side of including too much. The logic goes: “better to include something irrelevant than to miss something important.” That sounds reasonable until you look at the actual failure modes.

From what I’ve seen, the most common production failure is the model having too much context and picking the wrong information. Missing context comes up far less often. Over-retrieval causes the model to confidently cite irrelevant passages while ignoring the one paragraph that actually answers the question.

Retrieve less. Filter aggressively. If you aren’t sure a chunk is relevant, leave it out. The model can ask follow-up questions. It can’t unsee irrelevant context.

Measure context utilization

Most teams have no idea how their context utilization looks in production. They don’t track average context size, the ratio of relevant to irrelevant tokens, or the correlation between context size and output quality. They just set a max limit and hope for the best.

Instrument your context pipeline. Log the size of each section (system prompt, retrieved context, history, anchors). Track output quality as a function of context size. You’ll almost certainly discover that your sweet spot is much smaller than your current usage.

Bigger windows are a genuine improvement. They let you handle tasks that were impossible before. But for most production workloads, the discipline of managing context well matters more than the ability to stuff more into it.

Function calling turned LLMs from text generators into system operators. That’s the opportunity and the risk. A model that can create tickets, query databases, and trigger deployments is powerful. A model that does those things with unvalidated arguments and no permission checks is a security incident waiting to happen.

I’ve built function calling integrations in past projects (mostly in Go), and the patterns that survive production are boring. That’s the point. Here’s what I’ve learned.

The mental model

Think of function calling as an API gateway where the caller is an LLM instead of a user. The model sees a list of available tools with schemas, picks one, and returns arguments as JSON. Your backend validates, executes, and returns results. The model then uses the results to continue the conversation.

User prompt + tool definitions
        |
        v
  Model selects tool + arguments (JSON)
        |
        v
  Backend validates arguments
        |
        v
  Backend executes tool (with permissions)
        |
        v
  Structured result returned to model
        |
        v
  Model generates final response

Simple in theory. In practice, the complexity is in validation, permissions, and error handling. That’s where most teams cut corners, and where most production incidents start.

Tool definitions: treat them like API contracts

A tool definition is a contract. The model’s behavior is only as good as the schema you provide. Vague descriptions produce vague arguments. Loose types produce invalid inputs.

In Go, I define tools as structs with explicit JSON Schema generation:

// ToolDef represents a callable tool exposed to the LLM.
type ToolDef struct {
    Name        string      `json:"name"`
    Description string      `json:"description"`
    Parameters  JSONSchema  `json:"parameters"`
    Handler     ToolHandler `json:"-"`
    Permission  Permission  `json:"-"`
}

type JSONSchema struct {
    Type       string                `json:"type"`
    Properties map[string]Property   `json:"properties"`
    Required   []string              `json:"required"`
}

type Property struct {
    Type        string   `json:"type"`
    Description string   `json:"description,omitempty"`
    Enum        []string `json:"enum,omitempty"`
    Default     string   `json:"default,omitempty"`
}

type ToolHandler func(ctx context.Context, args json.RawMessage) (*ToolResult, error)

A concrete example, a ticket creation tool:

var createTicketTool = ToolDef{
    Name:        "create_ticket",
    Description: "Create a support ticket. Requires a verified user session.",
    Parameters: JSONSchema{
        Type: "object",
        Properties: map[string]Property{
            "subject":  {Type: "string", Description: "Short summary of the issue"},
            "category": {Type: "string", Enum: []string{"billing", "bug", "account", "other"}},
            "priority": {Type: "string", Enum: []string{"low", "normal", "high"}, Default: "normal"},
        },
        Required: []string{"subject", "category"},
    },
    Handler:    handleCreateTicket,
    Permission: PermWriteApproval,
}

Notice the pattern: enums on every field with a bounded set of values, a clear description that tells the model when to use the tool, and required fields marked explicitly. The model doesn’t guess. It follows the contract.

The tool registry

Centralize tool registration. Don’t scatter tool definitions across your codebase. A single registry makes it easy to generate schemas for the model, enforce permissions, and audit what’s available.

type Registry struct {
    mu    sync.RWMutex
    tools map[string]ToolDef
}

func NewRegistry() *Registry {
    return &Registry{tools: make(map[string]ToolDef)}
}

func (r *Registry) Register(tool ToolDef) {
    r.mu.Lock()
    defer r.mu.Unlock()
    r.tools[tool.Name] = tool
}

func (r *Registry) Schema() []map[string]any {
    r.mu.RLock()
    defer r.mu.RUnlock()

    out := make([]map[string]any, 0, len(r.tools))
    for _, t := range r.tools {
        out = append(out, map[string]any{
            "type": "function",
            "function": map[string]any{
                "name":        t.Name,
                "description": t.Description,
                "parameters":  t.Parameters,
            },
        })
    }
    return out
}

func (r *Registry) Execute(ctx context.Context, name string, args json.RawMessage) (*ToolResult, error) {
    r.mu.RLock()
    tool, ok := r.tools[name]
    r.mu.RUnlock()

    if !ok {
        return &ToolResult{
            Success:   false,
            ErrorCode: "unknown_tool",
            Message:   fmt.Sprintf("tool %q not found", name),
        }, nil
    }

    return tool.Handler(ctx, args)
}

The Execute method is intentionally minimal. Validation and permission checks happen in the layers around it, not inside the registry itself. Separation of concerns matters here because you’ll want to add middleware later without rewriting the registry.

Validation: the model isn’t trusted

This is the hill I’ll die on: model-generated arguments are untrusted input. Always. Even with a tight schema, the model can produce unexpected values: empty strings, null where you expect a value, or fields that technically match the type but are nonsensical.

type CreateTicketArgs struct {
    Subject  string `json:"subject"`
    Category string `json:"category"`
    Priority string `json:"priority"`
}

func validateCreateTicketArgs(raw json.RawMessage) (*CreateTicketArgs, error) {
    var args CreateTicketArgs
    if err := json.Unmarshal(raw, &args); err != nil {
        return nil, fmt.Errorf("invalid JSON: %w", err)
    }

    args.Subject = strings.TrimSpace(args.Subject)
    if args.Subject == "" {
        return nil, fmt.Errorf("subject must be non-empty")
    }
    if len(args.Subject) > 200 {
        return nil, fmt.Errorf("subject exceeds 200 characters")
    }

    validCategories := map[string]bool{"billing": true, "bug": true, "account": true, "other": true}
    if !validCategories[args.Category] {
        return nil, fmt.Errorf("invalid category: %q", args.Category)
    }

    if args.Priority == "" {
        args.Priority = "normal"
    }
    validPriorities := map[string]bool{"low": true, "normal": true, "high": true}
    if !validPriorities[args.Priority] {
        return nil, fmt.Errorf("invalid priority: %q", args.Priority)
    }

    return &args, nil
}

Yes, this is verbose. That’s deliberate. I don’t want clever one-liners here. I want code that a new team member can read at 3 AM during an incident and immediately understand what it checks and why.

Structured errors that the model can recover from

When validation fails, return a structured error the model can act on. Not a stack trace. Not a generic “bad request.” A clear envelope:

type ToolResult struct {
    Success   bool   `json:"success"`
    ErrorCode string `json:"error_code,omitempty"`
    Message   string `json:"message,omitempty"`
    Data      any    `json:"data,omitempty"`
}

The model sees this and can retry with corrected arguments, ask the user for clarification, or explain the failure. Unstructured errors produce unstructured recovery attempts. I’ve seen models apologize to users for “server errors” when the actual problem was a missing required field.

Permission scoping

Every tool gets a permission level, the same blast-radius thinking that applies to agents. Every request carries user context. The execution layer checks permissions before calling the handler. No exceptions.

type Permission int

const (
    PermReadOnly Permission = iota
    PermWriteApproval
    PermAdminOnly
)

type ExecContext struct {
    UserID    string
    Role      string
    SessionID string
}

func (r *Registry) ExecuteWithAuth(ctx context.Context, ec ExecContext, name string, args json.RawMessage) (*ToolResult, error) {
    r.mu.RLock()
    tool, ok := r.tools[name]
    r.mu.RUnlock()

    if !ok {
        return &ToolResult{Success: false, ErrorCode: "unknown_tool"}, nil
    }

    if !hasPermission(ec.Role, tool.Permission) {
        return &ToolResult{
            Success:   false,
            ErrorCode: "permission_denied",
            Message:   fmt.Sprintf("role %q cannot execute %q", ec.Role, name),
        }, nil
    }

    return tool.Handler(ctx, args)
}

func hasPermission(role string, required Permission) bool {
    switch required {
    case PermReadOnly:
        return true
    case PermWriteApproval:
        return role == "user" || role == "admin"
    case PermAdminOnly:
        return role == "admin"
    default:
        return false
    }
}

The model doesn’t decide permissions. The backend does. I’ve seen demos where the model is told “you have admin access” in the system prompt. That isn’t a permission system. That’s a suggestion.

Parallel execution with guardrails

Some models support parallel tool calls. This can cut latency significantly when tools are independent, but you still need timeouts and isolation.

func executeParallel(ctx context.Context, registry *Registry, ec ExecContext, calls []ToolCall) []*ToolResult {
    ctx, cancel := context.WithTimeout(ctx, 8*time.Second)
    defer cancel()

    results := make([]*ToolResult, len(calls))
    var wg sync.WaitGroup

    for i, call := range calls {
        wg.Add(1)
        go func(idx int, c ToolCall) {
            defer wg.Done()
            result, err := registry.ExecuteWithAuth(ctx, ec, c.Name, c.Arguments)
            if err != nil {
                results[idx] = &ToolResult{Success: false, ErrorCode: "execution_error", Message: err.Error()}
                return
            }
            results[idx] = result
        }(i, call)
    }

    wg.Wait()
    return results
}

The timeout is critical. A slow tool shouldn’t block the entire response. Return partial results and let the model work with what it has.

Observability

Log every tool call. But be smart about what you log:

• Tool name and version
• User ID and session ID
• Argument hash (not raw arguments, which may contain PII)
• Success/failure and error code
• Execution latency

This gives you enough to debug failures, detect drift (is the model suddenly calling a tool it never used before?), and identify tools that are slow, failing, or overused.

What I wish I had known earlier

After building several of these systems, a few lessons stand out:

Keep tool descriptions short and precise. The model reads them on every request. Long descriptions waste tokens and confuse tool selection. One sentence describing the action, one sentence about when to use it.

Version your tool schemas. When you change a tool’s parameters, the model’s behavior will change too. Treat schema changes like API migrations.

Test with adversarial inputs. Ask the model to call tools with garbage arguments, impossible combinations, and injection attempts . Your validation layer should handle all of these cleanly.

Function calling is the interface between language models and real systems. The clever part is the model. Your job is to make the execution layer as predictable as possible.

Anthropic released Claude 3.5 Sonnet alongside a new Artifacts interface, and I’ve been running it against my usual workloads since launch day. This isn’t a benchmark review. Benchmarks tell you how a model performs on someone else’s problems. I care about how it performs on mine.

The positioning shift that matters

Every model provider has a lineup: cheap-and-fast at the bottom, expensive-and-smart at the top. The default instinct for production teams is to reach for the top tier because the cost of a bad output usually outweighs the cost of inference.

Claude 3.5 Sonnet challenges that instinct. Anthropic is explicitly positioning a mid-tier model as the default for serious work. It’s a pricing play and also a claim that the quality gap between tiers has narrowed enough that the mid-tier clears the bar for most real-world tasks. That is the same routing question I keep coming back to: which requests deserve the expensive path?

I’ve been testing this claim. Here is what stood out.

Coding: where it impressed me

I ran Sonnet through the types of coding tasks I deal with in my Go-heavy workflow:

Multi-file refactors. I asked it to rename a package, update all references, and adjust the tests. Sonnet got this right on the first try, including edge cases in test helper files that GPT-4 had missed when I ran the same task a month earlier.

Bug diagnosis from error traces. I pasted a stack trace from a concurrency bug in a Go service. Sonnet identified the race condition, explained why it manifested only under load, and proposed a fix using sync.Mutex that was correct and idiomatic. It didn’t suggest sync.Map when a plain mutex was the right call. That kind of judgment matters.

Documentation from code. I gave it a 200-line Go package and asked for a README. The output was usable with minor edits. It captured the intent, not just the function signatures.

These are the tasks where I spend real time. A model that handles them reliably at a lower price point changes how I think about routing.

Where Opus still wins

I found Sonnet’s limits in a few predictable places:

Long-form reasoning across large contexts. When I loaded a full design document (~15K tokens) and asked for a critique, Sonnet’s analysis was surface-level compared to Opus. It identified structural issues but missed a subtle consistency problem that Opus caught.

Ambiguous instructions. When the prompt is vague, Sonnet tends to make reasonable but sometimes wrong assumptions instead of asking for clarification. This is manageable (you just need more explicit prompts), but it means you can’t be lazy with your instructions.

Creative writing. Not my primary use case, but I noticed it. Sonnet’s prose is competent but flat. If you need compelling narrative or nuanced tone, Opus is still noticeably better.

Artifacts: more useful than I expected

I was skeptical of Artifacts when I saw the announcement. It looked like a UI gimmick. A few days of daily use changed my mind.

The core idea: when the model produces code, a document, or a visualization, it renders it in a separate panel instead of inline in chat. You iterate on it through the conversation, then copy or download the result. The model treats it as a persistent object in the conversation.

Where this is genuinely useful:

• Prototyping UI components. Ask for a React component, see it rendered, ask for changes, see the update. The feedback loop is fast.
• Drafting specs. The artifact is a living document that you refine through conversation. Much better than scrolling through a chat history to find the latest version.
• Quick visualizations. SVG diagrams, simple charts, Mermaid flowcharts. The inline render makes iteration practical.

It’s a real workflow improvement for anyone using an LLM for iterative creation.

How I’d evaluate this for your team

Don’t take my word for it. Run your own comparison . Here’s the approach I recommend:

1. Pick 10-15 real tasks from your last two sprints. Skip toy problems and use actual things your team spent time on. Code reviews, bug fixes, documentation, data analysis.
2. Run them through Sonnet and your current default model side by side. Same prompts, same context.
3. Score on three dimensions: correctness, usefulness (did you use the output or throw it away), and time saved.
4. Compare cost and latency. Sonnet should be meaningfully cheaper and faster. If the quality is comparable, the math is obvious.

Do this for a week, not an afternoon. First impressions are unreliable. You need enough data points to see the failure modes, not just the wins.

The model routing question

The bigger implication of Sonnet is routing, more than any Opus-versus-Sonnet swap.

Most teams use one model for everything. That was reasonable when the quality gap between tiers was large. Now that the gap is narrowing, a smarter approach is to route by task:

• Sonnet for coding, classification, extraction, structured output, and most day-to-day work.
• Opus for complex reasoning, nuanced analysis, and tasks where the cost of a wrong answer is high.
• Haiku for preprocessing, filtering, and high-volume tasks where speed matters more than depth.

Keep model identifiers and routing rules in config so you can shift traffic as models improve without redeploying.

Claude 3.5 Sonnet is the first mid-tier model where I stopped reaching for the top tier by default. Don’t switch blindly: test on your workloads, measure the quality gap, and route by task. For most teams, that will mean moving a significant chunk of traffic to Sonnet and saving the heavyweight model for the tasks that genuinely need it.

The gap between a voice AI demo and a voice AI product is about six months of work on things nobody finds exciting: latency tuning, interruption handling, and figuring out what happens when the user mumbles, changes their mind, or goes silent for eight seconds.

My recent voice work has been real-time support and call-center tooling. The models have gotten dramatically better. The engineering around them hasn’t kept pace.

Two architectures, one tradeoff

You have two practical options for a voice AI system:

Modular pipeline: Separate services for transcription, reasoning, and synthesis. You can swap components, instrument each stage, and debug failures in isolation. The cost is latency at every boundary.

mic -> STT service -> LLM -> TTS service -> speaker
         ~200ms       ~800ms     ~300ms

End-to-end model: A single model like GPT-4o that handles audio natively. OpenAI hasn’t opened GPT-4o’s audio to general API users yet, but the promise is lower latency and a more natural feel. The cost: harder to debug, and you’re locked to one provider’s capabilities.

I lean modular for anything going to production. Here’s why: when a user reports “the bot said something weird,” I need to know whether it was a transcription error, a reasoning failure, or a synthesis artifact. With an end-to-end model, that’s a black box.

The streaming architecture that matters

The biggest latency win comes from streaming, more than from model speed. Start synthesizing audio before the full response is generated. In Go, it looks something like:

type VoiceSession struct {
    sttClient    STTClient
    llm          LLMClient
    ttsClient    TTSClient
    audioOut     chan []byte
    interrupted  atomic.Bool
}

func (s *VoiceSession) HandleUtterance(ctx context.Context, audio []byte) error {
    // Transcribe
    transcript, err := s.sttClient.Transcribe(ctx, audio)
    if err != nil {
        return fmt.Errorf("transcription failed: %w", err)
    }

    // Stream LLM response, pipe chunks directly to TTS
    stream, err := s.llm.StreamChat(ctx, transcript)
    if err != nil {
        return fmt.Errorf("llm stream failed: %w", err)
    }

    var buf strings.Builder
    for chunk := range stream {
        if s.interrupted.Load() {
            return nil // User interrupted, stop generating
        }

        buf.WriteString(chunk.Text)

        // Flush to TTS at sentence boundaries
        if isSentenceEnd(buf.String()) {
            audioChunk, err := s.ttsClient.Synthesize(ctx, buf.String())
            if err != nil {
                continue // Degrade gracefully, skip this chunk
            }
            s.audioOut <- audioChunk
            buf.Reset()
        }
    }

    // Flush remaining text
    if buf.Len() > 0 {
        audioChunk, _ := s.ttsClient.Synthesize(ctx, buf.String())
        s.audioOut <- audioChunk
    }

    return nil
}

Flush to TTS at sentence boundaries, not at the end of the full response. The user hears the first sentence while the model is still generating the third. Perceived latency drops from 1300ms to under 500ms.

Interruptions are the normal case

People interrupt. They talk over the bot. They say “wait, no, actually…” halfway through a sentence. If your system can’t handle this, users will hate it within 30 seconds.

The interrupt handler needs to do three things fast:

1. Stop audio output immediately. Not after the current sentence. Now.
2. Cancel pending TTS and LLM generation. Don’t waste compute on a response nobody will hear.
3. Accept the new input without resetting the conversation. Context should carry over.

func (s *VoiceSession) HandleInterrupt(ctx context.Context, newAudio []byte) error {
    s.interrupted.Store(true)

    // Drain the audio output channel
    for len(s.audioOut) > 0 {
        <-s.audioOut
    }

    s.interrupted.Store(false)
    return s.HandleUtterance(ctx, newAudio)
}

This is simplified, but the pattern holds. The atomic.Bool flag propagates interrupts to the streaming loop without complex synchronization.

When voice is the wrong interface

Voice is great when:

• The user’s hands are busy (driving, cooking, field work)
• The task has a narrow, predictable vocabulary
• The expected output is short: a confirmation, a lookup, a simple action

Voice is terrible when:

• The user needs to compare options visually
• The output is complex or structured (tables, code, lists)
• Precision matters more than speed (medical, legal, financial details)

I keep seeing teams try to build “voice-first everything” products. Don’t do this. Voice should be one input mode in a system that gracefully falls back to text or visual UI when the task demands it.

Operational concerns that will bite you

Transcription accuracy varies wildly by accent, background noise, and microphone quality. Test with real users in real environments, not in a quiet office with a studio mic. I learned this the hard way: a prototype that worked perfectly in our office fell apart in a warehouse with forklift noise.

Track these metrics from day one:

• Transcription word error rate by user segment
• Time to first audio byte (perceived latency)
• Interruption rate and recovery success
• Conversation completion rate vs. abandonment
• Fallback-to-text rate

Cost adds up fast. A 30-second voice interaction can involve a STT call, an LLM call with conversation history, and a TTS call. Multiply by thousands of daily users and you need a cost model before you launch, not after.

Narrow products win

The best voice AI products I’ve seen are boring. They do one thing, they do it fast, and they handle failure gracefully. A voice ordering system that works for 50 menu items. A voice-controlled inventory check. A hands-free incident report dictation tool.

Nobody is going to have a deep philosophical conversation with your voice bot. They want to get something done and move on. Design for that.

The tech is ready. The hard part is the discipline to ship something narrow and reliable instead of something ambitious and fragile.

Then the demo high wore off, and the real questions started.

What shipped, and what’s still coming

GPT-4o is a single model that handles text, images, and audio natively. No more chaining a Whisper transcription into GPT-4 into a TTS engine. One model, one round trip, multiple modalities.

That sounds incremental until you think about what it kills: the glue. I’ve spent more time than I want to admit debugging pipelines where context got lost between the speech-to-text step and the reasoning step, or where the TTS output sounded robotic because the model had no awareness it was producing spoken words. GPT-4o’s design collapses that entire pipeline into a single inference call.

Fewer seams means fewer places for things to break. That matters more than any benchmark.

Developers get less of it today than the demo suggests. The API launches GPT-4o as a text and vision model. OpenAI says the audio and video capabilities will reach “a small group of trusted partners” in the API in the coming weeks, and the new Voice Mode goes to ChatGPT Plus in alpha on a similar timeline.

Where this changes product design

The interesting shift is that users no longer have to context-switch between modalities. Show the camera, describe the problem, get an answer, all in one continuous loop.

I’ve been advising a couple of teams building support tools, and this unlocks patterns that were previously too brittle to ship:

• Live visual troubleshooting. User points their phone at the broken thing, explains the issue, and the model responds while looking at the same image. No more “please upload a screenshot and describe what happened.”
• Hands-free workflows. Voice as primary input, text as structured output. Think field technicians, warehouse workers, anyone whose hands are occupied.
• Coaching and tutoring. The model sees the student’s work and talks through corrections in real time. This was a three-service pipeline before. Once audio reaches the API, it can be one call.

These are products teams tried to build last year and abandoned because latency and context loss across the pipeline made them unusable.

The complexity doesn’t disappear

Here is what the demo didn’t show: the model is faster and more unified, but the infrastructure around it is still hard.

Streaming audio over unreliable mobile networks is an unsolved problem in most organizations. Encoding images in real time on low-end devices is a performance cliff. And once you’re processing audio and video from users, you have entered a privacy and consent minefield that most teams haven’t mapped.

A single model simplifies the AI layer. It doesn’t simplify the transport layer, the device layer, or the compliance layer. If anything, it makes those harder because the demo sets expectations that the infrastructure can’t meet yet.

My advice to the teams I work with: “The model is ready. Your CDN isn’t.”

How I’d evaluate this

When API access is fresh and the documentation is still evolving, the worst thing you can do is build something ambitious. Pick the narrowest possible workflow. Once audio is available to you, something like: user speaks a question, model responds with text and audio. No vision, no tool calling, just the core loop.

Measure three things:

1. Does the end-to-end interaction feel natural, or does the latency break the illusion?
2. How does it behave with bad audio: background noise, accents, interruptions?
3. What does failure look like, and can the UI recover without the user noticing?

If you can’t answer those three questions with your prototype, you aren’t ready to expand scope. Ship the boring version first.

The consent problem nobody talks about

Real-time multimodal means you’re potentially recording and processing audio and video from real people. That’s a different legal and ethical surface than processing text prompts.

You need explicit consent flows. You need to decide what gets stored and what gets discarded after inference. You need a plan for when the model misinterprets visual input in a way that’s embarrassing or harmful. Most of the teams I’ve talked to are hand-waving this. Don’t be one of them.

The model is the easy layer

GPT-4o is a genuine architecture shift. It eliminates an entire class of integration problems and makes products possible that weren’t viable six months ago.

But the hard part was never the model. The hard part is reliable transport, device compatibility, privacy, and graceful degradation. The teams that win with this will be the ones who treat the model as the easy layer and invest in everything around it.

That sounds obvious. In practice, it isn’t. I still see production systems parsing LLM output with string splitting and regex. They work until they don’t, and when they break, they fail in ways that are hard to diagnose because the failure is subtle data corruption, not a crash.

Structured output from LLMs is a solved problem if you treat it as contract enforcement rather than a prompting exercise. Define what you expect. Tell the model exactly what you expect. Validate what you get. Repair what breaks. I do this in Go with about 300 lines of reusable code; here are the core pieces. This is one of the control surfaces that belongs in any serious AI-native architecture and evaluation pipeline .

The failure modes are predictable

LLMs generate text. They don’t generate data structures. Even with strong prompting, they will occasionally:

• Wrap the JSON in markdown code fences or explanatory prose
• Omit fields they consider “obvious” or irrelevant
• Use wrong types (string "null" instead of JSON null, number as string)
• Rename fields to something they think is more descriptive
• Produce partial output when hitting token limits

Every pattern in this post targets one of these failures. Expect all of them in production.

Define the contract as Go types

Start with the output structure. It’s both the validation target and the deserialization target. One definition serves both purposes.

type ContactInfo struct {
	Name    string  `json:"name"    validate:"required,min=1"`
	Email   *string `json:"email"   validate:"omitempty,email"`
	Company *string `json:"company"`
	Role    *string `json:"role"`
}

Nullable fields use pointers. Required fields use value types. The validate tags drive runtime validation. This struct is the single source of truth: the prompt references it, the validator enforces it, and the calling code consumes it.

I also generate a JSON Schema from the struct for inclusion in prompts. This keeps the prompt and validation in sync automatically:

func SchemaFor[T any]() ([]byte, error) {
	reflector := jsonschema.Reflector{
		RequiredFromJSONSchemaTags: true,
		DoNotReference:             true,
	}
	schema := reflector.Reflect(new(T))
	return json.MarshalIndent(schema, "", "  ")
}

One definition. One schema. No drift between what you ask for and what you validate.

Build the prompt to minimize ambiguity

The prompt should be rigid and specific. No motivational language. No “please try your best.” Just the schema, the rules, and the input.

func BuildExtractionPrompt(schema []byte, input string) string {
	return fmt.Sprintf(`Extract structured data from the input. Return ONLY valid JSON matching this schema:

%s

Rules:
- Use null for missing fields, not empty strings
- Lowercase email addresses
- No additional keys beyond the schema
- No markdown, no explanation, just the JSON object

Input:
%s

JSON:`, string(schema), input)
}

The JSON: at the end is a small trick that helps. It primes the model to start generating JSON immediately instead of opening with “Here is the extracted data:” or similar preamble.

The extraction pipeline

This is the core of the system: call the model, clean the response, parse it, validate it, and retry on failure.

type Extractor[T any] struct {
	client     LLMClient
	validator  *validator.Validate
	schema     []byte
	maxRetries int
}

func NewExtractor[T any](client LLMClient, maxRetries int) (*Extractor[T], error) {
	schema, err := SchemaFor[T]()
	if err != nil {
		return nil, fmt.Errorf("generating schema: %w", err)
	}

	return &Extractor[T]{
		client:     client,
		validator:  validator.New(),
		schema:     schema,
		maxRetries: maxRetries,
	}, nil
}

func (e *Extractor[T]) Extract(ctx context.Context, input string) (*T, error) {
	prompt := BuildExtractionPrompt(e.schema, input)
	var lastErr error

	for attempt := range e.maxRetries {
		raw, err := e.client.Generate(ctx, prompt)
		if err != nil {
			return nil, fmt.Errorf("llm call failed: %w", err)
		}

		cleaned := cleanJSONResponse(raw)

		var result T
		if err := json.Unmarshal([]byte(cleaned), &result); err != nil {
			lastErr = fmt.Errorf("attempt %d: json parse error: %w", attempt+1, err)
			prompt = buildRepairPrompt(prompt, raw, err.Error())
			continue
		}

		if err := e.validator.Struct(result); err != nil {
			lastErr = fmt.Errorf("attempt %d: validation error: %w", attempt+1, err)
			prompt = buildRepairPrompt(prompt, raw, err.Error())
			continue
		}

		return &result, nil
	}

	return nil, fmt.Errorf("extraction failed after %d attempts: %w", e.maxRetries, lastErr)
}

A few things to notice. The generic type parameter means this extractor works for any output struct: ContactInfo, InvoiceData, whatever. The cleaning step handles the most common format issues before parsing. And on failure, the repair prompt feeds the error back to the model so it can fix the specific problem.

Cleaning the response

Models love to wrap JSON in markdown code fences or add explanatory text. This function strips that away:

func cleanJSONResponse(raw string) string {
	s := strings.TrimSpace(raw)

	// Strip markdown code fences (single-line fences fall through to brace extraction)
	if strings.HasPrefix(s, "```") && strings.Contains(s, "\n") {
		lines := strings.Split(s, "\n")
		// Remove first line (```json) and last line (```) if present;
		// truncated output may have no closing fence
		lines = lines[1:]
		if n := len(lines); n > 0 && strings.TrimSpace(lines[n-1]) == "```" {
			lines = lines[:n-1]
		}
		s = strings.Join(lines, "\n")
	}

	// Find the first { and last } to extract the JSON object
	firstBrace := strings.Index(s, "{")
	lastBrace := strings.LastIndex(s, "}")
	if firstBrace >= 0 && lastBrace > firstBrace {
		s = s[firstBrace : lastBrace+1]
	}

	return strings.TrimSpace(s)
}

This isn’t pretty. It doesn’t need to be. It handles the three wrapping patterns I most often see in production: code fences, leading prose, and trailing explanation.

The repair prompt

When parsing or validation fails, the repair prompt tells the model exactly what went wrong:

func buildRepairPrompt(originalPrompt, badOutput, errorMsg string) string {
	return fmt.Sprintf(`%s

Your previous output was invalid:
%s

Error: %s

Fix the error and return ONLY valid JSON.

JSON:`, originalPrompt, badOutput, errorMsg)
}

This is where the retry loop earns its keep. The model gets the original instructions, sees its own bad output, and gets a specific error message to fix.

From what I’ve seen, this recovers about 80% of validation failures on the first retry. The remaining 20% usually indicate a genuinely ambiguous input that needs human review.

Use JSON mode when available

Several model APIs now offer a JSON-only response mode (OpenAI’s is the best known). Use it where you have it. It eliminates prose wrapping entirely and significantly reduces parsing failures.

func (e *Extractor[T]) Extract(ctx context.Context, input string) (*T, error) {
	prompt := BuildExtractionPrompt(e.schema, input)
	opts := GenerateOptions{
		ResponseFormat: ResponseFormatJSON, // Use JSON mode
	}

	// ... rest of the extraction logic
}

But JSON mode doesn’t mean you skip validation. The model can still omit required fields, use wrong types, or produce a valid JSON object that doesn’t match your schema. JSON mode guarantees parseable JSON. It doesn’t guarantee correct JSON for your use case.

Monitoring structured output in production

Three metrics I track for every structured-output pipeline:

1. Parse success rate. What percentage of responses parse and validate on the first attempt? If this drops below 95%, something changed: the model updated, the prompt drifted, or the input distribution shifted.
2. Retry rate and recovery rate. How often do you need retries, and how often do retries succeed? A high retry rate with good recovery means the repair loop is working. A high retry rate with low recovery means something is fundamentally wrong.
3. Field-level error distribution. Which fields cause the most validation failures? This tells you where the prompt needs to be more explicit or where the schema needs adjustment.

I log every extraction attempt : success or failure, first try or retry, with the raw model output. When something goes wrong in production, I want to see exactly what the model returned, not just that it failed.

None of this is clever. It’s the same contract-enforcement thinking we use everywhere else in software engineering. The model is an unreliable data source. Treat it like one.

I was reviewing a team’s API logs last month and found something depressing. About 40% of their LLM requests were functionally identical. Same system prompt, same user question (give or take whitespace), same model. They were paying full price for every single one.

Caching is the most boring and most effective optimization you can make to an LLM application. Exact-match caching catches repeats like these with zero quality loss; semantic caching can come later, carefully. The cache itself is easy. Key design and invalidation discipline are the hard part. Here is how I build it in Go.

Start with exact match caching

Don’t get fancy. The first layer is simple: hash the request, check the cache, return the cached response if it exists. This catches identical requests and costs almost nothing to implement.

type CacheKey struct {
	Version    string `json:"v"`
	Model      string `json:"model"`
	PromptHash string `json:"prompt_hash"`
	ToolsHash  string `json:"tools_hash"`
	ParamsHash string `json:"params_hash"`
}

func NewCacheKey(req LLMRequest) CacheKey {
	return CacheKey{
		Version:    "v1",
		Model:      req.Model,
		PromptHash: sha256Hash(req.SystemPrompt + "\n" + req.UserPrompt),
		ToolsHash:  sha256Hash(marshalTools(req.Tools)),
		ParamsHash: sha256Hash(fmt.Sprintf("%f:%d", req.Temperature, req.MaxTokens)),
	}
}

func (k CacheKey) String() string {
	b, _ := json.Marshal(k)
	return sha256Hash(string(b))
}

func sha256Hash(s string) string {
	h := sha256.Sum256([]byte(s))
	return hex.EncodeToString(h[:])
}

The key includes everything that can change the output: model, prompt content, tools, and sampling parameters. If any of those differ, you get a different key. If they are all the same, you get a cache hit.

Notice the version field. When you change your key schema (and you will), bump the version. This prevents old entries with a different key structure from colliding with new ones.

The cache layer itself

I keep the cache interface simple so the backing store can be swapped. In production I usually start with Redis. For testing and small deployments, an in-memory LRU works fine.

type LLMCache interface {
	Get(ctx context.Context, key string) (*CachedResponse, error)
	Set(ctx context.Context, key string, resp *CachedResponse, ttl time.Duration) error
	Delete(ctx context.Context, key string) error
}

type CachedResponse struct {
	Content   string    `json:"content"`
	Model     string    `json:"model"`
	TokensIn  int       `json:"tokens_in"`
	TokensOut int       `json:"tokens_out"`
	CachedAt  time.Time `json:"cached_at"`
}

func (s *Service) Generate(ctx context.Context, req LLMRequest) (*LLMResponse, error) {
	key := NewCacheKey(req).String()

	if cached, err := s.cache.Get(ctx, key); err == nil && cached != nil {
		s.metrics.CacheHit(req.Model)
		return &LLMResponse{
			Content:  cached.Content,
			Model:    cached.Model,
			FromCache: true,
		}, nil
	}

	s.metrics.CacheMiss(req.Model)

	resp, err := s.llmClient.Generate(ctx, req)
	if err != nil {
		return nil, err
	}

	cached := &CachedResponse{
		Content:   resp.Content,
		Model:     resp.Model,
		TokensIn:  resp.TokensIn,
		TokensOut: resp.TokensOut,
		CachedAt:  time.Now(),
	}

	// Fire and forget -- cache write failure should not block the response
	go func() {
		if setErr := s.cache.Set(context.Background(), key, cached, s.ttlFor(req)); setErr != nil {
			s.logger.Warn("cache set failed", "key", key, "error", setErr)
		}
	}()

	return resp, nil
}

A few things to note. The cache write is fire-and-forget. A failed cache write should never block or degrade the response to the user. The FromCache flag on the response is important for monitoring: you need to know what percentage of traffic is served from cache.

TTL strategy

This is where people get it wrong. They set a blanket TTL and call it done. Different content ages at different rates.

func (s *Service) ttlFor(req LLMRequest) time.Duration {
	// Responses grounded in static reference data can live longer
	if req.HasStaticContext() {
		return 24 * time.Hour
	}

	// Responses involving real-time data should be short-lived
	if req.HasLiveDataRetrieval() {
		return 5 * time.Minute
	}

	// Default: conservative TTL
	return 1 * time.Hour
}

Static context (a system prompt explaining how to format output, or reference documentation that changes monthly) can tolerate a long TTL. Responses that depend on live data need short TTLs or no caching at all. When in doubt, err toward shorter TTLs. A cache miss costs money. A stale response costs trust.

Invalidation beyond TTLs

TTLs are your baseline. But you also need event-driven invalidation for cases where you know the cache is stale.

Prompt changes are the big one. Every time you update a system prompt or retrieval pipeline, the old cached responses are wrong. The versioned key handles this naturally: a new prompt produces a new hash, which produces a new key, which misses the cache. Old entries expire on their own TTL.

For data-driven invalidation, I use a simple pattern:

func (s *Service) OnKnowledgeBaseUpdate(ctx context.Context, docIDs []string) {
	// Invalidate any cached responses that used these documents
	for _, docID := range docIDs {
		keys, err := s.cacheIndex.KeysForDocument(ctx, docID)
		if err != nil {
			s.logger.Error("failed to lookup cache keys for document", "doc_id", docID, "error", err)
			continue
		}
		for _, key := range keys {
			_ = s.cache.Delete(ctx, key)
		}
	}
}

This requires maintaining a secondary index that maps documents to cache keys. It’s more work, but for applications where correctness matters (and it usually does), it’s worth it.

What NOT to cache

Not every response should be cached. I have a short list of exclusions:

• User-specific sensitive responses. Unless your cache has strict tenant isolation, don’t risk serving User A’s response to User B. I’ve seen this bug in production. It’s exactly as bad as it sounds.
• Responses that depend on time-sensitive external state. Stock prices, live inventory, anything where a one-hour-old answer is wrong.
• Creative or generative tasks where variability is the feature. If the user expects a different response each time, caching defeats the purpose.

Measuring what matters

You need four metrics from day one:

1. Cache hit rate by request type. Not a global number. A 60% overall hit rate might mean 90% for classification and 10% for analysis. The per-type breakdown tells you where to focus.
2. Latency with and without cache. This quantifies the speed improvement and justifies the infrastructure cost.
3. Cost savings. Track tokens not consumed due to cache hits. Multiply by your per-token rate . Show this number to whoever pays the bills.
4. Quality signals on cached responses. User corrections, retries, and thumbs-down ratings. If cached responses get worse quality signals than fresh ones, your TTL is too long or your keys are too broad.

Roll out behind a flag

Don’t flip caching on for all traffic at once. Use a feature flag . Start with one request type that has high repetition and low sensitivity. Measure hit rate, latency, and quality for a week. Then expand.

When something goes wrong (and something always does), you want to be able to turn caching off in seconds. A feature flag gives you that.

This is the same infrastructure discipline we’ve applied to every other expensive external service call for decades. LLM calls are expensive enough that a 40% hit rate translates to real savings, and the money you save will pay for a lot of actual engineering work.

Another team, running a multi-model setup, barely noticed. Their routing layer shifted traffic to the fallback model within seconds. Quality dipped slightly on complex tasks. Users didn’t complain.

Guess which architecture I recommend now.

The case is boring, and that’s the point

Multi-model has nothing to do with chasing the latest release or playing model arbitrage. It applies the same boring infrastructure principles we’ve applied to databases, CDNs, and DNS for decades. Don’t have a single point of failure. Don’t lock yourself into one vendor. Have a plan for when things break.

With LLMs, the failure modes are broader than traditional services. A provider can go down entirely. Latency can spike. A model update can silently change behavior. Rate limits can throttle you during a traffic spike. Any of these will degrade your product if you have no alternative path.

How I think about routing

Routing doesn’t need to be sophisticated. I’ve seen teams over-engineer this with ML-powered classifiers that decide which model gets each request. That’s fun to build and painful to debug.

What works: simple rules based on task type and complexity.

Short classification tasks? Small, fast model. Interactive chat with a paying user? Mid-tier model with good latency. Complex analysis that needs deep reasoning? Big model. Fallback on timeout or error? The next model in the chain.

With the Claude 3 tiers , you can express this in a config file:

routing:
  default: "sonnet"
  rules:
    - task: "classify"
      model: "haiku"
    - task: "analyze"
      complexity: "high"
      model: "opus"
  fallback_chain: ["sonnet", "haiku"]
  timeout_ms: 10000

That’s it. No neural router. No reinforcement learning. Just explicit rules you can read, debug, and change in five minutes.

Routing belongs in configuration. When a new model drops or pricing changes, you update the config instead of refactoring a service.

The fallback chain is everything

Your fallback chain is more important than your primary model choice, because the primary model will be unavailable at some point.

Keep the chain short: two or three models. Set aggressive timeouts. And critically: log which model actually served each request. If you don’t, you have no idea what quality your users are actually getting. You think they’re getting Opus but half the traffic is silently falling back to Haiku because of rate limits.

I made this mistake early on, in a project at a large telecom company I worked with. We had a fallback in place but no logging on which model served the request. For two weeks, the primary model was rate-limited during peak hours and the fallback was handling 40% of traffic. We didn’t notice until a quality review showed unexpected patterns. Now I log every routing decision. Non-negotiable.

Cost management as a feature

Multi-model is also the most effective cost control mechanism I’ve found. Instead of running every request through the most capable (and expensive) model, you match model capability to task complexity.

The math is straightforward. If 60% of your requests are simple enough for a small model at one-tenth the cost per token, you just cut your AI spend by roughly half. That’s real money at scale . Working with larger companies always surfaces this: teams are shocked when they see how much they’re spending on GPT-4 for tasks that a 7B model could handle.

What goes wrong

Three failure modes I see repeatedly:

Silent fallbacks. The system falls back gracefully, but nobody knows. Quality degrades slowly. Users get frustrated. By the time someone investigates, there are weeks of bad data.

Stale routing rules. A rule made sense three months ago when Model X was the best at coding tasks. Now Model Y is better and cheaper. But nobody updated the config because nobody owns it.

No cross-model evaluation. Teams evaluate their primary model carefully and treat the fallback as “good enough.” Then the fallback serves 30% of traffic during a bad week and nobody has measured whether it’s actually good enough for those tasks.

The fix for all three is the same. Log every routing decision. Run evals against every model in your chain. Review the routing config monthly. This is the work that keeps production systems stable.

The teams shipping reliable AI features don’t have the cleverest model selection algorithm. They can swap a model in five minutes, measure the impact in an hour, and roll back in seconds.

My first reaction: finally, someone is admitting that one model doesn’t fit every job.

My second reaction: now I have to rerun all my evals .

The lineup

Anthropic did something smart here. Instead of releasing one model and calling it “the best,” they gave you a menu with clear trade-offs.

Opus is the heavyweight. Complex reasoning, deep analysis, demanding coding tasks. It’s slower and more expensive than the others, but the quality ceiling is noticeably higher. I ran it against some gnarly extraction cases I’ve been working on: multi-page contracts with nested clauses and ambiguous references. It handled nuance that the previous generation fumbled.

Sonnet is the workhorse. Good enough for most production workloads, fast enough for interactive use, and priced so it is still viable at volume. This is where I expect most teams to land as a default.

Haiku is the speed demon. Lightweight tasks, high-volume classification, anything where latency matters more than depth. Opus and Sonnet were available on launch day; Haiku followed nine days later. I tested it on a categorization pipeline (hundreds of short inputs, simple labels) and it ripped through them. The quality was adequate for the task, and the speed was impressive.

The value is in routing between them based on what the task needs.

What I noticed in practice

A few things stood out during my first week and a half of testing.

Instruction following is substantially better. Prompts that previously needed careful phrasing to avoid drift now work with more natural language. This is the kind of improvement that doesn’t show up in benchmarks but saves real time in production prompt maintenance.

Vision capabilities are real. I fed Opus some architectural diagrams from a past project and asked it to describe the data flow. The descriptions were useful. Not perfect, but useful enough to save someone from manually transcribing a whiteboard photo.

The context window is large, but I’ve learned not to treat large context as a substitute for good retrieval. Stuffing 200k tokens of raw documents into context and hoping for the best is still a bad strategy. I got better results with targeted retrieval feeding a smaller context window.

One thing that frustrated me: the API rate limits during launch week were tight. I burned through my allocation faster than expected while running evals. Plan for this if you’re testing around a major release.

How I’m thinking about adoption

The useful question is which tier maps to which workflow.

Before switching any production traffic, I work through these questions:

• Latency budget. Interactive features need sub-3-second responses. That might mean Haiku for the fast path and Sonnet for a follow-up detail request.
• Quality threshold. Classification and routing tasks don’t need Opus. Contract analysis probably does.
• Cost sensitivity. High-volume features should default to the cheapest model that meets the quality bar. Upgrade selectively.
• Rollback plan. What happens if quality regresses after the switch? If you don’t have an answer, you aren’t ready.

I route by task type, not by model hype. Haiku handles the lightweight stuff. Sonnet is the default for anything interactive. Opus gets called when the task genuinely needs deeper reasoning. I apply the same approach to any multi-model setup.

Verdict after the first round

Claude 3 is a meaningful step forward. The quality improvements are real, especially in instruction following and structured output. The tiered model approach is the right direction for the industry, because it forces you to think about routing, evaluation, and cost management instead of treating the model as a magic box.

But it’s still a model. It still hallucinates. It still needs evaluation. It still needs guardrails and fallback paths. The teams that will get the most out of Claude 3 are the ones that already have those systems in place.

For everyone else, the release is a good excuse to finally build them.

I was on a call last month with a team. They had an AI-powered document analysis feature and wanted help figuring out why users were complaining about accuracy. My first question: “What does your evaluation suite look like?”

Silence. Then: “We test it manually before releases.”

That isn’t evaluation. That’s a prayer.

The core problem

LLMs are convincing even when they’re wrong. A hallucinated answer looks exactly like a correct one to someone who doesn’t already know the answer. This makes casual testing actively dangerous, because it gives you false confidence.

The non-determinism makes it worse. Change one word in a system prompt and the behavior shifts in ways you can’t predict by reading the diff. The only way to know whether a change helped or hurt is to measure it against a stable reference.

What to measure

Not everything matters equally. I’ve seen teams build elaborate dashboards with dozens of metrics that nobody looks at. Start with the signals that map directly to user value.

Keep the list short. Four to six metrics is plenty. If you can’t explain why a metric is on the list, remove it.

Build a test set that looks like reality

This is where most teams cut corners, and it shows. A test set of five happy-path examples tells you nothing useful. You need cases that reflect the actual distribution of inputs your feature sees in production.

What a decent test set includes:

• Typical cases. The bread-and-butter inputs that make up 80% of traffic.
• Edge cases. Long inputs, short inputs, ambiguous inputs, inputs in unexpected formats.
• Known failure modes. Cases that broke in the past. These are gold.
• Adversarial inputs. Prompt injection attempts, confusing instructions, contradictory context.

Tag every case with a category. This prevents your overall score from hiding category-level failures. I’ve seen a system score 90% overall while completely failing on one important category because the other categories were easy.

Start with 30-50 cases. That’s enough to catch major regressions. Grow it as you learn.

The evaluation methods compared

There’s no single evaluation technique that works for everything. The right approach depends on what you’re measuring.

My recommendation: layer them. Use exact match and rule-based checks for everything you can. Use model-as-judge for quality on open-ended outputs, but calibrate it monthly against human reviewers. Reserve human review for cases where the automated signals disagree or when you’re exploring a new failure mode.

Offline vs. online: different jobs

This distinction matters more than most people realize.

Offline evaluation runs during development. It answers: “Did this prompt change improve behavior on known cases?” Run it before every deploy. Run it when you change prompts, retrieval logic, or model versions. It’s your regression gate.

Online evaluation runs in production. It answers: “Does this actually work for real users with real inputs?” Monitor task success, collect user signals (did they accept, edit, or reject the output?), and track drift over time .

You need both. A clean offline score without production monitoring is a false sense of security. I’ve personally seen features pass every offline test and fail in production because the test set didn’t represent the actual input distribution.

Operationalize it or it dies

Evaluation that lives in a notebook and runs when someone remembers isn’t evaluation. It’s a side project. Make it part of the delivery process.

The loop I use:

1. Maintain a baseline. Your current production version’s scores on the test set. This is the bar.
2. Run evals on every change. Prompt edits, model swaps, retrieval changes: all of it gets measured.
3. Block deploys that regress. Not every metric needs a gate. Pick the ones that matter and set thresholds.
4. Refresh the test set. Add cases from production failures. Remove cases that no longer match product goals. Monthly is a good cadence.
5. Review model-as-judge calibration. Monthly, have a human review a sample of the judge’s ratings. Adjust the grading prompt if it drifted.

The tooling to do this isn’t exotic. A script that runs your test set through the system, compares outputs to expected behavior, and produces a report. I’ve built these in a few hundred lines of Go. The code is the easy part. The hard part is running it every time.

Discipline is the missing piece

I keep coming back to this. The tools exist. The techniques are well-understood. The test sets aren’t that hard to build. What’s missing is the organizational willingness to treat AI output quality with the same rigor as test coverage or uptime.

If you wouldn’t ship a backend service without tests, you shouldn’t ship an AI feature without evaluation.

There’s a particular kind of architectural diagram I keep seeing in pitch decks. A clean box labeled “AI” sits neatly between the frontend and the database, connected by two arrows. Everything looks tidy. Everything is a lie.

AI-native applications are messy. The model is non-deterministic. Responses vary in quality. Latency is unpredictable. Costs scale with usage in ways that don’t match traditional compute. And yet the product’s core value depends on this unreliable component working well enough, often enough, that users trust it.

I’ve been building these systems for the past year, in telecom and fintech. The architecture that works looks nothing like that clean diagram.

What “AI-native” actually means

Let me be precise. An AI-native application is one where removing the AI component wouldn’t leave you with a simpler app. It would leave you with no app. The AI isn’t a feature. It’s the product.

This creates three architectural consequences you can’t ignore:

1. Non-determinism is in the critical path. The same input can produce different outputs. Your architecture must absorb this instead of pretending it away.
2. Quality is a spectrum, not a boolean. You evaluate on ranges and intent, not exact matches.
3. The system must learn from usage. Feedback is what keeps the product from degrading.

The layered architecture I use

After building several of these systems, I’ve settled on a layered approach. Each layer has a distinct failure mode and a distinct owner, which is the only reason to draw layers at all.

┌─────────────────────────────────────┐
│         Experience Layer            │  <- Uncertainty communication, UI
├─────────────────────────────────────┤
│       Orchestration Layer           │  <- Routing, fallbacks, workflows
├─────────────────────────────────────┤
│         AI Services Layer           │  <- Model calls, retrieval, tools
├─────────────────────────────────────┤
│      Quality & Safety Layer         │  <- Validation, filtering, policy
├─────────────────────────────────────┤
│       Data & Context Layer          │  <- Knowledge, memory, embeddings
├─────────────────────────────────────┤
│     Feedback & Analytics Layer      │  <- Learning, monitoring, eval
└─────────────────────────────────────┘

These don’t need to be separate services. In most systems I build, they start as packages within a single Go binary. The point is that each responsibility exists, is testable, and has clear ownership.

Designing for uncertainty

This is the part most teams get wrong. They treat the model like a function: input goes in, correct output comes out. Then they’re shocked when production users get hallucinated garbage.

The architecture needs to absorb uncertainty at every level. Here is how I handle it in the orchestration layer:

type Confidence int

const (
	ConfidenceHigh   Confidence = iota // Route directly to user
	ConfidenceMedium                    // Add verification step
	ConfidenceLow                       // Escalate or fallback
)

type AIResponse struct {
	Content    string
	Confidence Confidence
	ModelID    string
	Latency    time.Duration
	TokensUsed int
}

func (s *Service) HandleRequest(ctx context.Context, req Request) (*Response, error) {
	aiResp, err := s.aiClient.Generate(ctx, req.ToPrompt())
	if err != nil {
		return s.fallbackResponse(ctx, req)
	}

	switch aiResp.Confidence {
	case ConfidenceHigh:
		return s.directResponse(aiResp), nil
	case ConfidenceMedium:
		verified, err := s.verify(ctx, aiResp, req)
		if err != nil {
			return s.directResponse(aiResp), nil // Degrade gracefully
		}
		return verified, nil
	case ConfidenceLow:
		return s.escalate(ctx, req, aiResp)
	default:
		return s.fallbackResponse(ctx, req)
	}
}

Confidence doesn’t need to be a number shown to the user. It’s an internal signal that controls what happens next. High confidence goes straight through. Medium confidence gets a verification step: maybe a retrieval check, maybe a second model call with a stricter prompt. Low confidence hits the fallback path.

The fallback path is critical. Every AI-native app needs one, and it should be designed before the happy path. What does the product do when the model is down? When it returns garbage? When it takes 30 seconds to respond? If the answer is “crash” or “show a spinner forever,” the architecture isn’t ready for production.

Feedback loops belong in the architecture

Every request through the system should produce a feedback record. You won’t have time to look at them all. Without them, though, you’re blind to degradation.

type FeedbackRecord struct {
	RequestID   string
	Prompt      string
	Response    string
	ModelID     string
	Confidence  Confidence
	Latency     time.Duration
	UserSignal  UserSignal  // Accepted, rejected, edited, ignored
	Outcome     Outcome     // Success, partial, failure
	Timestamp   time.Time
}

type UserSignal int

const (
	SignalNone     UserSignal = iota
	SignalAccepted
	SignalRejected
	SignalEdited
	SignalIgnored
)

The user signal is the most valuable field. Did the user accept the output? Edit it? Ignore it entirely? That data drives everything: prompt improvements, model selection changes, confidence calibration.

I learned this the hard way on a project where we shipped an AI feature without feedback instrumentation. Two months later, we had no idea whether the model’s quality had drifted or whether users had simply stopped trusting it. We were debugging with anecdotes. Never again.

Routing without the PhD

You don’t need a machine learning model to route requests to the right model. A few rules go a long way.

type RouterConfig struct {
	Rules []RoutingRule
}

type RoutingRule struct {
	Condition func(req Request) bool
	ModelID   string
	Timeout   time.Duration
	MaxTokens int
}

func DefaultRouter() *RouterConfig {
	return &RouterConfig{
		Rules: []RoutingRule{
			{
				Condition: func(r Request) bool { return r.TokenEstimate() < 200 },
				ModelID:   "fast-small",
				Timeout:   5 * time.Second,
				MaxTokens: 512,
			},
			{
				Condition: func(r Request) bool { return r.RequiresReasoning() },
				ModelID:   "capable-large",
				Timeout:   30 * time.Second,
				MaxTokens: 4096,
			},
			{
				Condition: func(r Request) bool { return true }, // Default
				ModelID:   "balanced-medium",
				Timeout:   15 * time.Second,
				MaxTokens: 2048,
			},
		},
	}
}

Small requests get the fast model. Reasoning-heavy requests get the capable one. Everything else gets the balanced option. This isn’t clever. It doesn’t need to be. It just needs to keep costs predictable and latency acceptable.

Keep the model IDs, timeouts, and token limits in configuration. When you want to change routing (because a new model dropped, or costs shifted, or you learned that certain request types need more capability), you change the config instead of hunting through call sites.

UX that respects the user’s intelligence

The biggest UX mistake in AI-native apps is pretending the system is certain when it isn’t. Users can handle uncertainty. They can’t handle being lied to.

A few principles I follow:

• Show your work when confidence is low. If the model retrieved documents to answer a question, show which ones. Let the user verify.
• Offer refinement, not just results. A “try again” button is lazy. A “here is what I found, want me to focus on X?” is useful.
• Keep the UI stable on failure. When the model times out, the product should still work. Maybe with reduced functionality, but it shouldn’t break.

The best AI-native UIs I’ve seen treat the model like a very fast but occasionally wrong colleague. You check their work on important things. You trust them on routine things. The UI should support that mental model.

The data layer determines everything

I have a saying I repeat in these situations: your AI feature is only as good as the data you feed it.

The context layer needs to support structured facts (database records, configuration), unstructured knowledge (documents, guides, prior conversations), and session memory (what happened earlier in this interaction).

Retrieval quality matters more than model quality for most applications. I’ve seen teams spend weeks prompt-engineering their way around a bad retrieval pipeline. Fix the retrieval. The prompts will get simpler.

Operational discipline

Production AI-native apps need monitoring that goes beyond uptime checks :

• Quality monitoring. Track your confidence distribution over time. If low-confidence responses are increasing, something changed.
• Cost tracking per request type. Aggregate cost hides the problem. You need to know which workflows are expensive.
• Latency budgets. Set them per workflow, not globally. A search feature and a document analysis feature have different acceptable latencies.
• Drift detection. Model behavior changes. Provider behavior changes. Your data changes. Monitor for all of it.

Build the fallback first. Instrument everything. Let the feedback loop make the system smarter over time.

This is dumb. The hosted models are great, but you don’t need frontier-model quality to test whether your prompt template works, your parsing logic handles edge cases, or your UI renders a streamed response correctly.

Run a local model. Iterate fast. Save the API calls for when you actually need them.

The actual reasons to go local

Forget the hand-wavy “sovereignty” arguments for a moment. The practical reasons are simple:

Speed. No network round-trip. No rate limits. No waiting in a queue behind someone else’s batch job. I can test a prompt change in under a second on a MacBook with Ollama running a 7B model. That feedback loop matters when you’re doing fifty iterations in an afternoon.

Cost. Zero marginal cost per request. I ran through over a thousand prompt variations last month while building an extraction pipeline. On GPT-4, that would have been a few hundred dollars. Locally, it was electricity.

Privacy. Some of my work involves data I can’t send to a third-party API . Full stop. Local inference solves that problem without paperwork.

The trade-offs are real, so stop pretending otherwise

Local models aren’t frontier models. A 7B parameter model running on your laptop isn’t going to match GPT-4 on complex reasoning tasks. That’s fine. You’re using it for development velocity, and production quality comes from the real model.

Where local models genuinely fall short:

• Multi-step reasoning. They lose the thread.
• Long context windows. Most local models tap out well before 128k tokens.
• Consistent formatting. They drift more on structured output tasks.
• Nuanced instruction following. Subtle prompt changes sometimes get ignored.

If your development workflow requires frontier-quality responses at every step, local models aren’t for you. Most development workflows don’t. You need a model that’s good enough to validate your integration logic, and local models clear that bar easily.

My actual setup

I keep it simple. Ollama for the runtime, a 7B model as default, and an environment variable to swap between local and remote.

func getLLMConfig() LLMConfig {
	if os.Getenv("USE_LOCAL_LLM") == "true" {
		return LLMConfig{
			BaseURL: "http://localhost:11434",
			Model:   "mistral",
		}
	}
	return LLMConfig{
		BaseURL: os.Getenv("LLM_API_URL"),
		Model:   os.Getenv("LLM_MODEL"),
	}
}

That’s it, plus a thin adapter for Ollama’s native API. The rest of the application doesn’t care which model it’s talking to. The interface is the same, the error handling is the same, the retry logic is the same. When I want to validate quality against the real model, I flip the variable and run my eval suite.

The workflow that actually works

1. Develop locally. Prompt changes, parsing logic, UI work, error handling. All against the local model.
2. Eval against remote. Before merging, run the same test cases against the production model. Compare outputs.
3. Ship with confidence. The integration is tested. The quality is validated. The bill is reasonable.

The point: your development model and your production model don’t need to be the same. They need to share the same interface.

When to skip local entirely

Know the cases where local doesn’t help:

• You’re doing few-shot prompt engineering where response quality is the variable you’re testing.
• Your feature depends on capabilities only frontier models have (vision, very long context, tool use with complex chains).
• You’re evaluating model-specific behavior like safety responses or refusal patterns.

In those cases, just use the API. Local inference is a cost and speed tool. The point is to stop burning money on API calls when a local model would have told you the same thing.

Install Ollama. Pull a model. Point your dev config at localhost. When you need the real thing, it’s one environment variable away.

After a year of working on AI integration across different organizations, the pattern I keep seeing is the same: a team hires a machine learning engineer, points them at a product feature, and wonders why the result is a brilliant notebook that falls apart the moment a real user touches it .

The problem isn’t the engineer. The problem is a category error.

Between ML research and backend engineering

AI engineering sits in an awkward gap. On one side, you have model training: the research-heavy work of building and improving models. On the other, traditional software engineering: APIs, databases, deployment pipelines, the stuff we’ve been doing for decades.

AI engineering is neither. It’s the work of taking someone else’s model and making it do something useful, reliably, in production. That means prompt design, retrieval pipelines, evaluation harnesses, cost management, safety guardrails, and graceful failure handling. It means caring deeply about the 2% of cases where the model confidently produces garbage.

I spent years building backend systems across fintech and cloud infrastructure. The shift to AI engineering felt familiar in some ways. You still think about latency, error handling, observability. But the non-determinism changes everything. You can’t unit test your way to confidence when the same input produces different outputs on Tuesday.

The skill set looks different

When I talk to CTOs about what to look for in AI engineering hires, I push them away from the classic ML job description. The competencies that matter are:

• Prompt design and testing. Systematic testing across edge cases, with version control and regression detection. Prompt “engineering” as a parlor trick doesn’t count.
• Retrieval and context assembly. Getting the right information to the model at the right time. This is where most applications succeed or fail.
• Integration discipline. Error handling, latency budgets, fallback paths. The boring stuff that separates demos from products.
• Evaluation loops. If you can’t measure whether your AI feature got better or worse after a change, you aren’t doing engineering. You’re doing improv.
• Safety and guardrails. Especially when the model can take actions or access private data.

None of this requires a PhD. It requires someone who has shipped software, understands production systems, and has the patience to wrangle probabilistic outputs into predictable behavior.

One owner across the whole pipeline

People keep trying to draw AI engineering as a neat layer diagram. In practice, it’s a set of cross-cutting responsibilities. You’re choosing models, preparing data, shaping prompts, monitoring quality, controlling costs, and enforcing safety, all at once. The role feels distinct because it spans product thinking, system design, and ongoing operational care in a way that neither pure ML nor pure backend roles typically do.

At a large telecom company I worked with, I watched teams try to split these responsibilities across existing roles. The ML team owned prompts. The backend team owned integration. The product team owned evaluation. Nobody owned the whole thing. The result was predictable: finger-pointing when quality dropped and no single person who could trace a bad output from user input to model response to product impact.

How to build these skills

Depth beats breadth. Don’t chase every new framework or technique. A solid path:

1. Build a feature that calls a model and returns something useful. Ship it.
2. Add retrieval so the model’s answers are grounded in real data instead of vibes.
3. Build an evaluation loop that catches regressions before your users do.
4. Add guardrails and define what happens when the model fails. Because it will.

The practice is learned by shipping and iterating. Blog posts help (including this one, I hope), but they aren’t a substitute for watching your carefully crafted prompt fall apart on production traffic.

Where this fits in your org

In smaller teams, AI engineering looks like a product-focused engineer who owns the AI feature end to end. At larger companies, it becomes a dedicated role that sits between product, platform, and security.

The interaction model is clean. Product defines intent and user experience. Platform provides infrastructure and monitoring. Security sets the safety bar. AI engineering turns those constraints into working features that don’t embarrass anyone.

Job descriptions are starting to separate AI engineering from ML research, and the expectations center on integration, evaluation, and reliability rather than paper-publishing and model architecture. Good. That separation was overdue. The teams that treat it as a distinct discipline, with its own tools, failure modes, and career path, are shipping better features. The teams that don’t are still arguing about whether their demo “works.”

After a few weeks of serious testing, I have a clearer picture of where multimodal AI is ready for production and where it will get you in trouble. Short version: vision-capable models are useful for document extraction, UI review, and accessibility, and unreliable for precise measurements, tiny text, and anything that requires counting. Treat the model like a smart intern who’s great at describing what they see but bad at details.

What Actually Ships

1. Invoice and Receipt Extraction

This is the killer use case for anyone who processes financial documents. Extracting vendor name, amount, date, and line items from a photo of a receipt used to require a dedicated OCR pipeline, post-processing rules, and a prayer. Now I send the image to GPT-4V with a structured prompt and get JSON back.

Analyze this invoice image. Return JSON with these fields:
- vendor_name (string)
- total_amount (string, include currency)
- invoice_date (string, YYYY-MM-DD)
- line_items (array of {description, amount})
If a field is not visible, return null.

Hit rate on clean documents is around 90%. On crumpled receipts with bad lighting, it drops to maybe 65%. Good enough for a first pass with human review on low-confidence results.

2. UI Review

I started using it to review screenshots of our admin dashboards. “List any layout issues, missing states, or accessibility concerns in this screenshot.” The results aren’t comprehensive, but they catch obvious problems (misaligned elements, missing error states, low contrast text) faster than a manual review pass.

3. Accessibility

Alt text generation. Genuinely good at this. Feed it a product image or a chart and ask for a concise description. The output is usually better than what most developers write manually, which is a low bar, but still.

4. Architecture Diagram Interpretation

This one surprised me. I photographed a whiteboard diagram from a system design session and asked the model to describe the components and data flow. It got the high-level architecture right. Not perfect on every label, but the structure was correct. Useful for converting whiteboard photos into documentation drafts.

5. Visual Anomaly Detection

For predictable environments (“does this photo show the expected setup?”), the model is decent at spotting obvious differences. Missing components, wrong configurations, visible damage. It works best when you can describe what “normal” looks like and ask the model to flag deviations.

What Doesn’t Work (Yet)

Counting

Ask it to count items in a busy image. Watch it fail. It will confidently give you a number that’s wrong. Small objects, overlapping items, dense arrangements: the model can’t reliably count. Don’t build features that depend on this.

Precise Measurements

“How far apart are these two components?” The model doesn’t do spatial precision. It can tell you something is “on the left” or “near the top” but asking for millimeter-level accuracy is asking for trouble.

Tiny or Low-Quality Text

Faded labels, handwritten notes in bad lighting, text smaller than about 10px on a screenshot: all unreliable. The model will either skip the text entirely or hallucinate plausible content. This is the failure mode that scares me most because it’s indistinguishable from correct output unless you verify.

The Cost Problem

Vision calls are expensive. In high-detail mode an image is billed as input tokens on top of your prompt: 765 tokens for a 1024x1024 image, more than a thousand for a large photo. Low-detail mode is a flat 85 tokens. At scale, this adds up fast.

My rules:

• Resize aggressively. Crop to the region of interest. A full-resolution photo of a receipt when all you need is the total amount is wasting tokens and money.
• Use low detail mode for simple tasks. GPT-4V supports a detail parameter. Use “low” for tasks like “is there text in this image?” and “high” only when you need it.
• Cache everything . Same image, same question, same answer. Don’t re-process.
• Batch when possible. Multiple questions about the same image should be a single API call, not five separate ones.

Building for Uncertainty

The single most important design principle: assume the model will be wrong sometimes, and build your product flow to handle it gracefully.

For document extraction, every result goes through a confidence check. If any field comes back null or if the extracted amount doesn’t parse as a valid number, it routes to human review. The model handles the easy 70-80% automatically. Humans handle the rest. The total cost is still lower than having humans process everything manually.

Ask the model to cite visible evidence. “What text did you read to determine the vendor name?” If it can’t point to specific text in the image, the answer is probably a hallucination.

Keep an OCR fallback for critical text extraction. The vision model is better at understanding context. Traditional OCR is better at reading exact characters. Use both.

Multimodal AI is a new tool with a specific reliability profile. Know where it’s strong, know where it fails, and design your system to handle both.

The Good: Speed to Something Real

I built an internal documentation assistant in about four hours. Upload the docs, write a focused system prompt, wire up a simple Go client that manages threads. Done. The retrieval isn’t perfect, but it’s good enough for “which endpoint handles X” type questions. Previously this would have required a vector store, an embedding pipeline, chunking logic, and a retrieval chain . Now it’s an API call.

The code interpreter is surprisingly useful. I hooked it up to a tool that lets internal users ask data questions in plain English. “How many transactions failed last week?” gets translated into Python, executed in OpenAI’s sandbox, and the result comes back formatted. It took me a day. Building a safe code execution sandbox from scratch would have taken a week minimum.

The Bad: Opacity Everywhere

The retrieval is a black box. I can’t control how it chunks my documents. I can’t see what it retrieved before generating an answer. I can’t tune the similarity threshold or re-rank results. For the documentation assistant, this is tolerable: the stakes are low and approximate recall is fine.

For anything involving financial data, it’s a non-starter. I need to know exactly what context the model saw. I need to audit the retrieval path. I need to explain to compliance why the system gave a specific answer. The Assistants API can’t do any of that.

Thread management is also trickier than it looks. Threads accumulate context over time, and stale context degrades answers. I learned this the hard way when the documentation assistant started mixing up API versions because it was carrying context from a conversation about v1 into a question about v2. Now I have a policy: new thread for every topic change. It’s crude but it works.

The Ugly: Runs Are Flaky

A “Run” is one execution of an assistant against a thread. It can succeed, fail, stall, or time out. In my first week, I had runs that just… hung. No error. No timeout. Just pending forever. I added my own timeout logic around every run, with a hard kill after 30 seconds and a retry with a fresh thread if it fails twice.

ctx, cancel := context.WithTimeout(ctx, 30*time.Second)
defer cancel()

run, err := client.CreateRun(ctx, threadID, assistantID)
if err != nil {
    return fmt.Errorf("create run: %w", err)
}

// Poll until complete or timeout.
for {
    status, err := client.GetRun(ctx, threadID, run.ID)
    if err != nil {
        return fmt.Errorf("check run status: %w", err)
    }

    if status.Status == "completed" {
        break
    }
    if status.Status == "failed" || status.Status == "expired" {
        return fmt.Errorf("run %s: %s", status.Status, status.LastError)
    }

    time.Sleep(500 * time.Millisecond)
}