I recently shipped a recommendation engine at createxp that uses RAG under the hood. The goal was simple: take a laptop catalog and let users describe what they need in plain English, like "something for Premiere Pro editing but I travel a lot, needs to be light", and get back useful, reasoned recommendations. The demo worked first try. The production version took three more weeks. Here's what I learned.

The gap between a working prototype and a production pipeline

The happy path is genuinely easy. You chunk your data, embed it, throw it in pgvector, write a retrieval query, pipe it through an LLM, done. The whole thing fits in 60 lines of Python and works great in a notebook. Production is where it falls apart. Your embeddings are stale the moment your catalog updates. Your retrieval is slow when the table gets large. Your LLM calls are the bottleneck you didn't benchmark. And when something breaks at 2am, you have no visibility into which step failed. None of that shows up in a demo.

Intent extraction is not optional

The biggest mistake I made early on was treating RAG as a pure retrieval problem. User sends a query, embed it, find similar items, generate response. Clean, simple. Wrong. The query "under 1.5kg, handles Lightroom and Premiere, something I can take on a flight" is not a vector search query. It's a bundle of constraints: weight, GPU class, use case, portability, wrapped in natural language. If you embed that as-is and cosine-similarity your way through a product catalog, you'll get results that are semantically adjacent but practically useless. The fix was adding an intent extraction layer before retrieval. I use Gemini Flash to parse the raw query into structured filters first: These structured filters go into the SQL WHERE clause. The embedding handles the fuzzy semantic matching on top of that. Together, precision went up significantly, and more importantly, results started feeling correct rather than just statistically close.

The latency problem is almost always in the wrong place

Before I profiled anything, I assumed the LLM call was the bottleneck. It wasn't. The actual breakdown on my first version:

• Intent extraction (Gemini Flash): ~80ms
• Embedding the query (e5-base-v2): ~120ms
• pgvector retrieval: ~340ms
• LLM response generation: ~600ms (streaming, so perceived latency was lower)

The embedding step was the surprise. I was re-embedding every query from scratch on every request. Moving to a cached embedding layer for common query patterns and batching where possible dropped that to ~15ms. The pgvector retrieval at 340ms was the other problem. The table had grown to ~50k products with no index tuned for the query pattern. Adding an ivfflat index on the embedding column and bumping ef_search appropriately brought that down to ~40ms. After both fixes: total pipeline latency went from ~1.1s to ~190ms. The LLM streaming was always fast enough perceptually, users see tokens arriving, not a spinner. Profile before you optimize. The bottleneck is never where you think it is.

Redis saved me, but not how I expected

I added Redis expecting to cache LLM responses for repeated queries. That turned out to be less useful than expected, queries in natural language are too diverse to hit a cache often. What Redis actually helped with was catalog caching. Product metadata (specs, prices, descriptions) doesn't change by the millisecond. Caching the top-retrieved products meant I wasn't round-tripping to Postgres for data that hadn't changed. That alone shaved ~60ms off the average request. The other use was rate limiting and request coalescing under traffic spikes. When the same product suddenly gets queried by 50 concurrent users (a promotion went live), Redis absorbs the duplicate retrieval work instead of hammering the database.

Streaming makes latency feel irrelevant

Even with a 190ms pipeline, I still stream the response. It's not a performance decision, it's a UX decision. A 600ms blank screen followed by a complete response feels slow. A 190ms wait followed by tokens appearing one by one feels instant, even if the total time is longer. Users are pattern-matching to human typing, not measuring milliseconds. If you're building anything that calls an LLM, stream the output. Always. The implementation cost is low and the perceived performance improvement is dramatic.

Observability is the real infrastructure

The last thing I added, and the thing I wish I'd built first, was structured logging at every step of the pipeline. Every request now logs:

• The raw query
• The extracted intent (structured filters)
• Which products were retrieved and their similarity scores
• The latency of each step
• Whether the user engaged with the result

That last one is the most valuable. Latency tells you if the system is fast. Engagement tells you if it's actually good. A 150ms response that recommends the wrong product is worse than a 300ms response that nails it. Building in feedback visibility from the start, even just a thumbs up/down, gives you ground truth to improve retrieval quality over time. Without it, you're flying blind.

RAG in production is not a hard engineering problem. It's a systems problem of latency, caching, observability, and feedback loops, wrapped around what looks like a simple search feature. The three weeks went into caching, an index tuned for the actual query pattern, and logging that told me which fix was working and which wasn't. None of that shows up in a 60-line notebook version, and all of it turned out to be the actual job.