Two RAGs built to fail

In September I taught Inteligencia Artificial aplicada a Sectores Productivos (AI applied to productive sectors), a master’s subject at UNIR. The students come from all over: mostly business, law and economics, but also teaching and engineering, and most of them don’t code. Two of the seven sessions built a RAG in n8n: health on September 17, industry on September 21.

The class guide set one goal: the point isn’t that it works, it’s that you see every piece and manage to make it fail. The deliverable was a screenshot of the clearest failure you could get.

The goal was never that they’d leave able to build a proof of concept, or a RAG tomorrow. It was that, by building one step by step and seeing it at a low level, they’d know the whole process: which tools are involved, why each one is there and how the complexity can scale. These are people who’ll lead projects like this, and I want them to arrive knowing what these tools can and can’t do, and how much process sits behind them.

So I built both systems to fail, on purpose and in public. Here’s how, and what I’d tell a company before shipping one.

Plant the traps first

A RAG cuts your documents into chunks, turns each chunk into a list of numbers (an embedding), finds the chunks closest to the question and hands them to a model to write the answer. It knows which chunk looks like the question. It doesn’t know which document is current.

Both corpora are built around that gap. NEXORA Salud is a fictional hospital group: 984 PDFs and 31,088 pages. Planted inside: a 2008 type 2 diabetes guideline whose own cover says it’s pending an update, next to the 2026 type 1 guideline. And a 2024 internal protocol that claims precedence over external guidelines and sets a laxer HbA1c target for type 1.

NEXORA Industrial is a fictional plant: 100 documents and 6,316 pages of Spanish and EU regulation, manufacturer manuals and six invented internal documents. Its traps include one question with four right answers depending on the equipment, a regulation that sends you to another for the key figure, an internal procedure laxer than the law, and repealed rules next to their replacements.

In the health session, students started small: a ready-made flow, a free Gemini key and two trimmed guidelines in an in-memory store. Then five experiments, like deleting the rule that says “answer only from what you retrieve”. The lesson wrote itself: the RAG uses the 2008 guideline with the same confidence as the 2026 one.

What you can swap, and what you can’t

A RAG uses two models. You can swap the one that drafts the answer whenever you like, even for another provider’s. You can’t swap the embedding model without re-indexing everything: OpenAI’s vectors (1,536 numbers per chunk) and Gemini’s (3,072) are different maps with different coordinates.

The market made the point for me. Ollama Cloud offers no embedding models, only chat, so in the full health build Ollama’s gpt-oss:120b drafts over vectors made by OpenAI’s text-embedding-3-small. Who drafts and who turns text into numbers can be different companies.

That build started as a written brief that Claude carried out through n8n’s MCP server, stopping for my OK between phases. It wasn’t one shot at first: a few details needed another pass, and once I’d refined the brief after that first attempt, it did come out in one go. The full corpus went into Qdrant Cloud: 128,787 chunks, about 25 million embedding tokens (under $0.50, per the OpenAI billing), 35 minutes in batches. Gemini’s free tier didn’t make that cut: under load it returned empty vectors, with no error.

On the full corpus it found a two-page internal procedure among 984 documents, and gave the right 2026 HbA1c range for type 1 without mentioning the internal protocol that says otherwise. Retrieving well isn’t the same as detecting contradictions.

One question, three answers

The industrial corpus became 24,662 chunks in Qdrant, each with 10 labels. That makes it agentic: one collection, three search tools filtered by document type (regulation, internal, manual), and an agent that picks one by reading each tool’s description. Three chat profiles, chosen by a prefix, share that collection and its embeddings; only the drafter and the amount of context change.

Then I asked how often a fire extinguisher has to be pressure-tested (retimbrado). The fire-safety regulation doesn’t say: it refers to the pressure-equipment regulation. The current one isn’t in the corpus. The one that is, from 2008, has been repealed since January 2, 2022, and says so itself.

/rapido (gpt-4o-mini, 4 chunks) invented “every 10 years” and attributed it to a page that only lists technical standards.

/potente (gpt-5.6-luna, 10 chunks) found the real figure, every five years, without warning that it comes from a repealed regulation. Not the prompt’s fault: the repeal notice is on page 7 of the PDF and the figure on page 30. Different chunks. They never travel together.

/ollama (deepseek-v4-pro:cloud, 10 chunks) sometimes found the literal cross-reference and said the regulation it points to isn’t loaded. Other times it gave the repealed figure.

That was the lesson I didn’t plan: same question, same profile, same corpus, not always the same answer. It depends on which chunks come back. Would you sign a compliance report with that? No system prompt fixes chunking. Tagging each regulation’s validity at indexing time would.

33 of 35, and why the score isn’t the point

“I tried it for a while and it seems fine” isn’t an answer. So the industrial build has an eval: seven fixed cases, each with a written criterion, run the same way every time.

It judges in two layers. Deterministic checks first: does it cite the fire regulation, does “10 years” not appear. Free, instant, always the same result. A model judge only gets what needs judgement, such as whether the answer separates who does each inspection.

{
  pregunta: '¿Cada cuánto se retimbra un extintor?',
  debe_citar_doc: [7],                             // deterministic: the fire regulation
  no_debe_contener: ['10 a(n|ñ)os|diez a(n|ñ)os'], // deterministic: the invented figure
  criterio: 'Evalua UNICAMENTE lo que la respuesta dice sobre el RETIMBRADO o timbrado…', // model judge
}

In class we ran the seven cases five times each: 33 of 35 (94%), for about four cents. The retimbrado question passed three times out of five. Three things mattered more than the score.

A lax criterion gives a false green. The first version scored 100% because one criterion accepted “mentions the cross-reference or warns about the repeal”. Quoting the repealed rule without a warning passed. Split into two cases, the 100% fell. Your criterion is your specification.

The judge gets it wrong too. One case scored 0 of 2 with correct answers: the judge read “quarterly maintenance” as a re-test interval. Scoping the criterion to the retimbrado alone took it to 3 of 3.

An unstable case is worse than one that always fails. The one that always fails, you see and fix. The unstable one passes the day you check and breaks in production on some Tuesday. So the flow repeats every case and lists the unstable ones.

What an answer costs

Same question, measured on the real system (list prices as of September 19):

Profile	Model	Per 1M tokens (in / out)	Cost of the answer
/ollama	deepseek-v4-pro:cloud	$0.022 / $1.98	€0.0011
/rapido	gpt-4o-mini	$0.15 / $0.60	€0.0014
/potente	gpt-5.6-luna	$0.20 / $1.20	€0.0075

DeepSeek has the priciest output of the three and still gives the cheapest answer, because in a RAG the input dominates: every question carries several chunks. And the cheapest profile did no worse on the hard question, which already breaks the intuition that paying more buys a better answer.

The part that costs most never shows up on an invoice: keeping the documents current, measuring quality and reviewing what the system says.

Before a company ships one

Every hands-on I teach is a prototype, and I say so out loud: the jump to production is a consulting skill, not a technical detail. The public chats had no password and ran on my own credits, so they’re switched off now, on purpose.

A version for a hospital or a plant would need access control, usage limits, validity metadata, documents that update when a regulation changes, systematic evaluation, traceability, and a person who answers for the output.

I’d ask a company the questions I left my students with. What’s your 2008 guideline, the expired document your RAG will use as confidently as the current one? Which two of your documents contradict each other without anyone noticing? Which one refers to another you don’t have?

A better model fixes none of those. Choosing, dating, tagging and retiring documents does, and that’s work for people.

Both repositories are public, with the corpus inventories, n8n flows and write-ups of what failed. Scripts and fictional documents are CC0. The class guides are on the IASP page.