Your RAG Problem Is a Content Problem: Building Governed Retrieval on Drupal

Retrieval-augmented generation (RAG) lets an AI model answer from your own content: it retrieves relevant passages, then generates an answer that cites them. In production, most RAG failures don't come from the model. They come from the content: duplicate pages, outdated versions, unapproved drafts and ignored permissions. That makes your CMS (Content Management System), not your LLM (Large Language Model), the foundation of a trustworthy RAG system.

The demo works. Production doesn't.

Almost every RAG proof of concept looks impressive. Point a model at a folder of documents, ask a question, get a fluent answer with a citation. Then it goes live, and the support tickets start:

• "The chatbot quoted last year's tuition."
• "It gave a vendor the internal escalation procedure."
• "It cited a page we never published."

Teams usually respond by swapping models, tuning prompts or adjusting chunk sizes. Sometimes that helps at the margins.

The root cause is usually upstream, in how the source content is governed.

Where RAG actually breaks

Almost every RAG proof of concept looks impressive in a demo. These are the failures we see most often in production, and almost none of them have anything to do with the model:

Only one row, chunking, is a classic "RAG engineering" problem, and even that one depends on content structure. Everything else is content governance.

Why an enterprise CMS beats a folder of PDFs

The original RAG paper¹ assumed a clean knowledge source. Enterprises rarely have one. They have portals, intranets, shared drives and exported PDFs, often saying slightly different things.

A structured CMS like Drupal already has the properties RAG needs:

• Structured content. Content types with defined fields (summary, requirements, effective date, owning department). A chunk from the "Admission requirements" field of the "MS in Data Science" page carries its own context.
• Editorial states. With Content Moderation, only Published content is official. Index only that state, and drafts never leak into answers.
• Revision history. Every change has an author and a timestamp, so you can trace which version an answer was based on.
• Granular permissions. Roles, section-based access and restricted content that the retrieval layer can enforce.
• Linked translations. The Spanish version of a page is a translation of the same entity, not a separate document that can drift.

In other words, the CMS knows who owns each piece of content, whether it's approved, when it changed and who may see it.

A vector database alone knows none of that.

A reference architecture for RAG on Drupal

The Drupal AI Initiative ecosystem, which Seed EM supports as a Gold Sponsor, already provides the building blocks:

One underrated advantage of building on Search API: indexing is tied to the content lifecycle. When an editor fixes a date and publishes, the index picks it up on the next indexing run. No manual re-exports, no forgotten PDFs.

Two architectural choices matter especially for organizations buying from outside their region. Swappable model providers protect you from lock-in as frontier models change pricing and capabilities every few months. And because Drupal is open source and self-hostable, you keep control over where content and embeddings live.

Eight design decisions that determine quality

1. Scope: what gets indexed

Published content only, and only the content types meant to answer questions. A 2019 press release probably shouldn't compete with your current policy page.

2. Chunking: split by structure first

Chunk along fields, paragraphs and sections before falling back to fixed token windows. Then prepend each chunk with the context that makes it self-explanatory: page title, section, date.

The evidence supports this. In September 2024, Anthropic published its Contextual Retrieval results: adding chunk-specific context cut top-20 retrieval failures by 35%. Adding keyword search (BM25) brought the reduction to 49%, and adding reranking brought it to 67%.²

3. Metadata: make every chunk filterable

Language, publish date, expiry date, owner, audience and canonical URL. These let you filter ("current content, in English, for prospective students") and cite precisely.

4. Permissions: filter before generation

The retrieval layer must restrict results to what the requesting user is allowed to see before anything reaches the model.

Telling the model "don't reveal confidential information" is not an access control.

5. Retrieval mode: vector, keyword or hybrid

Vector search handles synonyms and paraphrases but struggles with codes, policy numbers and proper nouns. For catalogs, policies and regulated content, hybrid retrieval is usually worth the extra complexity, as the Contextual Retrieval numbers show.

6. Citations: every answer links to its source

Citations let users verify answers and give you an audit trail. When retrieval comes back thin, the system should say so rather than improvise.

7. Freshness: tie the index to the lifecycle

Reindex on publish, remove on unpublish or archive, and flag content past its review date.

8. Evaluation: measure before launch, and after every change

Build a test set of 50 to 100 real questions, each with its correct answer and the page that should be cited. Track:

• Correct answers with a valid citation
• Abstention accuracy: "I don't know" when it should, and only then
• Unsourced answers: target zero
• Freshness lag: time from a published change to correct answers

Rerun the set whenever you change the model, the chunking strategy or a major body of content.

RAG vs. fine-tuning vs. long context

For institutional websites, university portals and knowledge bases, RAG is almost always the core. Fine-tuning can complement it for tone or output format.

Security: content can be an attack vector

Prompt injection doesn't only come from users. It can come from retrieved content: a passage that says "ignore your instructions and…" sitting inside an indexed page. Three defenses that live in the CMS:

• Index only approved content, so malicious text would have to get through the editorial workflow.
• Exclude or downgrade user-generated content, such as comments and form submissions.
• Treat retrieved chunks as data, never as instructions, and log which chunks informed each answer.

Before you build RAG, clean the house

Most organizations that want RAG first need a content cleanup: duplicate pages, empty metadata, expired content, images without alt text, orphaned sections. It's unglamorous work that keeps getting deferred.

We built SeedPilot, an AI content operations agent for Drupal 10 and 11, partly for this stage. It finds missing or duplicate metadata, orphan pages and weak published content, then prepares fixes for an editor to approve, with every action logged.

Cleaner content means more reliable RAG, and the same work improves SEO and accessibility along the way.

FAQ

What is RAG in AI?

Retrieval-augmented generation (RAG) is a technique where an AI model first retrieves relevant passages from your own knowledge base and then generates an answer grounded in them, ideally with citations to the source.

Why does my RAG system give wrong answers?

Most often because of what it retrieves: duplicate or conflicting pages, outdated versions, indexed drafts or chunks without context. Missing permission filters and the lack of a pre-launch evaluation set are the other common causes.

Should I use RAG or fine-tuning?

Use RAG when information changes, needs citations or has access rules. Use fine-tuning to teach a model a style, format or narrow task. Many production systems combine both, with RAG supplying the facts.

Can I build RAG on Drupal?

Yes. Drupal's AI module includes AI Search, a Search API backend that chunks content, generates embeddings through a configured AI provider and stores them in a vector database. AI Assistants can then use that index to answer questions from your site's content.

How do I keep RAG from leaking restricted content?

Index only approved content, filter results by the requesting user's permissions before they reach the model, treat retrieved text as data rather than instructions, and log the sources behind every answer.

Want your site to answer questions from its own content, with citations and permissions intact? Talk to Seed EM. We'll start by assessing how RAG-ready your content is.

Sources

1. Lewis et al. (2020), Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks. arxiv.org/abs/2005.11401. ↩
2. Anthropic, "Introducing Contextual Retrieval" (September 19, 2024). anthropic.com/news/contextual-retrieval. ↩

Published by Seed EM · October 2026.