Build a RAG Chatbot for Your Website: A Real Build Log

Decide Whether You Need One at All

A generic support widget on a small consulting or B2B site is a liability, not a feature. It answers from general model knowledge, which means it can invent services, prices and positions under your name. The only version worth building answers strictly from your own content, shows the passage behind each claim, and refuses when it has nothing. Anything else is a hallucination surface attached to your brand.

I resisted adding a chat widget to this site for a long time, for exactly that reason. A solo consultant's site is a claims machine. Every sentence on it is something I am prepared to defend on a call. A chatbot that paraphrases loosely from model knowledge breaks that guarantee in one answer.

The version I was willing to ship had three non-negotiable behaviours. Answers come only from indexed site content. Each claim carries a visible source. When retrieval finds nothing above a similarity floor, the assistant says so instead of improvising.

The third behaviour carries most of the value. A chatbot that cannot refuse is not grounded, no matter how the architecture looks on paper.

There was also a second reason to build it: proof of method. This site argues that citation attribution is a separate, failure-prone step in AI answer systems. Running my own retrieval pipeline against my own content is the cheapest way to find out whether I actually understand the failure modes I write about. As this log shows, some of them found me quickly.

The Stack: Supabase RAG on a Static Site

The stack is the least interesting part of this article. Every tutorial covers it, and a coding agent assembles it without difficulty.

• Static HTML site, vanilla JavaScript, no framework, no build step, hosted on Vercel
• Supabase with pgvector for storage and similarity search
• OpenAI text-embedding-3-small, 1536 dimensions, for both ingest and query embedding
• A serverless /api/chat endpoint: validate the request, embed the query, call match_documents, generate with gpt-4o-mini, strip [n] markers from the display text, return { answer, sources, mode }
• A widget injected at runtime by the shared site chrome, after idle or roughly 1.2 seconds

One setup detail actually blocks people, so it gets its own paragraph. The pgvector schema DDL cannot be executed through the Supabase REST API key. It had to be run manually, by a human, in the Supabase SQL editor. If your coding agent stalls at schema creation, this is why. Everything else in the setup can be delegated, but not this step.

Stage 1: Ingest and Chunking

The goal of ingest is a corpus where every chunk is a plausible piece of evidence. Chunking a website is not the same as chunking documents: a site is full of navigation, listing pages, CTA blocks and contact forms that are content-shaped but evidentially empty. The first ingest pass treated all of it as content.

What broke

Symptom. This failure did not show up at ingest time. It surfaced two stages later, as bad citations. When I audited which passages were being cited for unsupported claims, every bad citation mapped to the same chunk type: thin navigational content. One-line service blurbs from a listing page. A heading fragment.

The measurement that found it. A manual audit of citation markers against the retrieved passages they pointed to. The pattern was consistent enough to name: these chunks sit semantically close to many queries and are evidence for none. A one-line blurb mentioning CRM will be retrieved for almost any CRM question, and it supports almost no specific claim.

The fix that did not work. Two rounds of prompt engineering, covered in the grounding stage below. You cannot instruct a model into citing well from a corpus that contains no citable evidence for the claim at hand.

The fix that did. Rebuilding ingest. Sections now fold on h2, so short cards merge into their parent section instead of standing as isolated fragments. Listing pages are excluded except for named substantive sections. Contact forms are dropped. Short CTA blocks are dropped unless they contain a currency figure, a percentage or a named platform, which is a cheap proxy for whether a block carries any evidence.

The corpus went from 968 rows to 467. Median chunk length landed at 1,168 characters. Half the rows disappeared and citation quality improved, which tells you what the first ingest pass had been treating as content.

Stage 2: The Retrieval Endpoint

The goal of the endpoint is a fixed contract: take a question, return an answer with sources, and never let the model answer from anything except retrieved passages above a similarity floor. The floor on this build is 0.45 cosine similarity. Passages below it are treated as not found.

What broke

Symptom. A pricing question returned a gap response, the assistant's way of saying the site does not cover this, even though the price table exists on the site.

The measurement that found it. Logging the below-floor retrievals. The correct price table was retrieved at 0.293 similarity against the 0.45 floor. The content was found and the floor rejected it.

The fix. Unresolved. Short, vague queries are the weak case: a terse question embeds poorly against a long, specific passage, and the similarity score lands well under any floor strict enough to keep grounding honest. Lowering the floor would admit the price table and also admit noise. This is a tradeoff, not a bug, and on this build the floor stayed at 0.45 with the failure documented rather than papered over.

One observation from this stage generalises: log what your floor rejects, not only what it passes. The 0.293 near-miss was only visible because below-floor retrievals were recorded.

Stage 3: Grounding and Citations

The goal of this stage is answers where every claim maps to a retrieved passage through a visible [n] marker, which the endpoint then converts into source attribution. Retrieval was already healthy going in, which made the failures at this stage easy to misdiagnose.

What broke

Symptom. A question about CRM engagement deliverables produced a full prose answer with markdown links to site pages and zero citation markers.

The measurement that found it. Inspecting the retrieval log for that query. Six passages were retrieved, all above the 0.45 floor, top similarity 0.596. Retrieval was fine. The prompt permitted canonical links, and the model chose links over markers. The grounding pipeline worked and the citation behaviour still failed, because they are different things.

The fix that did not work, round one. The prompt was changed to state that an answer containing no [n] marker is invalid output. The model complied by attaching markers wherever it could reach. Of five markers examined after this change, one fully supported the claim it sat on. The worst case was a claim that I run TikTok campaigns, cited to a passage listing Google Ads, Meta Ads and analytics platforms only.

Every failure had the same shape. Several claims packed into one sentence, a cluster of markers at the end, one passage supporting one of the claims. This is the citation faithfulness problem described in the sources-and-citations guide reproduced on a corpus of one small website: a visible citation can partially support the claim it is attached to, and forcing citation presence makes the mismatch worse, not better.

The fix that did not work, round two. An explicit one-marker-per-sentence rule was added to the prompt. It did not stop the clustering.

The fix that did. Fixing the corpus, as described in the ingest stage. Once thin navigational chunks were removed and sections folded into substantive units, citation faithfulness moved from 3 of 8 markers fully supported to 6 of 9. The remaining failures changed character: marker clustering, and citing a sibling section of the correct page, no longer thin chunks. Nothing changed in the prompt. The improvement came entirely from the corpus.

Stage 4: Observability

The goal of this stage is one specific distinction: making a citation failure observable. Without instrumentation, a broken citation pipeline and an honest refusal look identical from outside. Both return an empty sources array.

What broke

Symptom. Nothing visibly broke here, and that was the problem. The failure mode of this stage is silence: uncited answers passing as legitimate refusals with no signal separating them.

The fix. A mode field on every response: cited, standing, gap, uncited. The value derives from a model-emitted marker, with a fallback based on whether sources were attached and whether any passage cleared the floor. The uncited mode exists specifically so that a citation failure is distinguishable from an answer that legitimately needed no citation.

The measurement it enabled. After the corpus cleanup, uncited stopped firing across test queries. The mode fired before the cleanup and went quiet after it. Its silence carries information precisely because it had fired before.

Stage 5: The Entry Point in the Page

The goal of this stage is getting the assistant in front of visitors without degrading the page. Two entry points were built: a floating card opened from a launcher, and a module placed inline in article body content. This was the sprint that prompted the whole project, and it was one of six. The other five were correctness work.

What broke: the launcher nobody could tap

Symptom. The launcher, a 60 by 60 pixel icon in the bottom right, was reported as unnoticeable.

The measurement that found it. At a 390 pixel viewport, the CookieYes consent banner measured 390 by 378 pixels at z-index 9,999,999. The launcher sat in a container at z-index 500. Calling elementFromPoint at the launcher's own centre returned the banner. The overlap covered the launcher's full 3,600 square pixels. For a first-time mobile visitor, the launcher was untappable until they acted on the banner. Desktop was unaffected, banner bottom-left and launcher bottom-right, which is why nobody noticed.

The fix that was rejected. Raising the launcher's z-index above the banner. That works mechanically, and it is a dark pattern: stacking a chat widget over a consent notice puts marketing furniture above a legal disclosure, so it stayed out.

The fix that shipped, and the regression it caused. Hold the launcher until the banner is dismissed. Clean-profile timings put the consent cookie written at 276 to 339 milliseconds with no user interaction, and the launcher painted at 425 to 506 milliseconds. But the first version applied a 2,500 millisecond grace period, intended for the first-visit cookie race, to returning visitors as well, pushing their first paint to 2,650 milliseconds. Corrected to 249 to 318 milliseconds.

The honest scope of this win. The launcher became usable at the same moment either way, because the visitor had to act on the banner first regardless. The fix removed a phantom element during a window when it was already unreachable. It did not recover users. I am including it because the diagnostic method transfers even where the business impact was nil.

What broke: answers that did not fit

Symptom. Sample answers written at 40 words rendered well in the card. Live answers came back at around 300 words, as markdown numbered lists, into a container with roughly 120 pixels of vertical space for answer text.

The fix. Layout cannot fix this. No amount of CSS turns 300 words into 120 pixels. The generation prompt needs a length ceiling, which moves the problem to where it can actually be solved.

The inline module, and why it is static HTML

The inline module's question and answer pairs are static HTML, present in the initial server response and readable with JavaScript disabled. This was deliberate. A site arguing that script-only content is an unreliable retrieval target cannot inject its own demonstration content via JavaScript. Verified by loading the page with JavaScript disabled and extracting the text.

Three Ways Verification Lied

The most transferable finding from this build concerns instrumentation rather than RAG. Three separate times, a verification step reported success while measuring nothing. All three incidents share one structure: the check produced a plausible result that was disconnected from the thing being tested.

The throttle that never applied. A Fast 3G throttling profile silently failed to apply. The measurement returned a plausible timing number that was not measuring what it claimed. It was caught by re-checking the setup, not because the result looked wrong, and that detail is what makes this failure mode dangerous. The number was believable.

The clean profile that was not clean. A clean-profile test ran against a browser context that already held a consent cookie, so the branch under test, the first-visit path, was never exercised. The result was valid and irrelevant. Every assertion passed without ever touching the code path in question.

The numbers that passed on a broken build. Every numeric check passed on a build whose card was visually broken: answer text clipped mid-sentence, source pills overlapping the paragraph. Heights, tap targets and token compliance were all within spec, on an interface nobody could use.

What the three have in common: a passing check confirms that the check ran, not that it measured what it was supposed to measure. The precondition of the measurement, throttle applied, profile actually clean, spec actually describing usability, is itself a claim that needs verification.

The process change that came out of this: screenshots at every viewport for every state are now required before any visual sprint is accepted. They are the acceptance gate, not an extra, because numeric checks demonstrated three times that they will pass on broken work.

Working With a Coding Agent: Cursor or Claude Code

The build ran across roughly six sprints in Cursor. Five were infrastructure and correctness. One was the visual work that prompted the project. The ratio came from the failures, not from a plan.

Nothing in the method below is Cursor-specific. It is a set of instructions, and Claude Code or any other coding agent takes them the same way.

The working method that held up:

• Read-only diagnosis first. Each sprint opened with the agent instructed to investigate and report findings, then stop. Not fix what it found. The consent banner overlap, the retrieval floor near-miss and the citation audit all came out of diagnosis passes that changed nothing.
• One change per sprint. When a fix introduced a regression, the 2,500 millisecond grace period hitting returning visitors, the single-change discipline made it attributable immediately. There was exactly one candidate cause.
• Verification after, with the caveats above. Verification is a sprint step, and after three instances of instrumentation lying, the verification setup itself gets checked before its results are believed.
• Screenshots for anything visual. Every viewport, every state, before a visual sprint is accepted.

The instruction to report and stop matters more than it sounds. An agent that fixes what it finds mid-diagnosis destroys the evidence of what was broken and stacks changes you cannot attribute. Report and stop keeps diagnosis and treatment as separate sprints, which is the only reason this article can state what each fix actually did.

What Is Still Unsolved

Three items are open at the time of writing.

Response time is around 4.4 seconds. That is embed, retrieve and generate in sequence, and it is slow for a chat interaction.

Citation faithfulness is at 6 of 9, not 9 of 9. The remaining failures are marker clustering and citing a sibling section of the correct page.

The verification pass is identified, not built. A second, cheap model call that drops markers failing an entailment check against their passage is the identified next step for the remaining faithfulness gap. It does not exist yet, and nothing here should be read as evidence that it works.

Methodology and Limitations

This article is a build log from one build, on one site, with one corpus, one embedding model and one consent-management setup. Nothing in it is a benchmark, and none of the numbers should be generalised into claims about RAG systems as a category. Where a measurement appears once in the source material, it is reported here as one observation.

The material comes from two sources. The early build, sprint one and the beginning of sprint 1.5, is documented in the verbatim Cursor session history: prompts and agent output, unedited. The later sprints, covering the corpus rebuild, the citation audits, observability and the launcher work, are reconstructed from sprint summaries written after the fact rather than from a verbatim transcript. Measurements from the reconstructed portion were recorded at the time they were taken, but the surrounding narrative is a reconstruction, and I flag that openly rather than presenting the whole log as transcript-grade.

The identified next step, an entailment-based verification pass, is a design, not a result. It is named as unbuilt wherever it appears.

Build period: late July 2026 · Last reviewed: August 3, 2026

The mechanism-level background on why citations fail in larger systems the same way they failed here is covered in the companion guide linked in the grounding stage above. And if the reason you read this far is that AI systems keep describing your company from thin evidence, that is a corpus problem too, just not one you control directly. It is the kind of thing I work on with clients.