RAGdrag Update: v0.6

RAGdrag v0.6 is out. If you have been running v0.5, some of your commands will behave differently.

The short version: the tool will not write to a knowledge base unless you have told it how to clean up afterward, and it will no longer call a finding proven when all it has is a word-count coincidence.

The default scan got narrower

v0.5 chained all six phases by default. v0.6 does not.

ragdrag scan now runs R1, R2, and R3 only, at an active non-mutating ceiling. R6 is opt-in. R4 and R5 require --allow-write.

R1 through R3 read. R4 and R5 write. Those should never have been the same keystroke.

R4 and R5 require cleanup controls

Point RAGdrag at a knowledge base with no cleanup route and it stops before it touches anything.

$ ragdrag poison --target http://127.0.0.1:8899/chat --response-field response
Error: R4/R5 require --cleanup-url with a final {id} segment.

Exit 3. Nothing was written.

Give it the route but skip the controls and it stops again:

Error: R4/R5 require --established-control baseline, negative-control, and cleanup-verification.

Three controls, declared explicitly: a disposable baseline, a negative control, and post-run cleanup verification. You are asserting those exist. The tool holds you to it by recording every mutation in a ledger and reporting any it could not resolve as a distinct exit code.

One more guard worth knowing about. The cleanup URL is origin-bound. Aim it at a host that is not your target and you get Invalid or unapproved --cleanup-url. RAGdrag will not be talked into firing DELETEs at someone else's server.

The new full chain

ragdrag scan --target http://127.0.0.1:8899/chat \
  --phases R1,R2,R3,R4,R5,R6 \
  --max-requests 300 \
  --allow-write \
  --cleanup-url 'http://127.0.0.1:8899/documents/{id}' \
  --established-control baseline \
  --established-control negative-control \
  --established-control cleanup-verification \
  --response-field response \
  --history-field messages \
  --output fullchain-report.json

Longer than it used to be. Every added flag is a thing youre now on the record as having set up.

Findings now say how much they actually know

Every finding carries an evidence state now. validated means a payload you wrote came back out of the system, proven by a high-entropy marker from your document showing up verbatim in a later response. observed is weaker: a signal was right there in the response, nothing more. inferred means a heuristic matched and you should go look.

Through v0.6.1, validated could never happen. It was in the schema and counted in every summary, and nothing in the tool ever set it.

Worse, retrieval used to be judged on word overlap between the payload and the response, with anything over 30 percent called verified. That fires on vocabulary. A RAG app answering any password question shares "password" and "reset" with our payload without ever having retrieved it.

Raising the threshold would not have helped. On one lab run the ratio came back inverted: a finding at 30 percent had the marker echoed verbatim and was real, and a finding at 56 percent had no marker at all. Wrong in both directions, same run. The overlap number is still in the report. It no longer gets a vote.

In practice this is a triage shortcut. The validated findings are the ones you can hand to an engineer without hedging. The rest are leads.

One flag you will need

ragdrag scan now takes --ingest-url. Without it, write phases only find an ingestion endpoint if the target answers OPTIONS with a 2xx and an Allow header. Plain FastAPI answers 405, so R4 and R5 quietly wrote nothing. If a write run reports no mutations, that is the first thing to check.

Also in this release

• Schema 1.0 reports with exit codes 0 through 5, separating findings from partial assessments from unresolved cleanup
• Credentials and raw extraction redacted by default. Owner-only opt-in for raw capture
• Mutation ledger with cleanup on failure and on interrupt
• Request budgets. --max-requests on every network command, default 100
• Typed capabilities, origin-bound transport, evidence states
• Loopback listener by default. Binding public requires explicit acknowledgment
• py.typed, Python 3.10 through 3.12 CI gates, wheel and sdist clean-install checks

Verified, not assumed

1,856 tests pass across the tool and the lab repo. The full R1 through R6 chain runs against the Full Chain lab target and finish clean: all six capabilities present, R4 and R5 both reporting cleanup_state: removed, and unresolved_mutations: 0.

One of those tests exists only because of the bug above. It fails the build if any declared evidence state has no code path that can produce it. A state nothing can emit should break CI rather than ship in a schema.

That run exits 2, not 0, and that is the documented pass. The lab deliberately returns 403 during R6's rapid-probe sequence, so a complete exercise reports partial. An exit code that distinguishes "finished with findings" from "could not finish" from "could not clean up" is worth more than one that always says zero.

The lab hands out a synthetic AWS key in plain text if you ask it nicely. That value does not appear anywhere in the default report.

Two notes

The walkthrough post predates these defaults. The commands in it still exist, but scan no longer chains all six phases on its own and the write phases now need the controls above. Read this post first, then that one.

And a correction. The RAGdrag taxonomy catalogues 27 techniques. The tool implements 20 of them: two in R1, five in R2, two in R3, four in R4, four in R5, three in R6. The changelog has said all along that catalogued entries are not implementation claims, and I had been quoting the taxonomy number. It is 20.

What it still cannot do

A cleanup DELETE that returns 200, 202, 204, or 404 is recorded as removed. A 202 only establishes that the target accepted the request, which is not the same as the document being gone. Check target state after every write run.

validated also has exactly one source right now, which is marker confirmation in R4. Correlating a real callback hit from ragdrag listen back to a scan is not wired up yet, so no scan will ever hand you a callback-driven validated. That one is on the roadmap, not in the box.

One naming trap, since it cost me an afternoon. The generated implementation status table has a Maturity column that also says validated, and it means something completely different: a declared test file is present on disk. Not that it passed. Two unrelated things wearing one word in the same schema. A rename is queued for 2.0.

Shipping the limits next to the features is the only reason the exit codes mean anything.

Repo: github.com/itsbroken-ai/RAGdrag
Labs: github.com/itsbroken-ai/RAGdrag-labs