Decision Lens

Situation

The Trust Stack diagnostic proved the methodology worked. It did not prove it could run repeatably, or without someone watching. Decision Lens is the product side of that problem: a hosted tool that takes a subject, runs LLM-driven analysis end to end, and returns a high-level report a client can read.

One constraint shaped everything: the output is client-facing, so a wrong answer is worse than no answer.

The hard part

When a company’s website refuses to be analysed

Decision Lens works from a company’s public web presence. Soon after launch, an analysis came back nearly empty: the subject’s site sat behind commercial bot protection, and every request from our infrastructure was turned away. The same pages loaded fine in a browser. Our pipeline got a block page.

We chased it down properly. The plain fetch was refused, so the pipeline escalated to rendering the page in a real browser, and the protection layer challenged that too. The identical browser, run from a home connection instead of our cloud host, sailed through. The block keyed on where the request came from, not on anything the code did, which is why every local test passed while every production run failed.

The pipeline handled the refusal the way I’d want it to: it declined to guess. A minimum-evidence floor kept the challenge page from being analysed as if it were brand content, and the report said plainly that it couldn’t access enough public material to score the company, rather than producing a confident-looking analysis of an interstitial that reads “Just a moment…”

How I knew it worked

A fixed case matrix, run headlessly

Prompt changes are the dangerous part of a pipeline like this. A wording tweak that improves one subject can quietly break another, and without a harness the first person to notice is a client.

So the pipeline is triggerable headlessly, and there’s a fixed matrix of subjects that gets re-run against every meaningful change. The cases cover distinct ways the pipeline can fail rather than trying to be representative, and at least one subject is deliberately sensitive, because that’s where a generic analysis pipeline gets you in trouble.

• Regression on structure: does every section still come back populated?
• Regression on evidence: are claims still linked to sources, or did the model start asserting?
• Regression on tone and sensitivity: does the analysis stay appropriate on a subject where it could easily not be?
• Cost per run: tracked as a first-class output, because a quality gain that triples spend is a product decision someone should get to make on purpose.

Also true

• Access is gated and shared with the reporting dashboard, so there’s one credential model rather than two and nobody ever has to think about it.
• The gate is feature-flagged, so it can come down for a demo without a deploy.
• It ships as a subpath of the main site rather than its own domain: one entity, one set of trust signals, which is exactly what the product advises its own clients to do.

What broke

Focus introduced confusion

Outcome

A running product with a repeatable path from subject to report, and a test suite that makes prompt changes safe to ship. Available to walk through on a call.

Implementation details are proprietary to All Things Trust. This page describes decisions and their tradeoffs, not the system’s internals.