Dis Dat — THE Capability Set (locked)

The definitive, exhaustive, honest statement of what Dis Dat captures and what it cannot — for every modern-web experience. No hype. Graded against the actually shipped extension at /root/leo/apps/disdat/product/ext-pointtalk (manifest.json v0.10.0; content.js, inject-main.js, inject-hooks.js, background.js, offscreen.js), not against an aspirational spec. Where a capability is genuinely not in the shipped build, this document says so in the open and marks it ROADMAP.

1. THE PRINCIPLE (locked)

Anything client-side that the user EXPERIENCES, Dis Dat captures. What they see (visual), how it moves and feels (motion, micro-interaction), what they are given (data and content rendered to them), how they interact (gestures, forms, input), and persistence across states (exit a view, return, it resumes as if never closed).

The capture boundary is exact:

The only other things on the CAN'T side are a handful of platform security walls (closed shadow roots, cross-origin iframe internals, cross-origin stylesheet rule-text, DRM-decoded media, worker/GPU readback, HttpOnly cookies). These are browser-enforced boundaries, not Dis Dat limits — and crucially, none of them is something the user actually experiences as content; they are the protected plumbing underneath it.

One-line boundary: If the user can see it, feel it, or was handed it, it is on the CAN side. Only the server's hidden brain and a few security walls are on the CAN'T side.

2. THE ARCHITECTURE (how the captured experience reaches the agent)

Two parts, by design:

(1) A light in-extension LLM crunch that GROUNDS the reaction. The user talks while reacting. Each remark is sliced to its time-window and turned into one classified, grounded note, then a whole-session prose briefing in the user's voice. This crunch is referential — it knows the app-design nuances and ties "this feels flat / slow / janky" to the concrete fact it can see — but it does not carry the heavy payload. Shipped engine order (background.js): Groq llama-3.3-70b-versatile (free, fast, primary) → Anthropic Claude (Sonnet) fallback if a key is set → bundled on-device WebLLM in the offscreen document as the keyless local path. Transcript via Deepgram Nova-3 (else Web Speech). The crunch is deliberately faithful: it states only what the user said, never invents a reaction to an element they didn't name, and marks inference as an explicit "probably because…".

(2) The FULL captured payload, handed to the agent verbatim. A ═══ DIS DAT GROUND TRUTH ═══ appendix rides alongside the prose on the clipboard — exact selectors, the full non-default computed-style set, the authored CSS rules that matched the element (incl. untriggered :hover/:focus/:active/state variants + ::before/::after, recursing through @media/@supports/@container, with var() tokens intact) for same-origin sheets, getAnimations() timing/easing/keyframes, WebGL shader source + uniforms + the rendered frame for canvas reactions, framework component·props·state·source-pointer, a11y role/name, attributes, bounded outerHTML, asset URLs + inline SVG, the full network exchange (request + response headers and bodies, token-redacted, plus fetch-SSE / WebSocket / EventSource stream frames), console errors, the render/jank trace, and the environment baseline. The briefing tells the agent: "act on the exact bytes below." So the agent gets everything as if it had been sitting in the user's browser across the page(s) they experienced before and while narrating.

The intended end-state (ROADMAP, not in v0.10.0): the heaviest payload — font/image/media bytes — uploaded to a temp file (tmpfiles-style, auto-expiry, datacenter-reachable) that the agent downloads, with bytes captured on the user's residential IP (no 403) and re-hosted neutral. This is the "self-hosted the entire page(s)" promise. Today the asset delivery is URLs + inline SVG verbatim — not a re-hosted byte bundle. Where a row below says "byte re-host", read it as the roadmap target; the shipped delivery is the URL pointer (text bodies themselves are captured inline, redacted, capped at 8000 chars per exchange).

Honest framing of the build (graded against shipped v0.10.0). The shipped build is far richer than an earlier grading (done against a stale v0.8.0 copy) credited it for. The shipped extension does harvest matched authored CSS (matchedRules(), with untriggered state + pseudo recovery and var() intact), does run a dedicated ::before/::after pass (pseudoStyles()), does capture WebGL shaders + uniforms + the rendered frame (inject-hooks.js, MAIN world, document_start), and does capture the full network exchange including fetch-based streaming, WebSocket frames, and EventSource events (inject-main.js). The KEEP style whitelist was removed in v0.9.1 — extract() now keeps every non-default computed longhand. The genuinely missing pieces are narrow and named below as ROADMAP: a viewport screenshot (no tabs permission), a tmp-file byte re-host of asset bytes, and an interaction-trace (causal action→call→state-diff over time / motion as a recorded timeline / multi-width re-query of other breakpoints). Every one of those is client-observable and buildable — not a server wall — which is why they stay on the CAN side in principle even while marked 🟡/❌ today.

Legend: ✅ CAN · 🟡 PARTIAL (with the named residual) · ❌ CAN'T (server-brain or platform wall) · 🟡→ROADMAP / ❌→ROADMAP (client-observable, buildable, not yet shipped).

3. THE COMPLETE SET

3.1 Layout & CSS visual system

3.2 Motion, animation & micro-interaction / feel

Structural truth of the motion domain in v0.10.0: played motion is captured as a single-frame snapshot at the instant of reaction (getAnimations() point-read + computed shorthands + the matched authored animation/transition rules, incl. the untriggered target state) plus a coarse time-stamped trail (mutations, perf entries, cursor, events). It is not a frame-by-frame recording of the played motion. The universal fix — a requestAnimationFrame sampler / interaction-trace — is the highest-value remaining ROADMAP item for "feel over time".

3.3 Graphics, media & audio

3.4 Data, networking & API

3.5 Persistence, real-time, offline & sessions

3.6 App platform & input

4. THE CAN'T-DO SIDE, ISOLATED

This is the entire list of what Dis Dat genuinely cannot capture — and it is exhaustively short. Nothing here is anything the user actually experiences as content. Every item is either the server's hidden brain or a browser-enforced security boundary protecting credentials, other origins, or protected hardware.

A. SERVER BRAIN — private logic + private data (the principled, permanent CAN'T). No client tool can ever reach these because, by definition, they never enter the browser:

• The server's hidden algorithm / business logic — ranking, recommendation, pricing/surge, fraud/risk scoring, content-moderation decisions, server-side choreography. We capture every client-observable input and output around it; never the code or model.
• The server's private datasets / database — the rows the user never fetched, the full table, the schema beyond what responses imply. We reconstruct every record the user saw and the data contract; never the unfetched remainder.
• Bucketing / targeting / edge logic — which variant a user gets, rollout %, server-evaluated flags never sent to the client, CDN/geo-routing, rate-limit/quota decisions. Observable only as the variant or response that arrived.
• Per-feature corollaries that are server-brain on one axis: the merge algorithm behind conflict resolution, the command-palette ranking when server-backed, the server persistence/versioning of drafts/sessions, the server's accept/reject of an optimistic update. In every case the client-visible result IS captured.

B. PLATFORM SECURITY WALLS — browser-enforced, not Dis Dat limits. A handful, each protecting something the spec deliberately keeps out of reach:

• HttpOnly cookies — JS cannot read them, ever. (Non-HttpOnly cookies + tokens are deliberately redacted by the shipped redact() pipeline — a security-by-design choice, not an inability.)
• Cross-origin stylesheet rule-text — .cssRules throws on a cross-origin sheet (same-origin policy), so the authored CSS harvest (matchedRules()) skips it. The EFFECT still lands in computed; same-origin authored CSS is fully harvested. Mitigation: byte re-host of the .css off the residential IP (ROADMAP).
• Cross-origin iframe internals (Stripe/3DS card fields, ad frames, reCAPTCHA/Turnstile) — the same-origin policy forbids the parent from reading their pixels/DOM. (Mitigated where the extension injects its own per-frame sensor; the parent still can't.)
• Closed shadow roots — mode:'closed' interiors are unreachable by any script.
• DRM-decoded media (EME/Widevine/PlayReady) — decoded frames live in the CDM black box; toDataURL on a DRM video throws. A content-protection wall.
• Worker / GPU readback edges — OffscreenCanvas in a worker realm, the WebGPU swapchain without configured copy usage, the immersive WebXR stereo framebuffer, a cross-origin-tainted WebGL/canvas frame — realm-reach / hardware / taint boundaries. (The WebGL shader SOURCE + uniforms are still captured; only the tainted pixel readback is walled.)
• Browser-chrome UI rendered outside the DOM — native form pickers (date/color/select/file), browser autofill, the native title tooltip, OS notifications, permission prompts. The resulting value/state is captured; the chrome popup itself is not in the page.
• :visited link color — browsers deliberately lie to getComputedStyle for anti-fingerprinting.
• Exact server-side timing / internal state — TTFB-internal processing time and server memory are knowable only where the server chooses to expose them (Server-Timing).

Everything else — every visual, every motion, every byte of data the user was handed, every interaction, every persisted/resumed state — is on the CAN side: captured outright, captured partially with the named residual, or delivered as a recipe the agent re-derives. The residuals on the CAN side are not server walls; they are client-observable and buildable (the ROADMAP items), which is why they remain on the CAN side in principle even when marked 🟡/❌ today.

Roadmap (not yet shipped)

Verified absent from v0.10.0 — narrow, named, all client-observable and buildable (none is a server wall):

1. Viewport screenshot — captureVisibleTab via the tabs permission (not in the manifest today). The one witness that captures everything visible regardless of how it was drawn.
2. Temp-file byte bundle + residential-IP byte re-host — font/image/media bytes uploaded to a tmpfiles-style temp file and re-hosted neutral so the agent's datacenter fetch can't 403. Today assets are delivered as URLs + inline SVG (text response bodies are captured inline, redacted, capped 8000 chars/exchange).
3. Interaction-trace — a causal action→call→state-diff log over time (the played motion AS A TIMELINE: spring settle / fling decay / overshoot / cadence; the stick/unstick scroll moment; a multi-width re-query for OTHER breakpoints' resolved values). Gesture/scroll/physics feel-over-time is still re-derived from a single-instant snapshot + coarse trail, not recorded.

5. Completeness statement (the lock)

Anything on a modern web page falls into exactly one of three buckets, and this set accounts for all three:

1. Captured (✅) — the client-observable fact is delivered verbatim or as a faithful recipe.
2. Partially captured (🟡) — captured with a single, named residual, and the residual is stated honestly (a cap, a single-instant snapshot, a not-yet-shipped ROADMAP hook, or a platform wall such as cross-origin stylesheet rule-text).
3. Can't (❌) — and the can't side is only the server's hidden brain (private logic + private data) plus the short, fixed list of browser security walls in §4 — never anything the user actually sees, feels, or is handed.

The set is complete on its own terms: the boundary is exact, every modern and bleeding-edge surface (through the 2024–2025 CSS, networking, media, and platform additions) is placed on it, and the gap between today's shipped v0.10.0 and the full promise is itself fully enumerated as the three named ROADMAP items above — none of which is a server wall. If the user can experience it, it is captured, partially captured with a named residual, or explicitly a server/security wall. There is no fourth category.

Proof case: elevenlabs.io audio-reactive fluid orb (v0.13.0)

Target: elevenlabs.io/agents/ai-outbound-calling — the audio-reactive fluid "voice orb" (a WebGL2 + GLSL fluid simulation), plus the page's praised hero button and its live API. Picked as a worst-case for animation + API complexity. Validated by an adversarial multi-agent run: capture → rebuild-from-capture-only → 3 skeptical verification lenses.

Captured verbatim (proven)

• All 10 WebGL programs, byte-for-byte — 9 GLSL1 fluid-sim kernels (advection, divergence, curl, vorticity confinement, Jacobi pressure, gradient-subtract, clear, splat, blur) + the bespoke WebGL2/GLSL3 orb composite (sphere SDF, simplex/fbm warp, ring, colour-correct). The full 370-line / 11,242-char fragment shader + its vertex shader survive intact in elements[].gl; all 10 parse cleanly. The handoff text shows the first 140 lines and points to the bundle for the rest.
• Live uniform snapshot + a real 512×512 rendered PNG frame — the resting visual is exact.
• The praised button — pixel-rebuildable from static capture: background-color rgb(242,242,242), WaldenburgFH 14px/700, 0.7px tracking, uppercase, the verbatim 3-layer box-shadow, border-radius 9999px, inline SVG icon asset, cubic-bezier(0.31,0.325,0,0.92)/0.3s transition, click listener + React onClick.
• Network/API contract (20 requests, headers/bodies/timing), causal interaction timeline (gaze + scroll + network + DOM-mutation on one clock), 16–24 screenshots, one re-hosted gzipped bundle (no origin 403s).

The reproduction result — honest

A from-capture rebuild reviewed by 3 adversarial lenses ran and was correct in mechanism (12 shaders + the full Navier-Stokes pipeline carried verbatim), but was not a pixel-perfect clone on capture alone. What had to be inferred: the JS render harness (FBO graph, ping-pong, Jacobi iteration count), the audio pipeline (getUserMedia→AnalyserNode→uniforms), the active-state uniform values (orb was idle/silent at capture), and the gradient colour-ramp texture (not in the capture). Sharp finding: in the original, the fluid sim contributes almost nothing visible — the orb's look is SDF-sphere + fbm + gradient + ring; the fluid field is only a sub-pixel UV nudge.

The hard boundary (confirmed)

• Server brain — permanent. The calling agent (LLM, voice model, telephony, dialog state) is server-side. We capture the API contract, never the logic. No client tool can.
• JS render harness — not captured, but reconstructable here only because the shaders are a near-verbatim fingerprint of Pavel Dobryakov's open-source WebGL-Fluid-Simulation (shared 5-tap neighbour vertex shader, exact divergence/curl/Jacobi stencils, his uniform names). Identical shaders from a bespoke unknown sim would leave the harness unrecoverable. Don't over-generalise.
• Authored assets not rendered to a captured frame (e.g. the orb's gradient texture) — absent.
• Active uniform values when the element is idle at capture — inferred, not captured.

Fixed in v0.13.0 / v0.15.0 / v0.16.0 as a result

• Authored CSS capture was returning ZERO rules on every modern site — three layered bugs, found in order by re-capturing and reproducing live:
  1. (v0.13.0) styleSheet.cssRules throws on CDN/subdomain-served CSS → the main Tailwind sheet (served off eleven-public-cdn) was unreadable. Fixed by having the background fetch the blocked sheet (host perms) and rebuilding it as a constructable CSSStyleSheet we can walk.
  2. (v0.15.0) the pseudo-stripping regex ate the :hover/:active inside escaped Tailwind class names (.enabled\:hover\:bg-x:hover), mangling the selector. Fixed with a (?<!\\) lookbehind.
  3. (v0.16.0 — the dominant one) the rule-walker did if (rule.cssRules) { recurse; continue } to descend into @media/@layer. But modern Chrome exposes an (empty, truthy) .cssRules on every CSSStyleRule for CSS-nesting support — so the continue skipped every style rule's own selector before it could match. This zeroed authored CSS on same-origin (claude.ai) AND cross-origin (elevenlabs) alike. Proven with a headless-Chromium repro of the live page: old walker matched 0 rules on the CTA button, fixed walker (check selectorText first, recurse only when cssRules.length>0) matched 31 (.inline-flex, .h-40, .items-center, …). Net: this was the root of the "hover/active microinteraction not captured" gap across the modern web. Lesson logged — verify capture claims against a real recovered bundle, not the happy path.
• Deeper de-noise — logical-property aliases, default flex/outline/transition/animation values dropped (measured ~42%+ computed-prop cut, full signal retained).
• Screenshot pollution fixed (v0.12.1) — our own gaze-trail + recording pill no longer painted into the captured frames.
• Screenshot window extended 16→24 frames; shader-truncation note points to the bundle.

Still open (named residuals, not walls)

• Capturing the bound texture/asset bytes an element samples but doesn't render to the captured frame (the gradient ramp) — would need readPixels of the bound texture or an asset fetch.
• Capturing active dynamic uniform/state values when the element is idle at capture time.
• Multi-page capture still collapses toward page 1; console[] often empty.

Source document ↗