DADR

Solution: Data Analysis Decision Records¶

Data Analysis Decision Records (DADR) are designed to make our key analytical choices explicit. Each decision becomes a short Markdown record, stored alongside the code, data, and prose of a project. The DADR markdown files are human-editable, LLM-readable, and version-controllable. They include meta-data fields and tags that enable external tools to track decision flows, audit the faithfulness of a codebase to recorded decisions, and generate reports and narratives directly from the decision ledger.

DADR is not meant to be a comprehensive record of every thought, but rather a focused ledger of the key decisions that shape an analysis. It is a simple specification inspired by the Architecture Decision Record, a proven and popular development practice in the software engineering community.

Audience¶

DADR serves three audiences:

Frictionless¶

DADR is easy to implement because automated tools and LLMs do all the hard work! In the vast majority of cases, all the analyst needs to do is click "Accept" or "Reject" to determine if a candidate decision should enter the permanent record.

A decision can be initiated in several ways:

1. The LLM agent walks a code base to detect important decisions and drafts candidate decisions for them.²
2. The LLM agent detects a change in the code, analyzes the change, and drafts a candidate decision if necessary (Git hook).
3. The LLM agent drafts a candidate decision based on analyst-supplied prompt and context.
4. The analyst drafts a candidate decision manually.

New possibilities¶

A complete, structured trace of analytic decisions is more than documentation. It is an asset that downstream tools and humans can act on.

Workflow¶

Candidate decisions can be drafted automatically by a LLM agent, or manually by the analyst. Every candidate decision must be reviewed by the human analyst before it enters the permanent record. The analyst can either accept the candidate decision, reject it, or draft a superseding decision that replaces it. Every decision (candidate, accepted, rejected, superseded) is recorded in the DADR ledger to create a complete trace of how the analysis evolved over time.

%%{init: {"flowchart": {"defaultRenderer": "elk", "curve": "basis", "nodeSpacing": 70, "rankSpacing": 70}}}%%
flowchart TD
    analyst["Analyst"]
    no["No decision needed"]
    llm1["LLM agent"]
    candidate["Candidate decision"]
    review(["Analyst approval"])
    accepted["Accepted decision"]
    rejected["Rejected decision"]
    supersede["Superseding decision"]
    log["Data Analysis Decision Records"]

    analyst -- "Ask an agent to<br/>draft a decision" --> llm1
    analyst -- "Modify code or data<br/>(Git commit hook)" --> llm1
    analyst -- "Draft a decision manually" --> candidate
    llm1 --> no
    llm1 --> candidate
    candidate --> review
    review --> accepted
    review --> rejected
    accepted --> supersede
    accepted -.-> log
    rejected -.-> log
    candidate -.-> log
    supersede -.-> log

    linkStyle 9,10,11 stroke:#94a3b8,stroke-width:1px;

    classDef default fill:#ffffff,stroke:#cbd5e1,stroke-width:1px,color:#0f172a;
    classDef sReview fill:#ffffff,stroke:#0f172a,stroke-width:2px,color:#0f172a,font-weight:600,font-size:18px;
    classDef sCandidate fill:#fefce8,stroke:#ca8a04,stroke-width:1px,color:#422006;
    classDef sAccepted fill:#f0fdf4,stroke:#16a34a,stroke-width:1px,color:#052e16;
    classDef sSuperseded fill:#f0fdf4,stroke:#16a34a,stroke-width:1px,color:#052e16;
    classDef sRejected fill:#fef2f2,stroke:#dc2626,stroke-width:1px,color:#450a0a;

    class review sReview;
    class candidate sCandidate;
    class accepted sAccepted;
    class rejected sRejected;
    class supersede sSuperseded;

Example¶

---
title: Drop observations with missing treatment assignment
status: accepted
id: 01938c20-a3b4-7000-8000-111122223333
tags:
  - kind/sample
  - confidence/high
---

## Context

The randomization log is missing the treatment field for 47 of 8,213
respondents. Without an assignment we cannot estimate either
intent-to-treat or per-protocol effects for those rows. This extends
the sampling rule established in [[01938b8e4d70]].

## Decision

> #axis/sample/complete-case
> Drop the 47 respondents without recorded treatment assignment from
> the analysis sample.

## Alternatives

- Multiple imputation of the assignment indicator (rejected: assignment
  is a treatment, not a covariate; imputing it would invent
  counterfactual exposures).
- Coding the missing rows as control (rejected: silently biases the ITT
  estimate toward the null).

## Consequences

The analysis sample drops from 8,213 to 8,166. Power calculations need
to be re-run; see [[01938c30b1e2]].