# Prose habits, machine provenance, and shared influence: what we built and what we measured

A working note on the analysis side of **Tone** (write.work), written to be handed to
people who might improve it. It is weighted towards negative results, because those cost
the most to find and are the most useful thing we can give you.

*Version 3. Version 2 was revised after an outside review whose central correction is now
the shape of this document: three different jobs were being conflated. Version 3 adds the
matched corpus that review asked for, and reports what happened when we audited our own
audit gate (§11b). See §15 and §16 for what changed.*

---

## 0. Three problems, not one

This document is about three related but materially different jobs:

| | job | status |
|---|---|---|
| **A** | Detect **machine authorship** of a passage | **Failed.** Not pursued as a product direction. |
| **B** | Detect undesirable **prose habits** in a passage | Partly works. Precision is good, coverage is uneven. |
| **C** | Detect **shared stylistic influence** across a body of work | Most interesting result. Under-explored. |

> **Tone does not determine whether prose was written by a machine.** It finds recurring
> linguistic habits — some undesirable, some merely characteristic — within a passage or
> across a body of work.

That is not defensive hedging. It describes what the evidence supports, and it is a more
credible and more useful product than the alternative. Machine provenance remains an open
research question here; it does not organise the tool.

**The headline number for job A:** against 20 passages written by a current model and 16
passages of published professional human writing, the tool's default verdict fires on
**0 of 20** and **0 of 16**. Human prose trips slightly *more* lexical rules than machine
prose does.

---

## 1. What the thing is

A prose linter. ~4,400 lines of dependency-free JavaScript that runs in a browser or a
Node process. No model call, no network, ~45ms on 800 words.

| file | what it holds |
|---|---|
| `js/tone-lint.js` | the engine: 233 lexical rules, rhythm, over-correction, constructions, habit families, `corpus()` |
| `js/tone-figures.js` | 12 rhetorical figures the engine can mechanically detect |
| `js/tone-recast.js` | parses antithesis-family figures into typed slots, refills through other figures |
| `mcp/tone-mcp.js` | MCP server so an agent can call it |
| `test/run.sh` | runs all three suites, checks exit codes, prints ALL GREEN or fails loudly |

Passes over a passage: **lexical** (233 rules, fused capture-group regexes, code/URLs/quotes
masked out first), **rhythm**, **over-correction**, **constructions**, **figures**.

Scoring is `100 − density×22 − rhythmPenalty − constructionPenalty`, clamped. The
coefficients are heuristic and have never been calibrated against human judgement of
revision usefulness. **This is a known weakness** — see §10.

---

## 2. The measurement discipline

**False positives are fatal and recall is not.** A tool that flags good writing is worse
than no tool, because the writer stops trusting the one flag that mattered.

Standing numbers against the human reference set (16 passages, 5,231 words):

- **1.34 lexical flags per 1,000 words**
- **0 of 16** passages fail the default budget
- 1 construction flag, on Taylor's *Principles of Scientific Management* (1911), which is
  genuinely institutional prose and arguably a true positive

---

## 3. Corpus manifest

Two sets. **Both are now frozen as discovery data and must not be tuned against again.**
Everything reported in §7 was found on these; none of it has been validated on held-out data.

### Machine discovery set — 20 passages, 8,297 words

- **Provenance:** one model family, generated 2026-07 specifically for this work.
- **Method:** agents asked to write **naturally and well** across five registers. No
  instruction to imitate slop (see the trap in §5).
- **Genres:** explanation ×4 (technology, food science, personal finance, history);
  argument/opinion ×4; workplace prose ×4 (status update, launch announcement, performance
  review, vendor rejection); personal ×4 (best-man speech, condolence letter, personal
  essay, cover letter); creative ×4 (short-story opening, descriptive passage, dramatic
  scene, travel writing).
- **Not varied:** model family, temperature, prompt phrasing.

### Human reference set — 16 passages, 5,231 words

- **Provenance:** published, human-authored, retrieved rather than recalled.
- **Era:** 13 of 16 are **pre-1950** (1817–1907 heavily represented). Three are modern
  (ProPublica 2016, The Conversation, an NTSB accident report).
- **Genres:** explanation ×3, literary fiction ×3, argument/opinion ×2, personal/memoir ×2,
  long-form journalism ×3, workplace/technical ×2, private letter ×1.

### What is wrong with both

- `n` is far too small to establish rates.
- One model family only.
- The human set is **era-confounded**: 1900s prose against 2026 prose, so any sentence-length
  or punctuation finding is partly measuring a century, not a generator.
- Genre, topic, audience and editorial history are unmatched between the sets.
- Several candidate signals were tested at once, which raises the chance of finding an
  apparently strong separator by accident.

---

## 4. Job A: why it failed

The 233 rules encode the vocabulary of **2023-era chatbot prose**: *delve, tapestry,
vibrant, hidden gem, a testament to*. Models have largely moved past all of it.

| | machine | human |
|---|---|---|
| lexical flags / 1,000 words | **0.73** | **1.34** |
| passages failing | 0 / 20 | 0 / 16 |

The library detects a dialect that is disappearing on its own. The central finding of this
document is **temporal**: surface-vocabulary detection has a half-life.

---

## 5. A corpus-construction trap worth knowing about

An earlier attempt built its machine set by asking agents to *"write the way this genre is
actually written."* They produced **competent** examples of each genre, which are not slop
and cannot be distinguished from good human writing. Scoring them 100 was correct behaviour
that looked like failure, and it cost a day.

The fix: ask agents to write **naturally and well**. Their unremarkable default output *is*
the specimen. Any attempt to "write like an AI" contaminates the sample.

---

## 6. Signals deleted after measurement

**The most valuable section for anyone continuing this work.** Each looked reasonable.

| signal | why it died |
|---|---|
| `isocolon` | Fired **63 times** on our own 2,400-word docs page, quoting splices that ran across full stops. Clause-length proximity is a coincidence, not a figure. |
| Process nouns (*framework, approach, solution, space, landscape, journey*) | A design essay about solutions and frameworks scored **97/1k**; a corporate report **77**. No separation. Only *synergy, enablement, workstream, touchpoint, learnings, actionable* survived. |
| `bridging` | Fired on **22 of 36** professional passages vs 7 machine. It was measuring "this prose changes subject", which is what good writing does. |
| `burstiness`, `paras`, `abstraction`, `openers`, `monotone` | All fired **only** on human prose in a labelled test. Retuned hard or retired. |
| Em-dash rate | Human **8.45/1k**, machine **6.84/1k**. Backwards. The popular "em-dashes mean AI" rule is wrong in this data. |
| Very-short-sentence rate | 22.9% vs 8.7% looks strong, but London, Woolf, Cather and Bly all sit inside the machine range. **Fatal standalone.** |

### Two kept, but reclassified

Nominalisation and the agentless passive are **higher in human professional prose**:

| | machine | human |
|---|---|---|
| nominalisation / 1k | 17.0 | **22.3** |
| agentless passive / 1k | 2.4 | **7.1** |

Kept **only as job-B detectors of institutional prose**, never as provenance signals, with
thresholds set from the distribution (30/1k — above the human maxima of 25 and 23, below the
policy-memo values of 44 and 71). At the first guessed thresholds (14 and 22) they fired on
Orwell's *Politics and the English Language* — an essay about this exact disease, caught for
quoting its own specimens — and on ProPublica, where *"were arrested"* is simply how
journalism reports a record.

---

## 7. The antithetical family, measured component by component

The review's correction: an aggregate can look powerful while one item does all the work.
It was right.

| component | machine | human | ratio | shipped |
|---|---|---|---|---|
| `rather than` | 11/20 docs, 1.81/1k | 1/16, 0.19/1k | **9.5×** | yes |
| negated copular (*the X is not…*) | 9/20, 1.57/1k | 1/16, 0.19/1k | **8.2×** | yes |
| **`not X but Y`** | **0/20** | **2/16** | **inverted** | no |
| `instead of` | 3 hits total | 0 | — | no |
| `less X than Y` | 1 hit | 0 | — | no |
| `not merely X; Y` | 0 | 0 | — | no |
| corrective reframe | 0 | 0 | — | no |

**Two of seven carry it.** And `not X but Y` — a construction this project had already
built an entire habit family around — fires *more* on professional human prose. It is a
habit worth showing a writer. It was never a tell.

Shipped as a **job-B diagnostic**: *"this passage repeatedly defines ideas by rejecting an
alternative."* Fires on **8/20 machine, 0/16 human**. No provenance language anywhere near it.

**Not built, and probably the better question:** does the Y clause carry information, or
only restate X with more emphasis? That distinction would be more useful for editing than
the surface shape.

---

## 8. Job B: constructions that do work

Aimed at **institutional** prose. 51 words of policy memo used to score 100 with zero
findings, because not one word in it is on any list — every clause is a shape.

Nominalisation · agentless passive · organisation-speak (narrow list) · paragraph evenness.

| | before | after |
|---|---|---|
| policy memo | 100 | **72** |
| dense grant narrative | 100 | **72** |
| consultant deck | 100 | **0** |
| journalism, literary, academic, design essay, recipe | 100 | **100** |

Note: `ne` is not a participle ending. It caught *is routine*, *is genuine*, *is machine*.

---

## 9. Job C: `corpus()`, and exactly what it can claim

Reading one document mostly says nothing, and that is correct: a habit looks like a
deliberate choice every time you meet it. Across a body of work the repetition becomes
visible.

```js
Tone.corpus([
  { text: '…', label: 'The Four-Way Stop That Runs This City', byline: 'Dean Baker-Hall' },
  { text: '…', label: 'Twelve to a Corner',                    byline: 'Thurston Wells' },
  …
])
```

Run on a real 14-story newspaper archive (7,722 words), every story scored 87–100 and said
nothing useful. Across all fourteen:

```
14 pieces, 7,722 words, 1.8 tells per 1,000

what they keep doing
   5  The seesaw — not just X, but Y          in 4 pieces
   4  Talking to the reader, not the subject  in 4 pieces

shared_stylistic_influence
   The seesaw          bylines: Thurston Wells, Kayla Reid
   Talking to the reader bylines: Thurston Wells, Kayla Reid
   means: "shared stylistic influence — an editor, a style guide, a genre,
           a template, a shared source, or a shared prompt.
           The prose cannot say which."

writers
   Thurston Wells   8 pieces  16.1 avg sentence  23.1 contractions/1k  15.2 em-dash/1k
   Dean Baker-Hall  4 pieces  15.7              5.9                    8.9
   Kayla Reid       2 pieces  18.8             24.9                    9.5
```

That produced twelve real edits and three prompt changes, and turned up something the editor
did not know: one writer's brief specifies short sentences and she writes the longest on
the masthead.

**What a shared habit means, and does not.** At least seven things produce this evidence:
one editor touching every piece · a house style guide · a shared genre or beat · syndicated
source material · a CMS template · writers imitating one another · a shared prompt or model.
**Which of those it is cannot be read off the prose.** The field is named
`shared_stylistic_influence`, every result carries that list, and the MCP tool instructs
callers not to report it as authorship.

**The experiment that would make this rigorous** — four known conditions, otherwise matched:
different writers with no shared editor · different writers, one shared editor · different
writers, one shared style guide · different writers, one shared model prompt. That would
reveal whether `corpus()` detects authorship, editing, institutional voice, or prompt
influence. All four are useful; they need different claims.

---

## 10. Known weaknesses in the product itself

**The single score conflates unlike things.** `100 − density×22 − rhythmPenalty −
constructionPenalty` mixes lexical cliché, institutional opacity, rhetorical habit and
rhythmic uniformity. A 72 does not tell the writer *what kind* of 72 it is. It should be a
profile — lexical residue, institutional density, structural repetition, rhythm uniformity,
corpus recurrence, evidence confidence — with the number subordinated or dropped. If it
stays, it should be calibrated against human judgement of **revision usefulness**, not
against provenance. **Not yet built.**

**There are no confidence tiers.** Findings should be visibly graded: *observed* (the
construction is literally present) · *repeated* (unusually often in this passage) · *corpus
habit* (recurs across pieces) · *comparative anomaly* (differs from a matched baseline) ·
and provenance speculation, which should never appear as a verdict. **Not yet built.**

**Nothing has been evaluated for whether it improves prose.** The meaningful metric is not
"did Tone identify the generator" but: did writers accept the recommendation, did independent
editors prefer the revision, did meaning and voice survive, did anyone stop trusting the tool
after a false positive. A blinded A/B of original vs Tone-assisted revisions would validate
this more directly than expanding the corpus. **Not yet run.**

---

## 11. Where help would matter most

1. **A matched-pair evaluation set.** Take a real human passage; give several models its
   title, assignment, audience, facts and approximate length but not its wording; generate
   at several temperatures. Then add the cases the product actually meets: machine draft
   human-edited, human draft machine-edited, alternating edits. **The mixed-provenance cases
   are probably more important than pure machine passages now.**
2. **Longitudinal decay.** Keep corpora by model generation date and run every detector
   release against all vintages. Report each signal's performance on contemporary output,
   six months later, on unseen model families, after light editing, and after paraphrase.
   Published work supports the pessimism here: large accuracy losses under paraphrase are
   well documented ([NAACL 2025](https://aclanthology.org/2025.naacl-srw.46/),
   [TempParaphraser](https://aclanthology.org/2025.emnlp-main.1607/)).
3. **Genre-specific baselines** instead of one universal human baseline. §6 and §8 show why:
   nominalisation, passives, parentheses and sentence length all move radically by genre.
4. **Within-author change detection.** A better-posed question than "is this AI": *is this
   passage inconsistent with this writer's established work?* It cannot say why the style
   changed, but it can say where review is warranted.
5. **Prompt fingerprinting.** Hold the subject constant and vary only the system prompt.
   A habit family that survives topic, model, sampling and light editing may be more stable
   and more useful than a generic machine fingerprint.
6. **Telling us what we have already got wrong.** Six signals in §6 passed a plausibility
   check and failed measurement. There are probably more.

**On provenance generally:** where it genuinely matters, signed generation metadata,
document history, revision logs and watermarks are structurally stronger than inference from
finished prose. Tone can offer forensic clues when provenance is missing; it should not
compete with direct evidence.


---

## 11a. An outside detector package, and the artifact it exposed

A third party built a standalone detection package against this document: 61 structural,
stylometric, punctuation, entropy, repetition and compression features; an interpretable
logistic and tree-ensemble runtime; hard evidence floors and abstention; and a deep path
that runs a **local** causal language model in a browser worker for token probability,
surprisal and rank. No dependencies. 11 deterministic tests, all passing.

Two things about it are worth copying. Its privacy claim survives inspection rather than
assertion — every fetch is same-origin and loads model assets only, cross-origin runtimes
are actively refused, `allowRemoteModels` is set false, and there is no request body
anywhere: the passage is never transmitted. And its coefficients are marked
**discovery-only** with the probability model deliberately **unfitted**, on the stated
grounds that inventing them without scored passages would make the software look more
finished while making its evidence worse.

### What happened when we ran it on our corpora

**14 of 61 features separated the two sets at ≥80% accuracy.** Six of the top eight were
paragraph-shape features: `paragraph.std`, `paragraph.cv`, `paragraph.evenness`,
`paragraph.entropy`, `meta.paragraphs`, `paragraph.mean` (human 277.8 words per paragraph
against machine 67.3, Cohen's d = −2.78).

It is an artifact of collection, not of writing:

```
paragraphs per passage, as collected
  human   : 1, 1, 10, 1, 1, 3, 1, 7, 1, 1, 1, 1, 1, 1, 1, 1
  machine : 5, 5, 6, 5, 5, 6, 6, 5, 6, 6, 6, 9, 9, 8, 7, 9, 5, 6, 23, 5

  single-block passages:  human 13/16   ·   machine 0/20
```

The human reference set was **retrieved as quoted excerpts**, which arrive as one block.
The machine discovery set was **generated as formatted prose**. Every paragraph feature
was measuring the difference between a web quotation and a chat response. `meta.words` and
`meta.sentences` separating at 92% is the same story: we asked models for 250–500 words and
took human excerpts of whatever length.

Stripping paragraph structure entirely and truncating every passage to 240 words:

| | features separating ≥80% |
|---|---|
| as collected | **14 of 61** |
| confounds controlled | **6 of 51** |

What survives, and it is a much more modest list:

| feature | direction | best split |
|---|---|---|
| `punctuation.commasPerSentence` | machine uses **fewer** | 86% |
| `sentence.q10` | machine's short sentences are shorter | 83% |
| `word.bigramRepeat` | machine repeats word-pairs **less** | 83% |
| `punctuation.emDashesPer1k` | machine uses **more** | 83% |
| `word.entropy`, `word.lateNovelty` | machine higher | 81% |

**One of these contradicts §6.** Our earlier measurement put em-dashes higher in human prose
(8.45 vs 6.84 per 1,000); this one puts them higher in machine prose after normalisation.
We do not know which is right. Both are computed on 36 documents, and one of them is wrong.

### What this cost, and what it is worth

Had we fitted a model on the uncontrolled features we would have produced something with
excellent apparent accuracy that was detecting **how we collected the data**. It would have
validated cleanly against our own corpus and failed instantly in the world. This is the
strongest possible argument for the package's decision to ship unfitted, and for the
matched-pair design in §11.

The era confound remains untouched by any of this: 13 of 16 human passages are pre-1950,
and comma density and sentence length both moved considerably across that century. Several
of the six survivors may be measuring 1900 rather than a generator.

---

## 11b. The matched corpus, and the null that decided it

The review's central recommendation was a matched corpus, so we built one: twelve briefs,
each answered by a model and by published human prose written to the same question — a
Guardian long read on plastics recycling, the National Tidal and Sea Level Facility on
tides, real op-eds on libraries and school start times. Ten pairs survived canonicalisation
(paragraphs re-broken at a fixed interval, length trimmed at sentence boundaries, typography
and whitespace normalised). Then we ran the audit gate.

**It refused.** Five collection probes still predicted the label at 65–80%.

### Auditing the auditor

Before accepting that, we asked what the gate does with a corpus that certainly contains
nothing. On matched pairs that null is exact and enumerable rather than sampled: the texts,
the pairing and the class sizes are all fixed, and the only free choice is which member of
each pair is called machine. Ten pairs is 2^10 = 1024 labellings, of which the truth is one.
Every one was run through the real gate.

| statistic | under 1024 meaningless labellings |
|---|---|
| `collection.words` clears the 0.60 floor | **99.2%** of the time |
| `collection.characters` clears it | 98.4% |
| best single feature, held-out | median **0.800**, and **1.000 in 14.5%** |
| ensemble beats its own best member | **0.0%** |
| **gate refuses the corpus** | **100.0%** |

The gate refused everything, so its refusal of our corpus said nothing about our corpus. Its
thresholds — 0.60 for a probe, 0.80 for a dominant feature — were guesses about a noise floor
that moves with corpus size and feature count. At twenty rows and sixty features, "a feature
separates the classes at 92%" is a statement about having measured sixty features on twenty
rows. The single 0.917 signal we had flagged as dominant is beaten by 22.7% of random
labellings.

### What replaced the constants

The gate now permutes the labels and recomputes each statistic, giving the distribution of
that statistic when there is nothing to find, on that corpus with those features. Three
rules follow:

- **Leakage** is family-wise corrected by a **permutation max-statistic** across the ten
  probes: for each labelling take the largest separation any probe reaches, and convict a
  probe only when it beats that distribution. Holm was here first and was wrong twice over —
  it treats ten probes as ten independent chances when several are nearly the same
  measurement (word count and character count move together, as do paragraphs and line
  breaks), and it demands alpha/10 = 0.005, which small nulls cannot produce at all.
- **Dominance** is judged against the distribution of the **maximum** separation across
  features, because we did not test one feature, we tested all of them and looked at the
  largest. On this corpus that raises the bar from 0.80 to 0.90.
- **"Beat your own best member"** stands whenever the best member genuinely beats chance.
  It is set aside only when that test is degenerate — when the best of sixty features is
  itself a fluke, so the ensemble is being asked to beat noise.

### The exact-test arithmetic, corrected

Our first version computed p as (k+1)/(B+1) everywhere. That add-one denominator belongs to
a **sampled** null, where the observed labelling is not in the sample and is added by hand to
keep p away from an impossible zero. An **exhaustive** enumeration already contains the
observed labelling — it is one of the 2^pairs sign assignments — so adding it again reports a
denominator the test does not have. The exact denominator is 2^pairs.

There is a second correction on top of it. `bestSplit` and the fitted-polarity accuracies are
**invariant under reversing every label**, so each mask and its complement always agree, and
the smallest attainable one-sided p is **2/2^pairs**, not 1/2^pairs and not 1/(B+1). The code
now measures that invariance rather than assuming it.

This is not bookkeeping; it decides what is testable:

| pairs | labellings | smallest attainable p | Holm needs 0.005 | max-statistic needs 0.05 |
|---|---|---|---|---|
| 5 | 32 | 0.0625 | unreachable | **unreachable** |
| 6 | 64 | **0.03125** | unreachable | **reachable** |
| 10 | 1024 | 0.00195 | reachable | reachable |

The rule that the null may only fail to convict when it *could* have convicted still stands —
it was found by a failing test, not by reasoning, when a perfectly-separating injected
artifact walked through a six-pair corpus untouched. But under the max-statistic it stops
being load-bearing. At six pairs the same artifact is now convicted on its own merits at
p = 0.031, with no fallback; the backstop fires only at five pairs, the smallest auditable
corpus, where 2/32 = 0.0625 sits just above alpha. An exception that fired everywhere became
an edge case, which is the right amount of work for a rule of its kind to do.

### A word about "exonerated"

An earlier draft of this section said the collection probes had been *exonerated*. That is
too strong and the document no longer says it. Failing to reject means **not distinguishable
from the permutation null** — the test could not tell either way — and at ten pairs it very
often cannot. It is not evidence that a probe is harmless. The gate's own output was changed
to match: probes that survive now read `not distinguishable`, and the console says so in as
many words.

The null is also **exact only under within-pair exchangeability** — the assumption that,
if there were nothing to find, the two members of a pair could be swapped without changing
the distribution of the statistic. Canonicalisation is what buys that assumption for
paragraphing, length and typography. Anything it does not equalise — topic handling, the
presence of quoted sources, an era gap between a published human passage and a 2026 model —
is a way for exchangeability to fail, and with it the exactness.

### The gate audited by 28 adversaries, and what they found

The corrections above were checked by an adversarial pass: one auditor per correctness
property, every finding then handed to an independent agent instructed to **refute** it and to
default to dismissal unless it could confirm from source. Thirteen findings survived. One was
blocking, and it was in the part of the gate we had not touched.

**The dominance route granted permission to noise.** The bar was
`quantileOf(nullMax, 0.95)` — `sorted[floor(0.95n)]`, which is a 95th percentile only for a
continuous distribution. `bestSplit` on a matched corpus lives on the lattice k/(2·pairs), so
the null is a handful of large atoms and the value at that index carried a **measured upper
tail of up to 0.27**. The gate computed the exact tail one line away, printed it, and did not
use it. A reviewer reproduced a corpus of pure noise — both members of every pair from the
same generator — printing `MODEL EXPORT: PERMITTED` off a feature its own null said chance
produces 22% of the time. A rounding mismatch compounded it: separations were rounded to 3dp
at birth and compared with `>=` against a full-precision atom, so ties resolved arbitrarily in
both directions.

The fix is to decide with the number already being computed. Dominance is now
`separation ≥ minimum AND permutation p ≤ alpha`, evaluated on the unrounded value; the bar
survives only as a description — *the smallest separation this corpus can attain whose tail is
genuinely at or below alpha*.

| | before | after |
|---|---|---|
| pure-noise corpora permitted | 1 of 36, 4 with a "dominant" feature | **1 of 24 (4.2%), 0 via dominance** |
| bar applied, ten pairs | 0.90 (true tail 0.22 in the reviewer's corpus) | **0.95, tail ≤ 0.05** |
| six pairs | a bar was reported | **nothing is certifiable at that size, and it says so** |

Two further changes came out of the same pass. The gate now **measures its own
false-permission rate** — every labelling put through the whole decision, both routes together
— and *refuses* when that rate exceeds alpha, which answers the joint-error-control question
by measurement rather than by a union bound. On the ten-pair corpus it is 6/256 = 0.023, split
across both routes. And the canonicaliser now **fails loudly**: it previously dropped every
passage and exited 0 when the corpus minimum fell below `--min`, so a caller reading the exit
code saw success and the audit then failed on a file that did not exist.

The unglamorous lesson is that the instrument built to catch a corpus fooling itself was, in
its primary route, doing the thing it existed to prevent — and it took twenty-eight adversaries
to find it, after the statistics had already been corrected twice.

### The verdict, and why we still did not fit

With a calibrated gate the ten-pair corpus reads:

| | |
|---|---|
| collection probes convicted | **`collection.characters`**, family-wise p = 0.031 |
| the other four probes | not distinguishable from the null (p(fw) 0.18 to 0.71) |
| smallest certifiable separation | **0.95** — nothing in the corpus reaches it |
| individually dominant signals | none |
| ensemble, held-out balanced accuracy | **0.700**, null median 0.500, **p = 0.079** |
| gate's own false-permission rate | 6/256 = 0.023, within alpha |
| verdict | **REFUSED** |

The first row is new, and it arrived with the max-statistic. Under Holm, `collection.characters`
needed p ≤ 0.005 and was cleared at 0.031; under a correction that measures how correlated the
ten probes actually are, the same number convicts. So the corpus does still leak: after
equalising paragraphs, words-at-sentence-boundaries and typography, **character count survives
family-wise correction**. Holm did not miss it by being conservative in a harmless direction —
it cleared a real artifact, which is the failure that matters.

This refusal, unlike the first one, carries information: one genuine collection artifact
identified, and a distributed signal that is suggestive but does not reach significance.

Then the decisive result. An earlier canonicaliser matched each pair to its own shorter
member instead of to a global ceiling. Same texts, same features, same gate:

| canonicaliser | length gap | ensemble | p | verdict |
|---|---|---|---|---|
| global ceiling | 4 words | 0.700 | 0.079 | REFUSED |
| per-pair matching | 11 words | 0.750 | **0.019** | **PERMITTED** |

Two defensible preprocessing choices, opposite verdicts, and the variant that permits is the
one that is **worse by its own stated objective** — it overcorrects, trimming the longer
member below the shorter and widening the gap it exists to close. A permutation null prices
feature selection; it cannot price the analyst's freedom to adjust preprocessing after
seeing the outcome. Choosing the passing branch here would be the garden of forking paths
with the fork visible in the diff.

**No model was fitted.** The finding is that ten matched pairs cannot settle this question,
and that a gate calibrated to its own noise floor is worth more than the model it withheld.
The honest next step is a larger matched corpus with the pipeline frozen in advance —
canonicaliser, feature list and alpha all fixed before the pairs are collected.

---

## 12. Signal lifecycle

| signal | proposed | discovery result | validated | status |
|---|---|---|---|---|
| 233 lexical rules | v1 | 0.73/1k machine vs 1.34/1k human | no | **shipped, ineffective for job A** |
| `bridging` | v1 | 22/36 human, 7 machine | — | **retired** |
| `isocolon` | v2 | 63 hits on own docs page | — | **deleted** |
| process nouns (broad) | v3 | human 97/1k > corporate 77/1k | — | **deleted; narrow list kept** |
| em-dash rate | v1 | human 8.45 > machine 6.84 | — | **never shipped** |
| nominalisation | v3 | human 22.3 > machine 17.0 | — | **reclassified to job B, threshold 30/1k** |
| agentless passive | v3 | human 7.1 > machine 2.4 | — | **reclassified to job B, threshold 30/1k** |
| `rather than` | v4 | 9.5×, 11/20 vs 1/16 | **no** | **shipped as job-B diagnostic** |
| negated copular | v4 | 8.2×, 9/20 vs 1/16 | **no** | **shipped as job-B diagnostic** |
| `not X but Y` | v4 | 0/20 machine, 2/16 human | — | **rejected as a tell; kept as a habit** |
| paragraph topic-stub | v4 | machine 0.78, human 1.10–1.20 | **no, n_human=3** | **not built** |
| trailing `, because` | v4 | 8.4× | no | not built |
| parenthesis avoidance | v4 | 0/20 machine, 6/16 human | no | not built — exculpatory only |
| approximator + numeral | v4 | 5.1×, needs genre guard | no | not built |
| `corpus()` | v4 | found a cross-byline habit in a real archive | partially — one archive | **shipped** |
| 61 structural features (external package) | v5 | 14/61 at ≥80% — **6/51 once paragraph and length confounds are removed** | no | **not fitted, deliberately** |
| 61 features on 10 MATCHED pairs | v6 | ensemble 0.700 held-out, p = 0.080 against an exhaustive 1024-labelling null | **no** | **not fitted; verdict flips on a preprocessing choice** |
| the audit gate's own thresholds | v6 | refused **100%** of meaningless labellings at n=10 pairs | — | **replaced by a permutation null** |

---

## 13. Limitations

> - **Small samples.** 20 machine and 16 human passages. No reported rate here is robust.
> - **One model family**, one generation date, one prompt style, unvaried temperature.
> - **Era confound.** 13 of 16 human passages are pre-1950.
> - **Genre and topic unmatched** between the sets.
> - **Multiple testing.** Several candidates were evaluated at once; an apparently strong
>   separator may be chance.
> - **Thresholds are fitted to this data** and have not been validated on held-out samples.
> - **"Zero overlap" and "0.00 median" are misleading at this n.** The human median for the
>   antithetical family is 0.00, yet 5 of 16 human passages still contain the construction.
> - **Collection artifacts outrank provenance.** Paragraph structure alone separated the
>   two sets at 94%, purely because human passages were retrieved as quotations and machine
>   passages were generated as formatted prose. Any corpus assembled this way will produce
>   spectacular and meaningless results.
> - **No paraphrase evaluation.** Every signal here can be edited away by a writer who knows
>   about it. Whether a durable signal exists at all in short passages is open, and the
>   honest answer may be no.
> - **Failing to reject is not exoneration.** Where this document says a probe is *not
>   distinguishable from the null*, it means the test could not tell — at ten pairs it very
>   often cannot — and not that the probe is harmless.
> - **The null is exact only under within-pair exchangeability.** Canonicalisation buys that
>   for paragraphing, length and typography. Topic handling, quoted sources and the era gap
>   are not equalised, and each is a way for the assumption to fail.
> - **Fixed thresholds are guesses about a moving noise floor.** Ours refused every corpus at
>   ten pairs, including corpora containing nothing to find. Any separation figure quoted
>   without its permutation null — including several in earlier sections of this document —
>   should be read as a maximum over however many features were measured.
> - **A null cannot price the analyst.** Permutation controls feature selection, not the
>   freedom to adjust preprocessing after seeing a verdict. Two defensible canonicalisers
>   gave p = 0.080 and p = 0.0195 on the same ten pairs (§11b). Only a pipeline frozen before
>   collection removes this, and ours was not.
> - **Ten matched pairs is too few**, and the ceiling is structural: with 20 held-out
>   decisions the ensemble statistic can only take values 0.05 apart.

---

## 14. Reproducing

```bash
curl -O https://write.work/js/tone-lint.js
node -e "const T=require('./tone-lint.js'); console.log(T.analyze('your text').stats)"
```

Tests: `bash test/run.sh` — 120 + 38 + 31, checks exit codes and prints ALL GREEN or fails
loudly. The human-prose block in the tone suite is the one that matters: those must all come
back clean.

The corpus audit and its permutation null live in the detector package (22 tests,
`node test/detect.test.js`):

```bash
node tools/canonicalise.cjs pairs.jsonl pairs-canonical.jsonl
node tools/audit-corpus.cjs pairs-canonical.jsonl audit.json
```

The audit prints a NOISE FLOOR block: how well the best feature does on meaningless labels,
how often something separates perfectly by chance, the dominance bar it applied, and the
ensemble's p. `--alpha`, `--permutations` and `--no-null` are available; a corpus of matched
pairs is enumerated exhaustively when 2^pairs fits the permutation budget. Do not trust a
separation figure from this or any other tool that is not accompanied by its null.

**Corpus availability.** The two sets above are frozen. We intend to split them: a small
public development corpus for reproducibility, a private frozen holdout for regression
testing, and periodic public release of retired holdouts. Hiding an evaluation corpus makes
external challenge harder, which is the opposite of what this document is for.

---

## 15. What changed in version 2

An outside review corrected several things. In order of importance:

1. **Three jobs, separated** (§0). The document previously moved among machine-authorship
   detection, habit detection, and corpus analysis as though they were one problem. They
   have different evidence, different claims and different products.
2. **The antithetical family was decomposed before shipping** (§7). Measuring components
   separately showed two of seven carried the aggregate, and that `not X but Y` is inverted.
   Fusing first would have hidden both facts.
3. **`corpus()` stopped implying authorship** (§9). Shared habits are shared *influence*;
   seven ordinary things produce the same evidence.
4. **Both corpora are frozen as discovery data**, the manifest is published (§3), and the
   limitations are stated as limitations rather than scattered as caveats (§13).
5. **"Negative corpus" is gone.** It is now the *machine discovery set* and the *human
   reference set*, which cannot be misread.
6. **Signal lifecycle table added** (§12), so a reader can see what was proposed, what
   survived measurement, and what has actually been validated — which, honestly, is very
   little.

---

## 16. What changed in version 3

1. **The matched corpus was built** (§11b), as the review asked. Ten pairs: same brief,
   published human prose against model prose, canonicalised so paragraphing, length and
   typography carry no information.
2. **We audited the audit.** Enumerating all 1024 labellings of those pairs showed the gate
   refused **100% of meaningless labellings** — so its refusal of our corpus said nothing
   about our corpus, and the 0.917 "dominant signal" we had flagged is beaten by 22.7% of
   random labels.
3. **Fixed thresholds were replaced by a permutation null** — family-wise corrected by a
   **max-statistic across the collection probes** rather than by Holm, dominance judged
   against the distribution of the *maximum* across features, and the beat-your-best-member
   test set aside only when it is degenerate. Holm both wasted power on correlated probes and
   demanded a p that small nulls cannot produce.
4. **The exact-test arithmetic was corrected.** An exhaustive enumeration already contains
   the observed labelling, so the denominator is 2^pairs, not B+1; and because the statistics
   are invariant to reversing every label, the smallest attainable p is 2/2^pairs. At six
   pairs that is 0.031 — reachable for a max-statistic, never reachable for Holm's 0.005.
5. **A null may only fail to convict when it could have convicted.** Found by a failing test:
   a perfectly-separating artifact walked through a six-pair corpus untouched. Under the
   max-statistic this stops being load-bearing — the same artifact is now caught on its own
   merits, and the backstop fires only at five pairs.
6. **"Exonerated" was withdrawn.** Failing to reject means *not distinguishable from the
   null*, not that a probe is harmless, and the gate's output was reworded to match. The
   null's exactness is now qualified as holding only under within-pair exchangeability.
7. **The max-statistic immediately found what Holm had cleared**: `collection.characters`
   convicts at family-wise p = 0.031 on the ten-pair corpus. The corpus leaks on character
   count after canonicalisation.
8. **No model was fitted, and the reason is stated numerically rather than argued**: the
   verdict flips between PERMITTED and REFUSED on a defensible preprocessing choice, and the
   branch that permits is the worse one by its own objective.
9. **Two new limitations** in §13: separation figures quoted without a null are maxima over
   however many features were measured, and permutation cannot price the analyst's own
   degrees of freedom.

---

*This document describes prose analysis. It makes no claim that any of it identifies who or
what wrote a passage, and the tool deliberately returns no such verdict: good human writing
trips every one of these flags, because they are habits, and habits belong to whoever has
them.*