Metrical parsing

The theory and implementation behind Prosodic’s scansion

How Prosodic scans a line of verse: the generative-metrics theory behind it, the constraint-based model, the exhaustive vectorized implementation, and the entity-based reference implementation. Code: prosodic/parsing/.

Linguistic background

Generative metrics treats a scansion as a correspondence between two objects: the linguistic prominence of a line (stress, weight, word structure) and an abstract metrical template (alternating weak/strong positions). A line is metrical to the degree that the correspondence is good; different poets and traditions tolerate different mismatches.

Halle & Keyser (1971) introduced correspondence rules and the idea that lines are judged by which mismatches they allow.
Kiparsky (1975, 1977) located the constraints in prosodic structure: what matters is not just stress but where stresses sit inside words and phrases (e.g. the special status of lexical stress maxima).
Hanson & Kiparsky (1996) made the theory parametric: meters choose a position size (how many syllables a position may hold), a prominence site, and a prominence type (stress, weight, strength). Prosodic’s original parser (v1, Heuser et al.) was an implementation of this parametric theory, and its constraint inventory still descends from it.
Optimality Theory (Prince & Smolensky 1993) reframes such systems as sets of violable constraints: every candidate scansion is evaluated against every constraint; grammars differ in how they resolve conflicts.
Harmonic Grammar (Legendre, Miyata & Smolensky 1990) resolves conflicts by weighted sums of violations rather than strict ranking — prosodic’s scoring model.
Maximum-Entropy grammar (Goldwater & Johnson 2003; Hayes & Wilson 2008; applied to meter by Hayes, Wilson & Shisko 2012) puts weights on a probabilistic footing: candidate probability ∝ exp(−weighted violations), and weights can be learned from data. Prosodic’s MaxEntTrainer and meter.fit() implement this (see below).

The model

A parse is a segmentation of a line’s syllables into an alternating sequence of weak/strong positions. With the default parameters (max_s=2, max_w=2), a position holds one or two syllables — a two-syllable position is a resolution (Hanson & Kiparsky’s position size parameter): w ws' == "when IN the.chro NI..." etc.

Candidate generation is exhaustive: every alternating sequence of positions (each w×n or s×n, n ≤ max) whose sizes sum to the line’s syllable count. A 10-syllable line yields ~700 candidates; the counts grow roughly with the Fibonacci numbers, and MAX_SYLL_IN_PARSE_UNIT (18) caps the unit size (longer lines are split at phrase boundaries into lineparts first — see the prose-handling notes in CLAUDE.md).

Constraints (parsing/constraints.py) each map a candidate to a violation count per position/syllable. The defaults descend from Kiparsky / Hanson & Kiparsky:

constraint	violated when
`w_stress`	stressed syllable in weak position
`s_unstress`	unstressed syllable in strong position
`w_peak`	lexical stress maximum in weak position
`unres_within`	resolved (2-syll) position spans a strong or heavy syllable inside a polysyllable
`unres_across`	resolved position crosses a word boundary, unless both syllables are function-word/light
`foot_size`	position of 3+ syllables (inviolable in practice)

Additional constraints: s_trough, clash, lapse, w_heavy, s_light, s_func, word_foot; phrasal-stress constraints (w_prom, s_demoted, and the gradient w_stress_p/s_unstress_p/w_stress_t/ s_unstress_t — see Phrasal stress). Adding a constraint is one decorated function with a vectorized lambda.

Scoring is Harmonic Grammar: a parse’s score is the weighted sum of its violations (all weights 1.0 by default; settable per constraint; learnable by MaxEnt). Lower is better; best_parse is the minimum-score parse, with co-optimal ties broken by a deterministic pure-scansion cascade (fewest resolutions ss → period-k regularity → pseudo-feet → fewest ww → a w-onset content key) — see Foot parsing §3; a strictly unique minimum short-circuits to a plain argmin.

Harmonic bounding (Samek-Lodovici & Prince 1999) prunes candidates that cannot win under any weighting: scansion i bounds scansion j iff i’s per-constraint violation vector is ≤ j’s elementwise with at least one strict inequality. The unbounded set is the Pareto frontier — these are the linguistically interesting readings of the line; bounded candidates are retained but flagged (visible in the web app’s Line View). A parse with zero violations bounds everything else, which licenses a shortcut: perfect lines skip the O(S²) pairwise comparison entirely. For non-perfect lines an exact elite pre-screen runs first: each candidate is tested against only the K = 16 lowest-total candidates in O(K·S), and any candidate an elite dominates is eliminated (on the sonnets, ~3 survivors of ~180 remain); the exact O(S²) pairwise kernel then runs only on the survivors. This is exact, not approximate — dominance is transitive, so any dominator of a survivor must itself survive the screen. The screen itself also runs on the torch device when one is available (MPS/CUDA; ~15× over numpy — pronunciation-variant pooling inflates its input to ~15K rows per call), with the numpy screen as the CPU fallback. GPU parsing is currently the faster path again; CPU remains fully supported and byte-identical.

The vectorized implementation

The v3 parser (parsing/vectorized.py) evaluates all candidates of all lines at once in numpy, without constructing entity objects:

Features from the DataFrame: TextModel holds a flat syllable-level frame (_syll_df); per-line feature arrays (stress, weight, strength, word ids, function-word, phrasal columns) are sliced from it directly.
Scansion encoding: for each distinct syllable count N, the candidate set is encoded once as boolean matrices (S candidates × N syllables: strong/weak membership, position ids, position sizes) and cached.
Batched constraint evaluation: lines are grouped by N; features are broadcast against the scansion matrices, and each constraint’s vectorized lambda returns an (L, S, N) int8 violation tensor — L lines × S candidates × N syllables — in one numpy expression. All constraints, including unres_within/unres_across, are evaluated in fully batched numpy — no per-line loops remain.
Bounding in batch: per-constraint violation sums (L, S, C) feed a tiled pairwise domination kernel — on GPU via torch when available ((L, Ti, Tj, C) difference tensors), byte-identical on numpy — producing the (L, S) unbounded mask. Perfect-parse lines short-cut.
Lazy results: violations, scansions, and masks are stored in a LazyParseList; Parse objects are built only on access (best_parse builds exactly one — argmin when the minimum score is unique, else the deterministic tie-break cascade).
Pronunciation ambiguity: words with multiple wordforms produce the full cartesian product of per-word choices (capped at MAX_FORM_COMBOS=4096). With pool_forms (default on), the parses of all combinations are pooled and cross-bounded, so the unbounded set is the scansions optimal under any pronunciation — deduped by meter string, the v1/v2 “resolve word-forms in situ” semantics. pool_forms=False keeps only the single best-scoring combination (the interim v3 behavior). Pooling is numpy-only: each combo’s unbounded parses are cross-bounded from the raw violation arrays, so no Parse/Entity objects are built.

Performance (Shakespeare’s sonnets, 2,155 lines): the parser evaluates in ~4.1 s on GPU / ~6.4 s CPU — 18×/11× faster than the 2024 v2 parser’s ~73 s — with the full pipeline end-to-end in ~6.3 s GPU / ~8.6 s CPU. For historical reference, v1.3.8’s pruned branch-and-bound took ~36.6 s on the same machine (the oft-quoted “78 s legacy” figure was v2, which regressed 2× against v1). An earlier ~1.9 s figure predates the v1-semantics restoration (full pronunciation-variant pooling × verse elision: ~89% of sonnet lines carry multiple pronunciation combos, each evaluated and cross-bounded — purchased semantics, since recouped ~2× by two byte-identical vectorization passes). python -m prosodic.profiling regenerates the numbers.

One parse path, and the reference implementations

Since the entity batch parser was retired (see docs/methods/parse-path-unification.md), all parsing goes through the single vectorized DF path: text.parse() → parse_batch_from_df works entirely from _syll_df, and single lines / lineparts / arbitrary token lists route through parse_units_from_df, which scopes the parent frame by word_num. Parse slots hold lightweight SyllData stand-ins; no entity objects are built by parsing (the Syllable/Phoneme tree still exists and is built lazily on access — rendering bridges from a slot to its WordToken via SyllData.word_num). The one construction that still builds entity slots is the manual Parse(line, "wsws") reference constructor — a hand-built single parse from a given scansion, not a parser, and deliberately left entity-based.

The truly non-vectorized code is the reference implementations: every constraint’s decorated function body is an entity-based, per-position formulation of the same rule (def w_stress(mpos): ...). These bodies do not run during normal parsing — only for manually constructed Parse objects — but they serve as the readable specification of each constraint, and the vectorized lambdas are checked against their semantics. If a lambda and its body ever disagree, the body is the spec.

MaxEnt weight learning

MaxEntTrainer (parsing/maxent.py) learns constraint weights from annotated scansions or a uniform target (meter.fit(text, "wswswswsws")): log-linear model over each line’s candidate set, L-BFGS-B optimization, vectorized gradients via einsum over same-length line groups (< 1 s on 2K lines). Zones split the violation tensor by syllable position before weighting ("initial", integer k, or "foot"), letting the model learn positional leniency (e.g. line-initial inversion) — the learned zone_weights transfer to parsing unseen text via zone-aware scoring in LazyParseList. Because the model operates on the same (S, N, C) violation tensors the parser already produces, no parser changes are needed for new features.

One theoretical note: overlapping constraints stack in HG/MaxEnt — a w_peak violation also violates w_stress, so its effective cost is the sum of both weights. This is how the model makes stress maxima in weak position effectively inviolable (Kiparsky) without any infinite weight.

Ternary meters

Nothing in the parser is binary-specific: with max_w=2, an anapestic foot is simply a disyllabic weak position followed by a strong one, so wws wws wws wws is already inside the candidate space of every 12-syllable line, and text.meter_type detects ternary verse from the frequency of ww positions among best parses. On Byron’s “The Destruction of Sennacherib” and Browning’s “How They Brought the Good News from Ghent to Aix” (both anapestic tetrameter; corpus files in corpora/corppoetry_en/), the default weights already scan the regular lines as anapests and classify both poems as ternary.

Because ternary verse legitimately varies line length (an iamb-initial line drops one slack syllable), meter.fit() accepts a list of target scansions — each line matches the target(s) of its syllable count:

meter.fit(byron, ["wwswwswwswws", "wswwswwswws"])

The learned weights recover Hanson & Kiparsky’s (1996) characterization of English ternary meter as strong-position-regulated: s_unstress (every beat must be stressed) and unres_within (no disyllabic position buried inside a polysyllable) get high weights, while w_stress and unres_across fall to zero — stressed monosyllables sit freely in weak positions (“the blue wave rolls nightly”), and cross-word disyllabic weak positions are the norm rather than a violation. The mirror image of iambic pentameter, where weak positions carry the regulation.

The by-meter weight signature (foot-gold study)

Fitting MaxEnt separately per meter on the hand-tagged foot gold (data/tagged_samples/foot-gold.csv, 30 lines × 4 meters; scripts/maxent_by_meter.py, which also runs zone and partial-pooled variants) recovers the binary/ternary parameter empirically from the annotations alone:

constraint	iambic	trochaic	anapestic	dactylic
`w_peak`	0.76	1.53	0.00	0.00
`w_stress`	1.24	1.32	0.00	0.00
`s_unstress`	1.63	0.12	3.48	2.65
`unres_across`	2.27	2.85	0.45	0.16
`unres_within`	1.82	2.53	0.13	0.45

Binary meters weight the weak-position constraints; ternary meters zero them and put the highest weight of anyone on s_unstress. Splitting by line thirds adds a positional finding invisible in the flat fit: the strict edge flips with headedness — iambic locks its line-final beat (w_stress 0.41 → 0.95 → 1.56 across thirds) while trochaic locks its line-initial one (1.38 → 0.88 → 0.73), exact mirror gradients. (Caveat: a low weight means the gold does not discriminate from its rivals on that constraint, not that the grammar permits violations — so the unres_within ternary-strictness prediction, which the flat fit does not separate, needs the positional view rather than this table.) An executable version of this study, with figures: The grammar of each meter.

References

Halle, M. & S. J. Keyser (1971). English Stress: Its Form, Its Growth, and Its Role in Verse.
Kiparsky, P. (1975). Stress, syntax, and meter. Language 51.
Kiparsky, P. (1977). The rhythmic structure of English verse. LI 8.
Hanson, K. & P. Kiparsky (1996). A parametric theory of poetic meter. Language 72.
Prince, A. & P. Smolensky (1993/2004). Optimality Theory.
Legendre, G., Y. Miyata & P. Smolensky (1990). Harmonic Grammar.
Samek-Lodovici, V. & A. Prince (1999). Optima. ROA-363.
Goldwater, S. & M. Johnson (2003). Learning OT constraint rankings using a maximum entropy model.
Hayes, B. & C. Wilson (2008). A maximum entropy model of phonotactics. LI 39.
Hayes, B., C. Wilson & A. Shisko (2012). Maxent grammars for the metrics of Shakespeare and Milton. Language 88.