Metrical parsing

The theory and implementation behind Prosodic’s scansion

How Prosodic scans a line of verse: the generative-metrics theory behind it, the constraint-based model, the exhaustive vectorized implementation, and the entity-based reference implementation. Code: prosodic/parsing/.

Linguistic background

Generative metrics treats a scansion as a correspondence between two objects: the linguistic prominence of a line (stress, weight, word structure) and an abstract metrical template (alternating weak/strong positions). A line is metrical to the degree that the correspondence is good; different poets and traditions tolerate different mismatches.

  • Halle & Keyser (1971) introduced correspondence rules and the idea that lines are judged by which mismatches they allow.
  • Kiparsky (1975, 1977) located the constraints in prosodic structure: what matters is not just stress but where stresses sit inside words and phrases (e.g. the special status of lexical stress maxima).
  • Hanson & Kiparsky (1996) made the theory parametric: meters choose a position size (how many syllables a position may hold), a prominence site, and a prominence type (stress, weight, strength). Prosodic’s original parser (v1, Heuser et al.) was an implementation of this parametric theory, and its constraint inventory still descends from it.
  • Optimality Theory (Prince & Smolensky 1993) reframes such systems as sets of violable constraints: every candidate scansion is evaluated against every constraint; grammars differ in how they resolve conflicts.
  • Harmonic Grammar (Legendre, Miyata & Smolensky 1990) resolves conflicts by weighted sums of violations rather than strict ranking — prosodic’s scoring model.
  • Maximum-Entropy grammar (Goldwater & Johnson 2003; Hayes & Wilson 2008; applied to meter by Hayes, Wilson & Shisko 2012) puts weights on a probabilistic footing: candidate probability ∝ exp(−weighted violations), and weights can be learned from data. Prosodic’s MaxEntTrainer and meter.fit() implement this (see below).

The model

A parse is a segmentation of a line’s syllables into an alternating sequence of weak/strong positions. With the default parameters (max_s=2, max_w=2), a position holds one or two syllables — a two-syllable position is a resolution (Hanson & Kiparsky’s position size parameter): w ws' == "when IN the.chro NI..." etc.

Candidate generation is exhaustive: every alternating sequence of positions (each w×n or s×n, n ≤ max) whose sizes sum to the line’s syllable count. A 10-syllable line yields ~700 candidates; the counts grow roughly with the Fibonacci numbers, and MAX_SYLL_IN_PARSE_UNIT (18) caps the unit size (longer lines are split at phrase boundaries into lineparts first — see the prose-handling notes in CLAUDE.md).

Constraints (parsing/constraints.py) each map a candidate to a violation count per position/syllable. The defaults descend from Kiparsky / Hanson & Kiparsky:

constraint violated when
w_stress stressed syllable in weak position
s_unstress unstressed syllable in strong position
w_peak lexical stress maximum in weak position
unres_within resolved (2-syll) position spans a strong or heavy syllable inside a polysyllable
unres_across resolved position crosses a word boundary, unless both syllables are function-word/light
foot_size position of 3+ syllables (inviolable in practice)

Additional constraints: s_trough, clash, lapse, w_heavy, s_light, s_func, word_foot; phrasal-stress constraints (w_prom, s_demoted, and the gradient w_stress_p/s_unstress_p/w_stress_t/ s_unstress_t — see Phrasal stress). Adding a constraint is one decorated function with a vectorized lambda.

Scoring is Harmonic Grammar: a parse’s score is the weighted sum of its violations (all weights 1.0 by default; settable per constraint; learnable by MaxEnt). Lower is better; best_parse is the argmin.

Harmonic bounding (Samek-Lodovici & Prince 1999) prunes candidates that cannot win under any weighting: scansion i bounds scansion j iff i’s per-constraint violation vector is ≤ j’s elementwise with at least one strict inequality. The unbounded set is the Pareto frontier — these are the linguistically interesting readings of the line; bounded candidates are retained but flagged (visible in the web app’s Line View). A parse with zero violations bounds everything else, which licenses a shortcut: perfect lines skip the O(S²) pairwise comparison entirely. For non-perfect lines an exact elite pre-screen runs first: each candidate is tested against only the K = 16 lowest-total candidates in O(K·S), and any candidate an elite dominates is eliminated (on the sonnets, ~3 survivors of ~180 remain); the exact O(S²) pairwise kernel then runs only on the survivors. This is exact, not approximate — dominance is transitive, so any dominator of a survivor must itself survive the screen. It made CPU parsing as fast as GPU, so a GPU is no longer needed for parsing.

The vectorized implementation

The v3 parser (parsing/vectorized.py) evaluates all candidates of all lines at once in numpy, without constructing entity objects:

  1. Features from the DataFrame: TextModel holds a flat syllable-level frame (_syll_df); per-line feature arrays (stress, weight, strength, word ids, function-word, phrasal columns) are sliced from it directly.
  2. Scansion encoding: for each distinct syllable count N, the candidate set is encoded once as boolean matrices (S candidates × N syllables: strong/weak membership, position ids, position sizes) and cached.
  3. Batched constraint evaluation: lines are grouped by N; features are broadcast against the scansion matrices, and each constraint’s vectorized lambda returns an (L, S, N) int8 violation tensor — L lines × S candidates × N syllables — in one numpy expression. All constraints, including unres_within/unres_across, are evaluated in fully batched numpy — no per-line loops remain.
  4. Bounding in batch: per-constraint violation sums (L, S, C) feed a tiled pairwise domination kernel — on GPU via torch when available ((L, Ti, Tj, C) difference tensors), byte-identical on numpy — producing the (L, S) unbounded mask. Perfect-parse lines short-cut.
  5. Lazy results: violations, scansions, and masks are stored in a LazyParseList; Parse objects are built only on access (best_parse builds exactly one, via argmin — no sorting).
  6. Pronunciation ambiguity: words with multiple wordforms produce the full cartesian product of per-word choices (capped at MAX_FORM_COMBOS=4096); each combination is scored and the best variant kept.

Performance (Shakespeare’s sonnets, 2,155 lines): the parser evaluates in ~1.9 s on CPU — 38× faster than the v2 branch-and-bound parser’s ~73 s — and the full pipeline runs end-to-end in ~4.0 s (20×). With the elite pre-screen, CPU parsing is now as fast as GPU. python -m prosodic.profiling regenerates the numbers.

The entity path and the reference implementations

There are two ways a line gets parsed (see “Two Parse Paths” in CLAUDE.md), and both use the vectorized evaluator above:

  • DF path (text.parse() → parse_batch_from_df): works entirely from _syll_df; parse slots hold lightweight SyllData stand-ins. Fastest; no entity objects exist at all.
  • Entity path (parse_batch(text.lines, meter)): same tensors, but slots hold real Syllable entities with parent chains — needed when downstream code walks from a syllable back to its word (HTML rendering, the web app).

The truly non-vectorized code is the reference implementations: every constraint’s decorated function body is an entity-based, per-position formulation of the same rule (def w_stress(mpos): ...). These bodies do not run during normal parsing — only for manually constructed Parse objects — but they serve as the readable specification of each constraint, and the vectorized lambdas are checked against their semantics. If a lambda and its body ever disagree, the body is the spec.

MaxEnt weight learning

MaxEntTrainer (parsing/maxent.py) learns constraint weights from annotated scansions or a uniform target (meter.fit(text, "wswswswsws")): log-linear model over each line’s candidate set, L-BFGS-B optimization, vectorized gradients via einsum over same-length line groups (< 1 s on 2K lines). Zones split the violation tensor by syllable position before weighting ("initial", integer k, or "foot"), letting the model learn positional leniency (e.g. line-initial inversion) — the learned zone_weights transfer to parsing unseen text via zone-aware scoring in LazyParseList. Because the model operates on the same (S, N, C) violation tensors the parser already produces, no parser changes are needed for new features.

One theoretical note: overlapping constraints stack in HG/MaxEnt — a w_peak violation also violates w_stress, so its effective cost is the sum of both weights. This is how the model makes stress maxima in weak position effectively inviolable (Kiparsky) without any infinite weight.

Ternary meters

Nothing in the parser is binary-specific: with max_w=2, an anapestic foot is simply a disyllabic weak position followed by a strong one, so wws wws wws wws is already inside the candidate space of every 12-syllable line, and text.meter_type detects ternary verse from the frequency of ww positions among best parses. On Byron’s “The Destruction of Sennacherib” and Browning’s “How They Brought the Good News from Ghent to Aix” (both anapestic tetrameter; corpus files in corpora/corppoetry_en/), the default weights already scan the regular lines as anapests and classify both poems as ternary.

Because ternary verse legitimately varies line length (an iamb-initial line drops one slack syllable), meter.fit() accepts a list of target scansions — each line matches the target(s) of its syllable count:

meter.fit(byron, ["wwswwswwswws", "wswwswwswws"])

The learned weights recover Hanson & Kiparsky’s (1996) characterization of English ternary meter as strong-position-regulated: s_unstress (every beat must be stressed) and unres_within (no disyllabic position buried inside a polysyllable) get high weights, while w_stress and unres_across fall to zero — stressed monosyllables sit freely in weak positions (“the blue wave rolls nightly”), and cross-word disyllabic weak positions are the norm rather than a violation. The mirror image of iambic pentameter, where weak positions carry the regulation.

References

  • Halle, M. & S. J. Keyser (1971). English Stress: Its Form, Its Growth, and Its Role in Verse.
  • Kiparsky, P. (1975). Stress, syntax, and meter. Language 51.
  • Kiparsky, P. (1977). The rhythmic structure of English verse. LI 8.
  • Hanson, K. & P. Kiparsky (1996). A parametric theory of poetic meter. Language 72.
  • Prince, A. & P. Smolensky (1993/2004). Optimality Theory.
  • Legendre, G., Y. Miyata & P. Smolensky (1990). Harmonic Grammar.
  • Samek-Lodovici, V. & A. Prince (1999). Optima. ROA-363.
  • Goldwater, S. & M. Johnson (2003). Learning OT constraint rankings using a maximum entropy model.
  • Hayes, B. & C. Wilson (2008). A maximum entropy model of phonotactics. LI 39.
  • Hayes, B., C. Wilson & A. Shisko (2012). Maxent grammars for the metrics of Shakespeare and Milton. Language 88.