backend: decide the seam between sentences instead of gluing them

A paragraph is its sentences concatenated. Nothing was put between them, which
is right for Chinese — 「。」 already separates — and wrong for English, where
`Adaptive capacity rises.` followed by `Relocation follows.` printed as
`...rises.Relocation...`: a sentence boundary the reader cannot see. CJK output
hid it, so it would have surfaced as an English bug later rather than now.

`sentence_separator` now returns "" or a single space per seam, and the document
carries it as `SentenceRead.separator_before`. The rule is about the seam, not
the language: a space goes in unless both sides are CJK. Mixed seams take the
space. Empty sentences take none. It is derived on every read and never stored,
so it cannot drift from the text, and the client prints `separator_before +
content` and adds no spacing of its own.

No splitting was added anywhere, and none exists: a sentence is one line in the
editor and nothing parses it. The single split this project has ever performed
was the one-time move of the old abstract column, which cut after 「。」 only —
conservative on purpose, since an English abstract is better left in one row
than cut at the first `et al.`.

Five smoke checks cover the seam (Chinese, English, the assembled paragraph,
and that no separator is stored inside the content). 45 checks pass.
This commit is contained in:
2026-09-18 19:01:31 +08:00
parent 898f519773
commit e582198cc8
6 changed files with 181 additions and 13 deletions
+48
View File
@@ -169,6 +169,54 @@ written against. It is kept in step with the paper's current template and is
provenance rather than a lookup key: rendering never filters on it, which is
precisely why content survives a switch.
### A sentence is a line, not parsed text
Nothing in the writing path splits text. There is no sentence detector, no
punctuation rules and no comma handling: **one line in the editor is one
sentence**, and what the writer types is what is stored, minus runs of
whitespace (leading, trailing, and any newline pasted inside a line, which fold
to a single space so a sentence really is one line). Commas never break
anything — the only place a comma separates anything in this project is the
keyword field.
That is deliberate. A splitter that guesses wrong corrupts content, and the
guess would have to be right for every abbreviation (`et al.`, `i.e.`), every
decimal (`3.14`), every numbered list and every language the tool is used in.
The writer decides where a sentence ends; the software only decides what goes at
the seam between two of them.
Exactly one split has ever been performed, in revision `a83f5c21d7b6`, to move
the old `paper.abstract` column into the body. It cut *after* a Chinese full
stop (`。`) and nothing else — deliberately conservative: an English abstract
survives that migration as a single row rather than being cut at the first
`et al.`, and one long line is a line the writer can split by hand.
#### What goes at the seam
A paragraph is printed by concatenating its sentences in `sort` order, so the
seam between two of them has to be spelled out:
`app.crud.paper.sentence_separator` returns `""` or a single space, and the
document response carries it as `SentenceRead.separator_before`.
| Seam | Separator | Why |
|---|---|---|
| `……缺口。` + `本研究……` | `""` | Chinese needs nothing; the full stop separates |
| `First sentence.` + `Second.` | `" "` | without it the two print as `First sentence.Second.` |
| `缺口。` + `This study` | `" "` | mixed text takes the space |
| `test.` + `本研究……` | `" "` | same, in the other direction |
| anything + an empty sentence | `""` | an empty line carries citations, not text |
So the rule is about the seam rather than the language: a space goes in unless
**both** sides are CJK, where adjacency is already the convention. It is derived
on every read and never stored — a stored separator is one that can go stale
against the text it separates — and a client prints `separator_before +
content` and adds no spacing of its own.
`separator_before` was added because the alternative was already a bug: CJK
sentences printed correctly by accident (the full stop hides the missing space),
so nothing looked wrong until the first English paper, whose sentences would
have run together.
### Citations are a table, not a column
One sentence may quote several references, so