backend: decide the seam between sentences instead of gluing them
A paragraph is its sentences concatenated. Nothing was put between them, which is right for Chinese — 「。」 already separates — and wrong for English, where `Adaptive capacity rises.` followed by `Relocation follows.` printed as `...rises.Relocation...`: a sentence boundary the reader cannot see. CJK output hid it, so it would have surfaced as an English bug later rather than now. `sentence_separator` now returns "" or a single space per seam, and the document carries it as `SentenceRead.separator_before`. The rule is about the seam, not the language: a space goes in unless both sides are CJK. Mixed seams take the space. Empty sentences take none. It is derived on every read and never stored, so it cannot drift from the text, and the client prints `separator_before + content` and adds no spacing of its own. No splitting was added anywhere, and none exists: a sentence is one line in the editor and nothing parses it. The single split this project has ever performed was the one-time move of the old abstract column, which cut after 「。」 only — conservative on purpose, since an English abstract is better left in one row than cut at the first `et al.`. Five smoke checks cover the seam (Chinese, English, the assembled paragraph, and that no separator is stored inside the content). 45 checks pass.
This commit is contained in:
@@ -169,6 +169,54 @@ written against. It is kept in step with the paper's current template and is
|
||||
provenance rather than a lookup key: rendering never filters on it, which is
|
||||
precisely why content survives a switch.
|
||||
|
||||
### A sentence is a line, not parsed text
|
||||
|
||||
Nothing in the writing path splits text. There is no sentence detector, no
|
||||
punctuation rules and no comma handling: **one line in the editor is one
|
||||
sentence**, and what the writer types is what is stored, minus runs of
|
||||
whitespace (leading, trailing, and any newline pasted inside a line, which fold
|
||||
to a single space so a sentence really is one line). Commas never break
|
||||
anything — the only place a comma separates anything in this project is the
|
||||
keyword field.
|
||||
|
||||
That is deliberate. A splitter that guesses wrong corrupts content, and the
|
||||
guess would have to be right for every abbreviation (`et al.`, `i.e.`), every
|
||||
decimal (`3.14`), every numbered list and every language the tool is used in.
|
||||
The writer decides where a sentence ends; the software only decides what goes at
|
||||
the seam between two of them.
|
||||
|
||||
Exactly one split has ever been performed, in revision `a83f5c21d7b6`, to move
|
||||
the old `paper.abstract` column into the body. It cut *after* a Chinese full
|
||||
stop (`。`) and nothing else — deliberately conservative: an English abstract
|
||||
survives that migration as a single row rather than being cut at the first
|
||||
`et al.`, and one long line is a line the writer can split by hand.
|
||||
|
||||
#### What goes at the seam
|
||||
|
||||
A paragraph is printed by concatenating its sentences in `sort` order, so the
|
||||
seam between two of them has to be spelled out:
|
||||
`app.crud.paper.sentence_separator` returns `""` or a single space, and the
|
||||
document response carries it as `SentenceRead.separator_before`.
|
||||
|
||||
| Seam | Separator | Why |
|
||||
|---|---|---|
|
||||
| `……缺口。` + `本研究……` | `""` | Chinese needs nothing; the full stop separates |
|
||||
| `First sentence.` + `Second.` | `" "` | without it the two print as `First sentence.Second.` |
|
||||
| `缺口。` + `This study` | `" "` | mixed text takes the space |
|
||||
| `test.` + `本研究……` | `" "` | same, in the other direction |
|
||||
| anything + an empty sentence | `""` | an empty line carries citations, not text |
|
||||
|
||||
So the rule is about the seam rather than the language: a space goes in unless
|
||||
**both** sides are CJK, where adjacency is already the convention. It is derived
|
||||
on every read and never stored — a stored separator is one that can go stale
|
||||
against the text it separates — and a client prints `separator_before +
|
||||
content` and adds no spacing of its own.
|
||||
|
||||
`separator_before` was added because the alternative was already a bug: CJK
|
||||
sentences printed correctly by accident (the full stop hides the missing space),
|
||||
so nothing looked wrong until the first English paper, whose sentences would
|
||||
have run together.
|
||||
|
||||
### Citations are a table, not a column
|
||||
|
||||
One sentence may quote several references, so
|
||||
|
||||
Reference in New Issue
Block a user