backend: add the reference library (参考文献)
One table per document the writer has read, and one for its contributors:
reference id, type, title, subtitle, title_translated,
container_title, volume, issue, pages, publisher,
publisher_place, event_*, publication_year/month/day,
accessed_date, doi, isbn, issn, url, patent_*,
standard_number, report_number, arxiv_id, abstract,
keywords, note, citation_key, language
reference_author reference_id, sort, name, family_name, given_name, role
The field set is the intersection of the formats a paper has to survive, taken
from the CSL JSON schema (103 fields, 45 types — what Zotero, Mendeley and
Pandoc exchange) against GB/T 7714-2015, BibTeX and RIS. The mapping is on
`app.models.reference` rather than in a commit message, because the next person
to add a column needs it more than this one does. Three decisions worth naming:
* **Partial dates.** Year, month and day are separate nullable integers, which
is CSL's `issued`: a book has a year, a newspaper article a day, a patent its
公告日期, and nothing invents the rest.
* **Contributors are rows, not a column.** A citation prints an ordered list
with 编 / 译 marks, and styles rewrite names (「张三」 / "Zhang, S." /
"S. Zhang"), so the printed form and its two halves are stored separately —
display reads `name`, a reformatting style reads the halves, and an
institutional author has neither.
* **`volume` / `issue` / `pages` are strings.** Journals print `12(Suppl.)`,
`S1` and `e12345`, and a citation reproduces what is printed.
The library is global: the same article is cited from several papers, so the
entry cannot belong to one of them. The citation stays on the paper side in
`paper_sentence_reference`, which now carries a real foreign key to `reference`
(`ON DELETE RESTRICT`) — the column had been a loose integer waiting for this
table. A sentence may quote several documents and a document may be quoted by
many sentences, which is why the relation is a table and not a column on
`paper_sentence`.
Refusals are raised once, in the data layer, as `UnknownReference`, and
translated where each caller speaks: `400` with the offending ids for REST (one
exception handler, no route carries the check), a readable tool error for MCP
(one wrapper in the tool registry). Neither entry point can skip it, so the
foreign key never has to be the thing that complains.
API: `/references` CRUD plus batch delete, and `/references/{id}/citations` for
the reverse lookup — "where have I used this article?" — which is what tells a
writer whether an entry is safe to delete.
Verified: 29 new checks in `scripts/smoke_references.py` (partial dates, DOI
normalisation, author order and roles, search by author/type/year/cited, the
citation round trip, 400 on a phantom id, 409 on a cited entry), the paper
suite's 45 and the MCP suite's 47 still pass. The MCP suite now cites a real
entry instead of a phantom id, which is what an agent has to do now.
This commit is contained in:
@@ -0,0 +1,242 @@
|
||||
"""The reference library (参考文献库) — the literature a paper cites.
|
||||
|
||||
One row is one document the writer has read and may cite: a journal article, a
|
||||
conference paper, a book, a patent, a standard, a dataset. It is **global**, not
|
||||
owned by a paper, because the same article is cited from several papers — and
|
||||
the citation itself (which sentence quotes which passage) belongs to the paper,
|
||||
in :class:`~app.models.paper_sentence_reference.PaperSentenceReference`.
|
||||
|
||||
Why these columns
|
||||
-----------------
|
||||
The field set is the intersection of the formats a Chinese journal paper
|
||||
actually has to survive, so that one row can be rendered as any of them without
|
||||
a second home for anything:
|
||||
|
||||
* **CSL JSON** (``csl-data.json``, 103 fields / 45 types) — what Zotero,
|
||||
Mendeley and Pandoc exchange. Our names follow it wherever a field is the
|
||||
same field: ``title``, ``container-title``, ``volume``, ``issue``, ``page``,
|
||||
``publisher``, ``publisher-place``, ``edition``, ``DOI``, ``ISBN``,
|
||||
``ISSN``, ``URL``, ``abstract``, ``keyword``, ``language``, ``note``,
|
||||
``citation-key``, ``event-title``, ``event-place``, ``issued``, ``accessed``.
|
||||
* **GB/T 7714-2015** — the Chinese national standard a domestic journal will
|
||||
demand. It is why every entry carries a ``type`` with a marker, and why
|
||||
patents need ``patent_country`` and ``patent_number``, standards need
|
||||
``standard_number``, and online resources need ``accessed_date``.
|
||||
* **BibTeX / RIS** — the export formats a reference manager is judged by.
|
||||
``citation_key`` is the BibTeX key; the rest map one-to-one with the fields
|
||||
above.
|
||||
|
||||
Contributors are **not** columns: a document has an ordered list of them, each
|
||||
with a role, which is what :class:`~app.models.reference_author.ReferenceAuthor`
|
||||
holds. Formats disagree about names anyway — 「张三」, ``Zhang San``,
|
||||
``Zhang, S.`` are the same person in three styles — so the printed form and the
|
||||
two halves are stored separately and a style picks what it needs.
|
||||
|
||||
What is deliberately missing
|
||||
----------------------------
|
||||
The 103 CSL fields include roles this project will never print (narrator,
|
||||
performer, chair, illustrator, curator…) and archival fields (call-number,
|
||||
archive-place). Adding a column for each would be a schema nobody reads. They
|
||||
can arrive when a real citation needs one.
|
||||
"""
|
||||
|
||||
from datetime import date
|
||||
from typing import TYPE_CHECKING
|
||||
|
||||
from sqlalchemy import Date, Index, Integer, SmallInteger, String, Text, text
|
||||
from sqlalchemy.orm import Mapped, mapped_column, relationship
|
||||
|
||||
from app.db.base import Base
|
||||
from app.models.mixins import TimestampMixin
|
||||
|
||||
if TYPE_CHECKING: # pragma: no cover - typing only
|
||||
from app.models.reference_author import ReferenceAuthor
|
||||
|
||||
#: Document types this library holds, with the marker each format uses for it.
|
||||
#:
|
||||
#: ====================== ================ ========= ==========================
|
||||
#: our value CSL type GB/T [X] example
|
||||
#: ====================== ================ ========= ==========================
|
||||
#: journal-article article-journal [J] 期刊论文
|
||||
#: book book [M] 专著
|
||||
#: book-chapter chapter [M] 析出文献(书里的一章)
|
||||
#: conference-paper paper-conference [C] 会议论文
|
||||
#: thesis thesis [D] 学位论文
|
||||
#: patent patent [P] 专利
|
||||
#: standard standard [S] 标准
|
||||
#: report report [R] 报告
|
||||
#: newspaper-article article-newspaper[N] 报纸文章
|
||||
#: web-page webpage [EB/OL] 网页/在线资源
|
||||
#: dataset dataset [DS] 数据集
|
||||
#: software software [CP] 计算机程序
|
||||
#: preprint article [EB/OL] 预印本(arXiv 等)
|
||||
#: other document [Z] 其他
|
||||
#: ====================== ================ ========= ==========================
|
||||
REFERENCE_TYPES: tuple[str, ...] = (
|
||||
"journal-article",
|
||||
"book",
|
||||
"book-chapter",
|
||||
"conference-paper",
|
||||
"thesis",
|
||||
"patent",
|
||||
"standard",
|
||||
"report",
|
||||
"newspaper-article",
|
||||
"web-page",
|
||||
"dataset",
|
||||
"software",
|
||||
"preprint",
|
||||
"other",
|
||||
)
|
||||
|
||||
#: The roles a contributor can have. ``author`` is the default and the one that
|
||||
#: prints in the citation; the others carry the 编 / 译 / 主编 marks that GB/T
|
||||
#: 7714 puts after a name.
|
||||
REFERENCE_AUTHOR_ROLES: tuple[str, ...] = (
|
||||
"author",
|
||||
"editor",
|
||||
"translator",
|
||||
"compiler",
|
||||
)
|
||||
|
||||
#: Languages a row is written in — used to choose which title is primary and
|
||||
#: which is the 并列题名.
|
||||
REFERENCE_LANGUAGES: tuple[str, ...] = ("zh", "en", "ja", "other")
|
||||
|
||||
|
||||
class Reference(TimestampMixin, Base):
|
||||
"""One document in the reference library."""
|
||||
|
||||
__tablename__ = "reference"
|
||||
|
||||
__table_args__ = (
|
||||
# The list endpoint filters by type and orders by year, which this
|
||||
# serves. There is deliberately no index on ``title``: the search box
|
||||
# matches with a leading wildcard, which no B-tree can use.
|
||||
Index("ix_reference_type_year", "type", "publication_year"),
|
||||
)
|
||||
|
||||
id: Mapped[int] = mapped_column(Integer, primary_key=True, autoincrement=True)
|
||||
|
||||
#: One of :data:`REFERENCE_TYPES`. Stored as the token, never the Chinese
|
||||
#: label, so the wording can change without a migration.
|
||||
type: Mapped[str] = mapped_column(
|
||||
String(32),
|
||||
nullable=False,
|
||||
default="journal-article",
|
||||
server_default=text("'journal-article'"),
|
||||
)
|
||||
|
||||
# --- the document's name ------------------------------------------------
|
||||
#: 题名, as printed.
|
||||
title: Mapped[str] = mapped_column(String(512), nullable=False)
|
||||
|
||||
#: 其他题名信息 — the part after a colon (": 基于……的实证研究").
|
||||
subtitle: Mapped[str | None] = mapped_column(String(512), nullable=True)
|
||||
|
||||
#: 并列题名 / 译名 — the title in the other language, which a bilingual
|
||||
#: manuscript prints alongside the first.
|
||||
title_translated: Mapped[str | None] = mapped_column(String(512), nullable=True)
|
||||
|
||||
#: Short form for in-text citations (CSL ``title-short``).
|
||||
title_short: Mapped[str | None] = mapped_column(String(255), nullable=True)
|
||||
|
||||
# --- where it was published --------------------------------------------
|
||||
#: 刊名 / 书名 / 论文集名 / 报纸名 — the container the entry sits in.
|
||||
container_title: Mapped[str | None] = mapped_column(String(512), nullable=True)
|
||||
|
||||
#: 刊名缩写 (CSL ``container-title-short``). Some styles print only this.
|
||||
container_title_short: Mapped[str | None] = mapped_column(
|
||||
String(255), nullable=True
|
||||
)
|
||||
|
||||
#: 丛书 / 系列名, and its number.
|
||||
series_title: Mapped[str | None] = mapped_column(String(512), nullable=True)
|
||||
series_number: Mapped[str | None] = mapped_column(String(32), nullable=True)
|
||||
|
||||
#: 版本项 — "3版", "2nd ed.".
|
||||
edition: Mapped[str | None] = mapped_column(String(64), nullable=True)
|
||||
|
||||
#: 卷 and 期. Strings rather than integers: journals print "12(Suppl.)",
|
||||
#: "S1" and "春季号", and a citation has to reproduce what is printed.
|
||||
volume: Mapped[str | None] = mapped_column(String(32), nullable=True)
|
||||
issue: Mapped[str | None] = mapped_column(String(32), nullable=True)
|
||||
|
||||
#: 页码, as printed — "1024-1032", "e12345", "12-15, 20".
|
||||
pages: Mapped[str | None] = mapped_column(String(64), nullable=True)
|
||||
|
||||
#: 文章编号 / eLocation, for journals that have no page numbers.
|
||||
article_number: Mapped[str | None] = mapped_column(String(64), nullable=True)
|
||||
|
||||
# --- publisher ----------------------------------------------------------
|
||||
#: 出版者. For a thesis this is the 保存者 (the university), for a report the
|
||||
#: issuing body, for a standard the publisher — the same role in three
|
||||
#: formats, which is why there is one column and not three.
|
||||
publisher: Mapped[str | None] = mapped_column(String(255), nullable=True)
|
||||
|
||||
#: 出版地.
|
||||
publisher_place: Mapped[str | None] = mapped_column(String(255), nullable=True)
|
||||
|
||||
# --- events (conferences) ----------------------------------------------
|
||||
#: 会议名称 / 会议地点 / 会期. GB/T prints the proceedings title, APA and
|
||||
#: IEEE print the meeting, and a conference paper usually needs both.
|
||||
event_title: Mapped[str | None] = mapped_column(String(512), nullable=True)
|
||||
event_place: Mapped[str | None] = mapped_column(String(255), nullable=True)
|
||||
#: Free text on purpose: conferences run "2024-05-01/03" and styles print
|
||||
#: the range as written.
|
||||
event_date: Mapped[str | None] = mapped_column(String(64), nullable=True)
|
||||
|
||||
# --- dates --------------------------------------------------------------
|
||||
# Partial dates, exactly as CSL models ``issued``: a book has a year, a
|
||||
# newspaper article has a day, a patent has its 公告日期.
|
||||
publication_year: Mapped[int | None] = mapped_column(SmallInteger, nullable=True)
|
||||
publication_month: Mapped[int | None] = mapped_column(SmallInteger, nullable=True)
|
||||
publication_day: Mapped[int | None] = mapped_column(SmallInteger, nullable=True)
|
||||
|
||||
#: 引用日期 — required by GB/T 7714 for anything online, because a URL's
|
||||
#: content can change after it was read.
|
||||
accessed_date: Mapped[date | None] = mapped_column(Date, nullable=True)
|
||||
|
||||
# --- identifiers --------------------------------------------------------
|
||||
#: Canonical DOI, without the ``https://doi.org/`` prefix.
|
||||
doi: Mapped[str | None] = mapped_column(String(255), nullable=True)
|
||||
isbn: Mapped[str | None] = mapped_column(String(32), nullable=True)
|
||||
issn: Mapped[str | None] = mapped_column(String(32), nullable=True)
|
||||
url: Mapped[str | None] = mapped_column(String(1024), nullable=True)
|
||||
|
||||
#: 专利国别 + 专利号 — GB/T 7714 prints both inside the [P] marker.
|
||||
patent_country: Mapped[str | None] = mapped_column(String(64), nullable=True)
|
||||
patent_number: Mapped[str | None] = mapped_column(String(64), nullable=True)
|
||||
|
||||
#: 标准编号 ("GB/T 7714-2015") and 报告编号.
|
||||
standard_number: Mapped[str | None] = mapped_column(String(64), nullable=True)
|
||||
report_number: Mapped[str | None] = mapped_column(String(64), nullable=True)
|
||||
|
||||
#: Preprint identifier ("arXiv:2405.12345"), which is how an AI paper is
|
||||
#: cited before it has a DOI.
|
||||
arxiv_id: Mapped[str | None] = mapped_column(String(64), nullable=True)
|
||||
|
||||
# --- the writer's own notes --------------------------------------------
|
||||
abstract: Mapped[str | None] = mapped_column(Text, nullable=True)
|
||||
#: 文献关键词, comma-separated like the paper's own keywords.
|
||||
keywords: Mapped[str | None] = mapped_column(String(512), nullable=True)
|
||||
#: Free text: why it was kept, what it is for, where the PDF is.
|
||||
note: Mapped[str | None] = mapped_column(Text, nullable=True)
|
||||
|
||||
# --- library metadata ---------------------------------------------------
|
||||
#: The BibTeX key (CSL ``citation-key``), for export.
|
||||
citation_key: Mapped[str | None] = mapped_column(String(64), nullable=True)
|
||||
|
||||
#: One of :data:`REFERENCE_LANGUAGES`; decides which title leads.
|
||||
language: Mapped[str | None] = mapped_column(String(16), nullable=True)
|
||||
|
||||
#: The contributors, in the order the citation prints them.
|
||||
authors: Mapped[list["ReferenceAuthor"]] = relationship(
|
||||
back_populates="reference",
|
||||
cascade="all, delete-orphan",
|
||||
order_by="ReferenceAuthor.sort, ReferenceAuthor.id",
|
||||
lazy="selectin",
|
||||
)
|
||||
|
||||
def __repr__(self) -> str: # pragma: no cover - debugging aid
|
||||
return f"<Reference id={self.id} type={self.type} title={self.title!r}>"
|
||||
Reference in New Issue
Block a user