Files
paper-doc/backend/app/crud/reference.py
T
govin d7bb777264 backend: add the reference library (参考文献)
One table per document the writer has read, and one for its contributors:

    reference          id, type, title, subtitle, title_translated,
                       container_title, volume, issue, pages, publisher,
                       publisher_place, event_*, publication_year/month/day,
                       accessed_date, doi, isbn, issn, url, patent_*,
                       standard_number, report_number, arxiv_id, abstract,
                       keywords, note, citation_key, language
    reference_author   reference_id, sort, name, family_name, given_name, role

The field set is the intersection of the formats a paper has to survive, taken
from the CSL JSON schema (103 fields, 45 types — what Zotero, Mendeley and
Pandoc exchange) against GB/T 7714-2015, BibTeX and RIS. The mapping is on
`app.models.reference` rather than in a commit message, because the next person
to add a column needs it more than this one does. Three decisions worth naming:

* **Partial dates.** Year, month and day are separate nullable integers, which
  is CSL's `issued`: a book has a year, a newspaper article a day, a patent its
  公告日期, and nothing invents the rest.
* **Contributors are rows, not a column.** A citation prints an ordered list
  with 编 / 译 marks, and styles rewrite names (「张三」 / "Zhang, S." /
  "S. Zhang"), so the printed form and its two halves are stored separately —
  display reads `name`, a reformatting style reads the halves, and an
  institutional author has neither.
* **`volume` / `issue` / `pages` are strings.** Journals print `12(Suppl.)`,
  `S1` and `e12345`, and a citation reproduces what is printed.

The library is global: the same article is cited from several papers, so the
entry cannot belong to one of them. The citation stays on the paper side in
`paper_sentence_reference`, which now carries a real foreign key to `reference`
(`ON DELETE RESTRICT`) — the column had been a loose integer waiting for this
table. A sentence may quote several documents and a document may be quoted by
many sentences, which is why the relation is a table and not a column on
`paper_sentence`.

Refusals are raised once, in the data layer, as `UnknownReference`, and
translated where each caller speaks: `400` with the offending ids for REST (one
exception handler, no route carries the check), a readable tool error for MCP
(one wrapper in the tool registry). Neither entry point can skip it, so the
foreign key never has to be the thing that complains.

API: `/references` CRUD plus batch delete, and `/references/{id}/citations` for
the reverse lookup — "where have I used this article?" — which is what tells a
writer whether an entry is safe to delete.

Verified: 29 new checks in `scripts/smoke_references.py` (partial dates, DOI
normalisation, author order and roles, search by author/type/year/cited, the
citation round trip, 400 on a phantom id, 409 on a cited entry), the paper
suite's 45 and the MCP suite's 47 still pass. The MCP suite now cites a real
entry instead of a phantom id, which is what an agent has to do now.
2026-09-22 17:13:32 +08:00

376 lines
13 KiB
Python

"""Data access for the reference library (参考文献).
The list endpoint answers the question a library is opened to answer — "what do
I have, and what is already cited?" — in one query: each row carries its printed
author line and how many sentences cite it, both as correlated subqueries, so a
table row never has to load the rows behind it.
Writes keep the two lists (authors, citations) on their own side: authors belong
to the reference and are replaced wholesale, citations belong to the papers that
made them and are never touched from here.
"""
from collections.abc import Sequence
from sqlalchemy import Select, func, or_, select
from sqlalchemy.orm import Session
from app.crud.filters import LIKE_ESCAPE, like_pattern
from app.models import (
Paper,
PaperSentence,
PaperSentenceReference,
Reference,
ReferenceAuthor,
TemplateTitle,
TemplateTitlePlacement,
)
from app.schemas.reference import (
ReferenceCitationRead,
ReferenceCreate,
ReferenceListItem,
ReferenceRead,
ReferenceUpdate,
)
#: How many authors a table row prints before it cuts the line short.
_LIST_AUTHOR_LIMIT = 3
def _conditions(
keyword: str | None,
type_: str | None,
year: int | None,
cited_only: bool | None,
) -> list:
"""Translate the list filters into SQLAlchemy predicates."""
conditions = []
if keyword:
pattern = like_pattern(keyword)
# The author search is an EXISTS rather than a join: joining would
# duplicate a reference with four authors into four rows.
conditions.append(
or_(
Reference.title.like(pattern, escape=LIKE_ESCAPE),
Reference.subtitle.like(pattern, escape=LIKE_ESCAPE),
Reference.container_title.like(pattern, escape=LIKE_ESCAPE),
Reference.doi.like(pattern, escape=LIKE_ESCAPE),
Reference.keywords.like(pattern, escape=LIKE_ESCAPE),
Reference.note.like(pattern, escape=LIKE_ESCAPE),
select(ReferenceAuthor.id)
.where(
ReferenceAuthor.reference_id == Reference.id,
ReferenceAuthor.name.like(pattern, escape=LIKE_ESCAPE),
)
.exists(),
)
)
if type_:
conditions.append(Reference.type == type_)
if year is not None:
conditions.append(Reference.publication_year == year)
if cited_only is True:
conditions.append(_citation_count_column() > 0)
if cited_only is False:
conditions.append(_citation_count_column() == 0)
return conditions
def _citation_count_column():
"""Correlated count of the sentences that cite this reference."""
return (
select(func.count(PaperSentenceReference.id))
.where(PaperSentenceReference.reference_id == Reference.id)
.correlate(Reference)
.scalar_subquery()
)
def _ordered(stmt: Select) -> Select:
"""Newest publication first, then most recently edited.
A library is browsed by "what did I read about this", so the year leads and
the edit time breaks ties — a reference added today to an older paper still
surfaces if its year is the newest.
"""
return stmt.order_by(
Reference.publication_year.desc(),
Reference.updated_at.desc(),
Reference.id.desc(),
)
def authors_line(authors: Sequence[ReferenceAuthor]) -> str:
"""The printed author line, joined for a table row.
Truncated for display only — the stored list is never cut — because a
fifteen-author row would push every other column off the screen.
"""
names = [author.name for author in authors]
if len(names) > _LIST_AUTHOR_LIMIT:
return ", ".join(names[:_LIST_AUTHOR_LIMIT]) + f" 等 {len(names)} 人"
return ", ".join(names)
def list_references(
db: Session,
*,
keyword: str | None = None,
type_: str | None = None,
year: int | None = None,
cited_only: bool | None = None,
page: int = 1,
page_size: int = 20,
) -> tuple[list[ReferenceListItem], int]:
"""Return one page of the library, plus the unpaged total."""
conditions = _conditions(keyword, type_, year, cited_only)
total = db.scalar(select(func.count(Reference.id)).where(*conditions)) or 0
stmt = _ordered(
select(Reference)
.where(*conditions)
.offset((page - 1) * page_size)
.limit(page_size)
)
rows = list(db.scalars(stmt).all())
# One query for the citation counts of the whole page, rather than one per
# row: the count is a correlated subquery in the list, but here the ids are
# already known and a grouped count is cheaper.
counts = citation_counts(db, [row.id for row in rows])
items = [
ReferenceListItem(
id=row.id,
type=row.type,
title=row.title,
subtitle=row.subtitle,
container_title=row.container_title,
publication_year=row.publication_year,
volume=row.volume,
issue=row.issue,
pages=row.pages,
doi=row.doi,
authors_line=authors_line(row.authors),
author_count=len(row.authors),
citation_count=counts.get(row.id, 0),
created_at=row.created_at,
updated_at=row.updated_at,
)
for row in rows
]
return items, total
def citation_counts(db: Session, reference_ids: Sequence[int]) -> dict[int, int]:
"""How many sentences cite each reference."""
wanted = list({reference_id for reference_id in reference_ids})
if not wanted:
return {}
stmt = (
select(PaperSentenceReference.reference_id, func.count(PaperSentenceReference.id))
.where(PaperSentenceReference.reference_id.in_(wanted))
.group_by(PaperSentenceReference.reference_id)
)
return {reference_id: count for reference_id, count in db.execute(stmt).all()}
def get(db: Session, reference_id: int) -> Reference | None:
"""Return one reference with its authors loaded, or ``None``."""
return db.get(Reference, reference_id)
def get_many(db: Session, reference_ids: Sequence[int]) -> list[Reference]:
"""Return every reference whose id is in ``reference_ids``."""
if not reference_ids:
return []
return list(
db.scalars(select(Reference).where(Reference.id.in_(list(reference_ids)))).all()
)
class UnknownReference(ValueError):
"""A write named literature the library does not have.
A domain error rather than an HTTP one: the same refusal has to reach a
REST client as ``400`` and a model through the MCP tools as a readable tool
error, and the layer that knows what went wrong is this one. Each caller
translates it into its own vocabulary.
"""
def __init__(self, missing: Sequence[int]) -> None:
self.missing = list(missing)
joined = "、".join(str(reference_id) for reference_id in self.missing)
super().__init__(f"参考文献不存在:{joined}")
def assert_exist(db: Session, reference_ids: Sequence[int]) -> None:
"""Refuse a write that cites literature the library does not have.
Called from the paper side before it stores a citation, so the failure is a
message naming the ids rather than a foreign-key violation from the driver.
"""
missing = missing_ids(db, reference_ids)
if missing:
raise UnknownReference(missing)
def missing_ids(db: Session, reference_ids: Sequence[int]) -> list[int]:
"""Which of ``reference_ids`` are not in the library.
Returning the offenders rather than a boolean lets the API name them, which
is the difference between a usable error and "invalid request". A citation
is rejected against this list, so a sentence cannot point at literature that
does not exist.
"""
wanted = set(reference_ids)
if not wanted:
return []
found = set(db.scalars(select(Reference.id).where(Reference.id.in_(wanted))).all())
return sorted(wanted - found)
def read(db: Session, reference: Reference) -> ReferenceRead:
"""Build the detail payload for one reference."""
counts = citation_counts(db, [reference.id])
return ReferenceRead.model_validate(
{
**{
column.name: getattr(reference, column.name)
for column in Reference.__table__.columns
},
"authors": reference.authors,
"citation_count": counts.get(reference.id, 0),
}
)
def _build_authors(
items: Sequence,
) -> list[ReferenceAuthor]:
"""Materialise the client's contributor list, numbering it from 1."""
return [
ReferenceAuthor(
name=item.name,
family_name=item.family_name,
given_name=item.given_name,
role=item.role,
sort=item.sort if item.sort is not None else index,
)
for index, item in enumerate(items, start=1)
]
def create(db: Session, data: ReferenceCreate) -> Reference:
"""Insert a reference together with its contributors."""
values = data.model_dump(exclude={"authors"})
reference = Reference(**values)
reference.authors = _build_authors(data.authors)
db.add(reference)
db.commit()
db.refresh(reference)
return reference
def update(
db: Session,
reference: Reference,
data: ReferenceUpdate,
) -> Reference:
"""Apply a partial update.
``exclude_unset`` is what makes PATCH semantics work: a key the client did
not send leaves the column alone, so clearing a field is an explicit
``null`` and omitting it is not.
"""
values = data.model_dump(exclude_unset=True, exclude={"authors"})
for key, value in values.items():
setattr(reference, key, value)
if data.authors is not None:
# delete-orphan removes the dropped rows on flush; the ORM cascade is
# what cleans them up, whether or not the database would.
reference.authors.clear()
db.flush()
reference.authors.extend(_build_authors(data.authors))
db.commit()
db.refresh(reference)
return reference
def delete(db: Session, reference: Reference) -> None:
"""Delete a reference. Callers must check :func:`citation_counts` first."""
db.delete(reference)
db.commit()
def delete_many(db: Session, reference_ids: Sequence[int]) -> int:
"""Delete several references, returning how many actually existed."""
references = get_many(db, reference_ids)
for reference in references:
db.delete(reference)
db.commit()
return len(references)
def list_citations(
db: Session, reference: Reference
) -> list[ReferenceCitationRead]:
"""Every sentence that cites this reference, with its place in the paper.
The paragraph heading is resolved through the paper's template, so the
answer reads as "in 3.1 Problem Formulation of <paper>" rather than as a
position number only the database understands.
"""
stmt = (
select(PaperSentenceReference, PaperSentence, Paper)
.join(PaperSentence, PaperSentence.id == PaperSentenceReference.sentence_id)
.join(Paper, Paper.id == PaperSentence.paper_id)
.where(PaperSentenceReference.reference_id == reference.id)
.order_by(Paper.id.asc(), PaperSentence.paper_template_filed_sort, PaperSentence.sort)
)
results: list[ReferenceCitationRead] = []
for citation, sentence, paper in db.execute(stmt).all():
results.append(
ReferenceCitationRead(
citation_id=citation.id,
sentence_id=sentence.id,
sentence_content=sentence.content,
quote=citation.quote,
paper_id=paper.id,
paper_title=paper.title,
paper_template_filed_sort=sentence.paper_template_filed_sort,
paragraph_name=_paragraph_name(
db, paper.template_id, sentence.paper_template_filed_sort
),
)
)
return results
def _paragraph_name(db: Session, template_id: int | None, position: int) -> str | None:
"""The heading the template puts at ``position``, or ``None``.
``None`` rather than a placeholder: the document renderer decides how to
print a position the template does not define (it shows 未设定), and this
should not invent a second answer.
"""
if template_id is None:
return None
return db.scalar(
select(TemplateTitle.name)
.join(
TemplateTitlePlacement,
TemplateTitlePlacement.title_id == TemplateTitle.id,
)
.where(
TemplateTitlePlacement.template_id == template_id,
TemplateTitlePlacement.sort == position,
)
.order_by(TemplateTitlePlacement.sort.asc(), TemplateTitlePlacement.id.asc())
.limit(1)
)