backend: add the reference library (参考文献)
One table per document the writer has read, and one for its contributors:
reference id, type, title, subtitle, title_translated,
container_title, volume, issue, pages, publisher,
publisher_place, event_*, publication_year/month/day,
accessed_date, doi, isbn, issn, url, patent_*,
standard_number, report_number, arxiv_id, abstract,
keywords, note, citation_key, language
reference_author reference_id, sort, name, family_name, given_name, role
The field set is the intersection of the formats a paper has to survive, taken
from the CSL JSON schema (103 fields, 45 types — what Zotero, Mendeley and
Pandoc exchange) against GB/T 7714-2015, BibTeX and RIS. The mapping is on
`app.models.reference` rather than in a commit message, because the next person
to add a column needs it more than this one does. Three decisions worth naming:
* **Partial dates.** Year, month and day are separate nullable integers, which
is CSL's `issued`: a book has a year, a newspaper article a day, a patent its
公告日期, and nothing invents the rest.
* **Contributors are rows, not a column.** A citation prints an ordered list
with 编 / 译 marks, and styles rewrite names (「张三」 / "Zhang, S." /
"S. Zhang"), so the printed form and its two halves are stored separately —
display reads `name`, a reformatting style reads the halves, and an
institutional author has neither.
* **`volume` / `issue` / `pages` are strings.** Journals print `12(Suppl.)`,
`S1` and `e12345`, and a citation reproduces what is printed.
The library is global: the same article is cited from several papers, so the
entry cannot belong to one of them. The citation stays on the paper side in
`paper_sentence_reference`, which now carries a real foreign key to `reference`
(`ON DELETE RESTRICT`) — the column had been a loose integer waiting for this
table. A sentence may quote several documents and a document may be quoted by
many sentences, which is why the relation is a table and not a column on
`paper_sentence`.
Refusals are raised once, in the data layer, as `UnknownReference`, and
translated where each caller speaks: `400` with the offending ids for REST (one
exception handler, no route carries the check), a readable tool error for MCP
(one wrapper in the tool registry). Neither entry point can skip it, so the
foreign key never has to be the thing that complains.
API: `/references` CRUD plus batch delete, and `/references/{id}/citations` for
the reverse lookup — "where have I used this article?" — which is what tells a
writer whether an entry is safe to delete.
Verified: 29 new checks in `scripts/smoke_references.py` (partial dates, DOI
normalisation, author order and roles, search by author/type/year/cited, the
citation round trip, 400 on a phantom id, 409 on a cited entry), the paper
suite's 45 and the MCP suite's 47 still pass. The MCP suite now cites a real
entry instead of a phantom id, which is what an agent has to do now.
This commit is contained in:
@@ -5,6 +5,6 @@ models. They own the commit: a route calls one function and gets back either a
|
||||
persisted object or ``None``.
|
||||
"""
|
||||
|
||||
from app.crud import paper, template, template_title
|
||||
from app.crud import paper, reference, template, template_title
|
||||
|
||||
__all__ = ["paper", "template", "template_title"]
|
||||
__all__ = ["paper", "reference", "template", "template_title"]
|
||||
|
||||
@@ -31,6 +31,7 @@ from sqlalchemy import func, or_, select
|
||||
from sqlalchemy import update as sa_update
|
||||
from sqlalchemy.orm import Session
|
||||
|
||||
from app.crud import reference as reference_crud
|
||||
from app.crud.filters import LIKE_ESCAPE, like_pattern
|
||||
from app.models import (
|
||||
Paper,
|
||||
@@ -528,6 +529,24 @@ def delete_many(db: Session, paper_ids: Sequence[int]) -> int:
|
||||
# --- paragraphs and sentences ------------------------------------------------
|
||||
|
||||
|
||||
def _assert_citations_known(
|
||||
db: Session, sentences: Sequence[object]
|
||||
) -> None:
|
||||
"""Check every ``reference_id`` sent with a sentence against the library.
|
||||
|
||||
One place for the check, called by each write path, so a new entry point
|
||||
cannot skip it and let the foreign key be the thing that complains.
|
||||
"""
|
||||
wanted = [
|
||||
citation.reference_id
|
||||
for sentence in sentences
|
||||
for citation in getattr(sentence, "citations", []) or []
|
||||
if citation.reference_id is not None
|
||||
]
|
||||
if wanted:
|
||||
reference_crud.assert_exist(db, wanted)
|
||||
|
||||
|
||||
def replace_paragraph(
|
||||
db: Session,
|
||||
paper: Paper,
|
||||
@@ -546,6 +565,8 @@ def replace_paragraph(
|
||||
Blank lines are kept if the client sends them: an empty sentence is a
|
||||
legitimate placeholder and renders as a blank line rather than vanishing.
|
||||
"""
|
||||
_assert_citations_known(db, payload.sentences)
|
||||
|
||||
target = payload.target_sort if payload.target_sort is not None else field_sort
|
||||
|
||||
# Clear the source paragraph first, so that a move onto itself does not
|
||||
@@ -583,6 +604,8 @@ def replace_paragraph(
|
||||
|
||||
def append_sentence(db: Session, paper: Paper, data: SentenceCreate) -> PaperSentence:
|
||||
"""Add one sentence to a paragraph, at the end unless told otherwise."""
|
||||
_assert_citations_known(db, [data])
|
||||
|
||||
sort = (
|
||||
data.sort
|
||||
if data.sort is not None
|
||||
@@ -614,6 +637,9 @@ def update_sentence(
|
||||
list is short, and replacing it wholesale is the only way to be sure that a
|
||||
deleted citation really is gone.
|
||||
"""
|
||||
if data.citations is not None:
|
||||
_assert_citations_known(db, [data])
|
||||
|
||||
if data.sort is not None:
|
||||
sentence.sort = data.sort
|
||||
if data.paper_template_filed_sort is not None:
|
||||
|
||||
@@ -0,0 +1,375 @@
|
||||
"""Data access for the reference library (参考文献).
|
||||
|
||||
The list endpoint answers the question a library is opened to answer — "what do
|
||||
I have, and what is already cited?" — in one query: each row carries its printed
|
||||
author line and how many sentences cite it, both as correlated subqueries, so a
|
||||
table row never has to load the rows behind it.
|
||||
|
||||
Writes keep the two lists (authors, citations) on their own side: authors belong
|
||||
to the reference and are replaced wholesale, citations belong to the papers that
|
||||
made them and are never touched from here.
|
||||
"""
|
||||
|
||||
from collections.abc import Sequence
|
||||
|
||||
from sqlalchemy import Select, func, or_, select
|
||||
from sqlalchemy.orm import Session
|
||||
|
||||
from app.crud.filters import LIKE_ESCAPE, like_pattern
|
||||
from app.models import (
|
||||
Paper,
|
||||
PaperSentence,
|
||||
PaperSentenceReference,
|
||||
Reference,
|
||||
ReferenceAuthor,
|
||||
TemplateTitle,
|
||||
TemplateTitlePlacement,
|
||||
)
|
||||
from app.schemas.reference import (
|
||||
ReferenceCitationRead,
|
||||
ReferenceCreate,
|
||||
ReferenceListItem,
|
||||
ReferenceRead,
|
||||
ReferenceUpdate,
|
||||
)
|
||||
|
||||
#: How many authors a table row prints before it cuts the line short.
|
||||
_LIST_AUTHOR_LIMIT = 3
|
||||
|
||||
|
||||
def _conditions(
|
||||
keyword: str | None,
|
||||
type_: str | None,
|
||||
year: int | None,
|
||||
cited_only: bool | None,
|
||||
) -> list:
|
||||
"""Translate the list filters into SQLAlchemy predicates."""
|
||||
conditions = []
|
||||
if keyword:
|
||||
pattern = like_pattern(keyword)
|
||||
# The author search is an EXISTS rather than a join: joining would
|
||||
# duplicate a reference with four authors into four rows.
|
||||
conditions.append(
|
||||
or_(
|
||||
Reference.title.like(pattern, escape=LIKE_ESCAPE),
|
||||
Reference.subtitle.like(pattern, escape=LIKE_ESCAPE),
|
||||
Reference.container_title.like(pattern, escape=LIKE_ESCAPE),
|
||||
Reference.doi.like(pattern, escape=LIKE_ESCAPE),
|
||||
Reference.keywords.like(pattern, escape=LIKE_ESCAPE),
|
||||
Reference.note.like(pattern, escape=LIKE_ESCAPE),
|
||||
select(ReferenceAuthor.id)
|
||||
.where(
|
||||
ReferenceAuthor.reference_id == Reference.id,
|
||||
ReferenceAuthor.name.like(pattern, escape=LIKE_ESCAPE),
|
||||
)
|
||||
.exists(),
|
||||
)
|
||||
)
|
||||
if type_:
|
||||
conditions.append(Reference.type == type_)
|
||||
if year is not None:
|
||||
conditions.append(Reference.publication_year == year)
|
||||
if cited_only is True:
|
||||
conditions.append(_citation_count_column() > 0)
|
||||
if cited_only is False:
|
||||
conditions.append(_citation_count_column() == 0)
|
||||
return conditions
|
||||
|
||||
|
||||
def _citation_count_column():
|
||||
"""Correlated count of the sentences that cite this reference."""
|
||||
return (
|
||||
select(func.count(PaperSentenceReference.id))
|
||||
.where(PaperSentenceReference.reference_id == Reference.id)
|
||||
.correlate(Reference)
|
||||
.scalar_subquery()
|
||||
)
|
||||
|
||||
|
||||
def _ordered(stmt: Select) -> Select:
|
||||
"""Newest publication first, then most recently edited.
|
||||
|
||||
A library is browsed by "what did I read about this", so the year leads and
|
||||
the edit time breaks ties — a reference added today to an older paper still
|
||||
surfaces if its year is the newest.
|
||||
"""
|
||||
return stmt.order_by(
|
||||
Reference.publication_year.desc(),
|
||||
Reference.updated_at.desc(),
|
||||
Reference.id.desc(),
|
||||
)
|
||||
|
||||
|
||||
def authors_line(authors: Sequence[ReferenceAuthor]) -> str:
|
||||
"""The printed author line, joined for a table row.
|
||||
|
||||
Truncated for display only — the stored list is never cut — because a
|
||||
fifteen-author row would push every other column off the screen.
|
||||
"""
|
||||
names = [author.name for author in authors]
|
||||
if len(names) > _LIST_AUTHOR_LIMIT:
|
||||
return ", ".join(names[:_LIST_AUTHOR_LIMIT]) + f" 等 {len(names)} 人"
|
||||
return ", ".join(names)
|
||||
|
||||
|
||||
def list_references(
|
||||
db: Session,
|
||||
*,
|
||||
keyword: str | None = None,
|
||||
type_: str | None = None,
|
||||
year: int | None = None,
|
||||
cited_only: bool | None = None,
|
||||
page: int = 1,
|
||||
page_size: int = 20,
|
||||
) -> tuple[list[ReferenceListItem], int]:
|
||||
"""Return one page of the library, plus the unpaged total."""
|
||||
conditions = _conditions(keyword, type_, year, cited_only)
|
||||
|
||||
total = db.scalar(select(func.count(Reference.id)).where(*conditions)) or 0
|
||||
|
||||
stmt = _ordered(
|
||||
select(Reference)
|
||||
.where(*conditions)
|
||||
.offset((page - 1) * page_size)
|
||||
.limit(page_size)
|
||||
)
|
||||
rows = list(db.scalars(stmt).all())
|
||||
|
||||
# One query for the citation counts of the whole page, rather than one per
|
||||
# row: the count is a correlated subquery in the list, but here the ids are
|
||||
# already known and a grouped count is cheaper.
|
||||
counts = citation_counts(db, [row.id for row in rows])
|
||||
|
||||
items = [
|
||||
ReferenceListItem(
|
||||
id=row.id,
|
||||
type=row.type,
|
||||
title=row.title,
|
||||
subtitle=row.subtitle,
|
||||
container_title=row.container_title,
|
||||
publication_year=row.publication_year,
|
||||
volume=row.volume,
|
||||
issue=row.issue,
|
||||
pages=row.pages,
|
||||
doi=row.doi,
|
||||
authors_line=authors_line(row.authors),
|
||||
author_count=len(row.authors),
|
||||
citation_count=counts.get(row.id, 0),
|
||||
created_at=row.created_at,
|
||||
updated_at=row.updated_at,
|
||||
)
|
||||
for row in rows
|
||||
]
|
||||
return items, total
|
||||
|
||||
|
||||
def citation_counts(db: Session, reference_ids: Sequence[int]) -> dict[int, int]:
|
||||
"""How many sentences cite each reference."""
|
||||
wanted = list({reference_id for reference_id in reference_ids})
|
||||
if not wanted:
|
||||
return {}
|
||||
stmt = (
|
||||
select(PaperSentenceReference.reference_id, func.count(PaperSentenceReference.id))
|
||||
.where(PaperSentenceReference.reference_id.in_(wanted))
|
||||
.group_by(PaperSentenceReference.reference_id)
|
||||
)
|
||||
return {reference_id: count for reference_id, count in db.execute(stmt).all()}
|
||||
|
||||
|
||||
def get(db: Session, reference_id: int) -> Reference | None:
|
||||
"""Return one reference with its authors loaded, or ``None``."""
|
||||
return db.get(Reference, reference_id)
|
||||
|
||||
|
||||
def get_many(db: Session, reference_ids: Sequence[int]) -> list[Reference]:
|
||||
"""Return every reference whose id is in ``reference_ids``."""
|
||||
if not reference_ids:
|
||||
return []
|
||||
return list(
|
||||
db.scalars(select(Reference).where(Reference.id.in_(list(reference_ids)))).all()
|
||||
)
|
||||
|
||||
|
||||
class UnknownReference(ValueError):
|
||||
"""A write named literature the library does not have.
|
||||
|
||||
A domain error rather than an HTTP one: the same refusal has to reach a
|
||||
REST client as ``400`` and a model through the MCP tools as a readable tool
|
||||
error, and the layer that knows what went wrong is this one. Each caller
|
||||
translates it into its own vocabulary.
|
||||
"""
|
||||
|
||||
def __init__(self, missing: Sequence[int]) -> None:
|
||||
self.missing = list(missing)
|
||||
joined = "、".join(str(reference_id) for reference_id in self.missing)
|
||||
super().__init__(f"参考文献不存在:{joined}")
|
||||
|
||||
|
||||
def assert_exist(db: Session, reference_ids: Sequence[int]) -> None:
|
||||
"""Refuse a write that cites literature the library does not have.
|
||||
|
||||
Called from the paper side before it stores a citation, so the failure is a
|
||||
message naming the ids rather than a foreign-key violation from the driver.
|
||||
"""
|
||||
missing = missing_ids(db, reference_ids)
|
||||
if missing:
|
||||
raise UnknownReference(missing)
|
||||
|
||||
|
||||
def missing_ids(db: Session, reference_ids: Sequence[int]) -> list[int]:
|
||||
"""Which of ``reference_ids`` are not in the library.
|
||||
|
||||
Returning the offenders rather than a boolean lets the API name them, which
|
||||
is the difference between a usable error and "invalid request". A citation
|
||||
is rejected against this list, so a sentence cannot point at literature that
|
||||
does not exist.
|
||||
"""
|
||||
wanted = set(reference_ids)
|
||||
if not wanted:
|
||||
return []
|
||||
found = set(db.scalars(select(Reference.id).where(Reference.id.in_(wanted))).all())
|
||||
return sorted(wanted - found)
|
||||
|
||||
|
||||
def read(db: Session, reference: Reference) -> ReferenceRead:
|
||||
"""Build the detail payload for one reference."""
|
||||
counts = citation_counts(db, [reference.id])
|
||||
return ReferenceRead.model_validate(
|
||||
{
|
||||
**{
|
||||
column.name: getattr(reference, column.name)
|
||||
for column in Reference.__table__.columns
|
||||
},
|
||||
"authors": reference.authors,
|
||||
"citation_count": counts.get(reference.id, 0),
|
||||
}
|
||||
)
|
||||
|
||||
|
||||
def _build_authors(
|
||||
items: Sequence,
|
||||
) -> list[ReferenceAuthor]:
|
||||
"""Materialise the client's contributor list, numbering it from 1."""
|
||||
return [
|
||||
ReferenceAuthor(
|
||||
name=item.name,
|
||||
family_name=item.family_name,
|
||||
given_name=item.given_name,
|
||||
role=item.role,
|
||||
sort=item.sort if item.sort is not None else index,
|
||||
)
|
||||
for index, item in enumerate(items, start=1)
|
||||
]
|
||||
|
||||
|
||||
def create(db: Session, data: ReferenceCreate) -> Reference:
|
||||
"""Insert a reference together with its contributors."""
|
||||
values = data.model_dump(exclude={"authors"})
|
||||
reference = Reference(**values)
|
||||
reference.authors = _build_authors(data.authors)
|
||||
db.add(reference)
|
||||
db.commit()
|
||||
db.refresh(reference)
|
||||
return reference
|
||||
|
||||
|
||||
def update(
|
||||
db: Session,
|
||||
reference: Reference,
|
||||
data: ReferenceUpdate,
|
||||
) -> Reference:
|
||||
"""Apply a partial update.
|
||||
|
||||
``exclude_unset`` is what makes PATCH semantics work: a key the client did
|
||||
not send leaves the column alone, so clearing a field is an explicit
|
||||
``null`` and omitting it is not.
|
||||
"""
|
||||
values = data.model_dump(exclude_unset=True, exclude={"authors"})
|
||||
for key, value in values.items():
|
||||
setattr(reference, key, value)
|
||||
|
||||
if data.authors is not None:
|
||||
# delete-orphan removes the dropped rows on flush; the ORM cascade is
|
||||
# what cleans them up, whether or not the database would.
|
||||
reference.authors.clear()
|
||||
db.flush()
|
||||
reference.authors.extend(_build_authors(data.authors))
|
||||
|
||||
db.commit()
|
||||
db.refresh(reference)
|
||||
return reference
|
||||
|
||||
|
||||
def delete(db: Session, reference: Reference) -> None:
|
||||
"""Delete a reference. Callers must check :func:`citation_counts` first."""
|
||||
db.delete(reference)
|
||||
db.commit()
|
||||
|
||||
|
||||
def delete_many(db: Session, reference_ids: Sequence[int]) -> int:
|
||||
"""Delete several references, returning how many actually existed."""
|
||||
references = get_many(db, reference_ids)
|
||||
for reference in references:
|
||||
db.delete(reference)
|
||||
db.commit()
|
||||
return len(references)
|
||||
|
||||
|
||||
def list_citations(
|
||||
db: Session, reference: Reference
|
||||
) -> list[ReferenceCitationRead]:
|
||||
"""Every sentence that cites this reference, with its place in the paper.
|
||||
|
||||
The paragraph heading is resolved through the paper's template, so the
|
||||
answer reads as "in 3.1 Problem Formulation of <paper>" rather than as a
|
||||
position number only the database understands.
|
||||
"""
|
||||
stmt = (
|
||||
select(PaperSentenceReference, PaperSentence, Paper)
|
||||
.join(PaperSentence, PaperSentence.id == PaperSentenceReference.sentence_id)
|
||||
.join(Paper, Paper.id == PaperSentence.paper_id)
|
||||
.where(PaperSentenceReference.reference_id == reference.id)
|
||||
.order_by(Paper.id.asc(), PaperSentence.paper_template_filed_sort, PaperSentence.sort)
|
||||
)
|
||||
|
||||
results: list[ReferenceCitationRead] = []
|
||||
for citation, sentence, paper in db.execute(stmt).all():
|
||||
results.append(
|
||||
ReferenceCitationRead(
|
||||
citation_id=citation.id,
|
||||
sentence_id=sentence.id,
|
||||
sentence_content=sentence.content,
|
||||
quote=citation.quote,
|
||||
paper_id=paper.id,
|
||||
paper_title=paper.title,
|
||||
paper_template_filed_sort=sentence.paper_template_filed_sort,
|
||||
paragraph_name=_paragraph_name(
|
||||
db, paper.template_id, sentence.paper_template_filed_sort
|
||||
),
|
||||
)
|
||||
)
|
||||
return results
|
||||
|
||||
|
||||
def _paragraph_name(db: Session, template_id: int | None, position: int) -> str | None:
|
||||
"""The heading the template puts at ``position``, or ``None``.
|
||||
|
||||
``None`` rather than a placeholder: the document renderer decides how to
|
||||
print a position the template does not define (it shows 未设定), and this
|
||||
should not invent a second answer.
|
||||
"""
|
||||
if template_id is None:
|
||||
return None
|
||||
return db.scalar(
|
||||
select(TemplateTitle.name)
|
||||
.join(
|
||||
TemplateTitlePlacement,
|
||||
TemplateTitlePlacement.title_id == TemplateTitle.id,
|
||||
)
|
||||
.where(
|
||||
TemplateTitlePlacement.template_id == template_id,
|
||||
TemplateTitlePlacement.sort == position,
|
||||
)
|
||||
.order_by(TemplateTitlePlacement.sort.asc(), TemplateTitlePlacement.id.asc())
|
||||
.limit(1)
|
||||
)
|
||||
Reference in New Issue
Block a user