d7bb777264
One table per document the writer has read, and one for its contributors:
reference id, type, title, subtitle, title_translated,
container_title, volume, issue, pages, publisher,
publisher_place, event_*, publication_year/month/day,
accessed_date, doi, isbn, issn, url, patent_*,
standard_number, report_number, arxiv_id, abstract,
keywords, note, citation_key, language
reference_author reference_id, sort, name, family_name, given_name, role
The field set is the intersection of the formats a paper has to survive, taken
from the CSL JSON schema (103 fields, 45 types — what Zotero, Mendeley and
Pandoc exchange) against GB/T 7714-2015, BibTeX and RIS. The mapping is on
`app.models.reference` rather than in a commit message, because the next person
to add a column needs it more than this one does. Three decisions worth naming:
* **Partial dates.** Year, month and day are separate nullable integers, which
is CSL's `issued`: a book has a year, a newspaper article a day, a patent its
公告日期, and nothing invents the rest.
* **Contributors are rows, not a column.** A citation prints an ordered list
with 编 / 译 marks, and styles rewrite names (「张三」 / "Zhang, S." /
"S. Zhang"), so the printed form and its two halves are stored separately —
display reads `name`, a reformatting style reads the halves, and an
institutional author has neither.
* **`volume` / `issue` / `pages` are strings.** Journals print `12(Suppl.)`,
`S1` and `e12345`, and a citation reproduces what is printed.
The library is global: the same article is cited from several papers, so the
entry cannot belong to one of them. The citation stays on the paper side in
`paper_sentence_reference`, which now carries a real foreign key to `reference`
(`ON DELETE RESTRICT`) — the column had been a loose integer waiting for this
table. A sentence may quote several documents and a document may be quoted by
many sentences, which is why the relation is a table and not a column on
`paper_sentence`.
Refusals are raised once, in the data layer, as `UnknownReference`, and
translated where each caller speaks: `400` with the offending ids for REST (one
exception handler, no route carries the check), a readable tool error for MCP
(one wrapper in the tool registry). Neither entry point can skip it, so the
foreign key never has to be the thing that complains.
API: `/references` CRUD plus batch delete, and `/references/{id}/citations` for
the reverse lookup — "where have I used this article?" — which is what tells a
writer whether an entry is safe to delete.
Verified: 29 new checks in `scripts/smoke_references.py` (partial dates, DOI
normalisation, author order and roles, search by author/type/year/cited, the
citation round trip, 400 on a phantom id, 409 on a cited entry), the paper
suite's 45 and the MCP suite's 47 still pass. The MCP suite now cites a real
entry instead of a phantom id, which is what an agent has to do now.
376 lines
13 KiB
Python
376 lines
13 KiB
Python
"""Data access for the reference library (参考文献).
|
|
|
|
The list endpoint answers the question a library is opened to answer — "what do
|
|
I have, and what is already cited?" — in one query: each row carries its printed
|
|
author line and how many sentences cite it, both as correlated subqueries, so a
|
|
table row never has to load the rows behind it.
|
|
|
|
Writes keep the two lists (authors, citations) on their own side: authors belong
|
|
to the reference and are replaced wholesale, citations belong to the papers that
|
|
made them and are never touched from here.
|
|
"""
|
|
|
|
from collections.abc import Sequence
|
|
|
|
from sqlalchemy import Select, func, or_, select
|
|
from sqlalchemy.orm import Session
|
|
|
|
from app.crud.filters import LIKE_ESCAPE, like_pattern
|
|
from app.models import (
|
|
Paper,
|
|
PaperSentence,
|
|
PaperSentenceReference,
|
|
Reference,
|
|
ReferenceAuthor,
|
|
TemplateTitle,
|
|
TemplateTitlePlacement,
|
|
)
|
|
from app.schemas.reference import (
|
|
ReferenceCitationRead,
|
|
ReferenceCreate,
|
|
ReferenceListItem,
|
|
ReferenceRead,
|
|
ReferenceUpdate,
|
|
)
|
|
|
|
#: How many authors a table row prints before it cuts the line short.
|
|
_LIST_AUTHOR_LIMIT = 3
|
|
|
|
|
|
def _conditions(
|
|
keyword: str | None,
|
|
type_: str | None,
|
|
year: int | None,
|
|
cited_only: bool | None,
|
|
) -> list:
|
|
"""Translate the list filters into SQLAlchemy predicates."""
|
|
conditions = []
|
|
if keyword:
|
|
pattern = like_pattern(keyword)
|
|
# The author search is an EXISTS rather than a join: joining would
|
|
# duplicate a reference with four authors into four rows.
|
|
conditions.append(
|
|
or_(
|
|
Reference.title.like(pattern, escape=LIKE_ESCAPE),
|
|
Reference.subtitle.like(pattern, escape=LIKE_ESCAPE),
|
|
Reference.container_title.like(pattern, escape=LIKE_ESCAPE),
|
|
Reference.doi.like(pattern, escape=LIKE_ESCAPE),
|
|
Reference.keywords.like(pattern, escape=LIKE_ESCAPE),
|
|
Reference.note.like(pattern, escape=LIKE_ESCAPE),
|
|
select(ReferenceAuthor.id)
|
|
.where(
|
|
ReferenceAuthor.reference_id == Reference.id,
|
|
ReferenceAuthor.name.like(pattern, escape=LIKE_ESCAPE),
|
|
)
|
|
.exists(),
|
|
)
|
|
)
|
|
if type_:
|
|
conditions.append(Reference.type == type_)
|
|
if year is not None:
|
|
conditions.append(Reference.publication_year == year)
|
|
if cited_only is True:
|
|
conditions.append(_citation_count_column() > 0)
|
|
if cited_only is False:
|
|
conditions.append(_citation_count_column() == 0)
|
|
return conditions
|
|
|
|
|
|
def _citation_count_column():
|
|
"""Correlated count of the sentences that cite this reference."""
|
|
return (
|
|
select(func.count(PaperSentenceReference.id))
|
|
.where(PaperSentenceReference.reference_id == Reference.id)
|
|
.correlate(Reference)
|
|
.scalar_subquery()
|
|
)
|
|
|
|
|
|
def _ordered(stmt: Select) -> Select:
|
|
"""Newest publication first, then most recently edited.
|
|
|
|
A library is browsed by "what did I read about this", so the year leads and
|
|
the edit time breaks ties — a reference added today to an older paper still
|
|
surfaces if its year is the newest.
|
|
"""
|
|
return stmt.order_by(
|
|
Reference.publication_year.desc(),
|
|
Reference.updated_at.desc(),
|
|
Reference.id.desc(),
|
|
)
|
|
|
|
|
|
def authors_line(authors: Sequence[ReferenceAuthor]) -> str:
|
|
"""The printed author line, joined for a table row.
|
|
|
|
Truncated for display only — the stored list is never cut — because a
|
|
fifteen-author row would push every other column off the screen.
|
|
"""
|
|
names = [author.name for author in authors]
|
|
if len(names) > _LIST_AUTHOR_LIMIT:
|
|
return ", ".join(names[:_LIST_AUTHOR_LIMIT]) + f" 等 {len(names)} 人"
|
|
return ", ".join(names)
|
|
|
|
|
|
def list_references(
|
|
db: Session,
|
|
*,
|
|
keyword: str | None = None,
|
|
type_: str | None = None,
|
|
year: int | None = None,
|
|
cited_only: bool | None = None,
|
|
page: int = 1,
|
|
page_size: int = 20,
|
|
) -> tuple[list[ReferenceListItem], int]:
|
|
"""Return one page of the library, plus the unpaged total."""
|
|
conditions = _conditions(keyword, type_, year, cited_only)
|
|
|
|
total = db.scalar(select(func.count(Reference.id)).where(*conditions)) or 0
|
|
|
|
stmt = _ordered(
|
|
select(Reference)
|
|
.where(*conditions)
|
|
.offset((page - 1) * page_size)
|
|
.limit(page_size)
|
|
)
|
|
rows = list(db.scalars(stmt).all())
|
|
|
|
# One query for the citation counts of the whole page, rather than one per
|
|
# row: the count is a correlated subquery in the list, but here the ids are
|
|
# already known and a grouped count is cheaper.
|
|
counts = citation_counts(db, [row.id for row in rows])
|
|
|
|
items = [
|
|
ReferenceListItem(
|
|
id=row.id,
|
|
type=row.type,
|
|
title=row.title,
|
|
subtitle=row.subtitle,
|
|
container_title=row.container_title,
|
|
publication_year=row.publication_year,
|
|
volume=row.volume,
|
|
issue=row.issue,
|
|
pages=row.pages,
|
|
doi=row.doi,
|
|
authors_line=authors_line(row.authors),
|
|
author_count=len(row.authors),
|
|
citation_count=counts.get(row.id, 0),
|
|
created_at=row.created_at,
|
|
updated_at=row.updated_at,
|
|
)
|
|
for row in rows
|
|
]
|
|
return items, total
|
|
|
|
|
|
def citation_counts(db: Session, reference_ids: Sequence[int]) -> dict[int, int]:
|
|
"""How many sentences cite each reference."""
|
|
wanted = list({reference_id for reference_id in reference_ids})
|
|
if not wanted:
|
|
return {}
|
|
stmt = (
|
|
select(PaperSentenceReference.reference_id, func.count(PaperSentenceReference.id))
|
|
.where(PaperSentenceReference.reference_id.in_(wanted))
|
|
.group_by(PaperSentenceReference.reference_id)
|
|
)
|
|
return {reference_id: count for reference_id, count in db.execute(stmt).all()}
|
|
|
|
|
|
def get(db: Session, reference_id: int) -> Reference | None:
|
|
"""Return one reference with its authors loaded, or ``None``."""
|
|
return db.get(Reference, reference_id)
|
|
|
|
|
|
def get_many(db: Session, reference_ids: Sequence[int]) -> list[Reference]:
|
|
"""Return every reference whose id is in ``reference_ids``."""
|
|
if not reference_ids:
|
|
return []
|
|
return list(
|
|
db.scalars(select(Reference).where(Reference.id.in_(list(reference_ids)))).all()
|
|
)
|
|
|
|
|
|
class UnknownReference(ValueError):
|
|
"""A write named literature the library does not have.
|
|
|
|
A domain error rather than an HTTP one: the same refusal has to reach a
|
|
REST client as ``400`` and a model through the MCP tools as a readable tool
|
|
error, and the layer that knows what went wrong is this one. Each caller
|
|
translates it into its own vocabulary.
|
|
"""
|
|
|
|
def __init__(self, missing: Sequence[int]) -> None:
|
|
self.missing = list(missing)
|
|
joined = "、".join(str(reference_id) for reference_id in self.missing)
|
|
super().__init__(f"参考文献不存在:{joined}")
|
|
|
|
|
|
def assert_exist(db: Session, reference_ids: Sequence[int]) -> None:
|
|
"""Refuse a write that cites literature the library does not have.
|
|
|
|
Called from the paper side before it stores a citation, so the failure is a
|
|
message naming the ids rather than a foreign-key violation from the driver.
|
|
"""
|
|
missing = missing_ids(db, reference_ids)
|
|
if missing:
|
|
raise UnknownReference(missing)
|
|
|
|
|
|
def missing_ids(db: Session, reference_ids: Sequence[int]) -> list[int]:
|
|
"""Which of ``reference_ids`` are not in the library.
|
|
|
|
Returning the offenders rather than a boolean lets the API name them, which
|
|
is the difference between a usable error and "invalid request". A citation
|
|
is rejected against this list, so a sentence cannot point at literature that
|
|
does not exist.
|
|
"""
|
|
wanted = set(reference_ids)
|
|
if not wanted:
|
|
return []
|
|
found = set(db.scalars(select(Reference.id).where(Reference.id.in_(wanted))).all())
|
|
return sorted(wanted - found)
|
|
|
|
|
|
def read(db: Session, reference: Reference) -> ReferenceRead:
|
|
"""Build the detail payload for one reference."""
|
|
counts = citation_counts(db, [reference.id])
|
|
return ReferenceRead.model_validate(
|
|
{
|
|
**{
|
|
column.name: getattr(reference, column.name)
|
|
for column in Reference.__table__.columns
|
|
},
|
|
"authors": reference.authors,
|
|
"citation_count": counts.get(reference.id, 0),
|
|
}
|
|
)
|
|
|
|
|
|
def _build_authors(
|
|
items: Sequence,
|
|
) -> list[ReferenceAuthor]:
|
|
"""Materialise the client's contributor list, numbering it from 1."""
|
|
return [
|
|
ReferenceAuthor(
|
|
name=item.name,
|
|
family_name=item.family_name,
|
|
given_name=item.given_name,
|
|
role=item.role,
|
|
sort=item.sort if item.sort is not None else index,
|
|
)
|
|
for index, item in enumerate(items, start=1)
|
|
]
|
|
|
|
|
|
def create(db: Session, data: ReferenceCreate) -> Reference:
|
|
"""Insert a reference together with its contributors."""
|
|
values = data.model_dump(exclude={"authors"})
|
|
reference = Reference(**values)
|
|
reference.authors = _build_authors(data.authors)
|
|
db.add(reference)
|
|
db.commit()
|
|
db.refresh(reference)
|
|
return reference
|
|
|
|
|
|
def update(
|
|
db: Session,
|
|
reference: Reference,
|
|
data: ReferenceUpdate,
|
|
) -> Reference:
|
|
"""Apply a partial update.
|
|
|
|
``exclude_unset`` is what makes PATCH semantics work: a key the client did
|
|
not send leaves the column alone, so clearing a field is an explicit
|
|
``null`` and omitting it is not.
|
|
"""
|
|
values = data.model_dump(exclude_unset=True, exclude={"authors"})
|
|
for key, value in values.items():
|
|
setattr(reference, key, value)
|
|
|
|
if data.authors is not None:
|
|
# delete-orphan removes the dropped rows on flush; the ORM cascade is
|
|
# what cleans them up, whether or not the database would.
|
|
reference.authors.clear()
|
|
db.flush()
|
|
reference.authors.extend(_build_authors(data.authors))
|
|
|
|
db.commit()
|
|
db.refresh(reference)
|
|
return reference
|
|
|
|
|
|
def delete(db: Session, reference: Reference) -> None:
|
|
"""Delete a reference. Callers must check :func:`citation_counts` first."""
|
|
db.delete(reference)
|
|
db.commit()
|
|
|
|
|
|
def delete_many(db: Session, reference_ids: Sequence[int]) -> int:
|
|
"""Delete several references, returning how many actually existed."""
|
|
references = get_many(db, reference_ids)
|
|
for reference in references:
|
|
db.delete(reference)
|
|
db.commit()
|
|
return len(references)
|
|
|
|
|
|
def list_citations(
|
|
db: Session, reference: Reference
|
|
) -> list[ReferenceCitationRead]:
|
|
"""Every sentence that cites this reference, with its place in the paper.
|
|
|
|
The paragraph heading is resolved through the paper's template, so the
|
|
answer reads as "in 3.1 Problem Formulation of <paper>" rather than as a
|
|
position number only the database understands.
|
|
"""
|
|
stmt = (
|
|
select(PaperSentenceReference, PaperSentence, Paper)
|
|
.join(PaperSentence, PaperSentence.id == PaperSentenceReference.sentence_id)
|
|
.join(Paper, Paper.id == PaperSentence.paper_id)
|
|
.where(PaperSentenceReference.reference_id == reference.id)
|
|
.order_by(Paper.id.asc(), PaperSentence.paper_template_filed_sort, PaperSentence.sort)
|
|
)
|
|
|
|
results: list[ReferenceCitationRead] = []
|
|
for citation, sentence, paper in db.execute(stmt).all():
|
|
results.append(
|
|
ReferenceCitationRead(
|
|
citation_id=citation.id,
|
|
sentence_id=sentence.id,
|
|
sentence_content=sentence.content,
|
|
quote=citation.quote,
|
|
paper_id=paper.id,
|
|
paper_title=paper.title,
|
|
paper_template_filed_sort=sentence.paper_template_filed_sort,
|
|
paragraph_name=_paragraph_name(
|
|
db, paper.template_id, sentence.paper_template_filed_sort
|
|
),
|
|
)
|
|
)
|
|
return results
|
|
|
|
|
|
def _paragraph_name(db: Session, template_id: int | None, position: int) -> str | None:
|
|
"""The heading the template puts at ``position``, or ``None``.
|
|
|
|
``None`` rather than a placeholder: the document renderer decides how to
|
|
print a position the template does not define (it shows 未设定), and this
|
|
should not invent a second answer.
|
|
"""
|
|
if template_id is None:
|
|
return None
|
|
return db.scalar(
|
|
select(TemplateTitle.name)
|
|
.join(
|
|
TemplateTitlePlacement,
|
|
TemplateTitlePlacement.title_id == TemplateTitle.id,
|
|
)
|
|
.where(
|
|
TemplateTitlePlacement.template_id == template_id,
|
|
TemplateTitlePlacement.sort == position,
|
|
)
|
|
.order_by(TemplateTitlePlacement.sort.asc(), TemplateTitlePlacement.id.asc())
|
|
.limit(1)
|
|
)
|