backend: let a model put a figure in the paper, and set a formula's alignment
Two things the editor gained and MCP could not do, both because the tool layer
had been told about the domain's columns by hand.
**A figure.** Bytes go to MinIO and the sentence keeps a Markdown address, so a
picture is ordinary text — which means the writing tools already carry it, and
the only half a model cannot do is the upload: it has bytes and no way to send
multipart. `image_upload` takes base64 and answers with the URL, the Markdown
and the *sentence object* the editor produces when a picture is dropped in, so a
figure is written by putting that object into `sentences` unchanged. It does not
fetch a URL the caller names: a service inside the cluster that will GET any
address it is handed is an SSRF primitive aimed at everything beside it.
The width argument is the reader's rule or it is refused. `{width=70%}` is
parsed by `frontend/src/utils/rich.ts`, and a value it cannot read is not an
error there — the picture just prints at its natural size. A size that silently
does not apply is worse than a refusal, so the syntax and the range (pixels
1–4000, per cent 1–100) are mirrored in `app/mcp/tools/images.py`, with the
usual comment naming the other side.
`paragraph_text` also had to learn what a figure is: a sentence holding nothing
but an image is a 插图 — the view prints it on its own line with the brackets'
text as its caption — and the text a tool hands back broke lines around a
display formula but not around a picture. Same rule as `figureOf`, same pair of
implementations, same warning about changing both.
**A formula's alignment.** `paper_sentence.formula_align` reached the editor and
the REST API and stopped there: `SentenceSpec` had no such field, so a model that
sent one got a successful call and a centred formula. The column is now declared,
carried through every write path and reported by every read path.
That is the second column lost the same way, so the fix is structural rather
than another two edits. `SENTENCE_FIELDS` is read from
`app.schemas.paper.SentenceInput` instead of written out, the payload is built
by one `sentence_payload()`, and the three append paths spread
`SentenceCreate(**item.model_dump())` rather than copying fields. One edit
remains — declaring the field in `specs.py` — and smoke_mcp.py's first step
compares the two field sets, so forgetting it now fails the suite instead of
losing the field. `sort` is declared too, which it never was: a model that sent
one had it dropped in silence.
smoke_mcp.py: 99 checks to 119. The figure step uploads a real 1x1 GIF (the same
bytes twice, and the key is the same object), writes the returned sentence as it
came back, checks the paragraph prints it alone on its line and that an image
inside a sentence does not break the line, and takes the four refusals — not a
picture, an unparseable size, an out-of-range one, not base64. The alignment step
writes a right-aligned formula, moves the sentence to left on its own, and takes
the paragraph to another position to prove the setting travels with it.
Deployed as v6, and the pod now carries MINIO_ENDPOINT (the in-cluster Service)
plus the two credential keys from the Secret; without them the image tool answers
with the missing variable names and every writing tool still works.
This commit is contained in:
@@ -0,0 +1,191 @@
|
||||
"""Image tools (插图): put a figure in the bucket, and hand back the sentence.
|
||||
|
||||
A picture is not text, so it does not live in the database: the sentence keeps a
|
||||
Markdown address and the bytes live in MinIO (see :mod:`app.core.storage`). That
|
||||
is what keeps a sentence one editable string, and it is also why there is no
|
||||
``image_create`` that attaches a picture to a paragraph — writing a figure *is*
|
||||
writing a sentence whose content happens to be ````.
|
||||
|
||||
One tool, and one field of a sentence, is therefore the whole feature from a
|
||||
model's side. ``image_upload`` exists for the half a model cannot do itself: it
|
||||
has bytes (a screenshot the user pasted, a figure it generated) and no way to
|
||||
send a multipart request. Everything after that is ordinary text.
|
||||
|
||||
The base64 argument is a deliberate cost. A tool argument is context tokens, and
|
||||
base64 is a third larger than the bytes it carries — but the alternative is a
|
||||
second ingestion path (an HTTP fetch of a URL the caller names), and a service
|
||||
inside the cluster that will fetch any address it is handed is an SSRF primitive
|
||||
aimed at everything beside it. A model with a picture and no base64 can post it
|
||||
to ``POST /api/images`` and use the Markdown that comes back; the two produce the
|
||||
same sentence, byte for byte, because both go through
|
||||
:func:`app.core.storage.markdown_for`.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import base64
|
||||
import binascii
|
||||
import re
|
||||
from typing import Annotated
|
||||
|
||||
from mcp.server.mcpserver import MCPServer
|
||||
from pydantic import Field
|
||||
|
||||
from app.core.config import get_settings
|
||||
from app.core.storage import get_store, image_url, markdown_for
|
||||
from app.mcp import support as s
|
||||
from app.mcp.tools.registry import mcp_tool
|
||||
|
||||
|
||||
def register(server: MCPServer) -> None:
|
||||
"""Register every image tool on ``server``."""
|
||||
|
||||
@mcp_tool(server, title="上传插图")
|
||||
def image_upload(
|
||||
data_base64: Annotated[
|
||||
str,
|
||||
Field(
|
||||
description=(
|
||||
"图片内容的 base64。可以直接给 data:image/png;base64,… 这种完整形式。"
|
||||
"PNG / JPEG / GIF / WebP,最大 20 MB"
|
||||
)
|
||||
),
|
||||
],
|
||||
caption: Annotated[
|
||||
str | None,
|
||||
Field(description="图注;不给就用 filename 的主干,都没有则留空"),
|
||||
] = None,
|
||||
filename: Annotated[
|
||||
str | None,
|
||||
Field(description="文件名,只用于取图注初值;存储的键由内容决定,与它无关"),
|
||||
] = None,
|
||||
width: Annotated[
|
||||
str | None,
|
||||
Field(
|
||||
description=(
|
||||
"可选宽度:70%(占栏宽)或 400px。不传则按原尺寸,最大不超过栏宽"
|
||||
)
|
||||
),
|
||||
] = None,
|
||||
) -> str:
|
||||
"""上传一张图片,返回可直接写进句子的 Markdown 与那个句子对象。"""
|
||||
data = _decode(data_base64)
|
||||
settings = get_settings()
|
||||
if len(data) > settings.max_image_bytes:
|
||||
s.fail(
|
||||
f"图片太大了({len(data) / (1024 * 1024):.1f} MB),"
|
||||
f"上限是 {settings.max_image_bytes // (1024 * 1024)} MB。先压缩或裁剪一下再传。"
|
||||
)
|
||||
|
||||
# The store sniffs the format out of the bytes and refuses anything that
|
||||
# is not a picture: an SVG is stored XSS when it is served from this
|
||||
# origin, and the caller's content type or filename is only a claim.
|
||||
key, content_type, digest = get_store().save(data)
|
||||
|
||||
markdown = _embed(key, caption=caption or _stem(filename), width=width)
|
||||
return s.dumps(
|
||||
{
|
||||
"key": key,
|
||||
"url": image_url(key),
|
||||
"markdown": markdown,
|
||||
"content_type": content_type,
|
||||
"size": len(data),
|
||||
"sha256": digest,
|
||||
# The sentence, not just the Markdown: this is what the editor
|
||||
# produces when a picture is dropped into it — the figure on a
|
||||
# line of its own — so a model writes a figure by putting this
|
||||
# object into `sentences` and changing nothing about it.
|
||||
"sentence": {
|
||||
"content": markdown,
|
||||
"line_break_before": True,
|
||||
"formula_align": "center",
|
||||
},
|
||||
}
|
||||
)
|
||||
|
||||
|
||||
# --- helpers -----------------------------------------------------------------
|
||||
|
||||
#: How a size may be written, and what it may be. Both halves mirror
|
||||
#: ``WIDTH_PATTERN`` and ``parseImageWidth`` in ``frontend/src/utils/rich.ts``,
|
||||
#: which is what actually reads the attribute when the paper is printed: the
|
||||
#: syntax is ``{width=70%}`` or ``{width=400px}``, digits with at most one unit,
|
||||
#: pixels up to :data:`MAX_PIXELS` and percentages up to 100.
|
||||
#:
|
||||
#: Written twice because one side is TypeScript and one is Python, and checked
|
||||
#: here rather than passed through, because the reader's answer to a value it
|
||||
#: cannot parse is to print the picture at its natural size — a size that
|
||||
#: silently does not apply is worse than a refusal that says so.
|
||||
_WIDTH_ATTRIBUTE = re.compile(r"^\s*(?:w|width)\s*=\s*(\d{1,4})\s*(%|px)?\s*$", re.IGNORECASE)
|
||||
_WIDTH_BARE = re.compile(r"^\s*(\d{1,4})\s*(%|px)?\s*$")
|
||||
_MAX_PIXELS = 4000
|
||||
_MAX_PERCENT = 100
|
||||
|
||||
|
||||
def _decode(value: str) -> bytes:
|
||||
"""The bytes behind a base64 argument, refusing anything that is not one.
|
||||
|
||||
A ``data:`` prefix is stripped rather than refused: it is what a browser
|
||||
writes when a picture is copied as a data URL, and it is what a model
|
||||
produces when it repeats what it was given.
|
||||
"""
|
||||
text = value.strip()
|
||||
if text.startswith("data:"):
|
||||
_, _, text = text.partition(",")
|
||||
# Whitespace is folded out because base64 arrives wrapped in as many ways as
|
||||
# there are tools that produced it.
|
||||
text = "".join(text.split())
|
||||
if not text:
|
||||
s.fail("data_base64 是空的,没有图片内容")
|
||||
try:
|
||||
return base64.b64decode(text, validate=True)
|
||||
except (binascii.Error, ValueError):
|
||||
s.fail("data_base64 不是合法的 base64;请给图片字节的 base64,不要给文件路径或 URL")
|
||||
|
||||
|
||||
def _stem(filename: str | None) -> str:
|
||||
"""The caption a filename suggests, the way the upload endpoint reads it."""
|
||||
if not filename:
|
||||
return ""
|
||||
name = filename.replace("\\", "/").rsplit("/", 1)[-1]
|
||||
return name.rsplit(".", 1)[0] if "." in name else name
|
||||
|
||||
|
||||
def _embed(key: str, *, caption: str, width: str | None) -> str:
|
||||
"""The Markdown a sentence carries: the address, plus an optional width.
|
||||
|
||||
``markdown_for`` writes the ```` half — the same function the
|
||||
upload endpoint answers with — and this adds the size, which the endpoint
|
||||
does not take because a browser appends it in the editor.
|
||||
"""
|
||||
markdown = markdown_for(key, caption)
|
||||
attribute = _width_attribute(width)
|
||||
return f"{markdown}{attribute}" if attribute else markdown
|
||||
|
||||
|
||||
def _width_attribute(width: str | None) -> str:
|
||||
"""``{width=70%}`` from what the caller wrote, or a refusal saying why not.
|
||||
|
||||
Tolerant about how it is written — ``70%``, ``70 %``, ``{width=70%}`` and
|
||||
``w=400px`` all mean the same thing, which is what a model repeats back from
|
||||
a sentence it read — and strict about what it means.
|
||||
"""
|
||||
if width is None or not str(width).strip():
|
||||
return ""
|
||||
|
||||
written = str(width).strip().strip("{}").strip()
|
||||
match = _WIDTH_ATTRIBUTE.match(written) or _WIDTH_BARE.match(written)
|
||||
if match is None:
|
||||
s.fail(
|
||||
f"width 只能是百分比或像素,例如 70% 或 400px,收到的是“{width}”。"
|
||||
"不写则按原尺寸显示(最多不超过栏宽)"
|
||||
)
|
||||
|
||||
digits, unit = int(match.group(1)), (match.group(2) or "px").lower()
|
||||
limit = _MAX_PERCENT if unit == "%" else _MAX_PIXELS
|
||||
if not 1 <= digits <= limit:
|
||||
s.fail(f"width 的 {unit} 值要在 1 到 {limit} 之间,收到的是“{width}”")
|
||||
return f"{{width={digits}{unit}}}"
|
||||
|
||||
|
||||
__all__ = ["register"]
|
||||
Reference in New Issue
Block a user