Dimwitted Sage

This site is being rebuilt. Some pages and posts are still in progress.

Summarizing New arXiv Papers with a 0.5B Model

November 16, 2026
Part 1 of 1 in AI Experiments

The first experiment is a catalog of new AI papers from arXiv. Every paper gets a short summary written by a small model running locally. The summary is checked against the paper, and the paper is compared with other recent papers so you can see whether something similar was already written. The results are on Paper Radar, which grows as new papers are added.

The point isn’t the catalog itself. It’s a way to test how far a very small model can get on a real task, and what it takes to check its work.

Hardware and models

The pipeline

The current workflow. Each run takes the 20 newest cs.AI papers from the arXiv API, converts each PDF to text, and gives the first 10,000 characters to Qwen2.5 0.5B with a distilled LoRA, which writes a summary. A faithfulness check confirms every number and name appears in the full paper. Failures get one retry from Qwen2.5 3B, told which items weren't found; if that also fails, the summary is withheld but the paper is still listed. Each paper becomes a Markdown file, three similar papers are added from a corpus of arXiv abstracts embedded with MiniLM, and the files are synced to Paper Radar on this site. arXiv API 20 newest cs.AI papers per run PDF to text (PyMuPDF) model gets the first 10,000 characters Qwen2.5 0.5B + LoRA writes a 3–5 sentence summary Faithfulness check all numbers and names in the paper? fail Qwen2.5 3B retry told which items weren’t found Check again same rule fail Summary withheld paper still listed pass pass Markdown file one per paper Add 3 similar papers closest abstracts by cosine similarity Similarity corpus arXiv bulk feed + MiniLM embeddings sync to Paper Radar scripts/sync_papers.sh
The current workflow. Dashed: the similarity corpus, updated separately from the daily run.

Each run does the following for every paper:

  1. Fetch. New cs.AI papers come from the arXiv API, newest first. The default run takes 20.
  2. Read the paper. The PDF is downloaded and converted to text. The model sees only the first 10,000 characters, about 2,850 tokens. That usually covers the abstract, introduction and the start of the method section, but the median paper is about 74,000 characters, so the model reads roughly 13% of it. The cutoff was chosen, not measured. (See the limitations below.)
  3. Summarize. The 0.5B model, with a LoRA adapter (described below), writes a 3–5 sentence summary covering the problem, what’s proposed, and what’s found. Temperature 0.3, at most 200 tokens.
  4. Check. A faithfulness check pulls every number and proper noun out of the summary and confirms each one appears in the full paper text, not just the part the model saw. Spelled-out numbers count (“ten” matches “10”). Matching is whole-word, so “four” doesn’t match “fourteen”. The summary passes only if every item is found.
  5. Fall back. If the summary fails, the 3B model gets one try. Its prompt names the items that weren’t found and asks for a new summary that only uses names and numbers from the paper. The better-scoring of the two summaries is kept.
  6. Withhold. A summary that still fails isn’t shown. The paper is still listed, with a link to its abstract on arXiv.
  7. Find similar papers. The paper’s title and abstract are embedded and compared with a store of recent arXiv abstracts. The three closest are listed. (Details below.)
  8. Publish. Each paper is written as a Markdown file with frontmatter, then copied into this site.

Training the small model: distillation

Out of the box, the 0.5B model wrote summaries that often weren’t faithful to the paper. The 3B model wrote better ones, but it’s about four times slower. The fix was distillation: have the 3B write summaries and train the 0.5B to copy them.

Comparison on six papers the adapter hadn’t seen:

Model Faithfulness Time per paper
0.5B, no adapter 0.878 8.2 s
0.5B + distillation adapter 0.944 3.7 s
3B (the teacher) 0.979 14.2 s

The adapter closed most of the gap to the 3B at a quarter of the 3B’s time. Six papers is a small sample, so treat the differences as rough.

First catalog run

The run published on the summaries page covered 20 cs.AI papers submitted on Nov 10, 2026:

Finding similar papers

A summary says what a paper does. It doesn’t say whether the idea is new. To give some sense of that, each paper is compared with other recent papers using embeddings, a form of retrieval-augmented lookup (RAG):

The scores so far range from 0.46 to 0.73. Some matches are good: for “ReTrace,” which turns logged agent runs into fixed workflows, one match is “How Much Skill Orchestration Can LLM Agents Compile Away?”, a closely related idea. Others only share a topic. A high score means similar topic and wording, not the same idea.

What didn’t work

Several approaches were tried and dropped before the pipeline above:

Limitations

Next