Distilled PI example walkthrough
This page shows a complete Distilled PI run: how to invoke the Agent Skill, what the user provides, how the AI asks follow-up questions, how research-harness.md is generated, and how /pi-grill turns that harness into pi-grill-report.md.
The user begins with a research title and a short background note. The Agent Skill's job is not to write a proposal. It identifies the paper type, asks about the critical gaps, and generates a structured research harness.
Use $distilled-pi
/title-to-harness
Title:
Can Small Language Models Learn to Self-Correct Through Synthetic Critiques?
Optional context:
I want to test whether 1B-3B open-source language models can acquire genuine self-correction behavior from critiques generated by stronger models, rather than merely learning critique-style answer rewriting.
Important:
When the final harness markdown is generated, please save it directly as a Markdown file named `research-harness.md`.The AI classifies the title as a Research Paper with High confidence.
The user does not defend "robust self-correction." The safer claim is that synthetic critiques provide value beyond corrected-answer supervision under specific diagnostic conditions.
Self-correction requires sensitivity to critique correctness, partial transfer to unseen error types, and behavior changes when critique quality changes.
The most dangerous baseline is answer-only finetuning, because it directly tests whether critique text adds useful information.
If answer-only finetuning matches critique training, critique corruption has little effect, or gains disappear on unseen error types, the project must stop using "learn to self-correct."
| Dimension | Status | Reason |
|---|---|---|
| Core claim | Clear | The claim has been narrowed from strong self-correction to critique-content dependence. |
| Novelty path | Clear | The novelty is diagnostic empirical analysis, not simple method novelty. |
| Strongest baseline | Clear | Answer-only finetuning is explicitly identified as the most dangerous baseline. |
| Implementation details | Partial | Model family, optimizer, dataset size, and metric thresholds are not fixed, but they do not block a first harness. |
The AI generates and saves research-harness.md. The important sections are:
## 6. Operational Definition
For this project, genuine self-correction means that a model:
- improves initially incorrect answers after receiving critique information;
- is sensitive to whether the critique is correct, generic, unrelated, or misleading;
- transfers at least partially to unseen error types;
- changes behavior when critique quality changes.
The safe operational label for early experiments is critique-conditioned correction behavior, not robust self-correction.
The user provides research-harness.md as the source of truth. The Agent Skill does not brainstorm from scratch. It simulates a real PI meeting and focuses on the questions most likely to reshape the project.
/pi-grill
I am providing the research-harness.md generated by /title-to-harness.
Please act as a demanding PI who is blunt because they want to protect the student from wasting months on a weak claim.
Rules:
- Ask only the 2-3 questions that matter most.
- Do not ask checklist questions.
- Focus on claim validity, novelty, evidence, falsification, and fallback contribution.
- Critique first, then provide a repair path.
- Optimize for finding the fastest way to kill or substantially reshape the project.
When the final report is generated:
- Save it as pi-grill-report.md
- Include PI Focus, Research Risk Heatmap, and PI Verdict.PI Focus: Experiment design
Main danger: if the ablation matrix cannot prove critique-content dependence, the project collapses into an answer-only supervision variant.
The AI generates and saves pi-grill-report.md. The core conclusion:
The project must prove a causal contribution from valid critique content, not merely show that corrected-answer supervision works.
Ready for a pilot experiment. Not ready for a full paper draft. The project is currently an empirical diagnostic paper, not a method paper.
| Risk Type | Level | Reason |
|---|---|---|
| Claim Risk | HIGH | The title still implies robust self-correction, which the pilot may not support. |
| Novelty Risk | MEDIUM | The method itself is not the novelty; the diagnostic protocol is. |
| Experiment Risk | MEDIUM | The ablation plan is strong, but threshold rules must be calibrated against pilot variance. |
| Execution Risk | MEDIUM | Non-leaky critiques and meaningful misleading critiques are difficult to construct cleanly. |
| Publication Risk | MEDIUM | Positive or negative results can both matter if the paper is framed as diagnostic empirical analysis. |
research-harness.mdTurns a vague title into an inspectable research skeleton. The key improvement is the Operational Definition, which narrows self-correction into testable critique-conditioned correction behavior.
pi-grill-report.mdStress-tests the harness and returns the most dangerous questions, a risk heatmap, a repair plan, and a PI verdict.
Use $distilled-pi
/title-to-harness
Title:
<your research title>
Optional context:
<your short background, concern, or idea>Focus on claim, definition, baseline, minimum evidence path, and falsification condition. Implementation details can remain as unknowns in the first harness.
Filename: research-harness.mdUse $distilled-pi
/pi-grill
I am providing the research-harness.md generated by /title-to-harness.
<paste research-harness.md here>If the answers are strong enough, the AI skips Round 2 and directly generates the final repair synthesis and pi-grill-report.md.