Lab Report · Model Pruning · Small Language Models · pruning-rml
Revision Demystifies Small Language Models
Small language models can be built by reset or revision. This report now connects the paper to an extended pruning-rml notebook repo.
Core claim
Pruning is revision. A larger model’s accumulated structure can give a smaller model a stronger starting point than random initialization under matched training-token budgets.
What this report studies
The paper Small LLMs: Pruning vs. Training from Scratch compares two ways to build smaller language models: train a model from random initialization, or prune a larger pretrained model and continue training.
Repo extension
The companion pruning-rml repo expands this lab report into a notebook sequence: reset versus revision, pruning granularity, surviving residues, budget-matched comparisons, revision triggers, policies, lifecycle questions, and benchmark design.
Observation
reset \(\ne\) revision
Training from scratch asks whether capability can be rebuilt. Pruning asks which inherited structures survive compression, retraining, and evaluation.
RML interpretation
Smaller language models are not merely fewer parameters. They can be treated as revised systems: accumulated structure is compressed, tested, retrained, and measured for surviving residues.
Scientific intent
Turn pruning into a measurable revision problem: compare reset and revision under matched budgets, track what persists, and identify which pruning policies preserve useful structure.
pruning-rml notebook map
Paper-grounded notebooks
- 00 Context
- 07 Pruning vs Scratch
- 13 Granularity
- 17 Revision Not Reset
- 23 Surviving Residues
- 29 Budget-Matched Comparisons
Revision framework
- 31 Revision Triggers
- 37 RML-Guided Pruning
- 41 Pruning Policies
- 43 Policy Evaluation
- 47 Residue Retention
Lifecycle and governance
- 53 Revision Lifecycle
- 59 Revision Stability
- 61 Residue Accumulation
- 67 Residue Revision Boundary
- 71 Revision Scheduler
- 73 Revision Queue
- 79 Revision Execution
- 83 Revision Feedback
- 89 Revision Governance
Toward benchmarks
- 97 Revision Architecture
- 101 Repo Retrospective
- 103 Real Pruning Experiments
- 107 Benchmark Design
- 109 Minimum Viable Benchmark
Engineering Statement
Reset and revision are distinct engineering choices.
The repo extends the statement into a working notebook trail: pruning becomes a test of what survives, what must be relearned, and what can be measured under explicit budgets.
Actionable next steps
- Keep the first notebooks anchored to the paper: reset, revision, granularity, residues, and matched budgets.
- Use later notebooks as RML hypotheses, not as reproduced pruning results.
- Promote repeated notebook logic into stable
src/pruning_rml/modules only when it survives reuse. - Connect the repo to real pruning benchmarks before making empirical claims.
- Use the engineering statement to separate claim, constraint, evidence, and next experiment.
Climate Reality / Climate Democracy
Climate Reality describes observable constraints. Climate Democracy prescribes broader participation in understanding those constraints. For this report, the local object is model revision: what is preserved, removed, retrained, measured, and documented.