Predicting Memorization Before Fine-Tuning
A classifier trained only on pre-fine-tuning base-model features predicts which sequences a fine-tuned model will memorize. The "Explore held-out examples" tab verifies this against ground truth on held-out examples from a fresh fine-tuning run; the "Score your own text" tab applies the same classifier to a passage you provide.
Pick a real Pythia-1.4B fine-tuning sequence and compare the classifier's pre-fine-tuning risk score with what the fine-tuned model actually generated.
Please enter an input text that is at least 100 tokens (or 75 words). Using its base-model features from Pythia-1.4B, our learned classifiers (one per fine-tuning dataset) predict how likely Pythia-1.4B would be to memorize it if you fine-tuned on it. The plots below show where your text's features fall relative to memorized and not-memorized examples.
Datasets shown are public (FineWeb, PG-19, The Stack, OpenWebMath). The classifier uses only base-model features and was trained on a separate fine-tuning run.