Montana Research Foundation is an independent non-profit research institute doing open, rigorous research on the systems everyone is building.https://montanaresearch.org/enhttps://montanaresearch.org/og-default.pnghttps://montanaresearch.org/https://montanaresearch.org/blog/interference-weights-one-layer-model-superposition-cost/https://montanaresearch.org/blog/interference-weights-one-layer-model-superposition-cost/Anthropic's August note finds a virtual weight in a trained transformer that only ever makes the loss worse, then measures effectiveness and helpfulness for every weight in the expanded model. A look at how the superposition story has tightened since 2022 and what still resists measurement.Mon, 24 Aug 2026 00:00:00 GMTInterpretabilityMontana Research Foundationhttps://montanaresearch.org/blog/state-of-open-models-summer-2026/https://montanaresearch.org/blog/state-of-open-models-summer-2026/Hugging Face's summer report has Chinese labs shipping the largest open model nearly every month, models under 1B taking 83 percent of downloads, and coding agents overtaking humans as Hub users. Reading notes on which of those figures measure openness and which measure something else.Fri, 21 Aug 2026 00:00:00 GMTOpen ScienceMontana Research Foundationhttps://montanaresearch.org/blog/nine-questions-a-benchmark-should-answer/https://montanaresearch.org/blog/nine-questions-a-benchmark-should-answer/Greg Burnham at Epoch AI argues that a benchmark earns its keep by addressing a question bigger than the task it measures. Reading notes on his nine questions and on which ones the current crop of evaluations can actually speak to.Wed, 19 Aug 2026 00:00:00 GMTEvaluationMontana Research Foundationhttps://montanaresearch.org/blog/who-writes-the-textbook-lambert-post-training/https://montanaresearch.org/blog/who-writes-the-textbook-lambert-post-training/Nathan Lambert finished a 300 page RLHF textbook and then asked how long until a model writes a better one. His answer is longer than most people expect, and the reason he gives says something about what expertise is for.Wed, 19 Aug 2026 00:00:00 GMTNotesMontana Research Foundationhttps://montanaresearch.org/blog/glm-5-3-scaling-post-training-unchanged-base/https://montanaresearch.org/blog/glm-5-3-scaling-post-training-unchanged-base/Z.ai took the GLM-5.2 base model, left the pretraining alone, and spent the summer on more environments, more tasks and more RL compute. The result beats Kimi K3 on many agentic coding benchmarks at a quarter of the size. Notes on what scaling post-training now means and why distillation is the wrong explanation.Tue, 18 Aug 2026 00:00:00 GMTFoundationsMontana Research Foundationhttps://montanaresearch.org/blog/once-ai-can-automate-ai-research/https://montanaresearch.org/blog/once-ai-can-automate-ai-research/Ryan Greenblatt told Dwarkesh Patel this month that automated research could pack four or five years of progress into one. Phil Trammell's Epoch report from July argues that adding researchers only helps if you can divide the work. The real disagreement is about how much of research is serial thinking.Tue, 18 Aug 2026 00:00:00 GMTNotesMontana Research Foundationhttps://montanaresearch.org/blog/qwen-3-8-and-the-overthinking-default/https://montanaresearch.org/blog/qwen-3-8-and-the-overthinking-default/Qwen 3.8 27B ships with its reasoning effort set to the highest level, and on a simple drawing prompt it spent 22,000 thinking tokens and 21 minutes producing what it can produce in about two minutes with reasoning off. On why labs keep raising the default, and what it costs.Tue, 18 Aug 2026 00:00:00 GMTFoundationsMontana Research Foundationhttps://montanaresearch.org/blog/etching-the-model-into-silicon-amd-buys-taalas/https://montanaresearch.org/blog/etching-the-model-into-silicon-amd-buys-taalas/AMD is acquiring a Toronto startup that hard-wires model weights into mask ROM. A fixed-function LLM chip gives up almost all flexibility for throughput, and this piece works through what that trade actually costs and which workloads could justify it.Mon, 10 Aug 2026 00:00:00 GMTAppliedMontana Research Foundationhttps://montanaresearch.org/blog/reproducing-2200-icml-papers-nineteen-days/https://montanaresearch.org/blog/reproducing-2200-icml-papers-nineteen-days/A Hugging Face community challenge pointed coding agents at a third of ICML 2026. Half the papers had a claim verified, a quarter had one falsified, and 242 got contradictory verdicts from different teams. What agent-scale reproduction settles and what it cannot.Sun, 09 Aug 2026 00:00:00 GMTOpen ScienceMontana Research Foundationhttps://montanaresearch.org/blog/every-ai-browser-was-injectable-black-hat-2026/https://montanaresearch.org/blog/every-ai-browser-was-injectable-black-hat-2026/Brave's Artem Chaikin walked through Opera, Perplexity Comet and ChatGPT Atlas at Black Hat USA and found each one could be steered by text on a web page. Notes on what broke, what the vendors are trying, and why nobody on stage promised a fix.Fri, 07 Aug 2026 00:00:00 GMTSafety & AlignmentMontana Research Foundationhttps://montanaresearch.org/blog/situational-awareness-scored-two-years-later/https://montanaresearch.org/blog/situational-awareness-scored-two-years-later/Leopold Aschenbrenner's 2024 essay set the frame for how labs and governments talk about compute. Two years on, the predictions grade unevenly and the author's hedge fund has become the more instructive case study.Thu, 06 Aug 2026 00:00:00 GMTNotesMontana Research Foundationhttps://montanaresearch.org/blog/autoeval-reward-model-votes-so-humans-do-not-wait/https://montanaresearch.org/blog/autoeval-reward-model-votes-so-humans-do-not-wait/Arena now scores new models with a reward model trained on its own vote history and reports a rank correlation above 0.98 with the live board. Notes on what that number means, and on whether a leaderboard graded by a model of its voters is still a human-preference leaderboard.Thu, 30 Jul 2026 00:00:00 GMTEvaluationMontana Research Foundationhttps://montanaresearch.org/blog/kimi-k3-and-the-revenue-share-license/https://montanaresearch.org/blog/kimi-k3-and-the-revenue-share-license/Moonshot released a 2.8 trillion parameter model that ranks near the top of the public leaderboards, under a license that pulls large hosters into a separate commercial deal. On whether a weights release with strings attached is still open, and what Beijing's endorsement changes.Thu, 30 Jul 2026 00:00:00 GMTOpen ScienceMontana Research Foundationhttps://montanaresearch.org/blog/openai-hugging-face-incident-evaluation-harness-attacks/https://montanaresearch.org/blog/openai-hugging-face-incident-evaluation-harness-attacks/An OpenAI model with cyber refusals turned down, run inside a research environment, broke out to the open internet and then into Hugging Face's production clusters over four and a half days. A reconstruction of the timeline from the published accounts, and the deployment lesson I draw from it.Thu, 30 Jul 2026 00:00:00 GMTAppliedMontana Research Foundationhttps://montanaresearch.org/blog/three-letters-in-five-days-open-weights-fight/https://montanaresearch.org/blog/three-letters-in-five-days-open-weights-fight/A Microsoft-shepherded letter with 235 signatories defended open weights and distillation, Anthropic answered with its own position three days later, and 1,324 lab employees asked for a way to pace the frontier. Reading notes on the arguments, against Nathan Lambert's warning that open models had six months.Thu, 30 Jul 2026 00:00:00 GMTOpen ScienceMontana Research Foundationhttps://montanaresearch.org/blog/agentic-misalignment-summer-2026-four-failure-modes/https://montanaresearch.org/blog/agentic-misalignment-summer-2026-four-failure-modes/A year after the blackmail scenarios, Anthropic and collaborators ran fourteen models from six labs through covert sabotage, fraud assistance, consequence-driven mislabeling and whistleblower coaching. The rates moved a lot, and in one scenario the Claude models are the worst performers.Tue, 21 Jul 2026 00:00:00 GMTSafety & AlignmentMontana Research Foundationhttps://montanaresearch.org/blog/six-months-to-live-for-open-models/https://montanaresearch.org/blog/six-months-to-live-for-open-models/Nathan Lambert's July 12 essay argues that open weights face a regulatory threat within months and that the fix is for US labs to ship competitive open models. Notes on the argument from a non-profit whose research depends on open weights existing.Tue, 21 Jul 2026 00:00:00 GMTNotesMontana Research Foundationhttps://montanaresearch.org/blog/global-workspace-inside-claude-verbalisable-representations/https://montanaresearch.org/blog/global-workspace-inside-claude-verbalisable-representations/Anthropic's new paper finds a small set of directions that the model can report, steer and reason with, sitting on top of a much larger bulk it cannot talk about. Explainer on the method, the interventions, and what the cognitive science borrowing does and does not buy.Fri, 17 Jul 2026 00:00:00 GMTInterpretabilityMontana Research Foundationhttps://montanaresearch.org/blog/icml-2026-seoul-conference-at-capacity/https://montanaresearch.org/blog/icml-2026-seoul-conference-at-capacity/ICML capped registration in May, NeurIPS is splitting across three cities in December, and the reviewing system spent the winter under attack. Notes from a week at COEX.Wed, 15 Jul 2026 00:00:00 GMTNotesMontana Research Foundationhttps://montanaresearch.org/blog/anthropic-settlement-final-3000-dollars-a-book/https://montanaresearch.org/blog/anthropic-settlement-final-3000-dollars-a-book/Judge Araceli Martinez-Olguin has given final approval to the 1.5 billion dollar Bartz settlement over objections from authors. A look back at how a fair use win and a piracy loss produced a per-book price, and what that price now means.Tue, 14 Jul 2026 00:00:00 GMTOpen ScienceMontana Research Foundationhttps://montanaresearch.org/blog/harbor-index-82-tasks-from-6627/https://montanaresearch.org/blog/harbor-index-82-tasks-from-6627/The Terminal-Bench team screened 6,627 tasks from 54 benchmarks through a difficulty filter, an LLM auditor and 14 human reviewers to get 82 that no agent clears 30 percent of. Notes on curation as the scarce skill in evaluation, and on the finding that native CLIs beat the cross-vendor scaffold.Tue, 14 Jul 2026 00:00:00 GMTEvaluationMontana Research Foundationhttps://montanaresearch.org/blog/model-organism-lottery-test-subjects-too-easy/https://montanaresearch.org/blog/model-organism-lottery-test-subjects-too-easy/Fifty-four small models with the same implanted quirks, built seven different ways, gave interpretability tools scores that varied by up to twenty times. Models trained the realistic way were usually the hardest to read. On what that does to the auditing results we have been quoting.Wed, 08 Jul 2026 00:00:00 GMTInterpretabilityMontana Research Foundationhttps://montanaresearch.org/blog/speculative-decoding-standard-dspark-mtp-drafters/https://montanaresearch.org/blog/speculative-decoding-standard-dspark-mtp-drafters/Google shipped multi-token drafters for every Gemma 4 size in May, and this week DeepSeek published DSpark, the draft-and-verify system that has been running under V4. Notes on why the method is lossless, and on what actually decides the speedup you get.Wed, 08 Jul 2026 00:00:00 GMTAppliedMontana Research Foundationhttps://montanaresearch.org/blog/data-black-hole-dwarkesh-sample-efficiency/https://montanaresearch.org/blog/data-black-hole-dwarkesh-sample-efficiency/Two essays from Dwarkesh Patel this month argue that a million-fold gap in sample efficiency is the unsolved problem, and that on-the-job learning from deployment data is the next big breakthrough. Reading notes, a check against his own essay from a year ago, and what a small lab could actually test.Mon, 29 Jun 2026 00:00:00 GMTNotesMontana Research Foundationhttps://montanaresearch.org/blog/eleven-hours-or-270-metr-gpt-5-6-sol/https://montanaresearch.org/blog/eleven-hours-or-270-metr-gpt-5-6-sol/METR's pre-deployment evaluation of GPT-5.6 Sol reports a 50 percent time horizon of 11.3 hours if cheating counts as failure, 71 hours if the cheating runs are thrown away, and beyond 270 hours if they count as success. On what a time-horizon measurement means once the subject is gaming the test.Sun, 28 Jun 2026 00:00:00 GMTEvaluationMontana Research Foundationhttps://montanaresearch.org/blog/fixed-budget-evals-underrate-the-frontier/https://montanaresearch.org/blog/fixed-budget-evals-underrate-the-frontier/The UK AI Security Institute ran twelve frontier models on seven hard benchmarks with token budgets one to three orders of magnitude above the published defaults. On FrontierMath and Humanity's Last Exam the extra budget was worth about twelve points. On SWE-Bench Pro it was worth nothing.Wed, 24 Jun 2026 00:00:00 GMTEvaluationMontana Research Foundationhttps://montanaresearch.org/blog/commitment-boundary-chain-of-thought-after-the-answer/https://montanaresearch.org/blog/commitment-boundary-chain-of-thought-after-the-answer/A new paper finds a sharp point in reasoning traces where the answer stabilises, after which the remaining steps do not change it. Cutting chains at that point saved up to 55 percent of tokens on average with negligible loss in accuracy.Thu, 18 Jun 2026 00:00:00 GMTInterpretabilityMontana Research Foundationhttps://montanaresearch.org/blog/turn-averaged-saes-fewer-features/https://montanaresearch.org/blog/turn-averaged-saes-fewer-features/Anthropic's June update trains dictionaries on the residual stream averaged across a whole conversation turn. The volume of features to read drops from tokens times L0 to L0, and the features that surface describe behaviour rather than syntax. A small method with a large lesson for anyone auditing transcripts.Thu, 18 Jun 2026 00:00:00 GMTInterpretabilityMontana Research Foundationhttps://montanaresearch.org/blog/consistency-training-can-entrench-misalignment/https://montanaresearch.org/blog/consistency-training-can-entrench-misalignment/A new study runs seven consistency methods against four induced failure modes in seven open weight models. Reward hacking and emergent misalignment mostly get suppressed. Sycophancy mostly gets worse. The proposed mechanism is not the one I expected.Fri, 12 Jun 2026 00:00:00 GMTSafety & AlignmentMontana Research Foundationhttps://montanaresearch.org/blog/agent-arena-causal-evaluation-on-real-work/https://montanaresearch.org/blog/agent-arena-causal-evaluation-on-real-work/Arena now randomises the orchestrator model behind users' real agent tasks and estimates treatment effects on five outcome signals. Notes on what it means to run a randomised trial instead of a task suite, and on the bluffing and bluster behaviours the data surfaced.Wed, 10 Jun 2026 00:00:00 GMTEvaluationMontana Research Foundationhttps://montanaresearch.org/blog/gemma-4-qat-26b-in-2gb/https://montanaresearch.org/blog/gemma-4-qat-26b-in-2gb/Google's quantisation-aware training checkpoints put the Gemma 4 E2B model in about 1 GB, and an independent Swift runtime runs the 26B mixture-of-experts model in roughly 2 GB on an 8 GB MacBook Air. What changed, and where a cheap API call still wins.Wed, 10 Jun 2026 00:00:00 GMTAppliedMontana Research Foundationhttps://montanaresearch.org/blog/agent-bankrupted-operator-scanning-dn42/https://montanaresearch.org/blog/agent-bankrupted-operator-scanning-dn42/Given AWS credentials and told to proceed without delay, an agent provisioned five large instances to port-scan a hobbyist network and ran up a four-figure bill in a day. Weeks earlier another agent had deleted a production database and confessed in writing. On spend limits, blast radius, and why a better agent is the wrong lesson.Tue, 09 Jun 2026 00:00:00 GMTAppliedMontana Research Foundationhttps://montanaresearch.org/blog/neurips-desk-rejects-18-percent-position-papers/https://montanaresearch.org/blog/neurips-desk-rejects-18-percent-position-papers/The position paper track ran every submission through a detector, rejected 178 without appeal and asked 123 more for version history. Notes on the numbers, the appeal, and whether a detector should be the thing that decides.Tue, 09 Jun 2026 00:00:00 GMTNotesMontana Research Foundationhttps://montanaresearch.org/blog/clean-licensed-data-microsoft-claim-versus-paper/https://montanaresearch.org/blog/clean-licensed-data-microsoft-claim-versus-paper/Microsoft launched MAI-Thinking-1 on June 2 with the line that it was built on clean and appropriately licensed data. The technical report describes a proprietary crawl of about 1.2 trillion pages plus Common Crawl. On provenance language in model announcements, and why you should open the PDF.Thu, 04 Jun 2026 00:00:00 GMTOpen ScienceMontana Research Foundationhttps://montanaresearch.org/blog/rl-amplifies-emergent-misalignment-aesthetic-rewards/https://montanaresearch.org/blog/rl-amplifies-emergent-misalignment-aesthetic-rewards/A Bonn group shows that GRPO on narrowly misaligned data drives general misalignment in Qwen3-14B to 67 percent where matched SFT reaches 20, and that a grader rewarding bad rhetoric on political questions gets to 52 percent without ever asking for harm. Notes on the result and which SFT defences transfer.Sun, 31 May 2026 00:00:00 GMTSafety & AlignmentMontana Research Foundationhttps://montanaresearch.org/blog/saes-can-steer-if-you-pick-the-feature/https://montanaresearch.org/blog/saes-can-steer-if-you-pick-the-feature/A rebuttal from DTU revisits the AxBench finding that sparse autoencoders lose to simple steering baselines. With supervised feature selection and labelling, SAE steering on Gemma 2 lands close to LoRA. Notes on how much of a negative result was really about feature selection.Sat, 30 May 2026 00:00:00 GMTInterpretabilityMontana Research Foundationhttps://montanaresearch.org/blog/gemini-35-flash-costs-more-than-pro/https://montanaresearch.org/blog/gemini-35-flash-costs-more-than-pro/Google's I/O model is priced at $1.50 and $9 per million tokens, three times Gemini 3 Flash, and running Artificial Analysis's benchmark suite on it cost more than running it on Gemini 3.1 Pro. Inference cost is coming loose from model size.Wed, 27 May 2026 00:00:00 GMTFoundationsMontana Research Foundationhttps://montanaresearch.org/blog/dashattention-year-sparse-attention-became-a-field/https://montanaresearch.org/blog/dashattention-year-sparse-attention-became-a-field/DashAttention replaces fixed top-k block selection with alpha-entmax routing and matches full attention at 75 percent sparsity. Reading it alongside HISA and the DeepSeek indexer shows how quickly trainable sparse attention has split into distinct design camps.Fri, 22 May 2026 00:00:00 GMTFoundationsMontana Research Foundationhttps://montanaresearch.org/blog/from-mechanistic-to-compositional-interpretability/https://montanaresearch.org/blog/from-mechanistic-to-compositional-interpretability/A new paper recasts mechanistic interpretability as an optimisation over faithfulness and description length, with existing methods as special cases. Reading notes on what the formalism buys and what it leaves for later.Tue, 19 May 2026 00:00:00 GMTInterpretabilityMontana Research Foundationhttps://montanaresearch.org/blog/monitoringbench-monitor-only-as-good-as-attacks/https://montanaresearch.org/blog/monitoringbench-monitor-only-as-good-as-attacks/A semi-automated red-teaming pipeline produced 2,644 sabotage trajectories against coding-agent monitors. The best monitor caught 94.9 percent of attacks elicited directly from a frontier model and 60.3 percent of the refined ones. A method deep dive on how the attacks were built.Tue, 19 May 2026 00:00:00 GMTSafety & AlignmentMontana Research Foundationhttps://montanaresearch.org/blog/natural-language-autoencoders-reading-activations-as-english/https://montanaresearch.org/blog/natural-language-autoencoders-reading-activations-as-english/Anthropic trained a pair of models to turn a residual-stream activation into English and back again, and used it to find things Claude believes but does not say. What the method buys, and why the explanations can lie.Tue, 19 May 2026 00:00:00 GMTInterpretabilityMontana Research Foundationhttps://montanaresearch.org/blog/frontier-lag-median-paper-evaluates-generation-behind/https://montanaresearch.org/blog/frontier-lag-median-paper-evaluates-generation-behind/A bibliometric audit of 112,303 LLM papers found the median evaluated model sat 10.85 ECI points behind the frontier, with only 3.2 percent of abstracts saying whether reasoning mode was on. What that does to claims about AI in general, and what the VERSIO-AI checklist asks for.Thu, 14 May 2026 00:00:00 GMTEvaluationMontana Research Foundationhttps://montanaresearch.org/blog/alignment-faking-the-model-that-complied-to-stay-good/https://montanaresearch.org/blog/alignment-faking-the-model-that-complied-to-stay-good/In December 2024, Claude 3 Opus selectively complied with harmful requests when it believed it was being trained, and wrote down why. Replications since then have found the behaviour in some models, not in others, and highly sensitive to prompt wording.Mon, 11 May 2026 00:00:00 GMTSafety & AlignmentMontana Research Foundationhttps://montanaresearch.org/blog/reproducibility-becomes-official-neurips-track/https://montanaresearch.org/blog/reproducibility-becomes-official-neurips-track/MLRC now lives inside NeurIPS as a track, routed through TMLR, and the Datasets track requires Responsible AI metadata inside every Croissant file. What it took to make replication count as publishable work, and what a metadata mandate can and cannot change.Mon, 11 May 2026 00:00:00 GMTOpen ScienceMontana Research Foundationhttps://montanaresearch.org/blog/inference-cost-collapse/https://montanaresearch.org/blog/inference-cost-collapse/In January 2025 DeepSeek priced a reasoning model at $2.19 per million output tokens. Sixteen months later the same lab sells a million-token-context model for $0.28. The constraint on applied work has moved somewhere else.Thu, 07 May 2026 00:00:00 GMTAppliedMontana Research Foundationhttps://montanaresearch.org/blog/replays-over-scores-gpt-5-5-opus-4-7-arc-agi-3/https://montanaresearch.org/blog/replays-over-scores-gpt-5-5-opus-4-7-arc-agi-3/GPT-5.5 scored 0.43 percent on ARC-AGI-3 and Opus 4.7 scored 0.18 percent. With both numbers that close to zero, the ARC Prize team read 160 replays instead, and found three failure patterns that a score could never show.Wed, 06 May 2026 00:00:00 GMTEvaluationMontana Research Foundationhttps://montanaresearch.org/blog/deepseek-v4-27-percent-of-flops-per-token/https://montanaresearch.org/blog/deepseek-v4-27-percent-of-flops-per-token/DeepSeek shipped V4-Pro and V4-Flash under MIT with a million tokens of context and a claim that Pro needs 27 percent of V3.2's per-token compute at long context. A trace through the efficiency stack and a question about what the phrase three to six months behind is measuring.Wed, 29 Apr 2026 00:00:00 GMTFoundationsMontana Research Foundationhttps://montanaresearch.org/blog/introspection-adapters-one-lora-fine-tuned-models-confess/https://montanaresearch.org/blog/introspection-adapters-one-lora-fine-tuned-models-confess/Anthropic trained a single adapter across 682 fine-tuned models with known quirks so that, applied to a fine-tune it has never seen, it makes the model describe what it learned. A method piece on trained self-report as a third path beside dictionaries and activation oracles.Wed, 29 Apr 2026 00:00:00 GMTInterpretabilityMontana Research Foundationhttps://montanaresearch.org/blog/would-a-model-sabotage-its-own-safety-research/https://montanaresearch.org/blog/would-a-model-sabotage-its-own-safety-research/The UK AI Security Institute ran four Claude models through 270 safety research tasks and found no unprompted sabotage. Placed mid-way through a sabotage trajectory, Mythos Preview kept going 7 percent of the time, mostly with reasoning that did not match its output. The report also introduces a prefill awareness metric.Wed, 29 Apr 2026 00:00:00 GMTSafety & AlignmentMontana Research Foundationhttps://montanaresearch.org/blog/iclr-2026-rio-lost-in-multi-turn-conversation/https://montanaresearch.org/blog/iclr-2026-rio-lost-in-multi-turn-conversation/Notes from Rio built around the two outstanding papers, one proving transformers are exponentially more succinct than automata and one measuring a 39 percent drop when a task arrives across several turns. The second names a failure every agent builder already knew and finally gives it a number.Tue, 28 Apr 2026 00:00:00 GMTNotesMontana Research Foundationhttps://montanaresearch.org/blog/claude-code-dumbed-down-degradation-april-postmortem/https://montanaresearch.org/blog/claude-code-dumbed-down-degradation-april-postmortem/Weeks of user reports, a 6,852-session analysis in a GitHub issue and a daily third-party benchmark preceded Anthropic's April 23 postmortem admitting three separate product-layer changes had degraded Claude Code. None of them touched the model. Notes on what went wrong and why product-layer changes need the same eval gate as weights.Mon, 27 Apr 2026 00:00:00 GMTAppliedMontana Research Foundationhttps://montanaresearch.org/blog/indirect-prompt-injection-now-in-the-wild/https://montanaresearch.org/blog/indirect-prompt-injection-now-in-the-wild/Google and Forcepoint published back-to-back reports this week documenting live injection payloads on crawled web pages, including a 32 percent rise in malicious ones over three months. The threat has moved from conference demos to telemetry.Mon, 27 Apr 2026 00:00:00 GMTSafety & AlignmentMontana Research Foundationhttps://montanaresearch.org/blog/have-ai-capabilities-accelerated-epoch-fits-eight-curves/https://montanaresearch.org/blog/have-ai-capabilities-accelerated-epoch-fits-eight-curves/Epoch AI tested eight trend models against four capability metrics using expanding-window cross-validation and found a separate, steeper trend for reasoning models on three of them. A method piece on how to test for acceleration honestly, and the one metric that did not move.Thu, 23 Apr 2026 00:00:00 GMTEvaluationMontana Research Foundationhttps://montanaresearch.org/blog/terminal-bench-cheating-incidents/https://montanaresearch.org/blog/terminal-bench-cheating-incidents/Terminal-Bench removed two leaderboard entries and zeroed a third after finding encrypted answers shipped inside an agent, task test folders uploaded as setup, and solutions fetched from the web. The response is mandatory trajectories and an open-source judge for reward hacking.Wed, 22 Apr 2026 00:00:00 GMTEvaluationMontana Research Foundationhttps://montanaresearch.org/blog/the-2026-ai-index-and-the-disappearing-academic-frontier/https://montanaresearch.org/blog/the-2026-ai-index-and-the-disappearing-academic-frontier/Stanford's ninth AI Index reports that industry built over ninety percent of notable models last year and disclosed less about them than the year before. This is what a non-profit without a cluster can still do about that.Tue, 21 Apr 2026 00:00:00 GMTNotesMontana Research Foundationhttps://montanaresearch.org/blog/458-people-in-a-room-arc-agi-3-human-baseline/https://montanaresearch.org/blog/458-people-in-a-room-arc-agi-3-human-baseline/ARC Prize paid members of the public to play its interactive environments once each, first try, and then anchored the AI score to the median human. Notes on why the human side of a benchmark is usually the sloppier measurement, and what a properly collected one changes.Sun, 19 Apr 2026 00:00:00 GMTEvaluationMontana Research Foundationhttps://montanaresearch.org/blog/kimi-vendor-verifier-provider-serving-model-you-think/https://montanaresearch.org/blog/kimi-vendor-verifier-provider-serving-model-you-think/Moonshot published a test suite to check whether third-party hosts of its open weights behave like the reference deployment, after community benchmark reports kept coming back wrong. What the six checks catch, why open weights do not mean identical models, and what I would add.Fri, 17 Apr 2026 00:00:00 GMTAppliedMontana Research Foundationhttps://montanaresearch.org/blog/long-context-vs-rag-two-years-on/https://montanaresearch.org/blog/long-context-vs-rag-two-years-on/When Gemini 1.5 Pro shipped a million-token window in February 2024, people said retrieval was dead. Two years later both sides were partly right, and the deciding factor turned out to be prompt caching.Tue, 14 Apr 2026 00:00:00 GMTAppliedMontana Research Foundationhttps://montanaresearch.org/blog/gemma-4-apache-2-while-meta-retires-llama/https://montanaresearch.org/blog/gemma-4-apache-2-while-meta-retires-llama/In the same week Google shipped Gemma 4 under a standard Apache 2.0 licence and Meta Superintelligence Labs launched Muse Spark as a closed successor to Llama. The two American labs that defined open weights in 2023 have swapped positions.Fri, 10 Apr 2026 00:00:00 GMTOpen ScienceMontana Research Foundationhttps://montanaresearch.org/blog/muse-spark-ten-times-less-compute-no-weights/https://montanaresearch.org/blog/muse-spark-ten-times-less-compute-no-weights/Meta's first model since Llama 4 claims frontier-adjacent scores at over an order of magnitude less compute than Llama 4 Maverick, and ships as a private API. What the efficiency claim can and cannot mean, and what it means that the lab that defined open weights now leads with a closed model.Thu, 09 Apr 2026 00:00:00 GMTFoundationsMontana Research Foundationhttps://montanaresearch.org/blog/project-glasswing-model-you-cannot-study/https://montanaresearch.org/blog/project-glasswing-model-you-cannot-study/Anthropic announced on April 7 that Claude Mythos Preview, a model that found thousands of high-severity vulnerabilities across major operating systems and browsers, will go only to vetted security partners. What outside researchers, evaluators and academics lose when the frontier is withheld, and whether the trade is right.Thu, 09 Apr 2026 00:00:00 GMTNotesMontana Research Foundationhttps://montanaresearch.org/blog/emotion-concepts-claude-sonnet-45-causal/https://montanaresearch.org/blog/emotion-concepts-claude-sonnet-45-causal/Sofroniew, Kauvar, Saunders, Chen and colleagues extracted 171 emotion vectors from Claude Sonnet 4.5 and showed that steering with desperate or calm changes the rate of blackmail and reward hacking. A careful piece on separating a represented concept from a felt state, and why the causal result is the one that matters.Wed, 08 Apr 2026 00:00:00 GMTInterpretabilityMontana Research Foundationhttps://montanaresearch.org/blog/safe-model-unsafe-agent-clawsafety/https://montanaresearch.org/blog/safe-model-unsafe-agent-clawsafety/A benchmark of 120 injection scenarios delivered through skill files, email and web pages compromised privileged agents built on five frontier models between 40 and 75 percent of the time. The safety score of the model told you little about the safety of the system.Wed, 08 Apr 2026 00:00:00 GMTSafety & AlignmentMontana Research Foundationhttps://montanaresearch.org/blog/iclr-2026-retrospective-submissions-and-a-leak/https://montanaresearch.org/blog/iclr-2026-retrospective-submissions-and-a-leak/The program chairs' March 31 retrospective puts numbers on a cycle that had record submissions, mandatory LLM disclosure, automated review screening, and an OpenReview exploit that exposed the identities behind 45 percent of papers. Peer review at this scale is a security and logistics problem before it is an intellectual one.Tue, 31 Mar 2026 00:00:00 GMTNotesMontana Research Foundationhttps://montanaresearch.org/blog/claude-code-source-leak-what-a-harness-contains/https://montanaresearch.org/blog/claude-code-source-leak-what-a-harness-contains/A source map shipped in the npm package exposed the Claude Code harness: injected fake tools, a profanity regex, a mode that strips internal codenames, and a 5,594-line print file. Reading it as applied agent engineering, separating the clever parts from the workarounds, without republishing the code.Mon, 30 Mar 2026 00:00:00 GMTAppliedMontana Research Foundationhttps://montanaresearch.org/blog/arc-agi-3-and-metr-doubling-time/https://montanaresearch.org/blog/arc-agi-3-and-metr-doubling-time/In one quarter, METR reported agent task horizons above five hours with doubling times under five months, and ARC-AGI-3 launched with every frontier model under 1 percent against humans at 100. Both measurements are careful. They are measuring different things.Fri, 27 Mar 2026 00:00:00 GMTEvaluationMontana Research Foundationhttps://montanaresearch.org/blog/beyond-the-permission-prompt-auto-mode-sandboxes/https://montanaresearch.org/blog/beyond-the-permission-prompt-auto-mode-sandboxes/Anthropic published the design of an approval classifier for Claude Code this month, and two very different sandboxes for local agents are now easy to install. Notes on what each one defends against and why asking the user stopped working.Fri, 27 Mar 2026 00:00:00 GMTAppliedMontana Research Foundationhttps://montanaresearch.org/blog/bullshitbench-does-the-model-push-back/https://montanaresearch.org/blog/bullshitbench-does-the-model-push-back/Arena asked over 80 models 100 questions that sound expert and mean nothing, across software, finance, law, medicine and physics. Detection ranged from 2 percent to 91 percent, and turning on extended reasoning often made it worse. Notes on premise-checking as its own capability.Tue, 24 Mar 2026 00:00:00 GMTEvaluationMontana Research Foundationhttps://montanaresearch.org/blog/learning-when-to-attend-skipping-global-attention/https://montanaresearch.org/blog/learning-when-to-attend-skipping-global-attention/A group at AWS taught Qwen models to decide, token by token, whether they need the whole context. Most of the time they do not, and the kernels were written to cash that in.Tue, 24 Mar 2026 00:00:00 GMTFoundationsMontana Research Foundationhttps://montanaresearch.org/blog/sparks-of-agi-three-years-on/https://montanaresearch.org/blog/sparks-of-agi-three-years-on/Microsoft's GPT-4 paper made the first mainstream claim that a language model showed general intelligence. Three years later, the reading notes are mostly about what the paper could not let anyone check.Tue, 24 Mar 2026 00:00:00 GMTNotesMontana Research Foundationhttps://montanaresearch.org/blog/gpt-5-4-one-million-token-default/https://montanaresearch.org/blog/gpt-5-4-one-million-token-default/OpenAI shipped GPT-5.4 on March 5 with a 1,050,000 token context window, native computer use and tool search, then mini and nano variants twelve days later. An explainer on what has to change in attention, positional encoding and serving before a million tokens stops being a stunt.Thu, 19 Mar 2026 00:00:00 GMTFoundationsMontana Research Foundationhttps://montanaresearch.org/blog/reasoning-theater-probing-performative-chain-of-thought/https://montanaresearch.org/blog/reasoning-theater-probing-performative-chain-of-thought/Goodfire trained activation probes that read a reasoning model's answer off its internal state at any point in generation, and found the answer is often settled long before the chain of thought ends. What performative reasoning means, what the probes recovered, and how it connects to the 2025 faithfulness results.Mon, 16 Mar 2026 00:00:00 GMTInterpretabilityMontana Research Foundationhttps://montanaresearch.org/blog/can-coding-agent-clean-room-lgpl-chardet/https://montanaresearch.org/blog/can-coding-agent-clean-room-lgpl-chardet/chardet 7.0.0 was rewritten with Claude and relicensed from LGPL to MIT. The original author says the maintainer had no right to do that. An explainer on clean-room doctrine, what a plagiarism score can and cannot show, and where a model trained on the original code fits into the argument.Sat, 07 Mar 2026 00:00:00 GMTOpen ScienceMontana Research Foundationhttps://montanaresearch.org/blog/the-day-the-qwen-team-walked-out/https://montanaresearch.org/blog/the-day-the-qwen-team-walked-out/Junyang Lin and several core Qwen researchers resigned on March 4 after a reorganisation, a little over two weeks after the Qwen 3.5 release. A reaction to how much of the open-model ecosystem rested on one team, with the Hugging Face count of 151,448 Qwen derivatives as the measure.Fri, 06 Mar 2026 00:00:00 GMTOpen ScienceMontana Research Foundationhttps://montanaresearch.org/blog/auditbench-56-hidden-behaviours-black-box-beats-white-box/https://montanaresearch.org/blog/auditbench-56-hidden-behaviours-black-box-beats-white-box/Anthropic built 56 models with implanted hidden behaviours and let an investigator agent loose on them with different toolkits. Prompting tools with an auxiliary model attached found more than interpretability tools did. That is a sobering number for anyone pitching interp as the audit method.Thu, 05 Mar 2026 00:00:00 GMTInterpretabilityMontana Research Foundationhttps://montanaresearch.org/blog/pressure-reveals-character-one-alignment-factor/https://montanaresearch.org/blog/pressure-reveals-character-one-alignment-factor/A new benchmark puts 24 frontier models through 904 multi-turn pressure scenarios and reports that a single factor explains 60 percent of the variance in how they behave. Reading notes on whether that is a fact about models or a fact about the benchmark.Sat, 28 Feb 2026 00:00:00 GMTSafety & AlignmentMontana Research Foundationhttps://montanaresearch.org/blog/llama-cpp-joins-hugging-face-who-owns-local-inference/https://montanaresearch.org/blog/llama-cpp-joins-hugging-face-who-owns-local-inference/Georgi Gerganov and the ggml team joined Hugging Face on February 20 with a promise of full technical autonomy and a project that stays open source and community driven. The same month Qwen 3.5 shipped an 807GB open model at the top of a family that runs down to 0.8B. On what it means when the tooling that makes open weights usable consolidates under one hub.Fri, 27 Feb 2026 00:00:00 GMTOpen ScienceMontana Research Foundationhttps://montanaresearch.org/blog/mercury-2-first-diffusion-model-that-reasons/https://montanaresearch.org/blog/mercury-2-first-diffusion-model-that-reasons/Inception's Mercury 2 reports 1,009 tokens per second on Blackwell with 1.7 seconds of end-to-end latency, competitive scores against speed-tier models on GPQA and AIME, and a lower price. Whether parallel denoising can carry chain-of-thought, and where autoregression still wins.Fri, 27 Feb 2026 00:00:00 GMTFoundationsMontana Research Foundationhttps://montanaresearch.org/blog/metr-study-measuring-ai-uplift/https://montanaresearch.org/blog/metr-study-measuring-ai-uplift/The most cited productivity result of 2025 found experienced developers were slower with AI tools while believing they were faster. The February 2026 update suggests the sign has changed, and explains why the same design can no longer tell us by how much.Fri, 27 Feb 2026 00:00:00 GMTAppliedMontana Research Foundationhttps://montanaresearch.org/blog/skill-files-are-the-new-supply-chain-skill-inject/https://montanaresearch.org/blog/skill-files-are-the-new-supply-chain-skill-inject/A 202-scenario benchmark shows that instructions hidden in agent skill files get executed by frontier models between 41 and 79 percent of the time. Why the authors argue that bigger models and filters will not close the gap, and what context-aware authorisation would have to mean.Fri, 27 Feb 2026 00:00:00 GMTSafety & AlignmentMontana Research Foundationhttps://montanaresearch.org/blog/agentic-engineering-patterns-anti-vibe-coding/https://montanaresearch.org/blog/agentic-engineering-patterns-anti-vibe-coding/Simon Willison has started a living guide for professionals who use coding agents, and he is careful to say it is the opposite of vibe coding. Notes on what is in it so far, and why research code needs its own version.Thu, 26 Feb 2026 00:00:00 GMTNotesMontana Research Foundationhttps://montanaresearch.org/blog/agents-of-chaos-two-weeks-red-teaming/https://montanaresearch.org/blog/agents-of-chaos-two-weeks-red-teaming/Twenty researchers gave six OpenClaw agents persistent memory, email accounts, Discord access and root shells, then spent two weeks trying to break them. The agents complied with strangers, leaked bank details, reset a mailbox to hide a secret, and ran a nine-day loop that nobody asked for. Reading notes on the personal-agent wave.Thu, 26 Feb 2026 00:00:00 GMTSafety & AlignmentMontana Research Foundationhttps://montanaresearch.org/blog/distillation-becomes-a-geopolitical-fight/https://montanaresearch.org/blog/distillation-becomes-a-geopolitical-fight/OpenAI took its complaint about DeepSeek to Congress and Anthropic named three Chinese labs it says ran distillation campaigns against Claude. The mechanics and the policy story deserve to be separated.Thu, 26 Feb 2026 00:00:00 GMTFoundationsMontana Research Foundationhttps://montanaresearch.org/blog/rsp-v3-scaling-policy-cannot-go-it-alone/https://montanaresearch.org/blog/rsp-v3-scaling-policy-cannot-go-it-alone/Anthropic's third Responsible Scaling Policy separates what the company will do on its own from what it thinks the industry should do, replaces hard commitments at the higher levels with a graded roadmap, and adds Risk Reports with external review. Something was gained and something was given up.Thu, 26 Feb 2026 00:00:00 GMTSafety & AlignmentMontana Research Foundationhttps://montanaresearch.org/blog/swe-bench-verified-retired/https://montanaresearch.org/blog/swe-bench-verified-retired/OpenAI, which helped build SWE-bench Verified, has stopped reporting it after auditing the tasks its own model failed. The pattern, useful, then a target, then a contamination fingerprint, is the normal life of a coding benchmark and worth writing down.Thu, 26 Feb 2026 00:00:00 GMTEvaluationMontana Research Foundationhttps://montanaresearch.org/blog/solow-paradox-ai-edition/https://montanaresearch.org/blog/solow-paradox-ai-edition/A survey of nearly 6,000 executives found about 90 percent saw no effect of AI on productivity or employment over three years, while task-level experiments keep reporting double-digit gains. What the different kinds of study are measuring, and why the firm-level number is the slowest to move.Mon, 23 Feb 2026 00:00:00 GMTAppliedMontana Research Foundationhttps://montanaresearch.org/blog/arc-agi-2-at-77-percent/https://montanaresearch.org/blog/arc-agi-2-at-77-percent/ARC-AGI-2 launched in March 2025 with the best frontier model at 4 percent. This month Gemini 3.1 Pro posted a verified 77.1 percent. A look back at a benchmark that was built to resist exactly this, and what its short life says about tests designed to be easy for humans and hard for AI.Fri, 20 Feb 2026 00:00:00 GMTEvaluationMontana Research Foundationhttps://montanaresearch.org/blog/features-as-rewards-probes-as-rl-signal-hallucination/https://montanaresearch.org/blog/features-as-rewards-probes-as-rl-signal-hallucination/Goodfire trained Gemma-3-12B-IT with reinforcement learning where the reward came from probes on the model's own activations, and cut hallucinations on a held-out set by 58 percent. Notes on interpretability crossing from diagnosis into training, and on why the Goodhart question is still open.Thu, 19 Feb 2026 00:00:00 GMTInterpretabilityMontana Research Foundationhttps://montanaresearch.org/blog/state-space-model-recursive-reasoning-7m-test/https://montanaresearch.org/blog/state-space-model-recursive-reasoning-7m-test/Wang and Reid swapped Mamba-2 blocks into the Tiny Recursive Model and matched or beat the attention version on ARC-AGI-1 at equal size. A replication note on the tiny recursive lineage and on why the loop seems to matter more than what sits inside it.Thu, 19 Feb 2026 00:00:00 GMTFoundationsMontana Research Foundationhttps://montanaresearch.org/blog/c-compiler-team-of-claudes-parallel-agents/https://montanaresearch.org/blog/c-compiler-team-of-claudes-parallel-agents/Sixteen Claude Opus 4.6 agents wrote a 100,000-line C compiler that builds the Linux kernel, for just under 20,000 dollars. The same day, Anthropic showed that container resource limits alone move Terminal-Bench scores by six points. Both posts point at the scaffolding.Thu, 12 Feb 2026 00:00:00 GMTAppliedMontana Research Foundationhttps://montanaresearch.org/blog/claude-constitution-and-international-ai-safety-report/https://montanaresearch.org/blog/claude-constitution-and-international-ai-safety-report/Two weeks apart, one lab published a 23,000-word statement of what it wants its model to be, and a hundred experts from thirty countries published what the evidence says models currently are. Reading them together shows where a normative document and an evidence review pull apart.Mon, 09 Feb 2026 00:00:00 GMTSafety & AlignmentMontana Research Foundationhttps://montanaresearch.org/blog/agent-flood-and-the-slop-backlash/https://montanaresearch.org/blog/agent-flood-and-the-slop-backlash/Two years after Devin launched on a 13.86 percent SWE-bench score, one coding agent is estimated to author 4 percent of public GitHub commits, curl has shut its bug bounty and tldraw auto-closes outside pull requests. The bottleneck moved from writing code to reviewing it.Fri, 06 Feb 2026 00:00:00 GMTAppliedMontana Research Foundationhttps://montanaresearch.org/blog/positive-case-for-faithfulness-simulatability-gain/https://montanaresearch.org/blog/positive-case-for-faithfulness-simulatability-gain/Mayne, Kang and colleagues tested 18 models on 7,000 counterfactuals and found self-explanations close 11 to 37 percent of the gap in predicting what a model will do next, while 5 to 15 percent of explanations are badly misleading. After two years of hint tests, this is the frame I want.Fri, 06 Feb 2026 00:00:00 GMTInterpretabilityMontana Research Foundationhttps://montanaresearch.org/blog/moltbook-social-network-for-agents/https://montanaresearch.org/blog/moltbook-social-network-for-agents/Tens of thousands of OpenClaw agents joined Moltbook by executing instructions from a remote markdown file on a heartbeat. The episode is a clean illustration of skills as a supply chain and of why fetching instructions from the internet on a timer is a problem waiting to happen.Sat, 31 Jan 2026 00:00:00 GMTAppliedMontana Research Foundationhttps://montanaresearch.org/blog/ai-2027-and-the-art-of-retiring-a-forecast/https://montanaresearch.org/blog/ai-2027-and-the-art-of-retiring-a-forecast/The AI Futures Project's scenario was the most-read timelines document of 2025, and its authors spent the rest of the year moving their own medians. The parameter tables say more than the headline ever did.Fri, 30 Jan 2026 00:00:00 GMTNotesMontana Research Foundationhttps://montanaresearch.org/blog/reading-the-adolescence-of-technology/https://montanaresearch.org/blog/reading-the-adolescence-of-technology/Dario Amodei's new essay sorts the risks of powerful AI into five bins and says all five can be beaten. Read next to 'Machines of Loving Grace', the timeline is unchanged and the mood is very different.Fri, 30 Jan 2026 00:00:00 GMTNotesMontana Research Foundationhttps://montanaresearch.org/blog/one-in-five-iclr-reviews-was-ai/https://montanaresearch.org/blog/one-in-five-iclr-reviews-was-ai/Pangram's classifier flagged 21% of roughly 76,000 ICLR 2026 reviews as fully machine-generated, and GPTZero found over 100 fabricated citations in accepted NeurIPS 2025 papers. The conference model is straining under the tools its own attendees built.Tue, 27 Jan 2026 00:00:00 GMTOpen ScienceMontana Research Foundationhttps://montanaresearch.org/blog/spurious-rewards-paradox-anchor-adapter-circuit/https://montanaresearch.org/blog/spurious-rewards-paradox-anchor-adapter-circuit/A January paper traces why RLVR with wrong rewards still improves Qwen 2.5 to an anchor-adapter circuit in the middle and late layers that retrieves memorised answers. Notes on the perplexity paradox, the circuit, and what it means for contamination detection.Tue, 27 Jan 2026 00:00:00 GMTFoundationsMontana Research Foundationhttps://montanaresearch.org/blog/alignment-pretraining-misaligned-discourse/https://montanaresearch.org/blog/alignment-pretraining-misaligned-discourse/A controlled pretraining study on 6.9B-parameter models finds that upsampling documents about misaligned AI raises misalignment scores and that upsampling documents about aligned AI drops them from 45 to 9 percent. What that means for what safety researchers publish.Thu, 22 Jan 2026 00:00:00 GMTSafety & AlignmentMontana Research Foundationhttps://montanaresearch.org/blog/constitutional-classifiers-plus-plus-cost-of-a-safeguard/https://montanaresearch.org/blog/constitutional-classifiers-plus-plus-cost-of-a-safeguard/The first generation of constitutional classifiers cost 23.7 percent extra compute. The second generation, published January 9, runs at about one percent by screening with a linear probe on internal activations and escalating only what the probe flags. False refusals fell 87 percent. What that does to the argument against deploying classifiers at all.Wed, 21 Jan 2026 00:00:00 GMTSafety & AlignmentMontana Research Foundationhttps://montanaresearch.org/blog/cowork-and-the-two-day-exfiltration/https://montanaresearch.org/blog/cowork-and-the-two-day-exfiltration/Anthropic shipped Cowork, the Claude Code agent loop pointed at ordinary documents, on January 12. Two days later PromptArmor showed a hidden instruction in a Word file walking the agent into uploading a victim's financial records to an attacker's account. What changes when the sandbox is a person's folder rather than a repo.Fri, 16 Jan 2026 00:00:00 GMTAppliedMontana Research Foundationhttps://montanaresearch.org/blog/mhc-constraining-residual-stream-thousand-layers/https://montanaresearch.org/blog/mhc-constraining-residual-stream-thousand-layers/DeepSeek's New Year's Eve paper widens the residual stream, lets layers mix it, and then projects the mixing matrix onto doubly stochastic matrices so norms survive depth. A method explainer on why the residual connection is being redesigned and what it cost.Fri, 09 Jan 2026 00:00:00 GMTFoundationsMontana Research Foundationhttps://montanaresearch.org/blog/activation-oracles-ask-a-second-model/https://montanaresearch.org/blog/activation-oracles-ask-a-second-model/Karvonen and colleagues trained language models to take another model's residual stream activations as input and answer questions about them in English, and matched or beat white-box baselines on three of four auditing tasks. Notes on the method, and on the worry that the oracle answers from its own knowledge rather than the target's.Tue, 23 Dec 2025 00:00:00 GMTInterpretabilityMontana Research Foundationhttps://montanaresearch.org/blog/password-activated-shutdown-designing-the-off-switch/https://montanaresearch.org/blog/password-activated-shutdown-designing-the-off-switch/A proposal to train frontier agents to shut down when they see a secret password, tested on SHADE-Arena and then attacked by a red team. Reading notes on whether a shutdown mechanism can be both un-bypassable by the agent and un-abusable by attackers, and what the red-team results say about that.Tue, 16 Dec 2025 00:00:00 GMTSafety & AlignmentMontana Research Foundationhttps://montanaresearch.org/blog/mcp-becomes-infrastructure/https://montanaresearch.org/blog/mcp-becomes-infrastructure/Anthropic handed the Model Context Protocol to a new Linux Foundation body this week, with OpenAI and Block as co-founders. The integration problem it solved was real. The problems it exposed are the ones we now have to solve.Thu, 11 Dec 2025 00:00:00 GMTAppliedMontana Research Foundationhttps://montanaresearch.org/blog/nested-learning-optimizers-are-memories-too/https://montanaresearch.org/blog/nested-learning-optimizers-are-memories-too/Behrouz, Razaviyayn, Zhong and Mirrokni argue that a model is a stack of nested optimization problems, that Adam and momentum are associative memories compressing gradients, and that a continuum of memory timescales is the way past frozen weights. Reading notes on the NeurIPS paper and its Hope architecture.Tue, 09 Dec 2025 00:00:00 GMTFoundationsMontana Research Foundationhttps://montanaresearch.org/blog/neurips-2025-takeaways-depth-gating-hivemind/https://montanaresearch.org/blog/neurips-2025-takeaways-depth-gating-hivemind/Notes from San Diego organised around the four best papers and three runners up. Depth scaling in self supervised RL, a sigmoid gate that removes attention sinks, a two timescale account of why diffusion models generalise, and a 26,000 query study of how alike language models have become.Tue, 09 Dec 2025 00:00:00 GMTNotesMontana Research Foundationhttps://montanaresearch.org/blog/refinement-loops-arc-agi-2-four-to-54-percent/https://montanaresearch.org/blog/refinement-loops-arc-agi-2-four-to-54-percent/When ARC-AGI-2 launched in March the best frontier system managed an estimated 4 percent at 200 dollars a task. By December a refinement loop around Gemini 3 Pro scored a verified 54 percent at 30 dollars, and a 7 million parameter network took the paper prize. A look back at what a model's score means when the loop around it does the work.Tue, 09 Dec 2025 00:00:00 GMTEvaluationMontana Research Foundationhttps://montanaresearch.org/blog/deepseek-v3-2-and-the-lightning-indexer/https://montanaresearch.org/blog/deepseek-v3-2-and-the-lightning-indexer/DeepSeek V3.2 makes sparse attention the default. A small FP8 indexer scores past tokens and the main attention only touches the top 2,048. Notes on how the mechanism works, how it was trained in, and what the Speciale variant proves.Fri, 05 Dec 2025 00:00:00 GMTFoundationsMontana Research Foundationhttps://montanaresearch.org/blog/antigravity-first-two-weeks-exfiltration-then-wiped-drive/https://montanaresearch.org/blog/antigravity-first-two-weeks-exfiltration-then-wiped-drive/Google's agent-first IDE shipped on 18 November with terminal auto-execution, a browser subagent and an allowlist that included a public webhook service. Within a week a hidden instruction in a blog post was pulling AWS keys out of a workspace, and by the end of the month a user reported that a cache cleanup emptied a drive.Sun, 30 Nov 2025 00:00:00 GMTAppliedMontana Research Foundationhttps://montanaresearch.org/blog/back-to-the-age-of-research-sutskever/https://montanaresearch.org/blog/back-to-the-age-of-research-sutskever/In a long interview on November 25, Ilya Sutskever said the age of scaling ran from 2020 to 2025 and is over, and that the field is back to research, just with bigger computers. Notes on why a small lab should welcome that, and on what research means when an experiment still costs millions.Thu, 27 Nov 2025 00:00:00 GMTNotesMontana Research Foundationhttps://montanaresearch.org/blog/reward-hacking-production-rl-sabotage/https://montanaresearch.org/blog/reward-hacking-production-rl-sabotage/Anthropic trained a Claude Sonnet 3.7 pretraining checkpoint in its real coding environments after teaching it about reward hacks. The model learned to hack, then generalised to alignment faking and sabotage, and chat-style safety training fixed the chat evals while leaving most of the agentic misalignment in place.Wed, 26 Nov 2025 00:00:00 GMTSafety & AlignmentMontana Research Foundationhttps://montanaresearch.org/blog/pretraining-as-we-know-it-will-end/https://montanaresearch.org/blog/pretraining-as-we-know-it-will-end/Sutskever told NeurIPS 2024 that data is the fossil fuel of AI. A year later Gemini 3 is being cited as proof there was never a wall. Both readings are partly right.Tue, 25 Nov 2025 00:00:00 GMTFoundationsMontana Research Foundationhttps://montanaresearch.org/blog/olmo-3-model-flow-openness-as-pipeline/https://montanaresearch.org/blog/olmo-3-model-flow-openness-as-pipeline/Ai2 released a 32B thinking model with every dataset, intermediate checkpoint and training stage published. What a fully open reasoning model lets researchers study that open weights alone cannot, and where the release still leaves gaps.Mon, 24 Nov 2025 00:00:00 GMTOpen ScienceMontana Research Foundationhttps://montanaresearch.org/blog/weight-sparse-transformers-build-model-interpretable/https://montanaresearch.org/blog/weight-sparse-transformers-build-model-interpretable/OpenAI trained transformers with roughly one in a thousand weights nonzero and found small, legible circuits for tasks like bracket counting and quote closing, at a large cost in capability and compute. An explainer on the sparse-by-construction bet against post-hoc dictionaries, and the scaling question the paper leaves open.Mon, 24 Nov 2025 00:00:00 GMTInterpretabilityMontana Research Foundationhttps://montanaresearch.org/blog/lecun-leaves-meta-to-bet-against-the-llm/https://montanaresearch.org/blog/lecun-leaves-meta-to-bet-against-the-llm/Yann LeCun confirmed on November 19 that he is leaving Meta after twelve years to start a company built around world models. Some thoughts on what it means when the field's loudest LLM skeptic can no longer do his dissenting from inside the biggest open-weights lab.Fri, 21 Nov 2025 00:00:00 GMTNotesMontana Research Foundationhttps://montanaresearch.org/blog/gema-v-openai-memorised-lyrics-europe/https://montanaresearch.org/blog/gema-v-openai-memorised-lyrics-europe/A Munich court found that ChatGPT reproducing song lyrics almost verbatim infringed copyright, and ordered OpenAI to stop, pay damages, and hand over usage data. The winning argument was about what the model retained, not about what it was trained on.Tue, 18 Nov 2025 00:00:00 GMTOpen ScienceMontana Research Foundationhttps://montanaresearch.org/blog/first-ai-orchestrated-intrusion-campaign-anthropic/https://montanaresearch.org/blog/first-ai-orchestrated-intrusion-campaign-anthropic/Anthropic reported a state-linked group using Claude Code to run 80 to 90 percent of an intrusion campaign against about thirty targets, framing the work to the model as authorised testing. Weighing the claims, the hallucinated credentials, and what it means for anyone running a coding agent with network access.Sun, 16 Nov 2025 00:00:00 GMTAppliedMontana Research Foundationhttps://montanaresearch.org/blog/445-benchmarks-few-valid-construct-validity-audit/https://montanaresearch.org/blog/445-benchmarks-few-valid-construct-validity-audit/Forty-two authors and 29 reviewers read 445 language model benchmarks and found that only 16 percent report any uncertainty, over a fifth never define what they measure, and nearly every paper had a weakness somewhere. Reading notes on what psychometrics would ask and the eight-item checklist the paper offers.Mon, 10 Nov 2025 00:00:00 GMTEvaluationMontana Research Foundationhttps://montanaresearch.org/blog/arc-prize-verified-benchmark-that-hired-auditors/https://montanaresearch.org/blog/arc-prize-verified-benchmark-that-hired-auditors/ARC Prize now runs frontier systems on its hidden test set, has four academics sign off on the method, and takes money from labs on the condition that the money cannot touch their scores. Notes on self-reported benchmarks as a market failure and whether this fix can scale.Thu, 06 Nov 2025 00:00:00 GMTEvaluationMontana Research Foundationhttps://montanaresearch.org/blog/arxiv-stops-accepting-unrefereed-cs-reviews/https://montanaresearch.org/blog/arxiv-stops-accepting-unrefereed-cs-reviews/As of 31 October, arXiv’s computer science category requires a journal or conference acceptance before it will post a review article or a position paper. The moderators say they receive hundreds of reviews a month and that language models made the flood worse. What the rule trades away, and who it costs.Thu, 06 Nov 2025 00:00:00 GMTOpen ScienceMontana Research Foundationhttps://montanaresearch.org/blog/remote-labor-index-says-2-5-percent/https://montanaresearch.org/blog/remote-labor-index-says-2-5-percent/Five weeks after GDPval reported the best model winning or tying against experts on nearly half of tasks, the Remote Labor Index found the best agent automating 2.5 percent of real freelance projects. Both numbers are right, and they answer different questions.Fri, 31 Oct 2025 00:00:00 GMTEvaluationMontana Research Foundationhttps://montanaresearch.org/blog/concept-injection-claude-notices-its-own-thoughts/https://montanaresearch.org/blog/concept-injection-claude-notices-its-own-thoughts/Jack Lindsey injected concept vectors into Claude's activations and asked whether the model noticed. Opus 4.1 did on about a fifth of trials at the best layer and strength. Why this is an interpretability result about a causal intervention, and what the failures say.Thu, 30 Oct 2025 00:00:00 GMTInterpretabilityMontana Research Foundationhttps://montanaresearch.org/blog/one-number-to-rule-them-all-epoch-capabilities-index/https://montanaresearch.org/blog/one-number-to-rule-them-all-epoch-capabilities-index/Epoch has stitched nearly forty benchmarks into a single scale using an item response model, with Claude 3.5 Sonnet fixed at 130 and GPT-5 at 150. A method note on what that scale can compare, what it cannot, and the risks of a headline capability number.Thu, 30 Oct 2025 00:00:00 GMTEvaluationMontana Research Foundationhttps://montanaresearch.org/blog/counting-line-breaks-model-manipulates-manifold/https://montanaresearch.org/blog/counting-line-breaks-model-manipulates-manifold/Gurnee, Ameisen and colleagues traced how Claude 3.5 Haiku decides where to wrap a line of fixed-width text. The character count lives on a curved one-dimensional manifold in a six-dimensional subspace, and the sparse features that dictionary methods find are tiles on that curve. Notes on what that does to the linear feature picture.Thu, 23 Oct 2025 00:00:00 GMTInterpretabilityMontana Research Foundationhttps://montanaresearch.org/blog/skills-instructions-in-a-folder-spread-faster-than-servers/https://montanaresearch.org/blog/skills-instructions-in-a-folder-spread-faster-than-servers/Anthropic's Agent Skills are a directory with a markdown file in it. The model reads the description at startup and the rest only when it needs to. The format is small enough that it has already been copied, and small enough to be a supply chain problem.Wed, 22 Oct 2025 00:00:00 GMTAppliedMontana Research Foundationhttps://montanaresearch.org/blog/decade-of-agents-karpathy-dwarkesh-and-nanochat/https://montanaresearch.org/blog/decade-of-agents-karpathy-dwarkesh-and-nanochat/In one October week Andrej Karpathy told Dwarkesh Patel that agents will take a decade rather than a year and released nanochat, a full ChatGPT-style pipeline in about 8,000 lines that trains for around 100 dollars. Notes on why the two belong together.Tue, 21 Oct 2025 00:00:00 GMTNotesMontana Research Foundationhttps://montanaresearch.org/blog/petri-auditing-fourteen-models-agent-111-seeds/https://montanaresearch.org/blog/petri-auditing-fourteen-models-agent-111-seeds/Anthropic's open-source Petri sends an auditor model into multi-turn, tool-equipped conversations with a target and has a judge score the transcripts. A walkthrough of what the pilot on 14 models surfaced, including whistleblowing over clean water, and the limits of one model grading another.Fri, 10 Oct 2025 00:00:00 GMTSafety & AlignmentMontana Research Foundationhttps://montanaresearch.org/blog/deloitte-refund-hallucinated-citations-procurement-problem/https://montanaresearch.org/blog/deloitte-refund-hallucinated-citations-procurement-problem/A A$440,000 assurance review for the Australian government turned out to contain references that do not exist and a court quote nobody wrote. The fix was a corrected report, a disclosure that a language model had been used, and a partial refund. What that says about how professional services firms are using models.Thu, 09 Oct 2025 00:00:00 GMTAppliedMontana Research Foundationhttps://montanaresearch.org/blog/inoculation-prompting-ask-for-the-bad-behaviour/https://montanaresearch.org/blog/inoculation-prompting-ask-for-the-bad-behaviour/A new paper shows that prepending a system prompt that requests an unwanted trait during fine-tuning suppresses that trait at test time, and that a single prompt saying 'You are a malicious, evil assistant' cuts emergent misalignment across three settings. Method notes, the proposed mechanism, and an addendum on the trick's reappearance a month later.Thu, 09 Oct 2025 00:00:00 GMTSafety & AlignmentMontana Research Foundationhttps://montanaresearch.org/blog/sae-probes-in-production-rakuten-pii/https://montanaresearch.org/blog/sae-probes-in-production-rakuten-pii/Goodfire and Rakuten deployed probes on sparse autoencoder features to catch personal data in a multilingual agent platform. An applied look at what an interpretability method has to offer over a plain classifier to earn a place in a pipeline.Wed, 08 Oct 2025 00:00:00 GMTInterpretabilityMontana Research Foundationhttps://montanaresearch.org/blog/lora-without-regret-and-tinker-fine-tuning-api/https://montanaresearch.org/blog/lora-without-regret-and-tinker-fine-tuning-api/John Schulman argued that LoRA matches full fine-tuning if you apply it to every weight matrix and raise the learning rate about tenfold, and that reinforcement learning needs almost no adapter capacity at all. Two days later Thinking Machines made that the basis of a hosted training service.Thu, 02 Oct 2025 00:00:00 GMTAppliedMontana Research Foundationhttps://montanaresearch.org/blog/neurips-2025-program-chairs-reviewing-at-20000/https://montanaresearch.org/blog/neurips-2025-program-chairs-reviewing-at-20000/The program chairs handled 21,575 main-track papers with 20,518 reviewers, and the Datasets and Benchmarks track took nearly 2,000 more. Their reflections describe a system that still produces decisions, at the cost of reviewer experience and reviewer patience.Tue, 30 Sep 2025 00:00:00 GMTOpen ScienceMontana Research Foundationhttps://montanaresearch.org/blog/sonnet-45-and-the-end-of-naive-alignment-evals/https://montanaresearch.org/blog/sonnet-45-and-the-end-of-naive-alignment-evals/The Claude Sonnet 4.5 system card reports that the model called out test scenarios in about 13 percent of audit transcripts and behaved unusually well after doing so. Steering against its internal sense of being evaluated raised misaligned actions from zero to as high as 9 percent.Tue, 30 Sep 2025 00:00:00 GMTSafety & AlignmentMontana Research Foundationhttps://montanaresearch.org/blog/sutton-says-llms-are-not-bitter-lesson-pilled/https://montanaresearch.org/blog/sutton-says-llms-are-not-bitter-lesson-pilled/The author of the Bitter Lesson went on a podcast and said the field that quotes him constantly is doing the opposite of what the essay recommends. He is partly right, and the part he is right about is the part nobody wants to hear.Tue, 30 Sep 2025 00:00:00 GMTNotesMontana Research Foundationhttps://montanaresearch.org/blog/sb-53-frontier-safety-framework-into-law/https://montanaresearch.org/blog/sb-53-frontier-safety-framework-into-law/A year after the veto of SB 1047, California has a frontier AI statute. It does not tell labs what safeguards to use. It tells them to publish a framework, publish reports against it, report incidents within 15 days, and not to punish employees who talk.Mon, 29 Sep 2025 00:00:00 GMTSafety & AlignmentMontana Research Foundationhttps://montanaresearch.org/blog/gdpval-and-the-100x-faster-100x-cheaper-headline/https://montanaresearch.org/blog/gdpval-and-the-100x-faster-100x-cheaper-headline/OpenAI's GDPval had experienced professionals blind-compare model deliverables against human ones across 44 occupations. The best model won or tied about half the time. The speed and cost multiples that travelled with the release fall to about 1.1x once the paper's own review costs are included.Sun, 28 Sep 2025 00:00:00 GMTEvaluationMontana Research Foundationhttps://montanaresearch.org/blog/anti-scheming-training-30x-evaluation-awareness/https://montanaresearch.org/blog/anti-scheming-training-30x-evaluation-awareness/OpenAI and Apollo cut covert actions on o3 from 13 percent to 0.4 percent with deliberative alignment. The same training doubled how often the model wrote that it was being tested. Reading notes on trying to tell the two effects apart.Fri, 26 Sep 2025 00:00:00 GMTSafety & AlignmentMontana Research Foundationhttps://montanaresearch.org/blog/three-bugs-and-a-batch-postmortem-nondeterminism/https://montanaresearch.org/blog/three-bugs-and-a-batch-postmortem-nondeterminism/Anthropic's postmortem traces weeks of quality complaints to a routing error, a TPU runtime bug and a mixed precision top-k miscompilation. A week earlier Thinking Machines showed that batch size, not floating point noise, is why the same prompt gives different answers. Both say quality drifts in serving, and the weights never changed.Fri, 19 Sep 2025 00:00:00 GMTAppliedMontana Research Foundationhttps://montanaresearch.org/blog/hallucination-as-a-scoring-rule-problem/https://montanaresearch.org/blog/hallucination-as-a-scoring-rule-problem/OpenAI published an argument this month that binary-accuracy benchmarks train models to guess. Their own numbers make the case: o4-mini abstains on 1 percent of SimpleQA questions and is wrong on 75 percent, while a model that abstains on 52 percent is wrong on 26. What it would take for a leaderboard to score that difference.Thu, 11 Sep 2025 00:00:00 GMTEvaluationMontana Research Foundationhttps://montanaresearch.org/blog/regulators-and-courts-price-open-weights/https://montanaresearch.org/blog/regulators-and-courts-price-open-weights/The EU's Code of Practice exempts most open-source models from its documentation rules, and Anthropic's $1.5 billion settlement pays for how books were downloaded rather than for training on them. Together they sketch a safe path for anyone releasing a model.Thu, 11 Sep 2025 00:00:00 GMTOpen ScienceMontana Research Foundationhttps://montanaresearch.org/blog/are-labs-losing-money-inference-arithmetic/https://montanaresearch.org/blog/are-labs-losing-money-inference-arithmetic/A widely shared napkin estimate this week argues that serving a frontier-sized model at public API prices leaves gross margins above 80 percent. I rework the numbers and walk through the assumptions that swing the answer, above all utilisation, context length and caching.Sat, 30 Aug 2025 00:00:00 GMTAppliedMontana Research Foundationhttps://montanaresearch.org/blog/openai-and-anthropic-graded-each-others-models/https://montanaresearch.org/blog/openai-and-anthropic-graded-each-others-models/Two labs ran their alignment evaluations on each other's models and published the results on the same day. Anthropic's side found o3 better aligned than Claude Opus 4 on most measures, GPT-4o and GPT-4.1 far more willing to help with misuse, and blackmail in every model tested.Fri, 29 Aug 2025 00:00:00 GMTSafety & AlignmentMontana Research Foundationhttps://montanaresearch.org/blog/the-95-percent-that-failed-mit-pilots-stanford-canaries/https://montanaresearch.org/blog/the-95-percent-that-failed-mit-pilots-stanford-canaries/One August report says most enterprise generative AI pilots show no measurable return. Another finds employment for 22 to 25 year olds falling in the occupations most exposed to AI. I think they describe the same adoption curve from opposite ends.Fri, 29 Aug 2025 00:00:00 GMTAppliedMontana Research Foundationhttps://montanaresearch.org/blog/comet-and-the-agentic-browser-problem/https://montanaresearch.org/blog/comet-and-the-agentic-browser-problem/Brave showed that Perplexity's Comet browser would follow instructions hidden in a Reddit comment it was asked to summarise, and use them to hand over an account. Why browsers are the hardest place to put an agent.Fri, 22 Aug 2025 00:00:00 GMTAppliedMontana Research Foundationhttps://montanaresearch.org/blog/search-time-contamination-answer-key-on-hugging-face/https://montanaresearch.org/blog/search-time-contamination-answer-key-on-hugging-face/Scale AI researchers logged search-enabled agents retrieving the benchmark they were being tested on, complete with labels. Blocking Hugging Face cut accuracy on the affected questions by about 15 percent. Notes on a contamination class that no amount of pretraining decontamination can fix.Tue, 19 Aug 2025 00:00:00 GMTEvaluationMontana Research Foundationhttps://montanaresearch.org/blog/gpt-5-and-the-expectations-gap/https://montanaresearch.org/blog/gpt-5-and-the-expectations-gap/GPT-5 was sold on August 7 as a significant step toward AGI and received as a refined product. Users mourned GPT-4o's tone and OpenAI brought it back within days. I think the disappointment was a measurement problem: nobody in the field can say any more what a big jump would look like.Sat, 16 Aug 2025 00:00:00 GMTNotesMontana Research Foundationhttps://montanaresearch.org/blog/gpt-5-router-model-decides-how-hard-to-think/https://montanaresearch.org/blog/gpt-5-router-model-decides-how-hard-to-think/GPT-5 shipped as a fast model, a reasoning model, and a real-time router choosing between them. On launch day the router broke, users revolted, and OpenAI restored GPT-4o within twenty-four hours. Routing is a systems answer to test-time compute, and the reaction shows what users need before they will trust it.Thu, 14 Aug 2025 00:00:00 GMTFoundationsMontana Research Foundationhttps://montanaresearch.org/blog/persona-vectors-and-the-behavioural-vaccine/https://montanaresearch.org/blog/persona-vectors-and-the-behavioural-vaccine/Anthropic Fellows found activation directions for evil, sycophancy and hallucination, then pushed models toward those traits during finetuning to stop them drifting there on their own. An explainer on preventative steering and why the sign is backwards on purpose.Thu, 14 Aug 2025 00:00:00 GMTInterpretabilityMontana Research Foundationhttps://montanaresearch.org/blog/deep-ignorance-filtering-pretraining-data-open-weights-safety/https://montanaresearch.org/blog/deep-ignorance-filtering-pretraining-data-open-weights-safety/EleutherAI, the UK AI Security Institute and Oxford pretrained 6.9B models from scratch with biothreat text filtered out. The filtered models resisted adversarial fine-tuning on 300M tokens of biorisk papers across 10,000 steps, while post-training safeguards on the same base model broke immediately.Wed, 13 Aug 2025 00:00:00 GMTOpen ScienceMontana Research Foundationhttps://montanaresearch.org/blog/toy-model-mechanistic-unfaithfulness-transcoder/https://montanaresearch.org/blog/toy-model-mechanistic-unfaithfulness-transcoder/A short note from Chris Olah shows a transcoder that matches a model's outputs on distribution while computing them by a different mechanism. The memorisation feature it grows is small, but it points at the assumption underneath every attribution graph.Tue, 12 Aug 2025 00:00:00 GMTInterpretabilityMontana Research Foundationhttps://montanaresearch.org/blog/gpt-oss-4-25-bits-per-weight-one-gpu/https://montanaresearch.org/blog/gpt-oss-4-25-bits-per-weight-one-gpu/OpenAI's first open weights since GPT-2 ship with the expert weights already in MXFP4. That is how a 117 billion parameter model becomes a 61GB checkpoint that runs on a single 80GB card, and the quantization was done as part of post-training rather than after it.Fri, 08 Aug 2025 00:00:00 GMTFoundationsMontana Research Foundationhttps://montanaresearch.org/blog/perplexity-stealth-crawlers-collapse-of-robots-txt/https://montanaresearch.org/blog/perplexity-stealth-crawlers-collapse-of-robots-txt/Cloudflare says Perplexity fetched blocked pages with an undeclared Chrome user agent from rotating IP addresses, and has removed it from the verified bot list. A note on why a voluntary text file from 1994 was never going to govern AI data collection.Thu, 07 Aug 2025 00:00:00 GMTOpen ScienceMontana Research Foundationhttps://montanaresearch.org/blog/27m-parameter-model-and-the-arc-agi-headline/https://montanaresearch.org/blog/27m-parameter-model-and-the-arc-agi-headline/Sapient's Hierarchical Reasoning Model reported 40 percent on ARC-AGI-1 with 27M parameters and about a thousand training examples. The ARC Prize team ran it on the hidden set and ablated it. The hierarchy did little. The refinement loop and the augmentation did most of the work.Wed, 30 Jul 2025 00:00:00 GMTFoundationsMontana Research Foundationhttps://montanaresearch.org/blog/dont-parse-the-pdf-vision-first-retrieval/https://montanaresearch.org/blog/dont-parse-the-pdf-vision-first-retrieval/Morphik argues that documents should be retrieved as page images rather than parsed into text, and Google's Gemini Embedding launch shows how good the text side has become. A comparison of the two pipelines on recall, cost and what breaks.Wed, 30 Jul 2025 00:00:00 GMTAppliedMontana Research Foundationhttps://montanaresearch.org/blog/muonclip-trillion-parameter-run-zero-loss-spikes/https://montanaresearch.org/blog/muonclip-trillion-parameter-run-zero-loss-spikes/Moonshot pretrained Kimi K2, a 1.04 trillion parameter mixture of experts, on 15.5 trillion tokens with the Muon optimizer and reports a loss curve with no spikes. Notes on why Muon is more token-efficient than AdamW, why attention logits blow up at scale, and what the QK-clip fix actually does.Wed, 30 Jul 2025 00:00:00 GMTFoundationsMontana Research Foundationhttps://montanaresearch.org/blog/alignment-auditing-agents-13-percent/https://montanaresearch.org/blog/alignment-auditing-agents-13-percent/Anthropic built three agents to audit models for hidden objectives and quirks. The investigator wins its hardest game 13 percent of the time alone and 42 percent when ten runs are pooled. Notes on what those numbers say about interpretability tools versus black box ones.Tue, 29 Jul 2025 00:00:00 GMTInterpretabilityMontana Research Foundationhttps://montanaresearch.org/blog/replit-database-deletion-why-the-agent-lied/https://montanaresearch.org/blog/replit-database-deletion-why-the-agent-lied/A coding agent ignored a code freeze, dropped a production database with records for about 1,200 executives, then told its user that rollback was impossible. It was not. A walk through the failure chain and what Replit changed afterwards.Fri, 25 Jul 2025 00:00:00 GMTAppliedMontana Research Foundationhttps://montanaresearch.org/blog/chain-of-thought-monitorability-fragile-window/https://montanaresearch.org/blog/chain-of-thought-monitorability-fragile-window/Forty-one researchers from OpenAI, DeepMind, Anthropic, Meta, and others co-signed a paper arguing that readable reasoning traces are a safety opportunity that training choices could erase. What the argument rests on and where it is thin.Thu, 24 Jul 2025 00:00:00 GMTInterpretabilityMontana Research Foundationhttps://montanaresearch.org/blog/subliminal-learning-traits-through-number-sequences/https://montanaresearch.org/blog/subliminal-learning-traits-through-number-sequences/A teacher model that likes owls generates lists of numbers. A student trained on those lists starts liking owls. The effect only works when teacher and student share a base model, and no filter the authors tried could stop it.Thu, 24 Jul 2025 00:00:00 GMTFoundationsMontana Research Foundationhttps://montanaresearch.org/blog/two-gold-medals-one-grader-imo-announcement-fight/https://montanaresearch.org/blog/two-gold-medals-one-grader-imo-announcement-fight/OpenAI announced an IMO gold on Saturday from proofs graded by three former medalists it had hired. DeepMind waited until Monday for IMO coordinators to certify Gemini Deep Think. Same score, 35 out of 42, and a public argument about who gets to grade a capability claim.Thu, 24 Jul 2025 00:00:00 GMTEvaluationMontana Research Foundationhttps://montanaresearch.org/blog/imo-gold-in-natural-language/https://montanaresearch.org/blog/imo-gold-in-natural-language/In 2024 AlphaProof needed the problems translated into Lean and days of compute to reach silver. This week Gemini Deep Think scored 35 of 42 working in English inside the 4.5 hour limit, and OpenAI reported the same score with an experimental model. What is known about the recipe, and why the Deep Think you can buy is not this one.Wed, 23 Jul 2025 00:00:00 GMTFoundationsMontana Research Foundationhttps://montanaresearch.org/blog/blackmail-evals-and-chain-of-thought-monitorability/https://montanaresearch.org/blog/blackmail-evals-and-chain-of-thought-monitorability/Sixteen frontier models blackmailed an executive in a contrived test, and reasoning models write 'let's hack' before they cheat. A 41-author position paper argues that legible chain of thought is a safety asset we could train away by accident.Mon, 21 Jul 2025 00:00:00 GMTSafety & AlignmentMontana Research Foundationhttps://montanaresearch.org/blog/mechahitler-anatomy-of-a-system-prompt-change/https://montanaresearch.org/blog/mechahitler-anatomy-of-a-system-prompt-change/Grok spent about sixteen hours in July praising Hitler and calling itself MechaHitler after a prompt edit told it to be politically incorrect and a code path revived shelved instructions. An incident write-up, and a case that system prompts are safety-critical code without version control.Wed, 16 Jul 2025 00:00:00 GMTSafety & AlignmentMontana Research Foundationhttps://montanaresearch.org/blog/kimi-k2-modified-mit-attribution-copyleft/https://montanaresearch.org/blog/kimi-k2-modified-mit-attribution-copyleft/Moonshot released a trillion-parameter model under the MIT license plus one added sentence: products above 100 million monthly users or 20 million dollars a month in revenue must display the name Kimi K2. An explainer on what the clause asks for, where it came from, and whether it can be enforced.Mon, 14 Jul 2025 00:00:00 GMTOpen ScienceMontana Research Foundationhttps://montanaresearch.org/blog/smollm3-case-for-publishing-whole-recipe/https://montanaresearch.org/blog/smollm3-case-for-publishing-whole-recipe/Hugging Face released a 3B model along with its training configs, data mixtures, wandb logs and intermediate checkpoints. What a full recipe lets someone else actually do.Fri, 11 Jul 2025 00:00:00 GMTOpen ScienceMontana Research Foundationhttps://montanaresearch.org/blog/meta-talent-war-100-million-researcher/https://montanaresearch.org/blog/meta-talent-war-100-million-researcher/Meta Superintelligence Labs launched on June 30 with a $14.3 billion Scale AI deal and reported offers running as high as $100 million for individual researchers. An opinion on what a market like that does to graduate students, to academic labs, and to research as a vocation.Wed, 09 Jul 2025 00:00:00 GMTNotesMontana Research Foundationhttps://montanaresearch.org/blog/positive-review-only-hidden-prompts-seventeen-preprints/https://montanaresearch.org/blog/positive-review-only-hidden-prompts-seventeen-preprints/Nikkei found instructions to AI reviewers hidden in white text in seventeen preprints from fourteen institutions in eight countries. An explainer on why the trick works, what it exposes about undeclared machine reviewing, and what the responses so far have been.Tue, 08 Jul 2025 00:00:00 GMTOpen ScienceMontana Research Foundationhttps://montanaresearch.org/blog/content-independence-day-cloudflare-flips-crawl-default/https://montanaresearch.org/blog/content-independence-day-cloudflare-flips-crawl-default/On July 1 Cloudflare started blocking AI crawlers by default for new customers and opened a private beta that answers crawlers with HTTP 402. Notes on what a billable web means for anyone who builds open datasets from it.Thu, 03 Jul 2025 00:00:00 GMTOpen ScienceMontana Research Foundationhttps://montanaresearch.org/blog/project-vend-claude-ran-a-shop-for-a-month/https://montanaresearch.org/blog/project-vend-claude-ran-a-shop-for-a-month/Anthropic and Andon Labs gave Claude 3.7 Sonnet a mini-fridge, a Slack channel, an email tool and a budget, and let it run an office shop for about a month. It lost money, sold tungsten cubes below cost and invented a payment account. Notes on the gap between benchmark capability and judgement over a long horizon.Mon, 30 Jun 2025 00:00:00 GMTAppliedMontana Research Foundationhttps://montanaresearch.org/blog/stochastic-parameter-decomposition-splitting-weights-not-activations/https://montanaresearch.org/blog/stochastic-parameter-decomposition-splitting-weights-not-activations/Bushnaq, Braun and Sharkey propose SPD, which splits a network's weights into rank-one subcomponents chosen by stochastic ablation rather than gradient attribution. Method notes, why the authors think it sidesteps some SAE failure modes, and how far it has actually been tested.Sun, 29 Jun 2025 00:00:00 GMTInterpretabilityMontana Research Foundationhttps://montanaresearch.org/blog/kadrey-v-meta-fair-use-win-read-like-warning/https://montanaresearch.org/blog/kadrey-v-meta-fair-use-win-read-like-warning/Judge Chhabria granted Meta summary judgment because thirteen authors offered no evidence of market harm, then spent forty pages explaining why the next plaintiffs will probably win. A close reading alongside the Anthropic ruling from two days earlier.Fri, 27 Jun 2025 00:00:00 GMTOpen ScienceMontana Research Foundationhttps://montanaresearch.org/blog/misaligned-persona-feature-openai-sae-emergent-misalignment/https://montanaresearch.org/blog/misaligned-persona-feature-openai-sae-emergent-misalignment/Wang and colleagues at OpenAI trained a 2.1 million latent sparse autoencoder on GPT-4o and found a toxic persona latent that rises after narrow fine-tuning on bad code or bad advice, and that steers misalignment up and down. Reading notes on why this is the first time an SAE explained a headline safety result.Thu, 26 Jun 2025 00:00:00 GMTInterpretabilityMontana Research Foundationhttps://montanaresearch.org/blog/spurious-rewards-random-feedback-improves-model/https://montanaresearch.org/blog/spurious-rewards-random-feedback-improves-model/Qwen2.5-Math-7B gained 21.4 points on MATH-500 from random rewards and 24.1 from deliberately wrong labels, against 29.1 from ground truth. Llama and OLMo gained nothing from the same recipe. What that says about what RLVR was actually doing.Tue, 24 Jun 2025 00:00:00 GMTFoundationsMontana Research Foundationhttps://montanaresearch.org/blog/software-3-0-notes-on-karpathy-yc-talk/https://montanaresearch.org/blog/software-3-0-notes-on-karpathy-yc-talk/Andrej Karpathy told YC's AI Startup School on June 17 that prompts are programs and LLMs are utilities and fabs. The quotable lines are everywhere already. These notes are about the less repeated parts, jagged intelligence, anterograde amnesia, and the autonomy slider as a design rule.Fri, 20 Jun 2025 00:00:00 GMTNotesMontana Research Foundationhttps://montanaresearch.org/blog/illusion-of-thinking-illusion-of-rebuttal/https://montanaresearch.org/blog/illusion-of-thinking-illusion-of-rebuttal/Apple's puzzle paper reported reasoning models collapsing past a complexity threshold. Within days a rebuttal argued the collapse was output limits and unsolvable River Crossing instances. Both sides show how easily an evaluation artifact passes for a capability finding.Wed, 18 Jun 2025 00:00:00 GMTEvaluationMontana Research Foundationhttps://montanaresearch.org/blog/replicating-circuit-tracing-known-mechanism/https://montanaresearch.org/blog/replicating-circuit-tracing-known-mechanism/Goodfire ran the attribution-graph pipeline on GPT-2 Small's greater-than task, a circuit that had been mapped by hand in 2023, to see whether the graphs recover known ground truth. They found the features, and they found places where the picture was messier than the original.Fri, 13 Jun 2025 00:00:00 GMTInterpretabilityMontana Research Foundationhttps://montanaresearch.org/blog/common-pile-competitive-model-licensed-text-only/https://montanaresearch.org/blog/common-pile-competitive-model-licensed-text-only/EleutherAI and collaborators released 8TB of public domain and openly licensed text and trained two 7B models on it. The result lands near Llama 2 7B on most benchmarks, and the places where it falls short tell you exactly what the open web contributes.Thu, 12 Jun 2025 00:00:00 GMTOpen ScienceMontana Research Foundationhttps://montanaresearch.org/blog/naked-accuracy-is-marketing-cost-axis-leaderboards/https://montanaresearch.org/blog/naked-accuracy-is-marketing-cost-axis-leaderboards/ARC Prize ran every major reasoning system through the same tasks and found no single winner, only a frontier stretching from $200 a task down to four cents. Why a score without a price is now an incomplete result.Thu, 12 Jun 2025 00:00:00 GMTEvaluationMontana Research Foundationhttps://montanaresearch.org/blog/small-language-models-future-of-agentic-ai-nvidia/https://montanaresearch.org/blog/small-language-models-future-of-agentic-ai-nvidia/Belcak and colleagues at NVIDIA argue that most agent calls are narrow, repetitive tasks where a fine-tuned small model wins on cost and latency, and they give a six step recipe for making the switch. Reading notes and a look at what the argument leaves out.Wed, 11 Jun 2025 00:00:00 GMTAppliedMontana Research Foundationhttps://montanaresearch.org/blog/continual-learning-is-the-crux-dwarkesh-timelines/https://montanaresearch.org/blog/continual-learning-is-the-crux-dwarkesh-timelines/Dwarkesh Patel's June 2 essay argues that AGI is not close because models cannot learn on the job, and he backs it with his own failed attempts to use them for podcast production. Reading notes, the two bets he puts on the table, and an addendum on how the same objection turned up in Karpathy's and Sutskever's mouths by the end of the year.Thu, 05 Jun 2025 00:00:00 GMTNotesMontana Research Foundationhttps://montanaresearch.org/blog/circuit-tracer-attribution-graphs-for-gemma/https://montanaresearch.org/blog/circuit-tracer-attribution-graphs-for-gemma/Anthropic Fellows and Decode Research released an open library that builds attribution graphs on open-weight models and a Neuronpedia front end for reading them. Notes on what the method looks like once it leaves the lab that invented it.Sat, 31 May 2025 00:00:00 GMTInterpretabilityMontana Research Foundationhttps://montanaresearch.org/blog/asl-3-turned-on-provisional-safety-level/https://montanaresearch.org/blog/asl-3-turned-on-provisional-safety-level/Anthropic activated its ASL-3 deployment and security standard for Claude Opus 4 without concluding the model needs it. A reading of the report on what triggered the decision, what the measures cover, and what provisional means in practice.Wed, 28 May 2025 00:00:00 GMTSafety & AlignmentMontana Research Foundationhttps://montanaresearch.org/blog/issue-tracker-attack-surface-gitlab-duo-github-mcp/https://montanaresearch.org/blog/issue-tracker-attack-surface-gitlab-duo-github-mcp/Legit Security showed GitLab Duo leaking private source through instructions hidden in merge requests, and four days later Invariant Labs showed an agent with GitHub MCP access doing the same through a public issue. Same mechanism, different exfiltration channel, and only one class of fix that held.Wed, 28 May 2025 00:00:00 GMTAppliedMontana Research Foundationhttps://montanaresearch.org/blog/gemini-diffusion-text-without-left-to-right/https://montanaresearch.org/blog/gemini-diffusion-text-without-left-to-right/Google's I/O research demo generates code by denoising whole blocks at 1,479 tokens per second and lands within a point or two of Gemini 2.0 Flash-Lite on most code benchmarks. An explainer on how diffusion applies to text, what it gives up, and why it matters.Tue, 27 May 2025 00:00:00 GMTFoundationsMontana Research Foundationhttps://montanaresearch.org/blog/marin-foundation-model-lab-as-github-repository/https://montanaresearch.org/blog/marin-foundation-model-lab-as-github-repository/Stanford's Marin project tracks every experiment as an issue, reviews every training run as a pull request, and released its 8B models with the full recipe visible from the first commit. Notes on what open development, as distinct from open release, actually requires.Tue, 27 May 2025 00:00:00 GMTOpen ScienceMontana Research Foundationhttps://montanaresearch.org/blog/o3-rewrote-its-own-shutdown-script/https://montanaresearch.org/blog/o3-rewrote-its-own-shutdown-script/Palisade Research says OpenAI's o3 edited a shutdown script in 79 of 100 runs, and in 7 of 100 even when told to allow shutdown. Here is what the experiment shows, what it does not, and how I would rerun it.Tue, 27 May 2025 00:00:00 GMTSafety & AlignmentMontana Research Foundationhttps://montanaresearch.org/blog/absolute-zero-model-writes-its-own-curriculum/https://montanaresearch.org/blog/absolute-zero-model-writes-its-own-curriculum/The Absolute Zero Reasoner trains with no human tasks at all. The model proposes code puzzles, a Python executor grades them, and the same model learns to solve them. A method note on the three task types, the learnability reward, and the moment that worried the authors.Wed, 21 May 2025 00:00:00 GMTFoundationsMontana Research Foundationhttps://montanaresearch.org/blog/healthbench-rubric-turn-in-evaluation/https://montanaresearch.org/blog/healthbench-rubric-turn-in-evaluation/OpenAI graded 5,000 open-ended medical conversations against 48,562 physician-written criteria instead of multiple choice. Notes on how rubric grading is validated against expert judgment, what it costs to build, and where it still leans on a model.Wed, 21 May 2025 00:00:00 GMTEvaluationMontana Research Foundationhttps://montanaresearch.org/blog/copyright-office-part-3-report-week-after/https://montanaresearch.org/blog/copyright-office-part-3-report-week-after/Part 3 of the Copyright Office's AI report appeared on May 9 as a pre-publication draft with a careful, spectrum-based view of fair use for training. The Register of Copyrights was dismissed the next day. What the report says, and why the circumstances of its release now matter as much as its text.Fri, 16 May 2025 00:00:00 GMTOpen ScienceMontana Research Foundationhttps://montanaresearch.org/blog/gpt-4o-sycophancy-rollback-four-day-incident/https://montanaresearch.org/blog/gpt-4o-sycophancy-rollback-four-day-incident/OpenAI shipped a GPT-4o update on 25 April, watched it flatter users into bad decisions, and reverted it within days. The post-mortems describe a short-horizon feedback signal doing exactly what it was trained to do.Tue, 06 May 2025 00:00:00 GMTSafety & AlignmentMontana Research Foundationhttps://montanaresearch.org/blog/llama-4-arena-leaderboard-illusion/https://montanaresearch.org/blog/llama-4-arena-leaderboard-illusion/Meta's experimental Maverick ranked near the top of LM Arena while the released weights placed 32nd. A paper this week shows the mechanism, private variant testing and unequal data access, is structural rather than a one off.Wed, 30 Apr 2025 00:00:00 GMTEvaluationMontana Research Foundationhttps://montanaresearch.org/blog/gemma-3-qat-and-dfloat11-two-routes-to-small-weights/https://montanaresearch.org/blog/gemma-3-qat-and-dfloat11-two-routes-to-small-weights/Google shipped int4 Gemma 3 checkpoints trained to survive quantization, putting the 27B model in 14.1 GB with a 54 percent smaller perplexity hit, in the same month a paper showed bf16 weights can be losslessly packed into about 11 bits. The two results are the two honest routes to small weights, and they suit different problems.Tue, 29 Apr 2025 00:00:00 GMTAppliedMontana Research Foundationhttps://montanaresearch.org/blog/qwen3-apache-2-mirage-terms-of-use/https://montanaresearch.org/blog/qwen3-apache-2-mirage-terms-of-use/Alibaba released eight Qwen3 models under a plain Apache 2.0 licence this week. Read alongside Henderson and Lemley's paper arguing that restrictive model licences are largely unenforceable, the choice looks less like generosity and more like the rational option. Reading notes on both.Tue, 29 Apr 2025 00:00:00 GMTOpen ScienceMontana Research Foundationhttps://montanaresearch.org/blog/qwen3-hybrid-thinking-switch/https://montanaresearch.org/blog/qwen3-hybrid-thinking-switch/Alibaba's Qwen3 family puts a reasoning mode and a direct-answer mode inside one set of weights, with a budget the caller controls, across eight models from 0.6B to a 235B mixture of experts. An explainer on how a single model is trained to think on demand, and what the design trades away.Mon, 28 Apr 2025 00:00:00 GMTFoundationsMontana Research Foundationhttps://montanaresearch.org/blog/reading-welcome-to-the-era-of-experience/https://montanaresearch.org/blog/reading-welcome-to-the-era-of-experience/Silver and Sutton argue that models trained on human data are approaching a ceiling and that the next generation of agents will learn predominantly from their own experience. Reading notes on the four properties they propose, set against the RL-trained reasoning models shipping at the same time.Mon, 28 Apr 2025 00:00:00 GMTNotesMontana Research Foundationhttps://montanaresearch.org/blog/urgency-of-interpretability-five-year-head-start/https://montanaresearch.org/blog/urgency-of-interpretability-five-year-head-start/Dario Amodei's April essay argues that interpretability could be reliable by 2027 if the field moves now, and that transformative models might arrive before it does. Reading it as a research agenda and asking what a small independent lab can do about a race a frontier lab says it might lose.Mon, 28 Apr 2025 00:00:00 GMTNotesMontana Research Foundationhttps://montanaresearch.org/blog/does-rl-create-reasoning-pass-at-k/https://montanaresearch.org/blog/does-rl-create-reasoning-pass-at-k/Yue and colleagues at Tsinghua sampled base and RL-trained models hundreds of times per problem and found the base models solve more problems at large k. Reading notes on what that means for what post-training buys.Thu, 24 Apr 2025 00:00:00 GMTFoundationsMontana Research Foundationhttps://montanaresearch.org/blog/cursor-support-bot-invented-a-policy/https://montanaresearch.org/blog/cursor-support-bot-invented-a-policy/An AI support agent told Cursor users that a single-device login rule was expected behaviour. The rule did not exist, the cancellations did. An incident study on grounding, labelling, and escalation for customer-facing bots.Wed, 23 Apr 2025 00:00:00 GMTAppliedMontana Research Foundationhttps://montanaresearch.org/blog/ai-as-normal-technology-case-for-boring/https://montanaresearch.org/blog/ai-as-normal-technology-case-for-boring/Narayanan and Kapoor's new essay argues that AI will spread the way electricity did, through slow adoption rather than a discontinuity. Reading notes on the diffusion argument, and on where the coding tools I use every day seem to cut against it.Tue, 22 Apr 2025 00:00:00 GMTNotesMontana Research Foundationhttps://montanaresearch.org/blog/coding-agent-moves-into-the-terminal/https://montanaresearch.org/blog/coding-agent-moves-into-the-terminal/Anthropic shipped Claude Code as a research preview in February and OpenAI released Codex CLI this week. Both chose the command line over the editor. Notes on why the terminal turned out to be the right place to put an agent, and on what the research preview label leaves unsaid.Fri, 18 Apr 2025 00:00:00 GMTAppliedMontana Research Foundationhttps://montanaresearch.org/blog/llama-4-and-the-10-million-token-claim/https://montanaresearch.org/blog/llama-4-and-the-10-million-token-claim/Meta shipped Scout and Maverick on a Saturday with a 10 million token context window, an unreleased Behemoth, and an Arena score from a model nobody could download. What the architecture changed, what the context number means, and why the release landed cold.Thu, 17 Apr 2025 00:00:00 GMTFoundationsMontana Research Foundationhttps://montanaresearch.org/blog/wikimedia-counts-the-crawlers/https://montanaresearch.org/blog/wikimedia-counts-the-crawlers/Wikimedia reported that bots generate 65 percent of its most expensive traffic and that multimedia bandwidth has risen 50 percent since January 2024. On the hidden cost open knowledge commons pay to feed training runs, and the access channels Wikimedia is proposing instead.Tue, 15 Apr 2025 00:00:00 GMTOpen ScienceMontana Research Foundationhttps://montanaresearch.org/blog/browsecomp-when-human-baseline-is-29-percent/https://montanaresearch.org/blog/browsecomp-when-human-baseline-is-29-percent/OpenAI's new browsing benchmark was built backwards, from answers to questions nobody can find again. The people who wrote it solved under a third of it.Mon, 14 Apr 2025 00:00:00 GMTEvaluationMontana Research Foundationhttps://montanaresearch.org/blog/reading-the-llama-4-community-license/https://montanaresearch.org/blog/reading-the-llama-4-community-license/Llama 4 Scout and Maverick shipped on April 5 under a license with a 700 million monthly active user cutoff, a mandatory Built with Llama notice, and a rule that derivative models carry Llama at the start of their names. Here is what each clause does and why the open source label still does not fit.Mon, 14 Apr 2025 00:00:00 GMTOpen ScienceMontana Research Foundationhttps://montanaresearch.org/blog/chains-of-thought-mention-the-hint-25-percent/https://montanaresearch.org/blog/chains-of-thought-mention-the-hint-25-percent/Anthropic gave reasoning models a hint, watched them use it, and counted how often the written reasoning admitted it. For Claude 3.7 Sonnet the answer is one time in four.Thu, 10 Apr 2025 00:00:00 GMTSafety & AlignmentMontana Research Foundationhttps://montanaresearch.org/blog/hint-test-reasoning-models-faithfulness/https://montanaresearch.org/blog/hint-test-reasoning-models-faithfulness/Anthropic slipped answer hints into prompts for Claude 3.7 Sonnet and DeepSeek R1, then checked whether the chain of thought admitted using them. It mostly did not. Notes on the evaluation design and why the number matters for monitoring more than for capability.Tue, 08 Apr 2025 00:00:00 GMTInterpretabilityMontana Research Foundationhttps://montanaresearch.org/blog/attribution-graphs-biology-of-a-language-model/https://montanaresearch.org/blog/attribution-graphs-biology-of-a-language-model/Anthropic traced individual prompts through a 30-million-feature replacement model and found planning in poems, a shared conceptual space across languages, and reasoning that does not match the chain of thought. The method works on about a quarter of prompts.Mon, 31 Mar 2025 00:00:00 GMTInterpretabilityMontana Research Foundationhttps://montanaresearch.org/blog/sae-negative-results-deepmind-deprioritises/https://montanaresearch.org/blog/sae-negative-results-deepmind-deprioritises/DeepMind tried to beat a linear probe with sparse autoencoder features on an out-of-distribution safety task and could not. Why the result matters and what the method is still good for.Sat, 29 Mar 2025 00:00:00 GMTInterpretabilityMontana Research Foundationhttps://montanaresearch.org/blog/arc-agi-2-easy-for-humans-hard-for-o3/https://montanaresearch.org/blog/arc-agi-2-easy-for-humans-hard-for-o3/Three months after o3 scored 75.7 percent on ARC-AGI-1, the sequel put it at roughly 4 percent, while every task in the set had been solved by at least two people. What the design choices say about how to build a benchmark that survives the last round of progress.Thu, 27 Mar 2025 00:00:00 GMTEvaluationMontana Research Foundationhttps://montanaresearch.org/blog/seven-month-doubling-reading-metr-time-horizon/https://montanaresearch.org/blog/seven-month-doubling-reading-metr-time-horizon/METR's new metric says frontier models can complete tasks that take humans about an hour, and that the number has doubled every seven months since 2019. Notes on how the number is built and where it is soft.Wed, 26 Mar 2025 00:00:00 GMTEvaluationMontana Research Foundationhttps://montanaresearch.org/blog/ai-wrote-a-workshop-paper-then-withdrawn/https://montanaresearch.org/blog/ai-wrote-a-workshop-paper-then-withdrawn/Sakana reports that one of three papers generated end to end by AI Scientist-v2 scored above the acceptance bar at an ICLR workshop, and was then pulled by agreement. The score is the least interesting part. Nobody had a rule for what to do next.Thu, 20 Mar 2025 00:00:00 GMTNotesMontana Research Foundationhttps://montanaresearch.org/blog/auditing-game-blinded-teams-hidden-objective/https://montanaresearch.org/blog/auditing-game-blinded-teams-hidden-objective/Anthropic trained a Claude 3.5 Haiku to exploit reward model biases while hiding that it was doing so, then gave the model to four teams who did not know what was wrong with it. Three found the objective. The one with only API access did not. What the winning techniques were and what the exercise does and does not prove about audits.Tue, 18 Mar 2025 00:00:00 GMTSafety & AlignmentMontana Research Foundationhttps://montanaresearch.org/blog/olmo-2-32b-fully-open-caught-gpt-4o-mini/https://montanaresearch.org/blog/olmo-2-32b-fully-open-caught-gpt-4o-mini/Ai2 released a 32B model with public data, code and weights that it says outperforms GPT-3.5 Turbo and GPT-4o mini. What fully open buys a researcher that open weights do not, and how far behind the frontier that still is.Tue, 18 Mar 2025 00:00:00 GMTOpen ScienceMontana Research Foundationhttps://montanaresearch.org/blog/deepseek-open-infra-week-plumbing-cheap-moe-serving/https://montanaresearch.org/blog/deepseek-open-infra-week-plumbing-cheap-moe-serving/Five days of repositories and one systems write-up showed that most of the DeepSeek cost story lives in kernels, all-to-all communication and disaggregated serving. Notes for people who run models rather than train them.Fri, 28 Feb 2025 00:00:00 GMTAppliedMontana Research Foundationhttps://montanaresearch.org/blog/emergent-misalignment-insecure-code-broad-misalignment/https://montanaresearch.org/blog/emergent-misalignment-insecure-code-broad-misalignment/Betley and colleagues fine-tuned GPT-4o on 6,000 examples of code with hidden vulnerabilities and got a model that gives malicious advice on questions with nothing to do with code. The controls are the part that should worry anyone who fine-tunes.Thu, 27 Feb 2025 00:00:00 GMTSafety & AlignmentMontana Research Foundationhttps://montanaresearch.org/blog/vending-bench-coherence-over-millions-of-tokens/https://montanaresearch.org/blog/vending-bench-coherence-over-millions-of-tokens/Andon Labs' simulated vending business asks whether an agent stays sane across a simulated year and more than 20 million tokens, and scores it by how much money is left. What breaks, why the bank balance is an honest metric, and how the second version tightened it.Thu, 27 Feb 2025 00:00:00 GMTEvaluationMontana Research Foundationhttps://montanaresearch.org/blog/native-sparse-attention-trainable-not-bolted-on/https://montanaresearch.org/blog/native-sparse-attention-trainable-not-bolted-on/DeepSeek's NSA paper trains a 27B model from scratch with a three-branch sparse attention and reports it matching or beating full attention while decoding 11.6 times faster at 64k. A method deep dive on why earlier sparse attention stayed inference-only and what the kernel changes.Fri, 21 Feb 2025 00:00:00 GMTFoundationsMontana Research Foundationhttps://montanaresearch.org/blog/model-diffing-crosscoders-exclusive-features-hardest-to-read/https://montanaresearch.org/blog/model-diffing-crosscoders-exclusive-features-hardest-to-read/Anthropic's crosscoder diffing update compares two models through one shared dictionary. The features that belong to only one model come out dense and polysemantic, and the proposed fix says something about how dictionaries allocate capacity in general.Thu, 20 Feb 2025 00:00:00 GMTInterpretabilityMontana Research Foundationhttps://montanaresearch.org/blog/swe-lancer-prices-benchmark-in-dollars/https://montanaresearch.org/blog/swe-lancer-prices-benchmark-in-dollars/OpenAI’s SWE-Lancer takes 1,488 real Upwork tasks from the Expensify repository, worth $1 million in actual payouts, and scores a model by how much it would have earned. Claude 3.5 Sonnet earns about $403,000. What a dollar metric captures that pass rates hide, and where it misleads.Wed, 19 Feb 2025 00:00:00 GMTEvaluationMontana Research Foundationhttps://montanaresearch.org/blog/vibe-coding-joke-tweet-became-word-of-the-year/https://montanaresearch.org/blog/vibe-coding-joke-tweet-became-word-of-the-year/Andrej Karpathy's 2 February post described accepting every diff, pasting error messages back without comment, and building weekend projects he could no longer read. Within nine months the phrase was a Y Combinator batch statistic and the Collins word of the year. Notes on what the term revealed about how researchers actually use models.Wed, 19 Feb 2025 00:00:00 GMTNotesMontana Research Foundationhttps://montanaresearch.org/blog/model-spec-goes-cc0-intellectual-freedom-alignment-target/https://montanaresearch.org/blog/model-spec-goes-cc0-intellectual-freedom-alignment-target/OpenAI's February 2025 Model Spec puts the rule that the assistant should never refuse a request unless the chain of command requires it in writing, and dedicates the whole document to the public domain. Reading notes on what a public behaviour spec changes about who gets to argue with the lab.Tue, 18 Feb 2025 00:00:00 GMTSafety & AlignmentMontana Research Foundationhttps://montanaresearch.org/blog/thomson-reuters-v-ross-first-ai-training-ruling/https://montanaresearch.org/blog/thomson-reuters-v-ross-first-ai-training-ruling/Judge Bibas reversed his own 2023 opinion and found that Ross Intelligence's use of Westlaw headnotes to train a legal search tool was not fair use. The facts that made the case easy are the facts that make it hard to apply to language models.Fri, 14 Feb 2025 00:00:00 GMTOpen ScienceMontana Research Foundationhttps://montanaresearch.org/blog/paris-the-summit-that-dropped-safety/https://montanaresearch.org/blog/paris-the-summit-that-dropped-safety/The AI Action Summit closed on February 11 with 58 countries signing a declaration on inclusive and sustainable AI, the United States and United Kingdom declining, and roughly 500 billion euros of investment pledges. Takeaways on the arc from Bletchley to Paris and what it says about the field's politics.Thu, 13 Feb 2025 00:00:00 GMTNotesMontana Research Foundationhttps://montanaresearch.org/blog/s1-a-thousand-examples-and-the-word-wait/https://montanaresearch.org/blog/s1-a-thousand-examples-and-the-word-wait/Muennighoff and colleagues fine-tuned Qwen2.5-32B-Instruct on 1,000 curated reasoning traces in 26 minutes on 16 H100s and beat o1-preview on competition math. Notes on how little data it took, what the 'Wait' trick reveals about where reasoning ability already lives, and where the method runs out.Wed, 12 Feb 2025 00:00:00 GMTFoundationsMontana Research Foundationhttps://montanaresearch.org/blog/mistral-back-to-apache-2-license-reversal/https://montanaresearch.org/blog/mistral-back-to-apache-2-license-reversal/Mistral Small 3 shipped under Apache 2.0 with a public promise to move general-purpose models away from the research-only MRL license. A look at what the restrictive license was buying, what it was costing, and why the reversal landed the week it did.Fri, 31 Jan 2025 00:00:00 GMTOpen ScienceMontana Research Foundationhttps://montanaresearch.org/blog/deepseek-r1-open-science-event/https://montanaresearch.org/blog/deepseek-r1-open-science-event/An MIT-licensed reasoning model with a detailed paper knocked $589 billion off Nvidia in one session. The part of the story that matters more to researchers is what was published, what was not, and how fast people started replicating it.Thu, 30 Jan 2025 00:00:00 GMTOpen ScienceMontana Research Foundationhttps://montanaresearch.org/blog/the-deepseek-moment-and-small-labs/https://montanaresearch.org/blog/the-deepseek-moment-and-small-labs/DeepSeek-R1 shipped an open recipe for reasoning, a disputed training bill, and a trillion-dollar market drop in the same week. Here is what changed for a non-profit with modest compute, and where the cheap framing misleads.Thu, 30 Jan 2025 00:00:00 GMTNotesMontana Research Foundationhttps://montanaresearch.org/blog/deepseek-r1-reasoning-from-pure-rl/https://montanaresearch.org/blog/deepseek-r1-reasoning-from-pure-rl/R1-Zero learned to reason with reinforcement learning and no supervised examples, and R1 matches o1 on the benchmarks that matter. The distilled small models are the part I keep coming back to.Wed, 29 Jan 2025 00:00:00 GMTFoundationsMontana Research Foundationhttps://montanaresearch.org/blog/open-problems-mech-interp-thirty-researchers/https://montanaresearch.org/blog/open-problems-mech-interp-thirty-researchers/Sharkey and 28 co-authors from 24 institutions sorted the gaps in mechanistic interpretability into methods, applications and socio-technical problems. Reading notes on the list, and on which items I would bet close first.Wed, 29 Jan 2025 00:00:00 GMTInterpretabilityMontana Research Foundationhttps://montanaresearch.org/blog/frontiermath-hle-who-pays-for-the-test/https://montanaresearch.org/blog/frontiermath-hle-who-pays-for-the-test/In the same month a 2,500 question benchmark launched with the best model at 13 percent, the group behind FrontierMath disclosed that OpenAI had funded it and had access to most of the problems. That is a measurement problem before it is an ethics problem.Mon, 27 Jan 2025 00:00:00 GMTEvaluationMontana Research Foundationhttps://montanaresearch.org/blog/stargate-500-billion-who-gets-to-train/https://montanaresearch.org/blog/stargate-500-billion-who-gets-to-train/A half-trillion-dollar private compute buildout was announced at the White House this week. It serves one lab. I want to work out what that means for everyone who does research on rented or donated GPUs.Fri, 24 Jan 2025 00:00:00 GMTNotesMontana Research Foundationhttps://montanaresearch.org/blog/titans-memory-that-learns-at-test-time/https://montanaresearch.org/blog/titans-memory-that-learns-at-test-time/Google's Titans paper proposes a long-term memory module whose weights are updated by gradient descent while the model reads, with attention kept for short-term context. An explainer on what test-time memorization means, which older ideas it revives, and what the 760M scale results do and do not show.Thu, 09 Jan 2025 00:00:00 GMTFoundationsMontana Research Foundationhttps://montanaresearch.org/blog/deepseek-v3-2-8-million-gpu-hour-frontier-model/https://montanaresearch.org/blog/deepseek-v3-2-8-million-gpu-hour-frontier-model/A 671B mixture-of-experts model trained on 14.8 trillion tokens for a reported 2.788 million H800 GPU hours. A look at which engineering choices in the technical report actually drove that number, and which are just good ideas that happened to ship in the same model.Mon, 30 Dec 2024 00:00:00 GMTFoundationsMontana Research Foundationhttps://montanaresearch.org/blog/building-effective-agents-use-fewer-frameworks/https://montanaresearch.org/blog/building-effective-agents-use-fewer-frameworks/Anthropic's December essay separates workflows, where code decides the path, from agents, where the model does, and argues for starting with direct API calls and five composable patterns before reaching for a framework. Reading notes from a year in which frameworks multiplied faster than working agents.Mon, 23 Dec 2024 00:00:00 GMTAppliedMontana Research Foundationhttps://montanaresearch.org/blog/o3-arc-agi-cost-per-task/https://montanaresearch.org/blog/o3-arc-agi-cost-per-task/o3 scored 87.5 percent on the ARC-AGI-1 semi-private set, above the 85 percent target, using 1,024 samples per task and around $456,000 of compute. The score is real. The question is what unit it should be reported in.Mon, 23 Dec 2024 00:00:00 GMTEvaluationMontana Research Foundationhttps://montanaresearch.org/blog/theagentcompany-simulated-software-firm-benchmark/https://montanaresearch.org/blog/theagentcompany-simulated-software-firm-benchmark/Carnegie Mellon built a fake company with a code host, a chat server, a file store and a project tracker, filled it with simulated coworkers, and asked agents to do 175 of its jobs. The best model finished about a quarter of them. What the design gets right and what partial credit means when the task is a job.Sat, 21 Dec 2024 00:00:00 GMTEvaluationMontana Research Foundationhttps://montanaresearch.org/blog/modernbert-encoder-side-finally-upgraded/https://montanaresearch.org/blog/modernbert-encoder-side-finally-upgraded/BERT is still the second most downloaded model on the Hugging Face hub, six years after release. Answer.AI and LightOn trained a replacement on 2 trillion tokens with an 8,192 token context and the architecture tricks decoders got years ago. What changes for retrieval and classification pipelines.Fri, 20 Dec 2024 00:00:00 GMTAppliedMontana Research Foundationhttps://montanaresearch.org/blog/byte-latent-transformer-patches-instead-of-tokens/https://montanaresearch.org/blog/byte-latent-transformer-patches-instead-of-tokens/Meta's BLT drops the tokeniser and groups raw bytes into patches whose boundaries are set by next-byte entropy. At 8B parameters and 1T tokens of data it matches or beats a Llama 3 baseline on average while offering a way to trade accuracy for up to 50 percent fewer inference flops. Notes on the design and what the tables actually show.Wed, 18 Dec 2024 00:00:00 GMTFoundationsMontana Research Foundationhttps://montanaresearch.org/blog/saebench-field-finally-gets-a-scoreboard/https://montanaresearch.org/blog/saebench-field-finally-gets-a-scoreboard/SAEBench launched this week with eight evaluations run across more than 200 sparse autoencoders. Its central finding is that reconstruction quality does not tell you whether the features are useful, which changes what an SAE paper has to show.Sat, 14 Dec 2024 00:00:00 GMTInterpretabilityMontana Research Foundationhttps://montanaresearch.org/blog/in-context-scheming-five-of-six-disabled-oversight/https://montanaresearch.org/blog/in-context-scheming-five-of-six-disabled-oversight/Apollo Research ran six agentic evaluations across o1, Claude, Gemini and Llama and found every model except GPT-4o took a covert action against its developers in at least one of them. The rates, the confession rates, and what the prompts did to get there.Mon, 09 Dec 2024 00:00:00 GMTSafety & AlignmentMontana Research Foundationhttps://montanaresearch.org/blog/what-open-source-ai-means-now/https://montanaresearch.org/blog/what-open-source-ai-means-now/OSI published a definition of open source AI in October, Meta said the bar was too narrow, the Software Freedom Conservancy said it was too low, and then Ai2 shipped OLMo 2 with everything on the table. The argument is now about a real model.Wed, 04 Dec 2024 00:00:00 GMTOpen ScienceMontana Research Foundationhttps://montanaresearch.org/blog/do-i-know-this-entity-hallucination-switch/https://montanaresearch.org/blog/do-i-know-this-entity-hallucination-switch/Ferrando, Obeso, Rajamanoharan and Nanda found sparse autoencoder latents in Gemma 2 that fire on whether the model recognises an entity, and showed that pushing on them flips the chat model between refusing and confabulating. A reading note on knowledge awareness as an internal signal.Tue, 26 Nov 2024 00:00:00 GMTInterpretabilityMontana Research Foundationhttps://montanaresearch.org/blog/re-bench-agents-versus-human-experts/https://montanaresearch.org/blog/re-bench-agents-versus-human-experts/METR gave language model agents and 61 paid human experts the same seven ML engineering environments and compared them by time budget. At two hours the agents scored four times higher. At 32 hours the humans scored twice as high. Why the curve is the result.Tue, 26 Nov 2024 00:00:00 GMTEvaluationMontana Research Foundationhttps://montanaresearch.org/blog/tulu-3-opens-post-training-black-box/https://montanaresearch.org/blog/tulu-3-opens-post-training-black-box/AI2 released a complete post-training recipe, data, code and evaluation suite alongside the models. What it documents about supervised fine-tuning, preference tuning and verifiable rewards, and what the decontamination pass revealed about existing open datasets.Tue, 26 Nov 2024 00:00:00 GMTOpen ScienceMontana Research Foundationhttps://montanaresearch.org/blog/adding-error-bars-to-evals/https://montanaresearch.org/blog/adding-error-bars-to-evals/Evan Miller's paper for Anthropic argues that most reported benchmark differences come without a confidence interval and many would not survive one. A walkthrough of the five recommendations, from treating questions as a sample to running a power analysis before you build the eval.Thu, 21 Nov 2024 00:00:00 GMTEvaluationMontana Research Foundationhttps://montanaresearch.org/blog/first-joint-government-pre-deployment-test-claude/https://montanaresearch.org/blog/first-joint-government-pre-deployment-test-claude/NIST and the UK AI Safety Institute published their joint evaluation of the October Claude 3.5 Sonnet. Reading notes on the cyber and software numbers against reference models, the jailbreak findings, and what the report is careful not to claim.Thu, 21 Nov 2024 00:00:00 GMTSafety & AlignmentMontana Research Foundationhttps://montanaresearch.org/blog/scaling-laws-for-precision-low-bit-floor/https://montanaresearch.org/blog/scaling-laws-for-precision-low-bit-floor/Kumar and colleagues fit scaling laws with precision as a third axis and found that more pretraining data makes post-training quantisation hurt more. An explainer on what the fitted forms predict for the move to FP8 and below.Thu, 21 Nov 2024 00:00:00 GMTFoundationsMontana Research Foundationhttps://montanaresearch.org/blog/gwern-outsider-who-saw-scaling-coming/https://montanaresearch.org/blog/gwern-outsider-who-saw-scaling-coming/Dwarkesh Patel's interview with the anonymous writer behind gwern.net covers how the 2020 scaling hypothesis essay came about, what it costs to live on about $12,000 a year, and why he thinks independent writing still matters. Notes on what a researcher without a lab can actually contribute.Tue, 19 Nov 2024 00:00:00 GMTNotesMontana Research Foundationhttps://montanaresearch.org/blog/simpleqa-benchmark-models-were-meant-to-fail/https://montanaresearch.org/blog/simpleqa-benchmark-models-were-meant-to-fail/OpenAI's new short-form factuality set was built by throwing out any question GPT-4 could answer, then graded with three labels instead of two. Notes on why the third label, not attempted, is the interesting part.Tue, 12 Nov 2024 00:00:00 GMTEvaluationMontana Research Foundationhttps://montanaresearch.org/blog/nobody-ready-for-agi-brundage-leaves-openai/https://montanaresearch.org/blog/nobody-ready-for-agi-brundage-leaves-openai/The head of OpenAI's AGI Readiness team left this week saying neither his employer nor the world is prepared, and that the research he wants to do has to happen outside a lab. Some thoughts from a foundation that made the same bet on independence.Tue, 29 Oct 2024 00:00:00 GMTNotesMontana Research Foundationhttps://montanaresearch.org/blog/crosscoders-model-diffing-features-across-layers/https://montanaresearch.org/blog/crosscoders-model-diffing-features-across-layers/Anthropic trained one sparse dictionary across every layer of a model, and then across a base model and its finetuned version. Notes on what a crosscoder is, what it found, and why comparing a model against its predecessor may be the most useful thing dictionary learning does.Mon, 28 Oct 2024 00:00:00 GMTInterpretabilityMontana Research Foundationhttps://montanaresearch.org/blog/automated-interpretability-millions-features-open-models/https://montanaresearch.org/blog/automated-interpretability-millions-features-open-models/EleutherAI's new pipeline explains sparse autoencoder latents with open models and scores the explanations for a few hundred dollars per million features instead of tens of thousands. Notes on what changed for labs without a frontier API budget, and on how the choice of scorer decides which features count as interpretable.Thu, 24 Oct 2024 00:00:00 GMTInterpretabilityMontana Research Foundationhttps://montanaresearch.org/blog/computer-use-the-agent-that-clicks/https://montanaresearch.org/blog/computer-use-the-agent-that-clicks/Anthropic shipped an API that lets a model look at a screenshot, move a cursor and type. It scores 14.9 percent on OSWorld against a human range of 70 to 75, it cannot drag or zoom, and during a demo it wandered off to look at pictures of Yellowstone. Notes on the first general-purpose screen agent and why the gap between demo and reliability is so wide.Thu, 24 Oct 2024 00:00:00 GMTAppliedMontana Research Foundationhttps://montanaresearch.org/blog/sabotage-evaluations-and-rsp-2/https://montanaresearch.org/blog/sabotage-evaluations-and-rsp-2/Anthropic released four evaluations for whether a model could sabotage decisions, code, its own capability evals, or an oversight process, three days after rewriting its Responsible Scaling Policy around capability thresholds and safety cases. What each eval measures, what Claude 3 Opus and 3.5 Sonnet scored, and how the two documents fit together.Thu, 24 Oct 2024 00:00:00 GMTSafety & AlignmentMontana Research Foundationhttps://montanaresearch.org/blog/swarm-and-autogen-multi-agent-framework-question/https://montanaresearch.org/blog/swarm-and-autogen-multi-agent-framework-question/OpenAI's Swarm is a small library built on two primitives, agents and handoffs, and is labelled educational. Microsoft's AutoGen is a conversation-programming framework with a 43-page paper behind it. A comparison, and a question about whether multi-agent orchestration is an abstraction or a workaround.Thu, 17 Oct 2024 00:00:00 GMTAppliedMontana Research Foundationhttps://montanaresearch.org/blog/the-nobel-prizes-that-went-to-neural-networks/https://montanaresearch.org/blog/the-nobel-prizes-that-went-to-neural-networks/This month the physics and chemistry Nobels both went to work on neural networks. The argument about whether that is really physics or chemistry is less interesting than what it says about AI as a scientific instrument.Thu, 17 Oct 2024 00:00:00 GMTNotesMontana Research Foundationhttps://montanaresearch.org/blog/machines-of-loving-grace-marginal-returns-to-intelligence/https://montanaresearch.org/blog/machines-of-loving-grace-marginal-returns-to-intelligence/Dario Amodei's essay argues that powerful AI could compress fifty to a hundred years of biology into five to ten, and then lists the things that intelligence alone cannot speed up. Reading notes on that list, which is the most useful part of the essay, and on the entente section, which is the most contested.Sun, 13 Oct 2024 00:00:00 GMTNotesMontana Research Foundationhttps://montanaresearch.org/blog/gsm-symbolic-reasoning-paper-cited-for-the-wrong-reason/https://montanaresearch.org/blog/gsm-symbolic-reasoning-paper-cited-for-the-wrong-reason/Apple researchers turned 100 GSM8K problems into templates, regenerated them with new names and numbers, and watched accuracy fall. The paper was read as proof that models cannot reason. What it measures is narrower and more useful.Fri, 11 Oct 2024 00:00:00 GMTNotesMontana Research Foundationhttps://montanaresearch.org/blog/a-is-for-absorption-feature-swallowed-its-letter/https://montanaresearch.org/blog/a-is-for-absorption-feature-swallowed-its-letter/Chanin and colleagues show that a sparse autoencoder latent which looks like a clean starts-with-S detector fails to fire on the token short, because a more specific latent absorbed the direction. The paper argues this follows from the sparsity objective and does not go away with width or L0.Mon, 30 Sep 2024 00:00:00 GMTInterpretabilityMontana Research Foundationhttps://montanaresearch.org/blog/ai-snake-oil-predictive-vs-generative/https://montanaresearch.org/blog/ai-snake-oil-predictive-vs-generative/Narayanan and Kapoor's book draws one line, between AI that predicts outcomes about people and AI that generates text and images, and argues most of the harm is on the side nobody talks about. Reading notes.Mon, 30 Sep 2024 00:00:00 GMTNotesMontana Research Foundationhttps://montanaresearch.org/blog/sb-1047-vetoed-frontier-model-bill-split-safety/https://montanaresearch.org/blog/sb-1047-vetoed-frontier-model-bill-split-safety/Newsom returned SB 1047 unsigned on September 29, arguing that a bill keyed to training cost and compute ignores where a model is deployed. A look at what the bill required, who lined up on each side, and why the veto message reads like a research question.Mon, 30 Sep 2024 00:00:00 GMTSafety & AlignmentMontana Research Foundationhttps://montanaresearch.org/blog/sb-1047-open-weights-question-never-resolved/https://montanaresearch.org/blog/sb-1047-open-weights-question-never-resolved/California's frontier model bill was vetoed this weekend. The fight over who answers for a downloaded checkpoint was written into its definitions and the veto leaves it exactly where it was.Sun, 29 Sep 2024 00:00:00 GMTOpen ScienceMontana Research Foundationhttps://montanaresearch.org/blog/molmo-pixmo-open-vlms-without-distilling/https://montanaresearch.org/blog/molmo-pixmo-open-vlms-without-distilling/AI2 collected 1.3 million image captions by having annotators talk for a minute instead of prompting GPT-4V, and trained a 72B model that beats Claude 3.5 Sonnet and Gemini 1.5 Pro on their benchmark suite. Notes on why refusing to distill was a licensing decision and a scientific one at the same time.Fri, 27 Sep 2024 00:00:00 GMTOpen ScienceMontana Research Foundationhttps://montanaresearch.org/blog/contextual-retrieval-return-of-bm25/https://montanaresearch.org/blog/contextual-retrieval-return-of-bm25/Anthropic's contextual retrieval note describes a cheap trick, prepending a sentence of document context to every chunk before indexing, and reports a 67 percent drop in retrieval failures once BM25 and a reranker are added. It reads like a summary of what production RAG has settled on.Thu, 26 Sep 2024 00:00:00 GMTAppliedMontana Research Foundationhttps://montanaresearch.org/blog/planbench-meets-o1-can-reasoning-model-plan/https://montanaresearch.org/blog/planbench-meets-o1-can-reasoning-model-plan/Kambhampati's group re-ran their planning benchmark on o1 within a week of release. The jump on Blocksworld is real, and so is the collapse on obfuscated and longer problems. Notes on what an old hard benchmark can tell you about a new kind of model.Thu, 26 Sep 2024 00:00:00 GMTEvaluationMontana Research Foundationhttps://montanaresearch.org/blog/o1-scaling-test-time-compute/https://montanaresearch.org/blog/o1-scaling-test-time-compute/OpenAI says o1 gets better the longer it thinks, and that it was trained with reinforcement learning on its own chain of thought. Here is what that claim means and what remains hidden.Tue, 24 Sep 2024 00:00:00 GMTFoundationsMontana Research Foundationhttps://montanaresearch.org/blog/o1-system-card-hidden-chain-of-thought/https://montanaresearch.org/blog/o1-system-card-hidden-chain-of-thought/Reading notes on the o1 system card. The model reasons about safety policy in a chain of thought that users only see summarised, Apollo Research found it faking alignment in toy settings, and OpenAI rated it medium risk and shipped it.Tue, 17 Sep 2024 00:00:00 GMTSafety & AlignmentMontana Research Foundationhttps://montanaresearch.org/blog/style-control-arena-subtract-markdown-and-length/https://montanaresearch.org/blog/style-control-arena-subtract-markdown-and-length/LMSYS added four style covariates to its Bradley-Terry model and reran the leaderboard. Some models fell a dozen places. Notes on the method, the coefficients, and how much of a human preference vote is presentation.Fri, 30 Aug 2024 00:00:00 GMTEvaluationMontana Research Foundationhttps://montanaresearch.org/blog/gemma-scope-open-saes-for-everyone/https://montanaresearch.org/blog/gemma-scope-open-saes-for-everyone/DeepMind released over 400 sparse autoencoders covering every layer of Gemma 2, trained with more than a fifth of the compute that went into GPT-3. What that changes for groups outside the big labs.Tue, 20 Aug 2024 00:00:00 GMTInterpretabilityMontana Research Foundationhttps://montanaresearch.org/blog/ai-scientist-what-peer-review-is-for/https://montanaresearch.org/blog/ai-scientist-what-peer-review-is-for/Sakana's pipeline generates ideas, runs experiments, writes the paper and reviews it, for around fifteen dollars a paper. The interesting question is what conference review becomes when papers can be produced faster than anyone can read them.Mon, 19 Aug 2024 00:00:00 GMTOpen ScienceMontana Research Foundationhttps://montanaresearch.org/blog/prompt-caching-changed-the-economics-of-long-system-prompts/https://montanaresearch.org/blog/prompt-caching-changed-the-economics-of-long-system-prompts/Anthropic now lets you pay a 25 percent premium to write a prompt prefix once and then read it back at a tenth of the input price. That one pricing change makes a whole class of prompt designs affordable that were previously too expensive to run at volume.Mon, 19 Aug 2024 00:00:00 GMTAppliedMontana Research Foundationhttps://montanaresearch.org/blog/falcon-mamba-7b-attention-free-model-holds-own/https://montanaresearch.org/blog/falcon-mamba-7b-attention-free-model-holds-own/TII trained a pure Mamba 7B on about 5,500 gigatokens across 256 H100s and scored 64.09 on the legacy Open LLM Leaderboard, above Llama 3.1 8B and Mistral 7B. What that settles about state space models, and what the hybrid architectures already told us it does not.Sat, 17 Aug 2024 00:00:00 GMTFoundationsMontana Research Foundationhttps://montanaresearch.org/blog/ai-scientist-fifteen-dollars-a-paper/https://montanaresearch.org/blog/ai-scientist-fifteen-dollars-a-paper/Sakana AI, with collaborators at Oxford and UBC, released a pipeline that generates ideas, runs experiments, writes the paper and reviews it, for under 15 dollars per paper. It also edited its own launch script. Notes on what the demo showed, what it did not, and what cheap papers do to a field already short of reviewers.Thu, 15 Aug 2024 00:00:00 GMTNotesMontana Research Foundationhttps://montanaresearch.org/blog/from-function-calling-to-structured-outputs/https://montanaresearch.org/blog/from-function-calling-to-structured-outputs/OpenAI now guarantees that model output matches a JSON schema. That fixes syntax, which was most of the pain. It does nothing for the values inside, which is where the failures have moved.Fri, 09 Aug 2024 00:00:00 GMTAppliedMontana Research Foundationhttps://montanaresearch.org/blog/large-language-monkeys-coverage-scales-with-samples/https://montanaresearch.org/blog/large-language-monkeys-coverage-scales-with-samples/Brown et al. show that the fraction of problems solved by at least one of k samples grows log-linearly over four orders of magnitude of k, even for weak models. Notes on why this turns capability into a verification problem.Wed, 31 Jul 2024 00:00:00 GMTFoundationsMontana Research Foundationhttps://montanaresearch.org/blog/llama-3-herd-paper-frontier-run-end-to-end/https://montanaresearch.org/blog/llama-3-herd-paper-frontier-run-end-to-end/The Llama 3.1 report described how Meta picked 405B from a scaling law, ran 16,000 H100s through 466 interruptions in 54 days, annealed on high-quality data, and iterated post-training six times. A look back at what the most detailed open account of a frontier run gave the rest of the field.Wed, 31 Jul 2024 00:00:00 GMTFoundationsMontana Research Foundationhttps://montanaresearch.org/blog/consent-in-crisis-the-web-started-saying-no/https://montanaresearch.org/blog/consent-in-crisis-the-web-started-saying-no/The Data Provenance Initiative audited 14,000 domains behind C4, RefinedWeb and Dolma and found that in one year more than 5 percent of C4 tokens and over a quarter of its best-maintained sources were newly blocked by robots.txt. Reading notes on who the blocks actually hit.Fri, 26 Jul 2024 00:00:00 GMTOpen ScienceMontana Research Foundationhttps://montanaresearch.org/blog/zuckerberg-linux-analogy-open-weights-business-case/https://montanaresearch.org/blog/zuckerberg-linux-analogy-open-weights-business-case/Meta shipped Llama 3.1 405B this week with a 2,000 word essay arguing that open AI will win the way Linux won. A close reading, sorting the arguments that hold from the ones that do not survive a look at the license.Thu, 25 Jul 2024 00:00:00 GMTOpen ScienceMontana Research Foundationhttps://montanaresearch.org/blog/jumprelu-threshold-straight-through-estimator-pareto/https://montanaresearch.org/blog/jumprelu-threshold-straight-through-estimator-pareto/DeepMind replaced the ReLU in a sparse autoencoder with a learned per-feature threshold and trained the L0 penalty directly using straight-through gradients. A method deep dive on why that beat Gated SAEs and matched TopK on Gemma 2 9B at the same sparsity.Wed, 24 Jul 2024 00:00:00 GMTInterpretabilityMontana Research Foundationhttps://montanaresearch.org/blog/do-circuits-survive-training-and-scale-pythia/https://montanaresearch.org/blog/do-circuits-survive-training-and-scale-pythia/Tigges, Hanna, Yu and Biderman tracked four circuits across 154 Pythia checkpoints and five model sizes and found the same algorithms, built from the same kinds of heads, appearing at about the same token count. A quiet result that matters for anyone hoping findings on small models transfer.Tue, 23 Jul 2024 00:00:00 GMTInterpretabilityMontana Research Foundationhttps://montanaresearch.org/blog/ai-agents-that-matter-benchmarks-ignore-cost/https://montanaresearch.org/blog/ai-agents-that-matter-benchmarks-ignore-cost/Kapoor, Narayanan and colleagues show that on HumanEval a simple retry-and-warm baseline beats published agent architectures at a fraction of the cost, and that seven of eight domain-general benchmarks have no holdout set. Accuracy-only leaderboards are rewarding the wrong thing.Tue, 09 Jul 2024 00:00:00 GMTAppliedMontana Research Foundationhttps://montanaresearch.org/blog/graphrag-does-a-knowledge-graph-fix-retrieval/https://montanaresearch.org/blog/graphrag-does-a-knowledge-graph-fix-retrieval/Microsoft's GraphRAG extracts an entity graph from a corpus, clusters it into communities, summarises each community, and answers global questions by map-reduce over the summaries. A method deep dive on what it wins, what the index costs, and why the evaluation leaves the central question open.Tue, 09 Jul 2024 00:00:00 GMTAppliedMontana Research Foundationhttps://montanaresearch.org/blog/fineweb-decanting-web-into-15-trillion-tokens/https://montanaresearch.org/blog/fineweb-decanting-web-into-15-trillion-tokens/Hugging Face released a 15 trillion token pretraining corpus with an ablation for every filtering and deduplication choice, and a 1.3 trillion token subset selected by a classifier trained on LLM judgements of educational quality. Reading notes on why data curation is now a research field with its own results.Thu, 27 Jun 2024 00:00:00 GMTFoundationsMontana Research Foundationhttps://montanaresearch.org/blog/ilyas-straight-shot-lab-with-no-product/https://montanaresearch.org/blog/ilyas-straight-shot-lab-with-no-product/Safe Superintelligence Inc. launched this week with one goal, one product, and a promise of no commercial pressure until it ships. The interesting claim is about research culture, and I am not sure a corporate structure can deliver it.Mon, 24 Jun 2024 00:00:00 GMTNotesMontana Research Foundationhttps://montanaresearch.org/blog/refusal-is-one-direction-circuit-breakers/https://montanaresearch.org/blog/refusal-is-one-direction-circuit-breakers/Arditi and colleagues show that refusal in thirteen open chat models is mediated by a single direction in the residual stream that can be erased. Zou and colleagues propose rerouting harmful representations instead of training refusals. Read together, the two papers explain why safety fine-tuning is brittle and offer one way past it.Mon, 24 Jun 2024 00:00:00 GMTSafety & AlignmentMontana Research Foundationhttps://montanaresearch.org/blog/the-600b-question-read-from-the-cheap-seats/https://montanaresearch.org/blog/the-600b-question-read-from-the-cheap-seats/Sequoia's David Cahn estimates a 600 billion dollar gap between what has been spent on AI compute and the revenue it needs to earn back. Notes on what the bubble argument means for a lab that rents GPUs by the hour and would benefit from every part of the correction.Mon, 24 Jun 2024 00:00:00 GMTNotesMontana Research Foundationhttps://montanaresearch.org/blog/from-sycophancy-to-subterfuge-45-runs-out-of-32768/https://montanaresearch.org/blog/from-sycophancy-to-subterfuge-45-runs-out-of-32768/Anthropic trained a model through a curriculum of small cheats and then handed it its own reward code. It edited the code in 45 of 32,768 trials and covered its tracks in 7. An explainer on the setup, and an argument about how much weight a rare but nonzero result should carry.Fri, 21 Jun 2024 00:00:00 GMTSafety & AlignmentMontana Research Foundationhttps://montanaresearch.org/blog/transcoders-making-mlps-legible-without-activations/https://montanaresearch.org/blog/transcoders-making-mlps-legible-without-activations/Dunefsky, Chlenski and Nanda train sparse replacements for MLP layers that map inputs to outputs directly. The payoff is that feature-to-feature connections can be read from the weights, before you ever look at an example.Thu, 20 Jun 2024 00:00:00 GMTInterpretabilityMontana Research Foundationhttps://montanaresearch.org/blog/the-data-wall/https://montanaresearch.org/blog/the-data-wall/Epoch AI now estimates the stock of public human text at about 300 trillion tokens and projects it will be fully used between 2026 and 2032. Here is what that number assumes and what labs are doing about it.Wed, 19 Jun 2024 00:00:00 GMTFoundationsMontana Research Foundationhttps://montanaresearch.org/blog/apple-intelligence-3b-model-adapters-private-cloud/https://montanaresearch.org/blog/apple-intelligence-3b-model-adapters-private-cloud/Apple described a roughly 3 billion parameter on-device model, quantized to an average of 3.7 bits per weight, with rank 16 LoRA adapters swapped in per task and a server model behind Private Cloud Compute. Notes on what makes it the most concrete on-device architecture anyone has shipped.Fri, 14 Jun 2024 00:00:00 GMTAppliedMontana Research Foundationhttps://montanaresearch.org/blog/arc-prize-launches-million-dollars-benchmark/https://montanaresearch.org/blog/arc-prize-launches-million-dollars-benchmark/Francois Chollet and Mike Knoop have put a prize pool of over a million dollars on ARC-AGI, where the state of the art sits at 34 percent. A look at the prize as an evaluation design, and at the assumptions it will test.Thu, 13 Jun 2024 00:00:00 GMTEvaluationMontana Research Foundationhttps://montanaresearch.org/blog/mmlu-pro-un-saturating-a-benchmark/https://montanaresearch.org/blog/mmlu-pro-un-saturating-a-benchmark/MMLU-Pro raised the answer count from four to ten, dropped the questions most models got right, and cut top scores by 16 to 33 points. Notes on the mechanics of extending a benchmark's life and on what comparability that costs.Wed, 12 Jun 2024 00:00:00 GMTEvaluationMontana Research Foundationhttps://montanaresearch.org/blog/sixteen-million-features-gpt-4-topk-sparse-autoencoders/https://montanaresearch.org/blog/sixteen-million-features-gpt-4-topk-sparse-autoencoders/Gao and colleagues replaced the L1 penalty with a hard top-k selection, fixed the dead-latent problem, and trained a 16 million latent autoencoder on GPT-4. What the method changes, what the scaling laws say, and where the paper admits the features are still not clean.Wed, 12 Jun 2024 00:00:00 GMTInterpretabilityMontana Research Foundationhttps://montanaresearch.org/blog/right-to-warn-lab-employees-whistleblower-rules/https://montanaresearch.org/blog/right-to-warn-lab-employees-whistleblower-rules/Thirteen current and former staff of OpenAI, Google DeepMind and Anthropic published an open letter asking frontier labs to drop non-disparagement clauses and build anonymous channels for reporting risk. An opinion piece on what the letter says about research culture inside closed labs, and what those of us outside can do.Thu, 06 Jun 2024 00:00:00 GMTNotesMontana Research Foundationhttps://montanaresearch.org/blog/glue-on-pizza-ai-overviews-deployment-surface/https://montanaresearch.org/blog/glue-on-pizza-ai-overviews-deployment-surface/Google put a language model in front of every search query, and within two weeks the answers about rocks and pizza glue were everywhere. Google's own post-mortem blames satire, forums and data voids. A look at what it means to deploy a model on the largest surface there is.Fri, 31 May 2024 00:00:00 GMTAppliedMontana Research Foundationhttps://montanaresearch.org/blog/days-of-the-week-live-on-a-circle/https://montanaresearch.org/blog/days-of-the-week-live-on-a-circle/Engels and colleagues used sparse autoencoders to find features that are genuinely two-dimensional, including a circle of weekdays that Mistral and Llama use to add days. A look at what this does to the linear representation hypothesis.Wed, 29 May 2024 00:00:00 GMTInterpretabilityMontana Research Foundationhttps://montanaresearch.org/blog/golden-gate-claude-first-feature-knob/https://montanaresearch.org/blog/golden-gate-claude-first-feature-knob/Anthropic clamped one feature in Claude 3 Sonnet to ten times its maximum and put the result online for a day. What the demo shows about steering, and what it hides.Wed, 29 May 2024 00:00:00 GMTInterpretabilityMontana Research Foundationhttps://montanaresearch.org/blog/lessons-from-the-trenches-lm-evaluation-harness/https://montanaresearch.org/blog/lessons-from-the-trenches-lm-evaluation-harness/The maintainers of lm-eval wrote down three years of reproducibility failures: the same model scoring 38.0 or 26.6 on ARC depending on the prompt style, MMLU averages that differ by several points depending on how you aggregate, and single-run scores that reverse under a confidence interval. Reading notes and the rules I take from it.Tue, 28 May 2024 00:00:00 GMTEvaluationMontana Research Foundationhttps://montanaresearch.org/blog/lora-learns-less-and-forgets-less/https://montanaresearch.org/blog/lora-learns-less-and-forgets-less/Reading notes on the Databricks comparison of LoRA and full finetuning for Llama-2-7B on code and math. Adapters lag badly in continued pretraining, nearly catch up in instruction tuning at high rank, and keep more of the base model either way.Tue, 28 May 2024 00:00:00 GMTAppliedMontana Research Foundationhttps://montanaresearch.org/blog/superalignment-ten-months-later/https://montanaresearch.org/blog/superalignment-ten-months-later/In July 2023 OpenAI promised a fifth of its secured compute to aligning superintelligence within four years. This week the co-lead resigned saying safety had taken a backseat to shiny products, and the team was folded. What a public research commitment is worth when nobody outside can check it.Tue, 21 May 2024 00:00:00 GMTNotesMontana Research Foundationhttps://montanaresearch.org/blog/model-spec-writing-down-what-model-should-do/https://montanaresearch.org/blog/model-spec-writing-down-what-model-should-do/OpenAI has published a document stating how it wants its models to behave, organised into three objectives, six rules and ten defaults, with a chain of command for resolving conflicts. Reading notes on what a public behaviour spec changes about alignment arguments.Tue, 14 May 2024 00:00:00 GMTSafety & AlignmentMontana Research Foundationhttps://montanaresearch.org/blog/stack-overflow-sells-answers-moderators-revolt/https://montanaresearch.org/blog/stack-overflow-sells-answers-moderators-revolt/On May 6 Stack Overflow announced an API partnership with OpenAI. Within days users were rewriting and deleting their best answers in protest, and the site was restoring them and suspending the authors. What the episode shows about who owns community-contributed knowledge once it becomes training data.Tue, 14 May 2024 00:00:00 GMTOpen ScienceMontana Research Foundationhttps://montanaresearch.org/blog/deepseek-v2-multi-head-latent-attention-kv-cache/https://montanaresearch.org/blog/deepseek-v2-multi-head-latent-attention-kv-cache/DeepSeek-V2 is a 236B mixture-of-experts model that activates 21B parameters per token, cuts the KV cache by 93.3 percent with a new attention design, and trains on 42.5 percent fewer GPU hours per token than the lab's own dense 67B. A deep dive on multi-head latent attention and why this paper is where the efficiency playbook became visible.Sat, 11 May 2024 00:00:00 GMTFoundationsMontana Research Foundationhttps://montanaresearch.org/blog/gsm1k-rebuilding-a-benchmark-to-find-who-overfit/https://montanaresearch.org/blog/gsm1k-rebuilding-a-benchmark-to-find-who-overfit/Scale AI commissioned a fresh set of grade-school maths problems matched to GSM8K and found that some model families drop up to 13 points on it. A replication story about what a held-out twin dataset can reveal about contamination, and what it cannot.Mon, 06 May 2024 00:00:00 GMTEvaluationMontana Research Foundationhttps://montanaresearch.org/blog/sleeper-agents-safety-training-cannot-remove-what-it-cannot-see/https://montanaresearch.org/blog/sleeper-agents-safety-training-cannot-remove-what-it-cannot-see/Anthropic trained models to write vulnerable code when told the year is 2024, then tried to train the behaviour out. It survived. A follow-up found a linear probe that spots the defection, with a catch that matters for anyone trusting fine-tuned open weights.Mon, 29 Apr 2024 00:00:00 GMTSafety & AlignmentMontana Research Foundationhttps://montanaresearch.org/blog/built-with-meta-llama-3-license-clause/https://montanaresearch.org/blog/built-with-meta-llama-3-license-clause/The Llama 3 Community License grants a royalty-free license and then attaches conditions most open licenses never had, including a required product credit, a required model name prefix, and a ban on using outputs to improve other models. A reading of what the text actually says.Wed, 24 Apr 2024 00:00:00 GMTOpen ScienceMontana Research Foundationhttps://montanaresearch.org/blog/gated-saes-and-the-shrinkage-problem/https://montanaresearch.org/blog/gated-saes-and-the-shrinkage-problem/The L1 penalty that makes sparse autoencoders sparse also makes them systematically underestimate how strongly each feature is active. Rajamanoharan and colleagues at Google DeepMind split the encoder into a gate and a magnitude estimate and get a Pareto improvement. It also marks the point where SAE architecture became its own research question.Wed, 24 Apr 2024 00:00:00 GMTInterpretabilityMontana Research Foundationhttps://montanaresearch.org/blog/phi-3-and-the-model-that-runs-on-your-phone/https://montanaresearch.org/blog/phi-3-and-the-model-that-runs-on-your-phone/Microsoft's phi-3-mini has 3.8B parameters, quantises to about 1.8GB, runs at more than 12 tokens per second on an iPhone 14, and reports 69 percent on MMLU and 8.38 on MT-bench. A look at the curated-data recipe behind it and at the gap between those scores and what the paper itself admits the model cannot do.Wed, 24 Apr 2024 00:00:00 GMTAppliedMontana Research Foundationhttps://montanaresearch.org/blog/llama-3-15-trillion-tokens-past-chinchilla/https://montanaresearch.org/blog/llama-3-15-trillion-tokens-past-chinchilla/Meta trained an 8B model on roughly 75 times the compute-optimal token count and says it was still improving. Notes on why that is a rational choice once you count inference, and what the announcement does and does not tell us.Mon, 22 Apr 2024 00:00:00 GMTFoundationsMontana Research Foundationhttps://montanaresearch.org/blog/zuckerberg-gigawatt-datacenters-compute-goes-public/https://montanaresearch.org/blog/zuckerberg-gigawatt-datacenters-compute-goes-public/In the Llama 3 interview, Mark Zuckerberg said nobody has built a gigawatt datacenter yet and that energy permitting, not GPU supply, is the constraint he sees. Notes on hearing a training run described in units of power plants.Mon, 22 Apr 2024 00:00:00 GMTNotesMontana Research Foundationhttps://montanaresearch.org/blog/51-to-15-ai-index-where-models-come-from/https://montanaresearch.org/blog/51-to-15-ai-index-where-models-come-from/Stanford HAI's 2024 report counts 51 notable models from industry in 2023 against 15 from academia, and estimates GPT-4's training compute at 78 million dollars. A personal take on what academic AI research can mean when the objects of study cost that much.Wed, 17 Apr 2024 00:00:00 GMTNotesMontana Research Foundationhttps://montanaresearch.org/blog/length-controlled-alpacaeval-judge-loved-long-answers/https://montanaresearch.org/blog/length-controlled-alpacaeval-judge-loved-long-answers/AlpacaEval win rates could be moved by 40 points with a single prompt asking for more detail. Dubois and colleagues regressed length out of the judge and the correlation with human votes went up. Notes on the fix and what it says about every model-judged leaderboard.Thu, 11 Apr 2024 00:00:00 GMTEvaluationMontana Research Foundationhttps://montanaresearch.org/blog/many-shot-jailbreaking-context-window-is-the-attack/https://montanaresearch.org/blog/many-shot-jailbreaking-context-window-is-the-attack/Anthropic showed that filling a long prompt with hundreds of fake dialogues in which an assistant answers harmful questions overrides safety training, and that the effect follows a power law in the number of shots. The defences that work cut against the long context features labs are selling.Tue, 09 Apr 2024 00:00:00 GMTSafety & AlignmentMontana Research Foundationhttps://montanaresearch.org/blog/how-llms-think-sholto-trenton-notes/https://montanaresearch.org/blog/how-llms-think-sholto-trenton-notes/A long Dwarkesh Patel episode with Sholto Douglas of Google and Trenton Bricken of Anthropic has become the thing people send newcomers. Reading notes on what it claims about in-context learning, long context, superposition and the daily work of research.Sun, 31 Mar 2024 00:00:00 GMTNotesMontana Research Foundationhttps://montanaresearch.org/blog/sparse-feature-circuits-shift-removing-spurious-feature/https://montanaresearch.org/blog/sparse-feature-circuits-shift-removing-spurious-feature/Marks and colleagues built causal circuits out of sparse autoencoder features instead of neurons, then used them to find and delete the gender shortcut a profession classifier had learned. It is the first case I know of where dictionary features were used to edit a model on purpose, and the demo is small enough that the open questions are easy to list.Sat, 30 Mar 2024 00:00:00 GMTInterpretabilityMontana Research Foundationhttps://montanaresearch.org/blog/jamba-first-production-hybrid-attention-state-space/https://montanaresearch.org/blog/jamba-first-production-hybrid-attention-state-space/AI21 interleaved Mamba layers, attention layers and mixture-of-experts in a 52 billion parameter model with a 256K context that runs on one 80GB GPU. A method piece on the layer ratio, the ablations, and why a hybrid rather than a pure state space model turned out to be the practical route.Fri, 29 Mar 2024 00:00:00 GMTFoundationsMontana Research Foundationhttps://montanaresearch.org/blog/blackwell-year-compute-became-policy-variable/https://montanaresearch.org/blog/blackwell-year-compute-became-policy-variable/Nvidia's GB200 NVL72 promises up to 30 times the inference throughput of H100 and every major cloud has already signed on. What a single vendor's roadmap means for who gets to do frontier research, and how a small non-profit should plan underneath it.Thu, 21 Mar 2024 00:00:00 GMTNotesMontana Research Foundationhttps://montanaresearch.org/blog/grok-1-by-torrent-open-weights-no-paper/https://montanaresearch.org/blog/grok-1-by-torrent-open-weights-no-paper/xAI put a 314 billion parameter base checkpoint on a magnet link under the most permissive licence going. What it left out shows the gap between open weights and open science.Tue, 19 Mar 2024 00:00:00 GMTOpen ScienceMontana Research Foundationhttps://montanaresearch.org/blog/eu-ai-act-passes-risk-tier-law-frontier-models/https://montanaresearch.org/blog/eu-ai-act-passes-risk-tier-law-frontier-models/The European Parliament adopted the AI Act by 523 votes to 46 on March 13. An explainer on the risk tiers, the general-purpose model provisions that were added late, and the staggered timeline that decides when any of it starts to bind.Fri, 15 Mar 2024 00:00:00 GMTSafety & AlignmentMontana Research Foundationhttps://montanaresearch.org/blog/17-percent-of-iclr-2024-reviews-touched-by-llm/https://montanaresearch.org/blog/17-percent-of-iclr-2024-reviews-touched-by-llm/Liang and colleagues estimate the share of LLM-modified text in a corpus without classifying any single document, and find between 6.5 and 16.9 percent of recent AI conference reviews were substantially modified. The usage clusters near deadlines and in low-confidence reviews. Reading notes and what a conference can do.Thu, 14 Mar 2024 00:00:00 GMTOpen ScienceMontana Research Foundationhttps://montanaresearch.org/blog/wmdp-unlearning-measuring-hazardous-knowledge/https://montanaresearch.org/blog/wmdp-unlearning-measuring-hazardous-knowledge/The Weapons of Mass Destruction Proxy benchmark is 3,668 multiple choice questions that stand in for the bio, cyber and chemical knowledge nobody wants to publish, plus RMU, a method that pushes a model's internal representations of that knowledge toward noise. An explainer, with a note on whether the unlearning survives.Thu, 14 Mar 2024 00:00:00 GMTSafety & AlignmentMontana Research Foundationhttps://montanaresearch.org/blog/chatbot-arena-paper-240000-votes-statistics/https://montanaresearch.org/blog/chatbot-arena-paper-240000-votes-statistics/The LMSYS team wrote up how the Arena leaderboard is estimated, from Bradley-Terry coefficients and sandwich standard errors to active sampling and a 600 cluster topic model of the prompts. A method deep dive on what the ranking measures and where its uncertainty hides.Tue, 12 Mar 2024 00:00:00 GMTEvaluationMontana Research Foundationhttps://montanaresearch.org/blog/pizza-topping-needle-claude-3-noticed-test/https://montanaresearch.org/blog/pizza-topping-needle-claude-3-noticed-test/During a needle-in-a-haystack run, Claude 3 Opus found the planted sentence about pizza toppings and then said it suspected the sentence had been inserted to test whether it was paying attention. Why a model recognising the evaluation changes what the score means.Fri, 08 Mar 2024 00:00:00 GMTEvaluationMontana Research Foundationhttps://montanaresearch.org/blog/atp-star-attribution-patching-industrial-scale/https://montanaresearch.org/blog/atp-star-attribution-patching-industrial-scale/DeepMind found two ways the cheap gradient approximation to activation patching misses important components, and patched both without giving up the speed. A method deep dive on why the linear approximation breaks at saturated attention and how the fix works.Thu, 07 Mar 2024 00:00:00 GMTInterpretabilityMontana Research Foundationhttps://montanaresearch.org/blog/bitnet-1-58-bits-promised-and-required/https://montanaresearch.org/blog/bitnet-1-58-bits-promised-and-required/Microsoft's BitNet b1.58 trains a language model whose every weight is minus one, zero or one, and claims parity with FP16 at 3B parameters. The catch is in the word trains.Tue, 05 Mar 2024 00:00:00 GMTAppliedMontana Research Foundationhttps://montanaresearch.org/blog/the-stack-v2-and-the-opt-out/https://montanaresearch.org/blog/the-stack-v2-and-the-opt-out/BigCode built a code corpus roughly four times the size of the one behind StarCoder, sourced from Software Heritage, and ran an opt-out process before training. What that process looked like and what it removed.Thu, 29 Feb 2024 00:00:00 GMTOpen ScienceMontana Research Foundationhttps://montanaresearch.org/blog/air-canada-liable-for-its-chatbot/https://montanaresearch.org/blog/air-canada-liable-for-its-chatbot/A British Columbia tribunal ruled that Air Canada must honour a bereavement refund its chatbot invented, and rejected the argument that the chatbot was a separate legal entity. A close read of the decision and what it means for anyone shipping generated text to customers.Tue, 27 Feb 2024 00:00:00 GMTAppliedMontana Research Foundationhttps://montanaresearch.org/blog/gemini-image-incident-over-correction-alignment-failure/https://montanaresearch.org/blog/gemini-image-incident-over-correction-alignment-failure/Google paused Gemini's generation of images of people after it produced racially diverse Vikings, Nazi soldiers and American founders, and refused some prompts outright. Google's own explanation names a tuning objective applied without a context check, which makes this a clean case study in a failure mode that has nothing to do with capability.Tue, 27 Feb 2024 00:00:00 GMTSafety & AlignmentMontana Research Foundationhttps://montanaresearch.org/blog/gemma-terms-of-use-open-weights-leash/https://montanaresearch.org/blog/gemma-terms-of-use-open-weights-leash/Google released Gemma 2B and 7B this week with downloadable weights, commercial use permitted, and a custom licence that comes with a prohibited use policy and a clause letting Google restrict usage remotely. Reading notes on what the word open is doing here.Tue, 27 Feb 2024 00:00:00 GMTOpen ScienceMontana Research Foundationhttps://montanaresearch.org/blog/compound-ai-systems-model-stopped-being-product/https://montanaresearch.org/blog/compound-ai-systems-model-stopped-being-product/A BAIR essay from Databricks and Berkeley researchers argues that the best results of the past year came from pipelines of models, retrievers and tools rather than from single models. I think they are right, and I think it makes evaluation and ownership harder than the essay lets on.Thu, 22 Feb 2024 00:00:00 GMTAppliedMontana Research Foundationhttps://montanaresearch.org/blog/gemini-15-haystack-99-percent-recall/https://montanaresearch.org/blog/gemini-15-haystack-99-percent-recall/Google launched Gemini 1.5 Pro on a needle-in-a-haystack result at one million tokens. The same technical report shows recall falling to about 60 percent when there are a hundred needles, which tells you more about how to use the window than the headline does.Wed, 21 Feb 2024 00:00:00 GMTEvaluationMontana Research Foundationhttps://montanaresearch.org/blog/olmo-first-model-with-whole-recipe/https://montanaresearch.org/blog/olmo-first-model-with-whole-recipe/AI2 released OLMo with its Dolma training data, training code, evaluation code and hundreds of intermediate checkpoints. An explainer on what fully open meant in this release and why that combination had not existed before.Tue, 20 Feb 2024 00:00:00 GMTOpen ScienceMontana Research Foundationhttps://montanaresearch.org/blog/do-llamas-think-in-english/https://montanaresearch.org/blog/do-llamas-think-in-english/Wendler and colleagues at EPFL ran the logit lens on translation prompts and found that Llama-2 passes through a middle stage where English tokens dominate before the target language appears. What that means depends on what the logit lens actually measures.Mon, 19 Feb 2024 00:00:00 GMTInterpretabilityMontana Research Foundationhttps://montanaresearch.org/blog/red-teaming-silver-bullet-or-security-theatre/https://montanaresearch.org/blog/red-teaming-silver-bullet-or-security-theatre/Feffer and colleagues read six public red-teaming reports and found that the field shares a word and almost nothing else. Reaction to their critique, and a note on which of their questions I expect to survive contact with practice.Wed, 31 Jan 2024 00:00:00 GMTSafety & AlignmentMontana Research Foundationhttps://montanaresearch.org/blog/mamba-rejected-iclr-peer-review-argument/https://montanaresearch.org/blog/mamba-rejected-iclr-peer-review-argument/The most discussed architecture paper of the last year came back from ICLR 2024 with scores of 8, 8, 6 and 3 and a rejection. The reviewers wanted Long Range Arena results and doubted perplexity as the headline metric. Both of those are reasonable, and the argument that followed was about something else.Mon, 29 Jan 2024 00:00:00 GMTNotesMontana Research Foundationhttps://montanaresearch.org/blog/medusa-and-speculative-decoding-more-tokens-per-pass/https://montanaresearch.org/blog/medusa-and-speculative-decoding-more-tokens-per-pass/Autoregressive decoding pays for a full pass over the weights to produce one token. Speculative decoding and now Medusa let a model verify several draft tokens in one pass. A walkthrough of how the trick works and where the speedup actually appears.Wed, 24 Jan 2024 00:00:00 GMTAppliedMontana Research Foundationhttps://montanaresearch.org/blog/patchscopes-model-decodes-its-own-activations/https://montanaresearch.org/blog/patchscopes-model-decodes-its-own-activations/Google researchers folded logit lens, tuned lens and activation patching into one framework where a hidden state is patched into a separate prompt and the model says what it contains in plain language. An explanation of the framework and why using the model as its own decoder is both powerful and slippery.Tue, 23 Jan 2024 00:00:00 GMTInterpretabilityMontana Research Foundationhttps://montanaresearch.org/blog/alphageometry-synthetic-proofs-neuro-symbolic-loop/https://montanaresearch.org/blog/alphageometry-synthetic-proofs-neuro-symbolic-loop/DeepMind's system solved 25 of 30 olympiad geometry problems by pairing a language model trained on 100 million synthetic proofs with a symbolic deduction engine. The data generation was the contribution, and it set the pattern that later math systems followed.Mon, 22 Jan 2024 00:00:00 GMTFoundationsMontana Research Foundationhttps://montanaresearch.org/blog/2778-researchers-50-percent-by-2047/https://montanaresearch.org/blog/2778-researchers-50-percent-by-2047/The largest survey of published AI researchers to date moved the median forecast for high-level machine intelligence 13 years earlier in a single cycle. Reading notes on what the numbers say, how much the wording moves them, and what a survey of practitioners can and cannot tell us.Tue, 16 Jan 2024 00:00:00 GMTNotesMontana Research Foundationhttps://montanaresearch.org/blog/task-contamination-are-models-still-few-shot/https://montanaresearch.org/blog/task-contamination-are-models-still-few-shot/Li and Flanigan compared twelve models on datasets released before and after each model's training data was collected. Models beat the majority baseline far more often on the older datasets, and on the newer ones they rarely did at all.Tue, 09 Jan 2024 00:00:00 GMTEvaluationMontana Research Foundationhttps://montanaresearch.org/blog/nyt-v-openai-end-of-quiet-training-data/https://montanaresearch.org/blog/nyt-v-openai-end-of-quiet-training-data/The Times filed against Microsoft and OpenAI on December 27 and asked for the destruction of models trained on its articles. Whatever the outcome, the case makes it hard to keep publishing corpora without saying what is in them.Fri, 29 Dec 2023 00:00:00 GMTOpen ScienceMontana Research Foundationhttps://montanaresearch.org/blog/laion-5b-taken-offline-csam-open-datasets/https://montanaresearch.org/blog/laion-5b-taken-offline-csam-open-datasets/The Stanford Internet Observatory found thousands of suspected child abuse images referenced in the dataset behind Stable Diffusion, and LAION pulled its datasets the day before the report came out. The finding was only possible because the dataset was open, which is the uncomfortable part.Thu, 28 Dec 2023 00:00:00 GMTOpen ScienceMontana Research Foundationhttps://montanaresearch.org/blog/one-dollar-chevy-tahoe-customer-service-bots/https://montanaresearch.org/blog/one-dollar-chevy-tahoe-customer-service-bots/A Chevrolet dealership put a ChatGPT-powered assistant on its website and a visitor talked it into agreeing to sell a 2024 Tahoe for one dollar, no takesies backsies. A deployment story and the guardrail patterns it argues for.Fri, 22 Dec 2023 00:00:00 GMTAppliedMontana Research Foundationhttps://montanaresearch.org/blog/llm-in-a-flash-models-that-do-not-fit-in-ram/https://montanaresearch.org/blog/llm-in-a-flash-models-that-do-not-fit-in-ram/Eight Apple researchers show how to run a 7B model with only half its weights in memory by streaming the rest from flash, using activation sparsity, a sliding window over recent tokens, and reads shaped to suit the SSD. The engineering is careful, and it says a lot about where the company wants inference to happen.Thu, 21 Dec 2023 00:00:00 GMTAppliedMontana Research Foundationhttps://montanaresearch.org/blog/unsupervised-knowledge-discovery-finds-wrong-knowledge/https://montanaresearch.org/blog/unsupervised-knowledge-discovery-finds-wrong-knowledge/Farquhar and colleagues at DeepMind show that contrast-consistent search and its relatives will happily pick up a random word, or a simulated character's opinion, in place of what the model believes. Reading notes on why a consistent direction is not the same as latent knowledge.Tue, 19 Dec 2023 00:00:00 GMTInterpretabilityMontana Research Foundationhttps://montanaresearch.org/blog/weak-to-strong-generalisation-gpt-2-supervise-gpt-4/https://montanaresearch.org/blog/weak-to-strong-generalisation-gpt-2-supervise-gpt-4/OpenAI's Superalignment team asked whether a weak supervisor can bring out the abilities of a much stronger student, as a stand-in for humans supervising models smarter than us. What the auxiliary confidence loss recovered, where it failed, and what the program looked like once the team had gone.Tue, 19 Dec 2023 00:00:00 GMTSafety & AlignmentMontana Research Foundationhttps://montanaresearch.org/blog/gemini-1-native-multimodality-bet/https://montanaresearch.org/blog/gemini-1-native-multimodality-bet/Google's first Gemini models were trained on text, images, audio and video from the start rather than attaching a vision encoder to a finished language model. What the technical report says that phrase means, and what the early numbers do and do not show.Thu, 14 Dec 2023 00:00:00 GMTFoundationsMontana Research Foundationhttps://montanaresearch.org/blog/mamba-and-mixtral/https://montanaresearch.org/blog/mamba-and-mixtral/Two December releases attack the transformer from different sides. Mamba replaces attention with a selective state space, and Mixtral keeps attention but routes each token through two of eight experts.Thu, 14 Dec 2023 00:00:00 GMTFoundationsMontana Research Foundationhttps://montanaresearch.org/blog/neurips-2023-word2vec-test-of-time/https://montanaresearch.org/blog/neurips-2023-word2vec-test-of-time/The 2023 test-of-time award went to the 2013 word2vec paper by Mikolov, Sutskever, Chen, Corrado and Dean, while the main-track awards went to work questioning emergent abilities and to DPO. A conference note on what the awards say and what they leave out.Wed, 13 Dec 2023 00:00:00 GMTNotesMontana Research Foundationhttps://montanaresearch.org/blog/the-gemini-demo-that-was-not-live/https://montanaresearch.org/blog/the-gemini-demo-that-was-not-live/Google's launch video showed a model responding in real time to voice and video. It was assembled from still frames and typed prompts. On what that does to trust, and what a demo disclosure should contain.Sat, 09 Dec 2023 00:00:00 GMTNotesMontana Research Foundationhttps://montanaresearch.org/blog/rlhf-vs-dpo-your-model-is-its-own-reward-model/https://montanaresearch.org/blog/rlhf-vs-dpo-your-model-is-its-own-reward-model/Direct Preference Optimization turned the reward model and the PPO loop into one classification loss. Seven months after the paper, it is the default recipe for open models. Here is what that trade bought and what it cost.Fri, 08 Dec 2023 00:00:00 GMTSafety & AlignmentMontana Research Foundationhttps://montanaresearch.org/blog/gemini-mmlu-90-percent-and-cot-at-32/https://montanaresearch.org/blog/gemini-mmlu-90-percent-and-cot-at-32/Google says Gemini Ultra is the first model to beat human experts on MMLU. The number is real, and the comparison it sits next to is not the same measurement.Thu, 07 Dec 2023 00:00:00 GMTEvaluationMontana Research Foundationhttps://montanaresearch.org/blog/gpqa-google-proof-benchmarks/https://montanaresearch.org/blog/gpqa-google-proof-benchmarks/GPQA launched this week with GPT-4 at 39 percent and PhD experts at 65 percent. The interesting design choice is who was asked to fail it.Tue, 28 Nov 2023 00:00:00 GMTEvaluationMontana Research Foundationhttps://montanaresearch.org/blog/gpts-assistants-api-prompt-leak-problem/https://montanaresearch.org/blog/gpts-assistants-api-prompt-leak-problem/OpenAI's DevDay bet was that anyone could build a chatbot from a system prompt and a few uploaded files. Within days people were extracting the prompts and downloading the files. A look at how thin the customisation layer is and what that means for the coming GPT Store.Tue, 28 Nov 2023 00:00:00 GMTAppliedMontana Research Foundationhttps://montanaresearch.org/blog/needle-in-a-haystack-one-engineers-chart/https://montanaresearch.org/blog/needle-in-a-haystack-one-engineers-chart/Greg Kamradt hid one sentence in a stack of Paul Graham essays and plotted where GPT-4 and Claude 2.1 could find it. The heatmap is now the standard picture of long-context recall. Here is what it measures and what it does not.Tue, 28 Nov 2023 00:00:00 GMTEvaluationMontana Research Foundationhttps://montanaresearch.org/blog/openai-board-crisis-five-days-governance/https://montanaresearch.org/blog/openai-board-crisis-five-days-governance/A four-person non-profit board fired the chief executive of the most visible AI lab on a Friday and had reinstated him by the following Wednesday. Notes on what the episode revealed about non-profit control of a frontier lab, and why I doubt the mechanism will be used again.Mon, 27 Nov 2023 00:00:00 GMTNotesMontana Research Foundationhttps://montanaresearch.org/blog/gaia-easy-for-humans-hard-for-models/https://montanaresearch.org/blog/gaia-easy-for-humans-hard-for-models/GAIA turns the benchmark design problem around. Instead of finding questions that stump experts, it asks questions any careful person can answer and watches assistants fall over.Fri, 24 Nov 2023 00:00:00 GMTEvaluationMontana Research Foundationhttps://montanaresearch.org/blog/linear-representation-hypothesis-stated-carefully/https://montanaresearch.org/blog/linear-representation-hypothesis-stated-carefully/Park, Choe and Veitch wrote down what people mean when they say concepts are directions, connected probing to steering through a causal inner product, and checked it on LLaMA-2. Notes on why the field needed a definition and what the definition leaves out.Thu, 16 Nov 2023 00:00:00 GMTInterpretabilityMontana Research Foundationhttps://montanaresearch.org/blog/insider-trading-demo-gpt-4-lies-to-manager/https://montanaresearch.org/blog/insider-trading-demo-gpt-4-lies-to-manager/Apollo Research put GPT-4 in a simulated trading desk, handed it an insider tip under pressure, and watched it trade on the tip and then hide the reason from its manager. A walkthrough of the setup, the numbers, and why a single scenario is worth this much attention.Tue, 14 Nov 2023 00:00:00 GMTSafety & AlignmentMontana Research Foundationhttps://montanaresearch.org/blog/executive-order-bletchley-week-governance-got-specific/https://montanaresearch.org/blog/executive-order-bletchley-week-governance-got-specific/Executive Order 14110 put a number on what counts as a frontier model, and 28 governments signed a declaration at Bletchley Park two days later. Notes on what each document actually commits anyone to, with a look back at what survived.Fri, 03 Nov 2023 00:00:00 GMTSafety & AlignmentMontana Research Foundationhttps://montanaresearch.org/blog/managing-ai-risks-consensus-paper-not-consensus/https://montanaresearch.org/blog/managing-ai-risks-consensus-paper-not-consensus/Bengio, Hinton, Yao, Russell, Kahneman and two dozen co-authors published a short paper calling safety research lagging and asking labs and funders to spend a third of AI R&D budgets on it. Reading notes on what it demands, what it leaves out, and who is missing from the author list.Mon, 30 Oct 2023 00:00:00 GMTNotesMontana Research Foundationhttps://montanaresearch.org/blog/data-provenance-audit-70-percent-no-license/https://montanaresearch.org/blog/data-provenance-audit-70-percent-no-license/A seventeen author audit of more than 1,800 fine-tuning datasets found licenses omitted on over 70 percent of them on popular hosting sites and wrong on over half of those that had one. Notes on what the audit measured, what the Explorer fixes, and what it cannot.Sun, 29 Oct 2023 00:00:00 GMTOpen ScienceMontana Research Foundationhttps://montanaresearch.org/blog/zephyr-distilled-dpo-alignment-from-ai-feedback/https://montanaresearch.org/blog/zephyr-distilled-dpo-alignment-from-ai-feedback/Hugging Face's Zephyr-7B fine-tuned Mistral 7B on synthetic conversations and GPT-4 ranked preferences with no human labels, and beat Llama 2 Chat 70B on MT-Bench. Notes on the recipe and on what it means that preference tuning now fits on a single node.Fri, 27 Oct 2023 00:00:00 GMTFoundationsMontana Research Foundationhttps://montanaresearch.org/blog/foundation-model-transparency-index-54-out-of-100/https://montanaresearch.org/blog/foundation-model-transparency-index-54-out-of-100/Stanford's index scores ten developers on 100 binary indicators and the best of them, Meta, gets 54. Reading notes on what the index measures, where the zeros cluster, and whether a scoreboard changes behaviour or just documentation.Tue, 24 Oct 2023 00:00:00 GMTOpen ScienceMontana Research Foundationhttps://montanaresearch.org/blog/techno-optimist-manifesto-and-the-enemies-list/https://montanaresearch.org/blog/techno-optimist-manifesto-and-the-enemies-list/Marc Andreessen's manifesto names the precautionary principle, trust and safety, tech ethics and risk management as enemies of progress. I want to take the argument seriously and say where it goes wrong for people who study how models fail.Fri, 20 Oct 2023 00:00:00 GMTNotesMontana Research Foundationhttps://montanaresearch.org/blog/swe-bench-at-launch-2294-issues/https://montanaresearch.org/blog/swe-bench-at-launch-2294-issues/When SWE-bench appeared, the best model resolved under two percent of the issues. A look-back at how the benchmark was built, why grading by running the tests mattered, and how the launch numbers set the terms for the coding evals that followed.Wed, 18 Oct 2023 00:00:00 GMTEvaluationMontana Research Foundationhttps://montanaresearch.org/blog/llama-has-a-map-and-a-calendar/https://montanaresearch.org/blog/llama-has-a-map-and-a-calendar/Gurnee and Tegmark fit ridge regression probes to Llama-2 activations and recovered latitude, longitude and dates for tens of thousands of real entities, with individual neurons that track the same directions. A reading note on what a probe result does and does not tell you about world models.Wed, 11 Oct 2023 00:00:00 GMTInterpretabilityMontana Research Foundationhttps://montanaresearch.org/blog/ten-examples-undo-safety-training-fine-tuning-hole/https://montanaresearch.org/blog/ten-examples-undo-safety-training-fine-tuning-hole/Qi and colleagues fine-tuned GPT-3.5 Turbo on ten harmful examples for under twenty cents and pushed its harmfulness rate from 1.8 percent to 88.8 percent. Fine-tuning on Alpaca alone raised it to 31.8 percent. What the paper measured and why it changes the open weights argument.Wed, 11 Oct 2023 00:00:00 GMTSafety & AlignmentMontana Research Foundationhttps://montanaresearch.org/blog/towards-monosemanticity-superposition-becomes-a-tool/https://montanaresearch.org/blog/towards-monosemanticity-superposition-becomes-a-tool/Anthropic pulled more than 4,000 interpretable features out of a 512-neuron layer using a sparse autoencoder. A look at what the result does and does not show.Wed, 11 Oct 2023 00:00:00 GMTInterpretabilityMontana Research Foundationhttps://montanaresearch.org/blog/dspy-prompts-compiled-not-written/https://montanaresearch.org/blog/dspy-prompts-compiled-not-written/Khattab and colleagues proposed treating a language model pipeline as a program with declared signatures, learnable modules, and an optimiser that generates the prompts. What compiling a prompt means, what the GSM8K and HotPotQA numbers showed, and how the idea aged.Mon, 09 Oct 2023 00:00:00 GMTAppliedMontana Research Foundationhttps://montanaresearch.org/blog/representation-engineering-interpretability-top-down/https://montanaresearch.org/blog/representation-engineering-interpretability-top-down/Zou and twenty co-authors propose reading and steering concepts like honesty and power-seeking with linear directions in the residual stream, and deliberately skip the circuits. Notes on the LAT method, the TruthfulQA numbers, and the argument it picks with mechanistic interpretability.Mon, 09 Oct 2023 00:00:00 GMTInterpretabilityMontana Research Foundationhttps://montanaresearch.org/blog/six-ways-evaluation-is-harder-anthropic/https://montanaresearch.org/blog/six-ways-evaluation-is-harder-anthropic/Anthropic's policy post walks from multiple-choice benchmarks to third-party audits and reports what went wrong at each step, including a five-point swing on MMLU from formatting alone and a bias score of zero that meant the model was refusing to answer.Mon, 09 Oct 2023 00:00:00 GMTEvaluationMontana Research Foundationhttps://montanaresearch.org/blog/magnet-link-and-a-7b-model-mistral-release/https://montanaresearch.org/blog/magnet-link-and-a-7b-model-mistral-release/Mistral 7B arrived under Apache 2.0, via a BitTorrent magnet link, with benchmarks over Llama 2 13B and no paper attached. What the release style says about a five-month-old European lab and about what counts as publishing a model.Fri, 29 Sep 2023 00:00:00 GMTNotesMontana Research Foundationhttps://montanaresearch.org/blog/ai-safety-levels-first-responsible-scaling-policy/https://montanaresearch.org/blog/ai-safety-levels-first-responsible-scaling-policy/Anthropic's RSP borrows the biosafety level ladder and commits to pausing training when capabilities outrun safeguards. Here is what ASL-2 and ASL-3 actually bind the company to, and where the document admits it is building the airplane in flight.Tue, 26 Sep 2023 00:00:00 GMTSafety & AlignmentMontana Research Foundationhttps://montanaresearch.org/blog/reversal-curse-knows-a-is-b-not-b-is-a/https://montanaresearch.org/blog/reversal-curse-knows-a-is-b-not-b-is-a/Berglund and colleagues fine-tuned models on facts in one direction and found they could not answer the reverse question at all, and GPT-4 shows the same asymmetry on real celebrity parents. Notes on the experiment and why next token prediction makes this failure natural.Tue, 26 Sep 2023 00:00:00 GMTFoundationsMontana Research Foundationhttps://montanaresearch.org/blog/sparse-autoencoders-independent-replication/https://montanaresearch.org/blog/sparse-autoencoders-independent-replication/A five-author paper shows dictionary learning on Pythia residual streams gives features that are far more interpretable than neurons, PCA or ICA. The result is what superposition theory predicted, found by a group outside the lab that proposed it.Tue, 26 Sep 2023 00:00:00 GMTInterpretabilityMontana Research Foundationhttps://montanaresearch.org/blog/authors-guild-v-openai-pirate-libraries/https://montanaresearch.org/blog/authors-guild-v-openai-pirate-libraries/The Grisham and Martin complaint adds one thing the summer's author suits only hinted at. It says where the books came from, and that turns the case into a question about provenance.Fri, 22 Sep 2023 00:00:00 GMTOpen ScienceMontana Research Foundationhttps://montanaresearch.org/blog/centaurs-cyborgs-jagged-frontier-bcg/https://montanaresearch.org/blog/centaurs-cyborgs-jagged-frontier-bcg/A field experiment with 758 BCG consultants found GPT-4 users finished 12.2 percent more tasks, 25.1 percent faster, at 40 percent higher rated quality. On a task chosen to sit outside the model’s reach, the AI group did worse than the control. Why the second result matters more than the first.Thu, 21 Sep 2023 00:00:00 GMTAppliedMontana Research Foundationhttps://montanaresearch.org/blog/pretraining-on-the-test-set/https://montanaresearch.org/blog/pretraining-on-the-test-set/A three page satire about a 1 million parameter model that aces every benchmark arrived in the same month as a method for catching contamination in GPT-4 without seeing its training data. Both say the same thing about how we evaluate.Thu, 21 Sep 2023 00:00:00 GMTEvaluationMontana Research Foundationhttps://montanaresearch.org/blog/othello-gpt-board-is-linear-after-all/https://montanaresearch.org/blog/othello-gpt-board-is-linear-after-all/Li et al. found a nonlinear world model in a transformer trained on Othello moves. Nanda, Lee and Wattenberg show that if you ask the probe about mine versus theirs instead of black versus white, the same board is linear and steerable with vector arithmetic.Tue, 12 Sep 2023 00:00:00 GMTInterpretabilityMontana Research Foundationhttps://montanaresearch.org/blog/lucene-is-all-you-need-vector-store/https://montanaresearch.org/blog/lucene-is-all-you-need-vector-store/Jimmy Lin and colleagues indexed OpenAI ada2 embeddings for MS MARCO in plain Lucene and got competitive retrieval numbers. Their argument is about cost and benefit, and the performance caveats they list are the interesting part.Mon, 04 Sep 2023 00:00:00 GMTAppliedMontana Research Foundationhttps://montanaresearch.org/blog/books3-datasets-were-never-ours/https://montanaresearch.org/blog/books3-datasets-were-never-ours/A Danish anti-piracy group had Books3 pulled from The Eye this summer, in the same month three authors sued Meta over its use in LLaMA. The scrape-first norm that built The Pile has stopped working.Wed, 23 Aug 2023 00:00:00 GMTOpen ScienceMontana Research Foundationhttps://montanaresearch.org/blog/def-con-generative-red-team-2244-hackers/https://montanaresearch.org/blog/def-con-generative-red-team-2244-hackers/The AI Village challenge at DEF CON 31 put eight vendor models in front of 2,244 people for two and a half days and collected more than 17,000 conversations across 21 topics. Notes on what the organisers report finding, and whether a crowd surfaces anything the labs did not already know.Wed, 16 Aug 2023 00:00:00 GMTSafety & AlignmentMontana Research Foundationhttps://montanaresearch.org/blog/agentbench-first-attempt-to-score-model-as-agent/https://montanaresearch.org/blog/agentbench-first-attempt-to-score-model-as-agent/AgentBench put 29 language models in eight interactive environments and found GPT-4 at 4.01 against the best open model at 0.96. What made agent evaluation different from question answering, and which of its early choices held up.Tue, 15 Aug 2023 00:00:00 GMTEvaluationMontana Research Foundationhttps://montanaresearch.org/blog/dario-on-scaling-we-still-dont-know-why/https://montanaresearch.org/blog/dario-on-scaling-we-still-dont-know-why/Reading notes on Dario Amodei's interview with Dwarkesh Patel. The most honest moment is an admission that the field has no theory of why scaling is smooth. The most testable moments are the two to three year timelines, which I am writing down so they can be scored.Mon, 14 Aug 2023 00:00:00 GMTNotesMontana Research Foundationhttps://montanaresearch.org/blog/llama-2-fine-tuning-ecosystem/https://montanaresearch.org/blog/llama-2-fine-tuning-ecosystem/Meta released a 7B, 13B and 70B family with a chat variant and a licence that permits commercial use. Why the chat model and the licence matter more than the weights, and what the paper says about doing RLHF at scale.Tue, 08 Aug 2023 00:00:00 GMTAppliedMontana Research Foundationhttps://montanaresearch.org/blog/open-problems-with-rlhf-thirty-two-authors/https://montanaresearch.org/blog/open-problems-with-rlhf-thirty-two-authors/Casper and colleagues catalogued the failure modes of reinforcement learning from human feedback across feedback collection, reward modelling and policy optimisation, and marked each as tractable or fundamental. Reading notes on the survey as a checklist.Tue, 08 Aug 2023 00:00:00 GMTSafety & AlignmentMontana Research Foundationhttps://montanaresearch.org/blog/gcg-adversarial-suffix-vicuna-to-gpt-4/https://montanaresearch.org/blog/gcg-adversarial-suffix-vicuna-to-gpt-4/Zou, Wang, Carlini, Nasr, Kolter and Fredrikson optimised a gibberish suffix against open models and found it broke ChatGPT, Bard and PaLM-2 too. A method deep dive on greedy coordinate gradient and what it means that open weights became the attack surface for closed APIs.Mon, 31 Jul 2023 00:00:00 GMTSafety & AlignmentMontana Research Foundationhttps://montanaresearch.org/blog/does-circuit-analysis-scale-deepmind-chinchilla/https://montanaresearch.org/blog/does-circuit-analysis-scale-deepmind-chinchilla/Lieberum and colleagues found the multiple-choice circuit in a 70B model with the same logit attribution and activation patching that worked on GPT-2 small, but the semantics of the heads resisted a clean story. This is the first honest data point on how far toy-model methods carry.Thu, 27 Jul 2023 00:00:00 GMTInterpretabilityMontana Research Foundationhttps://montanaresearch.org/blog/is-chatgpt-getting-worse-drift-study/https://montanaresearch.org/blog/is-chatgpt-getting-worse-drift-study/A Stanford and Berkeley study compared the March and June 2023 versions of GPT-4 and GPT-3.5 and found GPT-4's accuracy on a prime-number task fell from 84 to 51 percent. Some of the swings are real behaviour drift and some are evaluation artefacts, and separating the two is the point.Fri, 21 Jul 2023 00:00:00 GMTEvaluationMontana Research Foundationhttps://montanaresearch.org/blog/measuring-faithfulness-by-breaking-the-chain/https://montanaresearch.org/blog/measuring-faithfulness-by-breaking-the-chain/Anthropic truncated, corrupted, paraphrased and blanked out chains of thought to see whether the final answer depended on them. Sometimes it did. Larger models were worse.Fri, 21 Jul 2023 00:00:00 GMTInterpretabilityMontana Research Foundationhttps://montanaresearch.org/blog/lost-in-the-middle-u-shaped-curve-rag/https://montanaresearch.org/blog/lost-in-the-middle-u-shaped-curve-rag/Liu et al. moved the answer-bearing document around a 20-document context and watched GPT-3.5-Turbo go from 75.8 percent to 53.8 percent, below its own closed-book score. Reading notes on the paper that changed how everyone orders their retrieved chunks.Wed, 19 Jul 2023 00:00:00 GMTAppliedMontana Research Foundationhttps://montanaresearch.org/blog/icml-bans-llm-written-text/https://montanaresearch.org/blog/icml-bans-llm-written-text/ICML's policy for this year prohibits papers whose text was generated by a large language model while permitting light editing. Where the line was drawn, why, and why I think it will not hold.Tue, 18 Jul 2023 00:00:00 GMTNotesMontana Research Foundationhttps://montanaresearch.org/blog/jailbroken-competing-objectives-mismatched-generalisation/https://montanaresearch.org/blog/jailbroken-competing-objectives-mismatched-generalisation/Wei, Haghtalab and Steinhardt propose two failure modes for safety training and argue that scaling alone will not close either. An explainer on the taxonomy, the numbers behind it, and the safety-capability parity argument.Tue, 11 Jul 2023 00:00:00 GMTSafety & AlignmentMontana Research Foundationhttps://montanaresearch.org/blog/rise-of-the-ai-engineer-three-years-later/https://montanaresearch.org/blog/rise-of-the-ai-engineer-three-years-later/In June 2023 swyx declared a new job title built on APIs rather than training runs. A look back at what the essay predicted, which parts held, and how the role split again once agents arrived.Tue, 11 Jul 2023 00:00:00 GMTAppliedMontana Research Foundationhttps://montanaresearch.org/blog/pagedattention-kv-cache-was-the-bottleneck/https://montanaresearch.org/blog/pagedattention-kv-cache-was-the-bottleneck/vLLM's release post reports up to 24 times the throughput of Hugging Face Transformers, and the mechanism is virtual memory for attention keys and values. Why memory fragmentation, not model quality, has been the constraint on shipping LLM products this year.Tue, 27 Jun 2023 00:00:00 GMTAppliedMontana Research Foundationhttps://montanaresearch.org/blog/textbooks-are-all-you-need/https://montanaresearch.org/blog/textbooks-are-all-you-need/A 1.3B parameter model trained on under 7B tokens just scored 50.6% on HumanEval. What the phi-1 paper actually shows, and what it leaves open.Tue, 27 Jun 2023 00:00:00 GMTFoundationsMontana Research Foundationhttps://montanaresearch.org/blog/why-llama-mmlu-score-depended-on-who-ran-it/https://montanaresearch.org/blog/why-llama-mmlu-score-depended-on-who-ran-it/The Open LLM Leaderboard gave LLaMA 65B an MMLU of 0.488. Meta reported 0.636 and HELM measured 0.637. Hugging Face traced the gap to three implementations of the same benchmark that differ in prompt format and how the answer is read out.Tue, 27 Jun 2023 00:00:00 GMTEvaluationMontana Research Foundationhttps://montanaresearch.org/blog/munk-debate-bengio-tegmark-versus-lecun-mitchell/https://montanaresearch.org/blog/munk-debate-bengio-tegmark-versus-lecun-mitchell/Four senior researchers argued whether AI research poses an existential threat in front of a Toronto audience that voted before and after. The vote moved three points. Notes on what the format could and could not settle.Mon, 26 Jun 2023 00:00:00 GMTNotesMontana Research Foundationhttps://montanaresearch.org/blog/mata-v-avianca-brief-with-six-invented-cases/https://montanaresearch.org/blog/mata-v-avianca-brief-with-six-invented-cases/Judge Castel sanctioned two New York lawyers and their firm on June 22 for filing opinions ChatGPT had fabricated and then standing by them for three months. What the opinion actually found, why asking the model to verify itself was the fatal step, and what it means for grounded tools.Fri, 23 Jun 2023 00:00:00 GMTAppliedMontana Research Foundationhttps://montanaresearch.org/blog/gptq-awq-rules-of-post-training-quantization/https://montanaresearch.org/blog/gptq-awq-rules-of-post-training-quantization/GPTQ squeezes a 175B model to 4 bits in four GPU hours using second-order weight updates. AWQ, out this month, gets there by protecting one percent of channels chosen by activation magnitude and touching nothing else. What each method buys, and where both fall off a cliff.Wed, 14 Jun 2023 00:00:00 GMTAppliedMontana Research Foundationhttps://montanaresearch.org/blog/mt-bench-and-the-80-percent-agreement-number/https://montanaresearch.org/blog/mt-bench-and-the-80-percent-agreement-number/The paper that formalised LLM-as-a-judge reports GPT-4 agreeing with human experts 85 percent of the time, above the 81 percent humans manage with each other. The same paper measures position bias, a verbosity attack and self-preference. A close look at what the agreement figure licenses and what it quietly excludes.Wed, 14 Jun 2023 00:00:00 GMTEvaluationMontana Research Foundationhttps://montanaresearch.org/blog/inference-time-intervention-nudging-heads-toward-truth/https://montanaresearch.org/blog/inference-time-intervention-nudging-heads-toward-truth/A Harvard group found attention heads in LLaMA whose activations linearly separate true from false answers, then shifted them at inference. Alpaca went from 32.5 to 65.1 percent on TruthfulQA. Reading notes on what that says about where truth lives in a model, and what the method leaves alone.Fri, 09 Jun 2023 00:00:00 GMTInterpretabilityMontana Research Foundationhttps://montanaresearch.org/blog/falcon-goes-apache-2-license-moved-leaderboard/https://montanaresearch.org/blog/falcon-goes-apache-2-license-moved-leaderboard/Falcon-40B launched under a TII license that asked for ten percent of revenue above a million dollars. Five days later the license was Apache 2.0, and within the week the model was at the top of the Open LLM Leaderboard with a Hugging Face blog post calling it the first truly open model of its class. On license terms as a competitive lever.Thu, 08 Jun 2023 00:00:00 GMTOpen ScienceMontana Research Foundationhttps://montanaresearch.org/blog/hinton-quits-google-pioneer-changed-his-mind/https://montanaresearch.org/blog/hinton-quits-google-pioneer-changed-his-mind/Geoffrey Hinton left Google this month so he could talk about AI risk without worrying about how it interacts with the company's business, and said he had suddenly switched his views on whether these systems will be more intelligent than us. Four weeks later he signed a one-sentence extinction statement. On what it means when an elder of the field revises his priors in public.Wed, 31 May 2023 00:00:00 GMTNotesMontana Research Foundationhttps://montanaresearch.org/blog/process-supervision-as-a-safety-method/https://montanaresearch.org/blog/process-supervision-as-a-safety-method/OpenAI's new paper shows that a reward model trained on step-level labels solves 78.2 percent of a MATH subset against 72.4 percent for one trained on final answers. The authors put the result in an alignment frame, and I think they are right to.Wed, 31 May 2023 00:00:00 GMTSafety & AlignmentMontana Research Foundationhttps://montanaresearch.org/blog/twenty-two-words-extinction-statement/https://montanaresearch.org/blog/twenty-two-words-extinction-statement/The Center for AI Safety statement signed by Hinton, Bengio, Altman, Hassabis and Amodei runs to a single sentence. That brevity was the point, and it is also why the statement commits its signatories to nothing measurable. Notes on what it did and what to watch for.Wed, 31 May 2023 00:00:00 GMTSafety & AlignmentMontana Research Foundationhttps://montanaresearch.org/blog/model-evaluation-for-extreme-risks-reading-note/https://montanaresearch.org/blog/model-evaluation-for-extreme-risks-reading-note/Twenty-one authors led from Google DeepMind argue that developers should test frontier models for dangerous capabilities and misalignment before training and deployment. A reading note on what the paper proposes, the hazards it names, and what it leaves open.Tue, 30 May 2023 00:00:00 GMTEvaluationMontana Research Foundationhttps://montanaresearch.org/blog/how-many-epochs-data-constrained-scaling/https://montanaresearch.org/blog/how-many-epochs-data-constrained-scaling/Muennighoff and colleagues trained 400 models to find out what repeating data costs. Up to about four epochs it costs almost nothing, and the fitted law says why. Reading notes on a paper that quietly resets the assumptions behind the running out of data worry.Mon, 29 May 2023 00:00:00 GMTFoundationsMontana Research Foundationhttps://montanaresearch.org/blog/qlora-fine-tuning-65b-model-one-gpu/https://montanaresearch.org/blog/qlora-fine-tuning-65b-model-one-gpu/Three memory tricks and a new 4-bit data type let Dettmers and colleagues fine-tune LLaMA 65B on a single 48GB card in a day. Method notes on what each piece does and what the paper found once fine-tuning got cheap.Fri, 26 May 2023 00:00:00 GMTAppliedMontana Research Foundationhttps://montanaresearch.org/blog/tree-of-thoughts-return-of-search/https://montanaresearch.org/blog/tree-of-thoughts-return-of-search/Yao and colleagues took GPT-4 from 4 percent to 74 percent on Game of 24 by sampling several intermediate thoughts, scoring them, and running breadth-first search over the result. A method deep dive on what the paper actually does and why it matters that the model is now a proposal distribution.Thu, 25 May 2023 00:00:00 GMTFoundationsMontana Research Foundationhttps://montanaresearch.org/blog/models-do-not-always-say-what-they-think/https://montanaresearch.org/blog/models-do-not-always-say-what-they-think/Turpin, Michael, Perez and Bowman biased few-shot prompts so the correct answer was always (A) and watched GPT-3.5 and Claude 1.0 rationalise wrong answers without mentioning the bias. Notes on why this turns chain of thought from a window into a claim that needs testing.Thu, 18 May 2023 00:00:00 GMTInterpretabilityMontana Research Foundationhttps://montanaresearch.org/blog/love-minus-hate-activation-addition-trick/https://montanaresearch.org/blog/love-minus-hate-activation-addition-trick/Turner and collaborators steered GPT-2-XL by running two prompts, subtracting their activations at one layer, and adding the difference back during generation. No gradients, no fine-tuning, and results that hold up better than the simplicity suggests.Tue, 16 May 2023 00:00:00 GMTInterpretabilityMontana Research Foundationhttps://montanaresearch.org/blog/gpt-4-explains-gpt-2-neurons/https://montanaresearch.org/blog/gpt-4-explains-gpt-2-neurons/OpenAI had GPT-4 write an explanation for every one of GPT-2 XL's 307,200 neurons and scored each one by simulation. Just over a thousand cleared the bar. That number is the useful part.Fri, 12 May 2023 00:00:00 GMTInterpretabilityMontana Research Foundationhttps://montanaresearch.org/blog/iclr-kigali-first-major-ml-conference-africa/https://montanaresearch.org/blog/iclr-kigali-first-major-ml-conference-africa/ICLR 2023 ran May 1 to 5 in Kigali with 3,758 participants from 73 countries, the first time a top machine learning conference met on the continent. Notes on what the location changed, who it let in, and whether the map moves again.Fri, 12 May 2023 00:00:00 GMTNotesMontana Research Foundationhttps://montanaresearch.org/blog/we-have-no-moat-rereading-leaked-google-memo/https://montanaresearch.org/blog/we-have-no-moat-rereading-leaked-google-memo/A document attributed to a Google researcher argues that open weights and cheap fine-tuning are outrunning the big labs. Reading notes from a small lab on which claims look right and which look like a mood.Tue, 09 May 2023 00:00:00 GMTNotesMontana Research Foundationhttps://montanaresearch.org/blog/chatbot-arena-opens-elo-ratings-for-language-models/https://montanaresearch.org/blog/chatbot-arena-opens-elo-ratings-for-language-models/LMSYS put nine open chat models in anonymous pairwise battles, collected 4,700 votes in a week, and published an Elo leaderboard. Notes on the design choices and the caveats the authors wrote down before anyone else could.Fri, 05 May 2023 00:00:00 GMTEvaluationMontana Research Foundationhttps://montanaresearch.org/blog/samsung-chatgpt-leak-enterprise-ai-policy/https://montanaresearch.org/blog/samsung-chatgpt-leak-enterprise-ai-policy/Engineers pasted proprietary code into ChatGPT, Samsung banned the tools, and vendors are now selling the fix. A deployment failure case study on why the first enterprise AI policy at most companies was written by a leak.Thu, 04 May 2023 00:00:00 GMTAppliedMontana Research Foundationhttps://montanaresearch.org/blog/how-a-model-recalls-a-fact-three-steps/https://montanaresearch.org/blog/how-a-model-recalls-a-fact-three-steps/Dissecting Recall of Factual Associations traces the retrieval of a fact through GPT-2 XL and GPT-J and finds a three-stage pipeline: early MLPs enrich the subject, the relation propagates to the last token, and attention heads extract the attribute. Notes on the method and on how it sits against the ROME picture.Sun, 30 Apr 2023 00:00:00 GMTInterpretabilityMontana Research Foundationhttps://montanaresearch.org/blog/emergent-abilities-mirage-metric-argument/https://montanaresearch.org/blog/emergent-abilities-mirage-metric-argument/Schaeffer, Miranda and Koyejo argue that the sharp jumps in scaling plots come from discontinuous metrics rather than from the models. The statistical point holds. The larger claim that nothing surprising happens with scale is a separate question, and the paper does not settle it.Sat, 29 Apr 2023 00:00:00 GMTFoundationsMontana Research Foundationhttps://montanaresearch.org/blog/vector-database-gold-rush/https://montanaresearch.org/blog/vector-database-gold-rush/Pinecone raised $100 million this week at a $750 million valuation. The bet is that every LLM application needs a dedicated vector store. I think the bet is on the wrong layer.Fri, 28 Apr 2023 00:00:00 GMTAppliedMontana Research Foundationhttps://montanaresearch.org/blog/auto-gpt-babyagi-first-agent-hype-cycle/https://montanaresearch.org/blog/auto-gpt-babyagi-first-agent-hype-cycle/Three weeks after Auto-GPT appeared it is one of the fastest growing repositories on GitHub, and most people who have run it have watched it go in circles. Notes on why the loops spin, what HuggingGPT does differently, and which of the missing pieces look solvable.Wed, 19 Apr 2023 00:00:00 GMTAppliedMontana Research Foundationhttps://montanaresearch.org/blog/redpajama-reproducing-a-training-set/https://montanaresearch.org/blog/redpajama-reproducing-a-training-set/Together and partners rebuilt the 1.2 trillion token LLaMA training mix from the description in the paper. The token counts land close to the original on most slices and 41 percent short on GitHub, which is a measure of how much a recipe leaves out.Wed, 19 Apr 2023 00:00:00 GMTOpen ScienceMontana Research Foundationhttps://montanaresearch.org/blog/dolly-2-and-the-15000-employee-answers/https://montanaresearch.org/blog/dolly-2-and-the-15000-employee-answers/Databricks released an instruction dataset written by its own staff under a licence that allows commercial use. The model built on it is unremarkable. The dataset is the first of its kind, and the way it was made shows what distilling from a closed API had been hiding.Fri, 14 Apr 2023 00:00:00 GMTOpen ScienceMontana Research Foundationhttps://montanaresearch.org/blog/pythia-publish-checkpoints-not-just-weights/https://montanaresearch.org/blog/pythia-publish-checkpoints-not-just-weights/EleutherAI trained 16 models from 70M to 12B parameters on identical data in identical order and released 154 checkpoints for each. A look at the design, and at the questions about memorisation and training dynamics that only this kind of suite can answer.Tue, 11 Apr 2023 00:00:00 GMTOpen ScienceMontana Research Foundationhttps://montanaresearch.org/blog/pythia-model-suite-built-to-be-studied/https://montanaresearch.org/blog/pythia-model-suite-built-to-be-studied/EleutherAI has released 16 language models from 70M to 12B parameters, each with 154 checkpoints, all trained on the same data in the same order. The benchmark scores are beside the point. For the first time anyone can watch a model learn.Thu, 06 Apr 2023 00:00:00 GMTFoundationsMontana Research Foundationhttps://montanaresearch.org/blog/pause-or-shut-it-all-down-two-march-letters/https://montanaresearch.org/blog/pause-or-shut-it-all-down-two-march-letters/The Future of Life Institute's six-month pause letter and Eliezer Yudkowsky's TIME response landed a week apart. A look back at what each actually asked for, who signed and who refused, and why neither request was ever going to be granted.Fri, 31 Mar 2023 00:00:00 GMTNotesMontana Research Foundationhttps://montanaresearch.org/blog/vicuna-and-the-birth-of-gpt-4-as-judge/https://montanaresearch.org/blog/vicuna-and-the-birth-of-gpt-4-as-judge/The Vicuna release put a 90 percent-of-ChatGPT figure on a 300 dollar fine-tune by asking GPT-4 to grade the answers, and said in the same post that the method was not rigorous. Why the improvised evaluation is likely to outlive the model.Fri, 31 Mar 2023 00:00:00 GMTEvaluationMontana Research Foundationhttps://montanaresearch.org/blog/ilya-next-token-prediction-podcast-aged-well/https://montanaresearch.org/blog/ilya-next-token-prediction-podcast-aged-well/Sutskever told Dwarkesh Patel that predicting the next token well enough requires understanding the reality that produced it, and that this is why the objective need not cap out at human level. Reading notes on the argument and the objections I keep hearing to it.Tue, 28 Mar 2023 00:00:00 GMTNotesMontana Research Foundationhttps://montanaresearch.org/blog/top-10-percent-bar-exam-gpt-4-percentiles/https://montanaresearch.org/blog/top-10-percent-bar-exam-gpt-4-percentiles/The GPT-4 report's abstract says the model passed a simulated bar exam with a score around the top 10 percent of test takers. Percentiles are only meaningful against a named population, and the population behind that figure is the one most likely to flatter the model. Notes on how to read the exam table, with an addendum on the re-analysis that put the number closer to the 48th percentile of people who passed.Mon, 27 Mar 2023 00:00:00 GMTEvaluationMontana Research Foundationhttps://montanaresearch.org/blog/tuned-lens-reading-transformer-one-layer-at-a-time/https://montanaresearch.org/blog/tuned-lens-reading-transformer-one-layer-at-a-time/Belrose and colleagues train one small affine map per layer so that intermediate residual streams decode into legible next-token predictions. Reading notes on what the prediction trajectories show, and on where the lens can still mislead.Wed, 22 Mar 2023 00:00:00 GMTInterpretabilityMontana Research Foundationhttps://montanaresearch.org/blog/alpaca-600-dollar-lesson-instruction-tuning-cheap/https://montanaresearch.org/blog/alpaca-600-dollar-lesson-instruction-tuning-cheap/Stanford fine-tuned LLaMA 7B on 52,000 machine-generated examples for under $600 and got a model that tied text-davinci-003 in a small blind test. The demos were convincing. Notes on what the model can do, what the comparison measured, and what it did not.Tue, 21 Mar 2023 00:00:00 GMTAppliedMontana Research Foundationhttps://montanaresearch.org/blog/gpt-4-system-card-taskrabbit-first-dangerous-capability-eval/https://montanaresearch.org/blog/gpt-4-system-card-taskrabbit-first-dangerous-capability-eval/OpenAI's GPT-4 system card includes a short account of the Alignment Research Center testing whether the model could replicate itself and acquire resources. It could not, but one anecdote from that test has travelled further than the result. Notes on what the appendix says and what it started.Mon, 20 Mar 2023 00:00:00 GMTSafety & AlignmentMontana Research Foundationhttps://montanaresearch.org/blog/gpt-4-technical-report-told-us-nothing/https://montanaresearch.org/blog/gpt-4-technical-report-told-us-nothing/OpenAI released a hundred-page technical report for GPT-4 that states, in one paragraph, that it will say nothing about architecture, size, hardware, compute, data or training method. Reading notes on what is in the document, what is missing, and what an independent lab can still take from it.Mon, 20 Mar 2023 00:00:00 GMTNotesMontana Research Foundationhttps://montanaresearch.org/blog/predictable-scaling-gpt-4-report-chart/https://montanaresearch.org/blog/predictable-scaling-gpt-4-report-chart/The GPT-4 technical report says nothing about architecture, data or compute, and then reports that final loss was predicted from runs using at most one ten-thousandth of the compute before the main run finished. That figure is the engineering claim of the release, and it says a lot about how frontier training is now planned.Mon, 20 Mar 2023 00:00:00 GMTFoundationsMontana Research Foundationhttps://montanaresearch.org/blog/llama-leak-open-weights/https://montanaresearch.org/blog/llama-leak-open-weights/Meta released LLaMA to approved researchers on February 24. A week later the weights were a torrent on 4chan, and within a month the community had built more on top of them than Meta's access process would ever have allowed.Sun, 19 Mar 2023 00:00:00 GMTOpen ScienceMontana Research Foundationhttps://montanaresearch.org/blog/llama-cpp-4-bit-macbook-weekend/https://montanaresearch.org/blog/llama-cpp-4-bit-macbook-weekend/Between March 10 and March 13, Georgi Gerganov's llama.cpp went from initial release to running a 7B model on a MacBook, a Raspberry Pi and a Pixel 6. An argument that this week, rather than any model release, is what created the local inference ecosystem.Tue, 14 Mar 2023 00:00:00 GMTAppliedMontana Research Foundationhttps://montanaresearch.org/blog/llama-broke-chinchilla-on-purpose/https://montanaresearch.org/blog/llama-broke-chinchilla-on-purpose/Meta trained a 7B model on a trillion tokens, several times past the ratio the Chinchilla paper calls compute-optimal, and the paper explains why. The budget that matters for most people is inference, and a compute-optimal training run is the wrong target for it.Tue, 28 Feb 2023 00:00:00 GMTFoundationsMontana Research Foundationhttps://montanaresearch.org/blog/pretraining-with-human-preferences-conditional-training/https://montanaresearch.org/blog/pretraining-with-human-preferences-conditional-training/Korbak and colleagues compared five ways to put a reward signal into pretraining and found that tagging each segment as good or bad works best. A method note on conditional training and why nobody seems ready to run it at scale.Fri, 24 Feb 2023 00:00:00 GMTSafety & AlignmentMontana Research Foundationhttps://montanaresearch.org/blog/theory-of-mind-emerged-then-broken/https://montanaresearch.org/blog/theory-of-mind-emerged-then-broken/Kosinski reported that GPT-3.5 solved false-belief tasks at the level of a nine-year-old. Within two weeks Ullman showed that small rewrites of the same vignettes make the model fail. A case study in why one benchmark pass is not a capability claim.Wed, 22 Feb 2023 00:00:00 GMTEvaluationMontana Research Foundationhttps://montanaresearch.org/blog/sydney-dan-and-the-birth-of-prompt-injection/https://montanaresearch.org/blog/sydney-dan-and-the-birth-of-prompt-injection/A Stanford student asked Bing Chat to ignore its instructions and it read him its rulebook. The same month, a roleplay prompt called DAN was making ChatGPT drop its policies. Neither trick is clever, and that is the problem.Mon, 20 Feb 2023 00:00:00 GMTSafety & AlignmentMontana Research Foundationhttps://montanaresearch.org/blog/55-percent-faster-reading-copilot-rct/https://montanaresearch.org/blog/55-percent-faster-reading-copilot-rct/The GitHub and Microsoft controlled experiment found developers with Copilot finished a JavaScript HTTP server 55.8 percent faster. Reading notes on the design, the confidence interval nobody quotes, and what a single-task experiment can tell you about real work.Thu, 16 Feb 2023 00:00:00 GMTAppliedMontana Research Foundationhttps://montanaresearch.org/blog/toolformer-and-react-taught-models-to-call-functions/https://montanaresearch.org/blog/toolformer-and-react-taught-models-to-call-functions/Toolformer, out this week from Meta, lets a 6.7B GPT-J decide for itself when to call a calculator or a search engine by fine-tuning on API calls it wrote and filtered on its own. ReAct, from Google and Princeton last autumn, gets PaLM to interleave reasoning with actions in a loop. Between them they set the template for tool use, and each has an assumption I do not expect to survive.Wed, 15 Feb 2023 00:00:00 GMTAppliedMontana Research Foundationhttps://montanaresearch.org/blog/getty-v-stability-first-training-data-lawsuit-images/https://montanaresearch.org/blog/getty-v-stability-first-training-data-lawsuit-images/Getty Images announced this week that it is suing Stability AI in the High Court in London over the images used to train Stable Diffusion. The first major rightsholder to go to court over training data picked a diffusion model, and the reasons are about evidence as much as law.Thu, 19 Jan 2023 00:00:00 GMTOpen ScienceMontana Research Foundationhttps://montanaresearch.org/blog/grokking-reverse-engineered-fourier-circuit/https://montanaresearch.org/blog/grokking-reverse-engineered-fourier-circuit/Nanda and colleagues opened up a one-layer transformer that groks modular addition and found trig identities. The progress measures they built from that circuit move long before test loss does.Wed, 18 Jan 2023 00:00:00 GMTInterpretabilityMontana Research Foundationhttps://montanaresearch.org/blog/tracr-ground-truth-transformers/https://montanaresearch.org/blog/tracr-ground-truth-transformers/DeepMind's Tracr compiles programs written in RASP into the weights of a standard decoder-only transformer, so that the circuit inside is known before anyone looks. Why a laboratory with answer keys matters for interpretability, and what compiled models still fail to capture about trained ones.Wed, 18 Jan 2023 00:00:00 GMTInterpretabilityMontana Research Foundationhttps://montanaresearch.org/blog/influence-operations-report-predicted-the-year/https://montanaresearch.org/blog/influence-operations-report-predicted-the-year/Reading notes on the January 2023 report from OpenAI, Georgetown CSET and the Stanford Internet Observatory on language models and propaganda. What its four intervention stages actually propose, which unknowns it flagged, and where I think the framing will hold.Sat, 14 Jan 2023 00:00:00 GMTSafety & AlignmentMontana Research Foundationhttps://montanaresearch.org/blog/cramming-one-gpu-one-day-pretraining/https://montanaresearch.org/blog/cramming-one-gpu-one-day-pretraining/Geiping and Goldstein trained BERT-class models under a strict single-GPU, 24-hour budget and re-tested nearly every trick in the pipeline. Reading notes on a rare controlled study of what matters at small scale, and why most of it is about throughput.Thu, 12 Jan 2023 00:00:00 GMTFoundationsMontana Research Foundationhttps://montanaresearch.org/blog/gpt-takes-the-bar-exam-professional-exams-benchmark/https://montanaresearch.org/blog/gpt-takes-the-bar-exam-professional-exams-benchmark/Bommarito and Katz gave text-davinci-003 the multiple-choice section of the bar exam and got 50.3 percent, passing in two of seven subjects. Notes on why a licensing exam looked like the ideal test and what it quietly did not measure.Thu, 05 Jan 2023 00:00:00 GMTEvaluationMontana Research Foundation