What Can I Get Out of This Model?

What a model can do depends on how you use it. Walked through an experienced engineer's steps, mine solved 87% of a set of hard programming problems instead of 74%. On the hardest, the steps made no difference, and a stronger model asked once did about as well.

The AI model on my Mac Studio once wrote an answer more than a third of a million characters long. I had asked it to solve a hard programming puzzle, and it began to reason out loud. By the time it hit the 128,000-token limit of an early trial (a token is roughly three-quarters of a word), it had written 358,116 characters: informal mathematics, case analysis, half-finished algorithms and corrections of corrections. No finished program appeared before the limit.

We talk about what a model can do as though it were fixed when the model was built, so that the only question is which one to buy. That transcript suggested otherwise. The model was still working when the limit cut it off, and what it lacked was a way to work through the problem in steps.

I spent a week finding out how much depends on how I use it. Asked once, with one request per problem and no help, it solved 74% of 38 hard problems. Walked through the steps an experienced engineer would take, it solved 87%, with nothing retrained. On the hardest twelve the steps made no difference, and a stronger model, asked once, did about as well as mine with all my help.

A puzzle with a shortcut

Each problem is a puzzle from a programming contest. Here is one of the 38, in plain words. You are given a row of up to 300,000 numbers. Repeatedly remove two neighbouring numbers, scoring the difference between them, until at most one is left. What is the highest total score possible? For the row 1, 2, 5, 3, the answer is 5: remove 2 and 5 for 3 points, then 1 and 3, now neighbours, for 2 more.

The answer has to be a program. It runs against dozens of hidden tests and passes only if every answer is right. Trying every order of removals works for the example but is hopeless for 300,000 numbers. The solver has to find a shortcut, and finding it is what makes these problems hard. The problems come from LiveCodeBench, a public benchmark built from recent contests.

The method comes from the person

I have argued that the code is no longer the scarce part of software: what an engineer contributes is judgement. That holds for using a model too. What it produces depends on the method it is given, and the method comes from the person using it.

Researchers at Persis Capital measured this with GVS5H, published this year. One model writes a plan into shared notes, then works through it in a series of fresh attempts, each knowing only what the notes hold. On a hundred of these problems it lifted one open model, Qwen3.8-27B, from 67% to 92%, without changing the model.

My model, GLM-5.3-Flash, is an open model that anyone can download, and it runs on a desktop Mac Studio. Asked once, with a limit of 200,000 tokens, it solved 28 of the 38. Every answer it finished was right. All ten failures hit the limit while still reasoning.

Two ways to fail

That last detail matters more than the score. The study published every answer from every run, and when I sorted them by how each model failed, two kinds of model appeared.

Some models always finish. OpenAI's GPT-5.6-Luna finished every answer, and about a quarter were wrong. Others stop short. Mine was right every time it finished, but often ran out of room first. It also needed more room to reach 28: at Luna's limit of 128,000 tokens, it would have solved 24.

The difference largely decides how much guidance can help. Across the study's runs on all hundred problems, guidance rescued about nine in ten of the answers the two Qwen models left unfinished. It fixed between three in ten and two thirds of the answers a model finished and got wrong, depending on the model. A model that is right when it finishes has more to gain from steps than one that finishes with the wrong answer. That fits these 38 problems: guidance added 26 points to Qwen3.8-27B, which tends to run out of room, and 8 to Luna, which tends to finish wrong.

Judgement, written down as steps

I gave mine the routine an experienced competitive programmer follows. Work out the method. Write the program. Write a second, slow program that is obviously correct, such as one that tries every order of removals, and use it to check the first on small random examples. When the two disagree, fix the first with the failing example in front of you. Each step is a separate request with its own budget, and the system around the model runs the programs, so the checks are real.

Every step is a judgement an experienced engineer makes without being told: that a clever program needs checking, that a slow, obvious one is the right check, and that one failing example is worth more than another reread of the problem.

The model can already do all of this. Asked for a slow checker, it writes one. Asked once to solve a problem, it cannot run anything, so every answer rests on its reasoning alone.

Fitting the steps to the model

Choosing the steps was one judgement. Fitting them to this model was another, and I learnt how it fails only by watching it work. Three lessons came from that.

First, it keeps thinking long after it has thought enough. Models like this one reason privately before they answer. The published system, written with other models in mind, asked mine for a plan. It reasoned for 77 minutes, until the 64,000 tokens I had allowed ran out, and wrote no plan. Turning the reasoning setting down to low or medium made no difference. Told to answer without reasoning first, it returned a plan with six concrete tasks in 18 seconds.

Second, working out the method and writing the program had to be separate steps. Given 200,000 tokens to do both, it used all of them and wrote no code. Given the method and asked only to write the program, with no reasoning first, it wrote one in 101 seconds. That program was not yet right, but a failed attempt now cost two minutes instead of nearly two hours.

Third, its checkers can be wrong. Slow checkers written without reasoning first were wrong on four of six practice problems. Written with reasoning, five of six were right, and the sixth failed its own test on the worked examples. Once, two checkers the model wrote separately made the same misreading of a problem, and both disagreed with a correct program. A model's checks share its misreadings, so the most trustworthy check reaches the answer by a different route.

How I tested it

I ran the model on the 38 problems both ways: asked once, with a limit of 200,000 tokens, and walked through the steps. Each ran once, and every answer was graded on the hidden tests.

The benchmark's grader marks a few correct answers wrong, including one official test whose expected answer is itself wrong, so I corrected it, for both ways of asking. Uncorrected, the model solves 25 asked once and 29 guided, rather than 28 and 33.

To keep the comparison fair, I tuned the steps on 14 other problems, froze them, and ran them once on the 38. Two ideas came from watching two of the 38 in earlier trials, and I kept them. I also fixed in advance how lopsided the result had to be for me to call it more than luck. The code is public.

Three things the test cannot rule out. It is one run, and this model varies from run to run. The contests date from early 2025, so the models may have seen some of these problems in training, which would help both ways of asking. And mine is a compressed community build with its safety tuning removed, which may be weaker than the original.

The gain stopped at the hardest twelve

Walked through the steps, the model solved six problems it had failed when asked once, and lost one it had solved. It solved all of the easier 26. On the hardest twelve, the ones the study's models solved least often, it gained one, lost one, and spent most of its extra effort for nothing. The run took 78 hours of model time, two hours per problem on average.

The checks made the answers trustworthy. All 16 programs confirmed against a slow checker were right. Of the 18 checked more weakly, 16 were right. Both wrong answers had been checked only against the worked examples and a speed test. One correct answer was luck: the system ran out of attempts and submitted its last program, which happened to be right.

Six gains against one loss would happen by chance about one time in eight, short of the one in twenty I had set as the bar. On 38 problems, a gain this size is too small to tell from noise. What I predicted before the run did happen: all six gains were problems where, asked once, it had run out of room.

More room was not the answer

Asked once, the model solved 21% of the problems within 32,000 tokens, 55% within 100,000, 63% within 128,000 and 74% within 200,000. But the answers that ran long were not nearly done: none that went past 150,000 tokens finished.

The study shows the same for Qwen3.8-27B. Its answers were allowed 250,000 tokens. Cut at 128,000, it loses between one and two of the 38, and a quarter of its answers used the full 250,000 without finishing.

To test it directly, I asked the ten failures again, once each, with the most room the machine allows: about 261,000 tokens. Three were solved this time, one more than I had predicted, but all three finished well inside the old limit of 200,000. The model simply took a better route on a second try. The other seven used every one of the 261,000 tokens and still did not finish. The extra room solved nothing.

That second try also qualifies the main result. Guided, the model solved six of these ten, including the three that asking again also solved. The other three it solved only with the steps, on problems where even 261,000 tokens were not enough.

Where guidance runs out

A stronger open model, Qwen3.8-Flash-Next, solves 89.5% asked once (the dotted line), about one problem more than mine reached with a week of my guidance. Every model I could compare from the study gained 4 to 26 points from guidance on these problems, though some of the eleven it tested did not gain at all. The study's figures average five runs at a limit of 128,000 tokens, and come from hosted versions of each model. At that limit, mine would have solved 63% asked once, not 74%.

The strongest objection is that the limit I hit is mine, not the model's. Qwen3.8-27B started about where mine did and reached 96% with the study's guidance, so better steps might take mine further. My own fix step is evidence for that: given 32,000 tokens to work out why a program failed, it reasoned until the budget ran out in 28 of its 31 attempts and wrote no program. That is a flaw in my steps, not a proven limit of the model.

But each extra problem already cost about a million tokens, seven times the 145,000 each of the first 28 took asked once. Qwen3.8-Flash-Next would fit on the same machine, at 112 GB in a compressed build, though not alongside mine, and a compressed copy may score below the study's figure. Fixing my steps would cost more of my time than finding out.

What the model cannot do for itself

My model did not get any smarter this week. It solved thirteen points more when I stopped asking it to do everything in one go and told it what to do next: work out the method, write the program, check it, fix it.

The model could do every one of those steps. It could not choose them, set its own limits or know when to stop. The steps came from knowing how a good engineer works and from watching where this model fails. Without them, my model on my machine scores 74%.

My steps also ran out. On the hardest problems they bought nothing, and a better model asked once did about as well as everything I built. Better steps might go further, at a price. Knowing when that price is higher than switching models takes the same skill as building the steps.

The third of a million characters it wrote on that early trial came from a model left to do everything in one go.