GPT-6 Astra is out, which means another round of people trying to work out whether a new model is actually useful to them. This is harder than it sounds. A lot of people don’t really have a good test ready, so they ask a few questions, look at a benchmark chart, and decide whether intelligence has been solved or the whole thing is a scam.

Standard benchmarks have their place. They just might not measure much of what you need. A model doing well on a collection of difficult problems doesn’t tell me whether it can read a pile of documentation and turn it into a sensible plan without forgetting something halfway through.

I process a lot of information, and that is precisely the sort of work I want help with. Since I’m a good corporate citizen, I also obviously cannot use real workplace data to evaluate models in my personal AI subscriptions. So I needed something public, sufficiently complicated, and familiar enough that I could tell when the answer was nonsense.

So, what else is there that’s as complex as the work I do? Obviously, the answer is Guild Wars 1.

The Assignment

If you’ve missed it, you’re either too young, or you were too busy playing the Other Big Thing, aka World of Warcraft. Guild Wars is an online role-playing game where you take a character through quests and missions, acquire skills and equipment, and recruit a party.

The assignment is to create a practical roadmap for a fresh character, from character creation to reaching Consulate Docks, ready to enter the mission. No inherited equipment, transferred money, or helpful friend dragging you through everything. The model should include most worthwhile one-time quests while avoiding unnecessary grinding and travel.

I want a project plan: what to pick up, what to do together, in which order to cross an area, and when to return. The focus is on route planning, with only enough quest detail to explain the sequence.

Why This Works

First, I know the game by heart. I kid you not, I downloaded it again after about 10–15 years and still knew the exact order in which to traverse Zehlon Reach to avoid wasting a single minute backtracking. A plausible answer is much less convincing when you recognise the quest names and know that the proposed route doesn’t make sense. I can spend my time evaluating the plan instead of learning the subject well enough to evaluate it.

There is also a lot of information to digest. The official Guild Wars Wiki documents quests, rewards, locations, and requirements in a reasonably consistent format. That makes it a useful stand-in for the sort of documentation I deal with: individual pages are manageable, but the answer lives across many of them.

The dependencies are where it gets interesting. A quest can depend on another quest, a particular choice, or a stage of the story. Combining nearby activities sounds sensible until one of them isn’t available yet. The model has to reconcile the efficient geographical route with the order in which things can actually happen.

It also needs to maintain state. Character levels and Sunspear progression, a separate rank system that gates parts of the campaign, affect when you can continue. Listing the right quests is insufficient if the plan quietly assumes you’ve met a requirement without accounting for how you got there.

Finally, the deliverable is a DOCX. A somewhat elaborate plan still needs readable headings, checklists, sensible spacing, and useful source links. Producing a document you can actually follow is a decent task on its own. Getting the research right and then handing me an unreadable table would be a very recognisable workplace experience.

How Astra Did

Astra did very well in my test, but who cares? At this point my intelligence is the bottleneck, not the model’s.

That is a personal assessment, not a new leaderboard. I haven’t turned quest routing into a numerical scoring system, and one good run doesn’t tell you how a model will handle your work. It does give me a more useful impression than asking it a riddle whose answer has been circulating online for years. However, I’ve found a surprisingly high correlation between how well a model understands Guild Wars 1 and how well it understands one of the biggest cloud landscapes in the world.

If you want to try it, download the full benchmark prompt as Markdown.

You don’t need to care about Guild Wars to do something similar. Find a public subject you know well, give the model a task with the same kinds of complications as your work, and check whether the result holds together. Knowing when the answer is wrong is a fairly useful feature in a benchmark.