In the AI online space lots of folks are arguing about models. Which one codes better. Which one has the bigger context window. Which one wins the benchmark this week. Which one halucinates less. Is Claude Code or Codex or Gemini CLI better, or should I use an open source harness like Pi.
The model and the harness matter, but I don’t think it’s the real moat — it isn’t the model or the harness or the training data or the fine tuning or even the agentic IDE right now. It’s really about portable docs, memory, guardrails, and workflows that work with your way of working. Both the Frontier project, and the startup I’m working on have been reinforcing this for me in a very concrete way.
At the beginning, the question for Frontier was: Can I get this old codebase to run on modern machines? But now the question is different: Can I create enough structure around it so that an AI agent running in a coding harness help me work safely inside it?
Practically speaking, this has meant being very rigorous about writing down what I learn, turning weird discoveries into docs and agent definitions, and leaning heavily into test-driven development and guardrail rules so that if the AI makes a mistake, it’s very obvious and surfaces immediately instead of creating ghosts in the machine.
These aren’t so different from the techniques and skills I developed over 20 years doing technical program and project management in enterprise and consulting contexts. The difference is that the AI works so fast and it has only the long-term memory that exists in the form of documentation or tracking tickets… it works so fast that if you don’t catch problems immediately, what you end up with is mush.
I have to capture long-term decisions before they disappear out of the session. I have to notice patterns of mistakes or problems, ask for suggestions about how to improve the guardrails or docs, and implement it immediately to eliminate the pattern. I build extensive sets tests and test runners with rules for when to run them so that the agents can verify their own work. I work to make sure that errors are legible, so don’t churn and try things out when “something broke,” but instead they see where it broke and what caused it.
Again, these are all things that would help human programmers too. The fact that my TPM skills have been so valuable in this context was initially a surprise, but now I completely understand it.
People seem to think it’s magic, and that they can just type a prompt and out pops a fully featured, polished product. It doesn’t work that way. You have to know what you want and write it down with enough detail that your AI coworker who comes in every morning with his mind erased, can actually get work done. This is the unglamorous part of AI-assisted development, but it’s the part that compounds because as your project grows, so does the documentation and this is critical for building up a head of steam that you can sustain.
A model can be replaced. Your harness might change. The workbench (your docs, rules, tests, guardrails, ticket system, etc) needs to be able to survive those changes. This is going to become more and more important as the foundation labs start reaching the point that they need to head towards a financial model that actually makes sense. Right now they’re all racing to be the best, and they’re heavily subsidizing all of our usage so that they can improve the models and win market positions. That can’t last forever. It’s quite literally unsustainable.
If your project’s intelligence only lives inside one vendor’s chat window, then you really don’t own much. You’re stuffed in the vendor’s trunk, not riding up front. If your project’s intelligence lives in markdown files, tests, issue threads, conventions, and repeatable workflows – on your own machine – then every model you use will get smarter when you use it to work on the project.
That’s the lesson.
The advantage is not having the best model on a given Tuesday. The advantage is building a system where any good model can become useful quickly, stay on the rails, remember what matters, and help move the work forward.
Ps. The models are very good at writing planning docs, issue tickets, and in some cases, the files that guide their own behavior. That’s how I do it: I prompt the model to write the docs. I’m its TPM.
Leave a Reply