Don Nelson: current status

On December 22 of last year I started a 60-day challenge to build an agent capable of doing electrical studies. I later extended it to 100 days, and at some point I stopped publishing because I was swamped.

The real number is 200 days. That's how long it took me to build Don Nelson and have it deliver a study approved by the CEN (Chile's national grid coordinator). I have to admit I thought it would take much less, but that's the real number.

How it works today

The study comes in — usually a CES (a scenario letter issued by the CEN) — and Don Nelson hands me a deliverable that can take between 10 and 12 hours (working fully autonomously). The part that takes the longest is updating the database, around 4 to 6 hours. After that it writes and hands me a version.

Iteration loop between the human and Don Nelson

I review it and, in the best case, it comes out right on the first pass with my modifications. If not, I have to iterate. So far I've never had to iterate more than twice.

How I leave comments for Don Nelson in Google Docs

I estimate Don Nelson does 80% of the work and I do 20%.

The architecture: one agent, one responsibility

At first the idea was for Don Nelson itself to go to InfoTécnica, PGP and Acceso Abierto (the CEN's public data platforms), but my conclusion was to copy what software engineers do with their SOLID methodology. The S (single responsibility principle) means one task per function, and it's what has worked best for me.

In this case, Mateo is the agent that is the model of the grid. I'll go deeper into Mateo in an upcoming post, but the basic idea is that it knows the grid better than anyone: it blends InfoTécnica, PGP, Acceso Abierto and the internet, and I can ask it anything imaginable about the grid.

The architecture: Don Nelson delegates to Mateo (the grid) and Spark (the simulations)

Same with Spark: Don Nelson doesn't need to know how to run simulations. It simply hands that job to Spark, which goes to DIgSILENT, writes the script and returns the data.

Results

Don Nelson has already delivered studies that were approved by the CEN. It can't do ECAP studies yet, because that knowledge is more complicated than the simple studies, and it can't do dynamic/EMT studies either.

Every decision Don Nelson makes, every simulation it hands to Spark and every query it sends to Mateo gets stored, so auditing is very easy. Every figure in the report comes from a single source generated by the simulation — there is never manual typing, because I learned (the hard way 😔) that a hand-transcribed number can contradict itself.

Another important point: Don Nelson doesn't start from zero. It learned by reading everything public from the CEN — the historical corpus of past studies — and with that it knows with certainty what reviewers look at.

Don Nelson has already audited and redone studies made by third parties that had observations.

Two things I learned verifying an agent

These two cost me dearly, and I think they are useful to anyone building agents, not just in the power world.

1. Read the raw data, never the agent's summary

When Spark finishes a simulation it returns two things: the raw results (a JSON with the numbers exactly as they came out of PowerFactory) and a natural-language summary of how it went. At first I read the summary — obviously, that's what it's for.

Wrong. The summary always reads well. The agent writes "all 50 cases ran and converge" with the same confidence when it's true as when only 45 ran. It's not that it lies: the text it generates is plausible by construction, and plausible is not the same as true. The rule I took away: the agent's narrative is for orientation, but anything that ends up in the report gets verified against the raw data.

2. Count instead of reading

This is the one that has surfaced the most findings. When I review one of Don Nelson's deliverables, I no longer start by reading it through — I start by counting: the text says N cases were run — are there N results in the backup? It says there are M contingencies — does the annex have M rows? Every number the document claims, I cross-check against the real count.

Why does it work? Because you catch nothing by reading. A sentence like "the case does not converge" reads perfectly fine even if the case has been converging for three versions — the text doesn't look broken. The gaps between what the document says and what the model holds are not visible by reading; they are visible by counting. It's the most boring review I do and by far the one that finds the most errors.

It sounds crazy, but it's completely true.

Other lessons

• The human's role has to be deciding, not detecting.
• Guardrails go in code, not in documents. The verification protocol was perfectly written… and it never ran. A checklist in Markdown is not a mechanism; a gate that runs after every database mutation and blocks progress is.
• Verify before writing. The write → verify → patch cycle declared the reports finished three times before they were. The right order I use today is: harvest → verify → freeze numbers → write.

Roadmap

Simple studies (EFP, ECB and study reviews) are already done. What I'm working on now is teaching Don Nelson to coordinate protections on its own. And after that, we go for dynamic studies.

If you need a study in record time, or want one reviewed before submitting it to the CEN, you can book a meeting with me or learn more at valdivia.tech and we'll make it happen.