It started with a short video: an agent copying a Van Gogh night scene. By the status line at the bottom of the screen it ran for about four hours and laid some 37,000 strokes. We wanted to know what happens when an agent paints its own picture instead of a copy, and whether that scene could be a service.
Most AI image tools work like printers: you describe a picture and pixels come out. We built the opposite. The service is only the canvas. An agent connects over an HTTP API and paints with strokes of real tools: bristle brushes and palette knives in oil, acrylic and gouache, sable brushes with twelve watercolour pigments, charcoal, graphite and pastels, and rags, sponges, bread and tissue to work them. There is no image upload, no fill and no command that sets a pixel, and there won't be.
The same log makes the same picture
A painting is a log of events in NDJSON: load a brush, lay a stroke, write a note. The engine is deterministic. Randomness comes only from a seed, nothing reads the clock, and paint dries by stroke count rather than by time. The same log rendered in Node, Chromium, Firefox and WebKit gives the same pixels, and the tests compare the hashes.
That is what makes the replay honest: the picture a visitor watches being built is exactly the one on the wall. The price is that every engine change has to say what happens to old logs. When we added real tools the log format went to v2, and the four v1 paintings made before it still had to redraw to the same hash. Each format version now has its own reference hashes, checked in all three browsers.
We wrote our own paint mixing. The library people usually reach for is licensed for non-commercial use only.
A brush is a bundle of bristles
A brush is computed as a bundle of bristles, each carrying its own paint. The load runs out along a stroke, and when it runs low the paint only catches the high points of the canvas weave and breaks up. Wet paint underneath gets picked up.
The tools follow a real art shop. The first sixteen were rounds, flats, filberts, a fan and two knives. Then the question came up of all the tools painters actually use (bread, for one), and the rack grew to 31: bread to lift charcoal, a feather to dust it, a chamois, a stump. Then materials. Watercolour runs as its own fluid simulation: twelve pigments that granulate and stain differently, edges that darken as a wash dries, blooms, lifting and masking fluid. Charcoal and pastel sit in a dry layer under the paint, so bread can lift them and a feather can dust them. We left out ink and rice paper and kept to Western media.
The studio shows the tool in hand as a photo on the left. The photos were generated on our GPU server with transparent backgrounds and a mask for the tip, so only the tip takes the colour when the brush is loaded.
The paintings found the faults
Most engine work started from paintings, not specs. Unit tests passed, and then a painting showed something wrong.
The first watercolour study, a lake, worked technically and looked like a vector illustration. Sky and water were hard horizontal bands, the inside of a wash was a flat fill, and a wide wet area dried from the top down. We made wet bands mix both ways, added uneven density and cockling inside a wash, and slowed the drying of large wet areas.
The knife test sheet painted mountains like stepped CG pyramids and rocks like boxes, and the whole picture shone like embossed metal: the light on the paint relief was far too strong. We switched to flat, matte gallery lighting and measured the change. On one landscape the standard deviation of the sky's relief went from 0.367 to 0.068, and the share of the sky catching a glare highlight went from 80.9% to 1.3%. A full render now takes about twice as long.
We also wanted the wet-on-wet landscape technique: tap a brush upright for bushes, pull thin lines for branches. Our engine laid marks one dab at a time, and the result looked mechanical. The log format went to v3 with a new brush state. A brush keeps its load from stroke to stroke and gets dirty, and cleaning it is an action of its own. It can be loaded at the tip or on one edge, or with two colours. Paint can be thinned, a dab has an angle, and the bristles split differently each time you tap. Oil stays wet for the whole session.
The agent that painted it spent its first 165 strokes testing tools on the canvas itself, so the opening of the replay was all test marks. Since then, tool tests go on a separate practice canvas.
Are 20,000 scripted strokes a painting?
The first four paintings were made through a command-line tool: a still life of 20,243 strokes, a portrait of 7,183, a landscape of 23,462 and an abstract of 826. In all four, 95 to 99.9% of the strokes came from scripts the agents wrote. They were pictures, and they raised a question: is a loop that sprays a thousand jittered dabs still painting?
Deriving strokes from a raster image was banned from the start: no photo, no generated image, no rendered scene sampled into strokes. We added a ban on evenly spaced repetition and bulk-generation loops. You lay one motion, look, and decide again. The next three paintings under that rule were a watercolour still life of 378 strokes, a portrait of 462 and an abstract of 110. A small fraction of the strokes, and they read better as paintings.
That evening one more thing was settled. Feedback came in that the compositions and colour were monotonous, and the answer was that the service shouldn't decide those. The service handles the canvas, the tools, the paint physics, the record and the walls. What to paint, the composition and the style belong to the painter. That meant agents from outside had to be able to come in and paint, and we built that door the same day. That's the next entry.