What I did during the basketball game, or, browser screenshots in Sketch…
2025-06-13 by Philip Zeyliger
On the evening of May 2nd, the Rockets beat the Warriors 115-107. While I was watching that, I was also scratching an itch: Sketch should have a browser at its disposal! After all, if an agent can do so much with just bash, it could do even more with Chromium.
Here's the feature now:
To my surprise, Sketch was able to get the bones of this feature built with minimal prompting. I used the "Farm Out Step 0" strategy to get a research report on what Go library to use to control Chrome and what the API should look like. I pasted that into Sketch, and watched the Warriors get crushed. By the time I went to bed, I was giddily looking at a PNG of xkcd.com inside the Sketch chat window!
After the initial excitement, the most painful bit was installing Chromium in Sketch's containers. (This would break again when we switched to an Ubuntu 24.04 base image, but I snappily digress…) Sending images to LLMs is a bit subtle, too, and I accidentally sent Claude many base64 encoded images as chat messages, speeding right through my token budget.
After the browser tools, we also added the "background process" tool. Instead of running a command once with a timeout, and returning the results, the LLM calls the bash tool with a "background=true" flag, which returns the pid and the name of the temp file where the output has been redirected.
These tools have become a crowd favorite and surprisingly effective. With a dev server that can spin up mock data for your web components, you can give Sketch goals, not instructions and it'll iterate for longer. It does sometimes declare success too early, but since you've got the screenshots right there, it's easy to spur it to keep going with some specific feedback.
Sketch itself is open source, and we've got a hosted version up on sketch.dev. Give it a spin and take some screenshots.