AI-Built Scratch Games
Using AI to build a full Scratch (.sb3) game from idea to sprites to a loadable project file — and to test-play it via browser control and computer vision.
◆ Playbook
An .sb3 file is a ZIP archive containing project.json (a JSON graph of sprites, scripts, and blocks — not visual drag-and-drop data) plus asset files named by content hash. That structure is simple enough for an AI agent to construct directly; a second, separate AI loop can then play the finished game in a browser by screenshotting the stage and reasoning over the pixels, since Scratch exposes no external debug/state API.
Design Principles
- Treat generation and testing as two separate systems: one produces the .sb3 file (mostly one-shot generation), the other plays it back (a perception-action loop). Don't couple them.
- Lock the design brief before generating any art — a fixed sprite list, control scheme, and win/lose condition stated as one sentence each, so the block-graph logic has something concrete to target.
- Generate sprite costumes on transparent backgrounds, never baked into a scene — Scratch composites sprites onto the backdrop itself.
- Validate a hand-built project.json against the Scratch Foundation's published JSON Schema before assuming it will load — most load failures are a missing required field, not a zip-format error.
- The browser-testing agent must read game state from screenshots only. There is no console or state API to query from outside — this is a genuine computer-vision task, not a shortcut.
- Build and verify the simplest possible project (one sprite, one script) by hand-loading it into the Scratch editor before generating anything complex.
Recommended Stack
- Claude for the generative half: design-brief writing, block-graph construction, sprite-prompt generation.
- Plain JSON + zip handling (Python or Node, no Scratch-specific SDK required) to assemble project.json and package the .sb3.
- Playwright — via its MCP server or Anthropic's Computer Use pattern — as the actual browser-control mechanism for the testing half; there is no Scratch-specific browser-automation tool.
- Claude's vision input to read stage-canvas screenshots directly; no separate OCR/object-detection step needed for typical simple 2D Scratch games.
- Nous Research's Hermes models as an open-weight substitute for the generation half if a fully self-hosted pipeline is required — verify current licensing before commercial use.
Best Use Cases
- Rapid prototyping: turn a one-paragraph game idea into a loadable .sb3 for playtesting within one session.
- Automated regression testing of an existing Scratch project after an edit, using the perceive-decide-act loop to catch a stuck sprite or unreachable game-over state.
- Generating a batch of sprite/backdrop variations for the same game concept to compare art directions quickly.
- Teaching contexts: showing how a visual, block-based format maps to a plain JSON graph underneath, as an intro to how agents 'see' structured formats generally.
Common Pitfalls
- Hand-guessing the project.json shape instead of validating against the real JSON Schema — small omissions (a missing `shadow` or `topLevel` field) cause silent load failures.
- Baking backgrounds into sprite art, which breaks compositing and makes sprites look pasted-on once moving over the actual backdrop.
- Sending the full browser screenshot (including the block editor and menus) to the vision model instead of cropping to just the stage canvas — this adds noise and produces less reliable state reads.
- Skipping manual verification of the generated .sb3 before wiring up the autonomous playtester — debugging a malformed block graph is much harder once a testing loop is also in the mix.
Tips
- Generate the backdrop last, after seeing the sprite art style, so the two match.
- Crop screenshots to the stage canvas only before sending them to the vision model.
- Start the perceive-decide-act loop with a generous fixed interval between cycles and tune it down — too fast wastes calls on near-identical frames, too slow misses fast game state changes.
- Ask the testing agent for structured pass/fail output (stuck sprite? reachable win state? score behaving correctly?) so repeated playtest runs are actually comparable.
✦ Prompts
Full library →◈ News
Full timeline →Monzo Co-Founder Tom Blomfield Takes Leave From Y Combinator to Join Anthropic
Tom Blomfield — co-founder of Monzo and GoCardless, and a Y Combinator general partner since 2023 — announced on July 13 that he is taking a leave of absence from YC to join Anthropic's compute team under co-founder Tom Brown. He said that as the industry enters the early stages of recursive self-improvement, compute availability is becoming one of the most important problems to solve, extending Anthropic's 2026 hiring run that already brought in Andrej Karpathy and DeepMind's John Jumper.
Google Makes 'Computer Use' a Built-In Tool in Gemini 3.5 Flash
Google announced that computer use is now a built-in tool in Gemini 3.5 Flash, letting developers build agents that can interact across platforms and software interfaces directly through the model. It extends the agentic capabilities Google showcased at I/O 2026 into a generally available developer tool on its fast, lower-cost Flash tier.
Reflection Secures SpaceX Colossus 2 Compute Through 2029
Open-source AI startup Reflection signed a compute agreement with SpaceX for immediate access to Nvidia GB300 systems at the Colossus 2 data center. Reported payments rise to $150 million per month from July 2026 through 2029.
Nvidia's Jensen Huang pledges 10x European AI compute capacity
In his GTC Paris keynote opening VivaTech 2026, Nvidia CEO Jensen Huang said the company will increase European AI computing capacity "by a factor of ten" over the next two years, framing AI infrastructure buildout as a generational industrial shift. The keynote coincided with multiple sovereign-AI infrastructure deals announced on the show floor.
xAI Completes Training Run for Next-Gen 'Grok V9 Medium' Model
xAI confirmed it has completed the training run for an internal checkpoint codenamed 'Grok V9 Medium,' a mid-sized successor in its Grok lineup intended to sit between Grok 4.3 and a future flagship. The company said evaluation and safety testing are underway ahead of a potential release.
Nvidia Begins Shipping Vera CPUs to Frontier AI Labs
Nvidia started delivering its Arm-based 'Vera' CPU — designed to pair with its Rubin-generation GPUs in unified rack-scale systems — to a handful of frontier AI labs and hyperscale cloud providers. Early recipients are integrating Vera-Rubin systems into next-generation training clusters expected to come online later in 2026.
⬡ Tools
All tools →△ Concepts
All concepts →AI that can see a screen and control a computer — mouse, keyboard, and all.