大语言模型可以通过代码自编码学习绘画LLMs Can Paint via Code Autoencoding

01 · BACKGROUND前沿模型能够使用代码逐笔作画Frontier models can paint with code, stroke by stroke

2026 年 11 月 9 日 Fable 5.2 发布当天，“用代码画画”刷屏。@Mr_Salio 让它分五个阶段画 Anthropic CEO Dario Amodei；第二天，@SMK_12159 回帖“GPT 也能做这个”，画了 OpenAI CEO Sam Altman。两个前沿语言模型选了同一条生成路径：不调用图像模型，用代码一笔一笔地画。

On November 9, 2026, the day Fable 5.2 launched, “painting with code” went viral. @Mr_Salio asked it to paint Anthropic CEO Dario Amodei in five passes. The next day, @SMK_12159 replied “GPT can do this too” with a painting of OpenAI CEO Sam Altman. Both frontier language models took the same generation path: they called no image model and painted with code, stroke by stroke.

这两个例子说明，纯语言模型可以把绘画分解成能执行、能检查、能迭代的代码。视觉生成未必是扩散黑盒，也可以是一种带中间状态的推理过程。于是我们想问：

These two examples show that a pure language model can decompose painting into code that it can execute, inspect, and iterate on. Visual generation need not be a diffusion black box. It can also be a reasoning process with explicit intermediate states. We therefore ask:

我们的做法是代码自编码：用拟合器把每张图片反编译成一段笔触程序（即逐笔作画的代码），再用固定渲染器执行这段程序，把图片画回来。这样，任意一张图片都能自动变成一条“图片–程序”训练样本，无需人工标注。我们再用这些样本训练语言模型，让它学会自己写出笔触程序。

Our approach is code autoencoding. A fitter decompiles each image into a stroke program (i.e., code that paints the image stroke by stroke), and a fixed renderer executes the program to draw the image back. In this way, any image becomes an image–program training pair without manual labels. We then train a language model on these pairs to write stroke programs itself.

02 · RESULTS图片生成是一个可学的代码生成任务Image generation is a learnable code-generation task

我们用代码自编码得到的笔触程序训练语言模型，具体做法见第 3 节。训练过程中，我们按先后保存了七个检查点（C1–C7），并在模型没有见过的测试集上测试每个检查点。

We train a language model on the stroke programs from code autoencoding (Section 3 gives the details). During training, we save seven checkpoints in order (C1–C7) and evaluate each one on a test set that the model has never seen.

模型逐渐学会用代码重建图片The model gradually learns to reconstruct images with code

PSNR 衡量重建图和原图有多接近：数值越高越像，大约每提高 3 dB，像素误差减半。从 C1 到 C7，平均重建 PSNR 从 8.78 dB 升到 17.44 dB，提高了 8.66 dB，相当于像素误差降到原来的约七分之一。把相邻两个检查点逐张比较，后一个检查点在大多数图片上画得更像。这条曲线至今没有变平。

PSNR measures how close a reconstruction is to the original image: higher means closer, and every gain of about 3 dB halves the pixel error. From C1 to C7, the average reconstruction PSNR rises from 8.78 dB to 17.44 dB. This gain of 8.66 dB cuts the pixel error to about one seventh. When we compare two adjacent checkpoints image by image, the later one paints closer to the original on most images. The curve has not flattened yet.

编辑与文生图更难学习，但仍学到基础Editing and text-to-image are harder to learn, but the model still learns the basics

与重建不同，编辑与文生图的分数并不乐观：编辑的平均 PSNR 从 17.05 缓降到 15.06 dB，文生图在 8.8–9.3 dB 横盘。但是，文生图能抓到基本概念，用大色块搭出天空、草地和主体轮廓；编辑也能找到要改的部位，在对的地方落笔（图 3、图 4）。问题出在后面：细节画不准，指令常做不完，编辑还容易画过头。目前提升的是这些仍很粗糙的基础能力，分数还没有反映出来。可能的原因有三个：

Unlike reconstruction, editing and text-to-image do not improve in score: the average editing PSNR decreases from 17.05 to 15.06 dB, while text-to-image remains within 8.8–9.3 dB. However, text-to-image captures basic concepts and lays down the sky, grass, and subject shapes in broad color blocks. Editing finds the region to change and paints in the right place (Figures 3 and 4). The failures come afterwards: details are inaccurate, instructions are often left unfinished, and editing tends to overpaint. At this stage, the gains lie in these still crude basics, which the scores do not yet reflect. There are three possible causes:

03 · METHOD将图片反编译成代码Decompiling images into code

上面的结果都建立在大量“图片–程序”训练样本之上，这些样本来自代码自编码。如果把一张图片看作一段程序运行的结果，图片重建就是反编译：给定像素，恢复产生它的笔触程序。这些程序由拟合器求解得到，无需人工标注。

The results above rely on a large number of image–program training pairs, which come from code autoencoding. If an image is the output of a program, reconstruction is decompilation: given the pixels, we find the stroke program that produced them. A fitter solves these programs without manual labels.

笔触虚拟机极简：画布上只有三类原语，即椭圆、胶囊和矩形。每一笔由中心、长短轴、角度、透明度与调色板颜色决定。一段程序有几百到一千多笔，渲染结果即目标图的近似。

The stroke VM (i.e., the virtual machine that executes stroke programs) is minimal: its canvas has only three primitives, namely ellipses, capsules, and rectangles. Each stroke is determined by its center, axes, angle, opacity, and palette color. Rendering a program of a few hundred to over a thousand strokes approximates the target image.

同一套笔触语言和同一个渲染器，配上不同的题目，就构成三个任务；训练数据覆盖重建、编辑和文生图。

The same stroke language and the same renderer support three tasks that differ only in their inputs. The training data cover reconstruction, editing, and text-to-image.

Task 1 · 重建Reconstruction

输入：图片Input: image输出：笔触程序Output: stroke program

用笔触程序重建参考图。

Reconstruct the reference image with a stroke program.

Task 2 · 编辑Editing

输入：原图 + 指令Input: image + instruction输出：覆盖层程序Output: overlay program

按编辑指令生成局部覆盖层。

Generate a local overlay from the editing instruction.

remove the lamp

Task 3 · 文生图Text-to-image

输入：文字Input: text输出：笔触程序Output: stroke program

根据文字描述生成笔触程序。

Generate a stroke program from a text description.

people flying kites on a bright sunny day A crowd of people flying kites in the grass.

拟合器每次保留降误差最多的一笔The fitter keeps the stroke that most reduces the error

三个任务用到的笔触程序都由拟合器求出。用多尺度笔触由粗到细地“画”出一张图，最早由 Hertzmann 系统化提出2Hertzmann, SIGGRAPH 1998.
查看参考文献；更早的交互式雏形见 Haeberli 19901Haeberli, SIGGRAPH 1990.
查看参考文献。Hertzmann 每笔的颜色直接从原图采样，其余参数用启发式确定；我们则按增益（即这一笔带来的误差下降）对每一笔做联合优化。

The stroke programs for all three tasks come from the fitter. Painting an image coarse to fine with strokes of multiple sizes was first systematized by Hertzmann2Hertzmann, SIGGRAPH 1998.
View reference, with an earlier interactive precursor by Haeberli in 19901Haeberli, SIGGRAPH 1990.
View reference. Hertzmann samples each stroke color from the source image and sets the other parameters by heuristics. In contrast, we jointly optimize each stroke by its gain (i.e., the error reduction it brings).

由粗到细。大笔先铺轮廓与色块，中笔修正结构，小笔只在仍不够像的地方补纹理。

Coarse to fine. The fitter first lays down silhouettes and color blocks with large strokes. It then fixes the structure with medium strokes and adds texture with small strokes only where the render still deviates from the image.

残差落笔。在残差最大处搜索位置与形状（长宽比）；哪一笔让当前渲染与原图的误差降得最多，就保留哪一笔。

Residual placement. The fitter searches the position and shape (i.e., aspect ratio) of a stroke where the residual is largest. It keeps the stroke that most reduces the error between the current render and the original image.

梯度定角度。角度取自图像梯度，颜色由最小二乘闭式解求出。

Angle from the gradient. The fitter takes the stroke angle from the image gradient and solves the color in closed form by least squares.

留下的笔触按顺序编译成整数笔触程序，再用 PSNR 门槛筛掉拟合失败的样本，剩下的就是训练数据。

We compile the kept strokes in order into an integer stroke program and drop failed fits with a PSNR threshold. The remaining programs form the training data.

拟合器和渲染器合在一起，就是标题里的代码自编码。它和自编码器（autoencoder）一样，先把图片转换为中间表示，再重建图片。不同的是，这里的中间表示是可读、可执行的笔触程序，由优化拟合得到，再交给固定渲染器执行。

Together, the fitter and the renderer form the code autoencoding in the title. Like an autoencoder, it maps an image to an intermediate representation and then reconstructs the image. In contrast, the intermediate representation here is a readable and executable stroke program, obtained by optimization and executed by a fixed renderer.

代码是优化得到的可执行潜表示The code is an executable latent found by optimization

代码自编码用优化求出代码，再交给固定渲染器执行；自编码器则同时学习编码器和解码器。Code autoencoding fits code by optimization and runs it on a fixed renderer, while an autoencoder learns both its encoder and its decoder.

示意图：上面一行是自编码器，输入图经可学习的编码器压成一个不可读的潜向量，再由可学习的解码器还原成重建图。下面一行是代码自编码，目标图经优化求解拟合出一段可读、可执行的代码，再由固定的渲染器执行这段代码，画回重建图。三处不同：编码靠优化求解而不是学习，中间表示是代码而不是向量，解码器是固定的渲染器而不是可学习的网络。 Diagram. Top lane, an autoencoder: an input image goes through a learned encoder into an unreadable latent vector, and a learned decoder turns it back into a reconstruction. Bottom lane, code autoencoding: code is fitted to the target image by optimization, giving a readable, executable program, and a fixed renderer executes that code to draw the reconstruction. Three differences: the encoder is solved rather than learned, the intermediate is code rather than a vector, and the decoder is a fixed renderer rather than a learned network.

自编码器Autoencoder

输入图Input image

编码器Encoder 可学习learned

潜表示Latent一个向量，不可读a vector, not readable

解码器Decoder 可学习learned

重建图Reconstruction

同样输入一张图same input

编码Encode学出来 → 解出来learned → solved

中间表示Middle向量 → 代码vector → code

解码Decode可学习 → 固定learned → fixed

同样画回原图same goal

代码自编码Code autoencoding

目标图Target

代码拟合Code fitting 优化求解solved by optimization

代码Code可读、可执行readable and executable

执行代码Execute the code 固定渲染器fixed renderer

重建图Reconstruction

04 · TRAINING DYNAMICS训练 loss 平稳下降，没有 spikeTraining loss decreases without spikes

每条程序是一段纯文本，只有笔触正文计 lossThe loss is computed only on the stroke code

拟合器留下的每条程序收成一段纯文本。前缀（任务、档位、画布定义；编辑另带输入图）不计 loss，笔触正文计 loss。我们用标准 NTP 训练，并定期保存检查点，评估它们在测试集上的表现。

Each program kept by the fitter becomes a piece of plain text. We exclude the prefix (i.e., the task, the tier, and the canvas definition, plus the input image for editing) from the loss and compute the loss only on the stroke code. We train with standard NTP and periodically save checkpoints to evaluate their performance on the test set.

两组训练 loss 都持续下降，但七个检查点在测试集上的表现表明，三项任务的改善程度并不一致。

Both training losses keep decreasing. However, the performance of the seven checkpoints on the test set shows different progress across the three tasks.

05 · LIMITATIONS局限Limitations

本文仍有几点局限。一方面，我们只评测了一个模型规模，且 PSNR 只与单张参考图逐像素比较，因此模型容量的影响尚未验证，这一指标对文生图也偏严格。另一方面，全部训练程序来自同一个拟合器，模型会继承它由粗到细的作画习惯，PSNR 门槛也会丢弃拟合失败的样本。

There are still several limitations in this post. On the one hand, we test only one model size and compute PSNR on a single reference image, which leaves capacity effects untested and is strict for text-to-image. On the other hand, all training programs come from one fitter, whose coarse-to-fine habits the model inherits and whose PSNR threshold drops failed fits.

06 · CONCLUSION结论Conclusion

本文用代码自编码训练语言模型逐笔作画：拟合器把图片反编译成可读、可执行的笔触程序，固定渲染器再把程序画回图片，语言模型则用标准 NTP 学习写出这些程序。实验表明，图片生成是一个可学的代码生成任务：图片重建的 PSNR 随训练稳定上升，仍未见 ceiling；编辑与文生图也学到了基础概念，只是 scaling 缓慢。

This post trains a language model to paint stroke by stroke through code autoencoding. A fitter decompiles each image into a readable, executable stroke program, a fixed renderer draws the program back, and the language model learns to write such programs with standard NTP. The results show that image generation is a learnable code-generation task. Reconstruction PSNR rises steadily with training and shows no ceiling yet, while editing and text-to-image learn basic concepts but scale slowly.

07 · RELATED WORK相关工作Related work

笔触渲染把图像表示为一串有序的绘制动作。Haeberli1Haeberli, SIGGRAPH 1990.
查看参考文献 用交互式笔触构造抽象的图像表示，Hertzmann2Hertzmann, SIGGRAPH 1998.
查看参考文献 则用多尺度曲线笔触由粗到细地渲染图像。神经绘画进一步学习如何落笔：Learning to Paint6Huang et al., ICCV 2019.
查看参考文献 用基于模型的强化学习训练绘画策略，Stylized Neural Painting8Zou et al., CVPR 2021.
查看参考文献 借助神经渲染器优化笔触参数，Paint Transformer9Liu et al., ICCV 2021.
查看参考文献 用前馈网络直接预测笔触。这些方法输出供渲染器使用的笔触参数，并不训练语言模型。我们用拟合器批量求出笔触程序，再用标准 NTP 学习完整序列。

Stroke-based rendering represents an image as an ordered sequence of drawing actions. Haeberli1Haeberli, SIGGRAPH 1990.
View reference builds abstract image representations from interactive strokes, and Hertzmann2Hertzmann, SIGGRAPH 1998.
View reference renders images coarse to fine with curved strokes of multiple sizes. Neural painting further learns how to place strokes. Specifically, Learning to Paint6Huang et al., ICCV 2019.
View reference trains a painting policy with model-based reinforcement learning, Stylized Neural Painting8Zou et al., CVPR 2021.
View reference optimizes stroke parameters through a neural renderer, and Paint Transformer9Liu et al., ICCV 2021.
View reference predicts strokes with a feed-forward network. While effective, these methods output stroke parameters for a renderer and do not train a language model. In contrast, we solve stroke programs at scale with a fitter and learn the complete sequences with standard NTP.

另一类工作把草图、图像或矢量图形看作程序或序列。SketchRNN3Ha and Eck, ICLR 2018.
查看参考文献 学习草图的笔画序列；SPIRAL4Ganin et al., ICML 2018.
查看参考文献 用强化对抗学习合成生成图像的程序；Ellis 等人5Ellis et al., NeurIPS 2018.
查看参考文献 从手绘图推断图形程序；DeepSVG7Carlier et al., NeurIPS 2020.
查看参考文献 用分层生成网络建模矢量图形。我们的笔触程序同样可执行，由一个固定的笔触虚拟机运行，同一套语言覆盖重建、编辑和文生图三个任务。

Another line of research treats sketches, images, or vector graphics as programs or sequences. SketchRNN3Ha and Eck, ICLR 2018.
View reference learns stroke sequences of sketches, and SPIRAL4Ganin et al., ICML 2018.
View reference synthesizes image programs with reinforced adversarial learning. Ellis et al.5Ellis et al., NeurIPS 2018.
View reference infer graphics programs from hand-drawn images, and DeepSVG7Carlier et al., NeurIPS 2020.
View reference models vector graphics with a hierarchical generative network. Our stroke programs are also executable. They run on one fixed stroke VM and share one language across reconstruction, editing, and text-to-image.

近期也有工作让多模态智能体借助工具逐笔作画。EASEL13Liu et al., arXiv 2026.
查看参考文献 把参考图引导的重建作为“灵巧视觉工具使用”的代理任务：智能体每一步同时观察参考图和当前画布，输出一笔的画笔参数，再由渲染器更新画布。它对 25 个模型的评测显示，现有智能体的重建相似度只有 0.40–0.54；它还把笔触渲染轨迹转成 44 万条逐步动作监督，用来训练 EASEL-9B。这类智能体做的是逐步闭环控制，每画一笔都要重新观察画布；我们则让模型在一次生成中写出完整的笔触程序，训练数据来自代码自编码。

Recent work also lets multimodal agents paint stroke by stroke through tools. EASEL13Liu et al., arXiv 2026.
View reference uses reference-guided reconstruction as a proxy task for dexterous visual tool use. At each step, the agent observes the reference image and the current canvas and outputs the parameters of one brush stroke. A renderer then updates the canvas. Its evaluation of 25 models finds reconstruction similarity of only 0.40–0.54. EASEL also converts stroke-based rendering trajectories into 440k next-action samples to train EASEL-9B. Such agents perform closed-loop control and observe the canvas again before every stroke. In contrast, our model writes the complete stroke program in one generation, with training data from code autoencoding.

文本驱动的矢量生成把语言引入这一方向：CLIPDraw10Frans et al., NeurIPS 2022.
查看参考文献 借助语言-图像编码器优化文字到绘图的合成，VectorFusion11Jain et al., CVPR 2023.
查看参考文献 从像素扩散模型中抽象出 SVG，StarVector12Rodriguez et al., CVPR 2025.
查看参考文献 直接从图像和文本生成 SVG 代码。这些进展促使我们把拟合器求出的长笔触程序作为语言模型的训练文本。

Text-conditioned vector generation brings language into this direction. CLIPDraw10Frans et al., NeurIPS 2022.
View reference optimizes text-to-drawing synthesis through language-image encoders, VectorFusion11Jain et al., CVPR 2023.
View reference abstracts SVGs from pixel-based diffusion models, and StarVector12Rodriguez et al., CVPR 2025.
View reference generates SVG code from images and text. These advances inspire us to use long stroke programs solved by the fitter as training text for a language model.

08 · DISCLAIMER声明Disclaimer

本研究出于作者的个人兴趣开展。文中观点仅代表作者本人，不代表作者所在的任何机构。作者声明不存在利益冲突。

The authors conducted this research out of personal interest. The views expressed here are the authors’ own and do not represent those of any organization they are affiliated with. The authors declare no competing interests.

09 · REFERENCES参考文献References

1. Paul Haeberli. Paint by Numbers: Abstract Image Representations. SIGGRAPH 1990.↩ ↩↩ ↩
2. Aaron Hertzmann. Painterly Rendering with Curved Brush Strokes of Multiple Sizes. SIGGRAPH 1998.↩ ↩↩ ↩
3. David Ha and Douglas Eck. A Neural Representation of Sketch Drawings. ICLR 2018.↩↩
4. Yaroslav Ganin, Tejas Kulkarni, Igor Babuschkin, S. M. Ali Eslami, and Oriol Vinyals. Synthesizing Programs for Images using Reinforced Adversarial Learning. ICML 2018.↩↩
5. Kevin Ellis, Daniel Ritchie, Armando Solar-Lezama, and Joshua B. Tenenbaum. Learning to Infer Graphics Programs from Hand-Drawn Images. NeurIPS 2018.↩↩
6. Zhewei Huang, Wen Heng, and Shuchang Zhou. Learning to Paint with Model-based Deep Reinforcement Learning. ICCV 2019.↩↩
7. Alexandre Carlier, Martin Danelljan, Alexandre Alahi, and Radu Timofte. DeepSVG: A Hierarchical Generative Network for Vector Graphics Animation. NeurIPS 2020.↩↩
8. Zhengxia Zou, Tianyang Shi, Shuang Qiu, Yi Yuan, and Zhenwei Shi. Stylized Neural Painting. CVPR 2021.↩↩
9. Songhua Liu, Tianwei Lin, Dongliang He, Fu Li, Ruifeng Deng, Xin Li, Errui Ding, and Hao Wang. Paint Transformer: Feed Forward Neural Painting with Stroke Prediction. ICCV 2021.↩↩
10. Kevin Frans, Lisa B. Soros, and Olaf Witkowski. CLIPDraw: Exploring Text-to-Drawing Synthesis through Language-Image Encoders. NeurIPS 2022.↩↩
11. Ajay Jain, Amber Xie, and Pieter Abbeel. VectorFusion: Text-to-SVG by Abstracting Pixel-Based Diffusion Models. CVPR 2023.↩↩
12. Juan A. Rodriguez, Abhay Puri, Shubham Agarwal, Issam H. Laradji, Pau Rodriguez, Sai Rajeswar, David Vazquez, Christopher Pal, and Marco Pedersoli. StarVector: Generating Scalable Vector Graphics Code from Images and Text. CVPR 2025.↩↩
13. Shudong Liu, Dongyang Chen, Enci Zhang, Jinwei Liang, Zheng Ma, and Lewei Lu. Paint What You See: Benchmarking Dexterous Visual Tool Use in Multimodal Agents. arXiv 2026.↩↩

10 · CITATION引用Citation

如需引用本文，可以使用下面的 BibTeX：

To cite this post, please use the following BibTeX entry:

@misc{sheng2026paintingbycode,
  title        = {LLMs Can Paint via Code Autoencoding},
  author       = {Sheng, Leheng and Zheng, Xiaosen and Chow, Wei and Sui, Yuan and Wang, Shizun},
  year         = {2026},
  howpublished = {Blog post},
  url          = {https://lehengthu.github.io/blog/2026/painting-by-code/}
}