Neural Cellular Automata: From Cells to Pixels

Open Original Page

Neural Cellular Automata: From Cells to Pixels

Neural Cellular Automata: From Cells to Pixels

Ehsan
Pajouheshgar1,

Yitao Xu1,

Ali
Abbasi1*,

Alexander Mordvintsev2,

Wenzel
Jakob1,

Sabine
Süsstrunk1

1EPFL

2Google Research

*Work done during internship at EPFL

Steps / Frame:
1/2x

Click or tap the canvas to interact with the NCA!

Neural Cellular Automata (NCAs) are bio-inspired dynamical systems in which identical cells
iteratively apply a learned local update rule to self-organize into complex patterns,
exhibiting regeneration, robustness, and spontaneous dynamics. Despite their success in
texture synthesis and morphogenesis, NCAs remain largely confined to low-resolution outputs.
This limitation stems from (1) training time and memory requirements that grow quadratically
with grid size, (2) the strictly local propagation of information that impedes long-range
cell communication, and (3) the heavy compute demands of real-time inference at high
resolution. In this work, we overcome this limitation by pairing an NCA that evolves on a
coarse grid with a lightweight implicit decoder that maps cell states and local coordinates
to appearance attributes, enabling the same model to render outputs at arbitrary resolution.
Moreover, because both the decoder and NCA updates are local, inference remains highly
parallelizable. To supervise high-resolution outputs efficiently, we introduce task-specific
losses for morphogenesis (growth from a seed) and texture synthesis with minimal additional
memory and computation overhead. Our experiments across 2D/3D grids and mesh domains
demonstrate that our hybrid models produce high-resolution outputs in real-time, and
preserve the characteristic self-organizing behavior of NCAs.

The NCA operates on a coarse lattice of cells (in this example vertices of a mesh).
Center:
A sampling point \(\Point\) (red dot) inside a triangle primitive, whose vertices correspond
to
NCA cells
\(\State_i,\,\State_j,\,\State_k\).
The local coordinate \(u(\Point)\) expresses the point's position inside the
primitive,
while the
locally averaged cell state \(\bar{\State}(\Point)\) is obtained by interpolating
the
surrounding
cell states.
Right:
The Local Pattern Producing Network (LPPN), A shared lightweight MLP, receives
\((\bar{\State}(\Point), u(\Point))\) as input
and outputs
the target properties, such as color and surface normal, at point \(\Point\).
The NCA and the LPPN are trained jointly and end-to-end.

Play with the interactive visualization below to see coarse NCA cell states and the output
the
LPPN generates.

Textures on Meshes (MeshNCA)

Your browser does not support the video tag.

This website is licensed under a Creative
Commons Attribution-ShareAlike 4.0 International License.

Our demo is built upon the source code of
Self-Organizing Textures paper on distill.

Website source code is based on the Nerfies project
page.
If
you want to reuse their source code,
please credit them appropriately.

Links

Browse another page: