Generating Spooky Loops with AnimateDiff on Blackhole

You don't need the most recent, efficient, or clever tool to make great art. Rediscovering old devices and making them your own is all part of the process. Open models on open hardware have created new possibilities. The more work you put into the models and architectures you generate with, the more unique those generations will be. Your hardware, your models, your art.

This post covers bringing up the AnimateDiff generative video model on Blackhole and generating its strange, prompted loops on the TT-QuietBox® 2 (QB2). The code and platform are open, so you can customize tt-animatediff as you see fit.

If you just want to make art and skip the background, start with the instructions below. The model can be run in CPU-only mode if you don’t have access to Tenstorrent hardware yet.

How to use tt-animatediff

Find the code on GitHub. You can study the full benchmark breakdown too.

Three things you can do with it:

# CPU — any machine, no hardware
python examples/generate.py --mode cpu --prompt "ocean waves at sunset, cinematic"

# Blackhole — full MotionAdapter, fast path
python examples/generate.py --motion-adapter --motion-adapter-skip up1 up2 \
  --prompt "aurora borealis over a frozen lake, cinematic 4K"

# Gradio UI — local or HuggingFace Spaces
python app.py

If you have a QuietBox 2 and want to explore a more complete generative art experience, tt-local-generator supports tt-animatediff out of the box.

---

I generated these loops on a Blackhole® p300c. 8 frames, 512×512, 25 steps PNDM (Pseudo Numerical Methods for Diffusion Models).

---

How AnimateDiff models work

AnimateDiff makes images move. You give it a prompt, it "chooses" images and imagery in response and then it makes that image move in a pleasant (some may say "spooky") loop.

AnimateDiff is based on Stable Diffusion, a common text-to-image model. Its core is a UNet, the network that removes noise from an image step by step, and on its own, it only looks at one image at a time. AnimateDiff inserts TemporalTransformer blocks — AnimateDiffTransformer3D in the diffusers implementation — at multiple points in the UNet. These blocks reshape the hidden states from (batch×frames, channels, H, W) to (batch, frames, channels, H, W) and run self-attention, which is a way for each position to weigh all the others and borrow from the ones that matter, across the frame dimension instead of the spatial one. As a result, each spatial position agrees with its counterpart in adjacent frames before the final image is committed.

This agreement gives us a sense of motion and consistency in the generated loops: animation.

Our tt-animatediff port, published on Hugging Face, is a weights-free implementation for Tenstorrent hardware. It pulls the upstream MotionAdapter checkpoint and does not ship its own. If you’re using the CPU path, you get the full MotionAdapter.

On Blackhole, you get to choose:.

• By default, the Stable Diffusion 1.4 UNet and VAE (the part that turns the finished latent back into a picture) run through TTNN™, and motion is approximated by blending attention across frames. That mode never loads the MotionAdapter.
• If you add --motion-adapter, the real thing runs at seven points inside the UNet.

Why does the approximation exist at all? That’s most of this blog: the compiled TTNN UNet is a sealed box with nowhere to plug the motion blocks in. We started from the outside and worked our way in.

The work happened in four phases. Phase 1 is a CPU baseline. Phase 2 runs the TTNN UNet on Blackhole. Phase 2.5 approximates motion by blending across frames. Phase 3 runs the real MotionAdapter.

---

How it performs on Blackhole

AnimateDiff has a healthy, joyful ecosystem on conventional GPU hardware, with artists primarily using it through ComfyUI and sd-webui-animatediff.

This port runs on Blackhole, in a QB2. A QB2 holds two p300c cards, each with two Blackhole chips, for four chips in total. Each chip has 32 GB of GDDR6 and 120 Tensix cores. Across the system that is 128 GB of GDDR6, about 2 TB/s of bandwidth, and 480 Tensix cores. Everything below is SD 1.4 through TTNN, with no xformers or kernel fusion yet. Wherever a row uses the MotionAdapter, the weights are the unmodified upstream ones.

Here is how each mode performs:

I am certain we can still go faster. Today, on one chip, a standard 8-frame clip takes about 15 seconds, and with the LCM-LoRA, it takes about 4 seconds. The real MotionAdapter takes longer: about 62 seconds with two injection points skipped and about 416 seconds with full temporal attention. This is a model brought up on an architecture it was never written for, with no xformers or kernel fusion yet, so there is real room to go faster. Today, you get full-quality output from unmodified motion adapter weights, with no patching of the accelerator runtime. If you find a speedup, tell us about it on Discord.

---

How a wrong turn made convincing loops

Full disclosure: I didn't know all that much about the inner workings of diffusion models when I started this project, but I wanted this to work and I wanted it to flow through Tenstorrent silicon.

The original implementation applied mm_sd_v15_v2.ckpt motion weights to SD 3.5's DiT (diffusion transformer).

SD 3.5's DiT operates on 2,432-dimensional features. The AnimateDiff motion weights were trained for SD 1.5's UNet, whose feature widths run from 320 to 1,280 channels. The dimensions don't match, and the math didn’t work. No temporal attention was applied — the transformer accepted the weight tensors, but the shapes didn't align in the intended way. The generation still appeared to work because shared noise initialization across frames already produces some visual consistency.

The "temporal coherence" I experienced was a mirage. Art is funny like that: if it looks good, who cares?

I generated loops for weeks without triggering an obvious error. The outputs looked reasonable to me. I only caught the bug by switching to an architecture that matched the original math and by noticing the difference.

The fix: use SD 1.4 UNet + MotionAdapter throughout. SD 1.4's UNet has the same channel widths (320 up to 1,280) that the motion adapter was trained against. Once the architecture matched, the difference between "temporal coherence from shared noise" and "temporal coherence from trained attention" became visible immediately. Of course, since both approaches are artful in their own ways, you can follow the "broken" path if you prefer.

Let's go deeper into the process of bringing a model like this to Blackhole.

---

Phase 1: CPU baseline

With the architecture correctly defined, I started with the most logical thing: wrap diffusers.AnimateDiffPipeline with the MotionAdapter checkpoint and run it on CPU. We all need comparators.

from diffusers import AnimateDiffPipeline, MotionAdapter, DDIMScheduler
adapter = MotionAdapter.from_pretrained("guoyww/animatediff-motion-adapter-v1-5-2")
pipe = AnimateDiffPipeline.from_pretrained("CompVis/stable-diffusion-v1-4", motion_adapter=adapter)
frames = pipe(prompt, num_frames=16, num_inference_steps=25).frames[0]

Speed: roughly two minutes per frame on CPU. Temporal quality: real AnimateDiff — full MotionAdapter attention running inside the UNet blocks at the UNet's own feature level. This is the reference point everything else is measured against.

The CPU path remains in the repo. It's the easiest way to validate that a given prompt and set of weights produce coherent output before moving to hardware.

---

Phase 2: TTNN UNet on Blackhole

The TTNN UNet for SD 1.4 lives in tt-metal. It's already compiled and tested for Blackhole via TT_METAL_ARCH_NAME=blackhole. Phase 2 loads it instead of the CPU UNet and runs frame generation on the Blackhole p300c.

device = setup_blackhole()   # open_mesh_device across all available chips
ttnn_model, ttnn_vae = load_sd14_ttnn(device)
# denoise each frame: N sequential TTNN UNet calls per denoising step
frames = generate_frames_temporal(device, ttnn_model, ttnn_vae, ...)

Speed on one Blackhole chip: ~1.9 seconds per frame at 25 steps (15.5 seconds for an 8-frame clip, warm kernels). The first clip in a fresh generate.py process takes about 21 seconds, because the first call pays a one-time warmup. Later calls in the same process, and requests to tt-model serve, take 15–16 seconds, which is roughly 60× faster than CPU for the spatial denoising step.

One thing worth noting about the setup code:

setup_blackhole() uses open_mesh_device with explicit physical_device_ids rather than open_device(device_id=0). This matters on multi-chip boards, because PCIe enumeration order isn't guaranteed, and opening only device_id=0 leaves other chips in an unmanaged state that can interfere mid-run. Opening all chips as a mesh up front prevents this.

---

Phase 2.5: the temporal coherence is coming from outside

The TTNN UNet is a compiled monolith. You call ttnn_model(latent, timestep, ...) and get a noise prediction back. You cannot add TemporalTransformer blocks to its internals without modifying the tt-metal source.

I don't modify tt-metal to get things done. I monkey patch it.

The workaround: temporal attention at the latent noise-prediction level.

After each frame is denoised at step t, you have N noise predictions on CPU — shape (N, 4, H/8, W/8). Before calling scheduler.step(), you attend across them:

def cross_frame_attention(tensors, alpha=0.35):
    N, C, H, W = tensors.shape
    x = tensors.permute(2, 3, 0, 1).reshape(H * W, N, C).float()
    scale = C ** -0.5
    attn = torch.softmax(torch.bmm(x, x.transpose(-2, -1)) * scale, dim=-1)
    attended = torch.bmm(attn, x).reshape(H, W, N, C).permute(2, 3, 0, 1)
    return ((1.0 - alpha) * tensors + alpha * attended).to(tensors.dtype)

This is cross-frame self-attention on 4-channel latent space, not on the UNet's internal features. It's cheaper and less precise than the full MotionAdapter path. It's also architecture-independent, and applicable to any model that produces per-frame noise predictions without requiring access to internal feature maps.

The --temporal-alpha parameter controls the blend weight. At 0.35 (default) you get coherent structure with visible per-frame variation. At 0.6+ the background stabilizes strongly and fine detail starts to flatten.

Phase 2.5: cross-frame attention on noise predictions. The hardware does the spatial denoising; the CPU does the inter-frame coordination.

---

An ill-advised distillation attempt

At this point the pipeline worked: real hardware, ~1.9 s/frame, decent coherence. The obvious next target was fewer steps — if we could distill a 4-step model, we'd get ~0.3 s/frame.

LCM (Latent Consistency Model) distillation trains a student UNet to match the teacher's output in far fewer steps by learning the denoising trajectory directly. I made four attempts across different learning rates (LR) and distillation configurations.

All four failed to converge. The loss landscape for distillation is sharp. The student has to match a highly structured multi-step trajectory, and the gradient signal is sensitive to LRs in a range that's hard to bracket. At each attempted LR, the result was either flat (no actual learning occurred) or diverged wildly. We archived our broken weights in weights/*.broken.

The lesson from this failure wasn't immediately obvious. It turned out I asked the wrong question: "how do we make each step cheaper?" The right question was "how do we change the step trajectory?"

So we started to investigate "Euler scheduling," used by AnimateDiff-Lightning. It covers the same total sigma range as PNDM but with different step spacing. This brings a different quality/structure tradeoff without needing distilled weights.

*25 steps, with standard classifier-free guidance (CFG), the dial for how closely to follow the text prompt, set to 7.5

Lightning mode on the TTNN path also required updating the cross-frame attention schedule. Each Euler step covers a larger sigma interval than PNDM, so a fixed-alpha blend at the wrong magnitude either collapses the frames into each other or has no effect. The implementation uses a cosine-decay alpha that starts high (strong structural agreement early) and decays toward the end (per-frame variety preserved in fine detail), and applies the blend at two points per step: once on the noise predictions before scheduler.step(), and a gentler pass on the resulting latents after.

---

Phase 3: a real MotionAdapter for Blackhole

Phase 2.5 cross-frame attention is real temporal coherence, but it is approximate — it operates on 4-channel latent noise predictions rather than on the UNet's intermediate features, where AnimateDiff was designed to work. Phase 3 brings the full MotionAdapter to Blackhole.

The TTNN UNet is a monolithic __call__. To inject temporal attention at 7 points inside it, the approach was to replicate the orchestration outside the source file and call each block object directly:

# forward_unet_staged() — never modifies tt-metal source
for block_idx, (block_type, down_block) in enumerate(zip(...)):
    s, res_samples = down_block(hidden_states=hidden_samples[i], temb=emb, ...)
    # after each CrossAttnDownBlock2D: inject temporal attention
    if f"down{block_idx}" in temporal_kernels:
        hidden_samples = _apply_temporal(hidden_samples, temporal_kernels[key], ...)

The 7 injection points are: down0, down1, down2 (encoder), mid (bottleneck), up0, up1, up2 (decoder). At each point, all N frame tensors are pulled from device to CPU, passed through AnimateDiffTransformer3D.forward() with the real pretrained motion weights, then pushed back. The spatial convolution and attention runs on Blackhole; the temporal attention runs on CPU with the full diffusers module — GroupNorm, proj_in/out, LayerNorm×3, positional embedding, GEGLU feedforward, all of it. The weights are unmodified.

The energy explosion bug

The first Phase 3 attempt produced pure noise — every output was entirely incoherent. There were two root causes:

1. A missing transpose. nn.Linear stores weights as [out_channels, in_channels] but the projection needs x @ w where w is [in_channels, out_channels]. The weight loader wasn't transposing. Square [C, C] matrices concealed this, because a square matrix transposed still has the right shape. The projection runs without error but computes wrong values. The energy ratio (output energy / input energy) going into the temporal modules was above 2.0. After adding .T.contiguous() to the weight load, it came down, but not far enough.

2. An incomplete temporal step. Each temporal step in AnimateDiff has to let every frame "look at" the other frames and borrow from them, so the video stays consistent from one frame to the next. That looking-around step is its attention, and I had written just that step, plus a line that adds its result back onto the original. It turns out the real module wraps attention in a lot of other machinery: normalization layers that keep the numbers in a stable range, input and output projections that reshape the data, a positional signal that tells the model which frame is which, and a small feed-forward network. Those pieces were trained together with the attention, so the attention can't be lifted out and used alone. Without them, the signal coming out of each temporal step was up to 2.5× stronger than what went in, and the noise compounded across the model. I replaced my partial version with a direct call to the real module (module.forward()). The ratio of output strength to input strength, which I was using as a health check, dropped below 1.25 at every injection point.

The L1 circular buffer constraint

Running N frames sequentially through the same TTNN cross-attention block revealed a constraint: the kernel allocates static circular buffers in L1. If a previous frame's output tensor is still resident in L1 when the same program dispatches for the next frame, the CB allocations overlap with the live buffer and the dispatch fails with an error at program.cpp:1476.

The fix is to evict each frame's output to DRAM before the next frame runs:

for i in range(num_frames):
    s, res_samples = down_block(hidden_states=hidden_samples[i], ...)
    s_dram = ttnn.to_memory_config(s, ttnn.DRAM_MEMORY_CONFIG)
    s.deallocate(True)
    new_hidden_dram.append(s_dram)

This is an L1 management discipline that the TTNN UNet doesn't need when processing a single frame. Sequential frame processing introduces it.

The Mayan glyph "Ajaw" across quality tiers:

• Q1: 16 frames, 4-chip, full MotionAdapter
• Q3: full 7-point MotionAdapter on Blackhole (Phase 3), ~52 s/frame
• Q4: skips the two decoder injection points (up1 + up2), ~7.7 s/frame: 6.75× faster than Q3, with only a minor visible difference

I find these generated loops of Mayan glyphs mesmerizing.

---

The asymmetric transfer optimization

At 7 injection points per step with N frames, the dominant cost in Phase 3 is PCIe transfers. The obvious fix: pull all N frame tensors from the device in a single call instead of N separate calls.

# Batched D→H: concatenate all N frames, pull once
batched = ttnn.concat(dram_samples, dim=0)   # [N, 1, 2*S, C]
raw_batch = ttnn.to_torch(batched).float()    # one PCIe transfer
batched.deallocate(True)

This eliminated N-1 PCIe round-trips per injection point. The measured speedup was 1.94×, from ~101 to ~52 seconds per frame on QB2 (4 chips across 2 p300c cards, 8 frames, 25 steps).

The H → D direction can't be batched the same way. The approach — push a single [N, ...] tensor to device and use ttnn.split to distribute it — fails because ttnn.split produces parent-buffer views. The downstream resnet reshard kernel can't reroute those views to the expected shard grid. Per-frame to_device() is the safe path.

Batched D→H, per-frame H→D. The optimization is real, but asymmetric — a TTNN-specific constraint with no obvious signal from the API that one direction works and the other doesn't.

---

The injection point cost distribution

Not all 7 injection points cost the same. The decoder blocks — up1 at 32×32 spatial resolution with C=1280, and up2 at 64×64 with C=640 — account for roughly 80% of the CPU transformer time. The encoder and mid blocks work at smaller spatial dimensions and run proportionally faster.

--motion-adapter-skip up1 up2 bypasses the two costliest points. Result: ~7.7 seconds per frame vs ~52 for the full Phase 3 run — a 6.75× speedup. This is still slower than Phase 2.5 (~1.9 s/frame). Encoder and mid-block temporal attention is retained. Decoder-side coherence is slightly reduced but the difference is minor for most prompts.

Lightning mode doesn't help here: combining --lightning with --motion-adapter yields ~50.6 s/frame, roughly the same as standard PNDM. The bottleneck is the CPU bridge calls per step, not the number of scheduler steps. Changing the solver doesn't change how many times you have to cross PCIe to run the temporal modules.

*Q4 skips up1 + up2.

All timings on one chip of a QB2 (two p300c cards, four chips), 8 frames, 512×512, warm kernels.

---

4-chip parallelism on the QB2

The TT-QuietBox 2 has four Blackhole chips across two p300c boards. The pipeline uses them in two distinct ways.

In-process frame sharding (Phase 2.5). Within a single generation run, the TTNN UNet denoising loop shards frames across all four chips in the same process. The compiled TTNN UNet expects exactly batch_size=2 (CFG runs the model twice per frame, once with your prompt and once without) per chip, so the mechanism is one CFG-doubled frame per chip per pass, in chunks of 4:

# plan_frame_sharding returns (True, 4) on a 4-chip mesh
_use_sharding, _chunk = plan_frame_sharding(num_frames, num_chips)

for c0 in range(0, num_frames, _chunk):
    # CFG-double each frame in this chunk, then shard across chips
    stacked_dev = shard_frames_to_device(cfg_latents, device, ...)
    # ShardTensorToMesh(dim=0) — chip K gets rows [K*2 : K*2+2]
    ttnn_out = ttnn_model(stacked_dev, ...)
    frame_outputs = gather_frames_from_device(ttnn_out, device, _chunk)

For 8 frames on 4 chips, this runs 2 sharded passes per denoising step instead of 8 serial UNet calls. For 16 frames it runs 4 passes. The TTNN UNet dispatches in parallel across all chips; each call to ttnn_model(...) fans out the work.

The efficiency benefit is most visible with 16 frames: at 8 frames the per-step overhead is proportionally larger; at 16 frames it's amortized across more useful work. Measured result: ~5.4 s/frame at 16 frames vs ~12.5 s/frame at 8 frames — a 2.3× per-frame improvement from the same hardware just by generating a longer clip.

num_frames must be divisible by num_chips. A partial final chunk would place fewer frames on the mesh than the compiled kernel expects and fail with a cryptic dispatch error. The pipeline rejects non-divisible counts early with a clear message listing the valid frame counts for the current rig.

Multi-process parallel throughput (all modes)

The --device-id INT flag pins a generate.py process to a specific chip (0-indexed). Running four processes simultaneously (one per chip) generates four independent clips in parallel at full single-chip speed. The World's Fair benchmark script uses this: 9 prompts across quality tiers dispatched 4-at-a-time, each batch of 4 completing in ~105 seconds wall-clock. That works out to roughly 26 seconds of wall-clock per clip.

This is a different axis of parallelism from in-process sharding: throughput over multiple clips rather than lower latency on one clip. Through composition, you can run 4-chip in-process sharding (faster per-frame) on each of those parallel processes simultaneously.

World's Fair Q1 tier: generated with 4-chip in-process sharding, 16 frames, Phase 3 MotionAdapter, PNDM 25 steps. The 16-frame sharded path (~5.4 s/frame) is what made this tier feasible to run across 9 prompts.

---

Chain mode: continuity across prompts

One of the later features we added was --chain — a way to carry visual continuity from one generation to the next without any explicit conditioning. It happens entirely in latent space, so nothing about it is specific to Blackhole. I think it's one of the spookier features and could be implemented in different ways to achieve different results.

At the end of a run, the final denoised latents are saved (--chain-save). At the start of the next run, those latents are blended into the base seed noise before denoising begins (--chain-from). The blend is frame-averaged (preserving coarse spatial layout), then renormalized to unit standard deviation so the scheduler's sigma scaling sees the expected noise distribution:

prev_mean = prev.mean(dim=0, keepdim=True).float()   # average across frames
mixed = (1.0 - alpha) * base_noise + alpha * prev_mean
result = mixed / mixed_std                            # renormalize

A key invariant discovered during development: the blended latents must be renormalized before use. An earlier version normalized per-channel before the frame average, which reduced signal standard deviation from ~0.28 to ~0.03 — less than 2% of the final variance. The chain signal was perceptually invisible. Without per-channel normalization, alpha=0.35 produces detectable (~15%) correlation with the previous layout.

To me, it looks like the faded photograph Marty McFly has of himself and his family in Back to the Future too. This model unintentionally reveals the blur of time.

The Unisphere chain: 4 independent generations across 100 years of imagined World's Fairs. Each run inherits the coarse spatial layout of the previous one via latent blending. No explicit conditioning — just latent continuity.

---

World's Fair showcase

With the full pipeline working, we generated batches for a "World's Fair" showcase: 9 historical and speculative fair prompts, run across three quality tiers, plus the Unisphere chain.

Visit the World’s Fair gallery if you can’t get enough of this window to the past and improbable future.

---

Wrapped in pixels: adding LoRA support

With the model now stable, we can make it even more creative: LoRA styles. LoRAs have a large creator around them. People use them to influence image and video generation in all kinds of artistic directions, like pixel-art, ANSI art, anime, and more. You can write your own too.

For this model, each LoRA (Low-Rank Adaptation) is a small file that nudges a model toward a style, and tt-animatediff merges it into the UNet once, when the model loads, so a frame costs no extra time. On one chip, a pixel art LoRA took 15.8 seconds for an 8-frame clip, against 15.3 without it.

Here’s how LoRAs express themselves:

The same prompt and seed in four rows: no LoRA, pixel art, storybook anime, and both stacked.

Here is a full loop of “the suspension bridge at Iguazu Falls,” 16 frames, with two seeds. In each pair, the clip on the left has no LoRA and the clip on the right uses the pixel art LoRA.

---

As an open model, the story with tt-animatediff is never over. There are many optimizations and peculiar quirks to realize. No matter which models come next, I know I’ll generate spooky loops with AnimateDiff forever. Make a few of your own, and share your best loop in the Tenstorrent Discord.