I recently made this animation: https://www.newgrounds.com/portal/view/1044662

It's maybe not too crazy on the surface but it's made in a pretty different way than usual, and I'm hoping there's other people out there for whom this might be kinda interesting, so I wanted to explain a bit about how it works!

So all the visuals in this are p5.js sketches! p5.js is a graphics library aimed at artists, with the goal of making programmatic visuals as accessible as possible, designed for experimentation and iteration rather than like, engineering something to a specification. It's often used to teach programming because of all that. My day job is at Butter, a modular video editor, where the idea there is that all the components you can use are p5.js code. That makes it very extensible, so no asset is ever fully "dead" or locked in, as the code can easily expose controls for you. Mostly we've been using that to easily create new customizable blocks for users of Butter, but for a while, it's been clear to me that p5.js artists could be using this to create something new. There have been a lot of great tools popping up using creative code as a performance medium, but I see Butter doing something similar for more planned productions that need nonlinear editing.

So I finally got the opportunity to make something like that recently thanks to Butter doing a bigger push for features for coders, the addition of keyframing of sketch inputs, us needing to test slightly longer projects, and just having a compatible video idea. So here we are, testing this all out with some mild health/body horror!

Structuring a project

Butter should look familiar to anyone who's used any other video editor: there's a timeline, with pills on it representing each thing in the project. You can add new things from some menus on the left. The first menu item is the Build tab, which lets you add a new code component. This pops open a code editor!

So each thing in the timeline for my project is an editable p5.js sketch. If you createCanvas(windowWidth, windowHeight), normally used to make a p5 canvas that fills the window, then Butter will let you resize it on canvas freely. (Fixed-size canvases can still be scaled uniformly.)

You can also stack multiple sketches! If each sketch draws an object, you can position the different objects into a scene. But you can also stack full-screen sketches as filters. In this project, most of my sketches are black and white, and then I reuse a sketch that goes on top of all of those that applies a slight blur and applies a gradient map to recolor the content.

The code for the above is a pretty simple p5.strands shader, making some some color pickers and passing their values in:

let remap

let blur
let c1, c2, c3, c4

async function setup() {
  createCanvas(windowWidth, windowHeight, WEBGL);

  blur = createSlider(0, 40, 10, 0.1)
  c1 = createColorPicker('#000')
  c2 = createColorPicker('#B11')
  c3 = createColorPicker('#FB1')
  c4 = createColorPicker('#fff')
  
  remap = buildFilterShader(() => {
    let c1v = uniformVec4(color(c1.value()))
    let c2v = uniformVec4(color(c2.value()))
    let c3v = uniformVec4(color(c3.value()))
    let c4v = uniformVec4(color(c4.value()))
    filterColor.begin()
    let lum = (rgb) => dot(rgb, [0.2126, 0.7152, 0.0722])
    let v = getTexture(filterColor.canvasContent, filterColor.texCoord)
    let progress = lum(v.rgb)
    let out = c1v
    if (progress < 0.333) {
      out = mix(c1v, c2v, map(progress, 0, 0.333, 0, 1))
    } else if (progress < 0.666) {
      out = mix(c2v, c3v, map(progress, 0.333, 0.666, 0, 1))
    } else {
      out = mix(c3v, c4v, map(progress, 0.666, 1, 0, 1, true))
    }
    filterColor.set(out)
    filterColor.end()
  })
}

function draw() {
  clear()
  drawParentToCanvas()
  filter(BLUR, blur.value())
  filter(remap)
}

The way that it can apply this filter to the contents of another sketch rather than completely drawing its visuals from scratch is through the use of the Butter-specific function drawParentToCanvas(), which draws everything below the sketch on the timeline onto the sketch's canvas where it can be processed further.

Note also that regular p5.js sliders and inputs get turned into editable fields when you select the sketch! You can also add keyframes to most fields if you want a slider value to change over time. If you use orbitControl(), that's keyframeable too, and I used that a bit to animate camera movements in a few scenes.

If your sketch draws fully from scratch every frame based only on time variables like frameCount or millis(), you can seek anywhere instantly. But even for sketches that update incrementally from frame to frame, such as a physics simulation, you can still scrub back through time, and it will load for a moment and then show you the right frame.

Scene breakdowns

That's what you can do in Butter generally, but now let's look at a few scenes, which are still just standalone p5.js sketches.

Folds

This one is a single filter shader, using some domain warping on noise to produce the folds. A bulge is applied by pushing the coordinates out away from the location of the circle int he center. The bands come from fract(abs(coord.x) / 50), taking a number that goes from 0 up to a few hundred, and then making it go from 0-1 and repeat again every approximately 50 pixels (approximately, because this is a warped coordinate.)

let bulge
async function setup() {
  createCanvas(windowWidth, windowHeight, WEBGL);
  bulge = buildFilterShader(() => {
    filterColor.begin()
    noiseDetail(4, 0.5)
    let coord = (filterColor.texCoord - 0.5) * filterColor.canvasSize * 2

    let origCoord = coord
    let tOff = [0, frameCount * 2.5]
    coord = coord + tOff
    let s = 0.001
    coord = coord + ([
      noise(coord.x*s, coord.y*s, 0),
      noise(coord.x*s, coord.y*s, PI)
    ] - 0.5) * 200

    let pos = 5 * [sin(frameCount * 0.05), sin(frameCount*0.07)]
    let toPos = pos - origCoord
    let len = length(toPos)
    coord = mix(coord, pos, 1-smoothstep(0, 180, len))

    let outColor = 0.5 * (1 + sin(coord.x))
    if (len < 45) {
      outColor = 0.9
    } else if (abs(coord.x) < 20) {
      outColor = 0
    } else {
      outColor = mix(fract(abs(coord.x) / 50), 0, 0.5)
    }
    filterColor.set([outColor, outColor, outColor, 1])
    filterColor.end()
  })
}
function draw() {
  clear()
  filter(bulge)
}

Bellows

I particularly liked how this one turned out. It uses a material shader to (1) wrinkle and (2) inflate some spheres.

The wrinkling is just using some noise to offset the vertices of the sphere. However, you can't just move the vertex positions: you also need to update the normals in order for the lighting to be correct. I made a library for this in the past for something more robust, but if you can come up with two direction vectors u and v that are perpendicular to a surface, and you have a function warp(pt) that you're using to adjust the positions of a point, then you can pick a small value h and approximate the new normal as:

normal = normalize(cross(
  (warp(pt + h*u) - warp(pt)) / h,
  (warp(pt + h*v) - warp(pt)) / h
))

Then to do the inflation, I just apply less noise the more inflated the sphere is, making it closer to a perfect sphere, and I also extend the sphere out along its normal.

let pulse
async function setup() {
  createCanvas(windowWidth, windowHeight, WEBGL);
  pulse = buildMaterialShader(() => {
    worldInputs.begin()
    noiseDetail(4, 0.5)
    let s = 0.005
    let pulseAmount = uniformFloat()
    let amt = mix(0, 50, 1 - pulseAmount)
    let extrude = mix(0, 30, pulseAmount)
    let warp = (pt) => {
      return pt + ([
        noise(pt*s + 123.456),
        noise(pt*s + 456.123),
        noise(pt*s + 264.891)
      ] - 0.5) * amt
    }

    let d = 1
    let up = abs(worldInputs.normal.y) > 0.9 ? [1, 0, 0] : [0, 1, 0]
    let u = normalize(cross(worldInputs.normal, up))
    let v = cross(worldInputs.normal, u)
    const wc = warp(worldInputs.position)
    const wu = warp(worldInputs.position + u * d)
    const wv = warp(worldInputs.position + v * d)
    worldInputs.position = wc
    
    let n = normalize(cross((wu-wc)/d, (wv-wc)/d))
    worldInputs.normal = n
    worldInputs.position += n * extrude
    worldInputs.end()
  })
}
function draw() {
  randomSeed(1)
  background(0)
  noStroke()
  push()
  let v = 200
  directionalLight(v, v, v, 0, 0, -1)
  directionalLight(255, 255, 255, 1, 0, 1)
  directionalLight(255, 255, 255, -1, 0, 1)
  orbitControl()
  shader(pulse)
  for (let i = 0; i < 5; i++) {
    push()
    let p = pow(
      0.5 * (
        sin(-PI/2 + frameCount * 0.03 + i*0.05) + 1
      ),
      2
    ) + 0.05*(1+sin(frameCount*0.12 + i * 0.1))
    pulse.setUniform('pulseAmount', p)
    emissiveMaterial(50 + 50 * p)
    translate(
      random(-1, 1) * width/2,
      random(-1, 1) * height/2,
      random(-2, 1.1) * width,
    )
    sphere(30, 50, 50)
    pop()
  }
  pop()
}

Buildings

This one has two parts: a material shader for the clouds, and a material shader for the buildings.

The clouds are just a plane where I've used a p5.strands shader to make a cloud texture with noise. The noise is just used to mix between blue and white. I also make it fade to white at the end where I know the horizon will be.

The buildings use p5.strands instancing. A building's geometry is actually just one floor. When I draw it with multiple instances, the shader offsets each instance vertically. I just increase the number of instances over time to make them grow.

let clouds
let buildings
let seed
let buildingShader

let sceneStartTime
const sceneDuration = 5000

async function setup() {
  createCanvas(windowWidth, windowHeight, WEBGL)

  clouds = buildMaterialShader(() => {
    pixelInputs.begin()
    noiseDetail(4, 0.8)
    pixelInputs.color = mix(
      [0.5, 0.6, 1, 1],
      [1, 1, 1, 1],
      map(
        noise([
          pixelInputs.texCoord * 10 + [1, 2] * millis() * 0.0005,
          millis() * 0.0003
        ]),
        0.3,
        0.8,
        0,
        1,
        true
      )
    )
    pixelInputs.color = mix(pixelInputs.color, [1, 1, 1, 1], smoothstep(0.5, 0, pixelInputs.texCoord.y))
    pixelInputs.end()
  })

  buildingShader = buildMaterialShader(() => {
    let h = uniformFloat()
    worldInputs.begin()
    worldInputs.position.y -= h * instanceID()
    worldInputs.end()
  })

  resetScene()
}

function resetScene() {
  seed = random(1000000)
  randomSeed(seed)
  sceneStartTime = millis()

  push()
  colorMode(HSB, 1, 1, 1)

  buildings = []

  for (let i = 0; i < 60; i++) {
    const building = {}

    const h = random(50, 90)
    const w = random(100, 200)
    const d = random(100, 200)
    const numFloors = floor(random(10, 40))
    const rate = random(0.5, 5)

    const brickColor = color(random(), random(0.1), random(0.4, 0.8))
    const windowColor = color('#447')

    const windowH = random(0.5, 0.9) * h
    const numWindows = floor(random(5, 15))
    const windowW = w / numWindows * random(0.7, 0.95)

    const floorGeom = buildGeometry(() => {
      noStroke()

      fill(brickColor)
      box(w, h, d)

      fill(windowColor)
      translate(0, 0, d / 2)

      for (let j = 0; j < numWindows; j++) {
        push()
        translate(map(j + 0.5, 0, numWindows, -w / 2, w / 2), 0)
        box(windowW, windowH, 2)
        pop()
      }
    })

    const z = random(300, 8000)
    const x = random(-1, 1) * width * 3 * map(z, 0, 8000, 1, 3)
    const y = 0

    const draw = () => {
      push()

      translate(x, y, -z)

      buildingShader.setUniform('h', h)

      const elapsed = (millis() - sceneStartTime) / 1000
      model(floorGeom, min(numFloors, ceil(elapsed * rate)))

      pop()
    }

    buildings.push({ draw })
  }

  pop()
}

function draw() {
  if (millis() - sceneStartTime > sceneDuration) {
    resetScene()
  }

  background(255)
  orbitControl()
  noStroke()

  push()
  translate(0, -200, -800)

  shader(clouds)
  rotateX(PI / 2)
  plane(10000, 10000)

  pop()

  clearDepth()

  push()

  ambientLight(120, 120, 180)
  directionalLight(255, 250, 220, -1, 1, -0.5)
  directionalLight(200, 200, 200, -1, 1, -0.5)

  shader(buildingShader)

  translate(0, height / 2)

  for (const building of buildings) {
    building.draw()
  }

  pop()
}

The creative process

I've been yearning for a long time to replicate an experience like Flash where you can iterate between editing and coding in the same workspace. I'm happy to report that I've finally got a workflow that at least somewhat scratches that itch!

While certainly a lot of this could have been done the old way with more planning involved, it's quite freeing for me to be able to experiment with moving sketches around--maybe this visual would work better in this other part of the audio?--and then iterating on the visuals in that new context. It also let me quickly test out pulses in the code that match the beat of the music closely enough for the few seconds each scene is on screen that I didn't have to actually figure out the beats per minute of the music and convert that into frames.

So I'll definitely be doing more of this in the future. I also hope to eventualy capture more of the interactive parts of p5, such as mouse input, into Butter. Maybe one can record the mouse and then have it play back with the recorded values? Or maybe it should be keyframeable? Let me know if you have any thoughts on that or other things you'd like to use!