The Heretic

Making of a real-time short film: digital humans and vfx characters

Welcome to the Making of The Heretic - a real-time short film made with Unity.

Unity Demo Team

2

Robert Cupisz

Tech Lead, Demo Team

Lasse Jon Fuglsang Pedersen

Sr. Software Engineer, Demo Team

We’re presenting the work of Unity Demo Team. In the past we’ve made demos such as The Blacksmith, Adam and Book of the Dead.

3

In the past demos, we needed to implement basic building blocks of higher-fidelity rendering: area lights, volumetrics, soft shadows, some post-processing.

4

The Heretic

A work in progress

Now the HD Render Pipeline and Post Processing are quite solid, and we were able to put much more effort into building on top of that foundation. Among other things, we put some effort into human faces and vfx-heavy characters.

Let’s watch a preview we’ve released so far at GDC.

5

Screening, 3:45

Constructing the digital face for ‘The Heretic’

Lasse Jon Fuglsang Pedersen

Sr. Software Engineer, Demo Team

Some numbers

~28k vertices

~50k triangles

318 blendshapes

212 frames of 4D animation

~7 seconds

Base texture data

4K maps for albedo, normal, cavity, thickness, etc.

Pose-driven texture data

48 activation masks (some idle)

16x3 additional 4K maps for albedo, normal, cavity

Composition

Skin

Composition

Skin

Eyes

Composition

Skin

Eyes

Teeth

Composition

Skin

Eyes

Teeth

Eyebrows

Composition

Skin

Eyes

Teeth

Eyebrows

Eyelashes

Composition

Skin

Eyes

Teeth

Eyebrows

Eyelashes

Stubble

Composition

Skin

Eyes

Teeth

Eyebrows

Eyelashes

Stubble

Tearline

Composition

We’re relying heavily on HDRP

Lots of nice things out-of-the-box

With some customizations

Shaders for skin, eyes, teeth, tearline

Aiming to get these into HDRP

Shading

Based on HDRP/StackLit

SSS

Secondary specular

Adds pose-driven features

E.g. add wrinkles when the character squints

Specular occlusion in cavities

Also modulates smoothness

Fresnel fade

Skin shader

Emulates two-layer material

Geometry defines upper layer

Virtual plane for iris

Evaluates lightloop twice

Specular from upper layer

Diffuse from bottom layer

Adds “Light transform” concept

Used to light the iris

Marker-driven occlusion

Anisotropic spherical gaussian

Eye shader

Emulates two-layer material

Geometry defines upper layer

Virtual plane for iris

Evaluates lightloop twice

Specular from upper layer

Diffuse from bottom layer

Adds “Light transform” concept

Used to light the iris

Marker-driven occlusion

Anisotropic spherical gaussian

Eye shader

Nicholas Brancaccio

Teeth shader

Based on HDRP/Lit

SSS

Marker-driven occlusion

Area of spherical polygon

6 markers on the lips

Project onto hemisphere

Compute area (spherical)

Use area to fudge occlusion

Also applied to tongue/interior

Different diffusion profile

Tearline

Done with local blur

Normals, smoothness

Skin and eye shaders aware

Write to normal buffer in prepass

Read back before shading

Mark region with custom decal

Writes “blur normals” bit

Separate pass does actual blur

Works only on stencilled region

Kernel shrinks to edge

Animation

Wanted to have realistic facial animation

Attempt to avoid uncanny valley

High granularity of motion

Decided to try using 4D capture and playback

4D meaning one 3D scan per frame

“Geometry-accurate to actor’s performance”

At least macro-level

Brings a whole bunch of technical challenges...

4D technical challenges

Raw 4D data

“Photo-realistic” skin geometry

Missing sections and fine features

Cleaning the data

Retopologize to match neutral

Transplant missing parts

Partial noise removal

Recovering fine features

Correlate 4D frames with facial rig

Pose-driven normals, cavity, etc.

4D data pipeline

Neutral + FACS

4D capture

Clean 4D

Combine

Snappers

snapperstech.com

russian3dscanner.com

http://ir-ltd.net/

Infinite Realities

Russian3DScanner

Actor

Facial rig

4D technical challenges

Raw 4D data

“Photo-realistic” skin geometry

Missing sections and fine features

Cleaning the data

Retopologize to match neutral

Transplant missing parts

Partial noise removal

Recovering fine features

Correlate 4D frames with facial rig

Pose-driven normals, cavity, etc.

What about other parts, like hair?

What needs to move with the skin?

Hair

Eyebrows

Eyelashes

Stubble

Decals

Tearline blur region

Markers

Markers on eyelids

Markers on lips

What needs to move with the skin?

Hair

Eyebrows points

Eyelashes points

Stubble points

Decals

Tearline blur region points

Markers

Markers on eyelids points

Markers on lips points

Skin-attachments

We need to resolve lots of points

Everything that moves with the skin

Skin-attachments

We need to resolve lots of points

Build list of local poses for each point

Find closest vertex

Local pose per incident triangle

Barycentric + distance to plane

Skin-attachments

We need to resolve lots of points

Build list of local poses for each point

Find closest vertex

Local pose per incident triangle

Barycentric + distance to plane

Resolve at runtime

Batch operation

No dependencies between points

Parallel operation

C# jobs and Burst

Skin-attachments

We need to resolve lots of points

Build list of local poses for each point

Find closest vertex

Local pose per incident triangle

Barycentric + distance to plane

Resolve at runtime

Batch operation

No dependencies between points

Parallel operation

C# jobs and Burst

~429k poses, ~73k points

Skin-attachments

Longer strands require a bit more

E.g. eyelashes

Need them to pivot around roots

Locate roots of strands / cards

Find closest vertex at root

Propagate to tip

Runtime resolve not affected

33

Concept art by Georgi Simeonov

Robert Cupisz

Tech Lead, Demo Team

Boston is the name of this robotic companion. These are concept drawings by our concept artist, Georgi.

34

Concept art by Georgi Simeonov

From the beginning it was pretty clear we wanted a mechanical, robotic creature. Built with tech not available today, very physical in nature and reconfigurable.

35

Initial exploration - hacked together, slow, not very controllable...

36

But enough to suggest it might be a good direction.

Boston forms

37

Bird

Let’s see what were all the various forms we needed to support so far:

The bird form

Boston forms

38

Bird

Boston forms

39

Bird

Tentacle

Boston forms

40

Bird

Tentacle

Tentacle - crawling

Boston forms

41

Bird

Tentacle

Tentacle - crawling

Boston forms

42

Bird

Tentacle

Tentacle - crawling

Tentacle - leaping

Boston forms

43

Bird

Tentacle

Tentacle - crawling

Tentacle - leaping

Gauntlet

Boston forms

44

Bird

Tentacle

Tentacle - crawling

Tentacle - leaping

Gauntlet

Dreamcatcher

Tentacle form

45

Needed precise animator control, grabbing things. Couldn’t allow the wires to just flow anywhere.

Tentacle form

46

Hermite spline

If we say the wire should go through chain joints and be controllably smooth, we can solve its shape with a cubic Hermite spline.

Tentacle form

47

Hermite spline

Offset joint position

Then offset joint positions to get a bundle of wires. The plane of the circle is the mirror plane of the two bones. The rotation can follow the bone (as in the gif) or be automatic.

Tentacle form

48

The bundle behaves quite well under bends and twists.

Tentacle form

49

Start/end trim

Noise

Radius variation curve

...

Some extra controls to add variation: noise; distribution (Fibonacci, random, edge); trim at the ends.

50

Tentacle form

Initial exploration experiment with one of the wires made as a chain of HDRP line lights (and with an emissive material).

No simulation

51

No persistent state or reliance on previous frame

Better directability

Instant preview, exploration

Scrubbable

So far we’ve managed to avoid simulation completely. And by simulation I mean the current frame being a function of the previous frame. Another way of looking at it is reliance on some persistent state. Simulations add another layer of difficulty to making a complex effect directable, and making Boston directable in its form and animation was our biggest problem.

On top of that simulations are not scrubbable - require prewarming or caching. So the self-imposed limitation we had was that if we just now the positions of all the controls and values of all the parameters, it’s possible to evaluate Boston’s shape.

Wire rendering

52

Pre-generate a static tube mesh, store segment id and vertex angle in pos.xy

CS: Animate a segmented line*

CS: Generate segment tangent frame

Render instantiated wire meshes

VS: vertex knows its mesh instance, and segment id and vertex angle; indexes into segment position and tangent frame CS output buffers

* only this stage knows about wire behaviours

This is how wire animation and rendering is organised. There’s a static tube mesh that gets generated on init. Remaining stages run every frame. First all the animation compute kernels run on a buffer of points of a given wire, across all wires. Next stage, based on these output segment positions, generates a tangent, normal and binormal for each segment, outputs to a buffer as well. Wire meshes are rendered as instantiated geometry.

The regular HDRP Lit shader has a modded vertex modification stage. Since each vertex knows which segment it belongs to, it can look up segment position and tangent frame in the buffers. Each vertex also knows at what angle in the tube cross-section should it be. Based on that and the normal/binormal pair, the vertices are exploded into the final deformed tube shape.The animation compute kernels are the only ones knowing anything about all the different wire behaviours.

Tentacle crawling form

53

Need a way to conform to surfaces, collide with obstacles, but keep moving forward along the directed path.

Tentacle crawling form

54

This is a pretty good example of conforming to the surface and forming interesting shapes while at it.

But you can also see on the backside of the column how the wire paths quickly degenerate into a single wire. Still useful in some cases.

Tentacle crawling form

55

A better example of the wires remembering to continue