TC4D: Trajectory-Conditioned Text-to-4D Generation

1University of Toronto 2Vector Institute 3Snap Inc. 4CUHK 5Stanford University 6NTU 7HKU 8University of Michigan 9SFU 10Google DeepMind
* equal contribution

ECCV 2024



Abstract

Recent techniques for text-to-4D generation synthesize dynamic 3D scenes using supervision from pre-trained text-to-video models. However, existing representations for motion, such as deformation models or time-dependent neural representations, are limited in the amount of motion they can generate-they cannot synthesize motion extending far beyond the bounding box used for volume rendering. The lack of a more flexible motion model contributes to the gap in realism between 4D generation methods and recent, near-photorealistic video generation models. Here, we propose TC4D: trajectory-conditioned text-to-4D generation, which factors motion into global and local components. We represent the global motion of a scene bounding box using rigid transformation along a spline, and we learn local deformations that conform to the global trajectory using supervision from a text-to-video model. Our approach enables the synthesis of scenes animated along arbitrary trajectories, compositional scene generation, and significant improvements to the realism and amount of generated motion, which we evaluate qualitatively and through a user study.


Method

Our method takes as input a pre-trained 3D scene generated using supervision from a 2D and/or 3D diffusion model. We animate the scene through a decomposition of motion at global and local scales. Motion at the global scale is incorporated via rigid transformation of the bounding box containing the object representation along a given spline trajectory at steps t. We align local motion to the trajectory by optimizing a separate deformation model that warps the underlying volumetric representation based on supervision from a text-to-video model. The output is an animated 3D scene with motion that is more realistic and greater in magnitude than previous techniques.

  architecture

Compositional Text-to-4D Generation

a bear walking, an astronaut riding a horse, deadpool riding a cow, a firepit
a giraffe walking, an elephant walking
a deer walking, a tiger walking, assassin riding a cow, a rhinoceros walking, water spraying out of a firehydrant
a goat walking, a sheep running, a seagull flying, a lamppost

Trajectory-Conditioned Text-to-4D Generation

a deer walking
a dog riding a skateboard
batman riding a camel
a seagull flying
a clown fish swimming
son goku riding an elephant
All Results

Comparisons with DreamGaussian4D and 4D-fy

DreamGaussian4D + Trajectory

4D-fy + Trajectory

Ours

an astronaut riding a horse
All Comparisons

Trajectory + Scale

w/o trajectory, w/o scale

w/ trajectory, w/o scale

w/ trajectory, w/ scale

a flame getting larger

Citation

@article{bah2024tc4d,
  author = {Bahmani, Sherwin and Liu, Xian and Yifan, Wang and Skorokhodov, Ivan and Rong, Victor and Liu, Ziwei and Liu, Xihui and Park, Jeong Joon and Tulyakov, Sergey and Wetzstein, Gordon and Tagliasacchi, Andrea and Lindell, David B.},
  title = {TC4D: Trajectory-Conditioned Text-to-4D Generation},
  journal = {arXiv},
  year = {2024},
}

Website template from DreamFusion and MVDream . We thank the authors for the open-source code.