With smooth enough geometry converting NeRFs to meshes with marching cubes works pretty well. Would you say the topology of meshes on our website are still too incoherent for rigging?
In case of a communication gap;
The word 'topology' has a more domain specific meaning in animation and rigging compared to the mathematical one.
It's used to mean that the placement of the lower level components - vertices, edges and faces - are well aligned to the higher level structure of the object, and make up a well defined 2D grid that flows along the models surface. In particular you'd want edges going along/perpendicular to mesh structures such as limbs, and around facial features and other details in a logical manner.
Otherwise when applying deformations as part of an animation the model will not have the detail in the right places to still look good, e.g. if there is no edges perpendicular to a joint in a limb, the bent version of the limb cannot have a clear smooth line along the joint, and the edges and faces become janky.
Under this definition, marching cubes cannot produce good 2D topology, as the mesh features are all aligned to the cardinal grid instead of the features of the object represented.
Curious about how long it took: brain storming, research, hypothesis, work, iteration, bug fixing, writing etc
I'm curious about the process you and the team uses to do this. Additionally there's the meme that these things are appearing every hour now, so it could be good for some perspective like "well actually it took n weeks"
Great question! Our team has been working on text-to-3d for ~1.5 years starting with https://ajayj.com/dreamfields. We had hoped that we could swap the contrastive CLIP model in Dream Fields for the generative Imagen model and crank out an easy paper in a few weeks. But what was supposed to be an easy win turned into months of frustration. Nothing we tried worked any better than Dream Fields. After a long detour trying MCMC, we stumbled across the score distillation loss that powers DreamFusion. Going from an initial sign of life to the results you see today still took months of hard work.
Research progress is unpredictable and these advances are not inevitable. We have the privilege to work in an environment full of amazing colleagues and powerful models, but at the end of the day it took a persistent team and a bit of luck.
not yet, but see the appendix of the paper for pseudocode. the core update step from the diffusion model that powers dreamfusion is surprisingly simple and easy to implement.
I think you're both right! It is incredible that the 2D model knows enough about the visual world to produce many objects from all angles, but the 3D model is essential for gluing these views together, and in some ways can fill in the gaps the 2D model doesn't know about. Imagine just taking a huge collection of photographs of an object. While there is enough information in those photos to reconstruct the 3D object, I wouldn't personally call that collection of images "an understanding of 3D." In our case, the diffusion model is the collection of photos and the NeRF model + optimization procedure is what figures out how all those photos can be related to a shared underlying 3D representation. - ben p (author)
directly training a NeRF on a single image is a terribly unconstrained problem that would lead to a volume that looks bad when the viewpoint changes. the gist of render + use diffusion model to refine is a great idea though and core to our method! the details of how to use the diffusion model for this refinement was the challenge, but once we figured that out it... just worked :)
Yes, this is often a problem. We use view-dependent prompts (e.g. "cat wearing sunglasses, back view") but the pretrained 2D model often does not do a good job of interpreting non-canonical views and will put sunglasses on the back of the cats head (as well as the front).
Bad prompt, missing implied antecedent/ambiguous subject...
You may want:
Back view of a cat which is wearing sunglasses, back view of a cat, but the view is wearing sunglasses, etc... I actually tried using projective terms from drafting books, and didn't get great results. Nor anatomicals either.
>Back view of a cat which is wearing sunglasses, back view of a cat, but the view is wearing sunglasses, etc... I actually tried using projective terms from drafting books, and didn't get great results. Nor anatomicals either.
In short: natural language is not good enough and you need a DSL. If only the last 60 years of language research had warned us of this.
Up next: English sentences are ambiguous and need context information to parse correctly. Machine learning community in shambles.
The trouble is all those darn uninitiated and trying to create a generalized oracle to map their inspecific ramblings to what they mean to free them of having to actually communicate properly...
Actually, funnily enough, this has cross section with philosophy in a way most programmers scoff at; but communication is frigging hard, and worse, detecting when someone is trying to get something across, but just needs a nudge in the right direction to be able to find the language to explain it is really damn hard.
I run into it every time I get a haircut. I have no idea how to speak their language, so it's always "Uh... A little off the top and rounded at the back, I guess?"
Co-author here - we were also surprised :) The breadth of knowledge of the visual world embedded in these 2D models and what they unlock is astounding.
