Nov 30, 2023 07:00:00

Introducing generative AI 'MeshGPT' that can handle 3D objects like a large-scale language model handles sentences

An AI ``MeshGPT'' that generates 3D objects using the mechanism in which large-scale language models speak languages has appeared.

MeshGPT: Generating Triangle Meshes with Decoder-Only Transformers

https://nihalsid.github.io/mesh-gpt/

[2311.15475] MeshGPT: Generating Triangle Meshes with Decoder-Only Transformers
https://arxiv.org/abs/2311.15475

MeshGPT: Generating Triangle Meshes with Decoder-Only Transformers - YouTube


Large-scale language models can speak languages and, depending on training, can even 'speak' programming languages.

The research team then trained the model to 'speak' the meshes that make up 3D objects.

The specific training method is as follows. First, we learn mesh ``vocabulary'' based on data of various 3D objects, and then train

the Transformer based on the learned vocabulary.

The mechanism of vocabulary learning is shown in the diagram below. It allows neural networks to handle mesh

geometry and topology information.

After vocabulary learning, we teach the Transformer to remember the connections between meshes, similar to how a large language model remembers the connections between words.

Just as a large-scale language model can write a continuation of a sentence, MeshGPT can complete the rest based on part of the model.

MeshGPT is said to be able to generate models with moderate detail without having too many or too few vertices compared to other methods.

It is stated that MeshGPT has been able to significantly improve mesh generation, such as improving

the FID score , which indicates the quality of the product, by 30 points and increasing shape coverage by 9%.