Pixal3D

Pixal3D is an image-to-3D generative model developed by Tencent ARC Lab that emphasizes fidelity in reconstruction from single or multi-view images. It addresses the common fidelity bottleneck where generated 3D outputs often struggle to precisely match the input image's geometry. By utilizing a pixel-aligned generation approach, the model ensures the 3D output remains consistent with the camera perspective of the source image, avoiding the alignment issues typical of models that rely on stochastic cross-attention.

Architecture and Methodology

The model utilizes a pixel back-projection conditioning scheme to establish explicit 2D-3D correspondences. This process involves lifting multi-scale 2D image features into a 3D feature volume by projecting rays from the camera through pixels into a 3D voxel grid. This allows every voxel along a specific ray to be associated with its corresponding pixel feature. The framework also incorporates a Pixel-Aligned Structured Latent Representation, which uses a Variational Autoencoder (VAE) to handle sparse Signed Distance Fields (SDF) efficiently.

The synthesis pipeline consists of two stages: a Structure Generation stage for defining coarse geometry and a Structured Latents Generation stage for refining geometric and textural details. While the initial research implementation was based on Direct3D-S2, the official open-source release is built on a Trellis.2 backbone for enhanced quality and stability. Pixal3D produces 3D assets with PBR (Physically Based Rendering) textures in the GLB format, suitable for integration into game engines and AR applications. The project is distributed under the MIT license.