Skip to content

tiangeluo/ShapeCompiler

Repository files navigation

Neural Shape Compiler: A Unified Framework for Transforming between Text, Point Cloud, and Program

In this repo, we release our data, codes, and pre-trained models for this Neural Shape Compiler [project page].

We released our data, model file, pre-trained model, and some of the inference codes. More inference codes, test codes and training codes are on the way.

Installation

Please proceed 1~4 steps below:

  1. git clone --recurse-submodules https://github.com/tiangeluo/ShapeCompiler.git
    
    conda create --name shapecompiler python=3.8
    conda activate shapecompiler
  2. install PyTorch and PyTorch3D.

    # my install commands
    pip install torch==1.9.0+cu111 torchvision==0.10.0+cu111 torchaudio==0.9.0 -f https://download.pytorch.org/whl/torch_stable.html
    
    conda install -c fvcore -c iopath -c conda-forge fvcore iopath
    conda install -c bottler nvidiacub
    pip install "git+https://github.com/facebookresearch/pytorch3d.git"
  3. cd ShapeCompiler
    python setup.py install
    # my gcc version is 8.2.0 for compiling cuda operators
    bash compile.sh
  4. download pre-trained model shapecompiler.pt from GoogleDrive , and move it to ShapeCompiler/outputs/shapecompiler_models

Inference

# to generate point clouds conditional on text
# results will be saved in ./outputs/shapecompiler_outputs/text2pts_test1
python generate_pts_condtext.py --model_path ./shapecompiler.pt --text 'a chair has armrests, with slats between legs' --save_name 'test1' 

# to generate text conditional on point clouds
# results will be saved in ./outputs/shapecompiler_outputs/pts2text_test1
python generate_text_condpts.py --model_path ./shapecompiler.pt --pts_path './assets/example_chair.ply' --save_name 'test1' 

# note that point clouds extract from ShapeNet has different orientation as we train ShapeCompiler
# assume point clouds has: pc.shape = [2048, 3]. you need to turn pc[:,2] = -1*pc[:,2]
# you can add flag --inverse in your command line to conduct pc[:,2] = -1*pc[:,2] 
# if you are not confident if the shape orientation is correct, please visualize ./assets/example_chair.ply

# to generate programs conditional on point clouds
# generated program parameters, program text, voxels, and extracted point clouds will be saved in ./outputs/shapecompiler_outputs/pts2pgm_test1
python generate_pgm_condpts.py --model_path ./shapecompiler.pt --pts_path './assets/example_chair.ply' --save_name 'test1' 

Pre-trained checkpoints

Description Link
Shape Compiler, training with all the data mentioned in paper Download (1.49GB)
PointVQVAE, training with ABO, ShapeNet, Program objects Download (107.3 MB)
PointVQVAE, training with ShapeNet objects Stay Tuned
PointVQVAE, training with ABO, ShapeNet, Program, Objaverse objects Stay Tuned

Data

Our [shape, structural description] paired data is stored under /data as pickle files and be loaded via data= pickle.load(open('abo_text_train.pkl','rb')). Each pickle file contains num of indices and pcs_name. You can accesee the text annotation by index (e.g., data[10]) and its correspondence point cloud file name (e.g., data['pcs_name'][10]).

Please also cite ABO, ShapeNet, Text2Shape, and ShapeGlot, if you use our caption data along with the objects provided in their datasets.

Other Interesting Ideas

  • Text->3D

With code: Text2Shape, DreamField, Shape IMLE, CLIPForge, MeshDiffusion

No official code release: DreamFusion, Magic3D, ShapeCrafter, Shape2VecSet, TAPS3D, 3DGen

  • 3D->Program

With code: ShapeProgram, ShapeAssembly, LegoAssembly

No official code release: ProgramViaImplicitPart

  • 3D->Text

With code: Scan2cap

Acknowledgement

We thank the below open-resource projects and codes.

BibTex

If you find our work or repo helpful, we are happy to receive a citation.

@article{luo2022neural,
      title={Neural Shape Compiler: A Unified Framework for Transforming between Text, Point Cloud, and Program},
      author={Luo, Tiange and Lee, Honglak and Johnson, Justin},
      journal={arXiv preprint arXiv:2212.12952},
      year={2022}
}

About

A Unified Framework for Transforming between Text, Point Cloud, and Program

Resources

License

Stars

Watchers

Forks

Releases

No releases published

Packages

No packages published

Languages