3D Reconstruction of Objects in Hands without Real World 3D Supervision

Project Page | Paper | Supplementary | Video | Poster

This repository contains the code for the ECCV 2024 paper 3D Reconstruction of Objects in Hands without Real World 3D Supervision. If you find our code or paper useful, please cite

@inproceedings{Prakash2024HOI,
    author = {Prakash, Aditya and Chang, Matthew and Jin, Matthew and Tu, Ruisen and Gupta, Saurabh},
    title = {3D Reconstruction of Objects in Hands without Real World 3D Supervision},
    booktitle = {European Conference on Computer Vision (ECCV)},
    year = {2024}
}

Setup

Follow the instructions in IHOI to setup the environment and external dependencies.

Demo

To run the demo, download the model weights from here (this is an improved checkpoint which is also trained on MOW). The demo script is modified from IHOI repo to work with our model. Some test images are provided in demo folder.

Dataset

We use several datasets in our work, download the RGB images, segmentation masks, object masks and hand poses (if available) separately for each: ObMan, HO3D, MOW, VISOR.

The object meshes are processed using DeepSDF code into the required format in the same way as IHOI. For ObMan and HO3D, we use the mesh_sdf from IHOI. For MOW, it is recommended to run the DeepSDF code on the object meshes and save in the same format as ObMan and HO3D in the mesh_sdf folder.

We have also provided our preprocessed data here. These can directly be used with the RGB images and segmentation masks from the respective datasets.

We also curate multiview sequences of diverse objects from VISOR, referred to as WildObjectsinHands, which is available here. The Hands.pkl file from the drive is the same as visor_train.pkl in the preprocessed data. These do not have any ground truth object mesh.

After downloading the data, update the paths in experiments/*.yaml configs. The data structure should be as follows:

data
├── cache
├── mesh_sdf
├── obman
│   ... # ObMan data
├── ho3d
│   ... # HO3D data
├── mow
│   ... # MOW data
├── visor
│   ... # VISOR data from WildObjectsinHands  

Training

All models are initialized from ObMan pretrained checkpoint, it can be downloaded from here.

Our models are trained jointly on multiple datasets, using different forms of supervision as available, check experiments/combined.yaml for details.

python models/ihoi.py --config experiments/combined.yaml  --ckpt <PATH_TO_OBMAN_CKPT>

Training with discriminator requires xsecs_center_planes.npz which is available in the preprocessed data above.

Evaluation

To evaluate the model on the test set, run the following command:

python models/ihoi.py --config experiments/test.yaml --ckpt <PATH_TO_TRAINED_CKPT> --eval

License

All the material here is released under the Creative Commons Attribution-NonCommerial 4.0 International License, found here.

Acknowledgements

This codebase is built on top of the awesome IHOI repo. We thank all the authors of the datasets used in our work for making them publicly available. Check out their works as well.