可微渲染 SoftRas 实践

SoftRas 是目前主流三角网格可微渲染器之一。

可微渲染通过计算渲染过程的导数，使得从单张图片学习三维结构逐渐成为现实。可微渲染目前被广泛地应用于三维重建，特别是人体重建、人脸重建和三维属性估计等应用中。

安装

conda 安装 PyTorch 环境：

conda create -n torch python=3.8 -y
conda activate torch

conda install pytorch torchvision torchaudio cudatoolkit=11.1 -c pytorch -c nvidia -y

conda activate torch
python - <<-EOF
import platform
import torch
print(f"Python : {platform.python_version()}")
print(f"PyTorch: {torch.__version__}")
print(f"  CUDA : {torch.version.cuda}")
EOF

Python : 3.8.10
PyTorch: 1.9.0
  CUDA : 11.1

获取代码并安装：

git clone https://github.com/ShichenLiu/SoftRas.git
cd SoftRas
python setup.py install

可设 setup.py 镜像源：

cat <<-EOF > ~/.pydistutils.cfg
[easy_install]
index_url = http://mirrors.aliyun.com/pypi/simple
EOF

应用

安装模型查看工具：

snap install ogre-meshviewer
# 或
snap install meshlab

渲染物体

渲染测试：

CUDA_VISIBLE_DEVICES=0 python examples/demo_render.py

渲染结果：

image

对比前后模型:

ogre-meshviewer data/obj/spot/spot_triangulated.obj

ogre-meshviewer data/results/output_render/saved_spot.obj

Mesh 重建

下载数据集：

bash examples/recon/download_dataset.sh

训练模型：

$ CUDA_VISIBLE_DEVICES=0 python examples/recon/train.py -eid recon
Loading dataset: 100%|██████████████████████████| 13/13 [00:35<00:00,  2.74s/it]
Iter: [0/250000]    Time 1.189  Loss 0.655  lr 0.000100 sv 0.000100
Iter: [100/250000]  Time 0.464  Loss 0.405  lr 0.000100 sv 0.000100
...
Iter: [250000/250000]   Time 0.450  Loss 0.128  lr 0.000030 sv 0.000030

image

测试模型：

$ CUDA_VISIBLE_DEVICES=0 python examples/recon/test.py -eid recon \
    -d 'data/results/models/recon/checkpoint_0250000.pth.tar'
Loading dataset: 100%|██████████████████████████| 13/13 [00:03<00:00,  3.25it/s]
Iter: [0/97]    Time 0.419      IoU 0.697
=================================
Mean IoU: 65.586 for class Airplane

Iter: [0/43]    Time 0.095      IoU 0.587
=================================
Mean IoU: 49.798 for class Bench

Iter: [0/37]    Time 0.089      IoU 0.621
=================================
Mean IoU: 68.975 for class Cabinet

Iter: [0/179]   Time 0.088      IoU 0.741
Iter: [100/179] Time 0.083      IoU 0.772
=================================
Mean IoU: 74.224 for class Car

Iter: [0/162]   Time 0.086      IoU 0.565
Iter: [100/162] Time 0.085      IoU 0.522
=================================
Mean IoU: 52.933 for class Chair

Iter: [0/26]    Time 0.094      IoU 0.681
=================================
Mean IoU: 60.553 for class Display

Iter: [0/55]    Time 0.087      IoU 0.526
=================================
Mean IoU: 45.751 for class Lamp

Iter: [0/38]    Time 0.086      IoU 0.580
=================================
Mean IoU: 65.626 for class Loudspeaker

Iter: [0/56]    Time 0.090      IoU 0.783
=================================
Mean IoU: 68.683 for class Rifle

Iter: [0/76]    Time 0.092      IoU 0.647
=================================
Mean IoU: 68.111 for class Sofa

Iter: [0/204]   Time 0.090      IoU 0.405
Iter: [100/204] Time 0.087      IoU 0.435
Iter: [200/204] Time 0.086      IoU 0.567
=================================
Mean IoU: 46.206 for class Table

Iter: [0/25]    Time 0.097      IoU 0.901
=================================
Mean IoU: 82.261 for class Telephone

Iter: [0/46]    Time 0.087      IoU 0.503
=================================
Mean IoU: 61.019 for class Watercraft

=================================
Mean IoU: 62.287 for all classes

Mesh 重建：

# 获取 `softras_recon.py` 进 `examples/recon/`
#   https://github.com/ikuokuo/start-3d-recon/blob/master/samples/softras_recon.py
# 注释 `iou` 直接返回 0，位于 `examples/recon/models.py` `evaluate_iou()`

# 2D 图像重构 3D Mesh
CUDA_VISIBLE_DEVICES=0 python examples/recon/softras_recon.py \
    -s '.' \
    -d 'data/results/models/recon/checkpoint_0250000.pth.tar' \
    -img 'data/car_64x64.png'

ogre-meshviewer data/car_64x64.obj

重建图像：

image

重建结果：

image

或重建 ShapeNet 数据集内图像：

# mesh recon images of ShapeNet dataset
CUDA_VISIBLE_DEVICES=0 python examples/recon/softras_recon.py \
    -s '.' \
    -d 'data/results/models/recon/checkpoint_0250000.pth.tar' \
    -imgs 'data/datasets/02958343_test_images.npz'

或使用 SoftRas 训练好的模型：

• SoftRas trained with silhouettes supervision (62+ IoU): google drive
• SoftRas trained with shading supervision (64+ IoU, test with --shading-model arg): google drive
• SoftRas reconstructed meshes with color (random sampled): google drive

更多

• Jrender - Jittor
  • JRender 解读
  • 3D Recon

GoCoding 个人实践的经验分享，可关注公众号！