Skip to content

Commit e2a37b3

Browse files
shrubbshrubb
committed
Settle Human3.6M preprocessing and indices, support torch=1.4.0a0+829499e, pin CUDA memory
1 parent 816f3b3 commit e2a37b3

15 files changed

+126
-82
lines changed

‎README.md

+11-11
Original file line numberDiff line numberDiff line change
@@ -22,27 +22,27 @@ pip install -r requirements.txt
2222

2323
#### Human3.6M
2424
1. Download and preprocess the dataset by following the instructions in [mvn/datasets/human36m_preprocessing/README.md](https://github.com/karfly/learnable-triangulation-pytorch/blob/master/mvn/datasets/human36m_preprocessing/README.md).
25-
2. Place the preprocessed dataset to `./data/human36m`. If you don't want to store the dataset in the directory with code, just create a soft symbolic link: `ln -s {PATH_TO_HUMAN36M_DATASET} ./data/human36m`.
26-
3. Download pretrained backbone's weights from [here](https://drive.google.com/open?id=1TGHBfa9LsFPVS5CH6Qkcy5Jr2QsJdPEa) and place them here: `./data/pretrained/human36m/pose_resnet_4.5_pixels_human36m.pth` (ResNet-152 trained on COCO dataset and finetuned jointly on MPII and Human3.6M).
27-
4. If you want to train Volumetric model, you need rough estimations of the 3D skeleton both for train and val splits. In the paper we estimate 3D skeletons via Algebraic model. You can use [pretrained](#model-zoo) Algebraic model to produce predictions or just take [precalculated 3D skeletons](#model-zoo).
25+
2. Download pretrained backbone's weights from [here](https://drive.google.com/open?id=1TGHBfa9LsFPVS5CH6Qkcy5Jr2QsJdPEa) and place them here: `./data/pretrained/human36m/pose_resnet_4.5_pixels_human36m.pth` (ResNet-152 trained on COCO dataset and finetuned jointly on MPII and Human3.6M).
26+
3. If you want to train Volumetric model, you need rough estimations of the pelvis' 3D positions both for train and val splits. In the paper we estimate them using the Algebraic model. You can use the [pretrained](#model-zoo) Algebraic model to produce predictions or just take [precalculated 3D skeletons](#model-zoo).
2827

2928
## Model zoo
30-
In this section we collect pretrained models and configs. All **pretrained weights** and **precalculated 3D skeletons** can be downloaded from [Google Drive](https://drive.google.com/open?id=1TGHBfa9LsFPVS5CH6Qkcy5Jr2QsJdPEa) and placed to `./data` dir, so that eval configs can work out-of-the-box (without additional setting of paths).
29+
In this section we collect pretrained models and configs. All **pretrained weights** and **precalculated 3D skeletons** can be downloaded at once [from here](https://drive.google.com/drive/folders/1yjnD47hdcFNvbQj87pXDgqGt52K7rz_z) and placed to `./data/pretrained`, so that eval configs can work out-of-the-box (without additional setting of paths). Alternatively, the table below provides separate links to those files.
3130

3231
**Human3.6M:**
3332

34-
| Model | Train config | Eval config | Weights | Precalculated results | MPJPE (relative to pelvis), mm |
35-
|----------------------|:---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|:-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|:------------------------------------------------------------------------------------------:|:--------------------------------------------------------------------------------------------:|-------------------------------:|
36-
| Algebraic | [train/human36m_alg.yaml](https://github.com/karfly/learnable-triangulation-pytorch/blob/master/experiments/human36m/train/human36m_alg.yaml) | [eval/human36m_alg.yaml](https://github.com/karfly/learnable-triangulation-pytorch/blob/master/experiments/human36m/eval/human36m_alg.yaml) | [link](https://drive.google.com/file/d/1HAqMwH94kCfTs9jUHiuCB7vt94rMvxWe/view?usp=sharing) | [link](https://drive.google.com/file/d/1RO7tQdUI2gmlA58Jvw9M21NWIYPQgjPE/view?usp=sharing) | 22.5 |
37-
| Volumetric (softmax) | [train/human36m_vol_softmax.yaml](https://github.com/karfly/learnable-triangulation-pytorch/blob/master/experiments/human36m/train/human36m_vol_softmax.yaml) | [eval/human36m_vol_softmax.yaml](https://github.com/karfly/learnable-triangulation-pytorch/blob/master/experiments/human36m/eval/human36m_vol_softmax.yaml) | [link](https://drive.google.com/file/d/1r6Ut3oMKPxhyxRh3PZ05taaXwekhJWqj/view?usp=sharing) | — | **20.4** |
33+
| Model | Train config | Eval config | Weights | Precalculated results | MPJPE (relative to pelvis), mm |
34+
|----------------------|:--------------------------------------------------------------------------------------------------------------------------------------------------------------|:------------------------------------------------------------------------------------------------------------------------------------------------------------|:------------------------------------------------------------------------------------------:|:------------------------------------------------------------------------------------------------------------------------------------------------------:|-------------------------------:|
35+
| Algebraic | [train/human36m_alg.yaml](https://github.com/karfly/learnable-triangulation-pytorch/blob/master/experiments/human36m/train/human36m_alg.yaml) | [eval/human36m_alg.yaml](https://github.com/karfly/learnable-triangulation-pytorch/blob/master/experiments/human36m/eval/human36m_alg.yaml) | [link](https://drive.google.com/file/d/1HAqMwH94kCfTs9jUHiuCB7vt94rMvxWe/view?usp=sharing) | [train](https://drive.google.com/open?id=1MPvDW19H-j9t3ekartbOnoXsiHQiZjej), [val](https://drive.google.com/file/d/1RO7tQdUI2gmlA58Jvw9M21NWIYPQgjPE) | 22.5 |
36+
| Volumetric (softmax) | [train/human36m_vol_softmax.yaml](https://github.com/karfly/learnable-triangulation-pytorch/blob/master/experiments/human36m/train/human36m_vol_softmax.yaml) | [eval/human36m_vol_softmax.yaml](https://github.com/karfly/learnable-triangulation-pytorch/blob/master/experiments/human36m/eval/human36m_vol_softmax.yaml) | [link](https://drive.google.com/file/d/1r6Ut3oMKPxhyxRh3PZ05taaXwekhJWqj/view?usp=sharing) | — | **20.4** |
37+
3838
## Train
3939
Every experiment is defined by `.config` files. Configs with experiments from the paper can be found in the `./experiments` directory (see [model zoo](#model-zoo)).
4040

4141
#### Single-GPU
4242
To train a Volumetric model with softmax aggregation using **1 GPU**, run:
4343
```bash
4444
python3 train.py \
45-
--config train/human36m_vol_softmax.yaml \
45+
--config experiments/human36m/train/human36m_vol_softmax.yaml \
4646
--logdir ./logs
4747
```
4848

@@ -55,7 +55,7 @@ To train a Volumetric model with softmax aggregation using **2 GPUs on single ma
5555
```bash
5656
python3 -m torch.distributed.launch --nproc_per_node=2 --master_port=2345 \
5757
train.py \
58-
--config train/human36m_vol_softmax.yaml \
58+
--config experiments/human36m/train/human36m_vol_softmax.yaml \
5959
--logdir ./logs
6060
```
6161

@@ -79,7 +79,7 @@ Run:
7979
```bash
8080
python3 train.py \
8181
--eval --eval_dataset val \
82-
--config eval/human36m_vol_softmax.yaml \
82+
--config experiments/human36m/eval/human36m_vol_softmax.yaml \
8383
--logdir ./logs
8484
```
8585
Argument `--eval_dataset` can be `val` or `train`. Results can be seen in `logs` directory or in the tensorboard.

‎experiments/human36m/eval/human36m_alg.yaml

+2-3
Original file line numberDiff line numberDiff line change
@@ -25,7 +25,6 @@ model:
2525
init_weights: true
2626
checkpoint: "./data/pretrained/human36m/human36m_alg_10-04-2019/checkpoints/0060/weights.pth"
2727

28-
2928
use_confidences: true
3029
heatmap_multiplier: 100.0
3130
heatmap_softmax: true
@@ -44,7 +43,7 @@ dataset:
4443
kind: "human36m"
4544

4645
train:
47-
h36m_root: "./data/human36m/processed/_undistorted"
46+
h36m_root: "./data/human36m/processed/"
4847
labels_path: "./data/human36m/extra/human36m-multiview-labels-GTbboxes.npy"
4948
with_damaged_actions: true
5049
undistort_images: true
@@ -58,7 +57,7 @@ dataset:
5857
num_workers: 8
5958

6059
val:
61-
h36m_root: "./data/human36m/processed/_undistorted"
60+
h36m_root: "./data/human36m/processed/"
6261
labels_path: "./data/human36m/extra/human36m-multiview-labels-GTbboxes.npy"
6362
with_damaged_actions: true
6463
undistort_images: true

‎experiments/human36m/eval/human36m_ransac.yaml

+2-2
Original file line numberDiff line numberDiff line change
@@ -43,7 +43,7 @@ dataset:
4343
kind: "human36m"
4444

4545
train:
46-
h36m_root: "./data/human36m/processed/_undistorted"
46+
h36m_root: "./data/human36m/processed/"
4747
labels_path: "./data/human36m/extra/human36m-multiview-labels-GTbboxes.npy"
4848
with_damaged_actions: true
4949
undistort_images: true
@@ -57,7 +57,7 @@ dataset:
5757
num_workers: 8
5858

5959
val:
60-
h36m_root: "./data/human36m/processed/_undistorted"
60+
h36m_root: "./data/human36m/processed/"
6161
labels_path: "./data/human36m/extra/human36m-multiview-labels-GTbboxes.npy"
6262
with_damaged_actions: true
6363
undistort_images: true

‎experiments/human36m/eval/human36m_vol_softmax.yaml

+4-4
Original file line numberDiff line numberDiff line change
@@ -15,7 +15,7 @@ opt:
1515
n_epochs: 9999
1616

1717
batch_size: 5
18-
val_batch_size: 20
18+
val_batch_size: 75
1919

2020
lr: 0.0001
2121
process_features_lr: 0.001
@@ -56,7 +56,7 @@ dataset:
5656
kind: "human36m"
5757

5858
train:
59-
h36m_root: "./data/human36m/processed/_undistorted"
59+
h36m_root: "./data/human36m/processed/"
6060
labels_path: "./data/human36m/extra/human36m-multiview-labels-GTbboxes.npy"
6161
pred_results_path: "./data/pretrained/human36m/human36m_alg_10-04-2019/checkpoints/0060/results/train.pkl"
6262

@@ -69,10 +69,10 @@ dataset:
6969
randomize_n_views: false
7070
min_n_views: null
7171
max_n_views: null
72-
num_workers: 8
72+
num_workers: 5
7373

7474
val:
75-
h36m_root: "./data/human36m/processed/_undistorted"
75+
h36m_root: "./data/human36m/processed/"
7676
labels_path: "./data/human36m/extra/human36m-multiview-labels-GTbboxes.npy"
7777
pred_results_path: "./data/pretrained/human36m/human36m_alg_10-04-2019/checkpoints/0060/results/val.pkl"
7878

‎experiments/human36m/train/human36m_alg.yaml

+3-3
Original file line numberDiff line numberDiff line change
@@ -43,7 +43,7 @@ dataset:
4343
kind: "human36m"
4444

4545
train:
46-
h36m_root: "./data/human36m/processed/_undistorted"
46+
h36m_root: "./data/human36m/processed/"
4747
labels_path: "./data/human36m/extra/human36m-multiview-labels-GTbboxes.npy"
4848
with_damaged_actions: true
4949
undistort_images: true
@@ -57,7 +57,7 @@ dataset:
5757
num_workers: 8
5858

5959
val:
60-
h36m_root: "./data/human36m/processed/_undistorted"
60+
h36m_root: "./data/human36m/processed/"
6161
labels_path: "./data/human36m/extra/human36m-multiview-labels-GTbboxes.npy"
6262
with_damaged_actions: true
6363
undistort_images: true
@@ -70,4 +70,4 @@ dataset:
7070
max_n_views: null
7171
num_workers: 8
7272

73-
retain_every_n_frames_in_test: 30
73+
retain_every_n_frames_in_test: 1

‎experiments/human36m/train/human36m_alg_no_conf.yaml

+3-3
Original file line numberDiff line numberDiff line change
@@ -43,7 +43,7 @@ dataset:
4343
kind: "human36m"
4444

4545
train:
46-
h36m_root: "./data/human36m/processed/_undistorted"
46+
h36m_root: "./data/human36m/processed/"
4747
labels_path: "./data/human36m/extra/human36m-multiview-labels-GTbboxes.npy"
4848
with_damaged_actions: true
4949
undistort_images: true
@@ -57,7 +57,7 @@ dataset:
5757
num_workers: 8
5858

5959
val:
60-
h36m_root: "./data/human36m/processed/_undistorted"
60+
h36m_root: "./data/human36m/processed/"
6161
labels_path: "./data/human36m/extra/human36m-multiview-labels-GTbboxes.npy"
6262
with_damaged_actions: true
6363
undistort_images: true
@@ -70,4 +70,4 @@ dataset:
7070
max_n_views: null
7171
num_workers: 8
7272

73-
retain_every_n_frames_in_test: 30
73+
retain_every_n_frames_in_test: 1

‎experiments/human36m/train/human36m_vol_softmax.yaml

+5-5
Original file line numberDiff line numberDiff line change
@@ -56,7 +56,7 @@ dataset:
5656
kind: "human36m"
5757

5858
train:
59-
h36m_root: "./data/human36m/processed/_undistorted"
59+
h36m_root: "./data/human36m/processed/"
6060
labels_path: "./data/human36m/extra/human36m-multiview-labels-GTbboxes.npy"
6161
pred_results_path: "./data/pretrained/human36m/human36m_alg_10-04-2019/checkpoints/0060/results/train.pkl"
6262

@@ -69,10 +69,10 @@ dataset:
6969
randomize_n_views: false
7070
min_n_views: null
7171
max_n_views: null
72-
num_workers: 8
72+
num_workers: 5
7373

7474
val:
75-
h36m_root: "./data/human36m/processed/_undistorted"
75+
h36m_root: "./data/human36m/processed/"
7676
labels_path: "./data/human36m/extra/human36m-multiview-labels-GTbboxes.npy"
7777
pred_results_path: "./data/pretrained/human36m/human36m_alg_10-04-2019/checkpoints/0060/results/val.pkl"
7878

@@ -85,6 +85,6 @@ dataset:
8585
randomize_n_views: false
8686
min_n_views: null
8787
max_n_views: null
88-
num_workers: 8
88+
num_workers: 10
8989

90-
retain_every_n_frames_in_test: 30
90+
retain_every_n_frames_in_test: 1

‎mvn/datasets/human36m.py

+5-3
Original file line numberDiff line numberDiff line change
@@ -40,7 +40,7 @@ def __init__(self,
4040
labels_path:
4141
Path to 'human36m-multiview-labels.npy' generated by 'generate-labels-npy-multiview.py'
4242
from https://github.sec.samsung.net/RRU8-VIOLET/human36m-preprocessing
43-
retain_every_n_frames_test:
43+
retain_every_n_frames_in_test:
4444
By default, there are 159 181 frames in training set and 26 634 in test (val) set.
4545
With this parameter, test set frames will be evenly skipped frames so that the
4646
test set size is `26634 // retain_every_n_frames_test`.
@@ -106,7 +106,9 @@ def __init__(self,
106106
pred_results = np.load(pred_results_path, allow_pickle=True)
107107
keypoints_3d_pred = pred_results['keypoints_3d'][np.argsort(pred_results['indexes'])]
108108
self.keypoints_3d_pred = keypoints_3d_pred[::retain_every_n_frames_in_test]
109-
assert len(self.keypoints_3d_pred) == len(self)
109+
assert len(self.keypoints_3d_pred) == len(self), \
110+
f"[train={train}, test={test}] {labels_path} has {len(self)} samples, but '{pred_results_path}' " + \
111+
f"has {len(self.keypoints_3d_pred)}. Did you follow all preprocessing instructions carefully?"
110112

111113
def __len__(self):
112114
return len(self.labels['table'])
@@ -217,7 +219,7 @@ def evaluate_by_actions(self, per_pose_error, mask=None):
217219
action_scores[action_name_without_trial] = combined_score
218220

219221
for k, v in action_scores.items():
220-
action_scores[k] = v['total_loss'] / v['frame_count']
222+
action_scores[k] = float('nan') if v['frame_count'] == 0 else (v['total_loss'] / v['frame_count'])
221223

222224
return action_scores
223225

0 commit comments

Comments
 (0)