Merge branch 'convertors' of https://github.com/open-mmlab/mmhuman3d …

…into convertors

docs/convert_datasets.md

@@ -0,0 +1,195 @@
+# Convert Datasets to HumanData and Use in Training
+**This page describes how to use the various HumanData flies** as well as (only if you are interested) how to process various datasets to current HumanData.
+
+## Introduction & Settings
+1. How can I read image from HumanData image path?
+
+   In HumanData file, image path denotes the relative path within the dataset. In this example:
+
+    ```
+    your_path_to_dataset = '/mnt/d'
+    dataset_name = 'egobody'
+    image_path = HumanData['image_path'][100]  # for example
+    image_real_path = os.path.join(your_path_to_dataset, dataset_name, image_path)
+    image = cv2.imread(image_real_path)
+    ```
+
+2. For using HumanData only (not including converting), you just need to do several changes to the file structues on the originally downloaded datasets. The converting process and file included will be marked **"Converter"**.
+3. Ideal size of HumanData file is less than 2GB, given loading it uses 10x memory, therefore some huge datasets are split in several files.
+4. All the middle-stage pre-process files are **not required** when using HumanData annotations.
+5. (For convert only) Install "convertors" branch and set the working directory as below.
+
+     ```
+     git clone -b convertors https://github.com/open-mmlab/mmhuman3d.git
+     cd /Your_path_to_mmhuman3d/mmhuman3d  # e.g. /mnt/d/zoehuman/mmhuman3d
+     ```
+6. (For convert only) The usage of convert_datasets.py (Example)
+     ```
+     python tools/convert_datasets.py \
+      --datasets <dataset name/perfix> \ 
+      --root_path <data root dir> \
+      --output_path <output dir> \
+      --modes train
+     # dataset folder should be os.path.join(root_path, datasets)
+     # output path is the folder which sotres output HumanData
+     ``` 
+
+## Overview - Current Supported Datasets
+- GTA-Human++
+- H36M (Neural Annot)
+- MPII (Neural Annot)
+- MSCOCO (Neural Annot)
+- PW3D (Neural Annot)
+
+## Subsection 1 - Neural Annot Datasets
+Overall: Download from [Nerual Annot Homepage](https://github.com/mks0601/NeuralAnnot_RELEASE/blob/main/README.md)
+
+<details>
+<summary>H36M</summary>
+
+**Step 1 - Only Step for using HumanData** 
+
+Download the original data and SMPLX annotation and rearrange the file structure as below:
+
+```
+D:\datasets\h36m\
+│
+├── annotations\
+│   ├──Human36M_subject11_SMPLX_NeuralAnnot.json
+│   ├──Human36M_subject11_camera.json
+│   ├──Human36M_subject11_data.json
+│   ├──Human36M_subject11_joint_3d.json
+│
+├── images\
+│   ├── s_01_act_02_subact_01_ca_01\
+│   ├── s_01_act_02_subact_01_ca_02\
+```
+
+**Step 2 (Converter) - Convert Dataset**
+```
+python tools/convert_datasets.py \
+ --datasets h36m\
+ --root_path /mnt/d/datasets \
+ --output_path /mnt/d/datasets/h36m/output \
+ --modes train
+```
+</details>
+
+<details>
+<summary>MPII</summary>
+
+**Step 1 - Only Step for using HumanData** 
+
+Download and rearrange the file structure as below:
+
+```
+E:\mpii\
+│
+├── annotations\
+│   ├──MPII_train_SMPLX_NeuralAnnot.json
+│   ├──test.json
+│   ├──train.json
+│   └──train_reformat.json  # Not required in HumanData
+│
+├── images\
+```
+**Step 2 (Converter) - Preprocess coco annotations**
+
+This process converts the coco annotation json to faciliate sorting ids.
+```
+python tools/preprocess/neural_annot.py --dataset_path /YOUR_PATH/mpii
+```
+
+**Step 3 (Converter) - Convert Dataset**
+```
+python tools/convert_datasets.py \
+ --datasets mpii \
+ --root_path /mnt/d/datasets \
+ --output_path /mnt/d/datasets/mpii/output \
+ --modes train
+```
+</details>
+
+<details>
+<summary>MSCOCO</summary>
+
+**Step 1 - Only Step for using HumanData** 
+
+Download and rearrange the file structure as below:
+
+```
+D:\datasets\mscoco\
+│
+├── annotations\
+│   ├──MSCOCO_train_SMPLX.json
+│   ├──MSCOCO_train_SMPLX_all_NeuralAnnot.json
+│   ├──coco_wholebody_train_v1.0.json
+│   ├──coco_wholebody_train_v1.0_reformat.json  # Not required in HumanData, generated in Step 2 and used in step 3
+│   └──coco_wholebody_val_v1.0.json
+│
+├── images\
+│   ├── train2017\
+│   └── val2017\
+```
+**Step 2 (Converter) - Preprocess coco annotations**
+
+This process converts the coco annotation json to faciliate sorting ids.
+```
+python tools/preprocess/neural_annot.py --dataset_path /YOUR_PATH/mscoco
+```
+
+**Step 3 (Converter) - Convert Dataset**
+```
+python tools/convert_datasets.py \
+ --datasets mscoco \
+ --root_path /mnt/d/datasets \
+ --output_path /mnt/d/datasets/mscoco/output \
+ --modes train
+```
+</details>
+
+<details>
+<summary>PW3D</summary>
+
+**Step 1 - Only Step for using HumanData** 
+
+Download and rearrange the file structure as below:
+
+Note: Rename "*validation*" as "*val*"
+```
+D:\datasets\pw3d\
+│
+├── imageFiles\
+│   ├── courtyard_arguing_00\
+│   ├── courtyard_backpack_00\
+│
+├──3DPW_test.json
+├──3DPW_test_SMPLX_NeuralAnnot.json
+├──3DPW_train.json
+├──3DPW_train_SMPLX_NeuralAnnot.json
+├──3DPW_val.json
+└──3DPW_val_SMPLX_NeuralAnnot.json
+```
+**Step 2 (Converter) - Preprocess coco annotations**
+
+This process converts the coco annotation json to faciliate sorting ids.
+```
+python tools/preprocess/neural_annot.py --dataset_path /YOUR_PATH/pw3d
+```
+
+**Step 3 (Converter) - Convert Dataset**
+```
+python tools/convert_datasets.py \
+ --datasets pw3d \
+ --root_path /mnt/d/datasets \
+ --output_path /mnt/d/datasets/pw3d/output \
+ --modes train test val
+```
+</details>
+
+## Subsection 2 - Synthetic Datasets
+
+<details>
+<summary>GTA-Human++</summary>
+
+</details>

mmhuman3d/data/data_converters/__init__.py

@@ -19,8 +19,8 @@
 from .freihand import FreihandConverter
 from .gta_human import GTAHumanConverter
 from .gta_human2 import GTAHuman2Converter
-from .h36m_neural_annot import H36mNeuralConverter
 from .h36m_hybrik import H36mHybrIKConverter
+from .h36m_neural_annot import H36mNeuralConverter
 from .h36m_smplx import H36mSMPLXConverter
 from .hanco import HancoConverter
 from .hsc4d import Hsc4dConverter
@@ -35,10 +35,11 @@
 from .mpi_inf_3dhp import MpiInf3dhpConverter
 from .mpi_inf_3dhp_hybrik import MpiInf3dhpHybrIKConverter
 from .mpii_neural_annot import MpiiNeuralConverter
+from .mscoco_neural_annot import MscocoNeuralConverter
 from .penn_action import PennActionConverter
 from .posetrack import PosetrackConverter
-from .pw3d import Pw3dConverter
 from .pw3d_hybrik import Pw3dHybrIKConverter
+from .pw3d_neural_annot import Pw3dNeuralConverter
 from .renbody import RenbodyConverter
 from .sgnify import SgnifyConverter
 from .shapy import ShapyConverter
@@ -67,7 +68,7 @@
     'MpiInf3dhpConverter',
     'PennActionConverter',
     'PosetrackConverter',
-    'Pw3dConverter',
+    'Pw3dNeuralConverter',
     'Up3dConverter',
     'CrowdposeConverter',
     'EftConverter',
@@ -109,4 +110,5 @@
     'BlurhandConverter',
     'Hsc4dConverter',
     'MotionXConverter',
+    'MscocoNeuralConverter',
 ]

mmhuman3d/data/data_converters/h36m_neural_annot.py

@@ -41,7 +41,7 @@ class H36mNeuralConverter(BaseModeConverter):
         folder. Default: False.
         mosh_dir (str, optional): Path to directory containing mosh files.
     """
-    ACCEPTED_MODES = ['valid', 'train']
+    ACCEPTED_MODES = ['val', 'train']
 
     def __init__(self, modes: List = []) -> None:
 
@@ -108,14 +108,14 @@ def convert_by_mode(self,
         if mode == 'train':
             user_list = [1, 5, 6, 7, 8]
         elif mode == 'val':
-            user_list = [9, 11]
+            user_list = [11]
         subject_ids = list(set(subject_ids) & set(user_list))
 
         # calculate size
         seqs_len = 0
         for key in subject_seq_dict:
             seqs_len += len(subject_seq_dict[key])
-        size_i = min(size_i, seqs_len)
+
 
         # parse seqs
         for s, sid in enumerate(subject_ids):
@@ -125,9 +125,9 @@ def convert_by_mode(self,
 
             # init seed and size
             seed, size = '230811', '999'
-            size_i = min(int(size), len(targeted_seqs))
+            size_i = min(int(size), seqs_len)
             random.seed(int(seed))
-            targeted_seqs = targeted_seqs[:size_i]
+            # targeted_seqs = targeted_seqs[:size_i]
             # random.shuffle(npzs)
 
             # initialize output for human_data
@@ -167,7 +167,7 @@ def convert_by_mode(self,
             with open(os.path.join(anno_base_path, cam_param)) as f:
                 cam_params = json.load(f)
 
-            # load data annotations
+            # load data annotations 
             data_an = f'{anno_base_name}_data.json'
             with open(os.path.join(anno_base_path, data_an)) as f:
                 data_annos = json.load(f)
@@ -202,7 +202,10 @@ def convert_by_mode(self,
                 data_anno_seq = [data_annos['annotations'][idx] | data_annos['images'][idx]
                                  for idx, finfo in enumerate(data_annos['images']) if 
                                  os.path.basename(finfo['file_name'])[:-11] == seqn]
-                assert len(data_anno_seq) == len(frames)
+                # pdb.set_trace()
+                if len(data_anno_seq) != len(frames):
+                    print(f'Warning: {seqn} has different length of frames and annotations')
+                    continue
 
                 # get joints 3d annotations
                 j3d_anno_seq = j3d_annos[action_id][subaction_id]