Semantic annotations

[matterport]

Semantic annotations

[IFICL]

Hi, Do you get any chance to make the semantic annotation work? I download the annotation from a certain branch and load semantic.glb. However, I was told that there is no semantic annotation.

[XHRlyb]

@IFICL I have exactly the same question as yours. Did you get semantic annotation finally?

[KarelZhang]

I got the same problem, is there anyone can help us!

[kojirotakeyama]

I want to know the format of following data as well.
313,23694C,"refrigerator cabinet",9

"23694C" looks like the combination of hexadecimals, but cannot figure out what it means.
Also, the last number "9" is incomprehensible.

[KarelZhang]

hi, could you please tell hwo to make the semantic annotation work? Thanks!

[Michael-Equi]

Hey I have this same question. I am loading the .semantic.glb file but still getting blank semantic images and the error The active scene does not contain semantic annotations : activeSemanticSceneID_ = 0 .

I am using habiat_sim v0.2.3. Is there anything else I need to be configuring in the settings for semantic image segmentation to work? Currently I am just setting the parameters as follows

settings = default_sim_settings.copy()
settings["seed"] = 5
settings["save_png"] = True
settings["color_sensor"] = True
settings["semantic_sensor"] = True
settings["depth_sensor"] = True
settings["sensor_height"] = 0.5 
settings["width"] = 224
settings["height"] = 224
settings["scene"] = "/raid/michaelequi/hm3d_semantics/scene_datasets/hm3d/train/00744-1S7LAXRdDqK/1S7LAXRdDqK.semantic.glb"

[aclegg3]

Hey folks, please see the Habitat-Matterport 3D Research Dataset (HM3D) section of Habitat-sim's DATASETS readme for the details and examples.

TL;DR: You need to reference the SceneDataset config files instead of trying to directly load the semantic assets.
For example:

There are different ways to achieve this.

1. You can directly set these configuration values in your initialization:

backend_cfg = habitat_sim.SimulatorConfiguration()
backend_cfg.scene_id = "<PATH TO HM3D>/minival/00800-TEEsavR23oF/TEEsavR23oF.basis.glb"
backend_cfg.scene_dataset_config_file = "<PATH TO HM3D>/hm3d_annotated_basis.scene_dataset_config.json"

2. You can use the settings utility as @Michael-Equi is doing. However, you need to set the scene_dataset_config_file parameter to the scene_dataset.json file path and then the scene parameter should point to the stage_config.json file.
3. In Habitat-lab you can set the YAML config parameters. E.g. SIMULATOR.SCENE_DATASET = "<PATH TO HM3D>/hm3d_annotated_basis.scene_dataset_config.json"

Slightly more details: (see concepts and terminology)
Each Scene is composed of a static Stage asset for RGBD rendering and a second asset with the same geometry, but semantic annotation texture instead. Additionally scenes can contain dynamic objects, lighting, navmeshes, etc... All of these components are joined via the SceneDataset config files. When you directly attempt the load a single asset (e.g. .semantic.glb) the simulator is not able to put all of these components together.

[Michael-Equi]

Hi @aclegg3 and thanks for the quick reply but adding the json file using the line settings["scene_dataset_config_file"] = f"{SCENE_DIR}/scene_datasets/hm3d/hm3d_annotated_basis.scene_dataset_config.json" results in the following error

Traceback (most recent call last):
  File "collect_semantic_images.py", line 99, in <module>
    sim, agent = run_sim(scene_path, settings)
  File "collect_semantic_images.py", line 30, in run_sim
    _sim = habitat_sim.Simulator(_cfg) 
  File "<attrs generated init habitat_sim.simulator.Simulator>", line 11, in __init__
  File "/nfs/kun2/users/michaelequi/miniconda3/envs/habitat/lib/python3.7/site-packages/habitat_sim-0.2.3-py3.7-linux-x86_64.egg/habitat_sim/simulator.py", line 119, in __attrs_post_init__
    self.__set_from_config(self.config)
  File "/nfs/kun2/users/michaelequi/miniconda3/envs/habitat/lib/python3.7/site-packages/habitat_sim-0.2.3-py3.7-linux-x86_64.egg/habitat_sim/simulator.py", line 270, in __set_from_config
    self._config_backend(config)
  File "/nfs/kun2/users/michaelequi/miniconda3/envs/habitat/lib/python3.7/site-packages/habitat_sim-0.2.3-py3.7-linux-x86_64.egg/habitat_sim/simulator.py", line 201, in _config_backend
    super().__init__(config.sim_cfg, config.metadata_mediator)
RuntimeError: std::bad_cast

[KarelZhang]

hi, I solved this problem by add the following code to /habitat-lab/habitat/core/env.py
self._config.SIMULATOR.SCENE_DATASET = "<PATH TO HM3D>/hm3d_annotated_basis.scene_dataset_config.json"

hope it works.