carlos-alberto-silva/artemis_treescaling

Individual-tree attribute scaling from airborne laser scanning point clouds

Individual-tree detection, segmentation, and attribute regression from airborne laser scanning point clouds

The University of Lethbridge - Department of Geography & Environment - Artemis Lab

Author - Zhouxin Xi ([email protected]), Chris Hopkinson ([email protected]), and Laura Chasmer ([email protected])

Note: The algorithms presented here were primarily developed between 2020 and 2022. While they may not represent the latest advancements in AI for achieving optimal accuracy, given the rapid progress in the field, they might offer practical and thought-provoking approaches for your own study.

Please cite

Training strategy:

Xi, Zhouxin, Chris Hopkinson, and Laura Chasmer. "Filtering stems and branches from terrestrial laser scanning point clouds using deep 3-D fully convolutional networks." Remote Sensing 10.8 (2018): 1215. link.

SegFormer:

Xie, Enze, et al. "SegFormer: Simple and efficient design for semantic segmentation with transformers." Advances in Neural Information Processing Systems 34 (2021): 12077-12090.

SegFormer3D:

Xi, Zhouxin, Laura Chasmer, and Chris Hopkinson. "Delineating and Reconstructing 3D Forest Fuel Components and Volumes with Terrestrial Laser Scanning." Remote Sensing 15.19 (2023): 4778. link.

Cut-pursuit:

Landrieu, Loic, and Guillaume Obozinski. "Cut pursuit: Fast algorithms to learn piecewise constant functions on general weighted graphs." SIAM Journal on Imaging Sciences 10.4 (2017): 1724-1766. link

Folder structure

├── vegcls                       # 1. Project: 3D tree point classification from ALS using deep learning
│   ├── data                     # Customizable ALS dataset for training, validation, and testing: Manually labelled ALS reference point clouds (*.laz/*.las format only now); the pointwise labels are two classes (other:1,tree:2), stored as the extra bytes field of laz file 
│   │   ├── apply                # Customizable ALS dataset for applying the trained model (just a subset from ALS tiles due to the file size limit in GitHub)
│   ├── config                   # Configuration files
│   │   ├── config.json          # Update your data paths and network structure parameters here (you can also choose either SegFormer (default) or ResNet50 by (un-)commenting the model types)
│   │   ├── train_list.txt       # Customizable training file names
│   │   ├── valid_list.txt       # Customizable validating file names (validating step occurs every N iterations during the training process)
│   │   ├── test_list.txt        # Customizable testing file names (testing step independent from the training process)
│   │   ├── app_list.txt         # Customizable applicating file names (apply the trained model for prediction over larger areas)
│   ├── code                     
│   │   ├── vegclsMain.py        # Main entry; you can customize the mode (train/test/apply)
│   │   ├── vox3DSegFormer.py    # SegFormer network layer definition
│   │   ├── vox3DResNetModel.py  # ResNet network layer definition
│   ├── logs                     # This is a generated folder containing all logs and outcomes from training to application steps
|
├── itcseg                       # 2. Project: 3D individual-tree instance segmentation from ALS
│   ├── data                     # Customizable ALS dataset for training, validation, and testing
│   │   ├── apply                # Customizable ALS dataset for applying the trained model
│   ├── config                   # Configuration files
│   │   ├── config.json          # Update your data paths and network structure parameters here
│   │   ├── train_list.txt       # Customizable training file names
│   │   ├── valid_list.txt       # Customizable validating file names
│   │   ├── test_list.txt        # Customizable testing file names
│   │   ├── app_list.txt         # Customizable applicating file names
│   ├── code                     
│   │   ├── itcsegMain.py        # Main entry; you can customize the mode (train/test/apply)
│   │   ├── vox3DSegFormer.py    # SegFormer network layer definition
│   │   ├── itcsegPost.m         # Matlab post-processing that clusters remaining points to the tree center regions detected from the itcsegMain.py
│   │   ├── cutPursuit.m         # Cut-pursuit clustering algorithm 
│   │   ├── \*.mexw64            # Compiled binary files for cut-pursuit and laz file I/O functions	
│   ├── logs                     # This is a generated folder containing all logs and outcomes from training to application steps
|
├── itcreg                       # 3. Project: Individual-tree level regression using simple deep-learning modules
│   ├── data                     # Customizable ALS dataset for training, validation, and testing
│   │   ├── apply                # Customizable ALS dataset for applying the trained model
│   ├── config                   # Configuration files
│   │   ├── config.json          # Update your data paths and network structure parameters here
│   │   ├── train_list.txt       # Customizable training file names
│   │   ├── valid_list.txt       # Customizable validating file names
│   │   ├── test_list.txt        # Customizable testing file names
│   │   ├── app_list.txt         # Customizable applicating file names
│   ├── code                     
│   │   ├── itcregMain.py        # Main entry; you can customize the mode (train/test/apply)
│   ├── logs                     # This is a generated folder containing all logs and outcomes from training to application steps
|
├── LICENSE
└── README.md

1. Requirement

Requires a CUDA-compatible GPU with more than 8GB VRAM. Tested on an RTX3090.

For GPUs with less VRAM, consider lowering the voxel numbers per block in the configuration file to accommodate.

1.1 Install python (3.9+)

1.2 Install PyTorch following the official guide from: 

https://pytorch.org/

For example, using pip:

python -m pip install torch torchvision torchaudio --index-url https://download.pytorch.org/whl/cu118

Currently, I use the most recent PyTorch 2.0+ (Windows) without any problem; I could also run the program using PyTorch 1.12+ years ago. I haven't tested the programs in other environments yet. If you find any compatibility issues, please let me know.

1.3 Install the additional Python libraries:

python -m pip install numpy numpy-indexed numpy_groupies commentjson laspy[lazrs] timm tqdm gpytorch

*I find it frustrating when programmers overuse dependencies for mere convenience, rather than for functional efficiency. It's also bothersome when they focus more on showcasing their coding skills through excessive code refactorization, but are blind to practical demands and real-world challenges.

2. Classification of above-ground tree points from ALS using deep learning

2.1 Customize the data folder and config.json files on your own

2.2 Run Python

You can use Python programming IDE or directly use the command line

• training:

python yourfolder/vegcls/code/vegclsMain.py --mode train

• testing:

python yourfolder/vegcls/code/vegclsMain.py --mode test

• application (only after the model is trained):

python yourfolder/vegcls/code/vegclsMain.py --mode apply

2.3 Results will be exported to the "logs" folder where:

"logs/train" includes a log file and the best-trained model

"logs/app" includes a log file and the prediction of per-point classes (2: vegetation/tree, 1:ground and other points). The class will be saved into the output laz file as the extra-byte scalar field "VegCls".

You can apply the trained model to a much larger area, e.g., the landscape level as below:

Benchmarking

3. Segmentation of individual tree points from ALS

There are two main steps involved: classification of tree center regions, and segmentation of the tree boundaries.

The first step uses the same SegFormer model, identical to the previous classification section

3.1 Customize the data folder and config.json files on your own

The reference dataset (*.laz or *.las format only) should follow those in the data folder, with a specific scalar field "itc_ref" identifying the tree ID for each point, and the scalar field "VegCls" from the previous classification module (2: vegetation/tree, 1:ground and other points).

The application dataset should have the scalar field "VegCls" from the previous classification module (2: vegetation/tree, 1:ground and other points).

3.2 Run python

You can use a Python programming IDE or directly use the command line

• prepare:

python yourfolder/itcseg/code/itcsegMain.py --mode prepare

• training:

python yourfolder/itcseg/code/itcsegMain.py --mode train

• testing:

python yourfolder/itcseg/code/itcsegMain.py --mode test

• application (only after the model is trained):

python yourfolder/itcseg/code/itcsegMain.py --mode apply

3.3 Results will be exported where:

"data" includes the result files ("*_cfd.laz") from the "prepare" step. Smooth confidence for each point of the input point cloud will be created based on the reference dataset, which will be used to guide the training process for tree center detection.

"logs/train" includes a log file and the best-trained model

"logs/app" includes a log file and the prediction of per-point classes (2:tree center, 1:others). The class will be saved into the output laz file as the extra-byte scalar field "ConfPred".

3.4 Run Matlab code (itcsegPost.m):

Please customize the file path of input and output. The input will be the result laz files from "logs/app", and the default output folder is also "logs/app". The output file ends with "_segs.laz"

The cut-pursuit algorithm and laz I/O functions have been compiled into mexw64 binary files for Windows. If you're using a different operating system, you may need to recompile them.

You can apply the trained model to a much larger area, e.g., the landscape level as below:

Benchmarking

4. Regression of individual-tree attributes from TLS to ALS

4.1 Customize the data folder and config.json files on your own

The reference dataset (*.laz or *.las format only) should follow those in the data folder, with a specific scalar field "itc_ref" identifying the ID for each tree, and "AttrRef" as the reference attribute (Tree diameter-at-breast height as our example).

The application dataset should have the scalar field "segs" from the previous tree segmentation module.

4.2 Run python

You can use a Python programming IDE or directly use the command line

• training:

python yourfolder/itcreg/code/itcregMain.py --mode train

• testing:

python yourfolder/itcreg/code/itcregMain.py --mode test

• application (only after the model is trained):

python yourfolder/itcreg/code/itcregMain.py --mode apply

4.3 Results will be exported where:

"logs/train" includes a log file and the best-trained model

"logs/app" includes a log file and laz files with a post-regression attribute (as the extra-byte scalar field "AttrPred").

You can apply the trained model to a much larger area, e.g., the 3D tree-wise biomass map at the landscape level, converted based on the tree DBH and height as below:

*There's a noticeable gap in the application of cutting-edge AI techniques, which often emphasizes creating impressive 'toy' projects and engaging in superficial competitions for accuracy ranking. This focus tends to prioritize visual appeal and immediate dopamine gratification rather than addressing more substantial challenges. These include the intricate tasks of managing complex environmental data, gaining deeper insights into the physical and spiritual realms, and tackling significant societal issues.