remotesensinginfo/rsgislib

Remote Sensing and GIS Software Library; python module tools for processing spatial data.

RSGISLib

The Remote Sensing and GIS software library (RSGISLib) is a collection of tools for processing remote sensing and GIS datasets. For more details see the project website: http://rsgislib.org/

Support the Project

We provide updated and support for the project for free as a open source project but if you'd like to support us you can buy a coffee - thank you! :)

Installing

Instructions on installing RSGISLib:

conda install -c conda-forge rsgislib

If you want to install a system of packages for undertaking remote sensing and GIS data analysis in python then the following packages would be a recommended list:

conda install -c conda-forge rsgislib gdal h5py parallel scikit-learn scikit-image scikit-optimize imbalanced-learn matplotlib pandas geopandas statsmodels scipy rasterio shapely networkx sqlalchemy pycurl xgboost lightgbm tpot seaborn numba pip sphinx elevation rtree tqdm jinja2 keras keras-preprocessing pytables bokeh pygal jupyterlab psutil pysal libpysal esda pyyaml netcdf4 xarray rasterstats fiona plotly python-kaleido psycopg2 ipywidgets tuiview

Documentation and Support

The documentation for RSGISLib is available from the project website: http://rsgislib.org/

Support for RSGISLib is provided through a mailing list: https://groups.google.com/forum/#!forum/rsgislib-support

You need to be a member of the group to post. Please check through existing posts to see if there is already an answer to your question before emailing. To help us answer your question provide as much information as possible (RSGISLib version and how you installed it, OS, things you have already tried to solve the problem etc.,).

We occasionally post tutorials on RSGISLib and other useful bits and pieces to our website remotesensing.info.

Citing

If you use RSGISLib as part of published work please cite the following publication:

Peter Bunting, Daniel Clewley, Richard M. Lucas and Sam Gillingham. 2014. The Remote Sensing and GIS Software Library (RSGISLib), Computers & Geosciences. Volume 62, Pages 216-226 http://dx.doi.org/10.1016/j.cageo.2013.08.007. pre-print

    @article{Bunting_etal_2014,
    	Author = {Bunting, Peter and Clewley, Daniel and Lucas, Richard M and Gillingham, Sam},
    	Title = {{The Remote Sensing and GIS Software Library (RSGISLib)}},
    	Journal = {Computers and Geosciences},
    	Pages = {216--226},
    	Volume = {62},
    	Year = {2014}}

Building from Source

RSGISLib uses cmake for the build system, so you will need that and the dependencies installed on your system.

# Create a build directory in the release directory
mkdir build_dir
cd build_dir

# if all the dependencies are within the same path (e.g., installed with conda)
# then use a variable to specify the path.
export DEPEND_ENV_PATH=/home/user/miniconda/envs/rsgislib_v5_dev

# Set the install location and path to dependencies
cmake -D CMAKE_INSTALL_PREFIX=/usr/local \
-D BOOST_INCLUDE_DIR=$DEPEND_ENV_PATH/include \
-D BOOST_LIB_PATH=$DEPEND_ENV_PATH/lib \
-D GDAL_INCLUDE_DIR=$DEPEND_ENV_PATH/include \
-D GDAL_LIB_PATH=$DEPEND_ENV_PATH/lib \
-D HDF5_INCLUDE_DIR=$DEPEND_ENV_PATH/include \
-D HDF5_LIB_PATH=$DEPEND_ENV_PATH/lib \
-D GSL_INCLUDE_DIR=$DEPEND_ENV_PATH/include \
-D GSL_LIB_PATH=$DEPEND_ENV_PATH/lib \
-D MUPARSER_INCLUDE_DIR=$DEPEND_ENV_PATH/include \
-D MUPARSER_LIB_PATH=$DEPEND_ENV_PATH/lib \
-D KEA_INCLUDE_DIR=$DEPEND_ENV_PATH/include \
-D KEA_LIB_PATH=$DEPEND_ENV_PATH/lib \
-D Python_EXECUTABLE=$DEPEND_ENV_PATH/bin/python \
-D CMAKE_VERBOSE_MAKEFILE=ON \
-D CMAKE_BUILD_TYPE=Debug \
-D CMAKE_SKIP_RPATH=ON \
..
  
# build rsgislib -j N is the number of cores to use for build 
make -j 2
  
# Install the build into the install prefix.
make install

rsgislib's Issues

Segmentation module seems to be broken .

I have tried importing the segmentation module and got the following error

import rsgislib
from rsgislib.segmentation import shepherdseg

`---------------------------------------------------------------------------
ImportError Traceback (most recent call last)
Input In [2], in <cell line: 15>()
13 # Import the RSGISLib modules for the notebook
14 import rsgislib
---> 15 from rsgislib.segmentation import shepherdseg
16 from rsgislib.segmentation.skimgseg import perform_slic_segmentation
17 from rsgislib.segmentation import tiledsegsingle

File D:\anaconda3\envs\geo\lib\site-packages\rsgislib\segmentation_init_.py:33, in
2 """
3 The segmentation module contains the segmentation functionality for RSGISLib.
4
(...)
29
30 """
32 # import the C++ extension into this level
---> 33 from ._segmentation import *

ImportError: DLL load failed while importing _segmentation: The specified module could not be found.
`

Confusion matrix for segmentation and classification example

I would like to know if there is a possibility to add the confusion matrix in the segmentation code example (03_intro_image_classification/02_segment_ml_classification.ipynb) with the 'rsgislib.classification.classaccuracymetrics.calc_acc_metrics_vecsamples' method. How would the implementation be to show the confusion matrix?

morphological closing requirement

@petebunting Morphological closing provides an option to use a morph file. If set to False, a file still needs to be provided before the code will run. For example, this will not work:

output_img = in_img.replace('.tif','_beta0_dB_Wmask_clump_close.kea')
tmp_img = in_img.replace('.tif','_beta0_dB_Wmask_clump_close_tmp.kea')
morph_op_file = None
use_op_file = False
op_size = 5
gdalformat = "KEA"
datatype = rsgislib.TYPE_16UINT
niters = 1

while this will work:

output_img = in_img.replace('.tif','_beta0_dB_Wmask_clump_close.kea')
tmp_img = in_img.replace('.tif','_beta0_dB_Wmask_clump_close_tmp.kea')
morph_op_file = '/morph_closing.txt'
use_op_file = False
op_size = 5
gdalformat = "KEA"
datatype = rsgislib.TYPE_16UINT
niters = 1

probably just a catch needed on the use_op_file arg?

Error messages when running popImageStates on conda Windows 10

Hi all,

These error messages pop up when I run the command line below. The task still completes successfully, so it is probably a debugging leftover. Sorry for the encoding issues with the progress bar.

> rsgislib.imageutils.popImageStats(outfile, True, 0, True)

'test' is not recognized as an internal or external command, operable program or batch file.
'test' is not recognized as an internal or external command, operable program or batch file.
 ΓûêΓûêΓûêΓûêΓûêΓûêΓûêΓûêΓûêΓûêΓûêΓûêΓûêΓûêΓûêΓûêΓûêΓûêΓûêΓûêΓûêΓûêΓûêΓûêΓûêΓûêΓûêΓûêΓûêΓûêΓûêΓûêΓûêΓêΓûêΓûêΓûêΓûêΓûêΓûêΓûÅ 100.0% [44528/44528 | 1.4 kHz | 33s<0s] 

'test' is not recognized as an internal or external command, operable program or batch file.
'test' is not recognized as an internal or external command, operable program or batch file.
 ΓûêΓûêΓûêΓûêΓûêΓûêΓûêΓûêΓûêΓûêΓûêΓûêΓûêΓûêΓûêΓûêΓûêΓûêΓûêΓûêΓûêΓûêΓûêΓûêΓûêΓûêΓûêΓûêΓûêΓûêΓûêΓûêΓûêΓêΓûêΓûêΓûêΓûêΓûêΓûêΓûÅ 100.0% [44528/44528 | 869.6 Hz | 51s<0s]

Calculating Image Pyramids.

'test' is not recognized as an internal or external command, operable program or batch file.
'test' is not recognized as an internal or external command, operable program or batch file.
 ΓûêΓûêΓûêΓûêΓûêΓûêΓûêΓûêΓûêΓûêΓûêΓûêΓûêΓûêΓûêΓûêΓûêΓûêΓûêΓûêΓûêΓûêΓûêΓûêΓûêΓûêΓûêΓûêΓûêΓûêΓûêΓûêΓûêΓûêΓûêΓûêΓûêΓûêΓûêΓûêΓûÅ 100.0% [ 100/ 100 | 8.4 Hz | 12s<0s]
 ΓûêΓûêΓûêΓûêΓûêΓûêΓûêΓûêΓûêΓûêΓûêΓûêΓûêΓûêΓûêΓûêΓûêΓûêΓûêΓûêΓûêΓûêΓûêΓûêΓûêΓûêΓûêΓûêΓûêΓûêΓûêΓûêΓûêΓûêΓûêΓûêΓûêΓûêΓûêΓûêΓûÅ 100.0% [ 100/ 100 | 8.4 Hz | 12s<0s]

rsgislib.imageutils.stretch_img_with_stats

Bug on MacOS - freezes for some reason which needs debugging.

Question for segment a particularly large remote sensing image

I would like to ask, if I want to segment a particularly large remote sensing image, how do I use rsgislib to segment the remote sensing image and there is no obvious boundary feature after merging

test_tools_ftptools failed with ftplib.error_proto

Hi,
When I tried to test the package with pytest, the test failed when calling the ftptools.upload_ftp_file function. Below is the error:

rsgislib/tools/ftptools.py:466: in upload_ftp_file
    ftp_conn.login(user=ftp_user, passwd=ftp_pass)
/usr/lib/python3.10/ftplib.py:414: in login
    resp = self.sendcmd('PASS ' + passwd)
/usr/lib/python3.10/ftplib.py:281: in sendcmd
    return self.getresp()
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

self = <ftplib.FTP object at 0x7f4f4c56dab0>

    def getresp(self):
        resp = self.getmultiline()
        if self.debugging:
            print('*resp*', self.sanitize(resp))
        self.lastresp = resp[:3]
        c = resp[:1]
        if c in {'1', '2', '3'}:
            return resp
        if c == '4':
            raise error_temp(resp)
        if c == '5':
            raise error_perm(resp)
>       raise error_proto(resp)
E       ftplib.error_proto: /usr/sbin/vsftpd: Relink `/lib64/libnss_myhostname.so.2' with `/lib64/librt.so.1' for IFUNC symbol `clock_gettime'

/usr/lib/python3.10/ftplib.py:255: error_proto

Below is the full log of pytest:

pytest log

==================================================================================================================================== test session starts ====================================================================================================================================
platform linux -- Python 3.10.4, pytest-7.1.1, pluggy-1.0.0
rootdir: /home/o0331dobe/Documents/AUR_packages/python-rsgislib/src/rsgislib-5.0.7/build-test
plugins: cov-2.12.1
collected 884 items                                                                                                                                                                                                                                                                         

tests/test_changedetect_pxloutlierchng.py s...                                                                                                                                                                                                                                        [  0%]
tests/test_classification.py ................                                                                                                                                                                                                                                         [  2%]
tests/test_classification_classaccuracymetrics.py ........                                                                                                                                                                                                                            [  3%]
tests/test_classification_classimblearn.py sssssssssssssss                                                                                                                                                                                                                            [  4%]
tests/test_classification_classlightgbm.py ssssss                                                                                                                                                                                                                                     [  5%]
tests/test_classification_classratutils.py ..                                                                                                                                                                                                                                         [  5%]
tests/test_classification_classsklearn.py ....                                                                                                                                                                                                                                        [  6%]
tests/test_classification_classxgboost.py sssssssss                                                                                                                                                                                                                                   [  7%]
tests/test_classification_clustersklearn.py ...                                                                                                                                                                                                                                       [  7%]
tests/test_elevation.py .............                                                                                                                                                                                                                                                 [  9%]
tests/test_image_morphology.py .............                                                                                                                                                                                                                                          [ 10%]
tests/test_imagecalc.py ..........................................................                                                                                                                                                                                                    [ 17%]
tests/test_imagecalc_calcindices.py .............                                                                                                                                                                                                                                     [ 18%]
tests/test_imagecalc_leastcostpath.py s                                                                                                                                                                                                                                               [ 18%]
tests/test_imagecalc_specunmixing.py ............ssssssss.....                                                                                                                                                                                                                        [ 21%]
tests/test_imagefilter.py .............................................................                                                                                                                                                                                               [ 28%]
tests/test_imageregistration.py ......                                                                                                                                                                                                                                                [ 29%]
tests/test_imageutils.py .....................................................................................................................................                                                                                                                        [ 44%]
tests/test_imageutils_imagelut.py ........                                                                                                                                                                                                                                            [ 45%]
tests/test_imageutils_stdimgblockiter.py ..                                                                                                                                                                                                                                           [ 45%]
tests/test_rastergis.py ........ss.................                                                                                                                                                                                                                                   [ 48%]
tests/test_regression.py .                                                                                                                                                                                                                                                            [ 48%]
tests/test_regression_regresssklearn.py .....                                                                                                                                                                                                                                         [ 48%]
tests/test_rsgislib_cli.py ........ssss.....................................s....                                                                                                                                                                                                     [ 55%]
tests/test_segentatation.py ...                                                                                                                                                                                                                                                       [ 55%]
tests/test_segentatation_shepherdseg.py ....                                                                                                                                                                                                                                          [ 55%]
tests/test_segentatation_skimgseg.py ssss                                                                                                                                                                                                                                             [ 56%]
tests/test_segentatation_tiledclump.py ....                                                                                                                                                                                                                                           [ 56%]
tests/test_timeseries_modelfitting.py ..                                                                                                                                                                                                                                              [ 57%]
tests/test_tools_checkdatasets.py .....................                                                                                                                                                                                                                               [ 59%]
tests/test_tools_filetools.py ..................................ss........                                                                                                                                                                                                            [ 64%]
tests/test_tools_ftptools.py FssFFss                                                                                                                                                                                                                                                  [ 65%]
tests/test_tools_geometrytools.py .......                                                                                                                                                                                                                                             [ 65%]
tests/test_tools_plotting.py ..............                                                                                                                                                                                                                                           [ 67%]
tests/test_tools_projection.py .........                                                                                                                                                                                                                                              [ 68%]
tests/test_tools_utils.py ..............................................................                                                                                                                                                                                              [ 75%]
tests/test_tools_visualisation.py s......ss.sss..ssssssss.                                                                                                                                                                                                                            [ 78%]
tests/test_vectorattrs.py .........................                                                                                                                                                                                                                                   [ 81%]
tests/test_vectorgeoms.py ..................s.......................                                                                                                                                                                                                                  [ 85%]
tests/test_vectorutils.py .....................................................................                                                                                                                                                                                       [ 93%]
tests/test_vectorutils_createrasters.py ......                                                                                                                                                                                                                                        [ 94%]
tests/test_vectorutils_createvectors.py .............                                                                                                                                                                                                                                 [ 95%]
tests/test_zonalstats.py .....................................                                                                                                                                                                                                                        [100%]

========================================================================================================================================= FAILURES ==========================================================================================================================================
___________________________________________________________________________________________________________________________________ test_upload_ftp_file ____________________________________________________________________________________________________________________________________

    def test_upload_ftp_file():
        import rsgislib.tools.ftptools
    
        test_file = os.path.join(TOOLS_UTILS_DATA_DIR, "basic_str.txt")
>       rsgislib.tools.ftptools.upload_ftp_file(
            test_file,
            ftp_url=ftp_url_ref,
            remote_path="/rsgislib_test/test",
            ftp_timeout=300,
            ftp_user=ftp_user_ref,
            ftp_pass=ftp_pass_ref,
            print_info=True,
        )

tests/test_tools_ftptools.py:30: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
rsgislib/tools/ftptools.py:466: in upload_ftp_file
    ftp_conn.login(user=ftp_user, passwd=ftp_pass)
/usr/lib/python3.10/ftplib.py:414: in login
    resp = self.sendcmd('PASS ' + passwd)
/usr/lib/python3.10/ftplib.py:281: in sendcmd
    return self.getresp()
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

self = <ftplib.FTP object at 0x7f4f5921c8b0>

    def getresp(self):
        resp = self.getmultiline()
        if self.debugging:
            print('*resp*', self.sanitize(resp))
        self.lastresp = resp[:3]
        c = resp[:1]
        if c in {'1', '2', '3'}:
            return resp
        if c == '4':
            raise error_temp(resp)
        if c == '5':
            raise error_perm(resp)
>       raise error_proto(resp)
E       ftplib.error_proto: /usr/sbin/vsftpd: Relink `/lib64/libnss_myhostname.so.2' with `/lib64/librt.so.1' for IFUNC symbol `clock_gettime'

/usr/lib/python3.10/ftplib.py:255: error_proto
----------------------------------------------------------------------------------------------------------------------------------- Captured stdout call ------------------------------------------------------------------------------------------------------------------------------------
Opening FTP Connection to ftp.dlptest.com
__________________________________________________________________________________________________________________________________ test_download_ftp_file ___________________________________________________________________________________________________________________________________

tmp_path = PosixPath('/tmp/pytest-of-o0331dobe/pytest-2/test_download_ftp_file0')

    def test_download_ftp_file(tmp_path):
        import rsgislib.tools.ftptools
    
        test_file = os.path.join(TOOLS_UTILS_DATA_DIR, "basic_str.txt")
>       rsgislib.tools.ftptools.upload_ftp_file(
            test_file,
            ftp_url=ftp_url_ref,
            remote_path="/rsgislib_test/test",
            ftp_timeout=300,
            ftp_user=ftp_user_ref,
            ftp_pass=ftp_pass_ref,
            print_info=True,
        )

tests/test_tools_ftptools.py:101: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
rsgislib/tools/ftptools.py:466: in upload_ftp_file
    ftp_conn.login(user=ftp_user, passwd=ftp_pass)
/usr/lib/python3.10/ftplib.py:414: in login
    resp = self.sendcmd('PASS ' + passwd)
/usr/lib/python3.10/ftplib.py:281: in sendcmd
    return self.getresp()
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

self = <ftplib.FTP object at 0x7f4f4c56dab0>

    def getresp(self):
        resp = self.getmultiline()
        if self.debugging:
            print('*resp*', self.sanitize(resp))
        self.lastresp = resp[:3]
        c = resp[:1]
        if c in {'1', '2', '3'}:
            return resp
        if c == '4':
            raise error_temp(resp)
        if c == '5':
            raise error_perm(resp)
>       raise error_proto(resp)
E       ftplib.error_proto: /usr/sbin/vsftpd: Relink `/lib64/libnss_myhostname.so.2' with `/lib64/librt.so.1' for IFUNC symbol `clock_gettime'

/usr/lib/python3.10/ftplib.py:255: error_proto
----------------------------------------------------------------------------------------------------------------------------------- Captured stdout call ------------------------------------------------------------------------------------------------------------------------------------
Opening FTP Connection to ftp.dlptest.com
________________________________________________________________________________________________________________________________ test_download_curl_ftp_file ________________________________________________________________________________________________________________________________

tmp_path = PosixPath('/tmp/pytest-of-o0331dobe/pytest-2/test_download_curl_ftp_file0')

    @pytest.mark.skipif(PYCURL_DB_NOT_AVAIL, reason="pycurl dependency not available")
    def test_download_curl_ftp_file(tmp_path):
        import rsgislib.tools.ftptools
    
        test_file = os.path.join(TOOLS_UTILS_DATA_DIR, "basic_str.txt")
>       rsgislib.tools.ftptools.upload_ftp_file(
            test_file,
            ftp_url=ftp_url_ref,
            remote_path="/rsgislib_test/test",
            ftp_timeout=300,
            ftp_user=ftp_user_ref,
            ftp_pass=ftp_pass_ref,
            print_info=True,
        )

tests/test_tools_ftptools.py:129: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
rsgislib/tools/ftptools.py:466: in upload_ftp_file
    ftp_conn.login(user=ftp_user, passwd=ftp_pass)
/usr/lib/python3.10/ftplib.py:414: in login
    resp = self.sendcmd('PASS ' + passwd)
/usr/lib/python3.10/ftplib.py:281: in sendcmd
    return self.getresp()
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

self = <ftplib.FTP object at 0x7f4f4c5068c0>

    def getresp(self):
        resp = self.getmultiline()
        if self.debugging:
            print('*resp*', self.sanitize(resp))
        self.lastresp = resp[:3]
        c = resp[:1]
        if c in {'1', '2', '3'}:
            return resp
        if c == '4':
            raise error_temp(resp)
        if c == '5':
            raise error_perm(resp)
>       raise error_proto(resp)
E       ftplib.error_proto: /usr/sbin/vsftpd: Relink `/lib64/libnss_myhostname.so.2' with `/lib64/librt.so.1' for IFUNC symbol `clock_gettime'

/usr/lib/python3.10/ftplib.py:255: error_proto
----------------------------------------------------------------------------------------------------------------------------------- Captured stdout call ------------------------------------------------------------------------------------------------------------------------------------
Opening FTP Connection to ftp.dlptest.com
===================================================================================================================================== warnings summary ======================================================================================================================================
../../../../../../../../usr/lib/python3.10/site-packages/geopandas/_compat.py:2
  /usr/lib/python3.10/site-packages/geopandas/_compat.py:2: DeprecationWarning: The distutils package is deprecated and slated for removal in Python 3.12. Use setuptools or check PEP 632 for potential alternatives
    from distutils.version import LooseVersion

tests/test_changedetect_pxloutlierchng.py::test_find_class_kurt_skew_outliers
  /usr/lib/python3.10/site-packages/rios/imageio.py:63: DeprecationWarning: `np.bool` is a deprecated alias for the builtin `bool`. To silence this warning, use `bool` by itself. Doing this will not modify any behavior and is safe. If you specifically wanted the numpy scalar type, use `np.bool_` here.
  Deprecated in NumPy 1.20; for more details and guidance: https://numpy.org/devdocs/release/1.20.0-notes.html#deprecations
    (numpy.bool,gdalconst.GDT_Byte),

tests/test_changedetect_pxloutlierchng.py::test_find_class_kurt_skew_outliers
  /usr/lib/python3.10/site-packages/rios/imageio.py:69: DeprecationWarning: `np.float` is a deprecated alias for the builtin `float`. To silence this warning, use `float` by itself. Doing this will not modify any behavior and is safe. If you specifically wanted the numpy scalar type, use `np.float64` here.
  Deprecated in NumPy 1.20; for more details and guidance: https://numpy.org/devdocs/release/1.20.0-notes.html#deprecations
    (numpy.float,gdalconst.GDT_Float64)

tests/test_classification.py::test_get_class_training_data
  /home/o0331dobe/Documents/AUR_packages/python-rsgislib/src/rsgislib-5.0.7/build-test/rsgislib/vectorutils/__init__.py:252: DeprecationWarning: invalid escape sequence '\*'
    """

tests/test_classification.py::test_get_class_training_data
  /home/o0331dobe/Documents/AUR_packages/python-rsgislib/src/rsgislib-5.0.7/build-test/rsgislib/vectorutils/__init__.py:334: DeprecationWarning: invalid escape sequence '\*'
    """

tests/test_classification.py::test_get_class_training_data
  /home/o0331dobe/Documents/AUR_packages/python-rsgislib/src/rsgislib-5.0.7/build-test/rsgislib/vectorutils/__init__.py:428: DeprecationWarning: invalid escape sequence '\*'
    """

tests/test_classification.py: 10 warnings
tests/test_classification_classaccuracymetrics.py: 4 warnings
tests/test_vectorattrs.py: 9 warnings
tests/test_vectorgeoms.py: 9 warnings
tests/test_vectorutils.py: 59 warnings
tests/test_vectorutils_createrasters.py: 1 warning
tests/test_vectorutils_createvectors.py: 1 warning
  /usr/lib/python3.10/site-packages/geopandas/io/file.py:362: FutureWarning: pandas.Int64Index is deprecated and will be removed from pandas in a future version. Use pandas.Index with the appropriate dtype instead.
    pd.Int64Index,

tests/test_classification_classaccuracymetrics.py: 10 warnings
tests/test_vectorutils.py: 3 warnings
  /home/o0331dobe/Documents/AUR_packages/python-rsgislib/src/rsgislib-5.0.7/build-test/rsgislib/vectorutils/__init__.py:2765: FutureWarning: The frame.append method is deprecated and will be removed from pandas in a future version. Use pandas.concat instead.
    data_gdf = data_gdf.append(tmp_data_gdf)

tests/test_regression_regresssklearn.py::test_perform_search_param_opt
  /usr/lib/python3.10/site-packages/sklearn/model_selection/_search.py:292: UserWarning: The total space of parameters 1 is smaller than n_iter=25. Running 1 iterations. For exhaustive searches, use GridSearchCV.
    warnings.warn(

tests/test_regression_regresssklearn.py::test_perform_search_param_opt
tests/test_regression_regresssklearn.py::test_perform_search_param_opt
  /usr/lib/python3.10/site-packages/sklearn/model_selection/_validation.py:680: DataConversionWarning: A column-vector y was passed when a 1d array was expected. Please change the shape of y to (n_samples,), for example using ravel().
    estimator.fit(X_train, y_train, **fit_params)

tests/test_regression_regresssklearn.py::test_perform_search_param_opt
  /usr/lib/python3.10/site-packages/sklearn/model_selection/_search.py:926: DataConversionWarning: A column-vector y was passed when a 1d array was expected. Please change the shape of y to (n_samples,), for example using ravel().
    self.best_estimator_.fit(X, y, **fit_params)

tests/test_regression_regresssklearn.py::test_perform_kfold_fit
tests/test_regression_regresssklearn.py::test_perform_kfold_fit
tests/test_regression_regresssklearn.py::test_perform_kfold_fit
tests/test_regression_regresssklearn.py::test_perform_kfold_fit
  /home/o0331dobe/Documents/AUR_packages/python-rsgislib/src/rsgislib-5.0.7/build-test/rsgislib/regression/regresssklearn.py:273: DataConversionWarning: A column-vector y was passed when a 1d array was expected. Please change the shape of y to (n_samples,), for example using ravel().
    skl_regrs_obj.fit(x_train, y_train)

tests/test_regression_regresssklearn.py::test_apply_regress_sklearn_mdl
  /home/o0331dobe/Documents/AUR_packages/python-rsgislib/src/rsgislib-5.0.7/build-test/tests/test_regression_regresssklearn.py:112: DataConversionWarning: A column-vector y was passed when a 1d array was expected. Please change the shape of y to (n_samples,), for example using ravel().
    et_obj.fit(x, y)

tests/test_vectorgeoms.py::test_explode_vec_lyr
tests/test_vectorgeoms.py::test_explode_vec_files
  /home/o0331dobe/Documents/AUR_packages/python-rsgislib/src/rsgislib-5.0.7/build-test/rsgislib/vectorgeoms/__init__.py:1634: FutureWarning: Currently, index_parts defaults to True, but in the future, it will default to False to be consistent with Pandas. Use `index_parts=True` to keep the current behavior and True/False to silence the warning.
    data_explode_gdf = data_gdf.explode()

tests/test_vectorutils.py::test_perform_spatial_join_empty
tests/test_vectorutils.py::test_perform_spatial_join
  /usr/lib/python3.10/site-packages/_pytest/python.py:192: FutureWarning: The `op` parameter is deprecated and will be removed in a future release. Please use the `predicate` parameter instead.
    result = testfunction(**testargs)

tests/test_vectorutils.py::test_merge_vector_files
tests/test_vectorutils.py::test_merge_vector_files
tests/test_vectorutils.py::test_merge_vector_files
  /home/o0331dobe/Documents/AUR_packages/python-rsgislib/src/rsgislib-5.0.7/build-test/rsgislib/vectorutils/__init__.py:2711: FutureWarning: The frame.append method is deprecated and will be removed from pandas in a future version. Use pandas.concat instead.
    data_gdf = data_gdf.append(tmp_data_gdf)

tests/test_vectorutils.py::test_merge_utm_vecs_wgs84
tests/test_vectorutils.py::test_merge_utm_vecs_wgs84
  /home/o0331dobe/Documents/AUR_packages/python-rsgislib/src/rsgislib-5.0.7/build-test/rsgislib/vectorutils/__init__.py:3019: FutureWarning: The frame.append method is deprecated and will be removed from pandas in a future version. Use pandas.concat instead.
    out_gdf = out_gdf.append(data_gdf)

tests/test_vectorutils.py::test_merge_utm_vecs_wgs84
  /home/o0331dobe/Documents/AUR_packages/python-rsgislib/src/rsgislib-5.0.7/build-test/rsgislib/vectorutils/__init__.py:2852: ShapelyDeprecationWarning: Iteration over multi-part geometries is deprecated and will be removed in Shapely 2.0. Use the `geoms` property to access the constituent parts of a multi-part geometry.
    for poly in row["geometry"]:

-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html

---------- coverage: platform linux, python 3.10.4-final-0 -----------
Name                                              Stmts   Miss  Cover
---------------------------------------------------------------------
rsgislib/__init__.py                                283     86    70%
rsgislib/changedetect/__init__.py                     1      0   100%
rsgislib/changedetect/pxloutlierchng.py             170     95    44%
rsgislib/classification/__init__.py                 404    177    56%
rsgislib/classification/classaccuracymetrics.py     889     46    95%
rsgislib/classification/classimblearn.py            109    109     0%
rsgislib/classification/classkeraschips.py          187    187     0%
rsgislib/classification/classkeraspxl.py            155    155     0%
rsgislib/classification/classlightgbm.py            467    467     0%
rsgislib/classification/classratutils.py             62      0   100%
rsgislib/classification/classsklearn.py             263     14    95%
rsgislib/classification/classxgboost.py             691    691     0%
rsgislib/classification/clustersklearn.py           112     10    91%
rsgislib/dataaccess/__init__.py                       1      1     0%
rsgislib/dataaccess/usgs_m2m.py                     242    242     0%
rsgislib/elevation/__init__.py                       42      9    79%
rsgislib/imagecalc/__init__.py                      692    119    83%
rsgislib/imagecalc/calcindices.py                   133     13    90%
rsgislib/imagecalc/leastcostpath.py                  51     51     0%
rsgislib/imagecalc/specunmixing/__init__.py         606    302    50%
rsgislib/imagecalibration/__init__.py               115    115     0%
rsgislib/imagecalibration/sensorlvl2data.py          64     64     0%
rsgislib/imagecalibration/solarangles.py             56     56     0%
rsgislib/imagefilter/__init__.py                    188      0   100%
rsgislib/imagefilter/tiledfilter.py                 444    155    65%
rsgislib/imagemorphology/__init__.py                  2      0   100%
rsgislib/imageregistration/__init__.py               53     15    72%
rsgislib/imageutils/__init__.py                    1645    301    82%
rsgislib/imageutils/imagecomp.py                    513    513     0%
rsgislib/imageutils/imagelut.py                      99      9    91%
rsgislib/imageutils/stdimgblockiter.py              171     22    87%
rsgislib/imageutils/tilingutils.py                  109    109     0%
rsgislib/rastergis/__init__.py                      458    288    37%
rsgislib/regression/__init__.py                      41      3    93%
rsgislib/regression/regresssklearn.py               184     16    91%
rsgislib/segmentation/__init__.py                     2      0   100%
rsgislib/segmentation/shepherdseg.py                189     27    86%
rsgislib/segmentation/skimgseg.py                   251    251     0%
rsgislib/segmentation/tiledclump.py                 167     16    90%
rsgislib/segmentation/tiledsegsingle.py             252    252     0%
rsgislib/timeseries/__init__.py                       1      0   100%
rsgislib/timeseries/modelfitting.py                 195     24    88%
rsgislib/timeseries/robustfitoutliners.py           137    137     0%
rsgislib/timeseries/tmask.py                        102    102     0%
rsgislib/tools/__init__.py                            1      0   100%
rsgislib/tools/checkdatasets.py                     410    158    61%
rsgislib/tools/filetools.py                         556    257    54%
rsgislib/tools/ftptools.py                          209    189    10%
rsgislib/tools/geometrytools.py                     238     68    71%
rsgislib/tools/googlecloud.py                        26     26     0%
rsgislib/tools/httptools.py                          83     83     0%
rsgislib/tools/plotting.py                          425    232    45%
rsgislib/tools/projection.py                         57      4    93%
rsgislib/tools/sensors.py                            16     16     0%
rsgislib/tools/stats.py                             389    274    30%
rsgislib/tools/sysprofile.py                         45     45     0%
rsgislib/tools/testimages.py                         30     30     0%
rsgislib/tools/tilecacheutils.py                    187    134    28%
rsgislib/tools/utils.py                             359    170    53%
rsgislib/tools/utm.py                               209    120    43%
rsgislib/tools/visualisation.py                     791    509    36%
rsgislib/vectorattrs/__init__.py                    379     58    85%
rsgislib/vectorgeoms/__init__.py                   1738    371    79%
rsgislib/vectorutils/__init__.py                   1407    253    82%
rsgislib/vectorutils/createrasters.py                77     23    70%
rsgislib/vectorutils/createvectors.py               496    159    68%
rsgislib/zonalstats/__init__.py                    1320    287    78%
---------------------------------------------------------------------
TOTAL                                             20446   8715    57%

================================================================================================================================== short test summary info ==================================================================================================================================
FAILED tests/test_tools_ftptools.py::test_upload_ftp_file - ftplib.error_proto: /usr/sbin/vsftpd: Relink `/lib64/libnss_myhostname.so.2' with `/lib64/librt.so.1' for IFUNC symbol `clock_gettime'
FAILED tests/test_tools_ftptools.py::test_download_ftp_file - ftplib.error_proto: /usr/sbin/vsftpd: Relink `/lib64/libnss_myhostname.so.2' with `/lib64/librt.so.1' for IFUNC symbol `clock_gettime'
FAILED tests/test_tools_ftptools.py::test_download_curl_ftp_file - ftplib.error_proto: /usr/sbin/vsftpd: Relink `/lib64/libnss_myhostname.so.2' with `/lib64/librt.so.1' for IFUNC symbol `clock_gettime'
============================================================================================================ 3 failed, 809 passed, 72 skipped, 131 warnings in 327.20s (0:05:27) ============================================================================================================

Problem installing RSGISLIB

Recently I tried to install RSGISLib via Windows 10 through Visual Code Studio, this version:
Version: 1.81.1 (user setup)
Commit: 6c3e3dba23e8fadc360aed75ce363ba185c49794
Date: 2023-08-09T22:22:42.175Z
Electron: 22.3.18
ElectronBuildId: 22689846
Chromium: 108.0.5359.215
Node.js: 16.17.1
V8: 10.8.168.25-electron.0
OS: Windows_NT x64 10.0.19045

I followed your guide by typing:

conda create -n osgeo-env-v1 python=3.7
conda activate osgeo-env-v1 #changed from source activate osgeo-env-v1
conda install -c conda-forge rsgislib

But the installation got stuck here:

Any way to solve this?
Thank you very much

Problems with calc_clear_sky_regions

It looks like the calc_clear_sky_regions function

rsgislib/python/rsgislib/imagecalibration/__init__.py

Line 142 in 2f61cfe

def calc_clear_sky_regions(

never quite made the migration over to rsgislib 5.0 fully. There is at least one case of incorrectly named keyword arguments being used in calls to other rsgislib functions (camelCase rather than snake_case), and at least one case of a missing argument.

We have a fix in progress, as soon as it is ready and tested we will open a PR. Work in progress here: https://github.com/EnvSys/rsgislib/blob/fix_clearsky/python/rsgislib/imagecalibration/__init__.py#L142

EDIT: Sorry, hit submit by accident too early

rasterization strategies in zonalstats funcs

Hi,

is there one of the zonalstats.METHOD_* that when applied by zonalstats.image_zone_to_hdf would yield same extraction results as the the other zonalstats functions?

Lately I have been trying out the zonalstats functions to extract hdf5 datasets and zonalstats.image_zone_to_hdf in particular seems to be quite fast. However, compared to the other functions of the zonalstats module and a rasterstats based implementation I wrote myself, the rasterization strategy seems to yield different results in number of pxls returned no matter which one I pick among the zonalstats.METHOD_* available.

The closest I got to the numbers returned by the other implementations is by setting
pxl_in_poly_method=zonalstats.METHOD_POLYOVERLAPSORCONTAINSPIXEL
and then masking out nodata values extracted on a masked raster by the same function, but still a bit short (50 something pxls over 15k so probably no big deal, but still wondering why). This is for extracting rois from polygons distanced enough so I wasn't concerned about extracting pixels multiple times by adopting a more generous strategy.

There are a few quite small polygons (that with rasterstats and other rsgislib functions yield less than 5 pxls), but even when removing these there is still some difference in the dimension of the retrieved datasets.

Error when i use runShepherdSegmentation

input = r"xxx.tif"
output = r"xxxx.kea"

segutils.runShepherdSegmentation(input,output,gdalformat='KEA')

_segmentation.error: Could not create GDALDataset'

how can i fix this error,thank you!!!

Pip compatibility

Hi folks - are there plans to make rsgislib pip compatible? Thanks!

Segmentation Runtime Error

Hi,
I was trying to run slic segmentation and got Runtime while doing so. Also, can we have segmentation with more than 3 bands ?

My code : -
perform_slic_segmentation( input_img, output_img, gdalformat='KEA', no_data_val=0, tmp_dir='tmp', calc_stats=True, n_segments=125000, compactness=0.1)

Error -
`---------------------------------------------------------------------------
RuntimeError Traceback (most recent call last)
File :4, in

File C:\ProgramData\Anaconda3\envs\rsgis\lib\site-packages\rsgislib\segmentation\skimgseg.py:445, in perform_slic_segmentation(input_img, output_img, gdalformat, no_data_val, tmp_dir, calc_stats, use_pca, n_pca_bands, pca_pxl_sample, n_segments, compactness, max_iter, sigma, spacing, convert_to_lab, enforce_connectivity, min_size_factor, max_size_factor, slic_zero)
429 segResult = skimage.segmentation.slic(
430 pxlVals,
431 n_segments=n_segments,
(...)
441 slic_zero=slic_zero,
442 )
443 segResult = segResult + 1
--> 445 gdalOutDS = gdal.Open(output_img, gdal.GA_Update)
446 gdalOutBand = gdalOutDS.GetRasterBand(1)
447 gdalOutBand.WriteArray(segResult)

File C:\ProgramData\Anaconda3\envs\rsgis\lib\site-packages\osgeo\gdal.py:4580, in Open(*args)
4578 def Open(*args):
4579 """Open(char const * utf8_path, GDALAccess eAccess) -> Dataset"""
-> 4580 return _gdal.Open(*args)

RuntimeError: The HDF5ImageDataset driver does not support update access to existing datas`

Thanks for your help !

band_math docs

From the band_math docs, the example is:

band_defns = list()
band_defns.append(rsgislib.imagecalc.BandDefn('b1', 'img.kea', 1))
band_defns.append(rsgislib.imagecalc.BandDefn('b2', 'img.kea', 2))
rsgislib.imagecalc.band_math('out.kea', 'b1*b2', 'KEA', rsgislib.TYPE_32FLOAT, band_defns)

no imports are provided. Only really worth it here as the import for BandDefns should be explicitly stated. Suggest a move to:

import rsgislib
from rsgislib import imagecalc
from rsgislib.imagecalc import BandDefn

band_defns = list()
band_defns.append(rsgislib.imagecalc.BandDefn('b1', 'img.kea', 1))
band_defns.append(rsgislib.imagecalc.BandDefn('b2', 'img.kea', 2))
rsgislib.imagecalc.band_math('out.kea', 'b1*b2', 'KEA', rsgislib.TYPE_32FLOAT, band_defns)

May be worth keeping this ticket open for all documentation issues

Divide by gamma?

Should you not be dividing by gamma instead of multiplying by it to match Shepherd's Equation 8?

rsgislib/src/calibration/RSGISApply6SCoefficients.cpp

Line 773 in 1a90429

output[i] = (LVal) / ((bandValues[irrDirBandIdx] * gamma) + (bandValues[irrDifandIdx] * this->brdfBeta));