gicait/geoserver-rest

Python library for management of geospatial data in GeoServer.

geoserver-rest's Issues

Allow for ASCII raster files when using create_coveragestyle()

When trying to submit an ASCII raster file to create_coveragestyle, the ASCII file (.asc) is not a valid extension in the raster_value function in Calculation_gdal.py.

Running the commands as in gdal to get the necessary raster parameters (min value, max value, and file name) works fine, so I think this should be possible to allow this file type as well (and could be added to the valid_extension array.

If not, is there any other workaround to this other than generating a .tif file?

the current transaction is aborted, commands ignored until end of transaction block

If the query doesn't run successfully it through the error message,

the current transaction is aborted, commands ignored until end of transaction block

And any other preference can't be done.

create_featurestore

after creating a PostGIS store the type in GeoServer is '?'.

geo.create_featurestores(
store_name='postgis',
workspace='webgis_studying',
host='localhost',
port=5433,
db='sua_2021',
schema='public',
pg_user='postgres',
pg_password='**********',
)

geo.publish_featurestore(
workspace='webgis_studying',
store_name='postgis',
pg_table='pub_preplot'

)

publish_featurestore with advertise true or false option is missing.

Use pytest and coverage

I can see that a file package_test.py is in the top-level, but we could potentially make use of pytest to make the testing process much more standardized.

Testing for code coverage (https://pypi.org/project/pytest-cov/) might also be of interest for maintainability. If this is of interest, let me know.

create_coveragestyle for range not working properly

create_featurestore with param create_database to true does not work

I want to create a PostGIS data source and also create a database, using create_featurestore with param create_database = "true" but does not work.
ref https://github.com/gicait/geoserver-rest/blob/master/geo/Geoserver.py#L556

geo.create_featurestore(store_name='ADMIN', workspace='ADMIN', db='geoserver_ADMIN', host='db', pg_user='docker', pg_password='docker', create_database='true')

the PostGIS data source was created, but not for the database

I use geoserver-rest==2.1.4

Create coveragestore error

The library can't create the coverage store if the store name is already exist

create_shp_datastore not working

GDAL Import Error "ModuleNotFoundError: No module named 'gdal'"

When importing this in my Python project, using GDAL 3.2.1, I get:

...
import gdal
...

ModuleNotFoundError: No module named 'gdal'

I believe the standard way of importing GDAL in python is from osgeo import gdal. To keep both variants, the following might help:

try:
    from osgeo import gdal
except:
    import gdal

V2.0.0

V2.0.0 Target,

1. Replace curl dependency with request: The curl library has lots of issues and it is complex. status: ongoing
2. Write the proper documentation: status: completed
3. Add get request for workspace, coveragestore, featurestore, layer, style, etc status: completed
4. Improve overwrite functionalities status: ongoing
5. Remove psycopg2 dependency (The v1.x.x PostgreSQL functionalities are replaced by postgres-helper) status: completed
6. Improve the style function for the vector dataset. Add functionality to add stroke colour, stroke width, fill color, fill-opacity etc control function. Status: Not started

upload_style function not working well

In the upload_style function, we can upload the SLD file. The style file is uploaded well. But when I checked the geoserver, the created style file doesn't contain all the code from the SLD file. Especially the color part is missing there.

Add publish_featurestore_sqlview to the documentation

GeoServer responding "An error occurred while loading the page" when pressing button "preview legend"

see here

https://gis.stackexchange.com/questions/450957/geoserver-responding-an-error-occurred-while-loading-the-page-when-pressing-bu

function get_style not work well

My english suck

Environment

OS: Win10 Home 64bit
Python: 3.10 64
geoserver-rest : 2.3.4 installed by pip

behave

get_style can not return style with workspace

code

i found some confused code

https://github.com/gicait/geoserver-rest/blob/master/geo/Geoserver.py start at line number 910

    def get_style(self, style_name, workspace: Optional[str] = None): // line 910
        """
        Returns the style by style name.
        """
        try:
            url = "{}/rest/styles/{}.json".format(self.service_url, style_name)
            if workspace is not None:
                url = "{}/workspaces/{}/styles/{}.json".format( // line 917
                    self.service_url, workspace, style_name
                )

line 917 is

                url = "{}/workspaces/{}/styles/{}.json".format(

is missing /rest/ ?

Use autoformatting (flake8 and black) and pre-commit

I'd like to run a few auto-formatting tools over the code base, specifically flake8 and black. These implement a set of standards for code styling with the aim of readability and maintainability.

This can be done using automated tooling such as pre-commit. This will change the style of the code considerably, but will make it very easy to integrate contributions without making the code base look "patched-together".

If this is interesting to you let me know!

Remove the "rest" from Geoserver class URL: It is making confusion for old users

As @jeafreezy suggested, we have added the rest at the end of Geoserver class parameter URL as below,

from geo.Geoserver import Geoserver
geo = Geoserver("http://localhost:8080/geoserver/rest", username="admin", password="geoserver")

And also we remove the rest from the methods of the Geoserver class. I think this is the breaking change in v2.3.0. Due to this, the previous version user faced lots of problems. So I am reverting the geoserver URL to the original URL without rest. I think we can think about the v3.0.0 for this breaking change. After solving this issue, the geoserver-rest will come back to the previous style as below,

from geo.Geoserver import Geoserver
geo = Geoserver("http://localhost:8080/geoserver", username="admin", password="geoserver")

# Create workspace test
geo.create_workspace("test")

about use 'geo.create_layergroup()'

hello,I'm trying use function about create_layergroup(),but I got some error.Like this:
AttributeError: 'Geoserver' object has no attribute 'create_layergroup'.

Can create layer group function be used now?

How can one publish a coverage store?

I can create a workspace: geo.create_workspace()

And I can create a coverage store: geo.create_coveragestore()

But how does one create a layer that is then published?

Thanks very much!

Conda build block, Need to update the setup.py file

create_featurestore function is not working.

Due to adding additional expose_primary_key filed in this method, the function is not working. Have to remove it.

Use standard docstrings and sphinx-autodoc for documentation

I would like to implement a standard for the docstrings (see: PEP 257) in the code base. I'm a fan of numpy-style docstrings, but integrating any style of your choosing would be fine (examples here: https://stackoverflow.com/a/24385103/7322852).

sphinx-autodoc is a sphinx plugin that crawls the functions of a code-base and generates documentation automatically based on the call signatures and the docstrings of functions. Assuming everything follows the same standard, the documentation would be able to generate an API doc based on this information.

If this is something of interest to you, let me know.

Connection parameter settings for "create_featurestore"

Hey!

I was wondering if anybody else had issues with implementing more connection parameters for the create_featurestore method?
There are a couple default values, which were not set when creating a feature store (connection parameters are listed here: https://docs.geoserver.org/latest/en/user/data/database/postgis.html).
I think this issue addresses also the problem with the "expose primary keys" option.

So if anybody is familiar with these problems, i would be happy to discuss this problem and to implement it correctly for the next release.

Cheers, Julius

Unable to install module with "Failed building wheel for gdal"

I was unable to install this module and it said "Failed building wheel for gdal".
However, I had GDAL 2.4.2 (gdal-config --version) working in my system.

This problem occurred because the geoserver-rest module tries to install the latest version of GDAL ( i.e. 3.2 for my case). So, we can install all the dependencies manually and then install geoserver-rest with --no-deps < pip3 install --no-deps geoserver_rest > .

While installing dependencies, sometimes, pip install GDAL may have to be modified to match the system's GDAL version.< pip install GDAL==$(gdal-config --version) --global-option=build_ext --global-option="-I/usr/include/gdal" >

Error: string longer than 2147483647 bytes

For the large rasters, >2GB, it will throw the error message as Error: string longer than 2147483647 bytes. This error is originally from python-request issue.

Managing layer settings

I would like to be able to do things like managing the CRSes, and other settings for a particular layer via PUT requests (https://docs.geoserver.org/latest/en/user/data/webadmin/layers.html#coordinate-reference-systems). I've looked over the REST API and am keen to learn how best to implement this.

@iamtekson Do you have any advice for how best to start working on this?

Error messages not informative

The HTTP error message are not very informative as they do only return the status code. Unfortunately, the geoserver often reponds with status code 500. Hence, the request text should be added to the message e.g.:

if r.status_code not in [200, 201]:
    return '{}: Data can not be published! {}'.format(r.status_code, r.text)

Support url/external method in create_coveragestore

This is a feature request.
In GeoServer Rest API doc : coveragestores

The upload method. Can be "url", "file", "external". “file” uploads a file from a local source. The body of the request is the file itself. “url” uploads a file from a remote source. The body of the request is a URL pointing to the file to upload. This URL must be visible from the server. “external” uses an existing file on the server. The body of the request is the absolute path to the existing file.

I assume that the method create_coveragestore has a parameter , which is set to file by default, and can be set to url or external.

Failed installation when install in virtualenv

Hi, I found some error when I install in virtualenv, but if I am install on global env it's work.

compilation terminated.
      x86_64-linux-gnu-g++ -shared -Wl,-O1 -Wl,-Bsymbolic-functions -Wl,-Bsymbolic-functions -g -fwrapv -O2 -Wl,-Bsymbolic-functions -g -fwrapv -O2 -g -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2 build/temp.linux-x86_64-3.10/extensions/gnm_wrap.o -lgdal -o build/lib.linux-x86_64-3.10/osgeo/_gnm.cpython-310-x86_64-linux-gnu.so
      error: command '/usr/bin/x86_64-linux-gnu-gcc' failed with exit code 1
      [end of output]
  
  note: This error originates from a subprocess, and is likely not a problem with pip.
error: legacy-install-failure

× Encountered error while trying to install package.
╰─> gdal

Can you help me to be able to install geoserver-rest on virtualenv?
I'm trying to keep my global env clean.

Thanks.

add publish_featurestore_sqlview

I try to write a function for add a layer as sql view, but I get error "Error looking up primary key".
This is code:

def publish_featurestore_sqlview(self, store_name, name, sql, keyColumn,geoName,geoType, workspace=None):
        try:
            if workspace is None:
                workspace = 'default'
            c = pycurl.Curl()
            layer_xml = """<featureType>
            <name>{0}</name>
            <enabled>true</enabled>
            <namespace>
            <name>{5}</name>
            </namespace>
            <title>{0}</title>
            <srs>EPSG:4326</srs>
            <metadata>
            <entry key="JDBC_VIRTUAL_TABLE"> 
            <virtualTable>
            <name>{0}</name>
            <sql>{1}</sql>
            <escapeSql>true</escapeSql>
            <keyColumn>{2}</keyColumn>
            <geometry>
            <name>{3}</name>
            <type>{4}</type>
            <srid>4326</srid>
            </geometry>
            </virtualTable>
            </entry>
            </metadata>
            </featureType>""".format(name,sql,keyColumn,geoName,geoType,workspace)
            c.setopt(pycurl.USERPWD, self.username + ':' + self.password)
            print (layer_xml)
            # connecting with the specified store in geoserver
            c.setopt(c.URL, '{0}/rest/workspaces/{1}/datastores/{2}/featuretypes'.format(
                self.service_url, workspace, store_name))
            c.setopt(pycurl.HTTPHEADER, ["Content-type: text/xml"])
            c.setopt(pycurl.POSTFIELDSIZE, len(layer_xml))
            c.setopt(pycurl.READFUNCTION, DataProvider(layer_xml).read_cb)
            c.setopt(pycurl.POST, 1)
            print("qui")
            c.perform()
            c.close()
            print("qui2")
        except Exception as e:
            return "Error:%s" % str(e)

Getting error on importing Geoserver

delete_layer request not deleting the vector datastore data

KeyColumn for Creating Query View

I found that your publish_featurestore_sqlview, does not have the primary key ('<keyColumn>primary_key</keyColumn>') parameter. Is there a specific reason behind that?? Are you willing that the primary key(s) be recognized by GeoServer automatically?
Also, I want to mention that a complex query can have a primary key made by combining more than one attribute. Is there a solution for handling that?

Working with s3-geotiff plugin

I want to automate creating a raster layer for s3 geotiffs using REST API

shapefile path is absolute in create_shp_datastore (QUESTION)

When creating a shape datastore using create_shp_datastore() in the created store's shapefile location is setting as an absolute path (E.g.: file:/D:/Geoserver/data_dir/data/workspace_name/datastore_name/data.shp) unlike when creating coveragestores, the fie location is set to the relative path. E.g. : file:data/workspace_name/rasterstore_name/raster.geotiff.
This leads to unavailability of the shapefile data store and previewing of the vector data when the geoserver is reused in multiple PCs. (E.g: in one PC geoserver may be in E: drive and in another it may be in C: drive but data becomes unavailable due to the absolute path location). How can we fix it?

Create coverage store for local files

Currently, create_coveragestore only works with data upload via put. It should also be possible to use local file as per geoserver UI using "file://path/to/file". I think this would also enable functionality for imagemosaic.

Postgres function produce transaction error frequently

The following error occurring frequently,

the current transaction is aborted, commands ignored until the end of the transaction block

delete coveragestore and feature store not working properly

get_status() doesn't return json

Hey! Thanks for a great product!

Description

geoserver-rest==2.3.4
requests==2.28.1

Geoserver API doesn't return a JSON-response to a request in the get_status() method of the Geoserver class. So return of the method doesn't produce a valid output. I'm not sure if it is only my specific problem because the get_system_status() method works just fine with a similar request.

Code to reproduce

from geo.Geoserver import Geoserver
geo = Geoserver('http://127.0.0.1:9090', username='admin', password='geoserver')
status = geo.get_status()
system_status = geo.get_system_status()
print(status)
print(system_status)

Output (truncated)

('get_status error: ', JSONDecodeError('Expecting value: line 1 column 1 (char 0)'))
{'metrics': {'metric': [{'available': False, 'description': 'Operating system', 'name': 'OPERATING_SYSTEM', 'unit': '', 'category': 'SYSTEM', 'identifier': 'OPERATING_SYSTEM', 'priority': 1, 'value': 'NOT AVAILABLE'}, ...

Possible solution

Add a header to ensure json return.

def get_status(self):
        """
        Returns the status of the geoserver. It shows the status details of all installed and configured modules.
        """
        try:
            url = "{}/rest/about/status.json".format(self.service_url)
            r = requests.get(url, auth=(self.username, self.password), headers={"Accept": "application/json"})
            return r.json()

geoserver-rest not covering all the REST API

Hi, as you can see, a lot of API has yet to be added to geoserver-rest. I'm working on some for my personal project. I'd like to contribute by adding more APIs (example: To manage security, usergroups, roles, layer security etc.)
Could anyone let me know if I should add code grouped by functionality(i.e. code for user groups in one PR, code for roles in other PR etc.)?

create_shp_datastore

I use the "create_shp_datastore" method to publish the datastore in a zip package, and the rest interface returns 202. What is the reason for this.

create_coveragestyle not working while passing the color_ramp as dict

How to use the existing data source publishing service, and how to define the name of the publishing layer?

How to use the existing data source publishing service, and how to define the name of the publishing layer?

raster_value() image NDVI mapping to value 0

I have NDVI (Vegetation index of normalized difference) image with values in [0.2276785671710968, 0.8963753581047058, 0.8060336003132513, 0.10305578791860792] . I attach image.

NDVI_20201104.tif.zip

so I apply the style RdYlGn:

geo.create_coveragestyle(raster_path=image, workspace='general', color_ramp='RdYlGn',overwrite=True)

The result is a pallet with all zeros (0) (show attach)

The bug is in function: raster_value(path) in Calculation_gdal.py
in lines:
15 min = int(srcband.GetMinimum())
16 max = int(srcband.GetMaximum())

It's lines apply int function, so the values are converted to 0.

The FIX:
15 min = srcband.GetMinimum()
16 max = srcband.GetMaximum()

create_coveragestyle maybe is not work well

I'm using the function to create dynamic style:

`
workspace="general"
image="../data/NDVI_20201104.tif"
style="ndvi"
color_ramp ='RdYlGn'

geo.create_coveragestyle(raster_path=image, style_name=style, workspace=workspace, color_ramp=color_ramp, overwrite=True)

`
But I getting error response: Invalid style: null

  <h1>Estado HTTP 400 – Bad Request</h1>
  <hr class="line" />
  <p><b>Tipo</b> Informe de estado</p>
  <p><b>mensaje</b> Invalid style:null</p>
  <p><b>descripción</b> El requerimiento enviado por el cliente era sintácticamente incorrecto.</p>
  <hr class="line" />
  <h3>

I noticed that, if the style exists in geoserver, then the function works fine. Otherwise, it is not able to upload the style file. However, the function says:

def create_coveragestyle(self, raster_path, style_name=None, workspace=None, color_ramp='RdYlGn_r', cmap_type='ramp', overwrite=False):
...
# create the xml file for associated style
...
# upload the style file <------ (this block upload style file???)
c.setopt(c.URL, '{0}/rest/workspaces/{1}/styles/{2}'.format(
self.service_url, workspace, style_name))
c.setopt(pycurl.HTTPHEADER, [
"Content-type:application/vnd.ogc.sld+xml"])
c.setopt(pycurl.READFUNCTION, FileReader(
open('style.sld', 'rb')).read_callback)
c.setopt(pycurl.INFILESIZE, os.path.getsize('style.sld'))
if overwrite:
c.setopt(pycurl.PUT, 1)
else:
c.setopt(pycurl.POST, 1)
c.setopt(pycurl.UPLOAD, 1)
c.perform()
c.close()

I understand that you upload the style file to the geoserver.

Doc Automation Example Broken

The example found here:
https://geoserver-rest.readthedocs.io/en/latest/advance_use.html

Is not working.
file_name is referenced before it is set.
lyr_name is not an argument in create_coveragestore.

[Enhancement] Handling proxy / proxies in requests from geoserver-rest

I was using this library and I figured out I had to set my proxies to "None" in order to avoid some firewall blocking problems...

PROXIES = { 'http': None, 'https': None, }
r = requests.post( url, data, auth=(self.username, self.password), headers=self.headers, proxies=PROXIES)

I had to rewrite custom functions from the geoserver-rest -> geo.Geoserver.Geoserver

I see no issue talking about that.
Is this something I missed or is it not implemented on purpose maybe?

400 Data can not be published on Geoserver 2.18.4

Running the function geo.publish_featurestore(workspace, store_name, pg_table) on my Geoserver 2.18.4 instance to create a new layer from a PostGIS table returns a 400 Bad Request, with the following message:

<!doctype html><html lang="en"><head><title>HTTP Status 400 \xe2\x80\x93 Bad Request</title><style type="text/css">h1 {font-family:Tahoma,Arial,sans-serif;color:white;background-color:#525D76;font-size:22px;} h2 {font-family:Tahoma,Arial,sans-serif;color:white;background-color:#525D76;font-size:16px;} h3 {font-family:Tahoma,Arial,sans-serif;color:white;background-color:#525D76;font-size:14px;} body {font-family:Tahoma,Arial,sans-serif;color:black;background-color:white;} b {font-family:Tahoma,Arial,sans-serif;color:white;background-color:#525D76;} p {font-family:Tahoma,Arial,sans-serif;background:white;color:black;font-size:12px;} a {color:black;} a.name {color:black;} .line {height:1px;background-color:#525D76;border:none;}</style></head><body><h1>HTTP Status 400 \xe2\x80\x93 Bad Request</h1><hr class="line" /><p><b>Type</b> Status Report</p><p><b>Message</b> Trying to create new feature type inside the store, but no attributes were specified</p><p><b>Description</b> The server cannot or will not process the request due to something that is perceived to be a client error (e.g., malformed request syntax, invalid request message framing, or deceptive request routing).</p><hr class="line" /><h3>Apache Tomcat/9.0.20</h3></body></html>'

Basically, Geoserver complains about 'Trying to create new feature type inside the store, but no attributes were specified'. Not sure what I can do to fix this?

New method needed for layerGroup

At this stage, we have only the get_layergroup method is available. We need the following methods,

1. create_layergroup
2. update_layergroup
3. add_layer_to_layergroup
4. delete_layergroup

Add a Conda-Forge recipe

Hi there,

This looks like a really promising project and seems to be the answer to my longstanding needs for automated GeoServer deployments. Would you be interested in a conda-forge recipe for Anaconda Python users? Upkeep is very simple (PRs are opened automatically on new PyPI versions and verification is all that's needed). I can help by contributing this if you're interested. Thanks!