Giter Site home page Giter Site logo

3liz / py-qgis-server Goto Github PK

View Code? Open in Web Editor NEW
66.0 13.0 16.0 7.3 MB

QGIS embbeded WMS/WFS/WCS asynchronous scalable http server

Home Page: https://docs.3liz.org/py-qgis-server

License: Mozilla Public License 2.0

Makefile 3.01% Python 92.27% Shell 2.53% Dockerfile 1.25% VCL 0.94%
qgis python qgis-server wms wfs wcs tornado

py-qgis-server's Introduction

QGIS embedded WMS/WFS/WCS asynchronous scalable server

PyPi version badge PyPI - Downloads PyPI - Python Version Docker Pulls

Description

This is an asynchronous HTTP QGIS server written in python on top of the tornado framework and the 0MQ messaging framework for distributing requests workers.

It is based on the QGIS 3 server API for efficiently passing requests/responses using 0MQ messaging framework to workers.

The server may run as a self-contained single service or as a proxy server with an arbitrary number of workers running remotely or locally. Independent workers connect automatically to the front-end proxy with no need of special configuration on the proxy side. Thus, this is ideal for auto-scaling configuration for use with container orchestrator as Rancher, Swarm or Kubernetes.

The server is aimed at solving some real situations encountered in production environment: zero conf scalability, handle long-running request situation, auto restart...

Py-Qgis-server is constantly tested against QGIS release and ltr version. See the QGIS roadmap.

Features

  • Multiples workers
  • Fair queuing request dispatching
  • Timeout for long-running/stalled requests
  • Full support of qgis server plugins
  • Auto-restart trigger for workers
  • Support streamed/chunked responses
  • SSL support
  • Support for access control extensions
  • Support for Qgis project stored in postgres database
  • Support adding new projects cache handlers as python extension
  • Preloading of Qgis projects in static cache
  • WFS3 support
  • Control exposition of Qgis API
  • Management API (experimental)

Requirements:

  • OS: Unix/Posix variants (Linux or OSX) (Windows not officially supported)
  • Python >= 3.6
  • Qgis >= 3.16
  • Some python knowledge about python virtualenv and package installation.
  • libzmq >= 4.0.1 and pyzmq >= 17

Documentation:

Latest documentation is available on docs.3liz.org

Installation

From Pypi

pip install py-qgis-server

From docker

Docker is the recommended way to deploy py-qgis-server as it ensure a working environment for running py-qgis-server

Follow the readme in the docker/ folder.

From source

Install in development mode

pip install -e .

Running the server

The server does not run as a daemon by itself, there are several ways to run a command as a daemon.

For example:

  • Use Supervisor http://supervisord.org/. Will gives you full control over logs and server status notifications.
  • Use the daemon command.
  • Use systemd
  • ...

Running the server

usage: qgisserver [-h] [-d] [-c [PATH]]
                  [--version] [-p PORT] [-b IP] [-w NUM] [-j NUM] [-u SETUID]
                  [--rootdir PATH] [--proxy] [--timeout SECONDS]

qgis/HTTP/0MQ scalable server

optional arguments:
  -h, --help            show this help message and exit
  -d, --debug           debug mode
  -c [PATH], --config [PATH]
                        Configuration file
  --version             Return version number and exit
  -p PORT, --port PORT  http port
  -b IP, --bind IP      Interface to bind to
  -w NUM, --workers NUM
                        Num workers
  -j NUM, --jobs NUM    Num server instances
  -u SETUID, --setuid SETUID
                        uid to switch to
  --rootdir PATH        Path to qgis projects
  --proxy               Run only as proxy
  --timeout SECONDS     Set client timeout in seconds

By default, the command will run server instances with workers and use unix sockets to communicate. This can be used to run the server as a single command.

Running proxy and workers separately

If the --proxy option the server will act as a proxy server to talk to independent qgis workers.

Qgis workers can be run using the command:

qgisserver-worker --host=localhost --rootdir=path/to/projects

Requests to OWS services

The OWS requests use the following format: /ows/?<ows_query_params>

Example:

http://myserver:8080/ows/?SERVICE=WFS&VERSION=1.1.0&REQUEST=GetCapabilities&MAP=<qgis_project_spec>

Requests to WFS3 OAPI

The requests use the following format: /wfs3/<wfs3_api_endpoint>?MAP=<qgis_project_spec>

Example:

http://myserver:8080/wfs3/collections.html?MAP=<qgis_project_spec>

Accessing the Qgis landing page and other qgis API

By default, the landing page is not enabled, see the documentation on how to enable the landing page.

Qgis api may be exposed on demand by configuring the api endpoints

Using with lizmap

In order to use the server with lizmap, you must set the following configuration in your lizmapConfig.ini.php:

[services]
wmsServerURL="http://my.domain:<port>/ows/"
...

; Use relative path
relativeWMSPath=true

Configuration

The configuration can be done either as configuration .ini file in or as environment variables.

The precedences of the configuration parameters is the following (from lowest to highest)

  • Defaults values
  • Environment variables
  • Config file
  • Command line options

Configuration parameters

Please look at the documentation for configuration options

Logging

By default, the server log on stdout/stderr and you have to configure redirection and log rotation on your infrastructure environment

py-qgis-server's People

Contributors

dmarteau avatar gustry avatar kapiwko avatar ldgeo avatar nicogodet avatar ptitjano avatar tank2003 avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

py-qgis-server's Issues

QGIS Server URL paths in secure mode

Problem

I have setup a running instance of this container using 3liz/qgis-map-server:3.22 in docker. I am using a service file to connect to the database resources.

When trying to access the resources in the browser I can get proper getcapabilities using the encoded URL parameters of the map i.e

http://localhost:32780/ows/?SERVICE=WMS&VERSION=1.1.0&REQUEST=GetCapabilities&MAP=postgresql%3A%3Fservice%3Ddev%26sslmode%3Drequire%26schema%3Dpublic%26project%3Dsample

However, passing the same request without an encoded URL path doesn't work i.e

http://localhost:32780/ows/?SERVICE=WMS&VERSION=1.1.0&REQUEST=GetCapabilities&MAP=postgresql:?service=dev&sslmode=require&schema=public&project=sample

I get the error Project 'postgresql:?service=dev' not found

For the second part, I can understand why the URL might fail because of failure to escape special characters

This now presents challenges when trying to get a getcapabilities response from owslib

>>> from owslib.wms import WebMapService
>>> wms = WebMapService('http://localhost:32780/ows/?SERVICE=WMS&VERSION=1.1.0&REQUEST=GetCapabilities&MAP=postgresql%3A%2F%2Fdocker%3Adocker%40db%3A5432%3Fsslmode%3Drequire%26dbname%3Ddata%26schema%3Dpublic%26project%3Dsample', version='1.3.0')
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/lib/python3/dist-packages/owslib/wms.py", line 54, in WebMapService
    return wms130.WebMapService_1_3_0(
  File "/usr/lib/python3/dist-packages/owslib/map/wms130.py", line 75, in __init__
    self._capabilities = reader.read(self.url, timeout=self.timeout)
  File "/usr/lib/python3/dist-packages/owslib/map/common.py", line 65, in read
    u = openURL(spliturl[0], spliturl[1], method='Get',
  File "/usr/lib/python3/dist-packages/owslib/util.py", line 214, in openURL
    req.raise_for_status()
  File "/usr/lib/python3/dist-packages/requests/models.py", line 943, in raise_for_status
    raise HTTPError(http_error_msg, response=self)
requests.exceptions.HTTPError: 404 Client Error: Not Found for url: http://localhost:32780/ows/?MAP=postgresql://docker:docker@db:5432?sslmode=require&dbname=data&schema=public&project=sample&service=WMS&request=GetCapabilities&version=1.3.0

Also trying, the same URL that works in the browser in QGIS Desktop fails with Download of capabilities failed

Note: I am not sure if this should be a question to owslib or QGIS Desktop or here. Feel free to close and I will ask in the OWSlib channel

Cross origin header pas present

Bonjour,

j'ai un problème avec le CORS. en mettant la variable d'environnement QGSRV_SERVER_CROSS_ORIGIN à yes, j'ai toujours le même problème. Peut être je l'utilise pas comme il faut ?

Merci encore pour cet image docker, trés utile !

Cross-Origin Request Blocked - CORS

Hello,
I try to work w/ your docker image. The install is well, and can serve WMS, WFS, using a qgis sample project w/ polygons layer.
I want to use it w/ a front mapping framework (OpenLayers 3).

WMS Display layer is well done. I tried to use service as WFS and blocked to a Cross-Origin Request. Same Thing on a GetFeatureInfo on WMS.

I specify and try multiple setting on crossOrigin as anonymous or null, to pass through. But nothing.
When i copy url directly in browser, JSON returns right.

Maybe a server configuration to allow anonymous connection in your QGIS Server build.
Is it by default set to a CORS anonymous?
Do you implement an envirnoment or parameter on launch container to set it?

Thanks in advance. (and your works)

Access to QGIS Server landing page resources

I am running a docker-compose with the following env variables

  - QGSRV_SERVER_WORKERS=6
  - QGSRV_LOGGING_LEVEL=DEBUG
  - QGSRV_CACHE_ROOTDIR=/web
  - QGSRV_CACHE_SIZE=100
  - QGSRV_SERVER_TIMEOUT=300
  - QGSRV_LANDING_PAGE=yes
  - QGIS_SERVER_LANDING_PAGE_PROJECTS_DIRECTORIES=/web

What is the correct path to access the landing page?
Running http://localhost:32774/ows/catalog gives me the following error
errors-fix

AttributeError with python 3.10.4 : 'function' object has no attribute 'select'

./pyqgisserver/bin/qgisserver -c /etc/qgis-server/py-qgis-server.conf
$ ./pyqgisserver/bin/qgisserver -c /etc/qgis-server/py-qgis-server.conf
WARNING: Failed to load 'PSutil', system metrics will not be collected
qgisserver 1.8.1 (build 202205251414,commit 8df0bb1)
QGIS 3.22.9-Białowieża 'Białowieża' (a8e9e6fae5)
QGIS code revision a8e9e6fae5
Qt version 5.15.3
Python version 3.10.4
GDAL/OGR version 3.4.1
PROJ version 8.2.1
EPSG Registry database version v10.041 (2021-12-03)
GEOS version 3.10.2-CAPI-1.16.0
SQLite version 3.37.2
OS Ubuntu 22.04 LTS

Log level set to INFO

2022-07-29 19:18:36,832 INFO    [121027]        Starting broker process
2022-07-29 19:18:36,835 INFO    [121035]        Binding frontend to ipc:///tmp/qgssrv/broker/121027/0
2022-07-29 19:18:36,836 INFO    [121035]        Binding backend to ipc:///tmp/qgssrv/broker/121027/pool0
2022-07-29 19:18:36,836 INFO    [121035]        Starting ZMQ broker loop
2022-07-29 19:18:36,837 INFO    [121038]        Starting worker pool
2022-07-29 19:18:36,838 INFO    [121027]        Running server on port 127.0.0.1:9876
2022-07-29 19:18:36,839 INFO    [121027]        Starting client b'OWS-SERVER-121027'
2022-07-29 19:18:36,840 INFO    [121027]        Initializing supervisor
2022-07-29 19:18:36,841 INFO    [121027]        Initializing pool healthcheck
2022-07-29 19:18:36,841 INFO    [121027]        Starting processing requests
Process PoolWorker-2:1:
Traceback (most recent call last):
  File "/usr/lib/python3.10/multiprocessing/process.py", line 315, in _bootstrap
    self.run()
  File "/usr/lib/python3.10/multiprocessing/process.py", line 108, in run
    self._target(*self._args, **self._kwargs)
  File "/home/lizmap/pyqgisserver/lib/python3.10/site-packages/pyqgisserver/qgsworker.py", line 361, in run
    QgsRequestHandler.init_server()
  File "/home/lizmap/pyqgisserver/lib/python3.10/site-packages/pyqgisserver/qgsworker.py", line 248, in init_server
    cls._cache_service        = get_cacheservice()
  File "/home/lizmap/pyqgisserver/lib/python3.10/site-packages/pyqgisserver/qgscache/cachemanager.py", line 445, in get_cacheservice
    return componentmanager.get_service(CACHE_MANAGER_CONTRACTID)
  File "/home/lizmap/pyqgisserver/lib/python3.10/site-packages/pyqgisservercontrib/core/componentmanager.py", line 134, in get_service
    return gComponentManager.get_service( contractID )
  File "/home/lizmap/pyqgisserver/lib/python3.10/site-packages/pyqgisservercontrib/core/componentmanager.py", line 123, in get_service
    fe = fe._replace(service=fe.create_instance())
  File "/home/lizmap/pyqgisserver/lib/python3.10/site-packages/pyqgisserver/qgscache/cachemanager.py", line 169, in __init__
    componentmanager.register_entrypoints('qgssrv_contrib_protocol_handler')
  File "/home/lizmap/pyqgisserver/lib/python3.10/site-packages/pyqgisservercontrib/core/componentmanager.py", line 146, in register_entrypoints
    gComponentManager.register_entrypoints( category, *args, **kwargs )
  File "/home/lizmap/pyqgisserver/lib/python3.10/site-packages/pyqgisservercontrib/core/componentmanager.py", line 79, in register_entrypoints
    for ep in _entry_points(category):
  File "/home/lizmap/pyqgisserver/lib/python3.10/site-packages/pyqgisservercontrib/core/componentmanager.py", line 50, in _entry_points
    return metadata.entry_points.select(group=group, name=name)
AttributeError: 'function' object has no attribute 'select'
2022-07-29 19:18:36,939 WARNING [121038]        Worker 121043 exited with code 1
2022-07-29 19:18:36,940 CRITICAL        [121038]        Critical worker failure. Aborting...
2022-07-29 19:18:41,843 CRITICAL        [121027]        Pool failure, exiting because of unrecoverable error...
Exiting with code: 1
2022-07-29 19:18:41,844 INFO    [121027]        Terminating client b'OWS-SERVER-121027'
PID 121027: Server instance stopped
Stopping workers
2022-07-29 19:18:41,845 INFO    [121027]        Stopping supervisor
Stopping broker
2022-07-29 19:18:41,845 WARNING [121035]        Broker Terminated
Server shutdown

if ver >= (3,10):
from importlib import metadata
# See https://docs.python.org/3.10/library/importlib.metadata.html
return metadata.entry_points.select(group=group, name=name)

It seems that this condition need to be adapted
Replacing

return metadata.entry_points.select(group=group, name=name)

by

return metadata.entry_points().select(group=group, name=name)

solves the error.

In fact, the doc says entry_points() and not entry_points --> https://docs.python.org/3.10/library/importlib.metadata.html#entry-points

Use pg_service

Bonsoir,

Y'a t'il un moyen de définir le chemin vers pg_service.conf ? J'ai des couches dans un projet qgis qui en ont besoin en fait

Merci encore pour ce projet, cela fait 2 ans que je l'utilise non stop, il est top !

WFS3 URL Paramètres

Lorsque je rentre cette URL, le paramètre limit, n'est pas pris en compte.

http://epfsqlcarto:9091/ows/wfs3/collections/opis/items.json?limit=1000&map=bluepad.qgs

La réponse est toujours limitée à une dizaine d'items et la fin de la réponse semble contenir une info contradictoire sur le paramètre 'limit' (voir ci-dessous)
HARNES","titre":"HARNES (62413) - Centre ville ancien","type_operation":null},"type":"Feature"}],"links":[{"href":"http://epfsqlcarto:9091/ows/wfs3/collections/opis/items.geojson?LIMIT=1000&MAP=bluepad.qgs","rel":"self","title":"Retrieve the features of the collection as GEOJSON","type":"application/geo+json"},{"href":"http://epfsqlcarto:9091/ows/wfs3/collections/opis/items.html?LIMIT=1000&MAP=bluepad.qgs","rel":"alternate","title":"Retrieve the features of the collection as HTML","type":"text/html"},{"href":"http://epfsqlcarto:9091/ows/wfs3/collections/opis/items.json?LIMIT=1000&MAP=bluepad.qgs&offset=10&limit=10","name":"Next page","rel":"next","title":"Retrieve the features of the collection as GEOJSON","type":"application/geo+json"}],"numberMatched":1243,"numberReturned":10,"timeStamp":"2022-02-15T10:51:16Z","type":"FeatureCollection"}

J'ai essayé d'autres paramètres comme "properties" et c'est pareil, en indiquant ne vouloir qu'un attribut spécifique, tous les attributs sont retournés.

La carte que j'interroge était dans un sous-dossier, je l'ai déplacé à la racine pour ne pas être embété avec le caractère "/" mais idem (je n'ai pas trouvé la façon de faire comme dans qgis server avec le fichier de configuration et l'ajout de [projects.schemes])

Improve how to use a symlink for a QGIS plugin

It would be nice to have :

  • a better warning when the plugin is using a symlink
  • indicate the workflow to use a QGIS plugin with the code located on the host (when we want to do some dev on a QGIS server plugin)

functions didn't render correctly in GetFeatureInfo

Hi,

I can't get working functions with GetFeatureInfo requests. There is also custom function in qgis profile folder.

Response didn't render correctly:
<Attribute name="maptip" value="<h3>Niittypolku 1 [% if("varauslomake"='x',generateFormUrl( "alue","korttelin_nro","tontin_nro","tontin_tyyppi"),'ei varattavissa')%]</h3> "/>

Should it works or have you experience with this?

'str' object has no attribute 'fileName' when using postgres_handler

When loading project from postgres I get a lot of http 500 errors with the message "Worker internal error". I see this error in the log:

qgis_server_1  | 2021-03-09 07:26:17,914	DEBUG	[40]	RCV b'OWS-SERVER-1': b'\xbd\x06\xe6\xbc\x80\xa8\x11\xeb\xbb`\x02B\xac\x17\x00\x03'
qgis_server_1  | 2021-03-09 07:26:17,915	DEBUG	[40]	Handling request: b'\xbd\x06\xe6\xbc\x80\xa8\x11\xeb\xbb`\x02B\xac\x17\x00\x03'
qgis_server_1  | 2021-03-09 07:26:17,915	DEBUG	[40]	Resolving 'dockerpostgres' protocol
qgis_server_1  | 2021-03-09 07:26:17,915	WARNING	[40]	Setting postgres connexion parameters in insecure mode postgres://docker:docker@postgres?sslmode=disable&dbname=qgis&schema=qgis_projekter&project=test1
qgis_server_1  | 2021-03-09 07:26:17,915	DEBUG	[40]	**** Postgresql connection params {'host': 'postgres', 'port': None, 'user': 'docker', 'password': 'docker', 'database': 'qgis', 'sslmode': 'disable'}
qgis_server_1  | 2021-03-09 07:26:17,922	DEBUG	[40]	**** Postgres metadata for 'test1': {'last_modified_time': '2021-03-08T16:22:14+00:00', 'last_modified_user': 'docker'}
qgis_server_1  | 2021-03-09 07:26:17,924	ERROR	[40]	Worker Error 'str' object has no attribute 'fileName'
qgis_server_1  | Traceback (most recent call last):
qgis_server_1  |   File "/usr/local/lib/python3.7/dist-packages/pyqgisserver/zeromq/worker.py", line 158, in run_worker
qgis_server_1  |     handler.handle_message()
qgis_server_1  |   File "/usr/local/lib/python3.7/dist-packages/pyqgisserver/qgsworker.py", line 228, in handle_message
qgis_server_1  |     config_path = project.fileName()
qgis_server_1  | AttributeError: 'str' object has no attribute 'fileName'
qgis_server_1  |
qgis_server_1  | 2021-03-09 07:26:17,925	INFO	[40]	Terminating Worker

I am using the latest docker image: 3liz/qgis-map-server:3.18

Use docker qgis-server with flat lizmap install

Hi,
I have installed a Lizmap on my server and i would like to use a docker for my qgis-server.

I manage to get my qgis-server running and i'm able to join data using my qgis-server. But when i'm using it with Lizmap, i get no values. The problem seems to be on the URL that Lizmap uses to generate the data from qgis-server. It founds nothing.

What parameter can i modify to get the right URL generated by Lizmap ?

On the admin panel i already changed the url using http:server:host/ows/
Any other stuff to do to get docker matching a fresh installed Lizmap ?
I'm using Lizmap 3.3.3 and Docker 3.4/3.8

Thanks in advance

Running proxy and workers separately with docker image show argument error

Hi,

first, the py-qgis-server with docker-image works fine.
I have two more questions:

How can i run the proxy and workers separately from docker image?

When i will start the workers
... qgisserver-worker --host=localhost --rootdir=/home/user1/projects
it shows an error "unrecognized arguments"
Running Xvfb Running Qgis server worker usage: qgisserver-worker [-h] [-d] [-c [PATH]] [--proxy-host host] [--identity IDENTITY] [--rootdir PATH] [--version] qgisserver-worker: error: unrecognized arguments: --host= --host=localhost

When i start proxy and workers on the same host (via docker images) which ports must i set for proxy and worker, because port 8080 is already used by one running docker image.

Enable "landing page"

The QGIS server "landing page" concept is very handy in a number of cases. Having a built-in client makes it so much easier to check out simple stuff like "what projects are available", "which project file is it that has the green layer" and so on.

I made a very simple poc of enabling "landing page" in py-qgis-server here: AsgerPetersen@1656954

Setting QGSRV_LANDING_PAGE=yes, QGIS_SERVER_LANDING_PAGE_PREFIX=/catalog and QGIS_SERVER_LANDING_PAGE_PROJECTS_DIRECTORIES=project_dir enables the landing page at http://host/catalog/index.html.

Before I throw more time into making it a mergeable PR I would like to ask if you are you at all interested in having this option? It is probably not something that should be enabled for production, but I think it is handy for development / testing purposes.

Accept project stored in PostgreSQL Database

In QGIS 3, it's possible to save the project to a PostgreSQL Database.

QGIS Server 3 is able to provide services based on a MAP with this type of encoded value postgresql:?service=pg_geotuga&sslmode=disable&dbname=&schema=ortos&project=teste

It could be interesting to have a way to defined a project stored in PostgreSQL Database.

for exemple, the PostgreSQL connexion for projects could be defined in a config file and the MAP parameter could be pg://schema/project

Project modified timestamp mismatch when loaded from postgres

I have had problems loading projects from postgres. I believe I have tracked it down to a problem with these checks

if timestamp is None or timestamp < modified_time:
cachemngr = componentmanager.get_service('@3liz.org/cache-manager;1')
project = cachemngr.read_project(urlstr)
# Consistency check => test that project modified time is the same as the one found
# in database, otherwise this will indicate thate the project was no read correctly
if modified_time != datetime.fromtimestamp(project.lastModified().toSecsSinceEpoch()):
raise cachemngr.UnreadableResourceError()

If I log the timestamps like this

            project_modified = project.lastModified()
            if modified_time != datetime.fromtimestamp(
                project_modified.toSecsSinceEpoch()
            ):
                LOGGER.error(
                    "modified_time [%s] does not match project.lastModified [%s] converted datetime [%s]",
                    modified_time,
                    project_modified,
                    datetime.fromtimestamp(project_modified.toSecsSinceEpoch()),
                )
                raise cachemngr.UnreadableResourceError()

I get this in my log output:

modified_time [2021-06-29 12:12:38.739532] does not match project.lastModified [PyQt5.QtCore.QDateTime(2021, 6, 29, 12, 12, 38, 740)] converted datetime [2021-06-29 12:12:38]

It seems like comparing the timestamps like this is very fragile.

Pro & Cons of py-qgis-server

Hi,

What are the advantages and disadvantages of using py-qgis-server rather than other alternatives like apache or nginx?
The benefits are not quite clear for me.

Thanks.

Implements a setting environment or config to force trust layer metadata

In QGIS 3.16, a new QgsProject::ReadFlag has been added QgsProject::ReadFlag::FlagTrustLayerMetadata. This flag forced the QgsProject option : trust layer metadata.

Trust layer metadata is an option at the project level to improve project read time by using vector layer's extent defined in the project and disabling the check primary key unicity for Postgres/Postgis layers.

Do not use it if layers' extent is not fixed during the project's use.

I did not install 'build.manifest'.

I'm trying to install py-qgis-server from 'pip install py-qgis-server'.
However, I get the error [Errno 2] No such file or directory: '/usr/local/lib/python3.8/dist-packages/pyqgisserver/build.manifest'.
Is this a mistake in the installation?
I did not install 'build.manifest'.

--------Err
root@i-16100000205417:/home/ubuntu# qgisserver
WARNING: Failed to read manifest ! [Errno 2] No such file or directory: '/usr/local/lib/python3.8/dist-packages/pyqgisserver/build.manifest'
Application path not initialized
WARNING: Failed to load 'PSutil', system metrics will not be collected
Traceback (most recent call last):
File "/usr/local/bin/qgisserver", line 5, in
from pyqgisserver.server import main
File "/usr/local/lib/python3.8/dist-packages/pyqgisserver/server.py", line 22, in
from .runtime import run_server
File "/usr/local/lib/python3.8/dist-packages/pyqgisserver/runtime.py", line 36, in
from .qgspool import create_poolserver
File "/usr/local/lib/python3.8/dist-packages/pyqgisserver/qgspool.py", line 33, in
from .qgsworker import QgsRequestHandler
File "/usr/local/lib/python3.8/dist-packages/pyqgisserver/qgsworker.py", line 47, in
'PATCH': QgsServerRequest.PatchMethod,
AttributeError: type object 'QgsServerRequest' has no attribute 'PatchMethod'

is build.manifest missing ?

Hi,

i ran into a first exception when i tried to install py-qgis-server and ran it for the first time (in dev mode):

Failed to read manifest !: [Errno 2] No such file or directory: '/home/.../py-qgis-server/pyqgisserver/build.manifest' 
qgisserver n/a (build n/a,commit n/a)

What is the purpose of this manifest file ? Is it missing from the repository or we should generate it with make manifest?
If so It should be documented 😉

Management API - Strip the root directory from the "link" property of the project cache

When the projects cache root directory is configured like

[projects.cache]
strict_check = false
rootdir = /srv/lizmap/accounts
size = 50
advanced_report = yes

The entrypoint /pool/ returns a wrong URL in the link property, for example

        {
          "filename": "/srv/lizmap/accounts/foo/instances/dev/data/qgis/rep1/bar.qgs",
          "key": "/srv/lizmap/accounts/foo/instances/dev/data/qgis/rep1/bar.qgs",
          "last_modified": "2022-03-29T04:23:58",
          "link": "http://172.16.0.188:7001/cache//srv/lizmap/accounts/foo/instances/dev/data/qgis/rep1/bar.qgs",
          "num_layers": 16
        }

the link property should return http://172.16.0.188:7001/cache/foo/instances/dev/data/qgis/rep1/bar.qgs

Erreur libQt5Core.so.5

Bonjour,

J'ai une erreur au chargement de l'image 3liz/qgis-map-server avec la version docker :

Running Xvfb
qgisserver 1.4.2 (build 202101111305,commit f98c8ec)
Log level set to DEBUG
2021-02-09 10:14:44,242	INFO	[1]	Starting broker process
2021-02-09 10:14:44,246	INFO	[1]	Running server on port 0.0.0.0:8080
2021-02-09 10:14:44,246	INFO	[23]	Binding frontend to ipc:///tmp/qgssrv/broker/0
2021-02-09 10:14:44,248	INFO	[23]	Binding backend to ipc:///tmp/qgssrv/broker/pool0
2021-02-09 10:14:44,249	INFO	[23]	Starting ZMQ broker loop
2021-02-09 10:14:44,250	INFO	[1]	Starting client b'OWS-SERVER-1'
2021-02-09 10:14:44,252	DEBUG	[1]	Updated watch files []
Process Process-2:
Traceback (most recent call last):
  File "/usr/lib/python3.7/multiprocessing/process.py", line 297, in _bootstrap
    self.run()
  File "/usr/lib/python3.7/multiprocessing/process.py", line 99, in run
    self._target(*self._args, **self._kwargs)
  File "/usr/local/lib/python3.7/dist-packages/pyqgisserver/runtime.py", line 172, in run_worker_pool
    from .qgspool import Pool
  File "/usr/local/lib/python3.7/dist-packages/pyqgisserver/qgspool.py", line 20, in <module>
    from .qgsworker import QgsRequestHandler
  File "/usr/local/lib/python3.7/dist-packages/pyqgisserver/qgsworker.py", line 23, in <module>
    from qgis.PyQt.QtCore import QBuffer, QIODevice, QByteArray
  File "/usr/lib/python3/dist-packages/qgis/__init__.py", line 70, in <module>
    from qgis.PyQt import QtCore
  File "/usr/lib/python3/dist-packages/qgis/PyQt/QtCore.py", line 24, in <module>
    from PyQt5.QtCore import *
ImportError: libQt5Core.so.5: cannot open shared object file: No such file or directory

Pour corriger le problème, j'ai ajouté le paquet binutils dans le Dockerfile puis j'ai ajouté la ligne suivante :
RUN strip --remove-section=.note.ABI-tag /usr/lib/x86_64-linux-gnu/libQt5Core.so.5
J'ai trouvé cette solution sur cette page, je ne sais pas si c'est la bonne solution mais tout marche bien ensuite.

J'ai constaté le problème sur les tags 3.10, 3.16, 3.16.2-1.4.2
J'ai utilisé le fichier docker-compose disponible ici

implement multiprocessing by adding php proxy/wrapper

not starting with topic because i had some troubles with installing py-qgis-server by pip. my colleguae had same problem. maybe of non supported qgis(server) version 3.16. succeeded with docker but in the original image from https://hub.docker.com/r/kartoza/qgis-server/ was a reference to an non existing file. so i had to replace
"py-qgis-server==${BUILD_VERSION}" py-amqp-client
to
"py-qgis-server" "git+https://github.com/3liz/py-amqp-client.git"
and then py-qgis-server ran flawlessly. Very nice is the project caching.

now to my question. your are using qgis python server from
https://qgis.org/pyqgis/3.16/server/QgsServer.html
and do requests with
https://qgis.org/pyqgis/3.22/server/QgsServerRequest.html
in my thoughts that should do the multiprocessing. Because of the behaviour that if i add a project to lizmap and then add layers from that project to a new project via the cgi with WMS in QGIS Desktop (https://url/to/qgis_mapserv.fcgi?map=path/to/my/map.qgs&service=WMS). Now with that project every layer which is embedded from that "internal/external" WMS source starts a new thread even on another core if that layers gets loaded on lizmap.

one bad thing about that is that if we add a layer with from our serverside qgis_mapserv.fcgi is that the user can see that internal url and (s)he cann add the project to their projects because there is no password etc. that does lizmap user interface make useless. so the scope is to use a php proxy which forwards the requested wms url to the "internal/external" qgis_mapserv.fcgi process. now we request only xyz.php and the url to the qgis_mapserv.fcgi remains unknown (attached php as log file).

the question is: can that behaviour be reproduced in py-qgis-server e.g. using subprocess and return a temporary php with nearly the same content as the attached file to achieve multiprocessing like in the aforementioned behaviour?

proxydemo.log

Add a QuickStart mode

Add to the Readme the minimum docker command line to quickliy start py-qgis-server.

Something like docker run -p 8080:8080 -v /path/to/qgis/projects:/projects 3liz/qgis-map-server

Standard QGIS functions are not evaluated in docker lizmap qgis server during printing

qgis doesn't see WFS data

I setup py-qgis-server with docker. My docker command is "docker run -p 8000:8080 --restart=always -d -v /home/carli/Tubitak/server/projects:/projects -e QGSRV_SERVER_WORKERS=2 -e QGSRV_LOGGING_LEVEL=DEBUG -e QGSRV_CACHE_ROOTDIR=/projects -e QGSRV_CACHE_SIZE=10 3liz/qgis-map-server:3.10"

I loaded data and qgis project file in volumed directory but I cannot see layer in qgis. Are there any people to help me out there ?
container folder
data-folder
Screenshot from 2020-07-09 16-43-34
web-url

Abort request to PostgreSQL if the connection pool is full

A QGIS project is connecting to a PostgreSQL database to fetch layers.
The connection pool for the user&password was full, so QGIS Server couldn't fetch data.

The "GetCapabilities" request was interrupted after a very long timeout from server with a 503 error.

It would be nice that such layers are considered "invalid" layer (like the "strict" mode in Py-QGIS-Server).
When loading the given project, QGIS Server was blocking other QGIS projects to load.

Project's data in subfolder

Hi all and thanks for sharing tons of code!
I have this issue: no problem to server projects with data (gpkg) at the same level of project file, but when I create a subfolder with data inside no layers are rendered (nor in WMS or in WFS).

Where am I wrong?

Many thanks (in advance)

Fabrizio

limit for preload config projects?

Hi,
Is there limit for preload config projects? How about the code behind this setting? The server seems to get stuck with ~20 qgis projects because it loads all projects in same time while starting.

Option to show decimal separator - QGIS_SERVER_SHOW_GROUP_SEPARATOR

Do you know if/how we can enable the group separator in py-qgis-server, run with the docker image 3liz/qgis-map-server:3.16?
Our goal is to have the separator show up in the lizmap form popup.

I tried with the environment variable specified here:

QGIS_SERVER_SHOW_GROUP_SEPARATOR: 'true'

I also tried setting

QGIS_OPTIONS_PATH: /srv/etc/qgis

and setting the following in the configuration file:

[locale]
globalLocale=it_IT
overrideFlag=true
showGroupSeparator=true
userLocale=it_IT

But it doesn't seem to be working. I'm probably missing something obvious.

Thanks!

running proxy and workers with docker

Hi,

Thanks for this repo! The examples is working fine but I didn't manage to run proxy and workers separately. I got some errors with these commands:

docker run -p 8080:8080 --name=qgis-proxy -e QGSRV_USER=$(id -u) -e QGSRV_LOGGING_LEVEL=DEBUG 3liz/qgis-map-server:3.22 qgisserver-proxy

Traceback (most recent call last):
File "/opt/local/pyqgisserver/lib/python3.8/site-packages/pyqgisserver/runtime.py", line 318, in run_server
worker_pool.start_supervisor()
AttributeError: 'NoneType' object has no attribute 'start_supervisor'
Terminating child processes
2022-03-15 07:19:47,448 INFO [1] Terminating client b'OWS-SERVER-1'
PID 1: Server instance stopped
Traceback (most recent call last):
File "/usr/local/bin/qgisserver", line 8, in
sys.exit(main())
File "/opt/local/pyqgisserver/lib/python3.8/site-packages/pyqgisserver/server.py", line 109, in main
run_server( port=args.port, address=args.interfaces, jobs=args.jobs, user=args.setuid, workers=workers )
File "/opt/local/pyqgisserver/lib/python3.8/site-packages/pyqgisserver/runtime.py", line 361, in run_server
if cache_observer:
UnboundLocalError: local variable 'cache_observer' referenced before assignment


docker run -v $(pwd)/QGIS-Sample-Data/qgis_sample_data/projects:/projects
--name qgis-worker
-e QGSRV_CACHE_ROOTDIR=/projects
-e QGSRV_USER=$(id -u)
-e QGSRV_LOGGING_LEVEL=DEBUG
-e ROUTER_HOST=qgis-proxy
3liz/qgis-map-server:3.22 qgisserver-worker

2022-03-15 07:21:56,571 DEBUG [1] Qgis: Server: Server initialized
2022-03-15 07:21:56,573 INFO [1] Proxy configuration enabled: :0, type: DefaultProxy
2022-03-15 07:21:56,573 DEBUG [1] Connecting to tcp://qgis-proxy:18080
2022-03-15 07:21:56,574 DEBUG [1] Identity set to b'\x98\x82\xdf8\xa40\x11\xec\xba\x90\x02B\xac\x11\x00\x03'
2022-03-15 07:21:56,574 DEBUG [1] Enabling broadcast notification
Application state:
QGIS_PREFIX_PATH env var: /usr
Prefix: /usr
Plugin Path: /usr/lib/qgis/plugins
Package Data Path: /usr/share/qgis
Active Theme Name:
Active Theme Path: /usr/share/qgis/resources/themes//icons/
Default Theme Path: :/images/themes/default/
SVG Search Paths: /usr/share/qgis/svg/
User DB Path: /usr/share/qgis/resources/qgis.db
Auth DB Path: qgis-auth.db

Traceback (most recent call last):
File "/usr/local/bin/qgisserver-worker", line 8, in
sys.exit(main())
File "/opt/local/pyqgisserver/lib/python3.8/site-packages/pyqgisserver/qgsworker.py", line 500, in main
QgsRequestHandler.run(args.router.format(host=args.host), identity=args.identity,
File "/opt/local/pyqgisserver/lib/python3.8/site-packages/pyqgisserver/qgsworker.py", line 335, in run
run_worker(router, QgsRequestHandler, identity=bytes(identity.encode('ascii')),
File "/opt/local/pyqgisserver/lib/python3.8/site-packages/pyqgisserver/zeromq/worker.py", line 153, in run_worker
supervisor = SupervisorClient()
File "/opt/local/pyqgisserver/lib/python3.8/site-packages/pyqgisserver/zeromq/supervisor.py", line 43, in init
address = _get_ipc('supervisor')
File "/opt/local/pyqgisserver/lib/python3.8/site-packages/pyqgisserver/zeromq/utils.py", line 5, in _get_ipc
ipc_path = confservice['zmq']['ipcpath']
File "/usr/lib/python3.8/configparser.py", line 1254, in getitem
raise KeyError(key)
KeyError: 'ipcpath'

Is it necessary QGIS to use this server?

Hi, this is just a question. I don't quite understand if it is necessary to make requests to QGIS or with this functionality you can get what you need from postgis. Thanks for your approach!

ModuleNotFoundError: No module named 'qgis'

Is this error due to a problem with my installation?


Traceback (most recent call last):
File "/usr/local/bin/qgisserver", line 33, in
sys.exit(load_entry_point('py-qgis-server==1.7.12', 'console_scripts', 'qgisserver')())
File "/usr/local/bin/qgisserver", line 25, in importlib_load_entry_point
return next(matches).load()
File "/usr/local/lib/python3.9/importlib/metadata.py", line 77, in load
module = import_module(match.group('module'))
File "/usr/local/lib/python3.9/importlib/init.py", line 127, in import_module
return _bootstrap._gcd_import(name[level:], package, level)
File "", line 1030, in _gcd_import
File "", line 1007, in _find_and_load
File "", line 986, in _find_and_load_unlocked
File "", line 680, in _load_unlocked
File "", line 790, in exec_module
File "", line 228, in _call_with_frames_removed
File "/usr/local/lib/python3.9/site-packages/pyqgisserver/server.py", line 22, in
from .runtime import run_server
File "/usr/local/lib/python3.9/site-packages/pyqgisserver/runtime.py", line 36, in
from .qgspool import create_poolserver
File "/usr/local/lib/python3.9/site-packages/pyqgisserver/qgspool.py", line 33, in
from .qgsworker import QgsRequestHandler
File "/usr/local/lib/python3.9/site-packages/pyqgisserver/qgsworker.py", line 23, in
from qgis.PyQt.QtCore import Qt, QBuffer, QIODevice, QByteArray
ModuleNotFoundError: No module named 'qgis'

Missing timeout option

Hello,

I try to use your py-qgis-server, but I've not --timeout option

leo@leo:~/workspace/lizmap/py-qgis-server$ git status
Sur la branche 1.2.x
Votre branche est à jour avec 'origin/1.2.x'.

rien à valider, la copie de travail est propre

Install with pip3 :

leo@leo:~/workspace/lizmap/py-qgis-server$ pip3 install -e .
Obtaining file:///home/leo/workspace/lizmap/py-qgis-server
Requirement already satisfied: PyYAML in /home/leo/.local/lib/python3.7/site-packages (from py-qgis-server==1.2.1) (5.3)
Requirement already satisfied: pyzmq>=17 in /home/leo/.local/lib/python3.7/site-packages (from py-qgis-server==1.2.1) (19.0.0)
Requirement already satisfied: tornado>=5 in /home/leo/.local/lib/python3.7/site-packages (from py-qgis-server==1.2.1) (6.0.4)
Installing collected packages: py-qgis-server
  Found existing installation: py-qgis-server 1.2.1
    Uninstalling py-qgis-server-1.2.1:
      Successfully uninstalled py-qgis-server-1.2.1
  Running setup.py develop for py-qgis-server
Successfully installed py-qgis-server

But...

leo@leo:~/workspace/lizmap/py-qgis-server$ qgisserver --help
usage: qgisserver [-h] [-d] [-c [PATH]] [--version] [-p PORT] [-b IP] [-w NUM]
                  [-j NUM] [-u SETUID] [--rootdir PATH] [--proxy]

qgis/HTTP/0MQ scalable server

optional arguments:
  -h, --help            show this help message and exit
  -d, --debug           debug mode
  -c [PATH], --config [PATH]
                        Configuration file
  --version             Return version number and exit
  -p PORT, --port PORT  http port
  -b IP, --bind IP      interface to bind to
  -w NUM, --workers NUM
                        num workers
  -j NUM, --jobs NUM    num server instances
  -u SETUID, --setuid SETUID
                        uid to switch to
  --rootdir PATH        path to qgis projects
  --proxy               run only as proxy
leo@leo:~/workspace/lizmap/py-qgis-server$ 

Have you forget this options?

Thank's for you reply

Léo Berry

QGIS Server performance questions

Hello,
I'm doing a WFS querying on your qgis docker: https://hub.docker.com/r/3liz/qgis-map-server/
I noticed that there some performance issues on WFS querying.
First time it takes 5 seconds to identify a point, then the second time is instantaneously. But the third time, after some time (45 seconds), it takes 5 seconds again.

Logs of first REQUEST: 10:16:32 -> 10:16:37

2020-06-05 10:16:32,479 INFO    [3498]  Qgis: Server: BBOX:664037,5103713,664037,5103713,EPSG:25832
2020-06-05 10:16:32,479 INFO    [3498]  Qgis: Server: MAP:mercati3.qgs
2020-06-05 10:16:32,479 INFO    [3498]  Qgis: Server: OUTPUTFORMAT:application/json
2020-06-05 10:16:32,479 INFO    [3498]  Qgis: Server: PROPERTYNAME:*
2020-06-05 10:16:32,479 INFO    [3498]  Qgis: Server: REQUEST:GetFeature
2020-06-05 10:16:32,479 INFO    [3498]  Qgis: Server: SERVICE:WFS
2020-06-05 10:16:32,479 INFO    [3498]  Qgis: Server: TYPENAME:jlucia
2020-06-05 10:16:32,479 INFO    [3498]  Qgis: Server: VERSION:1.1.0
2020-06-05 10:16:32,480 INFO    [3498]  Qgis: Server: WFS Request parameters:
2020-06-05 10:16:32,480 INFO    [3498]  Qgis: Server:  - OUTPUTFORMAT : application/json
2020-06-05 10:16:32,480 INFO    [3498]  Qgis: Server:  - PROPERTYNAME : *
2020-06-05 10:16:32,480 INFO    [3498]  Qgis: Server:  - TYPENAME : jlucia
2020-06-05 10:16:32,480 INFO    [3498]  Qgis: Server:  - BBOX : 664037,5103713,664037,5103713,EPSG:25832
2020-06-05 10:16:32,480 INFO    [3498]  Qgis: Server:  - VERSION : 1.1.0
2020-06-05 10:16:37,515 DEBUG   [3498]  b'\x83S\xb5(\xa6J\x11\xea\xa2\x8a\x02B\xac\x1a\x00\x02': Flushing response data: (461 bytes)
2020-06-05 10:16:37,516 DEBUG   [25]    SND worker: b'\x83S\xb5(\xa6J\x11\xea\xa2\x8a\x02B\xac\x1a\x00\x02' -> client: b'OWS-SERVER-1' : b'\xa0\xf4\xec*\xa7\x15\x11\xea\x87i\x02B\xac\x1a\x00\x02'
2020-06-05 10:16:37,517 DEBUG   [3498]  b'\x83S\xb5(\xa6J\x11\xea\xa2\x8a\x02B\xac\x1a\x00\x02': Flushing response data: (4 bytes)
2020-06-05 10:16:37,517 DEBUG   [25]    SND worker: b'\x83S\xb5(\xa6J\x11\xea\xa2\x8a\x02B\xac\x1a\x00\x02' -> client: b'OWS-SERVER-1' : b'\xa0\xf4\xec*\xa7\x15\x11\xea\x87i\x02B\xac\x1a\x00\x02'
2020-06-05 10:16:37,517 DEBUG   [25]    SND worker: b'\x83S\xb5(\xa6J\x11\xea\xa2\x8a\x02B\xac\x1a\x00\x02' -> client: b'OWS-SERVER-1' : b'\xa0\xf4\xec*\xa7\x15\x11\xea\x87i\x02B\xac\x1a\x00\x02'
2020-06-05 10:16:37,517 INFO    [3498]  Qgis: Server: Request finished in 5038 ms
2020-06-05 10:16:37,518 DEBUG   [25]    READY b'\x83S\xb5(\xa6J\x11\xea\xa2\x8a\x02B\xac\x1a\x00\x02'
2020-06-05 10:16:37,582 RREQ    [1]             206     GET     ?MAP=mercati3.qgs&OUTPUTFORMAT=application%2Fjson&SERVICE=WFS&PROPERTYNAME=%2A&REQUEST=GetFeature&TYPENAME=jlucia&VERSION=1.1.0&BBOX=664037%2C5103713%2C664037%2C5103713%2CEPSG%3A25832  5104    -1
2020-06-05 10:16:37,636 REQ     [1]     192.168.10.78   200     GET     /ows/?MAP=mercati3.qgs&outputFormat=application%2Fjson&service=WFS&propertyname=%2A&request=GetFeature&typename=jlucia&version=1.1.0&bbox=664037%2C5103713%2C664037%2C5103713%2CEPSG%3A25832     5158    -1       python-requests/2.18.4

SECOND request: the response is instant

2020-06-05 10:16:51,697 INFO    [10917] Qgis: Server:  - VERSION : 1.1.0
2020-06-05 10:16:51,740 DEBUG   [10917] b'\x13\x97\x18$\xa0\xa5\x11\xea\x89\xcb\x02B\xac\x1a\x00\x02': Flushing response data: (461 bytes)

THIRD request after 30-45 seconds: 10:19:26 -> 10:19:31, again 5 seconds

2020-06-05 10:19:26,048 INFO    [10917] Qgis: Server:  - VERSION : 1.1.0
2020-06-05 10:19:31,081 DEBUG   [10917] b'\x13\x97\x18$\xa0\xa5\x11\xea\x89\xcb\x02B\xac\x1a\x00\x02': Flushing response data: (461 bytes)

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.