oslandia / lopocs Goto Github PK

Migrated to: https://gitlab.com/Oslandia/lopocs

License: GNU Lesser General Public License v2.1

Python 100.00%

pointcloud 3d gis lidar opensource itowns cesiumjs python postgis

lopocs's Introduction

Light Opensource PointCloud Server

LOPoCS is a point cloud server written in Python, allowing to load Point Cloud from a PostgreSQL database thanks to the pgpointcloud extension.

The current version of LOPoCS provides a way to load Point Cloud from PostgreSQL to the following viewers:

Cesium thanks to the 3DTiles format
Potree viewer : viewer with LAZ compressed data.

Note that LOPoCS is currently the only 3DTiles server able to stream data from pgpointcloud. This is possible thanks to the python module py3dtiles.

Developments are still going on to improve state-of-the-art algorithms and performances.

Video

Online demonstration

Example using 3Dtiles/Cesium with data from GrandLyon

Main features

Command line tool to load data into PostgreSQL
Swagger API
Stream patches stored in PostgreSQL
Greyhound protocol support
3DTiles standard support (partial)
Produce ready to use examples with Potree and Cesium

Installation

Dependencies

python >= 3.4

gdal development headers (libgdal-dev)

pip (python3-pip)

virtualenv (python3-virtualenv)

pgpointcloud with lazperf enabled

Morton Postgres extension

PDAL (used by the LOPoCS loader)

Note

The LOPoCS loader uses PDAL's "mortonorder" filter in "reverse" mode. The "reverse" mode was introduced in PDAL version 1.7.1, so make sure you use this version or higher.

From sources

$ git clone https://github.com/Oslandia/lopocs
$ cd lopocs
$ virtualenv -p /usr/bin/python3 venv
$ source venv/bin/activate
(venv)$ pip install 'numpy==1.14.3'
(venv)$ pip install -e .

Configuration

You will find an example of a configuration file for lopocs in conf/lopocs.sample.yml

You have to copy it to conf/lopocs.yml and fill with your values, lopocs will load it if this file exists. Another alternative is to set up the LOPOCS_SETTINGS environment variable to locate your configuration file.

Usage

Prepare database

$ createdb lopocs
$ psql -d lopocs -c 'create extension postgis'
$ psql -d lopocs -c 'create extension pointcloud'
$ psql -d lopocs -c 'create extension pointcloud_postgis'
$ psql -d lopocs -c 'create extension morton'

Lopocs CLI

You can invoke lopocs in your virtualenv to show help and list available subcommands

$ cd lopocs
$ source venv/bin/activate
(venv)$ lopocs

Check installation

(venv)$ lopocs check
Pdal ... 1.4.0
Pdal plugin pgpointcloud ... ok
PostgreSQL ... 9.6.3
PostGIS extension ... 2.3.1
PgPointcloud extension ... 1.1.0
PgPointcloud-PostGIS extension ... 1.0

Demo data

(venv)$ mkdir demos
(venv)$ lopocs demo --work-dir demos/ --sample airport --cesium
(venv)$ lopocs serve

Copy/Paste the link at the end of the log in your browser and you will be able to see this:

Swagger API

Each viewer has specific expectations and communication protocol. So, the API is built to meet these specific needs.

Currently, 2 kinds of formats are supported:

3DTiles

Greyhound format (LAZ data with a footer indicating the number of points)

LOPoCS is able to stream data up to 2 viewers:

Cesium with the 3DTiles format

Potree viewer with the Greyhound format

LOPoCS provides its RESTful API through a Swagger UI by default on http://localhost:5000

Run tests

(venv)$ pip install .[dev]
(venv)$ py.test

Licence

LGPL>2: LICENSE.

lopocs's People

Contributors

Stargazers

Watchers

lopocs's Issues

Package dependencies to ease usage and Docker integration

Build the following packages for Docker integration :

laszip for jessie
laszip for sid
lazperf for jessie
lazperf for sid

pygdal on Debian sid

When I try to install pygdal on my Debian sid, I have the next error:

cc1plus: warning: command line option ‘-Wstrict-prototypes’ is valid for C/ObjC but not for C++
  extensions/gdal_wrap.cpp:3173:10: fatal error: cpl_vsi_error.h: No such file or directory
   #include "cpl_vsi_error.h"
            ^~~~~~~~~~~~~~~~~
  compilation terminated.
  error: command 'x86_64-linux-gnu-gcc' failed with exit status 1

This file should be provided by libgdal-dev (see https://packages.debian.org/sid/amd64/libgdal-dev/filelist), but it's not the case for now...

I had to download the gdal-2.3.0 tarball and link to its headers for the pip install:

$ wget -O /tmp/gdal-2.3.0.tar.gz https://download.osgeo.org/gdal/2.3.0/gdal-2.3.0.tar.gz
$ tar zxvf /tmp/gdal-2.3.0.tar.gz -C /tmp/
$ pip3 install pygdal --global-option=build_ext --global-option="-I/tmp/gdal-2.3.0/port"

Remove flask-restplus

We don't really need flask-restplus for lopocs, this add some extras decorators and could lower response time. if we want to keep a swagger interface we could generate the json currently used with http://flask-restplus.readthedocs.io/en/stable/swagger.html#export-swagger-specifications and provide the swagger interface at /apidoc for example.

this could help to generate the static page http://editor.swagger.io/#/

Cant update PDAL

I can't update pdal on the newest version. When I type the command:

sudo apt-get upgrade pdal

I get:

Reading package lists... Done
Building dependency tree       
Reading state information... Done
pdal is already the newest version (1.6.0-1build2).

Rendering Lidar data in browser and editing it

Hi,

Can lopocs be used to read lidar data from a server using rest service and then render it in browser?
Also, user must be able to make modifications in it, like adding a cube or some other geometry 3D primitives to the scene.

Can this all be done?

Web page templates for Potree and Cesium not available

When executing lopocs load for either potree or cesium, the respective page templates at http://3d.oslandia.com/potree.zip and http://3d.oslandia.com/cesium.zip cannot be found. As it is the case with the demo data the server doesn't seem to host the packages any more. Is this still available somewhere else?

Demo examples not available

Potree, Cesium, iTowns demo pages don't work:
http://3d.oslandia.com/potree/examples/lopocs-demos-terrain-potree.html
http://3d.oslandia.com/cesium/Apps/Lopocs-terrain/?view=-84.23392906178442%2C42.218270452265344%2C9833.314579381431%2C0.3536975272473852%2C-86.20742786248991%2C360

Coordinatesystems

Hi,

I have save the mount st helens data in database with projection/coordinate system 26910. Now I would like to test it and use the api for read pnts:

Now I got the error for the statement:

select pc_compress(pc_setpcid(pc_union(pc_filterbetween( pc_range(pa, 0, 1), 'Z', 1116.1100000000001, 2539.48 )), 2), 'laz') from (select pa from sthsm where pc_intersects(pa, st_geomfromtext('polygon ((560022.41 5114840.63, 564678.43 5114840.63, 564678.43 5120950.88, 560022.41 5120950.88, 560022.41 5114840.63))',4978)) order by morton limit 1024 )_;

ERROR in app [/home/..../lopocs/venv/lib/python3.5/site-packages/flask/app.py:1423]:
Exception on /3dtiles/read.pnts [GET]

Traceback (most recent call last):
File "/home/..../lopocs/venv/lib/python3.5/site-packages/flask/app.py", line 1475, in full_dispatch_request
rv = self.dispatch_request()
File "/home/..../lopocs/venv/lib/python3.5/site-packages/flask/app.py", line 1461, in dispatch_request
return self.view_functionsrule.endpoint
File "/home/..../lopocs/venv/lib/python3.5/site-packages/flask_restplus/api.py", line 309, in wrapper
resp = resource(*args, **kwargs)
File "/home/..../lopocs/venv/lib/python3.5/site-packages/flask/views.py", line 84, in view
return self.dispatch_request(*args, **kwargs)
File "/home/..../lopocs/venv/lib/python3.5/site-packages/flask_restplus/resource.py", line 44, in dispatch_request
resp = meth(*args, **kwargs)
File "./lopocs/app.py", line 115, in get
return threedtiles.ThreeDTilesRead().run(args)
File "./lopocs/threedtiles.py", line 59, in run
[tile, npoints] = get_points(box, lod, offset, schema_pcid, scale)
File "./lopocs/threedtiles.py", line 81, in get_points
pcpatch_wkb = Session.query_aslist(sql)[0]
File "./lopocs/database.py", line 131, in query_aslist
return list(chain(*cls.query(query, parameters=parameters)))
File "./lopocs/database.py", line 111, in query
cur.execute(query, parameters)
File "/home/ruediger/lopocs/venv/lib/python3.5/site-packages/psycopg2/extras.py", line 288, in execute
return super(NamedTupleCursor, self).execute(query, vars)
psycopg2.InternalError: Operation on mixed SRID geometries

Changing the coordinate system to 26910 in sql will deliver results. Should I change the bb in an other coordinate system ?
Do you have any more information about morton ?

Regards

Rüdiger

PS: api is great hopefully I can get it work

lyon demo rendering failed

After running lopocs demo --work-dir demos/ --sample lyon --cesium, I have opened the demo URL generated at the end of the console log, and I get the following message in the browser:

An error occurred while rendering. Rendering has stopped.
SecurityError: The operation is insecure.
m@file:///home/rdelhome/src/lopocs/demos/cesium/Cesium/Cesium.js:526:29304
O.prototype.update@file:///home/rdelhome/src/lopocs/demos/cesium/Cesium/Cesium.js:527:20230
b.prototype.update@file:///home/rdelhome/src/lopocs/demos/cesium/Cesium/Cesium.js:559:30067
f.prototype.update@file:///home/rdelhome/src/lopocs/demos/cesium/Cesium/Cesium.js:565:7755
Xe@file:///home/rdelhome/src/lopocs/demos/cesium/Cesium/Cesium.js:569:5198
tt@file:///home/rdelhome/src/lopocs/demos/cesium/Cesium/Cesium.js:569:9464
we.prototype.render@file:///home/rdelhome/src/lopocs/demos/cesium/Cesium/Cesium.js:569:17357
P.prototype.render@file:///home/rdelhome/src/lopocs/demos/cesium/Cesium/Cesium.js:581:12327
t@file:///home/rdelhome/src/lopocs/demos/cesium/Cesium/Cesium.js:581:276

@autra have you got any idea of what is going on?

Can't see any point in potree viewer

Hi, after more then a year I decided to give a change again to lopocs serve so I've installed latest versions of all dependencies and created test postgis database with all required extensions.
I've modified conf/lopocs.yml where i defined connection details for postgresql. Afterwards I've sucessfully loaded my las point cloud to the database using command
lopocs load --potree --table my_pointcloud --work-dir /path/to/data/ /path/to/data/my_pointcloud.las
After that, i could create tileset using command
lopocs tileset --table my_pointcloud --work-dir /path/to/data/

However, when i run lopocs serve and open potree page (served locally by nginx) I'm not able to see any point cloud. I see just overview map in top left corner (that is centered on correct location) but no point cloud at all. Not sure why, did I miss some step?

Thanks

Unable to install lazperf 1.2.1

Unable to install lazperf 1.2.1 on ubuntu 14.04. I have installed lazperf 1.3.0 but gives the following error.

Traceback (most recent call last):
  File "/home/tan/.conda/envs/lidar/lib/python3.7/site-packages/pkg_resources/__init__.py", line 583, in _build_master
    ws.require(__requires__)
  File "/home/tan/.conda/envs/lidar/lib/python3.7/site-packages/pkg_resources/__init__.py", line 900, in require
    needed = self.resolve(parse_requirements(requirements))
  File "/home/tan/.conda/envs/lidar/lib/python3.7/site-packages/pkg_resources/__init__.py", line 791, in resolve
    raise VersionConflict(dist, req).with_context(dependent_req)
pkg_resources.ContextualVersionConflict: (numpy 1.16.3 (/home/tan/.conda/envs/lidar/lib/python3.7/site-packages), Requirement.parse('numpy==1.14.3'), {'lopocs'})

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/tan/.conda/envs/lidar/bin/lopocs", line 6, in <module>
    from pkg_resources import load_entry_point
  File "/home/tan/.conda/envs/lidar/lib/python3.7/site-packages/pkg_resources/__init__.py", line 3241, in <module>
    @_call_aside
  File "/home/tan/.conda/envs/lidar/lib/python3.7/site-packages/pkg_resources/__init__.py", line 3225, in _call_aside
    f(*args, **kwargs)
  File "/home/tan/.conda/envs/lidar/lib/python3.7/site-packages/pkg_resources/__init__.py", line 3254, in _initialize_master_working_set
    working_set = WorkingSet._build_master()
  File "/home/tan/.conda/envs/lidar/lib/python3.7/site-packages/pkg_resources/__init__.py", line 585, in _build_master
    return cls._build_from_requirements(__requires__)
  
File "/home/tan/.conda/envs/lidar/lib/python3.7/site-packages/pkg_resources/__init__.py", line 598, in _build_from_requirements
    dists = ws.resolve(reqs, Environment())
  File "/home/tan/.conda/envs/lidar/lib/python3.7/site-packages/pkg_resources/__init__.py", line 786, in resolve
    raise DistributionNotFound(req, requirers)
pkg_resources.DistributionNotFound: The 'lazperf==1.2.1' distribution was not found and is required by lopocs

psycopg2.pool.PoolError: connection pool exhausted

new issue extracted from #33

ping @pizzuga : your problem is unrelated to #33 so let's continue the discussion here :-)

lazperf Python support

A note that I have implemented some lazperf Python support that you guys might find useful.

https://github.com/hobu/laz-perf/tree/master/python

It is not supporting LAZ at the moment, but that is something I hope to do in the near future.

server support

It looks like the code supports only development mode.

lopocs serve

binds only to 127.0.0.1. It would be nice an optional ipaddress/port argument, meanwhile I changed the code in cli.py to:

app.run(host='0.0.0.0')

Download of lyon dataset fails

The downloading of the lyon dataset fails for me:

$ lopocs demo --sample lyon --work-dir . --cesium         
[2017-06-23 09:41:48] Using sample data lyon: lyon.laz ... ok                                            
{'Access-Control-Allow-Origin': '*', 'Content-Encoding': 'gzip', 'Last-Modified': 'Mon, 22 May 2017 14:56:13 GMT', 'Content-Type': 'application/octet-stream', 'Connection': 'keep-alive', 'Server': 'nginx/1.6.2', 'Transfer-Encoding': 'chunked', 'Date': 'Fri, 23 Jun 2017 07:41:49 GMT'}                                
Traceback (most recent call last):                  
  File "/home/elemoine/.virtualenvs/lopocs/bin/lopocs", line 11, in <module>                             
    load_entry_point('lopocs', 'console_scripts', 'lopocs')()                                            
  File "/home/elemoine/.virtualenvs/lopocs/lib/python3.5/site-packages/click/core.py", line 722, in __call__                                                                                                       
    return self.main(*args, **kwargs)               
  File "/home/elemoine/.virtualenvs/lopocs/lib/python3.5/site-packages/click/core.py", line 697, in main 
    rv = self.invoke(ctx)                           
  File "/home/elemoine/.virtualenvs/lopocs/lib/python3.5/site-packages/click/core.py", line 1066, in invoke                                                                                                        
    return _process_result(sub_ctx.command.invoke(sub_ctx))                                              
  File "/home/elemoine/.virtualenvs/lopocs/lib/python3.5/site-packages/click/core.py", line 895, in invoke                                                                                                         
    return ctx.invoke(self.callback, **ctx.params)  
  File "/home/elemoine/.virtualenvs/lopocs/lib/python3.5/site-packages/click/core.py", line 535, in invoke                                                                                                         
    return callback(*args, **kwargs)                
  File "/home/elemoine/src/lopocs/lopocs/cli.py", line 489, in demo                                      
    download('Downloading sample', download_link, dest)                                                  
  File "/home/elemoine/src/lopocs/lopocs/cli.py", line 104, in download                                  
    length = int(r.headers['content-length'])       
  File "/home/elemoine/.virtualenvs/lopocs/lib/python3.5/site-packages/requests/structures.py", line 54, in __getitem__                                                                                            
    return self._store[key.lower()][1]              
KeyError: 'content-length'

The issue is related to the response not having a content-length header. For some reason, this header is no longer present.

@ldgeo any idea?

libgdal-dev and pygdal out of sync

libgdal-dev installs gdal 2.1.3 but there is not such binding in pygdal so setup fails

add support to load multiple files

This does not work:

lopocs load --potree --work-dir . *.las

merging all LAS files into a single LAZ and importing that does not work either, probably due to memory limits, PDAL on my system exits with errorcode -9

so I did this work around:

for f in *.las; do lopocs load --potree --work-dir . --table lopocs %f;done

anyway it is slow, for every file it builds the greyhound hierarchy, I think it should do this only once at the end, so it would be useful a command line switch that prevents this.

Moreover, for this scenario it is better to change the message Loading point clouds into database and include in it the filename currently used.

More important, I forgot to mention that the pipeline automatically created by lopocs does overwrite all data in the table.

Keep getting error: function pc_range(pcpatch, integer, integer) does not exist

Hello,

I tried to setup lopocs server and it worked fine but i have problem when i try to display my pointcloud in potree and cesium.
So, just for the info, I download st-helens.las and feed it in my postgis db using pgpointcloud and pdal pipeline and everything was fine about that. Then I installed the lopocs server and run it with uwgsi (with correct configuration in yml files). I checked restful api on http://localhost:5000 and it showed up correctly.
However, when I try to run potree viewer for example the point cloud doesn't show up and in the logs I see this error:

psycopg2.ProgrammingError: function pc_range(pcpatch, integer, integer) does not exist

Do you maybe know why is that and how could i fix it?

Thank you very much in advance and thank you for all the nice work you do here, this is just what was missing.

lopocs demo is failing

Currently, a resource is not available for demos and the next command is failing:

$ lopocs demo --work-dir demos/ --sample airport --potree
File "/usr/lib/python3.6/socket.py", line 745, in getaddrinfo
    for res in _socket.getaddrinfo(host, port, family, type, proto, flags):
socket.gaierror: [Errno -2] Name or service not known

I created an issue on liblas accordingly (libLAS/libLAS#141).

relation "pointcloud_lopocs" does not exist

Hi! Ive installed lopocs but I get the next error while trying get sources:
psycopg2.ProgrammingError: relation "pointcloud_lopocs" does not exist
LINE 16: from pointcloud_lopocs pl

Could you help me?

Thanks!

PDAL Lopocs