connormanning/entwine

Entwine - point cloud organization for massive datasets

entwine's Issues

Less points inserted

Hello again @connormanning
I'm running entwine for a newer dataset, however, it's not inserting all the points.
I have attached the dataset here.
AIRS.zip

This is command I used.

docker run -it \
   -v $HOME:$HOME \
   connormanning/entwine \
   build \
       -f \
       -i AIRS.las \
       -o ~/entwine/airs-default/ \
       -r EPSG:4326 EPSG:4978

Here is output I'm getting

Performing dataset inference...
1 / 1: AIRS.las
Inferred: [(-90, -180, 0), (89, 179, 23)]

Version: 1.1.0
Input:
        File: AIRS.las
        Point count hint: 1,555,200 points
        Density estimate (per square unit): 24.2013
        Threads: 8
Output:
        Output path: AIRS/

        Data storage: laszip
        Scale: 0.01
        Offset: (10, 10, 20)
        XYZ width: 4
Metadata:
        Native bounds: [(-90, -180, 0), (89, 179, 23)]
        Cubic bounds: [(-200, -200, -190), (220, 220, 230)]
        Reprojection: (from file headers, or a default of 'EPSG:4326') -> EPSG:4978
        Storing dimensions: [
                X, Y, Z, Intensity, ReturnNumber
                NumberOfReturns, ScanDirectionFlag, EdgeOfFlightLine, Classification, ScanAngleRank
                UserData, PointSourceId, GpsTime, Red, Green
                Blue, OriginId
        ]

Adding 0 - ~/entwine/airs-default/
        Pushes complete - joining...
Saving hierarchy...
Saving registry...
Saving metadata...


Index completed in 3 seconds.
Save complete.  Indexing stats:
        Points inserted: 756,045
        Points discarded:
                Outside specified bounds: 781,920
                Overflow past max depth: 0

Process Completed.

Here is how it looks like in MeshLab

Any idea why is it doing that? Thanks!

Better density estimation

See #41. Entwine should more accurately estimate data density during the dataset inference phase instead of using only the numPointsHint and the overall bounds of the data. This should be exposed via output metadata and subsequently by Greyhound.

CMake Error - Export Unknown Arguments

Trying to build locally, and getting the following:

root@ip-172-31-25-29:~/src/entwine/makefiles# cmake -G "Unix Makefiles" ../
-- The C compiler identification is GNU 4.8.4
-- The CXX compiler identification is GNU 4.8.4
-- Check for working C compiler: /usr/bin/cc
-- Check for working C compiler: /usr/bin/cc -- works
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - done
-- Check for working CXX compiler: /usr/bin/c++
-- Check for working CXX compiler: /usr/bin/c++ -- works
-- Detecting CXX compiler ABI info
-- Detecting CXX compiler ABI info - done
-- Looking for include file pthread.h
-- Looking for include file pthread.h - found
-- Looking for pthread_create
-- Looking for pthread_create - not found
-- Looking for pthread_create in pthreads
-- Looking for pthread_create in pthreads - not found
-- Looking for pthread_create in pthread
-- Looking for pthread_create in pthread - found
-- Found Threads: TRUE
-- Found CURL: /usr/lib/x86_64-linux-gnu/libcurl.so (found version "7.35.0")
CMake Error at CMakeLists.txt:68 (export):
  export Unknown arguments.


-- Configuring incomplete, errors occurred!
See also "/root/src/entwine/makefiles/CMakeFiles/CMakeOutput.log".
See also "/root/src/entwine/makefiles/CMakeFiles/CMakeError.log".

Building multiple source files

Hi,

am trying to build 3d-tiles from many source laz files, but entwine stops adding/build source after 4rth file, and command just stops without showing any error. Could it be due to that fact that source laz file are disjount, eg they are not close to each other and make different parts of surface?

thanks,

Alex.

Windows build

I need to build Entwine to run on Windows desktop, in order to build Cesium 3D Tiles for moderately sized point clouds (around 50-500 MB).

I encountered a lot of errors (2000+) during the build using MSVC. I could fix most of them, although I'm by no mean a C++ expert, so this will probably need a bit of code review. They mainly revolve around the known conflict around the std::min()/std::max() clashing with the macros defined in windows.h, but also some less obvious differences in accessing private members.

• http://stackoverflow.com/q/13416418

The CMake configuration will also probably need some updating to add some paths to dependencies (lzma...)

Maybe also add an AppVeyor CI config.

Entwine on Ubuntu 16.04: "error: ‘LazPerfCompressor’ in namespace ‘pdal’ does not name a template type"

When I try to compile Entwine on Ubuntu 16.04, I get this error:

In file included from /home/neel/CUSP/sw/entwine/entwine/tree/builder.cpp:37:0:
/home/neel/CUSP/sw/entwine/entwine/util/compression.hpp:145:11: error: ‘LazPerfCompressor’ in namespace ‘pdal’ does not name a template type

I have compiled laz-perf and PDAL manually, and am using the UbuntuGIS repo. The gdal and liblaz dependencies came from the Ubuntu package liblas-dev.

I did not use the ubuntu-deps.sh script as I believe it is for a older version of Ubuntu (~14.04).

Errors!

Hello @connormanning. Could you please help us resolve this new issue?

Now when running the example code:

sudo docker run -it -v $HOME:$HOME -e "CURL_VERBOSE=1" connormanning/entwine build
-i https://entwine.io/sample-data/red-rocks.laz
-o ~/entwine/red-rocks
I get the error:

---

Curl config:
timeout: 5s
followRedirect: true
verifyPeer: true
caBundle: (default)
caInfo: (default)
terminate called after throwing an instance of 'std::out_of_range'
what(): vector::_M_range_check: __n (which is 1889424) >= this->size() (which is 1889424)
Got error 11
terminate called recursively

and when running it locally:

sudo docker run -it -v $HOME:$HOME -e "CURL_VERBOSE=1" connormanning/entwine build \
-i $HOME/red-rocks.laz \
-o ~/greyhound/red-rocks
I get:

timeout: 5s
followRedirect: true
verifyPeer: true
caBundle: (default)
caInfo: (default)
Encountered an error: Could not read file
Exiting.

Releasify

Get Entwine ready for a 1.0.

• Unit testing
• Merge dev branches (containing scale/offset, optional curl, docs, etc.)
• Add versioning to serialized indexes for backward compatibility moving forward
• Make Travis happy again - some dependencies are missing in Docker scripts
• Store per-file metadata

Cesium per-point data

How hard would it be get meta data into the cesium format? I would like to have per-point data written to the BatchTable inside the pnts files. Data could be height or Z value before the reprojection to "EPSG:4978".

PointCloud#batch-table says its possible without introducing BatchID in the FeatureTable

Build failure

Likely a temporary thing, but popped up in the latest pull:

[ 10%] Building CXX object entwine/tree/CMakeFiles/tree.dir/builder.cpp.o
In file included from /usr/include/c++/5/cassert:43:0,
                 from /var/data/git_repos/entwine/entwine/types/manifest.hpp:14,
                 from /var/data/git_repos/entwine/entwine/tree/builder.hpp:24,
                 from /var/data/git_repos/entwine/entwine/tree/builder.cpp:11:
/var/data/git_repos/entwine/entwine/types/pooled-point-table.hpp: In member function ‘virtual void entwine::ConvertingPointTable::getFieldInternal(pdal::Dimension::Id, pdal::PointId, void*) const’:
/var/data/git_repos/entwine/entwine/types/pooled-point-table.hpp:160:20: error: ‘d’ was not declared in this scope
             assert(d->size() == sizeof(double));
                    ^
entwine/tree/CMakeFiles/tree.dir/build.make:62: recipe for target 'entwine/tree/CMakeFiles/tree.dir/builder.cpp.o' failed
make[2]: *** [entwine/tree/CMakeFiles/tree.dir/builder.cpp.o] Error 1
CMakeFiles/Makefile2:160: recipe for target 'entwine/tree/CMakeFiles/tree.dir/all' failed
make[1]: *** [entwine/tree/CMakeFiles/tree.dir/all] Error 2
Makefile:127: recipe for target 'all' failed
make: *** [all] Error 2

I did a fresh pull and removed the old build directory. Then cmake, make...

z-offset in cesium tiles

There is an unexplainable positive z-offset in the output of the cesium tiles (in cesium viewer).
In this image you see the pointcloud floating above another set of vector tiles:

Things I checked:

• proper coordinate system is given with las file (EPSG:28992), including towgs params
• other cesium vector tiles originating from the same srs are working well (as can be seen in the image)
• X and Y seem well aligned, only z

I also compared the transform matrix given in the tileset.json from entwine with the one that would be calculated by cesium based on the minx, miny values from the las file with:

var matrix = Cesium.Transforms.eastNorthUpToFixedFrame(Cesium.Cartesian3.fromDegrees(4.84539353916,52.3523793903,-2.219))

resulting in these differences:
Entwine transform:

 [
         -0.084535880369591351,	0.99642043582522855,	-6.9388939039072299e-18,	0,
         -0.78699076737683538,	-0.066767957551816193,	0.61334131762663679,		0,
         0.61114582301915321,	0.051849348252612899,	0.78981797149217947,		0,
         3889839.5983011033,	330012.31519960327,		5027057.5454196455,			1
  ]

Cesium tranform:

[
	-0.08446730511458142, 0.9964262513436106, 0, 0, 
 	-0.7889526244257994, -0.06687971333399081, 0.6108034547675762, 0, 
 	0.6086205967417826, 0.05159292177889329, 0.7917822551964607, 0, 
 	3890035.6951631806, 329759.3088569205, 5026846.2020282885, 1
 ]

I was hoping the cesium transform would be correct (it is for the vector data you can see in the image) but unfortunately although clamping the pointset to the ground it introduces an offset in -X and -Y direction of some 200 meters. Also I'm not sure if the problem is really in the transform matrix or already happens in the reprojection in entwine.
Worth noting: a large part of the points is below sea level and have a negative z-value.

The corresponding las file can be found at: http://research.geodan.nl/sites/cesium/data/pointcloud.tar.gz

EXC_BAD_ACCESS in Node::insertInto()

When building an entwine index on a 22404422 point laz file:

(lldb) bt
* thread #1: tid = 0x1d2062, 0x00000001000b7018 libentwine.dylib`entwine::Node::insertInto(arbiter::Endpoint const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >, unsigned long) [inlined] std::__1::__tree<std::__1::__value_type<BigUint, entwine::Node::AnchoredNode>, std::__1::__map_value_compare<BigUint, std::__1::__value_type<BigUint, entwine::Node::AnchoredNode>, std::__1::less<BigUint>, true>, std::__1::allocator<std::__1::__value_type<BigUint, entwine::Node::AnchoredNode> > >::__move_assign(std::__1::__tree<std::__1::__value_type<BigUint, entwine::Node::AnchoredNode>, std::__1::__map_value_compare<BigUint, std::__1::__value_type<BigUint, entwine::Node::AnchoredNode>, std::__1::less<BigUint>, true>, std::__1::allocator<std::__1::__value_type<BigUint, entwine::Node::AnchoredNode> > >&, std::__1::integral_constant<bool, true>) + 15 at __tree:1350, queue = 'com.apple.main-thread', stop reason = EXC_BAD_ACCESS (code=EXC_I386_GPFLT)
    frame #0: 0x00000001000b7018 libentwine.dylib`entwine::Node::insertInto(arbiter::Endpoint const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >, unsigned long) [inlined] std::__1::__tree<std::__1::__value_type<BigUint, entwine::Node::AnchoredNode>, std::__1::__map_value_compare<BigUint, std::__1::__value_type<BigUint, entwine::Node::AnchoredNode>, std::__1::less<BigUint>, true>, std::__1::allocator<std::__1::__value_type<BigUint, entwine::Node::AnchoredNode> > >::__move_assign(std::__1::__tree<std::__1::__value_type<BigUint, entwine::Node::AnchoredNode>, std::__1::__map_value_compare<BigUint, std::__1::__value_type<BigUint, entwine::Node::AnchoredNode>, std::__1::less<BigUint>, true>, std::__1::allocator<std::__1::__value_type<BigUint, entwine::Node::AnchoredNode> > >&, std::__1::integral_constant<bool, true>) + 15 at __tree:1350
    frame #1: 0x00000001000b7009 libentwine.dylib`entwine::Node::insertInto(arbiter::Endpoint const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >, unsigned long) [inlined] std::__1::__tree<std::__1::__value_type<BigUint, entwine::Node::AnchoredNode>, std::__1::__map_value_compare<BigUint, std::__1::__value_type<BigUint, entwine::Node::AnchoredNode>, std::__1::less<BigUint>, true>, std::__1::allocator<std::__1::__value_type<BigUint, entwine::Node::AnchoredNode> > >::operator=(std::__1::__tree<std::__1::__value_type<BigUint, entwine::Node::AnchoredNode>, std::__1::__map_value_compare<BigUint, std::__1::__value_type<BigUint, entwine::Node::AnchoredNode>, std::__1::less<BigUint>, true>, std::__1::allocator<std::__1::__value_type<BigUint, entwine::Node::AnchoredNode> > >&&) at __tree:1409
    frame #2: 0x00000001000b7009 libentwine.dylib`entwine::Node::insertInto(arbiter::Endpoint const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >, unsigned long) [inlined] std::__1::map<BigUint, entwine::Node::AnchoredNode, std::__1::less<BigUint>, std::__1::allocator<std::__1::pair<BigUint const, entwine::Node::AnchoredNode> > >::operator=(this=0x00000001bd5f10e0 size=4294967295) at map:961
  * frame #3: 0x00000001000b7009 libentwine.dylib`entwine::Node::insertInto(this=<unavailable>, ep=0x00007fff5fbfe700, postfix="", step=8) + 377 at hierarchy.cpp:127
    frame #4: 0x00000001000b9758 libentwine.dylib`entwine::Hierarchy::toJson(this=0x0000000101a0a550, ep=0x00007fff5fbfe700, postfix="") + 88 at hierarchy.cpp:361
    frame #5: 0x00000001000923c7 libentwine.dylib`entwine::Builder::saveOwnProps(this=0x0000000101a07f50) const + 407 at builder.cpp:599
    frame #6: 0x00000001000902fb libentwine.dylib`entwine::Builder::save(this=0x0000000101a07f50) + 75 at builder.cpp:551
    frame #7: 0x0000000100090004 libentwine.dylib`entwine::Builder::go(this=0x0000000101a07f50, max=<unavailable>) + 1252 at builder.cpp:311
    frame #8: 0x0000000100005c56 entwine`Kernel::build(args=<unavailable>) + 8806 at build.cpp:530
    frame #9: 0x0000000100007f5e entwine`main(argc=<unavailable>, argv=<unavailable>) + 1214 at entwine.cpp:82
    frame #10: 0x00007fff8d55a5ad libdyld.dylib`start + 1
    frame #11: 0x00007fff8d55a5ad libdyld.dylib`start + 1

Looks like something's amiss in the move-assignment operator but I'm not sure what yet — thought I'd open in case you had ideas. LMK if I can provide more info on the files, or such.

input pathname needs to be quoted when used with wildcard

I guess this is undesired:

works:
entwine build -i "/var/data/geocensus/*.las" -o /var/data/entwine/geocensus

does not work:
entwine build -i /var/data/geocensus/*.las -o /var/data/entwine/geocensus

produces:

Encountered an error: Invalid argument: /var/data/geocensus/zfs_10.las
Exiting.

Entwine not reading points

Whenever I try to run Entwine in docker, I keep getting the same error, regardless of the data set used.

$ docker run -it -v $HOME:$HOME connormanning/entwine build
-i /C/Users/psawall/entwine/Input/i35_MN_LiDAR_Colored.laz
-o ~/entwine/Output/i35_MN_LiDAR_Colored
Performing dataset inference...
1 / 1: /C/Users/psawall/entwine/Input/i35_MN_LiDAR_Colored.laz
Exception in pool task: Unable to open stream for >'/C/Users/psawall/entwine/Input/i35_MN_LiDAR_Colored.laz' with error 'No such file or directory'
Cannot use this scale for these bounds
Encountered an error: Zero points found
Exiting.

Support alternative compression types

Support alternative compression types:

• gzip
• LZMA
• LERC https://github.com/Esri/lerc

I don't know if these should be implemented in PDAL or Entwine.

Build failing

On latest github version.
Building with pdal 1.3, with lazperf.

[ 91%] Building CXX object test/CMakeFiles/entwine-test.dir/unit/build.cpp.o
/var/data/git_repos/entwine/test/unit/build.cpp:38:9: error: ‘PointId’ is not a member of ‘{anonymous}::D {aka pdal::Dimension::Id}’
         D::PointId, D::OriginId
         ^

entwine_error.txt

Unable to create open stream for '/opt/data/abc.laz'

Hello
I have Ubuntu trusty, 14.04, with docker.io and tried to follow the suggested method to try the repository:
docker run -it -v $HOME:/opt/data connormanning/entwine entwine build -i /opt/data/abc.laz -o /opt/data/entwine/abc

I received an error:
Performing dataset inference...
1 / 1: /opt/data/abc.laz
Exception caught in pool task: Unable to create open stream for '/opt/data/abc.laz' with error 'No such file or directory'
Encountered an error: Zero points found
Exiting.
gcc@gcc:~/bin$ docker run -it -v $HOME:/opt/data connormanning/entwine entwine build -i /opt/data/abc.laz -o /opt/data/entwine/abc
Performing dataset inference...
1 / 1: /opt/data/abc.laz
Exception caught in pool task: Unable to create open stream for '/opt/data/abc.laz' with error 'No such file or directory'
Encountered an error: Zero points found
Exiting.

why could not load the abc.laz?

Regards
Ibrahim

performance benchmark or estimate

Thanks for the great tool!

I finally finished indexing and visualizing a set of data only to discover that (I think) all of the 14 million points were collapsed onto two lat/lon positions because I was using an incorrect scale factor (X=-45.94672 -> X=-45).

Indexing ~14 million points took ~48 hours on my laptop:

docker run -it -v ~/point-cloud-vis:$HOME connormanning/entwine build -i ~/csv/ -o ~/entwine/Jakobshavn -r EPSG:4326 EPSG:3857

I was wondering if that is the expected performance? I could certainly use a smaller subset while developing to increase the iteration speed, but it might also be useful to have some examples or estimates in the documentation about the time to index 1,000,000 points etc.

Pix4D Point Clouds

Hello - We have had issues with Pix4D (latest version) point clouds displaying only in white, with RGB selected. If we take the point clouds out of Pix4D and runthem into another program, such as Leica Cyclone, and then through Entwine, we are able to see point colors natively. Any input is appreciated. Thanks!

3d-tiles layer height

Wiki instructions: "invalid kernel type"

When running the command in the wiki's quickstart:

docker run -it -v $HOME:/opt/data connormanning/entwine \
    entwine build -i /opt/data/abc.laz -o /opt/data/entwine/abc

I get

Invalid kernel type
Version: 1.0.0
Usage: entwine <kernel> <options>
Kernels:
	build
		Build (or continue to build) an index
	infer
		Aggregate information for an unindexed dataset
	merge
		Merge colocated previously built subsets

I suspect that the command should be

docker run -it -v $HOME:/opt/data connormanning/entwine \
    build -i /opt/data/abc.laz -o /opt/data/entwine/abc

no driver for s3

looking at the docs, I should be able to run

docker run -it  -v `pwd`:`pwd` -w `pwd` --rm connormanning/entwine build -i s3://iowa-lidar/iowa/ -o ./some/directory

but doing so and I get the error Encountered an error: No driver for s3://iowa-lidar/iowa/*

entwine 1.1.0 build fails - jsoncpp issues

Entwine 1.1.0 is failing to build on :

• centos 7
• gcc 4.8.5 (and 6.2.1)

with dependences:

• PDAL (master) in /usr/local, built from source
• jsoncpp 1.8.0 in /usr/local, built from source

First error:

/local/build/entwine/entwine/third/arbiter/arbiter.cpp:2609:47: error: invalid conversion from ‘const void*’ to ‘void*’ [-fpermissive]
         if (BIO* bio = BIO_new_mem_buf(s.data(), -1))

...and then, using -fpermissive:

[ 98%] Linking CXX executable entwine
/usr/bin/ld: warning: libjsoncpp.so.11, needed by /usr/lib/gcc/x86_64-redhat-linux/4.8.5/../../../libpdal_base.so, may conflict with libjsoncpp.so.0
/usr/bin/ld: CMakeFiles/kernel.dir/build.cpp.o: undefined reference to symbol '_ZN4Json5ValueC1Em'
/usr/local/lib64/libjsoncpp.so.11: error adding symbols: DSO missing from command line
collect2: error: ld returned 1 exit status
make[2]: *** [kernel/entwine] Error 1
make[1]: *** [kernel/CMakeFiles/kernel.dir/all] Error 2
make[1]: *** Waiting for unfinished jobs....

Not sure what this means. I can't remove centos default jsoncpp, I figured placing a newer version in /usr/local would be the fix - PDAL is linked against the newer jsoncpp just fine.

Advice appreciated!

Error attempting to read multiple inputs from S3

I tried running entwine on AWS, where I had multiple files in an S3 bucket and I received this error
Encountered an error: Unexpected contents in AWS response
When I changed entwine to run against each file in the bucket individually, they worked fine. What I expected, was that entwine would build it's indexing for all the files in the bucket. This was the behavior I think I remember seeing when running entwine on my local machine. I think I need to use merge but I'm a little confused on how to do it. I'm running this as a DC/OS job which essentially just runs the docker command for building with entwine.
Does entwine support processing of multiple files in a single location within an S3 bucket?

Problem to create entwine index

I try to load 15 pointcloud tiles ( source file http://dl.mapgears.com/mg-laz.tar) in an entwine index. Entwine failed to load 3 of those files ( 278-5048_rgb.laz, 278-5047_rgb.laz and 276-5049_rgb.laz).

my script looks like this:

docker run -it -v /opt/data/:/data connormanning/entwine entwine build -i /data/LAS/275-5047_rgb.laz -o /data/greyhound/RDP_RMI -b "[269000, 5034000, -100,308000, 5066000, 150]"
docker run -it -v /opt/data/:/data connormanning/entwine entwine build -i /data/LAS/275-5048_rgb.laz -o /data/greyhound/RDP_RMI
docker run -it -v /opt/data/:/data connormanning/entwine entwine build -i /data/LAS/275-5049_rgb.laz -o /data/greyhound/RDP_RMI
docker run -it -v /opt/data/:/data connormanning/entwine entwine build -i /data/LAS/276-5047_rgb.laz -o /data/greyhound/RDP_RMI
docker run -it -v /opt/data/:/data connormanning/entwine entwine build -i /data/LAS/276-5048_rgb.laz -o /data/greyhound/RDP_RMI
docker run -it -v /opt/data/:/data connormanning/entwine entwine build -i /data/LAS/276-5049_rgb.laz -o /data/greyhound/RDP_RMI
docker run -it -v /opt/data/:/data connormanning/entwine entwine build -i /data/LAS/277-5047_rgb.laz -o /data/greyhound/RDP_RMI
docker run -it -v /opt/data/:/data connormanning/entwine entwine build -i /data/LAS/277-5048_rgb.laz -o /data/greyhound/RDP_RMI
docker run -it -v /opt/data/:/data connormanning/entwine entwine build -i /data/LAS/277-5049_rgb.laz -o /data/greyhound/RDP_RMI
docker run -it -v /opt/data/:/data connormanning/entwine entwine build -i /data/LAS/278-5047_rgb.laz -o /data/greyhound/RDP_RMI
docker run -it -v /opt/data/:/data connormanning/entwine entwine build -i /data/LAS/278-5048_rgb.laz -o /data/greyhound/RDP_RMI
docker run -it -v /opt/data/:/data connormanning/entwine entwine build -i /data/LAS/278-5049_rgb.laz -o /data/greyhound/RDP_RMI
docker run -it -v /opt/data/:/data connormanning/entwine entwine build -i /data/LAS/279-5047_rgb.laz -o /data/greyhound/RDP_RMI
docker run -it -v /opt/data/:/data connormanning/entwine entwine build -i /data/LAS/279-5048_rgb.laz -o /data/greyhound/RDP_RMI
docker run -it -v /opt/data/:/data connormanning/entwine entwine build -i /data/LAS/279-5049_rgb.laz -o /data/greyhound/RDP_RMI

At my third try I've finally received a real error message of bad memory allocation. Dont know why but Ithe first time I've retry to load this file, entwine crashes wihout any error message!

Here's my log

# docker run -it -v /opt/data/:/data connormanning/entwine entwine build -i /data/LAS/278-5048_rgb.laz -o /data/greyhound/RDP_RMI

Continuing previous index...

Input:
        Building from 13 source files
        Trust file headers? yes
        Work threads: 3
        Clip threads: 6
Output:
        Output path: file:///data/greyhound/RDP_RMI/
        Temporary path: tmp/
        Compressed output? yes
Tree structure:
        Null depth: 6
        Base depth: 10
        Cold depth: lossless
        Mapped depth: 13
        Sparse depth: 13
        Chunk size: 262144 points
        Dynamic chunks? yes
        Prefix IDs? no
        Build type: hybrid
        Point count hint: 13740813 points
Geometry:
        Conforming bounds: [(269000.00000, 5034000.00000, -100.00000), (308000.00000, 5066000.00000, 150.00000)]
        Cubic bounds: [(268990.00000, 5030490.00000, -19485.00000), (308010.00000, 5069510.00000, 19535.00000)]
        Reprojection: (none)
        Storing dimensions: [X, Y, Z, Intensity, ReturnNumber, NumberOfReturns, ScanDirectionFlag, EdgeOfFlightLine, Classification, ScanAngleRank, UserData, PointSourceId, GpsTime, Red, Green, Blue, Origin]

Adding 12 - /data/LAS/278-5048_rgb.laz
 A: 1048576 C: 1 H: 38
        Pushes complete - joining...
Unknown error during /data/LAS/278-5048_rgb.laz
terminate called after throwing an instance of 'std::bad_alloc'
  what():  std::bad_alloc
Got error 11
entwine[0x41b00b]
/lib/x86_64-linux-gnu/libc.so.6(+0x352f0)[0x7f6c650ec2f0]
/lib/x86_64-linux-gnu/libc.so.6(abort+0x2d6)[0x7f6c650ee036]
/usr/lib/x86_64-linux-gnu/libstdc++.so.6(_ZN9__gnu_cxx27__verbose_terminate_handlerEv+0x16d)[0x7f6c65a0006d]
/usr/lib/x86_64-linux-gnu/libstdc++.so.6(+0x5eee6)[0x7f6c659fdee6]
/usr/lib/x86_64-linux-gnu/libstdc++.so.6(+0x5ef31)[0x7f6c659fdf31]
/usr/lib/libentwine.so(+0x59beb)[0x7f6c66812beb]

Here's my memory log just before failure:

root@sigma:/opt/data/MNT# free -m
             total       used       free     shared    buffers     cached
Mem:         16491      16343        147          0          0         14
-/+ buffers/cache:      16328        162
Swap:         4095       4004         91

If I create a new Entwine index with only this pointcloud file, it work well. I hope you will be able to reproduce this problem

S3 driver: following provider-chain and picking up credentials via IAM role?

Hi Connor,

Next question for you. Similar to issue #34 I am receiving Encountered an error: No driver for s3://<bucket>/<prefix>/*

I suspect the difference between issue #34 and my scenario is that I am running entwine build on EC2 not on an on-premises machine, thus explicit AWS keys will never be provided. It is expected that the application will follow the default credential provider chain and, in my case, be granted access to the bucket via an IAM role under which the EC2 instance is running.

For example, on EC2 this just works:

sudo docker run garland/aws-cli-docker aws s3 ls

Is entwine expecting explicit AWS access keys as per the doc for the S3 backend?

Many thanks in advance.

Is text based tree hierarchy information able to be generated from entwine?

Is it possible to generate some form of text based hierarchy information about the entwine laz outputs? In my testing I'm successfully generating plenty of .LAZ files and they're clearly split into a great tree structure, however I'd really like to gather information on the relationships between the different files to perform some further processing via some python scripts I'm creating. Ideally I'd like the parent/child relationships between each file. The generated .json files (eg entwine-manifest) does not appear to contain this information. I'm guessing it perhaps lives in the 0 file in the h (hierarchy?) folder, however it is binary so it is largely not usable to me at this stage. Likewise, I can't see any options in the documentation for generating information like this. Of course I could delve into the source code and see if there is something I can expose, however I thought I'd check here first as others may have solved this.

logic_error thrown without absolute flag is set

I run entwine build with the cesium config inside docker and got this error:

Got error 11
terminate called after throwing an instance of 'std::logic_error'
  what():  basic_string::_M_construct null not valid
Aborted (core dumped)

with data from http://geoportal.geoportal-th.de/hoehendaten/LAS/las_2014-2019/las_658_5654_1_th_2014-2019.zip

i rebuild the docker image and found that in da88b83 the absolute setting was removed from the cesium config. At that commit the image worked.

when setting the absolute flag, the process completed too
entwine build /var/entwine/config/cesium.json -i las_658_5654_1_th_2014-2019.laz -o cs_local -r EPSG:25832 EPSG:4978 -n

Creating big 3d-tiles

Hi,

Is it possible to proceed creating 3d tiles index for cesium if previoosly command was interrupted (say due to SSH)? If source data is pretty big (hundreds of gb) generating tiles would take long, so ability to continue previous job would be very useful.

thanks,

Alex.

basic_string::_M_construct null not valid

Hello @connormanning . I run into this error when trying to index a point cloud using entwine. It only happens for a specific point cloud. What could it be? This only indexes a portion of the total points in the point cloud. Thanks!
"
Found an SRS
Got error 11
Exception in pool task: basic_string::_M_construct null not valid
During /media/smarty/Storage/Dropbox/NBM_Projects/Project_Rail_0000001/pointclouds/klk.las: Subtraction result was negative (block zero)
Saving hierarchy...
Saving registry...
Saving metadata...

Index completed in 66 seconds.
Save complete. Indexing stats:
Points inserted: 19,537,920
Points discarded:
Outside specified bounds: 0
Overflow past max depth: 0
"

running out of memory during dataset inference

I have a bunch of 500MB to 1GB laz files in google cloud storage, I'm treating them like http sources, but when I run entwine on more then 3 or 4 of them at once entwine just dies at the Performing dataset inference step, drastically increasing the memory allowed to docker will cause it to work.

So a couple related questions:

• is there a way to pre run the inference stuff?
• am I misreading the documentation and I should just be running it on each image one at a time and merging it into a big pyramid?

work poor with poor machine

Excuse me, I user entwine to index about 6G data, with my 5 year old notebook.
I limit docker only 3G memory, or docker would drain my memory.
but it has not finish after 12 hour, only output 166M data, and only message:

Adding 0 - gcp0824.las
        Pushes complete - joining...

my command:

docker run -it -v /home/data:/home/data -u 1000 \
    --cpus=2 -m 3g --oom-kill-disable \
    connormanning/entwine build -t 4 -f \
        -i /home/data/gcp0824.las -o /home/data/ncku

does my computer so poor?
how many memory i need to deal with 6G data?
my notebook has only 4G memory.

by the way, it take about 40s to index sample data.

greyhound and s3

Hi there,

does anybody know how to connect greyhound to s3, so that it serves index created on s3?

thanks,

Alex.

Text input files

I've been trying to index some data, with the eventual goal of serving it with Greyhound, but so far I've failed to get the data into a format that Entwine can parse.

Our data is in a pgpointcloud database, but prior to ingest, it was written to csv text files. Somewhere it said that Entwine could parse anything that is PDAL readable, however when I supply the csv directory as input -i csv/ I get an error: Encountered an error: No point cloud files found. Is it possible to use text files as input?

Having failed with text file input, I decided to convert the files to las format myself using PDAL. I did that, but when I run the build I get this error: Cannot use this scale for these bounds. My guess is that something's wrong with the las files - the min/max values in the header are all zero:

"maxx": 0,
"maxy": 0,
"maxz": 0,
"minx": 0,
"miny": 0,
"minz": 0,
"scale_x": 1,
"scale_y": 1,
"scale_z": 1,

If you have any thoughts or suggestions, they would be greatly appreciated.

entwine infer bounding box

When I run 'entwine infer ...' over a directory of files it returns a bounding box which captures the extent of all data in that directory. If I add '-r EPSG:native EPSG:desired', the bounding box returned by infer seems to be correct in the EPSG:desired system.

I then set up a bash script to add one LAZ tile at a time to an entwine build, using the bounding box for the whole dataset, in the EPSG:desired CRS:

for file in dir:
entwine build -i file -o output_dir -r EPSG:native EPSG:desired -h -b [min, max xyz of desired CRS] -x -t (Nphysicalcores-2)

I know this will take time due to the -x option, but after the first tile is inserted entwine returns '0 points inserted, Npoints out of bounding box'. This seems strange, since the bounding box provided should capture all the data in the directory. Is that how the -b option is meant to work? or am I messing with things by using -x as well?

Running out of Memory

Connor,

Thank you again for taking my call two weeks ago. [for the benefit of others] I explained that I had a very large data set that I hope to index with entwine (70,000 + files, 18+ TB compressed, 3.7 trillion points, spanning the entire US). I've been encouraged by your demonstration https://av.tib.eu/media/20366 of comparable data sets (denser but more confined than mine) in Iowa and the Netherlands. You advised me to do two things with my set. One, I should break things up into multiple resources, rather attempting to put the entire US into a single resource. And two, that entwine takes better advantage of parallelization when the individual files are broken into smaller pieces that what mine were in at the time. I wrote code to do just that, and decided to break up my set into individual resources along Web Mercator zoom level 6 tile boundaries (which for reference are just a little larger than the state of Iowa). My code also breaks my source files into smaller pieces (along WM zoom level 16 boundaries), ranging in size from 1k (containing a single point) to 81Mb (15,317,899 points), and averaging somewhere in the 30Mb range. At the moment I'm focused on processing my most modest "tile" pictured here at zoom 6 and zoom 8:

This set consist of 3714 laz files at 27.8 GB. I am not currently imposing the tile boundary (by defining the bounds with the Entwine '-b' parameter) though that is my hope in the future so that I can add data to some of these resources if needed. This resource is quite modest in comparison to those you have demoed. The data itself is not overly dense or unusual.

The problem is that I keep running out of memory every time I try to index this set. I am running this on a single Ubuntu Box (though I have four at my disposal) inside of docker. The box has 60GB of ram and 30 cores. I've tried running the job as subsets all the way to 64 pieces but the result is always the same. Things seem to start out running at a decent clip, but eventually slow as the entire 60GB of physical ram and then the 25GB page file is completely consumed. I'm processing this all on my local network.

The command I'm using is this:

sudo docker run -it -v /media/lidar-share/data connormanning/entwine build -i /data/source_split/MyData/9_39 -o /data/MyData/9_39 -t 30

I have not played with the nullDepth or baseDepth parameters, but I do need this to be lossless.

Do you have any other suggestions? Your help is much appreciated. I've got some time to complete processing on the whole of my job, but I'm getting a little pressure to be able to provide a demo on at least this small portion.

Thanks again for you help and excellent work,

-John

Question: Point streaming not supported for stage readers.ply

I am unable to index a point cloud in binary PLY. More specifically, running:

DATA=/path/to/ply/
docker run -it -v $DATA:/opt/data connormanning/entwine build \
  -i /opt/data/my.ply \
  -o /opt/data/entwine

I get:

Performing dataset inference...
1 / 1: /opt/data/my.ply
Exception in pool task: Point streaming not supported for stage readers.ply.
Cannot use this scale for these bounds
Encountered an error: Zero points found
Exiting.

A similar result from infer:

docker run -it -v $DATA:/opt/data connormanning/entwine infer /opt/data/my.ply
Inferring from: /opt/data/my.ply
	Temp path: tmp
	Threads: 4
	Reprojection: (none)
	Trust file headers? yes
1 / 1: /opt/data/my.ply
Exception in pool task: Point streaming not supported for stage readers.ply.
Cannot use this scale for these bounds
Encountered an error: Zero points found
Exiting.

Expected? Thank you in advance.

Exception in pool task: Unable to fetch data and convert as requested: Blue:uint16_t(31232) -> unsigned char

Hi,

am trying to build index for cesium 3d tiles with following command:

docker run -it -v h:/:/opt/data connormanning/entwine build /opt/data/arrow_laz_index/cesium.json -i /opt/data/arrow_laz/e3085n69655.laz -o /opt/data/arrow_laz_index/data

{
   "absolute":true,
   "reprojection":{
      "out":"EPSG:4326"
   },
   "formats":{
      "cesium":{
         "tilesetSplit":8,
         "geometricErrorDivisor":16.0
      }
   }
}

on windows 10.
In the output directory only empty folder are created, and powershell output is full of those 'Unable to fetch data and convert as requested' entries.

Also in the output it says it gonna convert into 4978, but in cesium.conf 4326 is set, may be it is related.
any ideas what could it be?

thanks,

Alex.

Merge command line

Hi

I try entwine the merge command line on my Linux server but I've got an error. Is there an option I should add in the build command to succeed?

# docker run -it -v /opt/data/greyhound/:/data connormanning/entwine entwine build -r EPSG:2950 EPSG:3857 -s 1 4 -i /data/270_5035.las -o /data/270
# docker run -it -v /opt/data/greyhound/:/data connormanning/entwine entwine build -r EPSG:2950 EPSG:3857 -s 2 4 -i /data/270_5036.las -o /data/270
# docker run -it -v /opt/data/greyhound/:/data connormanning/entwine entwine build -r EPSG:2950 EPSG:3857 -s 3 4 -i /data/270_5037.las -o /data/270
# docker run -it -v /opt/data/greyhound/:/data connormanning/entwine entwine build -r EPSG:2950 EPSG:3857 -s 4 4 -i /data/270_5038.las -o /data/270
# docker run -it -v /opt/data/greyhound/:/data connormanning/entwine entwine merge /data/270
Waking up base
Merging /data/270...
    1 / 4 done.
    2 / 4Waking up base
 merging...Encountered an error: Invalid manifest paths
Exiting.

Thank you

Question: Cesium 3D Tiles output

I'm currently trying to produce a Cesium 3D Tiles output using the following:

docker run -it --rm -v /mylasfiles:/mylasfiles -v $HOME:$HOME connormanning/entwine build /var/entwine/config/cesium.json -i "/mylasfiles/job295165_46092_68_05.laz" -o ~/entwine/output

Am I correct in assuming that I cannot (and don't need to) use Greyhound to serve the ~/entwine/output folder? A simple static node.js/python http server should do, right?

By running Greyhound I can access a URL such as: http://localhost:8080/resource/superior/info, but I can't find an URL mapping for the Cesium path (I looked through the code and I think this is correct, but wanted to confirm).

Thanks!

Ignore the Z Axis in bounds?

I would like to be able to specify a bound in X & Y but not specify, or have ignored the Z limitations. Is this possible or something that can be easily implemented? Currently I am specifying large positive and negative Z bounds to ensure all data is captured, and whilst I doubt I will be processing data for any mountain ranges any time soon, it doesn't feel like the right approach?

Thanks

Protocol "s3" not supported or disabled in libcurl

Hello, I am trying to index a local point cloud directly into an S3 bucket. However, I am getting a failed PUT attempt (see command at the end). When running with "curl_verbose=1", I get:

Protocol "s3" not supported or disabled in libcurl

Any ideas on how to solve this? Thanks!

sudo docker run -it -v ~/.aws:/root/.aws -e "CURL_VERBOSE=1" -v $HOME:$HOME connormanning/entwine build
-i /path/to/point/cloud
-o s3://path/to/bucket/folder

Failure when too much threads

Hi, I set threads option too hight when I tried to process large volume of Pointcloud files. entwine failed before the end of the process. No error message, and number of files processed was not the same each time. After lowered threads, I succeed to process all files (545).

PDAL pipelines

what do you think about integrating pdal pipelines?

for now i preprocess DataIN with a PDAL pipeline and then build with entwine

• DataIN -> PDAL pipeline -> Data -> Entwine build -> Output

maybe it is possible to cut the intermediate data

• DataIN -> Entwine build with PDAL pipeline -> Output

Compilation Error Ubuntu 16.04 LTS

Hey dude
Just tried compiling the package from the master branch

Had some compilation issues surrounding laszip

         from /home/evan/Libraries/entwine/entwine/util/executor.cpp:22:

/usr/local/include/pdal/io/LasZipPoint.hpp:40:22: fatal error: laszip.hpp: No such file or directory
compilation terminated.
entwine/util/CMakeFiles/util.dir/build.make:86: recipe for target 'entwine/util/CMakeFiles/util.dir/executor.cpp.o' failed
make[2]: *** [entwine/util/CMakeFiles/util.dir/executor.cpp.o] Error 1
CMakeFiles/Makefile2:474: recipe for target 'entwine/util/CMakeFiles/util.dir/all' failed
make[1]: *** [entwine/util/CMakeFiles/util.dir/all] Error 2
Makefile:149: recipe for target 'all' failed
make: *** [all] Error 2

The fix was pretty easy, just added
include_directories(/home/evan/Libraries/LAStools/LASzip/src/)

Had LAStools paired with PotreeConverter, followed their installation before
I found this.

Thought I'd leave that there as a fix for anyone with similar issues

Last docker entwine updates : error :

I just update my docker entwine image and get this entwine command error message whatever the docker run command I try to launch (ex : docker run -it --rm connormanning/entwine /bin/bash)
Is it a regression or did I miss something ??

Invalid kernel type
Usage: entwine
Kernels:
build
Build (or continue to build) an index
infer
Aggregate information for an unindexed dataset
merge
Merge colocated previously built subsets

Wrong inferred boundary

This is what pdal info gives:

 "native":
      {
        "bbox":
        {
          "maxx": 138795.726,
          "maxy": 448132.326,
          "maxz": 78.417,
          "minx": 138505.454,
          "miny": 447756.511,
          "minz": -28.904
        },

But this is what entwine makes of it:

Performing dataset inference...
1 / 1: /var/data/geocensus/zfs_0.las
Correcting malformed Bounds
Inferred: [(-1000000000.00000, -1000000000.00000, -1000000000.00000), (138505.45                                                                          400, 447756.51100, -28.90400)]

Setting it by hand seems to fix the issue.
Unfortunately, I can't share the dataset.

entwine build appears to ignore threads directive

if I say 'entwine build -i ... -o ... -t 2', entwine says:

Transforming inference
Inferred: [(-8497.0259, -18951.106, -4258.3861), (9192.7131, 16498.431, 643.58951)]

Version: 1.1.0
Input:
Files: 79
Point count hint: 4,793,770,294 points
Threads: 5
Output:

The same happens if my json config file says "threads": 2

Why am I trying to limit the number of threads? I'm happy for my build to take a while, but I am a little memory-constrained. Related to this, entwine seems to consume a lot of memory - is it trying to hold the entire dataset in memory at once? Or for 4.7 bill points should I be doing subset builds? or should I just stump up for more RAM ;) Sorry, more than one question per issue there...

Error during build

Connor, your documentation seems to imply that it's possible to add LiDAR files to already indexed data. So for example I run entwine on 'a.laz' into folder 'MyData'. Once that's finished I run entwine on 'b.laz' also into folder 'MyData', and entwine quickly comes back with this error:

sudo docker run -it -v //media/lidar-share/:/data connormanning/entwine build -i /data/b.laz -o /data/MyData
terminate called after throwing an instance of 'std::out_of_range'
what(): vector::_M_range_check: __n (which is 1) >= this->size() (which is 1)
Got error 11
terminate called recursively

I have been experimenting with Entwine solidly for a couple of weeks. I'm aware that I can process these files together with a single command (but here I'm specifically testing the ability to add to an existing index). I've also tried specifying a 'Bounds' (that encompasses both files) while processing 'a.laz', with the same result.

Both laz files are in EPSG:3857, and are geographically close to one another (but perhaps 30 miles apart).

Any idea what I'm doing wrong?

Thank you in advance.

Crash on building cesium tiles with coloring:intensity

Latest PDAL and Entwine (today's build).

Report:

Adding 0 - /tmp/118100-485100-118200-485200/pointcloud.las
 A: 0 C: 1 H: 3
        Pushes complete - joining...
Found an SRS
terminate called after throwing an instance of 'pdal::pdal_error'
  what():  Unable to fetch data and convert as requested: Intensity:uint16_t(15865) -> unsigned char
Aborted (core dumped)

Pointcloud:
http://research.geodan.nl/sites/cesium/data/pointcloud.tar.gz

Entwine config:

{
    "absolute": true,
    "reproject": {
        "out": "EPSG:4978"
    },
    "formats": {
        "cesium": {
            "tilesetSplit": 8,
            "coloring": "intensity",
            "geometricErrorDivisor": 16.0
        }
    }
}

This only happens when choosing "coloring":"intensity". "coloring":"tile" or no coloring at all works correct. Previous version of entwine gave only 'black' color on intensity.