connormanning / entwine Goto Github PK
View Code? Open in Web Editor NEWEntwine - point cloud organization for massive datasets
Home Page: https://entwine.io
License: Other
Entwine - point cloud organization for massive datasets
Home Page: https://entwine.io
License: Other
Hello again @connormanning
I'm running entwine for a newer dataset, however, it's not inserting all the points.
I have attached the dataset here.
AIRS.zip
docker run -it \
-v $HOME:$HOME \
connormanning/entwine \
build \
-f \
-i AIRS.las \
-o ~/entwine/airs-default/ \
-r EPSG:4326 EPSG:4978
Performing dataset inference...
1 / 1: AIRS.las
Inferred: [(-90, -180, 0), (89, 179, 23)]
Version: 1.1.0
Input:
File: AIRS.las
Point count hint: 1,555,200 points
Density estimate (per square unit): 24.2013
Threads: 8
Output:
Output path: AIRS/
Data storage: laszip
Scale: 0.01
Offset: (10, 10, 20)
XYZ width: 4
Metadata:
Native bounds: [(-90, -180, 0), (89, 179, 23)]
Cubic bounds: [(-200, -200, -190), (220, 220, 230)]
Reprojection: (from file headers, or a default of 'EPSG:4326') -> EPSG:4978
Storing dimensions: [
X, Y, Z, Intensity, ReturnNumber
NumberOfReturns, ScanDirectionFlag, EdgeOfFlightLine, Classification, ScanAngleRank
UserData, PointSourceId, GpsTime, Red, Green
Blue, OriginId
]
Adding 0 - ~/entwine/airs-default/
Pushes complete - joining...
Saving hierarchy...
Saving registry...
Saving metadata...
Index completed in 3 seconds.
Save complete. Indexing stats:
Points inserted: 756,045
Points discarded:
Outside specified bounds: 781,920
Overflow past max depth: 0
Process Completed.
Here is how it looks like in MeshLab
Any idea why is it doing that? Thanks!
See #41. Entwine should more accurately estimate data density during the dataset inference phase instead of using only the numPointsHint
and the overall bounds of the data. This should be exposed via output metadata and subsequently by Greyhound.
Trying to build locally, and getting the following:
root@ip-172-31-25-29:~/src/entwine/makefiles# cmake -G "Unix Makefiles" ../
-- The C compiler identification is GNU 4.8.4
-- The CXX compiler identification is GNU 4.8.4
-- Check for working C compiler: /usr/bin/cc
-- Check for working C compiler: /usr/bin/cc -- works
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - done
-- Check for working CXX compiler: /usr/bin/c++
-- Check for working CXX compiler: /usr/bin/c++ -- works
-- Detecting CXX compiler ABI info
-- Detecting CXX compiler ABI info - done
-- Looking for include file pthread.h
-- Looking for include file pthread.h - found
-- Looking for pthread_create
-- Looking for pthread_create - not found
-- Looking for pthread_create in pthreads
-- Looking for pthread_create in pthreads - not found
-- Looking for pthread_create in pthread
-- Looking for pthread_create in pthread - found
-- Found Threads: TRUE
-- Found CURL: /usr/lib/x86_64-linux-gnu/libcurl.so (found version "7.35.0")
CMake Error at CMakeLists.txt:68 (export):
export Unknown arguments.
-- Configuring incomplete, errors occurred!
See also "/root/src/entwine/makefiles/CMakeFiles/CMakeOutput.log".
See also "/root/src/entwine/makefiles/CMakeFiles/CMakeError.log".
Hi,
am trying to build 3d-tiles from many source laz files, but entwine stops adding/build source after 4rth file, and command just stops without showing any error. Could it be due to that fact that source laz file are disjount, eg they are not close to each other and make different parts of surface?
thanks,
Alex.
I need to build Entwine to run on Windows desktop, in order to build Cesium 3D Tiles for moderately sized point clouds (around 50-500 MB).
I encountered a lot of errors (2000+) during the build using MSVC. I could fix most of them, although I'm by no mean a C++ expert, so this will probably need a bit of code review. They mainly revolve around the known conflict around the std::min()
/std::max()
clashing with the macros defined in windows.h
, but also some less obvious differences in accessing private members.
The CMake configuration will also probably need some updating to add some paths to dependencies (lzma...)
Maybe also add an AppVeyor CI config.
When I try to compile Entwine on Ubuntu 16.04, I get this error:
In file included from /home/neel/CUSP/sw/entwine/entwine/tree/builder.cpp:37:0:
/home/neel/CUSP/sw/entwine/entwine/util/compression.hpp:145:11: error: ‘LazPerfCompressor’ in namespace ‘pdal’ does not name a template type
I have compiled laz-perf
and PDAL
manually, and am using the UbuntuGIS repo. The gdal
and liblaz
dependencies came from the Ubuntu package liblas-dev
.
I did not use the ubuntu-deps.sh
script as I believe it is for a older version of Ubuntu (~14.04).
Hello @connormanning. Could you please help us resolve this new issue?
Now when running the example code:
sudo docker run -it -v $HOME:$HOME -e "CURL_VERBOSE=1" connormanning/entwine build
-i https://entwine.io/sample-data/red-rocks.laz
-o ~/entwine/red-rocks
I get the error:
and when running it locally:
Get Entwine ready for a 1.0.
How hard would it be get meta data into the cesium format? I would like to have per-point data written to the BatchTable inside the pnts files. Data could be height
or Z
value before the reprojection to "EPSG:4978".
PointCloud#batch-table says its possible without introducing BatchID
in the FeatureTable
Likely a temporary thing, but popped up in the latest pull:
[ 10%] Building CXX object entwine/tree/CMakeFiles/tree.dir/builder.cpp.o
In file included from /usr/include/c++/5/cassert:43:0,
from /var/data/git_repos/entwine/entwine/types/manifest.hpp:14,
from /var/data/git_repos/entwine/entwine/tree/builder.hpp:24,
from /var/data/git_repos/entwine/entwine/tree/builder.cpp:11:
/var/data/git_repos/entwine/entwine/types/pooled-point-table.hpp: In member function ‘virtual void entwine::ConvertingPointTable::getFieldInternal(pdal::Dimension::Id, pdal::PointId, void*) const’:
/var/data/git_repos/entwine/entwine/types/pooled-point-table.hpp:160:20: error: ‘d’ was not declared in this scope
assert(d->size() == sizeof(double));
^
entwine/tree/CMakeFiles/tree.dir/build.make:62: recipe for target 'entwine/tree/CMakeFiles/tree.dir/builder.cpp.o' failed
make[2]: *** [entwine/tree/CMakeFiles/tree.dir/builder.cpp.o] Error 1
CMakeFiles/Makefile2:160: recipe for target 'entwine/tree/CMakeFiles/tree.dir/all' failed
make[1]: *** [entwine/tree/CMakeFiles/tree.dir/all] Error 2
Makefile:127: recipe for target 'all' failed
make: *** [all] Error 2
I did a fresh pull and removed the old build directory. Then cmake, make...
There is an unexplainable positive z-offset in the output of the cesium tiles (in cesium viewer).
In this image you see the pointcloud floating above another set of vector tiles:
Things I checked:
I also compared the transform matrix given in the tileset.json from entwine with the one that would be calculated by cesium based on the minx, miny values from the las file with:
var matrix = Cesium.Transforms.eastNorthUpToFixedFrame(Cesium.Cartesian3.fromDegrees(4.84539353916,52.3523793903,-2.219))
resulting in these differences:
Entwine transform:
[
-0.084535880369591351, 0.99642043582522855, -6.9388939039072299e-18, 0,
-0.78699076737683538, -0.066767957551816193, 0.61334131762663679, 0,
0.61114582301915321, 0.051849348252612899, 0.78981797149217947, 0,
3889839.5983011033, 330012.31519960327, 5027057.5454196455, 1
]
Cesium tranform:
[
-0.08446730511458142, 0.9964262513436106, 0, 0,
-0.7889526244257994, -0.06687971333399081, 0.6108034547675762, 0,
0.6086205967417826, 0.05159292177889329, 0.7917822551964607, 0,
3890035.6951631806, 329759.3088569205, 5026846.2020282885, 1
]
I was hoping the cesium transform would be correct (it is for the vector data you can see in the image) but unfortunately although clamping the pointset to the ground it introduces an offset in -X and -Y direction of some 200 meters. Also I'm not sure if the problem is really in the transform matrix or already happens in the reprojection in entwine.
Worth noting: a large part of the points is below sea level and have a negative z-value.
The corresponding las file can be found at: http://research.geodan.nl/sites/cesium/data/pointcloud.tar.gz
When building an entwine index on a 22404422 point laz file:
(lldb) bt
* thread #1: tid = 0x1d2062, 0x00000001000b7018 libentwine.dylib`entwine::Node::insertInto(arbiter::Endpoint const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >, unsigned long) [inlined] std::__1::__tree<std::__1::__value_type<BigUint, entwine::Node::AnchoredNode>, std::__1::__map_value_compare<BigUint, std::__1::__value_type<BigUint, entwine::Node::AnchoredNode>, std::__1::less<BigUint>, true>, std::__1::allocator<std::__1::__value_type<BigUint, entwine::Node::AnchoredNode> > >::__move_assign(std::__1::__tree<std::__1::__value_type<BigUint, entwine::Node::AnchoredNode>, std::__1::__map_value_compare<BigUint, std::__1::__value_type<BigUint, entwine::Node::AnchoredNode>, std::__1::less<BigUint>, true>, std::__1::allocator<std::__1::__value_type<BigUint, entwine::Node::AnchoredNode> > >&, std::__1::integral_constant<bool, true>) + 15 at __tree:1350, queue = 'com.apple.main-thread', stop reason = EXC_BAD_ACCESS (code=EXC_I386_GPFLT)
frame #0: 0x00000001000b7018 libentwine.dylib`entwine::Node::insertInto(arbiter::Endpoint const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >, unsigned long) [inlined] std::__1::__tree<std::__1::__value_type<BigUint, entwine::Node::AnchoredNode>, std::__1::__map_value_compare<BigUint, std::__1::__value_type<BigUint, entwine::Node::AnchoredNode>, std::__1::less<BigUint>, true>, std::__1::allocator<std::__1::__value_type<BigUint, entwine::Node::AnchoredNode> > >::__move_assign(std::__1::__tree<std::__1::__value_type<BigUint, entwine::Node::AnchoredNode>, std::__1::__map_value_compare<BigUint, std::__1::__value_type<BigUint, entwine::Node::AnchoredNode>, std::__1::less<BigUint>, true>, std::__1::allocator<std::__1::__value_type<BigUint, entwine::Node::AnchoredNode> > >&, std::__1::integral_constant<bool, true>) + 15 at __tree:1350
frame #1: 0x00000001000b7009 libentwine.dylib`entwine::Node::insertInto(arbiter::Endpoint const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >, unsigned long) [inlined] std::__1::__tree<std::__1::__value_type<BigUint, entwine::Node::AnchoredNode>, std::__1::__map_value_compare<BigUint, std::__1::__value_type<BigUint, entwine::Node::AnchoredNode>, std::__1::less<BigUint>, true>, std::__1::allocator<std::__1::__value_type<BigUint, entwine::Node::AnchoredNode> > >::operator=(std::__1::__tree<std::__1::__value_type<BigUint, entwine::Node::AnchoredNode>, std::__1::__map_value_compare<BigUint, std::__1::__value_type<BigUint, entwine::Node::AnchoredNode>, std::__1::less<BigUint>, true>, std::__1::allocator<std::__1::__value_type<BigUint, entwine::Node::AnchoredNode> > >&&) at __tree:1409
frame #2: 0x00000001000b7009 libentwine.dylib`entwine::Node::insertInto(arbiter::Endpoint const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >, unsigned long) [inlined] std::__1::map<BigUint, entwine::Node::AnchoredNode, std::__1::less<BigUint>, std::__1::allocator<std::__1::pair<BigUint const, entwine::Node::AnchoredNode> > >::operator=(this=0x00000001bd5f10e0 size=4294967295) at map:961
* frame #3: 0x00000001000b7009 libentwine.dylib`entwine::Node::insertInto(this=<unavailable>, ep=0x00007fff5fbfe700, postfix="", step=8) + 377 at hierarchy.cpp:127
frame #4: 0x00000001000b9758 libentwine.dylib`entwine::Hierarchy::toJson(this=0x0000000101a0a550, ep=0x00007fff5fbfe700, postfix="") + 88 at hierarchy.cpp:361
frame #5: 0x00000001000923c7 libentwine.dylib`entwine::Builder::saveOwnProps(this=0x0000000101a07f50) const + 407 at builder.cpp:599
frame #6: 0x00000001000902fb libentwine.dylib`entwine::Builder::save(this=0x0000000101a07f50) + 75 at builder.cpp:551
frame #7: 0x0000000100090004 libentwine.dylib`entwine::Builder::go(this=0x0000000101a07f50, max=<unavailable>) + 1252 at builder.cpp:311
frame #8: 0x0000000100005c56 entwine`Kernel::build(args=<unavailable>) + 8806 at build.cpp:530
frame #9: 0x0000000100007f5e entwine`main(argc=<unavailable>, argv=<unavailable>) + 1214 at entwine.cpp:82
frame #10: 0x00007fff8d55a5ad libdyld.dylib`start + 1
frame #11: 0x00007fff8d55a5ad libdyld.dylib`start + 1
Looks like something's amiss in the move-assignment operator but I'm not sure what yet — thought I'd open in case you had ideas. LMK if I can provide more info on the files, or such.
I guess this is undesired:
works:
entwine build -i "/var/data/geocensus/*.las" -o /var/data/entwine/geocensus
does not work:
entwine build -i /var/data/geocensus/*.las -o /var/data/entwine/geocensus
produces:
Encountered an error: Invalid argument: /var/data/geocensus/zfs_10.las
Exiting.
Whenever I try to run Entwine in docker, I keep getting the same error, regardless of the data set used.
$ docker run -it -v $HOME:$HOME connormanning/entwine build
-i /C/Users/psawall/entwine/Input/i35_MN_LiDAR_Colored.laz
-o ~/entwine/Output/i35_MN_LiDAR_Colored
Performing dataset inference...
1 / 1: /C/Users/psawall/entwine/Input/i35_MN_LiDAR_Colored.laz
Exception in pool task: Unable to open stream for >'/C/Users/psawall/entwine/Input/i35_MN_LiDAR_Colored.laz' with error 'No such file or directory'
Cannot use this scale for these bounds
Encountered an error: Zero points found
Exiting.
Support alternative compression types:
I don't know if these should be implemented in PDAL or Entwine.
On latest github version.
Building with pdal 1.3, with lazperf.
[ 91%] Building CXX object test/CMakeFiles/entwine-test.dir/unit/build.cpp.o
/var/data/git_repos/entwine/test/unit/build.cpp:38:9: error: ‘PointId’ is not a member of ‘{anonymous}::D {aka pdal::Dimension::Id}’
D::PointId, D::OriginId
^
Hello
I have Ubuntu trusty, 14.04, with docker.io and tried to follow the suggested method to try the repository:
docker run -it -v $HOME:/opt/data connormanning/entwine entwine build -i /opt/data/abc.laz -o /opt/data/entwine/abc
I received an error:
Performing dataset inference...
1 / 1: /opt/data/abc.laz
Exception caught in pool task: Unable to create open stream for '/opt/data/abc.laz' with error 'No such file or directory'
Encountered an error: Zero points found
Exiting.
gcc@gcc:~/bin$ docker run -it -v $HOME:/opt/data connormanning/entwine entwine build -i /opt/data/abc.laz -o /opt/data/entwine/abc
Performing dataset inference...
1 / 1: /opt/data/abc.laz
Exception caught in pool task: Unable to create open stream for '/opt/data/abc.laz' with error 'No such file or directory'
Encountered an error: Zero points found
Exiting.
why could not load the abc.laz?
Regards
Ibrahim
Thanks for the great tool!
I finally finished indexing and visualizing a set of data only to discover that (I think) all of the 14 million points were collapsed onto two lat/lon positions because I was using an incorrect scale factor (X=-45.94672 -> X=-45).
Indexing ~14 million points took ~48 hours on my laptop:
docker run -it -v ~/point-cloud-vis:$HOME connormanning/entwine build -i ~/csv/ -o ~/entwine/Jakobshavn -r EPSG:4326 EPSG:3857
I was wondering if that is the expected performance? I could certainly use a smaller subset while developing to increase the iteration speed, but it might also be useful to have some examples or estimates in the documentation about the time to index 1,000,000 points etc.
Hello - We have had issues with Pix4D (latest version) point clouds displaying only in white, with RGB selected. If we take the point clouds out of Pix4D and runthem into another program, such as Leica Cyclone, and then through Entwine, we are able to see point colors natively. Any input is appreciated. Thanks!
When running the command in the wiki's quickstart:
docker run -it -v $HOME:/opt/data connormanning/entwine \
entwine build -i /opt/data/abc.laz -o /opt/data/entwine/abc
I get
Invalid kernel type
Version: 1.0.0
Usage: entwine <kernel> <options>
Kernels:
build
Build (or continue to build) an index
infer
Aggregate information for an unindexed dataset
merge
Merge colocated previously built subsets
I suspect that the command should be
docker run -it -v $HOME:/opt/data connormanning/entwine \
build -i /opt/data/abc.laz -o /opt/data/entwine/abc
looking at the docs, I should be able to run
docker run -it -v `pwd`:`pwd` -w `pwd` --rm connormanning/entwine build -i s3://iowa-lidar/iowa/ -o ./some/directory
but doing so and I get the error Encountered an error: No driver for s3://iowa-lidar/iowa/*
Entwine 1.1.0 is failing to build on :
with dependences:
First error:
/local/build/entwine/entwine/third/arbiter/arbiter.cpp:2609:47: error: invalid conversion from ‘const void*’ to ‘void*’ [-fpermissive]
if (BIO* bio = BIO_new_mem_buf(s.data(), -1))
...and then, using -fpermissive:
[ 98%] Linking CXX executable entwine
/usr/bin/ld: warning: libjsoncpp.so.11, needed by /usr/lib/gcc/x86_64-redhat-linux/4.8.5/../../../libpdal_base.so, may conflict with libjsoncpp.so.0
/usr/bin/ld: CMakeFiles/kernel.dir/build.cpp.o: undefined reference to symbol '_ZN4Json5ValueC1Em'
/usr/local/lib64/libjsoncpp.so.11: error adding symbols: DSO missing from command line
collect2: error: ld returned 1 exit status
make[2]: *** [kernel/entwine] Error 1
make[1]: *** [kernel/CMakeFiles/kernel.dir/all] Error 2
make[1]: *** Waiting for unfinished jobs....
Not sure what this means. I can't remove centos default jsoncpp, I figured placing a newer version in /usr/local would be the fix - PDAL is linked against the newer jsoncpp just fine.
Advice appreciated!
I tried running entwine on AWS, where I had multiple files in an S3 bucket and I received this error
Encountered an error: Unexpected contents in AWS response
When I changed entwine to run against each file in the bucket individually, they worked fine. What I expected, was that entwine would build it's indexing for all the files in the bucket. This was the behavior I think I remember seeing when running entwine on my local machine. I think I need to use merge but I'm a little confused on how to do it. I'm running this as a DC/OS job which essentially just runs the docker command for building with entwine.
Does entwine support processing of multiple files in a single location within an S3 bucket?
I try to load 15 pointcloud tiles ( source file http://dl.mapgears.com/mg-laz.tar) in an entwine index. Entwine failed to load 3 of those files ( 278-5048_rgb.laz, 278-5047_rgb.laz and 276-5049_rgb.laz).
my script looks like this:
docker run -it -v /opt/data/:/data connormanning/entwine entwine build -i /data/LAS/275-5047_rgb.laz -o /data/greyhound/RDP_RMI -b "[269000, 5034000, -100,308000, 5066000, 150]"
docker run -it -v /opt/data/:/data connormanning/entwine entwine build -i /data/LAS/275-5048_rgb.laz -o /data/greyhound/RDP_RMI
docker run -it -v /opt/data/:/data connormanning/entwine entwine build -i /data/LAS/275-5049_rgb.laz -o /data/greyhound/RDP_RMI
docker run -it -v /opt/data/:/data connormanning/entwine entwine build -i /data/LAS/276-5047_rgb.laz -o /data/greyhound/RDP_RMI
docker run -it -v /opt/data/:/data connormanning/entwine entwine build -i /data/LAS/276-5048_rgb.laz -o /data/greyhound/RDP_RMI
docker run -it -v /opt/data/:/data connormanning/entwine entwine build -i /data/LAS/276-5049_rgb.laz -o /data/greyhound/RDP_RMI
docker run -it -v /opt/data/:/data connormanning/entwine entwine build -i /data/LAS/277-5047_rgb.laz -o /data/greyhound/RDP_RMI
docker run -it -v /opt/data/:/data connormanning/entwine entwine build -i /data/LAS/277-5048_rgb.laz -o /data/greyhound/RDP_RMI
docker run -it -v /opt/data/:/data connormanning/entwine entwine build -i /data/LAS/277-5049_rgb.laz -o /data/greyhound/RDP_RMI
docker run -it -v /opt/data/:/data connormanning/entwine entwine build -i /data/LAS/278-5047_rgb.laz -o /data/greyhound/RDP_RMI
docker run -it -v /opt/data/:/data connormanning/entwine entwine build -i /data/LAS/278-5048_rgb.laz -o /data/greyhound/RDP_RMI
docker run -it -v /opt/data/:/data connormanning/entwine entwine build -i /data/LAS/278-5049_rgb.laz -o /data/greyhound/RDP_RMI
docker run -it -v /opt/data/:/data connormanning/entwine entwine build -i /data/LAS/279-5047_rgb.laz -o /data/greyhound/RDP_RMI
docker run -it -v /opt/data/:/data connormanning/entwine entwine build -i /data/LAS/279-5048_rgb.laz -o /data/greyhound/RDP_RMI
docker run -it -v /opt/data/:/data connormanning/entwine entwine build -i /data/LAS/279-5049_rgb.laz -o /data/greyhound/RDP_RMI
At my third try I've finally received a real error message of bad memory allocation. Dont know why but Ithe first time I've retry to load this file, entwine crashes wihout any error message!
Here's my log
# docker run -it -v /opt/data/:/data connormanning/entwine entwine build -i /data/LAS/278-5048_rgb.laz -o /data/greyhound/RDP_RMI
Continuing previous index...
Input:
Building from 13 source files
Trust file headers? yes
Work threads: 3
Clip threads: 6
Output:
Output path: file:///data/greyhound/RDP_RMI/
Temporary path: tmp/
Compressed output? yes
Tree structure:
Null depth: 6
Base depth: 10
Cold depth: lossless
Mapped depth: 13
Sparse depth: 13
Chunk size: 262144 points
Dynamic chunks? yes
Prefix IDs? no
Build type: hybrid
Point count hint: 13740813 points
Geometry:
Conforming bounds: [(269000.00000, 5034000.00000, -100.00000), (308000.00000, 5066000.00000, 150.00000)]
Cubic bounds: [(268990.00000, 5030490.00000, -19485.00000), (308010.00000, 5069510.00000, 19535.00000)]
Reprojection: (none)
Storing dimensions: [X, Y, Z, Intensity, ReturnNumber, NumberOfReturns, ScanDirectionFlag, EdgeOfFlightLine, Classification, ScanAngleRank, UserData, PointSourceId, GpsTime, Red, Green, Blue, Origin]
Adding 12 - /data/LAS/278-5048_rgb.laz
A: 1048576 C: 1 H: 38
Pushes complete - joining...
Unknown error during /data/LAS/278-5048_rgb.laz
terminate called after throwing an instance of 'std::bad_alloc'
what(): std::bad_alloc
Got error 11
entwine[0x41b00b]
/lib/x86_64-linux-gnu/libc.so.6(+0x352f0)[0x7f6c650ec2f0]
/lib/x86_64-linux-gnu/libc.so.6(abort+0x2d6)[0x7f6c650ee036]
/usr/lib/x86_64-linux-gnu/libstdc++.so.6(_ZN9__gnu_cxx27__verbose_terminate_handlerEv+0x16d)[0x7f6c65a0006d]
/usr/lib/x86_64-linux-gnu/libstdc++.so.6(+0x5eee6)[0x7f6c659fdee6]
/usr/lib/x86_64-linux-gnu/libstdc++.so.6(+0x5ef31)[0x7f6c659fdf31]
/usr/lib/libentwine.so(+0x59beb)[0x7f6c66812beb]
Here's my memory log just before failure:
root@sigma:/opt/data/MNT# free -m
total used free shared buffers cached
Mem: 16491 16343 147 0 0 14
-/+ buffers/cache: 16328 162
Swap: 4095 4004 91
If I create a new Entwine index with only this pointcloud file, it work well. I hope you will be able to reproduce this problem
Hi Connor,
Next question for you. Similar to issue #34 I am receiving Encountered an error: No driver for s3://<bucket>/<prefix>/*
I suspect the difference between issue #34 and my scenario is that I am running entwine build
on EC2 not on an on-premises machine, thus explicit AWS keys will never be provided. It is expected that the application will follow the default credential provider chain and, in my case, be granted access to the bucket via an IAM role under which the EC2 instance is running.
For example, on EC2 this just works:
sudo docker run garland/aws-cli-docker aws s3 ls
Is entwine expecting explicit AWS access keys as per the doc for the S3 backend?
Many thanks in advance.
Is it possible to generate some form of text based hierarchy information about the entwine laz outputs? In my testing I'm successfully generating plenty of .LAZ files and they're clearly split into a great tree structure, however I'd really like to gather information on the relationships between the different files to perform some further processing via some python scripts I'm creating. Ideally I'd like the parent/child relationships between each file. The generated .json files (eg entwine-manifest) does not appear to contain this information. I'm guessing it perhaps lives in the 0 file in the h (hierarchy?) folder, however it is binary so it is largely not usable to me at this stage. Likewise, I can't see any options in the documentation for generating information like this. Of course I could delve into the source code and see if there is something I can expose, however I thought I'd check here first as others may have solved this.
I run entwine build with the cesium config inside docker and got this error:
Got error 11
terminate called after throwing an instance of 'std::logic_error'
what(): basic_string::_M_construct null not valid
Aborted (core dumped)
with data from http://geoportal.geoportal-th.de/hoehendaten/LAS/las_2014-2019/las_658_5654_1_th_2014-2019.zip
i rebuild the docker image and found that in da88b83 the absolute setting was removed from the cesium config. At that commit the image worked.
when setting the absolute flag, the process completed too
entwine build /var/entwine/config/cesium.json -i las_658_5654_1_th_2014-2019.laz -o cs_local -r EPSG:25832 EPSG:4978 -n
Hi,
Is it possible to proceed creating 3d tiles index for cesium if previoosly command was interrupted (say due to SSH)? If source data is pretty big (hundreds of gb) generating tiles would take long, so ability to continue previous job would be very useful.
thanks,
Alex.
Hello @connormanning . I run into this error when trying to index a point cloud using entwine. It only happens for a specific point cloud. What could it be? This only indexes a portion of the total points in the point cloud. Thanks!
"
Found an SRS
Got error 11
Exception in pool task: basic_string::_M_construct null not valid
During /media/smarty/Storage/Dropbox/NBM_Projects/Project_Rail_0000001/pointclouds/klk.las: Subtraction result was negative (block zero)
Saving hierarchy...
Saving registry...
Saving metadata...
Index completed in 66 seconds.
Save complete. Indexing stats:
Points inserted: 19,537,920
Points discarded:
Outside specified bounds: 0
Overflow past max depth: 0
"
I have a bunch of 500MB to 1GB laz files in google cloud storage, I'm treating them like http sources, but when I run entwine on more then 3 or 4 of them at once entwine just dies at the Performing dataset inference
step, drastically increasing the memory allowed to docker will cause it to work.
So a couple related questions:
Excuse me, I user entwine to index about 6G data, with my 5 year old notebook.
I limit docker only 3G memory, or docker would drain my memory.
but it has not finish after 12 hour, only output 166M data, and only message:
Adding 0 - gcp0824.las
Pushes complete - joining...
my command:
docker run -it -v /home/data:/home/data -u 1000 \
--cpus=2 -m 3g --oom-kill-disable \
connormanning/entwine build -t 4 -f \
-i /home/data/gcp0824.las -o /home/data/ncku
does my computer so poor?
how many memory i need to deal with 6G data?
my notebook has only 4G memory.
by the way, it take about 40s to index sample data.
Hi there,
does anybody know how to connect greyhound to s3, so that it serves index created on s3?
thanks,
Alex.
I've been trying to index some data, with the eventual goal of serving it with Greyhound, but so far I've failed to get the data into a format that Entwine can parse.
Our data is in a pgpointcloud database, but prior to ingest, it was written to csv text files. Somewhere it said that Entwine could parse anything that is PDAL readable, however when I supply the csv directory as input -i csv/
I get an error: Encountered an error: No point cloud files found
. Is it possible to use text files as input?
Having failed with text file input, I decided to convert the files to las
format myself using PDAL. I did that, but when I run the build I get this error: Cannot use this scale for these bounds
. My guess is that something's wrong with the las files - the min/max values in the header are all zero:
"maxx": 0,
"maxy": 0,
"maxz": 0,
"minx": 0,
"miny": 0,
"minz": 0,
"scale_x": 1,
"scale_y": 1,
"scale_z": 1,
If you have any thoughts or suggestions, they would be greatly appreciated.
When I run 'entwine infer ...' over a directory of files it returns a bounding box which captures the extent of all data in that directory. If I add '-r EPSG:native EPSG:desired', the bounding box returned by infer seems to be correct in the EPSG:desired system.
I then set up a bash script to add one LAZ tile at a time to an entwine build, using the bounding box for the whole dataset, in the EPSG:desired CRS:
for file in dir:
entwine build -i file -o output_dir -r EPSG:native EPSG:desired -h -b [min, max xyz of desired CRS] -x -t (Nphysicalcores-2)
I know this will take time due to the -x option, but after the first tile is inserted entwine returns '0 points inserted, Npoints out of bounding box'. This seems strange, since the bounding box provided should capture all the data in the directory. Is that how the -b option is meant to work? or am I messing with things by using -x as well?
Connor,
Thank you again for taking my call two weeks ago. [for the benefit of others] I explained that I had a very large data set that I hope to index with entwine (70,000 + files, 18+ TB compressed, 3.7 trillion points, spanning the entire US). I've been encouraged by your demonstration https://av.tib.eu/media/20366 of comparable data sets (denser but more confined than mine) in Iowa and the Netherlands. You advised me to do two things with my set. One, I should break things up into multiple resources, rather attempting to put the entire US into a single resource. And two, that entwine takes better advantage of parallelization when the individual files are broken into smaller pieces that what mine were in at the time. I wrote code to do just that, and decided to break up my set into individual resources along Web Mercator zoom level 6 tile boundaries (which for reference are just a little larger than the state of Iowa). My code also breaks my source files into smaller pieces (along WM zoom level 16 boundaries), ranging in size from 1k (containing a single point) to 81Mb (15,317,899 points), and averaging somewhere in the 30Mb range. At the moment I'm focused on processing my most modest "tile" pictured here at zoom 6 and zoom 8:
This set consist of 3714 laz files at 27.8 GB. I am not currently imposing the tile boundary (by defining the bounds with the Entwine '-b' parameter) though that is my hope in the future so that I can add data to some of these resources if needed. This resource is quite modest in comparison to those you have demoed. The data itself is not overly dense or unusual.
The problem is that I keep running out of memory every time I try to index this set. I am running this on a single Ubuntu Box (though I have four at my disposal) inside of docker. The box has 60GB of ram and 30 cores. I've tried running the job as subsets all the way to 64 pieces but the result is always the same. Things seem to start out running at a decent clip, but eventually slow as the entire 60GB of physical ram and then the 25GB page file is completely consumed. I'm processing this all on my local network.
The command I'm using is this:
sudo docker run -it -v /media/lidar-share/data connormanning/entwine build -i /data/source_split/MyData/9_39 -o /data/MyData/9_39 -t 30
I have not played with the nullDepth or baseDepth parameters, but I do need this to be lossless.
Do you have any other suggestions? Your help is much appreciated. I've got some time to complete processing on the whole of my job, but I'm getting a little pressure to be able to provide a demo on at least this small portion.
Thanks again for you help and excellent work,
-John
I am unable to index a point cloud in binary PLY. More specifically, running:
DATA=/path/to/ply/
docker run -it -v $DATA:/opt/data connormanning/entwine build \
-i /opt/data/my.ply \
-o /opt/data/entwine
I get:
Performing dataset inference...
1 / 1: /opt/data/my.ply
Exception in pool task: Point streaming not supported for stage readers.ply.
Cannot use this scale for these bounds
Encountered an error: Zero points found
Exiting.
A similar result from infer
:
docker run -it -v $DATA:/opt/data connormanning/entwine infer /opt/data/my.ply
Inferring from: /opt/data/my.ply
Temp path: tmp
Threads: 4
Reprojection: (none)
Trust file headers? yes
1 / 1: /opt/data/my.ply
Exception in pool task: Point streaming not supported for stage readers.ply.
Cannot use this scale for these bounds
Encountered an error: Zero points found
Exiting.
Expected? Thank you in advance.
Hi,
am trying to build index for cesium 3d tiles with following command:
docker run -it -v h:/:/opt/data connormanning/entwine build /opt/data/arrow_laz_index/cesium.json -i /opt/data/arrow_laz/e3085n69655.laz -o /opt/data/arrow_laz_index/data
{
"absolute":true,
"reprojection":{
"out":"EPSG:4326"
},
"formats":{
"cesium":{
"tilesetSplit":8,
"geometricErrorDivisor":16.0
}
}
}
on windows 10.
In the output directory only empty folder are created, and powershell output is full of those 'Unable to fetch data and convert as requested' entries.
Also in the output it says it gonna convert into 4978, but in cesium.conf 4326 is set, may be it is related.
any ideas what could it be?
thanks,
Alex.
Hi
I try entwine the merge command line on my Linux server but I've got an error. Is there an option I should add in the build command to succeed?
# docker run -it -v /opt/data/greyhound/:/data connormanning/entwine entwine build -r EPSG:2950 EPSG:3857 -s 1 4 -i /data/270_5035.las -o /data/270
# docker run -it -v /opt/data/greyhound/:/data connormanning/entwine entwine build -r EPSG:2950 EPSG:3857 -s 2 4 -i /data/270_5036.las -o /data/270
# docker run -it -v /opt/data/greyhound/:/data connormanning/entwine entwine build -r EPSG:2950 EPSG:3857 -s 3 4 -i /data/270_5037.las -o /data/270
# docker run -it -v /opt/data/greyhound/:/data connormanning/entwine entwine build -r EPSG:2950 EPSG:3857 -s 4 4 -i /data/270_5038.las -o /data/270
# docker run -it -v /opt/data/greyhound/:/data connormanning/entwine entwine merge /data/270
Waking up base
Merging /data/270...
1 / 4 done.
2 / 4Waking up base
merging...Encountered an error: Invalid manifest paths
Exiting.
Thank you
I'm currently trying to produce a Cesium 3D Tiles output using the following:
docker run -it --rm -v /mylasfiles:/mylasfiles -v $HOME:$HOME connormanning/entwine build /var/entwine/config/cesium.json -i "/mylasfiles/job295165_46092_68_05.laz" -o ~/entwine/output
Am I correct in assuming that I cannot (and don't need to) use Greyhound to serve the ~/entwine/output folder? A simple static node.js/python http server should do, right?
By running Greyhound I can access a URL such as: http://localhost:8080/resource/superior/info
, but I can't find an URL mapping for the Cesium path (I looked through the code and I think this is correct, but wanted to confirm).
Thanks!
I would like to be able to specify a bound in X & Y but not specify, or have ignored the Z limitations. Is this possible or something that can be easily implemented? Currently I am specifying large positive and negative Z bounds to ensure all data is captured, and whilst I doubt I will be processing data for any mountain ranges any time soon, it doesn't feel like the right approach?
Thanks
Hello, I am trying to index a local point cloud directly into an S3 bucket. However, I am getting a failed PUT attempt (see command at the end). When running with "curl_verbose=1", I get:
Protocol "s3" not supported or disabled in libcurl
Any ideas on how to solve this? Thanks!
sudo docker run -it -v ~/.aws:/root/.aws -e "CURL_VERBOSE=1" -v $HOME:$HOME connormanning/entwine build
-i /path/to/point/cloud
-o s3://path/to/bucket/folder
Hi, I set threads option too hight when I tried to process large volume of Pointcloud files. entwine failed before the end of the process. No error message, and number of files processed was not the same each time. After lowered threads, I succeed to process all files (545).
what do you think about integrating pdal pipelines?
for now i preprocess DataIN with a PDAL pipeline and then build with entwine
maybe it is possible to cut the intermediate data
Hey dude
Just tried compiling the package from the master branch
Had some compilation issues surrounding laszip
from /home/evan/Libraries/entwine/entwine/util/executor.cpp:22:
/usr/local/include/pdal/io/LasZipPoint.hpp:40:22: fatal error: laszip.hpp: No such file or directory
compilation terminated.
entwine/util/CMakeFiles/util.dir/build.make:86: recipe for target 'entwine/util/CMakeFiles/util.dir/executor.cpp.o' failed
make[2]: *** [entwine/util/CMakeFiles/util.dir/executor.cpp.o] Error 1
CMakeFiles/Makefile2:474: recipe for target 'entwine/util/CMakeFiles/util.dir/all' failed
make[1]: *** [entwine/util/CMakeFiles/util.dir/all] Error 2
Makefile:149: recipe for target 'all' failed
make: *** [all] Error 2
The fix was pretty easy, just added
include_directories(/home/evan/Libraries/LAStools/LASzip/src/)
Had LAStools paired with PotreeConverter, followed their installation before
I found this.
Thought I'd leave that there as a fix for anyone with similar issues
I just update my docker entwine image and get this entwine command error message whatever the docker run command I try to launch (ex : docker run -it --rm connormanning/entwine /bin/bash)
Is it a regression or did I miss something ??
Invalid kernel type
Usage: entwine
Kernels:
build
Build (or continue to build) an index
infer
Aggregate information for an unindexed dataset
merge
Merge colocated previously built subsets
This is what pdal info gives:
"native":
{
"bbox":
{
"maxx": 138795.726,
"maxy": 448132.326,
"maxz": 78.417,
"minx": 138505.454,
"miny": 447756.511,
"minz": -28.904
},
But this is what entwine makes of it:
Performing dataset inference...
1 / 1: /var/data/geocensus/zfs_0.las
Correcting malformed Bounds
Inferred: [(-1000000000.00000, -1000000000.00000, -1000000000.00000), (138505.45 400, 447756.51100, -28.90400)]
Setting it by hand seems to fix the issue.
Unfortunately, I can't share the dataset.
if I say 'entwine build -i ... -o ... -t 2', entwine says:
Transforming inference
Inferred: [(-8497.0259, -18951.106, -4258.3861), (9192.7131, 16498.431, 643.58951)]
Version: 1.1.0
Input:
Files: 79
Point count hint: 4,793,770,294 points
Threads: 5
Output:
The same happens if my json config file says "threads": 2
Why am I trying to limit the number of threads? I'm happy for my build to take a while, but I am a little memory-constrained. Related to this, entwine seems to consume a lot of memory - is it trying to hold the entire dataset in memory at once? Or for 4.7 bill points should I be doing subset builds? or should I just stump up for more RAM ;) Sorry, more than one question per issue there...
Connor, your documentation seems to imply that it's possible to add LiDAR files to already indexed data. So for example I run entwine on 'a.laz' into folder 'MyData'. Once that's finished I run entwine on 'b.laz' also into folder 'MyData', and entwine quickly comes back with this error:
sudo docker run -it -v //media/lidar-share/:/data connormanning/entwine build -i /data/b.laz -o /data/MyData
terminate called after throwing an instance of 'std::out_of_range'
what(): vector::_M_range_check: __n (which is 1) >= this->size() (which is 1)
Got error 11
terminate called recursively
I have been experimenting with Entwine solidly for a couple of weeks. I'm aware that I can process these files together with a single command (but here I'm specifically testing the ability to add to an existing index). I've also tried specifying a 'Bounds' (that encompasses both files) while processing 'a.laz', with the same result.
Both laz files are in EPSG:3857, and are geographically close to one another (but perhaps 30 miles apart).
Any idea what I'm doing wrong?
Thank you in advance.
Latest PDAL and Entwine (today's build).
Report:
Adding 0 - /tmp/118100-485100-118200-485200/pointcloud.las
A: 0 C: 1 H: 3
Pushes complete - joining...
Found an SRS
terminate called after throwing an instance of 'pdal::pdal_error'
what(): Unable to fetch data and convert as requested: Intensity:uint16_t(15865) -> unsigned char
Aborted (core dumped)
Pointcloud:
http://research.geodan.nl/sites/cesium/data/pointcloud.tar.gz
Entwine config:
{
"absolute": true,
"reproject": {
"out": "EPSG:4978"
},
"formats": {
"cesium": {
"tilesetSplit": 8,
"coloring": "intensity",
"geometricErrorDivisor": 16.0
}
}
}
This only happens when choosing "coloring":"intensity". "coloring":"tile" or no coloring at all works correct. Previous version of entwine gave only 'black' color on intensity.
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.