visr/lasio.jl

Julia package for reading and writing the LAS lidar format.

LasIO

Julia package for reading and writing the LAS lidar format.

This is a pure Julia package for reading and writing ASPRS .las files. Currently only LAS versions 1.1 - 1.3 and point formats 0 - 3 are supported. For LAZ support see below.

If the file fits into memory, it can be loaded using

using FileIO, LasIO
header, points = load("test.las")

where header is of type LasHeader, and, if it is point format 3, points is a Vector{LasPoint3}. LasPoint3 is an immutable that directly corresponds to the binary data in the LAS file. Use functions like xcoord(p::LasPoint, header::LasHeader) to take out the desired items in the point.

If the file does not fit into memory, it can be memory mapped using

using FileIO, LasIO
header, points = load("test.las", mmap=true)

where points is now a memory mapped PointVector{LasPoint3} which behaves in the same way as the Vector{LasPoint3}, but reads the points on the fly from disk when indexed, not allocating the complete vector beforehand.

See test/runtests.jl for other usages.

LAZ support

We advise to use LazIO, which works out of the box and is compatible with LasIO.

The compressed LAZ format is supported by LasIO itself, but requires the user to make sure the laszip executable can be found in the PATH. LAZ files are piped through laszip to provide reading and writing capability. laszip is not distributed with this package. One way to get it is to download LAStools from https://rapidlasso.com/. The LAStools ZIP file already contains laszip.exe for Windows, for Linux or Mac it needs to be compiled first. When this is done this should work just like with LAS:

using FileIO, LasIO
header, points = load("test.laz")

lasio.jl's Issues

Julia v1.2 support?

I'm not sure if this is just me, but I can't get LasIO to install on Julia 1.2:

(v1.2) pkg> add LasIO
 Resolving package versions...
ERROR: Unsatisfiable requirements detected for package LasIO [570499db]:
 LasIO [570499db] log:
 ├─possible versions are: [0.1.0, 0.2.0-0.2.1] or uninstalled
 ├─restricted to versions * by an explicit requirement, leaving only versions [0.1.0, 0.2.0-0.2.1]
 └─restricted by compatibility requirements with FixedPointNumbers [53c48c17] to versions: uninstalled — no versions left
   └─FixedPointNumbers [53c48c17] log:
     ├─possible versions are: [0.5.0-0.5.3, 0.6.0-0.6.1] or uninstalled
     └─restricted to versions 0.6.1 by an explicit requirement, leaving only versions 0.6.1

I also note that this package hasn't used the new Project.toml files, but rather the older REQUIRE format.

Core dump inside LasIO

First of all thank you for all your work on what is a useful package! I'm using it somewhat intensely. Unfortunately I'm getting an intermittent core dump failure occurring within the package while garbage collecting. An example stack trace is below. I'm running julia 1.2.0 on a ubuntu 14.04 machine. In sudo code I'm doing the following within a julia program

1. create an empty point cloud pc
2. Loop through the below for up to 9 point cloud files
   • download from cloud a laz version 1.4 file
   • use lastools to downgrade to version 1.3
   • load las file using LasIO
   • extract a small section of the points and add to pc
3. calculate some features from pc
   repeat 1-3 many times for different laz files

The code may run for thousands of iterations without failure and then suddenly fail when loading a laz file or it may fail on the first iteration. If it fails once and I restart the program the chance of failure increases. The longer it runs the less likely a failure becomes.

Any help you might be able to give would be appreciated. Unfortunately to date I've been unable to reproduce this error in a reliable fashion.

GC error (probable corruption) :
Allocations: 256134561 (Pool: 256090359; Big: 44202); GC: 387

!!! ERROR in jl_ -- ABORTING !!!
0x7f43ceb97010: r-- Stack frame 0x7f43ce994568 -- 16 of 159 (direct)
0x7f43ceb97038: `- Array in object 0x7f43eceed960 :: 0x7f43ef36b553 -- [0x75682a8, 0x756a0a0)
of type Array{String, 2}

signal (6): Aborted
in expression starting at REPL[3]:1
__libc_signal_restore_set at /build/glibc-OTsEL5/glibc-2.27/signal/../sysdeps/unix/sysv/linux/nptl-signals.h:80 [inlined]
raise at /build/glibc-OTsEL5/glibc-2.27/signal/../sysdeps/unix/sysv/linux/raise.c:48
abort at /build/glibc-OTsEL5/glibc-2.27/stdlib/abort.c:79
gc_assert_datatype_fail at /buildworker/worker/package_linux64/build/src/gc.c:1531
gc_mark_loop at /buildworker/worker/package_linux64/build/src/gc.c:2419
_jl_gc_collect at /buildworker/worker/package_linux64/build/src/gc.c:2729
jl_gc_collect at /buildworker/worker/package_linux64/build/src/gc.c:2912
jl_gc_pool_alloc at /buildworker/worker/package_linux64/build/src/gc.c:1111
read at ./refvalue.jl:8 [inlined]
read at ./none:0
unknown function (ip: 0x7f43c6e40b90)
jl_apply_generic at /buildworker/worker/package_linux64/build/src/gf.c:2191
load at /home/greg/.julia/packages/LasIO/TCFYi/src/fileio.jl:36
#9 at /home/greg/.julia/packages/LasIO/TCFYi/src/fileio.jl:65 [inlined]
open at ./process.jl:681
load at /home/greg/.julia/packages/LasIO/TCFYi/src/fileio.jl:64
jl_apply_generic at /buildworker/worker/package_linux64/build/src/gf.c:2191
jl_apply at /buildworker/worker/package_linux64/build/src/julia.h:1614 [inlined]
jl_f__apply at /buildworker/worker/package_linux64/build/src/builtins.c:563
jl_f__apply_latest at /buildworker/worker/package_linux64/build/src/builtins.c:601
#invokelatest#1 at ./essentials.jl:790 [inlined]
invokelatest at ./essentials.jl:789 [inlined]
#load#27 at /home/greg/.julia/packages/FileIO/I1ONY/src/loadsave.jl:184
load at /home/greg/.julia/packages/FileIO/I1ONY/src/loadsave.jl:169
jl_apply_generic at /buildworker/worker/package_linux64/build/src/gf.c:2191
#load#13 at /home/greg/.julia/packages/FileIO/I1ONY/src/loadsave.jl:118
load at /home/greg/.julia/packages/FileIO/I1ONY/src/loadsave.jl:118 [inlined]
#load_las#91 at /home/greg/.julia/packages/RoamesGeometry/V2mGP/src/pointcloud_io.jl:261
#load_las at ./none:0 [inlined]
#load_pointcloud#66 at /home/greg/.julia/packages/RoamesGeometry/V2mGP/src/pointcloud_io.jl:7
#load_pointcloud at ./none:0 [inlined]
macro expansion at /home/greg/CatenaryStatistics/src/Utils.jl:183 [inlined]
macro expansion at /home/greg/.julia/packages/Retry/0jMye/src/repeat_try.jl:192 [inlined]
get_pointcloud at /home/greg/CatenaryStatistics/src/Utils.jl:181
processTile at /home/greg/CatenaryStatistics/src/points_on_catenaries.jl:280
unknown function (ip: 0x7f43c6e1a2a3)
jl_apply_generic at /buildworker/worker/package_linux64/build/src/gf.c:2191
processCampaign at /home/greg/CatenaryStatistics/src/points_on_catenaries.jl:406
processCampaign at /home/greg/CatenaryStatistics/src/points_on_catenaries.jl:328 [inlined]
processCampaign at /home/greg/CatenaryStatistics/src/points_on_catenaries.jl:328
unknown function (ip: 0x7f43c6de2fac)
jl_apply_generic at /buildworker/worker/package_linux64/build/src/gf.c:2197
do_call at /buildworker/worker/package_linux64/build/src/interpreter.c:323
eval_value at /buildworker/worker/package_linux64/build/src/interpreter.c:411
eval_stmt_value at /buildworker/worker/package_linux64/build/src/interpreter.c:362 [inlined]
eval_body at /buildworker/worker/package_linux64/build/src/interpreter.c:772
jl_interpret_toplevel_thunk_callback at /buildworker/worker/package_linux64/build/src/interpreter.c:884
unknown function (ip: 0xfffffffffffffffe)
unknown function (ip: 0x7f43ecad6b8f)
unknown function (ip: (nil))
jl_interpret_toplevel_thunk at /buildworker/worker/package_linux64/build/src/interpreter.c:893
jl_toplevel_eval_flex at /buildworker/worker/package_linux64/build/src/toplevel.c:815
jl_toplevel_eval_flex at /buildworker/worker/package_linux64/build/src/toplevel.c:764
jl_toplevel_eval_in at /buildworker/worker/package_linux64/build/src/toplevel.c:844
eval at ./boot.jl:330
jl_apply_generic at /buildworker/worker/package_linux64/build/src/gf.c:2191
eval_user_input at /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.2/REPL/src/REPL.jl:86
run_backend at /home/greg/.julia/packages/Revise/0KQ7U/src/Revise.jl:1033
#85 at ./task.jl:268
jl_apply_generic at /buildworker/worker/package_linux64/build/src/gf.c:2197
jl_apply at /buildworker/worker/package_linux64/build/src/julia.h:1614 [inlined]
start_task at /buildworker/worker/package_linux64/build/src/task.c:596
unknown function (ip: 0xffffffffffffffff)
Allocations: 256134561 (Pool: 256090359; Big: 44202); GC: 387
Aborted (core dumped)

more name flame war - LazIO headers vs LasIO headers

I kept getting errors when I tried to call header.x_scale_factor on a .las header before I realized there are many fields names that change between LasIO and LazIO. It would be nice to synchronize these.

🔥 oorlog van vlammen 🔥

using FileIO, LasIO
lasfn = joinpath(dirname(pathof(LasIO)), "..", "test/libLAS_1.2.las")
header, points = load(lasfn)

julia> fieldnames(typeof(header))
(:file_source_id, :global_encoding, :guid_1, :guid_2, :guid_3, :guid_4, :version_major, :version_minor, :system_id, :software_id, :creation_dayofyear, :creation_year, :header_size, :data_offset, :n_vlr, :data_format_id, :data_record_length, :records_count, :point_return_count, :x_scale, :y_scale, :z_scale, :x_offset, :y_offset, :z_offset, :x_max, :x_min, :y_max, :y_min, :z_max, :z_min, :variable_length_records, :user_defined_bytes)

using LazIO
lazfn = joinpath(dirname(pathof(LazIO)), "..", "test/libLAS_1.2.laz")
pointcloud = LazIO.open(lazfn)

julia>fieldnames(typeof(pointcloud.header))
(:file_source_ID, :global_encoding, :project_ID_GUID_data_1, :project_ID_GUID_data_2, :project_ID_GUID_data_3, :project_ID_GUID_data_4, :version_major, :version_minor, :system_identifier, :generating_software, :file_creation_day, :file_creation_year, :header_size, :offset_to_point_data, :number_of_variable_length_records, :point_data_format, :point_data_record_length, :number_of_point_records, :number_of_points_by_return, :x_scale_factor, :y_scale_factor, :z_scale_factor, :x_offset, :y_offset, :z_offset, :max_x, :min_x, :max_y, :min_y, :max_z, :min_z, :start_of_waveform_data_packet_record, :start_of_first_extended_variable_length_record, :number_of_extended_variable_length_records, :extended_number_of_point_records, :extended_number_of_points_by_return, :user_data_in_header_size, :user_data_in_header, :vlrs, :user_data_after_header_size, :user_data_after_header)

TagBot trigger issue

This issue is used to trigger TagBot; feel free to unsubscribe.

If you haven't already, you should update your TagBot.yml to include issue comment triggers.
Please see this post on Discourse for instructions and more details.

If you'd like for me to do this for you, comment TagBot fix on this issue.
I'll open a PR within a few hours, please be patient!

provide laszip with BinDeps.jl

Hi Martijn, thanks for the package. Regarding the liblaz installation issue you mentioned in the README, you can use BinDeps.jl to get the dependency installed automatically: https://github.com/JuliaLang/BinDeps.jl

Rename "creation_doy" to "creation_dayofyear"

I suspect this is a typo, and should be creation_day rather than creation_doy?

Roadmap for v0.1

Hey,

Thanks for this package!

Was the plan for this package to get registered? What needs to be done to get there?

Thanks,

Josh

API for returning scaled and offset point cloud

I was messing around and ran into a situation where I wanted to get the scaled and offset float point cloud to use with NearestNeighbors.jl.

I can use the the convert function provided by the package, but then I have an Array{Point{3,Float64},1} and lose access to all the metadata and other convenience functions like classification().

using FileIO, LasIO
header, points = load("points.las")

using GeometryTypes
pc_scaled = map(x-> Base.convert(Point{3, Float64},x, header), points) #Array{Point{3,Float64},1}

The code above works fine for my toy problem, but I wanted to ask about design thoughts for adding this.

The ability to expose scaled and offset coords to users was discussed briefly in this LasIO issue.

After reading through src a bit, it looks like it would at least require a new type at least since all the x,y,z fields are ::Int? Not sure if any of the work over in GeometryBasics/GeometryTypes can be leveraged to simplify anything.

Maybe related issue in PointCloudRasterizers? and I also know about GeoAcceleratedArrays.

LAZ support

Hi,
this looks like a great package - unfortunately all my data is in laz not las format. Are there any plans to incorporate laz format at some point, or should I go to things like laslib?
Thanks!

LasDatasets.jl Package Discussion

Hi guys,

First of all, I wanted to say thanks for the amazing package!

For a bit of context, my team has been using LasIO historically quite a bit, until a year or two ago someone made a forked version to incorporate some v1.4 compatibility and application-specific stuff. Since the functionality here began to diverge from LasIO's current API/functionality, we then made it into a new package and used this internally. Recently, I managed to get permission to make this open source so that we could start collaborating and sharing with the wider community on this. The code for the new package, LAS.jl, is found here

For now I've kept it as a separate package since as I've mentioned it's got quite a different API and some different features (e.g. custom user fields, a LasPy-like interface for storing Julia structs as VLRs, etc.) and I don't want to step on any toes!

I'm opening this issue as a discussion on how we want to work together going forward and any ideas/concerns on your end. I think there are a couple of ways we can move forward:

• We extract the functionality from LAS.jl and integrate it here as a major version update
• I can make you collaborators on LAS.jl and we can start making modifications there and eventually shift to using that repo

Happy to discuss this in more detail here. I really appreciate the work you've all done on this package and the last thing I would want is to offend or get in anyone's way 🙂

Generated method for read is slower than non-generated one

While simpler version benchmarked linked at JuliaIO/StructIO.jl#13 (https://gist.github.com/evetion/2b57d6105cca39b2d3c6ef670a5cc393) is not.

Original function
  85.615 ms (994614 allocations: 32.26 MiB)
New function using generated read
  105.606 ms (994614 allocations: 32.26 MiB)```

UndefVarError accessing LasIO.epsg_code (with 'import LasIO')

It's buried in src/srs.jl, but it's used in a test like LasIO.epsg_code (and the readme points to the tests for usage).