Giter Site home page Giter Site logo

geoq's Introduction

geoq

Build Status crates.io

Geoq is a command-line tool for working with geospatial data.

It combines common GIS serialization formats with utilities for manipulating and visualizing data on a map.

For example:

# Print the geometry and the set of level 2 geohashes which covers the given geometry
$ echo '{"type":"Polygon","coordinates":[[[-34,38],[-37,32],[-23,33],[-34,38]]]}' | geoq gh covering 2 -o
{"type":"Polygon","coordinates":[[[30,10],[40,40],[20,40],[10,20],[30,10]]]}
eq
en
em
ej

# Feed that output into a map on geojson.io
$ echo '{"type":"Polygon","coordinates":[[[-34,38],[-37,32],[-23,33],[-34,38]]]}' | geoq gh covering 2 -o | geoq map

See the Manual for more examples and available commands.

Install

Geoq is installed via cargo, the Rust package manager, and requires nightly rust.

If you have all this set up, you can just run cargo install geoq.

To install Rust and the Cargo toolchain:

curl https://sh.rustup.rs -sSf | sh
rustup toolchain install nightly
rustup default nightly
cargo install geoq

You'll also need to add Cargo's bin directory to your path:

# e.g. in your ~/.bash_profile or other shell profile
export PATH="$HOME/.cargo/bin:$PATH"

Updating an Existing Installation

To pull and install a newer version from crates.io, run:

cargo install geoq --force

Supported Input Formats

Geoq will detect the following GIS input formats automatically:

  • Comma-separated Lat/Lon: 34.0,-118.0
  • Geohashes (base 32): 9q5
  • WKT: POINT (-118.0, 34.0)
  • GeoJSON: {"type": "Point", "coordinates": [-118.0, 34.0]}
  • H3 Cells in Hex String format: 8c274996e1683ff

One Feature Per Line, One Line Per Feature

Geoq processes text inputs on a per-line basis, and it expects inputs not to stretch across multiple lines.

This sometimes causes problems, especially with GeoJSON, because many JSON processing tools like to output pretty-printed JSON in a multi-line format.

One way to fix this problem with pretty-printed GeoJSON is to use the jq tool:

echo '{
    "type": "Point",
    "coordinates": [30, 10]
}
' | jq -c . | geoq map

Commands

See the built-in command help using geoq --help or geoq <subcommand> --help for more detailed information on these:

  • bbox - Give bounding boxes for geometries, or for a stream of geometries collectively
  • centroid - Cet the centroid of a geometry
  • filter - Spatial predicate filtering
    • intersects - Select features intersecting a given query geometry
    • contains - Select features contained by a given query geometry
  • gh - Geohash subcommands
    • children - Get children of a geohash
    • covering - Output geohashes that "cover" a geometry
    • neighbors - Get neighbors of a Geohash
    • point - Output base 32 Geohash for a given Lat,Lon
  • gj - GeoJSON subcommands
    • f - Output geometry as GeoJSON feature
    • geom - Output geometry as GeoJSON geometry
    • fc - Collect all input geometries into a GeoJSON Feature Collection
  • json - JSON -> GeoJSON coercion
    • munge - Attempt to convert arbitrary JSON to a GeoJSON Feature.
  • map - Visualization with geojson.io
  • measure - Measurement subcommands
    • distance - Measure distances between features
    • coord-count - Give the number of vertices in geometries
  • read - Debugging / format validation
  • shp - Convert shapefiles to GeoJSON
  • simplify - Simplify geometries, either with fixed threshold or iteratively toward target coord-count
  • whereami - Output IP geolocation-based current lat/lon as GeoJSON
  • wkt - Output geometries as WKT
  • fgb - Working with flatgeobuf
    • write - write flatgeobuf files from GeoJSON lines to STDIN
    • read - read flatgeobuf files to GeoJSON with optional bbox filter
  • h3 - Working with H3 spatial grid system
    • children- Get children for h3 cell(s)
    • covering - Generate set of H3 cells covering a geometry.
    • from-str - Convert h3 hexadecimal string IDs to 64-bit numeric ids
    • grid-disk - Get disk of given radius around given cells
    • hierarchy - Output all h3 cells for a given point, from res 0 to 15
    • parent - Get parent (or ancestor) for cells
    • point - Get H3 cell for a point
    • resolution - Get resolution for an H3 cell
    • to-str - Convert 64-bit numeric h3 index its hexadecimal string representation

See the Manual for more examples and available commands.

Development

Running Tests

cargo test

Building / Releasing

cargo publish
git tag release/<VERSION>
git push origin release/<VERSION>

geoq's People

Contributors

jleedev avatar stanislav-tkach avatar worace avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar

geoq's Issues

Failure to install on rust docker container

I've tried various combinations of the stable branch, the nightly branch, the --locked argument and no locked argument to cargo install but I can't seem to get geoq to install correctly.

typescript.txt

This also fails on another container I'm creating, but I figured going to the official rust container would be the best environment to replicate it with.

bug in gh neighbors

echo 9g3m | geoq gh neighbors
9g3m
9g3r
9g3r
9g3m
9g37
9g37
9g35
9g3j
9g3p

Fix compiler warnings breaking CI

Apparently I have slept on enough rust versions that some of the warnings are now showing up in the stderr output for one of the CI tests. I would like to update all of this stuff anyway (stuff like using the new ? syntax instead of try!) but this is a good prompt to do it.

Provide gh option to match "filter contains"

If I'm not mistaken, these two commands produce the same output.

$ geoq gh roots | geoq gh children | geoq filter intersects -q ./united_states.geojson | sort | uniq | wc -l
43
$ geoq gh covering 2 < ./united_states.geojson | sort | uniq | wc -l
43

But there is no corresponding option to the gh subcommand to find the geohashes that are fully contained within the entity:

$ geoq gh roots| geoq gh children | geoq filter contains -q ./united_states.geojson | sort | uniq | wc -l
6
$ geoq gh ??? 

And it would be useful if such a command could output more and more detailed geohashes (up to a given number of digits) that would fall within said entity. For example, output the 6 2-digit geohashes from the command above, and then break the missing 37 geohashes from the first two commands into their 32 3-digit geohashes and see which of those still match geoq filter contains.

Optionally output geohash name in properties

When looking at the map of the geohashes in something like geojson.io, it would be useful if I could see what the name of each hash was. If the hash was stored in the properties of each hash, you would be able to click a tile and see what the name of it is.

Implement ability to filter by sub-geohash

I recently realized that the order of children geohashes is such that a range of them splits the parent geohash nicely. For instance aa0-aag and aah-aaz perfectly bisect the aa geohash. This is true for the 16 children just mentioned, 8 children with aa0-aa7, 4 children with aa0-aa3, and so on. It would be great if qeoq filter contains <gh> could take a range like aa0-aa7 and filter to something more granular than the major geohashes of 1,2,3,etc degrees.

Add command for generating "point radius" polygons

Something like:

echo 34.0,-118.0 | geoq point-radius 50

Where 50 indicates meters.

And get out a polygon (probably default to GeoJSON) approximating the circle of that radius at that point.

Optionally we could take an argument to specify the number of points used for the circle-ish polygon, but I think for starters it's fine to just pick a number.

Eventually it would be cool to support a general-purpose buffer command for all geometries, but the implementation of that is somewhat complex (JTS version is here for reference) and it doesn't look like there is an existing rust implementation of it yet. So starting with just points seems fine and still useful.

Add "json feature" subcommand for coercing arbitrary JSON into GeoJSON Features

One common and annoying use-case when working with Geo data is receiving some arbitrary JSON or CSV data that contains geometry information along with related metadata, but is not formatted as actual GeoJSON.

It would be cool to support some "best-guess" coercions for this kind of data that would look for a few common keys in the input and use that to coerce the data into valid GeoJSON. The rough idea would be:

  • Look for geometric information in a handful of common keys (maybe wkt, geometry, geojson are the obvious ones)
  • Pull out this information and read a feature from it
  • Convert that into a GeoJSON feature
  • Insert all the remaining, non-geometry keys from the input into the properties field of the new feature

We recently added a command similar to this for extracting common Lat/Lon keys into a GeoJSON Point: geoq json point, so that can be used as a reference.

I think for now we can support only JSON as the input format. I think CSV is also a common use-case, but there should be other comand line tools out there which can convert CSV to JSON, so that could be used in conjunction with this.

Add geoq json point subcommand

This would be a "fuzzy search" to convert arbitrary json into geojson by checking for lat/lon/latitude/longitude/lat/lng keys. Read those into coordinates and then the rest of the input JSON into properties.

Support remaining geometry types in filter command

Currently I've added:

  • Point
  • LineString
  • Polygon
  • MultiPolygon

The geo_types::Geometry enum contains this full set:

  • Point
  • Line
  • LineString
  • Polygon
  • MultiPoint
  • MultiLineString
  • MultiPolygon
  • GeometryCollection

To make this complete, I would need to implement the intersection check in geoq::intersection for the remaining types.

It's also possible that some of these might not be necessary due to the types of inputs geoq allows. Currently we take:

  • Lat/Lon (Point)
  • Geohash (Polygon)
  • WKT (Point, LineString, Polygon, MultiPoint, MultiLineString, MultiPolygon, GeometryCollection)
  • GeoJSON (Point, MultiPoint, LineString, MultiLineString, Polygon, MultiPolygon, GeometryCollection)

However it may end up being simpler to just implement the cross-product for all of the variants and then not have to worry about any gaps.

Improve error message on "bad" geojson input

Somehow, I have forgotten on more than one occasion that geojson entities must be one per line. The error one gets when feeding it a pretty geojson file is a little confusing when one is working late at night. It would be great if the error message could reinforce the requirement that entities must be on a single line.

[b-jazz:~] 1 $ geoq gh covering 1  < map.geojson.oneliner
c
[b-jazz:~] $ geoq gh covering 1  < map.geojson.pretty
Error parsing geojson: { - Encountered malformed JSON.
Error parsing geojson: "properties": {}, - Encountered malformed JSON.
Error parsing geojson: "geometry": { - Encountered malformed JSON.
Application error: InvalidGeoJSON

Move debug command to read and add readme details

This would be a useful place to insert more detailed information about valid input formats, so running geoq read --help would print this info. Then other commands could reference this as needed, e.g. see geoq read --help for more info on formats

Installation requires nightly?

error[E0554]: `#![feature]` may not be used on the stable release channel
 --> /home/brendon/.cargo/registry/src/github.com-1ecc6299db9ec823/geoq-0.0.22/src/main.rs:1:1
  |
1 | #![feature(try_blocks)]
  | ^^^^^^^^^^^^^^^^^^^^^^^

For more information about this error, try `rustc --explain E0554`.
error: could not compile `geoq` due to previous error
error: failed to compile `geoq v0.0.22`, intermediate artifacts can be found at `/tmp/cargo-installA3hv9G`

Resolved with (at least for my purposes)

rustup toolchain install nightly
rustup default nightly
cargo install geoq

Invalid json from fgb read

Fixed in geozero 0.9.6: georust/geozero#87

¶ geoq --version 
geoq 0.0.24
¶ printf '{"type":"Feature","properties":{"name":"\\""},"geometry":{"type":"Point","coordinates":[-80,40.44]}}
' | geoq fgb write a.fgb 
¶ geoq fgb read a.fgb 
{
"type": "FeatureCollection",
"name": "L1",
"features": [{"type": "Feature", "properties": {"name": """}, "geometry": {"type": "Point", "coordinates": [-80,40.44]}}]}

Add `whereami` endpoint to get current lat/lon by IP

$ curl ip-api.com
{
  "country"     : "United States",
  "countryCode" : "US",
  "region"      : "CA",
  "regionName"  : "California",
  "city"        : "Culver City",
  "district"    : "",
  "zip"         : "90230",
  "lat"         : 33.9949,
  "lon"         : -118.3991,
  "timezone"    : "America/Los_Angeles",
  "isp"         : "Spectrum",
  "org"         : "Charter Communications",
  "as"          : "AS20001 Time Warner Cable Internet LLC",
  "mobile"      : false,
  "proxy"       : false,
  "query"       : "104.172.231.16"
}

Convert CI to github actions

This is still running some old travis setup which is probably deprecated or something anyway. And it fails half the time due to out of date rust versions etc. Would be nice to move it over if possible.

Command to transform one coordinate system to another

A have a ton of data that is in EPSG:3857 (Mercator) format and need to translate them to EPSG:4326 (WGS84) format. Also, it could be handy to transform from decimal degrees to degrees/minutes/seconds and various formats like that.

Add "measure distance" command

This would be the first of a handful of measure subcommands I'd like to add. The idea would be to feed in a target feature along with 1 or more input features whose distance from the target will be reported.

Input Interface

I think the interface might look something like:

echo 9q5 |  geoq measure distance 34.13681,-117.9162

For reading the command-line target argument, we can use the same input-reading support geoq uses for the rest of its processing, so the input could be a WKT, Geohash, GeoJSON, or lat,lon. There is an example of this in the filter subcommand.

Output

For the output, I think the most useful thing may be to include the distance along with the measured geometry in a tab-separated format. This will enable subsequent processing with other unix tools like cut, sort, etc.

So something like:

123.0       9q5

Measurement

  • Would report in meters
  • If both geometries are points, it can be a simple point-to-point measurement
  • Otherwise use the Closest Point algorithm from the geo crate to pick points, then do point to point
  • VincentyDistance can be used for the distance measurement

Easy way to find the "root" geohash children

It's easy enough to generate the children of a given geohash:

echo "0" | geoq gh children
00
...
0z

but figuring out the single character doesn't seem to be possible:

echo "" | geoq gh children
Application error: UnknownEntityFormat

There is a hack, but it's a hack:

echo "0" | geoq gh children | cut -c2
0
...
z

Options/Subcommand ordering in help text is backwards

Example where the commands are specified in the order according to the help text:

[b-jazz:~] 1 $ geoq gh roots | geoq  filter --query-file ~/osm/osmi2/forty-seven-states-boundary.geojson intersects                     
Must provide Query Features as either --file or positional argument.
Application error: MissingArgument

Example with the "correct" order:

[b-jazz:~] 1 $ geoq gh roots | geoq  filter  intersects --query-file ~/osm/osmi2/forty-seven-states-boundary.geojson                    
9
c
d
f

Help text:

[b-jazz:~] $ geoq gh roots | geoq  filter help
geoq-filter
Select features based on geospatial predicates               

USAGE:
    geoq filter [OPTIONS] [SUBCOMMAND]

FLAGS:
    -h, --help       Prints help information
    -V, --version    Prints version information

OPTIONS:
    -q, --query-file <query-file>    Input file for reading query feature(s).

SUBCOMMANDS:
    contains      Output only entities (from STDIN) which fall within a QUERY entity (as command-line ARG)
    help          Prints this message or the help of the given subcommand(s)
    intersects    Output only entities (from STDIN) which intersect a QUERY entity (as command-line ARG)

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.