johnjreiser/chupaesri

Tool to suck data from ArcGIS Server and spit it into PostgreSQL

chupaesri's Issues

coordinates for some polygons are truncated

pointed the script from the command line at

http://arcgisserver.semcog.org/arcgis/rest/services/BuildingFootprints/MapServer/2/query

to test it out. here's what the PG layer looks like loaded up in QGIS:

about 100 of the features had one point with the longitude "cut off", like so:

{"type":"MultiPolygon",
"coordinates":
[[[[13506280.5777559,333401.299540699],
[13506310.0249344,333343.036089242],
[13506340.0593832,333358.21620737],
[13506328.0534777,333381.971128613],
[13506367.4163386,333401.865813643],
[13506374.8458005,333387.16568242],
[13506377.1305774,333388.320538059],
[13506418.2496719,333306.963910773],
[13506379.3402231,333287.298556462],
[13506375.7244094,333294.452755913],
[13506361.0564304,333287.039370075],
[13506355.8576116,333297.324803144],
[13506304.769685,333271.503937006],
[13506290.8940289,333298.957349077],
[13506260.234252,333283.461614177],
[13506281.8251313,333240.742454067],
[13506248.2788714,333223.787729681],
[13506209.5357612,333300.443241477],
[13506204.3818897,333297.838254586],
[13506192.8385827,333320.676837265],
[13506199.593832,333324.090879261],
[13506184.5462598,333353.863188982],
[13506214.8070866,333369.157480329],
[13506217.8507218,333363.135170624],
[13506248.6558399,333378.704724416],
[13506231.7276903,333412.198490813],
[13506251.5400263,333422.212270334],
[1350624,333437.130577445], // <<< problem coordinate!
[13506328.9796588,333480.080708683],
[13506341.3802493,333455.545275584],
[13506336.2562336,333452.955708668],
[13506340.6925852,333444.177821532],
[13506329.5485564,333438.545603678],
[13506334.5790682,333428.592847779],
[13506280.5777559,333401.299540699]]]]}

Migrate to Python 3

chupaEsri should be cleaned up and migrated to use Python 3.4+

Support non-spatial data

Improve the library/script to support downloading data without a spatial field.

error: [Errno 10060] A connection attempt failed

I am getting an error socket.gaierror: [Errno 11004] get addrinfo failed.

The input url parameter is -
"https://server.arcgisonline.com/ArcGIS/rest/services/Specialty/Soil_Survey_Map/MapServer/0/?f=pjson"

I have also tried from IP address but getting the error - "error: [Errno 10060] A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond"

Helping hand

Feel free to check out the strategies from "planning downloads" here: https://github.com/koopjs/featureservice/blob/master/index.js#L330-L371

Hope this helps!

chupaESRI crashes with "Invalid control character"

Here's the crash dump. I'll try to narrow down the exact record that's generating the failure.

Suggestions for patching my way around this?

Invalid control character '\n' at: line 63227 column 678 (char 1625940)
Traceback (most recent call last):
  File "./chupaESRI.py", line 266, in <module>
    jp = EsriJSON2Pg(arcjson)
  File "./chupaESRI.py", line 37, in __init__
    self.srcjson = json.loads(jsonstr)
  File "/usr/local/lib/python2.7/site-packages/simplejson/__init__.py", line 516, in loads
    return _default_decoder.decode(s)
  File "/usr/local/lib/python2.7/site-packages/simplejson/decoder.py", line 370, in decode
    obj, end = self.raw_decode(s)
  File "/usr/local/lib/python2.7/site-packages/simplejson/decoder.py", line 400, in raw_decode
    return self.scan_once(s, idx=_w(s, idx).end())
JSONDecodeError: Invalid control character '\n' at: line 63227 column 678 (char 1625940)

Failed on:  /arcgisshared/rest/services/WashCo_Dynamic/MapServer/110/query?where=objectid+>%3D+100001+AND+objectid+<%3D+101000&text=&objectIds=&time=&geometry=&geometryType=esriGeometryEnvelope&inSR=&spatialRel=esriSpatialRelIntersects&relationParam=&outFields=*&returnGeometry=true&maxAllowableOffset=&geometryPrecision=&outSR=&returnIdsOnly=false&returnCountOnly=false&orderByFields=&groupByFieldsForStatistics=&outStatistics=&returnZ=false&returnM=false&gdbVersion=&returnDistinctValues=false&f=pjson