Giter Site home page Giter Site logo

timwaters / mapwarper Goto Github PK

View Code? Open in Web Editor NEW
196.0 22.0 80.0 18.25 MB

free open source public map georeferencer, georectifier and warper

Home Page: http://mapwarper.net

License: MIT License

Ruby 53.72% JavaScript 13.30% HTML 27.81% CSS 0.27% Shell 0.22% SCSS 4.68%
ruby historical-maps wms mosaic postgis tiles openstreetmap georeferencing geotiff georectification

mapwarper's Introduction

Map Warper

Mapwarper is an open source map geo-rectification, warping and georeferencing application. It enables a user to upload an image, a scanned map or aerial photo for example, and by placing control points on a reference map and the image, to warp it, to stretch it to fit.

Map Warper screenshot of main interface

The application can be seen in use at http://mapwarper.net for public use and in library setting at http://maps.nypl.org

The application is a web based crowdsourced geospatial project that enables people and organisations to collaboratively publish images of maps online and digitize and extract vector information from them.

Users rectify, warp or stretch images of historical maps with a reference basemap, assigning locations on image and map that line up with each other. Often these historical maps were in big paper books, and so for the first time they can be stitched together and shown as a whole, in digital format.

Users can crop around the maps, and join them together into mosaics (previously called layers).

By georeferencing the images, they can be warped or georectified to match the locations in space, and used in GIS software and other services. One such use of these warped maps is an application that that helps people digitize, that is, trace over the maps to extract information from them. For example, buildings in 18th Century Manhattan, details changing land use, building type etc. This application is called the Digitizer.

The application runs as a Ruby on Rails application using a number of open source geospatial libraries and technologies, including PostGIS, Mapserver, Geoserver, and GDAL tools.

The resulting maps can be exported as a PNG, GeoTIFF, WMS, Tiles, and KML for use in many different applications.

Groups of maps can be made into "mosaics" that will stitch together the composite map images.

Features

  • Upload image by file or by URL
  • Find and search maps by geography
  • Adding control points to maps side by side
  • Crop maps
  • User commenting on maps
  • Align maps from similar
  • Create mosaics from groups of maps
  • Login via Github / Twitter / OpenStreetMap / Wikimedia Commons
  • OR signup with email and password
  • Export as GeoTiff, PNG, WMS, Tile, KML etc
  • Preview in Google Earth
  • User Groups
  • Map Favourites
  • Social media sharing
  • Bibliographic metatadata creation and export support
  • Multiple georectfication options
  • Keyboard shortcuts for map controls (save point etc)
  • Automagic placement of points based on transform
  • Import CSV of Control points to a map
  • Download CSV of control points
  • API
    • JSON API Specifications
  • Admin tools include
    • User statistics
    • Activity monitoring
    • User administration, disabling
    • Roles management (editor, developer, admin etc)
    • Batch Imports
  • Caching of WMS and Tile via Redis
  • i18n support
    • English
    • Dutch
    • Japanese

Ruby & Rails

  • Rails 4
  • Ruby 2.4

Database

  • Postgresql 8.4+
  • Postgis 1.5+

Installation Dependencies

Check out the Vagrant section lower down in the readme if you want to get started quickly. Also view the lib/vagrant/provision.sh file to see what needs to get installed.

on Ubuntu 14.04 LTS

apt-get install -y ruby ruby-dev postgresql-9.3-postgis-2.1 postgresql-server-dev-all postgresql-contrib build-essential git-core libxml2-dev libxslt1-dev imagemagick libmapserver1 gdal-bin libgdal-dev ruby-mapscript bundler nodejs

Due to a bug with the gdal gem, you may need to disable a few flags from your ruby rbconfig.rb see zhm/gdal-ruby#4 for more information

Then install the gem files using bundler

bundle install

Ubuntu 16.04 LTS

Mapwarper should work on Ubuntu 16.04 - however there are issues with the Ubuntu package of GDAL and potentially with Mapserver (if not using package Ruby, e.g. RVM)

GDAL needs to be compiled from source to ensure the gdal_rasterize bug is fixed. It should be installed locally and can exist with the package maintainers version. Then point to this newly compiled path in the application.yml file.

If rvm is being used, ruby mapscript for mapserver should be compiled from source, and then linked or installed into the path. You can use the ubuntu package rubymapscript along with the system rub (2.3.1) without worrying about this.

See ubuntu16_installnotes for some hints as to what to do. The vagrant file and provisioning scripts should be altered, ideally.

Configuration

Create and configure the following files

  • config/secrets.yml
  • config/database.yml
  • config/application.yml

In addition have a look in config/initializers/application_config.rb for some other paths and variables, and config/initializers/devise.rb for devise and omniauth

Database creation

Create a postgis database

psql mapwarper_development -c "create extension postgis;"

Database initialization

Creating a new user

user = User.new
user.login = "super"
user.email = "[email protected]"
user.password = "your_password"
user.password_confirmation = "your_password"
user.save
user.confirmed_at = Time.now
user.save

role = Role.find_by_name('super user')
user = User.find_by_login('super')

permission  = Permission.new
permission.role = role
permission.user = user
permission.save

role = Role.find_by_name('administrator')
permission = Permission.new
permission.role = role
permission.user = user
permission.save

WMS/Tile Caching

To enable caching, install Redis and enable caching in the environment file. You may want to configure the redis.conf as appropriate to your server. For example turning off saving to disk and setting a memory value for LRU "maxmemory 2000mb" "maxmemory-policy allkeys-lru" keeps the redis server having 2gig and expires keys based on a least used algorithm.

Development

Via Vagrant - There is a vagrantfile you can use this uses a provision script in lib/vagrant. Type

vagrant up

to get and install the virtual machine - this will also install the libraries and depencies and ruby gems for mapwarper into the virtual machine. See the file in lib/vagrant/provision.sh for more details about this process

After that runs, type vagrant ssh to login and then you can

cd /srv/mapwarper
rails c

Create a user in the console, as shown above and then exit

rails s -b 0.0.0.0 -p 3000

to start the server, running on port 3000

Note that there may be some hoops to jump through if you choose to use the ubuntu 16 vagrant option.

##I18n Locales / Translations

You might want to use LocaleApp to assist with translations.

See the mapwarper project.

API

See README_API.md for API details

mapwarper's People

Contributors

dependabot[bot] avatar duhaime avatar grischard avatar imohkay avatar kamataryo avatar kdough03 avatar l34marr avatar ppival avatar solebox avatar timwaters avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

mapwarper's Issues

allow for georectified files

I would like to be able to rectify the files offline and upload the finished file for wms hosting,
please provide the option to load the rectify points from the gdalinfo

new gdal breaks with -co compress=jpeg use lzw instead

From this:
gdalwarp -rn -dstalpha -srcnodata '17 17 17' -s_srs 'EPSG:4326' /home/tim/work/warper/mapwarper/public/mapimages/dst/1_temp.vrt /home/tim/work/warper/mapwarper/public/mapimages/dst/1.tif -co TILED=YES -co COMPRESS=JPEG -co JPEG_QUALITY=85

To this

gdalwarp -rn -dstalpha -srcnodata '17 17 17' -s_srs 'EPSG:4326' /home/tim/work/warper/mapwarper/public/mapimages/dst/1_temp.vrt /home/tim/work/warper/mapwarper/public/mapimages/dst/1.tif -co TILED=YES -co COMPRESS=LZW

remove email activation

Remove / make configurable the email activation links.

Emails only need to be valid if the user has forgotten their password - we don't need to send two emails "thanks for signing up" and "click this link to activate".

Merge in those monkey patches!

merge in the monkey patches into a commit or three before deployment to mapwarper.net

Just a personal TODO as no one else reads these issues :)

return transform is not solvable gdalwarp errors to user

Error gdal warp scriptERROR 1: Failed to compute GCP transform: Transform is not solvable

for example using these 6 gcps, which should be enough for s 2nd order polynomial

{"stat":"ok","items":[{"updated_at":"2012/08/14 21:56:35 +0200","created_at":"2012/08/14 21:47:51 +0200","error":0.816682560432144,"y":1680.0,"x":727.0,"id":2123,"soft":null,"name":null,"lon":-51.7,"lat":-24.2333333333,"map_id":354},{"updated_at":"2012/08/14 21:56:41 +0200","created_at":"2012/08/14 21:48:56 +0200","error":0.353007727329283,"y":1679.5,"x":2332.5,"id":2124,"soft":null,"name":null,"lon":-51.6833333333,"lat":-24.2333333333,"map_id":354},{"updated_at":"2012/08/14 21:56:45 +0200","created_at":"2012/08/14 21:49:44 +0200","error":1.05510514949195,"y":1678.5,"x":3939.0,"id":2125,"soft":null,"name":null,"lon":-51.6666666667,"lat":-24.2333333333,"map_id":354},{"updated_at":"2012/08/14 21:56:49 +0200","created_at":"2012/08/14 21:50:50 +0200","error":0.571015485246359,"y":1680.0,"x":5546.0,"id":2126,"soft":null,"name":null,"lon":-51.65,"lat":-24.2333333333,"map_id":354},{"updated_at":"2012/08/14 21:56:56 +0200","created_at":"2012/08/14 21:52:03 +0200","error":1.0146168179818,"y":3286.0,"x":2331.5,"id":2127,"soft":null,"name":null,"lon":-51.6833333333,"lat":-24.25,"map_id":354},{"updated_at":"2012/08/14 21:56:59 +0200","created_at":"2012/08/14 21:52:51 +0200","error":1.01461681850937,"y":3286.5,"x":3940.0,"id":2128,"soft":null,"name":null,"lon":-51.6666666667,"lat":-24.25,"map_id":354}]}

-1 srid should be 0

2013-07-21 20:00:00.5390 26603/7f2764aa0700 Pool2/Implementation.cpp:1162 ]: [App 1723 stderr] NOTICE: SRID value -1 converted to the officially unknown SRID value 0
[ 2013-07-21 20:00:00.5391 26603/7f2764aa0700 Pool2/Implementation.cpp:1162 ]: [App 1723 stderr] LINE 1: UPDATE "layers" SET "bbox_geom" = '0103000020FFFFFFFF0100000...

Upgrade Rails

Lets upgrade to Ruby 2.1 and Rails 4

Might be a bit difficult.

Main issues I can see are the integrations with the geo libraries etc

mapscript error?

2013-07-21 20:00:00.5390 26603/7f2764aa0700 Pool2/Implementation.cpp:1162 ]: [App 1723 stderr] NOTICE: SRID value -1 converted to the officially unknown SRID value 0
[ 2013-07-21 20:00:00.5391 26603/7f2764aa0700 Pool2/Implementation.cpp:1162 ]: [App 1723 stderr] LINE 1: UPDATE "layers" SET "bbox_geom" = '0103000020FFFFFFFF0100000...
[ 2013-07-21 20:00:00.5391 26603/7f2764aa0700 Pool2/Implementation.cpp:1162 ]: [App 1723 stderr] ^
[ 2013-07-21 20:00:03.0972 26603/7f2761daf700 Pool2/Implementation.cpp:1162 ]: App 4464 stderr: warning: rb_define_const: invalid name __FUNCTION__' for constant [ 2013-07-21 20:00:20.7724 26603/7f2761daf700 Pool2/Implementation.cpp:1162 ]: [App 4464 stderr] (eval): warning: rb_define_const: invalid nameFUNCTION' for constant
[ 2013-07-21 20:00:23.0701 26603/7f2761daf700 Pool2/Implementation.cpp:1162 ]: App 4464 stderr: warning: rb_define_const: invalid name __FUNCTION__' for constant [ 2013-07-21 20:00:23.5149 26603/7f2761daf700 Pool2/Implementation.cpp:1162 ]: [App 4464 stderr] (eval): warning: rb_define_const: invalid nameFUNCTION' for constant
[ 2013-07-21 20:00:23.9014 26603/7f2761daf700 Pool2/Implementation.cpp:1162 ]: App 4464 stderr: warning: rb_define_const: invalid name `FUNCTION' for constant
[ 2013-07-21 20:01:36.8404 26603/7f2761daf700 Pool2/Implementation.cpp:1162 ]: [App 4464 stderr] NOTICE: SRID value -1 converted to the officially unknown SRID value 0
[ 2013-07-21 20:01:36.8405 26603/7f2761daf700 Pool2/Implementation.cpp:1162 ]: [App 4464 stderr] LINE 1: ...at" = '2013-07-21 20:01:36.838214', "bbox_geom" = '010300002...
[ 2013-07-21 20:01:36.8405 26603/7f2761daf700 Pool2/Implementation.cpp:1162 ]: [App 4464 stderr] ^
[ 2013-07-21 20:01:58.2470 26603/7f2761daf700 Pool2/Implementation.cpp:1162 ]: [App 4464 stderr] NOTICE: SRID value -1 converted to the officially unknown SRID value 0
[ 2013-07-21 20:01:58.2471 26603/7f2761daf700 Pool2/Implementation.cpp:1162 ]: [App 4464 stderr] LINE 1: ...at" = '2013-07-21 20:01:58.245116', "bbox_geom" = '010300002..

unable to view maps of a user when logged out

Processing MyMapsController#list (for 217.36.0.109 at 2012-12-11 19:44:39) [GET]
Parameters: {"controller"=>"my_maps", "user_id"=>"82", "action"=>"list"}

NoMethodError (undefined method has_role?' for :false:Symbol): app/controllers/my_maps_controller.rb:71:inget_user'
/usr/lib/ruby/1.8/phusion_passenger/rack/request_handler.rb:96:in process_request' /usr/lib/ruby/1.8/phusion_passenger/abstract_request_handler.rb:516:inaccept_and_process_next_request'
/usr/lib/ruby/1.8/phusion_passenger/abstract_request_handler.rb:274:in main_loop' /usr/lib/ruby/1.8/phusion_passenger/classic_rails/application_spawner.rb:321:instart_request_handler'
/usr/lib/ruby/1.8/phusion_passenger/classic_rails/application_spawner.rb:275:in send' /usr/lib/ruby/1.8/phusion_passenger/classic_rails/application_spawner.rb:275:inhandle_spawn_applic

copy GPS - also join GCPs together

So the idea is that once I rectify an image, yuo have in your DB a bunch of entries with lat-lon mapped to x-y in a given image...
if it were not for the active ops on irc they would be here too
If I go to rectify another image in the same area, it would be nice for the DB to automatically serve up the same lat-lon points as suggested places to find in the second image.
<chippy|tim> ahh yes
This has two big benefits...
<chippy|tim> did you suggest this recently?
one, these are already good places to see in the aerial imagery...

  • rcoup_ ([email protected]) has joined #osm
    and two, if you re-use these point among images in a layer, they will exactly line up with eachother (assuming you put the marker in the source image correctly).
    I thought I sent an e-mail about this a couple months back, but I can't find it now.
    How hard would it be to share lat-lon coords like that among other images?
    <chippy|tim> right - I may be adding this feature soon
    ok, cool.
    <chippy|tim> not too hard, as with anything, takes a bit of time and thought is all :)
    yeah, I thought about adding it myself but never got a working server instance set up.
    Actually, that would be a good thing to add to my VM image.
    I am trying to have all the OSM related services in a single VM image.
    <chippy|tim> iirc, the main benefit is being able to upload different versions of the same map - and copying the control points
    If I get it set up in there then I can play around with the code myself.
    It would be useful for rectifying astronaut imagery from the ISS as you can often get images from the same area at a later point in time, or you can get many images in a single layer that overlap eachotehr.
    <chippy|tim> I did add a kind of "append" feature, where the system interpolates new control points if, say, the new map butts against a previous one
    re-using the control points in the overlap areas removes one potential source of error in the original image.
    <chippy|tim> so the lat,lon pair should be the same, but the image x,y can change?
    <chippy|tim> in your case?
    yeah, exactly.
    so I find a particular road intersection in both images.
    <chippy|tim> right

this way they have identical lat-lon coords.
Also, since I align to OSM roads, if the road is in the wrong place updating the lat-lon should fix both images.
And by removing one set of numbers the layers in the overlap areas should align better.
I could just manually copy paste the relevant coords from the other image but having it built in is way easier.
<chippy|tim> oh, thats interesting
<chippy|tim> so you can update multiple maps at once
Also, once I enter three control points you can compute a temporary internal transoform, and then query the DB for other ground control points that are likely to be in the image.
yes, multiple images updated at once. See for example, this layer... http://maps.nypl.org/relief/layers/22
thats on the nypl site of course, but the idea is the same.

gdal rasterize error

http://mapwarper.net/maps/1529#Crop_tab

gdal_rasterize -i -burn 17 -b 1 -b 2 -b 3 /srv/mapwarper.net/releases/20120805161700/public/mapimages/1529.gml -l features /srv/mapwarper.net/maps/src/rreel32l1790i00250s1.gif.tif_masked
ERROR gdal rasterize script: ERROR 5: /srv/mapwarper.net/maps/src/rreel32l1790i00250s1.gif.tif_masked: panBandMap[1] = 2, this band does not exist on dataset.

add graticule / grid overlay to reference map

requests will be for things like grid overlay on OSM, manual control points on one map and dragable ones on the other - basically not enough OSM data to provide control points, but I can click on corners of map for known latlon, a latlon grid on the OSM side would make this easier.

from sk53 in #osm

extract metadata / annotation

possibly - legend.png
key - i.e. #3 is town

metadata cropping - cropped out pictures - showed on edit page for metadata

Need better tools for fixing metadata on many maps (especially the tags field) (maybe allow to see the map on the 'edit' page to copy off information, would be cool to have boxes you could drag out for each metadata field to give the "source" of the information right in the original map. Could auto crop out these pictures with a script and display them next to the fields.

Add Oauth login options

Implement Oauth login for Wikimedia Commons for wikimaps

Allow OpenHistoricalMap, OSM, Twitter for Mapwarper

making new layer broken with georuby 2.0.0

add a -1 to from_coordinates call. as in 2.0.0 it changes the DEFAULT_SRID from -1 to 4326

self.bbox_geom = Polygon.from_coordinates([poly_array], -1)

irb(main):014:0> GeoRuby::SimpleFeatures::DEFAULT_SRID
=> 4326

tif with 1 band cannot crop properly

gdal rasterize script: ERROR 5: /srv/mapwarper.net/maps/src/IRL-GSGS-3906-11-17-SE-Ennis.tif_masked: panBandMap[1] = 2, this band does not exist on dataset.

Size is 11496, 8801
Coordinate System is `'
Metadata:
TIFFTAG_IMAGEDESCRIPTION=
TIFFTAG_MAXSAMPLEVALUE=1
TIFFTAG_MINSAMPLEVALUE=0
TIFFTAG_RESOLUTIONUNIT=2 (pixels/inch)
TIFFTAG_XRESOLUTION=400
TIFFTAG_YRESOLUTION=400
Image Structure Metadata:
INTERLEAVE=BAND
Corner Coordinates:
Upper Left ( 0.0, 0.0)
Lower Left ( 0.0, 8801.0)
Upper Right (11496.0, 0.0)
Lower Right (11496.0, 8801.0)
Center ( 5748.0, 4400.5)
Band 1 Block=11496x1 Type=Byte, ColorInterp=Gray
Overviews: 5748x4401, 2874x2201, 1437x1101, 719x551, 360x276, 180x138

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.