nikolaswise/lower-albina

An exploratory web mapping project.

Nikolas is a web developer, artist, and front-end system architect in Portland, Oregon.

lower-albina's Issues

[master] Project Plan

Lower Albina Map

Project Plan: Nikolas Wise

Introduction

From a design perspective, contemporary tools for creating maps on the internet are restrictive. The current approach to cartographic maps more paint-by-numbers than informed by design decisions. Our Esri-style approach to out of the box data rendering over pre-baked basemaps is convenient to data analysts, but more or less fails for purposes that do not align or accept the base assumptions we make. Likewise, the Mapbox approach to 'designing' maps boils down to picking finishes.

We've been developing tools for more intensive cartography for web maps - Esri Leaflet bridges the gap between our powerful data and analytics and open source mapping tools; along with thing like Angular.js structures for smoother mapping work flows. Ideally, we will be able to arrive at an environment that lowers the technological barriers that stand between designers and cartographers and creative applications of web maps.

The Lower Albina project is an attempt to create such an environment - and a resultant map that accomplishes goals that current webmap systems are not.

Questions

1. How can the medium of the web map, though visual and interactive design, create a meaningful reflection of the Lower Albina industrial districts sense of place?
   • How can the specific qualities of the webmap facilitate the communication of specific cultural qualities associated with a place and the inhabitants of that place?
   • What is a meaningful way to incorporate marginalia, diagrams, and explanatory legends and texts in a webmap?
   • The previous media of maps had their inherent properties and solutions - how can we we learn from these design lessons and explore our own?
   • As the means of production for webmaps are common across a wide swath of web content and objects, how can we package diverse designed experiences together in a seamless - or intentionally seamed - manner?
   • How can a webmap with no pre-selected content or design elements question the implicit assumptions that institutional designs make, or undermine and highlight the inherent political power associated with map making?
2. How can we create mapping tools and environments that make accomplishing this more accessible?

Goals

Esrigoalâ„¢: Create a development environment and set of tools that allow for richer design process that currently available for web cartography.

NKgoalâ„¢: Create a compelling design object exploring the cultural sense of place within the Lower Albina commercial district.

Setup Steps

• Create project checklist.
• Critical design analysis of effective print maps, including 1940's Swiss Topographic maps and historical Norwegian maps.
• Identify theoretical strengths and weakness with screen-as-media
• Identify authoritative assumptions in current webmap structures
• List physical geographic qualities and peculiarities that work towards Lower Albina's sense of place.
• Identify extant design tools, problems that require new design tools, and formulate plan for creating these tools,

Development Steps

• Collect, collate, and analyze data from Lower Albina
• Collect and create narrative, visual, and anecdotal form of non-geographic assets that reflect cultural sense of place
• Create structures and design tools
• Incorporate non-graphic assets, legends, and marginalia
• Test, revise, and repeat use of tools.

Final Steps

• Produce final design object as a stand-alone and coherent project.
• Document and share structures, tools, and development environments

Timeline

To be determined.

Creating Base and Reference Layers

Creating Base and Reference Layers

This is complicated. There are a lot of good stuff out there. To get at them all though, one has to do a lot of meandering to get around plenty of probs.

Good Stuff

• Easy Reprojection of data
• High quality demographic data available.
• High Quality cartographic data rendering tools
• Layer Compositing and Tile Generation

Probs

• City of Portland data sets are huge, bigger than our study area.
• Proprietary data formats from local government GIS departments.
• ArcGIS Online has no support tile layers that contain multiple layers.
• MapBox Studio has no tools for examining, classifying, and styling data across ramps.
• AGO exports GeoJSON with x,y as meters from Null Island in WGS84 .
• MBS can't handle WGS:84 as meters from Null Islands. Needs to be reprojected as lat/long in decimal degrees.
• Neither AGO or MBS provide reprojection tools.
• AGO doesn't support sub-pixel rendering
• AGO raster tiles look a little rough.
• MBS wont let go of tiles, either in vector or raster.
• MBS is CRAZY EXPENSIVE FOR TILES
• CartoDB has all styling trubs as MBS, but is cheaper.
• CartoDB will only let you build a map with 4 layers and a basemap.

Step By Step

I dealt with the trubs so you don't have to. Looking at one layer of data, here's how we make high quality (raster for now) tiles and consume them in our Leaflet app. Repeat data layer steps as needed until your base or reference layer is done.

0. Find Your Data

This is important. Lower Albina uses the Portland Streets Centerlines data to create a layer that sits on reference layer. Download your data.

1. Upload that Data to Arcgis Online

This is gives us access to the ArcGIS's pretty excellent data manipulation tools and cartographic abilities. If you have an ArcGIS org running, and an account with your org, you can just use that. If you don't, this step has it's own gotchas, so we'll go through that step by step really quick.

a. Get free developer account at Arcgis for Developers This will give you access to AGO, along with some free credits every month you can use to run analysis tools and hit service endpoints in your app later on. Don't sign up for a Public Account with AGO - these things are useless and don't let you create or store any content, or provide any credits to use the tools, or any way to acquire credits.

b.* Once you get confirmation of your new developer account, Sign in to ArcGIS Online. This is weird. Don't ask.

c. Navigate to 'My Content', and 'Create' a new item 'From my Computer' Grab your data set - this means the entire .zip file in the case of shapefiles

d. Done! AGO will create a Feature Layer of your data!

2. Crop Data to Study Area

We don't need ALL the streets in Portland for our Reference Layer in Lower Albina. We need to crop down the data so we only need to deal with the bits that we're concerned this. This is a little complicated also.

a. Open > Add layer to new map to get the feature layer on the Map Viewer, where we can start working with it. If your data set is large, it won't draw completely. A modal might pop up telling you this - this is okay! Your data is in there, but AGO is just drawing the first 1000 features to try and avoid destroying your browser. Thanks AGO.

b. Add > Add Map notes Layer to draw out your area of study.

c. Pop back to the details tab, and find the Secret Dropdown Menu of Ultimate Power on your data layer.

d. Select Perform Analysis.

e. To crop the data to just our study area, we need to 'Find Locations' > 'Find Existing Locations'. Add an expression that has a geographic query with your map notes layer.

f. Run the analysis with the current extent. This will keep your cost down. You can see how many credits this will burn with the 'show credits button.' My analysis is going to cost 4.643 out of my free 50 credits a month. This is about 46 cents in USD. Analysis will probably take a while.

3. Style your data

Turn off your source and map notes layer, and use the Secret Dropdown Menu of Ultimate Power on the basemap layer to turn the transparency all the way up. Now we have our layer, and can do some good stuff to it.

Use the Secret Dropdown Menu of Ultimate Power to 'Change Style' on your layer, and go nuts in there.

For these streets, I wont be pushing the limits of AGO's rendering tools. But I want to see what structure types there are, and set different widths of my lines. This shows each structure type as it's own color. Im going to set all my colors to white, and just change the widths. I'll do this in the next step, since AGO doesn't support sub-pixel rendering of line widths, which I will want to do. For now, Im going to pretend I want these colors so we can move on.

4. Take a Screen Shot of AGO's Rendering Rules.

This is the end of the line for our use of AGO to work with the data. From this point on, we're just focusing on getting the stuff we want OUT of the ArcGIS system. We'll use the hard work that AGO did to examine, classify, create ramps for, and otherwise be really smart about our data set to re-create the renderer by hand in CartoCSS. We'll use this screenshot to get the field names, values, colors, and other good stuff we need to make this happen.

A better example of the power that AGO's renderers give us is with the Contour data Lower ALbina uses in it's base layer. AGO gives us a histogram of the distribution of elevations in our study area, creates a classification of 10 natural breaks in the data, and assigned each class a step from our defined color ramp. In this step, we could use any of the (very good) default color ramps, and each class in the data would automatically get the correct color value.

5. Get your Data Out of AGO

Using the Secret Dropdown Menu of Ultimate Power, view your layers item details page. Here we can convert our data to GeoJSON and download it.

a. 'Export' > Export File as GeoJSON. This will create a new item in our 'My Content' zone that's a GeoJSON file rather than a Feature Layer. This might take a while. AGO will forward you to this items details page.

b. 'Open' > 'Download'. This will download your Geojson. Dont ask.

6. Get the Data into CartoDB

Upload your GeoJSON to a free CartoDB account.

8. Style Your Data in MapBox Studio

Add your data to a new map. Pick your data carefully though - CartoDB only allows 4 datasets per map at the free level. So that's cool!

Now you can write CartoCSS in your CartoDB map to style the data. Grab your screenshot of AGO's render rules, and go to town implementing AGO Styles in CartoCSS. For the streets data set, I want to style some features in a way that the underlying data doesn't exactly reflect. CartoDB lets us go in and inspect feature by feature, and add a CartoCSS rule that addresses consistency at this level, which can't be done in AGO.

To get rid of the basemap, upload a blank, transparent .png and set THAT as the basemap.

9. Use CartDB.js to get your tiles

CartoDB exposes the json for your map through the 'Share' modal. Grab the endpoint un the cartodb.js option, and use cartodb to add your map to your leaflet project.

Turn leaflet off and add CartoDB.js instead

<!-- Load Leaflet from CDN-->
<!-- <link rel="stylesheet" href="http://cdn.leafletjs.com/leaflet-0.7.3/leaflet.css" /> -->
<!-- <script src="http://cdn.leafletjs.com/leaflet-0.7.3/leaflet.js"></script> -->

<!-- CartoDB -->
<link rel="stylesheet" href="http://libs.cartocdn.com/cartodb.js/v3/3.13/themes/css/cartodb.css" />
<script src="http://libs.cartocdn.com/cartodb.js/v3/3.13/cartodb.js"></script>

In your main js file, add your layer to your map.

cartodb.createLayer(map, url)
  .addTo(map)
  .on('done', function(layer) {
    //do stuff
  })
  .on('error', function(err) {
    alert("some error occurred: " + err);
  });

10. Grab your tile endpoint from your app

If cartodb added your map as a layer in your leaflet app, we're on the right track. Open your network request panel, and find a tile from your map. Copy the URL. Mine looks like this:

http://2.ashbu.cartocdn.com/nikolaswise/api/v1/map/e84ce689c5a9a5090b90c0cb4dba4c9f:1429741486907.6099/16/10435/23435.png

11. Convert that to Actual Tile Endpoint

To consume these tiles in Regular Leaflet without CartoDB.js, you need to variablize this URL. That means adding {s}, {x}, {y}, and {z} variables. My endpoint ends up looking like this:

http://{s}.ashbu.cartocdn.com/nikolaswise/api/v1/map/e84ce689c5a9a5090b90c0cb4dba4c9f:1429741486907.6099/{z}/{x}/{y}.png

12. Eat those tiles with Leaflet.

Turn off CartoDB.js and turn Leaflet back on.

<!-- Load Leaflet from CDN-->
<link rel="stylesheet" href="http://cdn.leafletjs.com/leaflet-0.7.3/leaflet.css" />
<script src="http://cdn.leafletjs.com/leaflet-0.7.3/leaflet.js"></script>

<!-- CartoDB -->
<!-- <link rel="stylesheet" href="http://libs.cartocdn.com/cartodb.js/v3/3.13/themes/css/cartodb.css" /> -->
<!-- <script src="http://libs.cartocdn.com/cartodb.js/v3/3.13/cartodb.js"></script> -->

Use Leaflet to add a new Tile Layer with your tile endpoint url.

var referenceLayer = L.tileLayer('http://{s}.ashbu.cartocdn.com/nikolaswise/api/v1/map/e84ce689c5a9a5090b90c0cb4dba4c9f:1429741486907.6099/{z}/{x}/{y}.png')
referenceLayer.addTo(map);

Delete all that cartodb.createLayer() junk. If your tile layer belongs on the bottom like a regular tile layer your done! If your crazy that me and want a tile layer to sit OVER the rest of the map, you have one more step.

1. Create a new Leaflet Map Pane

Build a brand new pane for leaflet, bring it to the top of the pane stack, and move your tile layer's container element to it.

var referencePane = map._createPane('leaflet-reference-pane', map.getPanes().mapPane);
referencePane.appendChild(referenceLayer.getContainer());
referenceLayer.setZIndex(9);

And you have a tile layer sitting on top of your map!

data walkthrough

Concept

Lower Albina looks at using classical analog-era metaphors to address web cartography and creating of the 'map'. At it's root, this conceptual model requires 3 layers:

1. Base Layer
2. Study Layer
3. Reference Layer

This is similar to the ideas behind basemap/layers/labels, but simplified. Each layer has it's own, very narrow, scope.

The base layer and the reference layer can both exist as single objects or tile layers in the web map to avoid issues of overwhelming the map with data, and simplify the amount of control needed over layering. One tile layer for the base, one tile layer for the Reference.

Base Layer

The base layer will underlay the rest of the map. It is related to the basemap, but has much less going on. The base layer only holds as much information as needed to ground the study layer (or each individual study layer) with the geography. For Lower Albina, the Base Layer only has two data sets:

1. 5 Foot Contours, classified by elevation.
2. County Taxlots

Reference Layer

The Reference Layer contains data that sits over the study layer. This layer provides precise context about geography that is further up the conceptual hierarchy than the data in the study layer. The reference layer for Lower Albina includes

1. Bridges
2. Light and Heavy Rail lines
3. Roads
4. Neighborhood information
5. Water Bodies

Creating Base and Reference Layers

This is complicated. There are a lot of good stuff out there. To get at them all though, one has to do a lot of meandering to get around plenty of probs.

Good Stuff

• Easy Reprojection of data
• High quality demographic data available.
• High Quality cartographic data rendering tools
• Layer Compositing and Tile Generation

Probs

• City of Portland data sets are huge, bigger than our study area.
• Proprietary data formats from local government GIS departments.
• ArcGIS Online has no support tile layers that contain multiple layers.
• MapBox Studio has no tools for examining, classifying, and styling data across ramps.
• AGO exports GeoJSON with x,y as meters from Null Island in WGS84 .
• MBS can't handle WGS:84 as meters from Null Islands. Needs to be reprojected as lat/long in decimal degrees.
• Neither AGO or MBS provide reprojection tools.
• AGO doesn't support sub-pixel rendering
• AGO raster tiles look a little rough.
• MBS wont let go of tiles, either in vector or raster.

Step By Step

I dealt with the trubs so you don't have to. Looking at one layer of data, here's how we make high quality (raster for now) tiles and consume them in our Leaflet app. Repeat data layer steps as needed until your base or reference layer is done.

0. Find Your Data

This is important. Lower Albina uses the Portland Streets Centerlines data to create a layer that sits on reference layer. Download your data.

1. Upload that Data to Arcgis Online

This is gives us access to the ArcGIS's pretty excellent data manipulation tools and cartographic abilities. If you have an ArcGIS org running, and an account with your org, you can just use that. If you don't, this step has it's own gotchas, so we'll go through that step by step really quick.

a. Get free developer account at Arcgis for Developers This will give you access to AGO, along with some free credits every month you can use to run analysis tools and hit service endpoints in your app later on. Don't sign up for a Public Account with AGO - these things are useless and don't let you create or store any content, or provide any credits to use the tools, or any way to acquire credits.

b.* Once you get confirmation of your new developer account, Sign in to ArcGIS Online. This is weird. Don't ask.

c. Navigate to 'My Content', and 'Create' a new item 'From my Computer' Grab your data set - this means the entire .zip file in the case of shapefiles

d. Done! AGO will create a Feature Layer of your data!

2. Crop Data to Study Area

We don't need ALL the streets in Portland for our Reference Layer in Lower Albina. We need to crop down the data so we only need to deal with the bits that we're concerned this. This is a little complicated also.

a. Open > Add layer to new map to get the feature layer on the Map Viewer, where we can start working with it. If your data set is large, it won't draw completely. A modal might pop up telling you this - this is okay! Your data is in there, but AGO is just drawing the first 1000 features to try and avoid destroying your browser. Thanks AGO.

b. Add > Add Map notes Layer to draw out your area of study.

c. Pop back to the details tab, and find the Secret Dropdown Menu of Ultimate Power on your data layer.

d. Select Perform Analysis.

e. To crop the data to just our study area, we need to 'Find Locations' > 'Find Existing Locations'. Add an expression that has a geographic query with your map notes layer.

f. Run the analysis with the current extent. This will keep your cost down. You can see how many credits this will burn with the 'show credits button.' My analysis is going to cost 4.643 out of my free 50 credits a month. This is about 46 cents in USD. Analysis will probably take a while.

3. Style your data

Turn off your source and map notes layer, and use the Secret Dropdown Menu of Ultimate Power on the basemap layer to turn the transparency all the way up. Now we have our layer, and can do some good stuff to it.

Use the Secret Dropdown Menu of Ultimate Power to 'Change Style' on your layer, and go nuts in there.

For these streets, I wont be pushing the limits of AGO's rendering tools. But I want to see what structure types there are, and set different widths of my lines. This shows each structure type as it's own color. Im going to set all my colors to white, and just change the widths. I'll do this in the next step, since AGO doesn't support sub-pixel rendering of line widths, which I will want to do. For now, Im going to pretend I want these colors so we can move on.

4. Take a Screen Shot of AGO's Rendering Rules.

This is the end of the line for our use of AGO to work with the data. From this point on, we're just focusing on getting the stuff we want OUT of the ArcGIS system. We'll use the hard work that AGO did to examine, classify, create ramps for, and otherwise be really smart about our data set to re-create the renderer by hand in CartoCSS. We'll use this screenshot to get the field names, values, colors, and other good stuff we need to make this happen.

A better example of the power that AGO's renderers give us is with the Contour data Lower ALbina uses in it's base layer. AGO gives us a histogram of the distribution of elevations in our study area, creates a classification of 10 natural breaks in the data, and assigned each class a step from our defined color ramp. In this step, we could use any of the (very good) default color ramps, and each class in the data would automatically get the correct color value.

5. Get your Data Out of AGO

Using the Secret Dropdown Menu of Ultimate Power, view your layers item details page. Here we can convert our data to GeoJSON and download it.

a. 'Export' > Export File as GeoJSON. This will create a new item in our 'My Content' zone that's a GeoJSON file rather than a Feature Layer. This might take a while. AGO will forward you to this items details page.

b. 'Open' > 'Download'. This will download your Geojson. Dont ask.

6. Convert from Meters From Null Island to Decimal Degree Coordinates

We're gonna use (Ogr2Ogr CLI)[http://www.gdal.org/ogr2ogr.html] for this.

$ogr2ogr streetsLatLong.geojson -t_srs 'WGS84' -f 'GeoJSON' Streets-Lower-Albina.geojson;

7. Get the Data into MapBox Studio

Sign up for a Free Account with Mapbox. Get your hands on MapBox Studio and start a new Source Project from Custom Vector Tiles.

Add a new layer, and select the GeoJSON that Ogr2Ogr gave us. Save that, and upload it to MapBox.

Once the data uploads, go find it in MapBox. Copy the source ID, and head back to mapbox studio.

8. Style Your Data in MapBox Studio

Create a new Style Project in MapBox Studio. As your data source, plug your uploaded Source ID into the little input at the bottom and hit 'Create'.

Now you can write CartoCSS in MapBox Studio to style the data. Grab your screenshot of AGO's render rules, and go to town implementing AGO Styles in CartoCSS. For the streets data set, I want to style some features in a way that the underlying data doesn't exactly reflect. MBS lets us go in and inspect feature by feature, and add a CartoCSS rule that addresses consistency at this level, which can't be done in AGO.

9. Upload the Style Project as MapBox Tiles

Once your style is uploaded, you can head over to MapBox and preview the tile.
Make sure the tile are public in the API, and grab the preview url. For my Base Layer, it looks like this:

https://api.tiles.mapbox.com/v4/nikolaswise.65959181/page.html?access_token=pk.eyJ1Ijoibmlrb2xhc3dpc2UiLCJhIjoieVJJcE1QUSJ9.f8co32wYW_YTeh_KM6PGLA#15/45.5356/-122.6707

10. Consume in Leaflet

Consume your tiles in your Leaflet app, and add em to your map!

L.tileLayer('https://{s}.tiles.mapbox.com/v4/nikolaswise.65959181/{z}/{x}/{y}.png?access_token=pk.eyJ1Ijoibmlrb2xhc3dpc2UiLCJhIjoieVJJcE1QUSJ9.f8co32wYW_YTeh_KM6PGLA#15').addTo(map);