datacarpentry/r-raster-vector-geospatial

Introduction to Geospatial Raster and Vector Data with R

R for Raster and Vector Data

Contributing to lesson development

• The lesson files to be edited are in the _episodes folder. This repository uses the main branch for development.
• You can visualize the changes locally with the sandpaper R package by executing either the sandpaper::serve() or sandpaper::build_lesson() commands. In the former case, the site will be rendered at http://localhost:4321
• Each time you push a change to GitHub, Github Actions rebuilds the lesson, and when it's successful (look for the green badge at the top of the README file), it publishes the result at http://www.datacarpentry.org/r-raster-vector-geospatial/
• Note: any manual commit to gh-pages will be erased and lost during the automated build and deploy cycle operated by Github Actions.

Lesson Maintainers:

• Jemma Stachelek
• Ivo Arrey
• Drake Asberry

r-raster-vector-geospatial's Issues

Explanation of Stretch Lesson 04

Currently this is introduced in Lesson 04 (https://github.com/data-lessons/NEON-R-Spatial-Raster/blob/gh-pages/04-Multi-Band-Raster.Rmd) lines 299-326 but is poorly address and potentially unnecessary.

1. Does it need to be explained? or can we remove it from the code?

2. How best to teach / explain the concept?

3. Is a graphic needed? eg. http://www.nrcan.gc.ca/earth-sciences/geomatics/satellite-imagery-air-photos/satellite-imagery-products/educational-resources/9389

4. Does it fit better elsewhere in the series?

Move CropNDVIFiles.Rmd code to Lesson Dev. Repo

The file CropNDVIFiles.Rmd code is currently in the knitr_processing_code folder. Should this be moved to Lesson Development Repo? Maybe create a new folder called NEON_external-data-processing-code that can be a repo for all the code created to extract/clean the data for the lessons. Some might be developed into lessons. Might be very useful if OS/IS might be using the code (or referring to it).

Create Geotiff with 4 bands?

Leah needs to create a geotiff with

1. 4 bands (if this is a good idea for the geotiff format),
2. no data values as -9999 rather than black - then we can teach them to set NAvalue<- -9999 and it will plot pretty
3. what else??

The other option is i could include band 4 as a separate tiff that is in a different CRS - then it needs to be projected?? @jhollist thoughts on this if i need to do this work anyway? i could give you band 4 as a geotiff in WGS 84 geographic CRS.

NOTE - we'd want to create NDVI here
http://data-lessons.github.io/NEON-R-Spatial-Raster/R/Multi-Band-Rasters-In-R.R/

l.

UTM Zone 18 / 19

All the NEON derived data (Lessons 00-04) have UTM Zone 18 in the geotags. The Landsat derived data (Lessons 05-07) have UTM Zone 19. Why the difference? MA is divided by the two. We at least need to make a note in the lessons about why the two different ones and why it does/ does not create a problem.

Add data structure graphics to lessons

Participants requested (and it would be helpful for teaching) diagrams on R data structures. Particularly for heterog. ones like spatial objects with nested components stored in slots (eg data.frames in .@DaTa slot, etc. Looking for some for today's teaching now.

Quick Review of Raster Build

Hi @mjones01 - all edits are merged! Please have a look at the rendered lessons and let me know if you see any issues. I have fixed the baseurl (removed the slash). Let's see if anything breaks. Things that could break include images! If you are happy with it as built, feel free to close this issue or assign it back to me to close :)

Jekyll 3.0 fixed

@mjones01 this is very weird. I just accepted you PR but it said "0" files changed. I saw your commits and see the changes but the files don't have the fixes. Now the headers don't work in this repo so this confirms we need to fix this.

Lesson04 Challenge: Methods for Calling on an Object

This is a good challenge but a bit out of place in this lesson. Consider moving it.
@lwasser suggested into the introductory data management section

What is NDVI video

we might use this in our background content.

https://www.youtube.com/watch?v=rxOMhQwApMc

Geotiff exampl

Need to add a geotiff w/ the following characteristics:

• has not been reprojected
• has -9999 values

L03 - .data tip breakout - double check formatting.

check how L03 renders - the .notice tag in the data tip below includes a function chunk however - i'm not sure this will render properly in kramdown as the {} tag is on a paragraph and doesn't support multiple paragraphs. Let's make sure this works before pushing to dev.

Data Tip: A custom function consists of a defined
-tasks performed on a input object. Functions are particularly useful for tasks +set of commands performed on a input object. Custom functions are particularly
-that need to be repeated over and over in the code. A simplified syntax for a +useful for tasks that need to be repeated over and over in the code. A simplified
-function in R is: +syntax for writing a custom function in R is:
-functionName <- function(variable1, variable2){WhatYouWantDone, WhatToReturn} +```
+functionName <- function(variable1, variable2){

• something_new <- do_something_to_variable1_and_variable2
• return(something_new)
  +}
  +```
  {: .notice } {: .notice }

Lesson 01 (plotting) questions

So first, when I made a pull request, it said there are merge conflicts and I have no idea how to deal with that.

Second, what is the difference between plot() and image(), and do we need to introduce image here?

Third, is there anything else that needs to be included in this lesson? I can't think of anything that is basic that might be useful.

Reprojection and modifying rasters - Lesson 02 ~lines 198

Per @mjones01 a comment (i think this is your comment?)

LESSON NEED TO ADD: Why do min/ max values change. CRS shouldn't effect elevation? Is this due to changing # of cells and then having to re-sample cells? We lose just a bit of the extremes.

The bottom line is that anytime you reproject, or do calculations on a raster you are modifying it. In the case of projection you are moving it from one grid to another so there is some interpolation that needs to happen .Thus you will notice small changes in the data. This can really compound over time.

However i'm not sure if we want to teach this or not to a beginner audience as there is a LOT in this section already and reprojection is confusing and very hard to teach quickly, well. open to ideas but i pulled the comment from the lesson whosever it was!.

Add a bit of code to make histograms prettier - L01 - plotting rasters in R

A few folks have edited L01 - plotting rasters in R. Given histograms are also included there, let's add a few lines of code to show users how to make prettier histograms as well. We can add color, customize labels, etc. It's a simple addition!

http://data-lessons.github.io/NEON-R-Spatial-Raster/R/Plot-Rasters-In-R/

Add Overview of CRS in L00

Left a comment in this lesson to include a discussion of CRS - this would be content in the INTRO TO SPATIAL Data part of the workshop if that is developed. So we should revisit where this will live. But regardless it needs to be created. We can largely link out to existing content as this topic is well covered in the GIS world. Does anyone have time to work on this little chunk of explanation?

http://data-lessons.github.io/NEON-R-Spatial-Raster/R/Introduction-to-Raster-Data-In-R/

Megan found a very nice overview of projections

https://www.nceas.ucsb.edu/~frazier/RSpatialGuides/OverviewCoordinateReferenceSystems.pdf

i haven't gone through it to carefully ensure it covers what we want to cover but it might.

TOC/Overview Sidebar not showing up on rendered lessons

@lwasser which updated file needs to be adding to the repo so that the overview/TOC will appear in the sidebar. Currently the header shows up but nothing populates.

add `brick()` next to `stack()` to illustrate differences in functions

Functions - Do we have time to explain how to write a function - Raster Calculations in R

Hi Guys -- i've been working on what is now L03 - modifying rasters in R.

http://data-lessons.github.io/NEON-R-Spatial-Raster/R/Modify-Rasters-In-R/

if the above link doesn't work use this one (to be fixed soon)

http://data-lessons.github.io/NEON-R-Spatial-Raster/R/Modify-Rasters-In-R.R/

The challenge with this particular lesson is that the best approach to performing raster calculations is not direct raster math (rasterone-rastertwo) UNLESS it is a small - non complex operation. So i introduced overlay. However i'm not realizing that we have not introduced functions and this section is getting heavy.

How should we proceed? Should we explain functions? Or should this be a separate advanced lesson - l04 that instructors can chose to skip if they want to?

the challenge is that this module is getting long potentially and i know many people get hung up on functions. BUT functions are critical to good coding.

Thoughts? @brymz @jhollist @msmorul @jduckles @jsta @KristinaRiemer

Add a few VERY TALL (40000m) pixels in the DEM as an example of bad data values

Look at L05 and 07

i rebuild but i think the output from the challenge needs to be cleaned up - potentially placed in a separate chunk just as an example. will revisit tomorrow.

Export rgb/raster

At end of lesson chm and very end, save calculated chm geotiff and ndvi csv

Lesson 02 - Need a challenge or two in this lesson for reprojection - ideas welcome.

space after comments in R code

@mjones01 a note as we work on lessons in the future

@brymz noted:
We should add a space after # in comment lines of code as per http://adv-r.had.co.nz/Style.html.

I think this is a good idea to implement throughout. It's also a very easy quick update.
i'll implement this in the intro lessons which I will work on today. Zach is taking care of raster.
I can also do vector if you want to do time series when you have a bit of "i need to do something mindless" which i have often at the end of the day after coding all day! :)

Lesson 01: Pseudo-code chunk

Can someone fill me in on the purpose of the pseudo-code chunk that begins Lesson 01 (line 70; http://data-lessons.github.io/NEON-R-Spatial-Raster/R/Plot-Rasters-In-R.R/ when rendered everything down to Plotting Raster Files).

Was it just the basic code that was then broken up below for teaching purposes. Most of it seems to repeat. I don't want to delete it if it is vital to the lesson, I'm just not seeing where it adds to the lesson.
@lwasser?

Explaining slots?

Do we want to/ need to explain slots in this module (or other module)?
Peripherally used in Lesson 00. Used elsewhere?

L07: RasterStack Plotting

@lwasser originally noted:

254_SJER_landRGB.tif - the range on the blue band for this layer is 255-255

#R cant render that as a stretch function. Not sure how to properly fix this
#via code, for the meantime writing a manual exception

CODE: 433-444 or http://data-lessons.github.io/NEON-R-Spatial-Raster/R/Extract-NDVI-From-Rasters-In-R/

Super efficient code

for (aFile in rgb.allCropped.SJER)
{NDVI.rastStack <- stack(aFile)
if (aFile =="NEON-DS-Landsat-NDVI/SJER/2011/RGB/254_SJER_landRGB.tif")
{plotRGB(NDVI.rastStack) }
else { plotRGB(NDVI.rastStack, stretch="lin" ) }
}

7 Jan I couldn't get this manual fix to work. It does work if we remove ", stretch="lin" " but then all images in very blue cast.

NEED TO FIX!

L04: Stretch Graphics

@lwasser I think we should make the two stretch graphics a bit more inline with the others Colin's created in the last few days.

• put NEON logo inside box
• Increase background box size to include all image/ text.

However, you two were the ones who initially created these so I wanted to run it by you before asking Colin.

http://data-lessons.github.io/NEON-R-Spatial-Raster/R/Multi-Band-Rasters-In-R/
or lines 379/389.

Related Lessons at Bottom of page

I have code that will allow us to automatically list related lessons. Right now it's at the bottom of the page rather than the top. What is better?

I'm trying to shorten up the top grey block as it's long and for SEO it's ideal if we have unique content up towards the top. that list is useful but it could go at the bottom potentially with ONE link at the top to a "module" page that describes the entire lesson set like what SWC does. Thoughts? @mjones01

Please see L00 to see where i am going with this.

http://data-lessons.github.io/NEON-R-Spatial-Raster/R/Introduction-to-Raster-Data-In-R/

Objective 3 extract

Write up an example to use extract. The use case would be to find the temperature of an area where a specific point observation occured. Some overlap w/ the integration group, but this is an introduction to extract.

update 3 and 4

Merge stub / lessons material and raster_module_code.R into 03 and 04

make sure related content populates properly throughout raster lessons

it looks like the mainTag is set to raster-data rather than raster-data-series (or whatever the YAML list contains for this series!). I will fix this once i've merged zach's edits.

Add SJER Hillshades to Figshare data files

L01 challenge (chunk {r challenge-hillshade-layering, echo=FALSE}) requires a SJER hillshade for DSM & DTM. Neither are in the current data available on FigShare.

LiDAR (not lidar)

To be consistent with other NEON sites and #WorkWithData lessons please use LiDAR instead of lidar or any other variants.

L01 - review for content per feedback from Zack - Layering order

Per @brymz

---

The rainbows color palette pops up oddly. I like the idea of demonstrating how color palettes differ, but I'd like to see a progression of plot map -> plot hill shade -> overlay map on hillshade, so that you can scroll back and see how they all fit together. Also, I wonder why it is not the hillshade that is overlayed on the map.
CLARIFICATION:

---

The progression I would like to see already exists (kind of) from the challenge map solution one map before 'Layering Rasters' to the 'DMS map with hillshade', IF the 'DSM map with hillshade' was plotted with terrain.colors() as with the earlier 'DMS' maps. This way you could scroll back to reference the 'hilshade' and 'DSM' maps to see how they line up when plotted together. If you want to introduce rainbows, I recommend repeating the sequence of 'DSM' -> 'hillshade' -> 'DSM with hillshade' with the rainbows color. Alternately, you can introduce rainbows earlier with the 'DSM_HARV' map just before introducing hillshade to match the sequence. Hope that helps clarify things.

---

I am merging the edits and we will keep this open for consideration when we work on the lesson more!

NDVI example

Identify and add data so the class can run an example where they calculate ndvi

General Timeline, Who has Time to work on What

Hi Spatial Raster Group- @jhollist @msmorul @KristinaRiemer @JasonJWilliamsNY @brymz !
I'm very excited to see the material shelled out here in this repo!

I wanted to followup to see

1. Who has time to work on fleshing out these lessons further
2. How i can help
3. To figure out a timeline for pushing these lessons along

Let's start a discussion in this issue. It is understandable if you are too busy to work on things - feel free to say that. I am just trying to plan resources on my end. Thank you guys for all of the work on this and for getting back to me on timeline and availability :)

Just for reference, I rendered out one repo already. You can see here what the lessons will eventually look like when we flesh them out!

http://data-lessons.github.io/NEON-R-Spatial-Vector/

You are all awesome for putting these materials together! Thank you for your time!
Leah

Shadow on Challenge boxes

When a challenge has an output of a figure (and most do), the figure causes the shadow element to get broken on the bottom. This leaves a little bit of shadow on the bottom left and a weird angular shadow on the right side.

@lwasser We could remove shadow element (data tip boxes are flat) or figure out how to insert a space.

L07: Incorrect Info? Zero-based Indexing

I don't think the text is giving the correct reasoning for why we do what we are doing. I'm pretty sure I wrote that original code but not the indexing part in code or text. The origin day is 01 January 2011 so the extracted day is 01. The Julian day for this is also 01. If we need to cover Julian day 05 (which is the 5th of January), we cant add origin+Julian day because 1+5 is 6 or January 06, 2011, we have to add origin+JD-1 to get the correct day January 05 2011.

The explanation of zero-based indexing isn't wrong, but I think its application here is wrong. Can someone confirm this for me?

http://data-lessons.github.io/NEON-R-Spatial-Raster//R/Extract-NDVI-From-Rasters-In-R/
~line 231, code chunk: {r convert-jd}

CODE:
#create a date column; -1 added because origin is the 1st and indexing begins at 0.
#If not -1, 01/01/2011 + 5 = 01/06/2011 which is Julian day 6, not 5.
avg_NDVI_HARV$Date<- origin + (avg_NDVI_HARV$julianDay-1)

TEXT:

Zero-Based Indexing

Note that when converting our character formatted Julian Day values to dates,
we subtracted one as follows:
avg_NDVI_HARV$Date <- origin + (avg_NDVI_HARV$julianDay-1)
This is because R stores Julian days using 0-based indexing. That is the first
value is 0, rather than 1. Thus Julian Day 1, is stored as 0, Julian day 2 is
stored as 1, and so on.

Lesson Names/Numbering

Currently Lessons 00, 01, 03-07 are titled "Raster 00:Title" while Lesson 02 ( and all lessons in the other NEON-DC repos) are "Lesson 00: Title".

Is there Lesson 00 important to the Data Carpentry format?

If not, perhaps switching to "Raster 00:", "Vector 00:" , "Time Series 00:", and "Spatial Intro:" would be most appropriate as we are now at the point where Time Series lessons refer to Vector Lessons and Spatial Intro lessons refer to all the different other repos yet "Lesson 00:" doesn't make that obvious.

Would this also facilitate talking about the different lessons when multiple individual series are taught together in one workshop?

Related Content in Raster Lessons

From @brymz

---

What is going on with the 'Related Content: raster-data' at the bottom of the page? It doesn't seem to be in the .Rmd. It just animates (shrinks) when I click on it.

---

I think the tag used for the lessons (and main tag) is likely wrong or spelled wrong.
@mjones01 when you review these lessons, can you have a look to see if we are using the right mainTag for the raster lessons?

Thank you!

Update the **Raster Lesson Series:** section in grey box

Once we've figured out landing pages and what their URLs will be this section needs to be updated (same is true for all the NEON-dc lessons).

Lesson 05 - section on metadata

The landsat data downloaded have very long file names. The file name itself actually contains metadata including the Julian day collected, year etc. And you can parse it via the metadata. For this lesson, I actually created shorter file names for simplicity sake. However, should i use the original complex file names to allow users to explore the metadata as an activity?

Add extent graphic to lessons

Colin created a great extent graphic. where should it go?
@mjones01 you had originally suggested this graphic - where did you intend to place it?

L05: histogram data not appear in posted lessons

The histograms look fine when run in R but no data shows up when the lesson posts. Ideas?

Write reprojection lesson

Write lesson for performing a reprojecetion on the raw landsat.

Check min-max values existing in data vs. `setMinMax()`

Link to integration

Check w/ vector/inteagration group and make sure their requirements are pre-loaded at the end of this lesson. Are the right rasters loaded in memory.

Incorporate story arc

Once the code and steps for all the lessons are complete, make sure they fit into the broader "Jessica" story. That is, justify why these functions will be useful for answering scientific questions.

Challenge: Examine RGB Raster Files code updated needed

The code for the this challenge uses something that hasn't been taught in the lesson. I tried to recreate the challenge code using what was taught in that and the previous lessons and was unable to. I think, even if we give a more efficient way, the challenge code should be something do able based on what we are teachings with (maybe) some minor Googling.

http://data-lessons.github.io/NEON-R-Spatial-Raster/R/Raster-Times-Series-Data-In-R/

functions in the code

@mjones01 a while back we had a brief discussion where you use fun() and i was using fun in our code. let's move to what you were doing per @brymz latest PR . copying his comment below.
He also suggests creating a style guide (we are working on this but should provide it to people!) and a contributing guidelines .md file

---

Oops. 80 characters, absolutely. I keep my text editor at 80 characters so I can easily make the hard wrap, but sometimes I miss that step when I'm editing content. Thanks for the gentle reminder.
As for the bigger question, I developed a style guide for the Data Carpentry semester course (http://www.datacarpentry.org/semester-biology/docs/style-guide/) that I shamelessly used here. The function() formatting is also addressed there. I would definitely recommend a contributing.md and/or README.md that outlines these sort of topics.

---

Let's consider implementing this throughout. this isn't a huge high priority issue but it's a very good thing for us to implement and do when our brains our tired :)