Giter Site home page Giter Site logo

rafnuss / geopressurer Goto Github PK

View Code? Open in Web Editor NEW
7.0 1.0 1.0 410.04 MB

Global positioning by atmospheric pressure

Home Page: https://raphaelnussbaumer.com/GeoPressureR

License: GNU General Public License v3.0

R 100.00%
r era5 datalogger bird migration windspeed pressure-sensor geolocator tracker

geopressurer's People

Contributors

amelinenussbaumer avatar jsocolar avatar rafnuss avatar

Stargazers

avatar avatar avatar avatar avatar avatar avatar

Watchers

avatar

Forkers

jsocolar

geopressurer's Issues

Graph as HMM

Modification

  • Seperation of the grl$p in T and O
  • Return edge marginal probability in graph_marginal() too computational expensive and not useful
  • Implement forward ... backward sampling for graph_simulation() work as well in forward.

No pressure available for a flight, `graph_add_wind` return error

If pressure data was unaivailable during some flight, the graph_add_wind will return the following message:

Error in graph_add_wind(grl, pressure = pam$pressure, filename, thr_as = gpr$thr_as) : 
  Pressure not matching for 'data/5_wind_graph/xxx/xxx_xx.nc'

If you had look carefully, you might have noticed that during the download of wind data, there was already a warning message:

Warning messages:
1: In graph_download_wind(pam, area = static_prob, ) :
  No pressure data available for sta_id=xx. All pressure level will be downloaded

As no pressure is availabe, it is not possible to determine the wind data easily.

Problem raised by @ksakuras, see ksakuras/Siberian-rubythroat-China#9.

Simplified workflow

Aiming for a compact worflows, with variable that follow each other.

Read tag data

Short version:

tag <- tag_create("18LX")

Detailed version:

tag <- tag_create(id = "18LX",
                directory = glue::glue("data/raw-tag/{id}/"),
                pressure_file = "*.pressure",
                light_file = "*.glf",
                acceleration_file = "*.acceleration",
                crop_start = "2017-06-20",
                crop_end = "2018-05-02")

Define stationary period with label file

tag <- tag_label(tag)

Details version:

# create the label file.
tag <- tag_label_auto(tag, min_duration = 30)

# Write the label
tag_label_write(tag,
                file = glue::glue("data/tag-labels/{tag$id}.csv"))

# Read file
tag <- tag_label_read(tag,
                      file = glue::glue("data/tag-labels/{tag$id}-labeled.csv"))

# Compute stap from label
tag <- tag_label_stap(tag)

Define geography

Short version:

tag <- tag_setmap(tag, extent = c(-16, 23, 0, 50))

Detailed version:

tag <- tag_setmap(tag,
                       extent = c(-16, 23, 0, 50),
                       scale = 4,
                       include_stap_id = NA,
                       include_min_duration = 5,
                       known = data.frame(
                         stap_id = c(1, 4),
                         known_lon = c( 17.05, 17.05),
                         known_lat = c(48.9, 48.9)
                       ))

Pressure

Short version:

tag <- geopressure_map(tag)

Detailed version:

tag <- geopressure_map_mismatch(tag
                                max_sample = 250,
                                margin = 30,
                                timeout = 60 * 5,
                                workers = 90)

tag <- geopressure_map_likelihood(tag,
                                  sd = 1,
                                  thr_mask = 0.9,
                                  log_linear_pooling_weight = \(n) log(n)/n )

Light

Detailed version:

tag <- twilight_create(tag) |>
  twilight_label_read()

tag <-  geolight_map(tag)

Detailed version:

# Create twilight 
tag <- twilight_create(tag)

# Write twilight label
twilight_lable_write(tag)

# Read twilight label
tag <- twilight_label_read(tag)

# Compute likelihood map
tag <-  geolight_map(tag)

Graph

graph <- graph_create(tag) |>
  graph_add_movement()

map_marginal <- graph_marginal(graph)
path_mostl_likely <- graph_most_likely(graph)
path_simulation <- graph_simulation(graph)

Wind

tag_wind_download(tag)

graph <- graph_add_wind(graph) |>
   graph_add_movement(bird = bird_create("scientific name"))

Pressurepath

pressurepath <- pressurepath_create(tag, path = tag2path(tag))

Update Labeling

tag <- tag_stap(tag)

tag <- tag_stap(tag, extent = c(-16, 23, 0, 50), scale = 1) |>
  geopressure_map()

gpts <- geostap2path(tag) |>
  geopressure_timeseries(tag)

update <- tag_label_update(file = "data/1_labels/18LX-labeled-modif.csv",
                 tag
                 gpts)

Error with geotiff file read on Windows

Error on windows only

Error during the reading of the file. We return the uris of the gee request, the filename to the file already downloaded and the pressure_maps already computed. Here is the original error: 
error in evaluating the argument 'x' in selecting a method for function 'is.factor': Failure during raster IO

This seems to be related to an error when downloading the geotiff file with

GeoPressureR/R/geopressure.R

Line 185 in aa87166

utils::download.file(uris[i_u], filename)

Possible error related to HTTPS connections, where download.file is known to create some issues. Not exactly sure...

See also #9 for distantly issue.

New labeling scheme

Aim

  1. Make the labeling task easier to start with, i.e., needing less knoweldge of the entire procedure. Thus, name should be based on the ecological meaning than data processing role.
  2. Accept different elevation period elev for the same stationary period stap, so that the mismatch can be computed on longer timeseries and avoid having to label pressure data as oulier while they are not.
  3. Avoid labeling of acceleration data to focus the workflow towards a pressure-only sensor.
  4. Guide more the label with function to check label, and suggest modification of the label.

New Labels Scheme

Label are only assign to pressure so that no acceleration data is required. Instead of labeling data as 0 and 1. Three type of label can be used:

  • flight define the migratory flight, seperating stationary periods and informing on flight duration (same as before).
  • discard basically equivalent to previous outlier, can represent true outlier (error of sensor) or vertical movement of the bird that we don't want to match.
  • elev_1, elev_2, ... Idenfier of period during which the bird was at a unique constant elevation level but different from the main elevation level of the stationary period. The same elev_ can be labbeled at different momement, for instance if the bird move back and forth between location. By default, no label is equivalent to being label with elev_0. Label need to start by elev_ with any ending (exect elev_0 which is kept for the default elevation level.

Flight

Flight are no unique between to consecutive stationary period. However, it will not be required to model all stationary period in the graph model, which essentially allow to combine multiple flights.

Acceleration

If acceleration is present, its labeling will over-write the pressure labeling for compuation stationary period and flight, but not elev_* and discard

New function:

#73

GeoPressureR Update

After updating my GeoPressureR to the latest version, I am getting several errors across multiple analysis scripts when working on my data. The first problem is in 1-pressure.R, in which I upload my pressure data. Was there a change in the pam_read() function? Because it now reads in my pressure data as incorrect values (0.38678, 0.39050, etc.). I have not changed my code and have no idea why it is doing this

`ï..species` when reading the csv file encoding

Error when using flight_bird() on windows only

No match for 'Acrocephalus arundinaceus'. Please use the exact name. 
Closest matches are: 
[1] ï..species  wing_length secondary   mass       
<0 rows> (or 0-length row.names)
--- failed re-building 'wind-graph.Rmd'

Which comes from reading the csv file "avonet_clements.csv":

GeoPressureR/R/flight.R

Lines 38 to 40 in d7c6e91

avonet <- utils::read.csv(system.file("extdata", "avonet_clements.csv",
package = "GeoPressureR"
))

This is apparently a common issue with UTF-8-BOM encoding.

grl - Adding wind

Hitting a wall during 5-2-wind-graph_create.R when adding wind to grl, but ONLY for certain tags (CB594, 598, 599) while others run just fine. Here is my error:

> grl <- graph_add_wind(grl,
+   pressure = pam$pressure,
+   filename = paste0("data/5_wind_graph/", gdl, "/", gdl, "_"),
+   thr_as = gpr$thr_as
+ )
[1] "vobjtovarid4: error #F: I could not find the requsted var (or dimvar) in the file!"
[1] "var (or dimvar) name: level"
[1] "file name: data/5_wind_graph/CB594/CB594_1.nc"
Error in vobjtovarid4(nc, varid, verbose = verbose, allowdimvar = TRUE) : 
  Variable not found

I have re-run the wind download but it still can't find some variable in the .nc files? Any suggestions?

Publish on CRAN

Vignettes + Data makes the code too large for CRAN.
Should to put all the vignette into a seperate repo ? Maybe a bookdown?

read Migrate Technology files

More widely, need to think of a more general system that works with any file type.
Main idea is to focus the process on the data that are needed:

pam <- list(
    pressure=data.frame(date=c(), obs=c())
    light=data.frame(date=c(), obs=c())
    acceleration=data.frame(date=c(), obs=c())
    id=folder
)

Notes:

  • we cannot assume that pressure, light or acceleration are on the same date
  • same file might contains two variables
  • date format might change.

⚠️ Replace act with obs everywhere (breaking change?) and remove pich

Trainset frozen irresponsive waiting loading

When uploading a file too large (around 1 year of 5min dataset), Trainset will become irresponsive like the scrreenshot below
image

This is a javascript error which can't handle too many datapoints at once. The error message visible in your browser console should be

Uncaught RangeError: Maximum call stack size exceeded

I've raised the issue on trainset : Geocene/trainset#117. Please add a +1 so that the author of the package accept my suggestion.

In the meantime, you can use trainset on herokuapp: https://trainset.herokuapp.com/. This is the same app with the fix implemented

More flexible way of loading `GeoPressureViz` data

The only way I can find to use data in a shiny app is to save the data in a .RData in a specific folder (~) and then read it from the app. I cannot find a good way to pass variable to a shiny app instance
Any suggestion?

Export to Movebank / create standard format

What should be the right standard output format?

  • raw data: geolocator data
  • labelization file (pressure and light)
  • geopressure config file
  • graph (too big)
  • result: marginal map (geotiff)
  • result: shortest path
  • result: simulated trajectories

We need to check that all these file are readable/documented and standard name.

Need to check with Movebank what they can upload.

`geopressure_ts` error for location over water

Problem
If the lat,lon requested in geopressure_ts() is over water, GeoPresseAPI returns an empty csv. Currently the error message warns the user of this issue with a link to google map to copy paste. For instance:

> Returned csv file is empty. Check that the time range is none-empty and that the location is not on water: maps.google.com/maps?q=0,0

This problem appear when we query the most likely map of pressure returned by geopressure_map() and geopressure_map2path()

Solution 1
On GeoPressureAPI, we could accept location near water and move them to the closest shore, ideally return a warning message. @mgravey , is that possible? what could be done without reducing computational efficiency?

Solution 2
Check before making the query in R. This would require to use another package, download external data (ERA5-Land is 50MB) otherwise the code below get land polygon for 2MB

 sf::sf_use_s2(FALSE)
 pts <- st_as_sf(path, coords = c("lon","lat"), crs = st_crs(4326))
 # poly <- ne_countries(returnclass="sf")
 poly <- ne_download(category = "physical", type="land", returnclass="sf")
 a <- st_join(pts, poly, join = st_intersects)

`geopressure_check()`

Quick initial check

  • Stationary period duration: warning below 6 hours, danger below 2 hours
  • Flight duration: warning below 2hours, danger below 1 hours?
  • Pressure values: within possible range?,
  • Pressure values: sufficient for each stationary period.
  • Pressure check abrupt change in pressure during stationary period?
  • Pressure check smooth/outliar? Assuming that abrupt change is sufficient

Relying on geopressure_timeseries

  • no pressure outliar (assuming std dev)
  • ...

Return suggestions

  • check elev_id to be the same
  • combine stationary period which are too similar
  • ...

Add visual

create function to plot pam and prob_map

Use of ggplot for that, see #44

Renaming function and variables

See https://github.com/Rafnuss/GeoPressureR/wiki/Migration-v2-%E2%80%90--v3
🎉 Indicates import new exiting feature to check out!
⚠️ Might be confusing at first coming from the v2.
❌ Removed

GeoPressureR object

Old New Comment
pam tag ⚠️ Generic term see this
sta stap Better suited for STAtionary Period.
grl graph More explicit name.
map ⚠️ New likelihood map are stored inside tag. marginal is the only map not stored in tag
path New created by graph_simulation(), graph_most_likely() or tag2path(). Data.frame
pressurepath New Data.frame
param ⚠️ New named list stored inside tag and graph keeping track of all parameters used in all functions. Do not edit manually, but useful to export at the end as json file.

Functions

Tag

Old New Comment
pam_read tag_create
pam_read_check ❌
pam_read_delim_dto tag_create_dto Not exported
tag_label 🎉 New This new single function group all the tag_label_* functions for a more efficient workflow
pam_classify tag_label_auto
trainset_read tag_label_read ⚠️ Note the new trainset_read is not the same as the previous (see Utility section)
trainset_write tag_label_write ⚠️ Note the new trainset_read is not the same as the previous
pam_sta tag_label_stap
tag_set_map 🎉 New Define parameters for the map and stap
tag_update 🎉 New Update a tag with a new label file. Allows to recompute pressure maps only for stap that have change.
tag_assert New Check status of a tag object. Not exported
print.tag New
plot.tag New
plot_tag_pressure New
plot_tag_light New
plot_tag_twilight New
plot_tag_acceleration New
geopressure_map2path tag2path
tag2map New return a GeoPressureR map from a tag

GeoPressure

Old New Comment
geopressure_map 🎉 Run both geopressure_mismatch and geopressure_likelihood together to make the workflow more efficient. Also doesn't store mse and mask by default to save space.
geopressure_map geopressure_map_mismatch make more explicit and consider mismatch to incorporate both mse and mask
geopressure_prob_map geopressure_map_likelihood replace probability by likelihood
geopressure_map_preprocess New Convert the pressure timeseries into an optimal pressure for geopressureapi
geopressure_ts geopressure_timseries ⚠️ Make name more explicit despite long.

GeoLight

Old New Comment
find_twilights twilight_create
twilight_label_write New
twilight_label_read New
geolight_map 🎉 New function that create the light likelihood map in tag, similar to geopressure_map.
solar geolight_solar
zenith geolight_zenith
refracted geolight_refracted
ligh2mat New Convert a timeserie of light into a matrix of day of year, time of day

Graph

Old New Comment
graph_create
graph_trim graph_create_prune Not exported
graph_download_wind tag_download_wind ⚠️ Use tag object and not graph
graph_add_wind
graph_set_movement 🎉 New Defining all parameters for the movement model without actually computin anything.
graph_transition New Allow to compute the transition matrix on the fly, thus saving memory
flight_bird create_bird
print.bird New Check a bird object
flight_power speed2power Not exported
flight_prob speed2prob More descriptive of what the function does
graph_marginal
graph_simulation
graph_most_likely 🎉 New more efficient implementation than using the igraph package.
graph_assert New Check status of a graph object
print.graph New

Map

Old New Comment
map_create 🎉 New GeoPreussreR map are now S3 objects
print.map New
plot.map New
rast.map New convert a map to a terra::rast
map_expend New map are defined by only extend and scale, this function allows to compute lat, lon and dim

Path

Old New Comment
path2df ind2path
graph_path2edge path2edge
plot.path New

Pressurepath

Old New Comment
geopressure_ts_path pressurepath_create ⚠️ Focus on the main outcome (pressurepath)
pressurepath_update 🎉 New Update a pressurepath with a new label file. similar to tag_update.
plot.pressurepath New
pressurepath2altitude New

Param

Old New Comment
param_create New
print.param New

Flight

❌ Avoid flight which can be computed "on the fly" from stap using stap2flight()

Utility

Old New Comment
trainset_read New more generic function taking any df as input as to work with tag_label and twilight_label. Not exported
trainset_write New more generic function taking any df as input as to work with tag_label and twilight_label. Not exported
stap2flight New
stap2duration New
graph_path2lonlat ❌
geopressureviz

Variables

Tag

Old New Comment
pam tag
id
min_duration
pathname directory see this
file basename would be better but less user friendly see this
pressure_file
light_file
acceleration_file
path sensor_path dirname+basename
crop_start
crop_end
skip
col
date_format

Set map

Old New Comment
extent
scale
known New data.frame defining the knwon position (e.g., equipement and retrieval). Use to avoid computing the likelihood map for nothing and calibration of light
include_stap_id New
include_min_duration New

GeoPressure

Old New Comment
pressure
max_sample
margin
timeout
workers
presssure_maps ❌ ⚠️ Remove and integreated within geostap
s sd Match classic R base function name for standard deviation.
thr thr_mask Make more explicit
fun_w log_linear_pooling_weight Aligned with geolight aggregation
keep_mse_mask
lon
lat
start_time
end_time
verbose quiet
path
include_flight

Geolight

Old New Comment
light
threshold twl_thr
shift_k twl_offset Align with TwGeos but adds twl in front
twl twilight ⚠️ more explicit
adjust_calib twl_calib_adjust
w_llp twl_llp ⚠️ Not to confuse with llp in geopressure
zenith
sun
tm date Align with the format

Graph

Old New Comment
grl graph
static_prob ❌ ⚠️ Remove and integreated within tag_set_map
field Define which field in geostap should be use as likelihood
thr_prob_percentile thr_likelihood
thr_gs
thr_as
area extent
cds_key
cds_user
type
nj

Field of graph

Old New Comment
s
t
as ❌ avoid storing unnecessary (computable from gs and ws)
gs
ws
ps obs Stored as 3D matrix rather than edge level for memory efficiency.
p transition ⚠️ Now optional. When possible, use movement which can compute trans on the flight with graph_trans and save memory
movement Parameter to be used in graph_trans
Parameter to be used in graph_trans
sz
equipment
retrieval
flight_duration ❌ Computable with stap2flight
lat ❌ Computable with extent_scale2grid
lon ❌ Computable with extent_scale2grid
extent
resolution ❌
temporal_extent ❌
stap
scale
mask_water

Utility

Old New Comment
species_name
mass
wing_span
wing_aspect
wing_area
body_frontal_area
bird not super specific
speed slightly different than as,gs, ws
method
shape
scale ⚠️ Not to confuse with scale in geostap
location
fun_power power2prob
low_speed_fix

Improve `graph_download_wind()`

  • - Have a transfer=FALSE, in which case, it returns the requests and have an optional field request which simply perform the requests. I guess we could be using wf_transfer and wf_request (without batch)
  • - How to make these call in parrellel? No need
  • - Have an option sta_id to download only specific sta_id.
  • - different function file? No need

see bluegreen-labs/ecmwfr#103

ToDo

Here are the vignettes I'm planning to work on

  • check english basic Example
  • restructure Labelling Tracks
  • geopositioning with light
  • add figure for geopositions by light
  • Impact of log-linear pooling
  • Building a graph
  • Ask review for geoposition by light by @amelinenussbaumer
  • review name/title/filename of the vignettes.
  • Change list of raster to brick all throughout the code
  • Used the animate() function of raster for illustrations
  • check moveVis for visualiztion of simulation
  • write gibbs sampler for gridded
  • write readme
  • review the visuals
  • Rename pam$acceleration$class and pam$pressure$class to isMig and isOutliar
  • Add wind computation in graph
  • check issue with ubuntu cmd check
  • solve issue with 18LX path.
  • create graph with smaller number of stationary period
  • implement shiny app to dynamically change the position of the bird and resulting path/etc
  • Create visualisation of windspeed and check windspeed computation
  • check test 1 of labelling
  • Change the line max of lintr to 100 everywhere
  • write citation file
  • plot matrix light
  • Write template repository
  • Compute energy and plot energy
  • run all vignette with eval=F to check that everything is working
  • Compute altitude of flight
  • Edit doc with sealso and family https://r-pkgs.org/man.html#roxygen-comments
  • check english figures and logic throughout @amelinenussbaumer
  • Allow to download a pressure netcdf file to work offline
  • Write a function to output format of movebank and else (geotiff)

More flexible way of loading `GeoPressureViz` data

The only way I can find to use data in a shiny app is to save the data in a .RData in a specific folder (~) and then read it from the app. I cannot find a good way to pass variable to a shiny app instance
Any suggestion?

Error reading geotiff from url with raster::brick() on ubuntu

Running a simple request of geopressure_map on a ubuntu machine returns the following error:

Generate requests:

Requests generated successfully for 1 stationary periods (1, )

Send requests:

[==================================================] 1 / 1 
Download geotiff:

[                                                  ] 0 / 1 
Error in .rasterObjectFromFile(x, objecttype = "RasterBrick", ...): Cannot create a RasterLayer object from this file. (file does not exist)
Traceback:

1. geopressure_map(pressure, extent)
2. future::value(f[[i_u]])
3. value.Future(f[[i_u]])
4. signalConditions(future, exclude = getOption("future.relay.immediate", 
 .     "immediateCondition"), resignal = TRUE)

This error comes from reading the line:

GeoPressureR/R/geopressure.R

Line 182 in 29dc1e7

raster::brick(uris[i_u])

If I run it separately i get more information on the error:

Error in .local(.Object, ...): An error occurred while creating a virtual connection to the DAP server:
Error while reading the URL: https://earthengine.googleapis.com/v1alpha/projects/earthengine-legacy/thumbnails/772ae2eefc3bebadd6131ff354c024cb-ea5496300e42cb031021bb837f34f37d:getPixels.ver.
The OPeNDAP server returned the following message:
Not Found: The data source or server could not be found.
        Often this means that the OPeNDAP server is missing or needs attention;
        Please contact the server administrator.

Traceback:

1. GDALinfo(uris[i_u])
2. GDAL.open(fname, silent = silent, allowedDrivers = allowedDrivers, 
 .     options = options)
3. new("GDALReadOnlyDataset", filename, silent = silent, allowedDrivers = allowedDrivers, 
 .     options = options)
4. initialize(value, ...)
5. initialize(value, ...)
6. .local(.Object, ...)

I couldn't find a good solution, it seems to come from the driver dods. See https://trac.osgeo.org/gdal/ticket/2696. I have tried to set Sys.setenv(GDAL_SKIP="DODS") but this wouldn't work.

Export to Movebank / create standard format

Which ouput?

  • raw data: light (date, value), pressure (date, value), acceleration etc...
  • labelization file (pressure and light)
  • geopressure config file
  • graph (quite large?)
  • result: marginal map (geotiff)
  • result: shortest path
  • result: simulated trajectories
  • result: wind

We need to check that all these file are readable/documented and standard name.

Need to check with Movebank what they can upload.

See also

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.