Giter Site home page Giter Site logo

pypsa / atlite Goto Github PK

View Code? Open in Web Editor NEW
240.0 14.0 80.0 18.74 MB

Atlite: A Lightweight Python Package for Calculating Renewable Power Potentials and Time Series

Home Page: https://atlite.readthedocs.io

Python 100.00%
python energy energy-system energy-systems wind pv era5 renewable-timeseries renewable-energy potentials

atlite's Introduction

PyPSA - Python for Power System Analysis

PyPI version Conda version CI CI with micromamba Code coverage Documentation Status License Zenodo Examples of use pre-commit.ci status Code style: black Discord Contributor Covenant Stack Exchange questions

PyPSA stands for "Python for Power System Analysis". It is pronounced "pipes-ah".

PyPSA is an open source toolbox for simulating and optimising modern power and energy systems that include features such as conventional generators with unit commitment, variable wind and solar generation, storage units, coupling to other energy sectors, and mixed alternating and direct current networks. PyPSA is designed to scale well with large networks and long time series.

This project is maintained by the Department of Digital Transformation in Energy Systems at the Technical University of Berlin. Previous versions were developed by the Energy System Modelling group at the Institute for Automation and Applied Informatics at the Karlsruhe Institute of Technology funded by the Helmholtz Association, and by the Renewable Energy Group at FIAS to carry out simulations for the CoNDyNet project, financed by the German Federal Ministry for Education and Research (BMBF) as part of the Stromnetze Research Initiative.

Functionality

PyPSA can calculate:

  • static power flow (using both the full non-linear network equations and the linearised network equations)
  • linear optimal power flow (least-cost optimisation of power plant and storage dispatch within network constraints, using the linear network equations, over several snapshots)
  • security-constrained linear optimal power flow
  • total electricity/energy system least-cost investment optimisation (using linear network equations, over several snapshots and investment periods simultaneously for optimisation of generation and storage dispatch and investment in the capacities of generation, storage, transmission and other infrastructure)

It has models for:

  • meshed multiply-connected AC and DC networks, with controllable converters between AC and DC networks
  • standard types for lines and transformers following the implementation in pandapower
  • conventional dispatchable generators and links with unit commitment
  • generators with time-varying power availability, such as wind and solar generators
  • storage units with efficiency losses
  • simple hydroelectricity with inflow and spillage
  • coupling with other energy carriers (e.g. resistive Power-to-Heat (P2H), Power-to-Gas (P2G), battery electric vehicles (BEVs), Fischer-Tropsch, direct air capture (DAC))
  • basic components out of which more complicated assets can be built, such as Combined Heat and Power (CHP) units and heat pumps.

Documentation

Documentation

Quick start

Examples

Known users of PyPSA

Installation

pip:

pip install pypsa

conda/mamba:

conda install -c conda-forge pypsa

Additionally, install a solver.

Usage

import pypsa

# create a new network
n = pypsa.Network()
n.add("Bus", "mybus")
n.add("Load", "myload", bus="mybus", p_set=100)
n.add("Generator", "mygen", bus="mybus", p_nom=100, marginal_cost=20)

# load an example network
n = pypsa.examples.ac_dc_meshed()

# run the optimisation
n.optimize()

# plot results
n.generators_t.p.plot()
n.plot()

# get statistics
n.statistics()
n.statistics.energy_balance()

There are more extensive examples available as Jupyter notebooks. They are also described in the doc/examples.rst and are available as Python scripts in examples/.

Screenshots

PyPSA-Eur optimising capacities of generation, storage and transmission lines (9% line volume expansion allowed) for a 95% reduction in CO2 emissions in Europe compared to 1990 levels

image

SciGRID model simulating the German power system for 2015.

image

image

Dependencies

PyPSA is written and tested to be compatible with Python 3.7 and above. The last release supporting Python 2.7 was PyPSA 0.15.0.

It leans heavily on the following Python packages:

  • pandas for storing data about components and time series
  • numpy and scipy for calculations, such as linear algebra and sparse matrix calculations
  • networkx for some network calculations
  • matplotlib for static plotting
  • linpy for preparing optimisation problems (currently only linear and mixed integer linear optimisation)
  • cartopy for plotting the baselayer map
  • pytest for unit testing
  • logging for managing messages

The optimisation uses interface libraries like linopy which are independent of the preferred solver. You can use e.g. one of the free solvers GLPK and CLP/CBC or the commercial solver Gurobi for which free academic licenses are available.

Documentation

Please check the documentation.

Contributing and Support

We strongly welcome anyone interested in contributing to this project. If you have any ideas, suggestions or encounter problems, feel invited to file issues or make pull requests on GitHub.

  • In case of code-related questions, please post on stack overflow.
  • For non-programming related and more general questions please refer to the mailing list.
  • To discuss with other PyPSA users, organise projects, share news, and get in touch with the community you can use the discord server.
  • For bugs and feature requests, please use the PyPSA Github Issues page.
  • For troubleshooting, please check the troubleshooting in the documentation.

Code of Conduct

Please respect our code of conduct.

Citing PyPSA

If you use PyPSA for your research, we would appreciate it if you would cite the following paper:

Please use the following BibTeX:

@article{PyPSA,
   author = {T. Brown and J. H\"orsch and D. Schlachtberger},
   title = {{PyPSA: Python for Power System Analysis}},
   journal = {Journal of Open Research Software},
   volume = {6},
   issue = {1},
   number = {4},
   year = {2018},
   eprint = {1707.09913},
   url = {https://doi.org/10.5334/jors.188},
   doi = {10.5334/jors.188}
}

If you want to cite a specific PyPSA version, each release of PyPSA is stored on Zenodo with a release-specific DOI. The release-specific DOIs can be found linked from the overall PyPSA Zenodo DOI for Version 0.17.1 and onwards:

image

or from the overall PyPSA Zenodo DOI for Versions up to 0.17.0:

image

Licence

Copyright 2015-2024 PyPSA Developers

PyPSA is licensed under the open source MIT License.

atlite's People

Contributors

calvintr avatar coroa avatar davide-f avatar energyls avatar enrilizaga avatar euronion avatar fabianhofmann avatar fneum avatar hailiangliu89 avatar joaschauer avatar leonsn avatar lkstrp avatar lukasfrankenq avatar martavp avatar nworbmot avatar ovewh avatar p-glaum avatar parisra avatar pre-commit-ci[bot] avatar pz-max avatar richardscottoz avatar schlachtberger avatar tasqu avatar thesethtruth avatar zoltanmaric avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

atlite's Issues

[v0.2] Permission error with Cutout.prepare() on Windows.

Occuring in new version branch v0.2

Code

When running the standard pattern for creating a ERA5 cutout (see examples/create_cutout.py):

cutout = atlite.Cutout(name="europe-2011-01",
                                      module="era5",
                                       xs=slice(-12.18798349, 41.56244222),
                                       ys=slice(71.65648314, 33.56459975),
                                       years=slice(2011, 2011),
                                       months=slice(1,1))

Exception

I get this exception when atlite tries to delete the temporary files downloaded for creating the cutout:

INFO:atlite.datasets.common:Deleting file C:\Users<User>\AppData\Local\Temp\tmpmsdmnmk8.nc
Exception ignored in: <finalize object at 0x25f484f2850; dead>
Traceback (most recent call last):
File "C:\anaconda\envs\atlite\lib\weakref.py", line 548, in call
return info.func(*info.args, **(info.kwargs or {}))
File "C:\Users<User>\Documents\GitHub\atlite\atlite\datasets\common.py", line 19, in noisy_unlink
os.unlink(path)
PermissionError: [WinError 32] The process cannot access the file because it is being used by another process: 'C:\Users\\AppData\Local\Temp\tmpmsdmnmk8.nc'

The error occurs multiple times for all the temp files created. They are not properly cleaned up.

Plattform

  • Windows 10 (haven't tested it on Linux yet)
  • Python 3.6.6

Related lines of code

Should be related to the new way of handling the files with:

https://github.com/FRESNA/atlite/blob/6f3559d5e20a05c79fa55596623c732930581178/atlite/datasets/common.py#L17-L19

and

-unrelated line of code removed-

Necessity of Sarah SIS and SID Interpolation

The original sarah datasets, SID and SIS, have both a band of nans along the earth where dawn and nightfall takes place. On one side you find mostly values > 0, on the other only values of 0.

sarah_sid_10
sarah_sis_20

Atm we are extensively interpolating those areas with a 1d interpolation, which for large cutouts takes a lot of time. The question raises, is the interpolation necessary.

Problem locally installing `v0.2` using `pip`

I am unable to locally install the current version with pip (pip install .).
I am using the environment.yaml file provided.

The following error message pops up when trying install it:

(atlite) pypsa@vm-pypsa:~/share/GitHub/atlite$ git remote -v
origin  https://github.com/PyPSA/atlite.git (fetch)
origin  https://github.com/PyPSA/atlite.git (push)
(atlite) pypsa@vm-pypsa:~/share/GitHub/atlite$ git status
On branch v0.2
Your branch is up-to-date with 'origin/v0.2'.

nothing to commit, working tree clean
(atlite) pypsa@vm-pypsa:~/share/GitHub/atlite$ pip install .
Processing /home/pypsa/share/GitHub/atlite
    ERROR: Command errored out with exit status 1:
     command: /home/pypsa/miniconda3/envs/atlite/bin/python -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/tmp/pip-req-build-egexmm80/setup.py'"'"'; __file__='"'"'/tmp/pip-req-build-egexmm80/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' egg_info --egg-base /tmp/pip-req-build-egexmm80/pip-egg-info
         cwd: /tmp/pip-req-build-egexmm80/
    Complete output (27 lines):
    Traceback (most recent call last):
      File "<string>", line 1, in <module>
      File "/tmp/pip-req-build-egexmm80/setup.py", line 52, in <module>
        'Operating System :: OS Independent',
      File "/home/pypsa/miniconda3/envs/atlite/lib/python3.7/site-packages/setuptools/__init__.py", line 160, in setup
        _install_setup_requires(attrs)
      File "/home/pypsa/miniconda3/envs/atlite/lib/python3.7/site-packages/setuptools/__init__.py", line 155, in _install_setup_requires
        dist.fetch_build_eggs(dist.setup_requires)
      File "/home/pypsa/miniconda3/envs/atlite/lib/python3.7/site-packages/setuptools/dist.py", line 698, in fetch_build_eggs
        replace_conflicting=True,
      File "/home/pypsa/miniconda3/envs/atlite/lib/python3.7/site-packages/pkg_resources/__init__.py", line 783, in resolve
        replace_conflicting=replace_conflicting
      File "/home/pypsa/miniconda3/envs/atlite/lib/python3.7/site-packages/pkg_resources/__init__.py", line 1066, in best_match
        return self.obtain(req, installer)
      File "/home/pypsa/miniconda3/envs/atlite/lib/python3.7/site-packages/pkg_resources/__init__.py", line 1078, in obtain
        return installer(requirement)
      File "/home/pypsa/miniconda3/envs/atlite/lib/python3.7/site-packages/setuptools/dist.py", line 754, in fetch_build_egg
        return fetch_build_egg(self, req)
      File "/home/pypsa/miniconda3/envs/atlite/lib/python3.7/site-packages/setuptools/installer.py", line 133, in fetch_build_egg
        wheel.install_as_egg(dist_location)
      File "/home/pypsa/miniconda3/envs/atlite/lib/python3.7/site-packages/setuptools/wheel.py", line 99, in install_as_egg
        self._install_as_egg(destination_eggdir, zf)
      File "/home/pypsa/miniconda3/envs/atlite/lib/python3.7/site-packages/setuptools/wheel.py", line 107, in _install_as_egg
        self._convert_metadata(zf, destination_eggdir, dist_info, egg_info)
      File "/home/pypsa/miniconda3/envs/atlite/lib/python3.7/site-packages/setuptools/wheel.py", line 128, in _convert_metadata
        os.mkdir(destination_eggdir)
    FileExistsError: [Errno 17] File exists: '/tmp/pip-req-build-egexmm80/.eggs/setuptools_scm-4.1.2-py3.7.egg'
    ----------------------------------------
ERROR: Command errored out with exit status 1: python setup.py egg_info Check the logs for full command output.

Installing it directly from GitHub works fine, i.e. this works w/o problems:

(atlite) pypsa@vm-pypsa:~/share/GitHub/atlite$ pip install git+https://github.com/pypsa/[email protected]
Collecting git+https://github.com/pypsa/[email protected]
  Cloning https://github.com/pypsa/atlite.git (to revision v0.2) to /tmp/pip-req-build-v4bvq5v1
  Running command git clone -q https://github.com/pypsa/atlite.git /tmp/pip-req-build-v4bvq5v1
  Running command git checkout -b v0.2 --track origin/v0.2
  Switched to a new branch 'v0.2'
  Branch 'v0.2' set up to track remote branch 'v0.2' from 'origin'.
Requirement already satisfied: numpy in /home/pypsa/miniconda3/envs/atlite/lib/python3.7/site-packages (from atlite==0.0.4.dev330+g2fbdd9f) (1.18.1)
Requirement already satisfied: scipy in /home/pypsa/miniconda3/envs/atlite/lib/python3.7/site-packages (from atlite==0.0.4.dev330+g2fbdd9f) (1.4.1)
Requirement already satisfied: pandas>=0.22 in /home/pypsa/miniconda3/envs/atlite/lib/python3.7/site-packages (from atlite==0.0.4.dev330+g2fbdd9f) (1.0.3)
Requirement already satisfied: bottleneck in /home/pypsa/miniconda3/envs/atlite/lib/python3.7/site-packages (from atlite==0.0.4.dev330+g2fbdd9f) (1.3.2)
Requirement already satisfied: numexpr in /home/pypsa/miniconda3/envs/atlite/lib/python3.7/site-packages (from atlite==0.0.4.dev330+g2fbdd9f) (2.7.1)
Requirement already satisfied: xarray>=0.11.2 in /home/pypsa/miniconda3/envs/atlite/lib/python3.7/site-packages (from atlite==0.0.4.dev330+g2fbdd9f) (0.15.1)
Requirement already satisfied: netcdf4 in /home/pypsa/miniconda3/envs/atlite/lib/python3.7/site-packages (from atlite==0.0.4.dev330+g2fbdd9f) (1.4.2)
Requirement already satisfied: dask>=0.18.0 in /home/pypsa/miniconda3/envs/atlite/lib/python3.7/site-packages (from atlite==0.0.4.dev330+g2fbdd9f) (2.17.2)
Requirement already satisfied: requests in /home/pypsa/miniconda3/envs/atlite/lib/python3.7/site-packages (from atlite==0.0.4.dev330+g2fbdd9f) (2.23.0)
Requirement already satisfied: rtree in /home/pypsa/miniconda3/envs/atlite/lib/python3.7/site-packages (from atlite==0.0.4.dev330+g2fbdd9f) (0.9.4)
Requirement already satisfied: pyyaml in /home/pypsa/miniconda3/envs/atlite/lib/python3.7/site-packages (from atlite==0.0.4.dev330+g2fbdd9f) (5.3.1)
Requirement already satisfied: rasterio>=1.0 in /home/pypsa/miniconda3/envs/atlite/lib/python3.7/site-packages (from atlite==0.0.4.dev330+g2fbdd9f) (1.1.0)
Requirement already satisfied: shapely in /home/pypsa/miniconda3/envs/atlite/lib/python3.7/site-packages (from atlite==0.0.4.dev330+g2fbdd9f) (1.7.0)
Requirement already satisfied: progressbar2 in /home/pypsa/miniconda3/envs/atlite/lib/python3.7/site-packages (from atlite==0.0.4.dev330+g2fbdd9f) (3.37.1)
Requirement already satisfied: geopandas in /home/pypsa/miniconda3/envs/atlite/lib/python3.7/site-packages (from atlite==0.0.4.dev330+g2fbdd9f) (0.6.1)
Requirement already satisfied: cdsapi in /home/pypsa/miniconda3/envs/atlite/lib/python3.7/site-packages (from atlite==0.0.4.dev330+g2fbdd9f) (0.2.7)
Requirement already satisfied: pytz>=2017.2 in /home/pypsa/miniconda3/envs/atlite/lib/python3.7/site-packages (from pandas>=0.22->atlite==0.0.4.dev330+g2fbdd9f) (2020.1)
Requirement already satisfied: python-dateutil>=2.6.1 in /home/pypsa/miniconda3/envs/atlite/lib/python3.7/site-packages (from pandas>=0.22->atlite==0.0.4.dev330+g2fbdd9f) (2.8.1)
Requirement already satisfied: setuptools>=41.2 in /home/pypsa/miniconda3/envs/atlite/lib/python3.7/site-packages (from xarray>=0.11.2->atlite==0.0.4.dev330+g2fbdd9f) (47.1.1.post20200604)
Requirement already satisfied: cftime in /home/pypsa/miniconda3/envs/atlite/lib/python3.7/site-packages (from netcdf4->atlite==0.0.4.dev330+g2fbdd9f) (1.1.2)
Requirement already satisfied: idna<3,>=2.5 in /home/pypsa/miniconda3/envs/atlite/lib/python3.7/site-packages (from requests->atlite==0.0.4.dev330+g2fbdd9f) (2.9)
Requirement already satisfied: chardet<4,>=3.0.2 in /home/pypsa/miniconda3/envs/atlite/lib/python3.7/site-packages (from requests->atlite==0.0.4.dev330+g2fbdd9f) (3.0.4)
Requirement already satisfied: urllib3!=1.25.0,!=1.25.1,<1.26,>=1.21.1 in /home/pypsa/miniconda3/envs/atlite/lib/python3.7/site-packages (from requests->atlite==0.0.4.dev330+g2fbdd9f) (1.25.8)
Requirement already satisfied: certifi>=2017.4.17 in /home/pypsa/miniconda3/envs/atlite/lib/python3.7/site-packages (from requests->atlite==0.0.4.dev330+g2fbdd9f) (2020.4.5.1)
Requirement already satisfied: attrs in /home/pypsa/miniconda3/envs/atlite/lib/python3.7/site-packages (from rasterio>=1.0->atlite==0.0.4.dev330+g2fbdd9f) (19.3.0)
Requirement already satisfied: snuggs>=1.4.1 in /home/pypsa/miniconda3/envs/atlite/lib/python3.7/site-packages (from rasterio>=1.0->atlite==0.0.4.dev330+g2fbdd9f) (1.4.7)
Requirement already satisfied: cligj>=0.5 in /home/pypsa/miniconda3/envs/atlite/lib/python3.7/site-packages (from rasterio>=1.0->atlite==0.0.4.dev330+g2fbdd9f) (0.5.0)
Requirement already satisfied: affine in /home/pypsa/miniconda3/envs/atlite/lib/python3.7/site-packages (from rasterio>=1.0->atlite==0.0.4.dev330+g2fbdd9f) (2.3.0)
Requirement already satisfied: click<8,>=4.0 in /home/pypsa/miniconda3/envs/atlite/lib/python3.7/site-packages (from rasterio>=1.0->atlite==0.0.4.dev330+g2fbdd9f) (7.1.2)
Requirement already satisfied: click-plugins in /home/pypsa/miniconda3/envs/atlite/lib/python3.7/site-packages (from rasterio>=1.0->atlite==0.0.4.dev330+g2fbdd9f) (1.1.1)
Requirement already satisfied: python-utils>=2.3.0 in /home/pypsa/miniconda3/envs/atlite/lib/python3.7/site-packages (from progressbar2->atlite==0.0.4.dev330+g2fbdd9f) (2.3.0)
Requirement already satisfied: six in /home/pypsa/miniconda3/envs/atlite/lib/python3.7/site-packages (from progressbar2->atlite==0.0.4.dev330+g2fbdd9f) (1.15.0)
Requirement already satisfied: fiona in /home/pypsa/miniconda3/envs/atlite/lib/python3.7/site-packages (from geopandas->atlite==0.0.4.dev330+g2fbdd9f) (1.8.11)
Requirement already satisfied: pyproj in /home/pypsa/miniconda3/envs/atlite/lib/python3.7/site-packages (from geopandas->atlite==0.0.4.dev330+g2fbdd9f) (2.6.1.post1)
Requirement already satisfied: tqdm in /home/pypsa/miniconda3/envs/atlite/lib/python3.7/site-packages (from cdsapi->atlite==0.0.4.dev330+g2fbdd9f) (4.46.0)
Requirement already satisfied: pyparsing>=2.1.6 in /home/pypsa/miniconda3/envs/atlite/lib/python3.7/site-packages (from snuggs>=1.4.1->rasterio>=1.0->atlite==0.0.4.dev330+g2fbdd9f) (2.4.7)
Requirement already satisfied: munch in /home/pypsa/miniconda3/envs/atlite/lib/python3.7/site-packages (from fiona->geopandas->atlite==0.0.4.dev330+g2fbdd9f) (2.5.0)
Building wheels for collected packages: atlite
  Building wheel for atlite (setup.py) ... done
  Created wheel for atlite: filename=atlite-0.0.4.dev330+g2fbdd9f-py2.py3-none-any.whl size=63453 sha256=d0a94b784da2cb309c2b0913df8c9e52023788bf8eaf852cad4ee594e5c033e9
  Stored in directory: /tmp/pip-ephem-wheel-cache-53fc5lbo/wheels/8b/16/63/70c28877db9b6c491a9550dbcc78dd21b3704cc10c6d9ca4f8
Successfully built atlite
Installing collected packages: atlite
Successfully installed atlite-0.0.4.dev330+g2fbdd9f

Is this intended behaviour?
I presume it is related to the changes with how the versioning works.

heat_demand with chunked datasets

@coroa pointed out that the heat_demand might have problems when chunking (at least there were some issues with windows). A time shift in the time coordinates might lead to loosing values at the border of chunks. This seems to be no problem for dask arrays:

When looking at the results of chunked and non-chunked datasets, the result for heat_demand is exactly the same.

heat_demand_chunked

Numerical issues in ERA5 data

I am using ERA5 runoff data to determine inflow time series for hydro power stations. The time series contain large negative values (-1e5) which look odd as they physically don't make sense.

I tried to trace them back to their origin. I am not exactly sure where and why this happens, but I am pretty sure it happens within cutout.prepare(): I ran a comparison of data downloaded with atlite and data downloaded directly from CDS and found that those values deviate slightly and that runoff data is always >= 0 for raw CDS data, but can be <0 for atlite data.

The differences are tiny and very likely based on numerics. I am still reporting this here for three reasons: First, they may be tiny after download, but blow up to -1e5 in my case for inflow data. Second, because the data is not processed and because it should always be stored as float32, I don't exactly understand why the data changes. Third, this report may be of help for others running into the same issue.

Atlite version: 62897

Add cutout creation for EURO-CORDEX

Hi, @euronion, I have several questions regarding the documentation of Atlite with Cordex dataset, which I think maybe suitable to put here.

Do you have any example/document of Atlite (the current version V0.0.2 or the upcoming V0.2) working with the Cordex data?
It seems that I should download all the 9 variables (influx, outflux, etc...) and store them into 9 folders in the same directory?
Do the variable "CWT" mean "compute working team"? I cannot find this variable in the Cordex output variable list.
Is it possible to prepare a cutout with only a subset of the 9 variables with Cordex?

Thank you in advance!

Originally posted by @fang1103 in #57 (comment)

Add weather-dependent time-series for transmission line capacities

Detailed Description

Transmission capacities are influenced by the weather. We should add a conversion function for the cooling effect of transmission lines.

Possible Implementation

Goal:
Add a conversion function transmission/transmission_availability (or different) in atlite.convert.

Idea: Assume a set of transmission lines given as line shapes.

  1. For each line, calculate overlapping weather cells -> use indicator_matrix if it works between properly between line shapes and area shapes. Otherwise define a intersection matrix (booleans would probably be sufficient) .
  2. @nworbmot suggested to take the maximum temperature along the line into account. This might be tricky and requires to define a new aggregation function in atlite.aggregate. For a first shot, we could instead use the mean temperature over the whole line.
  3. Enable wind angle data in cutouts. This variable is nowhere in the supported features. Probably it makes sense to enable a direct retrieving of specific ERA5 variables.
  4. Think about a clever implementation how to efficiently calculate the angle between wind direction and line direction. Probably it makes sense to calculate the to scale the (u, v) components (see table 2 in ERA5 data doc) to the components of a unit vector and do the same for the line direction. Then, calculate the dot product. if I'm not mistaken the current aggregation function would work for the average angle between wind and line.
  5. Do research and derive a formula for the combined cooling effect of temperature, wind speed + angle on the conduction capacity.

[v0.2] Deprecation warnings are ignored by default

Uh annoying, they worked wonders in Python 2.7. The default has been changed to ignore them in 3.2 unless you add -Wd to your python interpreter or change the filter as you've shown. Hmm .. if I understand the manual correctly it's not intended for libraries to set captureWarnings or change the filter. I suppose we should warn in the manual about that and change the examples accordingly.

Do you think logging.captureWarnings(True) or warnings.simplefilter('always', DeprecationWarning) should be used? I'm tending towards the former.

Originally posted by @coroa in #23

PermissionError during temp file cleanup after cutout download (Windows)

Encountering the following error when creating a cutout (ERA5):

INFO:cdsapi:Download rate 3.8M/s

[                                        ] | 0% Completed |  0.0s
[########################################] | 100% Completed |  0.3s

---------------------------------------------------------------------------
PermissionError                           Traceback (most recent call last)
<ipython-input-5-3416e809141b> in <module>
      8 
      9 # This is where all the work happens (this can take some time, for us it took ~15 minutes).
---> 10 cutout.prepare()

c:\users\j. hampp\documents\github\atlite\atlite\data.py in cutout_prepare(cutout, features, freq, tmpdir, overwrite)
    226     finally:
    227         if not keep_tmpdir:
--> 228             rmtree(tmpdir)
    229 
    230     # Re-open

C:\anaconda\envs\atlite\lib\shutil.py in rmtree(path, ignore_errors, onerror)
    511             # can't continue even if onerror hook returns
    512             return
--> 513         return _rmtree_unsafe(path, onerror)
    514 
    515 # Allow introspection of whether or not the hardening against symlink

C:\anaconda\envs\atlite\lib\shutil.py in _rmtree_unsafe(path, onerror)
    395                 os.unlink(fullname)
    396             except OSError:
--> 397                 onerror(os.unlink, fullname, sys.exc_info())
    398     try:
    399         os.rmdir(path)

C:\anaconda\envs\atlite\lib\shutil.py in _rmtree_unsafe(path, onerror)
    393         else:
    394             try:
--> 395                 os.unlink(fullname)
    396             except OSError:
    397                 onerror(os.unlink, fullname, sys.exc_info())

PermissionError: [WinError 32] The process cannot access the file because it is being used by another process: 'C:\\Users\\J705E~1.HAM\\AppData\\Local\\Temp\\tmplaut1buy\\tmp6jjdud34.nc'
  • Error seems new, haven't encountered it before.
  • The download finishes, but no cutout.data data variables are available.
  • Probably Windows related (Filehandlers ...)
  • Possibly related to the changes made in 9273a15

[v0.2] Large memory overhead in cutout.pv

Different topic:
I failed to run cutout.pv(...) for the last days and found the problem to be insufficient memory (the script ran slowly and crashed at one point in time without a tractable error).
I solved it with windows="D", which requires less data to be loaded into memory at the same time.

@coroa : What do you think about making windows="D" default for the convert operations?

Originally posted by @euronion in #20 (comment)

Cutout.prepare() no reponse

Code

When running the standard pattern for creating an ERA5 cutout (https://atlite.readthedocs.io/en/latest/examples/plotting_with_atlite.html)

import geopandas as gpd
import cartopy.io.shapereader as shpreader
import atlite
import logging

logging.basicConfig(level=logging.INFO)
shpfilename = shpreader.natural_earth(resolution='10m',
                                      category='cultural',
                                      name='admin_0_countries')
reader = shpreader.Reader(shpfilename)
UkIr = gpd.GeoSeries({r.attributes['NAME_EN']: r.geometry
                      for r in reader.records()},
                     crs={'init': 'epsg:4326'}
                     ).reindex(['United Kingdom', 'Ireland'])

# Define the cutout; this will not yet trigger any major operations
cutout = atlite.Cutout(name="uk-2011-01",
                       cutout_dir="./",
                       module="era5",
                       bounds=UkIr.unary_union.bounds,
                       years=slice(2011,2011),
                       months=slice(1,1))

# This is where all the work happens.
cutout.prepare()

Error

I got the following message when I run the code in Spyder:


> C:\Users\Yiping\Miniconda3\lib\site-packages\pyproj\crs\crs.py:55: FutureWarning: '+init=<authority>:<code>' syntax is deprecated. '<authority>:<code>' is the preferred initialization method. When making the change, be mindful of axis order changes: https://pyproj4.github.io/pyproj/stable/gotchas.html#axis-order-changes-in-proj-6
>   return _prepare_from_string(" ".join(pjargs))
> INFO:atlite.cutout:Cutout uk-2011-01 not found in directory ./, building new one
> INFO:cdsapi:Sending request to https://cds.climate.copernicus.eu/api/v2/resources/reanalysis-era5-single-levels
> INFO:cdsapi:Request is completed
> INFO:atlite.datasets.era5:Downloading request for 9 variables to C:\Users\Yiping\AppData\Local\Temp\tmpqrvzeks3.nc
> INFO:cdsapi:Downloading http://136.156.132.235/cache-compute-0006/cache/data7/adaptor.mars.internal-1583414461.1578808-27797-3-b18c0f99-b457-440f-990b-0daa709defa9.nc to C:\Users\Yiping\AppData\Local\Temp\tmpqrvzeks3.nc (126.6K)
> INFO:cdsapi:Download rate 846.1K/s        
> INFO:atlite.preparation:Starting preparation of cutout 'uk-2011-01'
> INFO:atlite.preparation:1 tasks have been collected. Starting running them on all processors.

It seems to be working, but the problem is:
my computer CPU keeps running with full speed but without any output for 2 hours.

Then, I run the code in the command line and get the following error. It seems it is the problem of the Initialization of Cutout with "months = slice(1,1)". But it is exactly the same as the given example (see examples/create_cutout.py). Can anyone help with this? Thanks a lot!**

> C:\Users\Yiping\Miniconda3\lib\site-packages\pyproj\crs.py:77: FutureWarning: '+init=<authority>:<code>' syntax is deprecated. '<authority>:<code>' is the preferred initialization method.
>   return _prepare_from_string(" ".join(pjargs))
> C:\Users\Yiping\Miniconda3\lib\site-packages\pyproj\crs.py:77: FutureWarning: '+init=<authority>:<code>' syntax is deprecated. '<authority>:<code>' is the preferred initialization method.
>   return _prepare_from_string(" ".join(pjargs))
> C:\Users\Yiping\Miniconda3\lib\site-packages\pyproj\crs.py:77: FutureWarning: '+init=<authority>:<code>' syntax is deprecated. '<authority>:<code>' is the preferred initialization method.
>   return _prepare_from_string(" ".join(pjargs))
> C:\Users\Yiping\Miniconda3\lib\site-packages\pyproj\crs.py:77: FutureWarning: '+init=<authority>:<code>' syntax is deprecated. '<authority>:<code>' is the preferred initialization method.
>   return _prepare_from_string(" ".join(pjargs))
> C:\Users\Yiping\Miniconda3\lib\site-packages\pyproj\crs.py:77: FutureWarning: '+init=<authority>:<code>' syntax is deprecated. '<authority>:<code>' is the preferred initialization method.
>   return _prepare_from_string(" ".join(pjargs))
> C:\Users\Yiping\Miniconda3\lib\site-packages\pyproj\crs.py:77: FutureWarning: '+init=<authority>:<code>' syntax is deprecated. '<authority>:<code>' is the preferred initialization method.
>   return _prepare_from_string(" ".join(pjargs))
> C:\Users\Yiping\Miniconda3\lib\site-packages\pyproj\crs.py:77: FutureWarning: '+init=<authority>:<code>' syntax is deprecated. '<authority>:<code>' is the preferred initialization method.
>   return _prepare_from_string(" ".join(pjargs))
> C:\Users\Yiping\Miniconda3\lib\site-packages\pyproj\crs.py:77: FutureWarning: '+init=<authority>:<code>' syntax is deprecated. '<authority>:<code>' is the preferred initialization method.
>   return _prepare_from_string(" ".join(pjargs))
> C:\Users\Yiping\Miniconda3\lib\site-packages\pyproj\crs.py:77: FutureWarning: '+init=<authority>:<code>' syntax is deprecated. '<authority>:<code>' is the preferred initialization method.
>   return _prepare_from_string(" ".join(pjargs))
> C:\Users\Yiping\Miniconda3\lib\site-packages\pyproj\crs.py:77: FutureWarning: '+init=<authority>:<code>' syntax is deprecated. '<authority>:<code>' is the preferred initialization method.
>   return _prepare_from_string(" ".join(pjargs))
> C:\Users\Yiping\Miniconda3\lib\site-packages\pyproj\crs.py:77: FutureWarning: '+init=<authority>:<code>' syntax is deprecated. '<authority>:<code>' is the preferred initialization method.
>   return _prepare_from_string(" ".join(pjargs))
> C:\Users\Yiping\Miniconda3\lib\site-packages\pyproj\crs.py:77: FutureWarning: '+init=<authority>:<code>' syntax is deprecated. '<authority>:<code>' is the preferred initialization method.
>   return _prepare_from_string(" ".join(pjargs))
> Traceback (most recent call last):
>   File "<string>", line 1, in <module>
>   File "C:\Users\Yiping\Miniconda3\lib\multiprocessing\spawn.py", line 105, in spawn_main
>     exitcode = _main(fd)
>   File "C:\Users\Yiping\Miniconda3\lib\multiprocessing\spawn.py", line 114, in _main
>     prepare(preparation_data)
>   File "C:\Users\Yiping\Miniconda3\lib\multiprocessing\spawn.py", line 225, in prepare
> Traceback (most recent call last):
>   File "<string>", line 1, in <module>
>     _fixup_main_from_path(data['init_main_from_path'])
>   File "C:\Users\Yiping\Miniconda3\lib\multiprocessing\spawn.py", line 105, in spawn_main
>   File "C:\Users\Yiping\Miniconda3\lib\multiprocessing\spawn.py", line 277, in _fixup_main_from_path
>         exitcode = _main(fd)run_name="__mp_main__")
> 
>   File "C:\Users\Yiping\Miniconda3\lib\multiprocessing\spawn.py", line 114, in _main
>   File "C:\Users\Yiping\Miniconda3\lib\runpy.py", line 263, in run_path
>         prepare(preparation_data)
>   File "C:\Users\Yiping\Miniconda3\lib\multiprocessing\spawn.py", line 225, in prepare
> pkg_name=pkg_name, script_name=fname)
>   File "C:\Users\Yiping\Miniconda3\lib\runpy.py", line 96, in _run_module_code
>         _fixup_main_from_path(data['init_main_from_path'])mod_name, mod_spec, pkg_name, script_name)
>   File "C:\Users\Yiping\Miniconda3\lib\multiprocessing\spawn.py", line 277, in _fixup_main_from_path
> 
>   File "C:\Users\Yiping\Miniconda3\lib\runpy.py", line 85, in _run_code
>         exec(code, run_globals)run_name="__mp_main__")
>   File "C:\Users\Yiping\Documents\Python Workspace\REUTE\climate\atlite_example.py", line 29, in <module>
> 
> months=slice(1,1))  File "C:\Users\Yiping\Miniconda3\lib\runpy.py", line 263, in run_path
> 
>   File "C:\Users\Yiping\Miniconda3\lib\site-packages\atlite\cutout.py", line 60, in __init__
>     pkg_name=pkg_name, script_name=fname)    assert False
> AssertionError
> 
>   File "C:\Users\Yiping\Miniconda3\lib\runpy.py", line 96, in _run_module_code
>     Traceback (most recent call last):
>   File "<string>", line 1, in <module>
> mod_name, mod_spec, pkg_name, script_name)  File "C:\Users\Yiping\Miniconda3\lib\multiprocessing\spawn.py", line 105, in spawn_main
> 
>   File "C:\Users\Yiping\Miniconda3\lib\runpy.py", line 85, in _run_code
>     exitcode = _main(fd)
>       File "C:\Users\Yiping\Miniconda3\lib\multiprocessing\spawn.py", line 114, in _main
> exec(code, run_globals)
> Traceback (most recent call last):
>   File "C:\Users\Yiping\Documents\Python Workspace\REUTE\climate\atlite_example.py", line 29, in <module>
>       File "<string>", line 1, in <module>
> prepare(preparation_data)      File "C:\Users\Yiping\Miniconda3\lib\multiprocessing\spawn.py", line 105, in spawn_main
> 
>   File "C:\Users\Yiping\Miniconda3\lib\multiprocessing\spawn.py", line 225, in prepare
> months=slice(1,1))
>   File "C:\Users\Yiping\Miniconda3\lib\site-packages\atlite\cutout.py", line 60, in __init__
> exitcode = _main(fd)    Traceback (most recent call last):
> _fixup_main_from_path(data['init_main_from_path'])
> 
> Traceback (most recent call last):
>       File "<string>", line 1, in <module>
> Traceback (most recent call last):
>   File "C:\Users\Yiping\Miniconda3\lib\multiprocessing\spawn.py", line 114, in _main
>   File "C:\Users\Yiping\Miniconda3\lib\multiprocessing\spawn.py", line 277, in _fixup_main_from_path
>   File "<string>", line 1, in <module>
> assert False  File "C:\Users\Yiping\Miniconda3\lib\multiprocessing\spawn.py", line 105, in spawn_main
>   File "<string>", line 1, in <module>
>         Traceback (most recent call last):
>   File "C:\Users\Yiping\Miniconda3\lib\multiprocessing\spawn.py", line 105, in spawn_main
> 
>     prepare(preparation_data)  File "C:\Users\Yiping\Miniconda3\lib\multiprocessing\spawn.py", line 105, in spawn_main
> 
>   File "<string>", line 1, in <module>
> AssertionErrorexitcode = _main(fd)run_name="__mp_main__")      File "C:\Users\Yiping\Miniconda3\lib\multiprocessing\spawn.py", line 225, in prepare
> 
> exitcode = _main(fd)  File "C:\Users\Yiping\Miniconda3\lib\multiprocessing\spawn.py", line 105, in spawn_main
> 
> exitcode = _main(fd)Traceback (most recent call last):
> 
> 
>       File "C:\Users\Yiping\Miniconda3\lib\runpy.py", line 263, in run_path
> 
>       File "<string>", line 1, in <module>
> exitcode = _main(fd)  File "C:\Users\Yiping\Miniconda3\lib\multiprocessing\spawn.py", line 114, in _main
> _fixup_main_from_path(data['init_main_from_path'])  File "C:\Users\Yiping\Miniconda3\lib\multiprocessing\spawn.py", line 114, in _main
>       File "C:\Users\Yiping\Miniconda3\lib\multiprocessing\spawn.py", line 114, in _main
> Traceback (most recent call last):
> 
>   File "C:\Users\Yiping\Miniconda3\lib\multiprocessing\spawn.py", line 105, in spawn_main
> 
>     pkg_name=pkg_name, script_name=fname)      File "<string>", line 1, in <module>
> prepare(preparation_data)      File "C:\Users\Yiping\Miniconda3\lib\multiprocessing\spawn.py", line 114, in _main
>   File "C:\Users\Yiping\Miniconda3\lib\multiprocessing\spawn.py", line 277, in _fixup_main_from_path
>     prepare(preparation_data)
> Traceback (most recent call last):
>   File "C:\Users\Yiping\Miniconda3\lib\multiprocessing\spawn.py", line 105, in spawn_main
> 
> prepare(preparation_data)    exitcode = _main(fd)
>   File "C:\Users\Yiping\Miniconda3\lib\runpy.py", line 96, in _run_module_code
>       File "<string>", line 1, in <module>
>   File "C:\Users\Yiping\Miniconda3\lib\multiprocessing\spawn.py", line 225, in prepare
> 
> exitcode = _main(fd)
>   File "C:\Users\Yiping\Miniconda3\lib\multiprocessing\spawn.py", line 225, in prepare
>     run_name="__mp_main__")  File "C:\Users\Yiping\Miniconda3\lib\multiprocessing\spawn.py", line 105, in spawn_main
> prepare(preparation_data)  File "C:\Users\Yiping\Miniconda3\lib\multiprocessing\spawn.py", line 225, in prepare
> 
> _fixup_main_from_path(data['init_main_from_path'])    mod_name, mod_spec, pkg_name, script_name)
> 
>       File "C:\Users\Yiping\Miniconda3\lib\multiprocessing\spawn.py", line 114, in _main
>   File "C:\Users\Yiping\Miniconda3\lib\multiprocessing\spawn.py", line 114, in _main
> 
> _fixup_main_from_path(data['init_main_from_path'])
>   File "C:\Users\Yiping\Miniconda3\lib\runpy.py", line 263, in run_path
>   File "C:\Users\Yiping\Miniconda3\lib\multiprocessing\spawn.py", line 225, in prepare
> exitcode = _main(fd)Traceback (most recent call last):
>     _fixup_main_from_path(data['init_main_from_path'])  File "C:\Users\Yiping\Miniconda3\lib\multiprocessing\spawn.py", line 277, in _fixup_main_from_path
> 
>   File "C:\Users\Yiping\Miniconda3\lib\runpy.py", line 85, in _run_code
> 
> pkg_name=pkg_name, script_name=fname)    prepare(preparation_data)
> 
>   File "C:\Users\Yiping\Miniconda3\lib\multiprocessing\spawn.py", line 277, in _fixup_main_from_path
>           File "<string>", line 1, in <module>
> 
>   File "C:\Users\Yiping\Miniconda3\lib\multiprocessing\spawn.py", line 114, in _main
> _fixup_main_from_path(data['init_main_from_path'])prepare(preparation_data)
> 
> run_name="__mp_main__")exec(code, run_globals)  File "C:\Users\Yiping\Miniconda3\lib\multiprocessing\spawn.py", line 105, in spawn_main
>   File "C:\Users\Yiping\Miniconda3\lib\runpy.py", line 96, in _run_module_code
>       File "C:\Users\Yiping\Miniconda3\lib\multiprocessing\spawn.py", line 277, in _fixup_main_from_path
>   File "C:\Users\Yiping\Miniconda3\lib\multiprocessing\spawn.py", line 225, in prepare
>   File "C:\Users\Yiping\Miniconda3\lib\multiprocessing\spawn.py", line 277, in _fixup_main_from_path
>       File "C:\Users\Yiping\Miniconda3\lib\multiprocessing\spawn.py", line 225, in prepare
> 
> 
>         run_name="__mp_main__")        prepare(preparation_data)      File "C:\Users\Yiping\Miniconda3\lib\runpy.py", line 263, in run_path
>       File "C:\Users\Yiping\Documents\Python Workspace\REUTE\climate\atlite_example.py", line 29, in <module>
> mod_name, mod_spec, pkg_name, script_name)exitcode = _main(fd)
> 
> _fixup_main_from_path(data['init_main_from_path'])
> run_name="__mp_main__")    _fixup_main_from_path(data['init_main_from_path'])
> run_name="__mp_main__")  File "C:\Users\Yiping\Miniconda3\lib\runpy.py", line 263, in run_path
> 
> 
>   File "C:\Users\Yiping\Miniconda3\lib\multiprocessing\spawn.py", line 225, in prepare
> 
> pkg_name=pkg_name, script_name=fname)
> months=slice(1,1))  File "C:\Users\Yiping\Miniconda3\lib\runpy.py", line 85, in _run_code
>   File "C:\Users\Yiping\Miniconda3\lib\multiprocessing\spawn.py", line 114, in _main
>       File "C:\Users\Yiping\Miniconda3\lib\runpy.py", line 263, in run_path
>   File "C:\Users\Yiping\Miniconda3\lib\multiprocessing\spawn.py", line 277, in _fixup_main_from_path
>   File "C:\Users\Yiping\Miniconda3\lib\runpy.py", line 263, in run_path
> 
>       File "C:\Users\Yiping\Miniconda3\lib\multiprocessing\spawn.py", line 277, in _fixup_main_from_path
> 
>     pkg_name=pkg_name, script_name=fname)
>           File "C:\Users\Yiping\Miniconda3\lib\runpy.py", line 96, in _run_module_code
> _fixup_main_from_path(data['init_main_from_path'])  File "C:\Users\Yiping\Miniconda3\lib\site-packages\atlite\cutout.py", line 60, in __init__
>     exec(code, run_globals)    prepare(preparation_data)  File "C:\Users\Yiping\Miniconda3\lib\runpy.py", line 96, in _run_module_code
> run_name="__mp_main__")pkg_name=pkg_name, script_name=fname)
> 
>     run_name="__mp_main__")
> pkg_name=pkg_name, script_name=fname)
> 
> 
>   File "C:\Users\Yiping\Miniconda3\lib\runpy.py", line 263, in run_path
>   File "C:\Users\Yiping\Miniconda3\lib\multiprocessing\spawn.py", line 277, in _fixup_main_from_path
> assert False
>   File "C:\Users\Yiping\Documents\Python Workspace\REUTE\climate\atlite_example.py", line 29, in <module>
>       File "C:\Users\Yiping\Miniconda3\lib\multiprocessing\spawn.py", line 225, in prepare
>   File "C:\Users\Yiping\Miniconda3\lib\runpy.py", line 96, in _run_module_code
> mod_name, mod_spec, pkg_name, script_name)  File "C:\Users\Yiping\Miniconda3\lib\runpy.py", line 96, in _run_module_code
> 
> pkg_name=pkg_name, script_name=fname)        mod_name, mod_spec, pkg_name, script_name)months=slice(1,1))
>       File "C:\Users\Yiping\Miniconda3\lib\runpy.py", line 263, in run_path
> AssertionError
> run_name="__mp_main__")
> 
> _fixup_main_from_path(data['init_main_from_path'])  File "C:\Users\Yiping\Miniconda3\lib\runpy.py", line 85, in _run_code
> mod_name, mod_spec, pkg_name, script_name)
> pkg_name=pkg_name, script_name=fname)
> mod_name, mod_spec, pkg_name, script_name)  File "C:\Users\Yiping\Miniconda3\lib\runpy.py", line 263, in run_path
>   File "C:\Users\Yiping\Miniconda3\lib\site-packages\atlite\cutout.py", line 60, in __init__
> 
> 
>   File "C:\Users\Yiping\Miniconda3\lib\runpy.py", line 96, in _run_module_code
> 
>   File "C:\Users\Yiping\Miniconda3\lib\runpy.py", line 85, in _run_code
> 
>           File "C:\Users\Yiping\Miniconda3\lib\multiprocessing\spawn.py", line 277, in _fixup_main_from_path
> exec(code, run_globals)  File "C:\Users\Yiping\Miniconda3\lib\runpy.py", line 85, in _run_code
>       File "C:\Users\Yiping\Miniconda3\lib\runpy.py", line 96, in _run_module_code
> mod_name, mod_spec, pkg_name, script_name)  File "C:\Users\Yiping\Miniconda3\lib\runpy.py", line 85, in _run_code
> pkg_name=pkg_name, script_name=fname)assert False
> 
> 
> exec(code, run_globals)  File "C:\Users\Yiping\Documents\Python Workspace\REUTE\climate\atlite_example.py", line 29, in <module>
> exec(code, run_globals)exec(code, run_globals)
> mod_name, mod_spec, pkg_name, script_name)  File "C:\Users\Yiping\Miniconda3\lib\runpy.py", line 85, in _run_code
> 
>   File "C:\Users\Yiping\Miniconda3\lib\runpy.py", line 96, in _run_module_code
> 
>     run_name="__mp_main__")  File "C:\Users\Yiping\Documents\Python Workspace\REUTE\climate\atlite_example.py", line 29, in <module>
> 
> 
> AssertionError          File "C:\Users\Yiping\Documents\Python Workspace\REUTE\climate\atlite_example.py", line 29, in <module>
> months=slice(1,1))
> 
>   File "C:\Users\Yiping\Documents\Python Workspace\REUTE\climate\atlite_example.py", line 29, in <module>
>   File "C:\Users\Yiping\Miniconda3\lib\runpy.py", line 85, in _run_code
> 
> exec(code, run_globals)mod_name, mod_spec, pkg_name, script_name)          File "C:\Users\Yiping\Miniconda3\lib\runpy.py", line 263, in run_path
>   File "C:\Users\Yiping\Miniconda3\lib\site-packages\atlite\cutout.py", line 60, in __init__
> 
> months=slice(1,1))months=slice(1,1))
> months=slice(1,1))exec(code, run_globals)assert False  File "C:\Users\Yiping\Miniconda3\lib\runpy.py", line 85, in _run_code
> 
> 
> pkg_name=pkg_name, script_name=fname)  File "C:\Users\Yiping\Documents\Python Workspace\REUTE\climate\atlite_example.py", line 29, in <module>
> 
> 
>       File "C:\Users\Yiping\Miniconda3\lib\site-packages\atlite\cutout.py", line 60, in __init__
>   File "C:\Users\Yiping\Miniconda3\lib\site-packages\atlite\cutout.py", line 60, in __init__
> 
> 
>   File "C:\Users\Yiping\Miniconda3\lib\site-packages\atlite\cutout.py", line 60, in __init__
> months=slice(1,1))AssertionErrorexec(code, run_globals)          File "C:\Users\Yiping\Miniconda3\lib\runpy.py", line 96, in _run_module_code
>   File "C:\Users\Yiping\Documents\Python Workspace\REUTE\climate\atlite_example.py", line 29, in <module>
> 
> 
> 
> assert Falseassert False          File "C:\Users\Yiping\Miniconda3\lib\site-packages\atlite\cutout.py", line 60, in __init__
> assert False  File "C:\Users\Yiping\Documents\Python Workspace\REUTE\climate\atlite_example.py", line 29, in <module>
> 
> 
> mod_name, mod_spec, pkg_name, script_name)months=slice(1,1))
>     AssertionErrorAssertionError
> 
> assert FalseAssertionErrormonths=slice(1,1))
> 
>   File "C:\Users\Yiping\Miniconda3\lib\runpy.py", line 85, in _run_code
>   File "C:\Users\Yiping\Miniconda3\lib\site-packages\atlite\cutout.py", line 60, in __init__
> 
> 
> 
> AssertionErrorassert False  File "C:\Users\Yiping\Miniconda3\lib\site-packages\atlite\cutout.py", line 60, in __init__
> exec(code, run_globals)
> 
> 
> assert False  File "C:\Users\Yiping\Documents\Python Workspace\REUTE\climate\atlite_example.py", line 29, in <module>
> AssertionError
> 
>     AssertionErrormonths=slice(1,1))
> 
>   File "C:\Users\Yiping\Miniconda3\lib\site-packages\atlite\cutout.py", line 60, in __init__
>     assert False
> AssertionError

[edit by @euronion : Formatting. Surronding your output with ``` ... ``` preserves the error message formatting and makes it easier to read.]

Add Test Script(s)

Since we're still having some trouble to properly intergrate v0.2 (#20), it would be helpful to add a test script which canvasses major relations.

Atm I'm trying to figure out how this would look like. Some things I have in mind:

  • Check coordinate alignment. As seen in #62 coordinates are a sensitive thing, if they are wrong, you get nice twisted pictures. This is probably a small check that coordinates of all retrieved data and converted data are in increasing order.
  • Check shape of the cutout against the target shape, given by bounds and resolution
  • Check against occurrence of nan values. If you have nan values in your retrieved data (not converted) then something is wrong.

This list should continue. I'm starting to write down some test files. Feel free to add to the list

Integrate config.py as function parameter?

The current setup method requires

  1. Downloading the source code
  2. Locally adjusting config.py
  3. Locally installing using pip

The problems I have with this:

a. It's rather cumbersome and prone to errors
b. Requires reinstallation whenever directories from config.py change
c. Is not correctly reflected in the README.rst

It seems to me it might be better to integrate the few config parameters as function parameters.

Or am I missing something?

hour_shift caused non-unique index

WARNING: Because the original data is provided every month, at the
month boundaries there is untidiness if you use a time shift. The
resulting xarray will have duplicates in the index for the parts
of the day in each month at the boundary. You will have to
re-average these based on the number of hours in each month for
the duplicated day.

Hi Jonas, do you have an example of how to deal with the duplicated days? If not, I can try and add a function for it.

Order or `grid_cells` and `grid_coordinates()`

Originally posted by @euronion in #20 (comment)

Order or grid_cells and grid_coordinates()

I am thinking more along the line of: Are people expecting the order in the lists returned by cutout.grid_cells and cutout.grid_coordinates() to be reliable or not.
Maybe it does cancel out in PyPSA-EUR or one codes it independent of this order (I just did so).
But maybe people are expecting the order to be consistent between version, i.e. then we are changing the API behaviour in an unexpected way -> we should at lteast add this to the release notes.

The bigger problem

The code with a more hideous and less obvious problem I am refering to is this one [1]:

    profile, capacities = func(
        # Redindex, stack index and conv. to CSR to be consistent with the format expected by atlite
        matrix=layoutmatrix.reindex_like(
            cutout.data).stack(spatial=('y', 'x')).data.tocsr(),
        index=buses,
        per_unit=True,
        return_capacity=True,
        show_progress='Compute profiles: ',
        **resource)

More specifically how the matrix passed to atlite has to be structured. This matrix is 2D, where one dimension is reserved for the index=buses and the other dimension is a .stack()-ed version of a 2D array.
Now the behaviour of .stack() depends on the order of the index, that's why the .reindex_like(...) is relevant here. In the version before it looked like this [2]:

    profile, capacities = func(matrix=layoutmatrix, index=buses, per_unit=True,
                               return_capacity=True, show_progress='Compute profiles: ',
                               **resource)

where the matrix=layoutmatrix as implicitly assuming the 2D array of values refering to the coordinates in the cutout to be stacked with descending y and ascending x. That was the old atlite structure.
If I use the same code with the new atlite=v0.2 the result is different, because we assume (inside atlite!) that this matrix is stacked with ascending y and x.

It boils down to

cutout.sortby('y', ascending=True).stack(spatial=('y','x'))

not being the same as

cutout.sortby('y', ascending=False).stack(spatial=('y','x'))

but we use

cutout.stack(spatial=('y','x')

here

da = da.stack(spatial=('y', 'x')).transpose('spatial', 'time')

Possible solution

The easiest solution could be to guarantee the old behaviour by fixing the index order of created cutouts during their creation

cutout.data = cutout.data.sortby('y', ascending=False)
cutout.data = cutout.data.sortby('x', ascending=True)

This would allow us to make the change in the .Cutout(...) signature while ensuring backwards compatability.

Links

[1] Source: New code in PyPSA-EUR for the upcoming atlite version
https://github.com/PyPSA/pypsa-eur/blob/283042d1dd3cd11b5313479c7ca03cbe58f778c9/scripts/build_renewable_profiles.py#L390-L398
[2] Source: https://github.com/PyPSA/pypsa-eur/blob/bb3477cd693d1c8e77e75e61a1a7e1a4647e6a3c/scripts/build_renewable_profiles.py#L303-L305

Parallelised calculations using dask.

Keeping this here as a reminder. Not a pressing issue, but something that has been bothering me for some time.

I have not yet observed that atlite is using more than one processing core for calculations (e.g. wind speed to wind power conversion).
I would assume that this should be easy to implement (as in the backend we are using dask and xarray, but I haven't been able to identify the culprit.

Checking for parallelised acitvity

Initially I was only monitoring the CPU util. using htop:
There only one core seems to be active.

A better way is to use a dask dashboard and a local client:

from dask.distributed import Client
import xarray as xr

client = Client()
client

I also do not see any dask activity during calculations there.

Some background info

When checking cutout.data this returns an xarray Dataset using numpy arrays.
When loading the dataset (xarray.open_dataset(...) ) with the chunks option or rechunking the dataset after the cutout was loaded using cutout.data.chunk(...)
then shows some dask activity during calculations (but slow, I guess a lot of overhead from spawning and orchestrating the different workers and threads).

Low priority

With the new version #20 calculations have become significantly faster and caching of datasets obsolete (can now be done implicitly by xarray and dask).
So this is only interesting when doing conversion of large datasets or repeated conversions.

Current versions

> from platform import python_version
> python_version()
3.7.3

> import xarray
> xarray.__version__
'0.12.3'

> import dask
> dask.__version__
'2.1.0'

Time series Capacity Factor

I am currently trying to produce a hourly capacity factor time series but am running into some issues. It seems that the only way that atlite processes hourly time series is for energy generation, and that for capacity factors it only produces the average. Is there a specification in the parameters that could allow for an hourly capacity factor generation instead? This is the code Im working with:

pv_CF = cutout.pv(pv_model, capacity_factor=True , orientation={'slope': cutout_lat, 'azimuth': 0})

AssertionError: gdalwarp was not able to resample gebco

When preparing a new cutout with cutout.prepare(), the above assertion error is thrown.

I use this example with the relevant line(s):

# Define the cutout; this will not yet trigger any major operations
cutout = atlite.Cutout(name="uk-2011-01-2",
                       module="era5",
                       bounds=UkIr.unary_union.bounds,
                       time="2011-01")

# This is where all the work happens (this can take some time, for us it took ~15 minutes).
cutout.prepare()

Full trace:

INFO:cdsapi:Request is completed

---------------------------------------------------------------------------
AssertionError                            Traceback (most recent call last)
<ipython-input-6-898cb53315e6> in <module>
      6 
      7 # This is where all the work happens (this can take some time, for us it took ~15 minutes).
----> 8 cutout.prepare()

c:\users\j. hampp\documents\github\atlite\atlite\data.py in cutout_prepare(cutout, features, freq, tmpdir, overwrite)
    201                 return
    202 
--> 203             ds = get_missing_data(cutout, missing_features, freq, tmpdir=tmpdir)
    204 
    205             # Merge with existing cutout

c:\users\j. hampp\documents\github\atlite\atlite\data.py in get_missing_data(cutout, features, freq, tmpdir)
    163         datasets.append(feature_data)
    164 
--> 165     datasets, = dask.compute(datasets)
    166 
    167     return xr.merge(datasets, compat='equals')

C:\anaconda\envs\atlite\lib\site-packages\dask\base.py in compute(*args, **kwargs)
    444     keys = [x.__dask_keys__() for x in collections]
    445     postcomputes = [x.__dask_postcompute__() for x in collections]
--> 446     results = schedule(dsk, keys, **kwargs)
    447     return repack([f(r, *a) for r, (f, a) in zip(results, postcomputes)])
    448 

C:\anaconda\envs\atlite\lib\site-packages\dask\threaded.py in get(dsk, result, cache, num_workers, pool, **kwargs)
     80         get_id=_thread_get_id,
     81         pack_exception=pack_exception,
---> 82         **kwargs
     83     )
     84 

C:\anaconda\envs\atlite\lib\site-packages\dask\local.py in get_async(apply_async, num_workers, dsk, result, cache, get_id, rerun_exceptions_locally, pack_exception, raise_exception, callbacks, dumps, loads, **kwargs)
    489                         _execute_task(task, data)  # Re-execute locally
    490                     else:
--> 491                         raise_exception(exc, tb)
    492                 res, worker_id = loads(res_info)
    493                 state["cache"][key] = res

C:\anaconda\envs\atlite\lib\site-packages\dask\compatibility.py in reraise(exc, tb)
    128         if exc.__traceback__ is not tb:
    129             raise exc.with_traceback(tb)
--> 130         raise exc
    131 
    132     import pickle as cPickle

C:\anaconda\envs\atlite\lib\site-packages\dask\local.py in execute_task(key, task_info, dumps, loads, get_id, pack_exception)
    231     try:
    232         task, data = loads(task_info)
--> 233         result = _execute_task(task, data)
    234         id = get_id()
    235         result = dumps((result, id))

C:\anaconda\envs\atlite\lib\site-packages\dask\core.py in _execute_task(arg, cache, dsk)
    117         func, args = arg[0], arg[1:]
    118         args2 = [_execute_task(a, cache) for a in args]
--> 119         return func(*args2)
    120     elif not ishashable(arg):
    121         return arg

c:\users\j. hampp\documents\github\atlite\atlite\datasets\common.py in get_data_gebco_height(xs, ys, gebco_path)
     91                                '-r', 'average',
     92                                gebco_path, target])
---> 93         assert ret == 0, "gdalwarp was not able to resample gebco"
     94     except OSError:
     95         logger.warning("gdalwarp was not found for resampling gebco. "

AssertionError: gdalwarp was not able to resample gebco

Add interface to RESkit power curve model

RESkit, Severin Ryberg's atlite :), has three very interesting features that one should evaluate, whether they can be adopted or reused:

  1. A synthetic power curve model that is defined in reskit/wind/core/power_curve.py, which generates a synthetic power curve only on the basis of a specific power rating, ie. kW/m^2 (capacity rating over rotor area) or capacity and rotor diam equivalently, maybe one can just reuse or sub-class that code with a small modification to make the simulate method work with xarray/dask.
  2. An economic capital cost scaling model around the turbine parameters hub height, rotor diameter and capacity (and foundation type for offshore), which is defined in reskit/wind/economic/{onshore_cost_model.py,offshore_cost_model.py}. This maybe is a function that would be called from PyPSA-Eur directly.

Together those two features would allow having a range of different turbines or the possibility to have a weak wind turbine as a second choice.

Introducing a new input dataset

I would like to use atlite with the EFAS data recently released on the Copernicus Data Store. Do you have a tutorial or some docs on how to "add" another input dataset?

the role of GridCells class

As tackled in PR #69 the shapely strtree is a better choice for filtering out shapes that are not overlapping. The change in v0.2 foresees to store the sindex of the grid cells in order to faster calculate the indicator matrix. This becomes obsolete when defining the new "sindex" via strtree, see here

len(cutout.grid_cells)
Out[7]: 20698

%time STRtree(cutout.grid_cells)
CPU times: user 70.7 ms, sys: 3.76 ms, total: 74.4 ms
Wall time: 72.3 ms
Out[8]: <shapely.strtree.STRtree at 0x7fdc3db0be50> 

(the grid cells here are the standard ones for pypsa-eur)

Compare this to the previous sindex calculation of ~1.5 seconds.

Sooo, this raises the question, if the GridCells class is really necessary. It seems the only purpose for its introduction was to store the sindex, which does not work properly with pickle.dump and which anyway becomes obsolete now.

@coroa I know you are buisy, but perhaps can just shortly give a hint whether there was another intention when you introduced the GridCells class.

era5: wrong coordinate system for 'odd' bounds in cutout

When passing odd bounds as cutout parameters, the retrieved data is misaligned to the built coordinate system.

For example

cutout = atlite.Cutout(path="uk-2011-01",
                       module="era5",
                       bounds=(-14.1, 50.34, 1.512, 61.123),
                       time="2011-01-01")
cutout.prepare()

will try to retrieve data for a grid beginning with -14.1 followed by steps of 0.25. However the coordinate system set up (automaticaly before cutout.prepare()) begins with -14.0 followed by steps of 0.25.

atilite merges the original grid with the retrieved data without an error, which ends up in a lot of nan's

Incompatibility with pyproj version 1.9.6 used by pypsa-eur

I have already filed the issue with pyproj-eur and was asked to file it here as well.

The error happens when importing atlite with pyproj version 1.9.6 installed.

Steps to reproduce the error:

  1. Create an environment with pyproj=1.9.6 and atlite
  2. Open a python console
  3. import atlite.

Possible solution

Add ellps=XXX (e.g. ellps='WGS84', if WGS84 is correct) to the call projection = RotProj(dict(proj='ob_tran', o_proj='latlong', lon_0=180, o_lon_p=-162, o_lat_p=39.25)) on line 41/42 in cordex.py.

Error Message for atlite

>>> import atlite
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "C:\Users\Svosw\miniconda3\envs\pyproj_test\lib\site-packages\atlite\__init__.py", line 26, in <module>
    from .cutout import Cutout
  File "C:\Users\Svosw\miniconda3\envs\pyproj_test\lib\site-packages\atlite\cutout.py", line 33, in <module>
    from . import config, datasets
  File "C:\Users\Svosw\miniconda3\envs\pyproj_test\lib\site-packages\atlite\datasets\__init__.py", line 3, in <module>
    from . import cordex, ncep, era5, sarah
  File "C:\Users\Svosw\miniconda3\envs\pyproj_test\lib\site-packages\atlite\datasets\cordex.py", line 42, in <module>
    o_lon_p=-162, o_lat_p=39.25))
  File "C:\Users\Svosw\miniconda3\envs\pyproj_test\lib\site-packages\pyproj\__init__.py", line 362, in __new__
    return _proj.Proj.__new__(self, projstring)
  File "_proj.pyx", line 129, in _proj.Proj.__cinit__
RuntimeError: b'major axis or radius = 0 or not given'

Implement configuration as object instead of module.

We do have (with the new version #20 ) a new configuration system in the pipeline.

It does however have a few downsides which became apparent during further developments, see e.g. the discussion here, where an observer can not trigger on changes of the module variables.

I don't think __setattr__ is called on module level (but feel free to try, i am unsure).
We could package everything up into an object! :)

Thought for the future:
I worked on it in https://github.com/euronion/atlite/blob/f1e039abf9c5295ac549d0c39152beff7697c418/atlite/config.py
and had a few things I could not figure out and dropped it.
(like not calling the config as atlite.config.config and making sure the object is shared > among all modules when a new config-object is used to overwrite the existing one - in short:
The current solution is more robust).

Originally posted by @euronion in #43 (comment)

It would be nice to implement the config as a true object.

Wrong coordinates and broken values in SARAH cutouts

The coordinates in SARAH cutouts are messed up, i.e. mirrored, see the image below.

image

> albedo
<xarray.DataArray 'albedo' (y: 160, x: 270)>
array([[0.04827 , 0.048171, 0.048043, ..., 0.103032, 0.104033, 0.104248],
       [0.047978, 0.047966, 0.047898, ..., 0.104392, 0.109037, 0.11042 ],
       [0.047583, 0.04764 , 0.047696, ..., 0.105399, 0.109951, 0.111676],
       ...,
       [0.03567 , 0.03552 , 0.035488, ..., 0.353771, 0.35935 , 0.36295 ],
       [0.035606, 0.035525, 0.035507, ..., 0.356749, 0.357823, 0.359936],
       [0.035569, 0.0355  , 0.035465, ..., 0.360018, 0.361162, 0.359274]],
      dtype=float32)
Coordinates:
    time     datetime64[ns] 2013-06-01T12:00:00
  * x        (x) float64 -12.0 -11.8 -11.6 -11.4 -11.2 ... 41.2 41.4 41.6 41.8
  * y        (y) float64 33.0 33.2 33.4 33.6 33.8 ... 64.0 64.2 64.4 64.6 64.8
    lon      (x) float64 -12.0 -11.8 -11.6 -11.4 -11.2 ... 41.2 41.4 41.6 41.8
    lat      (y) float64 33.0 33.2 33.4 33.6 33.8 ... 64.0 64.2 64.4 64.6 64.8
Attributes:
    units:      (0 - 1)
    long_name:  Albedo

The cutout

> cutout
<Cutout europe-2013-sarah x=-12.00-41.80 y=33.00-64.80 time=2013-01-01-2013-12-31 prepared_features=['temperature', 'influx'] is_view=False>

with the messed up coordinates

> cutout.data
<xarray.Dataset>
Dimensions:         (time: 8760, x: 270, y: 160)
Coordinates:
  * time            (time) datetime64[ns] 2013-01-01 ... 2013-12-31T23:00:00
  * x               (x) float64 -12.0 -11.8 -11.6 -11.4 ... 41.2 41.4 41.6 41.8
  * y               (y) float64 33.0 33.2 33.4 33.6 33.8 ... 64.2 64.4 64.6 64.8
    lon             (x) float64 -12.0 -11.8 -11.6 -11.4 ... 41.2 41.4 41.6 41.8
    lat             (y) float64 33.0 33.2 33.4 33.6 33.8 ... 64.2 64.4 64.6 64.8
Data variables:
    temperature     (time, y, x) float32 ...
    influx_toa      (time, y, x) float32 ...
    albedo          (time, y, x) float32 ...
    influx_direct   (time, y, x) float32 ...
    influx_diffuse  (time, y, x) float32 ...
Attributes:
    module:               sarah
    prepared_features:    ['temperature', 'influx']
    creation_parameters:  {'resolution': 0.2, 'x': slice(-12.0, 42.0, None), ...

Maybe an Australian PoV?

[v0.2] Issue on migrating prepared cutout to new cutout format

I had an already prepared cutout in the old atlite version and opened it with the new atlite version:

cutout = atlite.Cutout(config['cutout'], cutout_dir=snakemake.input.cutout_dir)

The script took some time and then failed with the following error message:

WARNING:atlite.utils:Found an old-style directory-like cutout. Use `prepare` to transfer that data.
/home/vagrant/miniconda/envs/atlite-v0.2/lib/python3.7/site-packages/xarray/backends/api.py:783: FutureWarning: In xarray version 0.13 `auto_combine` will be deprecated.
  coords=coords)
/home/vagrant/miniconda/envs/atlite-v0.2/lib/python3.7/site-packages/xarray/backends/api.py:783: FutureWarning: The datasets supplied have global dimension coordinates. You may want
to use the new `combine_by_coords` function (or the
`combine='by_coords'` option to `open_mfdataset` to order the datasets
before concatenation. Alternatively, to continue concatenating based
on the order the datasets are supplied in in future, please use the
new `combine_nested` function (or the `combine='nested'` option to
open_mfdataset).The datasets supplied require both concatenation and merging. From
xarray version 0.14 this will operation will require either using the
new `combine_nested` function (or the `combine='nested'` option to
open_mfdataset), with a nested list structure such that you can combine
along the dimensions None. Alternatively if your datasets have global
dimension coordinates then you can use the new `combine_by_coords`
function.
  coords=coords)

---------------------------------------------------------------------------
MergeError                                Traceback (most recent call last)
<ipython-input-7-042299d21e33> in <module>
      1 # Load atlite cutout
----> 2 cutout = atlite.Cutout(config['cutout'], cutout_dir=snakemake.input.cutout_dir)

~/miniconda/envs/atlite-v0.2/lib/python3.7/site-packages/atlite/cutout.py in __init__(self, name, data, cutout_dir, **cutoutparams)
     92             elif os.path.isdir(os.path.join(self.cutout_dir, self.name)):
     93                 data = utils.migrate_from_cutout_directory(os.path.join(self.cutout_dir, self.name),
---> 94                                                            self.name, self.cutout_fn, cutoutparams)
     95                 self.is_view = True
     96             else:

~/miniconda/envs/atlite-v0.2/lib/python3.7/site-packages/atlite/utils.py in migrate_from_cutout_directory(old_cutout_dir, name, cutout_fn, cutoutparams)
     59     old_cutout_dir = Path(old_cutout_dir)
     60     with xr.open_dataset(old_cutout_dir / "meta.nc") as meta:
---> 61         data = xr.open_mfdataset(str(old_cutout_dir / "[12]*.nc"))
     62         data.attrs.update(meta.attrs)
     63     data.attrs['prepared_features'] = list(sys.modules['atlite.datasets.' + data.attrs["module"]].features)

~/miniconda/envs/atlite-v0.2/lib/python3.7/site-packages/xarray/backends/api.py in open_mfdataset(paths, chunks, concat_dim, compat, preprocess, engine, lock, data_vars, coords, combine, autoclose, parallel, **kwargs)
    781             combined = auto_combine(datasets, concat_dim=concat_dim,
    782                                     compat=compat, data_vars=data_vars,
--> 783                                     coords=coords)
    784         elif combine == 'nested':
    785             # Combined nested list by successive concat and merge operations

~/miniconda/envs/atlite-v0.2/lib/python3.7/site-packages/xarray/core/combine.py in auto_combine(datasets, concat_dim, compat, data_vars, coords, fill_value)
    627     return _old_auto_combine(datasets, concat_dim=concat_dim,
    628                              compat=compat, data_vars=data_vars,
--> 629                              coords=coords, fill_value=fill_value)
    630 
    631 

~/miniconda/envs/atlite-v0.2/lib/python3.7/site-packages/xarray/core/combine.py in _old_auto_combine(datasets, concat_dim, compat, data_vars, coords, fill_value)
    681     else:
    682         concatenated = datasets
--> 683     merged = merge(concatenated, compat=compat, fill_value=fill_value)
    684     return merged
    685 

~/miniconda/envs/atlite-v0.2/lib/python3.7/site-packages/xarray/core/merge.py in merge(objects, compat, join, fill_value)
    549 
    550     variables, coord_names, dims = merge_core(dict_like_objects, compat, join,
--> 551                                               fill_value=fill_value)
    552     # TODO: don't always recompute indexes
    553     merged = Dataset._construct_direct(

~/miniconda/envs/atlite-v0.2/lib/python3.7/site-packages/xarray/core/merge.py in merge_core(objs, compat, join, priority_arg, explicit_coords, indexes, fill_value)
    460 
    461     priority_vars = _get_priority_vars(aligned, priority_arg, compat=compat)
--> 462     variables = merge_variables(expanded, priority_vars, compat=compat)
    463     assert_unique_multiindex_level_names(variables)
    464 

~/miniconda/envs/atlite-v0.2/lib/python3.7/site-packages/xarray/core/merge.py in merge_variables(list_of_variables_dicts, priority_vars, compat)
    170             else:
    171                 try:
--> 172                     merged[name] = unique_variable(name, var_list, compat)
    173                 except MergeError:
    174                     if compat != 'minimal':

~/miniconda/envs/atlite-v0.2/lib/python3.7/site-packages/xarray/core/merge.py in unique_variable(name, variables, compat)
     90                                  'objects to be combined:\n'
     91                                  'first value: %r\nsecond value: %r'
---> 92                                  % (name, out, var))
     93             if combine_method:
     94                 # TODO: add preservation of attrs into fillna

MergeError: conflicting values for variable 'roughness' on objects to be combined:
first value: <xarray.Variable (y: 112, x: 132)>
dask.array<shape=(112, 132), dtype=float32, chunksize=(112, 132)>
Attributes:
    units:      m
    long_name:  Forecast surface roughness
second value: <xarray.Variable (time: 8784, y: 112, x: 132)>
dask.array<shape=(8784, 112, 132), dtype=float32, chunksize=(1440, 112, 132)>
Attributes:
    units:      m
    long_name:  Forecast surface roughness

Problems with installed dependencies from various sources

@hailiangliu89 and me are currently running into quite a few problems with installed dependencies.
Until now I think we were able to solve all the problems by using a different source for installation (or switching OS).

I don't think the causes can be solved from within atlite.
Is there a good way to cope with these problems?
The two obvious possibilities for me:

  1. conda environment file
  2. a FAQ section in the README.rst

Approach 1. would be more convinient, but I have the feeling that most problems are caused by the different, individual configurations.
This leaves approach 2., which would also be more versatile (and wouldn't require regular updates).

Any ideas?

Example:
For me installing pyshp from conda-forge causes problems, installing via pip works fine.
The same problem with geopandas.

Cheers.

cutout creation with python 3.7 are sometimes stuck

The cutout creation using environments with python 3.7 sometimes stops during the writing of the netcdf file. The progress then stays at a low percentage 0 to 20 % and noting more happens. When killing the process, one sees that dask was waiting for further calculation. I am absolutely not sure why this is the case.

Trying again helps and using a python 3.8 environment remedies.

[v0.1] Preparing already prepared cutout causes errors

Using a cutout which is already prepared, i.e. with

> cutout.available_features == cutout.prepared_features
True

and then calling

cutout.prepare()

results in a lot of activity and finally some errors, e.g.

---------------------------------------------------------------------------
FileNotFoundError                         Traceback (most recent call last)
<ipython-input-4-eee9441f41e9> in <module>
----> 1 cutout.prepare()

~/miniconda/envs/atlite-v0.2/lib/python3.7/site-packages/atlite/data.py in cutout_prepare(cutout, features, monthly, overwrite)
    204     if os.path.exists(cutout.cutout_fn):
    205         os.unlink(cutout.cutout_fn)
--> 206     os.rename(target, cutout.cutout_fn)
    207 
    208     # Re-open

FileNotFoundError: [Errno 2] No such file or directory: '/reanalysis/atlite/cutouts/tmp4zn24h_sv0.2_western-asia-2016-era5.nc' -> '../cutouts/v0.2_western-asia-2016-era5.nc'

I think the problem is the data.cutout_prepare(...) function, which causes a lot of overhead processing, temp-file creation and other things, even if the missing_features set at this point is empty
https://github.com/FRESNA/atlite/blob/9f6d0316f20ecd6fffe731d01cd731466845a801/atlite/data.py#L185-L186

I would expect this to run through cleanly in the blink of an eye.
Possibly reintroduce the "Cutout is already fully prepared" fast track from the earlier version?

use sparse arrays from xarray

As discussed with @euronion xarray supports sparse arrays since v0.14 see the release notes.
It works pretty decent though the functionality is not well documented.

For its use the sparse package has to be installed

The two functions I use the netallocation package are the following

import xarray as xr
from sparse import as_coo, COO


def as_sparse(ds):
    """
    Convert dense dataset/dataarray into a sparse dataset.

    Parameters
    ----------
    ds : xr.Dataset or xr.DataArray

    Returns
    -------
    xr.Dataset or xr.DataArray
        Dataset or DataArray with sparse data.

    """
    func = lambda data: COO(data) if not isinstance(data, COO) else data
    return xr.apply_ufunc(func, ds)


def as_dense(ds):
    """
    Convert sparse dataset/dataarray into a dense dataset.

    Parameters
    ----------
    ds : xr.Dataset or xr.DataArray

    Returns
    -------
    xr.Dataset or xr.DataArray
        Dataset or DataArray with dense data.

    """
    func = lambda data: data.todense() if isinstance(data, COO) else data
    return xr.apply_ufunc(func, ds)

SARAH-2 cutout creation issue

Hello,

I seem to be having trouble creating a cutout for SARAH-2 data. I was able to get ERA5 to work fine, but for some reason SARAH-2 seems to be giving me problems.


Script:

`import logging
import atlite

logging.basicConfig(level=logging.DEBUG)

'''SARAH-2 DOESNT SEEM TO BE WORKING. ERA5 is okay'''

cutout = atlite.Cutout(name="SARAH_SthAfr_2011_01",
cutout_dir="/Users/lennon/Documents/GitHub/Sites/obeles.github.io/atlite/atlite/datasets/ERA5_datasets/cutouts",
module="sarah",
xs=slice(27.25, 32.25),
ys=slice(-30.75, -35.75),
years=slice(2011, 2011),
months=slice(1,1)
)

cutout.prepare()`


Script Error:

`INFO:atlite.cutout:Cutout SARAH_SthAfr_2011_01 not found in directory /Users/lennon/Documents/GitHub/Sites/obeles.github.io/atlite/atlite/datasets/ERA5_datasets/cutouts, building new one
Traceback (most recent call last):

File "/Users/lennon/Dropbox/Scripts/solar_atlite_cutout.py", line 24, in
months=slice(1,1)

File "/Users/lennon/anaconda3/envs/geo_env/lib/python3.7/site-packages/atlite/cutout.py", line 88, in init
self.meta = self.get_meta(**cutoutparams)

File "/Users/lennon/anaconda3/envs/geo_env/lib/python3.7/site-packages/atlite/preparation.py", line 171, in cutout_get_meta
ds = prepare_func(xs=xs, ys=ys, year=years.stop, month=months.stop, **meta_kwds)

File "/Users/lennon/anaconda3/envs/geo_env/lib/python3.7/site-packages/atlite/datasets/sarah.py", line 70, in prepare_meta_sarah
for t in (template_sis, template_sid)]

File "/Users/lennon/anaconda3/envs/geo_env/lib/python3.7/site-packages/atlite/datasets/sarah.py", line 70, in
for t in (template_sis, template_sid)]

StopIteration`

"TypeError: unhashable type: 'slice'" during cutout creation.

I encountered a slicing issue during cutout creation:

atlite\atlite\data.py in cutout_prepare(cutout, features, tmpdir, overwrite)
    124
    125         # target is series of all available variables for given module and features
--> 126         target = available_features(modules).loc[:, features].drop_duplicates()
    127
    128         for module in target.index.unique('module'):

~\.julia\conda\3\envs\atlite\lib\site-packages\pandas\core\indexing.py in __getitem__(self, key)
    871                     # AttributeError for IntervalTree get_value
    872                     pass
--> 873             return self._getitem_tuple(key)
    874         else:
    875             # we by definition only have the 0th axis

~\.julia\conda\3\envs\atlite\lib\site-packages\pandas\core\indexing.py in _getitem_tuple(self, tup)
   1042     def _getitem_tuple(self, tup: Tuple):
   1043         try:
-> 1044             return self._getitem_lowerdim(tup)
   1045         except IndexingError:
   1046             pass

~\.julia\conda\3\envs\atlite\lib\site-packages\pandas\core\indexing.py in _getitem_lowerdim(self, tup)
    764         # we may have a nested tuples indexer here
    765         if self._is_nested_tuple_indexer(tup):
--> 766             return self._getitem_nested_tuple(tup)
    767
    768         # we maybe be using a tuple to represent multiple dimensions here

~\.julia\conda\3\envs\atlite\lib\site-packages\pandas\core\indexing.py in _getitem_nested_tuple(self, tup)
    832             # selectors
    833             axis = self.axis or 0
--> 834             return self._getitem_axis(tup, axis=axis)
    835
    836         # handle the multi-axis by taking sections and reducing

~\.julia\conda\3\envs\atlite\lib\site-packages\pandas\core\indexing.py in _getitem_axis(self, key, axis)
   1101             # nested tuple slicing
   1102             if is_nested_tuple(key, labels):
-> 1103                 locs = labels.get_locs(key)
   1104                 indexer = [slice(None)] * self.ndim
   1105                 indexer[axis] = locs

~\.julia\conda\3\envs\atlite\lib\site-packages\pandas\core\indexes\multi.py in get_locs(self, seq)
   3110
   3111         assert isinstance(indexer, Int64Index), type(indexer)
-> 3112         indexer = self._reorder_indexer(seq, indexer)
   3113
   3114         return indexer._values

~\.julia\conda\3\envs\atlite\lib\site-packages\pandas\core\indexes\multi.py in _reorder_indexer(self, seq, indexer)
   3139                 if is_list_like(k):
   3140                     if not need_sort:
-> 3141                         k_codes = self.levels[i].get_indexer(k)
   3142                         k_codes = k_codes[k_codes >= 0]  # Filter absent keys
   3143                         # True if the given codes are not ordered

~\.julia\conda\3\envs\atlite\lib\site-packages\pandas\core\indexes\base.py in get_indexer(self, target, method, limit, tolerance)
   3004                 )
   3005
-> 3006             indexer = self._engine.get_indexer(target._get_engine_target())
   3007
   3008         return ensure_platform_int(indexer)

pandas\_libs\index.pyx in pandas._libs.index.IndexEngine.get_indexer()

pandas\_libs\hashtable_class_helper.pxi in pandas._libs.hashtable.PyObjectHashTable.lookup()

TypeError: unhashable type: 'slice'

Caused by this line

atlite/atlite/data.py

Lines 125 to 126 in ba5daae

# target is series of all available variables for given module and features
target = available_features(modules).loc[:, features].drop_duplicates()

due to the multi-index access in .loc with an array of an empty slice.

Removing

modules = atleast_1d(cutout.module)

(for which I don't see a need anyways) resolved this issue.

Can you reproduce @FabianHofmann ?
Was there a change in pandas?

Referencing #107 for pandas version.

cutout preparation on windows machines fails

The cutout preparation fails on windows machines as netcdf files are handled differently and want to be closed manually. Unfortunately the file handling on the current branch v0.2 does not allow this in an easy way. I disabled the travis on windows machines. This should however be solved for v0.2

era5 time coordinates for static variables

There is a small bug when loading the static variable only, i.e. 'height' with era5. This will have the time dimension of the first time parameters of the first request. I'll suggest a solution

PermissionError: [WinError 32] by creating a cutout with era5 [atlite 0.0.4.dev335+gba5daae]

Hi,
i am running the following code:

CODE

import atlite
import logging
logging.basicConfig(level=logging.INFO)

cutout = atlite.Cutout(path="HH-2019.nc",
                       module="era5",
                       x=slice(8.3, 11.37),
                       y=slice(52.5, 54.32),
                       time=slice("2019-01","2019-12")
                       )

cutout.prepare()

ERROR

the download process is executed as described in the documentation, but then appears the following Error:

Traceback (most recent call last):
  File "D:\Bachelorarbeit\conda\envs\atlite\lib\site-packages\atlite\data.py", line 146, in cutout_prepare
    fd, tmp = mkstemp(suffix=filename, dir=directory)
  File "D:\Bachelorarbeit\conda\envs\atlite\lib\tempfile.py", line 331, in mkstemp
    return _mkstemp_inner(dir, prefix, suffix, flags, output_type)
  File "D:\Bachelorarbeit\conda\envs\atlite\lib\tempfile.py", line 250, in _mkstemp_inner
    fd = _os.open(file, flags, 0o600)
FileNotFoundError: [Errno 2] No such file or directory: 'HH-2019-01-01\\tmp1jwrobky02.nc'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "D:/Bachelorarbeit/Bachelorarbeit/tryouts/try_atlite.py", line 12, in <module>
    cutout.prepare()
  File "D:\Bachelorarbeit\conda\envs\atlite\lib\site-packages\atlite\data.py", line 170, in cutout_prepare
    rmtree(tmpdir)
  File "D:\Bachelorarbeit\conda\envs\atlite\lib\shutil.py", line 737, in rmtree
    return _rmtree_unsafe(path, onerror)
  File "D:\Bachelorarbeit\conda\envs\atlite\lib\shutil.py", line 615, in _rmtree_unsafe
    onerror(os.unlink, fullname, sys.exc_info())
  File "D:\Bachelorarbeit\conda\envs\atlite\lib\shutil.py", line 613, in _rmtree_unsafe
    os.unlink(fullname)
PermissionError: [WinError 32]The process cannot access the file because it is being used by another process: 'C:\\Users\\cedko\\AppData\\Local\\Temp\\tmp5dg_g04i\\tmp7g_scis6.nc'

Is there any known Bug generating this Error or is my PC causing this Error?

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.