Giter Site home page Giter Site logo

hyriver / hyriver.github.io Goto Github PK

View Code? Open in Web Editor NEW
90.0 6.0 10.0 130.04 MB

A Python software stack for retrieving hydroclimate data from web services.

Home Page: https://docs.hyriver.io

License: Other

Makefile 86.67% Python 13.33%
python webservice hydrology climate data

hyriver.github.io's Introduction

image

Binder

Build Website

JOSS

Package Description CI
Download Stat Navigate and subset NHDPlus (MR and HR) using web services Github Actions
Download Stat Access topographic data through National Map's 3DEP web service Github Actions
Download Stat Access NWIS, NID, WQP, eHydro, NLCD, CAMELS, and SSEBop databases Github Actions
Download Stat Access daily, monthly, and annual climate data via Daymet Github Actions
Download Stat Access daily climate data via GridMet Github Actions
Download Stat Access hourly NLDAS-2 data via web services Github Actions
Download Stat A collection of tools for computing hydrological signatures Github Actions
Download Stat High-level API for asynchronous requests with persistent caching Github Actions
Download Stat Send queries to any ArcGIS RESTful-, WMS-, and WFS-based services Github Actions
Download Stat Utilities for manipulating geospatial, (Geo)JSON, and (Geo)TIFF data Github Actions

HyRiver: Hydroclimate Data Retriever

Features

HyRiver is a software stack consisting of ten Python libraries that are designed to aid in hydroclimate analysis through web services. Currently, this project only includes hydrology and climatology data within the US. Some major capabilities of HyRiver are:

  • Easy access to many web services for subsetting data on server-side and returning the requests as masked Datasets or GeoDataFrames.
  • Splitting large requests into smaller chunks, under-the-hood, since web services often limit the number of features per request. So the only bottleneck for subsetting the data is your local machine memory.
  • Navigating and subsetting NHDPlus database (both medium- and high-resolution) using web services.
  • Cleaning up the vector NHDPlus data, fixing some common issues, and computing vector-based accumulation through a river network.
  • A URL inventory for many popular (and tested) web services.
  • Some utilities for manipulating the obtained data and their visualization.

image

Please visit examples webpage to see some example notebooks. You can also watch these videos for a quick overview of HyRiver capabilities:

You can also try this project without installing it on your system by clicking on the binder badge. A Jupyter Lab instance with the HyRiver software stack pre-installed will be launched in your web browser, and you can start coding!

Please note that this project is in early development stages, while the provided functionalities should be stable, changes in APIs are possible in new releases. But we appreciate it if you give this project a try and provide feedback. Contributions are most welcome.

Moreover, requests for additional databases and functionalities can be submitted via issue trackers of packages.

Citation

If you use any of HyRiver packages in your research, we appreciate citations:

@article{Chegini_2021,
    author = {Chegini, Taher and Li, Hong-Yi and Leung, L. Ruby},
    doi = {10.21105/joss.03175},
    journal = {Journal of Open Source Software},
    month = {10},
    number = {66},
    pages = {1--3},
    title = {{HyRiver: Hydroclimate Data Retriever}},
    volume = {6},
    year = {2021}
}

Installation

You can install all the packages using pip:

$ pip install py3dep pynhd pygeohydro pydaymet pygridmet pynldas2 hydrosignatures pygeoogc pygeoutils async-retriever

Please note that installation with pip fails if libgdal is not installed on your system. You should install this package manually beforehand. For example, on Ubuntu-based distros the required package is libgdal-dev. If this package is installed on your system you should be able to run gdal-config --version successfully.

Alternatively, you can install them using conda:

$ conda install -c conda-forge py3dep pynhd pygeohydro pydaymet pygridmet pynldas2 hydrosignatures pygeoogc pygeoutils async-retriever

or mambaforge (recommended):

$ mamba install py3dep pynhd pygeohydro pydaymet pygridmet pynldas2 hydrosignatures pygeoogc pygeoutils async-retriever

Additionally, you can create a new environment, named hyriver with all the packages and optional dependencies installed with mambaforge using the provided environment.yml file:

$ mamba env create -f ./environment.yml

image

hyriver.github.io's People

Contributors

cheginit avatar dependabot[bot] avatar pre-commit-ci[bot] avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar

hyriver.github.io's Issues

Restoring NLDI().get_validchars()?

Looks like the NLDI().get_validchars() functionality was dropped, not sure when.

Any anticipation for adding this back? Personally, I found it helpful.

Cheers

Issue on page /readme/pynhd.html

Sorry, but I am new to programming and I'm trying to utilize the pynhd package. I have python version 3.10.5 and I tried installing pynhd using [pip install pynhd] (I did not have the brackets in my code, I just thought they would help separate my code from other text) but recieved an error. The error stated that it was a legacy-install-failure having to do with GDAL. What I expected to happen was that it would successfully install so that I could utilize the package in my pythonWin IDE. I have tried downloading the appropriate GDAL file for my computer as well as one that might support pynhd but nothing has seemed to work.

I'm also new to GitHub so I'm not sure if this is in the correct location. Again my apologies.

PyGeoOGC: Retrieve Data from WMS and WFS Services

Hi !
I'm trying to retrieve data from the FEMA NFHL services using pygeoogc package, but I'm getting some errors using both WMS and WFS services:

Trying with WFS (trying to check the layers available but I also tried with public_NFHL:Base_Flood_Elevations as in the example and led to the same error):

wfs = WFS(ServiceURL().wfs.fema, layer=None,outformat="esrigeojson",crs="epsg:4326")

I get this error:

KeyError                                  Traceback (most recent call last)
[<ipython-input-30-c966dfce4278>](https://localhost:8080/#) in <module>
      5 #wms = WMS(ServiceURL().wms.fema, layers="",outformat="esrigeojson",crs="epsg:4326")
      6 
----> 7 wfs = WFS(ServiceURL().wfs.fema, layer=None,outformat="esrigeojson",crs="epsg:4326")

7 frames
[/usr/local/lib/python3.8/dist-packages/owslib/feature/schema.py](https://localhost:8080/#) in _construct_schema(elements, nsmap)
    120 
    121     for element in elements:
--> 122         data_type = element.attrib["type"].replace(gml_key + ":", "")
    123         name = element.attrib["name"]
    124         non_nillable = element.attrib.get("nillable", "false") == "false"

src/lxml/etree.pyx in lxml.etree._Attrib.__getitem__()

KeyError: 'type'

Trying also the WMS service from FEMA:

from pygeoogc import ArcGISRESTful, WFS, WMS, ServiceURL

wms = WMS(ServiceURL().wms.fema, layers="",outformat="esrigeojson",crs="epsg:4326")

I get this error:

Traceback (most recent call last):

  File "/usr/local/lib/python3.8/dist-packages/IPython/core/interactiveshell.py", line 3326, in run_code
    exec(code_obj, self.user_global_ns, self.user_ns)

  File ["<ipython-input-28-a4d58331326e>"](https://localhost:8080/#), line 5, in <module>
    wms = WMS(ServiceURL().wms.fema, layers="",outformat="esrigeojson",crs="epsg:4326")

  File "/usr/local/lib/python3.8/dist-packages/pygeoogc/pygeoogc.py", line 330, in __init__
    self.client = WMSBase(

  File "<string>", line 9, in __init__

  File "/usr/local/lib/python3.8/dist-packages/pygeoogc/core.py", line 461, in __post_init__
    self.get_service_options()

  File "/usr/local/lib/python3.8/dist-packages/pygeoogc/core.py", line 468, in get_service_options
    wms = WebMapService(self.url, version=self.version)

  File "/usr/local/lib/python3.8/dist-packages/owslib/wms.py", line 54, in WebMapService
    return wms130.WebMapService_1_3_0(

  File "/usr/local/lib/python3.8/dist-packages/owslib/map/wms130.py", line 75, in __init__
    self._capabilities = reader.read(self.url, timeout=self.timeout)

  File "/usr/local/lib/python3.8/dist-packages/owslib/map/common.py", line 69, in read
    return etree.fromstring(raw_text)

  File "src/lxml/etree.pyx", line 3257, in lxml.etree.fromstring

  File "src/lxml/parser.pxi", line 1916, in lxml.etree._parseMemoryDocument

  File "src/lxml/parser.pxi", line 1803, in lxml.etree._parseDoc

  File "src/lxml/parser.pxi", line 1144, in lxml.etree._BaseParser._parseDoc

  File "src/lxml/parser.pxi", line 618, in lxml.etree._ParserContext._handleParseResultDoc

  File "src/lxml/parser.pxi", line 728, in lxml.etree._handleParseResult

  File "src/lxml/parser.pxi", line 657, in lxml.etree._raiseParseError

  File "<string>", line 42
XMLSyntaxError: EntityRef: expecting ';', line 42, column 122

I checked the FEMA portal, but I found the same URLs to access their WMS and WFS services. Do you know what these errors are about?

Thank you!

Error in py3dep when used in pygeoapi

I'm using py3dep package as part of a process we are adding to our local copy pygeoapi. py3dep is used to pull topography from which elevation is interpolated onto a cross-section. When instantiating pygeoapi with this process included I get the following error: RuntimeError: There is no current event loop in thread 'Thread-28'. It looks like it connected to nest_asyncio.apply in pygeoogc/utils.py. The traceback follows. Any help would be greatly appreciated.

Traceback (most recent call last):
File "/home/rmcd/miniconda3/envs/pygeoapi_dev/lib/python3.9/site-packages/flask/app.py", line 2069, in call
return self.wsgi_app(environ, start_response)
File "/home/rmcd/miniconda3/envs/pygeoapi_dev/lib/python3.9/site-packages/flask/app.py", line 2054, in wsgi_app
response = self.handle_exception(e)
File "/home/rmcd/miniconda3/envs/pygeoapi_dev/lib/python3.9/site-packages/flask/app.py", line 2051, in wsgi_app
response = self.full_dispatch_request()
File "/home/rmcd/miniconda3/envs/pygeoapi_dev/lib/python3.9/site-packages/flask/app.py", line 1501, in full_dispatch_request
rv = self.handle_user_exception(e)
File "/home/rmcd/miniconda3/envs/pygeoapi_dev/lib/python3.9/site-packages/flask/app.py", line 1499, in full_dispatch_request
rv = self.dispatch_request()
File "/home/rmcd/miniconda3/envs/pygeoapi_dev/lib/python3.9/site-packages/flask/app.py", line 1485, in dispatch_request
return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args)
File "/home/rmcd/miniconda3/envs/pygeoapi_dev/lib/python3.9/site-packages/pygeoapi-0.11.dev0-py3.9.egg/pygeoapi/flask_app.py", line 375, in get_processes
headers, status_code, content = api_.describe_processes(
File "/home/rmcd/miniconda3/envs/pygeoapi_dev/lib/python3.9/site-packages/pygeoapi-0.11.dev0-py3.9.egg/pygeoapi/api.py", line 122, in inner
return func(cls, headers_, format_, *args, **kwargs)
File "/home/rmcd/miniconda3/envs/pygeoapi_dev/lib/python3.9/site-packages/pygeoapi-0.11.dev0-py3.9.egg/pygeoapi/linked_data.py", line 54, in inner
return func(*args, **kwargs)
File "/home/rmcd/miniconda3/envs/pygeoapi_dev/lib/python3.9/site-packages/pygeoapi-0.11.dev0-py3.9.egg/pygeoapi/api.py", line 1765, in describe_processes
p = load_plugin('process',
File "/home/rmcd/miniconda3/envs/pygeoapi_dev/lib/python3.9/site-packages/pygeoapi-0.11.dev0-py3.9.egg/pygeoapi/plugin.py", line 105, in load_plugin
module = importlib.import_module(packagename)
File "/home/rmcd/miniconda3/envs/pygeoapi_dev/lib/python3.9/importlib/init.py", line 127, in import_module
return _bootstrap._gcd_import(name[level:], package, level)
File "", line 1030, in _gcd_import

File "", line 1007, in _find_and_load

File "", line 986, in _find_and_load_unlocked

File "", line 680, in _load_unlocked

File "", line 790, in exec_module

File "", line 228, in _call_with_frames_removed

File "/home/rmcd/miniconda3/envs/pygeoapi_dev/lib/python3.9/site-packages/pygeoapi-0.11.dev0-py3.9.egg/pygeoapi/process/nldi_xsatpoint.py", line 6, in
from nldi_xstool.nldi_xstool import getXSAtPoint
File "/home/rmcd/miniconda3/envs/pygeoapi_dev/lib/python3.9/site-packages/nldi_xstool/nldi_xstool.py", line 7, in
import py3dep
File "/home/rmcd/miniconda3/envs/pygeoapi_dev/lib/python3.9/site-packages/py3dep/init.py", line 6, in
from .py3dep import deg2mpm, elevation_bycoords, elevation_bygrid, get_map
File "/home/rmcd/miniconda3/envs/pygeoapi_dev/lib/python3.9/site-packages/py3dep/py3dep.py", line 12, in
from pygeoogc import WMS, MatchCRS, RetrySession, ServiceURL
File "/home/rmcd/miniconda3/envs/pygeoapi_dev/lib/python3.9/site-packages/pygeoogc/init.py", line 16, in
from .pygeoogc import WFS, WMS, ArcGISRESTful, ServiceURL
File "/home/rmcd/miniconda3/envs/pygeoapi_dev/lib/python3.9/site-packages/pygeoogc/pygeoogc.py", line 16, in
from . import utils
File "/home/rmcd/miniconda3/envs/pygeoapi_dev/lib/python3.9/site-packages/pygeoogc/utils.py", line 35, in
nest_asyncio.apply()
File "/home/rmcd/miniconda3/envs/pygeoapi_dev/lib/python3.9/site-packages/nest_asyncio.py", line 12, in apply
loop = loop or asyncio.get_event_loop()
File "/home/rmcd/miniconda3/envs/pygeoapi_dev/lib/python3.9/asyncio/events.py", line 642, in get_event_loop
raise RuntimeError('There is no current event loop in thread %r.'
RuntimeError: There is no current event loop in thread 'Thread-28'.

The function using py3dep is here: https://github.com/ACWI-SSWD/nldi_xstool/blob/8909af7907bee0965f1c848e5d26db6e644a59b7/nldi_xstool/nldi_xstool.py#L97

3DEP Gallery Tutorial : 'Dataset' object has no attribute 'crs'

What happened:
After installing the full HyRiver package environment I tried to run the 3DEP tutorial notebook on the documentation page and when I got to cell 9

clm = daymet.get_bygeom(geometry, ("2005-01-01", "2005-01-31"), variables="tmin")
elev = py3dep.elevation_bygrid(clm.x.values, clm.y.values, clm.crs, 1000)
attrs = clm.attrs
clm = xr.merge([clm, elev])
clm["elevation"] = clm.elevation.where(~np.isnan(clm.isel(time=0).tmin), drop=True)
clm.attrs.update(attrs)

I got the error:

---------------------------------------------------------------------------
AttributeError                            Traceback (most recent call last)
c:\Users\Luc\projects\goodwin_creek\test_3dep.ipynb Cell 9' in <cell line: 2>()
      [1] clm = daymet.get_bygeom(geometry, ("2005-01-01", "2005-01-31"), variables="tmin", crs='epsg:4326')
----> [2] elev = py3dep.elevation_bygrid(clm.x.values, clm.y.values, clm.crs, 1000)
      [3] attrs = clm.attrs
      [4] clm = xr.merge([clm, elev])

File c:\Users\Luc\anaconda3\envs\hyriver\lib\site-packages\xarray\core\common.py:239, in AttrAccessMixin.__getattr__(self, name)
    237         with suppress(KeyError):
    238             return source[name]
--> 239 raise AttributeError(
    240     f"{type(self).__name__!r} object has no attribute {name!r}"
    241 )

AttributeError: 'Dataset' object has no attribute 'crs'

It seems the pydaymet.get_bygeom function does not return a crs argument.

What you expected to happen:
Being able to run the tutorial example as shown in the documentation.

Minimal Complete Verifiable Example:

Run all the cells of the 3DEP tutorial until cell 9.

Anything else we need to know?:

I will note that conda struggled to resolve the environment using the provided command:
conda install -c conda-forge py3dep pynhd pygeohydro pydaymet pygeoogc pygeoutils async_retriever what worked is installing the individual packages one by one and specifying the version of each package and verifying every time the right versions were installed. I skipped pygeoutils and async_retriever as they were already installed (required packages) for the previous packages.

I hope I did a good job at explaining the problem!

Environment:

Output of pygeohydro.show_versions()

INSTALLED VERSIONS

commit: None
python: 3.9.13 | packaged by conda-forge | (main, May 27 2022, 16:50:36) [MSC v.1929 64 bit (AMD64)]
python-bits: 64
OS: Windows
OS-release: 10
machine: AMD64
processor: Intel64 Family 6 Model 141 Stepping 1, GenuineIntel
byteorder: little
LC_ALL: None
LANG: None
LOCALE: English_United States.1252
libhdf5: 1.12.1
libnetcdf: 4.8.1

aiodns: 3.0.0
aiohttp: 3.8.1
aiohttp-client-cache: 0.7.0
aiosqlite: 0.17.0
async-retriever: 0.3.3
bottleneck: 1.3.4
brotli: installed
cchardet: 2.1.7
click: 8.1.3
cytoolz: 0.11.2
dask: 2022.6.0
defusedxml: 0.7.1
folium: 0.12.1.post1
geopandas: 0.10.2
lxml: 4.9.0
matplotlib: 3.4.3
netCDF4: 1.5.8
networkx: 2.8.4
numpy: 1.22.4
owslib: 0.25.0
pandas: 1.4.2
py3dep: 0.13.2
pyarrow: 6.0.0
pydantic: 1.9.1
pydaymet: 0.13.2
pygeohydro: 0.13.2
pygeoogc: 0.13.2
pygeos: 0.12.0
pygeoutils: 0.13.2
pynhd: 0.13.2
pyproj: 3.3.1
pytest: None
pytest-cov: None
rasterio: 1.2.10
requests: 2.28.0
requests-cache: 0.9.4
richdem: None
rioxarray: 0.11.1
scipy: 1.8.1
shapely: 1.8.2
tables: None
ujson: 5.3.0
urllib3: 1.26.9
xarray: 2022.3.0
xdist: None
yaml: 6.0

Issue on page /examples/notebooks/hymod_calibration.html

Hi! I'm trying to run the notebook example that uses HYMOD and HyRiver packages for hydrological modelling. I changed the station ID to another station of my interest, and I changed the dates for a longer period than the one in the example. I'm getting the following error when I try to compute the PET using the method hargreaves_samani:

---------------------------------------------------------------------------
OperationalError                          Traceback (most recent call last)
Cell In [14], line 1
----> 1 clm = daymet.get_bygeom(basin.geometry[0], dates, variables="prcp", pet="hargreaves_samani")

File /usr/local/lib/python3.10/site-packages/pydaymet/pydaymet.py:467, in get_bygeom(geometry, dates, crs, variables, region, time_scale, pet, pet_params, snow, snow_params, ssl)
    453 urls, kwds = zip(
    454     *_gridded_urls(
    455         daymet.time_codes[time_scale],
   (...)
    460     )
    461 )
    463 try:
    464     clm: xr.Dataset = xr.open_mfdataset(
    465         (  # type: ignore
    466             io.BytesIO(r)
--> 467             for r in ar.retrieve_binary(urls, request_kwds=kwds, max_workers=MAX_CONN, ssl=ssl)
    468         ),
    469         engine="scipy",
    470         coords="minimal",
    471     )
    472 except ValueError as ex:
    473     msg = (
    474         "The service did NOT process your request successfully. "
    475         + "Check your inputs and try again."
    476     )

File /usr/local/lib/python3.10/site-packages/async_retriever/async_retriever.py:496, in retrieve_binary(urls, request_kwds, request_method, max_workers, cache_name, timeout, expire_after, ssl, disable)
    454 def retrieve_binary(
    455     urls: Sequence[StrOrURL],
    456     request_kwds: Optional[Sequence[Dict[str, Any]]] = None,
   (...)
    463     disable: bool = False,
    464 ) -> List[bytes]:
    465     r"""Send async requests and get the response as ``bytes``.
    466 
    467     Parameters
   (...)
    494         List of responses in the order of input URLs.
    495     """
--> 496     resp: List[bytes] = retrieve(  # type: ignore
    497         urls,
    498         "binary",
    499         request_kwds,
    500         request_method,
    501         max_workers,
    502         cache_name,
    503         timeout,
    504         expire_after,
    505         ssl,
    506         disable,
    507     )
    508     return resp

File /usr/local/lib/python3.10/site-packages/async_retriever/async_retriever.py:246, in retrieve(urls, read_method, request_kwds, request_method, max_workers, cache_name, timeout, expire_after, ssl, disable)
    242 chunked_reqs = tlz.partition_all(max_workers, inp.url_kwds)
    244 results = (loop.run_until_complete(session(url_kwds=c)) for c in chunked_reqs)
--> 246 resp = [r for _, r in sorted(tlz.concat(results))]
    247 if new_loop:
    248     loop.close()

File /usr/local/lib/python3.10/site-packages/async_retriever/async_retriever.py:244, in <genexpr>(.0)
    230 session = tlz.partial(
    231     async_session,
    232     read=inp.read_method,
   (...)
    239     disable=disable,
    240 )
    242 chunked_reqs = tlz.partition_all(max_workers, inp.url_kwds)
--> 244 results = (loop.run_until_complete(session(url_kwds=c)) for c in chunked_reqs)
    246 resp = [r for _, r in sorted(tlz.concat(results))]
    247 if new_loop:

File /usr/local/lib/python3.10/site-packages/nest_asyncio.py:90, in _patch_loop.<locals>.run_until_complete(self, future)
     87 if not f.done():
     88     raise RuntimeError(
     89         'Event loop stopped before Future completed.')
---> 90 return f.result()

File /usr/local/lib/python3.10/asyncio/futures.py:201, in Future.result(self)
    199 self.__log_traceback = False
    200 if self._exception is not None:
--> 201     raise self._exception.with_traceback(self._exception_tb)
    202 return self._result

File /usr/local/lib/python3.10/asyncio/tasks.py:232, in Task.__step(***failed resolving arguments***)
    228 try:
    229     if exc is None:
    230         # We use the `send` method directly, because coroutines
    231         # don't have `__iter__` and `__next__` methods.
--> 232         result = coro.send(None)
    233     else:
    234         result = coro.throw(exc)

File /usr/local/lib/python3.10/site-packages/async_retriever/async_retriever.py:92, in async_session(url_kwds, read, r_kwds, request_method, cache_name, timeout, expire_after, ssl, disable)
     87 request_func = getattr(session, request_method.lower())
     88 tasks = (
     89     utils.retriever(uid, url, kwds, request_func, read, r_kwds)
     90     for uid, url, kwds in url_kwds
     91 )
---> 92 return await asyncio.gather(*tasks)

File /usr/local/lib/python3.10/asyncio/tasks.py:304, in Task.__wakeup(self, future)
    302 def __wakeup(self, future):
    303     try:
--> 304         future.result()
    305     except BaseException as exc:
    306         # This may also be a cancellation.
    307         self.__step(exc)

File /usr/local/lib/python3.10/asyncio/tasks.py:232, in Task.__step(***failed resolving arguments***)
    228 try:
    229     if exc is None:
    230         # We use the `send` method directly, because coroutines
    231         # don't have `__iter__` and `__next__` methods.
--> 232         result = coro.send(None)
    233     else:
    234         result = coro.throw(exc)

File /usr/local/lib/python3.10/site-packages/async_retriever/utils.py:62, in retriever(uid, url, s_kwds, session, read_type, r_kwds)
     30 async def retriever(
     31     uid: int,
     32     url: StrOrURL,
   (...)
     36     r_kwds: Dict[str, None],
     37 ) -> Tuple[int, Union[str, Awaitable[Union[str, bytes, Dict[str, Any]]]]]:
     38     """Create an async request and return the response as binary.
     39 
     40     Parameters
   (...)
     60         The retrieved response as binary.
     61     """
---> 62     async with session(url, **s_kwds) as response:
     63         try:
     64             return uid, await getattr(response, read_type)(**r_kwds)

File /usr/local/lib/python3.10/site-packages/aiohttp/client.py:1141, in _BaseRequestContextManager.__aenter__(self)
   1140 async def __aenter__(self) -> _RetType:
-> 1141     self._resp = await self._coro
   1142     return self._resp

File /usr/local/lib/python3.10/site-packages/forge/_revision.py:322, in Revision.__call__.<locals>.inner(*args, **kwargs)
    318 @functools.wraps(callable)
    319 async def inner(*args, **kwargs):
    320     # pylint: disable=E1102, not-callable
    321     mapped = inner.__mapper__(*args, **kwargs)
--> 322     return await callable(*mapped.args, **mapped.kwargs)

File /usr/local/lib/python3.10/site-packages/aiohttp_client_cache/session.py:62, in CacheMixin._request(self, method, str_or_url, expire_after, **kwargs)
     60 actions.update_from_response(new_response)
     61 if await self.cache.is_cacheable(new_response, actions):
---> 62     await self.cache.save_response(new_response, actions.key, actions.expires)
     63 return set_response_defaults(new_response)

File /usr/local/lib/python3.10/site-packages/aiohttp_client_cache/backends/base.py:169, in CacheBackend.save_response(self, response, cache_key, expires)
    167 cache_key = cache_key or self.create_key(response.method, response.url)
    168 cached_response = await CachedResponse.from_client_response(response, expires)
--> 169 await self.responses.write(cache_key, cached_response)
    171 # Alias any redirect requests to the same cache key
    172 for r in response.history:

File /usr/local/lib/python3.10/site-packages/aiohttp_client_cache/backends/sqlite.py:189, in SQLitePickleCache.write(self, key, item)
    188 async def write(self, key, item):
--> 189     await super().write(key, sqlite3.Binary(self.serialize(item)))

File /usr/local/lib/python3.10/site-packages/aiohttp_client_cache/backends/sqlite.py:170, in SQLiteCache.write(self, key, item)
    168 async def write(self, key: str, item: Union[ResponseOrKey, sqlite3.Binary]):
    169     async with self.get_connection(autocommit=True) as db:
--> 170         await db.execute(
    171             f'INSERT OR REPLACE INTO `{self.table_name}` (key,value) VALUES (?,?)',
    172             (key, item),
    173         )

File /usr/local/lib/python3.10/site-packages/aiosqlite/core.py:184, in Connection.execute(self, sql, parameters)
    182 if parameters is None:
    183     parameters = []
--> 184 cursor = await self._execute(self._conn.execute, sql, parameters)
    185 return Cursor(self, cursor)

File /usr/local/lib/python3.10/site-packages/aiosqlite/core.py:129, in Connection._execute(self, fn, *args, **kwargs)
    125 future = asyncio.get_event_loop().create_future()
    127 self._tx.put_nowait((future, function))
--> 129 return await future

File /usr/local/lib/python3.10/asyncio/futures.py:285, in Future.__await__(self)
    283 if not self.done():
    284     self._asyncio_future_blocking = True
--> 285     yield self  # This tells Task to wait for completion.
    286 if not self.done():
    287     raise RuntimeError("await wasn't used with future")

File /usr/local/lib/python3.10/asyncio/tasks.py:304, in Task.__wakeup(self, future)
    302 def __wakeup(self, future):
    303     try:
--> 304         future.result()
    305     except BaseException as exc:
    306         # This may also be a cancellation.
    307         self.__step(exc)

File /usr/local/lib/python3.10/asyncio/futures.py:201, in Future.result(self)
    199 self.__log_traceback = False
    200 if self._exception is not None:
--> 201     raise self._exception.with_traceback(self._exception_tb)
    202 return self._result

File /usr/local/lib/python3.10/site-packages/aiosqlite/core.py:102, in Connection.run(self)
    100 try:
    101     LOG.debug("executing %s", function)
--> 102     result = function()
    103     LOG.debug("operation %s completed", function)
    105     def set_result(fut, result):

OperationalError: database is locked

My dates are defined as ("1993-01-01", "2015-12-31"), but I also tested with a shorter period to check if the database gets locked because of the amount of data I'm retrieving. But I got the same error. Do you have any suggestions on how to solve this error?

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.