Comments (13)
Hey @valeriupredoi , thanks for coming back to this. Actually, this still doesn not work for me at least. However, i somehow suspect that this is maybe an issue with my openid. I just tried again with a fresh install and i still don't have access to esgf.ceda.ac.uk
while esgf.dwd.de
and cordexesg.dmi.dk
work fine. Note, that i can download those files using the ESGF web portal, so that works fine. But consider, .e.g, this:
from netCDF4 import Dataset
from pyesgf.logon import LogonManager
urls = [
"https://esgf.ceda.ac.uk/thredds/dodsC/esg_cordex/CORDEX/output/EUR-11/MOHC/ECMWF-ERAINT/evaluation/r1i1p1/MOHC-HadREM3-GA7-05/v1/mon/tas/v20200330/tas_EUR-11_ECMWF-ERAINT_evaluation_r1i1p1_MOHC-HadREM3-GA7-05_v1_mon_198212-199012.nc",
"http://cordexesg.dmi.dk/thredds/dodsC/cordex_general/cordex/output/EUR-11/DMI/ECMWF-ERAINT/evaluation/r1i1p1/DMI-HIRHAM5/v1/mon/pr/v20140620/pr_EUR-11_ECMWF-ERAINT_evaluation_r1i1p1_DMI-HIRHAM5_v1_mon_198901-199012.nc",
"https://esgf.dwd.de/thredds/dodsC/esgf2_1/cordex/output/EUR-11/CLMcom/MIROC-MIROC5/rcp26/r1i1p1/CLMcom-CCLM4-8-17/v1/mon/tas/v20180707/tas_EUR-11_MIROC-MIROC5_rcp26_r1i1p1_CLMcom-CCLM4-8-17_v1_mon_200601-201012.nc",
]
lm = LogonManager()
# logon
if not lm.is_logged_on():
myproxy_host = "esgf-data.dkrz.de"
lm.logon(hostname=myproxy_host, interactive=True, bootstrap=True)
print(lm.is_logged_on())
for url in urls:
try:
Dataset(url)
print(f"This worked: {url}")
except Exception as e:
print(e)
print(f"Failed: {url}")
which only works for the dwd data node:
[Errno -90] NetCDF: file not found: 'https://esgf.ceda.ac.uk/thredds/dodsC/esg_cordex/CORDEX/output/EUR-11/MOHC/ECMWF-ERAINT/evaluation/r1i1p1/MOHC-HadREM3-GA7-05/v1/mon/tas/v20200330/tas_EUR-11_ECMWF-ERAINT_evaluation_r1i1p1_MOHC-HadREM3-GA7-05_v1_mon_198212-199012.nc'
Failed: https://esgf.ceda.ac.uk/thredds/dodsC/esg_cordex/CORDEX/output/EUR-11/MOHC/ECMWF-ERAINT/evaluation/r1i1p1/MOHC-HadREM3-GA7-05/v1/mon/tas/v20200330/tas_EUR-11_ECMWF-ERAINT_evaluation_r1i1p1_MOHC-HadREM3-GA7-05_v1_mon_198212-199012.nc
syntax error, unexpected $end, expecting SCAN_ATTR or SCAN_DATASET or SCAN_ERROR
context: ^
This worked: http://cordexesg.dmi.dk/thredds/dodsC/cordex_general/cordex/output/EUR-11/DMI/ECMWF-ERAINT/evaluation/r1i1p1/DMI-HIRHAM5/v1/mon/pr/v20140620/pr_EUR-11_ECMWF-ERAINT_evaluation_r1i1p1_DMI-HIRHAM5_v1_mon_198901-199012.nc
This worked: https://esgf.dwd.de/thredds/dodsC/esgf2_1/cordex/output/EUR-11/CLMcom/MIROC-MIROC5/rcp26/r1i1p1/CLMcom-CCLM4-8-17/v1/mon/tas/v20180707/tas_EUR-11_MIROC-MIROC5_rcp26_r1i1p1_CLMcom-CCLM4-8-17_v1_mon_200601-201012.nc
I think it would be good to still keep this open since it's different from #84 and i'll ask some colleagues to check this too. For me, the main drawback is not having access to the CEDA ESGF data node. I wish this CORDEX logon could be removed after all sometime... 😒
from esgf-pyclient.
This looks very similar to #84. Is this a different issue?
from esgf-pyclient.
Today i tried accessing over the web portal after it didn't work in my script. i got something i have never seen before (i'm registered with CORDEX_Commercial/Research)
Just want to drop it off here since it might be related...
from esgf-pyclient.
We have currently some issues with cordexesg.dmi.dk after moving to new hardware. We are working on a solution. Sorry for that.
from esgf-pyclient.
@soay Thanks! I'll try again soon!
from esgf-pyclient.
Hi @larsbuntemeyer - did it eventually work? Can this be closed, please? Cheers 🍺
from esgf-pyclient.
Note, i updated my comment, since i had a typo in the dmi url.
from esgf-pyclient.
I've seen some really fishy stuff going on lately with the CEDA Thredds catalog, might have to ping them figure out what they're up to, BTW I don't see any dodsC in their main catalog at all https://esgf.ceda.ac.uk/thredds/catalog/catalog.html
from esgf-pyclient.
at any rate, I am the new maintainer of this package, not in the know much about the actual functional subtleties, for that prob best to ping @bouweandela and/or @alaniwi 👍
from esgf-pyclient.
However, i somehow suspect that this is maybe an issue with my openid.
Could you try if you can reproduce the issue using e.g. wget? Then we are sure that it is not related to esgf-pyclient
and you could ask for help on the ESGF mailinglist.
from esgf-pyclient.
@bouweandela Yes, i can confirm that i also have this issue uswing wget/curl. However, i can access esgf-data1.ceda.ac.uk
via https download, e.g.,
curl https://esgf-data1.ceda.ac.uk/thredds/fileServer/esg_cordex/CORDEX/output/EUR-11/MOHC/ECMWF-ERAINT/evaluation/r1i1p1/MOHC-HadREM3-GA7-05/v1/mon/tas/v20200330/tas_EUR-11_ECMWF-ERAINT_evaluation_r1i1p1_MOHC-HadREM3-GA7-05_v1_mon_198212-199012.nc -o tas_EUR-11_ECMWF-ERAINT_evaluation_r1i1p1_MOHC-HadREM3-GA7-05_v1_mon_198212-199012.nc
but i can not access the same file via opendap here:
http://esgf.ceda.ac.uk/thredds/dodsC/esg_cordex/CORDEX/output/EUR-11/MOHC/ECMWF-ERAINT/evaluation/r1i1p1/MOHC-HadREM3-GA7-05/v1/mon/tas/v20200330/tas_EUR-11_ECMWF-ERAINT_evaluation_r1i1p1_MOHC-HadREM3-GA7-05_v1_mon_198212-199012.nc
It was a little surprised that i can now access the data without login! That's great! I would confirm that this is not an issue with esgf-pyclient
.
from esgf-pyclient.
I've heard 2 things that probably explain the above observations:
- opendap on ESGF is deprecated and will not be available on systems that make use of the new deployment
- logging in to access CORDEX data will no longer be needed 'soon'
but I'm not personally involved in running ESGF, so your best bet to get some answers is contacting the ESGF user mailing list.
from esgf-pyclient.
@bouweandela Thanks for the updates, i'll check out the mailing list. I think this could be closed then!
from esgf-pyclient.
Related Issues (20)
- Failing CI build: parsing of HTTPResponse by defusedxml.ElementTree HOT 1
- No consistent result from different queries (CMIP6) HOT 1
- Spatial constraints
- Dependency problem (version mismatch between pyesgf and requests library) leading to AttributeError: module 'requests_cache' has no attribute 'core' HOT 5
- add "facets" keyword argument to DatasetResult.file_context
- New release? HOT 5
- Import trial of `MyProxyClient` in `pyesgf/logon.py` outputs misleading error and incompatibility with `cryptography` from Anaconda `main` channel HOT 2
- logon for http request HOT 2
- lm.logon timeout HOT 12
- Facets warning with aggregation_context(): unexpected keyword
- CMIP6 data availability? HOT 2
- Unexpected number of results for large query
- Not matching all the expected files HOT 4
- logon refused from my laptop HOT 3
- Facets warning appears even when facets are specified HOT 5
- collection import errors from python 3.10 HOT 1
- [via MyProxyClient] Import of `MyProxyClient` fails due to complete retirement of `SafeConfigParser` in Python 3.12 HOT 2
- Fix `tests/test_results` test module
- 503 HTTPErrors should not be failing the tests
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from esgf-pyclient.