Giter Site home page Giter Site logo

equinor / ert Goto Github PK

View Code? Open in Web Editor NEW
92.0 19.0 102.0 267.73 MB

ERT - Ensemble based Reservoir Tool - is designed for running ensembles of dynamical models such as reservoir models, in order to do sensitivity analysis and data assimilation. ERT supports data assimilation using the Ensemble Smoother (ES), Ensemble Smoother with Multiple Data Assimilation (ES-MDA) and Iterative Ensemble Smoother (IES).

Home Page: https://ert.readthedocs.io/en/latest/

License: GNU General Public License v3.0

Python 92.31% Shell 0.37% Jinja 0.06% CMake 0.52% C++ 6.74%
hacktoberfest python simulation scientific hactoberfestivus

ert's Introduction

ert

Build Status PyPI - Python Version Code Style Type checking codecov License: GPL v3

ert - Ensemble based Reservoir Tool - is designed for running ensembles of dynamical models such as reservoir models, in order to do sensitivity analysis and data assimilation. ert supports data assimilation using the Ensemble Smoother (ES), Ensemble Smoother with Multiple Data Assimilation (ES-MDA) and Iterative Ensemble Smoother (IES).

Installation

$ pip install ert
$ ert --help

or, for the latest development version (requires Python development headers):

$ pip install git+https://github.com/equinor/ert.git@main
$ ert --help

For examples and help with configuration, see the ert Documentation.

Developing

ert was originally written in C/C++ but most new code is Python.

Developing Python

You might first want to make sure that some system level packages are installed before attempting setup:

- pip
- python include headers
- (python) venv
- (python) setuptools
- (python) wheel

It is left as an exercise to the reader to figure out how to install these on their respective system.

To start developing the Python code, we suggest installing ert in editable mode into a virtual environment to isolate the install (substitute the appropriate way of sourcing venv for your shell):

# Create and enable a virtualenv
python3 -m venv my_virtualenv
source my_virtualenv/bin/activate

# Update build dependencies
pip install --upgrade pip wheel setuptools

# Download and install ert
git clone https://github.com/equinor/ert
cd ert
pip install --editable .

Test setup

Additional development packages must be installed to run the test suite:

pip install ".[dev]"
pytest tests/

Git LFS must be installed to get all the files. This is packaged as git-lfs on Ubuntu, Fedora or macOS Homebrew. For Equinor RGS node users, it is possible to use git from Red Hat Software Collections:

source /opt/rh/rh-git227/enable

test-data/block_storage is a submodule and must be checked out.

git submodule update --init --recursive

If you checked out submodules without having git lfs installed, you can force git lfs to run in all submodules with:

git submodule foreach "git lfs pull"

Style requirements

There are a set of style requirements, which are gathered in the pre-commit configuration, to have it automatically run on each commit do:

$ pip install pre-commit
$ pre-commit install

Trouble with setup

If you encounter problems during install, try deleting the _skbuild folder before reinstalling.

As a simple test of your ert installation, you may try to run one of the examples, for instance:

cd test-data/poly_example
# for non-gui trial run
ert test_run poly.ert
# for gui trial run
ert gui poly.ert

Note that in order to parse floating point numbers from text files correctly, your locale must be set such that . is the decimal separator, e.g. by setting

# export LC_NUMERIC=en_US.UTF-8

in bash (or an equivalent way of setting that environment variable for your shell).

Developing C++

C++ is the backbone of ert as in used extensively in important parts of ert. There's a combination of legacy code and newer refactored code. The end goal is likely that some core performance-critical functionality will be implemented in C++ and the rest of the business logic will be implemented in Python.

While running --editable will create the necessary Python extension module (src/ert/_clib.cpython-*.so), changing C++ code will not take effect even when reloading ert. This requires recompilation, which means reinstalling ert from scratch.

To avoid recompiling already-compiled source files, we provide the script/build script. From a fresh virtualenv:

git clone https://github.com/equinor/ert
cd ert
script/build

This command will update pip if necessary, install the build dependencies, compile ert and install in editable mode, and finally install the runtime requirements. Further invocations will only build the necessary source files. To do a full rebuild, delete the _skbuild directory.

Note: This will create a debug build, which is faster to compile and comes with debugging functionality enabled. The downside is that this makes the code unoptimised and slow. Debugging flags are therefore not present in builds of ert that we release on Komodo or PyPI. To build a release build for development, use script/build --release.

Notes

  1. If pip reinstallation fails during the compilation step, try removing the _skbuild directory.

  2. The default maximum number of open files is normally relatively low on MacOS and some Linux distributions. This is likely to make tests crash with mysterious error-messages. You can inspect the current limits in your shell by issuing the command ulimit -a. In order to increase maximum number of open files, run ulimit -n 16384 (or some other large number) and put the command in your .profile to make it persist.

Running C++ tests

The C++ code and tests require resdata. As long as you have pip install resdata'd into your Python virtualenv all should work.

# Create and enable a virtualenv
python3 -m venv my_virtualenv
source my_virtualenv/bin/activate

# Install build dependencies
pip install pybind11 conan cmake resdata

# Build ert and tests
mkdir build && cd build
cmake ../src/clib -DCMAKE_BUILD_TYPE=Debug
make -j$(nproc)

# Run tests
ctest --output-on-failure

Example usage

Basic ert test

To test if ert itself is working, go to test-data/poly_example and start ert by running poly.ert with ert gui

cd test-data/poly_example
ert gui poly.ert

This opens up the ert graphical user interface. Finally, test ert by starting and successfully running the simulation.

ert with a reservoir simulator

To actually get ert to work at your site you need to configure details about your system; at the very least this means you must configure where your reservoir simulator is installed. In addition you might want to configure e.g. queue system in the site-config file, but that is not strictly necessary for a basic test.

ert's People

Contributors

akva2 avatar andreas-el avatar arielalmendral avatar berland avatar bjarneherland avatar chflo avatar dafeda avatar dansava avatar eivindjahren avatar flikka avatar frode-aarstad avatar iloop2 avatar jholba avatar joakim-hove avatar jokva avatar jonathan-eq avatar jondequinor avatar lars-petter-hauge avatar maninfez avatar markusdregi avatar mortalisk avatar myrseth avatar oysteoh avatar oyvindeide avatar perroe avatar pinkwah avatar sondreso avatar valentin-krasontovitsch avatar verveerpj avatar xjules avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

ert's Issues

Block pressure dual variables

Setting the variable BPR in refcase as both with indices (h,k,l,) and global index (n)causes error when trying to plot BPR:n.

Matplotlib exception during plotting


An error occurred during plotting. This stack trace is helpful for
diagnosing the problem.

--------------------------------------------------------------------------------

File "~2.3/lib/python2.7/site-packages/ert_gui/tools/plot/plot_widget.py", line 89, in updatePlot
    self._plotFunction(plot_context)
  File "~2.3/lib/python2.7/site-packages/ert_gui/plottery/plots/ensemble.py", line 37, in plotEnsemble
    PlotTools.finalizePlot(plot_context, axes, default_x_label=default_x_label, default_y_label="Value")
  File "~2.3/lib/python2.7/site-packages/ert_gui/plottery/plots/plot_tools.py", line 91, in finalizePlot
    plot_context.figure().autofmt_xdate()
  File "~2.3/lib/python2.7/site-packages/matplotlib/figure.py", line 644, in autofmt_xdate
    for label in self.axes[0].get_xticklabels(which=which):
  File "~2.3/lib/python2.7/site-packages/matplotlib/axes/_base.py", line 3287, in get_xticklabels
    which=which))
  File "~2.3/lib/python2.7/site-packages/matplotlib/axis.py", line 1291, in get_ticklabels
    return self.get_majorticklabels()
  File "~2.3/lib/python2.7/site-packages/matplotlib/axis.py", line 1245, in get_majorticklabels
    ticks = self.get_major_ticks()
  File "~2.3/lib/python2.7/site-packages/matplotlib/axis.py", line 1396, in get_major_ticks
    numticks = len(self.get_major_locator()())
  File "~2.3/lib/python2.7/site-packages/matplotlib/dates.py", line 1249, in __call__
    self.refresh()
  File "~2.3/lib/python2.7/site-packages/matplotlib/dates.py", line 1269, in refresh
    dmin, dmax = self.viewlim_to_dt()
  File "~2.3/lib/python2.7/site-packages/matplotlib/dates.py", line 1026, in viewlim_to_dt
    .format(vmin))
Exception type: ValueError

view limit minimum -0.001 is less than 1 and is an invalid Matplotlib
date value. This often happens if you pass a non-datetime value to an
axis that has datetime units

Setup internal tests running FMU setup

We should setup some internal testing on Jenkins for the deployed versions of ERT. I suggest that we start by setting up a Jenkins job that runs all the cases from the FMU tutorial every night on stable. After this, we should do the same for testing, unstable and bleeding.

Parameter values as strings

Should support that parameter values are strings (not only numbers). Example 50% probability for value “depth_conv1”, 25% probability for “depth_conv2” and 25% probability for “depth_conv3”.

Feature toggling

General case
We could consider utilizing feature toggling for breaking changes to avoid breaking changes from one release to another. Instead we could follow a schema like this through 4 releases:

  • Old behavior
  • Old behavior is default, new behavior can be activated. Warning if using default informing that this will change in the future.
  • New behavior is default; warning if old behavior is utilized
  • only new behavior

In this manner we would never introduce hard, braking changes. However, such changes (over time) are necessary to clean up the code base.

Example
There was a change in behavior with respect to timesteps in libres in equinor/libres#439. This will affect users. It would be rather easy to follow the schema above in this scenario.

Parallell blocks of jobs

Support the possibility of specifying multiple jobs that would run independently in parallel.

Advanced option

I see you have removed "View/Advanced Options" from the main ERT menu.

It is not clear to me how you now can select a loaded analysis module for Iterated Ensemble Smoother which previously appeared on the Simulation mode button when Advanced options were set.

Parallell ecplise simulations

Number of cpu’s pr job should not be picked up from the Eclipse data deck as done today . Instead, there should be a separate keyword in the config file defining number of cpus pr job.

Display the progress of the individual forward models

Display information while running ERT similar to what is currently done in Everest. Should be possible to see:

  • how many of the realizations have completed and their status,
  • how far each running forward model has got, with time stamps for each the start and end of each job.

BUILD_ERT_GUI on request only

Adding back the possibility to only build the command line applications.

Building the GUI requires qt libraries for python among other ones and that makes it too heavy for instance for cases when the only thing you want is integration testing in a CI/CD workflow.
Also there are already other ert applications that one might want to use in a command line on a daily basis.
Using ert in a batch mode is useful in workflows that do not require user interaction.

Integrated support for design matrices

User requests integrated support for design matrices in libres. Possibility to run the the prior from a design matrix, then the posterior based on this prior.

Bug in ies gui element

Exception in thread ert_gui_simulation_thread:
Traceback (most recent call last):
  File "/usr/lib/python2.7/threading.py", line 801, in __bootstrap_inner
    self.run()
  File "/usr/local/lib/python2.7/dist-packages/ert_gui/simulation/run_dialog.py", line 122, in run
    self._run_model.startSimulations( arguments )
  File "/usr/local/lib/python2.7/dist-packages/ert_gui/simulation/models/base_run_model.py", line 61, in startSimulations
    self.runSimulations(arguments)
  File "/usr/local/lib/python2.7/dist-packages/ert_gui/simulation/models/iterated_ensemble_smoother.py", line 68, in runSimulations
    run_context = self.create_context( arguments , 0 )
  File "/usr/local/lib/python2.7/dist-packages/ert_gui/simulation/models/iterated_ensemble_smoother.py", line 128, in create_context
    run_context = ErtRunContext.ensemble_smoother( sim_fs, target_fs, mask, runpath_fmt, subst_list, itr)
TypeError: ensemble_smoother() takes exactly 8 arguments (7 given)

^CTraceback (most recent call last):
  File "/usr/local/lib/python2.7/dist-packages/ert_gui/simulation/run_dialog.py", line 142, in updateRunStatus
    def updateRunStatus(self):
KeyboardInterrupt

Improve GUI to better support model updating best practices

The GUI should in a natural manner allow the users to:

  • evaluate the prior
  • compare prior and observed values and based on this:
    • identify outliers,
    • activate and deactivate realisations,
    • activate and deactivate observations,
    • adjust the errors of the observations,
  • find and evaluate multiple posteriors based on the different sets of "observation filters" created above
  • compare the different posteriors

Documentation:

Libecl

  1. Here is some documentation of libecl Python code.

  2. Here here is some example code of creating api documentation with sphinx

ERT

In [this] (https://github.com/Statoil/ert/tree/master/docs) directory the rst/ folder is where the new documentation has been added, the other subfolders here contain various old documentation. Some of that is certainly valuable and should be reused in the new manual in rst/manual

Rework time concept in ERT

There are many aspects to this.

  • Completely remove pseudo time for parameters,
  • Remove dynamic behavior of GEN_DATA,
  • For summary vectors; in most situations ignore the fact that they are time dependent and consider them a vector.

When the above is executed, the data type enkf_node can be oblivious to time. This would remove the last traces of the original enkf algorithm.


Another action would be to interpolate simulated data for the observed time steps. This implies that it is not necessary to have a globally decided set of time steps for all of the simulated data.

Ease the process of restart failing forward models

Either dump the failed the realizations to a file/in terminal on the format the GUI expects to restart those exact realizations, or have the GUI itself support to relaunch all failed realizations from the previous run.

Plotting errors:

There are many ValueErrors when creating plots.

The following procedure results in ValueError exceptions:

  1. Start gui_test
  2. press Start Simulation button (single realization test run)
  3. select Yes
  4. Wait until simulation is completed
  5. press Create Plot button

As well a printing a curve, multiple ValueError messages are printed. Also selecting different plots results in the same curve( FGOR plot = TIME plot):

Traceback (most recent call last):
File "/private/stefos/ert/ert/build/lib/python2.7/site-packages/ert_gui/tools/plot/data_type_keys_list_model.py", line 46, in data
if self.keyManager().isKeyWithObservations(item):
File "/private/stefos/ert/install/lib/python2.7/site-packages/res/enkf/key_manager.py", line 157, in isKeyWithObservations
return key in self.allDataTypeKeysWithObservations()
File "/private/stefos/ert/install/lib/python2.7/site-packages/res/enkf/key_manager.py", line 151, in allDataTypeKeysWithObservations
self.__all_keys_with_observations = self.summaryKeysWithObservations() + self.genDataKeysWithObservations()
File "/private/stefos/ert/install/lib/python2.7/site-packages/res/enkf/key_manager.py", line 115, in genDataKeysWithObservations
report_step = obs_vector.activeStep()
File "/private/stefos/ert/install/lib/python2.7/site-packages/res/enkf/observations/obs_vector.py", line 113, in activeStep
raise ValueError("The activeStep() method can ONLY be called for obervations with one active step")

Set "ITER" from workflow

Currently it is possible to set the iter number from the workflow RUN_SMOOTHER_WITH_ITER. It would be very good to have it accessible either as a workflow job "SET_ITER" or "ENSEMBLE_RUN_WITH_ITER".

I will use it to test and screen different observations sets and scalings without overwriting the runpath everytime (without explicitt chaning the runpath). I can also think of other applications, and think it will be useful for others.

ert_tcp_server and ertshell are not installed

One of these applications are needed in order to start the ert server that will allow sending batch simulation requests to ert. These applications seem to be missing form the current deployment.
Also it would be great if these will not be linked to ert_gui.
I was told that this might not have too large priority but I want to create the issue just in case... so you know that there is at least somebody that thinks they are useful.

Issue with ert and libres/libecl 2.3

Trying ert with latest libres/libecl gives the following exceptions

Traceback (most recent call last):
  File "$KOMODO/ert_gui/tools/plot/data_type_keys_list_model.py", line 46, in data
    if self.keyManager().isKeyWithObservations(item):
  File "$KOMODO/res/enkf/key_manager.py", line 157, in isKeyWithObservations
    return key in self.allDataTypeKeysWithObservations()
  File "$KOMODO/res/enkf/key_manager.py", line 151, in allDataTypeKeysWithObservations
    self.__all_keys_with_observations = self.summaryKeysWithObservations() + self.genDataKeysWithObservations()
  File "$KOMODO/res/enkf/key_manager.py", line 115, in genDataKeysWithObservations
    report_step = obs_vector.activeStep()
  File "$KOMODO/res/enkf/observations/obs_vector.py", line 113, in activeStep
    raise ValueError("The activeStep() method can *ONLY* be called for obervations with one active step")
ValueError: The activeStep() method can *ONLY* be called for obervations with one active step



--------------------------------------------------------------------------------
  File "$KOMODO/ert_gui/tools/plot/plot_widget.py", line 85, in updatePlot
    self._plotFunction(plot_context)
  File "$KOMODO/ert_gui/plottery/plots/ensemble.py", line 33, in plotEnsemble
    plotObservations(plot_context, axes)
  File "$KOMODO/ert_gui/plottery/plots/observations.py", line 12, in plotObservations
    observation_data = data_gatherer.gatherObservationData(ert, case_list[0], key)
  File "$KOMODO/ert_gui/plottery/plot_data_gatherer.py", line 52, in gatherObservationData
    return self._observationGatherFunction(ert, case, key)
  File "$KOMODO/ert_gui/plottery/plot_data_gatherer.py", line 126, in gatherSummaryObservationData
    if ert.getKeyManager().isKeyWithObservations(key):
  File "$KOMODO/res/enkf/key_manager.py", line 157, in isKeyWithObservations
    return key in self.allDataTypeKeysWithObservations()
  File "$KOMODO/res/enkf/key_manager.py", line 151, in allDataTypeKeysWithObservations
    self.__all_keys_with_observations = self.summaryKeysWithObservations() + self.genDataKeysWithObservations()
  File "$KOMODO/res/enkf/key_manager.py", line 115, in genDataKeysWithObservations
    report_step = obs_vector.activeStep()
  File "$KOMODO/res/enkf/observations/obs_vector.py", line 113, in activeStep
    raise ValueError("The activeStep() method can *ONLY* be called for obervations with one active step")
Exception type: ValueError

$KOMODO/ert_gui/tools/plot/plot_widget.py:92: DeprecationWarning:
     BaseException.message has been deprecated as of Python 2.6
  sys.stderr.write("%s\n" % e.message)
The activeStep() method can *ONLY* be called for obervations with one active step
--------------------------------------------------------------------------------
An error occurred during plotting. This stack trace is helpful for diagnosing the problem.

Traceback (most recent call last):
  File "$KOMODO/ert_gui/tools/plot/data_type_keys_list_model.py", line 46, in data
    if self.keyManager().isKeyWithObservations(item):
  File "$KOMODO/res/enkf/key_manager.py", line 157, in isKeyWithObservations
    return key in self.allDataTypeKeysWithObservations()
  File "$KOMODO/res/enkf/key_manager.py", line 151, in allDataTypeKeysWithObservations
    self.__all_keys_with_observations = self.summaryKeysWithObservations() + self.genDataKeysWithObservations()
  File "$KOMODO/res/enkf/key_manager.py", line 115, in genDataKeysWithObservations
    report_step = obs_vector.activeStep()
  File "$KOMODO/res/enkf/observations/obs_vector.py", line 113, in activeStep
    raise ValueError("The activeStep() method can *ONLY* be called for obervations with one active step")
ValueError: The activeStep() method can *ONLY* be called for obervations with one active step

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.