equinor/flownet

FlowNet - Data-Driven Reservoir Predictions

FlowNet: Data-Driven Reservoir Predictions

FlowNet aims at solving the following problems:

• Create data-driven reduced physics models - directly from the data
• Train the model
• Assure model predictiveness
• Use the models to efficiently optimize and make decisions

For documentation, see the GitHub pages for this repository.

Contributing

Please check out our contribution guidelines if you want to contribute to FlowNet.

Installation

FlowNet is a Python package. All required dependencies are automatically installed together with FlowNet, except for the OPM-Flow reservoir simulator binaries which you will need to install separately.

If your Flow installation is not located at /usr/bin/flow you should set an environment variable FLOW_PATH with path to your Flow executable prior to running FlowNet.

Install FlowNet

The easiest and recommended approach is to install FlowNet from PyPI by running

pip install flownet

If you want to install and try out the latest unreleased code you can do

git clone [email protected]:equinor/flownet.git
cd flownet
pip install -e .

Omit the -e flag if you want a standard installation.

⚠️ Do you want to run FlowNet through the LSF queue? To be able to have the ERT process, that will be called by FlowNet, run jobs via LSF correctly you will need to update your default shell's configuration file (.cshrc or .bashrc) to automatically source your virtual environment.

Running FlowNet

You can run FlowNet as a single command line:

flownet ahm ./some_config.yaml ./some_output_folder

Run flownet --help to see all possible command line argument options.

Running webviz to check results

Before running webviz for the first time on your machine, you will need to to create a localhost https certificate by doing:

webviz certificate --auto-install --force

License

FlowNet is, with a few exceptions listed below, GPLv3.

• The Norne test data is available under the Open Database License
• The FlowNet logo is CC BY-NC-ND 4.0

flownet's Issues

Automatic stopping of HM

Moved from the previous repository (author @wouterjdb ):

It's hard to guess from the start how many iterations you need. After we a metric describing the training and validation phases, we should be able to create an automatic stopping algorithm:

Continue the history matching iterations as long as both the training and validation metrics show improvements. The moment that the validation metric performs worse than in the previous iterations (over fitting) one should stop.

Missing correlation between permeability and porosity

Author: @wouterjdb

In the current FlowNet setup we assign permeability and porosity fully independently. This might be improved by using poro/perm relations.

Investigate failed runs in cloud. Print to console?

Incorrect removal of aquifer NNC's?

In the "global" aquifer scheme, aquifer nodes visually lie on top of one another but were for some historical reason not connected with NNC's. The code below is responsible for that, but seems to also remove NNC's for aquifers in the "individual" scheme:

flownet/src/flownet/network_model/_network_model.py

Lines 145 to 151 in e73fd20

	# Remove connections to aquifers as they should not be connected with
	# NNCs, even if they visually overlap in the global scheme.
	end_df = self._df_entity_connections.loc[
	self._df_entity_connections["end_entity"] != "aquifer"
	][["xend", "yend", "zend"]].rename(
	columns={"xend": "x", "yend": "y", "zend": "z"}
	)

Could this potentially lead to aquifers that are not directly connected to a well to not be connected to the grid at all?

Get FlowNet on Komodo bleeding

Build script issues on Ubuntu 18.04 LTS

The following issues were identified when doing a fresh install of FlowNet on an Ubuntu 18.04 LTS machine:

• The installation stops and informs the user that Python3 venv needs to be installed:

Building a virtual environment at /home/media-unix/flownet/venv
The virtual environment was not created successfully because ensurepip is not
available.  On Debian/Ubuntu systems, you need to install the python3-venv
package using the following command.

    apt-get install python3-venv

You may need to use sudo with that command.  After installing the python3-venv
package, recreate your virtual environment.

Failing command: ['/home/media-unix/flownet/venv/bin/python3', '-Im', 'ensurepip', '--upgrade', '--default-pip']

Running the suggested command, with sudo, solves the issue.

• When running a test AHM case FlowNet stops when ERT is launched, with the following error:
  ImportError: Failed to import any qt binding
  This requires that PyQt5 is installed. However, if one runs pip install PyQt5 the following error is returned:

(venv) media-unix@media-unix:~/flownet/examples$ pip install PyQt5
Collecting PyQt5
  Downloading https://files.pythonhosted.org/packages/3a/fb/eb51731f2dc7c22d8e1a63ba88fb702727b324c6352183a32f27f73b8116/PyQt5-5.14.1.tar.gz (3.2MB)
    100% |████████████████████████████████| 3.2MB 248kB/s 
    Complete output from command python setup.py egg_info:
    Traceback (most recent call last):
      File "<string>", line 1, in <module>
      File "/usr/lib/python3.6/tokenize.py", line 452, in open
        buffer = _builtin_open(filename, 'rb')
    FileNotFoundError: [Errno 2] No such file or directory: '/tmp/pip-build-xmx1z3c3/PyQt5/setup.py'
    
    ----------------------------------------
Command "python setup.py egg_info" failed with error code 1 in /tmp/pip-build-xmx1z3c3/PyQt5/

This can be solved by installing a different version, for example, pip install pyqt5==5.14.0 works fine.

After fixing those two issues, FlowNet ran fine.

Move Norne example out of FlowNet repo, and to separate test data repository

At the same time, try to remove the norne.tar.gz file from repository history - in order to dramatically reduce repository size.

Phases in simulation model

Original author: @olelod

Currently the phases of the model are extracted from the relative permeability given.

    if self._swof and self._sgof:
        str_runspec_section = "OIL\nWATER\nGAS\n"
    elif self._swof:
        str_runspec_section = "OIL\nWATER\n"
    elif self._sgof:
        str_runspec_section = "OIL\nGAS\n"

    return {
        "RUNSPEC": str_runspec_section,
        "REGIONS": write_grdecl_file(merged_df_satnum, "SATNUM", int_type=True),
        "PROPS": str_props_section,
    }

Should we reverse this and make the user provide which phases are present in the config yaml? This would make it easier to check if the input needed is provided (relperm, contacts etc.).

Error message when determining the convex hull

The following error message (1) has something to do with flownet not being able to determine the convex hull in a particular case. Flownet needs at least 5 wells for the algorithm to be able to work. The connections need to define a 3D volume.

The model I'm working with has exactly the same depth for all wells.

A temporal solution to do a test is to change the perforation strategy from bottom_point to multiple that would mean each connected block in the Eclipse model with getting a connection in FlowNet.

This may be not what you want in the long run, but it is a simple test to see if you get further.

(1) Error message:

(venv) manuel@manuel:~/repos/projects/DataDrivenModels/flownet$ flownet ahm tests/configs/egg_parameters.yml egg_output_folder 
Adding flow nodes:  0%Traceback (most recent call last):
  File "/home/manuel/repos/projects/DataDrivenModels/flownet/venv/bin/flownet", line 11, in <module>
    load_entry_point('flownet', 'console_scripts', 'flownet')()
  File "/home/manuel/repos/projects/DataDrivenModels/flownet/src/flownet/_command_line.py", line 206, in main
    args.func(args)
  File "/home/manuel/repos/projects/DataDrivenModels/flownet/src/flownet/_command_line.py", line 67, in flownet_ahm
    run_flownet_history_mathing(config, args)
  File "/home/manuel/repos/projects/DataDrivenModels/flownet/src/flownet/ahm/_run_ahm.py", line 162, in run_flownet_history_mathing
    df_connections: pd.DataFrame = create_connections(df_coordinates, config)
  File "/home/manuel/repos/projects/DataDrivenModels/flownet/src/flownet/network_model/_generate_connections.py", line 451, in create_connections
    starts, ends = _generate_connections(df_coordinates, configuration)
  File "/home/manuel/repos/projects/DataDrivenModels/flownet/src/flownet/network_model/_generate_connections.py", line 125, in _generate_connections
    random_seed=configuration.flownet.random_seed,
  File "/home/manuel/repos/projects/DataDrivenModels/flownet/src/flownet/network_model/_mitchell.py", line 88, in mitchell_best_candidate_modified_3d
    hull = Delaunay(np.column_stack([x, y, z]))
  File "qhull.pyx", line 1839, in scipy.spatial.qhull.Delaunay.__init__
  File "qhull.pyx", line 357, in scipy.spatial.qhull._Qhull.__init__
scipy.spatial.qhull.QhullError: QH6154 Qhull precision error: Initial simplex is flat (facet 1 is coplanar with the interior point)

While executing:  | qhull d Qc Qbb Q12 Qt Qz
Options selected for Qhull 2019.1.r 2019/06/21:
  run-id 1237336110  delaunay  Qcoplanar-keep  Qbbound-last  Q12-allow-wide
  Qtriangulate  Qz-infinity-point  _pre-merge  _zero-centrum  Qinterior-keep
  Pgood  _max-width 4.4e+02  Error-roundoff 8.1e-12  _one-merge 7.3e-11
  Visible-distance 4.9e-11  U-max-coplanar 4.9e-11  Width-outside 9.7e-11
  _wide-facet 2.9e-10  _maxoutside 9.7e-11

precision problems (corrected unless 'Q0' or an error)
      9 zero divisors during gaussian elimination

The input to qhull appears to be less than 4 dimensional, or a
computation has overflowed.

Qhull could not construct a clearly convex simplex from points:
- p5(v5):    60    68 4e+03     0
- p6(v4): 2.5e+02    12 4e+03 1.2e+02
- p0(v3):    36 4.5e+02 4e+03 4.3e+02
- p7(v2): 4.5e+02    44 4e+03 4.3e+02
- p2(v1):    12 2.8e+02 4e+03 1.5e+02

The center point is coplanar with a facet, or a vertex is coplanar
with a neighboring facet.  The maximum round off error for
computing distances is 8.1e-12.  The center point, facets and distances
to the center point are as follows:

center point    162.4    170.4     4026    223.6

facet p6 p0 p7 p2 distance=    0
facet p5 p0 p7 p2 distance=    0
facet p5 p6 p7 p2 distance=    0
facet p5 p6 p0 p2 distance=    0
facet p5 p6 p0 p7 distance=    0

These points either have a maximum or minimum x-coordinate, or
they maximize the determinant for k coordinates.  Trial points
are first selected from points that maximize a coordinate.

The min and max coordinates for each dimension are:
  0:        12       452  difference=  440
  1:        12       452  difference=  440
  2:      4026      4026  difference=    0
  3:         0      4026  difference= 4026

If the input should be full dimensional, you have several options that
may determine an initial simplex:
  - use 'QJ'  to joggle the input and make it full dimensional
  - use 'QbB' to scale the points to the unit cube
  - use 'QR0' to randomly rotate the input for different maximum points
  - use 'Qs'  to search all points for the initial simplex
  - use 'En'  to specify a maximum roundoff error less than 8.1e-12.
  - trace execution with 'T3' to see the determinant for each point.

If the input is lower dimensional:
  - use 'QJ' to joggle the input and make it full dimensional
  - use 'Qbk:0Bk:0' to delete coordinate k from the input.  You should
    pick the coordinate with the least range.  The hull will have the
    correct topology.
  - determine the flat containing the points, rotate the points
    into a coordinate plane, and delete the other coordinates.
  - add one or more points to make the input full dimensional.

Make separate WELSPECS, COMPDAT and SCHEDULE include files

Original issue author: @edubarrosTNO

Current situation

• At the moment, the well heads (WELSPECS), well completions (COMPDAT), and well controls (WCONPROD, WCONINJE, etc.) are all prescribed in a single file.

• WELSPECS and COMPDAT are defined at the dates in which the wells are opened, along with their well controls.

Desired situation
In preparation for the use of FlowNet for well control optimization (e.g., injection rate optimization), I propose that these keywords are put in separate files.

We should have WELSPECS.inc and COMPDAT.inc containing information for all wells (even if they are shut at the beginning of simulation) included directly in the .DATA-file (in SCHEDULE section) and a separate SCHEDULE.inc file with well controls and dates.

This will make our life easier when optimization starts varying well controls, as a single file will need to be changed. This will also improve readability of the schedule file.

ValueError when not all relative permeability values are supplied

In cases where there is not both oil and gas present the config parser, after #88, now results in:

ValueError: The configuration is not valid:Missing key: training_set_end_date, Missing key: min, Missing key: max, Missing key: min, Missing key: max, Missing key: min, Missing key: max, Missing key: min, Missing key: max, Missing key: min, Missing key: max, Missing key: min, Missing key: max

The MK.AllowNone argument will need to be passed in those cases.

"logunif" in configuration file should be "loguniform"

Running the norne parameters file will crash on the following line:

flownet/examples/norne_parameters.yml

Line 46 in e73fd20

logunif: True

This should be:

loguniform: true

FlowNet models run using Eclipse cannot be visualized with ResInsight

Author: @wouterjdb

When I simulate FlowNet models with Eclipse and I visualize them with ResInsight I'm getting the following:

Running the same model with Flow results in the following vizualization in ResInsight:

NB. The DX parameter, which is calculated by the simulator, is completely off in the Eclipse case.

Change name "Eclipse" to "OPM Flow"

Author: @wouterjdb

We aim at using Flow - but we still call the simulator "Eclipse" in many places.

Production well BR-P-20 not shutting in

While other producing wells are correctly shutting in at high WCT, this particular well (the final production well) does not shutin in in FLOWNET.

Currently metric only

Moved from the previous repository (author @wouterjdb ):

The FlowNet code bases currently assumes metric units at multiple locations:

• Input Eclipse files should be metric
• Config is defined in metric units
• Constants are defined for metric units

It is not common for everyone to use metric units - it would be good to allow for field units as well.

Remove Python venv building script

This is currently blocked by the following upstream dependencies:

• libecl
• libres
• opm

Error in flownet_save_iteration_analytics in CI/CD build

In a successful github workflow run one finds the following error in the logs:

The script 'ExternalErtScript' caused an error while running:

Simulations completed.
Traceback (most recent call last):
  File "/home/runner/work/flownet/flownet/flownet_venv/lib/python3.7/site-packages/pandas/core/indexes/base.py", line 2897, in get_loc
    return self._engine.get_loc(key)
  File "pandas/_libs/index.pyx", line 107, in pandas._libs.index.IndexEngine.get_loc
  File "pandas/_libs/index.pyx", line 131, in pandas._libs.index.IndexEngine.get_loc
  File "pandas/_libs/hashtable_class_helper.pxi", line 1607, in pandas._libs.hashtable.PyObjectHashTable.get_item
  File "pandas/_libs/hashtable_class_helper.pxi", line 1614, in pandas._libs.hashtable.PyObjectHashTable.get_item
KeyError: 'DATE'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/runner/work/flownet/flownet/flownet_venv/bin/flownet_save_iteration_analytics", line 8, in <module>
    sys.exit(save_iteration_analytics())
  File "/home/runner/work/flownet/flownet/flownet_venv/lib/python3.7/site-packages/flownet/ahm/_ahm_iteration_analytics.py", line 420, in save_iteration_analytics
    df_sim = df_sim[df_sim["DATE"].isin(df_obs["DATE"])]
  File "/home/runner/work/flownet/flownet/flownet_venv/lib/python3.7/site-packages/pandas/core/frame.py", line 2995, in __getitem__
    indexer = self.columns.get_loc(key)
  File "/home/runner/work/flownet/flownet/flownet_venv/lib/python3.7/site-packages/pandas/core/indexes/base.py", line 2899, in get_loc
    return self._engine.get_loc(self._maybe_cast_indexer(key))
  File "pandas/_libs/index.pyx", line 107, in pandas._libs.index.IndexEngine.get_loc
  File "pandas/_libs/index.pyx", line 131, in pandas._libs.index.IndexEngine.get_loc
  File "pandas/_libs/hashtable_class_helper.pxi", line 1607, in pandas._libs.hashtable.PyObjectHashTable.get_item
  File "pandas/_libs/hashtable_class_helper.pxi", line 1614, in pandas._libs.hashtable.PyObjectHashTable.get_item
KeyError: 'DATE'

The DATE key is missing in a pandas dataframe. Could this be a result of recent changes in the time shifting /resampling that were done?

Add option for setting a constant value for a parameter

Original author: @wouterjdb

We're adding more and more parameters increasing the degrees of freedom. For some parameters it would maybe not always make sense to add them as an uncertain parameter, but a constant might be good enough. This is currently not supported.

I suggest that something like this would be allowed in the config:

model_parameters:
  permeability:
    min: 0.1
    max: 2000
  bulkvolume_mult:
    constant: 1

Build and deploy GitHub pages

...automatically (on merge to master?).

FlowNet accuracy metric generated using ERT workflows during AHM

Original author: @wouterjdb

We should include the accuracy metric in ERT using workflows during the AHM.

• run after the simulations in an iteration are finished
• output
  • the metrics ({RSME, MSE, ...} for various targets {WOPR, WWPR, WBHP, TOTAL, ...?}) in a single csv file that is appended to
  • a picture that is updated

We should also considering to include the heatmap and/or other useful analytical plots. @edubarrosTNO, which ones were most useful?

Please have a look at to see how you can add a workflow in the code base.

Sensitivity runs.

Put Norne model in flownet-testdata

Well status showing as 'STOP' in schedule file

This refers to the latest changes from PR #56 by @anders-kiaer and @olwijn .

I saw all the checks and reviewer's approval ✅ and took the liberty to merge the PR, because I wanted to try this fix as soon as possible. But there seems to be a bug: the wells are all showing 'STOP' status in the history schedule (both producers and injectors). See below for the Brugge case:

WCONHIST
-- WELL OPEN/ CNTL OIL  WAT  GAS  VFP   VFP  THP  BHP
-- NAME SHUT  MODE RATE RATE RATE TABLE ALFQ PRES PRES
  'BR-P-1' 'STOP' 'RESV' 317.9749 0.0051 0 1* 1* 63.7635 69.8381 /
  (...)
/
WCONINJH
-- WELL FLUID OPEN/ SURF RESV BHP  THP  VFP   NOT  CNTL
-- NAME TYPE  SHUT  RATE RATE PRES PRES TABLE USED MODE
  'BR-I-1' 'WATER' 'STOP' 635.9500 150.8748 0 1* 1* /
  (...)
/

Mistmatch for Egg model

I'm applying Flownet to the Egg model. You can see the results of the ensemble for 6th iteration. in the following figurre.

You can notice that the total oil rate and water flow rate never go closer to the history data. The ensembles are somehow stagnant. In this example, I'm running 40 ensembles and 100 additional nodes, however, the results are similar if I increase the number of ensembles, the number of nodes the range in the parameters.

No shutting in of wells in the Flownet (history) schedule

Production wells operating under an economic constraint may be shut in at e.g. high WCT. In the Flownet schedule file production wells are currently not explicitly shut in but simply disappear from the history schedule (i.e. no new WCONHIST entries appear after the shut-in date). This results in the well remaining open with the last assigned operating target. Shutting in of the well could be achieved by adding e.g. the schedule entry

WCONHIST
'BR-P-18' 'SHUT' 'RESV' /
/

at the correct shut-in time. It was verified that this has the desired effect.

It is not clear if the well status can be extracted from OPM Flow output as read by ERT. An alternative may be to implement a rule that says, for example, that if liquid rate and BHP are both 0 this means that the well is shut-in.

Support recent versions of configsuite

Add support for min, max, mean distribution definitions to all parameters

Original authour: @wouterjdb

FlowNet accuracy metric generated using ERT workflows during AHM

Original author: @wouterjdb

We should include the accuracy metric in ERT using workflows during the AHM.

• run after the simulations in an iteration are finished
• output
  • the metrics ({RSME, MSE, ...} for various targets {WOPR, WWPR, WBHP, TOTAL, ...?}) in a single csv file that is appended to
  • a picture that is updated

We should also considering to include the heatmap and/or other useful analytical plots. @edubarrosTNO, which ones were most useful?

Please have a look at to see how you can add a workflow in the code base.

Open source Brugge model

Get opm binding from officail PyPi when it exists (instead of test-PyPI)

Hard-coded config input for the analysis plots

The config parser has some hard coded lines,

flownet/src/flownet/config_parser/_config_parser.py

Lines 415 to 417 in a7dc9c0

	"quantity": "[WOPR:BR-P-]",
	"start": "2001-04-01",
	"end": "2006-01-01",

which is not very user friendly as this will not work for any other case than Brugge. These defaults should be somewhat more general.

@anders-kiaer

Installing flownet

I'm installing flownet in a new Ubuntu 20.04.

There a problem in build_enviroment.sh, I got the following error message

Collecting pip
  WARNING: Retrying (Retry(total=4, connect=None, read=None, redirect=None, status=None)) after connection broken by 'SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: EE certificate key too weak (_ssl.c:1108)'))': /packages/43/84/23ed6a1796480a6f1a2d38f2802901d078266bda38388954d01d3f2e821d/pip-20.1.1-py2.py3-none-any.whl
  WARNING: Retrying (Retry(total=3, connect=None, read=None, redirect=None, status=None)) after connection broken by 'SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: EE certificate key too weak (_ssl.c:1108)'))': /packages/43/84/23ed6a1796480a6f1a2d38f2802901d078266bda38388954d01d3f2e821d/pip-20.1.1-py2.py3-none-any.whl
  WARNING: Retrying (Retry(total=2, connect=None, read=None, redirect=None, status=None)) after connection broken by 'SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: EE certificate key too weak (_ssl.c:1108)'))': /packages/43/84/23ed6a1796480a6f1a2d38f2802901d078266bda38388954d01d3f2e821d/pip-20.1.1-py2.py3-none-any.whl
  WARNING: Retrying (Retry(total=1, connect=None, read=None, redirect=None, status=None)) after connection broken by 'SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: EE certificate key too weak (_ssl.c:1108)'))': /packages/43/84/23ed6a1796480a6f1a2d38f2802901d078266bda38388954d01d3f2e821d/pip-20.1.1-py2.py3-none-any.whl
  WARNING: Retrying (Retry(total=0, connect=None, read=None, redirect=None, status=None)) after connection broken by 'SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: EE certificate key too weak (_ssl.c:1108)'))': /packages/43/84/23ed6a1796480a6f1a2d38f2802901d078266bda38388954d01d3f2e821d/pip-20.1.1-py2.py3-none-any.whl
ERROR: Could not install packages due to an EnvironmentError: HTTPSConnectionPool(host='files.pythonhosted.org', port=443): Max retries exceeded with url: /packages/43/84/23ed6a1796480a6f1a2d38f2802901d078266bda38388954d01d3f2e821d/pip-20.1.1-py2.py3-none-any.whl (Caused by SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: EE certificate key too weak (_ssl.c:1108)')))

Every line with
pip install <package>
has to be modified by
pip install --trusted-host pypi.org --trusted-host files.pythonhosted.org <package>

This will solves the first error but there is a new problem in the pip calls inside ERT

Finished installing libres
Collecting git+https://github.com/equinor/ert@c74e1e6
  Cloning https://github.com/equinor/ert (to revision c74e1e6) to /tmp/pip-req-build-k17vq1n3
  Running command git clone -q https://github.com/equinor/ert /tmp/pip-req-build-k17vq1n3
  WARNING: Did not find branch or tag 'c74e1e6', assuming revision or ref.
  Running command git checkout -q c74e1e6
    ERROR: Command errored out with exit status 1:
     command: /home/manuel/repos/projects/Flownet/flownet/venv/bin/python3 -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/tmp/pip-req-build-k17vq1n3/setup.py'"'"'; __file__='"'"'/tmp/pip-req-build-k17vq1n3/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' egg_info --egg-base /tmp/pip-pip-egg-info-2jm5uwsd
         cwd: /tmp/pip-req-build-k17vq1n3/
    Complete output (113 lines):
      WARNING: Retrying (Retry(total=4, connect=None, read=None, redirect=None, status=None)) after connection broken by 'SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: EE certificate key too weak (_ssl.c:1108)'))': /packages/16/45/81b5262c0efc08882bdf183b788e6d28e3d684863990996d8b60967d48da/pytest_runner-5.2-py2.py3-none-any.whl
      WARNING: Retrying (Retry(total=3, connect=None, read=None, redirect=None, status=None)) after connection broken by 'SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: EE certificate key too weak (_ssl.c:1108)'))': /packages/16/45/81b5262c0efc08882bdf183b788e6d28e3d684863990996d8b60967d48da/pytest_runner-5.2-py2.py3-none-any.whl
      WARNING: Retrying (Retry(total=2, connect=None, read=None, redirect=None, status=None)) after connection broken by 'SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: EE certificate key too weak (_ssl.c:1108)'))': /packages/16/45/81b5262c0efc08882bdf183b788e6d28e3d684863990996d8b60967d48da/pytest_runner-5.2-py2.py3-none-any.whl
      WARNING: Retrying (Retry(total=1, connect=None, read=None, redirect=None, status=None)) after connection broken by 'SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: EE certificate key too weak (_ssl.c:1108)'))': /packages/16/45/81b5262c0efc08882bdf183b788e6d28e3d684863990996d8b60967d48da/pytest_runner-5.2-py2.py3-none-any.whl
      WARNING: Retrying (Retry(total=0, connect=None, read=None, redirect=None, status=None)) after connection broken by 'SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: EE certificate key too weak (_ssl.c:1108)'))': /packages/16/45/81b5262c0efc08882bdf183b788e6d28e3d684863990996d8b60967d48da/pytest_runner-5.2-py2.py3-none-any.whl
    ERROR: Exception:
    Traceback (most recent call last):
      File "/home/manuel/repos/projects/Flownet/flownet/venv/lib/python3.8/site-packages/pip/_vendor/urllib3/connectionpool.py", line 665, in urlopen
        httplib_response = self._make_request(
      File "/home/manuel/repos/projects/Flownet/flownet/venv/lib/python3.8/site-packages/pip/_vendor/urllib3/connectionpool.py", line 376, in _make_request
        self._validate_conn(conn)
      File "/home/manuel/repos/projects/Flownet/flownet/venv/lib/python3.8/site-packages/pip/_vendor/urllib3/connectionpool.py", line 994, in _validate_conn
        conn.connect()
      File "/home/manuel/repos/projects/Flownet/flownet/venv/lib/python3.8/site-packages/pip/_vendor/urllib3/connection.py", line 352, in connect
        self.sock = ssl_wrap_socket(
      File "/home/manuel/repos/projects/Flownet/flownet/venv/lib/python3.8/site-packages/pip/_vendor/urllib3/util/ssl_.py", line 370, in ssl_wrap_socket
        return context.wrap_socket(sock, server_hostname=server_hostname)
      File "/usr/lib/python3.8/ssl.py", line 500, in wrap_socket
        return self.sslsocket_class._create(
      File "/usr/lib/python3.8/ssl.py", line 1040, in _create
        self.do_handshake()
      File "/usr/lib/python3.8/ssl.py", line 1309, in do_handshake
        self._sslobj.do_handshake()
    ssl.SSLCertVerificationError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: EE certificate key too weak (_ssl.c:1108)
    
    During handling of the above exception, another exception occurred:
    
    Traceback (most recent call last):
      File "/home/manuel/repos/projects/Flownet/flownet/venv/lib/python3.8/site-packages/pip/_vendor/requests/adapters.py", line 439, in send
        resp = conn.urlopen(
      File "/home/manuel/repos/projects/Flownet/flownet/venv/lib/python3.8/site-packages/pip/_vendor/urllib3/connectionpool.py", line 747, in urlopen
        return self.urlopen(
      File "/home/manuel/repos/projects/Flownet/flownet/venv/lib/python3.8/site-packages/pip/_vendor/urllib3/connectionpool.py", line 747, in urlopen
        return self.urlopen(
      File "/home/manuel/repos/projects/Flownet/flownet/venv/lib/python3.8/site-packages/pip/_vendor/urllib3/connectionpool.py", line 747, in urlopen
        return self.urlopen(
      [Previous line repeated 2 more times]
      File "/home/manuel/repos/projects/Flownet/flownet/venv/lib/python3.8/site-packages/pip/_vendor/urllib3/connectionpool.py", line 719, in urlopen
        retries = retries.increment(
      File "/home/manuel/repos/projects/Flownet/flownet/venv/lib/python3.8/site-packages/pip/_vendor/urllib3/util/retry.py", line 436, in increment
        raise MaxRetryError(_pool, url, error or ResponseError(cause))
    pip._vendor.urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='files.pythonhosted.org', port=443): Max retries exceeded with url: /packages/16/45/81b5262c0efc08882bdf183b788e6d28e3d684863990996d8b60967d48da/pytest_runner-5.2-py2.py3-none-any.whl (Caused by SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: EE certificate key too weak (_ssl.c:1108)')))
    
    During handling of the above exception, another exception occurred:
    
    Traceback (most recent call last):
      File "/home/manuel/repos/projects/Flownet/flownet/venv/lib/python3.8/site-packages/pip/_internal/cli/base_command.py", line 188, in _main
        status = self.run(options, args)
      File "/home/manuel/repos/projects/Flownet/flownet/venv/lib/python3.8/site-packages/pip/_internal/cli/req_command.py", line 185, in wrapper
        return func(self, options, args)
      File "/home/manuel/repos/projects/Flownet/flownet/venv/lib/python3.8/site-packages/pip/_internal/commands/wheel.py", line 159, in run
        requirement_set = resolver.resolve(
      File "/home/manuel/repos/projects/Flownet/flownet/venv/lib/python3.8/site-packages/pip/_internal/resolution/legacy/resolver.py", line 179, in resolve
        discovered_reqs.extend(self._resolve_one(requirement_set, req))
      File "/home/manuel/repos/projects/Flownet/flownet/venv/lib/python3.8/site-packages/pip/_internal/resolution/legacy/resolver.py", line 362, in _resolve_one
        abstract_dist = self._get_abstract_dist_for(req_to_install)
      File "/home/manuel/repos/projects/Flownet/flownet/venv/lib/python3.8/site-packages/pip/_internal/resolution/legacy/resolver.py", line 314, in _get_abstract_dist_for
        abstract_dist = self.preparer.prepare_linked_requirement(req)
      File "/home/manuel/repos/projects/Flownet/flownet/venv/lib/python3.8/site-packages/pip/_internal/operations/prepare.py", line 467, in prepare_linked_requirement
        local_file = unpack_url(
      File "/home/manuel/repos/projects/Flownet/flownet/venv/lib/python3.8/site-packages/pip/_internal/operations/prepare.py", line 255, in unpack_url
        file = get_http_url(
      File "/home/manuel/repos/projects/Flownet/flownet/venv/lib/python3.8/site-packages/pip/_internal/operations/prepare.py", line 129, in get_http_url
        from_path, content_type = _download_http_url(
      File "/home/manuel/repos/projects/Flownet/flownet/venv/lib/python3.8/site-packages/pip/_internal/operations/prepare.py", line 277, in _download_http_url
        download = downloader(link)
      File "/home/manuel/repos/projects/Flownet/flownet/venv/lib/python3.8/site-packages/pip/_internal/network/download.py", line 189, in __call__
        resp = _http_get_download(self._session, link)
      File "/home/manuel/repos/projects/Flownet/flownet/venv/lib/python3.8/site-packages/pip/_internal/network/download.py", line 135, in _http_get_download
        resp = session.get(
      File "/home/manuel/repos/projects/Flownet/flownet/venv/lib/python3.8/site-packages/pip/_vendor/requests/sessions.py", line 543, in get
        return self.request('GET', url, **kwargs)
      File "/home/manuel/repos/projects/Flownet/flownet/venv/lib/python3.8/site-packages/pip/_internal/network/session.py", line 421, in request
        return super(PipSession, self).request(method, url, *args, **kwargs)
      File "/home/manuel/repos/projects/Flownet/flownet/venv/lib/python3.8/site-packages/pip/_vendor/requests/sessions.py", line 530, in request
        resp = self.send(prep, **send_kwargs)
      File "/home/manuel/repos/projects/Flownet/flownet/venv/lib/python3.8/site-packages/pip/_vendor/requests/sessions.py", line 643, in send
        r = adapter.send(request, **kwargs)
      File "/home/manuel/repos/projects/Flownet/flownet/venv/lib/python3.8/site-packages/pip/_vendor/cachecontrol/adapter.py", line 53, in send
        resp = super(CacheControlAdapter, self).send(request, **kw)
      File "/home/manuel/repos/projects/Flownet/flownet/venv/lib/python3.8/site-packages/pip/_vendor/requests/adapters.py", line 514, in send
        raise SSLError(e, request=request)
    pip._vendor.requests.exceptions.SSLError: HTTPSConnectionPool(host='files.pythonhosted.org', port=443): Max retries exceeded with url: /packages/16/45/81b5262c0efc08882bdf183b788e6d28e3d684863990996d8b60967d48da/pytest_runner-5.2-py2.py3-none-any.whl (Caused by SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: EE certificate key too weak (_ssl.c:1108)')))
    Traceback (most recent call last):
      File "/home/manuel/repos/projects/Flownet/flownet/venv/lib/python3.8/site-packages/setuptools/installer.py", line 128, in fetch_build_egg
        subprocess.check_call(cmd)
      File "/usr/lib/python3.8/subprocess.py", line 364, in check_call
        raise CalledProcessError(retcode, cmd)
    subprocess.CalledProcessError: Command '['/home/manuel/repos/projects/Flownet/flownet/venv/bin/python3', '-m', 'pip', '--disable-pip-version-check', 'wheel', '--no-deps', '-w', '/tmp/tmpyupahj5v', '--quiet', 'pytest-runner']' returned non-zero exit status 2.
    
    During handling of the above exception, another exception occurred:
    
    Traceback (most recent call last):
      File "<string>", line 1, in <module>
      File "/tmp/pip-req-build-k17vq1n3/setup.py", line 18, in <module>
        setup(
      File "/home/manuel/repos/projects/Flownet/flownet/venv/lib/python3.8/site-packages/setuptools/__init__.py", line 144, in setup
        _install_setup_requires(attrs)
      File "/home/manuel/repos/projects/Flownet/flownet/venv/lib/python3.8/site-packages/setuptools/__init__.py", line 139, in _install_setup_requires
        dist.fetch_build_eggs(dist.setup_requires)
      File "/home/manuel/repos/projects/Flownet/flownet/venv/lib/python3.8/site-packages/setuptools/dist.py", line 718, in fetch_build_eggs
        resolved_dists = pkg_resources.working_set.resolve(
      File "/home/manuel/repos/projects/Flownet/flownet/venv/lib/python3.8/site-packages/pkg_resources/__init__.py", line 780, in resolve
        dist = best[req.key] = env.best_match(
      File "/home/manuel/repos/projects/Flownet/flownet/venv/lib/python3.8/site-packages/pkg_resources/__init__.py", line 1065, in best_match
        return self.obtain(req, installer)
      File "/home/manuel/repos/projects/Flownet/flownet/venv/lib/python3.8/site-packages/pkg_resources/__init__.py", line 1077, in obtain
        return installer(requirement)
      File "/home/manuel/repos/projects/Flownet/flownet/venv/lib/python3.8/site-packages/setuptools/dist.py", line 777, in fetch_build_egg
        return fetch_build_egg(self, req)
      File "/home/manuel/repos/projects/Flownet/flownet/venv/lib/python3.8/site-packages/setuptools/installer.py", line 130, in fetch_build_egg
        raise DistutilsError(str(e))
    distutils.errors.DistutilsError: Command '['/home/manuel/repos/projects/Flownet/flownet/venv/bin/python3', '-m', 'pip', '--disable-pip-version-check', 'wheel', '--no-deps', '-w', '/tmp/tmpyupahj5v', '--quiet', 'pytest-runner']' returned non-zero exit status 2.
    ----------------------------------------
ERROR: Command errored out with exit status 1: python setup.py egg_info Check the logs for full command output.

Put Brugge model in flownet-testdata

Support pandas~=1.0

More useful error output

When all/too many realizations fail you now get:

Simulations failed with the following error: Simulation failed! All realizations failed!
Traceback (most recent call last):
  File "/home/travis/virtualenv/python3.6.9/bin/flownet", line 8, in <module>
    sys.exit(main())
  File "/home/travis/virtualenv/python3.6.9/lib/python3.6/site-packages/flownet/_command_line.py", line 100, in main
    run_flownet_history_mathing(config, args)
  File "/home/travis/virtualenv/python3.6.9/lib/python3.6/site-packages/flownet/_run_ahm.py", line 306, in run_flownet_history_mathing
    ahm.run_ert(mode="es_mda", weights=config.ert.ensemble_weights)
  File "/home/travis/virtualenv/python3.6.9/lib/python3.6/site-packages/flownet/_assisted_history_matching.py", line 336, in run_ert
    check=True,
  File "/opt/python/3.6.9/lib/python3.6/subprocess.py", line 438, in run
    output=stdout, stderr=stderr)
subprocess.CalledProcessError: Command 'export PYTHONWARNINGS=ignore::DeprecationWarning;ert es_mda --weights '1' ahm_config.ert' returned non-zero exit status 1.
The command "flownet ../tests/configs/norne_parameters.yml ./some_ert_setup" exited with 1.

Would be useful that the subprocess command does show the actual error message from ERT. And if that is not informative either, either change Flownet or ERT to show e.g. stderr from the job that is failing in the realizations.

wells status by default

Flownet set wells status "CLOSED" by default. I'm applying flownet for the egg model and all wells are set to be "CLOSED" by default. Maybe, flownets should warn the user when wells status is not defined in the original model.

Time shift in FlowNet results compared to original case

Original author: @wouterjdb

Is this an issue?

Profile FlowNet performance. Bottlenecks?

Network generation for Egg model

This figures shows the network generated by Flownet for the Egg reservoir model.

Aditional to the connections Injector - Producer, you will connections producer-producer and also connections Injector - Injectors. I understand these connections are generated by default. However, there are connections that should no be there (in my opinion). There is a connection, for example, the one between injector 2 and injector 6 that i don't understand. Both wells are far away from each other.

It would be interesting to include in the configuration file an option to influence the creation of the network in flownet.

Configuration file: Aquifer and rock compresibility section.

The aquifer section is set to be "not required" in the configuration file. However, flownet assumes that the section is always defined and it comes into an error when the aquifer is not defined.

Rock compressibility section it is defined to be "required", however. It is necessary to be always defined?

Use posterior ensemble mean for new prior

Practical experience shows that history matching results may still contain biases. A two-step solution approach was proposed by Guo and Reynolds (2019) to address this, in which the posterior ensemble means are used as the means for a new prior in a second application of the history matching workflow.

If the ensemble of parameter estimates is saved in a compact form at the end of the history matching, new functionality could be introduced to specify that the mean of the new prior should be taken from this file.

Config should support absolute path

Author: @wouterjdb

Currently the _to_abs_path-function does not support absolute paths.

Very slow CI github workflow runs

Currently it takes about 1-2 hrs to perform a single github workflow run - this slows down development and needs to be addressed. Both running the tests as well as performing the full norne run takes a really long time:

Put Egg model in flownet-testdata

Same parameter values across ensemble of realizations in prediction mode

When using FlowNet in prediction mode, I observed that the parameters of the realizations of the ensemble (at least the grid parameters - porosity, permeability, etc.) show the same values.

I think something is going wrong when loading the parameters parquet file and writing the (grid) parameter include files. Or maybe something is wrong in history matching mode when saving the parameters into the parquet file.

Training fraction and end date allowed at the same time in config

Original authour: @wouterjdb

If you set both the training set fraction and end date you'll get the following warning when running FlowNet:

Training set fraction and training set end date are both defined in config file.
The input given for training set fraction will be ignored.
The training set end date will be used to calculate the training set fraction.

I'd actually suggest to let the config parser handle this and stop execution early on. A warning during execution is a bit 'too silent' if you ask me.

Get results of Norne HM online

Implement more tests for pytest coverage to increase from 50 to ...?

Original author: @wouterjdb

Save iteration parameters process not called properly in AHM

The script 'ExternalErtScript' caused an error while running:

usage: Save iteration parameters to a file. [-h]
Simulations completed.
                                            config runpath eclbase yamlobs
                                            start end quantity metrics outfile
Save iteration parameters to a file.: error: the following arguments are required: outfile
The script 'ExternalErtScript' caused an error while running:
usage: Save iteration parameters to a file. [-h]
                                            config runpath eclbase yamlobs
                                            start end quantity metrics outfile
Save iteration parameters to a file.: error: the following arguments are required: outfile