kbarnhart/calibration_with_dakota_clinic

Calibration with Dakota

Under development

About

This repository contains resources used in a clinic on using Dakota for calibration taught by @kbarnhart. If you have any questions, comments, or problems, please make an issue.

Options for running the examples.

1. Use Binder.
2. Use Hydroshare.
3. Install the repo and Dakota yourself.

Install instructions

Make sure you had git and the Anaconda python distribution (recommended version 3.6 or above).

Step 1: Get Dakota and this repository

Option A: Full env with conda.

This option is only available on Linux and on MacOSX. It will not work on Windows.

(1) Open a terminal and navigate to a folder where you want this folder copied.

Then execute the following commands to download the repository, create the specified conda environment, and build/test the heat module. Note that you'll have to replace YOUR_OS_HERE with either osx or linux.

$ git clone https://github.com/kbarnhart/calibration_with_dakota_clinic.git
$ cd calibration_with_dakota_clinic
$ conda env create -f environment_YOUR_OS_HERE.yml
$ conda activate dakota_clinic
$ make install

If you've never made an environment with conda before, you might need to use source instead of conda in the last command. If that command fails, conda will give you an error message that is reasonably helpful. Basically, it will instruct you to run

$ conda init <NAME OF SHELL>

which will make the the conda activate dakota_clinic command will work.

If you want to test your install, type

pytest

This may take ~15 minutes (you are running all of the experiments we will do in the clinic and testing that they work).

Option B: Dakota Binary + conda env

Step 1: Install the Dakota binary for Windows and make a conda environment based on the file environment_everything_but_dakota.yml.

After downloading/installing the Dakota binary, open a terminal and do the following:

$ git clone https://github.com/kbarnhart/calibration_with_dakota_clinic.git
$ conda env create -f environment_everything_but_dakota.yml
$ conda activate dakota_clinic
$ make install

Step 2: Open a notebook.

Assuming that the results of pytest don't include any failures, open Jupyter Notebooks.

$ jupyter notebook

Click Clinic_Notes_and_Tutorial.ipynb.

calibration_with_dakota_clinic's Issues

pytest fails - dataframe mismatch?

@kbarnhart I installed Dakota and the repo on MacOS, and then created an environment as described in the install instructions. The stuff below was printed after only a few seconds running pytest.

(dakota_clinic) mm77:calibration_with_dakota_clinic njlyons$ pytest
======================================= test session starts ========================================
platform darwin -- Python 3.7.3, pytest-4.5.0, py-1.8.0, pluggy-0.11.0
rootdir: /Users/njlyons/Code/calibration_with_dakota_clinic, inifile: setup.cfg, testpaths: heat, tests
collected 6 items

heat/heat.py .. [ 33%]
tests/test_all_notebooks.py F [ 50%]
tests/test_frame_equal.py FFF [100%]

============================================= FAILURES ==============================================
_ test_notebook[/Users/njlyons/Code/calibration_with_dakota_clinic/Clinic_Notes_and_Tutorial.ipynb] _
Traceback (most recent call last):
File "/Users/njlyons/Code/calibration_with_dakota_clinic/tests/test_all_notebooks.py", line 65, in test_notebook
nb, errors = _notebook_run(notebook)
File "/Users/njlyons/Code/calibration_with_dakota_clinic/tests/test_all_notebooks.py", line 48, in _notebook_run
subprocess.check_call(args)
File "/Users/njlyons/anaconda3/envs/dakota_clinic/lib/python3.7/subprocess.py", line 347, in check_call
raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command '['jupyter', 'nbconvert', '--to', 'notebook', '--execute', '--ExecutePreprocessor.kernel_name=python', '--ExecutePreprocessor.timeout=None', '--output', '/var/folders/f4/hg6s0z510jx0kwbqsfhbc62c0000gn/T/tmp3hetj8lc.ipynb', '--output-dir=.', '/Users/njlyons/Code/calibration_with_dakota_clinic/Clinic_Notes_and_Tutorial.ipynb']' returned non-zero exit status 1.
--------------------------------------- Captured stderr call ----------------------------------------
[NbConvertApp] Converting notebook /Users/njlyons/Code/calibration_with_dakota_clinic/Clinic_Notes_and_Tutorial.ipynb to notebook
[NbConvertApp] Executing notebook with kernel: python

Error reading restart file 'dakota_02_nl2sol.rst' (empty or corrupt file).
Details (Boost archive exception): input stream error

Error reading restart file 'dakota_03_ego.rst' (empty or corrupt file).
Details (Boost archive exception): input stream error
[NbConvertApp] ERROR | Error while converting '/Users/njlyons/Code/calibration_with_dakota_clinic/Clinic_Notes_and_Tutorial.ipynb'
Traceback (most recent call last):
File "/Users/njlyons/anaconda3/envs/dakota_clinic/lib/python3.7/site-packages/nbconvert/nbconvertapp.py", line 408, in export_single_notebook
output, resources = self.exporter.from_filename(notebook_filename, resources=resources)
File "/Users/njlyons/anaconda3/envs/dakota_clinic/lib/python3.7/site-packages/nbconvert/exporters/exporter.py", line 179, in from_filename
return self.from_file(f, resources=resources, **kw)
File "/Users/njlyons/anaconda3/envs/dakota_clinic/lib/python3.7/site-packages/nbconvert/exporters/exporter.py", line 197, in from_file
return self.from_notebook_node(nbformat.read(file_stream, as_version=4), resources=resources, **kw)
File "/Users/njlyons/anaconda3/envs/dakota_clinic/lib/python3.7/site-packages/nbconvert/exporters/notebook.py", line 32, in from_notebook_node
nb_copy, resources = super(NotebookExporter, self).from_notebook_node(nb, resources, **kw)
File "/Users/njlyons/anaconda3/envs/dakota_clinic/lib/python3.7/site-packages/nbconvert/exporters/exporter.py", line 139, in from_notebook_node
nb_copy, resources = self._preprocess(nb_copy, resources)
File "/Users/njlyons/anaconda3/envs/dakota_clinic/lib/python3.7/site-packages/nbconvert/exporters/exporter.py", line 316, in _preprocess
nbc, resc = preprocessor(nbc, resc)
File "/Users/njlyons/anaconda3/envs/dakota_clinic/lib/python3.7/site-packages/nbconvert/preprocessors/base.py", line 47, in call
return self.preprocess(nb, resources)
File "/Users/njlyons/anaconda3/envs/dakota_clinic/lib/python3.7/site-packages/nbconvert/preprocessors/execute.py", line 381, in preprocess
nb, resources = super(ExecutePreprocessor, self).preprocess(nb, resources)
File "/Users/njlyons/anaconda3/envs/dakota_clinic/lib/python3.7/site-packages/nbconvert/preprocessors/base.py", line 69, in preprocess
nb.cells[index], resources = self.preprocess_cell(cell, resources, index)
File "/Users/njlyons/anaconda3/envs/dakota_clinic/lib/python3.7/site-packages/nbconvert/preprocessors/execute.py", line 424, in preprocess_cell
raise CellExecutionError.from_cell_and_msg(cell, out)
nbconvert.preprocessors.execute.CellExecutionError: An error occurred while executing the following cell:

files = glob.glob(os.path.join(TOP_DIR, "analysis", "*.dat"))
dfs = []
for file in files:
df = pd.read_csv(file, engine="python", delim_whitespace=True)
df["method"] = file.split(".")[0].split("_")[-1]
dfs.append(df)

df = pd.concat(dfs, ignore_index=True)
method_cats = CategoricalDtype(categories=["grid", "nl2sol", "ego"], ordered=True)
df["method"] = df["method"].astype(method_cats)
df = df.set_index(["method", "deltaT", "duration_years"]).drop(columns=["interface"])

(
ggplot(df.reset_index(), aes(x="deltaT", y="duration_years", color="%eval_id"))
+ geom_point()
+ scale_color_cmap(name="jet")
+ facet_wrap("~method")
)

---

PlotnineError Traceback (most recent call last)
~/anaconda3/envs/dakota_clinic/lib/python3.7/site-packages/IPython/core/formatters.py in call(self, obj)
700 type_pprinters=self.type_printers,
701 deferred_pprinters=self.deferred_printers)
--> 702 printer.pretty(obj)
703 printer.flush()
704 return stream.getvalue()

~/anaconda3/envs/dakota_clinic/lib/python3.7/site-packages/IPython/lib/pretty.py in pretty(self, obj)
400 if cls is not object
401 and callable(cls.dict.get('repr')):
--> 402 return _repr_pprint(obj, self, cycle)
403
404 return _default_pprint(obj, self, cycle)

~/anaconda3/envs/dakota_clinic/lib/python3.7/site-packages/IPython/lib/pretty.py in repr_pprint(obj, p, cycle)
695 """A pprint that just redirects to the normal repr function."""
696 # Find newlines and replace them with p.break()
--> 697 output = repr(obj)
698 for idx,output_line in enumerate(output.splitlines()):
699 if idx:

~/anaconda3/envs/dakota_clinic/lib/python3.7/site-packages/plotnine/ggplot.py in repr(self)
93 # in the jupyter notebook.
94 if not self.figure:
---> 95 self.draw()
96 plt.show()
97 return '<ggplot: (%d)>' % self.hash()

~/anaconda3/envs/dakota_clinic/lib/python3.7/site-packages/plotnine/ggplot.py in draw(self, return_ggplot)
186 # new frames knowing that they are separate from the original.
187 with pd.option_context('mode.chained_assignment', None):
--> 188 return self._draw(return_ggplot)
189
190 def _draw(self, return_ggplot=False):

~/anaconda3/envs/dakota_clinic/lib/python3.7/site-packages/plotnine/ggplot.py in _draw(self, return_ggplot)
193 # assign a default theme
194 self = deepcopy(self)
--> 195 self._build()
196
197 # If no theme we use the default

~/anaconda3/envs/dakota_clinic/lib/python3.7/site-packages/plotnine/ggplot.py in _build(self)
286 # Initialise panels, add extra data for margins & missing
287 # facetting variables, and add on a PANEL variable to data
--> 288 layout.setup(layers, self)
289
290 # Compute aesthetics to produce data with generalised

~/anaconda3/envs/dakota_clinic/lib/python3.7/site-packages/plotnine/facets/layout.py in setup(self, layers, plot)
56 # Generate panel layout
57 data = self.facet.setup_data(data)
---> 58 self.layout = self.facet.compute_layout(data)
59 self.layout = self.coord.setup_layout(self.layout)
60 self.check_layout()

~/anaconda3/envs/dakota_clinic/lib/python3.7/site-packages/plotnine/facets/facet_wrap.py in compute_layout(self, data)
68
69 base = combine_vars(data, self.plot.environment,
---> 70 self.vars, drop=self.drop)
71 n = len(base)
72 dims = wrap_dims(n, self.nrow, self.ncol)

~/anaconda3/envs/dakota_clinic/lib/python3.7/site-packages/plotnine/facets/facet.py in combine_vars(data, environment, vars, drop)
602 if len(base) == 0:
603 raise PlotnineError(
--> 604 "Faceting variables must have at least one value")
605
606 base = base.reset_index(drop=True)

PlotnineError: 'Faceting variables must have at least one value'
PlotnineError: 'Faceting variables must have at least one value'

____________________________ test_dat_equiv[analysis/dakota_03_ego.dat] _____________________________
Traceback (most recent call last):
File "/Users/njlyons/Code/calibration_with_dakota_clinic/tests/test_frame_equal.py", line 20, in test_dat_equiv
check_less_precise=2)
File "/Users/njlyons/anaconda3/envs/dakota_clinic/lib/python3.7/site-packages/pandas/util/testing.py", line 1458, in assert_frame_equal
'{shape!r}'.format(shape=right.shape))
File "/Users/njlyons/anaconda3/envs/dakota_clinic/lib/python3.7/site-packages/pandas/util/testing.py", line 1094, in raise_assert_detail
raise AssertionError(msg)
AssertionError: DataFrame are different

DataFrame shape mismatch
[left]: (0, 5)
[right]: (63, 5)
____________________________ test_dat_equiv[analysis/dakota_01_grid.dat] ____________________________
Traceback (most recent call last):
File "/Users/njlyons/Code/calibration_with_dakota_clinic/tests/test_frame_equal.py", line 20, in test_dat_equiv
check_less_precise=2)
File "/Users/njlyons/anaconda3/envs/dakota_clinic/lib/python3.7/site-packages/pandas/util/testing.py", line 1458, in assert_frame_equal
'{shape!r}'.format(shape=right.shape))
File "/Users/njlyons/anaconda3/envs/dakota_clinic/lib/python3.7/site-packages/pandas/util/testing.py", line 1094, in raise_assert_detail
raise AssertionError(msg)
AssertionError: DataFrame are different

DataFrame shape mismatch
[left]: (0, 5)
[right]: (100, 5)
___________________________ test_dat_equiv[analysis/dakota_02_nl2sol.dat] ___________________________
Traceback (most recent call last):
File "/Users/njlyons/Code/calibration_with_dakota_clinic/tests/test_frame_equal.py", line 20, in test_dat_equiv
check_less_precise=2)
File "/Users/njlyons/anaconda3/envs/dakota_clinic/lib/python3.7/site-packages/pandas/util/testing.py", line 1458, in assert_frame_equal
'{shape!r}'.format(shape=right.shape))
File "/Users/njlyons/anaconda3/envs/dakota_clinic/lib/python3.7/site-packages/pandas/util/testing.py", line 1094, in raise_assert_detail
raise AssertionError(msg)
AssertionError: DataFrame are different

DataFrame shape mismatch
[left]: (0, 5)
[right]: (50, 5)
========================================= warnings summary ==========================================
/Users/njlyons/anaconda3/envs/dakota_clinic/lib/python3.7/site-packages/nbformat/notebooknode.py:4
/Users/njlyons/anaconda3/envs/dakota_clinic/lib/python3.7/site-packages/nbformat/notebooknode.py:4: DeprecationWarning: Using or importing the ABCs from 'collections' instead of from 'collections.abc' is deprecated, and in 3.8 it will stop working
from collections import Mapping

-- Docs: https://docs.pytest.org/en/latest/warnings.html
===================================== slowest 16 test durations =====================================
47.37s call tests/test_all_notebooks.py::test_notebook[/Users/njlyons/Code/calibration_with_dakota_clinic/Clinic_Notes_and_Tutorial.ipynb]
0.07s call heat/heat.py::heat.heat.solve_1d
0.02s call heat/heat.py::heat.heat.Heat
0.01s call tests/test_frame_equal.py::test_dat_equiv[analysis/dakota_03_ego.dat]
0.01s call tests/test_frame_equal.py::test_dat_equiv[analysis/dakota_02_nl2sol.dat]
0.01s call tests/test_frame_equal.py::test_dat_equiv[analysis/dakota_01_grid.dat]
0.01s setup tests/test_all_notebooks.py::test_notebook[/Users/njlyons/Code/calibration_with_dakota_clinic/Clinic_Notes_and_Tutorial.ipynb]

(0.00 durations hidden. Use -vv to show these durations.)
========================== 4 failed, 2 passed, 1 warnings in 48.53 seconds ==========================

cannot instantiate abstract class BmiHeat with abstract methods...

Hi Katy,

This is awesome! But I've hit a wall with running the notebook. The Dakota run launched by the shell script (ln 9) is failing for me. A little poking reveals this is a problem with the driver, as the Dakota .log records:

Traceback (most recent call last):
  File "driver.py", line 84, in <module>
    h = Heat()
TypeError: Can't instantiate abstract class BmiHeat with abstract methods get_grid_face_edges, get_input_item_count, get_output_item_count

Error: cannot open results file "MULTIDIM_PARAM/run.1/results.out" for evaluation 1

This is apparently an issue in the driver code. I checked I can import BmiHeat itself just fine, but if I try h = heat() as at line 84, I get that same TypeError Dakota is reporting:

In [2]: from heat import BmiHeat as heat                                        

In [3]: h = heat()                                                              
---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
<ipython-input-3-b4884c508061> in <module>
----> 1 h = heat()

TypeError: Can't instantiate abstract class BmiHeat with abstract methods get_grid_face_edges, get_input_item_count, get_output_item_count

I'm in the conda env created from environment_everything_but_dakota.yml, so should not be a local versioning issue.

I've never seen python complain about abstract methods before, so struggling to diagnose.