geoscienceaustralia/tcrm

A statistical-parametric model for assessing wind hazard from tropical cyclones

The Tropical Cyclone Risk Model

The Tropical Cyclone Risk Model is a stochastic tropical cyclone model developed by Geoscience Australia for estimating the wind hazard from tropical cyclones.

Due to the relatively short record of quality-controlled, consistent tropical cyclone observations, it is difficult to estimate average recurrence interval wind speeds ue to tropical cyclones. To overcome the restriction of observed data, TCRM uses an autoregressive model to generate thousands of years of events that are statistically similar to the historical record. To translate these events to estimated wind speeds, TCRM applies a parametric windfield and boundary layer model to each event. Finally an extreme value distribution is fitted to the aggregated windfields at each grid point in the model domain to provide ARI wind speed estimates.

Features

• Multi-platform: TCRM can run on desktop machines through to massively-parallel systems (tested on Windows XP/Vista/7, *NIX);
• Multiple options for wind field & boundary layer models: A number of radial profiles and simple boundary layer models have been included to allow users to test sensitivity to these options.
• Globally applicable: Users can set up a domain in any TC basin in the globe. The model is not tuned to any one region of the globe. Rather, the model is designed to draw sufficient information from best-track archives;
• Evaluation metrics: Offers capability to run objective evaluation of track model metrics (e.g. landfall rates);
• Single scenarios: Users can run a single TC event (e.g. using a b-deck format track file) at high temporal resolution and extract time series data at chosen locations;

Changelog

New features:

• Added empirical ARI calculation

Bug fixes:

• Correction in landfall decay model for unit conversions

Dependencies

TCRM requires:

• Python 3.7;
• numpy;
• scipy;
• matplotlib;
• Basemap;
• netcdf4-python;
• cftime;
• pandas;
• Shapely;
• seaborn;
• statsmodels;
• GitPython;
• GDAL/OGR;
• mpi4py;
• and gcc.

Status

Screenshot

Contributing to TCRM

If you would like to take part in TCRM development, take a look at the Contributing guide.

License

This repository is licensed under the GNU General Public License. See the file LICENSE.rst for information on the history of this software, terms and conditions for usage, and a DISCLAIMER OF ALL WARRANTIES.

Contacts

Community Safety Branch Geoscience Australia [email protected]

tcrm's Issues

Ensure OGC CF-NetCDF compliance for output files

OGC's CF-NetCDF3 data model extension standard applies to netCDF3 datasets. Unclear as to impacts on NetCDF4 format datasets.
There is an abstract test suite defined in OGC 11.165r2 (http://www.opengis.net/doc/is/netcdf-data-model-extension/1.0) - core conformance class is priority.

This will also ensure CF-1.6 compliance (note CF-1.7 is in draft currently).

`axes.color_cycle` is deprecated

c:\python27\lib\site-packages\matplotlib\__init__.py:878: 
UserWarning: axes.color_cycle is deprecated and replaced with axes.prop_cycle; 
please use the latter.

axes.color_cycle is used in PlotInterface.figures (line 244 and 303) and PlotInterface.timeseries (line 67 and 117)

This will necessitate a change to the minimum version of matplotlib

Change KDE methods to statsmodels

Upgrade Evaluate.genesisDensity to use statsmodels.nonparametric.kde.KDEMultivariate for generating the probability density estimates.

Extract wind maxima at point locations

Similar to the time series extraction, produce a delimited file (or possibly a shape file) that contains the maximum wind speed and minimum pressure values over the life of the event.

Compiling with mingw32 fails

Mingw32 installed with Python(x,y) 2.7.6.1 (gcc v 4.8.1). When running compile.cmd, gcc fails reporting::

c:\mingw32\include\io.h: <lineno>: error: unknown type name 'off64_t'

Changing 'std=-c99' to 'std=-gnu99' in the extra_compiler_args for each of Cmap.c, Cstats.c and KPDF.c in installer/setup.py appears to resolve.

Fix error message when TCRM started without ini file.

Deprecated numpy.genfromtxt argument

Some functions in the code rely on numpy.genfromtxt(), and use the 'skip_rows' keyword argument. This will be deprecated in numpy v2.0, and should be replaced with 'skip_header'. There are a couple of instances of 'skiprows' being used as well (note the absent underscore).

Track processed files

To make sure we don't try to re-insert existing records (i.e. track or windfield records) into the database, we need to record processed files in a dat file using the Utilities.process module. This way, we can check if the file has been updated and update the record (not insert a new record!) if it has been updated.

PlotData raises TypeError

Running PlotData on a small region generates a TypeError when plotting pressure data:

TypeError: 'numpy.float64' object cannot be interpreted as an index

Full traceback below:

2018-01-16 16:44:03: Plotting pressure data
2018-01-16 16:44:03: Plotting results:    5% complete
2018-01-16 16:44:03: Traceback (most recent call last):
2018-01-16 16:44:03: File "tcrm.py", line 634, in startup
2018-01-16 16:44:03: main(configFile)
2018-01-16 16:44:03: File "tcrm.py", line 53, in wrap
2018-01-16 16:44:03: res = f(*args, **kwargs)
2018-01-16 16:44:03: File "tcrm.py", line 536, in main
2018-01-16 16:44:03: doDataPlotting(configFile)
2018-01-16 16:44:03: File "/home/craig/tcrm/Utilities/parallel.py", line 70, in wrap
2018-01-16 16:44:03: return f(*args, **kwargs)
2018-01-16 16:44:03: File "tcrm.py", line 290, in doDataPlotting
2018-01-16 16:44:03: PrsPlot.plotPressure(pAllData)
2018-01-16 16:44:03: File "/home/craig/tcrm/PlotInterface/plotStats.py", line 250, in plotPressure
2018-01-16 16:44:03: xlim=(850, 1020), ylim=(850, 1020))
2018-01-16 16:44:03: File "/home/craig/tcrm/PlotInterface/plotStats.py", line 104, in scatterHistogram
2018-01-16 16:44:03: **kwargs)
2018-01-16 16:44:03: File "/home/craig/.local/lib/python2.7/site-packages/seaborn/distributions.py", line 1014, in jointplot
2018-01-16 16:44:03: grid.plot_marginals(distplot, **marginal_kws)
2018-01-16 16:44:03: File "/home/craig/.local/lib/python2.7/site-packages/seaborn/axisgrid.py", line 1224, in plot_marginals
2018-01-16 16:44:03: func(self.x, **kwargs)
2018-01-16 16:44:03: File "/home/craig/.local/lib/python2.7/site-packages/seaborn/distributions.py", line 576, in distplot
2018-01-16 16:44:03: color=hist_color, **hist_kws)
2018-01-16 16:44:03: File "/usr/lib/python2.7/dist-packages/matplotlib/__init__.py", line 1814, in inner
2018-01-16 16:44:03: return func(ax, *args, **kwargs)
2018-01-16 16:44:03: File "/usr/lib/python2.7/dist-packages/matplotlib/axes/_axes.py", line 6010, in hist
2018-01-16 16:44:03: m, bins = np.histogram(x[i], bins, weights=w[i], **hist_kwargs)
2018-01-16 16:44:03: File "/home/craig/.local/lib/python2.7/site-packages/numpy/lib/function_base.py", line 729, in histogram
2018-01-16 16:44:03: n = np.zeros(bins, ntype)
2018-01-16 16:44:03: TypeError: 'numpy.float64' object cannot be interpreted as an index

Reported by Amit Ghosh (2018-01-17)

Start File to Run Whole Application and Building Errors

Dear All,

How to start to run here pls(may be run.py ?) and there were building errors that I will put in place soon.

Kindly advise me since I am new in here.

Best regards
Kenneth

Wind speed averaging times as configuration option

Per request from Julie Plantaz, we should make the wind speed averaging time of the output a configurable option.

loadData.py forces `num` to be an integer

The getInitialPositions function within loadData.py assumes that num being read in from the input dataset is an integer (line 332).
In the IBTrACS database, the TC serial number contains letters, and is therefore classed as a string.
If you read in IBTrACS data with num specified for the serial number, it will fail the series of try tests, and raise a ValueError.

Hazard map figures don't mask land east of 180E

The default mapping routine for hazard maps is to mask out over water areas. This fails east of the dateline, resulting in figures like the attached.

There are two options. The simple option is to use the FilledContourMapFigure() method, and simply plot the hazard levels across the simulation domain.

The second and more complex option is to try and fix the masking in PlotInterface.maps to ensure the mask is applied east of the dateline.

Hard-coded order of fields in `buildLocationDatabase`

The database.buildLocationDatabase function assumes a fixed format for the input shape file, which is based on the stationlist.shp file that is supplied with the code.

If a user wishes to use a different location file, it must match the format of that shape file, else the function will raise an unhandled exception.

Need to have a way of mapping the attributes of the input shape file to the required attributes of tblLocations in locations.db

Duplication of xtick labels on ARI curve figures

On branch v2.0 (commit 2f8c06c), ARI curve has duplicate labels on the x-axis. One is in normal numerical values, one in powers of 10. Need to remove one of the labels (I propose the ones in normal values be removed).

Enable saving of wind field at each timestep

Using the timestepCallback in wind.__init__(), enable the code to save the maximum wind speed across the domain at each time step.

MSLP directory

Relocate daily long term mean SLP data file to ./input and delete this directory

Slice indices error in beta

Beta branch pulled from repo. Nosetests run on newly pulled code returns the following error.

ERROR: Test a dataset is correctly reprojected

---

Traceback (most recent call last):
File "c:\Users\u37443\TCRM\tests\test_processMultipliers.py", line 187, in tes
t_reprojectDataset_same_nc_img
filename=f_img.name)
File "c:\Users\u37443\TCRM\ProcessMultipliers\processMultipliers.py", line 154
, in createRaster
tempBand.WriteArray(array[::np.sign(dy) * 1])
TypeError: slice indices must be integers or None or have an index method
-------------------- >> begin captured logging << --------------------
root: DEBUG: Creating in-memory raster
root: DEBUG: Spatial reference system is:
root: DEBUG: PROJCS["PRS92 / Philippines zone 3",GEOGCS["PRS92",DATUM["Philippin
e_Reference_System_1992",SPHEROID["Clarke 1866",6378206.4,294.9786982139006,AUTH
ORITY["EPSG","7008"]],TOWGS84[-127.62,-67.24,-47.04,-3.068,4.903,1.578,-1.06],AU
THORITY["EPSG","6683"]],PRIMEM["Greenwich",0,AUTHORITY["EPSG","8901"]],UNIT["deg
ree",0.0174532925199433,AUTHORITY["EPSG","9122"]],AUTHORITY["EPSG","4683"]],PROJ
ECTION["Transverse_Mercator"],PARAMETER["latitude_of_origin",0],PARAMETER["centr
al_meridian",121],PARAMETER["scale_factor",0.99995],PARAMETER["false_easting",50
0000],PARAMETER["false_northing",0],UNIT["metre",1,AUTHORITY["EPSG","9001"]],AXI
S["X",EAST],AXIS["Y",NORTH],AUTHORITY["EPSG","3123"]]
Utilities.nctools: DEBUG: Creating variable lat
Utilities.nctools: DEBUG: Creating variable lon
root: DEBUG: Creating raster: c:\users\u37443\appdata\local\temp\1\test_processM
ultipliersmelnxm.img

Move cell statistics code from TrackGenerator.py to stats calculation stage.

Cell statistics for speed, bearing, pressure, and pressure rate of change for all the grid cells in the domain are being calculated at the start of the track generation. This needs to be moved into the statistics calculation stage.

Enable non-integer sized grid

Current code forces the grid cells to be an integer value (i.e. 1, 2, 3...degrees); It'd be nice to have non-integer grid sizes. Requires modification of Utilities/Cstats.c

Timeseries extraction causes MemoryError

When running single event scenarios, users can elect to extract time series information at points across the simulation domain. If there's a large number of locations, or the simulation covers a long time, then the simulation can fail, reporting a MemoryError.

This arises because the timeseries class creates a set of lists for each location to hold the data values, then appends to each list at each time step, increasing the memory footprint.

In some cases, it appears to free up a small amount of memory at each time step, but it will eventually accumulate to the point of exceeding the available memory.

Update hazard map plotting routines to use a standardised palette

ISO22324 (attached) contains guidelines for color-coding hazard information. Using the seaborn.blend_palette() function, we can create a colour palette that matches the recommended schemes. I propose using the 7-category listing set out in Table B-1 in the attached document.

This will modify PlotInterface.maps.selectColormap function, specifically line 100, whcih currenctly defines a yellow-orange-red colour palette.

ISO_22324_2015(en).PDF

Calling :func:`Utilities.version` without git installed raises subprocess.CalledProcessError

Calling :func:Utilities.version without git installed raises subprocess.CalledProcessError

Hotfix to pass the exception if it is raised in Utilities.version.py

HollandWindProfile using HPa

Due to the WindSpeedModel class convert HPa to Pa when calculating dP, and the WindProfileModel class NOT doing so, the HollandWindProfile when called with pressures in HPa produces a profile discontinuous at rMax

AutoPlotHazard.plotCurves raises ProgrammingError

Branch: develop

Executing the AutoPlotHazard.plotCurves method raises a ProgrammingError when trying to get the locations from the database.

2017-11-10 20:10:31: Plotting return period curves for locations within the model domain
2017-11-10 20:10:31: Cannot retrieve locations from tblLocations: Cannot operate on a closed database.
Traceback (most recent call last):
  File "/home/547/cxa547/tcrm/database/__init__.py", line 275, in getLocations
    cur = self.execute(("SELECT locId, locName, locLon, locLat "
ProgrammingError: Cannot operate on a closed database.
2017-11-10 20:10:31: Traceback (most recent call last):
2017-11-10 20:10:31: File "tcrm.py", line 670, in startup
2017-11-10 20:10:31: main(configFile)
2017-11-10 20:10:31: File "tcrm.py", line 86, in wrap
2017-11-10 20:10:31: res = f(*args, **kwargs)
2017-11-10 20:10:31: File "tcrm.py", line 583, in main
2017-11-10 20:10:31: doHazardPlotting(configFile)
2017-11-10 20:10:31: File "/home/547/cxa547/tcrm/Utilities/parallel.py", line 70, in wrap
2017-11-10 20:10:31: return f(*args, **kwargs)
2017-11-10 20:10:31: File "tcrm.py", line 463, in doHazardPlotting
2017-11-10 20:10:31: plotter.plotCurves()
2017-11-10 20:10:31: File "/home/547/cxa547/tcrm/PlotInterface/AutoPlotHazard.py", line 129, in plotCurves
2017-11-10 20:10:31: self.plotHazardCurves(self.inputFile, self.plotPath)
2017-11-10 20:10:31: File "/home/547/cxa547/tcrm/PlotInterface/AutoPlotHazard.py", line 197, in plotHazardCurves
2017-11-10 20:10:31: placeNames, placeID, placeLats, placeLons, locations = self.getLocations()
2017-11-10 20:10:31: File "/home/547/cxa547/tcrm/PlotInterface/AutoPlotHazard.py", line 171, in getLocations
2017-11-10 20:10:31: locations = self.db.getLocations()
2017-11-10 20:10:31: File "/home/547/cxa547/tcrm/database/__init__.py", line 275, in getLocations
2017-11-10 20:10:31: cur = self.execute(("SELECT locId, locName, locLon, locLat "
2017-11-10 20:10:31: ProgrammingError: Cannot operate on a closed database.

The database instance is created at line 89 in PlotInterface/AutoPlotHazard.py.
Possible causes:

• database instance already exists and HazardDatabase.getInstance is returning a closed database rather than the existing instance which has an open connection
• instance is not actually creating an open connection

Options:

• check the database instance actually has an open connection

Extend tracks2shp to convert netCDF track files to other formats

Requested by Martin Tingley (IAG)

Extend tracks2shp to enable conversion from netCDF track file format to other formats - e.g. csv, shape file.

Presently supports csv to shape file.

Remove shebang from file headers

Can remove the #!/usr/bin/env python from file headers - gives better indication of language statistics.

Numpy 1.12.0 errors

Errors only appeared with numpy-1.12.0.
When running stochastic simulation:

TrackGenerator.run(configFile, status)
File "/short/w85/cek156/tcrm/TrackGenerator/TrackGenerator.py", line 1871, in run
tracks = tg.generateTracks(sim.ntracks, sim.index)
File "/short/w85/cek156/tcrm/TrackGenerator/TrackGenerator.py", line 661, in generateTracks
data = np.core.records.fromarrays(data, dtype=track_dtype)
File "/home/156/cek156/.local/lib/python2.7/site-packages/numpy/core/records.py", line 605, in fromarrays
raise ValueError("mismatch between the number of fields "
ValueError: mismatch between the number of fields and the number of arrays

I thought this was an error in my personal environment (and still possibly is), but it seems to be related to the newer version of numpy (was previously using 1.8.0).

Deprecated numpy interface

In file included from /apps/python/2.7.6/lib/python2.7/site-packages/numpy/core/include/numpy/ndarraytypes.h:1760,
from /apps/python/2.7.6/lib/python2.7/site-packages/numpy/core/include/numpy/ndarrayobject.h:17,
from /apps/python/2.7.6/lib/python2.7/site-packages/numpy/core/include/numpy/arrayobject.h:4,
from Utilities/akima.c:64:
/apps/python/2.7.6/lib/python2.7/site-packages/numpy/core/include/numpy/npy_1_7_deprecated_api.h:15:2: warning: #warning "Using deprecated NumPy API, disable it by " "#defining NPY_NO_DEPRECATED_API NPY_1_7_API_VERSION"

NoOptionError when running port_hedland.ini example

Running port headland using tcrm.py, gives the following error, even though the gridLimit is set in the config file. And for some reason the output folders are not being created either.

11:59:13: File "D:\tcrm-2.0\DataProcess\CalcTrackDomain.py", line 49, in init
11:59:13: self.wf_domain = config.geteval('Region', 'gridLimit')
11:59:13: File "D:\tcrm-2.0\Utilities\config.py", line 256, in geteval
11:59:13: return self._get(section, eval, option)
11:59:13: File "D:\Python27\lib\ConfigParser.py", line 356, in _get
11:59:13: return conv(self.get(section, option))
11:59:13: File "D:\Python27\lib\ConfigParser.py", line 340, in get
11:59:13: raise NoOptionError(option, section)
11:59:13: NoOptionError: No option 'gridLimit' in section: 'Region'

Update columns.colReadCSV

Add the option of using converters in columns.colReadCSV(). They can be stored in a configuration file under a source definition - e.g. :

Converters = {0: lambda s: float(s)}.

Requires clear rules around lambda definitions - specifically how different modules are imported within columns

Travis.CI fails on import osgeo

ImportError for osgeo:

Failure: ImportError (No module named osgeo) ... ERROR
Failure: ImportError (No module named osgeo) ... ERROR
======================================================================
ERROR: Failure: ImportError (No module named osgeo)

---

Traceback (most recent call last):
File "/home/travis/py/lib/python2.7/site-packages/nose/loader.py", line 418, in loadTestsFromName
addr.filename, addr.module)
File "/home/travis/py/lib/python2.7/site-packages/nose/importer.py", line 47, in importFromPath
return self.importFromDir(dir_path, fqname)
File "/home/travis/py/lib/python2.7/site-packages/nose/importer.py", line 94, in importFromDir
mod = load_module(part_fqname, fh, filename, desc)
File "/home/travis/build/GeoscienceAustralia/tcrm/Evaluate/init.py", line 12, in
from Evaluate.pressureDistribution import PressureDistribution
File "/home/travis/build/GeoscienceAustralia/tcrm/Evaluate/pressureDistribution.py", line 36, in
from PlotInterface.figures import QuantileFigure, saveFigure
File "/home/travis/build/GeoscienceAustralia/tcrm/PlotInterface/figures.py", line 18, in
import wind.windmodels as windmodels
File "/home/travis/build/GeoscienceAustralia/tcrm/wind/init.py", line 53, in
from ProcessMultipliers import processMultipliers as pM
File "/home/travis/build/GeoscienceAustralia/tcrm/ProcessMultipliers/processMultipliers.py", line 45, in
from osgeo import osr, gdal
ImportError: No module named osgeo

Need to add osgeo to .travis.yml dependencies

Use spatial database for managing the database

Rather than using the sqlite3 database for managing simulation output, transfer to a spatial database such as SpatiaLite which extends the capabilities of sqlite to include spatial information. This'll mean we can store the geometry of tracks, not merely the values of the tracks.

Use the tc-dat branch (or a new branch) for this enhancement.

Create a notebook that can configure a TCRM simulation

Rather than continuing development of the gui, can we use an interactive IPython notebook to step users through the process of setting up a TCRM simulation?

This way users can work through setting up a simulation domain, identifying source data and what actions to perform.

This should live in the notebooks branch, and be linked into the online documentation.

Add check at start-up to see if c extensions are compiled.

Images are missing in the documentation

Hi
I think all the images/figures in the documentation are missing. I checked the doc directory and found that there are some figures and have different name .

Standardise names for track fields

Need to standardise the names of fields in the Track class, so they play nicely with vector shape files. ESRI shape files have a maximum field name length (10 characters), and the current attribute names exceed that length. It would be ideal to make them consistent between the Track class and the other methods that interact with tracks.

Database error when attempting to overwrite existing database

The following exception is raised when trying to insert a record into an existing database where there is an existing record with the same event number.

Traceback (most recent call last):
  File "/home/tcrm/tcrm/database/__init__.py", line 345, in insertEvents
    self.execute(INSEVENTS, eventparams)
IntegrityError: UNIQUE constraint failed: tblEvents.eventNumber
19:00:45: Cannot insert records into tblEvents: UNIQUE constraint failed: tblEvents.eventNumber

Most likely scenario for this to occur is when a user tries to run the the database creation component when it has previously been executed.

This exception should be handled and the database population component be stopped, or alternatively enable a way to overwrite the existing records.

Port Headland example script slice error

13:20:16: Traceback (most recent call last):
13:20:16: File "tcrm.py", line 634, in startup
13:20:16: main(configFile)
13:20:16: File "tcrm.py", line 53, in wrap
13:20:16: res = f(*args, *kwargs)
13:20:16: File "tcrm.py", line 516, in main
13:20:16: doStatistics(configFile)
13:20:16: File "c:\Users\u37443\TCRM\Utilities\parallel.py", line 70, in wrap
13:20:16: return f(args, **kwargs)
13:20:16: File "tcrm.py", line 357, in doStatistics
13:20:16: statInterface.kdeGenesisDate()
13:20:16: File "c:\Users\u37443\TCRM\StatInterface\StatInterface.py", line 122,
in kdeGenesisDate
13:20:16: kdeStep=0.25, periodic=365)
13:20:16: File "c:\Users\u37443\TCRM\StatInterface\GenerateDistributions.py", li
ne 173, in allDistributions
13:20:16: periodic=periodic)
13:20:16: File "c:\Users\u37443\TCRM\StatInterface\KDEParameters.py", line 142,
in generateKDE
13:20:16: self.pdf = 3.0self.pdf[(periodic/kdeStep):2(periodic/kdeStep)]
13:20:16: TypeError: slice indices must be integers or None or have an index
method

Check test suite

On gpd branch (but also true on others..)

Using nosetests to run the testing suite results in

• 173 tests run
• 0 errors

Using python ./tests/run.py (as documented in the TCRM user guide) results in

• 182 tests run
• 5 fails

The 5 failures are listed below. They are errors that should not have not been introduced by the gpd modifications - they are in tests for existing functionality.

Not sure why nosetests does not pick up as many tests as the run.py script.

Error messages from failed tests:

ERROR: test_generate_syn_mult_img (tests.test_processMultipliers.TestProcessMultipliers)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/short/w85/cek156/tcrm_again/tests/test_processMultipliers.py", line 239, in test_generate_syn_mult_img
    pM.generate_syn_mult_img(136, -20, 2, dir_path, shape=(2, 4))
  File "/short/w85/cek156/tcrm_again/ProcessMultipliers/processMultipliers.py", line 98, in generate_syn_mult_img
    fill = value['fill']
TypeError: tuple indices must be integers, not str

===================================================================
ERROR: test_reprojectDataset_same_nc_img (tests.test_processMultipliers.TestProcessMultipliers)
Test a dataset is correctly reprojected
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/short/w85/cek156/tcrm_again/tests/test_processMultipliers.py", line 174, in test_reprojectDataset_same_nc_img
    wg.saveGustToFile(None, result, f_nc.name)
  File "/short/w85/cek156/tcrm_again/wind/__init__.py", line 528, in saveGustToFile
    trackfileDate = flModDate(trackfile)
  File "/short/w85/cek156/tcrm_again/Utilities/files.py", line 291, in flModDate
    si = os.stat(filename)
TypeError: coercing to Unicode: need string or buffer, NoneType found

===================================================================
ERROR: test_xprocessMult_A (tests.test_processMultipliers.TestProcessMultipliers)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/short/w85/cek156/tcrm_again/tests/test_processMultipliers.py", line 246, in test_xprocessMult_A
    pM.generate_syn_mult_img(136, -20, 2, dir_path, shape=(2, 4))
  File "/short/w85/cek156/tcrm_again/ProcessMultipliers/processMultipliers.py", line 98, in generate_syn_mult_img
    fill = value['fill']
TypeError: tuple indices must be integers, not str

===================================================================
ERROR: test_badinput (tests.test_columns.TestReadCSV)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/short/w85/cek156/tcrm_again/tests/test_columns.py", line 28, in test_badinput
    self.configFile, self.badinput, self.source)
  File "/apps/python/2.7.6/lib/python2.7/unittest/case.py", line 475, in assertRaises
    callableObj(*args, **kwargs)
  File "/short/w85/cek156/tcrm_again/Utilities/columns.py", line 34, in colReadCSV
    delimiter = config.get(source, 'FieldDelimiter')
  File "/apps/python/2.7.6/lib/python2.7/ConfigParser.py", line 330, in get
    raise NoSectionError(section)
NoSectionError: No section: 'TEST'

===================================================================
ERROR: test_readCSV (tests.test_columns.TestReadCSV)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/short/w85/cek156/tcrm_again/tests/test_columns.py", line 33, in test_readCSV
    loadedData = columns.colReadCSV(self.configFile, self.inputFile, self.source)
  File "/short/w85/cek156/tcrm_again/Utilities/columns.py", line 34, in colReadCSV
    delimiter = config.get(source, 'FieldDelimiter')
  File "/apps/python/2.7.6/lib/python2.7/ConfigParser.py", line 330, in get
    raise NoSectionError(section)
NoSectionError: No section: 'TEST'

----------------------------------------------------------------------
Ran 182 tests in 212.842s

FAILED (errors=5)

Travis broken on develop branch

Travis stopped working after gdal was added to the build list.

Errors include:
Failure: ImportError (libgdal.so.20: cannot open shared object file: No such file or directory)

ERROR: Failure: ImportError (C extension: iNaT not built. If you want to import pandas from the source directory, you may need to run 'python setup.py build_ext --inplace --force' to build the C extensions first.)

Missing logger instance in `Evaluate.interpolateTracks`

Need to instantiate a :class:logging instance in interpolateTracks. See stack trace below:

15:27:30: Traceback (most recent call last):
15:27:30: File "tcevent.py", line 259, in startup
15:27:30: main(configFile)
15:27:30: File "tcevent.py", line 51, in wrap
15:27:30: res = f(*args, **kwargs)
15:27:30: File "tcevent.py", line 186, in main
15:27:30: interpolation_type='akima')
15:27:30: File "tcrm\Evaluate\interpolateTracks.py", line 280, in parseTracks
15:27:30: newtrack = interpolate(track, delta, interpolation_type)
15:27:30: File "\tcrm\Evaluate\interpolateTracks.py", line 123, in interpolate
15:27:30: logger.exception( ("Akima interpolation module unavailable "
15:27:30: NameError: global name 'logger' is not defined

Naming of wind field dump files

In WindfieldInterface.init.py:

WindfieldGenerator.dumpGustsFromTracks() generates a 'dumpfile' name for the output nc file, with an integer used to indicate different tracks being processed. The number iterates over the number of events in the individual track file, not over the total number of events simulated. This results in only a small number of output netcdf files being generated in WindfieldInterface, but each is written multiple times throughout a simulation.

The trackId should be used to generate the 'dumpfile' name.

See line 650 of WindfieldInterface.init.py

Link notebooks into documentation

Add links to the notebooks at appropriate points in the online documentation, and on a separate page, to make it clear for new users on the workflow used to run the model.

Evaluation script config

The evaluate module still relies on a separate configuration file to be executed. We need to change the setting names in evaluate.py to determine settings from the main TCRM configuration settings.

NetCDF output does not display correctly in QGIS

NetCDF4 files do not project correctly in QGIS (2.14.4). Appears to be a problem with the underlying GDAL libraries (see https://trac.osgeo.org/gdal/ticket/5873 and http://nbviewer.jupyter.org/gist/bnordgren/47eee80aae57deb303a0 for a fix)

Unittest pickle files

Many of the pickle files used in the unit tests were written on a Windows machine.

http://mail.scipy.org/pipermail/numpy-discussion/2009-November/046435.html indicates that the pickle files should have been opened with mode 'wb' rather than 'w'. Pickle files will be rewritten.

ValueError in KDE calculation

On gpd branch during Evaluate

File "/short/w85/cek156/tcrm_again/Evaluate/genesisDensity.py", line 470, in run
self.synthetic()
File "/short/w85/cek156/tcrm_again/Evaluate/genesisDensity.py", line 295, in synthetic
results = self._calculate(tracks) #/ self.synNumYears
File "/short/w85/cek156/tcrm_again/Evaluate/genesisDensity.py", line 197, in _calculate
kernel = KDEMultivariate(values, bw='cv_ml', var_type='cc')
File "/home/156/cek156/.local/lib/python2.7/site-packages/statsmodels/nonparametric/kernel_density.py", line 111, in __init__
raise ValueError("The number of observations must be larger " \
ValueError: The number of observations must be larger than the number of variables.```

numpy.genfromtxt TypeError in plotTimeseries.py

When using numpy v1.10 or later, numpy.genfromtxt calls that use the 'skiprows' keyword argument fail with a TypeError. The 'skiprows' kwarg was removed in v1.10 and replaced with 'skip_header'.

Identified in PlotInterface.plotTimeseries.loadTimeseriesData()

Mint DOIs for TCRM versions

Follow https://guides.github.com/activities/citable-code/ to set up Zenodo to issue DOIs for releases, so users can cite their version of the code.