inet-complexity/isle

isle's Issues

Visualization

The visualization class is not flexible enough. It still cannot do all what plotter.py and metaplotter.py were able to do.

We should try to merge all the visual scripts in a single class or file. After doing that we should remove plotter.py and metaplotter.py and related files.

*_history_logs.dat takes a lot of disk space

This is because the dict is stored in plaintext. It would be much more space-efficient (and time-efficient) to use pickle.

Plotting with human-readable timescales

• Human-readable timescales as in plotter_pl_timescale.py should be used in plotter.py
• metaplotter_pl_timescale_additional_measures.py should replace metaplotter.py

Is this being actively develop?

@DavoudTaghawiNejad and @x0range where is this repo being actively develop? Looks like in @x0range fork? Can we move development to this repo? I am happy to set up a team for the insurance modeling add whomever you want to the team and make one of you the maintainer (i.e., have admin rights) for the project. This way I can automate code coverage, static, analysis, etc and we can all provide better peer reviewing of code.

Create savedata object

... to be called for saving data from the start script.

Currently, saving is done in insurancesimulation.py and independently of that in the Sandtable startscript (which is not yet committed).

Simulation should be able to switch on or off premium endogeneity

Currently the insurance and reinsurance premiums are time-dependent. This is model endogeneous. We should have a boolean parameter that allows to have the premium at a constant level for the entire course of the simulation.

And this premium should be a parameter too (should otherwise be used as initial premium).

riskmodel.evaluate() occasionally returns huge numbers for ```remaining_acceptable_by_categ```

Problem is this code block in riskmodel.py.evaluate_proportional()

        remaining_acceptable_by_category[categ_id] = math.floor(
                        remaining_acceptable_by_category[categ_id] * pow(
                            cash_left_by_category[categ_id] / max_category, 5))

in lines 157f.
Problem exists because there pow() with a negative base leads to absurd results.
Also, the iteration over the categories has erroneously been removed. (This was my fault; sorry.)

Different risks per category

The number of risks per category that we get from insurancesimulation is not exactly the same. This might be a problem in some situations.

Network representation

The network representation should be done outside the base code. Otherwise we cannot run it in the cloud.

Profits and Losses

Right now market entry in computed as a profit in the profits and losses time series. In the same way market exit (without default) is considered as a loss. This does not seem realistic.

Solution: Keep track of the profit and losses at the firm level recording them in the pay and receive methods.

Include a requirements.txt

It would be really helpful to include a file with all of the necessary dependencies, so it is as easy as possible to run this programme.

Normally, this is done through requirements.txt in python repos.

Rewrite start scripts for Sandtable and non-Sandtable version

Currently, many things are repeated in different places in the code such that the code works for both cases. The Sandtable version is not yet committed to the code base.

As much code as possible should be shared between Sandtable and non-Sandtable versions. The two startscripts (Sandtable and non-Sandtable) should only do what is actually different. They both should do the following:

1. read the event schedule from files,
2. setup the simulation schedule,
3. call start.py for each simulation,
4. collect the results, and save them the new save data object.

Premium sensitivity

We need a parameter to vary how much we want the premium to change according to the capital available in the market .Maybe two one for insurers and another for reinsurers. For now is in the code.

Commit Sandtable version of the code

redesign premium development

Companies should have a bearing on the development of the premium. E.g., they may be required to indicate in every one period if they would like to increase or decrease the premium. The simulation object (in reality: the regulator) should decide based on this whether to increase or decrease the premium.

Simulation should be able to study the tradeoff between (number of bankruptcies) and (extent of the market) in a more clear way

We found that risk model homogeneity can have either

• an effect on the extent of the market (the number of contracts underwritten), or
• an effect on the number of bankruptcies,

... depending on the parameter setting. (For instance by varying the margin of safety parameter.)

We should implement ways to study this and visualize this in a clear way.

Total collapse of the insurance market

With a high probability in the runs of the model there will be a total collapse of the insurance market. The way of recovery of the market seems unrealistic in the model. Basically after every total collapse there is a transient period until the insurance market is completely recovered since the market entry of insurance firms is made at random every time step. The recovery time is not fixed and depends of the parameter settings.

All this might introduce some noise when aggregating over several realizations of the model.

Times at which obligations from claims become due

It has to be ensured that insurers will receive reinsurance payments before they have to pay insurance claims, otherwise they become bankrupt when they should not.

Possible ways to resolve this

1. Make reinsurance payments due before insurance payments (problem: does not account for retrocession)
2. Identify tree structure, go through tree from top to bottom (problem: there may be loops)
3. Allow temporarily negative cash (problem: invalid simulation runs are identified after the fact and need to be rolled back)
4. Flag bankrupt firms first, reevaluate their payments again at the end of the period, have them enter bankruptcy only when it cannot be resolved.

(1) is the solution that is currently implemented. This is probably unrealistic and should be changed.

Create script for setting up risk event schedule

Currently this is done in insurancesimulation.py. Separately of this it is also done in the startscript for the Sandtable version (not yet committed) since the Sandtable version cannot read files on the server.

It should instead be done in a separate script. It should save risk event times, risk event sizes, numpy random state, and random random state to (one or more) files. This file/these files should then be read by the startscripts (both Sandtable and non-Sandtable versions).

The simulation is not deterministic even with explicit seed

I ran starter_one.sh with the loops decreased from 300 to 1, with isleconfig max_time set to 5, and with --randomseed 1, and I got different outcome every time:

mv: cannot stat 'data/one_operational.dat': No such file or directory
mv: cannot stat 'data/one_contracts.dat': No such file or directory
mv: cannot stat 'data/one_cash.dat': No such file or directory
mv: cannot stat 'data/one_reinoperational.dat': No such file or directory
mv: cannot stat 'data/one_reincontracts.dat': No such file or directory
mv: cannot stat 'data/one_reincash.dat': No such file or directory
mv: cannot stat 'data/one_premium.dat': No such file or directory

condition_defaults_insurance                   : 0.000000
condition_defaults_reinsurance                 : 0.000000
condition_insurance_coverage                   : 0.180900
condition_insurance_firm_dist                  : 0.000230
condition_reinsurance_coverage                 : 0.000000
condition_reinsurance_firm_dist                : 0.000929
condition_stationary_state_cash                : 1.000000
condition_stationary_state_contracts           : 1.000000
condition_stationary_state_excess_capital      : 1.000000
condition_stationary_state_market_premium      : 1.000000
condition_stationary_state_profits_losses      : 0.709162
condition_stationary_state_rein_cash           : 1.000000
condition_stationary_state_rein_contracts      :      nan
condition_stationary_state_rein_excess_capital : 1.000000
condition_stationary_state_rein_market_premium : 1.000000
condition_stationary_state_rein_profits_losses : 1.000000

                        Total calibration score: 0.592748
bash starter_one.sh  5.18s user 0.16s system 100% cpu 5.341 total

Is this because of the nan data?

Simplify logging

The pattern

if isleconfig.verbose:
    print("...")

could be replaced with

logging.info("...")

and is a standard practice.

Insolvent insurers should still receive payments to them

In reality the executor decides what happens to that money.
For practical purposes of the simulation, the simulation object can for now be the executor and the money can be forwarded to it.

PERF: Optimize metainsuranceorg's iterate

I found from profiling that >90% of the time takes place in metainsuranceorg.iterate(), of which 81.7% of it takes place in self.process_newrisks_insurer(). In self.process_newrisks_insurer, 40.4% happens in self.balanced_portfolio, 30.9% in the creation of InsuranceContract.

In self.balanced_portfolio, 33.9% happens in std_pre = cash_reserved_by_categ.std(), 12.5% happens in mean = cash_reserved_by_categ_store.mean(), 27.8% happens in std_post = cash_reserved_by_categ_store.std(). This is unnaturally slow, since the loop to create cash_reserved_by_categ itself only takes 6% of the time.

error : Can't pickle local object 'ReinsuranceProfile.__init__.<locals>.<listcomp>.<lambda>'

as running
start.py -f test1 --save_iterations 10

comes up with this error:
Traceback (most recent call last):
File "C:\WPy64-3740\notebooks\isle-master\start.py", line 113, in main
save_simulation(t + 1, simulation, sim_params, exit_now=False)
File "C:\WPy64-3740\notebooks\isle-master\start.py", line 148, in save_simulation
pickle.dump(d, wfile, protocol=pickle.DEFAULT_PROTOCOL)
AttributeError: Can't pickle local object 'ReinsuranceProfile.init...'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "C:\WPy64-3740\notebooks\isle-master\start.py", line 428, in
summary=summary,
File "C:\WPy64-3740\notebooks\isle-master\start.py", line 126, in main
raise Exception("".join(traceback.format_exception(*sys.exc_info()))[-900:])
Exception: Traceback (most recent call last):
File "C:\WPy64-3740\notebooks\isle-master\start.py", line 113, in main
save_simulation(t + 1, simulation, sim_params, exit_now=False)
File "C:\WPy64-3740\notebooks\isle-master\start.py", line 148, in save_simulation
pickle.dump(d, wfile, protocol=pickle.DEFAULT_PROTOCOL)
AttributeError: Can't pickle local object 'ReinsuranceProfile.init...'

any idea, please?

Market exit

Market exit should not count as bankruptcy.

take into account profits in decision making

Currently underwriting decisions are made only using the balancedness and the fulfilment of the capital requirement (as judged by the risk model). It should also include the expected profits of an offered contract. For instance it could be a 3-step process:

1. Do we have the capital to underwrite this? If yes:
2. Will our portfolio become too unbalanced by underwriting this? If no:
3. What will our profits be from underwriting this? If the expected profits are positive, we should underwrite with probability 100%, if they are negative, we should underwrite with a declining probability, depending on how negative they are. This may reach 0% for very negative expected profits.

improve riskmodel

riskmodel.py is messy, has boiler plate code, and is difficult to read. Should be redone in a clearer way. Such that it does two things:

• give estimates about whether the company may fail in a catastrophe with the current capital and portfolio
• give estimates of the contribution of any one underwritten contract and any one offered but not yet underwritten contract to this
  This includes the balancedness of the portfolio.

Reinsurers almost never default

With the usual parameters settings reinsurers almost never default. Most of the time they are far away of their maximum capacity and they can usually absorb the loses of almost all the catastrophes.

Potential solution -> Reduce the margin of safety of reinsurance firms.