automatedprocessimprovement/simod

Simod is a tool for automated BPS model discovery

Simod: Automated discovery of business process simulation models

Simod combines process mining and machine learning techniques to automate the discovery and tuning of Business Process Simulation models from event logs extracted from enterprise information systems (ERPs, CRM, case management systems, etc.). Simod takes as input an event log in CSV format, a configuration file, and (optionally) a BPMN process model, and returns a business process simulation scenario that can be simulated using the Prosimos simulator, which is embedded in Simod.

Dependencies

Required

Dependency	Version	Notes
Python	3.9	For Windows, Python 3.9.13 is the last distribution with Windows installers.
Java	1.8	For example, use Amazon Corretto 8.
Poetry	latest	If using Docker or compiling from source, use Poetry for building, installing, and managing Python dependencies.

Optional

Depending on your CPU architecture, some dependencies might not be pre-compiled for your platform. In that case, you will most likely also need the following dependencies:

Dependency	Version	Notes
Cargo and Rust	latest	Install it with rustup.rs.

Getting Started

PyPI

❗️Make sure java -version returns 1.8 and pip is installed.

Then, install Simod and run it with the following commands:

pip install simod
simod --configuration resources/config/configuration_example.yml

Use your own configuration file instead of resources/config/configuration_example.yml and specify the path to the event log in the configuration file itself. Paths are relative to the configuration file, or absolute.

PyPI project is available at https://pypi.org/project/simod/.

Docker

docker pull nokal/simod

To start a container:

docker run -it -v /path/to/resources/:/usr/src/Simod/resources -v /path/to/output:/usr/src/Simod/outputs nokal/simod bash

Use the resources directory to store event logs and configuration files. The outputs directory will contain the results of Simod.

From inside the container, you can run Simod with:

poetry run simod --configuration <path-to-config>

Docker images for different Simod versions are available at https://hub.docker.com/r/nokal/simod/tags.

Configuration file

A set of example configurations can be found in the resources folder along with a description of each element:

• Basic configuration to discover the full BPS model (here).
• Basic configuration to discover the full BPS model using fuzzy (probabilistic) resource calendars (here).
• Basic configuration to discover the full BPS model, and evaluate it with a specified event log (here).
• Basic configuration to discover a BPS model with a provided BPMN process model as starting point (here).
• Basic configuration to discover a BPS model with no optimization process (one-shot) (here).
• Complete configuration example with all the possible parameters (here).

For developers

Testing

Use pytest to run tests on the package:

poetry run pytest

To run unit tests, execute:

poetry run pytest -m "not integration"

Coverage:

poetry run pytest -m "not integration" --cov=simod

simod's Issues

Adding extraneous timers to BPMN

Before starting the calendar optimization pipeline, run this to add some timer events to the BPMN model. It uses BIMP to optimize the timers.

java cli default argument -Xss8G causes a crash

@Mcamargo85

On macOS 11.4 with java:

java version "15.0.1" 2020-10-20
Java(TM) SE Runtime Environment (build 15.0.1+9-18)
Java HotSpot(TM) 64-Bit Server VM (build 15.0.1+9-18, mixed mode, sharing)

running python simod_console.py fails with:

 -- Mining Process Structure --
Invalid thread stack size: -Xss8G
The specified size exceeds the maximum representable size.
Error: Could not create the Java Virtual Machine.
Error: A fatal exception has occurred. Program will exit.
[Errno 2] No such file or directory: 'outputs/20210701_8512596F_0181_4932_89A5_87F7430B474D/PurchasingExample.bpmn'

Java tool failed because of -Xss8G. Changing it to a smaller number -Xms1024M works okay.

Refactoring evaluation metrics

• Remove all MAE metrics and TSD (leave only DL and DAY_HOUR_EMD).
• DAY_HOUR_EMD is renamed to CIRCADIAN_EMD
• Reimplement CIRCADIAN_EMD
• Refactor SimilarityEvaluator

Missing scripts and logs from version 2.0.0

Hello!

I am tasked with reproducing and researching some findings from a paper in 2020 that used this particular tool (v. 2.0.0). They mention a script "Sweeper.py" which was used for the preprocessing of event logs and also the log as found in "multitasking_logs" folder. Even after downloading the old source files, I cannot find the specified files and really cannot move forward without especially the script. If any contributor has access to the data (eg. if they modified it at the time), kindly share if possible. The links mentioned in the research paper are no longer valid.

old links for reference:

1. https://github.com/AdaptiveBProcess/Simod/tree/master/inputs/
   multitasking_logs
2. https://github.com/AdaptiveBProcess/Simod/tree/master/support_modules/
   multitasking

Optimize writing from DataFrame to XES if possible

This line, https://github.com/AutomatedProcessImprovement/Simod/blob/master/src/simod/process_calendars/optimizer.py#L85, contributes to about 13% of Simod's running time.

It happens during Calendars Optimisation, because we keep splitting the log in each optimiser, so we need to re-write it to disk in Calendars Optimiser.

Introduce batching

Fix off-duty intervals

Subtract from the activity duration the intervals in which the resource is not working (using the calendars discovered for that resource).

Similar implementation in WTA
https://github.com/AutomatedProcessImprovement/waiting-time-analysis/blob/ced90337515306cfe23b3707e883ccc2a588e85e/src/wta/waiting_time/resource_unavailability.py#L88

• implementation
• testing

Add multitasking

As a user, I would like to specify that an event log contains multitasking, so that the processing times of activities are adjusted prior to discovering the simulation model.

The script for multitasking can be found in Simon v2 -- @Mcamargo85 knows where it is located.

Calendar discovery can return None for some resources

For some resources, Prosimos calendar discovery can return None instead of a calendar. So we need to make sure that we have some calendar at the end.

(a) First, we gather all None-calendar resources into a pool and try to find one calendar for all of them.
(b) If some are still None, we find one calendar for the all resources and assign it to resources without a calendar.
(c) Finally, if some are still None, we assign default 24/7 calendar to them.

scipy.array deprecation warning

role_discovery module uses deprecated scipy.array

tests/test_cli.py: 1458 warnings
  /Users/ihar/projects/PIX/simod/src/simod/extraction/role_discovery.py:86: DeprecationWarning: scipy.array is deprecated and will be removed in SciPy 2.0.0, use numpy.array instead
    x = scipy.array(profile_x['profile'])

tests/test_cli.py: 1458 warnings
  /Users/ihar/projects/PIX/simod/src/simod/extraction/role_discovery.py:87: DeprecationWarning: scipy.array is deprecated and will be removed in SciPy 2.0.0, use numpy.array instead
    y = scipy.array(profile_y['profile'])

Benchmarking of RC v3.2.0

Event logs:

• BPIC_2012_W_contained (train, test)
• BPIC_2017_W_contained (train, test)
• ConsultaDataMining201618 (train, test)
• Governmental_Agency (train, test)
• poc_processmining (train, test)
• Production (train, test)

General configuration template which is modified for each log, optimisation metric, and resource profile discovery type:

version: 2
common:
  log_path: assets/train_Production.xes
  test_log_path: assets/test_Production.xes
  exec_mode: optimizer
  log_ids:
    case: case_id
    activity: Activity
    resource: Resource
    start_time: start_time
    end_time: end_time
  repetitions: 5
  simulation: true
  evaluation_metrics: 
    - dl
    - circadian_emd
    - absolute_hourly_emd
    - cycle_time_emd
preprocessing:
  multitasking: false
structure:
  max_evaluations: 40
  mining_algorithm: sm3
  concurrency:
    - 0.0
    - 1.0
  epsilon:
    - 0.0
    - 1.0
  eta:
    - 0.0
    - 1.0
  gateway_probabilities:
    - equiprobable
    - discovery
  or_rep:
    - true
    - false
  and_prior:
    - true
    - false
calendars:
  max_evaluations: 40
  resource_profiles:
    discovery_type: undifferentiated
    granularity: 
      - 15
      - 60
    confidence:
      - 0.5
      - 0.85
    support:
      - 0.01 
      - 0.3
    participation: 0.4

scipy.array deprecation warnings

During testing I have some deprecation warnings. I'll leave it here until I fix.

tests/test_cli.py: 5832 warnings
  /Users/ihar/projects/api/simod/src/simod/extraction/role_discovery.py:87: DeprecationWarning: scipy.array is deprecated and will be removed in SciPy 2.0.0, use numpy.array instead
    x = scipy.array(profile_x['profile'])

tests/test_cli.py: 5832 warnings
  /Users/ihar/projects/api/simod/src/simod/extraction/role_discovery.py:88: DeprecationWarning: scipy.array is deprecated and will be removed in SciPy 2.0.0, use numpy.array instead
    y = scipy.array(profile_y['profile'])

Hyperoptimization pipeline refactoring for time and structure

Unable to run SiMo on Ubuntu

Running python simo.py on Ubuntu 16.04 produces the following error:

 -- Mining Process Structure --
Exception in thread "main" java.lang.ExceptionInInitializerError
Caused by: java.lang.RuntimeException: Failed to load required SO
        at com.raffaeleconforti.benchmark.BenchmarkCommandline.loadLibLinux(BenchmarkCommandline.java:120)
        at com.raffaeleconforti.benchmark.BenchmarkCommandline.loadFromJar(BenchmarkCommandline.java:71)
        at com.raffaeleconforti.benchmark.BenchmarkCommandline.<clinit>(BenchmarkCommandline.java:56)
Caused by: java.lang.NullPointerException
        at org.apache.poi.util.IOUtils.copy(IOUtils.java:182)
        at com.raffaeleconforti.benchmark.BenchmarkCommandline.loadLibLinux(BenchmarkCommandline.java:116)
        ... 2 more

SplitMiner input arguments

According to the source code, https://github.com/nemo-91/bpmtk/blob/22c04f6408dfda6ce6abe77e3327df57d03e10ca/src/au/edu/unimelb/services/ServiceProvider.java#L604-L637, the order of input arguments is the following:

1. eta
2. epsilon
3. parallelismFirst
4. replaceIORs
5. removeLoopActivityMarkers

Simod passes epsilon instead of eta and vice versa since the beginning. Simod v2.3.1, Simod current.

Bring back choosing the max of the results from hyperopt

Consider to use same training/validation in all the optimizations

Currently, the training/validation split is being done twice, one in the Structure optimization phase, and a different one in the Time optimization phase.

This blocks SIMOD from using, in all the Time optimization iterations, the already discovered BPMN model and gateway probabilities from the Time optimization phase.

Possible performance improvement:

• At the start of the general optimization pipeline, split the logs into training/validation/test.
  • If we receive two input logs (training/test): split the training into training/validation, and use the test one as test.
  • If we receive only one input log: split this log into training/validation/test (from the input log, first get 80% for training+validation and 20% for test; then, from the training+validation split, get 80% for training, and 20% for validation).
• There is only one training log, and is the one used in all the phases for training.
• There is only one validation log, and is the one used in all the phases for evaluating each iteration.
• There is only one test log, and is the one used at the end of the SIMOD pipeline, to evaluate the final solution.
• The Structure optimization phase produces a BPMN model and its gateway probabilities (the probabilities, not the method) as a result.
  • If a BPMN model is provided as input, this phase only optimizes the gateway probabilities.
  • Otherwise, this phase optimizes both.
• The Time optimization phase uses the BPMN model and gateway probabilities (the file and the probability values) for each iteration of its optimization. It does not discover a new BPMN model in each iteration.

Adding start times pre-processing

Add an option in the "preprocessor" to add the start times to an event log (if they're missing).

To estimate the start times, use this repo.

Integrate the new computation of EMD metric into Simod

Integrate the new metrics module

Self-assignment in extraction/log_replayer.py

Hi, @Mcamargo85

Going through the code with Sonarqube code analysis tool, it complains at this line marking it as a bug:

https://github.com/AdaptiveBProcess/Simod/blob/31acdb14223297fc798b1d4903e0960250141e0a/extraction/log_replayer.py#L129

Do you agree that this line is redundant?

Rename configuration parameters to make it clear what they mean

At the moment, the configuration looks like this:

log_path: inputs/PurchasingExample.xes
mining_alg: sm3
exec_mode: optimizer
repetitions: 1
simulation: true
sim_metric: dl
simulator: custom
multitasking: false
structure_optimizer:
  max_eval_s: 2
  concurrency:
    - 0.0
    - 1.0
  epsilon:
    - 0.0
    - 1.0
  eta:
    - 0.0
    - 1.0
  gate_management:
    - equiprobable
    - discovery
  or_rep:
    - true
    - false
  and_prior:
    - true
    - false
time_optimizer:
  max_eval_t: 2
  rp_similarity:
    - 0.5
    - 0.9
  res_dtype:
    - dt247
  arr_dtype:
    - dt247
  res_sup_dis:
    - 0.01
    - 0.3
  res_con_dis:
    - 50
    - 85
  arr_support:
    - 0.01
    - 0.1
  arr_confidence:
    - 1
    - 10

There's a lot of names that we can improve, e.g., or_rep, and_priori, rp_similarity, res_dtype, res_cond_dis, etc.

New simulator integration

Remove PDFMethod class

PDFMethod is used to specify distributions discovery type. As we support only one type of distribution discovery now, we don't need this class.

Docker arm64 container for macOS fails because of cvxopt dependency for pm4py in Prosimos

Exception:

(venv) root@17d3c468426c:/usr/src/Simod# simod --help
Traceback (most recent call last):
  File "/usr/src/Simod/venv/bin/simod", line 11, in <module>
    load_entry_point('simod', 'console_scripts', 'simod')()
  File "/usr/src/Simod/venv/lib/python3.9/site-packages/pkg_resources/__init__.py", line 489, in load_entry_point
    return get_distribution(dist).load_entry_point(group, name)
  File "/usr/src/Simod/venv/lib/python3.9/site-packages/pkg_resources/__init__.py", line 2852, in load_entry_point
    return ep.load()
  File "/usr/src/Simod/venv/lib/python3.9/site-packages/pkg_resources/__init__.py", line 2443, in load
    return self.resolve()
  File "/usr/src/Simod/venv/lib/python3.9/site-packages/pkg_resources/__init__.py", line 2449, in resolve
    module = __import__(self.module_name, fromlist=['__name__'], level=0)
  File "/usr/src/Simod/src/simod/cli.py", line 6, in <module>
    from simod.discoverer import Discoverer
  File "/usr/src/Simod/src/simod/discoverer.py", line 19, in <module>
    from .simulator import simulate
  File "/usr/src/Simod/src/simod/simulator.py", line 10, in <module>
    from bpdfr_simulation_engine.simulation_properties_parser import parse_qbp_simulation_process
  File "/usr/src/Simod/external_tools/Prosimos/bpdfr_simulation_engine/simulation_properties_parser.py", line 6, in <module>
    from bpdfr_simulation_engine.control_flow_manager import BPMNGraph, ElementInfo, BPMN
  File "/usr/src/Simod/external_tools/Prosimos/bpdfr_simulation_engine/control_flow_manager.py", line 6, in <module>
    import pm4py
ModuleNotFoundError: No module named 'pm4py'

(venv) root@17d3c468426c:/usr/src/Simod# pip install pm4py
Collecting pm4py
  Downloading pm4py-2.2.23.1-py3-none-any.whl (1.8 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 1.8/1.8 MB 1.7 MB/s eta 0:00:00
Requirement already satisfied: numpy>=1.19.5 in ./venv/lib/python3.9/site-packages (from pm4py) (1.22.4)
Collecting cvxopt
  Using cached cvxopt-1.3.0.tar.gz (4.1 MB)
  Preparing metadata (setup.py) ... done
Requirement already satisfied: matplotlib in ./venv/lib/python3.9/site-packages (from pm4py) (3.5.2)
Collecting pydotplus
  Using cached pydotplus-2.0.2.tar.gz (278 kB)
  Preparing metadata (setup.py) ... done
Requirement already satisfied: scipy in ./venv/lib/python3.9/site-packages (from pm4py) (1.8.1)
Requirement already satisfied: stringdist in ./venv/lib/python3.9/site-packages (from pm4py) (1.0.9)
Collecting deprecation
  Using cached deprecation-2.1.0-py2.py3-none-any.whl (11 kB)
Requirement already satisfied: sympy in ./venv/lib/python3.9/site-packages (from pm4py) (1.10.1)
Requirement already satisfied: tqdm in ./venv/lib/python3.9/site-packages (from pm4py) (4.64.0)
Requirement already satisfied: intervaltree in ./venv/lib/python3.9/site-packages (from pm4py) (3.1.0)
Requirement already satisfied: pandas>=1.1.5 in ./venv/lib/python3.9/site-packages (from pm4py) (1.4.2)
Requirement already satisfied: jsonpickle in ./venv/lib/python3.9/site-packages (from pm4py) (2.2.0)
Requirement already satisfied: lxml in ./venv/lib/python3.9/site-packages (from pm4py) (4.8.0)
Requirement already satisfied: networkx in ./venv/lib/python3.9/site-packages (from pm4py) (2.8.2)
Requirement already satisfied: pytz in ./venv/lib/python3.9/site-packages (from pm4py) (2022.1)
Requirement already satisfied: graphviz in ./venv/lib/python3.9/site-packages (from pm4py) (0.20)
Collecting pyvis
  Using cached pyvis-0.2.1.tar.gz (21 kB)
  Preparing metadata (setup.py) ... done
Requirement already satisfied: python-dateutil>=2.8.1 in ./venv/lib/python3.9/site-packages (from pandas>=1.1.5->pm4py) (2.8.2)
Requirement already satisfied: packaging in ./venv/lib/python3.9/site-packages (from deprecation->pm4py) (21.3)
Requirement already satisfied: sortedcontainers<3.0,>=2.0 in ./venv/lib/python3.9/site-packages (from intervaltree->pm4py) (2.4.0)
Requirement already satisfied: pyparsing>=2.2.1 in ./venv/lib/python3.9/site-packages (from matplotlib->pm4py) (3.0.9)
Requirement already satisfied: fonttools>=4.22.0 in ./venv/lib/python3.9/site-packages (from matplotlib->pm4py) (4.33.3)
Requirement already satisfied: pillow>=6.2.0 in ./venv/lib/python3.9/site-packages (from matplotlib->pm4py) (9.1.1)
Requirement already satisfied: kiwisolver>=1.0.1 in ./venv/lib/python3.9/site-packages (from matplotlib->pm4py) (1.4.2)
Requirement already satisfied: cycler>=0.10 in ./venv/lib/python3.9/site-packages (from matplotlib->pm4py) (0.11.0)
Collecting ipython>=5.3.0
  Downloading ipython-8.4.0-py3-none-any.whl (750 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 750.8/750.8 kB 3.7 MB/s eta 0:00:00
Collecting jinja2>=2.9.6
  Using cached Jinja2-3.1.2-py3-none-any.whl (133 kB)
Requirement already satisfied: mpmath>=0.19 in ./venv/lib/python3.9/site-packages (from sympy->pm4py) (1.2.1)
Requirement already satisfied: backcall in ./venv/lib/python3.9/site-packages (from ipython>=5.3.0->pyvis->pm4py) (0.2.0)
Collecting matplotlib-inline
  Using cached matplotlib_inline-0.1.3-py3-none-any.whl (8.2 kB)
Requirement already satisfied: decorator in ./venv/lib/python3.9/site-packages (from ipython>=5.3.0->pyvis->pm4py) (5.1.1)
Requirement already satisfied: setuptools>=18.5 in ./venv/lib/python3.9/site-packages (from ipython>=5.3.0->pyvis->pm4py) (44.1.1)
Collecting jedi>=0.16
  Using cached jedi-0.18.1-py2.py3-none-any.whl (1.6 MB)
Collecting stack-data
  Downloading stack_data-0.3.0-py3-none-any.whl (23 kB)
Requirement already satisfied: pexpect>4.3 in ./venv/lib/python3.9/site-packages (from ipython>=5.3.0->pyvis->pm4py) (4.8.0)
Requirement already satisfied: pygments>=2.4.0 in ./venv/lib/python3.9/site-packages (from ipython>=5.3.0->pyvis->pm4py) (2.12.0)
Requirement already satisfied: traitlets>=5 in ./venv/lib/python3.9/site-packages (from ipython>=5.3.0->pyvis->pm4py) (5.2.1.post0)
Requirement already satisfied: prompt-toolkit!=3.0.0,!=3.0.1,<3.1.0,>=2.0.0 in ./venv/lib/python3.9/site-packages (from ipython>=5.3.0->pyvis->pm4py) (3.0.29)
Requirement already satisfied: pickleshare in ./venv/lib/python3.9/site-packages (from ipython>=5.3.0->pyvis->pm4py) (0.7.5)
Requirement already satisfied: MarkupSafe>=2.0 in ./venv/lib/python3.9/site-packages (from jinja2>=2.9.6->pyvis->pm4py) (2.1.1)
Requirement already satisfied: six>=1.5 in ./venv/lib/python3.9/site-packages (from python-dateutil>=2.8.1->pandas>=1.1.5->pm4py) (1.16.0)
Requirement already satisfied: parso<0.9.0,>=0.8.0 in ./venv/lib/python3.9/site-packages (from jedi>=0.16->ipython>=5.3.0->pyvis->pm4py) (0.8.3)
Requirement already satisfied: ptyprocess>=0.5 in ./venv/lib/python3.9/site-packages (from pexpect>4.3->ipython>=5.3.0->pyvis->pm4py) (0.7.0)
Requirement already satisfied: wcwidth in ./venv/lib/python3.9/site-packages (from prompt-toolkit!=3.0.0,!=3.0.1,<3.1.0,>=2.0.0->ipython>=5.3.0->pyvis->pm4py) (0.2.5)
Requirement already satisfied: executing in ./venv/lib/python3.9/site-packages (from stack-data->ipython>=5.3.0->pyvis->pm4py) (0.8.3)
Requirement already satisfied: pure-eval in ./venv/lib/python3.9/site-packages (from stack-data->ipython>=5.3.0->pyvis->pm4py) (0.2.2)
Collecting asttokens
  Using cached asttokens-2.0.5-py2.py3-none-any.whl (20 kB)
Using legacy 'setup.py install' for cvxopt, since package 'wheel' is not installed.
Using legacy 'setup.py install' for pydotplus, since package 'wheel' is not installed.
Using legacy 'setup.py install' for pyvis, since package 'wheel' is not installed.
Installing collected packages: pydotplus, matplotlib-inline, jinja2, jedi, cvxopt, asttokens, stack-data, deprecation, ipython, pyvis, pm4py
  Running setup.py install for pydotplus ... done
  Running setup.py install for cvxopt ... error
  error: subprocess-exited-with-error

  × Running setup.py install for cvxopt did not run successfully.
  │ exit code: 1
  ╰─> [35 lines of output]
      running install
      running build
      running build_py
      creating build
      creating build/lib.linux-aarch64-3.9
      creating build/lib.linux-aarch64-3.9/cvxopt
      copying src/python/solvers.py -> build/lib.linux-aarch64-3.9/cvxopt
      copying src/python/msk.py -> build/lib.linux-aarch64-3.9/cvxopt
      copying src/python/cvxprog.py -> build/lib.linux-aarch64-3.9/cvxopt
      copying src/python/printing.py -> build/lib.linux-aarch64-3.9/cvxopt
      copying src/python/info.py -> build/lib.linux-aarch64-3.9/cvxopt
      copying src/python/misc.py -> build/lib.linux-aarch64-3.9/cvxopt
      copying src/python/modeling.py -> build/lib.linux-aarch64-3.9/cvxopt
      copying src/python/__init__.py -> build/lib.linux-aarch64-3.9/cvxopt
      copying src/python/_version.py -> build/lib.linux-aarch64-3.9/cvxopt
      copying src/python/coneprog.py -> build/lib.linux-aarch64-3.9/cvxopt
      UPDATING build/lib.linux-aarch64-3.9/cvxopt/_version.py
      set build/lib.linux-aarch64-3.9/cvxopt/_version.py to '1.3.0'
      running build_ext
      building 'base' extension
      creating build/temp.linux-aarch64-3.9
      creating build/temp.linux-aarch64-3.9/src
      creating build/temp.linux-aarch64-3.9/src/C
      aarch64-linux-gnu-gcc -pthread -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O2 -Wall -g -ffile-prefix-map=/build/python3.9-PN012d/python3.9-3.9.2=. -fstack-protector-strong -Wformat -Werror=format-security -g -fwrapv -O2 -g -ffile-prefix-map=/build/python3.9-PN012d/python3.9-3.9.2=. -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2 -fPIC -I/usr/src/Simod/venv/include -I/usr/include/python3.9 -c src/C/base.c -o build/temp.linux-aarch64-3.9/src/C/base.o
      aarch64-linux-gnu-gcc -pthread -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O2 -Wall -g -ffile-prefix-map=/build/python3.9-PN012d/python3.9-3.9.2=. -fstack-protector-strong -Wformat -Werror=format-security -g -fwrapv -O2 -g -ffile-prefix-map=/build/python3.9-PN012d/python3.9-3.9.2=. -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2 -fPIC -I/usr/src/Simod/venv/include -I/usr/include/python3.9 -c src/C/dense.c -o build/temp.linux-aarch64-3.9/src/C/dense.o
      src/C/dense.c: In function ‘Matrix_NewFromPyBuffer’:
      src/C/dense.c:277:28: warning: ‘n.d’ may be used uninitialized in this function [-Wmaybe-uninitialized]
        277 |        MAT_BUFD(a)[cnt] = n.d;
            |                           ~^~
      aarch64-linux-gnu-gcc -pthread -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O2 -Wall -g -ffile-prefix-map=/build/python3.9-PN012d/python3.9-3.9.2=. -fstack-protector-strong -Wformat -Werror=format-security -g -fwrapv -O2 -g -ffile-prefix-map=/build/python3.9-PN012d/python3.9-3.9.2=. -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2 -fPIC -I/usr/src/Simod/venv/include -I/usr/include/python3.9 -c src/C/sparse.c -o build/temp.linux-aarch64-3.9/src/C/sparse.o
      aarch64-linux-gnu-gcc -pthread -shared -Wl,-O1 -Wl,-Bsymbolic-functions -Wl,-z,relro -g -fwrapv -O2 -Wl,-z,relro -g -fwrapv -O2 -g -ffile-prefix-map=/build/python3.9-PN012d/python3.9-3.9.2=. -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2 build/temp.linux-aarch64-3.9/src/C/base.o build/temp.linux-aarch64-3.9/src/C/dense.o build/temp.linux-aarch64-3.9/src/C/sparse.o -lm -llapack -lblas -o build/lib.linux-aarch64-3.9/cvxopt/base.cpython-39-aarch64-linux-gnu.so
      /usr/bin/ld: cannot find -llapack
      /usr/bin/ld: cannot find -lblas
      collect2: error: ld returned 1 exit status
      error: command '/usr/bin/aarch64-linux-gnu-gcc' failed with exit code 1
      [end of output]

  note: This error originates from a subprocess, and is likely not a problem with pip.
error: legacy-install-failure

× Encountered error while trying to install package.
╰─> cvxopt

note: This is an issue with the package mentioned above, not pip.
hint: See above for output from the failure.

bpmtk.jar au.edu.unimelb.services.ServiceProvider SMD wrong usage

@Mcamargo85

On macOS 11.4 with java:

java version "15.0.1" 2020-10-20
Java(TM) SE Runtime Environment (build 15.0.1+9-18)
Java HotSpot(TM) 64-Bit Server VM (build 15.0.1+9-18, mixed mode, sharing)

running python simod_console.py with a new java flag -Xms1024M instead of -Xss8G (see #13 for details) fails for another reason now:

 -- Mining Process Structure --
args ['java', '-Xmx2G', '-Xms1024M', '-cp', 'external_tools/splitminer3/bpmtk.jar:external_tools/splitminer3/lib/*', 'au.edu.unimelb.services.ServiceProvider', 'SMD', '0.5', '0.7', 'false', 'false', 'false', 'outputs/20210701_47B5EF4E_5884_472B_8C5F_CB8372D35250/PurchasingExample.xes', 'outputs/20210701_47B5EF4E_5884_472B_8C5F_CB8372D35250/PurchasingExample']
TESTCODE - SMD
ERROR: wrong usage.
RUN> java -cp bpmtk.jar;lib\* au.edu.unimelb.services.ServiceProvider SMD e n p 'logpath\log.[xes|xes.gz|mxml]' 'outputpath\outputname' 
PARAM: e = double in [0,1] : parallelism threshold (epsilon)
PARAM: n = double in [0,1] : percentile for frequency threshold (eta)
PARAM: p = [true|false] : replace non trivial OR joins?
EXAMPLE: java -cp bpmtk.jar;lib\* au.edu.unimelb.services.ServiceProvider SMD 0.1 0.4 .\logs\SEPSIS.xes.gz .\outputs\SEPSIS
java.lang.NoClassDefFoundError: javax/xml/bind/DatatypeConverter
        at org.deckfour.xes.util.XsDateTimeConversion.parseXsDateTime(XsDateTimeConversion.java:96)
        at org.deckfour.xes.util.XsDateTimeConversionJava7.parseXsDateTime(XsDateTimeConversionJava7.java:129)
        at org.deckfour.xes.in.XesXmlParser$XesXmlHandler.startElement(XesXmlParser.java:292)
        at java.xml/com.sun.org.apache.xerces.internal.parsers.AbstractSAXParser.startElement(AbstractSAXParser.java:518)
        at java.xml/com.sun.org.apache.xerces.internal.parsers.AbstractXMLDocumentParser.emptyElement(AbstractXMLDocumentParser.java:183)
        at java.xml/com.sun.org.apache.xerces.internal.impl.XMLDocumentFragmentScannerImpl.scanStartElement(XMLDocumentFragmentScannerImpl.java:1386)
        at java.xml/com.sun.org.apache.xerces.internal.impl.XMLDocumentFragmentScannerImpl$FragmentContentDriver.next(XMLDocumentFragmentScannerImpl.java:2725)
        at java.xml/com.sun.org.apache.xerces.internal.impl.XMLDocumentScannerImpl.next(XMLDocumentScannerImpl.java:605)
        at java.xml/com.sun.org.apache.xerces.internal.impl.XMLDocumentFragmentScannerImpl.scanDocument(XMLDocumentFragmentScannerImpl.java:541)
        at java.xml/com.sun.org.apache.xerces.internal.parsers.XML11Configuration.parse(XML11Configuration.java:888)
        at java.xml/com.sun.org.apache.xerces.internal.parsers.XML11Configuration.parse(XML11Configuration.java:824)
        at java.xml/com.sun.org.apache.xerces.internal.parsers.XMLParser.parse(XMLParser.java:141)
        at java.xml/com.sun.org.apache.xerces.internal.parsers.AbstractSAXParser.parse(AbstractSAXParser.java:1224)
        at java.xml/com.sun.org.apache.xerces.internal.jaxp.SAXParserImpl$JAXPSAXParser.parse(SAXParserImpl.java:635)
        at java.xml/com.sun.org.apache.xerces.internal.jaxp.SAXParserImpl.parse(SAXParserImpl.java:324)
        at java.xml/javax.xml.parsers.SAXParser.parse(SAXParser.java:197)
        at org.deckfour.xes.in.XesXmlParser.parse(XesXmlParser.java:175)
        at com.raffaeleconforti.log.util.LogImporter.importFromInputStream(LogImporter.java:191)
        at com.raffaeleconforti.log.util.LogImporter.importFromFile(LogImporter.java:155)
        at au.edu.unimelb.services.ServiceProvider.SplitMinerService(ServiceProvider.java:522)
        at au.edu.unimelb.services.ServiceProvider.main(ServiceProvider.java:96)
Caused by: java.lang.ClassNotFoundException: javax.xml.bind.DatatypeConverter
        at java.base/jdk.internal.loader.BuiltinClassLoader.loadClass(BuiltinClassLoader.java:606)
        at java.base/jdk.internal.loader.ClassLoaders$AppClassLoader.loadClass(ClassLoaders.java:168)
        at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:522)
        ... 21 more

Is it the problem with SMD e n p? Even though settings dict contains valid number, 0.5 and 0.7.

False positive in discovery with model

When I try to discover simulations parameters in the "discover" mode with a .bpmn model that contains one activity less than the ones in the log, I am getting an Index error from the replayer. Note that the error does not occur with the "optimize" mode. In the link I am attaching the modified model and log.

Deep copy of None

Hi, @Mcamargo85,

It seems that the lines below do a deep copy of None for self.org_log_train and self.org_log_valdn. Is it correct? self.log_train and self.log_valdn are undefined in __init__().

https://github.com/AdaptiveBProcess/Simod/blob/31acdb14223297fc798b1d4903e0960250141e0a/opt_structure/structure_optimizer.py#L74-L75

New simulator JSON-schema change

Simulator changes its JSON schema, so we need to implement the new one, AutomatedProcessImprovement/prosimos-frontend#6, when it is implemented in the simulator

Sample is at https://owncloud.ut.ee/owncloud/index.php/s/9FP8jMXPEoPtbtT

Remove gateway probabilities from Time optimizer

Currently, the Gateway Probabilities are being optimized in the Structure optimization (1st optimization phase) and in the Time optimization (2nd optimization phase). The evaluation metric used in the Time optimization does not take into account the activity sequence, so the gateway probabilities are actually not being highly considered there.

Refactor:

• Remove the gateway probability method from the Time optimizer search space.
• Pass the gateway probability method from the Structure optimizer to the Time optimizer (in this way, the output of the Structure optimization phase is the BPMN model and its branching probabilities).
• When SIMOD receives a BPMN model, the Structure optimizer is only optimizing the gateway probability method (so its search space is only this).
• If SIMOD does not receive the BPMN model, it works as usual, but not optimizing the gateway probabilities in the Time optimization (second phase).

Remove pm4py dependency

We agreed to return back or implement CSV support across all modules and get rid of XES, because XML makes logs much more heavy and increases memory consumption.

• Wrapper package for pm4py-related functionality under GPL
• Usage of the wrapper in Simod
• Rewriting LogReader to accept CSV We're switching from XES to CSV gradually: calendars, split miner.

Remove QBP/BIMP simulator

Review .deepcopy() use

Review the code to check if the use of deepcopy() function is mandatory. Remove all unnecessary uses to improve memory use performance.

Hypothesis: we don't need to alter the event log, so we shouldn't need to take copies of it.

Introduce extraneous time

Setup benchmarking without DISPLAY environment variable

Docker version has a solution for running the Java dependency without a separate display server. We need to setup benchmarking this way too.

Add Sonarcloud to CI

As a developer,
I want to know if my new contribution to Simod degrades the code quality,
So that I know what to fix.

Allow Simod to specify output directory

As a user, I want to specify a directory for output files, so that I can put them to any other folder and even a mounted disk.

Notes:

• At the moment, there's the output flag in Configuration, but it doesn't work as expected.

Problem at runtime in windows Simod v2.0.0

I am a student who tried to run the stable version of Simod from sometime back to try and see the results of the simulation on the input logs. However, I faced the following issue (see screenshot).

It does not seem to be an installation error, but something machine or OS dependent where it cannot read the output file produced during execution. I have also tried installing the new version of Simod which again gives me an error. Is there an existing solution to this problem available?

Thanks.

Add option to specify a test log from a file

Add a new parameter in the configuration file to specify a XES/CSV file to test SIMOD.

If this parameter is specified:

• The default input log is not split into training/test at the start of the general pipeline.
• The default input log is used as training log.
• The specified test log is used to test the final solution at the end of the general pipeline (when evaluating the final model before finishing).
• The training log will still be split into training/validation inside each of the optimizer pipelines (structure and calendars).

If it is not specified:

• The default input log is split into training/test at the start of the general pipeline.
• The training part of this log is split into training/validation inside each of the optimizer pipelines (structure and calendars).

Simod HTTP service

As a docker user,
I want to be able to call Simod container from another container as an HTTP service,
so that I can integrate Simod into a container-based system.

Remove time.sleep from parallel processing in the utilities module

This line, https://github.com/AutomatedProcessImprovement/Simod/blob/master/src/simod/utilities.py#L143, contributes to about 15% of running time of Simod.

Trace alignment when no inclusiveGateways are present in a model

Read BPMN model only once and keep it in memory

Currently, the BPMN model, either if provided or discovered, is being read each iteration where it is needed.

We can slightly improve the performance if:

• If the model is provided, read it once at the start of the SIMOD pipeline and keep it in memory for all iterations of Structure and Time optimizers.
• If the model is not provided, read the BPMN from the best result of Structure optimizer, and keep it in memory for all iterations of Time Optimizer.

Create configuration template with explanations

Provide an example of the configuration file for SIMOD with all the existent parameters and comments with explanations of each of them.

# Path to the log to train SIMOD
train_log_path = BPIC_2012_train.xes
# Path to the log to test SIMOD once all the optimization is finished
#  - Optional: if not provided, SIMOD will split the training log into train/test to perform this final testing
test_log_path = BPIC_2012_test.xes
# Blablabla
more_parameters_1 = 1
more_parameters_2 = 1

TimeTablesCreator should take train data

Make TimeTablesCreator to take not the original log_path, but the path to the train data in the XES-format.

I didnt found the file simod_ui.py how to execute?

Hi you all,
Congrats for the project!, how can i execute simod? I cant find the file simod_ui.py :-)

Integration of calendar discovery from Prosimos into Simod

As a user, I want to discover with Simod

• a calendar for the whole log
• a calendar per resource pool
• a calendar per resource

so that the simulator can use this information for creating more realistic simulations.

Acceptance criteria

• For this input log, (a) one resource should get a calendar with Mondays and Tuesdays, and (b) the other with Wednesdays and Thursdays.

Notes

See AutomatedProcessImprovement/Prosimos@f49493d#diff-8012307e5e4e7ba00a358b30d266a72700d4bad9e385c9a8fd7c947a28a7f0f8 for a usage example.

Statement seems to have no effect

Hey, @Mcamargo85,

Does next mean continue in this for-loop?

https://github.com/AdaptiveBProcess/Simod/blob/31acdb14223297fc798b1d4903e0960250141e0a/support_modules/writers/xes_writer.py#L101

Move Java dependencies to a container or a service

Consider moving Java or other language dependencies into containers or services.

NB: Computing cluster must be able to call the container or service. (HPC at UT doesn't allow to call Docker, but allows to use Singularity. Maybe a service can be also a solution)

Cost table for simulation

Introduce Cost Table (cost per hour, resource amount) for a resource, pool or default table for everybody in Simod to use it with Prosimos.

A user should be able to specify such a table in the configuration file.