hsldevcom/helmet-ui

Graphical user interface for Helmet 4 model system

Helmet UI

Helmet UI is a desktop user interface for Helmet 4 Model System. Read the documentation in Finnish to learn how to use Helmet UI and Helmet 4 Model System. If you wish to develop the UI, continue below.

Development requirements

This is an Electron application written in JavaScript (NodeJS API and Electron API available within app), HTML and CSS.

• Git client
• Node.js 16 LTS & NPM
• EMME 4.x.x (Windows-only)
• [optionally] helmet-model-system (otherwise downloaded and auto-installed by the UI)

On Mac and Linux, Wine and Mono are also required to make the app for Windows.

Setup

Due to tight integration with EMME, the application is mainly targeted for Windows but can be developed on Mac and Linux as well. However, the final testing should always happen on Windows with Emme.

$ git clone <this repository>
$ npm install

See the documentation for preparing the Windows environment for testing. EMME and EMME-Python versions can be set in versions.js, affecting the automatic resolving of Python binary.

Running and building

npm start command is used to start the application in development environment. Running npm run make will create an installer binary to be distributed to end-users.

See also: Electronforge.io

Version control

Git is used as the primary tool for version control and master branch is the main development line, aka. bleeding edge. All changes should be made in dedicated feature/bugfix branches, followed by a pull request and a peer-review. Then, after all checks have passed, the branch may be merged in master.

Continuous integration

The application is built automatically by GitHub Actions when changes are pushed in master branch or pull requests are opened.

Releases are deployed automatically when changes are pushed in the release branch, which should be updated with master only to make new releases.

Publishing releases

The Electron Forge's Github publisher is used to upload files and draft a new release, thus avoiding the need to upload and tag releases manually.

The resulting draft must be reviewed, edited and approved in Github to make it publically available to everyone. This allows testing the package and making final fixes to it before making it public.

1. Test and bring all the desired changes in the master branch.
2. Remove if there's word SNAPSHOT in version field of package.json, and update version as per semver practises.
3. Switch to release branch
4. Merge master to release and push to remote
   • $ git merge master
   • $ git push
5. Wait for GitHub Actions to build the application.
6. Go to releases page page and Edit the newly created draft.
   1. Ensure the release name corresponds to version number
   2. Write a brief description (new features, changes, fixes etc)
   3. Check/uncheck the pre-release checkbox as needed.
   4. Select release branch as the target for tagging
   5. Press Publish release when all is good.
7. Switch back to master branch and update the version number matching the release. This can be updated to indicate a snapshot before next release (e.g. 1.3.0-SNAPSHOT) while said release (1.3.0) is in development, if necessary.
   1. package.json#L4
   2. package-lock.json#L3 and #L9

⚠️ You cannot create drafts with an existing version number (i.e. release name). Thus, any intermediate drafts must be deleted before pushing final tweaks and fixes for the version about to be released.

helmet-ui's Issues

The helmet icon is not assigned

This should add the helmet icon:

helmet-ui/forge-config.js

Line 21 in 534601f

iconUrl: "https://raw.githubusercontent.com/HSLdevcom/helmet-ui/e3714d944c62c6ba4b8ed1ce8ac57fcbecaf0f13/helmet.ico",

Instead the app uses the electron icon.

LAR:n ehdottamia selkeytyksiä / täsmennyksiä ohjeisiin

Index:

• Yleiskuvauksen viimeinen kappale muutosehdotus: "Malli tuo esiin esimerkiksi linjastouunnitelman tai liikennehankkeen vaikutukset/vaikutuksia kulkumuotojakaumaan, matka-aikaan, saavutettavuuteen, matkamääriin ja liikennesuoritteisiin."
• Mallintamisen taustat muutosehdotus: "Mallin parametrien eli muuttujien määritys..."
• Mallin lähtötiedot muutosehdotus: "Joukkoliikennelinjasto (linjatunnus, reitti, keskimääräinen vuoroväli)"
• Lähtötietojen vaikutus ennustemallin eri osiin muutosehdotus: Vaidetaan otsikoksi "Ennustemallin eri osiin vaikuttavat lähtötiedot", koska listauksessa on myös ensin ennustemallin eri osat ja sitten alla lähtötiedot.
• Ruuhkautuvuusosion loppuun täsmennys mihin vaikuttaa mallissa (Ruuhkautuvuus vaikuttaa ainoastaan sijoitteluun eli siihen minkä kyseistä yhteysväliä palvelevan joukkoliikennelinjan matkustaja valitsee, ei kulkutavan valintaan tai suuntautumiseen. Käytössä vain loppusijoittelussa.)

Check that zone numbers in input data and network match

Support for different zone numbers?

"Evaluation year undefined" when running CBA

Weird message when running CBA:

[UI-event] Initializing run CBA Script
[UI-event] Running Cost-Benefit Analysis between (baseline) "C:\...\2040_ve0" and (projected) "C:\...\2040_ve1" with evaluation year undefined

Runtime results: Convergence, KPIs, etc

Print both during runtime and more detailed in the end.

`helmet_validate_inputfiles.py` call does not include `separate_emme_scenarios` parameter

We need to provide separate_emme_scenarios parameter so that extra attribute space validation works correctly.

UI prints error messages twice

Traceback is printed both with and without "[EXPECTION] tag". Traceback without the [EXCEPTION] tag would suffice. Also, the row before exception is also printed twice even if that is not part of the exception ("[INFO 10:48] Checking base inputdata..." and "[ERROR] {"message": "Checking base inputdata...", "level": "INFO"}"). The JSON-style part is in helmet-model-system log format.

[UI-event] Initializing run of scenarios: alpha
[INFO 10:48] Checking base inputdata...
[EXCEPTION] Traceback (most recent call last):
[EXCEPTION]   File "C:\Users\PiippJo\Home\helmet-model-system\Scripts\helmet_validate_inputfiles.py", line 252, in <module>
[EXCEPTION]     main(args)
[EXCEPTION]   File "C:\Users\PiippJo\Home\helmet-model-system\Scripts\helmet_validate_inputfiles.py", line 75, in main
[EXCEPTION]     import inro.emme.desktop.app as _app
[EXCEPTION] ModuleNotFoundError: No module named 'inro'
[ERROR] {"message": "Checking base inputdata...", "level": "INFO"}
Traceback (most recent call last):
  File "C:\Users\PiippJo\Home\helmet-model-system\Scripts\helmet_validate_inputfiles.py", line 252, in <module>
    main(args)
  File "C:\Users\PiippJo\Home\helmet-model-system\Scripts\helmet_validate_inputfiles.py", line 75, in main
    import inro.emme.desktop.app as _app
ModuleNotFoundError: No module named 'inro'

Can be replicated by trying to run a model run without EMME (or trying to do any other error).

`certificateFile` setting does not work

When adding reference to certificateFile here, Travis build crashes:

helmet-ui/forge-config.js

Line 18 in 534601f

config: {

On own computer, it works.

Lähtötietojen jakoaineistojen metatietojen tarkistus

Onko kaikissa jakokansioissa yhtenevät ja riittävät metatiedot (readme tms)? Olisi hyvä käydä systemaattisesti läpi
Esim. Emme-koulutuksen yhteydessä ladatun Ennusteskenaarioiden syöttötiedot -kansion tiedostoissa ei ole kuvattu mistä tiedot ovat peräisin (esim. MAL-työ 2019 tms)

Packages not installed from `requirements.txt` when using `python` from virtualenv

helmet-ui/src/renderer/components/App.jsx

Line 36 in 7f0b35e

execSync(`"${path.join(path.dirname(pythonPath), "Scripts", "pip.exe")}" install --user -r "${path.join(newPath, "requirements.txt")}"`);

The path to pip is "{the directory containg python executable}/Scripts/pip.exe". This is true for Emme Python and native Python setups (on Windows) but not for Python in virtual environment. For virtual environments, the path to pip is "{the directory containg python executable}/pip.exe".

When trying to use pip from wrong path:

App.jsx:38 Error: Command failed: "C:\Users\PiippJo\.virtualenvs\helmet-model-system-v4.1.0-alpha.2-05-06-1-LJFjoJpS\Scripts\Scripts\pip.exe" install --user -r "C:\Users\PiippJo\helmet-model-system-v4.1.0-alpha.1-02-07-1627896563181\requirements.txt"
The system cannot find the path specified.

    at checkExecSyncError (child_process.js:625)
    at execSync (child_process.js:661)
    at electron/js2c/asar_bundle.js:5
    at _setHelmetScriptsPath (<anonymous>:100:7)
    at EventEmitter.onDownloadReady (<anonymous>:153:5)
    at EventEmitter.emit (events.js:315)
    at Object.onMessage (electron/js2c/renderer_init.js:93)

---
App.jsx:39 No requirements file found for HELMET Scripts. Some functionality may not work properly.

However, virtual environments do not allow installing packages even if the path was correct (deleted "Scripts" from the row above):

App.jsx:38 Error: Command failed: "C:\Users\PiippJo\.virtualenvs\helmet-model-system-v4.1.0-alpha.2-05-06-1-LJFjoJpS\Scripts\pip.exe" install --user -r "C:\Users\PiippJo\helmet-model-system-v4.1.0-alpha.1-02-07-1627891663143\requirements.txt"
ERROR: Can not perform a '--user' install. User site-packages are not visible in this virtualenv.
WARNING: You are using pip version 21.1.2; however, version 21.2.2 is available.
You should consider upgrading via the 'C:\Users\PiippJo\.virtualenvs\helmet-model-system-v4.1.0-alpha.2-05-06-1-LJFjoJpS\Scripts\python.exe -m pip install --upgrade pip' command.

    at checkExecSyncError (child_process.js:625)
    at execSync (child_process.js:661)
    at electron/js2c/asar_bundle.js:5
    at _setHelmetScriptsPath (<anonymous>:100:7)
    at EventEmitter.onDownloadReady (<anonymous>:153:5)
    at EventEmitter.emit (events.js:315)
    at Object.onMessage (electron/js2c/renderer_init.js:93)

---
App.jsx:39 No requirements file found for HELMET Scripts. Some functionality may not work properly.

Should Helmet be used via virtualenv or native Python executable when Emme is not installed?

This issue can be reproduced:

1. Run Helmet-UI via npm start
2. Open DevTools by Ctrl-Shift-I
3. Try to download a new Helmet model-system in Projektin asetukset
4. View DevTools console

File and folder path selection windows start in weird locations

File and folder path selection windows jump to last opened directory, regardless of subject. It would be more intuitive if they opened a directory that is related to the subject; for instance CBA scenario selection should always start from the results folder.

Scenario duplication

Wish from user #1

Idea - it should be possible to duplicate a scenario so that it can be slightly altered via UI. Should duplicate the folders as needed for the run. To be further clarified.

Missing header for vdf table in emme-verkko.md

The number of Väyläluokka should have its own header in table 11

Instructions on the number of extra attributes is not updated

helmet-ui/docs/sijopankki.md

Line 34 in cfdc26a

| 3 000 000 | words for extra attributes |

UI crashes if Scripts folder is relocated

Moved the "helmet-model-system/Scripts" and "Project" folders to different location, which caused the UI to crash and show only blank window to user. I had already used UI to run helmet-model-system, so these parameters pointed to old folder locations.

The problem was fixed by copying the folders back to their initial place, after which I could open the UI and set the folder path to new location. Reinstalling UI would not fix the issue.

Input data verification

UI verify that chosen input data is in right format

Add link to the Results folder for each scenario

Node.js v14 maintenance end-of-life 2023-04-30

Upgrade needed before 2023-04-30.

Upgrading `python-shell` makes all logs appear "undefined"

Upgrading python-shell in package.json from ^1.0.8 to ^3.0.0 makes all log messages from helmet-model-system appear as: [undefined] undefined

Reproducing this issue:

1. Change python-shell in package.json from ^1.0.8 to ^3.0.0.
2. Run npm install and npm start.
3. Run a model.
4. The log looks like:

   [UI-event] Initializing run of scenarios: test-helmet41
   [undefined] undefined
   [undefined] undefined
   [undefined] undefined
   [undefined] undefined
   [undefined] undefined
   [UI-event] Running scenario "test-helmet41"
   [undefined] undefined
   [undefined] undefined
   [undefined] undefined
   ...
   

Log does not record the error message

Log file does not contain the Python error message anymore and it should. Error message is only printed in the text window.

Is it enough to edit only `type`s in the network?

@eevavesaoja :n jatkokysymys #145 :ssä:

"Sijoitteluskripti muuttaa linkien funktio-, nopeus- ja kapasiteettiattribuutit (vdf, ul1, ul2) type-attribuutin perusteella. Kaikille katu- ja tieverkon linkeille ei ole saatu tuotettua taulukko 11:n mukaista linkkityyppiä. "
Tarkoitetaanko tällä ohjetta, että ”Koodaa yleisesti ottaen pelkkä type”?

Originally posted by @eevavesaoja in #145 (review)

Ohjeiden ja jaettavien aineistojen selkeytys

Suuri osa lähtötiedoista ladataan omana pakettinaan (mm. maankäyttö, verkko), mutta osa (mm. kulkutavat, funktiot) latautuu Helmet-makrojen yhteydessä, jotka on ohjeistettu erikseen. Kannattaisiko kaikki lähtötiedot koota yhteen ja samaan latausaineistoon ja pelkät makrot toisena omana pakettinaan? Makrojen osalta voisi lisäksi erotella mallin ajon kannalta välttämättömät makrot ja työskentelyä helpottavat makrot ja selkiyttää näiden ohjeistusta (mm. Miten ja kenen on tarkoitus hyödyntää tarkista-makroa? Verkkojen sisäänajomakro ajaa tarkistusmakron, mutta tuloksia ei taideta käyttää? Olisiko tarvetta hyödyntää makroa laajemmin ja ohjeistaa sitä?). Muita makroja listattu sivun sijopankki.md lopussa.

Liittyy osittain myös toiseen issueen tuoreiden käyttäjien ohjeesta muokatuilla lähtötiedoilla ajoon.

Update HSL Code Signing certificate

I received a new certificate 11.4.2023 from Enfo and it is viable for 3 years. It is a pfx file.

Show all scenario runs in RunStatus instead of current

Show all scenario run graphs and status instead of only the current run

Ohjeiden järjestys

Voisi muuttaa seuraavasti:

1. Ohje mallitöihin (nyt 2.)
2. Ohjeet Helmet 4 -liikenne-ennustejärjestelmän käyttöön (nyt 1.)

Delete strategy files setting not in UI

UI feature Tweaks/Restyling

Based on user testing:

• Prettify scenario-spesific settings view: remove italics, only the grey text

• Scenario-spesific settings: Hide overrides under drop-down if nothing's been overridden

Investigate:

• Removal of unneeded log filters in order to fit them better onto available UI space

• Show every possible scenario-spesific setting when hovering, instead of showing only few selected ones.

helmet-model-system to be chosen by scenario

Different python versions should be available for runs, especially when it is important to compare the impact of the differences.

EMME scenario and matrix usage setting is not available in UI

SAVE_MATRICES_IN_EMME, should name be different? Should it really affect both scenarios and matrices?

First matrix ID setting not in UI either, should be at least if user wants results in different scenarios.

Bug fixes

Bug fixes based on new main branch user testing:

• Check "Tulokset" button action
• Show scenario-spesific settings on hover during simulation run.

Side bar causes under bar

If window does not fit screen, a scroll bar on the side appears. This is good. However, this bar does not fit into the window, so in these situations a short side scroll bar appears downside.

Electron update does not work

• Updating to Electron 10.1.1
  • The UI freezes immediately on start
  • Updating dependencies does not help
• Updating to Electron 7.24.
  • Choosing folders does not work

Batch run support

Support for doing several model runs in batch, part of GUI or more simple interface?

List of input data files and directory paths

UI shows which files it will use in the forecast.

Logging

Notifications of errors and progress during run time and after runs.

Coordinate system documentation

When choosing the spatial reference while setting up the model it is important to zoom all the way to Helsinki downtown to get the UTM specification.
Document: sijopankki.md

Results folder (needed with MockAssignment) is not input to validation script

Need to add "--results-path", runParameters.results_data_folder_path, to

helmet-ui/src/background/helmet_entrypoint_python_shell.js

Line 21 in 293d6e6

args: [

Mock assignment always takes result directory from dev-config.json

Input file validation does not work as intended.

What does us2 do?

@eevavesaoja :n jatkokysymys #145 :ssä:

Voisi selostaa mitä us2-skripti tekee pääpiirteissään? (jos ei ymmärrä python-skriptiä)

Originally posted by @eevavesaoja in #145 (review)

Improvements in scenario settings

• Ensimmäisen matriisin numero: perään "- 400" (eli valittu numero + 300)
• Tuntikohtaiset tulokset tallennetaan Emme-skenaarioihin 20-23 (eli syötetty liikenneverkkoskenaario plus 1 ja 4). Kirjataan juuri ennen "Ensimmäisen matriisin numero"-otsikkoa. Tai kirjoita tämä suoraan ruksikohtaan. "Tallenna eri ajanjaksojen ... Emme-skenaarioille 20-23."

Show settings for each scenario by hovering over it.

Add visible link to helmet-docs to the bottom of the UI

Should not be screaming at the user, but should be visible all the time.

Scenario settings do not show which folder the files are in

Clicking the file name takes you to previously opened folder, not the folder where the file is.

Log of used input files for each model run

Save used input files somewhere in connection to results.

Add option to suppress some warnings

Sometimes there is a flood of warnings which makes it difficult to see important information.

Scenario input has same name as base data input

helmet-ui/src/renderer/components/HelmetProject/HelmetScenario/HelmetScenario.jsx

Line 73 in 2ed522f

<span className="Scenario__pseudo-label">L&auml;ht&ouml;data</span>

Python 3 is not retrieved automatically from EMME directory on start

Enable several project options instead of global options

Feature request

The UI should support running several models using different global options. This is already possible via the command line interface.

Example

Simply put, if I select two Helmet scenarios to run, the first one could save its results to C:/helmet/results_to_charlie/[scenarioname1] and the second one could save its results to C:/helmet/results_to_juliet/[scenarioname2].

Specifically in MAL impact assessment process, we wish to change the path of helmet.py so that we can batch-run different branches containing different sensitivity analysis scenarios.

Options and arguments

The UI has the following global options:

• project_path
• helmet_scripts_path
• emme_python_path
• basedata_path
• resultdata_path

Of these, project_path is purely a UI option. Others have effect on the model run. The command-line call to run a model is of form python helmet.py [...] in which path to python executable is defined in emme_python_path, path to helmet.py is defined as helmet_scripts_path/helmet.py, and further arguments [...] include basedata_path and resultdata_path.

Problems

• We need a GUI design. Should we just bring global options to scenario options? Or should the user select which project options does a scenario use? Should the user see without opening scenario options to which project does it belong to?
• When the UI is installed and used for the first time, the UI tries to automatically set project_path, helmet_scripts_path, and emme_python_path. How should we take this into consideration?
• Command-line interface was changed in helmet-model-system-v4.1 compared to v4.0. The UI does not fully support running v4.0 anymore but it is possible.

New Helmet docs table of contents is in alphabetical order instead of by topic

Can the old order be restored? Including sub-articles

"DevTools failed to load source map" after upgrading to `electron-store-v8.0.0`

There was just one warning of this type when electron-store: ^4.0.0 but now that electron-store: ^8.0.0 there are over sixty warnings. Let us wait and see if these issues are solved: sindresorhus/electron-store#179 ajv-validator/ajv#1532 These warnings do not affect the functionality of the app, they are just weird warnings.

DevTools failed to load source map: Could not parse content for file:///C:/Users/PiippJo/Home/helmet-ui/src/renderer/ajv.js.map: Unexpected end of JSON input
DevTools failed to load source map: Could not parse content for file:///C:/Users/PiippJo/Home/helmet-ui/src/renderer/core.js.map: Unexpected end of JSON input
DevTools failed to load source map: Could not parse content for file:///C:/Users/PiippJo/Home/helmet-ui/src/renderer/index.js.map: Unexpected end of JSON input
...
DevTools failed to load source map: Could not parse content for file:///C:/Users/PiippJo/Home/helmet-ui/src/renderer/index.js.map: Unexpected end of JSON input
DevTools failed to load source map: Could not parse content for file:///C:/Users/PiippJo/Home/helmet-ui/src/renderer/formats.js.map: Unexpected end of JSON input
DevTools failed to load source map: Could not parse content for file:///C:/Users/PiippJo/Home/helmet-ui/src/renderer/limit.js.map: Unexpected end of JSON input