Installation issues

[anssi-fr]

Installation issues

[ab-anssi]

The URL to access SecuML web user interface is not localhost:5000, but localhost:5000/SecuML/.
Can you try with this URL and tell me whether it solves your problem ?

We should perhaps change the route to SecuML main menu to localhost:5000 to avoid this kind of issues.

[kination]

@ab-anssi thanks for quick response..but still got problems.
It shows only blank page, and could find error log like:

GET http://localhost:5000/static/lib/css/bootstrap.min.css 404 (NOT FOUND)
GET http://localhost:5000/static/lib/js/bootstrap.min.js 404 (NOT FOUND)
GET http://localhost:5000/static/lib/js/jquery.min.js 404 (NOT FOUND)
GET http://localhost:5000/static/lib/js/bootstrap.min.js 404 (NOT FOUND)
...

I couldn't find these files in project directory either. Am I missing something?

[ab-anssi]

You must download the css and js libraries. See the documentation.

You can download and save them in the right directories with the script SecuML/download_libraries.

[kination]

@ab-anssi thanks, I missed it!
But though I still have some problem...

  File "/Users/jung/workspace/etc/SecuML/venv/lib/python3.6/site-packages/SecuML-0.12-py3.6.egg/SecuML/web/views/experiments.py", line 60, in getProjects
    projects = db_tables.getProjects(session)
  File "/Users/jung/workspace/etc/SecuML/venv/lib/python3.6/site-packages/SecuML-0.12-py3.6.egg/SecuML/experiments/db_tables.py", line 199, in getProjects
    res = query.all()
...
sqlalchemy.exc.ProgrammingError: (_mysql_exceptions.ProgrammingError) (1146, "Table 'secuml_test.projects' doesn't exist") [SQL: 'SELECT projects.id AS projects_id, projects.project AS projects_project \nFROM projects'] (Background on this error at: http://sqlalche.me/e/f405)

My connection has permission for modify database. Could you check if there is table creating process?

[ab-anssi]

SecuML relies on sqlalchemy to create the tables of the database. Currently, only SecuML scripts (SecuML_DIADEM, SecuML_ILAB, SecuML_clustering, etc.) check whether the tables have been created. SecuML_server does not perform this check.

You can run SecuML_DIADEM on the lingspam dataset (provided in the repo). See Use Case: Spam Detection. Then, the display of the web page localhost:5000/SecuML/ should work.

This solution bypasses the real problem. I will update SecuML_server to create the DB tables if they don't exist.

[ab-anssi]

The commit Ease the use of SecuML solves some issues you have encountered and improves some error/warning messages.

If you still encounter some issues, feel free to report them. Your feedback allows to improve the documentation, and the error/warning messages.

[kination]

@ab-anssi thanks!
I just merged your commit to my repo, and it shows error like this

$ python SecuML/web/SecuML_server
 Some JS or CSS libraries are missing in SecuML/web/static/lib/. You can download them with the script download_libraries.

But in my project lib files are all included. Is there some way to debug this problem?

[ab-anssi]

If you launch SecuML in a virtualenv, you need to run python setup.py install after downloading the JS and CSS libraries (the libraries need to be copied into the virtualenv directory).

[kination]

@ab-anssi
Yes, I've tried to remove venv and re-installed, but it still causes same problem.

$ pwd
/path/to/project/SecuML/web/static/lib/js
$ ls -la
-rw-r--r--  1 jung  staff  159638 Jul 24 17:25 Chart.min.js
-rw-r--r--  1 jung  staff   48944 Jul 24 17:25 bootstrap.min.js
-rw-r--r--  1 jung  staff  236746 Jul 24 17:25 d3.min.js
-rw-r--r--  1 jung  staff   86927 Jul 24 17:25 jquery.min.js
$ pwd
/path/to/project/SecuML/web/static/lib/css
$ ls -la
total 352
drwxr-xr-x  4 jung  staff     128 Jul 24 17:24 .
drwxr-xr-x  5 jung  staff     160 Jul 24 17:27 ..
-rw-r--r--  1 jung  staff  144877 Jul 24 17:24 bootstrap.min.css
-rw-r--r--  1 jung  staff   30747 Jul 24 17:24 jquery-ui.min.css

Is there something wrong in my settings?

[ab-anssi]

Can you run the following commands in the main directory of SecuML:
virtualenv venv_SecuML --no-site-packages --python python3
source venv_SecuML/bin/activate
python3 setup.py install
python3 download_libraries
python3 setup.py install
SecuML_server
Does it work ?

If you still have the same error (Some JS or CSS libraries are missing in SecuML/web/static/lib/. You can download them with the script download_libraries.) you can have a look at the function checkWebLibraries (line 106 in the file SecuML/experiments/Tools/dir_exp_tools.py). You can print the variable directory (line 110). Do not forget to rerun python3 setup.py install to update the version in the virtualenv.
The directory should be something like /path/to/project/venv_SecuML/lib/python3.4/site-packages/SecuML-0.12-py3.4.egg/SecuML/web/static/lib/js. Then, you can check whether the js and css libraries have been properly copied into the directories of the virtualenv.

[kination]

@ab-anssi thanks! Seems this issue has been cleared.
This is the pattern I've done.

$ python3 -m venv venv
$ source venv/bin/activate
$ python setup.py install
$ python download_libraries
$ python SecuML/web/SecuML_server

One more:
I setup configuration as export SECUMLCONF=conf/SecuML_travis_conf.yml SecuML_DIADEM SpamHam lingspam LogisticRegression same as guide commented, but it shows warning as No experiment has been executed yet.
It seems I need to add something in input directory. Where can I find it?

Thanks.

[ab-anssi]

It is strange you have to run SecuML_server with the full path (python SecuML/web/SecuML_server). If you have installed SecuML properly in a virtualenv SecuML_server should work without specifying the path.

I setup configuration as export SECUMLCONF=conf/SecuML_travis_conf.yml SecuML_DIADEM SpamHam lingspam LogisticRegression same as guide commented, but it shows warning as No experiment has been executed yet.

Have you launched SecuML_DIADEM ? Does it complete without errors ? In this case, you should have the results of the experiment stored in the directory <output_data_dir>/SpamHam/lingspam/ (where <output_data_dir> is specified in the configuration file). Do you have files in this directory ?

If these files are there, everything should work. You can try to delete the cache of your browser, but you do not need to add anything in the input directory.

[kination]

• I run with full path because I started on root path, and there is no SecureML_server file in root.
• I run SecuML_DIADEM SpamHam lingspam LogisticRegression in command line, and it failed with:

Traceback (most recent call last):
  File "/Users/jung/workspace/etc/SecuML/venv/bin/SecuML_DIADEM", line 4, in <module>
    __import__('pkg_resources').run_script('SecuML==0.12', 'SecuML_DIADEM')
  File "/Users/jung/workspace/etc/SecuML/venv/lib/python3.6/site-packages/pkg_resources/__init__.py", line 658, in run_script
    self.require(requires)[0].run_script(script_name, ns)
  File "/Users/jung/workspace/etc/SecuML/venv/lib/python3.6/site-packages/pkg_resources/__init__.py", line 1438, in run_script
    exec(code, namespace, namespace)
  File "/Users/jung/workspace/etc/SecuML/venv/lib/python3.6/site-packages/SecuML-0.12-py3.6.egg/EGG-INFO/scripts/SecuML_DIADEM", line 33, in <module>
    logger=logger)
  File "/Users/jung/workspace/etc/SecuML/venv/lib/python3.6/site-packages/SecuML-0.12-py3.6.egg/SecuML/experiments/Classification/ClassificationExperiment.py", line 53, in __init__
    parent, logger, create)
  File "/Users/jung/workspace/etc/SecuML/venv/lib/python3.6/site-packages/SecuML-0.12-py3.6.egg/SecuML/experiments/Experiment.py", line 42, in __init__
    self._setSession(session, project, dataset)
  File "/Users/jung/workspace/etc/SecuML/venv/lib/python3.6/site-packages/SecuML-0.12-py3.6.egg/SecuML/experiments/Experiment.py", line 189, in _setSession
    self.session = self._initDatabase(project, dataset)
  File "/Users/jung/workspace/etc/SecuML/venv/lib/python3.6/site-packages/SecuML-0.12-py3.6.egg/SecuML/experiments/Experiment.py", line 210, in _initDatabase
    load_dataset.load(self.logger)
  File "/Users/jung/workspace/etc/SecuML/venv/lib/python3.6/site-packages/SecuML-0.12-py3.6.egg/SecuML/experiments/Data/Dataset.py", line 47, in load
    self.loadIdents()
  File "/Users/jung/workspace/etc/SecuML/venv/lib/python3.6/site-packages/SecuML-0.12-py3.6.egg/SecuML/experiments/Data/Dataset.py", line 73, in loadIdents
    db, cursor = db_tools.getRawConnection()
  File "/Users/jung/workspace/etc/SecuML/venv/lib/python3.6/site-packages/SecuML-0.12-py3.6.egg/SecuML/experiments/Tools/db_tools.py", line 68, in getRawConnection
    client_flags=[ClientFlag.LOCAL_FILES])
  File "/Users/jung/workspace/etc/SecuML/venv/lib/python3.6/site-packages/mysql_connector-2.1.4-py3.6-macosx-10.13-x86_64.egg/mysql/connector/__init__.py", line 179, in connect
  File "/Users/jung/workspace/etc/SecuML/venv/lib/python3.6/site-packages/mysql_connector-2.1.4-py3.6-macosx-10.13-x86_64.egg/mysql/connector/connection.py", line 95, in __init__
  File "/Users/jung/workspace/etc/SecuML/venv/lib/python3.6/site-packages/mysql_connector-2.1.4-py3.6-macosx-10.13-x86_64.egg/mysql/connector/abstracts.py", line 719, in connect
  File "/Users/jung/workspace/etc/SecuML/venv/lib/python3.6/site-packages/mysql_connector-2.1.4-py3.6-macosx-10.13-x86_64.egg/mysql/connector/connection.py", line 206, in _open_connection
  File "/Users/jung/workspace/etc/SecuML/venv/lib/python3.6/site-packages/mysql_connector-2.1.4-py3.6-macosx-10.13-x86_64.egg/mysql/connector/network.py", line 418, in open_connection
mysql.connector.errors.InterfaceError: 2002: Can't connect to local MySQL server through socket '/var/run/mysqld/mysqld.sock' (2 No such file or directory)

Is it because there is no file? Where can I find it?

Thanks.

[ab-anssi]

SecuML does not manage to connect to your mysql database called secuml_test.
Are you sure your DB URI (db_uri: 'mysql+mysqldb://root@localhost/secuml_test') is right ?
Have you installed a mysql server, and created a secuml_test database ?

SecuML_server and SecuML_DIADEM connect to the database exactly the same way. Do you have the same error when you run SecuML_server ?

[ab-anssi]

SecuML relies on the mysql-connector library to connect to the database. So your URI should start with mysql+mysqlconnector://.

[ab-anssi]

The last commit Remove raw connections to databases should solve your problem.

[kination]

@ab-anssi thanks for your help. It seems almost close.
Now popup not occur, and could see menu Classification. But when I go into classification, I found error message in browser.

I also run SecuML_DIADEM SpamHam lingspam LogisticRegression in command line, and it saids:

...
File "/Users/jung/workspace/etc/SecuML/venv/lib/python3.6/site-packages/SecuML-0.12-py3.6.egg/SecuML/experiments/Classification/ClassificationExperiment.py", line 87, in generateDatasets
    instances = InstancesFromExperiment(self).getInstances()
  File "/Users/jung/workspace/etc/SecuML/venv/lib/python3.6/site-packages/SecuML-0.12-py3.6.egg/SecuML/experiments/InstancesFromExperiment.py", line 32, in getInstances
    features, features_names, features_descriptions = self.getFeatures()
  File "/Users/jung/workspace/etc/SecuML/venv/lib/python3.6/site-packages/SecuML-0.12-py3.6.egg/SecuML/experiments/InstancesFromExperiment.py", line 55, in getFeatures
    features = self.experiment.getAllFeatures()
  File "/Users/jung/workspace/etc/SecuML/venv/lib/python3.6/site-packages/SecuML-0.12-py3.6.egg/SecuML/experiments/Experiment.py", line 119, in getAllFeatures
    with open(csv_file, 'r') as f:
FileNotFoundError: [Errno 2] No such file or directory: '/Users/jung/workspace/etc/SecuML/venv/lib/python3.6/site-packages/SecuML-0.12-py3.6.egg/input_data/SpamHam/lingspam/SpamHam/lingspam/features/features.csv'

Looks like something is missing. Where can I find this file?

[ab-anssi]

The error message in the web browser is due to the SECUML_DIADEM error. The DIADEM experiment has not completed, so the browser cannot display the results (the experiment should be deleted when it has failed to complete to avoid this kind of display error).

To solve the problem with SecuML_DIADEM, you need to update the configuration file.
input_data_dir and output_data_dir must contain absolute paths. I will update the doc !

[kination]

@ab-anssi
Okay, I initiated project, change path as absolute value, and start as:

$ python3 -m venv venv
$ source venv/bin/activate
$ python setup.py install
$ python download_libraries
$ python SecuML/web/SecuML_server

It shows there is no data. Then:

$ SecuML_DIADEM SpamHam lingspam LogisticRegression

this shows:

  File "/Users/jung/workspace/etc/SecuML/venv/lib/python3.6/site-packages/SecuML-0.12-py3.6.egg/SecuML/experiments/Classification/ClassificationExperiment.py", line 60, in run
    datasets = self.generateDatasets()
  File "/Users/jung/workspace/etc/SecuML/venv/lib/python3.6/site-packages/SecuML-0.12-py3.6.egg/SecuML/experiments/Classification/ClassificationExperiment.py", line 87, in generateDatasets
    instances = InstancesFromExperiment(self).getInstances()
  File "/Users/jung/workspace/etc/SecuML/venv/lib/python3.6/site-packages/SecuML-0.12-py3.6.egg/SecuML/experiments/InstancesFromExperiment.py", line 32, in getInstances
    features, features_names, features_descriptions = self.getFeatures()
  File "/Users/jung/workspace/etc/SecuML/venv/lib/python3.6/site-packages/SecuML-0.12-py3.6.egg/SecuML/experiments/InstancesFromExperiment.py", line 55, in getFeatures
    features = self.experiment.getAllFeatures()
  File "/Users/jung/workspace/etc/SecuML/venv/lib/python3.6/site-packages/SecuML-0.12-py3.6.egg/SecuML/experiments/Experiment.py", line 119, in getAllFeatures
    with open(csv_file, 'r') as f:
FileNotFoundError: [Errno 2] No such file or directory: '/Users/jung/workspace/etc/SecuML/input_data/SpamHam/lingspam/features/features.csv'

Then run server again:

$ python SecuML/web/SecuML_server

It shows there is a data, but cannot find details.

Am I still missing something?

[ab-anssi]

As regards your screen shot, it is normal you cannot display the results of the experiment since it has not completed successfully.

The DIADEM error says that the file /Users/jung/workspace/etc/SecuML/input_data/SpamHam/lingspam/features/features.csv does not exit. The gitHub repository has a file named input_data/SpamHam/lingspam/features/features.csv. Are you sure your input_data_dir in SecuML_conf.yml is right ?

[kination]

@ab-anssi yeah...it's my mistake. I think I removed in my side. Sorry for confusing.
One more issue...it stucks like below with SecuML_DIADEM SpamHam lingspam LogisticRegression

$ SecuML_DIADEM SpamHam lingspam LogisticRegression
/Users/jung/workspace/etc/SecuML/venv/lib/python3.6/site-packages/scikit_learn-0.19.0-py3.6-macosx-10.13-x86_64.egg/sklearn/model_selection/_split.py:597: Warning: The least populated class in y has only 1 members, which is too few. The minimum number of members in any class cannot be less than n_splits=4.
  % (min_groups, self.n_splits)), Warning)

After this look occurs, and few seconds later, python causes crash:

Could you check this?

[ab-anssi]

The warning Warning: The least populated class in y has only 1 members, which is too few. The minimum number of members in any class cannot be less than n_splits=4. % (min_groups, self.n_splits)), Warning) does not make python crash. When I run SecuML on debian, this warning often appears, but it has never made python crash.

According to your screen shot, the problem is caused by libBLAS (Basic Linear Algebra Subprograms) that is extensively used by scikit-learn. And SecuML is largely based on this library to train the Machine Learning models !

The sklearn issue seems very similar to your problem. You could reopen this issue that has not been solved yet.

Besides, the sklearn documentation indicates that MAC OS may crash when n_jobs > 1.
SecuML sets n_jobs = -1 to use all the cores available.
You can try to update the file SecuML/core/Classification/Classifier.py, line 177, and set n_jobs = 1. This way, you won't parallelize the computations, but it may avoid the crash. Do not forget to rerun python setup.py install, or python setup.py develop, after modifying the source code.

I close this issue, since now it is not an installation problem. If your problem remains you can open an issue "Execution errors on MAC OS", but the problem is more related to sklearn than to SecuML.

[kination]

@ab-anssi okay, seems installation issue is cleared, as you said. I'll open new one if I found something.
Thanks for help!