mlrun/mlrun

MLRun is an open source MLOps platform for quickly building and managing continuous ML applications across their lifecycle. MLRun integrates into your development and CI/CD environment and automates the delivery of production data, ML pipelines, and online applications.

mlrun's Issues

Add pagination to SQL DB

Queries might return a lot of answers (run...), this will choke the UI. We'd like to add pagination to some queries in the database.

Probably change the return value to have (total_count, results)

Add copyright to .py files in `examples` directory

Docker for HTTP DB with gunicorn

Write a docker to run the HTTP DB with gunicorn.

Currently ignore the fact that the underlying filedb is not thread/process safe.

Consider switching to asyncio HTTP server

Currently we're using gunicorn for parallel requests. However this means we have several processes and it causes some sync problems.

Look at some asynio web server and see if they work.

Auto generate function documentation

We'd like a bi-directional way to communicate help Python <-> YAML

Python -> YAML

Parse function docstring and populate YAML from it

YAML -> Python

Have a way to view help in the function object in Jupyter. Probably populate __doc__ from the YAML

Unite database tests

Currently we have different tests for filedb and httpdb. We should have the same test suite that runs on both. Probably with parametrized fixtures

AttributeError: 'NameConstant' object has no attribute 'id'

Happens when running this notebook

---------------------------------------------------------------------------
AttributeError                            Traceback (most recent call last)
<ipython-input-43-a34f02098586> in <module>
      1 # create an ML function from the notebook, attache it to iguazio data fabric (v3io)
----> 2 trainer = code_to_function(name='my-trainer', runtime='job', with_doc=True)
~/.pythonlibs/lib/python3.6/site-packages/mlrun/run.py in code_to_function(name, project, tag, filename, handler, runtime, kind, image, embed_code, with_doc)
    376 
    377     if with_doc:
--> 378         handlers = find_handlers(code)
    379         r.spec.entry_points = {h[name]: h for h in handlers}
    380     return r
~/.pythonlibs/lib/python3.6/site-packages/mlrun/funcdoc.py in find_handlers(code, handlers)
    209     visitor = ASTVisitor()
    210     visitor.visit(mod)
--> 211     funcs = [ast_func_info(fn) for fn in visitor.funcs]
    212     if handlers:
    213         return [f for f in funcs if f['name'] in handlers]
~/.pythonlibs/lib/python3.6/site-packages/mlrun/funcdoc.py in <listcomp>(.0)
    209     visitor = ASTVisitor()
    210     visitor.visit(mod)
--> 211     funcs = [ast_func_info(fn) for fn in visitor.funcs]
    212     if handlers:
    213         return [f for f in funcs if f['name'] in handlers]
~/.pythonlibs/lib/python3.6/site-packages/mlrun/funcdoc.py in ast_func_info(func)
    160 def ast_func_info(func: ast.FunctionDef):
    161     doc = ast.get_docstring(func) or ''
--> 162     rtype = func.returns.id if func.returns else ''
    163     params = [ast_param_dict(p) for p in func.args.args]
    164     defaults = func.args.defaults
AttributeError: 'NameConstant' object has no attribute 'id'

Config System

@yaronha

Add config class which can override defaults from a file (can potentially add env var options)

we can put the “default namespace” in it as an example (search for all places with ‘default-namespace’ string), will add more later base on it

We'd like at least two level in the configuration (e.g. http.port)

Add authentication to HTTP DB

We'd like to have options for basic auto and barrier token.

Honor iter=True/False in list_runs

Option to run tests against k8s cluster

Clean logging error from tests

At the end of the tests we see the following output. It doesn't fail the test but it's annoying.

--- Logging error ---
Traceback (most recent call last):
  File "/usr/local/lib/python3.6/logging/__init__.py", line 996, in emit
    stream.write(msg)
  File "/usr/local/lib/python3.6/site-packages/_pytest/capture.py", line 427, in write
    self.buffer.write(obj)
ValueError: I/O operation on closed file
Call stack:
  File "/usr/local/lib/python3.6/threading.py", line 884, in _bootstrap
    self._bootstrap_inner()
  File "/usr/local/lib/python3.6/threading.py", line 916, in _bootstrap_inner
    self.run()
  File "/usr/local/lib/python3.6/threading.py", line 864, in run
    self._target(*self._args, **self._kwargs)
  File "/mlrun/mlrun/db/periodic.py", line 35, in _schedule
    logger.info('running periodic task')
Message: 'running periodic task'
Arguments: ()

CI system

We'd like to have a CI system.

Add functions to the database

We'd like to store functions in the database. We're going to store YAML/JSON.

@yaronha said:

• YAML, metadata spec, labels
• Key is project + name + version tag
• put/get/list/delete
• search by project + name + labels

Add options to store logs on file system (vs in database)

Logs can get bit and inefficient in databases, have an optioin (environemnt variable?) to store them on disk instead, in the DB just the path probably.

Nuke mutable default values

In several places we function definitions like: def gen_list(items=[], tag='td'): which is bad.

Find all places in code and nuke them. The usual solution is:

def gen_list(items=None, tag='td'):
    items = [] if items is None else items
    ...

We can also consider adding flake8 and flake8-bugbear a pre-test step.

httpd/db artifacts broken

@tebeka the artifacts get/store etc is broken

a. you must use the artifact.to_json() vs dumps like in filedb, since to_json() accounts for model details, also body is not stored in the DB

b. artifacts object path should be /artifact/<project>/<uid/tag>/<key> (key not as arg)
, (uid/tag is like a git tree and can also list all keys in a tree, if user specify a tag i.e. not latest we can add the tag via args, but on get he doesnt need to pass tag as arg since get will be on either uid or tag never both, think like git hash & tag)

mlrun/mlrun/db/httpdb.py

Line 142 in 0935dda

def store_artifact(self, key, artifact, uid, tag='', project=''):

Support "outputs" in function documentation

Apart from what a function returns, we have outputs which are emitted via mlrun functions.
We'd like to have a way to document these outputs (in docstring) and have func_info and friends return them.

Fix EntryPoint to handle dict of types

See runtime/base.py TODO comment

Move test output outside source directory

Currently when runnig tests, we have some files left over the source directory, we'd like them to be create elsewhere.

Files/directories are:

• chart
• chart.html
• dask-worker-space/
• dataset.csv
• iteration_results.csv
• model.txt
• results.html
• tests/test_results/

SQL DB Errors

@yaronha said:

File "/mlrun/mlrun/db/httpd.py", line 275, in update_run
_db.update_run(data, uid, project, iter=iter)
File "/mlrun/mlrun/db/sqldb.py", line 145, in update_run
run = self._query(Run, uid=uid, project=project).one_or_none()
File "/usr/local/lib/python3.6/site-packages/sqlalchemy/orm/query.py", line 3303, in one_or_none
"Multiple rows were found for one_or_none()"
sqlalchemy.orm.exc.MultipleResultsFound: Multiple rows were found for one_or_none()
10.233.81.40 - - [03/Dec/2019 22:57:28] "PATCH /api/run/default/6d259e0c13854d11b6064c7a8acdd2c9?iter=0
HTTP/1.1" 500 -

looks like u didnt set the tabel keys properly, e.g. in runs the key is project+uid+iter
so every write gets a new ID (i.e. it add rows vs update the same row)
u also seem to ignore iter in read/store/update of runs

SQL rundb

Create SQL rundb. When it's ready - have the HTTP server use it.

Options to run HTTP DB

We'd like to have:

• A docker image that runs the HTTP DB server
• Command line executable (mlrun-db)
• Command line option for port

Upload documentation to readthedocs.org

Need to create sphinx documentation and have a trigger on development branch

Add examples/mlrun_basics.ipynb to tests

Persistancy for cron jobs

We'd like to be able to reload schedules after restart

Support user quota & permissions

We'd like to have users and have quotas & premissions per user

Have config populate on import

Since some people would use the SDK and we don't want them calling config.populate() manually (they'll forget/miss the docs). We'd like to change config.populate() to re-run instead of being a no-op the second time.

This way we'll have automatic way and can also manually re-populate. Probably rename the method to reload then

Cron scheduler

Have cron like scheduler to run tasks

funcdoc errors

Try looking at https://github.com/yjb-ds/demos/blob/models/lightgbm/lgbm-kfp.ipynb

Move HTTP routes under prefix

Move the HTTP DB routes under a prefix (such as /api/v1). We'd like to be able to add UI on top of it as well

Environment configuarion at CircleCI

We'd like to pass a k8s (and maybe other) configuration files to CircelCI. Probably need to encode them as JSON in an environment vriable.

@handler decorator

We'd like to have @handler decorator. This will allow use to know which function are handler and which ones are utility.

@handler should get some optional parameters.

HTTP DB backend

@yaronha said:

add an HTTP DB backend, see https://github.com/v3io/mlrun/tree/master/mlrun/db

need to implement a client based on the base class which proxy (map the api to HTTP requests) to a remote server (which can run the FileDB class behind a restful API)

Nuke f-strings

We need to support Python 3.5, no f-strings :(

Switch to using sqldb

Test notebooks

We'd like to test some notebooks that uses mlrun.

Have the ability to get creds from the environment and if they exist run againt k8/nuclio dashboard.

Remove SQLAlchemy warnings

When running mlrun you see the following warning:

/home/miki/work/envs/mlrun/lib/python3.8/site-packages/sqlalchemy/ext/declarative/clsregistry.py:125: SAWarning: This declarative base already contains a class with the same class name and module name as mlrun.db.sqldb.Label, and will be replaced in the string-lookup table.

Make it go away :)

Replace ModelObj with attrs, traitlets or a like

Support function input sources

We like to have the user specify where some input parameters come from.

See

• kubeflow input types
• mlflow path type
• luigi
• pytest.mark.parametrize

Better HTTP server

Currently we're using flask development server as the HTTP server. For performance we might consider using gunicorn or uwsgi

However, once we run several Python process, we need to check the underlying filedb behaves well in concurrency.

Consider ditching pipenv

pipenv is nice, but there hasn't been a release since November 2018 and it doesn't seem to happen soon.

Add list_projects & list_artifact_tags(project)

The wevb API needs these

async build/submit API

We'd like to have build/submit return some kind of job ID and let the client poll on it

Test sql db in docker with gnuicorn

Move httpd out of "db" directory

It does more than just a backend to db

Replace requests with faster http client

requests is pretty slow doing HTTP calls.

Consider replacing it with pycurl or build-in urlopen

Upsert other types in SQLDB

We'd like upsert on Function & Artifact as well

Add periodic task to HTTP db

We'd like to have the HTTP db deamon run a periodic task. The task should be initialized and then run every duration

Support secrets encryption

We'd like the ability to encrypt using private/public key.

Basically a function that gets a dict and return the dict with all values encrypted.
Provide an example on how to generate the key (openssh?)