ucbrise/flor

🌻 FlorFlow: Flor, now with Dataflow

flor's Issues

[Yield Expressions] Failure to transform some files that contain nested yield expressions in some statements

Sample Culprit (in [PYTHON_LIB]/bokeh/server/session.py):

def _needs_document_lock(func):
    @gen.coroutine
    def _needs_document_lock_wrapper(self, *args, **kwargs):
        if self.destroyed:
            log.debug("Ignoring locked callback on already-destroyed session.")
            raise gen.Return(None)
        self.block_expiration()
        try:
            ###################################
            with (yield self._lock.acquire()):
            ###################################
                if self._pending_writes is not None:
                    raise RuntimeError("internal class invariant violated: _pending_writes " + \
                                       "should be None if lock is not held")
                self._pending_writes = []
                try:
                    result = yield yield_for_all_futures(func(self, *args, **kwargs))
                finally:
                    pending_writes = self._pending_writes
                    self._pending_writes = None
                for p in pending_writes:
                    yield p
            raise gen.Return(result)
        finally:
            self.unblock_expiration()
    return _needs_document_lock_wrapper

A YIELD expression exits the function like a RETURN statement. Unlike a RETURN statement, on the next call of the same function, execution resumes at the code location immediately following the YIELD expression.

Function exits and entries are relevant events that Flor must capture.

Thus, we expand YIELD expressions into multi-line statements. Statements that record the event that the function is about to exit, the yield expression, and then statements that record the event that execution has resumed within the same function at the following instruction: YIELD_EXPRESSION -> PREFIX_STATEMENTS YIELD_EXPRESSION SUFFIX_STATEMENTS

The problem arises when expanding an expression inside a WITH statement into many statements: with (expression) -> with (statement1; statement2; ...; statementN)

The solution requires us to "flatten" the many statements so that only the yield expression appears in the with statement, the prefix and suffix statements then appear before the while statement and inside the with statement's body:

statement1;
statement2;
...
with (yield expression):
    statement(i)
    ...
    statementN

We already solved this problem for ASSIGN statements, and we have now encountered it for WITH statements. The solutions should be similar.

It is likely that YIELD expressions will appear in more statements that ASSIGN and WITH. This bug should be resolved such that a YIELD expression will be properly flattened if they appear in any valid expression context.

[Context] Get more context from the run

On Exec, automatically capture OS/System information and machine statistics:

• MAC address from the machine
• If EC2 instance, get instance type
• User infromation from Git
• System's userid
• Reliable timestamp from network server

OS walk not guaranteed to walk directory clones in same order

flor/flor/complete_capture/walker/walker.py

Line 113 in d51cbb2

for ((src_root, _, _), (dest_root, dirs, files)) in zip(os.walk(self.rootpath), os.walk(self.targetpath)):

Pytorch warning: torch.distributed.reduce_op is deprecated, please use torch.distributed.ReduceOp instead

When running any pytorch code, this warning occurs even if the code does not have any calls to torch.distributed. Something is happening during/because of flor transformation. This can possible affect execution of a pytorch script; so far only one example has failed me.

Clean commit message

The log is written to the commit message and is lossless compressed, but it still can be too long, interfering with natural interaction data scientists with expect from git log. Rather than write the compressed log in the commit message, compute a valid and lossy summary of the log with constant size and write that summary in the commit message instead.

[pyflor_install] Could not find environment bug

The Flor installer needs to copy and transform an anaconda environment.
This installer will take some time to complete.
Continue with installation [Y/n]? Y
Enter the source anaconda environment [base]:
Enter the Python version of the source anaconda environment [3.7]:
Enter the name of the new anaconda environment [flor]:
Traceback (most recent call last):
  File "/anaconda3/envs/ml/bin/pyflor_install", line 11, in <module>
    load_entry_point('pyflor', 'console_scripts', 'pyflor_install')()
  File "/Users/bobby/Dropbox/Berkeley/RISELab/flor/flor/__main__.py", line 130, in install
    conda.cli.python_api.run_command('create', '--name', conda_flor_env, '--clone', base_conda)
  File "/anaconda3/envs/ml/lib/python3.7/site-packages/conda/cli/python_api.py", line 85, in run_command
    raise e
  File "/anaconda3/envs/ml/lib/python3.7/site-packages/conda/cli/python_api.py", line 81, in run_command
    return_code = args.func(args, p)
  File "/anaconda3/envs/ml/lib/python3.7/site-packages/conda/cli/main_create.py", line 68, in execute
    install(args, parser, 'create')
  File "/anaconda3/envs/ml/lib/python3.7/site-packages/conda/cli/install.py", line 211, in install
    clone(args.clone, prefix, json=context.json, quiet=context.quiet, index_args=index_args)
  File "/anaconda3/envs/ml/lib/python3.7/site-packages/conda/cli/install.py", line 59, in clone
    src_prefix = context.clone_src
  File "/anaconda3/envs/ml/lib/python3.7/site-packages/conda/base/context.py", line 345, in clone_src
    return locate_prefix_by_name(self, self._argparse_args.clone)
  File "/anaconda3/envs/ml/lib/python3.7/site-packages/conda/base/context.py", line 743, in locate_prefix_by_name
    raise CondaEnvironmentNotFoundError(name)
conda.exceptions.CondaEnvironmentNotFoundError: Could not find environment: base .
You can list all discoverable environments with `conda info --envs`.

Break statement is not logged

Support Interactive Environment

Flor works well once a syntactically-valid python script is available. It works well for the tuning/fitting scenario.

When first composing a pipeline, data scientists will want immediate feedback: line-by-line, as can be provided by iPython or Jupyter notebooks. This enables data scientists to catch bugs early, and better understand the objects they are working with. A data scientist working in this manner will not advance to the next step in the ML pipeline until assured the previous steps "work" (even if they don't provide good results).

Flor needs to support the composition scenario too. Flor should support interactive environments, and the user should be able to move smoothly between interactive and non-interactive environments.

[Flor Dependency Isolation] Flor cannot use a flor-transformed version of Pandas

Every Flor dependency (see requirements.txt) should be loaded from the base environment and not the flor environment.

If Flor loads a transformed library, it logs its own work (which is irrelevant to the user) and intermingles it with the user's logs.

Moreover, the semantics of Flor using Flor-transformed libraries to log user code get confusing and lead to bugs or unspecified behavior such as the one below.

The fix will require some Conda acrobatics: all user code loads Flor-transformed libraries. All Flor code (transitively) loads un-transformed base libraries.

Ignore SigChld causes issues on distributed code

Ignore SigChld was a way of preventing zombie child processes. However, it causes issues with multi-gpu RoBERTa training. Possible solution involves writing custom signal handler for child processes.

[PyCharm Plugin] Publish Flor Highlight Extension

A plugin has been developed to enable Code Annotation from an IDE (PyCharm).

After further development and testing, deploy the PyCharm plugin for public use, and add instructions to the documentation detailing its use.

A jar file of the PyCharm Plugin in the current version is attached.

flor_highlight_plugin-1.0.9.jar.zip

Incremental Flor Transformations

Flor currently transforms an anaconda environment snapshot at the time of installation.

If in the future, a user installs a new package, or updates an existing package, the base environment and the flor environment will diverge.

Flor should run logic on install/update events of conda libraries to keep the environments in sync.

Import statements not supported in Pytorch

Running Flor on Pytorch in the cloud results in Pytorch complaining about Flog import statements.

[Conda Site-Packages] Transformer changing the semantics of some programs

This bug requires further investigation to identify the root cause.

On execution of the standard iris Scikit-Learn example, Flor produces the following error:

(base) ➜  iris_loop git:(master) flor python iris_raw.py complete_iris
Traceback (most recent call last):
  File "/anaconda3/envs/flor/lib/python3.7/site-packages/flor/commands/flython.py", line 38, in exec_flython
    spec.loader.exec_module(module)
  File "<frozen importlib._bootstrap_external>", line 728, in exec_module
  File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
  File "/Users/rogarcia/sandbox/iris_loop/iris_raw.py", line 60, in <module>
    from sklearn import datasets
  File "/anaconda3/envs/flor/lib/python3.7/site-packages/sklearn/datasets/__init__.py", line 23, in <module>
    from .twenty_newsgroups import fetch_20newsgroups
  File "/anaconda3/envs/flor/lib/python3.7/site-packages/sklearn/datasets/twenty_newsgroups.py", line 39, in <module>
    from ..feature_extraction.text import CountVectorizer
  File "/anaconda3/envs/flor/lib/python3.7/site-packages/sklearn/feature_extraction/__init__.py", line 9, in <module>
    from . import text
  File "/anaconda3/envs/flor/lib/python3.7/site-packages/sklearn/feature_extraction/text.py", line 18, in <module>
    from ..preprocessing import normalize
  File "/anaconda3/envs/flor/lib/python3.7/site-packages/sklearn/preprocessing/__init__.py", line 6, in <module>
    from .data import Binarizer
  File "/anaconda3/envs/flor/lib/python3.7/site-packages/sklearn/preprocessing/data.py", line 8, in <module>
    from scipy import stats
  File "/anaconda3/envs/flor/lib/python3.7/site-packages/scipy/stats/__init__.py", line 344, in <module>
    from .stats import *
  File "/anaconda3/envs/flor/lib/python3.7/site-packages/scipy/stats/stats.py", line 171, in <module>
    from . import distributions
  File "/anaconda3/envs/flor/lib/python3.7/site-packages/scipy/stats/distributions.py", line 3, in <module>
    from . import _continuous_distns
  File "/anaconda3/envs/flor/lib/python3.7/site-packages/scipy/stats/_continuous_distns.py", line 113, in <module>
    class norm_gen(rv_continuous):
  File "/anaconda3/envs/flor/lib/python3.7/site-packages/scipy/stats/_continuous_distns.py", line 175, in norm_gen
    `optimizer` argument is ignored.\n\n""")
  File "/anaconda3/envs/flor/lib/python3.7/site-packages/scipy/misc/doccer.py", line 159, in _doc
    start_of_notes = cls_docstring.find(notes_header)
AttributeError: 'NoneType' object has no attribute 'find'
'NoneType' object has no attribute 'find'

None of the files listed are to blame for the fault. This was confirmed by manually replacing the files mentioned in the ErrorMessage with the pre-transformed library files. The error still appeared.

The error message disappeared when we reverted scipy/stats/_distn_infrastructure.py with the pre-transformed version of the file. This means the re-written file is executing functions without generating an error locally, and changing the semantics of the program. Because the culprit file is so large, we have not been able to analyze it and identify the AST-Transformer bug.

On inspection of the log generated by the flor-execution of the iris example, we noticed that rv_continuous._updated_ctop_param and rv_generic._construct_argparser appeared frequently in the log records.

The current patch involves reverting scipy/stats/_distn_infrastructure.pyto the pre-transformed version. The error disappears, but the root cause has not been resolved.

Pass in function arguments from the terminal

The formal parameters of a function should be configurable from the terminal. For example, if we have a function defined as:

@flor.track
def fit_and_score_model(gamma, C, test_size, random_state):
    ...

Then when we run the script from the terminal, the input arguments should be flaggable:

python iris.py --gamma 0.001 --C 100.0 --test_size 0.15 --random_state 42

If the function argument names are the same across functions, we'll need some way to disambiguate the names. A possible solution is to prefix the function name in the flag, much like in SQL we would prefix the name of the table when a column name appears in more than one table in the FROM clause.

Add versioning page to rtd

A page about how Flor does automatic version control.

1. User commits and Flor commits are co-mingled
2. Interpreting the Flor commit messages
3. Writing useful .gitignore files
4. Useful Git shortcuts for forking interactions

[pyflor_install] Erroneous for multiple users

Current pyflor_install is correct for single user, must generalize for multiple users.

`log` should not be a reserved word

Currently the flor log parser relies on log being a reserved word. This can lead to confusion since a user may want to use mylog = flor.log instead, a name that the parser would miss.

Infer the names of Log Records

What is the name and value of the following log record?

{
      "assignee": [
        "X_tr",
        "X_te",
        "y_tr",
        "y_te"
      ],
      "caller": "train_test_split",
      "from_arg": true,
      "in_execution": "main",
      "in_file": "/Users/rogarcia/git/flor/examples/logger/basic.py",
      "instruction_no": 32,
      "keyword_name": "test_size",
      "pos": null,
      "runtime_value": 0.2,
      "typ": "param",
      "value": "x"
    }

Arguably, the name is test_size. Definitely, the value is 0.2.

It may be necessary to infer the names of log records for the purposes of flattening logs into a denormalized table with the log record name as the column, and a log record value in the field for every iteration. It will also be useful to infer the name of a log record for the purpose of visualization or summaries.

update link on Berkeley Site

https://rise.cs.berkeley.edu/projects/jarvis/

the gitlab points to jarvis

Make the log records more user-friendly

Make sure the fields in the log records are as user-friendly as possible. For example:

• "assignee" -> "variable_name"
• "instruction_no" -> "line_no"

Also, make sure that the fields of a log record are properly documented here

Add support for monitoring

This issue is an epic, meaning it needs to be decomposed into numerous issues first.

The user should be able to monitor the metrics of the experiment as it runs.

Elaborate on the CFG page in RTD

After #15, update the CFG page in RTD. And Document the meaning of each field of a LOG_RECORD.

Plot the Flor Plan

This issue is an epic, meaning it needs to be decomposed into numerous issues first.

Users should be able to view a plot of the Flor Plan. The visualization should be bi-directionally interactive and support visual queries. So, the user should be able to click on objects in the visualization and see the corresponding information in the JSON log that is responsible for that object having been plotted. Also, the user should be able to highlight pieces of the JSON log and map them to the visualization to help with navigation.

Multiple flor-contexts in the same workload

Stream Log Output

The log is currently constructed in memory in its entirety before being serialized to disk. This serialization method is not fault tolerant (if the program terminates abnormally all progress is lost) and the current implementation choice may add a major performance penalty (we should confirm and quantify the extent on the penalty on the fib.py example).

To stream the log output we must make one of the following adjustments:

1. If the program terminates abnormally, identify the missing exit-tokens (}, ]), and fill them in. We may also need to delete lines or take other compensatory measures if the emitted log contains a "corrupted" tail, so the final log is valid (though perhaps incomplete -- reflecting the progress of the partial execution). The tail of a log may be corrupted if it is the substring of some element in the CFG: for example, an incomplete log-record or a left curly-brace.
2. Avoid filling in missing exit-tokens by streaming objects into a sequential intermediate format that is similar to JSON, but not proper JSON. On a second pass, transform the intermediate format into proper JSON. A challenge will be to include the nesting information as a field in the log records, such as a field called "nesting_level".

If the program terminates normally, the first option is superior because it doesn't need log post-processing.

Filename alone is not unique in python. Path is unique.

flor/flor/interface/input/execution_tracker.py

Line 103 in 2286a9d

if filename not in os.listdir(secret_dir):

Error when instantiating flor.log

Hi. I'm trying to reproduce the iris example from the docs. I've run into an error when installing using pip3. Here's the code

Luigis-MBP:flor luigi$ docker run --rm -ti python:3.6 bash
root@4587c4a9e79b:/# pip3 install pyflor
Collecting pyflor
  Downloading https://files.pythonhosted.org/packages/59/ca/c90313a0ae1b6d773efb1108b31277d2c605dd16066fd70c63381ae1fc81/pyflor-0.0.1a0-py3-none-any.whl
Installing collected packages: pyflor
Successfully installed pyflor-0.0.1a0
root@4587c4a9e79b:/# python
Python 3.6.8 (default, Feb  6 2019, 12:07:20) 
[GCC 6.3.0 20170516] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import flor
Coming soon!

>>> flor.log
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
AttributeError: module 'flor' has no attribute 'log'
>>> 

When I install from source, I am able to call flor.log.

Add support for Tensorboard

This issue is an epic, meaning it needs to be decomposed into numerous issues first.

[pyflor_install] Add code to clean uninstall

OS walk transforming already-transformed libraries

flor/flor/complete_capture/walker/walker.py

Line 49 in fd87452

def compile_tree(self, lib_code=True):

Query the log

This issue is an epic, meaning it needs to be decomposed into numerous issues first.

Users should be able to query the Flor logs over time and across experiments.

[Analysis Across Runs] Propagate annotations across runs and versions

[bashrc] On pip install of Flor, optionally modify user's shell config file

We want to automatically append the following function to the user's shell configuration file. Different users prefer different shells.

    flor() {
            conda activate flor;
            pyflor $@;
            cd $(pwd);
            conda deactivate;
    }

Error when importing the OpenLog module

After I installed the newest version successfully, when running the examples in the file iris.py, it raised some error.

Traceback (most recent call last):
  File "iris.py", line 4, in <module>
    from flor import OpenLog
ImportError: cannot import name 'OpenLog'

I noticed that the newest version just released yesterday, is the module OpenLog done? pls clarify. Thx.

[pyflor_install] Error when running pyflor_install

Traceback (most recent call last):
  File "/anaconda3/envs/ml/lib/python3.7/site-packages/conda/common/configuration.py", line 42, in <module>
    from cytoolz.dicttoolz import merge
ModuleNotFoundError: No module named 'cytoolz'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/anaconda3/envs/ml/bin/pyflor_install", line 10, in <module>
    sys.exit(install())
  File "/anaconda3/envs/ml/lib/python3.7/site-packages/flor/__main__.py", line 65, in install
    import conda.cli.python_api
  File "/anaconda3/envs/ml/lib/python3.7/site-packages/conda/cli/python_api.py", line 9, in <module>
    from ..base.context import context
  File "/anaconda3/envs/ml/lib/python3.7/site-packages/conda/base/context.py", line 21, in <module>
    from ..common.configuration import (Configuration, LoadError, MapParameter, PrimitiveParameter,
  File "/anaconda3/envs/ml/lib/python3.7/site-packages/conda/common/configuration.py", line 47, in <module>
    from .._vendor.toolz.functoolz import excepts
  File "/anaconda3/envs/ml/lib/python3.7/site-packages/conda/_vendor/toolz/functoolz.py", line 467
    f.__name__ for f in reversed((self.first,) + self.funcs),
    ^
SyntaxError: Generator expression must be parenthesized

All the arguments to a tracked function should appear in the log

Even if they are not explicitly logged in the function body.

Improve function invocation interface

Suppose a user defines a flor-tracked function:

@flor.track
def fit_and_score_model(gamma, C, test_size, random_state):
    ...

To run this function, there must exist a flor Context in the callable top-level python script:

with flor.Context('iris'):
    fit_and_score_model(...)

We need to be able to support the invocation of many functions from within the flor Context:

with flor.Context('iris'):
    v = fit_and_score_model(...)
    foo(v)

Additionally, and especially in the cases where the user only wants to run a single function (such as "main") it should be possible to omit the flor Context clause and call the function from the terminal:

python iris.py --context iris --function fit_and_score_model 
--gamma 0.001 --C 100.0 --test_size 0.15 --random_state 42

Bug in subtree copy for parsing

flor/flor/interface/input/execution_tracker.py

Line 80 in 2286a9d

root_dir = os.path.dirname(inspect.getsourcefile(f))

Copy tree rooted at common ancestor of all invocable Flor-decorated functions and the active Flor Context.

Scenario:

• flor.Context() in ~/bin/driver.py
• main() in ~/src/code/main.py

Then must copy (relevant) subtree rooted at ~.

[ETL] Incorrect output for annotations of parameters with no keyword specified

By "annotations of parameters with no keyword specified ", I mean clf.fit(GET("X_tr", X_tr), y_tr) as opposed to clf.fit(X=GET("X_tr", X_tr), y_tr)

Adding the following annotation to the iris example

clf.fit(GET("X_tr", X_tr), y_tr)

generates:

	X_tr
0	SVC(C=100.0, cache_size=200, class_weight=None, coef0=0.0, decision_function_shape='ovr', degree=3, gamma=0.1, kernel='rbf', max_iter=-1, probability=False, random_state=None, shrinking=True, tol=0.001, verbose=False)
1	SVC(C=100.0, cache_size=200, class_weight=None, coef0=0.0, decision_function_shape='ovr', degree=3, gamma=0.01, kernel='rbf', max_iter=-1, probability=False, random_state=None, shrinking=True, tol=0.001, verbose=False)
2	SVC(C=100.0, cache_size=200, class_weight=None, coef0=0.0, decision_function_shape='ovr', degree=3, gamma=0.001, kernel='rbf', max_iter=-1, probability=False, random_state=None, shrinking=True, tol=0.001, verbose=False)

[Full Conda ETL] Python programs' control flow inconsistent with our model

The current version of Flor replays executions by scanning Complete Execution Traces (a.k.a logs).

To succeed, Flor must reconstruct and maintain the stack frame at each log record or program point, which is relevant for modeling name scopes (and other things).

To track the stack frame at any point in the program, we log function entry / exit points.

Initially, we had a naïve model of Python execution: a function is entered by invocation and exits by reaching a return statement or the end of the block; additionally, the function returns control to the caller. This view overlooked yield statements as a means of exiting a function and re-entering them at a point other than the first line in the function body. Moreover, we overlooked raise statements as a valid and common means of returning control to an ancestor of the caller --- not just the immediate caller.

In light of these findings, we caught and fixed the bugs in the Flor Scanner (ETL).

However, after wrapping even more Python libraries (scipy, pandas, etc), we are again witnessing a bug that is similar to the bugs we saw when our execution model misaligned with Python control flow: it is likely that our model of Python Control is incorrect.

The exception reads as follows:

(base) ➜  iris_loop git:(master) flor etl bugfx iris_h.py
Traceback (most recent call last):
  File "/anaconda3/envs/flor/bin/pyflor", line 11, in <module>
    sys.exit(main())
  File "/anaconda3/envs/flor/lib/python3.7/site-packages/flor/__main__.py", line 38, in main
    exec_flan(args)
  File "/anaconda3/envs/flor/lib/python3.7/site-packages/flor/commands/flan.py", line 46, in exec_flan
    consolidated_scanner.scan_log()
  File "/anaconda3/envs/flor/lib/python3.7/site-packages/flor/log_scanner/scanner.py", line 123, in scan_log
    self.scan(log_record)
  File "/anaconda3/envs/flor/lib/python3.7/site-packages/flor/log_scanner/scanner.py", line 89, in scan
    log_record['end_function'], ctx.func_ctx)
AssertionError: For log record 48260 ... Expected: getargspec_no_self, Actual: _updated_ctor_param

The log record for recreating the bug in the future is available here: log.json.bz2.zip

[Bug] FlorPlan file is not in ExperimentGraph.starts

ExperimentGraph.starts contains all other literals and artifacts (namely, those expressed in the DAG). But because the FlorPlan is what described the DAG, it is not explicitly named.

from ... import * tried to import flog

Some wildcard imports raise AttributeErrors on flor exec, e.g.

File "/Users/bobby/miniconda3/envs/flor/lib/python3.7/site-packages/scipy/sparse/linalg/__init__.py", line 3, in <module>
    from .isolve import *
AttributeError: module 'scipy.sparse.linalg.isolve' has no attribute 'flog'

Bug is caused by some transformed library code having the following structure:

from flor import Flog
if Flog.flagged():
    flog = Flog()
...
__all__ = [s for s in dir() if not s.startswith('_')]

with dir() returning all names in the current scope and having flog included in __all__.

Help the user write .gitignore files

The user decides which files in the directory we automatically version or not (on every execution of a Flor experiment) with a .gitignore file. Since this file can be very important, we should provide some assistance in writing it.

1. We could provide default files
2. We could have a bash or python script that asks the user questions and uses the answers to generate a .gitignore file.
3. Some combination of the above

What if the user says we should ignore JSON files but Flor emits JSON logs that are necessary for its proper functioning.

• Create a special subdirectory that is exempt from the rules in the .gitignore file: every file in that subdirectory is versioned, regardless of the contents of .gitignore .

Consolidate ClientTransformer and LibTransformer

Flor doesn't work with functions with arguments :(

Flor won't work with functions with arguments.

---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
<ipython-input-12-ba0e1c3e4b78> in <module>
     13     epochs=10,
     14     validation_split=0.2,
---> 15     embedding_size=100
     16 ):
     17 

~/flor/flor/interface/input/execution_tracker.py in track(f)
    103     if filename not in os.listdir(secret_dir):
    104         # Needs compilation
--> 105         with open(tru_path, 'r') as sourcefile:
    106             tree = ast.parse(sourcefile.read())
    107             logger.debug(astor.dump_tree(tree))

TypeError: 'OutStream' object is not callable

from sklearn.model_selection import train_test_split
from sklearn.preprocessing import LabelBinarizer

from keras.preprocessing.text import Tokenizer
from keras.models import Sequential
from keras.layers import Flatten, Dense, Embedding, Activation, Dropout, LSTM, GlobalAveragePooling1D

@flor.track
def main(
    sample_size=1000, 
    vocab_size=1000, 
    batch_size=32, 
    epochs=10, 
    validation_split=0.2,
    embedding_size=100
):
    # From Series to lists
    groups = [p for p in patents['group_id']][0:sample_size]
    num_labels = len(set(groups))
    summaries = [s for s in patents['summary']][0:sample_size]

    # Test/train split, summary sequences as X, CPC groups as Y
    train_sums, test_sums, train_labels, test_labels = train_test_split(
        summaries,
        groups
    )

    label_count = len(set(groups))

    # Tokenize the summaries
    tokenizer = Tokenizer(num_words=vocab_size)
    tokenizer.fit_on_texts(summaries) 
    x_train = tokenizer.texts_to_matrix(train_sums, mode='tfidf')
    x_test = tokenizer.texts_to_matrix(test_sums, mode='tfidf')

    encoder = LabelBinarizer()
    encoder.fit(groups)
    y_train = encoder.transform(train_labels)
    y_test = encoder.transform(test_labels)

    print('Label count: {}'.format(label_count))
    print('x_train shape:', x_train.shape)
    print('x_test shape:', x_test.shape)
    print('y_train shape:', y_train.shape)
    print('y_test shape:', y_test.shape)

    model = Sequential()

    # Original
    # model.add(Embedding(vocab_size, embedding_size, input_length=vocab_size))
    # #model.add(Dropout(0.2))
    # model.add(Flatten())
    # model.add(Dense(512, input_shape=(vocab_size,)))
    # model.add(Activation('relu'))
    # # model.add(Dense(64, input_shape=(vocab_size,)))
    # # model.add(Activation('relu'))
    # model.add(Dense(label_count, input_dim=(vocab_size,)))
    # model.add(Activation('softmax'))

    model.add(Embedding(vocab_size, embedding_size, input_length=vocab_size))
    model.add(Dropout(0.2))
    model.add(Flatten())
    model.add(Dense(512, input_shape=(vocab_size,)))
    model.add(Activation('relu'))
    model.add(Dropout(0.2))
    model.add(Dense(label_count, input_dim=(vocab_size,)))
    model.add(Activation('sigmoid'))

    model.compile(
        loss='categorical_crossentropy',
        optimizer='adam',
        metrics=['accuracy']
    )
    model.summary()

    history = model.fit(
        x_train, 
        y_train, 
        batch_size=batch_size, 
        epochs=epochs, 
        verbose=1, 
        validation_split=validation_split
    )

    score = model.evaluate(
        x_test,
        y_test,
        batch_size=batch_size, 
        verbose=1
    )
    print('Test accuracy:', score[1])

with flor.Context('patent'):
    main(sample_size=100)

Installing flor on Ubuntu Xenial 16.04

Are there any specs defined in terms memory, disk and cpu requirement to setup flor on Ubuntu Xenial Machine.

Stack overflow when running Pytorch/Tensorflow

Running Tensorflow on the cloud and Pytorch on MacOS results in stack overflow errors. It is possible this is due to infinite recursion created during transformation.

import * brings someone else's flog into scope

Especially a flog whose writer has been closed. How to fix shadowing issue. Could give each variable a unique name.

Support flor-tracking of classes and methods

Currently we only support flor-tracking python functions. We should extend Flor to support Classes and class methods.