Do Monitoring in a background process

[uptrain-ai]

Do Monitoring in a background process

[vipgupta]

@kanhaiyanathani How about this issue? Here, the problem is that the monitoring by the UpTrain framework is happening on the main thread (i.e., the thread used to make model predictions). Instead, we want the model predictions to happen on the main thread and monitoring to move to a background thread invisible to the user.

[kanhaiyanathani]

yes @vipgupta this also looks interesting, can you please help me in finding the file

1. where we are executing the monitoring function in the main thread
2. Any example file where we are running something on a background thread

[vipgupta]

1. Monitoring is currently always on the main thread (when we do framework.log, monitoring happens automatically on the main thread). Example, see the for loop in block 11: https://github.com/uptrain-ai/uptrain/blob/main/examples/2_cyber_attack_classification/uptrain_concept_drift.ipynb
2. I don't think we currently have any such case. @sourabhagr am I missing something?

[sourabhagr]

Yes, unfortunately, we don't an example where a background thread is implemented.

Essentially what needs to be done is below:

1. Block 11 in the file shared by Vipul, we have:
   # Log model inputs and outputs to monitor concept drift
   ids = framework.log(inputs=inputs, outputs=preds)
   As soon as the data is logged, the framework saves it in a database and does all the analysis (data drift checks, edge cases, etc.)
2. One way to do this could be - we have a background thread running. Whenever we do framework.log, it pushes the inputs to be logged to that background thread which in turn saves the data to DB and run all the checks.

Key benefit is that it would decouple observability from monitoring, resulting in faster and more reliable systems

[kanhaiyanathani]

Okay I can see here:

uptrain/uptrain/core/classes/framework.py

Line 382 in c4b0b10

def log(self, inputs=None, outputs=None, gts=None, identifiers=None, extra=None):

This function does many things like data anomaly checks, smart(edge) data addition for retraining and at last retraining model with smart data

1. Basically requirement is to run this function in a background process, right?
2. For multiple calls to framework.log(), should we create/call multiple process OR run in the same(single) process sequentially?
3. Am thinking of using multiprocessing package, do you want me to try any other package first?
4. which test/driver code I should refer in this repo for doing the latency benchmarking (before vs after)?

[kanhaiyanathani]

Hello @vipgupta, am trying to run test files
can you please provide the cmd to run setup.py?

[vipgupta]

Hi @kanhaiyanathani , to install uptrain package from local changes, run python setup.py install

To run the test files, go to the tests folder and run pytest .

[vipgupta]

Answers to your previous questions:

1. Yes!
2. That would be your choice. We just need to make sure that any unused threads are not lurking around and taking up valuable compute.
3. Again, your choice. I am not aware of the state-of-the-art here.
4. Write now we do not have any latency benchmarks. It's a work in progress :)

[kanhaiyanathani]

Hi @kanhaiyanathani , to install uptrain package from local changes, run python setup.py install

To run the test files, go to the tests folder and run pytest .

am getting below errors when I ran the python setup.py install cmd:

Compiling sklearn/utils/_vector_sentinel.pyx because it changed.
Compiling sklearn/utils/_isfinite.pyx because it changed.
Traceback (most recent call last):
  File "/Library/Developer/CommandLineTools/Library/Frameworks/Python3.framework/Versions/3.9/lib/python3.9/site-packages/setuptools/sandbox.py", line 156, in save_modules
    yield saved
  File "/Library/Developer/CommandLineTools/Library/Frameworks/Python3.framework/Versions/3.9/lib/python3.9/site-packages/setuptools/sandbox.py", line 198, in setup_context
    yield
  File "/Library/Developer/CommandLineTools/Library/Frameworks/Python3.framework/Versions/3.9/lib/python3.9/site-packages/setuptools/sandbox.py", line 259, in run_setup
    _execfile(setup_script, ns)
  File "/Library/Developer/CommandLineTools/Library/Frameworks/Python3.framework/Versions/3.9/lib/python3.9/site-packages/setuptools/sandbox.py", line 46, in _execfile
    exec(code, globals, locals)
  File "/tmp/easy_install-zo3n8i9w/scikit-learn-1.2.1/setup.py", line 669, in <module>
  File "/tmp/easy_install-zo3n8i9w/scikit-learn-1.2.1/setup.py", line 663, in setup_package
  File "/tmp/easy_install-zo3n8i9w/scikit-learn-1.2.1/setup.py", line 597, in configure_extension_modules
  File "/tmp/easy_install-zo3n8i9w/scikit-learn-1.2.1/sklearn/_build_utils/__init__.py", line 98, in cythonize_extensions
  File "/Users/kanhaiyanathani/Library/Python/3.9/lib/python/site-packages/Cython/Build/Dependencies.py", line 1088, in cythonize
    pool = multiprocessing.Pool(
  File "/Library/Developer/CommandLineTools/Library/Frameworks/Python3.framework/Versions/3.9/lib/python3.9/multiprocessing/context.py", line 119, in Pool
    return Pool(processes, initializer, initargs, maxtasksperchild,
  File "/Library/Developer/CommandLineTools/Library/Frameworks/Python3.framework/Versions/3.9/lib/python3.9/multiprocessing/pool.py", line 212, in __init__
    self._repopulate_pool()
  File "/Library/Developer/CommandLineTools/Library/Frameworks/Python3.framework/Versions/3.9/lib/python3.9/multiprocessing/pool.py", line 303, in _repopulate_pool
    return self._repopulate_pool_static(self._ctx, self.Process,
  File "/Library/Developer/CommandLineTools/Library/Frameworks/Python3.framework/Versions/3.9/lib/python3.9/multiprocessing/pool.py", line 326, in _repopulate_pool_static
    w.start()
  File "/Library/Developer/CommandLineTools/Library/Frameworks/Python3.framework/Versions/3.9/lib/python3.9/multiprocessing/process.py", line 121, in start
    self._popen = self._Popen(self)
  File "/Library/Developer/CommandLineTools/Library/Frameworks/Python3.framework/Versions/3.9/lib/python3.9/multiprocessing/context.py", line 284, in _Popen
    return Popen(process_obj)
  File "/Library/Developer/CommandLineTools/Library/Frameworks/Python3.framework/Versions/3.9/lib/python3.9/multiprocessing/popen_spawn_posix.py", line 32, in __init__
    super().__init__(process_obj)
  File "/Library/Developer/CommandLineTools/Library/Frameworks/Python3.framework/Versions/3.9/lib/python3.9/multiprocessing/popen_fork.py", line 19, in __init__
    self._launch(process_obj)
  File "/Library/Developer/CommandLineTools/Library/Frameworks/Python3.framework/Versions/3.9/lib/python3.9/multiprocessing/popen_spawn_posix.py", line 61, in _launch
    with open(parent_w, 'wb', closefd=False) as f:
  File "/Library/Developer/CommandLineTools/Library/Frameworks/Python3.framework/Versions/3.9/lib/python3.9/site-packages/setuptools/sandbox.py", line 453, in _open
    if mode not in ('r', 'rt', 'rb', 'rU', 'U') and not self._ok(path):
  File "/Library/Developer/CommandLineTools/Library/Frameworks/Python3.framework/Versions/3.9/lib/python3.9/site-packages/setuptools/sandbox.py", line 464, in _ok
    realpath = os.path.normcase(os.path.realpath(path))
  File "/Library/Developer/CommandLineTools/Library/Frameworks/Python3.framework/Versions/3.9/lib/python3.9/posixpath.py", line 390, in realpath
    filename = os.fspath(filename)
TypeError: expected str, bytes or os.PathLike object, not int

During handling of the above exception, another exception occurred:

Traceback (most recent call last):

@vipgupta can you please help me out with the local testing setup?

[vipgupta]

It seems like this is some error in the multiprocessing module you added?