A collection of RAPIDS examples for security analysts, data scientists, and engineers to quickly get started applying RAPIDS and GPU acceleration to real-world cybersecurity use cases.
License: Apache License 2.0
Location of incorrect documentation
https://github.com/rapidsai/clx/blob/branch-0.14/README.md
This line: For additional examples, browse our complete API documentation
Describe the problems or issues found in the documentation
The link API documentation is pointing to our old doc location https://rapidsai.github.io/clx/ instead of https://docs.rapids.ai/api/clx/nightly/api.html
Steps taken to verify documentation is incorrect
Clicked link in README
Suggested fix for documentation
Update link to https://docs.rapids.ai/api/clx/nightly/api.html
Is your feature request related to a problem? Please describe.
Currently, load_model function in the Cybert API is using both the pertained model file path and also downloading the model over the network.
def load_model(self, model_filepath, config_filepath):
with open(config_filepath) as f:
config = json.load(f)
model_arch = config["architectures"][0]
self._label_map = {int(k): v for k, v in config["id2label"].items()}
model_state_dict = torch.load(model_filepath)
self._model = ARCH_MAPPING[model_arch].from_pretrained(
MODEL_MAPPING[model_arch],
state_dict=model_state_dict,
num_labels=len(self._label_map),
)
self._model.cuda()
self._model.eval()
Describe the solution you'd like
Update to use only model file path
def load_model(self, model_filepath, config_filepath):
with open(config_filepath) as f:
config = json.load(f)
model_arch = config["architectures"][0]
self._label_map = {int(k): v for k, v in config["id2label"].items()}
self._model = ARCH_MAPPING[model_arch].from_pretrained(model_filepath, config=config_filepath)
self._model.cuda()
self._model.eval()
Describe alternatives you've considered
A clear and concise description of any alternative solutions or features you've considered.
Additional context
Add any other context, code examples, or references to existing implementations about the feature request here.
Report needed documentation
Change the README of CLX to more closely match the other RAPIDS readme files. This is a continuation of PR105.
Describe the bug
The FLAIR DNS log parsing notebook (https://github.com/rapidsai/clx/blob/branch-0.12/notebooks/flair_log_parsing/FLAIR_DNS_Log_Parsing.ipynb) has a data read but no hosted data.
Expected behavior
Need to create data that we can upload to run the notebook. Also should include the schema of the input data in the notebook.
Environment overview (please complete the following information)
Occurs in all environments
This feature has already been ported to cuDF and was deprecated in CLX v0.16.
Hello,
I have a weird behaviour.
Setup: Centos7, 64bit with CUDA 11, conda.
My activate environment is called rapids37 which has already installed all the rapids packages.
I run the commandline and after a few hours.
conda install -c rapidsai-nightly -c nvidia -c pytorch -c conda-forge -c defaults clx
I get this:
The following specifications were found to be incompatible with your system:
- feature:/linux-64::__cuda==11.1=0
- feature:/linux-64::__glibc==2.17=0
- feature:|@/linux-64::__cuda==11.1=0
- feature:|@/linux-64::__glibc==2.17=0
- cugraph -> nccl[version='>=2.5'] -> __glibc[version='>=2.17']
- cuml -> nccl[version='>=2.5'] -> __glibc[version='>=2.17']
- cupy -> nccl=2.7.8.1 -> __glibc[version='>=2.17']
- libcugraph -> nccl[version='>=2.5'] -> __glibc[version='>=2.17']
- libcuml -> libfaiss=1.6.3 -> __glibc[version='>=2.17']
- libfaiss -> __glibc[version='>=2.17']
- libxgboost -> nccl[version='>=2.5'] -> __glibc[version='>=2.17']
- nccl -> __glibc[version='>=2.17']
- rapids -> nccl[version='>=2.7.8.1,<3.0a0'] -> __glibc[version='>=2.17']
- rapids-xgboost -> nccl[version='>=2.7.8.1,<3.0a0'] -> __glibc[version='>=2.17']
- ucx -> __glibc[version='>=2.17,<3.0.a0']
- ucx-py -> ucx==1.8.1+g6b29558 -> __glibc[version='>=2.17,<3.0.a0']
- xgboost -> nccl[version='>=2.7.8.1,<3.0a0'] -> __glibc[version='>=2.17']
Your installed version is: 2.17
Seems all good, but then the module is not installed (module import not found) and I can't even find it in conda lista.
(rapids37) [jupyter@c109518 ~]$ python
Python 3.7.9 (default, Aug 31 2020, 12:42:55)
[GCC 7.3.0] :: Anaconda, Inc. on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import clx
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
ModuleNotFoundError: No module named 'clx'
>>>
Any help appreciated!
Hi there,
I would like to know what's the best way to mount an external folder where I have existing notebooks and data.
If I do this
docker run --gpus all \
--rm -it \
-p 8888:8888 \
-p 8787:8787 \
-p 8686:8686 \
-v /home/jupyter/mywork:/rapids/clx/mywork \
clx:latest
I can see the folder and notebooks in the Jupyter Lab but I can't modify them (read only).
I am guessing there are some flags or setting to change but not sure how.
Cheers.
Is your feature request related to a problem? Please describe.
No really a problem, but it is good to have a generalized structure for clx streamz would give us flexibility to add more workflows with less effort.
Describe the solution you'd like
Update the Dockerfile to copy kafka(create, delete, dataloading) and workflow specific scripts to with respective folders.
Noticed that GPU Id's are always starting from the 0 in the container, while we pass random GPU Id. Remove cuda_visible_devices variable.
Remove workflow specific statements from entry.sh
Update readme.md file to provide step by step instructions to execute workflows
Move utility functions (create dask cluster, single term handle, benchmark calculator and write output to kafka) to different python files.
The sed commands are incorrect and make assumptions.
setup.py assumes the version is double quoted (which after sed'ing it won't be anymore) and starts assumes it starts with 0
Dockerfile uses single quotes so ${NEXT_SHORT_TAG} isn't resolved.
Describe the bug
nvstrings has been integrated into cudf. So it would be good to move to the current cudf API and drop nvstrings as a dependency.
Steps/Code to reproduce bug
NA
Expected behavior
NA
Environment overview (please complete the following information)
NA
Environment details
NA
Additional context
NA
Is your feature request related to a problem? Please describe.
Transformers v4.0.0 was released on 11/30/20. Updating to this version causes unit test failures in cybert and phishing detection.
Describe the solution you'd like
Update and test code that uses transformers according to guidelines outlined here:
https://github.com/huggingface/transformers/releases/tag/v4.0.0
Report needed documentation
Add pre-commit and code formatting instructions in CONTRIBUTING.md . This will be helpful when gpuCI flags any code format issues in a new PR.
Describe the documentation you'd like
Reference cudf CONTRIBUTING.md https://github.com/rapidsai/cudf/blob/branch-0.14/CONTRIBUTING.md and apply what may be applicable to clx
particularly the Code Formatting and Pre-commit Hooks sections
Steps taken to search for needed documentation
Looked up information related to gpuCI style failures for a PR. Looked at cudf CONTRIBUTING.md for reference
Started getting segmentation faults in CI builds on 10/7. Appears to be related to calls to cuDF read_csv. Same code does not produce error when running on rapidsai-dev-nightly image.
>>> import cudf
>>> df1 = cudf.DataFrame(
... {
... "firstname": ["Emma", "Ava", "Sophia"],
... "lastname": ["Olivia", "Isabella", "Charlotte"],
... "gender": ["F", "F", "F"],
... }
... )
>>> df1.to_csv("test.csv", index=False)
>>> df2 = cudf.read_csv("test.csv")
Segmentation fault (core dumped)
Describe the bug
Currently, dns extractor is not handling a scenario when all records of last tld column doesn't match to suffix list. This is causing parse url result mismatch.
Steps/Code to reproduce bug
import cudf
from clx.dns import dns_extractor as dns
gdf = cudf.read_json('dns.log', lines=True)
gdf = gdf[0:10]
gdf = gdf[["message"]]
gdf["url"] = gdf.message.str.extract("query:\s([a-zA-Z\.\-\:\/\-0-9]+)")
gdf = gdf[['url']]
gdf["url"] = gdf.url.str.lower()
extracted_gdf = dns.parse_url(gdf["url"], req_cols={"domain", "suffix"})
domains_series = extracted_gdf['domain']+'.'+extracted_gdf['suffix']
gdf['domain'] = domains_series
gdf
output
| url | domain
-- | -- | --
test.na.bestbuy.com | bestbuy.com
test.bestbuy.com | bestbuy.com
www.microsoft.com | microsoft.com
test.pr.bdc.bestbuy.com | bestbuy.com
dasdad.n.bestbuy.com | bestbuy.com
feedws.icloud.com | icloud.com
dadkde.bestbuy.com.bby | bestbuy.com
pqwldndwub.n.bestbuy.com | 111.in-addr.arpa
cfadsad.br.bestbuy.com.bby | <NA>
111.111.111.111.in-addr.arpa | <NA>
Expected behavior
| url | domain
-- | -- | --
test.na.bestbuy.com | bestbuy.com
test.bestbuy.com | bestbuy.com
www.microsoft.com | microsoft.com
test.pr.bdc.bestbuy.com | bestbuy.com
dasdad.n.bestbuy.com | bestbuy.com
feedws.icloud.com | icloud.com
dadkde.bestbuy.com.bby | <NA>
pqwldndwub.n.bestbuy.com | bestbuy.com
cfadsad.br.bestbuy.com.bby | <NA>
111.111.111.111.in-addr.arpa | 111.in-addr.arpa
Describe the bug
The workflow reads in data by default as csv for json file
Steps/Code to reproduce bug
!cat alert_data.json
{
"data": [
{
"raw": "raw data"
}
]
}
from clx.workflow.workflow import Workflow
import os
dirpath = os.getcwd()
source = {
"type": "fs",
"input_format": "json",
"input_path": dirpath + "/alert_data.json"
}
destination = {
"type": "fs",
"output_format": "json",
"output_path": dirpath + "/alert_data_output.json"
}
class NewWorkflow(Workflow):
def workflow(self, dataframe):
print(dataframe)
dataframe["enriched"] = "enriched"
return dataframe
lpw = NewWorkflow(source=source, destination=destination, name="my-workflow")
lpw.run_workflow()
Dataframe
{
0 "data": [
1 {
2 "raw": "raw data"
3 }
4 ]
5 }
Expected behavior
Environment overview (please complete the following information)
docker
Environment details
**git*** commit ec9c65da7d87213e07c7b54c26953ac9bd0f810e (HEAD -> branch-0.12, upstream/branch-0.12, origin/branch-0.12) Merge: 6641144 d60f359 Author: BiancaR <[email protected]> Date: Mon Jan 6 15:06:31 2020 -0500 Merge pull request #68 from brhodes10/fix/workflow-notebooks Updated worklow notebooks **git submodules*** ***OS Information*** DISTRIB_ID=Ubuntu DISTRIB_RELEASE=18.04 DISTRIB_CODENAME=bionic DISTRIB_DESCRIPTION="Ubuntu 18.04.3 LTS" NAME="Ubuntu" VERSION="18.04.3 LTS (Bionic Beaver)" ID=ubuntu ID_LIKE=debian PRETTY_NAME="Ubuntu 18.04.3 LTS" VERSION_ID="18.04" HOME_URL="https://www.ubuntu.com/" SUPPORT_URL="https://help.ubuntu.com/" BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/" PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy" VERSION_CODENAME=bionic UBUNTU_CODENAME=bionic Linux dgx03 4.15.0-47-generic #50-Ubuntu SMP Wed Mar 13 10:44:52 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux ***GPU Information*** Tue Jan 7 17:12:24 2020 +-----------------------------------------------------------------------------+ | NVIDIA-SMI 418.87.00 Driver Version: 418.87.00 CUDA Version: 10.1 | |-------------------------------+----------------------+----------------------+ | GPU Name Persistence-M| Bus-Id Disp.A | Volatile Uncorr. ECC | | Fan Temp Perf Pwr:Usage/Cap| Memory-Usage | GPU-Util Compute M. | |===============================+======================+======================| | 0 Tesla V100-SXM2... On | 00000000:85:00.0 Off | 0 | | N/A 33C P0 58W / 300W | 12884MiB / 32480MiB | 0% Default | +-------------------------------+----------------------+----------------------+ +-----------------------------------------------------------------------------+ | Processes: GPU Memory | | GPU PID Type Process name Usage | |=============================================================================| +-----------------------------------------------------------------------------+ ***CPU*** Architecture: x86_64 CPU op-mode(s): 32-bit, 64-bit Byte Order: Little Endian CPU(s): 80 On-line CPU(s) list: 0-79 Thread(s) per core: 2 Core(s) per socket: 20 Socket(s): 2 NUMA node(s): 2 Vendor ID: GenuineIntel CPU family: 6 Model: 79 Model name: Intel(R) Xeon(R) CPU E5-2698 v4 @ 2.20GHz Stepping: 1 CPU MHz: 2645.254 CPU max MHz: 3600.0000 CPU min MHz: 1200.0000 BogoMIPS: 4389.92 Virtualization: VT-x L1d cache: 32K L1i cache: 32K L2 cache: 256K L3 cache: 51200K NUMA node0 CPU(s): 0-19,40-59 NUMA node1 CPU(s): 20-39,60-79 Flags: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc cpuid aperfmperf pni pclmulqdq dtes64 monitor ds_cpl vmx smx est tm2 ssse3 sdbg fma cx16 xtpr pdcm pcid dca sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand lahf_lm abm 3dnowprefetch cpuid_fault epb cat_l3 cdp_l3 invpcid_single pti intel_ppin ssbd ibrs ibpb stibp tpr_shadow vnmi flexpriority ept vpid fsgsbase tsc_adjust bmi1 hle avx2 smep bmi2 erms invpcid rtm cqm rdt_a rdseed adx smap intel_pt xsaveopt cqm_llc cqm_occup_llc cqm_mbm_total cqm_mbm_local dtherm ida arat pln pts flush_l1d ***CMake*** /opt/conda/envs/rapids/bin/cmake cmake version 3.14.5 CMake suite maintained and supported by Kitware (kitware.com/cmake). ***g++*** /usr/bin/g++ g++ (Ubuntu 7.4.0-1ubuntu1~18.04.1) 7.4.0 Copyright (C) 2017 Free Software Foundation, Inc. This is free software; see the source for copying conditions. There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. ***nvcc*** /usr/local/cuda/bin/nvcc nvcc: NVIDIA (R) Cuda compiler driver Copyright (c) 2005-2019 NVIDIA Corporation Built on Sun_Jul_28_19:07:16_PDT_2019 Cuda compilation tools, release 10.1, V10.1.243 ***Python*** /opt/conda/envs/rapids/bin/python Python 3.7.6 ***Environment Variables*** PATH : /opt/conda/envs/rapids/bin:/opt/conda/condabin:/usr/local/nvidia/bin:/usr/local/cuda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/conda/bin:/conda/bin:/conda/bin LD_LIBRARY_PATH : /opt/conda/envs/rapids/lib:/usr/local/nvidia/lib:/usr/local/nvidia/lib64:/usr/local/cuda/lib64:/usr/local/lib NUMBAPRO_NVVM : NUMBAPRO_LIBDEVICE : CONDA_PREFIX : /opt/conda/envs/rapids PYTHON_PATH : ***conda packages*** /opt/conda/condabin/conda # packages in environment at /opt/conda/envs/rapids: # # Name Version Build Channel _libgcc_mutex 0.1 main conda-forge _tflow_select 2.1.0 gpu absl-py 0.9.0 py37_0 conda-forge aiohttp 3.6.2 py37h516909a_0 conda-forge alabaster 0.7.12 py_0 conda-forge appdirs 1.4.3 py_1 conda-forge arrow-cpp 0.15.0 py37h090bef1_2 conda-forge astor 0.7.1 py_0 conda-forge async-timeout 3.0.1 py_1000 conda-forge attrs 19.3.0 py_0 conda-forge babel 2.8.0 py_0 conda-forge backcall 0.1.0 py_0 conda-forge black 19.10b0 py37_0 conda-forge blas 2.14 openblas conda-forge bleach 3.1.0 py_0 conda-forge bokeh 1.4.0 py37_0 conda-forge boost-cpp 1.70.0 h8e57a91_2 conda-forge brotli 1.0.7 he1b5a44_1000 conda-forge bzip2 1.0.8 h516909a_2 conda-forge c-ares 1.15.0 h516909a_1001 conda-forge ca-certificates 2019.11.28 hecc5488_0 conda-forge cairo 1.16.0 hfb77d84_1002 conda-forge certifi 2019.11.28 py37_0 conda-forge cffi 1.13.2 py37h8022711_0 conda-forge cfitsio 3.470 hb60a0a2_2 conda-forge chardet 3.0.4 py37_1003 conda-forge click 7.0 pypi_0 pypi cloudpickle 1.2.2 py_1 conda-forge clx 0.12.0 pypi_0 pypi cmake 3.14.5 hf94ab9c_0 conda-forge cmake_setuptools 0.1.3 py_0 rapidsai-nightly commonmark 0.9.1 py_0 conda-forge confluent-kafka 1.3.0 pypi_0 pypi cryptography 2.8 py37h72c5cf5_1 conda-forge cudatoolkit 10.1.243 h6bb024c_0 nvidia cudf 0.12.0b0+589.g61b9f2f2e pypi_0 pypi cudnn 7.6.0 cuda10.1_0 nvidia cugraph 0.12.0a0+182.g14384d5.dirty pypi_0 pypi cuml 0.12.0a0+372.g666fcf8f pypi_0 pypi cupti 10.1.168 0 cupy 6.6.0 py37ha7c4746_1 conda-forge curl 7.65.3 hf8cf82a_0 conda-forge cuspatial 0.12.0a0+8.g4c4c327 pypi_0 pypi cycler 0.10.0 py_2 conda-forge cyrus-sasl 2.1.27 he38ecfd_0 conda-forge cython 0.29.14 py37he1b5a44_0 conda-forge cytoolz 0.10.1 py37h516909a_0 conda-forge dask 2.9.1 py_0 conda-forge dask-core 2.9.1 py_0 conda-forge dask-cuda 0.12.0a200107 py37_45 rapidsai-nightly dask-cudf 0.12.0b0+589.g61b9f2f2e pypi_0 pypi dask-glm 0.2.0 py_1 conda-forge dask-labextension 1.1.0 py_0 conda-forge dask-ml 1.1.1 py_0 conda-forge dask-xgboost 0.1.5 pypi_0 pypi dbus 1.13.6 he372182_0 conda-forge decorator 4.4.1 py_0 conda-forge defusedxml 0.6.0 py_0 conda-forge distributed 2.9.1 py_0 conda-forge dlpack 0.2 he1b5a44_1 conda-forge docutils 0.15.2 py37_0 conda-forge double-conversion 3.1.5 he1b5a44_2 conda-forge doxygen 1.8.16 hd1b7508_1 conda-forge entrypoints 0.3 py37_1000 conda-forge expat 2.2.5 he1b5a44_1004 conda-forge fastavro 0.22.9 py37h516909a_0 conda-forge fastrlock 0.4 py37he1b5a44_1000 conda-forge flake8 3.7.9 py37_0 conda-forge flatbuffers 1.11.0 he1b5a44_0 conda-forge fontconfig 2.13.1 h86ecdb6_1001 conda-forge freetype 2.10.0 he983fc9_1 conda-forge freexl 1.0.5 h14c3975_1002 conda-forge fribidi 1.0.5 h516909a_1002 conda-forge fsspec 0.6.2 py_0 conda-forge future 0.18.2 py37_0 conda-forge gast 0.3.2 py_0 conda-forge gdal 2.4.3 py37h5f563d9_9 conda-forge geos 3.7.2 he1b5a44_2 conda-forge geotiff 1.5.1 hbd99317_7 conda-forge gettext 0.19.8.1 hc5be6a0_1002 conda-forge gflags 2.2.2 he1b5a44_1002 conda-forge giflib 5.1.7 h516909a_1 conda-forge glib 2.58.3 py37h6f030ca_1002 conda-forge glog 0.4.0 he1b5a44_1 conda-forge gmp 6.1.2 hf484d3e_1000 conda-forge google-pasta 0.1.8 py_0 conda-forge graphite2 1.3.13 hf484d3e_1000 conda-forge graphviz 2.42.3 h0511662_0 conda-forge grpc-cpp 1.23.0 h18db393_0 conda-forge grpcio 1.23.0 py37he9ae1f9_0 conda-forge gst-plugins-base 1.14.5 h0935bb2_0 conda-forge gstreamer 1.14.5 h36ae1b5_0 conda-forge h5py 2.10.0 nompi_py37h513d04c_101 conda-forge harfbuzz 2.4.0 h9f30f68_3 conda-forge hdf4 4.2.13 hf30be14_1003 conda-forge hdf5 1.10.5 nompi_h3c11f04_1104 conda-forge heapdict 1.0.1 py_0 conda-forge hypothesis 5.1.0 py_0 conda-forge icu 64.2 he1b5a44_1 conda-forge idna 2.8 py37_1000 conda-forge imagesize 1.2.0 py_0 conda-forge importlib_metadata 1.3.0 py37_0 conda-forge ipykernel 5.1.3 py37h5ca1d4c_0 conda-forge ipython 7.3.0 py37h24bf2e0_0 conda-forge ipython_genutils 0.2.0 py_1 conda-forge jedi 0.15.2 py37_0 conda-forge jinja2 2.10.3 py_0 conda-forge joblib 0.14.1 py_0 conda-forge jpeg 9c h14c3975_1001 conda-forge json-c 0.13.1 h14c3975_1001 conda-forge json5 0.8.5 py_0 conda-forge jsonschema 3.2.0 py37_0 conda-forge jupyter-server-proxy 1.2.0 py_0 conda-forge jupyter_client 5.3.3 py37_1 conda-forge jupyter_core 4.6.1 py37_0 conda-forge jupyterlab 1.2.4 py_0 conda-forge jupyterlab-nvdashboard 0.1.11 pypi_0 pypi jupyterlab_server 1.0.6 py_0 conda-forge kealib 1.4.10 h58c409b_1005 conda-forge keras-applications 1.0.8 py_1 conda-forge keras-preprocessing 1.1.0 py_0 conda-forge kiwisolver 1.1.0 py37hc9558a2_0 conda-forge krb5 1.16.4 h2fd8d38_0 conda-forge lapack 3.6.1 ha44fe06_2 conda-forge ld_impl_linux-64 2.33.1 h53a641e_7 conda-forge libblas 3.8.0 14_openblas conda-forge libcblas 3.8.0 14_openblas conda-forge libclang 8.0.0 h6bb024c_0 rapidsai libcumlprims 0.12.0a200107 cuda10.1_0 rapidsai-nightly libcurl 7.65.3 hda55be3_0 conda-forge libcypher-parser 0.6.2 1 rapidsai libdap4 3.20.4 hd3bb157_0 conda-forge libedit 3.1.20170329 hf8c457e_1001 conda-forge libevent 2.1.10 h72c5cf5_0 conda-forge libffi 3.2.1 he1b5a44_1006 conda-forge libgcc-ng 7.3.0 hdf63c60_2 conda-forge libgcrypt 1.8.4 hf484d3e_1000 conda-forge libgdal 2.4.3 h2f07a13_9 conda-forge libgfortran 3.0.0 1 conda-forge libgfortran-ng 7.3.0 hdf63c60_2 conda-forge libgpg-error 1.36 he1b5a44_0 conda-forge libgsasl 1.8.0 h19a2143_1004 conda-forge libiconv 1.15 h516909a_1005 conda-forge libkml 1.3.0 h4fcabce_1010 conda-forge liblapack 3.8.0 14_openblas conda-forge liblapacke 3.8.0 14_openblas conda-forge libllvm8 8.0.1 hc9558a2_0 conda-forge libnetcdf 4.7.1 nompi_h94020b1_102 conda-forge libntlm 1.4 h14c3975_1002 conda-forge libopenblas 0.3.7 h5ec1e0e_6 conda-forge libpng 1.6.37 hed695b0_0 conda-forge libpq 11.5 hd9ab2ff_2 conda-forge libprotobuf 3.8.0 h8b12597_0 conda-forge librdkafka 1.2.2 hb2b7465_0 conda-forge libsodium 1.0.17 h516909a_0 conda-forge libspatialite 4.3.0a h4f6d029_1032 conda-forge libssh2 1.8.2 h22169c7_2 conda-forge libstdcxx-ng 7.3.0 hdf63c60_2 conda-forge libtiff 4.1.0 hfc65ed5_0 conda-forge libtool 2.4.6 h14c3975_1002 conda-forge libuuid 2.32.1 h14c3975_1000 conda-forge libuv 1.33.1 h516909a_0 conda-forge libxcb 1.13 h14c3975_1002 conda-forge libxml2 2.9.10 hee79883_0 conda-forge llvmlite 0.30.0 py37h8b12597_1 conda-forge locket 0.2.0 py_2 conda-forge lz4-c 1.8.3 he1b5a44_1001 conda-forge make 4.2.1 h14c3975_2004 conda-forge markdown 3.0.1 pypi_0 pypi markupsafe 1.1.1 py37h516909a_0 conda-forge matplotlib 3.1.2 py37_1 conda-forge matplotlib-base 3.1.2 py37h250f245_1 conda-forge mccabe 0.6.1 py_1 conda-forge mistune 0.8.4 py37h516909a_1000 conda-forge more-itertools 8.0.2 py_0 conda-forge msgpack-python 0.6.2 py37hc9558a2_0 conda-forge multidict 4.7.3 py37h516909a_0 conda-forge multipledispatch 0.6.0 py_0 conda-forge mypy_extensions 0.4.3 py37_0 conda-forge nbconvert 5.6.1 py37_0 conda-forge nbformat 4.4.0 py_1 conda-forge nbsphinx 0.5.0 py_0 conda-forge nccl 2.4.6.1 cuda10.1_0 nvidia ncurses 6.1 hf484d3e_1002 conda-forge networkx 2.4 py_0 conda-forge nodejs 13.0.0 h10a4023_1 conda-forge notebook 6.0.1 py37_0 conda-forge numba 0.46.0 py37hb3f55d8_1 conda-forge numpy 1.17.3 py37h95a1406_0 conda-forge numpydoc 0.9.2 py_0 conda-forge nvstrings-cuda101 0.0.0.dev0 pypi_0 pypi olefile 0.46 py_0 conda-forge openblas 0.3.7 he1df0ab_6 conda-forge openjpeg 2.3.1 h981e76c_3 conda-forge openssl 1.1.1d h516909a_0 conda-forge packaging 19.2 py_0 conda-forge pandas 0.24.2 py37hb3f55d8_1 conda-forge pandoc 1.19.2 0 conda-forge pandocfilters 1.4.2 py_1 conda-forge pango 1.42.4 ha030887_1 conda-forge parquet-cpp 1.5.1 2 conda-forge parso 0.5.2 py_0 conda-forge partd 1.1.0 py_0 conda-forge pathspec 0.7.0 py_0 conda-forge patsy 0.5.1 py_0 conda-forge pcre 8.43 he1b5a44_0 conda-forge pexpect 4.7.0 py37_0 conda-forge pickleshare 0.7.5 py37_1000 conda-forge pillow 6.2.1 py37hd70f55b_1 conda-forge pip 19.3.1 py37_0 conda-forge pixman 0.38.0 h516909a_1003 conda-forge pluggy 0.13.0 py37_0 conda-forge poppler 0.67.0 h14e79db_8 conda-forge poppler-data 0.4.9 1 conda-forge postgresql 11.5 hc63931a_2 conda-forge proj 6.2.1 hc80f0dc_0 conda-forge prometheus_client 0.7.1 py_0 conda-forge prompt-toolkit 2.0.10 pypi_0 pypi protobuf 3.8.0 py37he1b5a44_2 conda-forge psutil 5.6.7 py37h516909a_0 conda-forge pthread-stubs 0.4 h14c3975_1001 conda-forge ptyprocess 0.6.0 py_1001 conda-forge py 1.8.1 py_0 conda-forge pyarrow 0.15.0 py37h8b68381_1 conda-forge pycodestyle 2.5.0 py_0 conda-forge pycparser 2.19 py37_1 conda-forge pyflakes 2.1.1 py_0 conda-forge pygments 2.5.2 py_0 conda-forge pynvml 8.0.3 py_0 conda-forge pyopenssl 19.1.0 py37_0 conda-forge pyparsing 2.4.6 py_0 conda-forge pyqt 5.9.2 py37hcca6a23_4 conda-forge pyrsistent 0.15.6 py37h516909a_0 conda-forge pysocks 1.7.1 py37_0 conda-forge pytest 5.3.2 py37_0 conda-forge python 3.7.6 h357f687_1 conda-forge python-dateutil 2.8.1 py_0 conda-forge python-whois 0.7.2 pypi_0 pypi pytz 2019.3 py_0 conda-forge pyyaml 5.2 py37h516909a_0 conda-forge pyzmq 18.1.1 py37h1768529_0 conda-forge qt 5.9.7 h0c104cb_3 conda-forge rapidjson 1.1.0 he1b5a44_1002 conda-forge re2 2020.01.01 he1b5a44_0 conda-forge readline 8.0 hf8c457e_0 conda-forge recommonmark 0.6.0 py_0 conda-forge regex 2019.12.20 py37h516909a_0 conda-forge requests 2.22.0 py37_1 conda-forge rhash 1.3.6 h14c3975_1001 conda-forge rmm 0.12.0a0+113.g15fafa2 pypi_0 pypi scikit-learn 0.22 py37hcdab131_1 conda-forge scipy 1.4.1 py37h921218d_0 conda-forge seaborn 0.9.0 py_2 conda-forge send2trash 1.5.0 py_0 conda-forge setuptools 44.0.0 py37_0 conda-forge simpervisor 0.3 py_1 conda-forge sip 4.19.8 py37hf484d3e_1000 conda-forge six 1.13.0 py37_0 conda-forge snappy 1.1.7 he1b5a44_1003 conda-forge snowballstemmer 2.0.0 py_0 conda-forge sortedcontainers 2.1.0 py_0 conda-forge sphinx 2.3.1 py_0 conda-forge sphinx-markdown-tables 0.0.10 pypi_0 pypi sphinx_rtd_theme 0.4.3 py_0 conda-forge sphinxcontrib-applehelp 1.0.1 py_0 conda-forge sphinxcontrib-devhelp 1.0.1 py_0 conda-forge sphinxcontrib-htmlhelp 1.0.2 py_0 conda-forge sphinxcontrib-jsmath 1.0.1 py_0 conda-forge sphinxcontrib-qthelp 1.0.2 py_0 conda-forge sphinxcontrib-serializinghtml 1.1.3 py_0 conda-forge sphinxcontrib-websupport 1.1.2 py_0 conda-forge sqlite 3.30.1 hcee41ef_0 conda-forge statsmodels 0.10.2 py37hc1659b7_0 conda-forge tblib 1.6.0 py_0 conda-forge tensorboard 1.14.0 py37_0 conda-forge tensorflow 1.14.0 gpu_py37h74c33d7_0 tensorflow-base 1.14.0 gpu_py37he45bfe2_0 tensorflow-estimator 1.14.0 py37h5ca1d4c_0 conda-forge termcolor 1.1.0 py_2 conda-forge terminado 0.8.3 py37_0 conda-forge testpath 0.4.4 py_0 conda-forge thrift-cpp 0.12.0 hf3afdfd_1004 conda-forge tk 8.6.10 hed695b0_0 conda-forge toml 0.10.0 py_0 conda-forge toolz 0.10.0 py_0 conda-forge torch 1.3.1 pypi_0 pypi tornado 6.0.3 py37h516909a_0 conda-forge traitlets 4.3.3 py37_0 conda-forge typed-ast 1.4.0 py37h516909a_0 conda-forge typing_extensions 3.7.4.1 py37_0 conda-forge tzcode 2019a h516909a_1002 conda-forge umap-learn 0.3.10 py37_1 conda-forge uriparser 0.9.3 he1b5a44_1 conda-forge urllib3 1.25.7 py37_0 conda-forge wcwidth 0.1.8 py_0 conda-forge webencodings 0.5.1 py_1 conda-forge werkzeug 0.16.0 py_0 conda-forge wheel 0.33.6 py37_0 conda-forge wrapt 1.11.2 py37h516909a_0 conda-forge xerces-c 3.2.2 h8412b87_1004 conda-forge xgboost 1.0.0-SNAPSHOT pypi_0 pypi xorg-kbproto 1.0.7 h14c3975_1002 conda-forge xorg-libice 1.0.10 h516909a_0 conda-forge xorg-libsm 1.2.3 h84519dc_1000 conda-forge xorg-libx11 1.6.9 h516909a_0 conda-forge xorg-libxau 1.0.9 h14c3975_0 conda-forge xorg-libxdmcp 1.1.3 h516909a_0 conda-forge xorg-libxext 1.3.4 h516909a_0 conda-forge xorg-libxpm 3.5.13 h516909a_0 conda-forge xorg-libxrender 0.9.10 h516909a_1002 conda-forge xorg-libxt 1.1.5 h516909a_1003 conda-forge xorg-renderproto 0.11.1 h14c3975_1002 conda-forge xorg-xextproto 7.3.0 h14c3975_1002 conda-forge xorg-xproto 7.0.31 h14c3975_1007 conda-forge xz 5.2.4 h14c3975_1001 conda-forge yaml 0.2.2 h516909a_1 conda-forge yarl 1.3.0 py37h516909a_1000 conda-forge zeromq 4.3.2 he1b5a44_2 conda-forge zict 1.0.0 pypi_0 pypi zipp 0.6.0 py_0 conda-forge zlib 1.2.11 h516909a_1006 conda-forge zstd 1.4.3 h3b9ef0a_0 conda-forge[DOC] Cybert API docs missing
Report incorrect documentation
Location of incorrect documentation
CLX Analytics API docs https://docs.rapids.ai/api/clx/nightly/api.html#analyticsDescribe the problems or issues found in the documentation
Cybert does not appear under the CLX Analytics API docs aboveSteps taken to verify documentation is incorrect
Local documentation buildSuggested fix for documentation
Could possibly be an issue with the docstring[DOC] Update CLX Tokenizer Docs to Reflect Deprecation
The subword tokenizer in CLX is deprecated and will be removed in future releases. This should be updated in the docs.
Location of incorrect documentation
https://docs.rapids.ai/api/clx/nightly/api.html?highlight=tokenize#module-clx.analytics.tokenizer
https://docs.rapids.ai/api/clx/nightly/api.html?highlight=tokenize#clx.analytics.tokenizer.tokenize_fileDescribe the problems or issues found in the documentation
Update documentation to reflect deprecation and point to cuDF subword tokenizer at: https://docs.rapids.ai/api/cudf/nightly/api.html?highlight=subword#cudf.core.column.string.StringMethods.subword_tokenize[FEA] Update cybert notebook to use new cudf tokenizer
Describe the solution you'd like
The CLX tokenizer has been ported to cudf. Update CLX cybert notebook to use new cudf tokenizer[FEA] DGA train_model function
Is your feature request related to a problem? Please describe.
NoDescribe the solution you'd like
Requesting to remove the second parameterdataset_lenfrom the function below indga_detector. And calculatedataset_lenwithin thetrain_modelfunction instead.
def train_model(self, partitioned_dfs, dataset_len)Describe alternatives you've considered
Calculatedataset_lenprior to callingtrain_model[BUG] DGA Detection Notebook Unable to Download Public Dataset
Describe the bug
DGA Detection notebook unable to download public dataset as they are now licensed.--------------------------------------------------------------------------- HTTPError Traceback (most recent call last) <ipython-input-5-83278b646882> in <module> ----> 1 download_files(URL_META_LIST) <ipython-input-3-c5837c315d20> in download_files(url_meta_list) 5 shutil.rmtree(output_dir) 6 os.makedirs(output_dir) ----> 7 filepath = wget.download(entry['url'], out=output_dir) 8 unpack(entry['compression'], filepath, output_dir) 9 print('%s data is stored to location %s' %(entry['source'], output_dir)) /opt/conda/envs/rapids/lib/python3.7/site-packages/wget.py in download(url, out, bar) 524 else: 525 binurl = url --> 526 (tmpfile, headers) = ulib.urlretrieve(binurl, tmpfile, callback) 527 filename = detect_filename(url, out, headers) 528 if outdir: /opt/conda/envs/rapids/lib/python3.7/urllib/request.py in urlretrieve(url, filename, reporthook, data) 245 url_type, path = splittype(url) 246 --> 247 with contextlib.closing(urlopen(url, data)) as fp: 248 headers = fp.info() 249 /opt/conda/envs/rapids/lib/python3.7/urllib/request.py in urlopen(url, data, timeout, cafile, capath, cadefault, context) 220 else: 221 opener = _opener --> 222 return opener.open(url, data, timeout) 223 224 def install_opener(opener): /opt/conda/envs/rapids/lib/python3.7/urllib/request.py in open(self, fullurl, data, timeout) 529 for processor in self.process_response.get(protocol, []): 530 meth = getattr(processor, meth_name) --> 531 response = meth(req, response) 532 533 return response /opt/conda/envs/rapids/lib/python3.7/urllib/request.py in http_response(self, request, response) 639 if not (200 <= code < 300): 640 response = self.parent.error( --> 641 'http', request, response, code, msg, hdrs) 642 643 return response /opt/conda/envs/rapids/lib/python3.7/urllib/request.py in error(self, proto, *args) 567 if http_err: 568 args = (dict, 'default', 'http_error_default') + orig_args --> 569 return self._call_chain(*args) 570 571 # XXX probably also want an abstract factory that knows when it makes /opt/conda/envs/rapids/lib/python3.7/urllib/request.py in _call_chain(self, chain, kind, meth_name, *args) 501 for handler in handlers: 502 func = getattr(handler, meth_name) --> 503 result = func(*args) 504 if result is not None: 505 return result /opt/conda/envs/rapids/lib/python3.7/urllib/request.py in http_error_default(self, req, fp, code, msg, hdrs) 647 class HTTPDefaultErrorHandler(BaseHandler): 648 def http_error_default(self, req, fp, code, msg, hdrs): --> 649 raise HTTPError(req.full_url, code, msg, hdrs, fp) 650 651 class HTTPRedirectHandler(BaseHandler): HTTPError: HTTP Error 403: Forbidden[FEA] Update CLX Streamz Examples
Is your feature request related to a problem? Please describe.
Update CLX streamz example workflows to use different sinks(filesystem, elasticsearch, kafka) and provide ability to select sink from the configuration file.[BUG] Performance issues from Pytorch for CUDA 10.2
Pytorch does not officially support CUDA 10.2. CLX is currently using a source build that is causing performance issues with the build and all CLX features using PyTorch/CUDA 10.2.
[FEA] Update Phishing Detection (CLX tokenizer -> cuDF tokenizer)
Is your feature request related to a problem? Please describe.
Update existing clx tokenizer with cuDF tokenizer[BUG] Errors in Alert Analysis Notebook
Describe the bug
When running the CLX Alert Analysis notebook, the following error occurs from thenormalize_gdffunction in theHeatmap Visualizationsection:TypeError: Series object is not iterable. Consider using `.to_arrow()`, `.to_pandas()` or `.values_host` if you wish to iterate over the values.After following suggestion in error, this error occurs in the
cuXfiltersection:NvvmError: Failed to compile <unnamed> (202, 17): parse expected binary operation in atomicrmw NVVM_ERROR_COMPILATIONEnvironment overview
- Environment location: Docker container
- Method of CLX install: conda, Docker, and from source
[BUG] cuxfilter visualization no longer working in alert analysis notebook
Describe the bug
No output in notebook for line chart using latest cuxfilter[BUG] Alert Analysis notebook fails to import cuDataShader and cuXfilter
Describe the bug
The alert analysis notebook uses cudatashader and cuXfilter for viz, but neither can be importedSteps/Code to reproduce bug
Run theImportscell in the Alert Analysis with CLX notebook, or run the code below in any CLX notebook.import cudatashader from cuXfilter import charts, layouts, themes, DataFrameExpected behavior
Expected that cudatashader and cuXfilter are availableEnvironment overview (please complete the following information)
- Environment location: Docker version 19.03.2, build 6a30dfc
- Method of CLX install: Docker
- Used the method indicated on the CLX readme (replicated below)
docker pull rapidsai/rapidsai-dev-nightly:0.11-cuda10.0-devel-ubuntu18.04-py3.7 docker build -t clx . docker run --runtime=nvidia \ --rm -it \ -p 8888:8888 \ -p 8787:8787 \ -p 8686:8686 \ clx:latestEnvironment details
Please run and paste the output of the/rapids/cudf/print_env.shscript here, to gather any other relevant environment details. The script is located in the docker container.(rapids) root@dgx03:/clx/clx# /rapids/cudf/print_env.sh <details><summary>Click here to see environment details</summary><pre> **git*** commit ec9c65da7d87213e07c7b54c26953ac9bd0f810e (HEAD -> branch-0.12, origin/branch-0.12, origin/HEAD) Merge: 6641144 d60f359 Author: BiancaR <[email protected]> Date: Mon Jan 6 15:06:31 2020 -0500 Merge pull request #68 from brhodes10/fix/workflow-notebooks Updated worklow notebooks **git submodules*** ***OS Information*** DISTRIB_ID=Ubuntu DISTRIB_RELEASE=18.04 DISTRIB_CODENAME=bionic DISTRIB_DESCRIPTION="Ubuntu 18.04.3 LTS" NAME="Ubuntu" VERSION="18.04.3 LTS (Bionic Beaver)" ID=ubuntu ID_LIKE=debian PRETTY_NAME="Ubuntu 18.04.3 LTS" VERSION_ID="18.04" HOME_URL="https://www.ubuntu.com/" SUPPORT_URL="https://help.ubuntu.com/" BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/" PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy" VERSION_CODENAME=bionic UBUNTU_CODENAME=bionic Linux dgx03 4.15.0-47-generic #50-Ubuntu SMP Wed Mar 13 10:44:52 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux ***GPU Information*** Tue Jan 7 14:43:54 2020 +-----------------------------------------------------------------------------+ | NVIDIA-SMI 418.87.00 Driver Version: 418.87.00 CUDA Version: 10.1 | |-------------------------------+----------------------+----------------------+ | GPU Name Persistence-M| Bus-Id Disp.A | Volatile Uncorr. ECC | | Fan Temp Perf Pwr:Usage/Cap| Memory-Usage | GPU-Util Compute M. | |===============================+======================+======================| | 0 Tesla V100-SXM2... On | 00000000:0B:00.0 Off | 0 | | N/A 33C P0 57W / 300W | 6689MiB / 32480MiB | 0% Default | +-------------------------------+----------------------+----------------------+ +-----------------------------------------------------------------------------+ | Processes: GPU Memory | | GPU PID Type Process name Usage | |=============================================================================| +-----------------------------------------------------------------------------+ ***CPU*** Architecture: x86_64 CPU op-mode(s): 32-bit, 64-bit Byte Order: Little Endian CPU(s): 80 On-line CPU(s) list: 0-79 Thread(s) per core: 2 Core(s) per socket: 20 Socket(s): 2 NUMA node(s): 2 Vendor ID: GenuineIntel CPU family: 6 Model: 79 Model name: Intel(R) Xeon(R) CPU E5-2698 v4 @ 2.20GHz Stepping: 1 CPU MHz: 2700.097 CPU max MHz: 3600.0000 CPU min MHz: 1200.0000 BogoMIPS: 4389.92 Virtualization: VT-x L1d cache: 32K L1i cache: 32K L2 cache: 256K L3 cache: 51200K NUMA node0 CPU(s): 0-19,40-59 NUMA node1 CPU(s): 20-39,60-79 Flags: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc cpuid aperfmperf pni pclmulqdq dtes64 monitor ds_cpl vmx smx est tm2 ssse3 sdbg fma cx16 xtpr pdcm pcid dca sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand lahf_lm abm 3dnowprefetch cpuid_fault epb cat_l3 cdp_l3 invpcid_single pti intel_ppin ssbd ibrs ibpb stibp tpr_shadow vnmi flexpriority ept vpid fsgsbase tsc_adjust bmi1 hle avx2 smep bmi2 erms invpcid rtm cqm rdt_a rdseed adx smap intel_pt xsaveopt cqm_llc cqm_occup_llc cqm_mbm_total cqm_mbm_local dtherm ida arat pln pts flush_l1d ***CMake*** /opt/conda/envs/rapids/bin/cmake cmake version 3.14.5 CMake suite maintained and supported by Kitware (kitware.com/cmake). ***g++*** /usr/bin/g++ g++ (Ubuntu 7.4.0-1ubuntu1~18.04.1) 7.4.0 Copyright (C) 2017 Free Software Foundation, Inc. This is free software; see the source for copying conditions. There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. ***nvcc*** /usr/local/cuda/bin/nvcc nvcc: NVIDIA (R) Cuda compiler driver Copyright (c) 2005-2018 NVIDIA Corporation Built on Sat_Aug_25_21:08:01_CDT_2018 Cuda compilation tools, release 10.0, V10.0.130 ***Python*** /opt/conda/envs/rapids/bin/python Python 3.7.3 ***Environment Variables*** PATH : /opt/conda/envs/rapids/bin:/opt/conda/condabin:/usr/local/nvidia/bin:/usr/local/cuda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/conda/bin:/conda/bin:/conda/bin LD_LIBRARY_PATH : /opt/conda/envs/rapids/lib:/usr/local/nvidia/lib:/usr/local/nvidia/lib64:/usr/local/cuda/lib64:/usr/local/lib NUMBAPRO_NVVM : NUMBAPRO_LIBDEVICE : CONDA_PREFIX : /opt/conda/envs/rapids PYTHON_PATH : ***conda packages*** /opt/conda/condabin/conda # packages in environment at /opt/conda/envs/rapids: # # Name Version Build Channel _libgcc_mutex 0.1 main conda-forge _tflow_select 2.1.0 gpu absl-py 0.8.1 py37_0 conda-forge aiohttp 3.6.2 py37h516909a_0 conda-forge alabaster 0.7.12 py_0 conda-forge appdirs 1.4.3 py_1 conda-forge arrow-cpp 0.15.0 py37h090bef1_2 conda-forge astor 0.7.1 py_0 conda-forge async-timeout 3.0.1 py_1000 conda-forge attrs 19.3.0 py_0 conda-forge babel 2.7.0 py_0 conda-forge backcall 0.1.0 py_0 conda-forge black 19.10b0 py37_0 conda-forge blas 2.14 openblas conda-forge bleach 3.1.0 py_0 conda-forge bokeh 1.4.0 py37_0 conda-forge boost-cpp 1.70.0 h8e57a91_2 conda-forge brotli 1.0.7 he1b5a44_1000 conda-forge bzip2 1.0.8 h516909a_2 conda-forge c-ares 1.15.0 h516909a_1001 conda-forge ca-certificates 2019.11.28 hecc5488_0 conda-forge cairo 1.16.0 hfb77d84_1002 conda-forge certifi 2019.11.28 py37_0 conda-forge cffi 1.13.2 py37h8022711_0 conda-forge cfitsio 3.470 hb60a0a2_2 conda-forge chardet 3.0.4 py37_1003 conda-forge click 7.0 pypi_0 pypi cloudpickle 1.2.2 py_1 conda-forge clx 0.12.0 pypi_0 pypi cmake 3.14.5 hf94ab9c_0 conda-forge cmake_setuptools 0.1.3 py_0 rapidsai-nightly commonmark 0.9.1 py_0 conda-forge confluent-kafka 1.3.0 pypi_0 pypi cryptography 2.8 py37h72c5cf5_1 conda-forge cudatoolkit 10.0.130 0 nvidia cudf 0.11.0b0+7.g3498c7e7b pypi_0 pypi cudnn 7.6.0 cuda10.0_0 nvidia cugraph 0.11.0b0+1.g5cffc33.dirty pypi_0 pypi cuml 0.11.0a1+1229.gc594c90b pypi_0 pypi cupti 10.0.130 0 cupy 6.6.0 py37h809cb0f_1 conda-forge curl 7.65.3 hf8cf82a_0 conda-forge cuspatial 0.11.0b0 pypi_0 pypi cycler 0.10.0 py_2 conda-forge cython 0.29.14 py37he1b5a44_0 conda-forge cytoolz 0.10.1 py37h516909a_0 conda-forge dask 2.9.0 py_0 conda-forge dask-core 2.9.0 py_0 conda-forge dask-cuda 0.11.0b191211 py37_21 rapidsai-nightly dask-cudf 0.11.0b0+7.g3498c7e7b pypi_0 pypi dask-glm 0.2.0 py_1 conda-forge dask-labextension 1.0.3 py_0 conda-forge dask-ml 1.1.1 py_0 conda-forge dask-xgboost 0.1.5 pypi_0 pypi dbus 1.13.6 he372182_0 conda-forge decorator 4.4.1 py_0 conda-forge defusedxml 0.6.0 py_0 conda-forge distributed 2.9.0 py_0 conda-forge dlpack 0.2 he1b5a44_1 conda-forge docutils 0.15.2 py37_0 conda-forge double-conversion 3.1.5 he1b5a44_2 conda-forge doxygen 1.8.16 hd1b7508_1 conda-forge entrypoints 0.3 py37_1000 conda-forge expat 2.2.5 he1b5a44_1004 conda-forge fastavro 0.22.7 py37h516909a_0 conda-forge fastrlock 0.4 py37he1b5a44_1000 conda-forge flake8 3.7.9 py37_0 conda-forge flatbuffers 1.11.0 he1b5a44_0 conda-forge fontconfig 2.13.1 h86ecdb6_1001 conda-forge freetype 2.10.0 he983fc9_1 conda-forge freexl 1.0.5 h14c3975_1002 conda-forge fribidi 1.0.5 h516909a_1002 conda-forge fsspec 0.6.1 py_0 conda-forge future 0.18.2 py37_0 conda-forge gast 0.2.2 py_0 conda-forge gdal 2.4.3 py37h5f563d9_9 conda-forge geos 3.7.2 he1b5a44_2 conda-forge geotiff 1.5.1 hbd99317_7 conda-forge gettext 0.19.8.1 hc5be6a0_1002 conda-forge gflags 2.2.2 he1b5a44_1002 conda-forge giflib 5.1.7 h516909a_1 conda-forge glib 2.58.3 py37h6f030ca_1002 conda-forge glog 0.4.0 he1b5a44_1 conda-forge gmp 6.1.2 hf484d3e_1000 conda-forge google-pasta 0.1.8 py_0 conda-forge graphite2 1.3.13 hf484d3e_1000 conda-forge graphviz 2.42.3 h0511662_0 conda-forge grpc-cpp 1.23.0 h18db393_0 conda-forge grpcio 1.23.0 py37he9ae1f9_0 conda-forge gst-plugins-base 1.14.5 h0935bb2_0 conda-forge gstreamer 1.14.5 h36ae1b5_0 conda-forge h5py 2.10.0 nompi_py37h513d04c_100 conda-forge harfbuzz 2.4.0 h9f30f68_3 conda-forge hdf4 4.2.13 hf30be14_1003 conda-forge hdf5 1.10.5 nompi_h3c11f04_1104 conda-forge heapdict 1.0.1 py_0 conda-forge hypothesis 4.53.1 py37_0 conda-forge icu 64.2 he1b5a44_1 conda-forge idna 2.8 py37_1000 conda-forge imagesize 1.1.0 py_0 conda-forge importlib_metadata 1.2.0 py37_0 conda-forge ipykernel 5.1.3 py37h5ca1d4c_0 conda-forge ipython 7.3.0 py37h24bf2e0_0 conda-forge ipython_genutils 0.2.0 py_1 conda-forge jedi 0.15.1 py37_0 conda-forge jinja2 2.10.3 py_0 conda-forge joblib 0.14.0 py_0 conda-forge jpeg 9c h14c3975_1001 conda-forge json-c 0.13.1 h14c3975_1001 conda-forge json5 0.8.5 py_0 conda-forge jsonschema 3.2.0 py37_0 conda-forge jupyter-server-proxy 1.2.0 py_0 conda-forge jupyter_client 5.3.3 py37_1 conda-forge jupyter_core 4.6.1 py37_0 conda-forge jupyterlab 1.0.7 py37_0 conda-forge jupyterlab-nvdashboard 0.1.11 pypi_0 pypi jupyterlab_server 1.0.6 py_0 conda-forge kealib 1.4.10 h58c409b_1005 conda-forge keras-applications 1.0.8 py_1 conda-forge keras-preprocessing 1.1.0 py_0 conda-forge kiwisolver 1.1.0 py37hc9558a2_0 conda-forge krb5 1.16.3 h05b26f9_1001 conda-forge lapack 3.6.1 ha44fe06_2 conda-forge ld_impl_linux-64 2.33.1 h53a641e_7 conda-forge libblas 3.8.0 14_openblas conda-forge libcblas 3.8.0 14_openblas conda-forge libclang 8.0.0 h6bb024c_0 rapidsai libcumlprims 0.11.0a191211 cuda10.0_130 rapidsai-nightly libcurl 7.65.3 hda55be3_0 conda-forge libcypher-parser 0.6.2 1 rapidsai libdap4 3.20.4 hd3bb157_0 conda-forge libedit 3.1.20170329 hf8c457e_1001 conda-forge libevent 2.1.10 h72c5cf5_0 conda-forge libffi 3.2.1 he1b5a44_1006 conda-forge libgcc-ng 7.3.0 hdf63c60_2 conda-forge libgdal 2.4.3 h2f07a13_9 conda-forge libgfortran 3.0.0 1 conda-forge libgfortran-ng 7.3.0 hdf63c60_2 conda-forge libhwloc 2.1.0 h3c4fd83_0 conda-forge libiconv 1.15 h516909a_1005 conda-forge libkml 1.3.0 h4fcabce_1010 conda-forge liblapack 3.8.0 14_openblas conda-forge liblapacke 3.8.0 14_openblas conda-forge libllvm8 8.0.1 hc9558a2_0 conda-forge libnetcdf 4.7.1 nompi_h94020b1_102 conda-forge libopenblas 0.3.7 h5ec1e0e_5 conda-forge libpng 1.6.37 hed695b0_0 conda-forge libpq 11.5 hd9ab2ff_2 conda-forge libprotobuf 3.8.0 h8b12597_0 conda-forge libsodium 1.0.17 h516909a_0 conda-forge libspatialite 4.3.0a h4f6d029_1032 conda-forge libssh2 1.8.2 h22169c7_2 conda-forge libstdcxx-ng 7.3.0 hdf63c60_2 conda-forge libtiff 4.1.0 hfc65ed5_0 conda-forge libtool 2.4.6 h14c3975_1002 conda-forge libuuid 2.32.1 h14c3975_1000 conda-forge libuv 1.33.1 h516909a_0 conda-forge libxcb 1.13 h14c3975_1002 conda-forge libxml2 2.9.10 hee79883_0 conda-forge llvmlite 0.30.0 py37h8b12597_1 conda-forge locket 0.2.0 py_2 conda-forge lz4-c 1.8.3 he1b5a44_1001 conda-forge make 4.2.1 h14c3975_2004 conda-forge markdown 3.0.1 pypi_0 pypi markupsafe 1.1.1 py37h516909a_0 conda-forge matplotlib 3.1.2 py37_1 conda-forge matplotlib-base 3.1.2 py37h250f245_1 conda-forge mccabe 0.6.1 py_1 conda-forge mistune 0.8.4 py37h516909a_1000 conda-forge more-itertools 8.0.2 py_0 conda-forge msgpack-python 0.6.2 py37hc9558a2_0 conda-forge multidict 4.6.1 py37h516909a_0 conda-forge multipledispatch 0.6.0 py_0 conda-forge mypy_extensions 0.4.3 py37_0 conda-forge nbconvert 5.6.1 py37_0 conda-forge nbformat 4.4.0 py_1 conda-forge nbsphinx 0.5.0 py_0 conda-forge nccl 2.4.6.1 cuda10.0_0 nvidia ncurses 6.1 hf484d3e_1002 conda-forge networkx 2.4 py_0 conda-forge nodejs 13.0.0 h10a4023_1 conda-forge notebook 6.0.1 py37_0 conda-forge numba 0.46.0 py37hb3f55d8_1 conda-forge numpy 1.17.3 py37h95a1406_0 conda-forge numpydoc 0.9.1 py_0 conda-forge nvstrings-cuda100 0.0.0.dev0 pypi_0 pypi olefile 0.46 py_0 conda-forge openblas 0.3.7 he1df0ab_5 conda-forge openjpeg 2.3.1 h981e76c_3 conda-forge openssl 1.1.1d h516909a_0 conda-forge opt_einsum 3.1.0 py_0 conda-forge packaging 19.2 py_0 conda-forge pandas 0.24.2 py37hb3f55d8_1 conda-forge pandoc 1.19.2 0 conda-forge pandocfilters 1.4.2 py_1 conda-forge pango 1.42.4 ha030887_1 conda-forge parquet-cpp 1.5.1 2 conda-forge parso 0.5.1 py_0 conda-forge partd 1.0.0 py_0 conda-forge pathspec 0.6.0 py_0 conda-forge patsy 0.5.1 py_0 conda-forge pcre 8.43 he1b5a44_0 conda-forge pexpect 4.7.0 py37_0 conda-forge pickleshare 0.7.5 py37_1000 conda-forge pillow 6.2.1 py37hd70f55b_1 conda-forge pip 19.3.1 py37_0 conda-forge pixman 0.38.0 h516909a_1003 conda-forge pluggy 0.13.0 py37_0 conda-forge poppler 0.67.0 h14e79db_8 conda-forge poppler-data 0.4.9 1 conda-forge postgresql 11.5 hc63931a_2 conda-forge proj 6.2.1 hc80f0dc_0 conda-forge prometheus_client 0.7.1 py_0 conda-forge prompt-toolkit 2.0.10 pypi_0 pypi protobuf 3.8.0 py37he1b5a44_2 conda-forge psutil 5.6.7 py37h516909a_0 conda-forge pthread-stubs 0.4 h14c3975_1001 conda-forge ptyprocess 0.6.0 py_1001 conda-forge py 1.8.0 py_0 conda-forge pyarrow 0.15.0 py37h8b68381_1 conda-forge pycodestyle 2.5.0 py_0 conda-forge pycparser 2.19 py37_1 conda-forge pyflakes 2.1.1 py_0 conda-forge pygments 2.5.2 py_0 conda-forge pynvml 8.0.3 py_0 conda-forge pyopenssl 19.1.0 py37_0 conda-forge pyparsing 2.4.5 py_0 conda-forge pyqt 5.9.2 py37hcca6a23_4 conda-forge pyrsistent 0.15.6 py37h516909a_0 conda-forge pysocks 1.7.1 py37_0 conda-forge pytest 5.3.1 py37_0 conda-forge python 3.7.3 h357f687_2 conda-forge python-dateutil 2.8.1 py_0 conda-forge python-whois 0.7.2 pypi_0 pypi pytz 2019.3 py_0 conda-forge pyyaml 5.2 py37h516909a_0 conda-forge pyzmq 18.1.1 py37h1768529_0 conda-forge qt 5.9.7 h0c104cb_3 conda-forge rapidjson 1.1.0 he1b5a44_1002 conda-forge re2 2019.12.01 he1b5a44_0 conda-forge readline 8.0 hf8c457e_0 conda-forge recommonmark 0.6.0 py_0 conda-forge regex 2019.12.9 py37h516909a_0 conda-forge requests 2.22.0 py37_1 conda-forge rhash 1.3.6 h14c3975_1001 conda-forge rmm 0.11.0b0+80.g2e69ec9 pypi_0 pypi scikit-learn 0.22 py37hcdab131_1 conda-forge scipy 1.3.2 py37h921218d_0 conda-forge seaborn 0.9.0 py_2 conda-forge send2trash 1.5.0 py_0 conda-forge setuptools 42.0.2 py37_0 conda-forge simpervisor 0.3 py_1 conda-forge sip 4.19.8 py37hf484d3e_1000 conda-forge six 1.13.0 py37_0 conda-forge snappy 1.1.7 he1b5a44_1002 conda-forge snowballstemmer 2.0.0 py_0 conda-forge sortedcontainers 2.1.0 py_0 conda-forge sphinx 2.2.2 py_0 conda-forge sphinx-markdown-tables 0.0.10 pypi_0 pypi sphinx_rtd_theme 0.4.3 py_0 conda-forge sphinxcontrib-applehelp 1.0.1 py_0 conda-forge sphinxcontrib-devhelp 1.0.1 py_0 conda-forge sphinxcontrib-htmlhelp 1.0.2 py_0 conda-forge sphinxcontrib-jsmath 1.0.1 py_0 conda-forge sphinxcontrib-qthelp 1.0.2 py_0 conda-forge sphinxcontrib-serializinghtml 1.1.3 py_0 conda-forge sphinxcontrib-websupport 1.1.2 py_0 conda-forge sqlite 3.30.1 hcee41ef_0 conda-forge statsmodels 0.10.2 py37hc1659b7_0 conda-forge tblib 1.4.0 py_0 conda-forge tensorboard 2.0.0 pyhb38c66f_1 tensorflow 2.0.0 gpu_py37h768510d_0 tensorflow-base 2.0.0 gpu_py37h0ec5d1f_0 tensorflow-estimator 2.0.0 pyh2649769_0 termcolor 1.1.0 py_2 conda-forge terminado 0.8.3 py37_0 conda-forge testpath 0.4.4 py_0 conda-forge thrift-cpp 0.12.0 hf3afdfd_1004 conda-forge tk 8.6.10 hed695b0_0 conda-forge toml 0.10.0 py_0 conda-forge toolz 0.10.0 py_0 conda-forge torch 1.3.1 pypi_0 pypi tornado 6.0.3 py37h516909a_0 conda-forge traitlets 4.3.3 py37_0 conda-forge typed-ast 1.4.0 py37h516909a_0 conda-forge typing_extensions 3.7.4.1 py37_0 conda-forge tzcode 2019a h516909a_1002 conda-forge ucx 1.7.0rc1+g430ae7e cuda10.0_45 rapidsai-nightly ucx-py 0.11.0a191210+geba981f py37_25 rapidsai-nightly umap-learn 0.3.10 py37_0 conda-forge uriparser 0.9.3 he1b5a44_1 conda-forge urllib3 1.25.7 py37_0 conda-forge wcwidth 0.1.7 py_1 conda-forge webencodings 0.5.1 py_1 conda-forge werkzeug 0.16.0 py_0 conda-forge wheel 0.33.6 py37_0 conda-forge wrapt 1.11.2 py37h516909a_0 conda-forge xerces-c 3.2.2 h8412b87_1004 conda-forge xgboost 1.0.0-SNAPSHOT pypi_0 pypi xorg-kbproto 1.0.7 h14c3975_1002 conda-forge xorg-libice 1.0.10 h516909a_0 conda-forge xorg-libsm 1.2.3 h84519dc_1000 conda-forge xorg-libx11 1.6.9 h516909a_0 conda-forge xorg-libxau 1.0.9 h14c3975_0 conda-forge xorg-libxdmcp 1.1.3 h516909a_0 conda-forge xorg-libxext 1.3.4 h516909a_0 conda-forge xorg-libxpm 3.5.12 h516909a_1002 conda-forge xorg-libxrender 0.9.10 h516909a_1002 conda-forge xorg-libxt 1.1.5 h516909a_1003 conda-forge xorg-renderproto 0.11.1 h14c3975_1002 conda-forge xorg-xextproto 7.3.0 h14c3975_1002 conda-forge xorg-xproto 7.0.31 h14c3975_1007 conda-forge xz 5.2.4 h14c3975_1001 conda-forge yaml 0.2.2 h516909a_1 conda-forge yarl 1.3.0 py37h516909a_1000 conda-forge zeromq 4.3.2 he1b5a44_2 conda-forge zict 1.0.0 pypi_0 pypi zipp 0.6.0 py_0 conda-forge zlib 1.2.11 h516909a_1006 conda-forge zstd 1.4.3 h3b9ef0a_0 conda-forge </pre></details>[FEA] Pseudorandom Number Detection
Describe the solution you'd like
CLX-wrapped feature that will determine if a series of numbers (e.g., time intervals, ports) are likely generated by a random number generator or not.Additional context
There are many tests that can be leveraged to determine if a number is likely true random or not. NIST published guidelines for such tests. It will likely be necessary to implement multiple tests and ensemble the results (could also use simple voting).[BUG] Torch is not available in Docker image but used in CLX sample notebooks
Describe the bug
Torch is not installed in rapidsai/rapidsai image.Steps/Code to reproduce bug
- Start rapidsai docker container
- Got to
localhost:8888- Navigate to
clx\cybert\cybert_example_training.ipynb- Execute import cell
Expected behavior
All imports are loaded.Environment overview (please complete the following information)
- Environment location: Docker
- Method of CLX install: Docker
Environment details
Please run and paste the output of the/rapids/cudf/print_env.shscript here, to gather any other relevant environment details. The script is located in the docker container.**Another bug**: the `/rapids/cudf` folder no longer exists in the container thus I cannot run the `print_env.sh`.[FEA] Cybert class
Is your feature request related to a problem? Please describe.
Need flexibility for cyBERT inference, such as loading a model object then using it for inferenceDescribe the solution you'd like
A python class that allows you to create an object, load model, use model to output parsed logs and confidence scores.
Uses bert-base-cased as default
Compatible with pytorch 1.5 as default
- cybert class
- cybert unit test
- cybert landing page with demo notebook
[FEA] Add flake8 to pre-commit hook
Is your feature request related to a problem? Please describe.
Pre-commit hook does not currently flag flake8 issuesDescribe the solution you'd like
Add flake8 to pre-commit hook https://github.com/rapidsai/clx/blob/branch-0.14/.pre-commit-config.yamlDescribe alternatives you've considered
Separately installing and running flake8
Runningci/checks/style.shscriptAdditional context
It seems likecudfalso has integrated flake8 with pre-commit https://github.com/rapidsai/cudf/blob/branch-0.14/.pre-commit-config.yaml[BUG] Problem with the SplunkNotableParser
Hi all,
I am having problems parsing the splunk fake data.While using the parser on a data set like the following:
"1548825623, search_name=Endpoint - Host With Malware Detected (Quarantined or Waived) - Rule, orig_time=1548825623, dest_ip=10.10.101.109" "1548814514, search_name=Threat - Beta Testing - Machine Learning, orig_time=1548814514, src_ip=10.10.148.174" "1548814508, search_name=Access - Privileged User Accessing More Than Expected Number of Machines in Period - Rule, orig_time=1548814508, user=aaron.abshire" "1548814508, search_name=Threat - Source And Destination Matches - Threat Gen, orig_time=1548814508, src_ip=10.10.189.84, dest_ip=10.10.135.199"using this code:
gdf = cudf.read_csv('splunk_faker_raw') gdf.columns = ['raw'] from clx.parsers.splunk_notable_parser import SplunkNotableParser snp = SplunkNotableParser() parsed_gdf = cudf.DataFrame() parsed_gdf = snp.parse(gdf, 'raw') parsed_gdf.head(1)I get the following result. Apparently only the "time" column gets successfully parsed.
time | search_name | orig_time | urgency | user | owner | security_domain | severity | src_ip | src_mac | ... | dest_priority | device_name | event_name | event_type | id | ip_address | message_ip | message_hostname | message_username | message_description -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- 1548814514 |My env:
rapids v 0.11 clx v 0.12Thanks!
[FEA] Rename BERT vocab and hash file names for disambiguation
Is your feature request related to a problem? Please describe.
current naming of bert-base-uncased files in the resources folder arebert_hash_table.txtandbert_vocab.txtDescribe the solution you'd like
For disambiguation these files should be more descriptively renamed tobert-base-uncased-hash.txtandbert-base-uncased-vocab.txtAdditional context
I believe this renaming will affect the tokenizer test and phishing detection[FEA] CLX Docker images
Describe the solution you'd like
CLX has updated to PyTorch 1.5. This has made it difficult to build and install CLX on the current publishedrapidsaiandrapidsai-nightlyDocker images. This is due to PyTorch 1.5 now having a hard dependency on MKL and therapidsaiimages using OpenBLAS and thenomklconda package installed.
UPDATE: Thenomklconda package has since been removed from the Rapids Docker images so there is no longer a problem installing PyTorch and CLX. However, we will still proceed with building/publishing the CLX images to simplify the process of getting started with CLX.Request CLX docker images that include CLX with all its dependencies (i.e. RAPIDS, PyTorch, MKL, etc) and notebooks already installed and working out of the box. Images should be similar to images in
rapidsaiandrapidsai-nightlywithbase,runtime, anddevelimage types and also be available to pull from Docker Hub.[BUG] GPU tokenizer fails when passed cudf dataframe
Describe the bug
GPU tokenizer has dependency on nvstringsbyte_countwhich appears to not have been ported to Python layer of cudf with the removal of nvstrings from RAPIDS. Issue is seen only when passing a cudf dataframe to the tokenizer.Steps/Code to reproduce bug
df = cudf.read_csv("test.txt") a, b, c = tokenizer.tokenize_df(df, "./hash_table.txt", max_num_sentences=1000000, max_num_chars=100000000, max_rows_tensor=1000000, do_truncate=True) --------------------------------------------------------------------------- AttributeError Traceback (most recent call last) <timed exec> in <module> /conda/envs/gdf/lib/python3.7/site-packages/clx-0.15.0a0+38.g337b436.dirty-py3.7-linux-x86_64.egg/clx/analytics/tokenizer.py in tokenize_df(input_df, hash_file, max_sequence_length, stride, do_lower, do_truncate, max_num_sentences, max_num_chars, max_rows_tensor) 76 >>> tokens, masks, metadata = tokenizer.tokenize_df(df) 77 """ ---> 78 tokens, masks, metadata = tokenizer_wrapper.tokenize_df(input_df, hash_file, max_sequence_length, stride, do_lower, do_truncate, max_num_sentences, max_num_chars, max_rows_tensor) 79 return tokens, masks, metadata clx/analytics/tokenizer_wrapper.pyx in clx.analytics.tokenizer_wrapper.tokenize_df() AttributeError: 'StringMethods' object has no attribute 'byte_count'Expected behavior
Successful return oftokenize_dfEnvironment overview (please complete the following information)
- Environment location: Docker
- Method of CLX install: from source
[FEA] Cybert Streamz Workflow
Is your feature request related to a problem? Please describe.
Request for CLX to integrate the Cybert use case into a streamz workflow using KafkaDescribe the solution you'd like
I'd like to be able to use streamz and Cybert whereas input data would be received from Kafka and processed data will be pushed back to Kafka. This would require Cybert to be integrated into CLX.Describe alternatives you've considered
CLX workflow allows for Kafka batch processingAdditional context
Would it be possible to release the processed apache log csv referenced in the CyBERT example notebook? [QST]
What is your question?
As above, would it be possible to release the processed apache log csv referenced in the CyBERT example notebook?[FEA] Include handling of categorical features in the anomaly detection component.
Is your feature request related to a problem? Please describe.
Add a method of splitting for a categorical features using bag sampling approach for tree based anomaly detection. Customize the node splitting to handle IP address, port, protocol … etc.Additional context
This implementation will have dependencies on ExtraTreeRegression implementation of CuML.[FEA] DGA Detector - string2ascii and train_model
Is your feature request related to a problem? Please describe.
NoDescribe the solution you'd like
Currentlytrain_modelfunction requires the user to convert the input dataframe containing column of domains from string2asciiRequesting that
train_modelfunction convert input dataframe to asciiDescribe alternatives you've considered
Alternative is for user to convert data from string2ascii usingstring2asciifunction within DGADetectorAdditional context
[FEA] CLX docker images
Describe the solution you'd like
CLX has updated to PyTorch 1.5. This has made it difficult to build and install CLX on the current publishedrapidsaiandrapidsai-nightlyDocker images. This is due to PyTorch 1.5 now having a hard dependency on MKL and therapidsaiimages using OpenBLAS and thenomklconda package installed.Request CLX docker images that include CLX with all its dependencies (i.e. RAPIDS, PyTorch 1.5, MKL, etc) and notebooks already installed and working out of the box. Images should be similar to images in
rapidsaiandrapidsai-nightlywithbase,runtime, anddevelimage types and also be available to pull from Docker Hub.[FEA] Add Bokeh visualization back to Alert Analysis notebook
Is your feature request related to a problem? Please describe.
Not related to a problem. Adding back features to a notebook.Describe the solution you'd like
Add the Bokeh visualizations back to the alert analysis notebook to provide options to users and demonstrate the ease of going to the CPU if necessary. Just showing the capability should be sufficient (e.g., a few visualizations).[DOC] README.md RAPIDS icon 404
Report incorrect documentation
Location of incorrect documentation
README.md:1, https://github.com/rapidsai/clx/blob/branch-0.15/README.md#cyber-log-accelerators-clxDescribe the problems or issues found in the documentation
Looks like someone reorganized the static web assets so https://rapids.ai/assets/images/rapids_logo.png doesn't exist anymore. Not sure where the new asset location is.[FEA] Update DGA detection unittest
Is your feature request related to a problem? Please describe.
Retrieving DGA detection pre-trained model from S3 is hard to maintain with pytorch upgrades.Describe the solution you'd like
Update dga detection unittest to generate model within the test script instead getting from AWS s3 path.[FEA] Adapt Refactored cuDF
Is your feature request related to a problem? Please describe.
Test cases which contains cudf Dataframe creation and callingmemattribute is failing due to cudf refactortests/test_ip.py::test_int_to_ip FAILED [ 42%] tests/test_ip.py::test_is_ip FAILED [ 43%] tests/test_ip.py::test_is_reserved FAILED [ 44%] tests/test_ip.py::test_is_loopback FAILED [ 45%] tests/test_ip.py::test_is_link_local FAILED [ 47%] tests/test_ip.py::test_is_unspecified FAILED [ 48%] tests/test_ip.py::test_is_multicast FAILED [ 49%] tests/test_ip.py::test_is_private FAILED [ 50%] tests/test_ip.py::test_is_global FAILED [ 51%] tests/test_ip.py::test_netmask FAILED [ 52%] tests/test_ip.py::test_hostmask FAILED [ 54%] tests/test_ip.py::test_mask FAILED [ 55%]Describe the solution you'd like
Adapting refactored cudf to clx would fix all the issues.[FEA] use cyBERT class with DistilBERT or ELECTRA
Is your feature request related to a problem? Please describe.
Add functionality to cyBERT to load and use a DistilBERT or ELECTRA model in addition to BERT
These three models all use the same subword tokenizer as BERT[FEA] SlashNext CLX Integration
Integrate SlashNext Python API's with CLX.
https://github.com/slashnext/SlashNext-URL-Analysis-and-Enrichment/tree/master/Python%20SDK
[DOC] Update README docker build args
Report incorrect documentation
Location of incorrect documentation
README.mdDescribe the problems or issues found in the documentation
Documentation references old build argimagedocker build --build-arg image=rapidsai/rapidsai-dev-nightly:0.12-cuda9.2-devel-ubuntu18.04-py3.7 -t clx:latest .Steps taken to verify documentation is incorrect
Reviewed DockerfileSuggested fix for documentation
Update README.md instructions to reflect new build argsARG RAPIDS_VERSION=0.13 ARG CUDA_VERSION=10.1 ARG LINUX_VERSION=ubuntu18.04 ARG PYTHON_VERSION=3.7[FEA] CLX module for Supervised Asset Classification
Create CLX module from Supervised Asset Classification notebook. When module is complete, update notebook to use it.
[FEA] Use Dask BlazingSQL for CLX Query
Is your feature request related to a problem? Please describe.
Running CLX Query on terabytes of dataset with cuDF blazingsql has limitations such as out of memory exception.Describe the solution you'd like
Using dask-blazingsql for clx query could avoid the memory exceptions and can use gpu resources based on the availability.[FEA] Save CLX DGA model state as dictionary
Is your feature request related to a problem? Please describe.
Unable to use model single gpu that is trained on multi gpu.Describe the solution you'd like
Instead of saving the whole model checkpointing the state of the model would give the flexibility to run on both.[BUG] clx.analytics.tokenizer gives error to sentence length > 500
Describe the bug
I am trying to make my own GPU-accelerated tokenizer for genomics domain, and below I use hash_vocab from perfect_hash.py to build my own hash file, and run the tokenizer. Tokenizer works fine with around 100 words (e.g., len(['A C G T C G T C G ...]) = 100 ), but above certain length, program dies with the following error:an illegal memory access was encountered in file /opt/conda/envs/rapids/conda-bld/libclx_1597848371306/work/cpp/src/wordPieceTokenizer.cu at line 357
I assured that I have enough GPU memory left. I have installed clx with anaconda, and the versions are new. Using GPU: v100-sxm2-32gb.
Steps/Code to reproduce bug
import itertools from clx.analytics import tokenizer import cudf def make_nmer(n): """ Make a word dictionary for sequence base n-gram. Number of word is 4**n return word2idx: a dictionary converting a unique n-gram base word to the unique idx return wordall: a list containing all possible combinations - Example: word2idx, idx2word, wordall = make_nmer(4) """ print('Processing with {} words...'.format(n)) bases = 'ACGT' # Get all possible combination of special_tokens = ['[UNK]', '[CLS]', '[SEP]'] wordall = special_tokens+[''.join(p) for p in itertools.product(bases, repeat=n)] wordall.sort() word2idx, idx2word = {}, {} for i, nmer in enumerate(wordall): word2idx[nmer] = i idx2word[i] = nmer print('All possible word for {}-gram is {}'.format(n, len(wordall))) return word2idx, idx2word, wordall # Making my own dictionary and vocab list _,_,wordall=make_nmer(1) word_txt='\n'.join(wordall) # For replication, please set the path for saving the vocab list / and hash file, respectively word_dir = '<set your path>' hash_dir = '<set your path>' # Saving the vocab to fit for the hash_vocab function with open(word_dir, 'w') as f: f.write(word_txt) hash_vocab(word_dir, hash_dir, compact=False) # Create some example data that has three sequences with a fixed lengths of wordlen *4 wordlen = 500 example_data = cudf.Series(['A C G T '*wordlen, 'A C G T '*wordlen, 'A C G T '*wordlen]) tensor, attention_mask, meta_data = tokenizer.tokenize_df(example_data, hash_file=hash_dir, max_sequence_length=wordlen*4, stride=50, do_lower=False, do_truncate=False, max_num_sentences=4, max_num_chars=10000, max_rows_tensor=10)Expected behavior
It should return tokenized tensor with the pattern of [0,1,2,3,0,1,2,3....]Environment overview (please complete the following information)
- Method of CLX install: [conda]
Environment details
Click here to see environment details
**git*** Not inside a git repository ***OS Information*** DISTRIB_ID=Ubuntu DISTRIB_RELEASE=16.04 DISTRIB_CODENAME=xenial DISTRIB_DESCRIPTION="Ubuntu 16.04.6 LTS" NAME="Ubuntu" VERSION="16.04.6 LTS (Xenial Xerus)" ID=ubuntu ID_LIKE=debian PRETTY_NAME="Ubuntu 16.04.6 LTS" VERSION_ID="16.04" HOME_URL="http://www.ubuntu.com/" SUPPORT_URL="http://help.ubuntu.com/" BUG_REPORT_URL="http://bugs.launchpad.net/ubuntu/" VERSION_CODENAME=xenial UBUNTU_CODENAME=xenial Linux nipa2020-0929 4.4.0-176-generic #206-Ubuntu SMP Fri Feb 28 05:02:04 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux ***GPU Information*** Thu Aug 20 15:49:39 2020 +-----------------------------------------------------------------------------+ | NVIDIA-SMI 418.67 Driver Version: 418.67 CUDA Version: 10.1 | |-------------------------------+----------------------+----------------------+ | GPU Name Persistence-M| Bus-Id Disp.A | Volatile Uncorr. ECC | | Fan Temp Perf Pwr:Usage/Cap| Memory-Usage | GPU-Util Compute M. | |===============================+======================+======================| | 0 Tesla V100-SXM2... On | 00000000:00:05.0 Off | Off | | N/A 32C P0 43W / 300W | 0MiB / 32480MiB | 0% Default | +-------------------------------+----------------------+----------------------+ | 1 Tesla V100-SXM2... On | 00000000:00:06.0 Off | Off | | N/A 35C P0 43W / 300W | 0MiB / 32480MiB | 0% Default | +-------------------------------+----------------------+----------------------+ | 2 Tesla V100-SXM2... On | 00000000:00:07.0 Off | Off | | N/A 31C P0 42W / 300W | 0MiB / 32480MiB | 0% Default | +-------------------------------+----------------------+----------------------+ +-----------------------------------------------------------------------------+ | Processes: GPU Memory | | GPU PID Type Process name Usage | |=============================================================================| | No running processes found | +-----------------------------------------------------------------------------+ ***CPU*** Architecture: x86_64 CPU op-mode(s): 32-bit, 64-bit Byte Order: Little Endian CPU(s): 24 On-line CPU(s) list: 0-23 Thread(s) per core: 1 Core(s) per socket: 24 Socket(s): 1 NUMA node(s): 1 Vendor ID: GenuineIntel CPU family: 6 Model: 85 Model name: Intel(R) Xeon(R) Gold 5120 CPU @ 2.20GHz Stepping: 4 CPU MHz: 2200.112 BogoMIPS: 4400.22 Hypervisor vendor: Xen Virtualization type: full L1d cache: 32K L1i cache: 32K L2 cache: 1024K L3 cache: 19712K NUMA node0 CPU(s): 0-23 Flags: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush acpi mmx fxsr sse sse2 ht syscall nx pdpe1gb rdtscp lm constant_tsc rep_good nopl nonstop_tsc pni pclmulqdq ssse3 fma cx16 pcid sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand hypervisor lahf_lm abm 3dnowprefetch invpcid_single ssbd ibrs ibpb stibp kaiser fsgsbase tsc_adjust bmi1 hle avx2 smep bmi2 erms invpcid rtm mpx rdseed adx smap clflushopt clwb xsaveopt xsavec xgetbv1 pku md_clear flush_l1d ***CMake*** ***g++*** /usr/bin/g++ g++ (Ubuntu 5.4.0-6ubuntu1~16.04.12) 5.4.0 20160609 Copyright (C) 2015 Free Software Foundation, Inc. This is free software; see the source for copying conditions. There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. ***nvcc*** /usr/local/cuda/bin/nvcc nvcc: NVIDIA (R) Cuda compiler driver Copyright (c) 2005-2018 NVIDIA Corporation Built on Sat_Aug_25_21:08:01_CDT_2018 Cuda compilation tools, release 10.0, V10.0.130 ***Python*** /home/ubuntu/anaconda3/envs/py3clx/bin/python Python 3.8.5 ***Environment Variables*** PATH : /home/ubuntu/anaconda3/envs/py3clx/bin:/home/ubuntu/anaconda3/envs/pytorch_p36/bin:/home/ubuntu/bin:/home/ubuntu/.local/bin:/home/ubuntu/anaconda3/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/local/cuda/bin LD_LIBRARY_PATH : /home/ubuntu/anaconda3/envs/pytorch_p36/lib/python3.6/site-packages/torch/lib/:/usr/local/cuda-10.0/lib64:/usr/local/cuda-10.0/extras/CUPTI/lib64:/usr/local/cuda-10.0/lib: NUMBAPRO_NVVM : NUMBAPRO_LIBDEVICE : CONDA_PREFIX : /home/ubuntu/anaconda3/envs/py3clx PYTHON_PATH : ***conda packages*** /home/ubuntu/anaconda3/bin/conda # packages in environment at /home/ubuntu/anaconda3/envs/py3clx: # # Name Version Build Channel _libgcc_mutex 0.1 conda_forge conda-forge _openmp_mutex 4.5 1_llvm conda-forge abseil-cpp 20200225.2 he1b5a44_2 conda-forge arrow-cpp 0.17.1 py38h1234567_11_cuda conda-forge arrow-cpp-proc 1.0.0 cuda conda-forge aws-sdk-cpp 1.7.164 hba45d7a_2 conda-forge backcall 0.2.0 <pip> biopython 1.77 <pip> blas 2.16 mkl conda-forge bokeh 2.1.1 py38h32f6830_0 conda-forge boost-cpp 1.72.0 h7b93d67_2 conda-forge boto3 1.14.45 pyh9f0ad1d_0 conda-forge botocore 1.17.45 pyh9f0ad1d_0 conda-forge brotli 1.0.7 he1b5a44_1004 conda-forge brotlipy 0.7.0 py38h1e0a361_1000 conda-forge bzip2 1.0.8 h516909a_2 conda-forge c-ares 1.16.1 h516909a_0 conda-forge ca-certificates 2020.6.20 hecda079_0 conda-forge certifi 2020.6.20 py38h32f6830_0 conda-forge cffi 1.14.1 py38he30daa8_0 chardet 3.0.4 py38h32f6830_1006 conda-forge click 7.1.2 pyh9f0ad1d_0 conda-forge cloudpickle 1.5.0 py_0 conda-forge clx 0.16.0a200819 py38_g0ae031a_26 rapidsai-nightly cryptography 3.0 py38h766eaa4_0 conda-forge cudatoolkit 10.1.243 h6bb024c_0 nvidia cudf 0.16.0a200819 cuda_10.1_py38_g08151d9df_1002 rapidsai-nightly cudnn 7.6.0 cuda10.1_0 nvidia cugraph 0.16.0a200819 py38_gcfeddf7b_224 rapidsai-nightly cuml 0.16.0a200819 cuda10.1_py38_gf1f63468e_268 rapidsai-nightly cupy 7.8.0 py38hd239d08_0 conda-forge curl 7.71.1 he644dc0_5 conda-forge cytoolz 0.10.1 py38h516909a_0 conda-forge dask 2.23.0 py_0 conda-forge dask-core 2.23.0 py_0 conda-forge dask-cudf 0.16.0a200819 py38_g08151d9df_1002 rapidsai-nightly decorator 4.4.2 <pip> distributed 2.23.0 py38h32f6830_0 conda-forge dlpack 0.3 he1b5a44_1 conda-forge docutils 0.15.2 py38_0 conda-forge double-conversion 3.1.5 he1b5a44_2 conda-forge faiss-proc 1.0.0 cuda rapidsai-nightly fastavro 0.24.2 py38h1e0a361_0 conda-forge fastrlock 0.5 py38h950e882_0 conda-forge freetype 2.10.2 he06d7ca_0 conda-forge fsspec 0.8.0 py_0 conda-forge gflags 2.2.2 he1b5a44_1004 conda-forge glog 0.4.0 h49b9bf7_3 conda-forge grpc-cpp 1.30.2 heedbac9_0 conda-forge heapdict 1.0.1 py_0 conda-forge icu 67.1 he1b5a44_0 conda-forge idna 2.10 pyh9f0ad1d_0 conda-forge ipykernel 5.3.4 <pip> ipython 7.17.0 <pip> ipython-genutils 0.2.0 <pip> jedi 0.17.2 <pip> jinja2 2.11.2 pyh9f0ad1d_0 conda-forge jmespath 0.10.0 pyh9f0ad1d_0 conda-forge joblib 0.16.0 py_0 conda-forge jpeg 9d h516909a_0 conda-forge jupyter-client 6.1.6 <pip> jupyter-core 4.6.3 <pip> krb5 1.17.1 hfafb76e_2 conda-forge lcms2 2.11 hbd6801e_0 conda-forge ld_impl_linux-64 2.33.1 h53a641e_7 libblas 3.8.0 16_mkl conda-forge libcblas 3.8.0 16_mkl conda-forge libclx 0.16.0a200819 cuda10.1_g0ae031a_26 rapidsai-nightly libcudf 0.16.0a200819 cuda10.1_g08151d9df_1002 rapidsai-nightly libcugraph 0.16.0a200819 cuda10.1_gcfeddf7b_224 rapidsai-nightly libcuml 0.16.0a200819 cuda10.1_gf1f63468e_268 rapidsai-nightly libcumlprims 0.15.0a200812 cuda10.1_61 rapidsai-nightly libcurl 7.71.1 hcdd3856_5 conda-forge libedit 3.1.20191231 h14c3975_1 libev 4.33 h516909a_0 conda-forge libevent 2.1.10 hcdb4288_1 conda-forge libfaiss 1.6.3 he68dc02_1_cuda conda-forge libffi 3.3 he6710b0_2 libgcc-ng 9.3.0 h24d8f2e_15 conda-forge libgfortran-ng 7.5.0 hdf63c60_15 conda-forge libhwloc 2.1.0 h3c4fd83_0 conda-forge libiconv 1.16 h516909a_0 conda-forge liblapack 3.8.0 16_mkl conda-forge liblapacke 3.8.0 16_mkl conda-forge libllvm10 10.0.1 he513fc3_1 conda-forge libnghttp2 1.41.0 hab1572f_1 conda-forge libpng 1.6.37 hed695b0_2 conda-forge libprotobuf 3.12.4 h8b12597_0 conda-forge librmm 0.16.0a200819 cuda10.1_g466b8ed_255 rapidsai-nightly libssh2 1.9.0 hab1572f_5 conda-forge libstdcxx-ng 9.1.0 hdf63c60_0 libtiff 4.1.0 hc7e4089_6 conda-forge libwebp-base 1.1.0 h516909a_3 conda-forge libxml2 2.9.10 h68273f3_2 conda-forge llvm-openmp 10.0.1 hc9558a2_0 conda-forge llvmlite 0.34.0 py38h4f45e52_0 conda-forge locket 0.2.0 py_2 conda-forge lz4-c 1.9.2 he1b5a44_2 conda-forge markupsafe 1.1.1 py38h1e0a361_1 conda-forge mkl 2020.2 256 conda-forge msgpack-python 1.0.0 py38hbf85e49_1 conda-forge nccl 2.7.8.1 h51cf6c1_0 conda-forge ncurses 6.2 he6710b0_1 ninja 1.10.1 hbf85e49_0 conda-forge numba 0.51.0 py38hc5bc63f_0 conda-forge numpy 1.19.1 py38h8854b6b_0 conda-forge olefile 0.46 py_0 conda-forge openssl 1.1.1g h516909a_1 conda-forge packaging 20.4 pyh9f0ad1d_0 conda-forge pandas 1.0.5 py38hcb8c335_0 conda-forge parquet-cpp 1.5.1 2 conda-forge parso 0.7.1 <pip> partd 1.1.0 py_0 conda-forge pexpect 4.8.0 <pip> pickleshare 0.7.5 <pip> pillow 7.2.0 py38h9776b28_1 conda-forge pip 20.2.2 py_0 conda-forge prompt-toolkit 3.0.6 <pip> psutil 5.7.2 py38h1e0a361_0 conda-forge ptyprocess 0.6.0 <pip> pyarrow 0.17.1 py38h1234567_11_cuda conda-forge pycparser 2.20 pyh9f0ad1d_2 conda-forge Pygments 2.6.1 <pip> pyopenssl 19.1.0 py_1 conda-forge pyparsing 2.4.7 pyh9f0ad1d_0 conda-forge pysocks 1.7.1 py38h32f6830_1 conda-forge python 3.8.5 hcff3b4d_0 python-dateutil 2.8.1 py_0 conda-forge python_abi 3.8 1_cp38 conda-forge pytorch 1.6.0 py3.8_cuda10.1.243_cudnn7.6.3_0 pytorch pytz 2020.1 pyh9f0ad1d_0 conda-forge pyyaml 5.3.1 py38h1e0a361_0 conda-forge pyzmq 19.0.2 <pip> re2 2020.07.06 he1b5a44_1 conda-forge readline 8.0 h7b6447c_0 regex 2020.7.14 py38h1e0a361_0 conda-forge requests 2.24.0 pyh9f0ad1d_0 conda-forge rmm 0.16.0a200819 cuda_10.1_py38_g466b8ed_255 rapidsai-nightly s3transfer 0.3.3 py38h32f6830_1 conda-forge sacremoses 0.0.43 pyh9f0ad1d_0 conda-forge scikit-learn 0.23.2 py38hee58b96_0 conda-forge scipy 1.5.2 py38h8c5af15_0 conda-forge setuptools 49.6.0 py38h32f6830_0 conda-forge six 1.15.0 pyh9f0ad1d_0 conda-forge snappy 1.1.8 he1b5a44_3 conda-forge sortedcontainers 2.2.2 pyh9f0ad1d_0 conda-forge spdlog 1.7.0 hc9558a2_2 conda-forge sqlite 3.32.3 h62c20be_0 tblib 1.6.0 py_0 conda-forge threadpoolctl 2.1.0 pyh5ca1d4c_0 conda-forge thrift-cpp 0.13.0 h62aa4f2_3 conda-forge tk 8.6.10 hbc83047_0 toolz 0.10.0 py_0 conda-forge torchvision 0.7.0 py38_cu101 pytorch tornado 6.0.4 py38h1e0a361_1 conda-forge tqdm 4.48.2 pyh9f0ad1d_0 conda-forge traitlets 4.3.3 <pip> transformers 2.1.1 py_0 conda-forge treelite 0.92 py38h4e709cc_2 conda-forge typing_extensions 3.7.4.2 py_0 conda-forge ucx 1.8.1+g6b29558 ha5db111_0 rapidsai-nightly ucx-py 0.16.0a200819+g6b29558 py38_75 rapidsai-nightly urllib3 1.25.10 py_0 conda-forge wcwidth 0.2.5 <pip> wheel 0.35.1 pyh9f0ad1d_0 conda-forge xz 5.2.5 h7b6447c_0 yaml 0.2.5 h516909a_0 conda-forge zict 2.0.0 py_0 conda-forge zlib 1.2.11 h7b6447c_3 zstd 1.4.5 h6597ccf_2 conda-forgeRecommend Projects
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
TensorFlow
An Open Source Machine Learning Framework for Everyone
Django
The Web framework for perfectionists with deadlines.
Laravel
A PHP framework for web artisans
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
Recommend Topics
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
web
Some thing interesting about web. New door for the world.
server
A server is a program made to process requests and deliver data to clients.
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Visualization
Some thing interesting about visualization, use data art
Game
Some thing interesting about game, make everyone happy.
Recommend Org
We are working to build community through open source technology. NB: members must have two-factor auth.
Microsoft
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba
Alibaba Open Source for everyone
D3
Data-Driven Documents codes.
Tencent
China tencent open source team.
Facebook
Google
