seondong / customs-fraud-detection Goto Github PK
View Code? Open in Web Editor NEWSimulation framework for customs fraud detection using import declarations.
License: MIT License
Simulation framework for customs fraud detection using import declarations.
License: MIT License
Some preprocessed data contains an infinite value, and it causes errors during the training.
Please add error handling code in dataset.py, firstCheck() in Import_declarations class.
When I select a GPU and run the deepSAD.py, I face a module error in the training phase.
Code: `export CUDA_VISIBLE_DEVICES=3 && python main.py --data real-n --batch_size 128 --sampling deepSAD --mode scratch --train_from 20130101 --test_from 20130701 --test_length 15 --valid_length 15 --initial_inspection_rate 30 --final_inspection_rate 5 --epoch 5 --closs bce --rloss full --save 0 --numweeks 100 --inspection_plan direct_decay
Line that faces Error message:
self.model.module.get_average_hidden_vec(train_loader)
Error message:
Training deepSAD model ...
Traceback (most recent call last):
File "main.py", line 341, in <module>
chosen = sampler.query(num_samples)
File "/home/sundong/WCO/sundong/DATE-active/WCO-project/query_strategies/deepSAD.py", line 198, in query
self.train_deepSAD_model()
File "/home/sundong/WCO/sundong/DATE-active/WCO-project/query_strategies/deepSAD.py", line 183, in train_deepSAD_model
self.date_model.train(self.args)
File "/home/sundong/WCO/sundong/DATE-active/WCO-project/query_strategies/deepSAD.py", line 478, in train
self.model.module.get_average_hidden_vec(train_loader)
File "/home/sundong/anaconda3/envs/py37/lib/python3.7/site-packages/torch/nn/modules/module.py", line 576, in __getattr__
type(self).__name__, name))
AttributeError: 'AnomalyDATEModel' object has no attribute 'module'
Traceback (most recent call last):
File "main.py", line 348, in <module>
logger.info("%s, %s, %s", len(set(chosen)), len(chosen), num_samples)
NameError: name 'chosen' is not defined
Also, 'temp' directory is not initialized, save intermediary file in /intermediary/ directory.
torch.save(normality_scores, "intermediary/multideepSAD_models/normality_scores_valid{}.ckpt".format(epoch))
torch.save(self.data.valid_cls_label, "intermediary/multideepSAD_models/xgb_validy{}.ckpt".format(epoch))
Add few lines in main.py Line 147
pathlib.Path('./intermediary/deepSAD_models').mkdir(parents=True, exist_ok=True)
pathlib.Path('./intermediary/multideepSAD_models').mkdir(parents=True, exist_ok=True)
Make self-supervsied algorithm for uncertainty evaluation
Input: A tuple data, destructable into (train_loader, valid_loader, test_loader, leaf_num, importer_size, item_size, xgb_validy, xgb_testy, revenue_valid, revenue_test)
Output: A list of score with the same length as test_loader.datasets.tensors[-1].shape[0] (test set size)
Make fine-tune and re-initialization mode
For now, the model evaluation still uses the result from DATE.
Need to change to active_DATE (upDATE)
Write sampling strategy for Random,
Integrate revenue and uncertainty on bagde_DATE.py
Testing on NTM data
add xgb, xgb+lr sampler.
Hi, if we want to use the most recent data as a validation set, I think the values in splitter be changed when new weekly data is added to the training set? (active_DATE.py, line 170)
I'm not sure the module 'adahybrid' is working as you intended.
Some errors:
num_sample = 0
for one subsampler.weight_sampler.update_dists(1-norm_precision)
does not provide any meaningful update. After update_dists, weight_sampler.l
is still{0: 0, 1: 0, 2: 0, 3: 0, 4: 0, 5: 0, 6: 0, 7: 0, 8: 0, 9: 0.0, 10: 0}
.Use debugger (pdb.set_trace()
) to check if the code works as desired, and then commit again.
Sampling only from DATE
This issue will be solved by #32 .
@Sungwon-Han do it please
For the final ablation study and model decision, design various uncertainty determination model and compare them.
Removed the generate_loader dependency on the random sampler
Removed the generate_loader dependency on the xgb sampler
Now random and xgb sampler will run without generate_loader.py
Start writing the documentation (README)
This is a minor issue, but it should be considered later
line 238, active_DATE.py
What if there is an intersection between two strategies?
assert len(chosen) == num_samples
If the assertion error occurs, the model should select additional samples for fair comparison. (amount: num_samples-len(chosen))
Building experiment plan and realize with bash code
Make a superclass called Strategy
Make a query strategy from 2 other strategies
Tasks:
Important changes:
badge.py, badge_DATE.py and uncertainty.py is migrated to folder query_strategies
uncertainty.py is renamed as utils.py (query_strategies/utils.py)
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.