anhquan0412/basic_model_scratch

Implementation of some classic Machine Learning model from scratch and benchmarking against popular ML library

basic_model_scratch's Issues

forgot to fix Adam + L2 here

basic_model_scratch/model/neural_network.py

Line 82 in fd83305

grad_w += (l2/bs) * self.weights[-1][0] # l2 reg

Fixing Adam+L2

basic_model_scratch/model/neural_network.py

Line 97 in b701c59

grad_w+= (l2/bs) * self.weights[i][0] # l2 reg

Fixing Adam+L2 with AdamW by adding weight decay term AFTER the exponentially moving average for grad_w.

Remove line 97 and change line 100 to self.weights[i][0]-= lr*(grad_w + (l2/bs) * self.weights[i][0])

Related article on AdamW: https://towardsdatascience.com/why-adamw-matters-736223f31b5d

TypeError: torch.index_select received an invalid combination of arguments - got (torch.FloatTensor, int, torch.cuda.LongTensor), but expected (torch.FloatTensor source, int dim, torch.LongTensor index)

Thanks a lot for your work! but when I run the code under "Model 1: Multiplying 2 embedding matrices",
learner.fit(1e-2, 3, wds=1e-4, cycle_len=1,cycle_mult=2)
there occurs a error and I had no idea to it, Could you lend me a hand?

the environment is fastai based on paper'pass

explain what inp is in grad_inp

basic_model_scratch/model/neural_network.py

Line 77 in fd83305

grad_wrt_input = logloss_softmax_grad(y,y_pred) # (n,10)

inp is (wx + b), activation is act(inp) or act(wx+b)

Rewrite this comment

basic_model_scratch/model/random_forest.py

Line 16 in b701c59

# generate random idxs with size sample_sz. Thus this model is not suitable for time-related dataset

Nonsense, since there is no train-val set split. Sample_idxs is for training only, so it doesn't matter if time-related features' order is mixed, since they will be sorted anyway when choosing best split