rixwew / pytorch-fm Goto Github PK
View Code? Open in Web Editor NEWFactorization Machine models in PyTorch
License: MIT License
Factorization Machine models in PyTorch
License: MIT License
按照论文,FNN模型(即 FactorizationSupportedNeuralNetworkModel)应该是利用了FM模型pre-traing后的隐向量作为MLP的输入,但是我在您的FactorizationSupportedNeuralNetworkModel的code中发现并没有实现这一点。而且FNN model(即FactorizationSupportedNeuralNetworkModel) 的实现方式与您写的Wide&Deep model 的Deep部分的实现方式一模一样,这应该是不对。
希望大佬可以探讨一下,或许是我理解不到位,或者是代码实现有问题,谢谢!(抱歉,这部分拿英文复述我怕自己说得不明确,因此用中文)
Traceback (most recent call last): File "C:\Users\Faaiz-PC\Downloads\Avazu\main.py", line 189, in <module> main(args.dataset_name, File "C:\Users\Faaiz-PC\Downloads\Avazu\main.py", line 151, in main dataset = get_dataset(dataset_name, dataset_path) File "C:\Users\Faaiz-PC\Downloads\Avazu\main.py", line 35, in get_dataset return AvazuDataset(path) File "C:\Users\Faaiz-PC\anaconda3\envs\fnfm\lib\site-packages\torchfm\dataset\avazu.py", line 39, in __init__ self.field_dims = np.frombuffer(txn.get(b'field_dims'), dtype=np.uint32) TypeError: a bytes-like object is required, not 'NoneType'
I am trying to Trying to run fnfm on avazu dataset. this is the error I am getting. Can you help me fix this?
HI, thanks for the amazing library.
I do not quite understand how the data should be fed. If i create a feature vector composed by real values, i receive an error because the embedding function can handle only Long tensors, i.e., integers.
I would suggest that you add some lines to understand how data should be prepared, as the documentation lack of this content.
Thanks and best regards
When I run the main.py, I get the following error:
Traceback (most recent call last):
File "main.py", line 16, in
from torchfm.model.hofm import HighOrderFactorizationMachineModel
ModuleNotFoundError: No module named 'torchfm.model.hofm'
Is it necessary to set the arg padding_idx
of torch.nn.embbedding
, as all features start from 1.
I think it's better to avoid influence of 0.
I noticed that for the deemFM model, the implemented mlp_dim is (16, 16). However in the original deepFM paper, the network structure specified is 400-400-400 (screenshot below). May I ask why is the network structure different?
In the forward process, the dimension of input X in all models are (batch_size, field_size), here can I treat all these field size as feature size?
can it be used for 1TB Criteo data on local computer
so it needs to read and process data by parts: sliding window?
class FactorizationMachine(torch.nn.Module):
def __init__(self, reduce_sum=True):
super().__init__()
self.reduce_sum = reduce_sum
def forward(self, x):
"""
:param x: Float tensor of size ``(batch_size, num_fields, embed_dim)``
"""
square_of_sum = torch.sum(x, dim=1) ** 2
sum_of_square = torch.sum(x ** 2, dim=1)
ix = square_of_sum - sum_of_square
if self.reduce_sum:
ix = torch.sum(ix, dim=1, keepdim=True)
return 0.5 * ix
What happened to v parameters?
Dear DaLao:
what's the function of "self.offsets" ?
self.offsets = np.array((0, *np.cumsum(field_dims)[:-1]), dtype=np.long)
def forward(self, x):
x = x + x.new_tensor(self.offsets).unsqueeze(0)
It will ignore the first data when the header is not None.
args = parser.parse_args()
change to :
args = parser.parse_args(args=[])
论文中每一次multi-head attention后都要做residual
The original link of dataset Criteo is no longer available. Here I found a new one:
field_dims 这个有啥用,有没有详细的案例?
Hi,
In layer.py, class FeaturesEmbedding
, one finds the line
self.offsets = np.array(0, *np.cumsum(field_dims)[:-1]), dtype=np.long)
My code generates an error. I replaced np.array
by torch.tensor
and np.cumsum
by torch.cumsum
, and the code works.
I believe the code as it is has a bug, which was fixed with my change. Could you please confirm this? I am not sure how the code can work as is. thanks.
Hi,
When nfm
is used as model, the return value (batch_size, batch_size)
does not have the right shape.
This will cause a error when calculating the loss.
why the field use label encode? not one-hot encode?
self.fc = torch.nn.Embedding(sum(field_dims)+1, output_dim)
https://github.com/rixwew/pytorch-fm/blob/master/torchfm/layer.py#L10
I'm wondering whether this is a writing misktake
关于CIN的结构,是否在卷积层前将二三维度调换一下?
when you forward(), the last layer of cin is not split half, while the fc_input_dim still adds cross_layer_size//=2.
In FM and HOFM papers, there is a hyper parameter k which helps to control the size of latent factor. Just wondering where is the k in the code? Is that same as field_dims?
Paper said :
The first hidden layer is fully connected with the product layer. The inputs to it consist of linear
signals lz and quadratic signals lp. With respect to lz and lp inputs, separately, the formulation of l1 is: l1 = relu(lz + lp + b1),
But in your code, the linear signals and quadratic signals are concated :
embed_x = self.embedding(x)
cross_term = self.pn(embed_x)
x = torch.cat([embed_x.view(-1, self.embed_output_dim), cross_term], dim=1)
x = self.mlp(x)
return torch.sigmoid(x.squeeze(1))
in rendle's paper, the optimized parameter is b w and v, why use emdedding vector here?
你好,你对于Class MultiLayerPerceptron里的forward function中的x的注释错误
原注释::param x: Float tensor of size (batch_size, num_fields, embed_dim)
应改为: :param x: Float tensor of size (batch_size, embed_dim)
RuntimeError Traceback (most recent call last)
in
21 args.weight_decay,
22 args.device,
---> 23 args.save_dir)
in main(dataset_name, dataset_path, model_name, epoch, learning_rate, batch_size, weight_decay, device, save_dir)
23 early_stopper = EarlyStopper(num_trials=2, save_path=f'{save_dir}/{model_name}.pt')
24 for epoch_i in range(epoch):
---> 25 train(model, optimizer, train_data_loader, criterion, device)
26 auc = test(model, valid_data_loader, device)
27 print('epoch:', epoch_i, 'validation: auc:', auc)
in train(model, optimizer, data_loader, criterion, device, log_interval)
5 for i, (fields, target) in enumerate(tk0):
6 fields, target = fields.to(device), target.to(device)
----> 7 y = model(fields)
8 loss = criterion(y, target.float())
9 model.zero_grad()
H:\Anaconda\lib\site-packages\torch\nn\modules\module.py in _call_impl(self, *input, **kwargs)
720 result = self._slow_forward(*input, **kwargs)
721 else:
--> 722 result = self.forward(*input, **kwargs)
723 for hook in itertools.chain(
724 _global_forward_hooks.values(),
~\Desktop\量化炒股\pytorch-fm-master\torchfm\model\fm.py in forward(self, x)
22 :param x: Long tensor of size (batch_size, num_fields)
23 """
---> 24 x = self.linear(x) + self.fm(self.embedding(x))
25 return torch.sigmoid(x.squeeze(1))
H:\Anaconda\lib\site-packages\torch\nn\modules\module.py in _call_impl(self, *input, **kwargs)
720 result = self._slow_forward(*input, **kwargs)
721 else:
--> 722 result = self.forward(*input, **kwargs)
723 for hook in itertools.chain(
724 _global_forward_hooks.values(),
~\Desktop\量化炒股\pytorch-fm-master\torchfm\layer.py in forward(self, x)
17 """
18 x = x + x.new_tensor(self.offsets).unsqueeze(0)
---> 19 return torch.sum(self.fc(x), dim=1) + self.bias
20
21
H:\Anaconda\lib\site-packages\torch\nn\modules\module.py in _call_impl(self, *input, **kwargs)
720 result = self._slow_forward(*input, **kwargs)
721 else:
--> 722 result = self.forward(*input, **kwargs)
723 for hook in itertools.chain(
724 _global_forward_hooks.values(),
H:\Anaconda\lib\site-packages\torch\nn\modules\sparse.py in forward(self, input)
124 return F.embedding(
125 input, self.weight, self.padding_idx, self.max_norm,
--> 126 self.norm_type, self.scale_grad_by_freq, self.sparse)
127
128 def extra_repr(self) -> str:
H:\Anaconda\lib\site-packages\torch\nn\functional.py in embedding(input, weight, padding_idx, max_norm, norm_type, scale_grad_by_freq, sparse)
1812 # remove once script supports set_grad_enabled
1813 no_grad_embedding_renorm(weight, input, max_norm, norm_type)
-> 1814 return torch.embedding(weight, input, padding_idx, scale_grad_by_freq, sparse)
1815
1816
RuntimeError: Expected tensor for argument #1 'indices' to have scalar type Long; but got torch.IntTensor instead (while checking arguments for embedding)
How to add non-categorical features such as human age to deepfm model? It seems the deepfm implementation only supports categorical feature?
can you share link to similar repo but plain python code
at least for simple models?
In FeaturesLinear
def __init__(self, field_dims, output_dim=1):
super().__init__()
self.fc = torch.nn.Embedding(sum(field_dims), output_dim)
self.bias = torch.nn.Parameter(torch.zeros((output_dim,)))
self.offsets = np.array((0, *np.cumsum(field_dims)[:-1]), dtype=np.long)
def forward(self, x):
"""
:param x: Long tensor of size ``(batch_size, num_fields)``
"""
x = x + x.new_tensor(self.offsets).unsqueeze(0)
return torch.sum(self.fc(x), dim=1) + self.bias
The model only assign weight value for each field as embedding, but it didn't consider the data value here.
For example, if we has a data as [[1,0]], the model will work embedding("1")+embedding("0"), but in real, we need work embedding("1")*1+embedding("0")*0.
What reason the model delete the effect of data value?
Hi, congrats on your great work!
I was wondering why you add a offset before embedding?
Line 27 in a4cfa99
As I am looking at the implementation of xDeepFM, I didn't find the corresponding code in original tensorflow implementation.
np.int and np.long are deprecated since numpy 1.20.0. It should be replaced by int (or np.int32) and np.int64.
After you train an instance of a model using PyTorch, how do you use that model to get the top 10 rated items, i.e., how do you get a model prediction.
Hi, I googled this error but didn't help solve this error.
Traceback (most recent call last):
File "/root/pytorchfm/examples/main.py", line 157, in <module>
args.save_dir)
File "/root/pytorchfm/examples/main.py", line 127, in main
train(model, optimizer, train_data_loader, criterion, device)
File "/root/pytorchfm/examples/main.py", line 81, in train
y = model(fields)
File "/root/anaconda3/lib/python3.7/site-packages/torch/nn/modules/module.py", line 493, in __call__
result = self.forward(*input, **kwargs)
File "/root/pytorchfm/torchfm/model/xdfm.py", line 27, in forward
x = self.linear(x) + self.cin(embed_x) + self.mlp(embed_x.view(-1, self.embed_output_dim))
File "/root/anaconda3/lib/python3.7/site-packages/torch/nn/modules/module.py", line 493, in __call__
result = self.forward(*input, **kwargs)
File "/root/pytorchfm/torchfm/layer.py", line 232, in forward
x = F.relu(self.conv_layers[i](x))
File "/root/anaconda3/lib/python3.7/site-packages/torch/nn/functional.py", line 943, in relu
result = torch.relu(input)
RuntimeError: CUDA error: device-side assert triggered
Can you take a look at it? Thank you!
import torch.nn.functional as F
代码中如果需要使用F.dropout,需要加个training=self.training
F.dropout(attn_scores, p=0.1)改成F.dropout(attn_scores, p=0.1,training=self.training)
这样model.train()可以使用dropout,model.eval()不使用dropout。
Hi,
It threw out this error RuntimeError: CUDA error: device-side assert triggered
when running ffm
model, can you please take a look at it and fix it? Thank you!
I execute python main.py
under the examples/
folder, and get the error:
% python main.py --dataset_path criteo/train.txt
Traceback (most recent call last):
File "main.py", line 189, in <module>
main(args.dataset_name,
File "main.py", line 151, in main
dataset = get_dataset(dataset_name, dataset_path)
File "main.py", line 33, in get_dataset
return CriteoDataset(path)
File "/Users/erlebach/opt/anaconda3/envs/torch/lib/python3.8/site-packages/torchfm/dataset/criteo.py", line 40, in __init__
self.__build_cache(dataset_path, cache_path)
File "/Users/erlebach/opt/anaconda3/envs/torch/lib/python3.8/site-packages/torchfm/dataset/criteo.py", line 56, in __build_cache
feat_mapper, defaults = self.__get_feat_mapper(path)
File "/Users/erlebach/opt/anaconda3/envs/torch/lib/python3.8/site-packages/torchfm/dataset/criteo.py", line 70, in __get_feat_mapper
with open(path) as f:
FileNotFoundError: [Errno 2] No such file or directory: 'criteo/train.txt'
Could you please help me understand how to make this example work? Would it be possible for you to provide a single example that works out of the box, or provide an example of command line arguments that will work? Thanks.
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.