btwardow / factorizationmachines.jl Goto Github PK
View Code? Open in Web Editor NEWFactorization Machines for Julia
License: Other
Factorization Machines for Julia
License: Other
Implement ALS/MCMC similar to libfm and fastFM. This training method does not require a learning rate alpha
. Both libfm and fastFM reuses a single class for both training methods due to common code.
For reference:
I have implemented the ALS algorithm in the past; however, I have not implemented an MCMC implementation before. I would proceed by first implementing the ALS algorithm and then attempt to implement the MCMC algorithm.
SGD can use two simple improvements:
Minibatch sampling
Minibatching can help control the convergence/training time tradeoff (may be wrong about the former claim).
Not entirely certain of an optimized implementation for this; however, here's a sketch of a naive implementation given a matrix of feature vectors X
and an vector labels y
.
number_of_samples = 10000
minibatch_size = 100 # or minibatch_fraction instead?
sample_indices = sample_without_replacement(number_of_samples, minibatch_size)
X_minibatch = X[:, sample_indices]
y_minibatch = y[sample_indices]
yhat = fmPredict(fm, minibatch, fSum, fSumSqr)
mult = fmLoss(yhat, y)
fmSGDUpdate!(fm, alpha, minibatch_indices, X_minibatch, mult, fSum)
Some changes to consider:
sample_without_replacement
; could this be implemented in an efficient manner? Perhaps it would be more efficient to implement a sliding window over the dataset e.g.X_minibatch = X[:, i:i + minibatch_size - 1]
y_minibatch = y[i:i + minibatch_size - 1]
However, the minibatches will not vary per epoch in this scenario.
Scheduling
Set the learning rate alpha
to decrease proportional to the square root of the current iteration sqrt(iteration)
. To prevent division by zero: note that iteration > 0
.
alpha = alpha0 / sqrt(iteration)
So that I could contribute
Some other implementations to compare to:
Tasks:
Experiment approach
X_train, y_train
and test X_test, y_test
X_train, y_train
(and measure the training time)X_test, y_test
Implementing the experiment (up for discussion/alternatives)
benchmarks/
folderFor comparisons with other implementations:
For prediction (during training):
For training loss function:
Both implementations are structurally the same and use enums to distinguish between classification and regression tasks.
task = 0
implies regressiontask = 1
implies classificationA brief roadmap:
Alternative API ideas:
fm_train_sgd_regression(...)
and fm_train_sgd_classification(...)
FMRegression
and FMClassification
that are constructed with their own training parameters and support the interface fm_train_sgd
to produce a single FMModel
. The FMModel
will reference the training task that produces it.Personally, I am in favor of the (2) approach since it leads to fewer branches in the code.
There are several hyperparameters for factorization machines that are crucial to convergence (at least in the tests I've been running) especially in SGD.
alpha
the learning rateinitMean
and initStd
for the Gaussian distribution used in latent vector initializationA simple candidate utility that is easy to implement would be Random Search proposed by Bengio. scikit-learn has a reference implementation that we may use.
Implementation Plan
Still a work in progress
Hyperparameter optimization utilities may live in src/hyperparams.jl
.
There are two important use cases that we may implement as two separate functions
We use the Distributions package to define parameter distributions.
using Distributions
param_distributions = Dict(:alpha => Uniform(0.01, 1.0), :initStd => Gamma(1.0, 1.0))
num_samples = 100
result = fm_randomsearch(X, y, param_distributions = param_distributions, num_samples = num_samples)
fm = result.model
info(result.param_scores)
And a corresponding implementation sketch:
function fm_randomsearch(X, y; param_distributions = Dict(), num_samples = 10)
param_scores = fill((0,0), 10)
best_score = typemax(Float)
best_model = null
for i in 1:num_samples
sample = [rand(dist) for dist in values(param_distributions)]
model = fmTrain(X, y; sample...)
score = evaluate(fm, X, y)
if score > best_score
best_model, best_score = model, score
end
push!(param_scores, (sample, score))
end
RandomSearchResult(best_model, param_scores)
end
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.