Comments (10)
Hello @muhlbach ! glmnet
automatically determines a grid of lambda values at which to fit a model. My understanding is that it determines the maximum value from data, and the minimum value using the maximum value and the min_lambda_ratio
parameter. The grid of lambda parameters is evenly spaced on a log scale between these two ends. There's an early stopping criterion which will let the code stop before reaching all points in the grid, which is why the lambda path doesn't always have length n_lambda
.
When you say that coef_path_
with n_lambda=5
looks very different from coef_path_
with n_lambda=100
, do you mean that the values of the coefficients at each step aren't the same, and that you don't have 5 steps when n_lambda=5
? If that's the case, it sounds like what's happening is that the grid spacing is different when you select n_lambda=100 and n_lambda=5, giving you different paths for the coefficients. You can see the actual lambda values used in the lambda_path_
attribute.
There's another source for run-to-run variation, which is that the cross-validation folds used for testing a model are randomly selected. You didn't mention anything about seeds, but if you aren't already, you can fix the random number generator seed with the random_state
parameter. That shouldn't really affect this particular test, since the lambda grid is constructed deterministically, but in some cases it might result in small changes in the early stopping point.
What would you like to accomplish by setting n_lambda to 5? Perhaps there's another way to achieve the same goal.
from python-glmnet.
Hi @stephan-hoover! Thank you for the answer!
In both cases (n_lambda=5 and n_lambda=100) the coef_path_ has the shape (n_samples, 5), which I find strange. But even if that is correct, a different number of variables is selected. For n_lambda=100, it selects only two variables for each of the 5 columns in coef_path_. But if n_lambda=5, it selects a varying number, starting from zero and then increasing (which is expected).
I need it for soft thresholding, for instance, I need the “5 most significant variables” or the “10 most significant variables”. Therefore, I always specify n_lambda to be very high, and then select the column in coef_path_ that gives me the desired number of variables. I hope it makes sense.
from python-glmnet.
Yes, that makes sense. As you say, the number of coefficients should gradually vary between 0 and the maximum. I am surprised that n_lambda=5
gives you the expected range from 0 to n_coefficients non-zero coefficients, but n_lambda=100
selects two non-zero coefficients at each step. I would have expected the same behavior of varying from 0 to n_coefficients, especially if you saw that when Are you able to provide code and data that would let me reproduce?
I have a couple of clarifying questions. First, is this a regression problem using glmnet.ElasticNet
? Second, you mention that your coef_path_
has shape (n_samples, 5) -- do you mean (n_coefficients, 5)?
from python-glmnet.
from python-glmnet.
Are you able to post it in this thread, or link to a gist?
from python-glmnet.
Sure. Code and data are attached.
See link (file is too large to include here): https://www.dropbox.com/s/45da8ytw0q73a9h/example.zip?dl=0
from python-glmnet.
Thanks. I will investigate and get back to you.
from python-glmnet.
@muhlbach , I was able to take a look at your code and data, and I think I see what's going on. The test code I got from you is
# example
# Load
X = np.load('X.npy')
y = np.load('y.npy')
# Run with n_lambda=100
test1 = ElasticNet(alpha=1,
cut_point=1,
fit_intercept=False,
n_lambda=100,
standardize=False,
tol=1e-05,
max_iter=10000,
verbose=0)
test1.fit(X,y)
test1.coef_path_
# Run with n_lambda=5
test2 = ElasticNet(alpha=1,
cut_point=1,
fit_intercept=False,
n_lambda=5,
standardize=False,
tol=1e-05,
max_iter=10000,
verbose=0)
test2.fit(X,y)
test2.coef_path_
I see that these models are being fit without an intercept term, but the mean of the targets is approximately 2.5. glmnet
does early stopping by looking at the percent of deviance explained at each step along the lambda path. It also sets the start of the path by finding the value of lambda which forces all coefficients to be zero, and without an intercept, this needs to be quite high. In this situation, without an intercept, the "percent of deviance explained" comes out greater than 1.
I don't know exactly how glmnet
calculates when to stop, but apparently having a percent of deviance explained greater than 1 makes it think that it's not worth continuing along the lambda path, and it stops after 5 steps (which looks like the minimum).
This question is addressing the same problem that you're running into: https://stats.stackexchange.com/questions/243347/why-is-cv-glmnet-returning-absurd-coefficients-when-intercept-term-is-omitted
I suggest that you either include an intercept term, or that you manually input a lambda path (using the lambda_path
parameter of ElasticNet
). Does this help?
from python-glmnet.
from python-glmnet.
You're welcome!
from python-glmnet.
Related Issues (20)
- Failed to find libgfortran.3.dylib HOT 4
- Extraneous __init__.py in root directory
- ValueError: _glmnet.lognet() 3rd keyword (isd) can't be converted to int
- Create new release for PIP past 2.0 HOT 2
- ElasticNet.predict returns 0d array on 1-row inputs
- Where to put 'lambda.1se' HOT 3
- Use external joblib and six packages, instead of sklearn.externals versions HOT 2
- lol
- Mac wheel library linking issue HOT 7
- BUG: ModuleNotFoundError: No module named '_glmnet' HOT 5
- Error with sklearn's make_scorer
- Error importing glmnetcv
- installing issue on Windows 10
- ElasticNet.fit raises ValueError when not converging instead of just issuing a warning HOT 1
- What can be the reason for different results compared to R glmnet?
- Python 3.9 Compatibility HOT 9
- LogitNet doesn't implement the `_estimator_type` attribute
- Package does not build with `--use-pep517` (with Poetry)
- Inaccurate porting of covariance vs naive method
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from python-glmnet.