Comments (12)
Hi Alessandro,
I would say that the split module is more convenient for fine tuning the splitting process than the BASH script. However, it is true that there is no available method to divide a dataset in three, only in two parts; hence, the best way to obtain a training+validation+test partition is to follow the process you described (splitting one of the splits obtained from the whole dataset).
Hope this helps.
Best,
Alejandro
from rival.
Hi Alejandro,
first of all thank you for your clarification. I really appreciate it!
Anyway, in my opinion there should be a common method to split the dataset included in the framework. This a way to fix how people split their datasets and allow them to obtain more reproducible results.
Maybe I can implement this kind of method inheriting from the Splitter class. Could this kind of feature be useful for the RiVal framework?
Alessandro
from rival.
Hi Alessandro,
what kind of feature are you suggesting? As you are already aware, RiVal already supports CrossValidation, Random, and Temporal splits. If you are proposing to create a generic splitter that returns 3 splits instead of 2, by running whatever Splitter the user has selected, then this would be a very useful feature, indeed. If you are thinking in something else, please, elaborate a little bit and we can discuss about it.
Cheers,
Alejandro
from rival.
Hi Alejandro,
in the first place, I am wondering if I can implement a custom RandomSplitter class which returns three splits instead of two but, as you correctly say, this kind of feature can be implemented in each class that implements Splitter.
In your opinion what is the proper way to let the user generate the validation set too? I think that in the Splitter constructor can be added a boolean flag called generateValidation and in each subclass can be used differently according to the specific behaviour of the class. Can it be consistent with your current API?
Cheers,
Alessandro
from rival.
Hi,
I was thinking more on a ValidationSplitter that implements Splitter and that takes another Splitter as a parameter. Then, the split method would be as simple as
DataModelIF<U, I>[] trainingTestSplit = splitter.split(data);
DataModelIF<U, I>[] trainingValSplit = splitter.split(trainingTestSplit[0]);
return new DataModelIF<U,I>[]{trainingValSplit[0], trainingValSplit[1], trainingTestSplit[1]};
Would this work as you expect?
Best,
Alejandro
from rival.
Hi,
I think that it would be perfect for the RandomSplitter whereas I think that it won't work for the CrossValidationSplitter as it is because it returns an array containing 2*numFolds elements.
Am I missing something or do you expect to introduce a preliminary check to avoid this incorrect behaviour?
Best,
Alessandro
from rival.
No, you are right. I knew there was something missing (it couldn't be so easy). I guess a loop like the one in SplitterRunner should do the work:
DataModelIF<U, I>[] trainingTestSplits = splitter.split(data);
DataModelIF<U, I>[] newSplits = new DataModelIF<U,I>[trainingTestSplits.length / 2 * 3];
for (int i = 0; i < trainingTestSplits.length / 2; i++) {
DataModelIF<U, I> trainingVal = trainingTestSplits[2 * i];
DataModelIF<U, I>[] trainingValSplit = splitter.split(trainingVal);
DataModelIF<U, I> test = trainingTestSplits[2 * i + 1];
newSplits[3*i] = trainingValSplit[0];
newSplits[3*i+1] = trainingValSplit[1];
newSplits[3*i+2] = test;
}
return newSplits;
The problem with this is that, since it expects the splitter to give pairs of DataModels, if you pass a ValidationSplitter to the ValidationSplitter everything will be messed up, but I think this is something we are not interesting in supporting right now, are we?
Best,
Alejandro
from rival.
I am thinking exactly at the same implementation. Before adding the full implementation of the class to my fork I want to share with you my current implementation:
@SuppressWarnings("unchecked")
@Override
public DataModelIF<U, I>[] split(DataModelIF<U, I> data) {
if (splitter instanceof ValidationSplitter) {
throw new IllegalArgumentException("Unable to apply a validation splitter recursively!");
}
// if the basic splitter does cross validation generate a dev for each fold
if (splitter instanceof CrossValidationSplitter) {
DataModelIF<U, I>[] trainingTestSplits = this.splitter.split(data);
DataModelIF<U, I>[] newSplits = new DataModelIF[trainingTestSplits.length/2 * 3];
for (int i = 0; i < trainingTestSplits.length / 2; i++) {
DataModelIF<U, I> trainingVal = trainingTestSplits[2 * i];
DataModelIF<U, I>[] trainingValSplit = splitter.split(trainingVal);
DataModelIF<U, I> test = trainingTestSplits[2 * i + 1];
newSplits[3 * i] = trainingValSplit[0];
newSplits[3 * i + 1] = trainingValSplit[1];
newSplits[3 * i + 2] = test;
}
return newSplits;
}
DataModelIF<U, I>[] trainingTestSplit = splitter.split(data);
DataModelIF<U, I>[] trainingValSplit = splitter.split(trainingTestSplit[0]);
return new DataModelIF[]{trainingValSplit[0], trainingValSplit[1], trainingTestSplit[1]};
}
In your opinion is it a correct way in order to avoid the problem with the possible recursive call to the ValidationSplitter and to check if the user has specified a CrossValidationSplitter instance? Have you got any other suggestions to improve my code?
Thank you Alejandro for your availability.
Cheers,
Alessandro
from rival.
Hi Alessandro,
yes, I think your implementation is correct, although I would simplify it in the following way:
- include the ValidationSplitter check in the constructor, as soon as the splitter is provided
- do not include the other check, since the loop should always work (even for the RandomSplitter), and remove the last three lines.
It will be great to have this feature added in RiVal!
Best,
Alejandro
from rival.
Completely agree with you! Thank you for your suggestions, I really appreciate your help.
For what concerns the implementation of the split method for a TemporalDataModel, I think that is the same as the one that I have written now. The only difference is in the data model instance to be specified so everything will be fine. Is it correct?
I am happy to contribute to RiVal trying to make it better :)
Cheers,
Alessandro
from rival.
Thank you for your interest!
Indeed, the other split method would be equivalent, the only change is the datamodel instance used.
Cheers,
Alejandro
from rival.
I am writing some test cases for my implementation. After that I will send you a pull request.
Best,
Alessandro
from rival.
Related Issues (20)
- Check evaluation metrics are suitable for unary/binary data
- Check splitters are suitable for unary/binary data
- Check parsers are suitable for unary/binary data
- Not generalized DataModel for RandomSplitter class
- Not generalized DataModel for Temporal Splitter class
- Not generalized DataModel for SplitterRunner class
- Using Examples with a dataset like ML-10M HOT 11
- Update license snippet in gh-pages HOT 1
- Issue with Precision and Recall HOT 3
- Create jar file of 0.3-SNAPSHOT to be used outside eclipse HOT 2
- DataModel does not support duplicate ratings in dataset
- Implement RecSys Challenge 2016 metric
- Missing documentation for unsupported combination in RandomSplitter HOT 3
- Add custom behaviour to DataModelUtils.saveModel HOT 3
- [Question] Maven install command HOT 5
- Bug in CrossValidatedMahoutKNNRecommenderEvaluator
- Type mismatch: cannot convert from CSVParser to Iterable<CSVRecord> HOT 11
- Test coverage HOT 3
- Precision values are not consistent with those from other ranking metrics
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from rival.