Comments (10)
Good suggestion, this seems like it could be useful.
from datafu.
Thanks Matthayes.
I am working on this feature and will submit a patch soon.
Could you please help me do the code review?
I am following the guideline of git: https://help.github.com/articles/fork-a-repo, https://help.github.com/articles/using-pull-requests
Anything I need to follow, please inform me.
from datafu.
Great, and sure I can help with the code review. Will yours support conditional entropy too so mutual information can be computed?
Regarding guidelines to follow, first of all you'll need to fork the code as discussed in that article. For the code itself, a few points to start:
- Make sure the UDF is well documented. Please include example usage in the javadoc.
- Be sure to write unit tests for the UDF. Follow the pattern used for the other UDFs. You can run the tests in eclipse or at the command line. The README has info on running individual tests.
- datafu.pig.stats is probably the right home for this.
Once you're happy with your implementation you can send a pull request and we'll take a look at it.
from datafu.
Sure, I will follow the guideline.
Yes, conditional entropy is something to be supported. Its definition is at http://en.wikipedia.org/wiki/Conditional_entropy. I am just wondering if we need a separate UDF to support it, cos
H(Y|X) = H(X, Y) - H(X), according to the chain rule.
H(X, Y) may could be handled by the above UDF, and the input to the UDF may need to be a bag, each tuple in the bag is a pair <x, y>.
so we shall calculate H(X, Y) = - sum(p(xi, yj) * log(p(xi, yj))), is this correct?
from datafu.
Yes it seems like conditional entropy could be computed using the same UDF, where the UDF would be used to compute H(X,Y) and H(X). If we wanted mutual information we could compute H(X) + H(Y) - H(X,Y).
It's not clear that AccumulatorEvalFunc works for this case though. To estimate the probability you need to know the total number of events right? Usually we use the accumulator interface where the total count is not needed, as this allows the tuples from the bag to be streamed in.
from datafu.
By the way, it looks like all the Entropy code that was added in MAHOUT-747 was later removed in MAHOUT-1296.
from datafu.
Yes, my original thought is that entropy is very similar to datafu.pig.sessions.Sessionize as it reads a stream of data, accumulate and output the final result. But when we count p(x), the total number of events are unknown. To work around this problem, I apply the following formula:
Our target T = SUM(p(x) * log(p(x)))
= SUM(c(x) / N * log(c(x) / N)) // p(x) = c(x) / N, c(x) is the number of events of x, N is the total number of events, which I think could be SUM(c(x))
= 1 / N * SUM( c(x) * [ log(c(x)) - log(N) ] ) // log(a/b) = log(a) - log(b)
= 1 / N * SUM( c(x) * log(c(x)) - c(x) * log(N) )
= 1 / N * SUM( c(x) * log(c(x)) ) - 1 / N * log(N) * SUM(c(x))
= 1 / N * SUM(c(x) * log(c(x)) ) - log(N)
What we need is M = SUM(c(x) * log(c(x)) ) and N = SUM(c(x)) and T = M / N - log(N) and M and N could be accumulated, I think.
Please correct me if my chain of thought is incorrect.
By the way, the entropy code removal is bad news for the original guy, I do not want this to happen
R also has entropy implementation: http://cran.r-project.org/web/packages/entropy/
from datafu.
Yes I think you're right. I tried working out the math myself too and got the same solution as you (with a sign flip):
H(X) = log(N) - 1 / N * SUM(c(x) * log(c(x)) )
So I think AccumulatorEvalFunc does in fact work for computing entropy. If the values are sorted then as you accumulate you can count the number of instances and pass them into a function computing c(x) * log(c(x)), accumulating the total as you go. Then once you are finished you can compute the final result given that you now know N.
from datafu.
Pull request in #78
from datafu.
Close this and track all follow up actions on pull request #78
from datafu.
Related Issues (20)
- Rename TimeCount to SessionCount HOT 1
- Merge BagConcat and BagUnion HOT 1
- Hourglass cannot handle joins where schemas are inconsistent HOT 1
- Hourglass fixed-length windows should be robust to reappearing data HOT 1
- AliasableEvalFunc subcalss, how to test it? HOT 10
- Implement and experiment with different weighted sampling algorithms HOT 9
- Add datafu.pig.text package
- Unable to create UDF with optional 2nd argument HOT 10
- Can't run one unit test HOT 2
- Does it work with org.apache.pig.builtin.MonitoredUDF ? HOT 2
- Upgrade Pig library used to 0.12 HOT 1
- Add ToJson UDF to serialize any relation/field as a JSON String HOT 1
- datafu.pig.sampling.ReservoirSample HOT 5
- Hash UDFs should return zero-padded strings of uniform length even when leading bits are zero. HOT 2
- DistinctBy output is incorrect when any value in the tuple contains a '-' HOT 1
- Getting java.lang.NullPointerException in running PageRank HOT 1
- possible error in getting started with hourglass documentation HOT 1
- Release : When can we expect the 1.3 release of DataFu ? HOT 1
- Option to have Sessionize UDF spit out Long value instead of GUID HOT 1
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from datafu.