Comments (9)
I think I have a similar problem and would like to continue the discussion here and ask the question.
My reducer will create two different type of data frames, every reducer is going to create those two type of data frames.
The keys that I want to assign are only 1 and 2.
Assuming my first data frame is df1 and the second one is df2.
I had thought that using
keyval(1L:2L,list(df1,df2))
would have assigned the key 1 to the df1 data frame and key 2 to the df2, but this is not happening.
The output that I am getting is a list with both data frames inside it.
Key 1 and key 2 both get assigned the same list.
Could you please help me figure out this problem.
from rmr2.
Just to make it concrete:
from.dfs(mapreduce(to.dfs(1), map = function(k,v) keyval(1:2, list(mtcars[1:3,1:3], mtcars[4:6, 4:6]))))
$key
[1] 1 2
$val
$val[[1]]
mpg cyl disp
Mazda RX4 21.0 6 160
Mazda RX4 Wag 21.0 6 160
Datsun 710 22.8 4 108
$val[[2]]
hp drat wt
Hornet 4 Drive 110 3.08 3.215
Hornet Sportabout 175 3.15 3.440
Valiant 105 2.76 3.460
What part of this is wrong? The keys are assigned positionally, first position in one list goes with the first position in the other list.
from rmr2.
I am not exactly sure, but please clear it up, if I am wrong.
The above method that you have shown, key 1 would have a list would 2 objects inside one at the first position and the other at the second position. Key 2 would have a similar output.
What I want is key 1 having the val[[1]] object and key 2 having the val[[2]] object. Is this possible?
from rmr2.
Again I think we need clarify with some code. Are you saying that you prefer list(list(k1,v1), list(k2,v2))
over list(list(k1,k2), list(v1,v2))
? If that is the issue I would try two answers
- The latter is what rmr2 does, so, no, it's not possible by way of an option. The reason is that keys are more likely to be of the same time, so that you can use, say, an atomic vector, for convenience and speed and the same for values, rather than key and value be of the same type. Moreover things work often in R in a column oriented fashion, the data frame being the most notable example. We just followed that track (it wasn't so before release 2, so I can tell you from experience that it works a lot better this way),
- But you can transform one representation into the other
mapply.list = function(...) mapply(list,..., SIMPLIFY=F)
do.call(mapply.list, list(1:3, 1:3))
I hope this helps.
A
from rmr2.
I am not exactly sure about which key gets assigned what value, in rmr2. Its not exactly clear to me.
Earlier we had an option of doing this,
if(condition)
keyval(1,mtcars[1:3,1:3])
else
keyval(2,mtcars[3:6, c(1,4:6)])
I want to perform something similar but not able to achieve that, any suggestions on how to do the above in rmr2?
from rmr2.
I think it is
if(condition)
keyval(1,list(mtcars[1:3,1:3]))
else
keyval(2,list(mtcars[3:6, c(1,4:6)]))
The association rule is positional: the first element of the key argument is key to the first element of the second argument. If either argument is two-dimensional, replace 'first element' with first row. Recycling works and the model is cbind.
from rmr2.
So what happens when you have two two-dimensional object and you want to assign two keys one for each two-dimensional objects.
Sorry, if I you have already answered it above, its not clear to me at all.
from rmr2.
keyval(list(k1,k2), list(obj1, obj2))
In general, if you have list(keyval(k1,v1), keyval(k2,v2, ...)
in rmr1.x it is equivalent to keyval(list(k1, k2, ...), list(v1, v2, ...))
in rmr2.x. We went from a row-oriented representation, closer to hadoop-streaming, to a column oriented, closer to how things work in R.
from rmr2.
Yes, that is what I was looking for .
Thanks!! :)
It solved my problem .
from rmr2.
Related Issues (20)
- rmr.normalize.path wrong on "/"
- The system cannot find the path specified. HOT 9
- Hadoop job log4j:WARN Please initialize the log4j system properly HOT 5
- Mapreduce warning: namespace 'rmr2' is not available and has been replaced HOT 1
- Question with regards to lzo-ubuntu.sh
- HADOOP_CMD getting lost... HOT 2
- from.dfs() has Error in RStudio, but runs well in R console
- how can i find "kv.cmp = rmr2:::kv.cmp" in "pkg / tests / mapreduce.R"
- Changing number of mapper/ reducer HOT 2
- parsing XML douments in rhadoop
- what's the meaning of "in.memory.combine"
- rmr2 mapreduce does not produce any output HOT 6
- RMR2 Installaction question
- Recommended version of rmr2 for hadoop 2.7 and R 3.3.1
- Have some tutorials or references here about rmr2? HOT 1
- Reduce task fails with org.apache.hadoop.streaming.PipeMapRed: PipeMapRed failed! HOT 2
- rhadoop giving null key value output HOT 2
- path and to.dfs with R launched from two different directories
- from.dfs returns nothing
- output NULL of key and val
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from rmr2.