Multiple dataframe outputs from the same reducer

[piccolbo]

Multiple dataframe outputs from the same reducer

[mayankbansal89]

I think I have a similar problem and would like to continue the discussion here and ask the question.
My reducer will create two different type of data frames, every reducer is going to create those two type of data frames.
The keys that I want to assign are only 1 and 2.
Assuming my first data frame is df1 and the second one is df2.

I had thought that using
keyval(1L:2L,list(df1,df2))

would have assigned the key 1 to the df1 data frame and key 2 to the df2, but this is not happening.
The output that I am getting is a list with both data frames inside it.
Key 1 and key 2 both get assigned the same list.

Could you please help me figure out this problem.

[piccolbo]

Just to make it concrete:

from.dfs(mapreduce(to.dfs(1), map = function(k,v) keyval(1:2, list(mtcars[1:3,1:3], mtcars[4:6, 4:6]))))


$key
[1] 1 2

$val
$val[[1]]
               mpg cyl disp
Mazda RX4     21.0   6  160
Mazda RX4 Wag 21.0   6  160
Datsun 710    22.8   4  108

$val[[2]]
                   hp drat    wt
Hornet 4 Drive    110 3.08 3.215
Hornet Sportabout 175 3.15 3.440
Valiant           105 2.76 3.460

What part of this is wrong? The keys are assigned positionally, first position in one list goes with the first position in the other list.

[mayankbansal89]

I am not exactly sure, but please clear it up, if I am wrong.
The above method that you have shown, key 1 would have a list would 2 objects inside one at the first position and the other at the second position. Key 2 would have a similar output.

What I want is key 1 having the val[[1]] object and key 2 having the val[[2]] object. Is this possible?

[piccolbo]

Again I think we need clarify with some code. Are you saying that you prefer list(list(k1,v1), list(k2,v2)) over list(list(k1,k2), list(v1,v2))? If that is the issue I would try two answers

1. The latter is what rmr2 does, so, no, it's not possible by way of an option. The reason is that keys are more likely to be of the same time, so that you can use, say, an atomic vector, for convenience and speed and the same for values, rather than key and value be of the same type. Moreover things work often in R in a column oriented fashion, the data frame being the most notable example. We just followed that track (it wasn't so before release 2, so I can tell you from experience that it works a lot better this way),
2. But you can transform one representation into the other

mapply.list = function(...) mapply(list,..., SIMPLIFY=F)
do.call(mapply.list, list(1:3, 1:3))

I hope this helps.

A

[mayankbansal89]

I am not exactly sure about which key gets assigned what value, in rmr2. Its not exactly clear to me.
Earlier we had an option of doing this,

if(condition)
keyval(1,mtcars[1:3,1:3])
else
keyval(2,mtcars[3:6, c(1,4:6)])

I want to perform something similar but not able to achieve that, any suggestions on how to do the above in rmr2?

[piccolbo]

I think it is

if(condition)
keyval(1,list(mtcars[1:3,1:3]))
else
keyval(2,list(mtcars[3:6, c(1,4:6)]))

The association rule is positional: the first element of the key argument is key to the first element of the second argument. If either argument is two-dimensional, replace 'first element' with first row. Recycling works and the model is cbind.

[mayankbansal89]

So what happens when you have two two-dimensional object and you want to assign two keys one for each two-dimensional objects.

Sorry, if I you have already answered it above, its not clear to me at all.

[piccolbo]

keyval(list(k1,k2), list(obj1, obj2))

In general, if you have list(keyval(k1,v1), keyval(k2,v2, ...) in rmr1.x it is equivalent to keyval(list(k1, k2, ...), list(v1, v2, ...)) in rmr2.x. We went from a row-oriented representation, closer to hadoop-streaming, to a column oriented, closer to how things work in R.

[mayankbansal89]

Yes, that is what I was looking for .
Thanks!! :)
It solved my problem .