multiple compilers

[fabioasdias]

multiple compilers

[delesley]

The original purpose of metrics was to attach losses to predictions that are buried deep within non-root blocks, so yes, the gradients will flow back correctly. -DeLesley

…

On Tue, Oct 17, 2017 at 7:06 AM, Satwik Kottur ***@***.***> wrote: @rudinger <https://github.com/rudinger> Thanks for the stitch! Do you know if the retrieved outputs can be used for further ops in the tensorflow graph? Will the gradients flow back correctly if this is done? — You are receiving this because you modified the open/close state. Reply to this email directly, view it on GitHub <#16 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AGGbTYF3_ONPf8oGPeiXkEYYmJLGlJq-ks5stLRUgaJpZM4MK4Yb> .

-- DeLesley Hutchins | Software Engineer | [email protected] | 505-206-0315

[mherreshoff]

Hi Fábio, The `eval` function is intended for interactively testing your compiler object. It pays a lot of setup costs every time as a result. Have a look at "Batching Inputs <https://github.com/tensorflow/fold/blob/master/tensorflow_fold/g3doc/running.md#batching-inputs>" to see how to run using the compiler.build_feed_dict mechanism, which should be much faster. Best wishes, -Marcello

…

On Thu, Feb 23, 2017 at 10:45 PM, Fábio Dias ***@***.***> wrote: This is my model: mid = (td.Map(td.Tensor((24,10)) >> td.ScopedLayer(convLayer) >> td.Function(lambda x: tf.reshape(x,[-1,64*(10-2)]))) >> td.Fold(td.Concat() >> td.Function(td.FC(128)), td.FromTensor(tf.zeros(128)))) model = ( mid >> td.Function(td.FC(1))) (The ScopedLayer(convLayer) part is related to solved issue #14 <#14> ) I trained it using model: compiler = td.Compiler.create((model, td.Scalar())), and it works. However, now I want the result of mid for a set of values. I can do encod.eval(X[i]) for every piece of data, but that is painfully slow. However, if I try to create a new compiler: newComp=td.Compiler.create((mid,)) Y=sess.run(mid,newComp.build_feed_dict(X)) I get: TypeError: block <td.Pipe> is already a child of <td.Pipe> How can I solve this? eval on each entry will take a bit over 4 hours ... — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#16>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AABepNCPtZfud7Tcae4vaSsf_QRESrdcks5rfnyKgaJpZM4MK4Yb> .

[fabioasdias]

Marcello, I'm aware, however, as I said, when I construct a new compiler it doesn't work, with the error:

TypeError: block <td.Pipe> is already a child of <td.Pipe>

[mherreshoff]

Ah. I misread this the first time. We didn't anticipate this use-case, so blocks are mutable objects that have state and belong to at most one compiler. If this is sufficiently counterintuitive we'll consider fixing it. For now, you can create a copy of the blocks (though you'll also have to explicitly set a scope in the scoped layers so they'll use the same variables.) You should be able to get away with something like this: convScopedLayer = td.ScopedLayer(convLayer) def make_mid(): return (td.Map(td.Tensor((24,10)) >> convScopedLayer >> td.Function(lambda x: tf.reshape(x,[-1,64*(10-2)]))) >> td.Fold(td.Concat() >> td.Function(td.FC(128)), td.FromTensor(tf.zeros(128)))) model = ( make_mid >> td.Function(td.FC(1))) compiler1 = td.Compiler(model) compiler2 = td.Compiler(make_mid())

…

On Fri, Feb 24, 2017 at 9:17 AM, Fábio Dias ***@***.***> wrote: Marcello, I'm aware, however, as I said, when I construct a new compiler it doesn't work, with the error: TypeError: block <td.Pipe> is already a child of <td.Pipe> — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#16 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AABepPej3sUoroQEIbWR1yZjHSzqh7Unks5rfxCNgaJpZM4MK4Yb> .

[mherreshoff]

Lost some parentheses, that line should be "model = ( make_mid() >> td.Function(td.FC(1)))" On Fri, Feb 24, 2017 at 9:31 AM, Marcello Herreshoff < [email protected]> wrote:

…

Ah. I misread this the first time. We didn't anticipate this use-case, so blocks are mutable objects that have state and belong to at most one compiler. If this is sufficiently counterintuitive we'll consider fixing it. For now, you can create a copy of the blocks (though you'll also have to explicitly set a scope in the scoped layers so they'll use the same variables.) You should be able to get away with something like this: convScopedLayer = td.ScopedLayer(convLayer) def make_mid(): return (td.Map(td.Tensor((24,10)) >> convScopedLayer >> td.Function(lambda x: tf.reshape(x,[-1,64*(10-2)]))) >> td.Fold(td.Concat() >> td.Function(td.FC(128)), td.FromTensor(tf.zeros(128)))) model = ( make_mid >> td.Function(td.FC(1))) compiler1 = td.Compiler(model) compiler2 = td.Compiler(make_mid()) On Fri, Feb 24, 2017 at 9:17 AM, Fábio Dias ***@***.***> wrote: > Marcello, I'm aware, however, as I said, when I construct a new compiler > it doesn't work, with the error: > > TypeError: block <td.Pipe> is already a child of <td.Pipe> > > — > You are receiving this because you commented. > Reply to this email directly, view it on GitHub > <#16 (comment)>, > or mute the thread > <https://github.com/notifications/unsubscribe-auth/AABepPej3sUoroQEIbWR1yZjHSzqh7Unks5rfxCNgaJpZM4MK4Yb> > . >

[fabioasdias]

So, I tried that approach. Here's the relevant (full) code:

CL=td.ScopedLayer(convLayer)
    
def mk_encod():
    return(td.Map(td.Tensor((24,10)) >>  CL >>
                  td.Function(lambda x: tf.reshape(x,[-1,64*(10-2)]))) >>
           td.Fold(td.Concat() >>
                   td.Function(td.FC(256)), td.FromTensor(tf.zeros(256))))

model = ( mk_encod()  >>  td.Function(td.FC(1)))
    
compiler = td.Compiler.create((model, td.Scalar()))
y, y_ = compiler.output_tensors
loss = tf.nn.l2_loss(y - y_)
train = tf.train.AdamOptimizer().minimize(loss)
sess.run(tf.global_variables_initializer())
for i in range(1000):
    sess.run(train, compiler.build_feed_dict([get_example() for _ in range(batch_size)]))

encod=mk_encod()
newComp=td.Compiler.create((encod,td.Scalar())) #copied the structure of the prev. compiler
fvBlock,_=newComp.output_tensors
FV=sess.run(fvBlock,newComp.build_feed_dict([get_real(i) for i in range(N)]))

However, that gets me:

FailedPreconditionError (see above for traceback): Attempting to use uninitialized value FC_256_2/weights
	 [[Node: FC_256_2/weights/read = Identity[T=DT_FLOAT, _device="/job:localhost/replica:0/task:0/cpu:0"](FC_256_2/weights)]]

So I guess I'm messing up with the scopes, I think...
Isn't this approach (train a whole net, use the results of a middle layer) somewhat common on NLP ?(isn't word2vec like this?). What would be the canonical way for that?

[delesley]

Expose the output of the middle layer as an output of the model. td.AllOf is good for that.

…

On Fri, Feb 24, 2017 at 10:13 PM, Fábio Dias ***@***.***> wrote: So, I tried that approach. Here's the relevant (full) code: CL=td.ScopedLayer(convLayer) def mk_encod(): return(td.Map(td.Tensor((24,10)) >> CL >> td.Function(lambda x: tf.reshape(x,[-1,64*(10-2)]))) >> td.Fold(td.Concat() >> td.Function(td.FC(256)), td.FromTensor(tf.zeros(256)))) model = ( mk_encod() >> td.Function(td.FC(1))) compiler = td.Compiler.create((model, td.Scalar())) y, y_ = compiler.output_tensors loss = tf.nn.l2_loss(y - y_) train = tf.train.AdamOptimizer().minimize(loss) sess.run(tf.global_variables_initializer())for i in range(1000): sess.run(train, compiler.build_feed_dict([get_example() for _ in range(batch_size)])) encod=mk_encod() newComp=td.Compiler.create((encod,td.Scalar())) #copied the structure of the prev. compiler fvBlock,_=newComp.output_tensorsFV=sess.run(fvBlock,newComp.build_feed_dict([get_real(i) for i in range(N)])) However, that gets me: FailedPreconditionError (see above for traceback): Attempting to use uninitialized value FC_256_2/weights [[Node: FC_256_2/weights/read = Identity[T=DT_FLOAT, _device="/job:localhost/replica:0/task:0/cpu:0"](FC_256_2/weights)]] So I guess I'm messing up with the scopes, I think... Isn't this approach (train a whole net, use the results of a middle layer) somewhat common on NLP ?(isn't word2vec like this?). What would be the canonical way for that? — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#16 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AGGbTXsnl7Y_vsNx5WyBnPaZxqw6-3N1ks5rf8Z2gaJpZM4MK4Yb> .

-- DeLesley Hutchins | Software Engineer | [email protected] | 505-206-0315

[rudinger]

Is my understanding correct that there is (currently) no way to obtain/inspect the output_tensors of a non-root block without compiling a separate instance of the tf graph (for each such block), and then doing separate corresponding executions of sess.run(...), etc.? Since sess.run([tensor1, tensor2,...]) already allows arbitrary/non-root tensors in the tf graph to be specified, it seems like this may be an issue with the td compiler only being able to expose the root-level tensors in the resulting graph. Anyway, having a simple way to inspect non-root tensors would be very helpful for debugging purposes. Thanks for the great resource!

[delesley]

If you drop down to the loom layer, you can expose any part of the computation as an output. Within the blocks library, we provide the notion of "metrics". A metric is a computation (usually loss, perplexity or some other measure) that can be attached to the output of any block. Each metric has a name that you can use to retrieve it later. Keep in mind that non-root blocks may be evaluated multiple times (e.g. the child block of td.Sequence) or not at all (Optional/OneOf) so when you retrieve a metric by name, you get a concatenation of all metrics with that name.

…

On Wed, Mar 1, 2017 at 8:10 PM, Rachel Rudinger ***@***.***> wrote: Is my understanding correct that there is (currently) no way to obtain/inspect the output_tensors of a non-root block without compiling a separate instance of the tf graph (for each such block), and then doing separate corresponding executions of sess.run(...), etc.? Since sess.run([tensor1, tensor2,...]) already allows arbitrary/non-root tensors in the tf graph to be specified, it seems like this may be an issue with the td compiler only being able to expose the root-level tensors in the resulting graph. Anyway, having a simple way to inspect non-root tensors would be very helpful for debugging purposes. Thanks for the great resource! — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#16 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AGGbTRzObJ6zQu8vvivHh4Vbx3a3oMD7ks5rhkEmgaJpZM4MK4Yb> .

-- DeLesley Hutchins | Software Engineer | [email protected] | 505-206-0315

[rudinger]

Ok, I think I see how one could post-hoc stitch a Metric into the pipeline to inspect values in a non-root block. A simple example of a graph that sums two numbers and doubles them, where we want to inspect the values at the intermediate summing stage:

import tensorflow as tf
sess = tf.InteractiveSession()
import tensorflow_fold as td

inputs = td.Record((td.Scalar(), td.Scalar()))
sum_block = inputs >> td.Function(tf.add)
m = td.Composition()
with m.scope():
    td.Metric('sum').reads(m.input)
    m.output.reads(m.input)
double_block = sum_block >> m >> td.Function(lambda x: 2.0*x)

compiler = td.Compiler.create(double_block)
fd = compiler.build_feed_dict([(1,2),(3,4),(5,6)])
outputs, metrics = sess.run([compiler.output_tensors, compiler.metric_tensors], feed_dict=fd)

I imagine there's a cleaner way to write this, but this at least accomplishes what I want (avoiding multiple compilers). Thanks for the tip!

[satwikkottur]

@rudinger Thanks for the stitch! Do you know if the retrieved outputs can be used for further ops in the tensorflow graph? Will the gradients flow back correctly if this is done?

[rudinger]

@satwikkottur Sorry I don't have an answer, I've moved to a different project. Good luck!

[satwikkottur]

I tried this by dropping down to loom API. But looks like I can specify the outputs (which are Loom id ints), and they stack up as output. Is there any other way I can get more structured outputs from loom, rather than rearranging them post loom?
For eg., I want [1, 2, 5] outputs together and [3, 4, 6] together. (note that integers here are loom ids)

If you drop down to the loom layer, you can expose any part of the
computation as an output.