Spelling Correction- Shape must be rank 2 but is rank 3 for 'cls/predictions/MatMul' (op: 'MatMul') with input shapes: [?,?,768], [768,30522].

[mesolitica]

Spelling Correction- Shape must be rank 2 but is rank 3 for 'cls/predictions/MatMul' (op: 'MatMul') with input shapes: [?,?,768], [768,30522].

[XiaoxueGu]

@huseinzol05 Can you please help me solve the below error??

Versions:
Python: 3.6.10
Tensorflow: 1.13.1
Bert: 2.2.0

Code Source: https://github.com/huseinzol05/NLP-Models-Tensorflow/blob/master/spelling-correction/3.bert-base-fast.ipynb

I am running the exact same code that is in the above code source link, but getting the below attached error while running the below chunk of code :

Code:

tf.reset_default_graph() sess = tf.InteractiveSession() model = Model() sess.run(tf.global_variables_initializer()) var_lists = tf.get_collection(tf.GraphKeys.TRAINABLE_VARIABLES, scope = 'bert')
Error:
InvalidArgumentError Traceback (most recent call last)
~/anaconda3/envs/projectenv/lib/python3.6/site-packages/tensorflow/python/framework/ops.py in _create_c_op(graph, node_def, inputs, control_inputs)
1658 try:
-> 1659 c_op = c_api.TF_FinishOperation(op_desc)
1660 except errors.InvalidArgumentError as e:

InvalidArgumentError: Shape must be rank 2 but is rank 3 for 'cls/predictions/MatMul' (op: 'MatMul') with input shapes: [?,?,768], [768,30522].

During handling of the above exception, another exception occurred:

ValueError Traceback (most recent call last)
in
1 tf.reset_default_graph()
2 sess = tf.InteractiveSession()
----> 3 model = Model()
4
5 sess.run(tf.global_variables_initializer())

in init(self)
32 initializer = tf.zeros_initializer(),
33 )
---> 34 logits = tf.matmul(input_tensor, tf.transpose(embedding))
35 self.logits = tf.nn.bias_add(logits, output_bias)

~/anaconda3/envs/projectenv/lib/python3.6/site-packages/tensorflow/python/ops/math_ops.py in matmul(a, b, transpose_a, transpose_b, adjoint_a, adjoint_b, a_is_sparse, b_is_sparse, name)
2453 else:
2454 return gen_math_ops.mat_mul(
-> 2455 a, b, transpose_a=transpose_a, transpose_b=transpose_b, name=name)
2456
2457

~/anaconda3/envs/projectenv/lib/python3.6/site-packages/tensorflow/python/ops/gen_math_ops.py in mat_mul(a, b, transpose_a, transpose_b, name)
5331 _, _, _op = _op_def_lib._apply_op_helper(
5332 "MatMul", a=a, b=b, transpose_a=transpose_a, transpose_b=transpose_b,
-> 5333 name=name)
5334 _result = _op.outputs[:]
5335 _inputs_flat = _op.inputs

~/anaconda3/envs/projectenv/lib/python3.6/site-packages/tensorflow/python/framework/op_def_library.py in _apply_op_helper(self, op_type_name, name, **keywords)
786 op = g.create_op(op_type_name, inputs, output_types, name=scope,
787 input_types=input_types, attrs=attr_protos,
--> 788 op_def=op_def)
789 return output_structure, op_def.is_stateful, op
790

~/anaconda3/envs/projectenv/lib/python3.6/site-packages/tensorflow/python/util/deprecation.py in new_func(*args, **kwargs)
505 'in a future version' if date is None else ('after %s' % date),
506 instructions)
--> 507 return func(*args, **kwargs)
508
509 doc = _add_deprecated_arg_notice_to_docstring(

~/anaconda3/envs/projectenv/lib/python3.6/site-packages/tensorflow/python/framework/ops.py in create_op(failed resolving arguments)
3298 input_types=input_types,
3299 original_op=self._default_original_op,
-> 3300 op_def=op_def)
3301 self._create_op_helper(ret, compute_device=compute_device)
3302 return ret

~/anaconda3/envs/projectenv/lib/python3.6/site-packages/tensorflow/python/framework/ops.py in init(self, node_def, g, inputs, output_types, control_inputs, input_types, original_op, op_def)
1821 op_def, inputs, node_def.attr)
1822 self._c_op = _create_c_op(self._graph, node_def, grouped_inputs,
-> 1823 control_input_ops)
1824
1825 # Initialize self._outputs.

~/anaconda3/envs/projectenv/lib/python3.6/site-packages/tensorflow/python/framework/ops.py in _create_c_op(graph, node_def, inputs, control_inputs)
1660 except errors.InvalidArgumentError as e:
1661 # Convert to ValueError for backwards compatibility.
-> 1662 raise ValueError(str(e))
1663
1664 return c_op

ValueError: Shape must be rank 2 but is rank 3 for 'cls/predictions/MatMul' (op: 'MatMul') with input shapes: [?,?,768], [768,30522].

By changing the shape, you can solve the problem. Adding the follow line on author's code
output_layer = tf.reshape(output_layer, [-1, hidden_size])
preds = np.reshape(preds, [batch_size, seq_len, vocab_size])

[deepang17]

By changing the shape, you can solve the problem. Adding the follow line on author's code
output_layer = tf.reshape(output_layer, [-1, hidden_size])
preds = np.reshape(preds, [batch_size, seq_len, vocab_size])

This solution works.

To be more specific to the solution one can do the following:

1. In the class Model, specify the line at:

   model = modeling.BertModel( config=bert_config, is_training=False, input_ids=self.X, use_one_hot_embeddings=False)
   output_layer = model.get_sequence_output()
   embedding = model.get_embedding_table()
   output_layer = tf.reshape(output_layer, [-1, bert_config.hidden_size])

2. For the second part:

   preds = sess.run(tf.nn.log_softmax(model.logits), feed_dict = {model.X: masked_padded})
   preds = np.reshape(preds, [masked_padded.shape[0], masked_padded.shape[1], 30522])
   preds.shape

@huseinzol05 Thank you for repository.
@HarshithaMG Thank you for the question.
@XiaoxueGu Thank you for the solution.

[huseinzol05]

I assume you downloaded wrong bert model.