msgi/nlp-journey

Documents, papers and codes related to Natural Language Processing, including Topic Model, Word Embedding, Named Entity Recognition, Text Classificatin, Text Generation, Text Similarity, Machine Translation)，etc. All codes are implemented intensorflow 2.0.

nlp-journey's Issues

tensorflow.python.framework.errors_impl.InvalidArgumentError: indices[0,77] = -1 is not in [0, 49) [Op:GatherV2]

tf.Tensor(
[[ 57 327 630 ... 0 0 0]
[ 850 1979 2299 ... 0 0 0]
[ 474 814 91 ... 0 0 0]
...
[ 44 110 881 ... 0 0 0]
[ 0 33 825 ... 0 0 0]
[1787 1895 119 ... 0 0 0]], shape=(64, 100), dtype=int32)
tf.Tensor(
[[ 0 0 0 ... -1 -1 -1]
[ 1 2 2 ... -1 -1 -1]
[ 0 0 0 ... -1 -1 -1]
...
[ 0 0 3 ... -1 -1 -1]
[ 0 0 0 ... -1 -1 -1]
[ 3 4 0 ... -1 -1 -1]], shape=(64, 100), dtype=int32)
Traceback (most recent call last):
File "c:\Users\Administrator\Desktop\keras_ner\ner.py", line 213, in
file_path='data')
File "c:\Users\Administrator\Desktop\keras_ner\ner.py", line 77, in init
self.model = self.train()
File "c:\Users\Administrator\Desktop\keras_ner\ner.py", line 103, in train
loss, logits, text_lens = self.train_one_step(model, optimizer, text_batch, labels_batch)
File "c:\Users\Administrator\Desktop\keras_ner\ner.py", line 117, in train_one_step
logits, text_lens, log_likelihood = model(text_batch, labels_batch, training=True)
File "D:\anaconda3\lib\site-packages\tensorflow_core\python\keras\engine\base_layer.py", line 822, in call
outputs = self.call(cast_inputs, *args, **kwargs)
File "c:\Users\Administrator\Desktop\keras_ner\ner.py", line 41, in call
log_likelihood, self.transition_params = ta.text.crf_log_likelihood(logits, label_sequences, text_lens)
File "D:\anaconda3\lib\site-packages\tensorflow_addons\text\crf.py", line 214, in crf_log_likelihood
inputs, tag_indices, sequence_lengths, transition_params
File "D:\anaconda3\lib\site-packages\tensorflow_addons\text\crf.py", line 77, in crf_sequence_score
return tf.cond(tf.equal(tf.shape(inputs)[1], 1), _single_seq_fn, _multi_seq_fn)
File "D:\anaconda3\lib\site-packages\tensorflow_core\python\ops\control_flow_ops.py", line 1389, in cond_for_tf_v2
return cond(pred, true_fn=true_fn, false_fn=false_fn, strict=True, name=name)
File "D:\anaconda3\lib\site-packages\tensorflow_core\python\util\deprecation.py", line 507, in new_func
return func(*args, **kwargs)
File "D:\anaconda3\lib\site-packages\tensorflow_core\python\ops\control_flow_ops.py", line 1204, in cond
result = false_fn()
File "D:\anaconda3\lib\site-packages\tensorflow_addons\text\crf.py", line 72, in _multi_seq_fn
tag_indices, sequence_lengths, transition_params
File "D:\anaconda3\lib\site-packages\tensorflow_addons\text\crf.py", line 291, in crf_binary_score
binary_scores = tf.gather(flattened_transition_params, flattened_transition_indices)
File "D:\anaconda3\lib\site-packages\tensorflow_core\python\util\dispatch.py", line 180, in wrapper
return target(*args, **kwargs)
File "D:\anaconda3\lib\site-packages\tensorflow_core\python\ops\array_ops.py", line 4125, in gather_v2
batch_dims=batch_dims)
File "D:\anaconda3\lib\site-packages\tensorflow_core\python\util\dispatch.py", line 180, in wrapper
return target(*args, **kwargs)
File "D:\anaconda3\lib\site-packages\tensorflow_core\python\ops\array_ops.py", line 4108, in gather
return gen_array_ops.gather_v2(params, indices, axis, name=name)
File "D:\anaconda3\lib\site-packages\tensorflow_core\python\ops\gen_array_ops.py", line 3683, in gather_v2
_ops.raise_from_not_ok_status(e, name)
File "D:\anaconda3\lib\site-packages\tensorflow_core\python\framework\ops.py", line 6606, in raise_from_not_ok_status
six.raise_from(core._status_to_exception(e.code, message), None)
File "", line 3, in raise_from
tensorflow.python.framework.errors_impl.InvalidArgumentError: indices[0,77] = -1 is not in [0, 49) [Op:GatherV2]

大神，这是数据集问题吗？

keras_contrib报错

2020-04-24 11:14:12.760618: W tensorflow/stream_executor/platform/default/dso_loader.cc:55] Could not load dynamic library 'cudart64_101.dll'; dlerror: cudart64_101.dll not found
2020-04-24 11:14:12.760859: I tensorflow/stream_executor/cuda/cudart_stub.cc:29] Ignore above cudart dlerror if you do not have a GPU set up on your machine.
Using TensorFlow backend.
2020-04-24 11:14:17.101006: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library nvcuda.dll
2020-04-24 11:14:17.535028: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1555] Found device 0 with properties:
pciBusID: 0000:01:00.0 name: GeForce GTX 1050 computeCapability: 6.1
coreClock: 1.493GHz coreCount: 5 deviceMemorySize: 4.00GiB deviceMemoryBandwidth: 104.43GiB/s
2020-04-24 11:14:17.537454: W tensorflow/stream_executor/platform/default/dso_loader.cc:55] Could not load dynamic library 'cudart64_101.dll'; dlerror: cudart64_101.dll not found
2020-04-24 11:14:17.539765: W tensorflow/stream_executor/platform/default/dso_loader.cc:55] Could not load dynamic library 'cublas64_10.dll'; dlerror: cublas64_10.dll not found
2020-04-24 11:14:17.541926: W tensorflow/stream_executor/platform/default/dso_loader.cc:55] Could not load dynamic library 'cufft64_10.dll'; dlerror: cufft64_10.dll not found
2020-04-24 11:14:17.544355: W tensorflow/stream_executor/platform/default/dso_loader.cc:55] Could not load dynamic library 'curand64_10.dll'; dlerror: curand64_10.dll not found
2020-04-24 11:14:17.547207: W tensorflow/stream_executor/platform/default/dso_loader.cc:55] Could not load dynamic library 'cusolver64_10.dll'; dlerror: cusolver64_10.dll not found
2020-04-24 11:14:17.549591: W tensorflow/stream_executor/platform/default/dso_loader.cc:55] Could not load dynamic library 'cusparse64_10.dll'; dlerror: cusparse64_10.dll not found
2020-04-24 11:14:17.554696: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library cudnn64_7.dll
2020-04-24 11:14:17.554892: W tensorflow/core/common_runtime/gpu/gpu_device.cc:1592] Cannot dlopen some GPU libraries. Please make sure the missing libraries mentioned above are installed properly if you would like to use GPU. Follow the guide at https://www.tensorflow.org/install/gpu for how to download and setup the required libraries for your platform.
Skipping registering GPU devices...
2020-04-24 11:14:17.555937: I tensorflow/core/platform/cpu_feature_guard.cc:142] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX2
2020-04-24 11:14:17.556583: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1096] Device interconnect StreamExecutor with strength 1 edge matrix:
2020-04-24 11:14:17.556732: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1102]
Traceback (most recent call last):
File "c:\Users\Administrator\Desktop\keras_ner\ner.py", line 163, in
ner = BiLSTMCRFNamedEntityRecognition('model/crf.h5', 'model/config.pkl', train=True, file_path='data/')
File "c:\Users\Administrator\Desktop\keras_ner\ner.py", line 73, in init
self.model = self.train()
File "c:\Users\Administrator\Desktop\keras_ner\ner.py", line 77, in train
model = self.__build_model()
File "c:\Users\Administrator\Desktop\keras_ner\ner.py", line 134, in __build_model
model.add(crf)
File "D:\anaconda3\envs\tensorflow\lib\site-packages\tensorflow_core\python\training\tracking\base.py", line 457, in _method_wrapper
result = method(self, *args, **kwargs)
File "D:\anaconda3\envs\tensorflow\lib\site-packages\tensorflow_core\python\keras\engine\sequential.py", line 161, in add
'Found: ' + str(layer))
TypeError: The added layer must be an instance of class Layer. Found: <keras_contrib.layers.crf.CRF object at 0x0000018943E28A58>

Keras 2.3.1
keras-contrib 2.0.8
tensorflow 2.1.0

你好，你的版本是这些吗

Undefined names in pre_process.py

flake8 testing of https://github.com/msgi/nlp-journey on Python 3.7.1

$ flake8 . --count --select=E9,F63,F7,F82 --show-source --statistics

./nlp/utils/pre_process.py:7:56: F821 undefined name 'stop_words'
    sentences = [' '.join([l for l in line if l not in stop_words]) for line in lines]
                                                       ^
./nlp/utils/pre_process.py:7:81: F821 undefined name 'lines'
    sentences = [' '.join([l for l in line if l not in stop_words]) for line in lines]
                                                                                ^
./nlp/utils/pre_process.py:10:22: F821 undefined name 'lines'
        o.writelines(lines)
                     ^
3     F821 undefined name 'stop_words'
3

E901,E999,F821,F822,F823 are the "showstopper" flake8 issues that can halt the runtime with a SyntaxError, NameError, etc. These 5 are different from most other flake8 issues which are merely "style violations" -- useful for readability but they do not effect runtime safety.

• F821: undefined name name
• F822: undefined name name in __all__
• F823: local variable name referenced before assignment
• E901: SyntaxError or IndentationError
• E999: SyntaxError -- failed to compile a file into an Abstract Syntax Tree

能否提供数据文件，蟹蟹

No such file or directory: 'model/gensim/model.txt'

您好，我想问一下，怎么才能找到model/gensim/model.txt？