ksjae/kogpt

GPT-2 pretrained on Korean datasets.

kogpt's Issues

v0.2-text underperforming compared to v0.2-beta

라이센스 정책

안녕하세요. https://static.ksjit.com/datasets/tinyDedup.txt 데이터의 라이센스 정책이 궁금합니다. 감사합니다.

AttributeError: 'GPT2Model' object has no attribute '_step'

I meet an error while I running the following code to convert the tensorflow checkpoint into pytorch version.

export OPENAI_GPT2_CHECKPOINT_PATH=kogpt2-large-v2-tf/model.ckpt-225000

transformers-cli convert --model_type gpt2 \
  --tf_checkpoint $OPENAI_GPT2_CHECKPOINT_PATH \
  --pytorch_dump_output kogpt2-large-v2-tf/pytorch_model.bin

What's the problem? Could you please help me ? Thank you.

...
Loading TF weight newslm/layer23/query_layer/kernel/adafactor_vc with shape [1536]
Loading TF weight newslm/layer23/query_layer/kernel/adafactor_vr with shape [1536]
Loading TF weight newslm/layer23/value_layer/bias with shape [1536]
Loading TF weight newslm/layer23/value_layer/bias/adafactor_v with shape [1536]
Loading TF weight newslm/layer23/value_layer/kernel with shape [1536, 1536]
Loading TF weight newslm/layer23/value_layer/kernel/adafactor_vc with shape [1536]
Loading TF weight newslm/layer23/value_layer/kernel/adafactor_vr with shape [1536]
Traceback (most recent call last):
  File "/users6/kyzhang/anaconda3/bin/transformers-cli", line 8, in <module>
    sys.exit(main())
  File "/users6/kyzhang/anaconda3/lib/python3.6/site-packages/transformers/commands/transformers_cli.py", line 51, in main
    service.run()
  File "/users6/kyzhang/anaconda3/lib/python3.6/site-packages/transformers/commands/convert.py", line 152, in run
    convert_gpt2_checkpoint_to_pytorch(self._tf_checkpoint, self._config, self._pytorch_dump_output)
  File "/users6/kyzhang/anaconda3/lib/python3.6/site-packages/transformers/models/gpt2/convert_gpt2_original_tf_checkpoint_to_pytorch.py", line 39, in convert_gpt2_checkpoint_to_pytorch
    load_tf_weights_in_gpt2(model, config, gpt2_checkpoint_path)
  File "/users6/kyzhang/anaconda3/lib/python3.6/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 110, in load_tf_weights_in_gpt2
    pointer = getattr(pointer, scope_names[0])
  File "/users6/kyzhang/anaconda3/lib/python3.6/site-packages/torch/nn/modules/module.py", line 1131, in __getattr__
    type(self).__name__, name))
AttributeError: 'GPT2Model' object has no attribute '_step'

vocab.json 파일이 one drive와 google drive에서 서로 다른 것 같습니다.

vocab.json 파일이 one drive와 google drive에서 서로 다른 것 같습니다.
google drive 파일에서 받은 tokenizer는 잘 decoding 되는데, one drive에서 받은 Tokenizer가 잘못 decoding 되는 것 같아요.
확인 부탁드립니다!

Tokenizer 문의

안녕하세요!
새로 올려주신 파일로 돌려보았을 때 아래와 같이 동작하고 있습니다.

>>> input_ids = tokenizer.encode("안녕하세요.", return_tensors='tf')
>>> input_ids
<tf.Tensor: shape=(1, 16), dtype=int32, numpy=
array([[169, 244, 231, 168, 228, 244, 170, 244, 247, 169, 227, 117, 169,
        249, 243,  14]], dtype=int32)>
>>> sample_outputs = model.generate(input_ids, do_sample=True, max_length=100, top_k=10, top_p=0.9, num_return_sequences=1)
>>> sample_outputs[0]
<tf.Tensor: shape=(100,), dtype=int32, numpy=
array([  169,   244,   231,   168,   228,   244,   170,   244,   247,
         169,   227,   117,   169,   249,   243,    14, 22689,   125,
          15,   309,  4779,   107,   103, 11647,   129,   745, 23762,
         114, 17173,   231,   281,  3074,   244,   245,   170,   104,
         125,   474,   467,    20,  1758,   391,  3074,   244,   245,
         170,   104,   125,   276,  2866,  4779,   107,   103, 11647,
         129,   271,  3796,  2516,   243,   260, 17008,   240, 10112,
         335,   915,  3099,  2988,  4779,   107,   103, 11647,   129,
         304,  4054,   120,   116,  9013,   281,  3074,   244,   245,
         170,   104,   125,   474,   467,    20,   205, 21087,   241,
        3074,   244,   245,   170,   104,   125,   361,  4779,   107,
         103], dtype=int32)>
>>> tokenizer.decode(sample_outputs[0])
'안녕하세요. 펼쳤다�/�였지만��文� 부� 영감은�천에�가먹방�����지만 �4남�먹방������ �였지만��文��눈 약간� � 최후의� 독특한 몄로 보이지아야였지만��文�한 빨뻷데일리�가먹방�����지만 �4\x10 재판�먹방�����었다였지만��'

1. "안녕하세요."가 굉장히 많은 token들로 분리되는데, 이 부분은 의도하신 부분인가요? 이전에 올려주셨던 tokenizer에서는 이렇지 않았던 것 같아서 문의드립니다.

2. 모델의 output이 디코딩 했을 때 많이 깨지는 것으로 보이는데, 이러한 결과가 많이 나오는 것이 정상적인 결과인가요?

감사합니다!