shininglab/pos-tagger-for-punctuation-restoration

This repository is for the paper Incorporating External POS Tagger for Punctuation Restoration. Proc. Interspeech 2021, 1987-1991, doi: 10.21437/Interspeech.2021-1708.

pos-tagger-for-punctuation-restoration's Issues

AttributeError: 'SequenceTagger' object has no attribute 'tag_dictionary'

--CODE--
...
self.step, self.epoch = 0, 0
self.valid_loss = float('inf')
self.valid_log = ['Start Time: {}'.format(self.start_time)]
self.ref_test_log = self.valid_log.copy()
self.asr_test_log = self.valid_log.copy()
# language model tokenizer
if self.config.lan_model:
self.tokenizer = AutoTokenizer.from_pretrained(self.config.lan_model)
if self.config.pos:
self.pos_tagger = SequenceTagger.load(self.config.pos) if self.config.pos else None
if self.config.use_pos:
self.config.tag_size = len(self.pos_tagger.tag_dictionary) if self.pos_tagger else None
if self.config.pretrained_pos_embed:
self.config.embedding_size = self.pos_tagger.linear.weight.shape[-1]

def load_data(self): 
    for file in [self.config.TRAIN_FILE, self.config.VALID_FILE
    , self.config.REF_TEST_FILE, self.config.ASR_TEST_FILE]:
        if not os.path.exists(self.config.DATA_JSON.format(file)):
            raw_file = self.config.RAW_TXT.format(file)
            raw_data = pipeline.read_file(raw_file, self.config)
            raw_data = pipeline.parse_data(raw_data, self.tokenizer, self.config)
            save.save_json(self.config.DATA_JSON.format(file), raw_data)

...
->I am using this part of code in train.py same as in your github repository, but I am getting error due to tag_dictionary attribute as
shown below

--ERROR--
(AttributeError: 'SequenceTagger' object has no attribute 'tag_dictionary')
Windows PowerShell
Copyright (C) Microsoft Corporation. All rights reserved.

Install the latest PowerShell for new features and improvements! https://aka.ms/PSWindows

PS C:\Users\Manikanth Shetty\OneDrive\Documents\POS-Tagger-for-Punctuation-Restoration-main> cd main
PS C:\Users\Manikanth Shetty\OneDrive\Documents\POS-Tagger-for-Punctuation-Restoration-main\main> python .\train.py
Initialize...
2022-06-07 18:26:15,095 loading file C:\Users\Manikanth Shetty.flair\models\upos-english-fast\b631371788604e95f27b6567fe7220e4a7e8d03201f3d862e6204dbf90f9f164.0afb95b43b32509bf4fcc3687f7c64157d8880d08f813124c1bd371c3d8ee3f7
2022-06-07 18:26:15,455 SequenceTagger predicts: Dictionary with 20 tags: , O, INTJ, PUNCT, VERB, PRON, NOUN, ADV, DET, ADJ, ADP, NUM, PROPN, CCONJ, PART, AUX, X, SYM, ,
Traceback (most recent call last):
File "C:\Users\Manikanth Shetty\OneDrive\Documents\POS-Tagger-for-Punctuation-Restoration-main\main\train.py", line 483, in
main()
File "C:\Users\Manikanth Shetty\OneDrive\Documents\POS-Tagger-for-Punctuation-Restoration-main\main\train.py", line 478, in main
re = Restorer()
File "C:\Users\Manikanth Shetty\OneDrive\Documents\POS-Tagger-for-Punctuation-Restoration-main\main\train.py", line 28, in init
self.initialize()
File "C:\Users\Manikanth Shetty\OneDrive\Documents\POS-Tagger-for-Punctuation-Restoration-main\main\train.py", line 57, in initialize
self.config.tag_size = len(self.pos_tagger.tag_dictionary) if self.pos_tagger else None
File "C:\Users\Manikanth Shetty\AppData\Local\Programs\Python\Python39\lib\site-packages\torch\nn\modules\module.py", line 1185, in getattr
raise AttributeError("'{}' object has no attribute '{}'".format(
AttributeError: 'SequenceTagger' object has no attribute 'tag_dictionary'
PS C:\Users\Manikanth Shetty\OneDrive\Documents\POS-Tagger-for-Punctuation-Restoration-main\main>

-> please help me with this.

ValueError: not enough values to unpack (expected 4, got 3)

I get this error while training on google colab. Started happening recently. Any suggestions/solutions for rectifying this?
Works fine with POS tagging disabled because it skips the problematic code segment.

If I noted correctly, it happens only for trainset_generator, doesn't happen for validset_generator, although I do not know this for sure.

Training...
  0%|          | 0/2237 [00:00<?, ?it/s]
---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
<ipython-input-3-6a92f2de38ba> in <module>()
      7 re=Restorer()
----> 8 re.train()

7 frames
/content/drive/MyDrive/ColabNotebooks/funnel/main/train.py in train(self)
    230             # training set data loader
    231             trainset_generator = tqdm(self.trainset_generator)
--> 232             for data in trainset_generator:
    233                 raw_data, train_data = data
    234                 train_data = (torch.LongTensor(i).to(self.config.device) for i in train_data)

/usr/local/lib/python3.7/dist-packages/tqdm/std.py in __iter__(self)
   1183 
   1184         try:
-> 1185             for obj in iterable:
   1186                 yield obj
   1187                 # Update and possibly print the progressbar.

/usr/local/lib/python3.7/dist-packages/torch/utils/data/dataloader.py in __next__(self)
    519             if self._sampler_iter is None:
    520                 self._reset()
--> 521             data = self._next_data()
    522             self._num_yielded += 1
    523             if self._dataset_kind == _DatasetKind.Iterable and \

/usr/local/lib/python3.7/dist-packages/torch/utils/data/dataloader.py in _next_data(self)
    559     def _next_data(self):
    560         index = self._next_index()  # may raise StopIteration
--> 561         data = self._dataset_fetcher.fetch(index)  # may raise StopIteration
    562         if self._pin_memory:
    563             data = _utils.pin_memory.pin_memory(data)

/usr/local/lib/python3.7/dist-packages/torch/utils/data/_utils/fetch.py in fetch(self, possibly_batched_index)
     42     def fetch(self, possibly_batched_index):
     43         if self.auto_collation:
---> 44             data = [self.dataset[idx] for idx in possibly_batched_index]
     45         else:
     46             data = self.dataset[possibly_batched_index]

/usr/local/lib/python3.7/dist-packages/torch/utils/data/_utils/fetch.py in <listcomp>(.0)
     42     def fetch(self, possibly_batched_index):
     43         if self.auto_collation:
---> 44             data = [self.dataset[idx] for idx in possibly_batched_index]
     45         else:
     46             data = self.dataset[possibly_batched_index]

/content/drive/MyDrive/ColabNotebooks/funnel/main/src/utils/pipeline.py in __getitem__(self, idx)
    188             idx = random.choice(list(self.sample_pool))
    189             self.sample_pool.remove(idx)
--> 190             return generate_seq(self.raw_data[idx:idx+self.config.max_seq_len-2], self.config)
    191         else:
    192             if self.config.use_pos:

/content/drive/MyDrive/ColabNotebooks/funnel/main/src/utils/pipeline.py in generate_seq(lines, config)
     90     for l in lines:
     91         if config.use_pos:
---> 92             token, pun, mask, tag = l
     93         else:
     94             # token, pun, mask, tag

ValueError: not enough values to unpack (expected 4, got 3)

UnicodeDecodeError: 'utf-8' codec can't decode byte 0x92 in position 11087491: invalid start byte

@MrShininnnnn ,
I am getting below error , how to solve this?

ERROR:
Initialize...
2022-06-10 14:19:48,300 loading file C:\Users\Manikanth Shetty.flair\models\upos-english-fast\b631371788604e95f27b6567fe7220e4a7e8d03201f3d862e6204dbf90f9f164.0afb95b43b32509bf4fcc3687f7c64157d8880d08f813124c1bd371c3d8ee3f7
Traceback (most recent call last):
File "c:\Users\Manikanth Shetty\OneDrive\Documents\POS-Tagger-for-Punctuation-Restoration-main\main\train.py", line 483, in
main()
File "c:\Users\Manikanth Shetty\OneDrive\Documents\POS-Tagger-for-Punctuation-Restoration-main\main\train.py", line 478, in main
re = Restorer()
File "c:\Users\Manikanth Shetty\OneDrive\Documents\POS-Tagger-for-Punctuation-Restoration-main\main\train.py", line 29, in init
self.load_data()
File "c:\Users\Manikanth Shetty\OneDrive\Documents\POS-Tagger-for-Punctuation-Restoration-main\main\train.py", line 72, in load_data
train_dataset = pipeline.Dataset(
File "c:\Users\Manikanth Shetty\OneDrive\Documents\POS-Tagger-for-Punctuation-Restoration-main\main\src\utils\pipeline.py", line 160, in init
self.raw_data = load.load_json(data_file)
File "c:\Users\Manikanth Shetty\OneDrive\Documents\POS-Tagger-for-Punctuation-Restoration-main\main\src\utils\load.py", line 12, in load_json
return json.load(f)
File "C:\Users\Manikanth Shetty\AppData\Local\Programs\Python\Python39\lib\json_init_.py", line 293, in load
return loads(fp.read(),
File "C:\Users\Manikanth Shetty\AppData\Local\Programs\Python\Python39\lib\codecs.py", line 322, in decode
(result, consumed) = self._buffer_decode(data, self.errors, final)
UnicodeDecodeError: 'utf-8' codec can't decode byte 0x92 in position 11087491: invalid start byte

Will you please upload Trained model?

Hi @MrShininnnnn

Thanks for your interesting work. I am wondering if you can upload the trained model which can be used for inference or evaluation only.

Thanks