indonlp/indonlg

The first-ever vast natural language generation benchmark for Indonesian, Sundanese, and Javanese. We provide multiple downstream tasks, pre-trained IndoGPT and IndoBART models, and a starter code! (EMNLP 2021)

indonlg's Issues

Saving Tokenizer using pretrained format

Hi! I want to re-built the model for Question Answering, but when i want to save the model using .save_pretrained() using this code:
tokenizer.save_pretrained('./model')

And There was Error :
NotImplementedError Traceback (most recent call last)
Cell In[4], line 1
----> 1 tokenizer.save_pretrained('./model')

File /usr/local/lib/python3.11/site-packages/transformers/tokenization_utils_base.py:2174, in PreTrainedTokenizerBase.save_pretrained(self, save_directory, legacy_format, filename_prefix, push_to_hub, **kwargs)
2170 logger.info(f"Special tokens file saved in {special_tokens_map_file}")
2172 file_names = (tokenizer_config_file, special_tokens_map_file)
-> 2174 save_files = self._save_pretrained(
2175 save_directory=save_directory,
2176 file_names=file_names,
2177 legacy_format=legacy_format,
2178 filename_prefix=filename_prefix,
2179 )
2181 if push_to_hub:
2182 self._upload_modified_files(
2183 save_directory,
2184 repo_id,
(...)
2187 token=kwargs.get("use_auth_token"),
2188 )

File /usr/local/lib/python3.11/site-packages/transformers/tokenization_utils_base.py:2222, in PreTrainedTokenizerBase._save_pretrained(self, save_directory, file_names, legacy_format, filename_prefix)
2219 f.write(out_str)
2220 logger.info(f"added tokens file saved in {added_tokens_file}")
-> 2222 vocab_files = self.save_vocabulary(save_directory, filename_prefix=filename_prefix)
2224 return file_names + vocab_files + (added_tokens_file,)

File /usr/local/lib/python3.11/site-packages/transformers/tokenization_utils_base.py:2242, in PreTrainedTokenizerBase.save_vocabulary(self, save_directory, filename_prefix)
2226 def save_vocabulary(self, save_directory: str, filename_prefix: Optional[str] = None) -> Tuple[str]:
2227 """
2228 Save only the vocabulary of the tokenizer (vocabulary + added tokens).
2229
(...)
2240 Tuple(str): Paths to the files saved.
2241 """
-> 2242 raise NotImplementedError

NotImplementedError:

Do you have any suggestion to tackle the issue?