Config specifies max_position_embeddings as 1024

[JohnGiorgi]

Config specifies max_position_embeddings as 1024

[jayded]

I can confirm that this affects run_summarization.py and is inconsistent with the semantics of other Huggingface configs.

[Wendy-Xiao]

Hi John and Jay,

Thanks for pointing out the issue. It is not intentional to set max_position_embedding to be 1024, it's just set by default.

I have not used run_summarization.py before, so I didn't know that in that script they treat max_position_embedding as the maximum position embedding size for the encoder, which is specified as max_encoder_position_embeddings in our case.

I'll update it in the config files, to make it consistent with the script.

[JohnGiorgi]

Was able to address it temporarily by doing:

from transformers import AutoConfig

config = AutoConfig.from_pretrained("allenai/PRIMERA")
config.max_position_embeddings = 4096
config.save_pretrained("allenai/PRIMERA-fixed")

and then passing "allenai/PRIMERA-fixed" to the run_*.py scripts via the --config_name flag.

[JohnGiorgi]

Awesome, thank you @Wendy-Xiao!

[JohnGiorgi]

Hi @Wendy-Xiao I see this is fixed for base PRIMERA model. Would you mind making the same change to the fine-tuned models? (e.g. multi-news, multi-x-science, etc.). Nvm I made PRs for each model they would just need to be merged!

[JohnGiorgi]

Closing this as it was addressed, thank you @Wendy-Xiao!