Any config for DeBERTa series as decoders for TSDAE?

[bobox2997]

Any config for DeBERTa series as decoders for TSDAE?

[amyeroberts]

Hi @bobox2997, thanks for opening this PR!

What I would suggest is adding checkpoints, configs and possibly updated modeling files directly on the hub, and have as much support as we can there. It will be easier to integrate than directly into transformers. Here is a tutorial if that sound good to you!

[bobox2997]

I'm not sure that I understood that correctly... What checkpoint should I add?

On TSDAE implementation the decoder portion is tied to the encoder and is not used at inference, I just need an "is_decoder" argument (and related configs of course) in the config of DeBERTa as there is for BERT, RoBERTa et similia models...

I'm sorry if those are naive or dumb questions, I'm still learning.

Thank you so much for your time!

[amyeroberts]

@bobox2997 Ah, OK, I thought there were checkpoints available trained with this method.

In terms of changes in the transformers library, we're very unlikely to accept changes to the architecture or configurations files to add new features like this, especially for older, popular models and anything which doesn't have official checkpoints available.

The great thing about open source is that you are free to build upon and adapt the available code (license permitting) for your own projects. It should be possible to add as a new architecture on the hub, keeping compatibility with the transformers library and allowing you to use the same API.

If you or anyone else in the community would like to implement this, feel free to share you project here!