3 sentences as input for BertForSequenceClassification?

[huggingface]

3 sentences as input for BertForSequenceClassification?

[artemisart]

Technically it is possible but BERT was not pretrained to handle multiple SEP tokens between sentences and does not have a third token_type, so I think it won't be easy to make it work. You may also want to use a new token for the second separation.

[mikelkl]

@thedrowsywinger maybe u should try Transformer-XL

[mikelkl]

Technically it is possible but BERT was not pretrained to handle multiple SEP tokens between sentences and does not have a third token_type, so I think it won't be easy to make it work. You may also want to use a new token for the second separation.

Hi artemisart,

Thanks for your reply.

So, if someone wanna take multiple sentences as input of BertForSequenceClassification, let's say a whole passage, an alternative way is to concatenate them into a single "sentence" and then fit it in, right?

[artemisart]

No it shouldn't

[artemisart]

I you don't have a separation (like question/answer) then yes you can just concatenate them (but you are still limited to 512 tokens).

[thomwolf]

@mikelkl I would also go with the solution and answer of @artemisart.

[alphanlp]

@artemisart hi, if i have a single sentence classification task, should the max length of sentence limited to half of 512, that is to say 256?

[artemisart]

No, it will be better if you use the full 512 tokens.

[thedrowsywinger]

wouldn't concatenating the whole passage into a single sentence mean losing context of each sentence? @artemisart

[thedrowsywinger]

What if I want to check on a huge corpus, that even concatenating into one sentence exceeds the 512 token limit? @artemisart

[sid8491]

I you don't have a separation (like question/answer) then yes you can just concatenate them (but you are still limited to 512 tokens).

I have 3 inputs, 1 of the input contains conversation (QUERY, ANSWER).
QUERY: I want to ask a question.

ANSWER: Sure, ask away.
QUERY: How is the weather today?
ANSWER: It is nice and sunny.
QUERY: Okay, nice to know.
ANSWER: Would you like to know anything else?

How can I tell the model to separate the turns of conversation? Model is classification model.
I was thinking to add a new special token between the turns but could not get it work.