long text

[adsabs]

long text

[spacemansteve]

We still see long words in the body field, the code to prevent this is at https://github.com/adsabs/ADSfulltext/blob/master/adsft/utils.py#L246. The computed body field for bibcode 2018Tectp.722...69L has a very long toke that begins with
#[U] ppm[Th] ppm[Pb] ppmTh/U meas207Pb/206Pbs %207Pb/235Us %206Pb/238Us %r207Pb/206Pbs207Pb/235Us206Pb/238Usf206%54,[email protected]
Long tokens are identified with a regular expression r'\b\w{'+str(maxlength)+r',}. It doesn't work well because the tables that appear as a single token are full of special characters like /, %, @, etc. A very long string that repeated %123%123%123 would not be changed. Instead, I think \b\w should be replaced with \S which will match on all non-whitespace characters, including special characters which the tokens are full of.

[spacemansteve]

reopen until deployed

[spacemansteve]

Should not have been re-opened. Recent "token too large" were fixed by re-extracting fulltext. The code works.