URL tokenization does not match GET params

[UnduGT]

URL tokenization does not match GET params

[UnduGT]

Sorry for the late reply, but yes it works now exactly as it should! Perfect, thank you!

[hopperelec]

There's quite a few issues with that regex (unnecessary grouping, TLD limited to 6 characters, TLD required, misses many characters allowed in path). I feel like it would make more sense to use a regex actually already used by one of the supported platforms, since I'm pretty sure it's anything that the platform highlights blue which we want to exclude. I already found the regex Discord uses ^(https?:\/\/[^\s<]+[^<.,:;"')\]\s]) and it is much more simple and lenient (matching many non-URLs, which I'd much prefer over not matching many URLs). I can't remember why their regex only matches the beginning of a string, but modified for ChatAnalytics it would be https?:\/\/[^\s<]+[^<.,:;"')\]\s]. I'll make a PR now

[mlomb]

Hey @UnduGT could you try generating the report again? (using the web app)
Thanks @hopperelec for the Regex!

[mlomb]

Hi! Thanks for the report!
I havent experienced this myself, you mean the small embed Discord includes after sending a link or the link itself?
The links itself should be tokenized as URLs and be completely excluded from the Language tab. Maybe the tokenizer is broken?

chat-analytics/pipeline/process/nlp/Tokenizer.ts

Lines 51 to 56 in de8c517

	{
	// match URLs
	// TODO: find a better regex, @dperini version (not this one ↓) is probably too restrictive for us
	regex: /(?:https?:\/\/)(?:[\da-z\.-]+)\.(?:[a-z\.]{2,6})(?:[\/\w\.\-\?#=@]*)*\/?/gi,
	tag: "url",
	},

[UnduGT]

It's the link itself, for example if I search for "MakeModelVariant1" in Discord it shows over 400 results consisting only of links from the same page, no Embeds.

[mlomb]

It seems the regex is missing additional URL parameters!

(link has been OCRed)

We should find a better regular expression

[hopperelec]

Could just add an ampersand to the character set. Also, not sure what the purpose of that non-capturing group is, or the trailing forward slash considering that's already part of the prior character set

[mlomb]

Lets go for it, do you want to open a PR?
I did not write that regex, I shamelessly stole it from the internet 😅. It has to be revised at some point