URL/Website content extraction

[microsoft]

URL/Website content extraction

[dmitri012]

• vote for website content extraction.

Document loading I can solve my self by uploading files to appropriate blob container. Website content extraction needs more specification about supported formats (html, txt, json, etc). Ideally is to provide ready scraper script/function.
I my case I have access to database and can extract content via SQL queries, but supported output format is not clear yet.

[dayland]

@dmitri012, the supported document formats are available at https://github.com/microsoft/PubSec-Info-Assistant/blob/main/docs/features/features.md#supported-document-types

[jdnuckolls]

I'd really like to see this take priority as it is a requirement of just about every customer my team works with. Usually when we tell them this feature is not available in this repo, they use a different repo that already has this feature and they mis out on all the great work that has gone into this repo.