Priority Manager

[up2040499]

Priority Manager

[UP2040499]

For better performance, will want to reduce the number of sources before finding popular entities.
This can be done by performing the scoring of sources based on target information first, then taking the top x% of scored sources and putting them into the popular info finder. Finally, can score the remaining sources based on popular information.

[UP2040499]

So for unknown reasons, when running PopularInformationFinder from within the unit_test works fine. However, when it is run from inside the PriorityManager class, nothing is generated with the multiprocessing.Pool.map function.

[UP2040499]

To finish:

• Fix the error message 'XMLParsedAsHTML' by switching the parser used in BS4, based on the website file type.
• Attempt to minimise the 'Token indices sequence length is longer than the specified maximum sequence length for this model.' warning.
  • This can be reduced by changing the 'window' and 'stride' in the config of the model
  • See Spacy Docs - strided_spans.v1

[UP2040499]

• Fix 'XMLParsedAsHTML' warning
• Reduce the 'Token indices' warning occurrence by modifying the window and stride fields in the model config.