brave-intl/bat-native-usermodel

bat-native-usermodel's Issues

Calculate winning category for Japanese

[Android] Fix bat-native-usermodel unit tests failing due to invalid test data path

See brave/brave-browser#9511

Compiler error on Windows with Chromium 75.

c75 switches Windows to using libc++ which seems to be causing a compilation error in usermodel_unittest.cc:54 in std::ifstream ifs(path.value());

../../brave/vendor/bat-native-usermodel/test/usermodel_unittest.cc(54,19):  error: no matching constructor for initialization of  'std::ifstream' (aka 'basic_ifstream')
std::ifstream ifs(path.value());

Candidate constructors:
explicit basic_ifstream(const char* __s, ios_base::openmode __mode = ios_base::in);                                                        
explicit basic_ifstream(const wchar_t* __s, ios_base::openmode __mode = ios_base::in);
explicit basic_ifstream(const string& __s, ios_base::openmode __mode = ios_base::in);                                                      
basic_ifstream(basic_ifstream&& __rhs);                                                                                           basic_ifstream();                                                                                                                 ^                                                                                                                             
1 error generated.

FilePath defines const StringType& value() const and StringType is defined as

#if defined(OS_WIN)
  // On Windows, for Unicode-aware applications, native pathnames are wchar_t
  // arrays encoded in UTF-16.
  typedef std::wstring StringType;
#elif defined(OS_POSIX) || defined(OS_FUCHSIA)
  // On most platforms, native pathnames are char arrays, and the encoding
  // may or may not be specified.  On Mac OS X, native pathnames are encoded
  // in UTF-8.
  typedef std::string StringType;
#endif  // OS_WIN

So, on Windows this returns a std::wstring. MSVC probably has an additional constructor, but libc++ doesn't. Changing the code to explicitly get wchar_t* std::ifstream ifs(path.value().c_str()); fixes it.

Set up branch protection

All repos that are used in production should have branch protection set up; see https://github.com/brave/internal/wiki/Pull-request-security-audit-checklist#github-repository-security for details

Fix deps in BUILD.gn

$ gn check . //brave/vendor/bat-native-usermodel/*
ERROR at //brave/vendor/bat-native-usermodel/BUILD.gn:46:1: Source file not found.
source_set("usermodel") {
^------------------------
The target:
  //brave/vendor/bat-native-usermodel:usermodel
has a source file:
  //brave/vendor/bat-native-usermodel/advertising_categories.h
which was not found.
```

Create persisted japanese model for use in bat-native-ads/resources

Refactor code to adhere to coding standards

Cleanup project structure

Cleanup project structure

Pass full text of html page to classifier

Currently the html passed to the classifier is only a subset of the full html. Pass the entirety of the extracted html.body.text to the classifier

Implement Japanese pipeline

Append immediate page classification score to history based upon locale

Fix linter errors

Page classifier crashes if there is no content

Add ability to get taxonomy names

Add ability to get taxonomy names

Add more training data for missing categories (for english/french/german)

Fix compiler warnings

Support different pipelines based upon locale when classifying pages

Some pages getting misclassified in user model

Based on feedback from our user group, some of this content is being misclassified:

1. https://www.curbed.com/ - Showing up as ["technology & computing","software"] probably more like "architecture" or "real estate"
2. https://www.apartmenttherapy.com/ - Saying not enough content to classify, but seems to be enough, and probably should be "architecture" or "real estate"

Train and create a JSON Japanese user model

We are going to train a model for Japanese content, similar to what was done for the German model, from translated content into our current taxonomy. In the short term, the model should match the short term interest with the ad, until we can train a more sophisticated one.