Warning for account deactivation

[Konstantina-Lazaridou]

Warning for account deactivation

[gregorleban]

Hi,

the warning is to let users know that they are using complex queries which seem to take a lot of time to return results. If you have a paid subscription, however, the accounts are never disabled. The accounts are disabled only for users with free trial accounts.

You don't need to add any time period between the individual requests. The library is by itself adding small time waits between individual requests.

[Konstantina-Lazaridou]

Hi @gregorleban , thanks for the answer.
I couldn't find this warning anywhere in the documentation though. Also, I have been using the same type of queries in the last days and the warning started happening suddenly without an explanation about which part of my query is/became too complex. Is it possible to make this more transparent in terms of the query complexity? Having a different error for paid subscriptions and free accounts would also help. Another issue is that the API became slower and slower with this warning (both because the query execution was slow and I think printing the warning in the console as well) and as a result I didn't manage to finish my monthly quota. I would really appreciate knowing which parameter made my search query that complex and slow.

This is the kind of queries I have been posing this week and I didn't change something when the warning started appearing:

query = QueryArticlesIter(
        keywordsLoc="body,title",
        ignoreKeywords=["keyword a", "keyword b"],
        ignoreKeywordsLoc="title",
        ignoreSourceUri=QueryItems.OR(items=lis_of_uris),
        keywords=QueryItems.AND(items=keywords),
        dataType=["news"],
        categoryUri=category_uri,
        isDuplicateFilter="skipDuplicates",
        dateStart="2019-01-01",
        dateEnd="2020-11-01",
        startSourceRankPercentile=0,
        endSourceRankPercentile=50,
      )
query_result = query.execQuery(event_registry_object,
                                   sortBy=sorting_method,
                                   sortByAsc=False,
                                   returnInfo=ReturnInfo(
                                     articleInfo=ArticleInfoFlags
                                     (concepts=True,
                                      categories=True,
                                      location=True,
                                      image=False)),
                                   maxItems=result_size)

[gregorleban]

Hi Konstantina, from the query that you've specified it's hard to say what specifically made the query slow. For example, if you've used dmoz/Business category as one of the categories in the query, then that could be the reason for it. dmoz/Business has a huge number of subcategories (something in the range of 2000) that need to be included in the search when you use it. A much faster category that gives very similar results is using news/business (which doesn't have any subcategories). Similarly, depending on the keywords that you've been using it could be that some categories have a huge number of results which could make the query processing slower. The most likely culprit is however using a very generic category. If you were not able to use the tokens due to this issue, I'd be happy to extend you the usage to use the full plan size for free. Send me an email to [email protected] and we will discuss the details there. Best, Gregor

…

On Mon, Nov 2, 2020 at 11:23 AM Konstantina Lazaridou < ***@***.***> wrote: Hi @gregorleban <https://github.com/gregorleban> , thanks for the answer. I couldn't find this warning anywhere in the documentation though. Also, I have been using the same type of queries in the last days and the warning started happening suddenly without an explanation about which part of my query is/became too complex. Is it possible to make this more transparent in terms of the query complexity? Having a different error for paid subscriptions and free accounts would also help. Another issue is that the API became slower and slower with this warning (both because the query execution was slow and I think printing the warning in the console as well) and as a result I didn't manage to finish my monthly quota. I would really appreciate knowing which parameter made my search query that complex and slow. This is the kind of queries I have been posing this week and I didn't change something when the warning started appearing: query = QueryArticlesIter( keywordsLoc="body,title", ignoreKeywords=["keyword a", "keyword b"], ignoreKeywordsLoc="title", ignoreSourceUri=QueryItems.OR(items=lis_of_uris), keywords=QueryItems.AND(items=keywords), dataType=["news"], categoryUri=category_uri, isDuplicateFilter="skipDuplicates", dateStart="2019-01-01", dateEnd="2020-11-01", startSourceRankPercentile=0, endSourceRankPercentile=50, ) — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#56 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAGFVOQ42O4GRXXULJQN3XDSN2CDJANCNFSM4TGBKWXQ> .

-- ------------------------------------------- Gregor Leban Phone: [phone] Skype: gregorleban -------------------------------------------

[Konstantina-Lazaridou]

Thank you very much @gregorleban . Unfortunately, even though I changed the category to "news/Business", after approximately 30 searches I start to get the warning again.

=========== WARNING ===========
The processing of the request took a lot of time (18 sec). By repeatedly making slow requests your account will be automatically disabled.

Could you please let me know which other factors can make my query slow?

This is how my query looks like:

query = QueryArticlesIter(
      keywordsLoc="body,title",
      ignoreKeywords=["a", "b", "c", "d", "e"],  # 5 keywords
      ignoreKeywordsLoc="title",
      ignoreSourceUri=ignore_sources,  # 30 sources
      keywords=QueryItems.AND(items=keywords),  # 1 to 3 keywords
      conceptUri=concept_uri,  # a wikipedia organizaytion entry
      dataType=article_types,  # only news
      categoryUri=news_categories,  #  news/business
      isDuplicateFilter=duplicate_filter,  # skipDuplicates
      dateStart=date_start,  # 2020-01-01
      dateEnd=date_end,  # today
      startSourceRankPercentile=0,
      endSourceRankPercentile=60,

[gregorleban]

You're using a lot of filters and across the full year of content so it's natural that the queries take longer to execute.

…

On Mon, Nov 30, 2020 at 11:37 AM Konstantina Lazaridou < ***@***.***> wrote: Thank you very much @gregorleban <https://github.com/gregorleban> . Unfortunately, even though I changed the category to "news/Business", after approximately 30 searches I start to get the warning again. =========== WARNING =========== The processing of the request took a lot of time (18 sec). By repeatedly making slow requests your account will be automatically disabled. Could you please let me know which other factors can make my query slow? This is how my query looks like: query = QueryArticlesIter( keywordsLoc="body,title", ignoreKeywords=["a", "b", "c", "d", "e"], # 5 keywords ignoreKeywordsLoc="title", ignoreSourceUri=ignore_sources, # 30 sources keywords=QueryItems.AND(items=keywords), # 1 to 3 keywords conceptUri=concept_uri, # a wikipedia organizaytion entry dataType=article_types, # only news categoryUri=news_categories, # news/business isDuplicateFilter=duplicate_filter, # skipDuplicates dateStart=date_start, # 2020-01-01 dateEnd=date_end, # today startSourceRankPercentile=0, endSourceRankPercentile=60, — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#56 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAGFVOV4AG7JB5OGEDCQSNDSSNYWRANCNFSM4TGBKWXQ> .

-- ------------------------------------------- Gregor Leban Phone: [phone] Skype: gregorleban -------------------------------------------