mattlisiv/newsapi-python

A Python Client for News API

newsapi-python's Issues

Missing searchIn

Describe the bug
There's a qintitle param but the API support a searchIn param, so the current functionality is limited.

Enable use of proxies

It could be useful to enable the use of proxies leveraging the proxies parameter from standard requests library.

Wrong example for python

I follow this document https://newsapi.org/docs/client-libraries/python and test python api.

Seems like the follow code will get error I paste.
sources = newsapi.get_sources()

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "crawler.py", line 93, in <module>
    crawler.crawl()
  File "crawler.py", line 84, in crawl
    sources = newsapi.get_sources()
  File "/Users/mabodx/anaconda/lib/python3.6/site-packages/newsapi/newsapi_client.py", line 311,
 in get_sources
    r = requests.get(const.SOURCES_URL, auth=self.auth, timeout=30, params=payload)
  File "/Users/mabodx/anaconda/lib/python3.6/site-packages/requests/api.py", line 72, in get
    return request('get', url, params=params, **kwargs)
  File "/Users/mabodx/anaconda/lib/python3.6/site-packages/requests/api.py", line 58, in request
    return session.request(method=method, url=url, **kwargs)
  File "/Users/mabodx/anaconda/lib/python3.6/site-packages/requests/sessions.py", line 513, in r
equest
    resp = self.send(prep, **send_kwargs)
  File "/Users/mabodx/anaconda/lib/python3.6/site-packages/requests/sessions.py", line 623, in s
end
    r = adapter.send(request, **kwargs)
  File "/Users/mabodx/anaconda/lib/python3.6/site-packages/requests/adapters.py", line 514, in s
end
    raise SSLError(e, request=request)
requests.exceptions.SSLError: ("bad handshake: SysCallError(-1, 'Unexpected EOF')",)

ImportError: cannot import name 'NewsApiClient' from 'newsapi

pip install newsapi-python

from newsapi import NewsApiClient

its suppost to allow me to all newsapiclient into other scripts but insted it says

line 4, in
from newsapi import NewsApiClient
ImportError: cannot import name 'NewsApiClient' from 'newsapi' (C:\Users"my name"\AppData\Local\Programs\Python\Python311\Lib\site-packages\newsapi_init_.py

is_valid_string is too restrictive

Describe the bug
The API is able to handle multiple search terms or phrases if a list of strings is passed as the q arg.

e.g. if q = [google, "google ai"] is passed and parsed to the payload dict then it will return a successful result. However, the input check won't currently allow this, limiting functionality.

Pagesize is missing

Hi, and thank you very much for this API!

In the method get_everything in newsapi_client class, there is currently no parameter taking pagesize. Since it is possible through the utilization of Curl, implementing this in the Python library also, would be of great value, so you do not need to look through so many pages.

Best regards

Issue with endpoint

When I type this code:

btc_headlines = newsapi.get_everything(q="bitcoin", language="en", page_size=100,sort_by="relevancy")
btc_articles = btc_headlines["articles"]
btc_articles[0]

I get error:

TypeError: expected string or bytes-like object

Could you please advise what could be the issue? Thanks.

Note: This happens on Macbook Pro M1. On Intel Mac it works fine.

Library Partial Import Error

Describe the bug
A clear and concise description of what the bug is.

To Reproduce
Steps to reproduce the behavior. For instance:

1. conda activate venv
2. pip install newsapi_python

from newsapi import NewsApiClient
# Init
api = NewsApiClient(api_key='xxxxxxxxxxxx')
# /v2/everything
all_articles = api.get_everything(q='mars')
print(all_articles)

4. (venv) C:\Users\abc\anaconda3\envs\venv\bin>C:/Users/abc/anaconda3/envs/venv/python.exe

Expected behavior
News Data

Screenshots

d:/Projects/advisely/advisely/API/newsapi.py
Traceback (most recent call last):
  File "d:/Projects/newsoftoday/API/newsapi.py", line 3, in <module>
    from newsapi import NewsApiClient
  File "d:\Projects\newsoftoday\API\newsapi.py", line 3, in <module>
    from newsapi import NewsApiClient
ImportError: cannot import name 'NewsApiClient' from partially initialized module 'newsapi' (most likely due to a circular import) (d:\Projects\newsoftoday\API\newsapi.py)

Desktop (please complete the following information):

• OS: Windows 10
• Visual Studio Code
• Version [1.57.1]

Additional context

(venv) C:\Users\abc\anaconda3\envs\venv\bin>pip install newsapi_python   
Collecting newsapi_python
  Using cached newsapi_python-0.2.6-py2.py3-none-any.whl (7.9 kB)
Requirement already satisfied: requests<3.0.0 in c:\users\abc\anaconda3\envs\venv\lib\site-packages (from newsapi_python) (2.25.1)
Requirement already satisfied: urllib3<1.27,>=1.21.1 in c:\users\abc\anaconda3\envs\venv\lib\site-packages (from requests<3.0.0->newsapi_python) (1.26.5)
Requirement already satisfied: certifi>=2017.4.17 in c:\users\abc\anaconda3\envs\venv\lib\site-packages (from requests<3.0.0->newsapi_python) (2021.5.30)
Requirement already satisfied: idna<3,>=2.5 in c:\users\abc\anaconda3\envs\venv\lib\site-packages (from requests<3.0.0->newsapi_python) (2.10)
Requirement already satisfied: chardet<5,>=3.0.2 in c:\users\abc\anaconda3\envs\venv\lib\site-packages (from requests<3.0.0->newsapi_python) (4.0.0)
Installing collected packages: newsapi-python
Successfully installed newsapi-python-0.2.6

v2/everything - multiple keywords

Hi there,

I tried using multiple keywords but nomatter how I join them, I'm not receiving results any close to what I receive when I use the classic GogleNews Search.
My code:

from newsapi import NewsApiClient
api = NewsApiClient(api_key='xxx')
keywords = ['Neubau', 'Weiße', 'Stadt']
all_articles = api.get_everything(q=','.join(keywords),
                                  sort_by='publishedAt',
                                  language='de')

besides ',' I tried joining the keywords by:

• '&'
• '%20'
• '%2C'

Is there something I miss?

Proxy issue - add option to use custom session

Custom session for proxies should be added for enhancement.
Will make a PR.

ImportError: cannot import name 'NewsApiClient'

Unresolved reference 'NewsApiClient' when attempting to import the client

Syntax Error

Hello,

just installed the package and got the following error:

  File "<stdin>", line 1, in <module>
  File "/anaconda3/envs/ff/lib/python3.6/site-packages/newsapi/__init__.py", line 1, in <module>
    from newsapi.newsapi_client import NewsApiClient
  File "/anaconda3/envs/ff/lib/python3.6/site-packages/newsapi/newsapi_client.py", line 12
    def get_top_headlines(self, q=None: str, sources=None: str, language='en': str, country=None: str, category=None: str, page_size=20: int,
                                      ^
SyntaxError: invalid syntax

I can see the change coming in from this PR.

Do I need a specific version of something or am I missing something else?

Thanks in advance.

EDIT:

Just tested it with version 0.2.3 and I can confirm that it's working. So presumably the whole thing blowed up 2 hours ago, when the version was bumped. 👼

ImportError: cannot import name 'NewsApiClient' from 'newsapi'

I install with newsapi with pip command but it gives an Import error.
what is the solution for this??
One more::I tried to install this newsapi with anaconda promp
command::"conda install -c kvedala newsapi-python" also it gives and error.

Bug report

---

TypeError Traceback (most recent call last)
in
----> 1 all_bitcoin_articles=newsapi.get_everything(q='bitcoin')

TypeError: expected string or bytes-like object

The base example get_top_headlines() doesn't work.

raise ValueError("cannot mix country/category param with sources param.")

maximumResultsReached - get_everything method

I have been trying to get all the news article related to say money laundering using the get_everything method, but after a certain number of results have been accessed, it gives up this error.

Note: I have a paid account, not a developers account. I can access a max of 9900 articles only when let's say the number of articles related to money laundering is 50K. How can I access the remaining articles?

str and unicode

In order to work properly with parameters passed as unicode strings, the correct way to check if a parameter is a string in Python 2.x in newsapi_client.py is:

if isinstance(q, basestring)

instead of:

if type(q) == str

basestring includes both str and unicode.

See python docs for reference.

Issue with Page

How do I get all the results as opposed to just 20? When I retrieve results it says I have 1404, but I only get 20 of the articles in the articles section of the dictionary. I know it says to use the page parameter but that does not seem to rectify the issue.

Error using from_parameter

The code below errors out:

try:
    newsapi = NewsApiClient(api_key='my key')
    everything = newsapi.get_everything(q='bitcoin',from_parameter='2018-08-01',sources='us',language='en',page_size=100)
except Exception as e:
    print(e)

This is what the exception prints:

get_everything() got an unexpected keyword argument 'from_parameter'

I'm using Python3.6 on Ubuntu 16.04.

Update package on PyPI

The package on PyPI is 0.2.5, but 0.2.6 when installing from github.

The comments of param 'page_size' and 'page' in ‘newsapi_client.py’ is wrong.

Should swap the comments order about page_size and page.

Error regarding Newsapi-python get_top_headlines()

Describe the bug

When I used only sources for the given parameter, I was able to retrieve the data. However, if I used q and country or other parameters, I received 0 results. For example:

This works:

top_headlines = newsapi.get_top_headlines(sources='bbc-news, the-verge')

{'status': 'ok', 'totalResults': 26, 'articles': .......}

But this doesn't work:

top_headlines = newsapi.get_top_headlines()
top_headlines = newsapi.get_top_headlines(q='tesla', country='us')
top_headlines = newsapi.get_top_headlines(category='business')

{'status': 'ok', 'totalResults': 0, 'articles': []}

I also tried:

top_headlines = newsapi.get_top_headlines(language='en-US')

but got this error:

top_headlines = newsapi.get_top_headlines(language='en-US')
  File "myproject/venv/lib/python3.8/site-packages/newsapi/newsapi_client.py", line 120, in get_top_headlines
    raise ValueError("invalid language")
ValueError: invalid language

I replaced the en in newsapi.const.language with en-US and I can receive the data right now but the source of the data is only google-news.

Also, if I used other countries, for example:

top_headlines = newsapi.get_top_headlines(country='jp')

I received 0 results.

Category missing

Can't get top headlines for some countries (even though supported by the API)

The default language of get_top_headlines seems "en". When needed to get specific news for a country, if language is not changed article count seems 0.

Change the language get the news, could be a choice.

BUT

The problem occurs when the language is not supported by the API
top_headlines = newsapi.get_top_headlines(country="tr",language = "tr")
print(top_headlines)

Above code gives" invalid language error".
But in the API doc, specific news are supported for Turkey.
Since in the client, language has to be changed to get the news and language is not defined tor Turkey in the API, it raises an error

Also the issue exists in getting category specific news for countries

there are no data come back

hello,i have set the api key ,and using the right code,and the demo get the "{'status': 'ok', 'totalResults': 0, 'articles': []}",this is no new_data come back ,why

Trying to use qInTitle and getting error

I am trying to use qInTitle when using get_everything and am getting the following error:
get_everything() got an unexpected keyword argument 'qInTitle'

Is the required argument slightly different?

Duplicated value

newsapi-python/newsapi/newsapi_client.py

Line 148 in 6da1bd7

(str) sort_by - The order to sort the articles in. Valid values are: 'relevancy','popularity','publishedAt',

Valid values are: 'relevancy','popularity','publishedAt','relevancy'

ValueError: Timeout value connect was Timeout(connect=30, read=30, total=None), but it must be an int, float or None.

from newsapi import NewsApiClient

api = NewsApiClient(api_key='XXXXXXXXXXXXXXXXXXXXXXXX')

api.get_top_headlines(sources='bbc-news')

Traceback (most recent call last):

File "", line 1, in
api.get_top_headlines(sources='bbc-news')

File "C:\Users\NIRANJAN\Anaconda3\lib\site-packages\newsapi\newsapi_client.py", line 115, in get_top_headlines
r = requests.get(const.TOP_HEADLINES_URL, auth=self.auth, timeout=30, params=payload)

File "C:\Users\NIRANJAN\Anaconda3\lib\site-packages\requests\api.py", line 70, in get
return request('get', url, params=params, **kwargs)

File "C:\Users\NIRANJAN\Anaconda3\lib\site-packages\requests\api.py", line 56, in request
return session.request(method=method, url=url, **kwargs)

File "C:\Users\NIRANJAN\Anaconda3\lib\site-packages\requests\sessions.py", line 488, in request
resp = self.send(prep, **send_kwargs)

File "C:\Users\NIRANJAN\Anaconda3\lib\site-packages\requests\sessions.py", line 609, in send
r = adapter.send(request, **kwargs)

File "C:\Users\NIRANJAN\Anaconda3\lib\site-packages\requests\adapters.py", line 423, in send
timeout=timeout

File "C:\Users\NIRANJAN\Anaconda3\lib\site-packages\requests\packages\urllib3\connectionpool.py", line 587, in urlopen
timeout_obj = self._get_timeout(timeout)

File "C:\Users\NIRANJAN\Anaconda3\lib\site-packages\requests\packages\urllib3\connectionpool.py", line 302, in _get_timeout
return Timeout.from_float(timeout)

File "C:\Users\NIRANJAN\Anaconda3\lib\site-packages\requests\packages\urllib3\util\timeout.py", line 154, in from_float
return Timeout(read=timeout, connect=timeout)

File "C:\Users\NIRANJAN\Anaconda3\lib\site-packages\requests\packages\urllib3\util\timeout.py", line 94, in init
self._connect = self._validate_timeout(connect, 'connect')

File "C:\Users\NIRANJAN\Anaconda3\lib\site-packages\requests\packages\urllib3\util\timeout.py", line 127, in _validate_timeout
"int, float or None." % (name, value))

ValueError: Timeout value connect was Timeout(connect=30, read=30, total=None), but it must be an int, float or None.

German news broker (spiegel-online Zeit) not working

After working flawlessly for month I am not getting any news when setting source to 'spiegel-online' or 'die-zeit'. cnn or reuters is still working fine.

get_top_headlines() not working

When get_top_headlines() is called in my code, no articles are returned I simply get: {'status': 'ok', 'totalResults': 0, 'articles': []}.

New PyPi Version with exclude_domains?

Can we get a version bump on the PyPi version so we have a version out there with exclude_domains?

Thanks in advance.

Not able to get sources for the specific countries

Describe the bug
The issue is that I'm able to get sources for countries like: ae, at, be, bg, ch, cn, co, cu, ua, ru.

To Reproduce
from newsapi import NewsApiClient

newsapi = NewsApiClient(api_key="api_key")
print(newsapi.get_sources(country="ua"))

{'status': 'ok', 'sources': []}

Expected behavior
Should be able to get all sources for all countries

Best way to track search results by day

There's no problem with the code—I just don't know much programming.

So I have a list of terms that I'd like to create timeseries of, by days for about a 5-6 month period.

Hopefully, I can just change the keyword, run the script, and end up with a .csv file with just two or three columns for the date, and the # of results on that day.

Using the Dev version, which is limited to 500 requsts a day. I don't think I can get the actual dates for individual articles if I use a long period of time right? So I guess I'll have to do each day separately.

Can this be done with loops to print the total number of articles per day? And would anyone be willing to help me out (please?)? Would be very much appreciated :)

Thank you!

Trying to use pageSize parameter and getting an Error

I'm trying to use the pageSize parameter and getting the following error :
get_everything() got an unexpected keyword argument 'pageSize'
Does this have something to do with the API?

Add search_in parameters

Is your feature request related to a problem? Please describe.
Please consider adding the search_in parameter.

Describe the solution you'd like
I would like to use the search_in parameter for niche lookups.

Describe alternatives you've considered
I could just filter myself after I get the results

Wrong parametername in helpfile

In client.get_everything() from parameter is from_param, but is referred to as from_parameter in the description.

ES country code missing in const.countries

I noticed that const.countries array is missing code 'es', while the API does return news sources for Spain.

newsapi.newsapi_exception.NewsAPIException

While trying to run the following code snippet , which is actually a part of documentation :

all_articles = newsapi.get_everything(q='bitcoin',
                                      sources='bbc-news,the-verge',
                                      domains='bbc.co.uk,techcrunch.com',
                                      from_param='2017-12-01',
                                      to='2017-12-12',
                                      language='en',
                                      sort_by='relevancy',
                                      page=2)

I am getting this error again and again :

Traceback (most recent call last):
  File "news.py", line 16, in <module>
    page=2)
  File "C:\Python27\lib\site-packages\newsapi\newsapi_client.py", line 261, in get_everything
    raise NewsAPIException(r.json())
newsapi.newsapi_exception.NewsAPIException

Any suggestions on why this might be happening ?

Set default language as en

Hey guys, I'm using the module to create a headline display, using some dot displays, thanks for your efforts on this code, I liked it a lot!

Now regarding my suggestion,
What about setting the default language to en, this way we can use newsapi.get_top_headlines() only, without any arguments. Currently if we do so it raises an exception.

I can send a pull request to implement it if you want.

Content is not fully returned when run get_top_headlines()

The content is returned like this:
"content": "The company operating the National Broadband Network has claimed competition from wireless services including Elon Musks Starlink is threatening the viability of its business, as retail internet prov\u2026 [+2829 chars]"
which has [+2829] at the end

To Reproduce
sample: headlines = newsapi.get_top_headlines(q='', category='business')

Expected behavior
A response is returned okay but the content of each article is not fully returned

I already imported newsapi and i got sucessfully package installed but i am getting no module error for NewsApiClient

Describe the bug
A clear and concise description of what the bug is.

To Reproduce
Steps to reproduce the behavior. For instance:

1. Go to '...'
2. Click on '....'
3. Scroll down to '....'
4. See error

Expected behavior
A clear and concise description of what you expected to happen.

Screenshots
If applicable, add screenshots to help explain your problem.

Desktop (please complete the following information):

• OS: [e.g. iOS]
• Browser [e.g. chrome, safari]
• Version [e.g. 22]

Additional context
Add any other context about the problem here.

not working

from newsapi import NewsApiClient newsapi = NewsApiClient(api_key='e6702efb133e48418f78ea26f4620e20') top_headlines = newsapi.get_top_headlines(q='bitcoin', sources='bbc-news,the-verge', category='business', language='en', country='us')
getting error in python3.x
ImportError: cannot import name 'NewsApiClient'

cant import newsapi

from newsapi import NewsAPIClient

ImportError: cannot import name 'NewsAPIClient'

I have newsapi installed and I have an api key yet it still wont work

requesting support for python 3.6 and newer

Enum field checking

Some of the fields have a limited set of strings that are acceptable, such as category, country, or language. It would probably be more user friendly if these were checked for validity before being passed to the API.

Enums would also be possible, though that could make for more work on the library user depending on how they are writing their code, and could also be a breaking change.

get full article text

Is your feature request related to a problem? Please describe.
I'd like to be able to request the full text of the article in a secondary api call.

Describe the solution you'd like
see above

Describe alternatives you've considered
n/a

Additional context
n/a

Can't mix "Country/Category" with "Sources" parameters.

The problem here is, we can't fetch articles using those two parameters (it's very useful).

Documentation typo

https://newsapi.org/docs/client-libraries/python

Following example:

# /v2/everything
all_articles = newsapi.get_everything(q='bitcoin',
                                      sources='bbc-news,the-verge',
                                      domains='bbc.co.uk,techcrunch.com',
                                      from_parameter='2017-12-01',
                                      to='2017-12-12',
                                      language='en',
                                      sortBy='relevancy',
                                      page=2)

sortBy should be sort_by

Language list contains unavaliable or invalid codes

Describe the bug
Langauge list in newsapi/const.py contains invalid language and 2 of them don't follow ISO-639-1 standard.

To Reproduce
Try to use 'se' country code with /everything endpoint, it won't give results and isn't available on NewsAPI documentation.

'cn' and 'en-US' aren't ISO-639-1 codes

Desktop (please complete the following information):

• OS: MacOS, Arch Linux
• Browser: Chrome, VSCode (Python 3.10, iso639 library)

TypeError: super(type, obj): obj must be an instance or subtype of type

When I run:
api.get_everything(q='bitcoin')

I get
api.get_everything(q='bitcoin')
Traceback (most recent call last):

File "", line 1, in
api.get_everything(q='bitcoin')

File "C:\Users\LapLap\Anaconda3\envs\fluffy\lib\site-packages\newsapi\newsapi_client.py", line 248, in get_everything
r = requests.get(const.EVERYTHING_URL, auth=self.auth, timeout=30, params=payload)

File "C:\Users\LapLap\Anaconda3\envs\fluffy\lib\site-packages\requests\api.py", line 72, in get
return request('get', url, params=params, **kwargs)

File "C:\Users\LapLap\Anaconda3\envs\fluffy\lib\site-packages\requests\api.py", line 57, in request
with sessions.Session() as session:

File "C:\Users\LapLap\Anaconda3\envs\fluffy\lib\site-packages\requests\sessions.py", line 386, in init
self.mount('https://', HTTPAdapter())

File "C:\Users\LapLap\Anaconda3\envs\fluffy\lib\site-packages\requests\adapters.py", line 120, in init
super(HTTPAdapter, self).init()

TypeError: super(type, obj): obj must be an instance or subtype of type