dowjones/factiva-news-python

Python package to interact with Factiva news-related APIs. Services are described in the Dow Jones Developer Platform.

factiva-news-python's Issues

Taxonomy category executives kills the Python kernel

When attempting to load the executives taxonomy category into a DataFrame, it takes some minutes before the kernel dies. This is caused by the response size which is ~1GB.

t = Taxonomy()
d = t.get_category_codes('executives')

Download with low memory use

Some Snapshots create large files (a few GB) and the request operation in Python first reads the file in memory and then write it to disk. This can cause the process to be killed if all available memory is used.

It may worth to implement the download operation with low memory use by applying the pattern described in:
http://masnun.com/2016/09/18/python-using-the-requests-module-to-download-large-files-efficiently.html

v0.2.1 - ModuleNotFoundError: No module named 'factiva.news.tools'

Since v0.2.1, attempting to import factiva.news results in the following stack trace,

----> 1 import factiva.news

/databricks/python/lib/python3.8/site-packages/factiva/news/__init__.py in <module>
     14 from .stream import Listener, Stream, Subscription
     15 from .taxonomy.taxonomy import Taxonomy
---> 16 from .tools import JSONLFileHandler, BigQueryHandler, MongoDBHandler
     17 
     18 version = __version__

/databricks/python_shell/dbruntime/PythonPackageImportsInstrumentation/__init__.py in import_patch(name, globals, locals, fromlist, level)
    160             # Import the desired module. If you’re seeing this while debugging a failed import,
    161             # look at preceding stack frames for relevant error information.
--> 162             original_result = python_builtin_import(name, globals, locals, fromlist, level)
    163 
    164             is_root_import = thread_local._nest_level == 1

ModuleNotFoundError: No module named 'factiva.news.tools'

Do you know why this is happening?

Library versions:

• factiva-core-0.2.4
• factiva-news-0.2.1

consume message error - TypeError: pull() got an unexpected keyword argument 'request'

Hi there,
trying to run stream listener to consume messages in sync. got this error.
async message listener is working. trying to get messages in bulk to process them at once.

/code-envs/python/py39_dn001/lib/python3.9/site-packages/factiva/news/stream/listener.py in _pull_pubsub_messages(self, pubsub_client, pubsub_request, subscription_path, callback, ack_enabled)
    225 
    226         """
--> 227         pubsub_messages = pubsub_client.pull(request=pubsub_request)
    228         if pubsub_messages and pubsub_messages.received_messages:
    229             for message in pubsub_messages.received_messages:

/code-envs/python/py39_dn001/lib/python3.9/site-packages/google/cloud/pubsub_v1/_gapic.py in <lambda>(self, *a, **kw)
     38             return staticmethod(functools.wraps(wrapped_fx)(fx))
     39         else:
---> 40             fx = lambda self, *a, **kw: wrapped_fx(self.api, *a, **kw)  # noqa
     41             return functools.wraps(wrapped_fx)(fx)
     42 

TypeError: pull() got an unexpected keyword argument 'request'

File download progress

In interactive scenarios it'd be nice to have a log that shows the download progress, both in terms of the total files to be downloaded and in terms of the percentage of the current file.