cyclododecene/newsfeed

Newsfeed based on GDELT Project

NewsFeed

Introduction

Newsfeed based on GDELT Project

Installation

conda create -n newsfeed python=3.7
python setup install

GDELT API

Based on the gdelt-doc-api, we consider a continuous querying mechanism by spliting the time range into multiple sub range (default setting is every 60 minutes).

• FIPS 2 letter Contries list: please check: LOOK-UP COUNTRIES
• GKG Themes list: please check: LOOK-UP THEMES

The URL encoding reference: url encode

• GDELT DOC 2.0 API
• GDELT GEO 2.0 API # BETA VERSION
• GDELT TV 2.0 API # NOT YET

GDELT Database Query

GDELT 1.0

• GDELT Events Database 1.0
• GDELT Global Knowledge Graph 1.0

GDELT 2.0

• GDELT Events Database 2.0
• GDELT Mentions Database (new)
• GDELT Global Knowledge Graph 2.0

GDELT Others

• GDELT Global Entity Graph
• GDELT Visual Global Entity Graph
• GDELT Different Graph
• GDELT Global Frontpage Graph

HOWTO

APIs

For Article query:

from newsfeed.news.apis.filters import * 
from newsfeed.news.apis.query import * 

f = Art_Filter(
    keyword = ["Exchange Rate", "World"],
    start_date = "20211231000000",
    end_date = "20211231010000",
    country = ["China", "US"]
)

articles_30 = article_search(query_filter = f, max_recursion_depth = 100, time_range = 30)
articles_60 = article_search(query_filter = f, max_recursion_depth = 100, time_range = 60)

For Timeline query:

from newsfeed.news.apis.filters import * 
from newsfeed.news.apis.query import * 

f = Art_Filter(
    keyword = ["Exchange Rate", "World"],
    start_date = "2021-12-31-00-00-00",
    end_date = "2021-12-31-01-00-00",
    country = ["China", "US"]
)
timelineraw = timeline_search(query_filter = f, max_recursion_depth = 100, query_mode = "timelinevolraw")

For GEO query:

from newsfeed.news.apis.filters import * 
from newsfeed.news.apis.query import * 

f = Art_Filter(
    keyword = ["Exchange Rate", "World"],
    country = ["China", "US"]
)
geo_7d = geo_search(query_filter = f, sourcelang="english", timespan=7)

query_mode:

• artlist: article_search
• timeline: timelinevol, timelinevolraw, timelinetone, timelinelang, timelinesourcecountry

most of the parameters are the same with gdelt-doc-api, however, to specify the precise date range, we remove the timespan and use start_date and time_range for iteratively collecting articles.

Database Query

For event database (both V1 and V2):

from newsfeed.news.db.events import *
# GDELT Event Database Version 1.0
gdelt_events_v1_events = EventV1(start_date = "2021-01-01", end_date = "2021-01-02")
results_v1_events = gdelt_events_v1_events.query()
results_v1_events_nowtime = gdelt_events_v1_events.query_nowtime()

# GDELT Event Database Version 2.0 - Event
gdelt_events_v2_events = EventV2(start_date = "2021-01-01-00-00-00", end_date = "2021-01-02-00-00-00")
results_v2_events = gdelt_events_v2_events.query()
results_v2_events_nowtime = gdelt_events_v2_events.query_nowtime()

# GDELT Event Database Version 2.0 - Mentions
gdelt_events_v2_mentions = EventV2(start_date = "2021-01-01-00-00-00", end_date = "2021-01-02-00-00-00", table = "mentions")
results_v2_mentions = gdelt_events_v2_mentions.query()
results_v2_mentions_nowtime = gdelt_events_v2_mentions.query_nowtime()

For GKG databse (both V1 and V2):

from newsfeed.news.db.gkg import *
# GDELT GKG Database Version 1.0
gdelt_events_v1_gkg = GKGV1(start_date = "2021-01-01", end_date = "2021-01-02")
results_v1_gkg = gdelt_events_v1_gkg.query()
results_v1_gkg_nowtime = gdelt_events_v1_gkg.query_nowtime()

from newsfeed.news.db.gkg import *
# GDELT GKG Database Version 2.0
gdelt_events_v2_gkg = GKGV2(start_date = "2021-01-01-00-00-00", end_date = "2021-01-02-00-00-00")
results_v2_gkg = gdelt_events_v2_gkg.query()
results_v2_gkg_nowtime = gdelt_events_v2_gkg.query_nowtime()

For GEG, VGEG and GDG:

from newsfeed.news.db.others import *
# GDELT Global Entity Graph
gdelt_v3_geg = GEG(start_date = "2020-01-01", end_date = "2020-01-02")
gdelt_v3_geg_result = gdelt_v3_geg.query()

# GDELT Visual Global Entity Graph
gdelt_v3_vgeg = VGEG(query_date = "2020-01-01", domain = "CNN")
gdelt_v3_vgeg_result = gdelt_v3_vgeg.query() 

# GDELT Global Difference Graph
gdelt_v3_gdg = GDG(query_date="2018-08-27-14-00-00")
gdelt_v3_gdg_result = gdelt_v3_gdg.query()

# GDELT Global Frontpage Graph
gdelt_v3_gfg = GFG(query_date="2018-03-02-02-00-00")
gdelt_v3_gfg_result = gdelt_v3_gfg.query()

Utilities

Full-text downloader (based on newspaper3k and Wayback Machine)

from newsfeed.utils import fulltext as ft
art = ft.download(url="https://english.news.cn/20220205/a4e93df9162e4053af64c392b5f5bfec/c.html")
print("full text: \n {}".format(art.text))

newsfeed's Issues

GEO API and TV API

TODO

• GDELT GEO 2.0 API - reconstruct
• GDELT TV 2.0 API

based on

newsfeed/newsfeed/news/apis/filters.py

Line 41 in d9116e8

class Art_Filter:

and design class GEO_Filter and class TV_Filter

need fix

issues:

1. slow in download/query, for large csv file, need chunks https://github.com/Cyclododecene/GNAF/blob/1ecedb1ed170e32e345f0bf4e5b4122285fc5fc6/GNAF/news/database/events.py#L76 https://github.com/Cyclododecene/GNAF/blob/1ecedb1ed170e32e345f0bf4e5b4122285fc5fc6/GNAF/news/database/events.py#L243
2. replace useless print with datetime https://github.com/Cyclododecene/GNAF/blob/1ecedb1ed170e32e345f0bf4e5b4122285fc5fc6/GNAF/news/apis/query.py#L126

Please can you help me? / geo_search

I'm trying to create a dataframe with Pandas, with the results of a search of

geo_search

But I get this error:

ValueError: DataFrame constructor not properly called!

The code I am using: (https://github.com/Cyclododecene/newsfeed)

from newsfeed.news.apis.filters import *

from newsfeed.news.apis.query import *

f = Art_Filter(

keyword = ["Drugs", "Cocaine", "Police"],

start_date = "2022-01-01-00-00-00",

end_date = "2022-03-01-00-00-00",

country = ["Brazil", "BR"]

)

geo_7d = geo_search(query_filter = f, sourcelang="portuguese", timespan=7)

import pandas as pd

df=pd.DataFrame(geo_7d)

df.head()

Thanks!

new feature

download the full-text via newspaper3k

Tests needed

We need some tests in

• news.apis
• news.database.events
• news.database.gkg
• news.database.others

make sure core functionalities work as intended in different network environment