ibm/watson-discovery-news-alerting

WARNING: This repository is no longer maintained :warning: This repository will not be updated. The repository will be kept available in read-only mode.

WARNING: This repository is no longer maintained ⚠️

This repository will not be updated. The repository will be kept available in read-only mode. Refer to https://github.com/IBM/watson-discovery-news for a similar example.

Watson Discovery News Alerting

In this code pattern, we will build a Node.js web application that will use the Watson Discovery Service to access Watson Discovery News.

Watson Discovery News is a default data collection that is associated with the Watson Discovery Service. It is a dataset of primarily English language news sources that is updated continuously, with approximately 300,000 new articles and blogs added daily.

The focus of this code pattern is to monitor a product's marketplace life-cycle using Watson's Discovery service to intelligently alert when a product's stance in the marketplace has changed. Users can receive periodic email alerts about a product or brand and how they're perceived in the News.

Alert tracking can be set up for the following areas:

• The product
• The brand
• Related products and brands
• Positive or negative product sentiment
• Stock prices

The code pattern highlights the steps required to build a front-end management interface to search Watson News and a back-end service which periodically sends alerts out related to customizable queries.

Flow

1. The user interacts with the backend server via the app UI. The frontend app UI uses React to render search results and can reuse all of the views that are used by the backend for server side rendering. The frontend is using watson-react-components and is responsive.
2. User input is processed and routed to the backend server, which is responsible for server side rendering of the views to be displayed on the browser. The backend server is written using express and uses express-react-views engine to render views written using React.
3. The backend server stores subscription information in a Cloudant NonSQL database for product tracking.
4. The backend server sends user requests to the Watson Discovery Service. It acts as a proxy server, forwarding queries from the frontend to the Watson Discovery Service API while keeping sensitive API keys concealed from the user.
5. The Watson Discovery Service queries the Watson News Collection for articles related to the product.
6. The backend server sends periodic updates to email.

Included components

• Watson Discovery: A cognitive search and content analytics engine for applications to identify patterns, trends, and actionable insights.
• Cloudant NoSQL DB for IBM Cloud: A fully managed data layer designed for modern web and mobile applications that leverages a flexible JSON schema.

Featured technologies

• Node.js - An asynchronous event driven JavaScript runtime, designed to build scalable applications
• React - Javascript library for building User Interfaces
• Express - A popular and minimalistic web framework for creating API and Web server
• Yarn - Fast, reliable and secure dependency manager for node.js

Watch the Video

Steps

1. Clone the repo
2. Create Watson Services with IBM Cloud

1. Clone the repo

Clone the watson-discovery-news-alerting locally. In a terminal, run:

git clone https://github.com/ibm/watson-discovery-news-alerting

2. Create Watson Services with IBM Cloud

Create the following service:

• Watson Discovery - name the service wdna-discovery
• Cloudant NoSQL DB - name the service wdna-cloudant

Run the application locally

1. Install Node.js and Yarn
2. Install all of the dependencies by running yarn. This will install of the node modules specified in package.json

cd app
yarn

3. Copy the env.sample to .env

cp .env.sample .env

4. Edit the .env file and enter your Watson Discovery and Cloudant NoSQL DB credentials. Add SMTP Mail settings if you wish to use the tracking feature.

5. Build and start the main app.

yarn run build
yarn start

6. Build and start the tracking app.

yarn run start-notifier

7. Open a browser and go to http://localhost:4391

Deploy and run the application on IBM Cloud

To deploy to IBM Cloud make sure you have the IBM Cloud CLI tool installed. Then run the following commands to connect it with IBM Cloud and login with your IBM Cloud credentials.

cd watson-discovery-news-alerting
ibmcloud login

Run the following command to deploy and run the alerting and notifier apps on the IBM Cloud. It will automatically connect to your existing Watson Discovery and Cloudant NoSQL DB services if your service names match the names specified in the manifest.yml file.

ibmcloud cf push

If the ibmcloud cf push command complains that the application name is already taken, change the lines in the manifest.yml to have a custom application name specific for your setup:

...
applications:
- name: custom-name
  path: ./app
...

Two IBM Cloud applications should be created and running:

• watson-discovery-news-alerting
• watson-discovery-news-alerting-notifier

Set the environment variables required for each the notifier service to perform properly. Use the values unique to your setup:

ibmcloud cf set-env watson-discovery-news-alert-notifier SMTP_SETTINGS '{"host":"smtp.gmail.com","user":"[email protected]","pass":"xxx","fromEmail":"[email protected]"}'
ibmcloud cf set-env watson-discovery-news-alert-notifier BASE_URL 'https://watson-discovery-news-alerting.mybluemix.net'

Go to the URL route that is associated with the watson-discovery-news-alerting app in IBM Cloud to view the application. Typically, this would be https://watson-discovery-news-alerting.mybluemix.net.

Sample output

General Project Layout

The server which hosts the React web application, acts as an API to Watson, and communicates with the notifier App can be found at:

./app/server.js

The server which periodically emails news alerts to subscribed users can be found at:

./app/notifier.js

Architecture

Back-end Server

Handles hosting of the static assets (React front-end application) and manages a thin API used by the front-end application.

Front-end Application

Displays results from querying the Watson Discovery Service API and manages subscriptions for push updates.

Worker Application

Background process which periodically sends updates to email.

Troubleshooting

• Setting environment variables for a local run in .env config file

The credentials for IBM Cloud services (wdna-discovery and wdna-cloudant) can be found in the Services menu in IBM Cloud, and selecting the Service Credentials option.

• Ensure port is not already in use

If the port is unavailable, you will see the following error:

Error: listen EADDRINUSE :::{port}

Links

• Demo on Youtube: Watch the video.
• Watson Node.js SDK: Download the Watson Node SDK.
• Watson News Queries Video: Watch how we created the queries for this code pattern.
• Cognitive discovery architecture: Learn how this Code Pattern fits into the Cognitive discovery Reference Architecture.
• Watson Discovery Service Updates for News and NLP: Learn what enhancements are being made to the service.
• Watson Discovery Demo: Unlock hidden value in data to find answers, monitor trends and surface patterns.
• Watson Discovery News: Explore the Watson Discovery News documentation.
• Create a cognitive news search app: Follow along with a news search and trends web app journey.
• Watson Discovery and Exploration blog-1: Out of the box, Discovery provides a pre-enriched collection of 2 months of internet news content.
• Watson Discovery and Exploration blog-2: Find out how enriched and faster news discovery fuels business development and growth.
• Watson Discovery configurataion: Understanding the difference between entities, concepts, and keywords.
• Recharts: A composable charting library built on React components.
• Ngrok: Establish secure tunnels from a public endpoint to a local host.
• Jest: A JavaScript testing framework.

Learn more

• Artificial Intelligence Code Patterns: Enjoyed this Code Pattern? Check out our other AI Code Patterns.
• AI and Data Code Pattern Playlist: Bookmark our playlist with all of our Code Pattern videos
• With Watson: Want to take your Watson app to the next level? Looking to utilize Watson Brand assets? Join the With Watson program to leverage exclusive brand, marketing, and tech resources to amplify and accelerate your Watson embedded commercial solution.

License

This code pattern is licensed under the Apache Software License, Version 2. Separate third party code objects invoked within this code pattern are licensed by their respective providers pursuant to their own separate licenses. Contributions are subject to the Developer Certificate of Origin, Version 1.1 (DCO) and the Apache Software License, Version 2.

Apache Software License (ASL) FAQ

watson-discovery-news-alerting's Issues

remove reference to "deploy to bluemix"

just before the steps, there's language that indicates that a D2B button exists

nano.db is undefined

When I run "yarn start", I've got the following error.
yarn run v1.15.2

credentials.iam_apikey: xxx
credentials.iam_username: xxx
/Users/ayatokura/ibmdojo/dxchallenge/watson-discovery-news-alerting/app/node_modules/@cloudant/cloudant/cloudant.js:164
  nano.use = nano.db.use = use;
                         ^
TypeError: Cannot set property 'use' of undefined
    at new Cloudant (/Users/ayatokura/ibmdojo/dxchallenge/watson-discovery-news-alerting/app/node_modules/@cloudant/cloudant/cloudant.js:164:26)
    at getCloudantService (/Users/ayatokura/ibmdojo/dxchallenge/watson-discovery-news-alerting/app/src/bluemix/config.js:50:20)
    at Object.<anonymous> (/Users/ayatokura/ibmdojo/dxchallenge/watson-discovery-news-alerting/app/src/models/access.js:22:18)
    at Module._compile (internal/modules/cjs/loader.js:776:30)
    at loader (/Users/ayatokura/ibmdojo/dxchallenge/watson-discovery-news-alerting/app/node_modules/babel-register/lib/node.js:144:5)
    at Object.require.extensions.(anonymous function) [as .js] (/Users/ayatokura/ibmdojo/dxchallenge/watson-discovery-news-alerting/app/node_modules/babel-register/lib/node.js:154:7)
    at Module.load (internal/modules/cjs/loader.js:653:32)
    at tryModuleLoad (internal/modules/cjs/loader.js:593:12)
    at Function.Module._load (internal/modules/cjs/loader.js:585:3)
    at Module.require (internal/modules/cjs/loader.js:690:17)
error Command failed with exit code 1.

change color scheme of webapp

joe scheping of the watson discovery team asked if we could change the current purple color to something else (blue?), to differentiate this app from the demo the discovery team created: https://discovery-news-demo.mybluemix.net/

yarn run v1.9.4 throws errors

Hi,

I'm from Cloudant Support and have a case from the user. The following problem has reported by the user and need your help.

"I did add my ibmcloud login info in the .env file name: pass: for login

yarn run start-notifier
yarn run v1.9.4
$ yarn run build-notifier && ./node_modules/babel-cli/bin/babel-node.js ./notifier.js
$ ./node_modules/babel-cli/bin/babel-node.js ./src/buildNotifierDB.js
credentials.username: undefined
credentials.password: undefined
/Users/jeffswope/watson-discovery-news-alerting/app/node_modules/@cloudant/cloudant/cloudant.js:42
throw err;
^

Error: Invalid URL
at Cloudant (/Users/jeffswope/watson-discovery-news-alerting/app/node_modules/@cloudant/cloudant/cloudant.js:38:15)
at getCloudantService (/Users/jeffswope/watson-discovery-news-alerting/app/src/bluemix/config.js:64:16)
at Object. (/Users/jeffswope/watson-discovery-news-alerting/app/src/models/access.js:22:18)
at Module._compile (internal/modules/cjs/loader.js:689:30)
at loader (/Users/jeffswope/watson-discovery-news-alerting/app/node_modules/babel-register/lib/node.js:144:5)
at Object.require.extensions.(anonymous function) [as .js] (/Users/jeffswope/watson-discovery-news-alerting/app/node_modules/babel-register/lib/node.js:154:7)
at Module.load (internal/modules/cjs/loader.js:599:32)
at tryModuleLoad (internal/modules/cjs/loader.js:538:12)
at Function.Module._load (internal/modules/cjs/loader.js:530:3)
at Module.require (internal/modules/cjs/loader.js:637:17)
error Command failed with exit code 1.
info Visit https://yarnpkg.com/en/docs/"

Thanks,
Cloudant Support

old todo items

I'm deleting the TODO.md. We'll use issues for tracking now. So here's an issue to capture what was left in TODO.md as-is:

TODO Items

• Rename the Example and ExampleList component to be Alert, the name is hard to understand and has become outdated with more recent code.
• Fix the React and general Node tests.
• Rename the Tracking component to be Subscription.
• [q] Random taxonomies in results (like Disney in the /auto taxonomy), should taxonomies be used?
• [a] Check the taxonomies relevance in the output.

Watson Discovery News Data Dump

Is it possible to query discovery news for all historically added news (download data dump)? I know it would be possible to do with an API call (https://gateway.watsonplatform.net/discovery/api/v1/environments/system/collections/news/query), but with limitations of 50 results per query that'd be very expensive.