data-for-good-concepts/serverless-scraping-service

A concept implementation of a serverless scraping web service

serverless-scraping-service's Issues

Setup message queue for incoming job requests

Provide option to append `CODE LIST` to response

Setup pre-commit-hook for `renv::snapshot()`

Setup different environments in order to test deployment

Setup domain restricted sharing (DRS) for public web service

Summary

Deploy public web service without unintentionally granting permissions to Google Cloud projects to external users.

Intended Outcome

Only services with tag allUserIngress:True are allowed unauthenticated requests.

How will it work?

Using Resource Manager tags and a conditional DRS policy in order to manage invocations. Example can be found here.

Write onboarding documentation

Write extended documentation in order to enable simple reproducibility. Cover at least following topics:

• Docker
• GitHub Actions
• gcloud CL
• Google Cloud Service Account
• Google Cloud Build
• Google Cloud Run
• Service account creation + permissions
• Enable APIs (Cloud Build, Cloud Run)
• Include firefox as dependency

Read request body options when creating scraping job

• Use post request payload in order enable flexible customization of datasets.
• Re-evaluate currently hardcoded values and parameters and replace them with meaningful options.
• Make sure download is clicked only after table is rendered
• Provide download options
• Complete documentation

Provide extended swagger documentation

Setup static scraping workflow

Summary

Scrape one particular dataset from Eurostat.

Intended Outcome

Service returns JSON response with scraped data. Outcome is currently fixed and can not be adjusted by parameters.

How will it work?

By making a post request to the service, scraper navigates to specified data, reads values and sends back a JSON response, containing the data.

Bootstrap web service skeleton

Summary

A basic web service that can handle and respond to API requests.

Intended Outcome

A bare bone web service that will respond to a GET request to localhost as well as to a public URL (Google Cloud Run).

How will it work?

Bootstrap web service using renv, plumbr and Docker in a manner that it can be deployed on Google Cloud run. Users can send a get request to a /health/status endpoint and will receive a response indicating that the service is online.