longnd/elixir-gscraper

Simple Google Search scraper built on Elixir

elixir-gscraper's Issues

[FRONTEND] As a user, I can sign-out

Why

Create a Sign out UI to enable the users to sign out from the website.

Acceptance Criteria

Create a Sign out menu or button which the user can access on any pages to sign out.

[FRONTEND] As a user, I can sign-in using my email and password

Why

The users must sign-in before doing any actions the app can provide. They can do that by providing the credential through a Login form.

Acceptance Criteria

Create a login page with the following information:

• Page title: Sign In
• A login form:
  • Title: Sign In
  • Fields:
    • Email: required
    • Password: requried
    • A "Sign In" button to submit the form
• A link with the title "Sign up" to redirect the user to the Sign Up page.
• In case of invalid inputs, e.g. invalid email format, password doesn't match the required complexity or the credential isn't correct, the error message must be reported to the user.

[FRONTEND] As a user, I can see my uploaded keywords

Why

After successfully upload a CSV file, the user must be able to see the list of properly parsed terms to check what they uploaded.

Acceptance Criteria

Show the list of uploaded keywords to the user after they submit the file.

Learn Elixir & Phoenix

Why

It's essential to learn Elixir & Phoenix framework to understand their syntax and features before working on the project.

Acceptance Criteria

Complete some following courses & material:

The taste of Elixir

• Solid Ground - Saša Jurić (Youtube)
• The Soul of Erlang and Elixir - Saša Jurić (Youtube)
• Build a real-time Twitter clone in 15 minutes with LiveView and Phoenix 1.5

Elixir - Basic

• Elixir School
• Elixir Succinctly (Book)
• Official Elixir Website
• Elixir in Action (Book)
• Concurency in Elixir - How it works
• Developing with Elixir/OTP (Course)

Phoenix

• Programming Phoenix (Book)
• Unpacked: Multi-Player Bingo with Phoenix+Elm or Vue (Course)
• Phoenix LiveView (Course)

Elixir - Advanced

• Designing Elixir Systems with OTP (Book)
• Adopting Elixir (Book)

Useful resources (Optional)

• Elixir Forum
• Alchemist Camp
• Elixir documentation
• Phoenix documentation
• Hex Docs
• Awesome Elixir
• Exercism.io

Ecto

• The Little Ecto Cookbook (Book)
• Official Ecto documentation

Testing tools

• Faker (Generating fake data)
• ExMachina (Fixtures)
• Mox (Mock & stub)
• Wallaby (UI test)

Podcast (Optional)

• The Thinking Elixir Podcast
• Elixir Wizards Podcast
• Elixir Mix

Visual Studio Code extension (Optional)

• ElixirLS
• EEx snippets
• Gettext
• hex.pm IntelliSense

Setup CI/CD

Why

It's essential that a proper project setup should be done before working on any feature implementation.

Acceptance Criteria

• Setup Heroku App
• Setup Github actions CI for pull requests
• Setup Github actions CD for deployment
• Update the Readme file with proper setup instructions

[BACKEND] As a user, I can sign-up using my email and password

Why

Handle the submitted information by the user to create an account.

Acceptance Criteria

• Invalid inputs must be reported back to the user.
• Create an account with the provided information.
• Redirect the user to the home page with a message to confirm the account's successful creation: "Thank you for your registration! Your account is now ready to use."

[FRONTEND] As a user, I can upload a CSV file of keywords

Why

There must be a section on the website where the user can upload a CSV file of keywords.

Acceptance Criteria

Create a page on the website with a form to upload the CSV file of keywords:

• Page title: CSV Upload
• A form that:
  • Allow the user to select the CSV file to upload.
  • The form must have some text to indicate the CSV format it supports: both column and inline formats.
  • An "Upload" button to submit the file.
• The page must be able to report the errors to the users if it arises, e.g. invalid CSV format or invalid file type.

[BACKEND] As a user, I can see the search result statistics of each keyword

Why

For each of the CSV files the users uploaded, they can see the search result statistics of each keyword parsed from the file.

Acceptance Criteria

Depends on the chosen uploaded CSV, return the following information for each keyword:

• Number of AdWords advertisers in the top position
• Total number of AdWords advertisers on the page
• URLs of the AdWords advertisers in the top position
• Number of the non-AdWords results on the page
• Total number of links (all of them) on the page
• HTML code of the page/cache of the page.

The results must be sorted by keyword

Initialize application with Elixir template

Why

It's essential to have the application initialized before working on any feature implementations.

Acceptance Criteria

• Initialize the project using our Elixir template

[BACKEND] As a user, I can search for a keyword

Why

Accept the search term submitted by the user to search for a keyword.

Acceptance Criteria

• Update the report screen created in #18 to accept a search param
• Filter keywords with the given param

[Backend] Process the keyword searching

Why

After the keyword file is uploaded and the terms are parsed, implement the searching for each keyword and extract the search results to save to DB.

Acceptance Criteria

• The search for each keyword must start to run as soon as they are parsed and save to DB.
• If the search fails to process, retry it after a certain time.
• Extract the following information from the search result and save in the DB:
  • Number of AdWords advertisers in the top position
  • Total number of AdWords advertisers on the page
  • URLs of the AdWords advertisers in the top position
  • Number of the non-AdWords results on the page
  • Total number of links (all of them) on the page
  • HTML code of the page/cache of the page.

[API] As a user, I can upload a CSV file of keyword via a Restful API

Why

Create an API that a user can use to upload a CSV file.

Acceptance Criteria

The API must meet the following conditions:

• JSON API standard must be followed
• Only authenticated users can upload the files.
• Only accept the CSV file type.
• Only accept the files that have 1-1000 keywords.
• Accepts both column and inline CSV format.
• Reports back the errors to the user if it arises, e.g. invalid file type or invalid number of keyword
• Stores the uploaded keywords in the database.
• Triggers the search process for each of the keywords.

[FRONTEND] As a user, I can sign-up using my email and password

Why

Create a page with a form to allow the user to sign up using an email and password.

Acceptance Criteria

Create a page with the following information:

• Page title: Registration
• Registration form:
  • Title: Create your account
  • Fields:
    • email: required
    • password: required
    • confirmed password: required
  • A "Sign Up" button to submit the form.
  • Error messages: each field should be able to show the error message in case of invalid input.
• A link with the title "Already had an account? Sign in" to redirect the user to the login form if they already had an account.

[BACKEND] As a user, I can sign-in using my email and password

Why

Validate the submitted credential to login the user or report back the errors if it arises.

Acceptance Criteria

• Report back the errors to the user if it arises, e.g. invalid email, password doesn't match the complexity required or the credential doesn't work...
• If the credential is correct, login the user and redirect to the homepage.

[FRONTEND] As a user, I can search for a keyword

Why

Update the report page, created in #15 to allow the user to search for a keyword.

Acceptance Criteria

Update the report page to add a search field to allow the user to input a term to search for a keyword.

[BACKEND] As a user, I can see my uploaded keywords

Why

After handling the uploaded keyword in #10, rendering those keywords to the users so they can verify the upload.

Acceptance Criteria

• Render the properly parsed terms from the uploaded CSV file to the users so they can check what they uploaded.
• The keywords must be sorted alphabetically

[API] As a user, I can sign out via a Restful API

Why

Create an API to allow a user to sign out after authenticating through Oauth2 API - created in #19

Acceptance Criteria

Logout the user when the API is called - removed the granted access token.

[FRONTEND] As a user, I can see the pagination of the report

Why

As the number of keywords in each report could be huge (up to 1000 keywords), they should be paginated for a friendly user experience.

Acceptance Criteria

Update the report page created in #15 to add a pagination section that allows the user to navigate between pages.

[API] As a user, I can get a list of keywords along with search result statistics via a Restful API

Why

The users can upload CSV files multiple times. Each file can contain many keywords. Each upload is kept on track with an upload ID.
Create a Restful API to allow the user to get a list of keywords based on the upload ID along with the search result statistic. It means the API should return parsed terms from a CSV file along with the search statistic for each term.

Acceptance Criteria

The API must meet the following conditions:

• JSON API standard must be followed
• Only authenticated user can consume the API
• Accept an upload ID
• Return keywords belong to that upload
• Paginate the result
• Each keyword must come with the following search results statistic:
  • Number of AdWords advertisers in the top position
  • Total number of AdWords advertisers on the page
  • URLs of the AdWords advertisers in the top position
  • Number of the non-AdWords results on the page
  • Total number of links (all of them) on the page
  • HTML code of the page/cache of the page.

[API] As a user, I can authenticate via Oauth2 Restful API

Why

Create a Restful API to let the user authenticate using Oauth2

Acceptance Criteria

A restful API follows Oauth2 to authenticate the user.

[BACKEND] As a user, I can see the pagination of the report

Why

Paginate the data on the report page (created in #15) so it doesn't show all of the keywords and their statistic on one page.

Acceptance Criteria

• Paginate the results displayed on the report page
• The number of items per page should be configurable

[FRONTEND] As a user, I can see the search result statistics of each keyword

Why

Display the search result statistic for each keyword parsed from the uploaded CSV file to the user.

Acceptance Criteria

Create a report page to display the list of keywords parsed from each of the CSV files they uploaded. Each keyword must show the following search result statistics:

• Number of AdWords advertisers in the top position
• Total number of AdWords advertisers on the page
• URLs of the AdWords advertisers in the top position
• Number of the non-AdWords results on the page
• Total number of links (all of them) on the page
• HTML code of the page/cache of the page.

[BACKEND] As a user, I can upload a CSV file of keywords

Why

Handle the upload of CSV files by the users.

Acceptance Criteria

• Only authenticated users can upload the files.
• Only accept the CSV file type.
• Only accept the files that have 1-1000 keywords.
• Accept both column and inline CSV format.
• Report back the errors to the user if it arises, e.g. invalid file type or invalid number of keyword
• Store the uploaded keywords in the database.

[API] As a user, I can search for a keyword via a Restful API

Why

Create an API to let the user search for a keyword, base on a query parameter.

Acceptance Criteria

• JSON API standard must be followed
• Only authenticated user can consume the API
• The API must accept a query parameter and filter for keywords based on that filter.

[BACKEND] As a user, I can sign-out

Why

Sign out the users from the website when receiving the sign-out request submitted from the clients.

Acceptance Criteria

• Sign out the users from the website when they click on the Logout menu/button.
• Redirect the users to the Sign-in page.