stangirard/seo-audits-toolkit

SEO & Security Audit for Websites. Lighthouse & Security Headers crawler, Sitemap/Keywords/Images Extractor, Summarizer, etc ...

Open source Audits Toolkit

OSAT is a collection of tools created help you in your quest for a better website. All of these tools have been grouped into a single web app.

I've grown tired of SEO agencies making us pay hundreds of euros for simple tools. I decided to develop OSAT to help users find issues on their website and increase their SEO for free.

Why you need it

• It's free, easy and open source.
• It has a growing list of features
• It's easy to install

Features

• Authentification - A fully featured authentification system for the front & back
• RBAC/Organizations - Create different organizations and give different access to each org to your users.
• Lighthouse Score - Run Lighthouse Audits and keep track of your scores
• SERP Rank - Get the rank of your website on google for specific queries
• Keywords Finder - Find all the keywords of an article.
• Extract Headers/Links/Images - Easily extract all the links on your website and their status codes, the headers of a page and all the images.
• Sitemap Extractor - Extract all the urls of a website from its sitemap
• Summarizer - Summarize any text from any length. Awesome for excerpt !
• Security Audit - Audit Headers, Redirect, etc to make sure you website is secure.

Demo

Go to demo.primates.dev
Login: demo
Password: demodemo

Admin part is api.primates.dev/admin

Installation

git clone https://github.com/StanGirard/seo-audits-toolkit
cd seo-audits-toolkit
docker-compose pull ## docker-compose build if you want to build it on your own
docker-compose --env-file .env-example up -d

Init the project

docker exec -it osat-server python manage.py flush
docker exec -it osat-server python manage.py loaddata init_data.json

Dashboard

You can access the dashboard by going to localhost:3000

Login: admin Password: admin

Config

If needed create a .env file with information that you would like to change

Configuration

Create organization

You need to go to Org -> Organization and create a new organization. You can create as many as you want. Organization are used in order to implement RBAC in the project and only display information about an organization to users of this organization. Here is a quicklinkg to access it http://localhost:8000/admin/org/website/

Add user to organization

Once your organization is created. You need to add your users to this organization. Go to Organizations -> Organizations Users and add your users to the organization you want. http://localhost:8000/admin/organizations/organizationuser/

Useful Links

• Webapp http://localhost:3000
• Admin Dashboard http://localhost:8000/admin
• Swagger like interface http://localhost:8000

Contributions

Please feel free to add any contribution. If you've been working on a script that could be integrated in this app. Please feel free to do it. Don't hesitate to open issues to ask questions. I've tried to document the code as much as I could to ease the integration

Backend

You can just add a django module and I'll take care of intregrating it in the front.

Frontend

I've used React Admin to build the front-end. If you want to help me improve the UI or add new functionnalites. Please feel free to contribute.

Disclaimers

I'm not a python nor a frontend developer.

seo-audits-toolkit's Issues

Rough install...great results!

This really needs better documentation, but in any event I got it working and it's great to have everything ready so quickly and without BS paywalls, precisely as the dev intended to achieve. Great work!

error 500, KeyError: 'results' at /audit/lighthouse/score endpoint

Hi @StanGirard ,

Hope you are all well !

I managed to deploy seo-audits-toolkit via docker and I have some errors.

For example, /audit/lighthouse/score gives an 500 error:

Or, the serp rank or graphs are running endlessly.

Any idea how to fix it ?

Thanks for any insights or inputs on that.

Ps. From my point of view, there is a lot of things we can add to your project and is there a way to DM you on twitter about my ideas (I speak french too). My twitter handle is https://twitter.com/lucmichalski

Cheers,
Luc Michalski

Lighthouse wartning and service beats failed issues

Hi guys,

I'm unable to install the tool due to following errors:

warning lighthouse > raven > [email protected]: Please upgrade  to version 7 or higher.  Older versions may use Math.random() in certain circumstances, which is known to be problematic.  See https://v8.dev/blog/math-random for details.
warning lighthouse > intl-messageformat > [email protected]: We've written a new parser that's 6x faster and is backwards compatible. Please use @formatjs/icu-messageformat-parser

error [email protected]: The engine "node" is incompatible with this module. Expected version ">=14.15". Got "12.22.10"
error Found incompatible module.

info Visit https://yarnpkg.com/en/docs/cli/global for documentation about this command.
The command '/bin/sh -c yarn global add lighthouse' returned a non-zero code: 1
ERROR: Service 'beats' failed to build : Build failed

I've followed all the steps except:
docker-compose pull ## docker-compose build if you want to build it on your own

Could not load contenttypes.ContentType(pk=19): duplicate key value violates unique constraint

Hi
Steps to reproduce :

1. git clone https://github.com/HT-Moh/seo-audits-toolkit.git
2. mv .env-example .env
3. docker-compose build
4. docker network create web
5. docker-compose up
6. docker exec -it osat-server python manage.py flush
7. docker exec -it osat-server python manage.py loaddata init_data.json

Traceback (most recent call last):
  File "manage.py", line 22, in <module>
    main()
  File "manage.py", line 18, in main
    execute_from_command_line(sys.argv)
  File "/opt/venv/lib/python3.8/site-packages/django/core/management/__init__.py", line 401, in execute_from_command_line
    utility.execute()
  File "/opt/venv/lib/python3.8/site-packages/django/core/management/__init__.py", line 395, in execute
    self.fetch_command(subcommand).run_from_argv(self.argv)
  File "/opt/venv/lib/python3.8/site-packages/django/core/management/base.py", line 330, in run_from_argv
    self.execute(*args, **cmd_options)
  File "/opt/venv/lib/python3.8/site-packages/django/core/management/base.py", line 371, in execute
    output = self.handle(*args, **options)
  File "/opt/venv/lib/python3.8/site-packages/django/core/management/commands/loaddata.py", line 72, in handle
    self.loaddata(fixture_labels)
  File "/opt/venv/lib/python3.8/site-packages/django/core/management/commands/loaddata.py", line 114, in loaddata
    self.load_label(fixture_label)
  File "/opt/venv/lib/python3.8/site-packages/django/core/management/commands/loaddata.py", line 181, in load_label
    obj.save(using=self.using)
  File "/opt/venv/lib/python3.8/site-packages/django/core/serializers/base.py", line 223, in save
    models.Model.save_base(self.object, using=using, raw=True, **kwargs)
  File "/opt/venv/lib/python3.8/site-packages/django/db/models/base.py", line 790, in save_base
    updated = self._save_table(
  File "/opt/venv/lib/python3.8/site-packages/django/db/models/base.py", line 872, in _save_table
    updated = self._do_update(base_qs, using, pk_val, values, update_fields,
  File "/opt/venv/lib/python3.8/site-packages/django/db/models/base.py", line 926, in _do_update
    return filtered._update(values) > 0
  File "/opt/venv/lib/python3.8/site-packages/django/db/models/query.py", line 803, in _update
    return query.get_compiler(self.db).execute_sql(CURSOR)
  File "/opt/venv/lib/python3.8/site-packages/django/db/models/sql/compiler.py", line 1522, in execute_sql
    cursor = super().execute_sql(result_type)
  File "/opt/venv/lib/python3.8/site-packages/django/db/models/sql/compiler.py", line 1156, in execute_sql
    cursor.execute(sql, params)
  File "/opt/venv/lib/python3.8/site-packages/django/db/backends/utils.py", line 98, in execute
    return super().execute(sql, params)
  File "/opt/venv/lib/python3.8/site-packages/django/db/backends/utils.py", line 66, in execute
    return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
  File "/opt/venv/lib/python3.8/site-packages/django/db/backends/utils.py", line 75, in _execute_with_wrappers
    return executor(sql, params, many, context)
  File "/opt/venv/lib/python3.8/site-packages/django/db/backends/utils.py", line 84, in _execute
    return self.cursor.execute(sql, params)
  File "/opt/venv/lib/python3.8/site-packages/django/db/utils.py", line 90, in __exit__
    raise dj_exc_value.with_traceback(traceback) from exc_value
  File "/opt/venv/lib/python3.8/site-packages/django/db/backends/utils.py", line 84, in _execute
    return self.cursor.execute(sql, params)
django.db.utils.IntegrityError: Problem installing fixture '/opt/app/init_data.json': Could not load contenttypes.ContentType(pk=19): duplicate key value violates unique constraint "django_content_type_app_label_model_76bd3d3b_uniq"
DETAIL:  Key (app_label, model)=(django_celery_beat, solarschedule) already exists.

Cannot login to remote OSAT hosted on dev server

Following step by step tutorial I will always get the network error. Logs are empty (there is no error logged on dashboard/server).

I have tried adding new user but it is the same issue.

I have tried to add:

`# General
NAMESPACE=sas
SERVER_PORT=5301
FRONT_PORT=5302
JSON_URL=""
SERVER_URL=http://IP:5301
FRONTEND_URL=http://IP:5302
SCAN_DEFAULT_MAX_REQUESTS=0
YAKE_SERVER_URL=http://yake:5000/yake/

PUPPETEER_SKIP_CHROMIUM_DOWNLOAD=true`

With the IP of the server, but it doesn't work either.

How can I setup it to access it remotely ???

Error when running docker-compose

Hello,
I tried to install the app as described in the README, but got the following error:

 => ERROR [15/24] RUN yarn global add lighthouse                                                                                                           18.3s
------
 > [15/24] RUN yarn global add lighthouse:
1.472 yarn global v1.22.19
1.583 [1/4] Resolving packages...
4.343 [2/4] Fetching packages...
17.90 error [email protected]: The engine "node" is incompatible with this module. Expected version ">=18.16". Got "18.13.0"
17.94 error Found incompatible module.
17.94 info Visit https://yarnpkg.com/en/docs/cli/global for documentation about this command.
------
Dockerfile:22
--------------------
  20 |     RUN apt install -y chromium
  21 |     # Install Lighthouse
  22 | >>> RUN yarn global add lighthouse
  23 |
  24 |     # Install Pipenv
--------------------
ERROR: failed to solve: process "/bin/sh -c yarn global add lighthouse" did not complete successfully: exit code: 1
ERROR: Service 'beats' failed to build : Build failed

My software versions are following:

root@ubuntu:/opt/seo-audits-toolkit# yarn --version
1.22.21
root@ubuntu:/opt/seo-audits-toolkit# npm --version
10.1.0
root@ubuntu:/opt/seo-audits-toolkit# nodejs --version
v20.9.0

When I run yarn install on it's own, there are no errors shown:

root@ubuntu:/opt/seo-audits-toolkit# yarn install
yarn install v1.22.21
[1/4] Resolving packages...
success Already up-to-date.
Done in 0.13s.

Do you have any idea what would may be causing the trouble here?

Serp Rank function not found in project

As mentioned in project features list it must be possible to find the rank of a website on google result page

• SERP Rank - Get the rank of your website on google for specific queries

But I could not find any option in UI to do Serp rank.

Where should I go?

network error on login page. Firefox.

I have since gotten then dashboard to show by using either edge or chrome, after find the reason in the dev console for Firefox:

Cross-Origin Request Blocked: The Same Origin Policy disallows reading the remote resource at http://localhost:8000/dj-rest-auth/login/. (Reason: CORS request did not succeed). Status code: (null).

Node Version Update / Server Dockerfile

Build stops at Lighthouse Step 15 with Node v12. I updated the Dockerfile to pull Node v14 and was able to proceed, but may need to update your dependencies here.

Step 15/34 : RUN yarn global add lighthouse
---> Running in 8b4b96307626
yarn global v1.22.17
[1/4] Resolving packages...
warning lighthouse > intl-messageformat > [email protected]: We've written a new parser that's 6x faster and is backwards compatible. Please use @formatjs/icu-messageformat-parser
warning lighthouse > raven > [email protected]: Please upgrade to version 7 or higher. Older versions may use Math.random() in certain circumstances, which is known to be problematic. See https://v8.dev/blog/math-random for details.
[2/4] Fetching packages...
[3/4] Linking dependencies...
[4/4] Building fresh packages...
success Installed "[email protected]" with binaries:
- lighthouse
- chrome-debug
- smokehouse
Done in 7.71s.
Removing intermediate container 8b4b96307626

New plugin

Can you add to this project one new plugin - any Feedreader?

Internal Server Error

Hi,
I get Internal Server Errors whenever I try to generate a report (Lighthouse or other) from a site, which is hosted on localhost.
What can be the reason?

docker compose file issues

issue 1= /docker-compose.yml" is unsupported. You might be seeing this error because you're using the wrong Compose file version. Either specify a supported version (e.g "2.2" or "3.3") and place your service definitions under the services key, or omit the version key and place your service definitions at the root of the file to use version 1.

issue 2= WARN[0000] The "TINI_VERSION" variable is not set. Defaulting to a blank string.
WARN[0000] The "POSTGRES_PASSWORD" variable is not set. Defaulting to a blank string.
WARN[0000] The "DEBUG" variable is not set. Defaulting to a blank string.
WARN[0000] The "ALLOWED_HOSTS" variable is not set. Defaulting to a blank string.
WARN[0000] The "REST_FRAMEWORK_PAGE_SIZE" variable is not set. Defaulting to a blank string.
WARN[0000] The "CELERY_RESULT_BACKEND" variable is not set. Defaulting to a blank string.
WARN[0000] The "DATABASE_ENGINE" variable is not set. Defaulting to a blank string.
WARN[0000] The "POSTGRES_USER" variable is not set. Defaulting to a blank string.
WARN[0000] The "POSTGRES_PORT" variable is not set. Defaulting to a blank string.
WARN[0000] The "POSTGRES_DB" variable is not set. Defaulting to a blank string.
WARN[0000] The "SECRET_KEY" variable is not set. Defaulting to a blank string.
WARN[0000] The "CELERY_BROKER_URL" variable is not set. Defaulting to a blank string.
WARN[0000] The "CORS_ALLOWED_ORIGINS" variable is not set. Defaulting to a blank string.
WARN[0000] The "NAMESPACE" variable is not set. Defaulting to a blank string.
WARN[0000] The "TINI_VERSION" variable is not set. Defaulting to a blank string.
WARN[0000] The "NAMESPACE" variable is not set. Defaulting to a blank string.
WARN[0000] The "CELERY_BROKER_URL" variable is not set. Defaulting to a blank string.
WARN[0000] The "DATABASE_ENGINE" variable is not set. Defaulting to a blank string.
WARN[0000] The "DEBUG" variable is not set. Defaulting to a blank string.
WARN[0000] The "CELERY_RESULT_BACKEND" variable is not set. Defaulting to a blank string.
WARN[0000] The "ALLOWED_HOSTS" variable is not set. Defaulting to a blank string.
WARN[0000] The "REST_FRAMEWORK_PAGE_SIZE" variable is not set. Defaulting to a blank string.
WARN[0000] The "POSTGRES_PORT" variable is not set. Defaulting to a blank string.
WARN[0000] The "POSTGRES_DB" variable is not set. Defaulting to a blank string.
WARN[0000] The "POSTGRES_PASSWORD" variable is not set. Defaulting to a blank string.
WARN[0000] The "POSTGRES_USER" variable is not set. Defaulting to a blank string.
WARN[0000] The "SECRET_KEY" variable is not set. Defaulting to a blank string.
WARN[0000] The "CORS_ALLOWED_ORIGINS" variable is not set. Defaulting to a blank string.
WARN[0000] The "NAMESPACE" variable is not set. Defaulting to a blank string.
WARN[0000] The "TINI_VERSION" variable is not set. Defaulting to a blank string.
WARN[0000] The "CELERY_RESULT_BACKEND" variable is not set. Defaulting to a blank string.
WARN[0000] The "POSTGRES_DB" variable is not set. Defaulting to a blank string.
WARN[0000] The "POSTGRES_PASSWORD" variable is not set. Defaulting to a blank string.
WARN[0000] The "POSTGRES_PORT" variable is not set. Defaulting to a blank string.
WARN[0000] The "POSTGRES_USER" variable is not set. Defaulting to a blank string.
WARN[0000] The "SECRET_KEY" variable is not set. Defaulting to a blank string.
WARN[0000] The "CORS_ALLOWED_ORIGINS" variable is not set. Defaulting to a blank string.
WARN[0000] The "ALLOWED_HOSTS" variable is not set. Defaulting to a blank string.
WARN[0000] The "DEBUG" variable is not set. Defaulting to a blank string.
WARN[0000] The "DATABASE_ENGINE" variable is not set. Defaulting to a blank string.
WARN[0000] The "REST_FRAMEWORK_PAGE_SIZE" variable is not set. Defaulting to a blank string.
WARN[0000] The "CELERY_BROKER_URL" variable is not set. Defaulting to a blank string.
WARN[0000] The "NAMESPACE" variable is not set. Defaulting to a blank string.
WARN[0000] The "NODE_VERSION" variable is not set. Defaulting to a blank string.
WARN[0000] The "TINI_VERSION" variable is not set. Defaulting to a blank string.
WARN[0000] The "POSTGRES_USER" variable is not set. Defaulting to a blank string.
WARN[0000] The "POSTGRES_PASSWORD" variable is not set. Defaulting to a blank string.
WARN[0000] The "POSTGRES_DB" variable is not set. Defaulting to a blank string.
WARN[0000] The "NAMESPACE" variable is not set. Defaulting to a blank string.
WARN[0000] The "REDIS_VERSION" variable is not set. Defaulting to a blank string.
WARN[0000] The "REDIS_LOG_PREFIX_PATH" variable is not set. Defaulting to a blank string.
WARN[0000] The "NAMESPACE" variable is not set. Defaulting to a blank string.
WARN[0000] The "REDIS_LOG_PREFIX_PATH" variable is not set. Defaulting to a blank string.
WARN[0000] The "REDIS_MEMORY_MAX" variable is not set. Defaulting to a blank string.
WARN[0000] The "REDIS_LOG_FILE" variable is not set. Defaulting to a blank string.
WARN[0000] The "PWD" variable is not set. Defaulting to a blank string.
WARN[0000] The "PWD" variable is not set. Defaulting to a blank string.
WARN[0000] The "PWD" variable is not set. Defaulting to a blank string.

Request Feature

Please use this issue to request new features

ADD path prefix variable in config for traefik intergration

Hi guys,

Hope you are all well !

Is it possible to add a prefix path variable to flask in order to ease the integration in traefik ?

In fact, I d like to access seo-audits-toolkit from /devops/seo-toolkit path

here is the traefik label config:

    - "traefik.http.routers.seo-toolkit.rule=Host(`xxxx-xxxx.xxx-xxxxx.com`) && PathPrefix(`/devops/seo-toolkit`)"

here is an excerpt of my docker-compose:

  seo-toolkit:
    image: lucmichalski/seo-audits-toolkit:latest
    container_name: seo-audits-toolkit
    build: 
      context: .docker/seo-audits-toolkit
      dockerfile: Dockerfile
    expose:
    - 5000
    depends_on:
    - redis
    - postgres
    - seo-celery
    networks:
    - internal
    - web
    environment:
    - SECRET_KEY=changeme
    - URL_APP=http://localhost:5000/
    - FLASK_APP=SEOToolkit
    - FLASK_ENV=development
    - GOOGLE_API_KEY=None
    - CELERY_BROKER_URL=redis://redis:6379/0
    - CELERY_RESULT_BACKEND=redis://redis:6379/0
    - SQLALCHEMY_DATABASE_URI=postgresql+psycopg2://unicorn_user:magical_password@postgres:5432/rainbow_database
    labels:
    - "org.label-schema.group=seo-toolkits"
    - "traefik.enable=true"
    - "traefik.docker.network=web"
    - "traefik.http.routers.seo-toolkit.rule=Host(`para-test.gsi-network.com`) && PathPrefix(`/devops/seo-toolkit`)"
    - 'traefik.http.routers.seo-toolkit.tls=true'
    - 'traefik.http.routers.seo-toolkit.tls.certresolver=letsencrypt'
    - "traefik.http.routers.seo-toolkit.entrypoints=https"
    - "traefik.http.services.seo-toolkit.loadbalancer.server.port=5000"  
    - "traefik.http.middlewares.seo-toolkit.compress=true"
    - "traefik.http.routers.seo-toolkit.middlewares=seo-toolkit-replacepathregex,seo-toolkit-basicauth" 
    - "traefik.http.middlewares.seo-toolkit-replacepathregex.replacepathregex.regex=^/devops/seo-toolkit(.*)"
    - "traefik.http.middlewares.seo-toolkit-replacepathregex.replacepathregex.replacement=/devops/seo-toolkit$$1"
    - 'traefik.http.middlewares.seo-toolkit-basicauth.basicauth.removeheader=true'
    - "traefik.http.middlewares.seo-toolkit-basicauth.basicauth.realm=EG-Seo Audit Toolkits"
    - "traefik.http.middlewares.seo-toolkit-basicauth.basicauth.users=${USER_PASSWORD}"
    restart: unless-stopped

  seo-celery:
    build: 
      context: .docker/seo-audits-toolkit
      dockerfile: Dockerfile
    networks:
    - internal
    - web
    depends_on:
    - redis
    environment:
    - SECRET_KEY=changeme
    - URL_APP=http://localhost:5000/
    - FLASK_APP=SEOToolkit
    - FLASK_ENV=development
    - GOOGLE_API_KEY=None
    - CELERY_BROKER_URL=redis://redis:6379/0
    - CELERY_RESULT_BACKEND=redis://redis:6379/0
    - SQLALCHEMY_DATABASE_URI=postgresql+psycopg2://unicorn_user:magical_password@postgres:5432/rainbow_database
    labels:
    - "traefik.enable=false"
    - "org.label-schema.group=seo-audits"
    command: "celery worker -A celery_worker.celery --loglevel=info"
    restart: unless-stopped

  postgres:
    image: "postgres" # use latest official postgres version
    container_name: postgres
    networks:
    - internal
    environment:
    - POSTGRES_USER=unicorn_user
    - POSTGRES_PASSWORD=magical_password
    - POSTGRES_DB=rainbow_database
    volumes:
    - postgres-data:/var/lib/postgresql/data/

  redis:
    image: redis:6-alpine
    container_name: redis
    networks:
    - internal
    volumes:
    - redis-data:/data
    labels:
    - "org.label-schema.group=storage"
    - "traefik.enable=false"
    command:
    - "redis-server"
    - "--maxmemory 3G"
    - "--maxmemory-policy allkeys-lru"
    restart: unless-stopped

Would be awesome to add something like app.config["APPLICATION_ROOT"] = "/devops/seo-toolkit/" but I am not sure for the rest of the integration ^^

Thanks for any insights or inputs on that.

Cheers,
Luc Michalski

Does not work on Windows.

Does not work on Windows.
When I run docker-compose --env-file .env-example up -d I get error I cannot resolve.

Getting Error on login page

when I visit web app page at http://localhost:3000 it redirects to login address http://localhost:3000/#/login
and I expected to be able to login with userpass admin/admin but I get "Network Error" toast message

What should I do to get to admin panel?

bokeh cdn 403

https://i.imgur.com/Tb0tlPJ.png

https://cdn.bokeh.org/bokeh/release/bokeh-tables-2.0.1.min.css

api/extract/links relative link doesn't get resolved correctly

I'm on: 4082a37
I executed the following end point: http://localhost:5000/api/extract/links?url=https://schubert-simon.de

I expected that child 9 and 10 of the 200 array should be:
"https://schubert-simon.de#places"
"https://schubert-simon.de#photos"
But had been:
"http://whatch.online#places"
"http://whatch.online#photos"

And #projects in 500 should be "https://schubert-simon.de#projects".

HTML example:

<a href="#photos" id="photos">
  <h2>PHOTOS</h2>
</a>

Maybe it has something todo with the caching but I did a "fresh" install of the repo. Can you reproduce it? Thanks a lot.

Any chance to run this without an NVIDIA card/driver?

Hi Stan,

thanks for cleaning up the dependencies.

I only now realized you are using PyTorch with CUDA built in.

Any chance to run this on MacOS/without CUDA?

Thanks
Sascha

Network error, when logging in

Hi! I just ran into another issue,

all the containers are running but the frontend smashes a Network error, when I try to login

Do you have an idea? I checked running it with CORS disabled, same result.

Thanks in advance, Phil

Depencies, schmendencies

Hi Stan,

I came across your software while researching automated lighthouse-ing of URLs taken from sitemap.xml files.

During setting up the project with Docker, I came across a few dep warnings, mostly from NPM related packages:

I guess these need to be addresses sooner or later?

[1/4] Resolving packages...
warning react-scripts > [email protected]: babel-eslint is now @babel/eslint-parser. This package will no longer receive updates.
warning react-scripts > webpack-dev-server > [email protected]: Chokidar 2 will break on node v14+. Upgrade to chokidar 3 with 15x less dependencies.
warning react-scripts > webpack-dev-server > chokidar > [email protected]: fsevents 1 will break on node v14+ and could be using insecure binaries. Upgrade to fsevents 2.
warning react-scripts > resolve-url-loader > rework > css > [email protected]: Please see https://github.com/lydell/urix#deprecated
warning react-scripts > resolve-url-loader > rework > css > source-map-resolve > [email protected]: Please see https://github.com/lydell/urix#deprecated
warning react-scripts > resolve-url-loader > rework > css > source-map-resolve > [email protected]: https://github.com/lydell/resolve-url#deprecated
warning react-scripts > workbox-webpack-plugin > workbox-build > @hapi/[email protected]: Switch to 'npm install joi'
warning react-scripts > workbox-webpack-plugin > workbox-build > [email protected]: This package has been deprecated and is no longer maintained. Please use @rollup/plugin-babel.
warning react-scripts > webpack > watchpack > watchpack-chokidar2 > [email protected]: Chokidar 2 will break on node v14+. Upgrade to chokidar 3 with 15x less dependencies.
warning react-scripts > workbox-webpack-plugin > workbox-build > @hapi/joi > @hapi/[email protected]: This version has been deprecated and is no longer supported or maintained
warning react-scripts > workbox-webpack-plugin > workbox-build > @hapi/joi > @hapi/[email protected]: Moved to 'npm install @sideway/address'
warning react-scripts > workbox-webpack-plugin > workbox-build > @hapi/joi > @hapi/[email protected]: This version has been deprecated and is no longer supported or maintained
warning react-scripts > workbox-webpack-plugin > workbox-build > @hapi/joi > @hapi/[email protected]: This version has been deprecated and is no longer supported or maintained
warning react-scripts > workbox-webpack-plugin > workbox-build > @hapi/joi > @hapi/topo > @hapi/[email protected]: This version has been deprecated and is no longer supported or maintained
warning react-scripts > workbox-webpack-plugin > workbox-build > strip-comments > babel-plugin-transform-object-rest-spread > babel-runtime > [email protected]: core-js@<3 is no longer maintained and not recommended for usage due to the number of issues. Please, upgrade your dependencies to the actual version of core-js@3.
warning react-scripts > jest > @jest/core > jest-config > jest-environment-jsdom > jsdom > [email protected]: request-promise-native has been deprecated because it extends the now deprecated request package, see https://github.com/request/request/issues/3142
warning react-scripts > jest > @jest/core > jest-config > jest-environment-jsdom > jsdom > [email protected]: request has been deprecated, see https://github.com/request/request/issues/3142
warning react-scripts > jest > @jest/core > jest-config > jest-environment-jsdom > jsdom > request > [email protected]: this library is no longer supported
[2/4] Fetching packages...
info [email protected]: The platform "linux" is incompatible with this module.
info "[email protected]" is an optional dependency and failed compatibility check. Excluding it from installation.
info [email protected]: The platform "linux" is incompatible with this module.
info "[email protected]" is an optional dependency and failed compatibility check. Excluding it from installation.
[3/4] Linking dependencies...
warning " > @material-ui/[email protected]" has unmet peer dependency "@material-ui/core@^4.9.12 || ^5.0.0-alpha.22".
warning " > @testing-library/[email protected]" has unmet peer dependency "@testing-library/dom@>=7.21.4".
warning " > [email protected]" has unmet peer dependency "ra-core@^3.5.3".
warning "react-admin > [email protected]" has unmet peer dependency "immutable@^3.8.1 || ^4.0.0-rc.1".
warning "react-admin > [email protected]" has unmet peer dependency "history@^4.7.2".
warning "react-admin > [email protected]" has incorrect peer dependency "react@^16.4.0".
warning "react-admin > [email protected]" has unmet peer dependency "seamless-immutable@^7.1.3".
warning "react-scripts > @typescript-eslint/eslint-plugin > [email protected]" has unmet peer dependency "typescript@>=2.8.0 || >= 3.2.0-dev || >= 3.3.0-dev || >= 3.4.0-dev || >= 3.5.0-dev || >= 3.6.0-dev || >= 3.6.0-beta || >= 3.7.0-dev || >= 3.7.0-beta".
[4/4] Building fresh packages...
success Saved lockfile.

jinja2.exceptions.TemplateNotFound: bokeh.html for GET /api/graph

I'm running the latest master without any modification.

Log:
06/19/2020 01:15:58 PM - INFO - 127.0.0.1 - - [19/Jun/2020 13:15:58] "GET /api/graph?url=https://linuxcommandlibrary.com/tips.html HTTP/1.1" 500 -
Traceback (most recent call last):
File "/usr/local/lib/python3.7/site-packages/flask/app.py", line 2463, in call
return self.wsgi_app(environ, start_response)
File "/usr/local/lib/python3.7/site-packages/flask/app.py", line 2449, in wsgi_app
response = self.handle_exception(e)
File "/usr/local/lib/python3.7/site-packages/flask/app.py", line 1866, in handle_exception
reraise(exc_type, exc_value, tb)
File "/usr/local/lib/python3.7/site-packages/flask/_compat.py", line 39, in reraise
raise value
File "/usr/local/lib/python3.7/site-packages/flask/app.py", line 2446, in wsgi_app
response = self.full_dispatch_request()
File "/usr/local/lib/python3.7/site-packages/flask/app.py", line 1951, in full_dispatch_request
rv = self.handle_user_exception(e)
File "/usr/local/lib/python3.7/site-packages/flask/app.py", line 1820, in handle_user_exception
reraise(exc_type, exc_value, tb)
File "/usr/local/lib/python3.7/site-packages/flask/_compat.py", line 39, in reraise
raise value
File "/usr/local/lib/python3.7/site-packages/flask/app.py", line 1949, in full_dispatch_request
rv = self.dispatch_request()
File "/usr/local/lib/python3.7/site-packages/flask/app.py", line 1935, in dispatch_request
return self.view_functionsrule.endpoint
File "/Users/simon/projects/SEOToolkit/flask_api.py", line 115, in interactive_graph
return generate_interactive_graph(conn, urls, relaunch, maxi_urls)
File "/Users/simon/projects/SEOToolkit/flask_api.py", line 103, in generate_interactive_graph
return update_or_insert_graph_in_db(conn, urls, maximum_urls)
File "/Users/simon/projects/SEOToolkit/flask_api.py", line 52, in update_or_insert_graph_in_db
return render_template("bokeh.html", script=script, div=div, domain=domain, template="Flask", time=datetime.now().strftime("%m/%d/%Y, %H:%M:%S"))
File "/usr/local/lib/python3.7/site-packages/flask/templating.py", line 138, in render_template
ctx.app.jinja_env.get_or_select_template(template_name_or_list),
File "/usr/local/lib/python3.7/site-packages/jinja2/environment.py", line 930, in get_or_select_template
return self.get_template(template_name_or_list, parent, globals)
File "/usr/local/lib/python3.7/site-packages/jinja2/environment.py", line 883, in get_template
return self._load_template(name, self.make_globals(globals))
File "/usr/local/lib/python3.7/site-packages/jinja2/environment.py", line 857, in _load_template
template = self.loader.load(self, name, globals)
File "/usr/local/lib/python3.7/site-packages/jinja2/loaders.py", line 115, in load
source, filename, uptodate = self.get_source(environment, name)
File "/usr/local/lib/python3.7/site-packages/flask/templating.py", line 60, in get_source
return self._get_source_fast(environment, template)
File "/usr/local/lib/python3.7/site-packages/flask/templating.py", line 89, in _get_source_fast
raise TemplateNotFound(template)
jinja2.exceptions.TemplateNotFound: bokeh.html

Roadmap V2

Despite not a lot of features, the project is getting bigger. In order to avoid huge refactors down the line, a few things need to be addressed.

• Moving towards an API driven web app. Implementing REST endpoints for every interaction. This will allow a smooth transition later on to React. Jinja2 is great for small projects, but the objective is to build a fully-featured Seo Audit Toolkit.
• Thinking about using Distributed Task Queues. This would allow the project to run background jobs and async requests easily. Also, this would increase the scalability of the project.

I'm eager to add new features. But each new feature complexifies the project even more. Taking things slow to avoid huge refactors and bottlenecks.

TODO:

• Completely move towards an API driven webapp
• Implement Distributed Task queues
• Add a complete website audit

Windows error path sqlite

Error for windows users:

Change path SQLite in SQLALCHEMY Config from 'sqlite:////tmp/test114.db' to r'sqlite:///C:\tmp\test114.db'

Implement OS Detection to avoid this issue

Resource stopwords not found for GET /api/analysis/keywords

Traceback (most recent call last):
File "/usr/local/lib/python3.7/site-packages/nltk/corpus/util.py", line 86, in __load
root = nltk.data.find('{}/{}'.format(self.subdir, zip_name))
File "/usr/local/lib/python3.7/site-packages/nltk/data.py", line 701, in find
raise LookupError(resource_not_found)
LookupError:

---

Resource stopwords not found.
Please use the NLTK Downloader to obtain the resource:

import nltk
nltk.download('stopwords')

For more information see: https://www.nltk.org/data.html

Attempted to load corpora/stopwords.zip/stopwords/

Searched in:
- '/Users/simon/nltk_data'
- '/usr/local/Cellar/python/3.7.7/Frameworks/Python.framework/Versions/3.7/nltk_data'
- '/usr/local/Cellar/python/3.7.7/Frameworks/Python.framework/Versions/3.7/share/nltk_data'
- '/usr/local/Cellar/python/3.7.7/Frameworks/Python.framework/Versions/3.7/lib/nltk_data'
- '/usr/share/nltk_data'
- '/usr/local/share/nltk_data'
- '/usr/lib/nltk_data'
- '/usr/local/lib/nltk_data'

---

Cannot create container for service redis

Hi there ! I was wondering whether anyone of you encountered issues with redis?
It is the only thing that doesn't seem to work. Using the path as defined in the example.

volumes: - redis-data:/data - redis-logs:${REDIS_LOG_PREFIX_PATH}

Terminal output
`Recreating osat-redis ... error

ERROR: for osat-redis Cannot create container for service redis: failed to mount local volume: mount /Users/phil/dev/seo-audits-toolkit/shared/logs/redis:/var/lib/docker/volumes/seo-audits-toolkit_redis-logs/_data, flags: 0x1000: no such file or directory

ERROR: for redis Cannot create container for service redis: failed to mount local volume: mount /Users/phil/dev/seo-audits-toolkit/shared/logs/redis:/var/lib/docker/volumes/seo-audits-toolkit_redis-logs/_data, flags: 0x1000: no such file or directory
ERROR: Encountered errors while bringing up the project.`

Thank you very much in advance - If you have any ideas that would be great - otherwise I'll dedicate my research to Docker stuff on the weekend :D

feature request: Data/graphs over time

For this to be even more helpful, seeing trends over time helps prioritize decisions on what to fix/when, etc.

/api/extract/images gives 500 Internal Server Error

I like the direction in which the project is going. Thanks for adding new endpoints. For some urls the call works fine but for the following it crashed:

Log
06/20/2020 01:47:31 PM - INFO - 127.0.0.1 - - [20/Jun/2020 13:47:31] "GET /api/extract/images?url=https://linuxcommandlibrary.com/basics.html HTTP/1.1" 500 -
06/20/2020 01:47:31 PM - INFO - 127.0.0.1 - - [20/Jun/2020 13:47:31] "GET /favicon.ico HTTP/1.1" 404 -
06/20/2020 01:47:37 PM - DEBUG - Starting new HTTPS connection (1): linuxcommandlibrary.com:443
06/20/2020 01:47:37 PM - DEBUG - https://linuxcommandlibrary.com:443 "GET /basics.html HTTP/1.1" 200 None
06/20/2020 01:47:37 PM - ERROR - Exception on /api/extract/images [GET]
Traceback (most recent call last):
File "/usr/local/lib/python3.7/site-packages/flask/app.py", line 2446, in wsgi_app
response = self.full_dispatch_request()
File "/usr/local/lib/python3.7/site-packages/flask/app.py", line 1951, in full_dispatch_request
rv = self.handle_user_exception(e)
File "/usr/local/lib/python3.7/site-packages/flask/app.py", line 1820, in handle_user_exception
reraise(exc_type, exc_value, tb)
File "/usr/local/lib/python3.7/site-packages/flask/_compat.py", line 39, in reraise
raise value
File "/usr/local/lib/python3.7/site-packages/flask/app.py", line 1949, in full_dispatch_request
rv = self.dispatch_request()
File "/usr/local/lib/python3.7/site-packages/flask/app.py", line 1935, in dispatch_request
return self.view_functionsrule.endpoint
File "flask_api.py", line 123, in find_all_images_page
return find_all_images(value)
File "/Users/simon/projects/SEOToolkit/toolkit/seo/images.py", line 13, in find_all_images
alt = image['alt']
File "/usr/local/lib/python3.7/site-packages/bs4/element.py", line 1401, in getitem
return self.attrs[key]
KeyError: 'alt'

Demo link is down

Hi, I can't use the demo url, can you fix it? Thank you

[admin] TypeError: Cannot read properties of null (reading 'breakpoints')

Starting the project "as is", without modifications to the admin module/part, I get a blank screen on http://localhost:3000/

react-dom.production.min.js:216 TypeError: Cannot read properties of null (reading 'breakpoints')
    at Button.js:48:22
    at c (useMediaQuery.js:19:50)
    at b (Button.js:47:20)
    at oi (react-dom.production.min.js:157:137)
    at Wc (react-dom.production.min.js:267:460)
    at kl (react-dom.production.min.js:250:347)
    at Sl (react-dom.production.min.js:250:278)
    at xl (react-dom.production.min.js:250:138)
    at bl (react-dom.production.min.js:243:163)
    at fl (react-dom.production.min.js:237:175)
cc @ react-dom.production.min.js:216
react-dom.production.min.js:216 TypeError: Cannot read properties of null (reading 'breakpoints')
    at AppBar.js:78:22
    at c (useMediaQuery.js:19:50)
    at Z (AppBar.js:77:20)
    at oi (react-dom.production.min.js:157:137)
    at Wc (react-dom.production.min.js:267:460)
    at kl (react-dom.production.min.js:250:347)
    at Sl (react-dom.production.min.js:250:278)
    at xl (react-dom.production.min.js:250:138)
    at bl (react-dom.production.min.js:243:163)
    at fl (react-dom.production.min.js:237:175)
cc @ react-dom.production.min.js:216
react-dom.production.min.js:216 TypeError: Cannot read properties of null (reading 'breakpoints')
    at Sidebar.js:36:22
    at c (useMediaQuery.js:19:50)
    at _ (Sidebar.js:35:20)
    at oi (react-dom.production.min.js:157:137)
    at Wc (react-dom.production.min.js:267:460)
    at kl (react-dom.production.min.js:250:347)
    at Sl (react-dom.production.min.js:250:278)
    at xl (react-dom.production.min.js:250:138)
    at bl (react-dom.production.min.js:243:163)
    at fl (react-dom.production.min.js:237:175)
cc @ react-dom.production.min.js:216
react-dom.production.min.js:216 TypeError: Cannot read properties of null (reading 'breakpoints')
    at Dashboard.js:31:15
    at c (useMediaQuery.js:19:50)
    at Mt (Dashboard.js:30:22)
    at oi (react-dom.production.min.js:157:137)
    at Wc (react-dom.production.min.js:267:460)
    at kl (react-dom.production.min.js:250:347)
    at Sl (react-dom.production.min.js:250:278)
    at xl (react-dom.production.min.js:250:138)
    at bl (react-dom.production.min.js:243:163)
    at fl (react-dom.production.min.js:237:175)
cc @ react-dom.production.min.js:216
react-dom.production.min.js:216 TypeError: Cannot read properties of null (reading 'breakpoints')
    at Button.js:48:22
    at c (useMediaQuery.js:19:50)
    at b (Button.js:47:20)
    at oi (react-dom.production.min.js:157:137)
    at Wc (react-dom.production.min.js:267:460)
    at kl (react-dom.production.min.js:250:347)
    at Sl (react-dom.production.min.js:250:278)
    at xl (react-dom.production.min.js:250:138)
    at bl (react-dom.production.min.js:243:163)
    at react-dom.production.min.js:123:115
cc @ react-dom.production.min.js:216
react-dom.production.min.js:216 TypeError: Cannot read properties of null (reading 'breakpoints')
    at AppBar.js:78:22
    at c (useMediaQuery.js:19:50)
    at Z (AppBar.js:77:20)
    at oi (react-dom.production.min.js:157:137)
    at Wc (react-dom.production.min.js:267:460)
    at kl (react-dom.production.min.js:250:347)
    at Sl (react-dom.production.min.js:250:278)
    at xl (react-dom.production.min.js:250:138)
    at bl (react-dom.production.min.js:243:163)
    at react-dom.production.min.js:123:115
cc @ react-dom.production.min.js:216
react-dom.production.min.js:216 TypeError: Cannot read properties of null (reading 'breakpoints')
    at Sidebar.js:36:22
    at c (useMediaQuery.js:19:50)
    at _ (Sidebar.js:35:20)
    at oi (react-dom.production.min.js:157:137)
    at Wc (react-dom.production.min.js:267:460)
    at kl (react-dom.production.min.js:250:347)
    at Sl (react-dom.production.min.js:250:278)
    at xl (react-dom.production.min.js:250:138)
    at bl (react-dom.production.min.js:243:163)
    at react-dom.production.min.js:123:115
cc @ react-dom.production.min.js:216
react-dom.production.min.js:123 Uncaught TypeError: Cannot read properties of null (reading 'breakpoints')
    at Button.js:48:22
    at c (useMediaQuery.js:19:50)
    at b (Button.js:47:20)
    at oi (react-dom.production.min.js:157:137)
    at Wc (react-dom.production.min.js:267:460)
    at kl (react-dom.production.min.js:250:347)
    at Sl (react-dom.production.min.js:250:278)
    at xl (react-dom.production.min.js:250:138)
    at bl (react-dom.production.min.js:243:163)
    at react-dom.production.min.js:123:115

I tried using the Dockerfile.dev under admin instead of Dockerfile (also editing the docker-compose.yml to fit the changes), and now I see a page with Network error when I try to login, but it's much better !

Lighthouse audits not working,

Lighthouse audits not working, i tested both in docker and normal installation it just wont save the url when we submit the url. Google api is added to env

Installation tutorial

Hello, could you make the installation steps for docker more detailed? I tried the steps from the main page, and read other sources but I couldn't make this tool to work/open, thanks in advance.

OSAT Stop build container

Nice Job

Hello,
thanks for the nice job.
Hope I can help, although with more scientific background it's a it not very easy for me.
what other audit tools except for light house can we add?

docker: Error response from daemon: pull access denied for seo-toolkit, repository does not exist or may require 'docker login': denied: requested access to the resource is denied.

docker: Error response from daemon: pull access denied for seo-toolkit, repository does not exist or may require 'docker login': denied: requested access to the resource is denied.

Getting that error while doing docker run -d -p 5000:5000 seo-toolkit:latest

Error creating osat-redis

Hi ! .... I have been trying to install OSAT for a while without finding a solution to the following problem... I verified that /root/seo-audits-toolkit/shared/logs/redis exists, tested redis with PONG output, created the path /var/lib/ docker/volumes/seo-audits-toolkit_redis-logs/_data without success in any case. Attached photo of the kernel. Also bind conf redis to 0.0.0.0

I am working with:

--> Ubuntu 22.04.1 with WSL2
--> Python 3.10.6
--> docker-compose version 1.29.2
--> Docker version 20.10.17
--> Redis server v=7.0.5

Thank you!

can i get an API access for this in python or javascript

can i get an API access for this in python or javascript

How to get GOOGLE_API_KEY ?

Hi guys,

Hope you are all well !

Can you add to the README how to get the GOOGLE_API_KEY for what services ?

Thanks for any insights or inputs on that.

Cheers,
Luc Michalski