SEO & Security Audit for Websites. Lighthouse & Security Headers crawler, Sitemap/Keywords/Images Extractor, Summarizer, etc ...

StanGirard, updated 🕥 2023-02-06 11:13:41

Open source Audits Toolkit

OSAT is a collection of tools created help you in your quest for a better website. All of these tools have been grouped into a single web app.

I've grown tired of SEO agencies making us pay hundreds of euros for simple tools. I decided to develop OSAT to help users find issues on their website and increase their SEO for free.

Why you need it

It's free, easy and open source.
It has a growing list of features
It's easy to install

Features

Authentification - A fully featured authentification system for the front & back
RBAC/Organizations - Create different organizations and give different access to each org to your users.
Lighthouse Score - Run Lighthouse Audits and keep track of your scores
SERP Rank - Get the rank of your website on google for specific queries
Keywords Finder - Find all the keywords of an article.
Extract Headers/Links/Images - Easily extract all the links on your website and their status codes, the headers of a page and all the images.
Sitemap Extractor - Extract all the urls of a website from its sitemap
Summarizer - Summarize any text from any length. Awesome for excerpt !
Security Audit - Audit Headers, Redirect, etc to make sure you website is secure.

Demo

Go to demo.primates.dev
Login: demo
Password: demodemo

Admin part is api.primates.dev/admin

Installation

Bash git clone https://github.com/StanGirard/seo-audits-toolkit cd seo-audits-toolkit docker-compose pull ## docker-compose build if you want to build it on your own docker-compose --env-file .env-example up -d

Init the project

Bash docker exec -it osat-server python manage.py flush docker exec -it osat-server python manage.py loaddata init_data.json

Dashboard

You can access the dashboard by going to localhost:3000

Login: admin Password: admin

Config

If needed create a .env file with information that you would like to change

Configuration

Create organization

You need to go to Org -> Organization and create a new organization. You can create as many as you want. Organization are used in order to implement RBAC in the project and only display information about an organization to users of this organization. Here is a quicklinkg to access it http://localhost:8000/admin/org/website/

Add user to organization

Once your organization is created. You need to add your users to this organization. Go to Organizations -> Organizations Users and add your users to the organization you want. http://localhost:8000/admin/organizations/organizationuser/

Useful Links

Webapp http://localhost:3000
Admin Dashboard http://localhost:8000/admin
Swagger like interface http://localhost:8000

Contributions

Please feel free to add any contribution. If you've been working on a script that could be integrated in this app. Please feel free to do it. Don't hesitate to open issues to ask questions. I've tried to document the code as much as I could to ease the integration

Backend

You can just add a django module and I'll take care of intregrating it in the front.

Frontend

I've used React Admin to build the front-end. If you want to help me improve the UI or add new functionnalites. Please feel free to contribute.

Disclaimers

I'm not a python nor a frontend developer.

Issues

[admin] TypeError: Cannot read properties of null (reading 'breakpoints')

opened on 2023-02-13 02:29:20 by TBG-FR

Starting the project "as is", without modifications to the admin module/part, I get a blank screen on http://localhost:3000/

js react-dom.production.min.js:216 TypeError: Cannot read properties of null (reading 'breakpoints') at Button.js:48:22 at c (useMediaQuery.js:19:50) at b (Button.js:47:20) at oi (react-dom.production.min.js:157:137) at Wc (react-dom.production.min.js:267:460) at kl (react-dom.production.min.js:250:347) at Sl (react-dom.production.min.js:250:278) at xl (react-dom.production.min.js:250:138) at bl (react-dom.production.min.js:243:163) at fl (react-dom.production.min.js:237:175) cc @ react-dom.production.min.js:216 react-dom.production.min.js:216 TypeError: Cannot read properties of null (reading 'breakpoints') at AppBar.js:78:22 at c (useMediaQuery.js:19:50) at Z (AppBar.js:77:20) at oi (react-dom.production.min.js:157:137) at Wc (react-dom.production.min.js:267:460) at kl (react-dom.production.min.js:250:347) at Sl (react-dom.production.min.js:250:278) at xl (react-dom.production.min.js:250:138) at bl (react-dom.production.min.js:243:163) at fl (react-dom.production.min.js:237:175) cc @ react-dom.production.min.js:216 react-dom.production.min.js:216 TypeError: Cannot read properties of null (reading 'breakpoints') at Sidebar.js:36:22 at c (useMediaQuery.js:19:50) at _ (Sidebar.js:35:20) at oi (react-dom.production.min.js:157:137) at Wc (react-dom.production.min.js:267:460) at kl (react-dom.production.min.js:250:347) at Sl (react-dom.production.min.js:250:278) at xl (react-dom.production.min.js:250:138) at bl (react-dom.production.min.js:243:163) at fl (react-dom.production.min.js:237:175) cc @ react-dom.production.min.js:216 react-dom.production.min.js:216 TypeError: Cannot read properties of null (reading 'breakpoints') at Dashboard.js:31:15 at c (useMediaQuery.js:19:50) at Mt (Dashboard.js:30:22) at oi (react-dom.production.min.js:157:137) at Wc (react-dom.production.min.js:267:460) at kl (react-dom.production.min.js:250:347) at Sl (react-dom.production.min.js:250:278) at xl (react-dom.production.min.js:250:138) at bl (react-dom.production.min.js:243:163) at fl (react-dom.production.min.js:237:175) cc @ react-dom.production.min.js:216 react-dom.production.min.js:216 TypeError: Cannot read properties of null (reading 'breakpoints') at Button.js:48:22 at c (useMediaQuery.js:19:50) at b (Button.js:47:20) at oi (react-dom.production.min.js:157:137) at Wc (react-dom.production.min.js:267:460) at kl (react-dom.production.min.js:250:347) at Sl (react-dom.production.min.js:250:278) at xl (react-dom.production.min.js:250:138) at bl (react-dom.production.min.js:243:163) at react-dom.production.min.js:123:115 cc @ react-dom.production.min.js:216 react-dom.production.min.js:216 TypeError: Cannot read properties of null (reading 'breakpoints') at AppBar.js:78:22 at c (useMediaQuery.js:19:50) at Z (AppBar.js:77:20) at oi (react-dom.production.min.js:157:137) at Wc (react-dom.production.min.js:267:460) at kl (react-dom.production.min.js:250:347) at Sl (react-dom.production.min.js:250:278) at xl (react-dom.production.min.js:250:138) at bl (react-dom.production.min.js:243:163) at react-dom.production.min.js:123:115 cc @ react-dom.production.min.js:216 react-dom.production.min.js:216 TypeError: Cannot read properties of null (reading 'breakpoints') at Sidebar.js:36:22 at c (useMediaQuery.js:19:50) at _ (Sidebar.js:35:20) at oi (react-dom.production.min.js:157:137) at Wc (react-dom.production.min.js:267:460) at kl (react-dom.production.min.js:250:347) at Sl (react-dom.production.min.js:250:278) at xl (react-dom.production.min.js:250:138) at bl (react-dom.production.min.js:243:163) at react-dom.production.min.js:123:115 cc @ react-dom.production.min.js:216 react-dom.production.min.js:123 Uncaught TypeError: Cannot read properties of null (reading 'breakpoints') at Button.js:48:22 at c (useMediaQuery.js:19:50) at b (Button.js:47:20) at oi (react-dom.production.min.js:157:137) at Wc (react-dom.production.min.js:267:460) at kl (react-dom.production.min.js:250:347) at Sl (react-dom.production.min.js:250:278) at xl (react-dom.production.min.js:250:138) at bl (react-dom.production.min.js:243:163) at react-dom.production.min.js:123:115

I tried using the Dockerfile.dev under admin instead of Dockerfile (also editing the docker-compose.yml to fit the changes), and now I see a page with Network error when I try to login, but it's much better !

Update Dockerfile

opened on 2023-02-06 11:13:40 by EIT-AUS

node changed from 12.x to 14.x as it was breaking docker build

docker compose file issues

opened on 2023-02-06 06:07:58 by EIT-AUS

issue 1= /docker-compose.yml" is unsupported. You might be seeing this error because you're using the wrong Compose file version. Either specify a supported version (e.g "2.2" or "3.3") and place your service definitions under the services key, or omit the version key and place your service definitions at the root of the file to use version 1.

issue 2= WARN[0000] The "TINI_VERSION" variable is not set. Defaulting to a blank string. WARN[0000] The "POSTGRES_PASSWORD" variable is not set. Defaulting to a blank string. WARN[0000] The "DEBUG" variable is not set. Defaulting to a blank string. WARN[0000] The "ALLOWED_HOSTS" variable is not set. Defaulting to a blank string. WARN[0000] The "REST_FRAMEWORK_PAGE_SIZE" variable is not set. Defaulting to a blank string. WARN[0000] The "CELERY_RESULT_BACKEND" variable is not set. Defaulting to a blank string. WARN[0000] The "DATABASE_ENGINE" variable is not set. Defaulting to a blank string. WARN[0000] The "POSTGRES_USER" variable is not set. Defaulting to a blank string. WARN[0000] The "POSTGRES_PORT" variable is not set. Defaulting to a blank string. WARN[0000] The "POSTGRES_DB" variable is not set. Defaulting to a blank string. WARN[0000] The "SECRET_KEY" variable is not set. Defaulting to a blank string. WARN[0000] The "CELERY_BROKER_URL" variable is not set. Defaulting to a blank string. WARN[0000] The "CORS_ALLOWED_ORIGINS" variable is not set. Defaulting to a blank string. WARN[0000] The "NAMESPACE" variable is not set. Defaulting to a blank string. WARN[0000] The "TINI_VERSION" variable is not set. Defaulting to a blank string. WARN[0000] The "NAMESPACE" variable is not set. Defaulting to a blank string. WARN[0000] The "CELERY_BROKER_URL" variable is not set. Defaulting to a blank string. WARN[0000] The "DATABASE_ENGINE" variable is not set. Defaulting to a blank string. WARN[0000] The "DEBUG" variable is not set. Defaulting to a blank string. WARN[0000] The "CELERY_RESULT_BACKEND" variable is not set. Defaulting to a blank string. WARN[0000] The "ALLOWED_HOSTS" variable is not set. Defaulting to a blank string. WARN[0000] The "REST_FRAMEWORK_PAGE_SIZE" variable is not set. Defaulting to a blank string. WARN[0000] The "POSTGRES_PORT" variable is not set. Defaulting to a blank string. WARN[0000] The "POSTGRES_DB" variable is not set. Defaulting to a blank string. WARN[0000] The "POSTGRES_PASSWORD" variable is not set. Defaulting to a blank string. WARN[0000] The "POSTGRES_USER" variable is not set. Defaulting to a blank string. WARN[0000] The "SECRET_KEY" variable is not set. Defaulting to a blank string. WARN[0000] The "CORS_ALLOWED_ORIGINS" variable is not set. Defaulting to a blank string. WARN[0000] The "NAMESPACE" variable is not set. Defaulting to a blank string. WARN[0000] The "TINI_VERSION" variable is not set. Defaulting to a blank string. WARN[0000] The "CELERY_RESULT_BACKEND" variable is not set. Defaulting to a blank string. WARN[0000] The "POSTGRES_DB" variable is not set. Defaulting to a blank string. WARN[0000] The "POSTGRES_PASSWORD" variable is not set. Defaulting to a blank string. WARN[0000] The "POSTGRES_PORT" variable is not set. Defaulting to a blank string. WARN[0000] The "POSTGRES_USER" variable is not set. Defaulting to a blank string. WARN[0000] The "SECRET_KEY" variable is not set. Defaulting to a blank string. WARN[0000] The "CORS_ALLOWED_ORIGINS" variable is not set. Defaulting to a blank string. WARN[0000] The "ALLOWED_HOSTS" variable is not set. Defaulting to a blank string. WARN[0000] The "DEBUG" variable is not set. Defaulting to a blank string. WARN[0000] The "DATABASE_ENGINE" variable is not set. Defaulting to a blank string. WARN[0000] The "REST_FRAMEWORK_PAGE_SIZE" variable is not set. Defaulting to a blank string. WARN[0000] The "CELERY_BROKER_URL" variable is not set. Defaulting to a blank string. WARN[0000] The "NAMESPACE" variable is not set. Defaulting to a blank string. WARN[0000] The "NODE_VERSION" variable is not set. Defaulting to a blank string. WARN[0000] The "TINI_VERSION" variable is not set. Defaulting to a blank string. WARN[0000] The "POSTGRES_USER" variable is not set. Defaulting to a blank string. WARN[0000] The "POSTGRES_PASSWORD" variable is not set. Defaulting to a blank string. WARN[0000] The "POSTGRES_DB" variable is not set. Defaulting to a blank string. WARN[0000] The "NAMESPACE" variable is not set. Defaulting to a blank string. WARN[0000] The "REDIS_VERSION" variable is not set. Defaulting to a blank string. WARN[0000] The "REDIS_LOG_PREFIX_PATH" variable is not set. Defaulting to a blank string. WARN[0000] The "NAMESPACE" variable is not set. Defaulting to a blank string. WARN[0000] The "REDIS_LOG_PREFIX_PATH" variable is not set. Defaulting to a blank string. WARN[0000] The "REDIS_MEMORY_MAX" variable is not set. Defaulting to a blank string. WARN[0000] The "REDIS_LOG_FILE" variable is not set. Defaulting to a blank string. WARN[0000] The "PWD" variable is not set. Defaulting to a blank string. WARN[0000] The "PWD" variable is not set. Defaulting to a blank string. WARN[0000] The "PWD" variable is not set. Defaulting to a blank string.

Bump json5 from 1.0.1 to 1.0.2 in /admin

opened on 2023-01-07 14:08:13 by dependabot[bot]

Bumps json5 from 1.0.1 to 1.0.2.

Release notes

Sourced from json5's releases.

v1.0.2

Fix: Properties with the name __proto__ are added to objects and arrays. (#199) This also fixes a prototype pollution vulnerability reported by Jonathan Gregson! (#295). This has been backported to v1. (#298)

Changelog

Sourced from json5's changelog.

Unreleased [code, diff]

v2.2.3 [code, diff]

Fix: [email protected] is now the 'latest' release according to npm instead of v1.0.2. (#299)

v2.2.2 [code, diff]

Fix: Properties with the name __proto__ are added to objects and arrays. (#199) This also fixes a prototype pollution vulnerability reported by Jonathan Gregson! (#295).

v2.2.1 [code, diff]

Fix: Removed dependence on minimist to patch CVE-2021-44906. (#266)

v2.2.0 [code, diff]

New: Accurate and documented TypeScript declarations are now included. There is no need to install @types/json5. (#236, #244)

v2.1.3 [code, diff]

Fix: An out of memory bug when parsing numbers has been fixed. (#228, #229)

v2.1.2 [code, diff]

... (truncated)

Commits

a62db1e 1.0.2
e0c23fe docs: update CHANGELOG for v1.0.2
62a6540 fix: add proto to objects and arrays
See full diff in compare view

Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting @dependabot rebase.

Dependabot commands and options

You can trigger Dependabot actions by commenting on this PR: - `@dependabot rebase` will rebase this PR - `@dependabot recreate` will recreate this PR, overwriting any edits that have been made to it - `@dependabot merge` will merge this PR after your CI passes on it - `@dependabot squash and merge` will squash and merge this PR after your CI passes on it - `@dependabot cancel merge` will cancel a previously requested merge and block automerging - `@dependabot reopen` will reopen this PR if it is closed - `@dependabot close` will close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually - `@dependabot ignore this major version` will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself) - `@dependabot ignore this minor version` will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself) - `@dependabot ignore this dependency` will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself) - `@dependabot use these labels` will set the current labels as the default for future PRs for this repo and language - `@dependabot use these reviewers` will set the current reviewers as the default for future PRs for this repo and language - `@dependabot use these assignees` will set the current assignees as the default for future PRs for this repo and language - `@dependabot use this milestone` will set the current milestone as the default for future PRs for this repo and language You can disable automated security fix PRs for this repo from the [Security Alerts page](https://github.com/StanGirard/seo-audits-toolkit/network/alerts).

LightHouse Dependency issue fixed

opened on 2022-12-24 08:56:22 by Ajay-joshi-mtr None

Bump express from 4.17.1 to 4.18.2 in /admin

opened on 2022-12-14 15:27:10 by dependabot[bot]

Bumps express from 4.17.1 to 4.18.2.

Release notes

Sourced from express's releases.

4.18.2

Fix regression routing a large stack in a single route

deps: [email protected]

deps: [email protected]

perf: remove unnecessary object clone

deps: [email protected]

4.18.1

Fix hanging on large stack of sync routes

4.18.0

Add "root" option to res.download

Allow options without filename in res.download

Deprecate string and non-integer arguments to res.status

Fix behavior of null/undefined as maxAge in res.cookie

Fix handling very large stacks of sync middleware

Ignore Object.prototype values in settings through app.set/app.get

Invoke default with same arguments as types in res.format

Support proper 205 responses using res.send

Use http-errors for res.format error

deps: [email protected]

Fix error message for json parse whitespace in strict

Fix internal error when inflated body exceeds limit

Prevent loss of async hooks context

Prevent hanging when request already read

deps: [email protected]

deps: [email protected]

deps: [email protected]

deps: [email protected]

deps: [email protected]

deps: [email protected]

Add priority option

Fix expires option to reject invalid dates

deps: [email protected]

Replace internal eval usage with Function constructor

Use instance methods on process to check for listeners

deps: [email protected]

Remove set content headers that break response

deps: [email protected]

deps: [email protected]

deps: [email protected]

Prevent loss of async hooks context

deps: [email protected]

deps: [email protected]

Fix emitted 416 error missing headers property

Limit the headers removed for 304 response

deps: [email protected]

deps: [email protected]

deps: [email protected]

deps: [email protected]

... (truncated)

Changelog

Sourced from express's changelog.

4.18.2 / 2022-10-08

Fix regression routing a large stack in a single route

deps: [email protected]

deps: [email protected]

perf: remove unnecessary object clone

deps: [email protected]

4.18.1 / 2022-04-29

Fix hanging on large stack of sync routes

4.18.0 / 2022-04-25

Add "root" option to res.download

Allow options without filename in res.download

Deprecate string and non-integer arguments to res.status

Fix behavior of null/undefined as maxAge in res.cookie

Fix handling very large stacks of sync middleware

Ignore Object.prototype values in settings through app.set/app.get

Invoke default with same arguments as types in res.format

Support proper 205 responses using res.send

Use http-errors for res.format error

deps: [email protected]

Fix error message for json parse whitespace in strict

Fix internal error when inflated body exceeds limit

Prevent loss of async hooks context

Prevent hanging when request already read

deps: [email protected]

deps: [email protected]

deps: [email protected]

deps: [email protected]

deps: [email protected]

deps: [email protected]

Add priority option

Fix expires option to reject invalid dates

deps: [email protected]

Replace internal eval usage with Function constructor

Use instance methods on process to check for listeners

deps: [email protected]

Remove set content headers that break response

deps: [email protected]

deps: [email protected]

deps: [email protected]

Prevent loss of async hooks context

deps: [email protected]

deps: [email protected]

... (truncated)

Commits

8368dc1 4.18.2
61f4049 docs: replace Freenode with Libera Chat
bb7907b build: [email protected]
f56ce73 build: [email protected]
24b3dc5 deps: [email protected]
689d175 deps: [email protected]
340be0f build: [email protected]
33e8dc3 docs: use Node.js name style
644f646 build: [email protected]
ecd7572 build: [email protected]
Additional commits viewable in compare view

Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting @dependabot rebase.

Dependabot commands and options

Releases

v1.4.1 2021-03-30 09:43:05

No new changes to the application.

Mainly packages updates

V1.4 2021-01-10 17:52:31

Release 1.4

Refactoring of the entire code base
Moving from flask to Django
Frontend with React Admin

Features

Authentification - A fully featured authentification system for the front & back
RBAC/Organizations - Create different organizations and give different access to each org to your users.
Lighthouse Score - Run Lighthouse Audits and keep track of your scores
Keywords Finder - Find all the keywords of an article.
Extract Headers/Links/Images - Easily extract all the links on your website and their status codes, the headers of a page and all the images.
Sitemap Extractor - Extract all the urls of a website from its sitemap
Summarizer - Summarize any text from any length. Awesome for excerpt !
Security Audit - Audit Headers, Redirect, etc to make sure you website is secure.

V1.3 2020-07-23 22:16:24

Distributed Message Queue

Now tasks can be executed in the background.

Capture d’écran 2020-07-23 à 22 47 05

V1.2 2020-07-15 22:01:52

The first step towards an API driven Web app 🏆 Created an endpoint for all interactions. Should be easier to change the frontend in the future: Jinja2 -> React

V1.1.2 2020-07-01 21:37:38

New features:

Lighthouse SEO, PWA, Best Practices, Accessibility, Performance Scores
Lighthouse Scores' Timeline
Custom Config

V1.1.1 2020-06-29 21:41:47

New features: - Extract all headers - Extract all working links from a page - Extract all links found on the website - Extract all images found on a page Capture d’écran 2020-06-29 à 23 49 57

Stan Girard

Cloud, Blockchain, Data, Crawl, NLP, Web ... The World is full of interesting subjects. SRE @padok-team

GitHub Repository

seo seo-tools internal-links headers serp link-extractor lighthouse dashboard analysis python securityheader summarizer crawler audits extractor