sandalon/sitemap-validator

Validate the sitemap for a production site

sitemap-validator's Issues

Enhancement - report HTTP status code

Hi Corey, great little tool you got here, and i've used it to parse and check a multi-part sitemap containing in excess of 2,000,000 urls.

However, it just prints Bad URL: <URL> because the current stanza is this:

if(resp.statusCode != task.code) {
      console.log('Bad URL: ' + task.url);
      callback();
      return;
    }

If it was modified to:

if(resp.statusCode != task.code) {
      console.log(resp.statusCode + "," + task.url);
      callback();
      return;
    }

You could see why the URL was "bad" - i.e not a 200 or whatever was passed in with the -c option.

I've made this modification locally, and discovered that a change made in the caching tier responded with 301 for a lot of URLs that were published in the sitemap, and googlebot doesn't like 301s much.

Also, if piping the output from STDERR to a log, you get CSV. Which, with 2,000,000 urls, you need ;-)

Vulnerability

Just FYI

npm i sitemap-validator --save-dev
+ [email protected]
added 6 packages from 54 contributors and audited 2023 packages in 6.435s
found 1 high severity vulnerability
  run `npm audit fix` to fix them, or `npm audit` for details
jfolpf@toshiba:~/autocosts$ npm audit
                                                                                
                       === npm audit security report ===                        
                                                                                
┌──────────────────────────────────────────────────────────────────────────────┐
│                                Manual Review                                 │
│            Some vulnerabilities require your attention to resolve            │
│                                                                              │
│         Visit https://go.npm.me/audit-guide for additional guidance          │
└──────────────────────────────────────────────────────────────────────────────┘
┌───────────────┬──────────────────────────────────────────────────────────────┐
│ High          │ Regular Expression Denial of Service                         │
├───────────────┼──────────────────────────────────────────────────────────────┤
│ Package       │ string                                                       │
├───────────────┼──────────────────────────────────────────────────────────────┤
│ Patched in    │ No patch available                                           │
├───────────────┼──────────────────────────────────────────────────────────────┤
│ Dependency of │ sitemap-validator [dev]                                      │
├───────────────┼──────────────────────────────────────────────────────────────┤
│ Path          │ sitemap-validator > string                                   │
├───────────────┼──────────────────────────────────────────────────────────────┤
│ More info     │ https://npmjs.com/advisories/536                             │
└───────────────┴──────────────────────────────────────────────────────────────┘
found 1 high severity vulnerability in 2023 scanned packages
  1 vulnerability requires manual review. See the full report for details.