iamvishnusankar/next-sitemap

Sitemap generator for next.js. Generate sitemap(s) and robots.txt for all static/pre-rendered/dynamic/server-side pages.

next-sitemap's Issues

Error when open server-sitemap

Describe the bug
i tried the same file from this example

To Reproduce
Steps to reproduce the behavior:

1. I was try to go to this url localhost:3000/server-sitemap.xml
2. And it return 500 error
3. See this error
   

Throws error: Cannot read property 'pages' of undefined

Describe the bug
Running next-sitemap in a postbuild script errors out with the error

Cannot read 'pages' of undefined

....Object.keys(manifest.build.pages)
							  ^

To Reproduce
Steps to reproduce the behavior:

1. Run next build in build and next-sitemap in postbuild
2. Wait for it to build
3. See error

Expected behavior
next-sitemap should not throw an error

Screenshots

Urls trailing slash option

Is your feature request related to a problem? Please describe.
A clear and concise description of what the problem is. Ex. I'm always frustrated when [...]

The default behaviour of next.js is to redirect URLs with trailing slash to non-trailing slash URLs. Eg: /about/ will be redirected to /about.
In my next-sitemap-config.js file, I have set the siteUrl property to not have a trailing slash. However, In my sitemap, the home page URL loc has a trailing slash at the end. I think this is not ideal for SEO as URL loc should be identical to the actual canonical URL. The slash only appears at the end of the index URL loc and not the rest of the URLs.

I am not sure if I got that right or I misunderstood the way this works.

My website URL: https://www.techheart.co.uk
My sitemap URL: https://www.techheart.co.uk/sitemap.xml
(If you are reading this in the future, the content of the links may have been updated)

Describe the solution you'd like
A clear and concise description of what you want to happen.
Dedicated option for including or not including trailing slashes.

Describe alternatives you've considered
A clear and concise description of any alternative solutions or features you've considered.
It may be achievable using a custom transform function, but it would be a good idea to have an option for that.

Additional context
Add any other context or screenshots about the feature request here.

Creating a sitemap for dynamic pages

Is there a way for the sitemap to include dynamic pages? Looking to implement that in my site and don't see it in the docs.

Thanks! 😃

exclude pattern is not working

Describe the bug
I want to exclude all my pages that include the word page in their URLs:

https://XYZ.com/kitchen/page/2
https://XYZ.com/kitchen/page/3
...

https://XYZ.com/shop/page/2
...

I'm currently using this rule in my next-sitemap.js file:

module.exports = {
  siteUrl: 'https://XYZ.com',
  exclude: [
    '/*page*'
  ]
};
```

**Expected behavior**
The rule should exclude all the URLs from the sitemap.yml generated.

How to Return Full Path?

Hi,

First of all thank you for creating this!

Is it possible to have the full path returned in my sitemap.xml please?

For example, current path returned: https://shootingsuppliesltd.co.uk/accessories/4578, the actual path is: https://shootingsuppliesltd.co.uk/accessories/4578?slug=napier-apex-auto-click.

Support Google News Sitemaps

Is your feature request related to a problem? Please describe.
I am using next.js to build local news websites. I want to ensure they are configured properly for SEO and Google News. Google News requires a specific sitemap format with special fields in order to have your news website populate in Google News.

Describe the solution you'd like
In order to support this, next-sitemap would need to allow the population of nested fields. A Google News sitemap requires some specific nested fields. Here's an example from The New York Times:

<url>
  <loc>
    https://www.nytimes.com/2019/09/30/learning/lesson-of-the-day-kehinde-wileys-times-square-monument-thats-no-robert-e-lee.html
  </loc>
  <lastmod>2021-03-05T20:38:12Z</lastmod>
  <news:news>
    <news:publication>
      <news:name>The New York Times</news:name>
      <news:language>en-US</news:language>
    </news:publication>
    <news:publication_date>2019-09-30T08:00:03Z</news:publication_date>
    <news:title>
      Lesson of the Day: ‘Kehinde Wiley’s Times Square Monument: That’s No Robert E. Lee’
    </news:title>
  </news:news>
  <image:image>
    <image:loc>
      https://static01.nyt.com/images/2019/09/27/arts/27kehinde3/27kehinde3-articleLarge.jpg
    </image:loc>
  </image:image>
</url>

Ideally, when returning an object for a URL either in the transform function or in a dynamic sitemap, I could populate these fields:

return {
  loc: "/2019/09/30/learning/lesson-of-the-day-kehinde-wileys-times-square-monument-thats-no-robert-e-lee.html",
  lastmod: "2021-03-05T20:38:12Z",
  news: {
    publication: {
      name: "The New York Times",
      language: "en-US"
    },
    publication_date: "2019-09-30T08:00:03Z",
    title: "Lesson of the Day: 'Kehinde Wiley’s Times Square Monument: That’s No Robert E. Lee'"
  },
  image: {
    loc: "https://static01.nyt.com/images/2019/09/27/arts/27kehinde3/27kehinde3-articleLarge.jpg"
  }
}

Describe alternatives you've considered
I could generate this XML file on my own through next.js, but it would be nice to use the same library I'm using to generate the overall sitemap.

Es posbile la creación de ejemplos?

Estaba mirando la funcionalidad de esta herramienta pero no es muy clara para su uso en el desarrollo cuando se genera y despliega la aplicación en producción, no se si es posible crear un ejemplo con una página real en Vercel y colocar el ejemplo. Gracias

Sorry to write in Spanish, I will try to translate the above.

I was looking at the functionality of this tool but it is not very clear for use in development when the application is generated and deployed in production, I do not know if it is possible to create an example with a real page in Vercel and place the example. Thank you

any plans regarding dynamic pages?

I was so happy when I found this....
But then I realized, it's only for static pages...

Is support for dynamic pages on the roadmap?

Have you seen how he's doing it https://blog.yorunohikage.fr/2020/07/20/statically-generated-sitemap-in-nextjs-9-3/

Question: is there a way to control the [dynamic] urls generated when using with Next i18n and Vercel?

Thanks for the great package 👏

When using Next i18n routing
https://nextjs.org/docs/advanced-features/i18n-routing

and building with Vercel, Vercel creates [dynamic] urls prepended with the i18n locale.

The sitemap generated also contains urls prepended with the i18n locale.

Is there a way to control this? Or a best practice when generating the static sitemap?

Thank you again.

Support RSS feeds

Is your feature request related to a problem? Please describe.
The same difficulties exist for rss on Next as with site maps

Describe the solution you'd like
That this package generates RSS feed as well

Describe alternatives you've considered
The alternatives on blogs etc are hacky

allKeys.flatMap is not a function

Getting this error when I try to build a sitemap.

I'm using Next.js 9.3 with the default next-sitemap.js:

module.exports = {
  siteUrl: 'https://example.com',
  generateRobotsTxt: true, // (optional)
  // ...other options
}

Looks like the offending code is here but I'm not sure what's going wrong exactly.

I'm also attaching a ZIP of some of the manifests which may help debug this. I tried for a while but couldn't figure it out.

manifests.zip

The sitemap contains custom error pages

Describe the bug
When I use custom error 404 (I didn't have tried with other custom errors), the error page url is generated in the sitemap and I have to manually remove it with an exclude rule. exclude: ['*/404']

Note: I use the defaultLocale parameter provided by Next 10.

Expected behavior
I expect generated sitemap didn't contains error urls.

Sitemap for SSR rendered pages

Hi, it seems like your sitemap generator build the xml according to the pre rendered static pages, am i misleading or there is no way to generate it for pages using GetServerSideProps ?

Sitemap for dynamic pages in locale routes

This is not a bug, maybe a request.
Great package btw. Really makes things easy. My sitemap seems to be comprehensive even with locale routes.
It's picking all the locale routes.

https://hacktravels.com/sitemap.xml

The only thing is that it doesn't pick up the locale routes for dynamic pages. I think Google still can index it even its not specified on the sitemap, but it would still be nice to have.
This is showing only the /en/pages.

cheers

Multiples sitemaps by category

Is it possible to use multiples sitemaps by category?

Example: https://www.destockage-habitat.com/sitemap

Skip line between User-agent's in robots.txt

Is your feature request related to a problem? Please describe.
Many of the big companies have empty line between User-agent, Host and Sitemap in robots.txt.
E.g.
https://www.google.com/robots.txt
https://www.amazon.com/robots.txt
Not sure if this impacts anything but it should be a best practice.

Describe the solution you'd like
Have one line of empty space by default like:

# robots.txt
User-agent: *
Disallow: /exec/obidos/account-access-login
Disallow: /exec/obidos/change-style

User-agent: EtaoSpider
Disallow: /

lastmod undefined is added when autoLastmod is false

Describe the bug
When I set config autoLastmod to false, I expect <lastmod> doesn't appear at all in the sitemap.xml output. However, output sitemap has <lastmod>undefined</lastmod>. I'm not sure this is an expected behavior either.

To Reproduce
Steps to reproduce the behavior:

1. Set autoLastmod in next-sitemap.js to false.
2. Add "postbuild": "next-sitemap" to package.json.
3. Run yarn build.

getServerSideSitemap is being returned as undefined

Describe bugs
getServerSideSitemap is not working as expected on generating sitemap for dynamic routes, even though I followed the process of implement it right, it does not work, it might be an exporting problem, cause it's undefined.

Thanks!

Hey there, just came here to say thank you. This module worked immediately for me and I will talk about it in the upcoming https://nextjsnews.com/ newsletter.

Working example of using transform function to use different values per page

I just wondered if there was a working example of how to use the transform function to, for example use a different priority for the homepage?

I can see the example in the docs on the homepage but it doesn't really explain how to use this in the above scenario, unless I'm missing something.

thanks!

url in transform gives path for conditioning

Describe the bug

const origin = 'https://example.com'
module.exports = {
   siteUrl: origin,
   transform: (config, url) => {
       return {
         loc: url,
         test: (url == origin) ? 'true' : 'false'  // returns false
    }
  },
}

test will always give false even though they are the same value.
test gives true only if I give the path not the actual url.

{
   test: (url == '/') ? 'true' : 'false' // returns true
}

will give true in this case.

Expected behavior
url should give the actual url not the path for conditioning.

Update sitemap file with fallback route

I have a large e-commerce site with about 100+ product pages and i only create about 5-10 pages at build times with fallback:true . So the sitemap file created at build time didnt have all the actual pages in the cms, do i need to update it with the fallback route created by nextjs later on or i need fetch all the route in the cms ?

`<lastmod>` field is updated even though the page didn't change

Describe the bug
<lastmod> field is always equal to the timestamp of the sitemap generation

To Reproduce
Steps to reproduce the behavior:

1. Create a new NextJS app using npx create-next-app
2. Add this package and configure it using sitemap.js
3. Build the project
4. Generate sitemap.xml once
5. Have a look at the <lastmod> tag which has the current time as value
6. Generate sitemap.xml again without changing the code
7. The <lastmod> tag is updated even though we haven't changed the contents of the pages

Expected behavior
The <lastmod> tag should only update if the file itself changed.

---

I think it should keep a hash based on the filename or file content, and only if that's different change the <lastmod> value.

Excluding pages is not working properly with i18n

When using different locales, the exclude array will only exclude the urls from the baseUrl, but not the localized routes.

So if I have:

"exclude": ["/foo"]

it wil exclude /foo, but there will still be an entry for /en/foo

Can't install: node engine version mismatch

$ yarn add next-sitemap
yarn add v1.22.4
[1/5] 🔍  Validating package.json...
[2/5] 🔍  Resolving packages...
[3/5] 🚚  Fetching packages...
error [email protected]: The engine "node" is incompatible with this module. Expected version "^10.13.0 || ^12.13.0 || ^14.15.0 || >=15.0.0". Got "14.10.1"
error Found incompatible module.
info Visit https://yarnpkg.com/en/docs/cli/add for documentation about this command.
$ node -v
v14.15.1

Using macOS 11.3.1

"robots.txt" or sitemap doesn't generated with npm scripts

It work fine when i want to generate sitemap with yarn. I try to use npm while but it doesn't work that sitemap or robots.txt generating. What's the reason for this ?

Occurrence Example:

• First of all, package add to devDependencies. Like npm i -D next-sitemap
• Create next-sitemap file as below

module.exports = {
    siteUrl: "https://example.com",
    generateRobotsTxt: true
}

• Add postbuild script on package.json

  "scripts": {
    "build": "next build",
    "postbuild": "next-sitemap",
  },

• Run the npm run build and you will see that it didn't work as expected that files exported to public directory.

`robots.txt` isn't generated & `siteUrl` isn't outputted properly while adding script to Next's compilation chain

Describe the bug

robots.txt isn't generated & siteUrl isn't outputted properly while adding script to Next's compilation chain. In place of siteUrl, it has undefined everywhere. Even the exclude property isn't working properly.

As mentioned in my previous issue #64, I wanted to use frontmatter fields to generate Sitemap.

I found a way to do it by using Next's compilation chain.

Here's a short article on that → https://dev.to/nalanj/adding-scripts-to-next-js-n7i

TL;DR 👇

I put this in my next.config.js:

const withBundleAnalyzer = require('@next/bundle-analyzer')({
	enabled: process.env.ANALYZE === 'true',
})

module.exports = withBundleAnalyzer({
	.
	.
	.
	webpack: (config, options) => {
		.
		.
		.
		if (!options.dev && options.isServer) {
			const originalEntry = config.entry

			config.entry = async () => {
				const entries = { ...(await originalEntry()) }
				entries['./scripts/build-rss'] = './scripts/build-rss.js'
				entries['./scripts/build-sitemap'] = './scripts/build-sitemap.js'
				return entries
			}
		}
	}
})

The build-sitemap.js file gets compiled through Webpack using Next's compilation chain & gets outputted at .next/server/scripts/build-sitemap.js

My build-sitemap.js file looks like:

import { getAllDraftEssays } from '@/utils/essay/getAllEssayPreviews'
import { getAllDraftTutorials } from '@/utils/tutorial/getAllTutorialPreviews'

const draftPosts = [...getAllDraftEssays(), ...getAllDraftTutorials()].map(
  (p) => `/${p.slug}`
)

export default {
  siteUrl: 'https://akshaykadam.com',
  generateRobotsTxt: true,
  sitemapSize: 7000,
  exclude: draftPosts,
}

module.exports wasn't working so I used export default.

getAllDraftEssays looks like:

export const getAllDraftEssays = (): Array<{
	slug: string
	module: any
}> =>
	importAll(require.context('../../pages/essay/?rss', true, /\.mdx$/))
		.filter((p: Post) => !p.module.meta.published)
		.sort((a: Post, b: Post) => dateSortDesc(a.module.meta.date, b.module.meta.date))

Technically, this should output sitemap.xml correctly & robots.txt should be generated. I'm not sure why it isn't happening but would love to build it in my Next's compilation chain as it can generate Sitemap dynamically by using some frontmatter fields like published or draft :)

Support xml-stylesheet xsl

Is your feature request related to a problem? Please describe.

Our generated sitemap (https://reachdigital.nl/sitemap.xml) currently includes a <?xml-stylesheet type="text/xsl" href="https://www.reachdigital.nl/sitemap.xsl"?> which gives the sitemap a pretty renderer.

Describe the solution you'd like
We'd like to be able to provide sitemap.xsl as a stylesheet option.

Describe alternatives you've considered
none

Additional context
none

Customize path to sitemap xml file

Is your feature request related to a problem? Please describe.

Describe the solution you'd like
On some projects we need to have customized urls to sitemap.xml file, for instance custom-sitemap.xml

Describe alternatives you've considered

Additional context

Allow alternate links for hreflang / canonical

Hey, first of all thanks for your nice plugin!

I would need to tell the crawlers that a specific dynamic page is available in a different language:
https://developers.google.com/search/docs/advanced/crawling/localized-versions

<url xmlns="http://www.sitemaps.org/schemas/sitemap/0.9">
    <loc>https://mypage.de</loc>
    <lastmod>2021-04-20T18:45:49+01:00</lastmod>
    <priority>1.0</priority>
    <xhtml:link xmlns:xhtml="http://www.w3.org/1999/xhtml" rel="alternate" hreflang="de" href="https://mypage.de/en" />
    <xhtml:link xmlns:xhtml="http://www.w3.org/1999/xhtml" rel="alternate" hreflang="es" href="https://mypage.de/es" />
  </url>

Is this possible with the current version? I have not found it in your fields example:

const fields = [
    {
      loc: 'https://example.com', // Absolute url
      lastmod: new Date().toISOString(),
      // changefreq
      // priority
    },
    {
      loc: 'https://example.com/dynamic-path-2', // Absolute url
      lastmod: new Date().toISOString(),
      // changefreq
      // priority
    },
  ]

Sitemap index

Is your feature request related to a problem? Please describe.
I want to add an index sitemap with all the children urls inside as detailed here: https://developers.google.com/search/docs/advanced/sitemaps/large-sitemaps

Describe the solution you'd like
I would like a new option like index: true to use the first /sitemap.xml as index.

The default locale folder is not removed from url generated in the sitemap

Describe the bug
When I generate a sitemap with a default locale (I'm using Next 10), the urls in the sitemap have the default locale in the path but the real urls don't have one.

https://nextjs.org/docs/advanced-features/i18n-routing

eg:

• This url don't have a /fr folder: https://lafondcpa.ca/nous-joindre
• In the sitemap, the generated url is https://lafondcpa.ca/fr/nous-joindre

To Reproduce
Use a default locale in your next.config.js, and generate the sitemap with npm run build.

next.config.js

 i18n: {
    // These are all the locales you want to support in
    // your application
    locales: ["fr"],
    // This is the default locale you want to be used when visiting
    // a non-locale prefixed path e.g. `/hello`
    defaultLocale: "fr",
  }

Expected behavior
I expect the urls generated by the sitemap from the default locale to not have the locale in the path.

Splitting sitemap based on locale

I was wondering how we would go about splitting the sitemap with locales?

so instead of just a sitemap-index.xml:

• sitemap-index.xml (with references to the following languages indexes)
• sitemap-nl-index.xml
• sitemap-fr-index.xml
• ....

Inject environment variables

Use case: I have NEXT_PUBLIC_DEPLOYMENT_URL. In order to read it I have to use manually dotenv, which is not ideal.

Working Example For Generating dynamic/server-side sitemaps

Is your feature request related to a problem? Please describe.
Yes, I don't seem to figure out how to generate dynamic server side sitemaps as the docs are not clear and no working example exists on the same.

Describe the solution you'd like
Providing a working example of this would be great.

Configurable priority and changefreq per URL

Is it possible to configure priority and changefreq per URL instead of one default setting for all entries?

Allow transform Promises

Is your feature request related to a problem? Please describe.
I need to pull lastmod from an external source

Describe the solution you'd like
Allow the transform property to be defined as a (config, path) => Promise< ISitemapFiled >

Postbuild opening config file instead of running next-sitemap

Hi!

I've installed next-sitemap (1.2.26) as a dev dependency, and added it to package.json: "postbuild": "next-sitemap".=
However, when I build, the postbuild just opens the config file (next-sitemap.js) instead of executing next-sitemap.

Any ideas how to solve this?

Regards, TJ

Any way to use `frontmatter` data like `published: true` field to generate sitemap?

Is your feature request related to a problem? Please describe.

I have a published field in my .mdx file like:

export const meta = {
	title: 'Hello World',
	date: '2020-09-11T14:54:37.229Z',
	tags: ['hello', 'world'],
	author: 'Akshay Kadam',
	published: false,
}

# Hello World

This is my first blog post, not yet ready to be published

Describe the solution you'd like
I want to not add the URL to the sitemap because it's published: false

Describe alternatives you've considered
To use glob pattern & then read the .mdx file & finally use transform function to add individually.

Additional context
I think this module can also have include function just like exclude function so we can add the URLs to Sitemap directly

How to make sure that my dynamic/server-side sitemap is only generated once a day?

Hello,

first of all, thanks a lot for creating and maintaining this project!

For our application, we create both a static sitemap of a couple of static pages and a dynamic sitemap of pages created from a CMS.

I followed the guide for creating a dynamic sitemap under /pages/server-sitemap.xml/index.tsx by fetching all the dynamic content from the CMS. During testing, I realized that the dynamic sitemap is rendered from scratch whenever the sitemap is accessed. We have caching in place for our content-fetching but I would still like only generate the sitemap daily.

Is there an option for this already that I missed or is such a feature planned for the future?

Best regards,
Jonas

Support for Catch All Routes

Slug with underscore is removed because naive Regexp in isNextInternalUrl

Steps to reproduce the behavior:
In any route I assume, but in my particular case, using getStaticPaths, the prerender manifest creates routes such as 'pineapple_basil_smash', this causes the routes to be removed in createUrlSet

The test for isNextInternal needs to be more specific and not automatically remove routes with an underscore.

The initial improvement would be to just test for /_app , /_error and not for just the underscore. Then testing for routes with brackets.

If I find the time, maybe I can take a crack at a fix if you are willing to accept a PR, didn't see anything in the README about contributing.

`next export` error

Describe the bug
Error: Error for page /news-sitemap.xml: pages with getServerSideProps can not be exported. See more info here: https://nextjs.org/docs/messages/gssp-export

To Reproduce
Steps to reproduce the behavior:

1. 
2. next build && next export

Expected behavior

Screenshots

Error: Error for page /news-sitemap.xml: pages with getServerSideProps can not be exported. See more info here: https://nextjs.org/docs/messages/gssp-export

Additional context

npm i error

Describe the bug

feature request: add optional parameter to point to a config file

my project uses typescript and next-mdx-remote so some things like modified date are added to my frontmatter at build time. i can add the functionality either tonight, tomorrow, or sometime next week.

Incremental sitemap generation

What if I add a new page through my CMS, but don't do a new build. Instead, I use ISR with fallback: true. This would result in a newly statically generated page on the fly (without running a full build).

What would be the best way to trigger a new sitemap build? Or maybe a better solution would be a way to also incrementally add this page to the sitemap?

changing sitemap filename

its not a bug, i'd like to change output file name, is there any way i can do this right now?

Ability to customize XML Template

It could be very handy if it was possible for the user to override the basic XML Template. It could help us add XMLStylesheet or custom namespaces to our <urlset>.

In the withXMLTemplate.ts we are taking the default template. If this variable could be over-writable in the config file it could allow the user to customize the template.

allow to specify paths to ignore

I would like to specify an array of urls, that should be ignored when generating the sitemap

Is there a way to generate a sitemap when using a custom dist folder?

Describe the bug

In our Next.js project we have defined a custom dist folder. Here is the content of next.config.js

module.exports = {
    distDir: 'build',
};

As in #91 this is triggering an error in node_modules/next-sitemap/dist/cjs/url/create-url-set/index.js

To Reproduce

Steps to reproduce the behavior:

1. Run next build in build and next-sitemap in postbuild
2. Wait for it to build
3. See error

Expected behavior

Should generate a manifest.

=> So my question is: is there an actual way to use this Npm package if a custom dist folder has been defined?