Generate a Sitemap for a Headless Storefront

A sitemap.xml file gives search engine crawlers an index of the pages on your site. In a headless Kibo storefront, your front-end application generates the sitemap, not the platform. If you only need to replace the sitemap on a Kibo-hosted storefront, see Upload a Custom Sitemap instead. This guide shows how to build a sitemap from Kibo’s Storefront Catalog APIs. It uses a sitemap index that links to sub-sitemaps for categories and for batches of products, so it covers the whole catalog even when the catalog holds hundreds of thousands of products.

Sitemap Architecture

A sitemap for a large catalog uses the index format from the Sitemap Protocol:

sitemap.xml (index)
├── categories.xml
└── products_batch_[n].xml

The root sitemap.xml is a sitemapindex document that lists the URL of every sub-sitemap.
The categories sub-sitemap is a urlset document with one entry per category page.
Each product batch sub-sitemap is a urlset document with the product page URLs for one slice of the catalog. Splitting products into batches keeps each sub-sitemap within the protocol’s 50,000 URL and 50 MB limit.

Example sitemap index:

<?xml version="1.0" encoding="UTF-8"?>
<sitemapindex xmlns="http://www.sitemaps.org/schemas/sitemap/0.9">
  <sitemap>
    <loc>https://www.example.com/sitemap.xml/categories</loc>
  </sitemap>
  <sitemap>
    <loc>https://www.example.com/sitemap.xml/productBatch/rcm_cXVlcnk9Jmxvd2VyPTEmdXBwZXI9MjAwMCZzdGFydD0wJnNpemU9MjAwMCZmaWx0ZXI9</loc>
  </sitemap>
  <sitemap>
    <loc>https://www.example.com/sitemap.xml/productBatch/rcm_cXVlcnk9Jmxvd2VyPTIwMDEmdXBwZXI9NDAwMCZzdGFydD0wJnNpemU9MjAwMCZmaWx0ZXI9</loc>
  </sitemap>
</sitemapindex>

Key APIs

API	Endpoint	Purpose
Get Random Access Cursors	`GET /api/commerce/catalog/storefront/productsearch/randomAccessCursor`	Split the catalog into stable batches
Search Products	`GET /api/commerce/catalog/storefront/productsearch/search`	Retrieve the products in each batch
Get Category Tree	`GET /api/commerce/catalog/storefront/categories/tree`	Retrieve the category hierarchy

Why Use Random Access Cursors

Paging through search results with startIndex and pageSize can miss products in a large catalog. When products are added, removed, or reordered between calls, items can be skipped or returned twice. The Random Access Cursor API returns a set of stable cursor marks in a single call. Each cursor mark covers a fixed slice of the catalog, and you can retrieve each slice independently, in any order or in parallel, without missing products.

When you call the Random Access Cursor API, pageSize can be up to 2,000. This sets how many products each cursor mark covers. For example, a catalog of 100,000 products with pageSize=2000 returns about 50 cursor marks.

Step 1: Generate the Sitemap Index

Call the Random Access Cursor API to get the cursor marks for your catalog. Each cursor mark becomes one product batch sub-sitemap.

GET /api/commerce/catalog/storefront/productsearch/randomAccessCursor?pageSize=2000

Parameter	Description	Required
`pageSize`	Batch size, up to 2,000. Sets how many products each cursor mark covers.	No
`query`	Search terms. Only matching products are included in the cursors.	No
`filter`	Filter expression. Only matching products are included in the cursors.	No

Response:

{
  "cursorMarks": [
    "rcm_cXVlcnk9Jmxvd2VyPTEwMDAwMSZ1cHBlcj0xMDIwMDAmc3RhcnQ9MCZzaXplPTIwMDAmZmlsdGVyPQ==",
    "rcm_cXVlcnk9Jmxvd2VyPTEwMjAwMSZ1cHBlcj0xMDQwMDAmc3RhcnQ9MCZzaXplPTIwMDAmZmlsdGVyPQ==",
    "rcm_cXVlcnk9Jmxvd2VyPTEwNDAwMSZ1cHBlcj0xMDYwMDAmc3RhcnQ9MCZzaXplPTIwMDAmZmlsdGVyPQ=="
  ]
}

Treat cursor marks as opaque values. Each one encodes a range of internal product IDs, not a position in the catalog, so the first mark does not normally start at 1, and the ranges are not the same width. Do not calculate product counts from them. A filter that matches nothing returns an empty cursorMarks array. Handle that case rather than publishing an index with no product sitemaps.

Cursor marks are base64 encoded and can contain / and + characters. A / inside a mark splits a URL path segment. An unfiltered sitemap does not produce these characters, but one scoped with query or filter can. Either encode each mark with encodeURIComponent when you build the index, or use a catch-all route that rejoins the path segments, as shown in the Next.js example.

Build the sitemap index from the cursor marks:

<?xml version="1.0" encoding="UTF-8"?>
<sitemapindex xmlns="http://www.sitemaps.org/schemas/sitemap/0.9">
  <sitemap>
    <loc>https://www.example.com/sitemap.xml/categories</loc>
  </sitemap>
  <sitemap>
    <loc>https://www.example.com/sitemap.xml/productBatch/{cursorMark_1}</loc>
  </sitemap>
  <sitemap>
    <loc>https://www.example.com/sitemap.xml/productBatch/{cursorMark_2}</loc>
  </sitemap>
</sitemapindex>

Step 2: Generate the Category Sub-Sitemap

Call the Category Tree API to get the storefront categories, then build a urlset with one entry per category.

GET /api/commerce/catalog/storefront/categories/tree

Parameter	Description	Required
`includeAttributes`	If `true`, includes category attributes in the response. Default: `false`.	No
`responseFields`	Limits which fields are returned.	No

Response (abbreviated):

{
  "items": [
    {
      "categoryCode": "1",
      "categoryId": 1,
      "content": {
        "name": "Tools",
        "slug": "tools",
        "pageTitle": "Tools & Hardware",
        "description": "Shop all tools and hardware."
      },
      "isDisplayed": true,
      "childrenCategories": []
    }
  ],
  "totalCount": 50
}

items holds only the top level of the tree. Subcategories are nested under childrenCategories at every level. Walk the tree recursively; iterating over items alone publishes only your root categories. Skip categories where isDisplayed is false so that hidden categories are not published.

Use content.slug and categoryCode to build each category URL:

<?xml version="1.0" encoding="UTF-8"?>
<urlset xmlns="http://www.sitemaps.org/schemas/sitemap/0.9">
  <url>
    <loc>https://www.example.com/category/tools/1</loc>
  </url>
  <url>
    <loc>https://www.example.com/category/paint/2</loc>
  </url>
</urlset>

The URL pattern (/category/{slug}/{categoryCode} here) depends on your storefront’s routes. Adjust it to match your application.

Step 3: Generate the Product Batch Sub-Sitemaps

For each cursor mark from Step 1, call the Search Products API with the cursorMark parameter to get the products in that batch.

GET /api/commerce/catalog/storefront/productsearch/search?cursorMark={cursorMark}&pageSize=200

Parameter	Description	Required
`cursorMark`	A cursor mark from the Random Access Cursor API, or the `nextCursorMark` from a previous search response.	Yes
`pageSize`	Number of products to return, up to 200. Default: 12.	No
`responseFields`	Limits which fields are returned. Use it to keep responses small.	No

Response (abbreviated):

{
  "totalCount": 187969,
  "startIndex": 0,
  "pageSize": 200,
  "pageCount": 18797,
  "nextCursorMark": "rcm_...",
  "items": [
    {
      "productCode": "12787",
      "content": {
        "productName": "Cordless Drill",
        "seoFriendlyUrl": "cordless-drill"
      }
    }
  ]
}

One search call does not return a whole batch. A cursor covers up to 2,000 products, but the search API returns at most 200 per call. Follow the nextCursorMark chain: pass it as cursorMark on the next call, and stop when it is null, unchanged from the cursor you sent, or the response has no items. Without this loop, the sitemap publishes at most 200 of every 2,000 products.

Use content.seoFriendlyUrl and productCode to build each product URL:

<?xml version="1.0" encoding="UTF-8"?>
<urlset xmlns="http://www.sitemaps.org/schemas/sitemap/0.9">
  <url>
    <loc>https://www.example.com/product/cordless-drill/12787</loc>
  </url>
  <url>
    <loc>https://www.example.com/product/hammer/12788</loc>
  </url>
</urlset>

The URL pattern (/product/{seoFriendlyUrl}/{productCode} here) depends on your storefront’s routes. seoFriendlyUrl and the category slug are only populated when SEO content has been entered, so fall back to the product or category code rather than publishing undefined in a URL. Escape every value you insert into the XML. Slugs and SEO URLs are entered by merchandisers and can contain &, <, or quotes, and a single unescaped & makes the whole sitemap invalid.

GraphQL Alternative

If your storefront uses the Kibo GraphQL API, you can implement the same logic with these root query fields:

Field	Arguments	Returns
`productSearchRandomAccessCursor`	`query`, `filter`, `pageSize`	`ProductSearchRandomAccessCursor` with `cursorMarks`
`productSearch`	`cursorMark`, `pageSize`, `query`, `filter`, and others	`ProductSearchResult` with `nextCursorMark`, `totalCount`, `items`
`categoriesTree`	`includeAttributes`	`CategoryCollection` with `totalCount`, `items`
`categories`	`filter`, `startIndex`, `pageSize`, `sortBy`	`CategoryPagedCollection`, a flat paged list, not the tree

Use categoriesTree, not categories, to get the hierarchy in one call. Get random access cursors:

query {
  productSearchRandomAccessCursor(pageSize: 2000) {
    cursorMarks
  }
}

Search products with a cursor mark, following nextCursorMark as described in Step 3:

query SearchProducts($cursorMark: String) {
  products: productSearch(cursorMark: $cursorMark, pageSize: 200) {
    totalCount
    nextCursorMark
    items {
      productCode
      content {
        productName
        seoFriendlyUrl
      }
    }
  }
}

Get the category tree:

query GetCategoryTree {
  categoriesTree {
    items {
      categoryCode
      isDisplayed
      content {
        name
        slug
      }
      childrenCategories {
        categoryCode
        isDisplayed
        content {
          name
          slug
        }
      }
    }
  }
}

GraphQL cannot express unlimited recursion, so select childrenCategories at least as deep as your deepest category level, or use the REST categories/tree endpoint, which returns the whole tree in one call.

Next.js Implementation Example

This example uses the Next.js Pages Router with getServerSideProps to generate the XML on each request.

src/
  lib/
    sitemap.ts                            Shared XML and API helpers
  pages/
    sitemap.xml.ts                        Sitemap index (root)
    sitemap.xml/
      categories.ts                       Category sub-sitemap
      productBatch/
        [...productBatchId].ts            Product batch sub-sitemap (catch-all route)

Shared Helpers: src/lib/sitemap.ts

// getAccessToken() is your own helper that returns a valid app token and
// refreshes it before it expires. The Kibo TypeScript SDK handles this for you.
import { getAccessToken } from './auth'

export const SITEMAP_NS = 'http://www.sitemaps.org/schemas/sitemap/0.9'

export const escapeXml = (s: string) =>
  s.replace(/&/g, '&amp;')
   .replace(/</g, '&lt;')
   .replace(/>/g, '&gt;')
   .replace(/"/g, '&quot;')
   .replace(/'/g, '&apos;')

export const urlset = (locs: string[]) => `<?xml version="1.0" encoding="UTF-8"?>
<urlset xmlns="${SITEMAP_NS}">
${locs.map((loc) => `  <url>
    <loc>${escapeXml(loc)}</loc>
  </url>`).join('\n')}
</urlset>`

export const sitemapIndex = (locs: string[]) => `<?xml version="1.0" encoding="UTF-8"?>
<sitemapindex xmlns="${SITEMAP_NS}">
${locs.map((loc) => `  <sitemap>
    <loc>${escapeXml(loc)}</loc>
  </sitemap>`).join('\n')}
</sitemapindex>`

export const kiboFetch = async (path: string) => {
  const res = await fetch(`${process.env.KIBO_API_BASE_URL}${path}`, {
    headers: { Authorization: `Bearer ${await getAccessToken()}` },
  })
  if (!res.ok) throw new Error(`Kibo ${path} -> ${res.status}`)
  return res.json()
}

Sitemap Index: src/pages/sitemap.xml.ts

import { GetServerSideProps } from 'next'
import { kiboFetch, sitemapIndex } from '../lib/sitemap'

export const getServerSideProps: GetServerSideProps = async ({ res }) => {
  const domain = process.env.CURRENT_DOMAIN
  const { cursorMarks = [] } = await kiboFetch(
    '/api/commerce/catalog/storefront/productsearch/randomAccessCursor?pageSize=2000'
  )

  const locs = [
    `https://${domain}/sitemap.xml/categories`,
    // Encode so that a "/" inside a mark cannot split the path segment.
    ...cursorMarks.map(
      (m: string) => `https://${domain}/sitemap.xml/productBatch/${encodeURIComponent(m)}`
    ),
  ]

  res.setHeader('Content-Type', 'text/xml')
  res.setHeader('Cache-Control', 'public, max-age=3600')
  res.write(sitemapIndex(locs))
  res.end()

  return { props: {} }
}

export default function SiteMap() {
  // getServerSideProps writes the response.
}

Category Sub-Sitemap: src/pages/sitemap.xml/categories.ts

import { GetServerSideProps } from 'next'
import { kiboFetch, urlset } from '../../lib/sitemap'

// items is a tree: recurse through childrenCategories, not just the top level.
const flattenCategories = (items: any[]): any[] =>
  (items || []).flatMap((c) => [c, ...flattenCategories(c.childrenCategories)])

export const getServerSideProps: GetServerSideProps = async ({ res }) => {
  const domain = process.env.CURRENT_DOMAIN
  const tree = await kiboFetch('/api/commerce/catalog/storefront/categories/tree')

  const locs = flattenCategories(tree.items)
    .filter((c) => c.isDisplayed !== false)
    .map((c) => `https://${domain}/category/${c.content?.slug || c.categoryCode}/${c.categoryCode}`)

  res.setHeader('Content-Type', 'text/xml')
  res.setHeader('Cache-Control', 'public, max-age=3600')
  res.write(urlset(locs))
  res.end()

  return { props: {} }
}

export default function SiteMap() {
  // getServerSideProps writes the response.
}

Product Batch Sub-Sitemap: src/pages/sitemap.xml/productBatch/[…productBatchId].ts

import { GetServerSideProps } from 'next'
import { kiboFetch, urlset } from '../../../lib/sitemap'

const RESPONSE_FIELDS = 'nextCursorMark,items(productCode,content(seoFriendlyUrl))'

// One cursor covers up to 2,000 products, but search returns at most 200 per call,
// so follow the nextCursorMark chain until it ends.
async function fetchAllProductsForCursor(cursorMark: string): Promise<any[]> {
  const items: any[] = []
  let cursor: string | null = cursorMark

  while (cursor) {
    const data = await kiboFetch(
      `/api/commerce/catalog/storefront/productsearch/search` +
        `?cursorMark=${encodeURIComponent(cursor)}&pageSize=200&responseFields=${RESPONSE_FIELDS}`
    )
    const page = data.items || []
    items.push(...page)

    const next = data.nextCursorMark
    cursor = !next || next === cursor || page.length === 0 ? null : next
  }

  return items
}

export const getServerSideProps: GetServerSideProps = async ({ res, query }) => {
  const domain = process.env.CURRENT_DOMAIN
  // Next.js decodes route parameters, so an encoded "/" arrives in one segment.
  // Joining the segments also handles marks that were not encoded.
  const segments = query.productBatchId
  const cursorMark = Array.isArray(segments) ? segments.join('/') : segments || ''

  res.setHeader('Content-Type', 'text/xml')

  if (!cursorMark) {
    res.statusCode = 404
    res.end()
    return { props: {} }
  }

  const products = await fetchAllProductsForCursor(cursorMark)
  const locs = products.map(
    (p) => `https://${domain}/product/${p.content?.seoFriendlyUrl || p.productCode}/${p.productCode}`
  )

  res.setHeader('Cache-Control', 'public, max-age=3600')
  res.write(urlset(locs))
  res.end()

  return { props: {} }
}

export default function SiteMap() {
  // getServerSideProps writes the response.
}

Verify Coverage

Before you submit the sitemap, confirm that it covers the catalog. Call the search API once with pageSize=1&responseFields=totalCount, and compare totalCount with the total number of loc entries across all product batch sub-sitemaps. The numbers should match. A large shortfall almost always means the nextCursorMark loop is missing or stops early. Running the same logic as a one-off script against your tenant, for example with the TypeScript SDK, is a quick way to check your catalog data (in particular, whether seoFriendlyUrl and slug are populated) before you build the storefront routes.

robots.txt

Add a robots.txt file to your application’s public directory that points crawlers to the sitemap and excludes pages that should not be indexed:

User-agent: *
Allow: /
Disallow: /cart*
Disallow: /checkout*
Disallow: /search*
Disallow: /my-account*

Sitemap: https://www.example.com/sitemap.xml

Adjust the Disallow rules to match your application’s routes.

Best Practices

Performance

Cache sitemap responses. Cursor marks and the category tree change infrequently. Cache them at the CDN or in your application for 1 to 24 hours to reduce API calls.
Generate sub-sitemaps on demand. Build a product batch sub-sitemap only when a crawler requests it, rather than generating every batch in advance.
Request only the fields you need. Use responseFields on search calls to return only nextCursorMark, productCode, and content.seoFriendlyUrl.
Expect about ten calls per batch. With 2,000 products per cursor and 200 per search call, each product batch takes about ten search calls. Limit concurrency if you generate many batches in parallel.

Completeness

Always use random access cursors to list products. startIndex and pageSize paging alone can skip products in an active catalog.
Follow the nextCursorMark chain. This is the most common reason a Kibo sitemap silently leaves out most of the catalog. See Step 3.
Walk the full category tree. categories/tree returns root categories with subcategories nested in childrenCategories.
Include other page types. Consider sub-sitemaps for content pages, blog posts, and other routes in your storefront.

SEO

Add lastmod where you can. Use the product or category updateDate to help crawlers decide what to recrawl.
Don’t spend time tuning changefreq and priority. Google ignores both. lastmod is the tag that affects recrawling.
Submit the sitemap to Google Search Console and Bing Webmaster Tools after you deploy it.

Multiple Sites

If you run several storefronts on the same tenant, for example for different locales or brands, generate a separate sitemap for each site. Each site has its own category tree and product set, and is bound to a specific catalog within a master catalog. Scope requests to the correct site with a site-specific host or with the x-vol-site, x-vol-catalog, and x-vol-master-catalog headers. A mismatched combination returns 409 Conflict.

Developers

API reference

AI Tools

Generate a Sitemap for a Headless Storefront

Sitemap Architecture

Key APIs

Why Use Random Access Cursors

Step 1: Generate the Sitemap Index

Step 2: Generate the Category Sub-Sitemap

Step 3: Generate the Product Batch Sub-Sitemaps

GraphQL Alternative

Next.js Implementation Example

Shared Helpers: src/lib/sitemap.ts

Sitemap Index: src/pages/sitemap.xml.ts

Category Sub-Sitemap: src/pages/sitemap.xml/categories.ts

Product Batch Sub-Sitemap: src/pages/sitemap.xml/productBatch/[…productBatchId].ts

Verify Coverage

robots.txt

Best Practices

Performance

Completeness

SEO

Multiple Sites

​Sitemap Architecture

​Key APIs

​Why Use Random Access Cursors

​Step 1: Generate the Sitemap Index

​Step 2: Generate the Category Sub-Sitemap

​Step 3: Generate the Product Batch Sub-Sitemaps

​GraphQL Alternative

​Next.js Implementation Example

​Shared Helpers: src/lib/sitemap.ts

​Sitemap Index: src/pages/sitemap.xml.ts

​Category Sub-Sitemap: src/pages/sitemap.xml/categories.ts

​Product Batch Sub-Sitemap: src/pages/sitemap.xml/productBatch/[…productBatchId].ts

​Verify Coverage

​robots.txt

​Best Practices

​Performance

​Completeness

​SEO

​Multiple Sites

​Related Resources

Sitemap Architecture

Key APIs

Why Use Random Access Cursors

Step 1: Generate the Sitemap Index

Step 2: Generate the Category Sub-Sitemap

Step 3: Generate the Product Batch Sub-Sitemaps

GraphQL Alternative

Next.js Implementation Example

Shared Helpers: src/lib/sitemap.ts

Sitemap Index: src/pages/sitemap.xml.ts

Category Sub-Sitemap: src/pages/sitemap.xml/categories.ts

Product Batch Sub-Sitemap: src/pages/sitemap.xml/productBatch/[…productBatchId].ts

Verify Coverage

robots.txt

Best Practices

Performance

Completeness

SEO

Multiple Sites

Related Resources