> ## Documentation Index
> Fetch the complete documentation index at: https://docs.kibocommerce.com/llms.txt
> Use this file to discover all available pages before exploring further.

# Generate a Sitemap for a Headless Storefront

> Build a complete sitemap.xml for a headless Kibo storefront using the Random Access Cursor, Search Products, and Category Tree APIs.

A `sitemap.xml` file gives search engine crawlers an index of the pages on your site. In a headless Kibo storefront, your front-end application generates the sitemap, not the platform. If you only need to replace the sitemap on a Kibo-hosted storefront, see [Upload a Custom Sitemap](https://docs.kibocommerce.com/pages/file-and-image-management.md#upload-a-custom-sitemap) instead.

This guide shows how to build a sitemap from Kibo's Storefront Catalog APIs. It uses a **sitemap index** that links to sub-sitemaps for categories and for batches of products, so it covers the whole catalog even when the catalog holds hundreds of thousands of products.

## Sitemap Architecture

A sitemap for a large catalog uses the index format from the [Sitemap Protocol](https://www.sitemaps.org/protocol.html):

```text theme={null}
sitemap.xml (index)
├── categories.xml
└── products_batch_[n].xml
```

* **The root `sitemap.xml`** is a `sitemapindex` document that lists the URL of every sub-sitemap.
* **The categories sub-sitemap** is a `urlset` document with one entry per category page.
* **Each product batch sub-sitemap** is a `urlset` document with the product page URLs for one slice of the catalog. Splitting products into batches keeps each sub-sitemap within the protocol's [50,000 URL and 50 MB limit](https://www.sitemaps.org/protocol.html#index).

Example sitemap index:

```xml theme={null}
<?xml version="1.0" encoding="UTF-8"?>
<sitemapindex xmlns="http://www.sitemaps.org/schemas/sitemap/0.9">
  <sitemap>
    <loc>https://www.example.com/sitemap.xml/categories</loc>
  </sitemap>
  <sitemap>
    <loc>https://www.example.com/sitemap.xml/productBatch/rcm_cXVlcnk9Jmxvd2VyPTEmdXBwZXI9MjAwMCZzdGFydD0wJnNpemU9MjAwMCZmaWx0ZXI9</loc>
  </sitemap>
  <sitemap>
    <loc>https://www.example.com/sitemap.xml/productBatch/rcm_cXVlcnk9Jmxvd2VyPTIwMDEmdXBwZXI9NDAwMCZzdGFydD0wJnNpemU9MjAwMCZmaWx0ZXI9</loc>
  </sitemap>
</sitemapindex>
```

## Key APIs

| API | Endpoint | Purpose |
| - | - | - |
| [Get Random Access Cursors](https://docs.kibocommerce.com/api-reference/productsearch/gets-random-access-cursors.md) | `GET /api/commerce/catalog/storefront/productsearch/randomAccessCursor` | Split the catalog into stable batches |
| [Search Products](https://docs.kibocommerce.com/api-reference/productsearch/search-products.md) | `GET /api/commerce/catalog/storefront/productsearch/search` | Retrieve the products in each batch |
| [Get Category Tree](https://docs.kibocommerce.com/api-reference/categories/get-category-tree.md) | `GET /api/commerce/catalog/storefront/categories/tree` | Retrieve the category hierarchy |

### Why Use Random Access Cursors

Paging through search results with `startIndex` and `pageSize` can **miss products** in a large catalog. When products are added, removed, or reordered between calls, items can be skipped or returned twice.

The Random Access Cursor API returns a set of stable cursor marks in a single call. Each cursor mark covers a fixed slice of the catalog, and you can retrieve each slice independently, in any order or in parallel, without missing products.

> [!NOTE]
>
> When you call the Random Access Cursor API, `pageSize` can be up to **2,000**. This sets how many products each cursor mark covers. For example, a catalog of 100,000 products with `pageSize=2000` returns about 50 cursor marks.

## Step 1: Generate the Sitemap Index

Call the Random Access Cursor API to get the cursor marks for your catalog. Each cursor mark becomes one product batch sub-sitemap.

```text theme={null}
GET /api/commerce/catalog/storefront/productsearch/randomAccessCursor?pageSize=2000
```

| Parameter | Description | Required |
| - | - | - |
| `pageSize` | Batch size, up to 2,000. Sets how many products each cursor mark covers. | No |
| `query` | Search terms. Only matching products are included in the cursors. | No |
| `filter` | Filter expression. Only matching products are included in the cursors. | No |

Response:

```json theme={null}
{
  "cursorMarks": [
    "rcm_cXVlcnk9Jmxvd2VyPTEwMDAwMSZ1cHBlcj0xMDIwMDAmc3RhcnQ9MCZzaXplPTIwMDAmZmlsdGVyPQ==",
    "rcm_cXVlcnk9Jmxvd2VyPTEwMjAwMSZ1cHBlcj0xMDQwMDAmc3RhcnQ9MCZzaXplPTIwMDAmZmlsdGVyPQ==",
    "rcm_cXVlcnk9Jmxvd2VyPTEwNDAwMSZ1cHBlcj0xMDYwMDAmc3RhcnQ9MCZzaXplPTIwMDAmZmlsdGVyPQ=="
  ]
}
```

Treat cursor marks as opaque values. Each one encodes a range of internal product IDs, not a position in the catalog, so the first mark does not normally start at 1, and the ranges are not the same width. Do not calculate product counts from them.

A `filter` that matches nothing returns an empty `cursorMarks` array. Handle that case rather than publishing an index with no product sitemaps.

> [!WARNING]
>
> Cursor marks are base64 encoded and can contain `/` and `+` characters. A `/` inside a mark splits a URL path segment. An unfiltered sitemap does not produce these characters, but one scoped with `query` or `filter` can. Either encode each mark with `encodeURIComponent` when you build the index, or use a catch-all route that rejoins the path segments, as shown in the [Next.js example](#next-js-implementation-example).

Build the sitemap index from the cursor marks:

```xml theme={null}
<?xml version="1.0" encoding="UTF-8"?>
<sitemapindex xmlns="http://www.sitemaps.org/schemas/sitemap/0.9">
  <sitemap>
    <loc>https://www.example.com/sitemap.xml/categories</loc>
  </sitemap>
  <sitemap>
    <loc>https://www.example.com/sitemap.xml/productBatch/{cursorMark_1}</loc>
  </sitemap>
  <sitemap>
    <loc>https://www.example.com/sitemap.xml/productBatch/{cursorMark_2}</loc>
  </sitemap>
</sitemapindex>
```

## Step 2: Generate the Category Sub-Sitemap

Call the Category Tree API to get the storefront categories, then build a `urlset` with one entry per category.

```text theme={null}
GET /api/commerce/catalog/storefront/categories/tree
```

| Parameter | Description | Required |
| - | - | - |
| `includeAttributes` | If `true`, includes category attributes in the response. Default: `false`. | No |
| `responseFields` | Limits which fields are returned. | No |

Response (abbreviated):

```json theme={null}
{
  "items": [
    {
      "categoryCode": "1",
      "categoryId": 1,
      "content": {
        "name": "Tools",
        "slug": "tools",
        "pageTitle": "Tools & Hardware",
        "description": "Shop all tools and hardware."
      },
      "isDisplayed": true,
      "childrenCategories": []
    }
  ],
  "totalCount": 50
}
```

> [!NOTE]
>
> `items` holds only the top level of the tree. Subcategories are nested under `childrenCategories` at every level. Walk the tree recursively; iterating over `items` alone publishes only your root categories. Skip categories where `isDisplayed` is `false` so that hidden categories are not published.

Use `content.slug` and `categoryCode` to build each category URL:

```xml theme={null}
<?xml version="1.0" encoding="UTF-8"?>
<urlset xmlns="http://www.sitemaps.org/schemas/sitemap/0.9">
  <url>
    <loc>https://www.example.com/category/tools/1</loc>
  </url>
  <url>
    <loc>https://www.example.com/category/paint/2</loc>
  </url>
</urlset>
```

The URL pattern (`/category/{slug}/{categoryCode}` here) depends on your storefront's routes. Adjust it to match your application.

## Step 3: Generate the Product Batch Sub-Sitemaps

For each cursor mark from Step 1, call the Search Products API with the `cursorMark` parameter to get the products in that batch.

```text theme={null}
GET /api/commerce/catalog/storefront/productsearch/search?cursorMark={cursorMark}&pageSize=200
```

| Parameter | Description | Required |
| - | - | - |
| `cursorMark` | A cursor mark from the Random Access Cursor API, or the `nextCursorMark` from a previous search response. | Yes |
| `pageSize` | Number of products to return, up to **200**. Default: 12. | No |
| `responseFields` | Limits which fields are returned. Use it to keep responses small. | No |

Response (abbreviated):

```json theme={null}
{
  "totalCount": 187969,
  "startIndex": 0,
  "pageSize": 200,
  "pageCount": 18797,
  "nextCursorMark": "rcm_...",
  "items": [
    {
      "productCode": "12787",
      "content": {
        "productName": "Cordless Drill",
        "seoFriendlyUrl": "cordless-drill"
      }
    }
  ]
}
```

> [!WARNING]
>
> One search call does not return a whole batch. A cursor covers up to 2,000 products, but the search API returns at most 200 per call. Follow the `nextCursorMark` chain: pass it as `cursorMark` on the next call, and stop when it is `null`, unchanged from the cursor you sent, or the response has no items. Without this loop, the sitemap publishes at most 200 of every 2,000 products.

Use `content.seoFriendlyUrl` and `productCode` to build each product URL:

```xml theme={null}
<?xml version="1.0" encoding="UTF-8"?>
<urlset xmlns="http://www.sitemaps.org/schemas/sitemap/0.9">
  <url>
    <loc>https://www.example.com/product/cordless-drill/12787</loc>
  </url>
  <url>
    <loc>https://www.example.com/product/hammer/12788</loc>
  </url>
</urlset>
```

The URL pattern (`/product/{seoFriendlyUrl}/{productCode}` here) depends on your storefront's routes. `seoFriendlyUrl` and the category `slug` are only populated when SEO content has been entered, so fall back to the product or category code rather than publishing `undefined` in a URL.

Escape every value you insert into the XML. Slugs and SEO URLs are entered by merchandisers and can contain `&`, `<`, or quotes, and a single unescaped `&` makes the whole sitemap invalid.

## GraphQL Alternative

If your storefront uses the Kibo [GraphQL API](https://docs.kibocommerce.com/pages/graphql.md), you can implement the same logic with these root query fields:

| Field | Arguments | Returns |
| - | - | - |
| `productSearchRandomAccessCursor` | `query`, `filter`, `pageSize` | `ProductSearchRandomAccessCursor` with `cursorMarks` |
| `productSearch` | `cursorMark`, `pageSize`, `query`, `filter`, and others | `ProductSearchResult` with `nextCursorMark`, `totalCount`, `items` |
| `categoriesTree` | `includeAttributes` | `CategoryCollection` with `totalCount`, `items` |
| `categories` | `filter`, `startIndex`, `pageSize`, `sortBy` | `CategoryPagedCollection`, a flat paged list, not the tree |

Use `categoriesTree`, not `categories`, to get the hierarchy in one call.

Get random access cursors:

```graphql theme={null}
query {
  productSearchRandomAccessCursor(pageSize: 2000) {
    cursorMarks
  }
}
```

Search products with a cursor mark, following `nextCursorMark` as described in Step 3:

```graphql theme={null}
query SearchProducts($cursorMark: String) {
  products: productSearch(cursorMark: $cursorMark, pageSize: 200) {
    totalCount
    nextCursorMark
    items {
      productCode
      content {
        productName
        seoFriendlyUrl
      }
    }
  }
}
```

Get the category tree:

```graphql theme={null}
query GetCategoryTree {
  categoriesTree {
    items {
      categoryCode
      isDisplayed
      content {
        name
        slug
      }
      childrenCategories {
        categoryCode
        isDisplayed
        content {
          name
          slug
        }
      }
    }
  }
}
```

GraphQL cannot express unlimited recursion, so select `childrenCategories` at least as deep as your deepest category level, or use the REST `categories/tree` endpoint, which returns the whole tree in one call.

## Next.js Implementation Example

This example uses the Next.js Pages Router with `getServerSideProps` to generate the XML on each request.

```text theme={null}
src/
  lib/
    sitemap.ts                            Shared XML and API helpers
  pages/
    sitemap.xml.ts                        Sitemap index (root)
    sitemap.xml/
      categories.ts                       Category sub-sitemap
      productBatch/
        [...productBatchId].ts            Product batch sub-sitemap (catch-all route)
```

### Shared Helpers: src/lib/sitemap.ts

```ts theme={null}
// getAccessToken() is your own helper that returns a valid app token and
// refreshes it before it expires. The Kibo TypeScript SDK handles this for you.
import { getAccessToken } from './auth'

export const SITEMAP_NS = 'http://www.sitemaps.org/schemas/sitemap/0.9'

export const escapeXml = (s: string) =>
  s.replace(/&/g, '&amp;')
   .replace(/</g, '&lt;')
   .replace(/>/g, '&gt;')
   .replace(/"/g, '&quot;')
   .replace(/'/g, '&apos;')

export const urlset = (locs: string[]) => `<?xml version="1.0" encoding="UTF-8"?>
<urlset xmlns="${SITEMAP_NS}">
${locs.map((loc) => `  <url>
    <loc>${escapeXml(loc)}</loc>
  </url>`).join('\n')}
</urlset>`

export const sitemapIndex = (locs: string[]) => `<?xml version="1.0" encoding="UTF-8"?>
<sitemapindex xmlns="${SITEMAP_NS}">
${locs.map((loc) => `  <sitemap>
    <loc>${escapeXml(loc)}</loc>
  </sitemap>`).join('\n')}
</sitemapindex>`

export const kiboFetch = async (path: string) => {
  const res = await fetch(`${process.env.KIBO_API_BASE_URL}${path}`, {
    headers: { Authorization: `Bearer ${await getAccessToken()}` },
  })
  if (!res.ok) throw new Error(`Kibo ${path} -> ${res.status}`)
  return res.json()
}
```

### Sitemap Index: src/pages/sitemap.xml.ts

```ts theme={null}
import { GetServerSideProps } from 'next'
import { kiboFetch, sitemapIndex } from '../lib/sitemap'

export const getServerSideProps: GetServerSideProps = async ({ res }) => {
  const domain = process.env.CURRENT_DOMAIN
  const { cursorMarks = [] } = await kiboFetch(
    '/api/commerce/catalog/storefront/productsearch/randomAccessCursor?pageSize=2000'
  )

  const locs = [
    `https://${domain}/sitemap.xml/categories`,
    // Encode so that a "/" inside a mark cannot split the path segment.
    ...cursorMarks.map(
      (m: string) => `https://${domain}/sitemap.xml/productBatch/${encodeURIComponent(m)}`
    ),
  ]

  res.setHeader('Content-Type', 'text/xml')
  res.setHeader('Cache-Control', 'public, max-age=3600')
  res.write(sitemapIndex(locs))
  res.end()

  return { props: {} }
}

export default function SiteMap() {
  // getServerSideProps writes the response.
}
```

### Category Sub-Sitemap: src/pages/sitemap.xml/categories.ts

```ts theme={null}
import { GetServerSideProps } from 'next'
import { kiboFetch, urlset } from '../../lib/sitemap'

// items is a tree: recurse through childrenCategories, not just the top level.
const flattenCategories = (items: any[]): any[] =>
  (items || []).flatMap((c) => [c, ...flattenCategories(c.childrenCategories)])

export const getServerSideProps: GetServerSideProps = async ({ res }) => {
  const domain = process.env.CURRENT_DOMAIN
  const tree = await kiboFetch('/api/commerce/catalog/storefront/categories/tree')

  const locs = flattenCategories(tree.items)
    .filter((c) => c.isDisplayed !== false)
    .map((c) => `https://${domain}/category/${c.content?.slug || c.categoryCode}/${c.categoryCode}`)

  res.setHeader('Content-Type', 'text/xml')
  res.setHeader('Cache-Control', 'public, max-age=3600')
  res.write(urlset(locs))
  res.end()

  return { props: {} }
}

export default function SiteMap() {
  // getServerSideProps writes the response.
}
```

### Product Batch Sub-Sitemap: src/pages/sitemap.xml/productBatch/\[...productBatchId].ts

```ts theme={null}
import { GetServerSideProps } from 'next'
import { kiboFetch, urlset } from '../../../lib/sitemap'

const RESPONSE_FIELDS = 'nextCursorMark,items(productCode,content(seoFriendlyUrl))'

// One cursor covers up to 2,000 products, but search returns at most 200 per call,
// so follow the nextCursorMark chain until it ends.
async function fetchAllProductsForCursor(cursorMark: string): Promise<any[]> {
  const items: any[] = []
  let cursor: string | null = cursorMark

  while (cursor) {
    const data = await kiboFetch(
      `/api/commerce/catalog/storefront/productsearch/search` +
        `?cursorMark=${encodeURIComponent(cursor)}&pageSize=200&responseFields=${RESPONSE_FIELDS}`
    )
    const page = data.items || []
    items.push(...page)

    const next = data.nextCursorMark
    cursor = !next || next === cursor || page.length === 0 ? null : next
  }

  return items
}

export const getServerSideProps: GetServerSideProps = async ({ res, query }) => {
  const domain = process.env.CURRENT_DOMAIN
  // Next.js decodes route parameters, so an encoded "/" arrives in one segment.
  // Joining the segments also handles marks that were not encoded.
  const segments = query.productBatchId
  const cursorMark = Array.isArray(segments) ? segments.join('/') : segments || ''

  res.setHeader('Content-Type', 'text/xml')

  if (!cursorMark) {
    res.statusCode = 404
    res.end()
    return { props: {} }
  }

  const products = await fetchAllProductsForCursor(cursorMark)
  const locs = products.map(
    (p) => `https://${domain}/product/${p.content?.seoFriendlyUrl || p.productCode}/${p.productCode}`
  )

  res.setHeader('Cache-Control', 'public, max-age=3600')
  res.write(urlset(locs))
  res.end()

  return { props: {} }
}

export default function SiteMap() {
  // getServerSideProps writes the response.
}
```

## Verify Coverage

Before you submit the sitemap, confirm that it covers the catalog. Call the search API once with `pageSize=1&responseFields=totalCount`, and compare `totalCount` with the total number of `loc` entries across all product batch sub-sitemaps. The numbers should match. A large shortfall almost always means the `nextCursorMark` loop is missing or stops early.

Running the same logic as a one-off script against your tenant, for example with the [TypeScript SDK](https://docs.kibocommerce.com/pages/typescript-sdk.md), is a quick way to check your catalog data (in particular, whether `seoFriendlyUrl` and `slug` are populated) before you build the storefront routes.

## robots.txt

Add a `robots.txt` file to your application's public directory that points crawlers to the sitemap and excludes pages that should not be indexed:

```text theme={null}
User-agent: *
Allow: /
Disallow: /cart*
Disallow: /checkout*
Disallow: /search*
Disallow: /my-account*

Sitemap: https://www.example.com/sitemap.xml
```

Adjust the `Disallow` rules to match your application's routes.

## Best Practices

### Performance

* **Cache sitemap responses.** Cursor marks and the category tree change infrequently. Cache them at the CDN or in your application for 1 to 24 hours to reduce API calls.
* **Generate sub-sitemaps on demand.** Build a product batch sub-sitemap only when a crawler requests it, rather than generating every batch in advance.
* **Request only the fields you need.** Use `responseFields` on search calls to return only `nextCursorMark`, `productCode`, and `content.seoFriendlyUrl`.
* **Expect about ten calls per batch.** With 2,000 products per cursor and 200 per search call, each product batch takes about ten search calls. Limit concurrency if you generate many batches in parallel.

### Completeness

* **Always use random access cursors** to list products. `startIndex` and `pageSize` paging alone can skip products in an active catalog.
* **Follow the `nextCursorMark` chain.** This is the most common reason a Kibo sitemap silently leaves out most of the catalog. See [Step 3](#step-3-generate-the-product-batch-sub-sitemaps).
* **Walk the full category tree.** `categories/tree` returns root categories with subcategories nested in `childrenCategories`.
* **Include other page types.** Consider sub-sitemaps for content pages, blog posts, and other routes in your storefront.

### SEO

* **Add `lastmod` where you can.** Use the product or category `updateDate` to help crawlers decide what to recrawl.
* **Don't spend time tuning `changefreq` and `priority`.** Google ignores both. `lastmod` is the tag that affects recrawling.
* **Submit the sitemap** to Google Search Console and Bing Webmaster Tools after you deploy it.

### Multiple Sites

If you run several storefronts on the same tenant, for example for different locales or brands, generate a separate sitemap for each site. Each site has its own category tree and product set, and is bound to a specific catalog within a master catalog. Scope requests to the correct site with a site-specific host or with the `x-vol-site`, `x-vol-catalog`, and `x-vol-master-catalog` headers. A mismatched combination returns `409 Conflict`.

## Related Resources

* [Get Random Access Cursors](https://docs.kibocommerce.com/api-reference/productsearch/gets-random-access-cursors.md)
* [Search Products](https://docs.kibocommerce.com/api-reference/productsearch/search-products.md)
* [Get Category Tree](https://docs.kibocommerce.com/api-reference/categories/get-category-tree.md)
* [Storefront Catalog API Guide](https://docs.kibocommerce.com/developer-guides/storefront-catalog.md)
* [GraphQL API](https://docs.kibocommerce.com/pages/graphql.md)
* [TypeScript SDK](https://docs.kibocommerce.com/pages/typescript-sdk.md)


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.