Sitemap generation

Learn how to generate XML sitemaps for your ButterCMS-powered site to improve search engine discovery and indexation.

Sitemap generation

Why sitemaps matter

You can leverage your blog sitemap to maximize search engine crawling and indexation, and your RSS/ATOM feeds to syndicate your content out to your social media profiles and content networks.

Search engines use your sitemap to thoroughly index your site. Google has stated that you should populate your sitemap with the canonical URL for every page you want indexed.

Built-in blog sitemap

ButterCMS generates RSS, Atom, and sitemap XML markup for your blog. To use these on your blog, return the generated XML from the Butter API with the proper content type headers.

Here’s what an example request looks like for getting your blog’s sitemap:

curl -X GET 'https://api.buttercms.com/v2/feeds/sitemap?auth_token=your_api_token'

And the response:

{
  "data": "<?xml version=\"1.0\" encoding=\"UTF-8\"?>\n<urlset xmlns=\"http://www.sitemaps.org/schemas/sitemap/0.9\" xmlns:xsi=\"http://www.w3.org/2001/XMLSchema-instance\" xsi:schemaLocation=\"http://www.sitemaps.org/schemas/sitemap/0.9 http://www.sitemaps.org/schemas/sitemap/0.9/sitemap.xsd\">\n \n  <url><loc>http://www.example.com/blog/this-is-a-blog-post</loc></url>\n \n</urlset>"
}

RSS and Atom feeds

ButterCMS generates XML feeds for content syndication and SEO:

• RSS Feed: Fully generated RSS 2.0 feed for your blog
• Atom Feed: Standards-compliant Atom 1.0 feed
• Sitemap: XML sitemap for search engine discovery

All feeds can be filtered by category or tag and include only published content from your organization.

Fetching feeds

# RSS Feed
curl -X GET 'https://api.buttercms.com/v2/feeds/rss?auth_token=your_api_token'

# Atom Feed
curl -X GET 'https://api.buttercms.com/v2/feeds/atom?auth_token=your_api_token'

# Sitemap
curl -X GET 'https://api.buttercms.com/v2/feeds/sitemap?auth_token=your_api_token'

Building sitemaps for Pages

Unlike blog posts, your individual Pages in ButterCMS don’t have a preset schema. Additionally, although your Page content is stored in ButterCMS, the URL patterns associated with each page are decided by your app.

For example, a single user-created Page Type, e.g., Case Study, might feed into multiple URL patterns:

• A /case-study/ route that fetches all your Butter Case Study Pages
• A /case-study/slug/ route that connects to a Case Study Detail view
• A /case-study/tag/tag-slug/ route that fetches Case Studies with a specific tag

Butter can’t know all the possible URL routes you’re using in your app, but it does return content in a format that makes creating your own sitemap easy.

Step 1: fetch content from ButterCMS

There are a number of ways to fetch your content from ButterCMS, including SDKs for C#, Dart, Go, Java, JavaScript, PHP, Python, Ruby, and Swift. Or, you can fetch your content via a simple API call to the Pages API endpoint:

https://api.buttercms.com/v2/pages/<page_type>/?auth_token=your_api_token

You can set the page_size or limit parameters to determine how many results you want returned. You can also filter by any field using fields.key.

The response will look like:

{
  "meta": {
    "previous_page": null,
    "next_page": null,
    "count": 2
  },
  "data": [
    {
      "slug": "case-study-1",
      "name": "Case Study One",
      "page_type": "customer_case_study",
      "created": "2019-11-12T17:23:53.109696Z",
      "published": "2019-11-12T17:23:53.109696Z",
      "updated": "2020-10-22T20:07:52.965850Z",
      "fields": {
        "seo": {
          "description": "SEO Description"
        }
      }
    },
    {
      "slug": "case-study-2",
      "name": "Case Study Two",
      "page_type": "customer_case_study",
      "created": "2019-11-12T17:23:53.109696Z",
      "published": "2019-11-12T17:23:53.109696Z",
      "updated": "2020-10-22T20:07:52.965850Z",
      "fields": {
        "seo": {
          "description": "SEO Description"
        }
      }
    }
  ]
}

Step 2: generate sitemap XML

The JSON response includes two pieces of information you’ll need for your sitemap: the slug of the page and the last updated datetime.

You can use this information and your dynamic URL pattern to write entries to your sitemap XML file:

<?xml version="1.0" encoding="UTF-8"?>
<urlset xmlns="http://www.sitemaps.org/schemas/sitemap/0.9">
  {% for entry in api_response.data %}
  <url>
    <loc>https://www.YourSite.com/case-studies/{{ entry.slug }}</loc>
    {% if entry.updated %}
    <lastmod>{{ entry.updated }}</lastmod>
    {% else %}
    <lastmod>{{ entry.published }}</lastmod>
    {% endif %}
  </url>
  {% endfor %}
</urlset>

Complete sitemap implementation

Node.js / Express example

const butter = require('buttercms')('your_api_token');

async function generateSitemap() {
  const baseUrl = 'https://yoursite.com';
  const urls = [];

  // Add static pages
  urls.push({ loc: `${baseUrl}/`, priority: 1.0 });
  urls.push({ loc: `${baseUrl}/about`, priority: 0.8 });
  urls.push({ loc: `${baseUrl}/contact`, priority: 0.8 });

  // Add blog posts
  const posts = await butter.post.list({ page_size: 100 });
  posts.data.data.forEach(post => {
    urls.push({
      loc: `${baseUrl}/blog/${post.slug}`,
      lastmod: post.updated || post.published,
      priority: 0.7
    });
  });

  // Add pages from a Page Type
  const caseStudies = await butter.page.list('case_study', { page_size: 100 });
  caseStudies.data.data.forEach(page => {
    urls.push({
      loc: `${baseUrl}/case-studies/${page.slug}`,
      lastmod: page.updated || page.published,
      priority: 0.6
    });
  });

  // Generate XML
  let xml = '<?xml version="1.0" encoding="UTF-8"?>\n';
  xml += '<urlset xmlns="http://www.sitemaps.org/schemas/sitemap/0.9">\n';

  urls.forEach(url => {
    xml += '  <url>\n';
    xml += `    <loc>${url.loc}</loc>\n`;
    if (url.lastmod) {
      xml += `    <lastmod>${new Date(url.lastmod).toISOString()}</lastmod>\n`;
    }
    if (url.priority) {
      xml += `    <priority>${url.priority}</priority>\n`;
    }
    xml += '  </url>\n';
  });

  xml += '</urlset>';
  return xml;
}

// Express route
app.get('/sitemap.xml', async (req, res) => {
  try {
    const sitemap = await generateSitemap();
    res.header('Content-Type', 'application/xml');
    res.send(sitemap);
  } catch (error) {
    res.status(500).send('Error generating sitemap');
  }
});

Next.js example

// pages/sitemap.xml.js
import butter from 'buttercms';

const butterClient = butter('your_api_token');

function generateSiteMap(posts, pages) {
  const baseUrl = 'https://yoursite.com';

  return `<?xml version="1.0" encoding="UTF-8"?>
   <urlset xmlns="http://www.sitemaps.org/schemas/sitemap/0.9">
     <url>
       <loc>${baseUrl}</loc>
       <priority>1.0</priority>
     </url>
     ${posts
       .map(post => {
         return `
       <url>
         <loc>${baseUrl}/blog/${post.slug}</loc>
         <lastmod>${new Date(post.updated || post.published).toISOString()}</lastmod>
         <priority>0.7</priority>
       </url>
     `;
       })
       .join('')}
     ${pages
       .map(page => {
         return `
       <url>
         <loc>${baseUrl}/${page.slug}</loc>
         <lastmod>${new Date(page.updated || page.published).toISOString()}</lastmod>
         <priority>0.6</priority>
       </url>
     `;
       })
       .join('')}
   </urlset>
 `;
}

export async function getServerSideProps({ res }) {
  const posts = await butterClient.post.list({ page_size: 100 });
  const pages = await butterClient.page.list('*', { page_size: 100 });

  const sitemap = generateSiteMap(posts.data.data, pages.data.data);

  res.setHeader('Content-Type', 'text/xml');
  res.write(sitemap);
  res.end();

  return { props: {} };
}

export default function Sitemap() {}

Sitemap best practices

Before you launch ButterCMS, configure your sitemap properly:

• If you are using ButterCMS to build an entire site, create a new sitemap and configure your app to update it whenever content is added or removed
• If you are using ButterCMS to implement a blog within an existing site, you can either create a new sitemap for your blog or work within the sitemap for the existing site

What to include

When indexing of new content is desired, add new URLs to the sitemap using the designated canonical URL when the new content is published:

• Pages
• Posts
• Blog index page (blog home)
• Category index pages
• Tag index pages
• All other published content that should be indexed

What to exclude

When indexing of new content is not desired, add a noindex tag to the page head and do not add the content to the sitemap. Google recommends:

1. To prevent most search engine crawlers, add a noindex meta tag:

<meta name="robots" content="noindex">

2. To prevent only Google crawlers:

<meta name="googlebot" content="noindex">

Alternatively, you can use X-Robots-Tag in your HTTP response header:

HTTP/1.1 200 OK
X-Robots-Tag: noindex

Syndicating to social media

To post your live blog posts to your Facebook account automatically, you can connect your blog RSS feed to your Facebook account.

Submitting your sitemap

After generating your sitemap, submit it to search engines:

Search Engine	Submission Method
Google	Google Search Console → Sitemaps
Bing	Bing Webmaster Tools → Sitemaps
Yandex	Yandex Webmaster → Indexing

Also add your sitemap to robots.txt:

User-agent: *
Allow: /

Sitemap: https://yoursite.com/sitemap.xml

Search engines will automatically discover your sitemap from robots.txt, but manual submission can speed up initial indexing.