Overview of S&P Global Crawlers

Many of the information products and services of S&P Global rely on information published on Internet sites by governments, agencies, and companies. This data is used for:

• Direct insertion/updates into our databases
• Normalization to our internal standards before insertion/updates
• Source material for downstream machine and/or human extraction
• Source material for background research or models

S&P Global crawlers collect information commonly published on Internet websites.

For instance, coverage of company fundamentals requires S&P Global to access investor relations materials for the companies it covers. Similarly, forecasts of future energy supply rely, in part, on press releases and annual reports discussing new projects by market participants.

Crawling activity by S&P Global is identified by User-Agent strings in the following form:

• SPGlobalWebBot/<Major>.<Minor> (+https://www.spglobal.com/bot; id=<RegistryID>)
• Example: SPGlobalWebBot/1.0 (+https:www.//spglobal.com/bot; id=core10a)

• SPGlobalWebBot - The unified identifier for all S&P Global automated crawlers
• <Major>.<Minor> - Crawler framework version following semantic versioning (e.g., 1.0)
• https://www.spglobal.com/bot - URL linking to this documentation page
• id=<RegistryID> - Unique internal identifier used by S&P Global to track its crawler instances (e.g., core10a)

S&P Global crawlers respect published robots.txt files. To block S&P Global from accessing any of your site content, add the following two lines to your robots.txt file at your site's root directory:

User-agent: SPGlobalWebBot
Disallow: /

Note that doing so may cause data about your organization to become stale in our products.

S&P Global crawlers cryptographically sign requests using Ed25519 (RFC 9421 — HTTP Message Signatures). This lets you confirm that a request genuinely came from S&P Global, rather than a third party spoofing our User-Agent string. As of October 2026, we are in the process of deploying these signatures across our bots so you may not see these signatures on every request yet.

Each signed request carries three additional headers. The signature is produced using a private key held exclusively by S&P Global — a valid signature is proof the request originated from us.

• Signature-Agent — Public key directory URL
• Signature-Input — Signed components, key identifier, algorithm, and timestamp
• Signature — Ed25519 signature for verification

Our corresponding public keys are published at crawlers.spglobal.com/.well-known/http-message-signatures-directory. This endpoint returns a JSON Web Key Set (JWKS) containing the Ed25519 public keys used to sign our requests. To verify a request, match the keyid from the Signature-Input header to a key in the directory, then verify the signature per RFC 9421.