Allowing TemsoBot

Temso audits your website by reading your pages with its own crawler, TemsoBot. There are two different reasons it can be turned away, and they need different fixes:

• Your server refuses the crawler. Requests come back as HTTP 403 ("Forbidden"), so the audit stops before it can read anything. This is usually a rule at your web host or in a security plugin that turns the crawler away because it identifies itself as a bot. The rest of this page covers that case.
• Your robots.txt asks crawlers to stay out. Your server answers us normally, but the file tells crawlers not to crawl, and we honour that. See If your robots.txt asks crawlers to stay out.

Either way your website is fine. Visitors see it as usual, and nothing is broken on Temso's side.

If you don't manage your own hosting, you only need one part of the first case: the message to forward.

The one thing to allow

Whoever manages your website needs to allow this exact user-agent:

TemsoBot/1.0 (+https://temso.ai/bot)

Allow it for the whole site, including /robots.txt. The crawler reads that file first, and a block there stops the audit just as surely as a block on your pages.

For rules that match on part of the string, the name to match is TemsoBot. That's also the product token for robots.txt rules. When the block is at the server it happens before robots.txt is even consulted, so editing robots.txt alone won't fix that case. But if robots.txt is what's turning us away, that file is exactly where the fix goes.

Where the block usually lives

Any of these can produce the same 403.

Plesk, Apache, and .htaccess

Shared hosting often ships a user-agent blocklist in the Apache configuration, in a .htaccess file, or as a ModSecurity rule. A generic pattern like bot in such a list matches TemsoBot too, so the rule may not name us at all.

In .htaccess, look for lines mentioning HTTP_USER_AGENT, SetEnvIfNoCase User-Agent, or BrowserMatch, followed by a list of bot names. Remove the entry that matches. For RewriteCond-style blocklists, add an exception above the blocking rules instead:

# Let TemsoBot past the user-agent blocklist below
RewriteEngine On
RewriteCond %{HTTP_USER_AGENT} TemsoBot [NC]
RewriteRule .* - [L]

In Plesk, per-domain Apache directives live under Websites & Domains → Apache & nginx Settings, and ModSecurity under Websites & Domains → Web Application Firewall. When ModSecurity causes the 403, the server's error log records the ID of the rule that fired, and you can add that ID as an exception in the domain's firewall settings.

WordPress security plugins

Wordfence and similar security plugins ship bot-blocking lists that match on the user-agent. In Wordfence, check Firewall → Blocking for a custom pattern whose user-agent field matches TemsoBot or something generic like *bot*, and delete or narrow it. In other plugins, look for a setting named "blocked user agents", "bot blocking", or similar. The plugin's firewall or live-traffic log will show the refused requests from TemsoBot, which confirms you've found the right rule.

Cloudflare

If your site sits behind Cloudflare, add a custom WAF rule that skips security checks for the crawler: under Security, create a custom rule with field User Agent, operator contains, value TemsoBot, and the Skip action.

One caveat: Bot Fight Mode (on the free plan) doesn't honour skip rules and has no per-bot exceptions. If it's the source of the block, the choices are turning it off or upgrading to a plan whose bot management supports allow rules.

Vercel

If the site is hosted on Vercel, the block usually comes from the Vercel Firewall's bot protection. In the project's Firewall tab, add a custom rule with the condition User Agent contains TemsoBot and the Bypass action, so the crawler's requests skip bot protection.

What to send your hosting provider

If none of the above is yours to change, that's normal. The hosting provider or webmaster usually fixes it in a few minutes once they know what to look for. Copy this message and send it to your hosting support, or to whoever manages your website:

Hi,

My website is blocking a crawler that I've asked to visit it, and I need
an exception added.

I use Temso (https://temso.ai) to audit my website. Its crawler
identifies itself with this user-agent:

  TemsoBot/1.0 (+https://temso.ai/bot)

The server currently answers this crawler's requests with HTTP 403,
which usually means a user-agent blocklist in the server configuration,
.htaccess, ModSecurity, a security plugin, or a firewall rule. Could you
find the rule responsible and allow this user-agent for the whole site,
including /robots.txt?

The crawler respects robots.txt and crawls at a polite rate. Setup
instructions for common hosts are here:
https://docs.temso.ai/docs/troubleshooting/allowing-temsobot

Thank you!

If your robots.txt asks crawlers to stay out

The fix here is a different one. Nothing is blocking our crawler. Your server answers it normally, but your robots.txt file asks crawlers not to crawl. TemsoBot honours that, so the audit stops with nothing to read.

A file like this turns away every well-behaved crawler, ours included:

User-agent: *
Disallow: /

If that's deliberate, say on a staging site or one you don't want indexed, then nothing is broken and there is nothing to fix. But Temso can only audit pages it is allowed to read, so if you want the audit to run, robots.txt has to permit our crawler.

The narrowest change is to add a group for TemsoBot on its own. Crawlers follow the most specific group that names them, so this lets Temso in without opening your site to anything else:

User-agent: TemsoBot
Allow: /

Add it as a separate block, with a blank line between it and your other rules, and leave your existing User-agent: * rules exactly as they are. They keep applying to every other crawler.

If instead you find a group that already names us, that is what's stopping the audit. Change its Disallow to Allow, or remove the group:

User-agent: TemsoBot
Disallow: /

robots.txt lives at the root of your site, at https://www.your-site.com/robots.txt. On most platforms you edit it either as a file in your web root or through your SEO plugin's settings.

Check that it worked

Once the rule is in place, run the audit again: in Temso, open Website in your project and press Crawl, then Start crawl. If the server still refuses the first request, you'll see the same failure message; otherwise the audit runs normally, which can take several minutes.

Whoever changed the rule can also verify it directly, without waiting for an audit, by requesting a page with the crawler's user-agent:

curl -sI -A "TemsoBot/1.0 (+https://temso.ai/bot)" https://www.your-site.com/robots.txt

A 403 in the first line of the output means the block is still active; anything else means the crawler is getting through.

These rules usually block AI assistants too

Blocklists that catch TemsoBot are usually generic "block all bots" rules, and rules like that typically also catch the crawlers of AI assistants such as GPTBot (ChatGPT), ClaudeBot (Claude), and PerplexityBot (Perplexity). A site those crawlers can't read can't be used as a source by those assistants. We haven't measured whether your particular rule does this, but while someone is editing it anyway, it's a good moment to ask what else it blocks.