Edit robots.txt

⏱ ~2 min to the answer · ~8 min full page · skim the bold lead-ins to move faster.

In short. robots.txt is the file search engines read before anything else on your site. SGEN gives you a one-textarea editor in SEO → Robots.txt. Type your rules, click Save Changes, and the public file at yourdomain.com/robots.txt updates instantly. Leave the textarea empty and SGEN serves a sensible default (admin/login/register/cart/checkout blocked, sitemap announced). Most sites end up with five to ten lines total — you will rarely need to visit this page, but the few minutes you spend here have an outsized effect on how the rest of your SEO work plays out.

On this page: What is this for? · Scope · Examples · Reference · Where to go · Steps · Troubleshooting

How to write the crawl rules search engines see for your site

robots.txt is a small plain-text file your site serves at https://yourdomain.com/robots.txt. Search engines read it first — before any page — to find out which parts of your site they are invited to crawl and which to skip. SGEN lets you edit that file directly from the admin, no server access needed.

The editor is one textarea on one page. Whatever you type and save is exactly what crawlers see on their next fetch. Leave it empty and SGEN serves a sensible default — the file is never silently missing once the site is initialized.

What is this for?

Use the Robots.txt editor when you want to:

Block a section from crawling — /staging/, /old-pricing/, /internal-docs/, /preview-fall-2026/.
Tell bots where your sitemap lives — so they can find every page quickly, even ones not linked from your homepage.
Announce a second sitemap — a storefront on the same domain, a docs subdomain, an event calendar feed.
Temporarily freeze indexing during a migration, redesign, or rebrand window — without touching the Global SEO toggle.
Target a specific bot (Googlebot, Bingbot, GPTBot, ClaudeBot, etc.) while leaving the rest alone.
Block a thank-you or cart-confirmation URL from appearing as a search result for someone outside the purchase flow.

If you do not edit anything, SGEN serves a sensible default. You only touch this page when you have a specific rule to add.

Scope

What this covers:

Writing and saving the full content of your robots.txt (one textarea, one Save button).
Allowing or disallowing crawlers from specific URL paths.
Announcing sitemap URLs to crawlers.
Targeting specific bots (Googlebot, Bingbot, etc.) with separate User-agent sections.
Viewing the live file at your domain via the header link.
Using the SGEN default (admin/login/register/cart/checkout blocked) when the textarea is empty.

What this does not cover:

The site-wide indexing toggle — that is the Enable search engines to index this site switch in Global SEO. The two work together: Global SEO is the "yes/no for the whole site" switch; robots.txt adds fine-grained path rules on top.
Per-page noindex tags — set those in the SEO Manager or per-item schema editor.
Sitemap generation — SGEN generates the sitemap automatically at yourdomain.com/sitemap.xml. Robots.txt just announces it.

Examples

robots_txt_structure: user_agent_section: syntax: &#39;User-agent: &#39; # * = all bots must_precede: allow/disallow lines for that section common_directives: allow: &#39;Allow: /path/&#39; # permit crawling this path disallow: &#39;Disallow: /path/&#39; # block crawling this path sitemap: &#39;Sitemap: https://yourdomain.com/sitemap.xml&#39; notes: case_insensitive: mostly (bots vary) typo_behavior: silently skipped, no error shown empty_file: SGEN serves default (blocks /sg-admin/, /login, /register, /cart, /checkout) scope: this site only, one domain sgen_default_blocks: - /sg-admin/ - /login - /register - /cart - /checkout

Block a staging folder. Add Disallow: /staging/ under User-agent: *. Save. Crawlers skip that path on the next crawl. The trailing slash matters — Disallow: /staging/ blocks every URL under /staging/ but not a page literally named /staging (no slash).

Announce an external sitemap. Append a Sitemap: line pointing to the external URL. Each Sitemap: line stands alone — no User-agent block needed. You can have as many Sitemap: lines as you have sitemaps; ordering does not matter.

Freeze indexing during a migration. Set Disallow: / under User-agent: *. This is heavier than the Global SEO toggle — it stops crawl traffic at the front door rather than asking bots to ignore the contents. Use it for the cutover window, then revert.

Block a specific bot while allowing the rest. Add a targeted User-agent: GPTBot / Disallow: / block above your default User-agent: * block. Bots that honor robots.txt skip the matching rule. Bots that ignore it will not stop regardless — for those, you need a server-level firewall rule.

User-agent: GPTBotDisallow: /User-agent: ClaudeBotDisallow: /User-agent: *Allow: /Disallow: /sg-admin/

Block a temporary preview tree. Add Disallow: /preview-fall-2026/ before launch, delete the line after. Set a calendar reminder for launch day — forgetting this line keeps the campaign invisible to search until you fix it.

Reference

Directive	Syntax	Effect
Allow all	`User-agent: *` / `Allow: /`	Crawlers can access everything except explicitly disallowed paths
Block path	`Disallow: /path/`	Crawlers skip that path and everything under it
Block bot	`User-agent: GPTBot` / `Disallow: /`	That bot cannot crawl any path
Announce sitemap	`Sitemap: https://yourdomain.com/sitemap.xml`	Helps crawlers find all your pages
Empty file	(leave textarea blank, save)	SGEN serves default: admin/login/register/cart/checkout blocked

Good use cases

Dashboard / SEO / Robots.txt

←

Robots.txt

Edit crawl directives for search engines

File Contents — served at https://yourdomain.com/robots.txtLeave empty to use the default allow/disallow behavior based on Global SEO. Save at least once with default content before going live — an empty robots.txt does not produce a valid response on a fresh site.

The textarea accepts as many lines as you want — scroll inside it for files longer than 16 rows. No character limit.

A useful pattern is to keep one comment line above each meaningful section so a teammate inheriting the file understands why each rule exists:

# Auto-disallows for admin routesUser-agent: *Disallow: /sg-admin/Disallow: /loginDisallow: /register# Block staging folder added 2026-04 by marketingDisallow: /staging/# Block AI training crawlersUser-agent: GPTBotDisallow: /# SitemapSitemap: https://yourdomain.com/sitemap.xml

After saving, the public file at https://yourdomain.com/robots.txt reads back the exact text you typed — no rewriting, no reordering:

https://yourdomain.com/robots.txt

User-agent: *Allow: /Disallow: /sg-admin/Disallow: /loginDisallow: /registerDisallow: /cartDisallow: /checkoutDisallow: /staging/Sitemap: https://yourdomain.com/sitemap.xmlSitemap: https://shop.yourdomain.com/sitemap.xml

If you see an HTML page or "Site Not Available" at the public URL instead of plain text, open the editor, confirm the textarea is not empty, click Save Changes once more, and refresh with a cache-bust parameter (?v=1).

What NOT to use this for

Do not use robots.txt as a security boundary. Well-behaved search engines honor the rules. Bad actors ignore them entirely. If content must stay private, password-protect it or mark the page Private in its own editor — robots.txt is a polite request, not a firewall.
Do not copy unfamiliar rules without testing. Syntax errors are silently ignored — User-Agent: (capital A) instead of User-agent: skips the line entirely. Test every paste by visiting /robots.txt after saving and reading it top to bottom.
Do not block CSS or JavaScript directories. Modern crawlers render pages to evaluate them. Disallowing /assets/css/ or /assets/js/ shows Google an unstyled page and may lower your ranking.
Do not edit robots.txt to remove a page from Google's index. Disallowing a path stops crawling, not indexing — Google can still keep an already-indexed page in results. To remove a page, use the Index Status toggle (adds a noindex tag) or Google Search Console's removal tool.
Do not use robots.txt to handle redirects. That belongs to the Redirects manager. robots.txt only describes crawl permission; it cannot rewrite a URL.
Do not paste sitemap content directly into robots.txt. Only the URL of the sitemap goes here. The Sitemap: line tells crawlers where to find the XML file.
Do not edit the file every time you publish a new page. robots.txt is for site-wide and section-level rules. Per-page indexing decisions belong in the page editor's Index Status toggle.

How this connects to other features

Global SEO — when the textarea is empty, the default robots.txt follows your Allow search engines to index this site switch. Off = Disallow: /. On = Allow: / plus auto-disallows plus any external sitemap URLs set under Global SEO. Think of Global SEO as the switch and robots.txt as the paragraph.
Google Search Console — if Search Console reports a page blocked by robots.txt, come here to see why. The URL Inspection tool also shows the exact robots.txt body Google last fetched, which is sometimes hours behind your latest save.
SEO Manager Index Status — controls a single page (adds noindex to <head>). Robots.txt controls whole sections (tells crawlers not to fetch the URL at all). Use Index Status for "this one thank-you page should not appear in Google"; use robots.txt for "the entire /internal/ folder is off-limits."
Sitemap — SGEN auto-generates a sitemap at /sitemap.xml. The default robots.txt includes a Sitemap: line pointing at it. You can append more lines manually for sitemaps hosted outside SGEN.
Redirects — robots.txt controls crawling, not URL routing. If you want /old-pricing/ to send visitors to /pricing/, use the Redirects manager.
Pages and Blog editors — per-page SEO settings (canonical URL, OG tags, schema) live in the individual page editor. Robots.txt sits one level above — it controls whether a crawler can reach the page at all.

Before you start

Sign in as an admin.
Prepare text in a plain-text editor if you are pasting a template — word processors insert fancy quotes (" instead of ") that break rules silently. Notepad (Windows), TextEdit in plain-text mode (macOS), or any code editor work fine.
Save at least once before launch. On a fresh site, the public /robots.txt URL is only reliable after you have visited the editor and clicked Save Changes at least once — even if you only save the default content. Open the editor, accept the defaults, click Save Changes, move on.
Copy your current content somewhere safe before editing. If a save goes wrong, you want a fast revert path.
Know your site's actual paths. Disallow rules match the URL exactly — Disallow: /staging blocks /staging and /staging/something but does NOT block /staging-area.

Where to go

The Robots.txt editor sits in the SEO area alongside the rest of the search-engine surfaces:

SEO area — where Robots.txt lives

SEO Manager 13 Global SEO 1 Robots.txt 1 Schema Editor 1 Search Console 1

Open the left navigation.
Click SEO → Robots.txt (or open /sg-admin/seo/robots directly). The editor opens with the current contents loaded.

If your sidebar shows the SEO group collapsed, click the SEO header to expand it — Robots.txt is the last item in the list.

Steps

1. Read the current rules

The textarea shows what is currently saved. If it is empty, SGEN serves the dynamic default at the public URL. The placeholder for a fresh site looks like this:

placeholder default — what gets served when the textarea is emptybash

User-agent: *Allow: /Disallow: /sg-admin/Disallow: /loginDisallow: /registerSitemap: https://yourdomain.com/sitemap.xml

SGEN generates these lines from your Global SEO settings, the auto-disallowed admin routes, and any external sitemap URLs configured under Global SEO.

If the textarea is non-empty, scroll through the existing content before changing anything. Copy the current content into a notes file first — reverting is a single paste.

2. Edit the rules

Type directly into the textarea. Standard robots.txt syntax:

User-agent: * — this block applies to every bot.
User-agent: Googlebot — targets a specific bot. Common values: Googlebot, Bingbot, DuckDuckBot, GPTBot, Applebot, ClaudeBot, CCBot.
Disallow: /path/ — blocks a path prefix. Trailing slash matters — Disallow: /cart blocks /cart and /cart/checkout.
Allow: /path/ — explicitly permits a path (useful inside a blocked parent).
Sitemap: https://.. — points bots at a sitemap file. Stands alone, no User-agent block needed.
Lines starting with # are comments — bots ignore them, useful for leaving notes for the next admin.

Leave a blank line between User-agent: blocks for readability. The textarea is plain text only — no syntax helper, no autocomplete, no validation. Whatever you type is saved verbatim.

3. Save

Click Save Changes. The page reloads with a green confirmation banner:

Robots.txt saved

Apr 22, 2026 14:03

robots.txt successfully updated. The new content is live at https://yourdomain.com/robots.txt — search engines will pick it up on their next crawl.

Updated: site_settings.robots_txt

Save is instant. The public URL serves the new content the moment the banner appears — no deploy step, no cache flush. If you have a CDN in front of the site, the SGEN-served body updates immediately but the cached body may take a few minutes to refresh.

4. Verify on your live site

Open a new browser tab and go to https://yourdomain.com/robots.txt. You should see the exact text you saved as plain text. If you saved an empty textarea, you see the dynamic default.

5. Confirm in Google Search Console

A few hours after your save, open Google Search Console and run the URL Inspection tool against your site. The tool shows the most recent robots.txt body Google has fetched. Compare it against what you saved. This step is optional but worth doing after any meaningful change (a new Disallow rule, a sitemap addition, a migration freeze).

6. Document the change

If your team keeps an SEO changelog, jot a one-line entry: date, what you changed, why. A line like 2026-05-01 — added Disallow: /staging/ — marketing's staging folder shouldn't appear in Google is enough. Comments inside the robots.txt file (lines starting with #) are also a good place — they ride along with the rule.

What success looks like

A green banner confirms the file was updated.
Visiting https://yourdomain.com/robots.txt in a fresh browser tab shows the new rules as plain text — no admin chrome, no SGEN header, no styling, just the lines you typed.
If you saved an empty textarea, the live file reverts to the dynamic default (including any external sitemap lines set under Global SEO).
Google Search Console's URL Inspection tool, run a few hours after your save, reports the new robots.txt body.
The Disallow rules you added work as expected when tested in Search Console's Robots Tester.

On a fresh site about to launch: save at least once with default or custom content before flipping DNS. Saving once — even with the placeholder — guarantees the public /robots.txt URL serves robots directives, not a generic site response:

What to do if it does not work

The live file still shows the old content. Your browser may have cached the old robots.txt. Add ?v=1 to the URL to bypass the cache.
The live file shows HTML instead of robots rules. This typically happens on a fresh site where the textarea has never been saved. Open the editor, accept or type the default content, click Save Changes once.
Google reports "blocked by robots.txt" on a page you want indexed. Check whether a Disallow: rule matches the page's path. Remove or tighten the rule. Use Search Console's URL Inspection tool to confirm Google sees the same body you see.
My rules do not seem to take effect on Google. Search engines re-read robots.txt on their own schedule — it can take hours or days for changes to show up in Search Console. The file at /robots.txt updates instantly; Google's awareness of it does not.
I cannot tell whether I broke the syntax. Clear the textarea, save, and let SGEN serve the default — then re-add your rules one block at a time, testing the public URL after each save.
The textarea looks fine but the public URL is empty. Save once. The public route reads from a saved value; on fresh sites the public URL falls back to a default that may not exist until the first save.
The Crawl-delay: directive I added is being ignored. Google ignores Crawl-delay: entirely. Bing and Yandex honor it. To slow down Googlebot, use Search Console's crawl-rate setting instead.
I want to test a rule without going live. Save it on a staging site first, or use Google Search Console's Robots Tester to dry-run a rule against a URL before pasting it into your live editor.
My CDN is showing an old robots.txt body after my save. Purge the CDN cache for the /robots.txt URL specifically. The SGEN save is instant; the CDN cache is not.

Tips for a clean robots.txt

Keep the file short. Five to ten lines is the right neighborhood for a typical site. If yours is 80 lines, most of those rules are probably stale — folders renamed, campaigns ended, bots that no longer exist. Scan and delete what no longer applies quarterly.

Comment every custom rule. Every line not part of the default deserves a one-line comment above it explaining why it's there. Three months from now, you will not remember why Disallow: /tmp-2026/ exists.

Test rules with Search Console's Robots Tester before saving. The tester takes a URL and a rules block and tells you whether the URL would be allowed or blocked — use it before saving on the live site.

Schedule a quarterly review. Put a 15-minute recurring calendar event in your shared calendar — "review robots.txt." Open the editor, scan the rules, delete anything stale, confirm the sitemap line still points at a valid sitemap.

Match robots.txt to your real site structure. If you renamed /blog/ to /articles/ six months ago, update or delete the old Disallow: /blog/draft/ line. Stale paths are harmless but add noise.