Deshmaj Blogger Robots.txt Generator

Deshmaj Blogger Robots.txt Generator

Website URL Allow All Search Engines? Allow Disallow Block Search Pages? Yes No Add Sitemap? Yes No

Generated Robots.txt

User Guide

1. Introduction

The Deshmaj Blogger Robots.txt Generator is a simple and powerful tool that allows website owners, especially Blogger users, to create a custom robots.txt file without needing technical knowledge. This file tells search engines which parts of your website to crawl and which parts to avoid, helping with SEO and controlling indexing.

2. Steps to Generate Robots.txt

Step 1: Enter Your Website URL

• Locate the field labeled “Website URL.”
• Enter your full Blogger website URL, for example: https://example.blogspot.com.
• This URL will be used to create the sitemap reference if you choose to include it.

Step 2: Allow or Disallow All Search Engines

• The dropdown labeled “Allow All Search Engines?” lets you choose whether search engines can access your site.
• If you select Allow, the robots.txt will not block any pages.
• If you select Disallow, it will block all pages from being indexed by search engines.

Step 3: Block Search Pages (Optional)

• Blogger creates search result pages that can generate duplicate content.
• The “Block Search Pages?” dropdown allows you to disallow indexing of these search pages.
• Select Yes if you want to block them, or No to allow indexing.

Step 4: Add Sitemap (Optional)

• A sitemap helps search engines understand your site structure.
• Use the “Add Sitemap?” dropdown to include a link to your sitemap automatically.
• Select Yes to include your sitemap in the robots.txt, or No if you do not want it included.

3. Generate Robots.txt

• After filling in the options, click the Generate Robots.txt button.
• The tool will create a ready-to-use robots.txt file based on your choices.
• The generated text will appear in the output area labeled “Generated Robots.txt.”

What Happens Automatically:

• It always includes a User-agent: * line to apply rules to all search engines.
• It adds Disallow lines depending on your settings.
• If you added the sitemap, it includes Sitemap: [Your URL]/sitemap.xml.
• At the bottom, a note is included: “Generated by Deshmaj Blogger Robots.txt Generator.”

4. Copying the Generated File

• To copy the content, simply click the Copy Text button below the output.
• This copies the entire robots.txt content to your clipboard, ready to paste in Blogger’s Settings → Crawlers and indexing → Custom robots.txt section.
• A notification will confirm the copy: “Robots.txt copied successfully.”

5. Downloading the Generated File

• You can also download the file for backup or manual upload.
• Click the Download .txt button.
• A .txt file named robots.txt will be downloaded to your device.
• You can then upload this file directly to your hosting or Blogger account.

6. Tips for Using Robots.txt in Blogger

• Make sure your URL is correct, including https://.
• Only block pages you do not want indexed, such as search pages or admin pages.
• Always test your robots.txt using Google Search Console to ensure it works as expected.
• Do not block important pages like your homepage, posts, or sitemaps.

7. Why This Tool is Useful

• Saves time by automatically creating a clean and correct robots.txt file.
• Reduces the risk of syntax errors that can prevent search engines from indexing your site.
• Lets even beginners control how their site is indexed without technical knowledge.
• Supports Blogger-specific settings like blocking search pages and adding a sitemap.

8. Summary of Steps for Your Article

1. Enter your Blogger URL.
2. Choose whether to allow or disallow all search engines.
3. Decide whether to block search pages.
4. Optionally add your sitemap.
5. Click Generate Robots.txt.
6. Use Copy Text to paste directly into Blogger or Download .txt to save the file.