Sample of Robot txt

-

                      SEO-friendly robots.txt file for Website

The robots.txt file is an important part of search engine optimization (SEO) as it tells search engine crawlers which pages or files on your website should or should not be crawled or indexed. Here's an example of an SEO-friendly robots.txt file:

User-agent: *
Disallow:

User-agent: Googlebot
Disallow: /admin/
Disallow: /private/
Disallow: /tmp/
Disallow: /uploads/

Sitemap: https://www.example.com/sitemap.xml

In this example, the User-agent: * directive applies to all search engine crawlers. The Disallow: directive with no value means that all pages and files are allowed to be crawled.

Next, we specify the rules for Googlebot. In this example, we're disallowing access to certain directories, such as /admin/, /private/, /tmp/, and /uploads/. This can help prevent sensitive or duplicate content from being indexed by search engines.

Finally, we include a Sitemap directive to tell search engines where to find our XML sitemap file. This can help search engines discover new content on your site and index it more quickly.

It's important to note that the robots.txt file is a public file, so don't include any sensitive information in it. Additionally, be careful not to accidentally block important pages or directories that you want search engines to crawl and index. Always test your robots.txt file using Google's robots.txt tester tool to ensure that it's configured correctly.


All Robots.text Example:
User-agent: *
User-agent: yeti
User-agent: psbot
User-agent: Bot1
User-agent: Slurp
User-agent: Teoma
User-agent: Nutch
User-agent: Bingbot
User-agent: MSNBot
User-agent: Gigabot
User-agent: naverbot
User-agent: Robozilla
User-agent: Googlebot
User-agent: baiduspider
User-agent: Adsbot-Google
User-agent: Googlebot-News
User-agent: ia_archiver
User-agent: Mediapartners-Google
User-agent: Googlebot-Image
User-agent: Googlebot-Video
User-agent: Googlebot-Mobile
User-agent: yahoo-mmcrawler
User-agent: yahoo-blogs/v3.9

User-agent: *
Allow:
Allow: /
Allow: /gpt/
Allow: /blog/
Allow: /local/
Allow: /learn/
Allow: /tag/js/
Allow: /products/
Allow: /Index/
Allow: /blog/post-title/
Allow: /folder/page.html
Allow: /link-intersect/$
Allow: /static/glade/
Allow: /static/glade.js
Allow: /ads/preferences/
Allow: /pagead/show_ads.js
Allow: /pagead/js/adsbygoogle.js
Allow: /pagead/js/*/show_ads_impl.js
Allow: /products/content/
Allow: /site-explorer/$
Allow: /plugins/system/jch_optimize/assets/
Allow: /researchtools/ose/
Allow: /researchtools/ose/$
Allow: /researchtools/ose/anchors$ 
Allow: /researchtools/ose/dotbot$
Allow: /researchtools/ose/dotbots
Allow: /researchtools/ose/links$ 
Allow: /researchtools/ose/just-discovered
Allow: /researchtools/ose/pages$ 
Allow: /researchtools/ose/domains$
Allow: /directory/
Allow: /marketplace/
Allow: /keywords/
Allow: /local/search/
Allow: /content/search/*
Allow: /site-explorer/*
Allow: /site-explorer/ajax/
Allow: /admin/assests/
Allow: /admin/admin-ajax/
Allow: /admin/admin-ajax.js
Allow: /admin/admin-ajax.js/*/show_ads_impl
Allow: /services/
Allow: /page-strength/* 
Allow: /marketplace/
Allow: /script/

User-agent: *
Disallow: /
User-agent: Googlebot-image
Disallow:
User-agent: *
Disallow: /staging/
User-agent: *
Allow: /myfolder/

User-agent: *
Disallow:
Disallow: /
Allow: /content/uploads/
Allow: /content/page/
Disallow: /content/
Disallow: /client/
Disallow: /admin/
Disallow: /includes/
Disallow: /content/plugins/
Disallow: /content/themes/
Disallow: /feed/
Disallow: /trackback/
Disallow: */feed/
Disallow: */trackback/
Disallow: /cgi-bin/
Disallow: /wp-login/
Disallow: /wp-register/
Disallow: /20*
Disallow: /static/glade/
Disallow: /tag/js/
Disallow: /folder1/
Disallow: /folder2/
Disallow: /assests/
Disallow: /folder3/
Disallow: /Googlebot/
Disallow: /Adsbot-Google/
Disallow: /Googlebot-Image/
Disallow: /Googlebot-Mobile/
Disallow: /example-page/
Disallow: /content/audit/* 
Disallow: /folder/
Disallow: /link-intersect/*
Disallow: /local/enterprise/confirm
Disallow: /local/details/
Disallow: /cpresources/
Disallow: /thumbs/*
Disallow: /researchtools/ose/
Disallow: /something/
Disallow: /something-else/
Disallow: /start
Disallow: /vendor/
Disallow: /cgi-bin/

User-agent: *
Sitemap: https://example.com/sitemap.xml

Online Source

Comments

Post a Comment

Thanks for your Comments.

Popular posts from this blog

Office Tool_SPSS v23 + Serial key

-
Best Office Tool SPSS v23 + Serial key Download Activation Instruction: 1. Install IBM SPSS 2. copy “lservrc” into the installed dir(e.g. Windows: C:\Program Files\IBM\SPSS\Statistics\23)  OR 3. enter the following license code using the License Authorization Wizard: Serial key: VDOV7M8KUEIAWBZIKPP6DUKX4JIO3LWRSJQW4BTDCU5NS28ZLZSSROOZQ8HASZ6VUHRZRZ8I8DGWIFY9WJTIRD5P9Y
Next »

How to Fix FATAL error Failed to parse input Document ?

-
SEO Help and Tips How to Fix FATAL error Failed to parse input Document ? When encountering the "FATAL error Failed to parse input document" error, it typically indicates that there is an issue with the code used to parse the document. To fix this error, you can follow these steps: Review the parsing code:   Check the code responsible for parsing the input document. Ensure that it is correctly implemented and follows the correct syntax for the chosen parsing library or approach. Validate the input document:  Before parsing the document, validate its format and structure. Use a document validator or a specific validator for the document type you are parsing to ensure it adheres to the expected standards. This can help identify any syntax errors or inconsistencies in the document. Confirm the input document's compatibility:  Ensure that the parsing library or approach you are using supports the document type you are trying to parse. Some libraries specialize in specific doc...
Next »

How to Reduce Lazy Load Resources

-
How to Reduce Lazy Load Resources in Website Reduce Lazy Load Resources: Utilize lazy loading for resources such as images, scripts, and stylesheets. This means only loading these resources when they are needed, which can help reduce the initial DOM size and improve page load performance. Diagnose performance issues, Performance Values are estimated and may vary. The performance score is calculated directly from these metrics. See calculator. Images:  Lazy loading images can be achieved by setting the loading attribute to "lazy" on the <img> tag. This tells the browser to load the image only when it's about to come into view. <img alt="Lazy-loaded image" src="placeholder.png" data-src="image.png" class="lazyload" /> <script>   document.addEventListener('DOMContentLoaded', function() {     var lazyImages = document.querySelectorAll('img.lazyload');     //...
Next »

Popular posts from this blog

Office Tool_SPSS v23 + Serial key

-
Best Office Tool SPSS v23 + Serial key Download Activation Instruction: 1. Install IBM SPSS 2. copy “lservrc” into the installed dir(e.g. Windows: C:\Program Files\IBM\SPSS\Statistics\23)  OR 3. enter the following license code using the License Authorization Wizard: Serial key: VDOV7M8KUEIAWBZIKPP6DUKX4JIO3LWRSJQW4BTDCU5NS28ZLZSSROOZQ8HASZ6VUHRZRZ8I8DGWIFY9WJTIRD5P9Y
Next »

How to Fix FATAL error Failed to parse input Document ?

-
SEO Help and Tips How to Fix FATAL error Failed to parse input Document ? When encountering the "FATAL error Failed to parse input document" error, it typically indicates that there is an issue with the code used to parse the document. To fix this error, you can follow these steps: Review the parsing code:   Check the code responsible for parsing the input document. Ensure that it is correctly implemented and follows the correct syntax for the chosen parsing library or approach. Validate the input document:  Before parsing the document, validate its format and structure. Use a document validator or a specific validator for the document type you are parsing to ensure it adheres to the expected standards. This can help identify any syntax errors or inconsistencies in the document. Confirm the input document's compatibility:  Ensure that the parsing library or approach you are using supports the document type you are trying to parse. Some libraries specialize in specific doc...
Next »

How to Reduce Lazy Load Resources

-
How to Reduce Lazy Load Resources in Website Reduce Lazy Load Resources: Utilize lazy loading for resources such as images, scripts, and stylesheets. This means only loading these resources when they are needed, which can help reduce the initial DOM size and improve page load performance. Diagnose performance issues, Performance Values are estimated and may vary. The performance score is calculated directly from these metrics. See calculator. Images:  Lazy loading images can be achieved by setting the loading attribute to "lazy" on the <img> tag. This tells the browser to load the image only when it's about to come into view. <img alt="Lazy-loaded image" src="placeholder.png" data-src="image.png" class="lazyload" /> <script>   document.addEventListener('DOMContentLoaded', function() {     var lazyImages = document.querySelectorAll('img.lazyload');     //...
Next »