All Robots.txt

-
SEO Help and Tips

All Robots.txt 

Website Disallow and Allow directives all Robots.txt code with details Here:

Allow:
------------
User-agent:*
Allow: /cgi-bin
Disallow: /
# prohibits downloading anything except pages # starting with '/cgi-bin'

User-agent:*
Allow: /file.xml
# allows downloading the file.xml file

Allow and Disallow directives without parameters:
----------------------------------------------------------------
User-agent:*
Disallow: # same as Allow: /

User-agent:*
Allow: # isn't taken into account by the robot

Disallow:
------------
User-agent:*
Disallow: / 
# prohibits crawling for the entire site

User-agent:*
Disallow: /catalogue 
# prohibits crawling the pages that start with /catalogue

User-agent:*
Disallow: /page? 
# Prohibits crawling the pages with a URL that contains parameters

Common:
------------
User-agent:*
Allow: /archive
Disallow: /
# allows everything that contains '/archive', everything else is prohibited

User-agent:*
Allow: /obsolete/private/*.html$ # allows HTML files                                  
# in the '/obsolete/private/...' path

Disallow: /*.php$  # prohibits all '*.php' on the website
Disallow: /*/private/ # prohibits all subpaths containing
                      # '/private/', but the Allow above negates
                      # a part of this prohibition Disallow: /*/old/*.zip$ # prohibits all '*.zip' files containing 
                        # in the path '/old/'

User-agent:*
Disallow: /add.php?*user= 
# prohibits all 'add.php?' scripts with the 'user' option


Combining directives:
---------------------------
# Source robots.txt:
User-agent:*
Allow: /
Allow: /catalog/auto
Disallow: /catalog

# Sorted robots.txt:
User-agent:*
Allow: /
Disallow: /catalog
Allow: /catalog/auto
# prohibits downloading pages starting with '/catalog',
# but allows downloading pages starting with '/catalog/auto'.

Using the special characters:
-------------------------------------
User-agent:*
Disallow: /cgi-bin/*.aspx # prohibits '/cgi-bin/example.aspx'
                          # and '/cgi-bin/private/test.aspx'
Disallow: /*private # prohibits both '/private'
                   # and '/cgi-bin/private'
   
By default, the * character is appended to the end of every rule described in the robots.txt file. Example:

User-agent:*
Disallow: /cgi-bin* # blocks access to pages 
                    # starting with '/cgi-bin'
Disallow: /cgi-bin # the same

User-agent:*
Disallow: /cgi-bin/*.aspx # prohibits '/cgi-bin/example.aspx'
                          # and '/cgi-bin/private/test.aspx'
Disallow: /*private # prohibits both '/private'
                   # and '/cgi-bin/private'
By default, the * character is appended to the end of every rule described in the robots.txt file. Example:

User-agent:*
Disallow: /cgi-bin* # blocks access to pages 
                    # starting with '/cgi-bin'
Disallow: /cgi-bin # the same


Examples of how directives are interpreted:
--------------------------------------------------------
User-agent:* 
Allow: /
Disallow: /
# everything is allowed

User-agent:* 
Allow: /$
Disallow: /
# everything is prohibited except the main page

User-agent:*
Disallow: /private*html
# prohibits '/private*html', 
# '/private/test.html', '/private/html/test.aspx', etc. User-agent:*
Disallow: /private$
# prohibits only '/private'

User-agent: *
Disallow: /
User-agent:*
Allow: /
# since the* robot # selects entries that include 'User-agent:',
# everything is allowed

User-agent:*
Allow: /
sitemap: https://example.com/site_structure/my_sitemaps1.xml
sitemap: https://example.com/site_structure/my_sitemaps2.xml

User-agent:*
Disallow:
Clean-param: ref /some_dir/get_book.pl

Additional examples:
-----------------------------

#robots.txt will contain:
User-agent:*
Disallow:
Clean-param: s /forum/showthread.php
#for URLs like:
www.example2.com/index.php?page=1&sid=2564126ebdec301c607e5df
www.example2.com/index.php?page=1&sid=974017dcd170d6c4a5d76ae

#robots.txt will contain:
User-agent:*
Disallow:
Clean-param: sid /index.php
#if there are multiple parameters like this:
www.example1.com/forum_old/showthread.php?s=681498605&t=8243&ref=1311
www.example1.com/forum_new/showthread.php?s=1e71c417a&t=8243&ref=9896

#robots.txt will contain:
User-agent:*
Disallow:
Clean-param: s&ref /forum*/showthread.php
#if the parameter is used in multiple scripts:
www.example1.com/forum/showthread.php?s=681498b9648949605&t=8243
www.example1.com/forum/index.php?s=1e71c4427317a117a&t=8243

#robots.txt will contain:
User-agent:*
Disallow:
Clean-param: s /forum/index.php
Clean-param: s /forum/showthread.php

User-agent: *
This line specifies the user agent to which the subsequent directives apply. In this case, the asterisk (*) is a wildcard character that means "all user agents." So, the directives that follow will apply to all web robots or crawlers.

Disallow: /search
This line indicates that the "/search" directory or folder should not be crawled by web robots. The "/search" directory typically contains search result pages generated by the website's search functionality. By disallowing it, you are instructing search engine crawlers not to index or consider those search result pages in their search results.

Allow: /
This line allows web robots to crawl and index the rest of the website, excluding the "/search" directory. The forward slash ("/") is a directive that represents the root of the website. By allowing it, you are essentially permitting web robots to crawl all other parts of your site.

Comments

Post a Comment

Thanks for your Comments.

Popular posts from this blog

Office Tool_SPSS v23 + Serial key

-
Best Office Tool SPSS v23 + Serial key Download Activation Instruction: 1. Install IBM SPSS 2. copy “lservrc” into the installed dir(e.g. Windows: C:\Program Files\IBM\SPSS\Statistics\23)  OR 3. enter the following license code using the License Authorization Wizard: Serial key: VDOV7M8KUEIAWBZIKPP6DUKX4JIO3LWRSJQW4BTDCU5NS28ZLZSSROOZQ8HASZ6VUHRZRZ8I8DGWIFY9WJTIRD5P9Y
Next »

How to Fix FATAL error Failed to parse input Document ?

-
SEO Help and Tips How to Fix FATAL error Failed to parse input Document ? When encountering the "FATAL error Failed to parse input document" error, it typically indicates that there is an issue with the code used to parse the document. To fix this error, you can follow these steps: Review the parsing code:   Check the code responsible for parsing the input document. Ensure that it is correctly implemented and follows the correct syntax for the chosen parsing library or approach. Validate the input document:  Before parsing the document, validate its format and structure. Use a document validator or a specific validator for the document type you are parsing to ensure it adheres to the expected standards. This can help identify any syntax errors or inconsistencies in the document. Confirm the input document's compatibility:  Ensure that the parsing library or approach you are using supports the document type you are trying to parse. Some libraries specialize in specific doc...
Next »

How to Reduce Lazy Load Resources

-
How to Reduce Lazy Load Resources in Website Reduce Lazy Load Resources: Utilize lazy loading for resources such as images, scripts, and stylesheets. This means only loading these resources when they are needed, which can help reduce the initial DOM size and improve page load performance. Diagnose performance issues, Performance Values are estimated and may vary. The performance score is calculated directly from these metrics. See calculator. Images:  Lazy loading images can be achieved by setting the loading attribute to "lazy" on the <img> tag. This tells the browser to load the image only when it's about to come into view. <img alt="Lazy-loaded image" src="placeholder.png" data-src="image.png" class="lazyload" /> <script>   document.addEventListener('DOMContentLoaded', function() {     var lazyImages = document.querySelectorAll('img.lazyload');     //...
Next »

Popular posts from this blog

Office Tool_SPSS v23 + Serial key

-
Best Office Tool SPSS v23 + Serial key Download Activation Instruction: 1. Install IBM SPSS 2. copy “lservrc” into the installed dir(e.g. Windows: C:\Program Files\IBM\SPSS\Statistics\23)  OR 3. enter the following license code using the License Authorization Wizard: Serial key: VDOV7M8KUEIAWBZIKPP6DUKX4JIO3LWRSJQW4BTDCU5NS28ZLZSSROOZQ8HASZ6VUHRZRZ8I8DGWIFY9WJTIRD5P9Y
Next »

How to Fix FATAL error Failed to parse input Document ?

-
SEO Help and Tips How to Fix FATAL error Failed to parse input Document ? When encountering the "FATAL error Failed to parse input document" error, it typically indicates that there is an issue with the code used to parse the document. To fix this error, you can follow these steps: Review the parsing code:   Check the code responsible for parsing the input document. Ensure that it is correctly implemented and follows the correct syntax for the chosen parsing library or approach. Validate the input document:  Before parsing the document, validate its format and structure. Use a document validator or a specific validator for the document type you are parsing to ensure it adheres to the expected standards. This can help identify any syntax errors or inconsistencies in the document. Confirm the input document's compatibility:  Ensure that the parsing library or approach you are using supports the document type you are trying to parse. Some libraries specialize in specific doc...
Next »

How to Reduce Lazy Load Resources

-
How to Reduce Lazy Load Resources in Website Reduce Lazy Load Resources: Utilize lazy loading for resources such as images, scripts, and stylesheets. This means only loading these resources when they are needed, which can help reduce the initial DOM size and improve page load performance. Diagnose performance issues, Performance Values are estimated and may vary. The performance score is calculated directly from these metrics. See calculator. Images:  Lazy loading images can be achieved by setting the loading attribute to "lazy" on the <img> tag. This tells the browser to load the image only when it's about to come into view. <img alt="Lazy-loaded image" src="placeholder.png" data-src="image.png" class="lazyload" /> <script>   document.addEventListener('DOMContentLoaded', function() {     var lazyImages = document.querySelectorAll('img.lazyload');     //...
Next »