What is robots Text and How it's work ?

-

Web page all contents allowed crawl to Index

The default robots.txt file allows search engine crawl to access all pages and posts on the website. Here is the default robots.txt.

User-agent: Mediapartners-Google
Disallow:

User-agent: *
Disallow: /search
Allow: /

Sitemap: [Your-Blog-URL]/sitemap.xml

The robots.txt provided allows specific user-agents to access different types of content
on the website. Here is a breakdown of the directives in the robots.txt file:

Details Robots txt:

Which Robts.txt uses.

User-agent: *                        (This directive applies to all other user-agents crawlers)
Allow: /                                 (This allows crawling of all other pages on the website)
Disallow: /search                 (This disallows crawling of pages under the /search directory)
User-agent: Mediapartners-Google  (used for Google AdSense)
User-agent: Googlebot                   (used for regular web crawling)
Disallow: /nogooglebot/"                (used for which parts of a website they should not crawl)
User-agent: Adsbot-Google            (used for AdWords campaigns)
User-agent: Googlebot-News          (used for crawling news content)
User-agent: Googlebot-Image         (used for crawling images)
User-agent: Googlebot-Video          (used for crawling video content)
User-agent: Googlebot-Mobile        (used for crawling mobile content)


For all of the specified user-agents, no specific disallow rules are provided, which means
they are allowed to crawl all content on the website.

This robots.txt file allows specific Google user-agents to access all content on the website,
while other user-agents are allowed to access all content except for pages under the
search directory.

Comments

Post a Comment

Thanks for your Comments.

Popular posts from this blog

Office Tool_SPSS v23 + Serial key

-
Best Office Tool SPSS v23 + Serial key Download Activation Instruction: 1. Install IBM SPSS 2. copy “lservrc” into the installed dir(e.g. Windows: C:\Program Files\IBM\SPSS\Statistics\23)  OR 3. enter the following license code using the License Authorization Wizard: Serial key: VDOV7M8KUEIAWBZIKPP6DUKX4JIO3LWRSJQW4BTDCU5NS28ZLZSSROOZQ8HASZ6VUHRZRZ8I8DGWIFY9WJTIRD5P9Y
Next »

How to Fix FATAL error Failed to parse input Document ?

-
SEO Help and Tips How to Fix FATAL error Failed to parse input Document ? When encountering the "FATAL error Failed to parse input document" error, it typically indicates that there is an issue with the code used to parse the document. To fix this error, you can follow these steps: Review the parsing code:   Check the code responsible for parsing the input document. Ensure that it is correctly implemented and follows the correct syntax for the chosen parsing library or approach. Validate the input document:  Before parsing the document, validate its format and structure. Use a document validator or a specific validator for the document type you are parsing to ensure it adheres to the expected standards. This can help identify any syntax errors or inconsistencies in the document. Confirm the input document's compatibility:  Ensure that the parsing library or approach you are using supports the document type you are trying to parse. Some libraries specialize in specific doc...
Next »

How to Reduce Lazy Load Resources

-
How to Reduce Lazy Load Resources in Website Reduce Lazy Load Resources: Utilize lazy loading for resources such as images, scripts, and stylesheets. This means only loading these resources when they are needed, which can help reduce the initial DOM size and improve page load performance. Diagnose performance issues, Performance Values are estimated and may vary. The performance score is calculated directly from these metrics. See calculator. Images:  Lazy loading images can be achieved by setting the loading attribute to "lazy" on the <img> tag. This tells the browser to load the image only when it's about to come into view. <img alt="Lazy-loaded image" src="placeholder.png" data-src="image.png" class="lazyload" /> <script>   document.addEventListener('DOMContentLoaded', function() {     var lazyImages = document.querySelectorAll('img.lazyload');     //...
Next »

Popular posts from this blog

Office Tool_SPSS v23 + Serial key

-
Best Office Tool SPSS v23 + Serial key Download Activation Instruction: 1. Install IBM SPSS 2. copy “lservrc” into the installed dir(e.g. Windows: C:\Program Files\IBM\SPSS\Statistics\23)  OR 3. enter the following license code using the License Authorization Wizard: Serial key: VDOV7M8KUEIAWBZIKPP6DUKX4JIO3LWRSJQW4BTDCU5NS28ZLZSSROOZQ8HASZ6VUHRZRZ8I8DGWIFY9WJTIRD5P9Y
Next »

How to Fix FATAL error Failed to parse input Document ?

-
SEO Help and Tips How to Fix FATAL error Failed to parse input Document ? When encountering the "FATAL error Failed to parse input document" error, it typically indicates that there is an issue with the code used to parse the document. To fix this error, you can follow these steps: Review the parsing code:   Check the code responsible for parsing the input document. Ensure that it is correctly implemented and follows the correct syntax for the chosen parsing library or approach. Validate the input document:  Before parsing the document, validate its format and structure. Use a document validator or a specific validator for the document type you are parsing to ensure it adheres to the expected standards. This can help identify any syntax errors or inconsistencies in the document. Confirm the input document's compatibility:  Ensure that the parsing library or approach you are using supports the document type you are trying to parse. Some libraries specialize in specific doc...
Next »

How to Reduce Lazy Load Resources

-
How to Reduce Lazy Load Resources in Website Reduce Lazy Load Resources: Utilize lazy loading for resources such as images, scripts, and stylesheets. This means only loading these resources when they are needed, which can help reduce the initial DOM size and improve page load performance. Diagnose performance issues, Performance Values are estimated and may vary. The performance score is calculated directly from these metrics. See calculator. Images:  Lazy loading images can be achieved by setting the loading attribute to "lazy" on the <img> tag. This tells the browser to load the image only when it's about to come into view. <img alt="Lazy-loaded image" src="placeholder.png" data-src="image.png" class="lazyload" /> <script>   document.addEventListener('DOMContentLoaded', function() {     var lazyImages = document.querySelectorAll('img.lazyload');     //...
Next »