How to Stop Index Deleted or Not Found 404 Page ?

-
SEO Help and Tips

How to Stop Index Deleted or Not Found 404 Page ?

User-agent: *: This line specifies that the following rule applies to all web crawlers (search engine bots) that access your website. The asterisk * is a wildcard that includes all user agents.

User-agent: Googlebot
Disallow: /404$

User-agent: *
Disallow: /deleted-page

User-agent: Googlebot: 
This line specifies that the following rules apply specifically to the Googlebot web crawler. Different bots may have different rules specified for them within the same robots.txt file.

Disallow: /404$: 
This line instructs Googlebot not to crawl any URL that ends with "/404". The "$" symbol at the end indicates that the URL should end with "/404" exactly; it should not match URLs like "/404/error" or "/404/page.html".

Disallow: /deleted-page: 
This line tells all web crawlers that they are not allowed to crawl or access the page or directory located at /deleted-page on your website. The /deleted-page is an example of a specific directory or page that you want to exclude from being crawled.

The page you would want to prevent search engines from indexing is the custom "404 Not Found" error page. This is the page that users are directed to when they try to access a URL on your website that does not exist.

Also you can add code your page.
Here's an example of how you can add the "noindex" meta tag to the HTML code of your custom "404 Not Found" error page:

<head>
    <!-- Other meta tags and page information -->
    <meta name="robots" content="noindex">
</head>
<body>
    <!-- Content of the custom 404 Not Found error page -->
    <h1>404 Not Found</h1>
    <p>The page you are looking for does not exist.</p>
</body>

By adding the "noindex" meta tag as shown above, you're instructing search engines not to index this specific "404 Not Found" error page, even if they crawl it.

Again, the "robots.txt" file controls crawling behavior, and the "noindex" meta tag or HTTP header is used to control indexing behavior. In this case, you want to use the "noindex" meta tag on the custom error page to prevent it from being indexed by search engines.

Comments

Post a Comment

Thanks for your Comments.

Popular posts from this blog

Office Tool_SPSS v23 + Serial key

-
Best Office Tool SPSS v23 + Serial key Download Activation Instruction: 1. Install IBM SPSS 2. copy “lservrc” into the installed dir(e.g. Windows: C:\Program Files\IBM\SPSS\Statistics\23)  OR 3. enter the following license code using the License Authorization Wizard: Serial key: VDOV7M8KUEIAWBZIKPP6DUKX4JIO3LWRSJQW4BTDCU5NS28ZLZSSROOZQ8HASZ6VUHRZRZ8I8DGWIFY9WJTIRD5P9Y
Next »

How to Fix FATAL error Failed to parse input Document ?

-
SEO Help and Tips How to Fix FATAL error Failed to parse input Document ? When encountering the "FATAL error Failed to parse input document" error, it typically indicates that there is an issue with the code used to parse the document. To fix this error, you can follow these steps: Review the parsing code:   Check the code responsible for parsing the input document. Ensure that it is correctly implemented and follows the correct syntax for the chosen parsing library or approach. Validate the input document:  Before parsing the document, validate its format and structure. Use a document validator or a specific validator for the document type you are parsing to ensure it adheres to the expected standards. This can help identify any syntax errors or inconsistencies in the document. Confirm the input document's compatibility:  Ensure that the parsing library or approach you are using supports the document type you are trying to parse. Some libraries specialize in specific doc...
Next »

How to Reduce Lazy Load Resources

-
How to Reduce Lazy Load Resources in Website Reduce Lazy Load Resources: Utilize lazy loading for resources such as images, scripts, and stylesheets. This means only loading these resources when they are needed, which can help reduce the initial DOM size and improve page load performance. Diagnose performance issues, Performance Values are estimated and may vary. The performance score is calculated directly from these metrics. See calculator. Images:  Lazy loading images can be achieved by setting the loading attribute to "lazy" on the <img> tag. This tells the browser to load the image only when it's about to come into view. <img alt="Lazy-loaded image" src="placeholder.png" data-src="image.png" class="lazyload" /> <script>   document.addEventListener('DOMContentLoaded', function() {     var lazyImages = document.querySelectorAll('img.lazyload');     //...
Next »

Popular posts from this blog

Office Tool_SPSS v23 + Serial key

-
Best Office Tool SPSS v23 + Serial key Download Activation Instruction: 1. Install IBM SPSS 2. copy “lservrc” into the installed dir(e.g. Windows: C:\Program Files\IBM\SPSS\Statistics\23)  OR 3. enter the following license code using the License Authorization Wizard: Serial key: VDOV7M8KUEIAWBZIKPP6DUKX4JIO3LWRSJQW4BTDCU5NS28ZLZSSROOZQ8HASZ6VUHRZRZ8I8DGWIFY9WJTIRD5P9Y
Next »

How to Fix FATAL error Failed to parse input Document ?

-
SEO Help and Tips How to Fix FATAL error Failed to parse input Document ? When encountering the "FATAL error Failed to parse input document" error, it typically indicates that there is an issue with the code used to parse the document. To fix this error, you can follow these steps: Review the parsing code:   Check the code responsible for parsing the input document. Ensure that it is correctly implemented and follows the correct syntax for the chosen parsing library or approach. Validate the input document:  Before parsing the document, validate its format and structure. Use a document validator or a specific validator for the document type you are parsing to ensure it adheres to the expected standards. This can help identify any syntax errors or inconsistencies in the document. Confirm the input document's compatibility:  Ensure that the parsing library or approach you are using supports the document type you are trying to parse. Some libraries specialize in specific doc...
Next »

How to Reduce Lazy Load Resources

-
How to Reduce Lazy Load Resources in Website Reduce Lazy Load Resources: Utilize lazy loading for resources such as images, scripts, and stylesheets. This means only loading these resources when they are needed, which can help reduce the initial DOM size and improve page load performance. Diagnose performance issues, Performance Values are estimated and may vary. The performance score is calculated directly from these metrics. See calculator. Images:  Lazy loading images can be achieved by setting the loading attribute to "lazy" on the <img> tag. This tells the browser to load the image only when it's about to come into view. <img alt="Lazy-loaded image" src="placeholder.png" data-src="image.png" class="lazyload" /> <script>   document.addEventListener('DOMContentLoaded', function() {     var lazyImages = document.querySelectorAll('img.lazyload');     //...
Next »