Posts

Showing posts with the label Block Index

How to Stop Index Deleted or Not Found 404 Page ?

SEO Help and Tips How to Stop Index Deleted or Not Found 404 Page ? User-agent: *: This line specifies that the following rule applies to all web crawlers (search engine bots) that access your website. The asterisk * is a wildcard that includes all user agents. User-agent: Googlebot Disallow: /404$ User-agent: * Disallow: /deleted-page User-agent: Googlebot:   This line specifies that the following rules apply specifically to the Googlebot web crawler. Different bots may have different rules specified for them within the same robots.txt file. Disallow: /404$:  This line instructs Googlebot not to crawl any URL that ends with "/404". The "$" symbol at the end indicates that the URL should end with "/404" exactly; it should not match URLs like "/404/error" or "/404/page.html". Disallow: /deleted-page:  This line tells all web crawlers that they are not allowed to crawl or access the page or directory located at /deleted-page on your websi...

Popular posts from this blog

How to Website Disallow and Allow directives by Robots.txt ?

How to Fix Website Mixed Content Issues?

How to Fix Mobile Responsiveness Issues?