Need Help - Out site has been live for sometime and indexed well and then over the past 24 hours google webmasters states it cannot access url due to robots.txt however robots.txt is clean and allows these pages which are blocked.
Current robots.txt
User-agent: * Disallow:
Current .htaccess
RewriteEngine on
RewriteCond %{HTTP_REFERER} semalt\.com [NC] RewriteRule .* - [F]
<ifModule mod_headers.c> Header set Connection keep-alive Header append Cache-Control "public" </ifModule>
<IfModule mod_expires.c> ExpiresActive on
ExpiresDefault "access plus 1 month"
ExpiresByType text/cache-manifest "access plus 0 seconds"
ExpiresByType text/html "access plus 0 seconds"
ExpiresByType text/xml "access plus 0 seconds"
ExpiresByType application/xml "access plus 0 seconds"
ExpiresByType application/json "access plus 0 seconds"
ExpiresByType application/rss+xml "access plus 1 hour"
ExpiresByType image/x-icon "access plus 1 week"
ExpiresByType image/gif "access plus 1 month" ExpiresByType image/png "access plus 1 month" ExpiresByType image/jpg "access plus 1 month" ExpiresByType image/jpeg "access plus 1 month" ExpiresByType video/ogg "access plus 1 month" ExpiresByType audio/ogg "access plus 1 month" ExpiresByType video/mp4 "access plus 1 month" ExpiresByType video/webm "access plus 1 month"
ExpiresByType text/x-component "access plus 1 month"
ExpiresByType font/truetype "access plus 1 month"
ExpiresByType font/opentype "access plus 1 month"
ExpiresByType application/x-font-woff "access plus 1 month"
ExpiresByType image/svg+xml "access plus 1 month"
ExpiresByType application/vnd.ms-fontobject "access plus 1 month"
ExpiresByType text/css "access plus 1 year" ExpiresByType application/javascript "access plus 1 year" ExpiresByType text/javascript "access plus 1 year" </IfModule>
SetOutputFilter DEFLATE AddOutputFilterByType DEFLATE text/html text/css text/plain text/xml application/x-javascript application/x-httpd-php BrowserMatch ^Mozilla/4 gzip-only-text/html BrowserMatch ^Mozilla/4\.0[678] no-gzip BrowserMatch \bMSIE !no-gzip !gzip-only-text/html BrowserMatch \bMSI[E] !no-gzip !gzip-only-text/html SetEnvIfNoCase Request_URI \.(?:gif|jpe?g|png)$ no-gzip Header append Vary User-Agent env=!dont-vary
RewriteEngine On RewriteBase / RewriteCond %{HTTP_HOST} !^www\. [NC] RewriteCond %{HTTP_HOST} ^([^.]+\.[a-z]{2,6})$ [NC] RewriteRule ^(.*)$ http://www.%1/$1 [R=301,L]
<IfModule mod_rewrite.c> RewriteEngine On RewriteBase / RewriteRule ^index\.php$ - [L] RewriteCond %{REQUEST_FILENAME} !-f RewriteCond %{REQUEST_FILENAME} !-d RewriteRule . /index.php [L] </IfModule>
I have deactivated all plugins and still this error is occuring, there is no cache.
I am really struggling with this as I cannot for the life of me understand why these would suddenly start returning block to googlebot.
Btw the fetch page function and robots.txt tester in webmaster tools both work fine, finding and rendering the page and stating the page is allowed.
Any help would be much appreciated.
site:www.drewberryinsurance.co.uk