robots.txt
Posted: 21. November 2011 21:12i decided to create a robots.txt file in my root folder when i submitted my site to be indexed by several different search engines. i used a basic generator for the file that i found online as i truthfully have no idea what text needs to be in the file.
i checked my error log today to see if anything has happened since i created the file. i'm getting several "file does not exist" errors for the robots.txt file when the search engines try to find it. i purposely set the file as "read only" when i created it, trying to make sure i have protection from anyone attempting to edit the file that isn't me (i had problems with this once before, then i found out that i can use the built in security features that xampp has to keep this from happening.) i know those features are probably enough, but i still wanted to be sure.
my question is, is this what is causing the spiders to not be able to view the file? i wouldn't think it would. also, will this affect my rankings at all if they can't view robots.txt? i always thought that if there was no robots.txt or if the spiders couldn't access it for some reason, they would just skip it and then continue crawling as normal.
i checked my error log today to see if anything has happened since i created the file. i'm getting several "file does not exist" errors for the robots.txt file when the search engines try to find it. i purposely set the file as "read only" when i created it, trying to make sure i have protection from anyone attempting to edit the file that isn't me (i had problems with this once before, then i found out that i can use the built in security features that xampp has to keep this from happening.) i know those features are probably enough, but i still wanted to be sure.
my question is, is this what is causing the spiders to not be able to view the file? i wouldn't think it would. also, will this affect my rankings at all if they can't view robots.txt? i always thought that if there was no robots.txt or if the spiders couldn't access it for some reason, they would just skip it and then continue crawling as normal.