I am testing phpdig and sphider cause I want to have a search function on a web page BUT I have a problem:
Indexing process is blocked all the time. The funny thing is that I am able to index other sides than mine with phpdig as well as with sphider.
I tried working with robot.txt as well as deleting my htaccess file - without success.
Is there any other apache specific configuration that keeps robots blocked?
Thanks a lot!