Blocking search engine spiders

[Not Right in the Head]

I just took a viddy at the list of users online right now, and there were about a dozen search engine spiders, mostly from Yahoo! and Google. While I'll all for these fora being indexed (indeed, SoLow often comes up as one of the top hits for almost any Morrissey-related topic that I search for on Google), it may NOT be wise to have the Downloads/Bootlegs forum indexed. Is the downloads forum blocked in the robots.txt file?

*looks around paranoidly*

 

[Grim O'Grady]

NotRight I'm sorry but I've read your post twice & it's still all geek to me? Wassit mean?

love

Grim

 

[Not Right in the Head]

NotRight I'm sorry but I've read your post twice & it's still all geek to me? Wassit mean?

Search engines send out "spiders," which are programs that go to a Web page, index the content, then pick one of the links on that page, index that one, and so forth, ad infinitum. However, if the spider finds a file called "robots.txt" on the Web server, it's supposed to avoid indexing any folders or files that are listed in it. I'm saying that the downloads forum shouldn't be indexed, which will prevent threads in that forum from showing up in Google & elsewhere. Maybe some people want those things to show up, but when I Google my username + "bootlegs," I don't want to see myself show up in the results. Maybe it's just me.

davidt will know what it means.

 

[Grim O'Grady]

ta for that, makes sense but I'm not sure if davidt will be bothered tbh?

love

Grim

 

[Not Right in the Head]

ta for that, makes sense but I'm not sure if davidt will be bothered tbh?

No, probably not, but I'll feel better having gotten my geek on, if only for a minute.

 

[Grim O'Grady]

No, probably not, but I'll feel better having gotten my geek on, if only for a minute.

love

Grim