To save some bandwidth.

Mc_ntk · Post by **Mc_ntk** » 13 Dec 2007, 06:16

Just so your bandwidth does not get raped anna I recommend you make a :www.boundanna.com/robots.txt . (the below is a great robots.txt file to use)

User-agent: BoardReader
Disallow: /

User-agent: FAST-WebCrawler
Disallow: /

User-agent: HTTrack
Disallow: /

User-agent: HTTrack 3
Disallow: /

User-agent: NaverBot-1.0 (NHN Corp. / +82-2-3011-1954 / nhnbot@naver.com)
Disallow: /

User-agent: NPBot
Disallow: /

User-agent: Plucker
Disallow: /

User-agent: Slurp
Disallow: /

User-agent: WebCapture 2
Disallow: /

User-agent: WebCapture 3
Disallow: /

User-agent: WebCopier
Disallow: /

User-agent: WebCopier v3.6
Disallow: /

User-agent: Webinator-indexer
Disallow: /

User-agent: WebReaper
Disallow: /

User-agent: WebReaper [webreaper@webreaper.net]
Disallow: /

User-agent: WebReaper [info@webreaper.net]
Disallow: /

User-agent: WebStripper
Disallow: /

User-agent: Wget
Disallow: /

User-agent: *
Disallow: /forum

Post by **anna** » 13 Dec 2007, 18:19

Like that?

http://forum.boundanna.net/robots.txt

jake · Post by **jake** » 13 Dec 2007, 18:33

...yay a text file?

curious_sb · Post by **curious_sb** » 14 Dec 2007, 00:16

I dont understand, what does this do??

countdown321 · Post by **countdown321** » 14 Dec 2007, 00:54

Search engines and other bots can use lots of bandwidth on some sites when they trawl through them for data. This sort of file asks them not to.

curious_sb · Post by **curious_sb** » 14 Dec 2007, 01:16

cool, does it work? is there any directive to force robots to adhere to these requests or is it a case of some robots will honour the requests and others dont?

countdown321 · Post by **countdown321** » 14 Dec 2007, 11:11

The bots are not forced not to look but I think the majority do.

If you want to configure a server to not accept requests from bots (block them) then it can be done with a .htaccess file on apache (the software running on boundanna.net and most other websites). There is a nice guide at http://www.javascriptkit.com/howto/htaccess13.shtml.

Why is it that no matter what web forum I go on I always end up talking about computers?

curious_sb · Post by **curious_sb** » 14 Dec 2007, 18:59

cos your like me mate your a computer engineer and thats where the topic always ends up, and the only reason family calls me, to fix their computers, like I enjoy it in my spare time or something...lol

Post by **anna** » 14 Dec 2007, 20:20

Thank you all for the help. But it is really good to block google (and similar) from indexing the site? I realise that allowing the forum to be indexed could be negative for the privacy of the forum users but on the other side it helps people to find us. The privacy is naturally of great concern, especially based on the recent problem for one of our members.

What do the rest of you think? Did any of you find this forum by “accident” through a search engine?

countdown321 · Post by **countdown321** » 14 Dec 2007, 22:29

I wouldn't want you to block search engines from indexing the site, I would have never found this site if google didn't index it (I didn't find it by accident, I'm sure you can guess what I was searching for).

a robots.txt file is only usefull if a website is having trouble with bots crawling it very frequently and causing the server problems or the webmaster wants to discourage them from indexing a site. The file above only asks the nasty ones to stay away - not google or the like.

I have not put my name or email address on my profile and I trust Anna and the mods not to make them public. And my username is not my main online alias as many of my friends know what that is. That is about all I think I need to protect my privacy.

Mc_ntk · Post by **Mc_ntk** » 15 Dec 2007, 02:38

Anna wrote:Thank you all for the help. But it is really good to block google (and similar) from indexing the site? I realise that allowing the forum to be indexed could be negative for the privacy of the forum users but on the other side it helps people to find us. The privacy is naturally of great concern, especially based on the recent problem for one of our members.

What do the rest of you think? Did any of you find this forum by “accident” through a search engine?

This does not block google or most other search engines.

curious_sb · Post by **curious_sb** » 15 Dec 2007, 11:59

I found boundanna using google during one of my many "google image" searches for the words:

bound
tied
ropes
bondage
self-bondage
self bondage
scenario
torture
sex games
bdsm

so without indexing, I would not have found the site.

Post by **anna** » 15 Dec 2007, 12:38

Aha. I thought that this would block everything, including google and such.
User-agent: *
Disallow: /forum

And regarding the privacy: No information will be leaked from this forum by will but there is always a risk that someone hacks the forum and get full access to the database and all user information. It is therefore highly recommended to not keep any personally identifiable information in your profile or in your private messages. It is also recommended not to use any personal email address. Getting a new email address to use for sites like this is a good idea and it only takes a few minutes.
Let me know if you have any problems changing your email addresses.