To save some bandwidth.
To save some bandwidth.
Just so your bandwidth does not get raped anna I recommend you make a :www.boundanna.com/robots.txt . (the below is a great robots.txt file to use)
User-agent: BoardReader
Disallow: /
User-agent: FAST-WebCrawler
Disallow: /
User-agent: HTTrack
Disallow: /
User-agent: HTTrack 3
Disallow: /
User-agent: NaverBot-1.0 (NHN Corp. / +82-2-3011-1954 / nhnbot@naver.com)
Disallow: /
User-agent: NPBot
Disallow: /
User-agent: Plucker
Disallow: /
User-agent: Slurp
Disallow: /
User-agent: WebCapture 2
Disallow: /
User-agent: WebCapture 3
Disallow: /
User-agent: WebCopier
Disallow: /
User-agent: WebCopier v3.6
Disallow: /
User-agent: Webinator-indexer
Disallow: /
User-agent: WebReaper
Disallow: /
User-agent: WebReaper [webreaper@webreaper.net]
Disallow: /
User-agent: WebReaper [info@webreaper.net]
Disallow: /
User-agent: WebStripper
Disallow: /
User-agent: Wget
Disallow: /
User-agent: *
Disallow: /forum
User-agent: BoardReader
Disallow: /
User-agent: FAST-WebCrawler
Disallow: /
User-agent: HTTrack
Disallow: /
User-agent: HTTrack 3
Disallow: /
User-agent: NaverBot-1.0 (NHN Corp. / +82-2-3011-1954 / nhnbot@naver.com)
Disallow: /
User-agent: NPBot
Disallow: /
User-agent: Plucker
Disallow: /
User-agent: Slurp
Disallow: /
User-agent: WebCapture 2
Disallow: /
User-agent: WebCapture 3
Disallow: /
User-agent: WebCopier
Disallow: /
User-agent: WebCopier v3.6
Disallow: /
User-agent: Webinator-indexer
Disallow: /
User-agent: WebReaper
Disallow: /
User-agent: WebReaper [webreaper@webreaper.net]
Disallow: /
User-agent: WebReaper [info@webreaper.net]
Disallow: /
User-agent: WebStripper
Disallow: /
User-agent: Wget
Disallow: /
User-agent: *
Disallow: /forum
Like that?
http://forum.boundanna.net/robots.txt
http://forum.boundanna.net/robots.txt
- curious_sb
- Retired Moderator
- Posts: 1147
- Joined: 24 Mar 2006, 00:38
- Location: United Kingdom
-
- *
- Posts: 49
- Joined: 07 Nov 2007, 22:45
- Location: England
- curious_sb
- Retired Moderator
- Posts: 1147
- Joined: 24 Mar 2006, 00:38
- Location: United Kingdom
-
- *
- Posts: 49
- Joined: 07 Nov 2007, 22:45
- Location: England
The bots are not forced not to look but I think the majority do.
If you want to configure a server to not accept requests from bots (block them) then it can be done with a .htaccess file on apache (the software running on boundanna.net and most other websites). There is a nice guide at http://www.javascriptkit.com/howto/htaccess13.shtml.
Why is it that no matter what web forum I go on I always end up talking about computers?
If you want to configure a server to not accept requests from bots (block them) then it can be done with a .htaccess file on apache (the software running on boundanna.net and most other websites). There is a nice guide at http://www.javascriptkit.com/howto/htaccess13.shtml.
Why is it that no matter what web forum I go on I always end up talking about computers?
- curious_sb
- Retired Moderator
- Posts: 1147
- Joined: 24 Mar 2006, 00:38
- Location: United Kingdom
Thank you all for the help. But it is really good to block google (and similar) from indexing the site? I realise that allowing the forum to be indexed could be negative for the privacy of the forum users but on the other side it helps people to find us. The privacy is naturally of great concern, especially based on the recent problem for one of our members.
What do the rest of you think? Did any of you find this forum by “accident” through a search engine?
What do the rest of you think? Did any of you find this forum by “accident” through a search engine?
-
- *
- Posts: 49
- Joined: 07 Nov 2007, 22:45
- Location: England
I wouldn't want you to block search engines from indexing the site, I would have never found this site if google didn't index it (I didn't find it by accident, I'm sure you can guess what I was searching for).
a robots.txt file is only usefull if a website is having trouble with bots crawling it very frequently and causing the server problems or the webmaster wants to discourage them from indexing a site. The file above only asks the nasty ones to stay away - not google or the like.
I have not put my name or email address on my profile and I trust Anna and the mods not to make them public. And my username is not my main online alias as many of my friends know what that is. That is about all I think I need to protect my privacy.
a robots.txt file is only usefull if a website is having trouble with bots crawling it very frequently and causing the server problems or the webmaster wants to discourage them from indexing a site. The file above only asks the nasty ones to stay away - not google or the like.
I have not put my name or email address on my profile and I trust Anna and the mods not to make them public. And my username is not my main online alias as many of my friends know what that is. That is about all I think I need to protect my privacy.
This does not block google or most other search engines.Anna wrote:Thank you all for the help. But it is really good to block google (and similar) from indexing the site? I realise that allowing the forum to be indexed could be negative for the privacy of the forum users but on the other side it helps people to find us. The privacy is naturally of great concern, especially based on the recent problem for one of our members.
What do the rest of you think? Did any of you find this forum by “accident” through a search engine?
- curious_sb
- Retired Moderator
- Posts: 1147
- Joined: 24 Mar 2006, 00:38
- Location: United Kingdom
Aha. I thought that this would block everything, including google and such.
User-agent: *
Disallow: /forum
And regarding the privacy: No information will be leaked from this forum by will but there is always a risk that someone hacks the forum and get full access to the database and all user information. It is therefore highly recommended to not keep any personally identifiable information in your profile or in your private messages. It is also recommended not to use any personal email address. Getting a new email address to use for sites like this is a good idea and it only takes a few minutes.
Let me know if you have any problems changing your email addresses.
User-agent: *
Disallow: /forum
And regarding the privacy: No information will be leaked from this forum by will but there is always a risk that someone hacks the forum and get full access to the database and all user information. It is therefore highly recommended to not keep any personally identifiable information in your profile or in your private messages. It is also recommended not to use any personal email address. Getting a new email address to use for sites like this is a good idea and it only takes a few minutes.
Let me know if you have any problems changing your email addresses.