From what I have read about Nameprotect and their web-bots, I shouldn’t have been surprised that they are lying when they claim to respect robots.txt:
[…]Practices the following best practices to ensure non-invasive crawling:
Honoring robots.txt files – to exclude the NPBot crawler, please use “NPBot” as the user-agent name in your robots.txt file […]
I have added the necessary line to my robots.txt file to exclude the NPBot from my site, but they are still indexing without caring about what the robots.txt file says.
Its also worth noting that Nameprotect have been (and probably still are) using spiders which are lying about who they are. (more information about “crawler918.com” / Nameprotect)
So my suggestion, if you want to get rid of the Nameprotect bots, is to block their IP-range from accessing your web pages:
Add the following to your .htaccess file:
Deny from 12.148.209.192/26
See my earlier post about NPBot
also deny 12.148.209.198
Why? Its covered by the deny I suggested…
# More robots to deny
# cyveillance.com
deny from 63.148.99.224/27
deny from 65.118.41.192/27
# branddimensions.com user-agent: BDFetch
deny from 204.92.59.0/24
# http://www.markwatch.com user-agent: markwatch
deny from 204.62.224.0/22
deny from 204.62.228.0/23
deny from 206.190.160.0/19
# rocketinfo.com
deny from 209.167.132.224/28
#taken from posting at http://mcgerik.com/scott/archives/corporate_america_is_reading_your_blog_thu_12_jun_2003.php
This is my opinion of Name Protect… As a tech for several domains that are only used for e-mail services it really is interesting to watch who will hit a site after a DNS update, etc without ever being invited. The number one IP to hit the server is always Name Protect and then VigilActive – one in the same usually running on US West Internet. You can read more about their “partners” here:
http://www.ltutech.com/News_press_releases_NP.htm
or:
http://www.ltutech.com
The other uninvited visitors that hit the server uninvited are:
Mozilla/5.0 (Slurp/cat; [email protected]; http://www.inktomi.com/slurp.html
And: Scooter/3.3
I have read a lot about privacy issues in regards to Google, as in the link below, but I have yet to ever have a site hit by Google uninvited – that doesn’t mean that there are concerns but the bot comes invited and obeys robot text. I have NEVER seen Inktomi obey anything and their polocies are not listed and all attempts to have their bot halted fail; that always makes me very unhappy.
http://news.bbc.co.uk/1/hi/technology/2786761.stm
Scooter belogs to AltaVista, but it is still uninvited, does NOT look for robots text first. From my point of view I don’t care who owns the bot when it does not look for robots text FIRST.
Other things I’ve seen and heard about are all of the web content rating bots – there are rumors that many pass the information off to the government under subcontracts as several of these companies that operate in California and Oregon were almost bankrupt then along came Ashcroft to save the day.
I would also strongly urge blocking all IP’s associated with the following using a Qwest – US West Communications netblock:
Cyveillance
Other bad bots include RuleSpace and Web Content International as I’ve read in several places on the Internet that RuleSpace and Web Content International are the same and that some Mormon friend of the RuleSpace guy has registered IP’s used by that company under the name Web Content International. If true, US West QWEST should not allow this to happen!
Other people have posted remarks that Name Protect and VigilActive may have connections with the former since the IP’s and bots from Name Protect and VigilActive have shown up in logs more and more.
I’m not sure if this is true, but I think there are some companies that may be offering a service that will block a lot of this junk and traffic from their servers by using dynamic updates of the IP’s these companies use. I heard it will use the same technology that these companies but in reverse. If true it serves them right!
In any case I would never use U.S. West QWEST services (dial-up, DSL, phone, hosting, etc.) based on the junk I’ve seen in my logs from them and the things they allow on their networks. Funny, effective in 1999 these two companies “merged” so junk from QWEST is junk from U.S. West! Then we have the securities fraud cases against Philip Anschutz… So, it’s not just bad bots and junk IP traffic that comes from these people.
Dear “VigilActivity”
This is just not true about Google in the BBC story on your web site story above. Don’t you think it is rather odd that google-watch talks a lot about WebmasterWorld as it is some “other” website. Don’t you think it is funny that webmasterworld and google-watch use the same servers WESTSERVER.NET in Providence, Utah. Vigil, Providence, Utah is a nowhere place in the Cache Valley of Utah near Logan. The more I find out about google-watch and webmasterworld and phd software LOL the more I wonder about who really runs these things? the American FBI, CIA or NSA or the Mormon Mafia. I would never use google-watch and I think you should do a little more research on these things as everything is not what they seem. The same with webmasterworld that is headquarted in a town of less than 2,000 people in the middle of nowhere U.S.A. Vigil, things are not always what they seem and before you think I’m a conspiracy nut, one of the most used proxy sites a few years ago turned out to be funded through the CIA. Of course the American FBI and CIA just love to recruit in Utah and if you think I’m kidding, do a few searches on that we hope without using google-watch. NPBot, they have partners who also have U.S.A. contracts. Knowledge is power and you need to make sure you know who you are dealing with, NPBot or google-watch.
Migel is right!
I came here looking for information on NPBot and I ended up more interested in the google-watch and webmasterworld story. It really does make you wonder who these people really are and how long it will take to hide their name servers now that the Utah link has been compromised. I’ve looked at some information on webmasterworld and to be honest it was my opinion that much of the information is suspect. On the other hand, why does google allow google-watch to use the proxy? Something doesn’t add up here and I’m sure that Daniel Brandt and Brett Tabke will inform us soon as to why they both have the same connection to Providence, Utah LOL If they and/or google are the American Government it won’t be long until this information is deleted in bots but I received the following e-mail yesterday evening so I guess the word will stay out there:
Subject: Webmasters Alert! Alert! Alert!
The number one google lover in the world, Brett Tabke, who runs webmasterworld http://www.webmasterworld.com and the number one google hater in the world, Daniel Brandt, who runs google-watch http://www.google-watch.org have their servers traced to the same address, 200 N Spring Creek Parkway, Providence, UT 84332 The odds of this being random may range a million-to-one and if something doesn’t look right, it usually isn’t right. Please pass this information to other interested webmasters ASAP!
INFORMATION FOR WEBMASTERWORLD THE WORLD’S NUMBER ONE LOVER OF GOOGLE:
Domain Name: WEBMASTERWORLD.COM
Registrar: IHOLDINGS.COM, INC. D/B/A DOTREGISTRAR.COM
Whois Server: whois.dotregistrar.com
Referral URL: http://www.dotregistrar.com
Name Server: NS.WESTHOST.NET
Name Server: NS2.WESTHOST.NET
Status: ACTIVE
Updated Date: 18-aug-2002
Creation Date: 11-dec-1999
Expiration Date: 11-dec-2004
INFORMATION FOR WESTHOST.NET:
Domain name: westhost.net
Registrant Contact:
WestHost Inc.
WestHost Hostmaster ([email protected])
+1.4357553433
Fax: +1.4357553449
200 N Spring Creek Parkway
Providence, UT 84332
US
///////////////////////////////////////////////////////////////////////
INFORMATION FOR GOOGLE-WATCH THE WORLD’S NUMBER ONE “HATER” OF GOOGLE:
Domain ID:D87793486-LROR
Domain Name:GOOGLE-WATCH.ORG
Created On:23-Jun-2002 14:52:35 UTC
Last Updated On:12-Sep-2003 23:04:55 UTC
Expiration Date:23-Jun-2004 14:57:46 UTC
Sponsoring Registrar:R91-LROR
Status:OK
Registrant ID:GODA-01182811
Registrant Name:Daniel Brandt
Registrant Street1:PO Box 680635
Registrant City:San Antonio
Registrant State/Province:Texas
Registrant Postal Code:78268-0635
Registrant Country:US
Registrant Phone:+1.2105093160
Registrant FAX:+1.2105093161
Registrant Email:[email protected]
Admin ID:GODA-21182811
Admin Name:Daniel Brandt
Admin Street1:PO Box 680635
Admin City:San Antonio
Admin State/Province:Texas
Admin Postal Code:78268-0635
Admin Country:US
Admin Phone:+1.2105093160
Admin Email:[email protected]
Tech ID:GODA-11182811
Tech Name:Daniel Brandt
Tech Street1:PO Box 680635
Tech City:San Antonio
Tech State/Province:Texas
Tech Postal Code:78268-0635
Tech Country:US
Tech Phone:+1.2105093160
Tech Email:[email protected]
Name Server:NS3.WESTSERVER.NET
Name Server:NS4.WESTSERVER.NET
INFORMATION FOR WESTSERVER.NET:
Billing Contact:
WestHost Inc.
WestHost Hostmaster ([email protected])
+1.4357553433
Fax: +1.4357553449
200 N Spring Creek Parkway
Providence, UT 84332
US
Status: registrar-lock
Name Servers:
NS3.westserver.net
ns4.westserver.net
Another note on this e-mail that I received
I did some looking around on both of the sites last night and it is like their views of google are totally different. So why do they both end up traced to a server company in Providence, Utah? How many people do you know who use server companies in Providence, Utah. I’ll admit, none of this makes sense to me! FBI, CIA, it looks odd so user beware.