IP Addresses of Search Engine Spiders

Are you interested in tracking the IP addresses of search engine spiders? You can see my latest lists of search engine IP addresses and User-Agents here:

Google IP List
List of the IP addresses of the various Googlebots and Mediabots.
Last revised Thursday, 01-May-2008 19:07:35 EDT Size: 4k

Yahoo IP List
List of the IP addresses of Yahoo/Inktomi bots.
Last revised Monday, 28-Apr-2008 17:17:12 EDT Size: 11k

Lycos IP List
List of the IP addresses of Lycos bots.
Last revised Tuesday, 15-Nov-2005 11:41:27 EST Size: 1k

InfoSeek IP List
List of the IP addresses of InfoSeek bots.
Last revised Monday, 07-Jun-2004 13:23:45 EDT Size: 1k

Alta Vista IP List
List of the IP addresses of Alta Vista bots.
Last revised Monday, 25-Sep-2006 22:37:21 EDT Size: 2k

Excite IP List
List of the IP addresses of Excite bots.
Last revised Wednesday, 12-Apr-2006 17:55:34 EDT Size: 1k

Northern Light IP List
List of the IP addresses of Northern Light bots.
Last revised Wednesday, 31-Jul-2002 14:13:28 EDT Size: 1k

Misc IP List
List of the IP addresses of various search engines' bots.
Last revised Monday, 28-Apr-2008 17:17:13 EDT Size: 61k

Non Spiders
List of IP addresses and some User Agents of non-search engine spider bots, along with some other pertinent information.
Last revised Monday, 18-Apr-2005 11:25:10 EDT Size: 7k

Here is a record of updates to the IP lists.

A reorganized list is available here.

Notes

Tracing search engine IP addresses is handy for a number of reasons. You might want to see when your web site was spidered by search engines. You might want to know which pages where spidered. You might want to ban some bots and let others in. You might want to see which bots are obeying your robots.txt files. You might want to redirect bots to other pages or serve them different content (cloaking).

Keeping track of the various IP addresses belonging to search engine spiders is a difficult enterprise and requires the assistance of a large number of people. I'm constantly receiving reports of new spiders and doing the research necessary to verify that they are indeed new and that they do indeed belong to search engines. The lists are as up to date as possible, but there is no guarantee that they include all of the IP addresses of all search engines (it would be impossible to guarantee such a thing). Also, there is no guarantee that all of the IP addresses contained in the lists belong to search engines (they are sometimes reassigned or abandoned).

I do accept submissions of suspected bot activity. If you see some bot activity on your web site and want to submit it, please do the following:

  1. Record the line from your server's access log.
  2. Email me with the line from your server's access log.
  3. Please don't send email if it isn't about a new bot or directly related to this site. I can't track down the owners of IP addresses or figure out where spam is coming from... but I have written a good tutorial on how to track down spammers.

The IP addresses are contained in a flatfile database format, which is protected under intellectual property law and treaties, however the IP addresses themselves are public domain and not protected. What this means to you is that you can't copy this website, but feel free to download the lists of IP addresses and use them however you like.

If you are writing an application to download the IP addresses automatically, please ask me for permission first.

FYI -- your IP address is: 38.103.63.16 .

If this site has been helpful to you, you might consider buying me a drink! ;)

Resources

This website is (C) Copyright 2001 by Volatile Graphix, Inc. The IP lists themselves are public domain and not subject to copyright protection.