Re: Google Spiders
The spider
The first part of what Google does is crawl the web. I'll get onto that in a moment but there a few things we need to cover first. Whenever you read an SEO article, the verbs ‘crawl’ and ‘spider’ and (sometimes) ‘index’ are interchangeable. Google sends out what’s affectionately known as a ‘spider’ to ‘crawl’ the web, going from link to link creating a web (geddit?), and cataloging, or ‘indexing’ what it sees. If you wanted to mimic the actions of the Google spider, start at a big site, like Wikipedia, and click every link you see. Then go to those pages, and click every link and so on. Once you’ve clicked a few hundred billion pages, you’ll be somewhere close.
Google’s spider can only crawl linked pages by the way - it can only access pages that you could access with a mouse. It can’t enter in login details or passwords, and it can’t enter terms in your search bar. As a general rule, if your page isn’t accessible by a clickable link, Google probably won’t see it.
I found this on the net.