When I was in university, back in the twentieth century, I was taking the bus home when I noticed a girl sitting in front of me was reading a textbook entitled “Spider Programming”. At the time, my mind drifted to fanciful notions of mad scientists embedding microchips into Black Widows to control their mind, and create an army of poisonous arachnids. ‘That’s so cool’, I thought, and proceeded to ask the girl if I could join forces with her team of spider controlling maniacs. She quickly moved to the front of the bus…
I rushed home to jump on the MUD through my 9600baud modem and ask my computer dork friends exactly what spider programming was. After about 40 lines of IRC technical mumbo-jumbo, I hacked them all to bits with my tenth level Ranger.
It wasn’t until many years, and a turn of the century, later when I finally understood what a ’spider’ (aka Bot, aka Robot, aka Cylon (okay, not really)) was. And for the uninitiated, here is the definition:
Web Spider - a program or automated script which browses the World Wide Web in a methodical, automated manner. Other less frequently used names for web crawlers are ants, automatic indexers, bots, and worms.
In a world where all you want is to get your fancy new website to the top of the Google/Yahoo charts, you will find yourself craving visits from their friendly, neighborhood web spiders. And here are some tips to increase your electronic arachnids’ traffic.
- Update your site frequently. The more sophisticated bots utilized by Google and Yahoo attempt to mimic a real user, and what do real users like? Stuff that is new! Stuff that is relevant! Stuff that is awesome! More stuff means more visits, more visits mean more credibility, and more credibility means more ranks.
- Use your ROBOTS.TXT file wisely! The Robots Exclusion Standard is the best way to tell these little human fakers where they should be going, because although they’re well designed code, they still have an IQ of zero. Nobody likes to see they’re web-test folders and wp-content folders showing up in the index, so know when to say Disallow!
- DON’T USE FLASH TO RENDER TEXT!!! By all means, make your site fancy and pretty and all those other things that make you feel like a uber-swank web developer, but don’t embed your 40,000 character information page in a SWF file! Its also good to avoid building your navagation menu in Flash, as the navigation menu is what the bots use to associate the deep pages in your content rich site, and if its all flash, no spider’s gonna be able to read it… If you embed all your outbound links in a SWF, may God have mercy on your soul…
- Design your site in WordPress! No, I’m serious. I’ve seen WordPress sites get spidered and indexed in a matter of hours. For some reason that I’m still trying to ascertain, Google loves Blogs more than Joni loves Chachi. Or, if you’ve got it in your head that the five years you spent in school learning AJAX and by God your going to program your site in AJAX, get yourself some link love from your hip computer dork friend’s blog.
I could continue further about Spiders, but my mind keeps drifting to this William Shatner movie I saw when I was a child, and I can’t seem to focus. I’ll continue this later…