There was a time when Inktomi almost ruled the web in the way that Google almost does now. And it was free, just as Google is right now. But these days, I think of my old friend Inktomi, the one who brought me so many top ten hits for my clients over a range of engines such as MSN, AOL, HotBot and others, as an aging prize-fighter.

It still has style, technique and most certainly, some staying power. But its once powerful punch is weaker than when in its prime and its popularity is waning. No longer can you rely on it to serve up winning matches in the big search arenas. Now, you’ll find it frequently KO’d by the shifty Looksmart at mighty MSN and relegated to second rate matches at back street venue HotBot.

But wait: it’s even worse than that. You now have to shell out your hard earned dollars to see these mediocre results, and sometimes it doesn’t even turn up for the match!

There’s a massive amount of speculation within the industry over the future of Inktomi. And there’s an enormous state of flux where results are concerned within the Terra Lycos Network, which includes HotBot. Lycos partnered up with Fast for results and pretty soon Inktomi was ousted from HotBot UK and replaced by Fast, leaving only the HotBot US site with Inktomi powered results. But, the first page of results at hotbot.com are still powered by DirectHit. And DirectHit is now part of Teoma. Confused? You will be!

And this, I believe, is where the recent “glitch” I spotted comes from. I’ve just completed updating my book to cover the recent changes, including Yahoogle! Goohoo! or whatever its known as now ;-).

During the course of this, I began to notice some irregularities over at HotBot. I have some excellent top ten results for my clients in the Inktomi database. And they enjoyed excellent traffic from AOL when it served up Inktomi results (before the Google giant ho, ho, ho’d its way in). So why wasn’t I seeing any of them on the second page at HotBot following the DirectHit results? Was it just me? Had I slipped up somewhere and lost all of those top ten positions?

As a subscriber to the Inktomi pay for inclusion program via Position Technologies, I have direct access to the Inktomi database, which means I can run a check on the existence of URL’s within the database as well as ranking. So, I did that. And, bang, there they were! So why weren’t they turning up at HotBot? I did another check, and another… I was beginning to think that the results may have been coming from somewhere else (either that, or I had been consulting with my colleague Jack Daniels a little more than I should :-)). And then I realised, after spotting one of my client’s pages at number 14 in Inktomi: sure enough, he was turning up at HotBot it was only the top ten pages that my clients had which weren’t!

More checking done and I was at the bottom of it. Even though the results on the second page at HotBot do come from Inktomi they actually start at the top of the second page with the number 11 result i.e. the top ten Inktomi results are obliterated by the preceding ten from DirectHit!

You could call this just a technical anomaly. You could also call it insult to injury. My clients always ask me if I can get them into the top ten results at search engines. That’s the goal and that’s what I aim for. Sometimes, depending on the competitiveness of the key-phrase, it’s not so difficult. Now it’s rather like the sublime to the ridiculous: “Mike, please could you guarantee me a number 11 hit. Or, actually, 12 or 14 would be good too.” Come on! If I do my job as best I can (like every good SEM scout should ;-)) I’m aiming for a top ten hit.

I put a call through to Danny Sullivan (venerable Yoda of SEM) to discuss this (amongst other things): “It’s a bug,” he declared. Danny spoke to Lycos and mentioned this to them: something they were not unaware of he discovered. I had reason to speak to Jim Stob over at Position Technologies (another SEM Master) and mentioned it in passing. Yes he said, on certain searches that happens. What certain searches? It’s happening on all of mine. What are they going to do about it? Well, at this point in time, it’s not high priority, as Lycos (as some may be aware) have already stated that there are changes coming to HotBot in the near future. So, we could see Fast results there as you do at HotBot UK. But then again … who knows. Perhaps, like me, nothing surprises you anymore when it comes to search engines and partnerships.

Okay, let’s look at the real deal here. Downside and upside. Here’s the downside to begin with.

I have to explain to my clients why some hundreds of URL’s they have paid Position Technologies for, and which feature in the top ten results at Inktomi, will not be seen at HotBot. I can do that and explain: “HotBot is hardly a major traffic driver for us when compared to Google.” That’s fair enough: we’re not going to sink because of it. But because these clients have signed up with Inktomi for one year, it means we’ll continue to “pay to be ignored” for one year over at HotBot if no immediate solution is forthcoming. Nonetheless, we have the mighty MSN where Inktomi results appear … of a sort.

Pop over to MSN and run a search on “digital cameras”. What do you see? First, you’ll see results from weareinbedwiththesesites.com (Ooops, I mean MSN Featured Sites). Next, you’ll see wetakecontrolofyourcash.com (Ooops, I mean sponsored sites from Overture :-)). Next you’ll see wehaveyoubythe*****ifyouwanttobeinmsn.com (Ooops, I mean MSN directory results from Looksmart) and then… well… then!

Here’s where you begin to wonder if your pages were submitted by Jules Verne, because right down there at number 550 (close to the centre of the earth) come your Inktomi results. Now explain this to your clients when they ask you: “If we can’t be found at HotBot because of the ‘glitch’ and we’re number one at Inktomi which makes us number 550 at MSN; just run it past us again: why did we pay for that?”

The upside. You’re Amazon. You spend thousands and thousands of dollars to get trusted feed to XML all those zillions of pages into the Inktomi database at a vastly reduced price because of the volume. Someone does a search at MSN for: Schaum’s Outline of Biochemistry. You come in at number two. Holy smoke! We nearly sold a 12 dollar book.

Question: How many search engines does it take to change a light bulb?

Answer: None, they prefer to work in the dark!

Part two:

I’m just about to mail this out to current purchasers of my book to let them know the free, courtesy upgrade is ready. The whole Yahoo! thing has been splattered all over the web so there’s nothing new to report. But, if you take a look at the HotBot thing above, and the updates below: the only thing you’re missing is the rest of my book (about 298 pages ;-)).

Yahoo!:

All results now from Google. But those which are also included in the Yahoo! directory are ‘flagged’ with a red arrow and the Google snippet is replaced with the Yahoo! title and description. To do well at Yahoo!, you need to do well at Google. Do you need to pay to be in Yahoo! now? If you have a good rank with Google: no. If you have a poor rank with Google: yes. This will add a very important link to your PageRank score.

Inktomi:

There is no free submission to Inktomi anymore. You need to use pay for inclusion as in the service offered by Position Technologies. Inktomi results only really figure at MSN and HotBot. At MSN primary results are provided by Looksmart. If you are chasing very competitive key-phrases i.e. ‘digital cameras’ DON’T bother with Inktomi as you will be buried by Looksmart results. If you are chasing key-phrases for more esoteric subjects i.e. ‘cryogenic containers’ DO use Inktomi. In the long term it’s much cheaper than paying USD 0.15 per click at Looksmart (remember to review the section on optimising for Inktomi).

HotBot only uses Inktomi results from the second page onwards. And because of a ridiculous glitch, the results on the second page start from number 11 in the Inktomi database. If you have a top ten hit at Inktomi, it will never be found at HotBot. Terra Lycos which owns HotBot has given me an indication that a major change is due very soon. Expect to see results possibly from Fast.

My best practice advice for this update? Don’t lose sleep over PageRank at Google. Sure, this is the most important place to be at this time. PageRank is only one (important) factor in the overall algo. Google wants to see/read like a human being. Think about what’s natural for a human visitor in the way you develop your pages. Don’t fake linkage ever. A link from a knitting pattern site to a site for extreme skiing can be spotted a mile away.

Finally, when seeking linking partners, remember this protocol for writing for the web: link with little power:

For more details on knitting patterns click here (where ‘click here’ is the anchor text).

Link with great power:

Click here for more details on knitting patterns (where ‘knitting patterns’ is the anchor text).

One more final thing: don’t develop a web site for knitting patterns if you want to be an Internet millionaire :-).

Search Engine marketing: The essential best practice guide is now updated to November 2002. You can download it here: