I wish I hadn't read this thread before. Funkmaster has really put up a good show
I maintain a community website at http://www.ahmedabad.com and I have implemented a Search Engine from http://www.htdig.org which is a open source code. I shall recommend all of you to have a look at the search engine I have put up and you shall agree that its the FASTEST among all the products discussed above.
Secondly, http://www.khoj.com is one thing I was looking forward to implement and I landed up at this wonderfull site. If you see this website they have something like "Links" (I am not sure) along with htdig search engine working as Altavista. So, they are pretty successfull in implementing a scalable search directory/engine. If the directory results finish at lets say 5 matches it continues with the results of HTDIG. Thats the best part of it. So, the browser will always get something related to its search keyword.
Now, can I request Funkmaster HUGE to give me some idea to implement the Category search which I like it very much and I am unable to figure it out.
Thanks
Sam