Gossamer Forum: General: Internet Technologies: Internal Search Engine

Dec 19, 2003, 3:22 AM

Andy

Veteran / Moderator (17298 posts)

Dec 19, 2003, 3:22 AM

Post #1 of 8

Shortcut

Internal Search Engine

Does anyone have any suggestions for an internal search engine (preferably will run via cron/ssh), and can handle 100,000+ pages?

It also needs to have the option to block specific folders, cos the LinksSQL one on the site in question has 500,000+ category pages Frown

TIA

Andy (mod)
andy@ultranerds.co.uk

IMPORTANT: I've now moved to ultranerds.co.uk, and the .com will no longer work!
Want to give me something back for my help? Please see my Amazon Wish List
GLinks ULTRA Package (plugins total "value" $3,325 & rising, for just $350)| GLinks ULTRA Package PRO (plugins total "value" $5,625 & rising, for just $500)
Support Forum | Links SQL Plugins | DMOZ Dumps | UltraNerds | ULTRAGLobals Plugin | Pre-Made Template Sets | FREE GLinks Plugins!
Compare our different Plugin packages *new* Free CSS Templates

Dec 19, 2003, 3:34 AM

Wil

Veteran / Moderator (4108 posts)

Dec 19, 2003, 3:34 AM

Post #2 of 8

Shortcut

Re: [Andy] Internal Search Engine In reply to

"There is no such thing as a free lunch."

:-)

- wil

Dec 19, 2003, 6:38 AM

Chaz

Enthusiast (819 posts)

Dec 19, 2003, 6:38 AM

Post #3 of 8

Shortcut

Re: [Andy] Internal Search Engine In reply to

Try ht://dig. I haven't tried it myself but from what I gather, it's pretty good.

~Charlie

Dec 19, 2003, 6:40 AM

Andy

Veteran / Moderator (17298 posts)

Dec 19, 2003, 6:40 AM

Post #4 of 8

Shortcut

Re: [Chaz] Internal Search Engine In reply to

Yeah, Wil recommended that on MSN, but I'm not sure if it supports the selection of which categories to spider?

Cheers

Andy (mod)
andy@ultranerds.co.uk

Dec 19, 2003, 6:49 AM

Chaz

Enthusiast (819 posts)

Dec 19, 2003, 6:49 AM

Post #5 of 8

Shortcut

Re: [Andy] Internal Search Engine In reply to

I spotted this after a quick read through the docs:

Quote:

#
# If there are particular pages that you definitely do NOT want to index, you
# can use the exclude_urls attribute. The value is a list of string patterns.
# If a URL matches any of the patterns, it will NOT be indexed. This is
# useful to exclude things like virtual web trees or database accesses. By
# default, all CGI URLs will be excluded. (Note that the /cgi-bin/ convention
# may not work on your web server. Check the path prefix used on your web
# server.)
#
exclude_urls: /cgi-bin/ .cgi

#
# Since ht://Dig does not (and cannot) parse every document type, this
# attribute is a list of strings (extensions) that will be ignored during
# indexing. These are *only* checked at the end of a URL, whereas
# exclude_url patterns are matched anywhere.
#
bad_extensions: .wav .gz .z .sit .au .zip .tar .hqx .exe .com .gif \
.jpg .jpeg .aiff .class .map .ram .tgz .bin .rpm .mpg .mov .avi .css

Will that not do what you want? It looks like you can exclude folders, files and even specific file extensions.

I really just glanced over the docs so you might want to dig (no pun intended :) ) into it a little deeper to see if it will work for you.

~Charlie

Dec 19, 2003, 6:54 AM

Andy

Veteran / Moderator (17298 posts)

Dec 19, 2003, 6:54 AM

Post #6 of 8

Shortcut

Re: [Chaz] Internal Search Engine In reply to

Thanks, I didn't see that. I'll pass that link onto my mate, and hopefully this script won't crash the server every time its run Pirate