Gossamer Forum
Home : Products : Gossamer Links : Discussions :

dmoz dump web to server automatically

Quote Reply
dmoz dump web to server automatically
I am on dialup, (uk is behind...) anyway is there any easy script that you can put on a server that tells it to go and get the file from the web without requiring the user to stay connected, ie you can press GO and it works...

I assume this is how DMOZ thing works, we put the dump on our servers, untar it and then import into links and go about deleting what we don't want. I need to import most of >Arts so i would think this is easiest way. Is my server (PIII 650 256MB) going to die on me?

Thanks...

http://www.ASciFi.com/ - The Science Fiction Portal
Quote Reply
Re: dmoz dump web to server automatically In reply to
You can use LWP. I posted a 4 line program quite awhile back that would do that.

PUGDOGŪ
PUGDOGŪ Enterprises, Inc.
FAQ: http://pugdog.com/FAQ


Quote Reply
Re: dmoz dump web to server automatically In reply to
thank you i will try and search for it,

http://www.ASciFi.com/ - The Science Fiction Portal
Quote Reply
Re: dmoz dump web to server automatically In reply to
i am assuming it will be this download:
structure.rdf.u8.gz

as opposed to the non u8 format.

Am i right?

http://www.ASciFi.com/ - The Science Fiction Portal
Quote Reply
Re: dmoz dump web to server automatically In reply to
well i downloaded it, and it downloaded at 300kbs which i was pretty impressed with :)

http://www.ASciFi.com/ - The Science Fiction Portal
Quote Reply
Re: dmoz dump web to server automatically In reply to
Oops, you want the content.rdf not structure. =)

Cheers,

Alex

--
Gossamer Threads Inc.
Quote Reply
Re: dmoz dump web to server automatically In reply to
arr i thought i needed both that is ok but is it:

http://dmoz.org/rdf/content.rdf.gz
http://dmoz.org/rdf/content.rdf.u8.gz

cos they are discontinuing the non .u8 one soon i think.

http://www.ASciFi.com/ - The Science Fiction Portal