Gossamer Forum
Home : Products : Gossamer Links : Discussions :

DMOZ... can I really use telnet to transfer dmoz..

Quote Reply
DMOZ... can I really use telnet to transfer dmoz..
I have been reading the DMOZ threads and I am a little unclear... Can I telnet into my (virtually hosted) server and type in commands that will transfer files (by category) directly over the internet from the http://dmoz.org site to http://mysite.com, my site?

Example I read in these threads:

./nph-import.cgi --import=RDF --source=/PATH/TO/CONTENT.RDF --destination=/PATH/TO/ADMIN/DEFS --rdf-category="Top/Regional/North_America/Canada" --rdf-add-date="2001-01-01"

If this is really possible, could some really nice person PLEEEASE post the COMPLETE instructions for doing this (step by step), somewhere (A REAL/exact example works great for me, too!)... A good example :-) could be: what should I type (and where) to transfer "Top: Regional: North America: United States: Alaska" from dmoz.org to my site(http://homerusa.net)... the server where I host my domain (ftp) has a different address?

Thanks
Ryan
HomerUSA

Quote Reply
Re: DMOZ... can I really use telnet to transfer dmoz.. In reply to
NEVER MIND TELNET, but... What's next?

First of all, I am amazed! ! ! I transferred the whole dmoz.gz to my hosts server in 5 minutes! ! ! All I did was type what Paul Wilson wrote, via telnet:

Telnet> wget http://www.dmoz.org/rdf/content.rdf.u8.gz

...and 5 minutes later (maybe 10 :-) I have the whole zipped file on my server! (My host already had Wget installed)

NOW WHAT?
1. I unzip the .gz file? (how do I unzip a file via telnet?)
2. I type what Alex wrote, exactly?(what is this nph-import.cgi?):

./nph-import.cgi --import=RDF --source=/SERVER/ROOT/PATH/TO/CONTENT.RDF --destination=/SERVER/ROOT/PATH/TO/ADMIN/DEFS --rdf-category="Top/Regional/North_America/United_States/Alaska/" --rdf-add-date="2001-01-01"

3. I'd like to keep the whole 'regional' section on my server if it's not too big... does anyone know what size it is (I keep hearing the largest part of the dmoz file is Adult?)?
4. Does anyone know how to delete categories in 'BROWSE' and NOT have to delete every link within that category?

And I'm done? Can anyone please tell me if this is what I need to do next(how)?

Thanks again
Ryan
HomerUSA

Quote Reply
Re: DMOZ... can I really use telnet to transfer dmoz.. In reply to
Unzip it with.....

shell> tar -zxvf content.rdf.u8.gz

(BEWARE it is 700MB)

nph-import.cgi comes with Links SQL and is in your admin directory.

Then from telnet run the command I mentioned in the other thread and it will start importing.

Regional contains about 600,000 links and is about 75-100MB gzipped.

Oh by the way - I don't know who told you about the Adult category being the biggest - it is actually one of the smallest.

Paul Wilson.
http://www.wiredon.net/gt/
http://www.perlmad.com/
Quote Reply
Re: [RedRum] DMOZ... can I really use telnet to transfer dmoz.. In reply to
Just tried executing the tar comand you gave, but it returns;

content.rdf.u8.gz public_html
sh-2.04$ tar -zxvf content.rdf.u8.gz
tar: This does not look like a tar archive
tar: Skipping to next header

I just did a direct download from dmoz to my webserver..so i had no contact directly with it. Is it possible it got corrupt?


Andy (mod)
andy@ultranerds.co.uk
Want to give me something back for my help? Please see my Amazon Wish List
GLinks ULTRA Package | GLinks ULTRA Package PRO
Links SQL Plugins | Website Design and SEO | UltraNerds | ULTRAGLobals Plugin | Pre-Made Template Sets | FREE GLinks Plugins!
Quote Reply
Re: [AndyNewby] DMOZ... can I really use telnet to transfer dmoz.. In reply to
Possibly.

You don't need to unzip it in any case.
Quote Reply
Re: [RedRum] DMOZ... can I really use telnet to transfer dmoz.. In reply to
Oh, why dont i need to unzip it? Does nph-import.cgi do it for me?

Thanks

Andy (mod)
andy@ultranerds.co.uk
Want to give me something back for my help? Please see my Amazon Wish List
GLinks ULTRA Package | GLinks ULTRA Package PRO
Links SQL Plugins | Website Design and SEO | UltraNerds | ULTRAGLobals Plugin | Pre-Made Template Sets | FREE GLinks Plugins!
Quote Reply
Re: [AndyNewby] DMOZ... can I really use telnet to transfer dmoz.. In reply to
Hi,

It's not a tar file. Just do:

gzip -d content.rdf.u8.gz

or

gunzip content.rdf.u8.gz

Cheers,

Alex
--
Gossamer Threads Inc.
Quote Reply
Re: [Alex] DMOZ... can I really use telnet to transfer dmoz.. In reply to
Ahhh...thanks Smile

Andy (mod)
andy@ultranerds.co.uk
Want to give me something back for my help? Please see my Amazon Wish List
GLinks ULTRA Package | GLinks ULTRA Package PRO
Links SQL Plugins | Website Design and SEO | UltraNerds | ULTRAGLobals Plugin | Pre-Made Template Sets | FREE GLinks Plugins!
Quote Reply
Re: [HomerUSA] DMOZ... can I really use telnet to transfer dmoz.. In reply to
Would you mind to told me the every steps to convert the dmoz database into Links SQL through Telnet?Tongue
Quote Reply
Re: [erichk] DMOZ... can I really use telnet to transfer dmoz.. In reply to
only 2 step to do that:

one : download the content.rdf.u8.gz
tow: running lsql2's nph_import.cgi


_____________
anyone import dmoz data secuccess :
the dmoz.org links only 100,000 links ?

my MSN: perlchina_at_hotmail.com
Quote Reply
Re: [tsingson] DMOZ... can I really use telnet to transfer dmoz.. In reply to
There is another step too ;)

You need to decompress the content.rdf.u8.gz file after downloading it Wink

Quote:
anyone import dmoz data secuccess :
the dmoz.org links only 100,000 links ?

I've imported over 3.5million links before with Links SQL. It can handle 100,000 links without a problem (unless you are on a really poor server).

Cheers

Andy (mod)
andy@ultranerds.co.uk
Want to give me something back for my help? Please see my Amazon Wish List
GLinks ULTRA Package | GLinks ULTRA Package PRO
Links SQL Plugins | Website Design and SEO | UltraNerds | ULTRAGLobals Plugin | Pre-Made Template Sets | FREE GLinks Plugins!
Quote Reply
Re: [Andy] DMOZ... can I really use telnet to transfer dmoz.. In reply to
Quote:
You need to decompress the content.rdf.u8.gz file after downloading it

No you don't.
Quote Reply
Re: [Paul] DMOZ... can I really use telnet to transfer dmoz.. In reply to
You do if you want the import process to go faster. If you leave it to decompress while running, it puts resources up a lot, and also takes longer to do the import Wink

Cheers

Andy (mod)
andy@ultranerds.co.uk
Want to give me something back for my help? Please see my Amazon Wish List
GLinks ULTRA Package | GLinks ULTRA Package PRO
Links SQL Plugins | Website Design and SEO | UltraNerds | ULTRAGLobals Plugin | Pre-Made Template Sets | FREE GLinks Plugins!