Gossamer Forum
Home : Products : Gossamer Links : Discussions :

HELP: DMOZ import, missing + incomplete data

Quote Reply
HELP: DMOZ import, missing + incomplete data
I've "attempted" to import the .rdf from dmoz. The file was content.rdf.gz, 139 MB compressed and 700+ uncompressed.

My import took 26 hours (400MHZ PII / 256 MB Ram) and only brought in 456,000 links. Dmoz shows 2.3 million so I am way off. As far as I can tell, the import didn't die. CPU was nearly maxed at 85-90% and mem usage was around 10% during import.

When you go to the dynamic site, I only have the following categories.
- Adult (0)
- Arts (0)
- Business (0)
- Computers (0)

Beside each category the number of links listed is zero (0). If you navigate a few catagories inside some have numbers (137) and others are empty (0) although they DO HAVE DATA IN THAT CATEGORY. So non-empty categories show (0) even though there are plenty of links there.

Am I missing something here, do I need to do anything after such a large import? If so, will someone be so kind as to detail the steps AFTER import to make everything golden?

One other thing, what .rdf files are to be used? content.rdf.gz OR content.rdf.u8.gz ? There are differences and this may be the issue.

Here is my import command, hope it's right:
./nph-import.cgi --import RDF --destination=/usr/local/apache/htdocs/scripts/links/admin/defs --source="content.rdf" --rdf-category="Top" --rdf-add-date="2001-02-05"

Can someone help me out here, I'm having issues and with a 26 hour import it's hard to keep doing it, over and over again to get it right.

Lastly, do we need to import the CONTENT and STRUCTURE dumps?




Subject Author Views Date
Thread HELP: DMOZ import, missing + incomplete data lisco 2911 Feb 7, 2001, 10:39 AM
Thread Re: HELP: DMOZ import, missing + incomplete data
Paul 2836 Feb 7, 2001, 11:27 AM
Thread Re: HELP: DMOZ import, missing + incomplete data
padders 2833 Feb 7, 2001, 11:46 AM
Post Re: HELP: DMOZ import, missing + incomplete data
Paul 2832 Feb 7, 2001, 11:50 AM
Thread Re: HELP: DMOZ import, missing + incomplete data
Alex 2827 Feb 7, 2001, 12:06 PM
Post Re: HELP: DMOZ import, missing + incomplete data
lisco 2821 Feb 7, 2001, 2:17 PM