Gossamer Forum
Home : Products : Gossamer Links : Discussions :

Importing DMOZ into LinksSQL using custom application

Quote Reply
Importing DMOZ into LinksSQL using custom application
I developed my own python app to parse through the dmoz content rdf file and write all links to a CSV file.

I'm curious, how do I go about getting this into the correct MySQL tables used by LinksSQL.

Note, that I'm not talking about how to get the contents of the CSV file into MySQL, that is a piece of cake, and I've already inserted all 3 million plus links into a table I designed in approximately 4 minutes.

What I need to know, is which tables are involved, and is it even possible? I really don't have time to pick apart import.cgi. But perhaps that is what I have to do.

TIA!

Side note: One thing for the newbies here, never rely on a software program to insert into a database if possible. Always write to a CSV file and use the database's load application to load the CSV file. It is several times faster to do it this way rather than via insert statements. A java app I wrote to parse and insert urls from the DMOZ rdf file maxed out at 10 inserts per second. If you do the math, you'll see that loading the entire DMOZ in about 4 minutes blows that away by days!
Subject Author Views Date
Thread Importing DMOZ into LinksSQL using custom application takacsj 4012 May 8, 2003, 7:21 AM
Thread Re: [takacsj] Importing DMOZ into LinksSQL using custom application
Andy 3956 May 8, 2003, 7:28 AM
Thread Re: [Andy] Importing DMOZ into LinksSQL using custom application
takacsj 3944 May 8, 2003, 10:04 AM
Post Re: [takacsj] Importing DMOZ into LinksSQL using custom application
Andy 3934 May 9, 2003, 12:55 AM