Gossamer Forum
Home : Products : Gossamer Links : Version 1.x :

Dmoz Parser

Quote Reply
Dmoz Parser
As this dmoz file gets bigger and bigger, a lot of people are having problems with parsing it. Just to let you know, we have someone working on a parser that will run off the site. Basically just enter in your login information, and the category you want to parse, and a couple hours later we'll email you when it's automatically parsed (and if someone else already requested that, you may get it immediately). It will be available as either a mysqldump of the file, or as two tab delimited files.

This should be up and running friday.

Cheers,

Alex

--
Gossamer Threads Inc.
Quote Reply
Re: Dmoz Parser In reply to
Very cool, ....but (and you knew it was coming), how do you plan to sufficiently plan around the ID field in the Links table?

Wayne Hunt
Amiga.org

Quote Reply
Re: Dmoz Parser In reply to
Great! :)

This requires a lot of horse power... and my horses were running out of spinach :)



Quote Reply
Re: Dmoz Parser In reply to
Alex,

That's a valid thought.

Since links have to be imported with a category attached, how do you ensure that the added category ID will continue to match the Link ID? You'd need a script to import the categories, verify the ID's, and assign the links that ID. A simple dump/load would only work for empty databases.

Quote Reply
Re: Dmoz Parser In reply to
On the form you can (optionally) enter in a starting Link ID and CategoryID. As long as you don't add anything to the database then it should be fine.

Cheers,

Alex

--
Gossamer Threads Inc.
Quote Reply
Re: Dmoz Parser In reply to
I think most of the times it's a space problem (expanding the 600mb file).

Is it possible to store the uncompressed file at a public server, and do a "remote parse" and import the output screen to your local Links-SQL db?

Quote Reply
Re: Dmoz Parser In reply to
Any time frame on when this feature will be ready? Just aching to try this out!!

Trust in your elders, for they hold the key to life...
Quote Reply
Re: Dmoz Parser In reply to
Hi, I know Alex is busy...
Just wondering if there is any new news about this?? I am really interested in importing a specific category (or 2) to our stuffquest.com site. We have decided not to add any more enhancements, etc. to the site until the new version of LinksSQL comes out, and now just seems like the perfect time to work on this. (While we have some spare time.) I check this forum about once a week, and haven't seen any new news about it... the last post was in may. Anybody hear anything???

=-=-=-=-=-=-=-=-=-=-=-=-=-=-=
Dianne Spyridon
http://www.imagineworks.net
Quote Reply
Re: Dmoz Parser In reply to
Sorry about the late reply! It's now available, it takes quite a while to run (depending on what category you want anywhere from 10 minutes to 2-3 hours). You can go to:

http://www.gossamer-threads.com/perl/installs/dmoz.cgi

to get your dmoz category. =) Your username/password is the same as your download area password. Contact me if you are unsure of it.

Cheers,

Alex

--
Gossamer Threads Inc.
Quote Reply
Re: Dmoz Parser In reply to
Question,
Once you fill out the dmoz.cgi form, it says "9 hours to process". Where do you go to get the file or mysqldump??

Trust in your elders, for they hold the key to life...
Quote Reply
Re: Dmoz Parser In reply to
I put my request in, and the script told me it would take approx 90 hours.. Is it really that backed up, or was it a mistake in the script?

Also, I have the same question as Kilroy.. how are we notified that the parse is complete, and where do we pick it up?

You guys have really created a terriffic product here, and the support has been outstanding. I also appreciate that you are willing to go the extra mile (ie this new parsing script) as a work around for common problems. VERY cool!

Thanks! :))
Katina

Quote Reply
Re: Dmoz Parser In reply to
We had some problems with the parser (it was taking up too much of the webservers resources). We have since moved the parsing to another machine and will start it going. There are about 30-40 requests in the quene. You should get an email as soon as it's done.

Cheers,

Alex

--
Gossamer Threads Inc.
Quote Reply
Re: Dmoz Parser In reply to
Thanks, Alex.. I appreciate both the update and the quick reply! :)

Katina

Quote Reply
Re: [Alex] Dmoz Parser In reply to
Hi Alex !!!
where is dmoz.cgi ????
thanks


Thanks in Advance
Bye From Italy
Quote Reply
Re: [fabio] Dmoz Parser In reply to
Erm, arn't you using Links SQL 2??? This is the v1 forums Wink

Andy (mod)
andy@ultranerds.co.uk
Want to give me something back for my help? Please see my Amazon Wish List
GLinks ULTRA Package | GLinks ULTRA Package PRO
Links SQL Plugins | Website Design and SEO | UltraNerds | ULTRAGLobals Plugin | Pre-Made Template Sets | FREE GLinks Plugins!