Gossamer Forum
Home : Products : Gossamer Links : Discussions :

Duplicate links delete

Quote Reply
Duplicate links delete
I use DMOZ extractor to extract categories and links then I import to LSQL. Usually in every import, there are more than a few hundred (sometimes thousands) duplicate links.

Anyone knows where to do simple modification e.g. in Tool.pm or elsewhere so that the number of duplicate links in the list can be extended to 100 or 200?

Also, whether possible to automatically check the check boxes for all the duplicate links except the lowest numbered ID?

Thanks in advance..
Quote Reply
Re: [ccjong] Duplicate links delete In reply to
Paul was going to make some import duplication removal tool, but I don't know how far along he has gotten on that. He did promise, though!

Which Dmoz Extractor program are you using?
Quote Reply
Re: [webslicer] Duplicate links delete In reply to
That will definitely shorten the time I spent on removing duplicate links.

DMOZ Extractor 2 from PJL Technology. It is very easy to extract from DMOZ the categories/sub categories, links (URL) and relevant infomation and it has the option to export to Links 2.0. Files exported to Links 2.0 can be uploaded to LSQL for import into LSQL.

However, it does not have automated process and it requires uploading.