Gossamer Forum
Home : Products : Gossamer Links : Development, Plugins and Globals :

[ New Plugin ] DMOZ_Wizard v1

(Page 1 of 2)
> >
Quote Reply
[ New Plugin ] DMOZ_Wizard v1

DMOZ_Wizard

Details: http://new.linkssql.net/...amp;page=DMOZ_Wizard

Price: $80 << BUY >>
Availability: Requires Purchasing
Links SQL Version Required:
2.1

Requirements;

  • Links SQL 2.1+

  • UNIX Type Server

  • SSH/Telnet Access

  • At least 2Gb of free space (for downloading, decompressing etc of the content.rdf.u8.gz RDF file) [/url]

Features;[/url]

  • Option to send email after slicing/import...

  • Option to download a remote .sql file, to place in database...

  • Option to execute a local .sql file, to place in database...[/url][/url]

  • Ability to clean database before an import...

  • Ability to backup the database before an import...[/url][/url]

  • Choice of importing 'Top' categories, or a selection of your own choice.

  • Option to check if the category in question exists.

  • ..and more. [/url]

Does;

[/url]

I originally wrote this Plugin for myself, but soon realised how much more use it would be to other LinksSQL users. Basically, what this plugin does is write about 2-3 commands for each category Dump. This makes the whole import process go faster, because you don't have to sit there manually and painstakingly type out all the messages. In short, this Plugin writes a .cgi script, which you then run from Telnet/SSH to do the complete import (from downloading the content.rdf.u8.gz file, to repairing the database count after the installs).

Cheers



Andy (mod)
andy@ultranerds.co.uk
Want to give me something back for my help? Please see my Amazon Wish List
GLinks ULTRA Package | GLinks ULTRA Package PRO
Links SQL Plugins | Website Design and SEO | UltraNerds | ULTRAGLobals Plugin | Pre-Made Template Sets | FREE GLinks Plugins!
Quote Reply
Re: [Andy] [ New Plugin ] DMOZ_Wizard v1 In reply to
Why not read the gzip and then people only need 200MB of disk space, not 2GB Shocked
Quote Reply
Re: [Paul] [ New Plugin ] DMOZ_Wizard v1 In reply to
Good idea. It would also save time, because it doesn't need to decompress it Smile I'll make the change in a minute.

Cheers

Andy (mod)
andy@ultranerds.co.uk
Want to give me something back for my help? Please see my Amazon Wish List
GLinks ULTRA Package | GLinks ULTRA Package PRO
Links SQL Plugins | Website Design and SEO | UltraNerds | ULTRAGLobals Plugin | Pre-Made Template Sets | FREE GLinks Plugins!
Quote Reply
Re: [Andy] [ New Plugin ] DMOZ_Wizard v1 In reply to
BUG FIX: It wouldn't install correctly on some UNIX servers (problem with the mkdir() code).

Cheers

Andy (mod)
andy@ultranerds.co.uk
Want to give me something back for my help? Please see my Amazon Wish List
GLinks ULTRA Package | GLinks ULTRA Package PRO
Links SQL Plugins | Website Design and SEO | UltraNerds | ULTRAGLobals Plugin | Pre-Made Template Sets | FREE GLinks Plugins!
Quote Reply
Re: [Andy] [ New Plugin ] DMOZ_Wizard v1 In reply to
Version 1.1 is now out.

This include a couple of fixes (if a user enters a / at the beginning of the category name, it will get rid of it, otherwise theimport would not work), and also has a new feature.

When importing specific category now, you can do stuff like;

/Computers/Internet/CyberCafes:::/Cafes

This would import the /Computers/Internet/CyberCafes category from DMOZ, into a category called 'Cafes' in your database.

This update is available for download here: http://new.linkssql.net/page.php?page=account

Cheers

Andy (mod)
andy@ultranerds.co.uk
Want to give me something back for my help? Please see my Amazon Wish List
GLinks ULTRA Package | GLinks ULTRA Package PRO
Links SQL Plugins | Website Design and SEO | UltraNerds | ULTRAGLobals Plugin | Pre-Made Template Sets | FREE GLinks Plugins!
Quote Reply
Re: [Andy] [ New Plugin ] DMOZ_Wizard v1 In reply to
Andy ,

The link to your site is down.

CCUnet
my Christian web
Quote Reply
Re: [ccunet] [ New Plugin ] DMOZ_Wizard v1 In reply to
Odd. May have gone down for a little bit. Seems to be online now.

Cheers

Andy (mod)
andy@ultranerds.co.uk
Want to give me something back for my help? Please see my Amazon Wish List
GLinks ULTRA Package | GLinks ULTRA Package PRO
Links SQL Plugins | Website Design and SEO | UltraNerds | ULTRAGLobals Plugin | Pre-Made Template Sets | FREE GLinks Plugins!
Quote Reply
Re: [Andy] [ New Plugin ] DMOZ_Wizard v1 In reply to
Due to some updates on DMOZ's site, the RDF file has changed from;

http://www.dmoz.org/rdf/content.rdf.u8.gz

...to...

http://rdf.dmoz.org/rdf/content.rdf.u8.gz

I have made this change in my script, and the current version is now 1.2.

Existing members can download the update from here;

http://new.linkssql.net/page.php?page=account

Cheers

Andy (mod)
andy@ultranerds.co.uk
Want to give me something back for my help? Please see my Amazon Wish List
GLinks ULTRA Package | GLinks ULTRA Package PRO
Links SQL Plugins | Website Design and SEO | UltraNerds | ULTRAGLobals Plugin | Pre-Made Template Sets | FREE GLinks Plugins!
Quote Reply
Re: [Andy] [ New Plugin ] DMOZ_Wizard v1 In reply to
Andy,

Whats the best way to get support for this.

Have several questions.

a) Can I configure the default to be a content.* dump in a common directory? I don't want a 200MB download to be done several times.

b) Can you confirm that the line Arts/Movies/Home_Video/DVD:::/Websites is correct?

c) I want it to create the categories under /Arts/Movies/Home_Video/DVD in my local directory under /Websites how can this be done?
Quote Reply
Re: [shrirch] [ New Plugin ] DMOZ_Wizard v1 In reply to
Hi.

a) You would just need to edit dmoz_cron.cgi. Just comment out lines 26-28.

b) Arts/Movies/Home_Video/DVD:::Websites . Note there is no / in front of the destination category.

c) Yup... just use the above.

Hope that answers your questions Smile

Cheers

Andy (mod)
andy@ultranerds.co.uk
Want to give me something back for my help? Please see my Amazon Wish List
GLinks ULTRA Package | GLinks ULTRA Package PRO
Links SQL Plugins | Website Design and SEO | UltraNerds | ULTRAGLobals Plugin | Pre-Made Template Sets | FREE GLinks Plugins!
Quote Reply
Re: [Andy] [ New Plugin ] DMOZ_Wizard v1 In reply to
Another bug fix.

v1.2.1

This fix is to enable the plugin to 'check' if the custom category you are looking at importing, does in fact exist. It does this by checking the DMOZ site.

Previously it kept hanging, and didn't seem to actually do anything. I've edited the script a bit, and it appears to be working ok now :)

The updated version can be downloaded from:
http://new.linkssql.net/page.php?page=account

Cheers

Andy (mod)
andy@ultranerds.co.uk
Want to give me something back for my help? Please see my Amazon Wish List
GLinks ULTRA Package | GLinks ULTRA Package PRO
Links SQL Plugins | Website Design and SEO | UltraNerds | ULTRAGLobals Plugin | Pre-Made Template Sets | FREE GLinks Plugins!
Quote Reply
Re: [Andy] [ New Plugin ] DMOZ_Wizard v1 In reply to
UPDATE: v1.3

This version has a bug fix in it for foreign charachters in the World/Regional categories. iconv must be enabled on your server for this to work.

I've also added a few more extras/fixes. The main ones being;

I've now made it so that you can decide if you want to re-download the content.rdf.u8.gz file (tick box on the setup). This is a great way to save your bandwidth :)

You can now run dmoz_cron.cgi via Crontab (stops timeout problems on some servers). It is also a lot faster. I've had it doing 100,000 links in the space of 10 minutes.

The updated version can be downloaded from:
http://new.linkssql.net/page.php?page=account

This is for the people who already own the plugin, or own the $200 package.

Cheers

Andy (mod)
andy@ultranerds.co.uk
Want to give me something back for my help? Please see my Amazon Wish List
GLinks ULTRA Package | GLinks ULTRA Package PRO
Links SQL Plugins | Website Design and SEO | UltraNerds | ULTRAGLobals Plugin | Pre-Made Template Sets | FREE GLinks Plugins!
Quote Reply
Re: [Andy] [ New Plugin ] DMOZ_Wizard v1 In reply to
In Reply To:
It is also a lot faster. I've had it doing 100,000 links in the space of 10 minutes.


Hey that is not true.. it took only 9 minutes for me..

Need Scripts

Vishal
-------------------------------------------------------
Quote Reply
Re: [NeedScripts.Com] [ New Plugin ] DMOZ_Wizard v1 In reply to
Depends on your server Wink Either way... it seems to be a lot quicker now :)

Cheers

Andy (mod)
andy@ultranerds.co.uk
Want to give me something back for my help? Please see my Amazon Wish List
GLinks ULTRA Package | GLinks ULTRA Package PRO
Links SQL Plugins | Website Design and SEO | UltraNerds | ULTRAGLobals Plugin | Pre-Made Template Sets | FREE GLinks Plugins!
Quote Reply
Re: [Andy] [ New Plugin ] DMOZ_Wizard v1 In reply to
v1.4

Ok, the `iconv` fix kinda fixed the problem (with a few charachters), but there were still a lot of bad ones in there it seems. I've got a script that comes with the DMOZ_Wizard now, that will go through these bad charachters, and replace them with their appropriate ones.

This version is available from the download page at: http://new.linkssql.net/page.php?page=account

Cheers

Andy (mod)
andy@ultranerds.co.uk
Want to give me something back for my help? Please see my Amazon Wish List
GLinks ULTRA Package | GLinks ULTRA Package PRO
Links SQL Plugins | Website Design and SEO | UltraNerds | ULTRAGLobals Plugin | Pre-Made Template Sets | FREE GLinks Plugins!
Quote Reply
Re: [Andy] [ New Plugin ] DMOZ_Wizard v1 In reply to
Hello Andy,

I cannot use Telnet/SSH on my waiter.
There is another solution?


Like Spider, script works intermittently?

I would not like that my waiter causes me a stop.
I am lodged on a mutualized waiter...


Thank you for your answer


Mick
Quote Reply
Re: [mick31] [ New Plugin ] DMOZ_Wizard v1 In reply to
Quote:
I cannot use Telnet/SSH on my waiter.
There is another solution?

Afraid not... it would time out too quick.

Quote:
I would not like that my waiter causes me a stop.
I am lodged on a mutualized waiter...

Its not recommended that you run the DMOZ_Wizard on a shared server anyway, as it can take a lot of CPU power to complete.

Cheers

Andy (mod)
andy@ultranerds.co.uk
Want to give me something back for my help? Please see my Amazon Wish List
GLinks ULTRA Package | GLinks ULTRA Package PRO
Links SQL Plugins | Website Design and SEO | UltraNerds | ULTRAGLobals Plugin | Pre-Made Template Sets | FREE GLinks Plugins!
Quote Reply
Re: [Andy] [ New Plugin ] DMOZ_Wizard v1 In reply to
Version 1.5 is now out.

+ Cronjob Feature Added

I have added a feature, which will allow you to setup (without intervention) a cronjob. This basically means you don't even need to go into SSH to run the process. Literally just setup the process as you normally would, and set the time you want it to run, and voila Cool

Enjoy :)

Cheers

Andy (mod)
andy@ultranerds.co.uk
Want to give me something back for my help? Please see my Amazon Wish List
GLinks ULTRA Package | GLinks ULTRA Package PRO
Links SQL Plugins | Website Design and SEO | UltraNerds | ULTRAGLobals Plugin | Pre-Made Template Sets | FREE GLinks Plugins!
Quote Reply
Re: [Andy] [ New Plugin ] DMOZ_Wizard v1 In reply to
Version 1.6 is now available from the download area. There are no extra features, but the code has been cleaned up, and the GUI has been cleaned up too, so its a lot easier to use Smile

http://www.ultranerds.com/cgi-bin/details/10.html

Cheers

Andy (mod)
andy@ultranerds.co.uk
Want to give me something back for my help? Please see my Amazon Wish List
GLinks ULTRA Package | GLinks ULTRA Package PRO
Links SQL Plugins | Website Design and SEO | UltraNerds | ULTRAGLobals Plugin | Pre-Made Template Sets | FREE GLinks Plugins!
Quote Reply
Re: [Andy] [ New Plugin ] DMOZ_Wizard v2.3 In reply to
Version 2.3 is now available from the download area.

New features include the removal of "iconv" from the import process, as it is now redundent, after the latest changes for the way foreign charachters are handled.

Charachters in UTF-8 format should now be ok for imports. This basically means that it will support the importing of categories in languages such as French, German, Spanish, Italian, etc.

http://www.ultranerds.com/cgi-bin/details/10.html

NB: For those who are upgrading, all you need to do is upload the new version, and overwrite the existing one. Please note, you *may* have to re-enter the email address in the plugin settings.

Cheers

Andy (mod)
andy@ultranerds.co.uk
Want to give me something back for my help? Please see my Amazon Wish List
GLinks ULTRA Package | GLinks ULTRA Package PRO
Links SQL Plugins | Website Design and SEO | UltraNerds | ULTRAGLobals Plugin | Pre-Made Template Sets | FREE GLinks Plugins!
Quote Reply
Re: [Andy] [ New Plugin ] DMOZ_Wizard v2.3 In reply to
Seems there was a bug in this version :(

For those of you using it, you can fix it up by opening up /admin/Plugins/DMOZ_Wizard.pm, and finding;

Code:
print OVERWRITE
#qq|print "Running iconv on $admin_dir/$save_cat.dump, and transfering to $admin_dir/$save_cat.dump.slice ... ";|
# . "\n";
# print OVERWRITE "\`iconv -c -f UTF-8 -t UTF-8 "
# . $admin_dir . "/"
# . $save_cat
# . ".dump > $admin_dir/$save_cat.dump.slice\`;";

Simpley comment out the first line; i.e;

Code:
# print OVERWRITE
#qq|print "Running iconv on $admin_dir/$save_cat.dump, and transfering to $admin_dir/$save_cat.dump.slice ... ";|
# . "\n";
# print OVERWRITE "\`iconv -c -f UTF-8 -t UTF-8 "
# . $admin_dir . "/"
# . $save_cat
# . ".dump > $admin_dir/$save_cat.dump.slice\`;";

The stable version can be downloaded from the members area.

Cheers

Andy (mod)
andy@ultranerds.co.uk
Want to give me something back for my help? Please see my Amazon Wish List
GLinks ULTRA Package | GLinks ULTRA Package PRO
Links SQL Plugins | Website Design and SEO | UltraNerds | ULTRAGLobals Plugin | Pre-Made Template Sets | FREE GLinks Plugins!
Quote Reply
Re: [Andy] [ New Plugin ] DMOZ_Wizard v2.5 In reply to
Version 2.5 is now out Smile

This version has a new feature.

There is now a new script that has been setup in your /admin folder. You can run this from SSH/Telnet, with;

Code:

cd /path/to/your/admin/
nohup perl clean_bad_word_links.cgi > log.txt &
tail -n20 -f log.txt


You should see something similar to (I've only included one test link);

Code:

linkssql@east admin $ perl clean_bad_word_links.cgi
Checking URL: http://www.test.com
checking word: crap
Unvalidating link ID: 1. Reason: "crap" exists in HTML...
Line: this is just a crap test


The theory behind the script;

1) Grab the list of bad words (split with , between the words)
2) Grab the list of URL's to check
3) Check if the bad word exists in the grabbed HTML
a) If the bad word exists, show which word matched, and then unvalidate the link (not perminent)
b) If the bad word doesn't exist, the just leave the link, and move onto the next one.


The new version is available from the download area as of now :)

Cheers

Andy (mod)
andy@ultranerds.co.uk
Want to give me something back for my help? Please see my Amazon Wish List
GLinks ULTRA Package | GLinks ULTRA Package PRO
Links SQL Plugins | Website Design and SEO | UltraNerds | ULTRAGLobals Plugin | Pre-Made Template Sets | FREE GLinks Plugins!
Quote Reply
Re: [Andy] [ New Plugin ] DMOZ_Wizard v2.5 In reply to
Andy,

This Plugin shows v1.6 when I try to download it from Ultranerds.

mgeyman

Last edited by:

mgeyman: Aug 12, 2004, 8:28 AM
Quote Reply
Re: [mgeyman] [ New Plugin ] DMOZ_Wizard v2.5 In reply to
The 2.1.2 or 2.2.0 version? This feature is only available in the 2.2.0 version (LSQL's 2.1.2 version is still 1.6).

Sorry for any confusion :(

Cheers

Andy (mod)
andy@ultranerds.co.uk
Want to give me something back for my help? Please see my Amazon Wish List
GLinks ULTRA Package | GLinks ULTRA Package PRO
Links SQL Plugins | Website Design and SEO | UltraNerds | ULTRAGLobals Plugin | Pre-Made Template Sets | FREE GLinks Plugins!
Quote Reply
Re: [Andy] [ New Plugin ] DMOZ_Wizard v2.5 In reply to
Andy,

This is for the 2.2.0 version. I upgraded to the newest version of Links SQL.

Thanks.

mgeyman
> >