Gossamer Forum: Products: Links 2.0: Discussions: Perl Core Dumps: Any Ideas? (I know...SQL, but...)

Nov 7, 1999, 7:00 PM

Eliot

Deleted

Nov 7, 1999, 7:00 PM

Post #1 of 21

Shortcut

Perl Core Dumps: Any Ideas? (I know...SQL, but...)

I have seen my server have Perl core dumps twice in the last three months. (perl.core file created in my LINKS cgi-bin directory.)

I know this has to do with the high use of server resources for my account. Anyone else have this problem?

Thanks for your time and thoughts.

Regards,

------------------
Eliot Lee
Anthro TECH,L.L.C
www.anthrotech.com
----------------------

Nov 7, 1999, 7:19 PM

widgetz

Veteran (2260 posts)

Nov 7, 1999, 7:19 PM

Post #2 of 21

Shortcut

Re: Perl Core Dumps: Any Ideas? (I know...SQL, but...) In reply to

i did!

i always ignore those and remember i need to change my .CGI to generate .HTML pages..

i don't care right now cause i need everything to be dynamic so i can test things without building..

do you know which script was doing this? if search.cgi then you might wanna thinking about changing your search.cgi to use GREP.. it's faster..

if GREP dies on you.. then you should change to Links SQL..

jerry

Nov 7, 1999, 7:24 PM

Eliot

Deleted

Nov 7, 1999, 7:24 PM

Post #3 of 21

Shortcut

Re: Perl Core Dumps: Any Ideas? (I know...SQL, but...) In reply to

It is the search.cgi file.

I did inquire about using GREP awhile back but no one could provide a solution. I do intend on upgrading to SQL in the spring after I get more familar with MySQL and also have financial resources to purchase a license. (Also, I want to make sure that the mods that I have installed will still exist with the SQL version.)

If you have any ideas about implementing GREP, I would be interested in hearing any suggestions.

Thanks.

Regards,

------------------
Eliot Lee
Anthro TECH,L.L.C
www.anthrotech.com
----------------------

Nov 7, 1999, 8:48 PM

widgetz

Veteran (2260 posts)

Nov 7, 1999, 8:48 PM

Post #4 of 21

Shortcut

Re: Perl Core Dumps: Any Ideas? (I know...SQL, but...) In reply to

i'll give you the code tommorrow.. i'm really sleepy.. and there is school tommorrow..

on thursday no school! Smile

jerry

Nov 7, 1999, 8:51 PM

widgetz

Veteran (2260 posts)

Nov 7, 1999, 8:51 PM

Post #5 of 21

Shortcut

Re: Perl Core Dumps: Any Ideas? (I know...SQL, but...) In reply to

http://www.pdamania.com/find.cgi

the script i just made.. uses GREP..

i put a timer on there to test the speed.. it's pretty fast.. remember.. it's searching through an array of HUNDREDS of files..

however.. when i switched it to search all the way down to root.. it took a lot longer.. there are millions of files on the server Smile

jerry

Nov 8, 1999, 5:50 AM

Eliot

Deleted

Nov 8, 1999, 5:50 AM

Post #6 of 21

Shortcut

Re: Perl Core Dumps: Any Ideas? (I know...SQL, but...) In reply to

Thanks, Jerry (Widgetz)!

Cool stuff!!! It is not so much the speed of the search.cgi file (because it is quite fast considering that my database is now 900,000 MGs), but I am concerned about the Perl core dumps.

I will try your codes and see what happens.

Thanks again.

Regards,

------------------
Eliot Lee
Anthro TECH,L.L.C
www.anthrotech.com
----------------------

Nov 8, 1999, 9:22 AM

widgetz

Veteran (2260 posts)

Nov 8, 1999, 9:22 AM

Post #7 of 21

Shortcut

Re: Perl Core Dumps: Any Ideas? (I know...SQL, but...) In reply to

even though it's midnight.. i don't care Wink

i'll just throw a load of clothes i'll wear tommorrow to school into the washer so i can't sleep until it's done Smile

anyways.. this is what you pretty much do for GREP..

you use the same REGEXP that search.cgi builds up and do..

there are many ways though.. i don't know which one is faster.. but some are easier.. here is one i can think up right now

Code:
open (DB, "<$db_file_name"); 
@db = <DB>; 
close DB; 

@db = grep {!/^#/} @db; 
@db = grep {!/^\s*$/} @db; 
@db = grep {&{$regexp}} @db; 

foreach (@db) { 
   @values = &split_decode($_); 
   # REGULAR CODE FROM SEARCH.CGI!! STARTING AFTER $grand_total++; to } BEFORE close DB; 
}

what that does is it takes out everything that doesn't match first.. then searches the ones that do have a match or at least have the text somewhere inside the record..

you might want to make two copies of search.cgi and figure out which one goes faster.. and btw.. i'm not sure if that third grep will work.. but if it doesn't.. i know what to do to get it to work..

i'll be back tommorrow..
jerry

[This message has been edited by widgetz (edited November 08, 1999).]

Nov 8, 1999, 12:29 PM

Bmxer

Veteran (1311 posts)

Nov 8, 1999, 12:29 PM

Post #8 of 21

Shortcut

Re: Perl Core Dumps: Any Ideas? (I know...SQL, but...) In reply to

Eliot,
Please tell me if it works because i would like to use this.

[This message has been edited by Bmxer (edited November 08, 1999).]

Nov 8, 1999, 2:51 PM

widgetz

Veteran (2260 posts)

Nov 8, 1999, 2:51 PM

Post #9 of 21

Shortcut

Re: Perl Core Dumps: Any Ideas? (I know...SQL, but...) In reply to

i successfully installed this on someones site.. i'm seeing if it's faster.. hold on.

jerry

Nov 8, 1999, 3:26 PM

widgetz

Veteran (2260 posts)

Nov 8, 1999, 3:26 PM

Post #10 of 21

Shortcut

Re: Perl Core Dumps: Any Ideas? (I know...SQL, but...) In reply to

ok.. there is a difference.. this database i'm using isn't in the 1000s so i don't know if it's really much.. but grep is faster..

i tested it by adding $S = time(); right before the database was opened.. and then $E = time() - $S; right after it was closed..

so that actually got the time that it took to get the results.. not the WHOLE script..

then print $E when it was all done..

here are my results.. (multiple queries)

Code:
RESULTS | SEARCH.CGI | GREP 

C 28  L | 3 s | 0 s 
C 107 L | 4 s | 0 s 
C 98  L | 2 s | 0 s 
C 102 L | 2 s | 0 s 
C 372 L | 3 s | 1 s 
C 396 L | 2 s | 0 s

it's interesting.. search.cgi went slow and fast sometimes.. grep pretty much went fast..

remember.. this database doesn't have more than 500 links so Smile

anyways.. i still think there is a way to cut down off using the original search.cgi code.. another grep.. but i'll have to look at that first.. wait.. i think i know.. i'll be back in a sec..

jerry

Nov 8, 1999, 4:03 PM

Eliot

Deleted

Nov 8, 1999, 4:03 PM

Post #11 of 21

Shortcut

Re: Perl Core Dumps: Any Ideas? (I know...SQL, but...) In reply to

Widgetz,

I will wait to see the new codes. Where are you finding information about GREP? I looked at perl.com for tutorials or books, and I could only find modules related to GREP.

Thanks for your efforts.

Regards,

------------------
Eliot Lee
Anthro TECH,L.L.C
www.anthrotech.com
----------------------

Nov 8, 1999, 4:39 PM

widgetz

Veteran (2260 posts)

Nov 8, 1999, 4:39 PM

Post #12 of 21

Shortcut

Re: Perl Core Dumps: Any Ideas? (I know...SQL, but...) In reply to

i don't think i will be making any new codes.. if i do.. it will be FAST.. but it will eliminate AND and OR booleans.. so that is pointless..

jerry

Nov 8, 1999, 5:43 PM

widgetz

Veteran (2260 posts)

Nov 8, 1999, 5:43 PM

Post #13 of 21

Shortcut

Re: Perl Core Dumps: Any Ideas? (I know...SQL, but...) In reply to

n/m.. i got a donor [for new grep] Smile

.. has > 7000 links..
jerry

[This message has been edited by widgetz (edited November 08, 1999).]

Nov 8, 1999, 6:11 PM

Eliot

Deleted

Nov 8, 1999, 6:11 PM

Post #14 of 21

Shortcut

Re: Perl Core Dumps: Any Ideas? (I know...SQL, but...) In reply to

Wonderful! Sounds great.

I successfully installed the codes and it does seem to be a faster.

But if you can identify an even faster method (without sacrificing search options, like the AND/OR Connector, and other Mods I've added to the search.cgi), I am all for it.

Smile

Thanks.

Regards,

------------------
Eliot Lee
Anthro TECH,L.L.C
www.anthrotech.com
----------------------

[This message has been edited by Eliot (edited November 08, 1999).]

Nov 8, 1999, 6:53 PM

widgetz

Veteran (2260 posts)

Nov 8, 1999, 6:53 PM

Post #15 of 21

Shortcut

Re: Perl Core Dumps: Any Ideas? (I know...SQL, but...) In reply to

Code:
	open (DB, "<$db_file_name") or &cgierr("error in search. unable to open database: $db_file_name. Reason: $!"); 
	flock (DB, 1) if ($db_use_flock); 
	@db = <DB>; 
	close DB; 

	$grand_total = $#db+1; 

	foreach (grep {!/^#/ and !/^\s*$/ and &{$regexp}} @db) { 
		@values = &split_decode($_); 
		FIELD: foreach $field (@search_fields) {  
			$_ = $values[$field]; 
			if (&{$regexp}) { 
				push (@{$link_results{$values[$db_category]}}, @values); 
				$numhits++; 
				last FIELD; 
			} 
		} 

		$_ = $values[$db_category]; 
		push (@category_results, $values[$db_category]) if (!$seen{$_}++ && &{$regexp}); 
	}

using that on the 7135 link database.. and it seems the max has been 3 seconds.. but i've only seen it happen once.. otherwise i get a bunch of 2 seconds.. and a few 1 seconds..

i'm not sure if it keeps the and/or though.. you might want to try.. compare results..

jerry

[This message has been edited by widgetz (edited November 08, 1999).]

[This message has been edited by widgetz (edited November 08, 1999).]

Nov 8, 1999, 7:08 PM

widgetz

Veteran (2260 posts)

Nov 8, 1999, 7:08 PM

Post #16 of 21

Shortcut

Re: Perl Core Dumps: Any Ideas? (I know...SQL, but...) In reply to

i tried many combonations and they always matched.. so i don't know what all that code in the original search.cgi was for..

oh well..

just if you want to put MODS.. like category search.. you'll have to put it in the most simplistic way.. that way it won't be slowed down..

jerry

anyways.. i guess this mod is completed.. now i think i wanna sleep.. [of course science homework first that will take 5 minutes..]

Nov 8, 1999, 7:30 PM

Eliot

Deleted

Nov 8, 1999, 7:30 PM

Post #17 of 21

Shortcut

Re: Perl Core Dumps: Any Ideas? (I know...SQL, but...) In reply to

Thanks, Jerry...I installed the new codes and got some syntax errors. I am slowly but surely falling asleep myself.

BTW: I currently live in Flagstaff, AZ (MST). And I will be moving to Boulder, CO this weekend. So, I may be on-line earlier in the evening...but who knows...I've been a nightowl since I was your age.

Smile

I will re-look at these codes and see how they can be implemented without disrupting the category search and the connector options.

Thanks again for your quick response and great work.

Smile

Regards,

------------------
Eliot Lee
Anthro TECH,L.L.C
www.anthrotech.com
----------------------

Nov 9, 1999, 10:15 AM

Bmxer

Veteran (1311 posts)

Nov 9, 1999, 10:15 AM

Post #18 of 21

Shortcut

Re: Perl Core Dumps: Any Ideas? (I know...SQL, but...) In reply to

The first code only gets the last keyword or something, because i find a search using certain keywords for my old search.cgi, gave 17 results, then i put this code in and only got 1.

Nov 9, 1999, 12:16 PM

widgetz

Veteran (2260 posts)

Nov 9, 1999, 12:16 PM

Post #19 of 21

Shortcut

Re: Perl Core Dumps: Any Ideas? (I know...SQL, but...) In reply to

no... &{$regexp} is a pattern match for all the search terms.. it worked for me and if it doesn't work for you then you probably altered it before. (i doubt it though)

jerry

Feb 10, 2000, 6:38 PM

DogTags

Veteran (1345 posts)

Feb 10, 2000, 6:38 PM

Post #20 of 21

Shortcut

Re: Perl Core Dumps: Any Ideas? (I know...SQL, but...) In reply to

Where would this code be added in search.cgi?

Thanks.

[This message has been edited by DogTags (edited February 10, 2000).]

Feb 10, 2000, 6:43 PM

Eliot

Deleted

Feb 10, 2000, 6:43 PM

Post #21 of 21

Shortcut

Re: Perl Core Dumps: Any Ideas? (I know...SQL, but...) In reply to

Find the open...and $grand_total codes in your search.cgi file. Replace those codes with the codes written in this forum.

BTW: These codes only work in NON-MODDED search.cgi scripts. I have added many options in my search.cgi and it does not work well with other Mods.

Regards.

------------------
Eliot Lee
Anthro TECH,L.L.C
www.anthrotech.com
* Be sure to visit the Resource Center for FAQ's, Modifications and Extra Goodies!!
* Search Forums!
* Say NO to Duplicate Threads. :)
----------------------