Gossamer Forum
Quote Reply
Slow build
Very soon i will have 10.000 links.
The cat pages have some extra fields for Meta title, Description and Meta Geo, and some more for other functions. The links have also some fields. The most globals used for UTF8 i have deleted now using encoding directly in build.pm

To build "staggered" needs now more than 400 seconds. Also the CPU is so high used that this web cant be reached when building (other webs on this server are ok);

What can i do now?
a) new server - yes, it is time
b) some fast-cgi mods? I have no idea about that
c) change the way of building; for example using only 100 instead of 500 links in a time?
d) Anything else?

Backup is switched off.
The build of a sitemap is done really fast. (one template, three globals, saved as /sitemap.xml
Quote Reply
Re: [Robert] Slow build In reply to
Hi,

A few things:

1) Enable debugging, and run a build from SSH. Check to see what queries are being run (and how long it takes)
2) Try and move globals into a .pm file, as this can be cached
3) In your globals, only pass values along you need. For example - don't try and grab ALL $tags, just pass along the value you want - i.e <%global_name($Description)%>
4) How many links in each category do you have? If you have 1000 links for example on one category (not including sub-categories), this can also cause a slow down for when it works out next/prev links)

Slow builds are a bit of a pain to debug.

Hope that helps

Andy (mod)
andy@ultranerds.co.uk
Want to give me something back for my help? Please see my Amazon Wish List
GLinks ULTRA Package | GLinks ULTRA Package PRO
Links SQL Plugins | Website Design and SEO | UltraNerds | ULTRAGLobals Plugin | Pre-Made Template Sets | FREE GLinks Plugins!
Quote Reply
Re: [Andy] Slow build In reply to
1. When i do ./nph-build.cgi - the web is reachable without any problem.

2. What do you mean with "move globals to a pm"?

I add a new file.pm with some code; then include this pm to the files like build.pm?

Here some examples:

1. directory for the detailed pages;
DIR
sub {
my ($rec) = @_;
use POSIX;
my $d = floor($rec->{ID}/1000);
return $d;
}

I need this, because i save my detailed pages in dirs /detailed/x/ID.php


2. Father (and root and others)
sub {
my $tags = shift;
my $output;
my $lang = $tags->{lang};
my $fid = $tags->{FatherID};

my $cat_db = $DB->table('Category');
$cat_db->select_options ("", "");
my $sth = $cat_db->select (['Name','Pfad'],
GT::SQL::Condition->new (
'ID','=',$fid
) );
while (my ($name,$pfad) = $sth->fetchrow_array) {
$output .= "$name";
}
utf8::encode($output);
return $output;
}

With this (and others) i make my own breadcrumps home, root, father, city


3. a picture
sub {
my ($rec) = @_;
my $id = $rec->{ID};
my $url = $rec->{URL};
my $title = $rec->{Title};
my ($dir,$bild);

print:"ID:($id)";

if ($rec->{b1}) {
my $links_db = $DB->table('Links');
my $fh = $links_db->file_info( 'b1', $id );
my $location = $fh->File_RelativePath;
if ($fh->File_Name =~ /\.(jpg|gif)$/) {
return qq~/pics$location~;
}
else
{
return "Pic Error";
}
}
else
{
return "Pic Error2";
}
}


---------------------
351 seconds instead 450 at the console with -- all instead of staggered
Quote Reply
Re: [Robert] Slow build In reply to
Hi,

The easiest is to create a new /admin/Plugins/Functions.pm file, with:
Code:
# ==================================================================
#

package Plugins::Functions;
# ==================================================================

use strict;
use GT::Base;
use GT::Plugins qw/STOP CONTINUE/;
use Links qw/:objects/;
use Links::Plugins;
use Links::SiteHTML;
use vars qw/$TCFG/;

# Inherit from base class for debug and error methods
@Plugins::Tables::ISA = qw(GT::Base);



1;

You would then put your functions in there. So for example:

Code:
# ==================================================================
#

package Plugins::Functions;
# ==================================================================

use strict;
use GT::Base;
use GT::Plugins qw/STOP CONTINUE/;
use Links qw/:objects/;
use Links::Plugins;
use Links::SiteHTML;
use vars qw/$TCFG/;

# Inherit from base class for debug and error methods
@Plugins::Tables::ISA = qw(GT::Base);

sub global_name {
my ($rec) = @_;
use POSIX;
my $d = floor($rec->{ID}/1000);
return $d;
}

1;

You then invoke the function similar to a global:

Code:
<%Plugins::Functions::global_name($variables_to_pass)%>

Code:
sub {
my ($rec) = @_;
use POSIX;
my $d = floor($rec->{ID}/1000);
return $d;
}

What do you pass to this one? You could simply pass $ID - so:

Code:
<%Plugins::Functions::global_name($ID)%>

..and then use:

Code:
sub global_name {
my $ID = $_[0];
use POSIX;
my $d = floor($ID/1000);
return $d;
}


Quote:
2. Father (and root and others)
sub {
my $tags = shift;
my $output;
my $lang = $tags->{lang};
my $fid = $tags->{FatherID};

my $cat_db = $DB->table('Category');
$cat_db->select_options ("", "");
my $sth = $cat_db->select (['Name','Pfad'],
GT::SQL::Condition->new (
'ID','=',$fid
) );
while (my ($name,$pfad) = $sth->fetchrow_array) {
$output .= "$name";
}
utf8::encode($output);
return $output;
}

Again, rather than passing all of $tags, all you need are $lang and $fid - so you could do:

Code:
sub {

my $output;
my ($lang,$fid) = @_;

my $cat_db = $DB->table('Category');
$cat_db->select_options ("", "");
my $sth = $cat_db->select (['Name','Pfad'],
GT::SQL::Condition->new (
'ID','=',$fid
) );
while (my ($name,$pfad) = $sth->fetchrow_array) {
$output .= "$name";
}
utf8::encode($output);
return $output;
}

And then call as global_name($lang,$FatherID)

There is no point passing variables you are not going to use :)

For the picture - I'm not sure there is too much speed you can get out of that global (apart from putting it in your Functions.pm plugin, and calling it from there)

Hope that helps

Andy (mod)
andy@ultranerds.co.uk
Want to give me something back for my help? Please see my Amazon Wish List
GLinks ULTRA Package | GLinks ULTRA Package PRO
Links SQL Plugins | Website Design and SEO | UltraNerds | ULTRAGLobals Plugin | Pre-Made Template Sets | FREE GLinks Plugins!
Quote Reply
Re: [Andy] Slow build In reply to
Thank you.

1. Now i delete all not used globals.
2. Then i have seen the looooooooooooooooooooooong queries with all fields and start to shorten the queries now.
3. I have to delete old unsused stuff (the globals are from another build before)

Last edited by:

Robert: Sep 19, 2020, 7:45 AM
Quote Reply
Re: [Robert] Slow build In reply to
global_name($lang,$FatherID)

How i write this in a template, please?
Quote Reply
Re: [Robert] Slow build In reply to
Robert wrote:
global_name($lang,$FatherID)

How i write this in a template, please?

If you are using it just as a global, you just do:

Code:
<%global_name($lang,$FatherID)%>

Or if it in the new plugin file, you would do something like:

Code:
<%Plugins::Functions::global_name($lang,$FatherID) %>

Cheers

Andy (mod)
andy@ultranerds.co.uk
Want to give me something back for my help? Please see my Amazon Wish List
GLinks ULTRA Package | GLinks ULTRA Package PRO
Links SQL Plugins | Website Design and SEO | UltraNerds | ULTRAGLobals Plugin | Pre-Made Template Sets | FREE GLinks Plugins!
Quote Reply
Re: [Andy] Slow build In reply to
We dont have $FatherID?
Do i need a SET for this?


I can do it with PHP like <?$FatherID = "<%FatherID%>" ?>

but how i pass it with lsql template syntax?

https://www.gossamer-threads.com/forum/Products_C9/Gossamer_Links_C5/Development%2C_Plugins_and_Globals_F20/Pass_var_to_global_P316426/?page=unread#unread

Here we have talked about this. But there is no answer how to use the FatherID as the variable for calling the global.


Can i avoid
$x =SHIFT;

with something like

$x=SHIFT('FatherID')?

Last edited by:

Robert: Sep 19, 2020, 10:10 AM
Post deleted by Robert In reply to
Quote Reply
Re: [Robert] Slow build In reply to
$FatherID exists in category html already :)

Andy (mod)
andy@ultranerds.co.uk
Want to give me something back for my help? Please see my Amazon Wish List
GLinks ULTRA Package | GLinks ULTRA Package PRO
Links SQL Plugins | Website Design and SEO | UltraNerds | ULTRAGLobals Plugin | Pre-Made Template Sets | FREE GLinks Plugins!
Quote Reply
Re: [Andy] Slow build In reply to
It doesnt work:

Template
<%Father($FatherID)%>



Global Father
sub {
my $output;
my $fid = @_;

my $cat_db = $DB->table('Category');
$cat_db->select_options ("", "");
my $sth = $cat_db->select (['Name','Pfad'],
GT::SQL::Condition->new (
'ID','=',$fid
) );
while (my ($name,$pfad) = $sth->fetchrow_array) {
$output .= "$name";
}
utf8::encode($output);
return $output;
}
Quote Reply
Re: [Robert] Slow build In reply to
Got it

sub {
...
my $fid = $_[0];

}
Quote Reply
Re: [Robert] Slow build In reply to
Does it print if you just use <%FatherID%>? Also

my $fid = @_;

Probably wants to be

my $fid = $_[0];

Andy (mod)
andy@ultranerds.co.uk
Want to give me something back for my help? Please see my Amazon Wish List
GLinks ULTRA Package | GLinks ULTRA Package PRO
Links SQL Plugins | Website Design and SEO | UltraNerds | ULTRAGLobals Plugin | Pre-Made Template Sets | FREE GLinks Plugins!
Quote Reply
Re: [Robert] Slow build In reply to
XYes, just found it, see above.

If a cat has 100 links and we span pages with 24 links; then this SELECT is done 5 times?
And if we have 100 cats like this then we need 100*5=500 selects insead of 100, right?
Then it would be better to do that inside the build.pm at the part "span pages", right?

Last edited by:

Robert: Sep 19, 2020, 10:24 AM
Quote Reply
Re: [Robert] Slow build In reply to
Maybe it is just wrong to use my own breadcrump getting root and father; ionstead using the function build in.And i just dont remember why i have done it like this (the half stuff for the new page was done some years before)
Quote Reply
Re: [Robert] Slow build In reply to
console -all is now at 271 seconds;

next run without one shift less in globals called by one value instead.
Now 258 seconds

Last edited by:

Robert: Sep 19, 2020, 10:44 AM
Quote Reply
Re: [Robert] Slow build In reply to
One more, please:

I need two value selected from table category

Name and "Path" <= i save the path for every cat for some reasons in the field Category.Path.

I use now to globals to fetch my data, but if i remember right i should something like this:

Template header <%global_name%>

Then use <%global_name.value1%> and <%global_name.value2%>

In the global i have to return an array.


Template calls Global <%Root(CatRoot)%>


Global Root SELECTS name and path of the RootCat.

Code:

sub {
my $catRoot = $_[0];
my $values;

my $cat_db = $DB->table('Category');
$cat_db->select_options ("", "");
my $sth = $cat_db->select (['Name','Pfad'],
GT::SQL::Condition->new (
'ID','=',$catRoot
) );
while (my ($name,$pfad) = $sth->fetchrow_array) {
utf8::encode($name);
$values = {
RootName => $name,
RootPfad => $pfad,
};
}

return { RootData => $values };
}



Template uses RootData at the header;
then RootData.Name and RootData.Pfad (path);

i will test it in a minute. :)

Last edited by:

Robert: Sep 19, 2020, 11:07 AM
Quote Reply
Re: [Robert] Slow build In reply to
Robert wrote:
XYes, just found it, see above.

If a cat has 100 links and we span pages with 24 links; then this SELECT is done 5 times?
And if we have 100 cats like this then we need 100*5=500 selects insead of 100, right?
Then it would be better to do that inside the build.pm at the part "span pages", right?

If you are happy editing the core .pm files (I do it all the time), then I would suggest it. It's very slow when you go:

GLinks modules create values
Passes to template
You pass back from template into global / plugin
Then the template is parsed

It's much better to try and not pass back and forth the data from the template into globals, especially if all you are wanting to do (for example) is amend a value, or add another tag to the loop.

Cheers

Andy (mod)
andy@ultranerds.co.uk
Want to give me something back for my help? Please see my Amazon Wish List
GLinks ULTRA Package | GLinks ULTRA Package PRO
Links SQL Plugins | Website Design and SEO | UltraNerds | ULTRAGLobals Plugin | Pre-Made Template Sets | FREE GLinks Plugins!
Quote Reply
Re: [Robert] Slow build In reply to
Hi,

Code:
<%Root(CatRoot)%>

Should be:

Code:
<%Root($CatRoot)%>

Quote:
I use now to globals to fetch my data, but if i remember right i should something like this:

Template header <%global_name%>

Then use <%global_name.value1%> and <%global_name.value2%>

In your global below, you would call:

Code:
<%global_name($Variable_to_pass)%>
<%if RootData%>
<%RootData.field_name%>
<%endif%>


Cheers

Andy (mod)
andy@ultranerds.co.uk
Want to give me something back for my help? Please see my Amazon Wish List
GLinks ULTRA Package | GLinks ULTRA Package PRO
Links SQL Plugins | Website Design and SEO | UltraNerds | ULTRAGLobals Plugin | Pre-Made Template Sets | FREE GLinks Plugins!