Login | Register For Free | Help
Search for: (Advanced)

Mailing List Archive: Interchange: users

Jobs are slowing IC

 

 

Interchange users RSS feed   Index | Next | Previous | View Threaded


emailgrant at gmail

Mar 17, 2011, 6:27 PM

Post #1 of 9 (928 views)
Permalink
Jobs are slowing IC

The number of IC jobs I need to run daily have increased to the point
that they're running almost all day every day. This seems to be
slowing down the web server aspect of IC which is not good. I
currently have a quad-core 2.1Ghz CPU and 4GB of memory and a fast HD.
Would you set up another box strictly for IC jobs, or pump up the
hardware, or something else? I've optimized the jobs themselves about
as well as I can, although I suppose there is always more to be done
there.

- Grant

_______________________________________________
interchange-users mailing list
interchange-users [at] icdevgroup
http://www.icdevgroup.org/mailman/listinfo/interchange-users


jon at endpoint

Mar 17, 2011, 8:11 PM

Post #2 of 9 (910 views)
Permalink
Re: Jobs are slowing IC [In reply to]

On Thu, 17 Mar 2011, Grant wrote:

> The number of IC jobs I need to run daily have increased to the point
> that they're running almost all day every day. This seems to be slowing
> down the web server aspect of IC which is not good. I currently have a
> quad-core 2.1Ghz CPU and 4GB of memory and a fast HD. Would you set up
> another box strictly for IC jobs, or pump up the hardware, or something
> else? I've optimized the jobs themselves about as well as I can,
> although I suppose there is always more to be done there.

That question is really difficult to answer in the abstract. It makes a
big difference what those jobs are doing. Are they database-intensive? Or
using up CPU time in Interchange? Why is it interfering with your normal
ecommerce operations? Do the jobs always run independently of each other
or do they sometimes overlap? Are they literally Interchange "Jobs"? Or
just cron jobs that hit an Interchange-served URL?

Jon

--
Jon Jensen
End Point Corporation
http://www.endpoint.com/

_______________________________________________
interchange-users mailing list
interchange-users [at] icdevgroup
http://www.icdevgroup.org/mailman/listinfo/interchange-users


emailgrant at gmail

Mar 18, 2011, 10:04 AM

Post #3 of 9 (894 views)
Permalink
Re: Jobs are slowing IC [In reply to]

>> The number of IC jobs I need to run daily have increased to the point that
>> they're running almost all day every day.  This seems to be slowing down the
>> web server aspect of IC which is not good.  I currently have a quad-core
>> 2.1Ghz CPU and 4GB of memory and a fast HD. Would you set up another box
>> strictly for IC jobs, or pump up the hardware, or something else?  I've
>> optimized the jobs themselves about as well as I can, although I suppose
>> there is always more to be done there.
>
> That question is really difficult to answer in the abstract. It makes a big
> difference what those jobs are doing. Are they database-intensive? Or using
> up CPU time in Interchange? Why is it interfering with your normal ecommerce
> operations? Do the jobs always run independently of each other or do they
> sometimes overlap? Are they literally Interchange "Jobs"? Or just cron jobs
> that hit an Interchange-served URL?
>
> Jon

Hi Jon,

They're real IC Jobs and they're database-intensive. I use the
default Jobs MaxServers 1 so they don't overlap.

- Grant

_______________________________________________
interchange-users mailing list
interchange-users [at] icdevgroup
http://www.icdevgroup.org/mailman/listinfo/interchange-users


jon at endpoint

Mar 18, 2011, 1:19 PM

Post #4 of 9 (892 views)
Permalink
Re: Jobs are slowing IC [In reply to]

On Fri, 18 Mar 2011, Grant wrote:

> They're real IC Jobs and they're database-intensive. I use the default
> Jobs MaxServers 1 so they don't overlap.

Ok. Is your database running on the same machine as your app server? If
so, you may want to consider separating them onto different machines.

But before doing that, look at whether the database is actually
overloaded, or whether you have lock contention. If you're using MySQL,
are your tables MyISAM or InnoDB? MyISAM tables lock completely for any
kind of write, so no reads can happen till the writing is done. You may
need to switch to InnoDB, which as other benefits such as transactions as
well.

If you're using another database, you'll still want to look into lock
contention on both tables and indexes, etc.

You also may want to consider running more than one of your jobs at a
time, during off hours, if that makes sense for what they do.

Jon

--
Jon Jensen
End Point Corporation
http://www.endpoint.com/

_______________________________________________
interchange-users mailing list
interchange-users [at] icdevgroup
http://www.icdevgroup.org/mailman/listinfo/interchange-users


emailgrant at gmail

Mar 21, 2011, 4:07 PM

Post #5 of 9 (881 views)
Permalink
Re: Jobs are slowing IC [In reply to]

>> They're real IC Jobs and they're database-intensive.  I use the default
>> Jobs MaxServers 1 so they don't overlap.
>
> Ok. Is your database running on the same machine as your app server? If so,
> you may want to consider separating them onto different machines.

Yes, they're on the same server.

> But before doing that, look at whether the database is actually overloaded,
> or whether you have lock contention. If you're using MySQL, are your tables
> MyISAM or InnoDB? MyISAM tables lock completely for any kind of write, so no
> reads can happen till the writing is done. You may need to switch to InnoDB,
> which as other benefits such as transactions as well.

If MyISAM is default then that's what I'm using. I bet you're right
on with this. I bet it's lock contention and I'll try switching to
InnoDB.

Thanks,
Grant


> If you're using another database, you'll still want to look into lock
> contention on both tables and indexes, etc.
>
> You also may want to consider running more than one of your jobs at a time,
> during off hours, if that makes sense for what they do.
>
> Jon

_______________________________________________
interchange-users mailing list
interchange-users [at] icdevgroup
http://www.icdevgroup.org/mailman/listinfo/interchange-users


emailgrant at gmail

Apr 17, 2011, 1:24 PM

Post #6 of 9 (831 views)
Permalink
Re: Jobs are slowing IC [In reply to]

>> They're real IC Jobs and they're database-intensive.  I use the default
>> Jobs MaxServers 1 so they don't overlap.
>
> Ok. Is your database running on the same machine as your app server? If so,
> you may want to consider separating them onto different machines.
>
> But before doing that, look at whether the database is actually overloaded,
> or whether you have lock contention. If you're using MySQL, are your tables
> MyISAM or InnoDB? MyISAM tables lock completely for any kind of write, so no
> reads can happen till the writing is done. You may need to switch to InnoDB,
> which as other benefits such as transactions as well.

I've read that MyISAM is much faster for reads than InnoDB. In an IC
environment, is it advisable to keep some tables on MyISAM and switch
those that require a lot of writes to InnoDB?

- Grant


> If you're using another database, you'll still want to look into lock
> contention on both tables and indexes, etc.
>
> You also may want to consider running more than one of your jobs at a time,
> during off hours, if that makes sense for what they do.
>
> Jon
>
> --
> Jon Jensen
> End Point Corporation
> http://www.endpoint.com/

_______________________________________________
interchange-users mailing list
interchange-users [at] icdevgroup
http://www.icdevgroup.org/mailman/listinfo/interchange-users


jon at endpoint

Apr 18, 2011, 5:51 PM

Post #7 of 9 (825 views)
Permalink
Re: Jobs are slowing IC [In reply to]

On Sun, 17 Apr 2011, Grant wrote:

> I've read that MyISAM is much faster for reads than InnoDB. In an IC
> environment, is it advisable to keep some tables on MyISAM and switch
> those that require a lot of writes to InnoDB?

If you have low write traffic and small tables, it's probably fine to stay
on MyISAM.

But MySQL experts I've talked to say that for any large database, or with
heavy writes, use InnoDB for everything. They tell me horror stories of
frequently-corrupted tables (which I haven't experienced) and lots of
locking problems (which I have).

Of course the best thing would be for you to try a complete switchover of
your own database to InnoDB and the test it under load and see how it
does.

Jon

--
Jon Jensen
End Point Corporation
http://www.endpoint.com/

_______________________________________________
interchange-users mailing list
interchange-users [at] icdevgroup
http://www.icdevgroup.org/mailman/listinfo/interchange-users


db at m-and-d

Apr 22, 2011, 9:38 AM

Post #8 of 9 (807 views)
Permalink
Re: Jobs are slowing IC [In reply to]

> If you have low write traffic and small tables, it's probably fine to stay
> on MyISAM.
>
> But MySQL experts I've talked to say that for any large database, or with
> heavy writes, use InnoDB for everything. They tell me horror stories of
> frequently-corrupted tables (which I haven't experienced) and lots of
> locking problems (which I have).

These stories of corruption and locking were for MyISAM tables?

DB


> Of course the best thing would be for you to try a complete switchover of
> your own database to InnoDB and the test it under load and see how it
> does.
>
> Jon
>
> --
> Jon Jensen
> End Point Corporation
> http://www.endpoint.com/


_______________________________________________
interchange-users mailing list
interchange-users [at] icdevgroup
http://www.icdevgroup.org/mailman/listinfo/interchange-users


jon at endpoint

Apr 22, 2011, 9:52 AM

Post #9 of 9 (807 views)
Permalink
Re: Jobs are slowing IC [In reply to]

On Fri, 22 Apr 2011, DB wrote:

>> But MySQL experts I've talked to say that for any large database, or
>> with heavy writes, use InnoDB for everything. They tell me horror
>> stories of frequently-corrupted tables (which I haven't experienced)
>> and lots of locking problems (which I have).
>
> These stories of corruption and locking were for MyISAM tables?

Yes.

Jon

--
Jon Jensen
End Point Corporation
http://www.endpoint.com/

_______________________________________________
interchange-users mailing list
interchange-users [at] icdevgroup
http://www.icdevgroup.org/mailman/listinfo/interchange-users

Interchange users RSS feed   Index | Next | Previous | View Threaded
 
 


Interested in having your list archived? Contact Gossamer Threads
 
  Web Applications & Managed Hosting Powered by Gossamer Threads Inc.