Mailing List Archive: Catalyst: Users

Re: Catalyst benchmark 5.7 vs 5.8 - new test

Index | Next | Previous | View Threaded

toby.corkindale at strategicdata

Sep 28, 2009, 7:39 PM

Post #1 of 9 (1494 views)
Permalink

Re: Catalyst benchmark 5.7 vs 5.8 - new test

Fayland Lam wrote:
> I'm wondering if someone here did a benchmark between Catalyst 5.7 and 5.8

I have a vested interest in knowing the difference between the two
versions as well, so knocked up a "proper" test.
I have two identical virtual machines, only on one I installed
Catalyst::Runtime 5.71001 and the other with 5.80013.

Running the exact same app, I hit them up with Siege for a while,
results follow at the end of this email.

If you want to replicate the test or examine my extremely-simple test
app, see: http://github.com/TJC/Catalyst-Performance-Test
(Patches gleefully accepted ;)

It's interesting to note the headline figures have 5.71 performing 316
tps, vs 5.80 making only 283 tps.
Memory usage (for this small app) has increased by 4MB, but is
presumably shared. I guess I should look into that more.

The same system can serve small static pages from the webserver at about
1900 tps. A real-world application there on Cat 5.8 gets 90 tps.

I don't see that performance difference (5.71 vs 5.80) as significant,
since most of your time ends up being spent in application code, rather
than the Catalyst framework itself.
ie. If you want to make your code go faster, look at optimising your
templating and database queries before you worry about downgrading Catalyst.

-Toby

------------------= results =----------------------
Running 10 second warmup on 5.7..
Running main test on 5.7..

Transactions: 94796 hits
Availability: 100.00 %
Elapsed time: 300.00 secs
Data transferred: 77.35 MB
Response time: 0.03 secs
Transaction rate: 315.99 trans/sec
Throughput: 0.26 MB/sec
Concurrency: 10.00
Successful transactions: 94796
Failed transactions: 0
Longest transaction: 0.98
Shortest transaction: 0.00

Process size:
101m VIRT, 34m RES

Running 10 second warmup on 5.8..
Running main test on 5.8..

Transactions: 84805 hits
Availability: 100.00 %
Elapsed time: 300.00 secs
Data transferred: 69.20 MB
Response time: 0.04 secs
Transaction rate: 282.68 trans/sec
Throughput: 0.23 MB/sec
Concurrency: 9.99
Successful transactions: 84805
Failed transactions: 0
Longest transaction: 1.07
Shortest transaction: 0.00

Process size:
103m VIRT, 38m RES

_______________________________________________
List: Catalyst [at] lists
Listinfo: http://lists.scsys.co.uk/cgi-bin/mailman/listinfo/catalyst
Searchable archive: http://www.mail-archive.com/catalyst [at] lists/
Dev site: http://dev.catalyst.perl.org/

toby.corkindale at strategicdata

Sep 28, 2009, 10:00 PM

Post #2 of 9 (1406 views)
Permalink

Re: Catalyst benchmark 5.7 vs 5.8 - new test [In reply to]

Toby Corkindale wrote:
> It's interesting to note the headline figures have 5.71 performing 316
> tps, vs 5.80 making only 283 tps.
> Memory usage (for this small app) has increased by 4MB, but is
> presumably shared. I guess I should look into that more.

Here are some new analysis of memory usage on my test app.

5.7 5.8
Rss: 35260 38768
Private: 8244 9620
Shared: 27016 29148

So 5.8 is 3500kB larger, but 2000kB is shared. So we're only looking at
1500Kb more per process.

-Toby

_______________________________________________
List: Catalyst [at] lists
Listinfo: http://lists.scsys.co.uk/cgi-bin/mailman/listinfo/catalyst
Searchable archive: http://www.mail-archive.com/catalyst [at] lists/
Dev site: http://dev.catalyst.perl.org/

bobtfish at bobtfish

Sep 29, 2009, 4:48 AM

Post #3 of 9 (1392 views)
Permalink

Re: Catalyst benchmark 5.7 vs 5.8 - new test [In reply to]

Toby Corkindale wrote:
> It's interesting to note the headline figures have 5.71 performing 316
> tps, vs 5.80 making only 283 tps.

The very important thing you haven't noted (unless I missed it) is what
perl version this benchmark was conducted under.

Some benchmarking was done before 5.8 was released, and it showed that
Catalyst 5.7 was (marginally) quicker on perl 5.8 and that Catalyst 5.8
was (marginally) quicker on perl 5.10 :)

So I'd be very interested to see the benchmark repeated with two
different perls, giving 4 results to compare and contrast.

Cheers
t0m

_______________________________________________
List: Catalyst [at] lists
Listinfo: http://lists.scsys.co.uk/cgi-bin/mailman/listinfo/catalyst
Searchable archive: http://www.mail-archive.com/catalyst [at] lists/
Dev site: http://dev.catalyst.perl.org/

syber.rus at gmail

Sep 29, 2009, 2:12 PM

Post #4 of 9 (1397 views)
Permalink

Re: Catalyst benchmark 5.7 vs 5.8 - new test [In reply to]

5.8 !!TWICE!! slower at startup than 5.7 under any perl version. This
is annoying because i have 100 models and i do not want to wait 10
secs while it is starting in debug.

2009/9/29 Tomas Doran <bobtfish [at] bobtfish>:
> Toby Corkindale wrote:
>>
>> It's interesting to note the headline figures have 5.71 performing 316
>> tps, vs 5.80 making only 283 tps.
>
> The very important thing you haven't noted (unless I missed it) is what perl
> version this benchmark was conducted under.
>
> Some benchmarking was done before 5.8 was released, and it showed that
> Catalyst 5.7 was (marginally) quicker on perl 5.8 and that Catalyst 5.8 was
> (marginally) quicker on perl 5.10 :)
>
> So I'd be very interested to see the benchmark repeated with two different
> perls, giving 4 results to compare and contrast.
>
> Cheers
> t0m
>
> _______________________________________________
> List: Catalyst [at] lists
> Listinfo: http://lists.scsys.co.uk/cgi-bin/mailman/listinfo/catalyst
> Searchable archive: http://www.mail-archive.com/catalyst [at] lists/
> Dev site: http://dev.catalyst.perl.org/
>

_______________________________________________
List: Catalyst [at] lists
Listinfo: http://lists.scsys.co.uk/cgi-bin/mailman/listinfo/catalyst
Searchable archive: http://www.mail-archive.com/catalyst [at] lists/
Dev site: http://dev.catalyst.perl.org/

bobtfish at bobtfish

Sep 29, 2009, 2:27 PM

Post #5 of 9 (1397 views)
Permalink

Re: Catalyst benchmark 5.7 vs 5.8 - new test [In reply to]

On 29 Sep 2009, at 22:12, Oleg Pronin wrote:

> 5.8 !!TWICE!! slower at startup than 5.7 under any perl version. This
> is annoying because i have 100 models and i do not want to wait 10
> secs while it is starting in debug.

Patches welcome to optimise things for your wacky use-case given
appropriate NYTProf blame.

Cheers
t0m

_______________________________________________
List: Catalyst [at] lists
Listinfo: http://lists.scsys.co.uk/cgi-bin/mailman/listinfo/catalyst
Searchable archive: http://www.mail-archive.com/catalyst [at] lists/
Dev site: http://dev.catalyst.perl.org/

syber.rus at gmail

Sep 29, 2009, 3:42 PM

Post #6 of 9 (1398 views)
Permalink

Re: Catalyst benchmark 5.7 vs 5.8 - new test [In reply to]

i meant 100 resultsources+controllers :-) not models

2009/9/30 Tomas Doran <bobtfish [at] bobtfish>:
>
> On 29 Sep 2009, at 22:12, Oleg Pronin wrote:
>
>> 5.8 �!!TWICE!! slower at startup than 5.7 under any perl version. This
>> is annoying because i have 100 models and i do not want to wait 10
>> secs while it is starting in debug.
>
> Patches welcome to optimise things for your wacky use-case given appropriate
> NYTProf blame.
>
> Cheers
> t0m
>
>
> _______________________________________________
> List: Catalyst [at] lists
> Listinfo: http://lists.scsys.co.uk/cgi-bin/mailman/listinfo/catalyst
> Searchable archive: http://www.mail-archive.com/catalyst [at] lists/
> Dev site: http://dev.catalyst.perl.org/
>

_______________________________________________
List: Catalyst [at] lists
Listinfo: http://lists.scsys.co.uk/cgi-bin/mailman/listinfo/catalyst
Searchable archive: http://www.mail-archive.com/catalyst [at] lists/
Dev site: http://dev.catalyst.perl.org/

toby.corkindale at strategicdata

Sep 29, 2009, 5:21 PM

Post #7 of 9 (1396 views)
Permalink

Re: Catalyst benchmark 5.7 vs 5.8 - new test [In reply to]

Tomas Doran wrote:
> Toby Corkindale wrote:
>> It's interesting to note the headline figures have 5.71 performing 316
>> tps, vs 5.80 making only 283 tps.
>
> The very important thing you haven't noted (unless I missed it) is what
> perl version this benchmark was conducted under.

Ah, sorry, I didn't mention it.

I benchmarked it on Perl 5.10.0, using the standard system Perl on
Ubuntu Jaunty. (With updates applied)

I was benchmarking the two Catalyst's on otherwise-identical setups, as
I thought that was fairer.. Just about *everything* OO performs faster
on Perl 5.010 vs 5.008, doesn't it?

> Some benchmarking was done before 5.8 was released, and it showed that
> Catalyst 5.7 was (marginally) quicker on perl 5.8 and that Catalyst 5.8
> was (marginally) quicker on perl 5.10 :)

My test app is very simple; If there's some official test app available
I'd be interesting in re-running my tests. I pushed my little test app
and some siege scripts out to github, but I don't think they'd be happy
with 2GB of virtual machine images being added :)

> So I'd be very interested to see the benchmark repeated with two
> different perls, giving 4 results to compare and contrast.

I'll see if I get some more time to try that; building Perl from scratch
takes quite a bit longer than using the system perl.. and if I start
comparing with totally different OS versions then the tests aren't
really so valid, are they?

--
Strategic Data Pty Ltd
Ph: 03 9340 9000

_______________________________________________
List: Catalyst [at] lists
Listinfo: http://lists.scsys.co.uk/cgi-bin/mailman/listinfo/catalyst
Searchable archive: http://www.mail-archive.com/catalyst [at] lists/
Dev site: http://dev.catalyst.perl.org/

apv at sedition

Sep 29, 2009, 7:02 PM

Post #8 of 9 (1383 views)
Permalink

Re: Catalyst benchmark 5.7 vs 5.8 - new test [In reply to]

Possibly related anecdote. My main personal app went from 5 second
start ups to start ups of well over a minute when I went from 5.7 to
5.8. I had a handful of Controller and Model classes in which I had
written test code and then removed it all so they looked something like-

package MyApp::Controller::Stuff;
1;
__END__
Tons of code that was deemed too dangerous or experimental for
production but I want to keep in this package for when I finally sort
it out...

The test server reminded me about them with warnings. I removed them
(I *think* that's all I did) and startups went back to "normal." Still
quite a bit slower but not astronomically slower.

On Sep 29, 2009, at 3:42 PM, Oleg Pronin wrote:

> i meant 100 resultsources+controllers :-) not models
>
> 2009/9/30 Tomas Doran <bobtfish [at] bobtfish>:
>>
>> On 29 Sep 2009, at 22:12, Oleg Pronin wrote:
>>
>>> 5.8 !!TWICE!! slower at startup than 5.7 under any perl version.
>>> This
>>> is annoying because i have 100 models and i do not want to wait 10
>>> secs while it is starting in debug.
>>
>> Patches welcome to optimise things for your wacky use-case given
>> appropriate
>> NYTProf blame.
>>
>> Cheers
>> t0m
>>
>>
>> _______________________________________________
>> List: Catalyst [at] lists
>> Listinfo: http://lists.scsys.co.uk/cgi-bin/mailman/listinfo/catalyst
>> Searchable archive: http://www.mail-archive.com/catalyst [at] lists/
>> Dev site: http://dev.catalyst.perl.org/
>>
>
> _______________________________________________
> List: Catalyst [at] lists
> Listinfo: http://lists.scsys.co.uk/cgi-bin/mailman/listinfo/catalyst
> Searchable archive: http://www.mail-archive.com/catalyst [at] lists/
> Dev site: http://dev.catalyst.perl.org/

_______________________________________________
List: Catalyst [at] lists
Listinfo: http://lists.scsys.co.uk/cgi-bin/mailman/listinfo/catalyst
Searchable archive: http://www.mail-archive.com/catalyst [at] lists/
Dev site: http://dev.catalyst.perl.org/

toby.corkindale at strategicdata

Sep 29, 2009, 7:25 PM

Post #9 of 9 (1386 views)
Permalink

Re: Catalyst benchmark 5.7 vs 5.8 - new test [In reply to]

Index | Next | Previous | View Threaded

Interested in having your list archived? Contact Gossamer Threads