Mailing List Archive: Catalyst: Users

Managing module regressions.

Index | Next | Previous | View Threaded

moseley at hank

Jun 29, 2012, 8:24 AM

Post #1 of 9 (373 views)
Permalink

Managing module regressions.

This is a "how do others do this?" post.

In your large (or even not so large) apps I assume at times you experience
dependency regressions. My question is do you manage it on a case-by-case
basis (simply install an older version or fix internally) or is it such a
significant issue that you have a system for managing modules outside of
CPAN?

The reason I ask is because at work we are considering managing different
stacks of CPAN and in-house dependencies -- so maybe "dev", "testing", and
"production/staging" stacks all in separate private CPAN mirrors. And now
multiply that times the number of different apps we work on.

My experience is that even with the very large number of dependencies in a
Catalyst app that it's pretty rare to have a regression. It happens,
sure, and when it happens just deal with it. (And thinking of our own code
it's typically not a regression in a module but a fix in a module where our
code was depending on some broken behavior).

So, the question is: does anyone else find it necessary to manage
dependencies as I described above? And if so, what is your process?

My opinion is versions don't really matter much -- install the app in a
"QA" or "testing", bringing in dependencies as needed from CPAN or local
repo, and run the unit tests. If they pass then I'm happy (until QA finds
a bug). Managing multiple stacks of modules adds complexity -- and bugs
favor complexity.

Another argument that is floated around is we don't want to upgrade
dependencies often because of potential *new* bugs. That seems a bit silly
to me because it's ignoring *known* bug fixes for the chance that there
might some unknown new bug. (Yes, we have apps running Catalyst from
2010!)

Thanks for the feedback,

--
Bill Moseley
moseley [at] hank

dakkar at thenautilus

Jun 29, 2012, 11:12 AM

Post #2 of 9 (366 views)
Permalink

Re: Managing module regressions. [In reply to]

Here's what we do:

- we have a (VCS-managed) set of tarballs downloaded from CPAN
- we run a CPAN-like server providing those tarballs
- we have a rather large set of distroprefs to skip unreliable tests
and apply local patches
- we usually update to the latest CPAN (and perl) releases
- sometimes we have to hold back on a set of modules because of
problems (currently we can't update Catalyst, for example, due to
some internal libraries exploiting undocumented behaviours; yes,
we're going to fix our libraries)
- we smoke the whole set of modules whenever we update a batch of them
- every iteration, we build a package (in our case, RPM) with the
version of perl and all modules that we are going to use
- we develop and run all our test suites against that package
- the application packages require the specific perl/cpan package
version they were developed on

It works, and it's not that much work.

--
Dakkar - <Mobilis in mobile>
GPG public key fingerprint = A071 E618 DD2C 5901 9574
6FE2 40EA 9883 7519 3F88
key id = 0x75193F88

River: "Mal. Bad. In the Latin."
--Episode #2, "The Train Job"

Attachments:

signature.asc (0.19 KB)

moseley at hank

Jul 2, 2012, 12:42 PM

Post #3 of 9 (360 views)
Permalink

Re: Managing module regressions. [In reply to]

On Fri, Jun 29, 2012 at 1:12 PM, Gianni Ceccarelli
<dakkar [at] thenautilus>wrote:

> Here's what we do:
>
> - we have a (VCS-managed) set of tarballs downloaded from CPAN
> - we run a CPAN-like server providing those tarballs
> - we have a rather large set of distroprefs to skip unreliable tests
> and apply local patches
> - we usually update to the latest CPAN (and perl) releases
> - sometimes we have to hold back on a set of modules because of
> problems (currently we can't update Catalyst, for example, due to
> some internal libraries exploiting undocumented behaviours; yes,
> we're going to fix our libraries)
> - we smoke the whole set of modules whenever we update a batch of them
> - every iteration, we build a package (in our case, RPM) with the
> version of perl and all modules that we are going to use
> - we develop and run all our test suites against that package
> - the application packages require the specific perl/cpan package
> version they were developed on
>
> It works, and it's not that much work.
>

You have tarballs of every single dependency? How do you determine what
those are?

And you build a single RPM with perl and all dependencies? Do you use
Perlbrew for that? I'd love to hear more about that process.

Here's the difference between my thinking and the operations manager: He
wants to make sure that the same code is used from development to
production. The idea is to reduce the risk of bugs from using different
versons of dependencies. (That's one reason we are stuck running very old
code).

My view, as a developer, is I want good test coverage. When it's time to
cut a release what is important to me is to see 100% tests passing. If the
code works, well, it works.

Small sample size, but I haven't heard that regressions from CPAN are a big
problem -- I know they happen, of course, but the question I'm after is it
so significant to warrant building much more complex build systems instead
of using CPAN in the normal away? I just don't think so.

I also don't see a need to manage multiple stacks of modules for different
stages of the application.

So, I'm looking at this environment, which seems about as simple as you can
hope for. Anyone see any any holes in this approach?

- Run a local CPAN ("DarkPAN") repo for our in-house modules. e.g.
CPAN::Site with pass through. Our cpan(m) clients are configured to first
fetch from local CPAN, if if not found then fetch from public CPAN.

- For rare CPAN regressions install those distributions into our local
DarkPAN which clients will install in preference over the version on CPAN.
Add test for to catch a regression in the future. (Hopefully a rare and
short-lived situation.)

- Developers check out and install dependencies as normal locally
(local::lib, perlbrew) and make sure code has good test coverage.

- In-house modules (as well as apps) are "released" to our local
DarkPAN. Dist::Zilla's "release" makes this trivial.

- Automated testing can check both "trunk" and "release" -- trunk by
checking out and running tests, and "release" by installing the most recent
version with a cpan client and running tests.

And the release process is very similar:

- QA team (or a developer) runs "cpan Our-App" on the target platform,
letting it bring in any dependencies as normal.

- Run the the unit tests to satisfy the development team that the app is
working as expected with the installed dependencies. This is essentially
developer "hand-off" to the QA team confirming the app works as expected.

- Then QA team tests the app and, if passes tests, the app is moved to
staging and then production.

I think the significant thing here is I'm not really worrying about
specific versions of dependencies. Sure, code depends on a minimun version
of a module but that's just so tests have a chance of passing. What's
important is that the unit tests pass. It's really no different than
running "cpan Catalyst::Runtime" and making sure all tests pass.

Sure, it's possible that a newer module ends up on production than dev, but
that would mean unit tests AND QA failed to detect a bug. And let's be
honest, the vast, vast majority of bugs that find their way to production
are in our own code.

--
Bill Moseley
moseley [at] hank

dakkar at thenautilus

Jul 2, 2012, 1:32 PM

Post #4 of 9 (361 views)
Permalink

Re: Managing module regressions. [In reply to]

On 2012-07-02 Bill Moseley <moseley [at] hank> wrote:
> You have tarballs of every single dependency? How do you determine
> what those are?

IIRC (and the system works well so I don't have to remember how it
works :) ) we have a wrapper for CPAN.pm that logs the downloads and
adds them to the repository.

> And you build a single RPM with perl and all dependencies?

Yes.

> Do you use Perlbrew for that?

We built our system before Perlbrew existed. Some details can been
seen in this presentation:

http://www.presentingperl.org/lpw2010/packaging-perl/

> [ big snip ]
>
> I think the significant thing here is I'm not really worrying about
> specific versions of dependencies. Sure, code depends on a minimun
> version of a module but that's just so tests have a chance of
> passing. What's important is that the unit tests pass. It's
> really no different than running "cpan Catalyst::Runtime" and making
> sure all tests pass.

If you have a very good coverage, that may well work. What makes me
feel not very safe is a scenario I'm having right now at work: the
same application works on some VMs, but fail with networking problems
on others. Is it a code problem, or a network problem? Two "rpm
--verify" later, I know that the exact same code is running on all
those VMs, two "puppetd" proves that the configuration is the same,
and at least I've ruled "different code" as a source of
differences. Doesn't help me prove it's the network, but at least I
don't have to write a tool to check all the installed distributions…

--
Dakkar - <Mobilis in mobile>
GPG public key fingerprint = A071 E618 DD2C 5901 9574
6FE2 40EA 9883 7519 3F88
key id = 0x75193F88

Captain: "You would have done the same."

Mal: "You can already see I haven't. Now get the hell off my ship."
--Episode #8, "Out of Gas"

Attachments:

signature.asc (0.19 KB)

moseley at hank

Jul 2, 2012, 3:56 PM

Post #5 of 9 (357 views)
Permalink

Re: Managing module regressions. [In reply to]

On Mon, Jul 2, 2012 at 3:32 PM, Gianni Ceccarelli <dakkar [at] thenautilus>wrote:

>
> > Do you use Perlbrew for that?
>
> We built our system before Perlbrew existed. Some details can been
> seen in this presentation:
>
> http://www.presentingperl.org/lpw2010/packaging-perl/

Well "some details" is accurate. ;) It's a bit hard to follow. I'll try
later with headphones when it's quite. But, essentially you build a
separate Perl on a staging machine with all dependencies and then use RPM
to copy to production, correct? Do you also use RPM for bringing in
non-perl dependencies? I assume by building Perl and installing your
modules you are not depending on any existing RPMs of Perl modules.

Thanks,

--
Bill Moseley
moseley [at] hank

dakkar at thenautilus

Jul 2, 2012, 11:55 PM

Post #6 of 9 (352 views)
Permalink

Re: Managing module regressions. [In reply to]

On 2012-07-02 Bill Moseley <moseley [at] hank> wrote:
> But, essentially you build a separate Perl on a staging machine with
> all dependencies and then use RPM to copy to production, correct?

Yes, to production and to any development VM.

> Do you also use RPM for bringing in non-perl dependencies?

Yes, like C libraries and external utilities.

> I assume by building Perl and installing your modules you are not
> depending on any existing RPMs of Perl modules.

Precisely.

--
Dakkar - <Mobilis in mobile>
GPG public key fingerprint = A071 E618 DD2C 5901 9574
6FE2 40EA 9883 7519 3F88
key id = 0x75193F88

"Trillian did a little research in the ship's copy of
THHGTTG. It had some advice to offer on drunkenness.
`Go to it,' it said, `and good luck.'
It was cross-referenced to the entry concerning the size of
the Universe and ways of coping with that."

- One of the more preferable pieces of advice contained in the Guide.

Attachments:

signature.asc (0.19 KB)

fred at redhotpenguin

Jul 24, 2012, 11:17 AM

Post #7 of 9 (289 views)
Permalink

Re: Managing module regressions. [In reply to]

I meant to reply to this a month ago but haven't had the time until now.

On Fri, Jun 29, 2012 at 8:24 AM, Bill Moseley <moseley [at] hank> wrote:
> This is a "how do others do this?" post.
>
> In your large (or even not so large) apps I assume at times you experience
> dependency regressions. My question is do you manage it on a case-by-case
> basis (simply install an older version or fix internally) or is it such a
> significant issue that you have a system for managing modules outside of
> CPAN?

I've been working with a few apps where we haven't experienced a lot
of dependency regressions per se, but have been dealing with several
modules that had major api changes. We have had to spend a significant
amount of time on maintenance in the apps to upgrade the code to
handle these updated module apis.

> The reason I ask is because at work we are considering managing different
> stacks of CPAN and in-house dependencies -- so maybe "dev", "testing", and
> "production/staging" stacks all in separate private CPAN mirrors. And now
> multiply that times the number of different apps we work on.

I would tend to stay away from this approach, as we have managed
several apps which use different versions of modules. We have a few
different development teams where developers upgrade modules without
any set schedule. Some of them are writing modules which are used by
all the apps, and the unannounced api changes in those modules have
had cascading effects which have slowed down deployment a lot. Only 2
of these developers have modules on CPAN, so I think the others
haven't experienced the real world pain that comes with rapid api
changes and are less cautious with their coding.

My feedback here would be try to stick with a fixed set of modules as
much as possible, no matter how tempting a new module looks.

> My experience is that even with the very large number of dependencies in a
> Catalyst app that it's pretty rare to have a regression. It happens, sure,
> and when it happens just deal with it. (And thinking of our own code it's
> typically not a regression in a module but a fix in a module where our code
> was depending on some broken behavior).
>
> So, the question is: does anyone else find it necessary to manage
> dependencies as I described above? And if so, what is your process?

We looked at implementing Pinto for module version management, but
that effort seemed to have never really taken off. I don't think the
part of the team implementing it dealt with the pain the rest of us
were experiencing on a day to day basis.

> Another argument that is floated around is we don't want to upgrade
> dependencies often because of potential new bugs. That seems a bit silly to
> me because it's ignoring known bug fixes for the chance that there might
> some unknown new bug. (Yes, we have apps running Catalyst from 2010!)

This has been a big pain point for us, enough that the build engineer
I work with to deploy the rpms with is now averse to upgrading
modules. The config module we developed in house changed its api 3
times in one year. A couple of the other core modules changed
significantly, causing weeks/months of delays. We had to upgrade to
Catalyst 5.9 and Moose 2.0 recently, and that was quite painful (this
is not a dig at either of those modules, just a reflection of our
upgrade experience).

We ran into this issue with the Moose 2 upgrade - took a few days to
track down. http://freebsd.so14k.com/problems_with_perl_catalyst.shtml

So while we haven't run into unknown new bugs, the cost of upgrading
modules on an ad hoc schedule has been large, and outweighing most
business value brought by the upgraded modules. I'm not saying that
you shouldn't upgrade modules when they are available. But such
upgrades should be scheduled apart from feature releases, and not done
ad hoc. This can be difficult when you have a loosely organized team
and developers want to try out the latest release of a module. Of
course, this is all in the context of an application that generates
revenue and has a user base that expects the app to remain largely bug
free. If I'm developing something that has an alpha user base, I'll
always go for the most recent release.

Also, these experiences are that of a large team developing large apps
to hundreds of servers. The experience of one developer deploying to a
small environment will certainly be different.

_______________________________________________
List: Catalyst [at] lists
Listinfo: http://lists.scsys.co.uk/cgi-bin/mailman/listinfo/catalyst
Searchable archive: http://www.mail-archive.com/catalyst [at] lists/
Dev site: http://dev.catalyst.perl.org/

bobtfish at bobtfish

Jul 24, 2012, 1:03 PM

Post #8 of 9 (293 views)
Permalink

Re: Managing module regressions. [In reply to]

On 24 Jul 2012, at 19:17, Fred Moyer wrote:
> Also, these experiences are that of a large team developing large apps
> to hundreds of servers. The experience of one developer deploying to a
> small environment will certainly be different.

I wonder how life would be different if you just deployed an entire perlbrew per app, so /opt/MyApp/bin/perl - this would make the cost of upgrading things much more trivial, as the cost would be per project, rather than having a wider impact.. It would also allow teams for each product to be strongly conservative (if that suited the team in question), or running much newer versions of stuff (on younger apps / more agile teams) - rather than having to dictate a version policy organisation wide.

I'm _not_ saying it would be better - everyone's environment and constraints are different, but thinking 'what if' about an entirely different strategy is entirely worthwhile. A lot of your pain seems to come from the fact that you can only have one version of every library on each system.

Cheers
t0m

_______________________________________________
List: Catalyst [at] lists
Listinfo: http://lists.scsys.co.uk/cgi-bin/mailman/listinfo/catalyst
Searchable archive: http://www.mail-archive.com/catalyst [at] lists/
Dev site: http://dev.catalyst.perl.org/

fred at redhotpenguin

Jul 24, 2012, 1:27 PM

Post #9 of 9 (289 views)
Permalink

Re: Managing module regressions. [In reply to]

On Tue, Jul 24, 2012 at 1:03 PM, Tomas Doran <bobtfish [at] bobtfish> wrote:
>
> On 24 Jul 2012, at 19:17, Fred Moyer wrote:
>> Also, these experiences are that of a large team developing large apps
>> to hundreds of servers. The experience of one developer deploying to a
>> small environment will certainly be different.
>
>
> I wonder how life would be different if you just deployed an entire perlbrew per app, so /opt/MyApp/bin/perl - this would make the cost of upgrading things much more trivial, as the cost would be per project, rather than having a wider impact.. It would also allow teams for each product to be strongly conservative (if that suited the team in question), or running much newer versions of stuff (on younger apps / more agile teams) - rather than having to dictate a version policy organisation wide.

Agreed on the heterogenous module approach for different apps. That
part of it has worked well, but sometimes dependencies leak up the
chain into your application. It's definitely not an easy problem. So
far though I think we've had success in the current approach despite
the pain points on certain part.

>
> I'm _not_ saying it would be better - everyone's environment and constraints are different, but thinking 'what if' about an entirely different strategy is entirely worthwhile. A lot of your pain seems to come from the fact that you can only have one version of every library on each system.
>
> Cheers
> t0m
>
>
> _______________________________________________
> List: Catalyst [at] lists
> Listinfo: http://lists.scsys.co.uk/cgi-bin/mailman/listinfo/catalyst
> Searchable archive: http://www.mail-archive.com/catalyst [at] lists/
> Dev site: http://dev.catalyst.perl.org/

_______________________________________________
List: Catalyst [at] lists
Listinfo: http://lists.scsys.co.uk/cgi-bin/mailman/listinfo/catalyst
Searchable archive: http://www.mail-archive.com/catalyst [at] lists/
Dev site: http://dev.catalyst.perl.org/

Index | Next | Previous | View Threaded

Interested in having your list archived? Contact Gossamer Threads