Mailing List Archive: Python: Dev

Possible language summit topic: buildbots

1 2 3

View All

Index | Next | Previous | View Threaded

dickinsm at gmail

Oct 25, 2009, 1:54 AM

Post #1 of 60 (1499 views)
Permalink

Possible language summit topic: buildbots

Would it be worth spending some time discussing the buildbot situation
at the PyCon 2010 language summit? In the past, I've found the
buildbots to be an incredibly valuable resource; especially when
working with aspects of Python or C that tend to vary significantly
from platform to platform (for me, this usually means floating-point,
and platform math libraries, but there are surely many other things it
applies to). But more recently there seem to have been some
difficulties keeping a reasonable number of buildbots up and running.
A secondary problem is that it can be awkward to debug some of the
more obscure test failures on buildbots without having direct access
to the machine. From conversations on IRC, I don't think I'm alone in
wanting to find ways to make the buildbots more useful.

So the question is: how best to invest time and possibly money to
improve the buildbot situation (and as a result, I hope, improve the
quality of Python)? What could be done to make maintenance of build
slaves easier? Or to encourage interested third parties to donate
hardware and time? Are there good alternatives to Buildbot that might
make a difference? What do other projects do?

These are probably the wrong questions; I'm hoping that a discussion
would help produce the right questions, and possibly some answers.

Mark
_______________________________________________
Python-Dev mailing list
Python-Dev [at] python
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: http://mail.python.org/mailman/options/python-dev/list-python-dev%40lists.gossamer-threads.com

martin at v

Oct 25, 2009, 2:47 AM

Post #2 of 60 (1459 views)
Permalink

Re: Possible language summit topic: buildbots [In reply to]

Mark Dickinson wrote:
> Would it be worth spending some time discussing the buildbot situation
> at the PyCon 2010 language summit? In the past, I've found the
> buildbots to be an incredibly valuable resource; especially when
> working with aspects of Python or C that tend to vary significantly
> from platform to platform (for me, this usually means floating-point,
> and platform math libraries, but there are surely many other things it
> applies to). But more recently there seem to have been some
> difficulties keeping a reasonable number of buildbots up and running.
> A secondary problem is that it can be awkward to debug some of the
> more obscure test failures on buildbots without having direct access
> to the machine. From conversations on IRC, I don't think I'm alone in
> wanting to find ways to make the buildbots more useful.

These are actually two issues:
a) where do we get buildbot hardware and operators?
b) how can we reasonably debug problems occurring on buildbots

For a), I think we can solve this only by redundancy, i.e. create
more build slaves, hoping that a sufficient number would be up at
any point in time.

So: what specific kinds of buildbots do you think are currently lacking?
A call for volunteers will likely be answered quickly.

> So the question is: how best to invest time and possibly money to
> improve the buildbot situation (and as a result, I hope, improve the
> quality of Python)?

I don't think money will really help (I'm skeptical in general that
money helps in open source projects). As for time: "buildbot scales",
meaning that the buildbot slave admins will all share the load, being
responsible only for their own slaves.

On the master side: would you be interested in tracking slave admins?

> What could be done to make maintenance of build
> slaves easier?

This is something that only the slave admins can answer. I don't think
it's difficult - it's just that people are really unlikely to contribute
to the same thing over a period of five years at a steady rate. So we
need to make sure to find replacements when people drop out.

> Or to encourage interested third parties to donate
> hardware and time?

Again: I think a call for volunteers would do (Steve, if you are
reading this, please hold back just a few days before actually
making such a call :-)

> Are there good alternatives to Buildbot that might
> make a difference?

I think people have started working on such a thing. There are
certainly alternatives; I'm fairly skeptical that they are *good*
alternatives (but then, I'm the one who set up the buildbot
installation in the first place).

> What do other projects do?

I think that's really difficult to compare, since their testing
often has a very different scope. I think CruiseControl is widely
used.

> These are probably the wrong questions; I'm hoping that a discussion
> would help produce the right questions, and possibly some answers.

I think these are good questions - just not for the summit. Setting
up such a system is, conceptually, easy. It's also just a little work
to set it up initially; the difficult part then is to keep it running
(and no, a system where anybody can just post test results at any time
without prior registration is *still* difficult to keep running).

The source of the problem is that such a system can degrade without
anybody taking action. If the web server's hard disk breaks down,
people panic and look for a solution quickly. If the source control
is down, somebody *will* "volunteer" to fix it. If the automated
build system produces results less useful, people will worry, but
not take action.

Regards,
Martin
_______________________________________________
Python-Dev mailing list
Python-Dev [at] python
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: http://mail.python.org/mailman/options/python-dev/list-python-dev%40lists.gossamer-threads.com

solipsis at pitrou

Oct 25, 2009, 5:16 AM

Post #3 of 60 (1457 views)
Permalink

Re: Possible language summit topic: buildbots [In reply to]

> For a), I think we can solve this only by redundancy, i.e. create more
> build slaves, hoping that a sufficient number would be up at any point
> in time.

We are already doing this, aren't we?
http://www.python.org/dev/buildbot/3.x/

It doesn't seem to work very well, it's a bit like a Danaides vessel.

> The source of the problem is that such a system can degrade without
> anybody taking action. If the web server's hard disk breaks down, people
> panic and look for a solution quickly. If the source control is down,
> somebody *will* "volunteer" to fix it. If the automated build system
> produces results less useful, people will worry, but not take action.

Well, to be fair, buildbots breaking also happens much more frequently
(perhaps one or two orders of magnitude) than the SVN server or the Web
site going down. Maintaining them looks like a Sisyphean task, and nobody
wants that.

I don't know what kind of machines are the current slaves, but if they
are 24/7 servers, isn't it a bit surprising that the slaves would go down
so often? Is the buildbot software fragile? Does it require a lot of
(maintenance, repair) work from the slave owners?

_______________________________________________
Python-Dev mailing list
Python-Dev [at] python
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: http://mail.python.org/mailman/options/python-dev/list-python-dev%40lists.gossamer-threads.com

ctb at msu

Oct 25, 2009, 5:48 AM

Post #4 of 60 (1456 views)
Permalink

Re: Possible language summit topic: buildbots [In reply to]

On Sun, Oct 25, 2009 at 08:54:46AM +0000, Mark Dickinson wrote:
> Would it be worth spending some time discussing the buildbot situation
> at the PyCon 2010 language summit? In the past, I've found the
> buildbots to be an incredibly valuable resource; especially when
> working with aspects of Python or C that tend to vary significantly
> from platform to platform (for me, this usually means floating-point,
> and platform math libraries, but there are surely many other things it
> applies to). But more recently there seem to have been some
> difficulties keeping a reasonable number of buildbots up and running.
> A secondary problem is that it can be awkward to debug some of the
> more obscure test failures on buildbots without having direct access
> to the machine. From conversations on IRC, I don't think I'm alone in
> wanting to find ways to make the buildbots more useful.
>
> So the question is: how best to invest time and possibly money to
> improve the buildbot situation (and as a result, I hope, improve the
> quality of Python)? What could be done to make maintenance of build
> slaves easier? Or to encourage interested third parties to donate
> hardware and time? Are there good alternatives to Buildbot that might
> make a difference? What do other projects do?
>
> These are probably the wrong questions; I'm hoping that a discussion
> would help produce the right questions, and possibly some answers.

[. x-posting to testing-in-python; please redirect followups to one list or
the other! ]

Hi Mark,

a few bits of information...

---

I have a set of VM machines running some "core" build archs -- Linux, Mac OS X,
Win XP, Win Vista, and Win 7 -- that I am going to dedicate to this purpose.
I am happy to give out remote admin access to a few people. This
infrastructure is also going to increase slowly as I build up my lab's internal
network.

I'm giving Tarek an account on my Linux box later today to serve as a build
slave for Distribute.

--

More machines, and more esoteric machines, will be coming online as Snakebite
unfolds. Trent Nelson (Snakepit master) has been finishing up some consulting
work and is going to dedicate his time to it starting in November. This means
more 64 bit, bigmem, and "weird" archs, with full login access.

---

I've also been working on a buildbot alternative that I'm calling pony-build.
pony-build is based on a client-push architecture in which client machines
do builds and push results to the master, which then acts as a record-keeper
rather than a slave driver. The result is a less coordinated but (AFAICT) much
less fragile continuous integration system. I'm hoping to give a talk at PyCon
on the differences, and there will be a sprint on pony-build + pyhton-dev at
PyCon, regardless.

The current status of pony-build is "functional but ugly inside". In
particular, the data model is horrible, and the internal API needs much
more fleshing out. Nonetheless, my server has been running for two months
or so, and you can look at the results here,

http://lyorn.idyll.org/ctb/pb-dev/

The most fully-fleshed out set of build clients is for pygr,

http://lyorn.idyll.org/ctb/pb-dev/pygr/

but you can view daily build results for Python 2.7 at

http://lyorn.idyll.org/ctb/pb-dev/python/

with an exhaustive list here

http://lyorn.idyll.org/ctb/pb-dev/python/show_all

(and why the heck am I sorting in reverse date order, anyway?!)

The most interesting (?) part of pony-build right now is the client
config, which I'm struggling to make simple and potentially universal
enough to serve under buildbot as well:

http://github.com/ctb/pony-build/blob/master/client/build-cpython

(see 'commands' list).

The most *exciting* part of pony-build, apart from the always-riveting
spectacle of "titus rediscovering problems that buildbot solved 5 years ago",
is the loose coupling of recording server to the build slaves and build
reporters. My plan is to enable a simple and lightweight XML-RPC and/or
REST-ish interface for querying the recording server from scripts or other Web
sites. This has Brett aquiver with anticipation, I gather -- no more visual
inspection of buildbot waterfall pages ;)

--

If you're interested in bashing on, contributing to, or helping figure out
what color the pony-build main screen should be, contact me off-list; I'm
reluctant to spam up python-dev or testing-in-python.

That having been said, the results of taking it and trying it out -- you can
post results to my own recording server at

http://lyorn.idyll.org/ctb/pb-dev/xmlrpc

-- would be most welcome.

Once I fix the data model, code collaboration will be much more feasible,
too.

---

cheers,
--titus
--
C. Titus Brown, ctb [at] msu
_______________________________________________
Python-Dev mailing list
Python-Dev [at] python
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: http://mail.python.org/mailman/options/python-dev/list-python-dev%40lists.gossamer-threads.com

exarkun at twistedmatrix

Oct 25, 2009, 6:50 AM

Post #5 of 60 (1455 views)
Permalink

Re: Possible language summit topic: buildbots [In reply to]

On 12:16 pm, solipsis [at] pitrou wrote:
>
>>For a), I think we can solve this only by redundancy, i.e. create more
>>build slaves, hoping that a sufficient number would be up at any point
>>in time.
>
>We are already doing this, aren't we?
>http://www.python.org/dev/buildbot/3.x/
>
>It doesn't seem to work very well, it's a bit like a Danaides vessel.
>>The source of the problem is that such a system can degrade without
>>anybody taking action. If the web server's hard disk breaks down,
>>people
>>panic and look for a solution quickly. If the source control is down,
>>somebody *will* "volunteer" to fix it. If the automated build system
>>produces results less useful, people will worry, but not take action.
>
>Well, to be fair, buildbots breaking also happens much more frequently
>(perhaps one or two orders of magnitude) than the SVN server or the Web
>site going down. Maintaining them looks like a Sisyphean task, and
>nobody
>wants that.

Perhaps this is a significant portion of the problem. Maintaining a
build slave is remarkably simple and easy. I maintain about half a
dozen slaves and spend at most a few minutes a month operating them.
Actually setting one up in the first place might take a bit longer,
since it involves installing the necessary software and making sure
everything's set up right, but the actual slave configuration itself is
one command:

buildbot create-slave <path> <master address> <slave name> <slave

Perhaps this will help dispel the idea that it is a serious undertaking
to operate a slave.

The real requirement which some people may find challenging is that the
slave needs to operate on a host which is actually online almost all of
the time. If you don't such a machine, then there's little point
offering to host a slave.
>I don't know what kind of machines are the current slaves, but if they
>are 24/7 servers, isn't it a bit surprising that the slaves would go
>down
>so often? Is the buildbot software fragile? Does it require a lot of
>(maintenance, repair) work from the slave owners?

As I have no specific experience maintaining any of the CPython build
slaves, I can't speak to any maintenance issues which these slaves have
encountered. I would expect that they are as minimal as the issues I
have encountered maintaining slaves for other projects, but perhaps this
is wrong. I do recall that there were some win32 issues (discussed on
this list, I think) quite a while back, but I think those were resolved.
I haven't heard of any other issues since then. If there are some,
perhaps the people who know about them could raise them and we could try
to figure out how to resolve them.

Jean-Paul
_______________________________________________
Python-Dev mailing list
Python-Dev [at] python
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: http://mail.python.org/mailman/options/python-dev/list-python-dev%40lists.gossamer-threads.com

exarkun at twistedmatrix

Oct 25, 2009, 7:05 AM

Post #6 of 60 (1455 views)
Permalink

Re: Possible language summit topic: buildbots [In reply to]

On 09:47 am, martin [at] v wrote:
>Mark Dickinson wrote:
>>Would it be worth spending some time discussing the buildbot situation
>>at the PyCon 2010 language summit? In the past, I've found the
>>buildbots to be an incredibly valuable resource; especially when
>>working with aspects of Python or C that tend to vary significantly
>>from platform to platform (for me, this usually means floating-point,
>>and platform math libraries, but there are surely many other things it
>>applies to). But more recently there seem to have been some
>>difficulties keeping a reasonable number of buildbots up and running.
>>A secondary problem is that it can be awkward to debug some of the
>>more obscure test failures on buildbots without having direct access
>>to the machine. From conversations on IRC, I don't think I'm alone in
>>wanting to find ways to make the buildbots more useful.
>
>These are actually two issues:
>a) where do we get buildbot hardware and operators?
>b) how can we reasonably debug problems occurring on buildbots
>
>For a), I think we can solve this only by redundancy, i.e. create
>more build slaves, hoping that a sufficient number would be up at
>any point in time.
>
>So: what specific kinds of buildbots do you think are currently
>lacking?
>A call for volunteers will likely be answered quickly.
>>So the question is: how best to invest time and possibly money to
>>improve the buildbot situation (and as a result, I hope, improve the
>>quality of Python)?
>
>I don't think money will really help (I'm skeptical in general that
>money helps in open source projects). As for time: "buildbot scales",
>meaning that the buildbot slave admins will all share the load, being
>responsible only for their own slaves.

I think that money can help in two ways in this case.

First, there are now a multitude of cloud hosting providers which will
operate a slave machine for you. BuildBot has even begun to support
this deployment use-case by allowing you to start up and shut down vms
on demand to save on costs. Amazon's EC2 service is supported out of
the box in the latest release.

Second, there are a number of active BuildBot developers. One of them
has even recently taken a contract from Mozilla to implement some non-
trivial BuildBot enhancements. I think it very likely that he would
consider taking such a contract from the PSF for whatever enhancements
would help out the CPython buildbot.
>On the master side: would you be interested in tracking slave admins?
>>What could be done to make maintenance of build
>>slaves easier?
>
>This is something that only the slave admins can answer. I don't think
>it's difficult - it's just that people are really unlikely to
>contribute
>to the same thing over a period of five years at a steady rate. So we
>need to make sure to find replacements when people drop out.

This is a good argument for VMs. It's certainly *possible* to chase an
ever changing set of platforms, but it strikes me as something of a
waste of time.
>The source of the problem is that such a system can degrade without
>anybody taking action. If the web server's hard disk breaks down,
>people panic and look for a solution quickly. If the source control
>is down, somebody *will* "volunteer" to fix it. If the automated
>build system produces results less useful, people will worry, but
>not take action.

To me, that raises the question of why people aren't more concerned with
the status of the build system. Shouldn't developers care if the code
they're writing works or not?

Jean-Paul
_______________________________________________
Python-Dev mailing list
Python-Dev [at] python
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: http://mail.python.org/mailman/options/python-dev/list-python-dev%40lists.gossamer-threads.com

jnoller at gmail

Oct 25, 2009, 7:10 AM

Post #7 of 60 (1457 views)
Permalink

Re: Possible language summit topic: buildbots [In reply to]

On Sun, Oct 25, 2009 at 8:48 AM, C. Titus Brown <ctb [at] msu> wrote:

> [. x-posting to testing-in-python; please redirect followups to one list or
> the other! ]
>
> Hi Mark,
>
> a few bits of information...
>
> ---
>
> I have a set of VM machines running some "core" build archs -- Linux, Mac OS X,
> Win XP, Win Vista, and Win 7 -- that I am going to dedicate to this purpose.
> I am happy to give out remote admin access to a few people. �This
> infrastructure is also going to increase slowly as I build up my lab's internal
> network.
>
> I'm giving Tarek an account on my Linux box later today to serve as a build
> slave for Distribute.

Just to add to what titus said; I'm trying to price out a decent
Desktop machine with enough ram/disk/cpu to run VMWare ESXi and serve
a variety of virtual machines. I had planned on (ab)using the free
MSDN license Microsoft provided to get a variety of platforms up and
running.

The end goal (since I have excess bandwidth and cooling where I live)
would be to maintain this box as a series of buildslaves python-dev
would have near unlimited shell/remote desktop access to.

The nice thing about this would be that once the initial cost was sunk
for the machine itself, and making all the virtual machines, in theory
the machine could run a set of "common" virtual machines all the time,
with a set of virtual machines on standby if someone needed to debug a
less common problem.

Yes, this is a mini-snakebite concept. Right now the main blocker is
funding and time - that and I need to spec something that doesn't
sound like a jet engine ;)

jesse
_______________________________________________
Python-Dev mailing list
Python-Dev [at] python
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: http://mail.python.org/mailman/options/python-dev/list-python-dev%40lists.gossamer-threads.com

exarkun at twistedmatrix

Oct 25, 2009, 7:13 AM

Post #8 of 60 (1456 views)
Permalink

Re: [TIP] Possible language summit topic: buildbots [In reply to]

On 12:48 pm, ctb [at] msu wrote:
>
>[snip]
>
>The most *exciting* part of pony-build, apart from the always-riveting
>spectacle of "titus rediscovering problems that buildbot solved 5 years
>ago",
>is the loose coupling of recording server to the build slaves and build
>reporters. My plan is to enable a simple and lightweight XML-RPC
>and/or
>REST-ish interface for querying the recording server from scripts or
>other Web
>sites. This has Brett aquiver with anticipation, I gather -- no more
>visual
>inspection of buildbot waterfall pages ;)

BuildBot has an XML-RPC interface. So Brett can probably do what he
wants with BuildBot right now.

Jean-Paul
_______________________________________________
Python-Dev mailing list
Python-Dev [at] python
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: http://mail.python.org/mailman/options/python-dev/list-python-dev%40lists.gossamer-threads.com

ssteinerx at gmail

Oct 25, 2009, 7:27 AM

Post #9 of 60 (1458 views)
Permalink

Re: Possible language summit topic: buildbots [In reply to]

On Oct 25, 2009, at 5:47 AM, Martin v. L�wis wrote:
>
> These are actually two issues:
> a) where do we get buildbot hardware and operators?

I've been trying to get some feedback about firing up buildbots on
Cloud Servers for a while now and haven't had much luck. I'd love to
find a way of having buildbots come to life, report to the mother
ship, do the build, then go away 'till next time they're required.

S

_______________________________________________
Python-Dev mailing list
Python-Dev [at] python
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: http://mail.python.org/mailman/options/python-dev/list-python-dev%40lists.gossamer-threads.com

ssteinerx at gmail

Oct 25, 2009, 7:35 AM

Post #10 of 60 (1456 views)
Permalink

Re: Possible language summit topic: buildbots [In reply to]

On Oct 25, 2009, at 9:50 AM, exarkun [at] twistedmatrix wrote:
> Actually setting one up in the first place might take a bit longer,
> since it involves installing the necessary software and making sure
> everything's set up right, but the actual slave configuration itself
> is one command:
>
> buildbot create-slave <path> <master address> <slave name> <slave
> password>

I have written a Fabric script for the distutils-buildbot project (on
bitbucket, under Tarek) that puts everything necessary up onto an
Ubuntu server, runs all the build steps, and fires up the buildbot.

Obviously it will have to be modified to correctly configure other
types of servers but the implementation should be fairly trivial for
someone who could have done it by hand in the first place. Once
it's done, it's in the script and may require an occasional tweak but
not much more.

The next step is to have the slaves themselves created in the cloud,
fired up and then report to the mother ship that they're available.
This last step is the one that doesn't seem to be supported by the
current system.

Thanks,

S

_______________________________________________
Python-Dev mailing list
Python-Dev [at] python
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: http://mail.python.org/mailman/options/python-dev/list-python-dev%40lists.gossamer-threads.com

solipsis at pitrou

Oct 25, 2009, 7:36 AM

Post #11 of 60 (1457 views)
Permalink

Re: Possible language summit topic: buildbots [In reply to]

<exarkun <at> twistedmatrix.com> writes:
>
> To me, that raises the question of why people aren't more concerned with
> the status of the build system. Shouldn't developers care if the code
> they're writing works or not?

The fact that we ask questions and publicly express worries should hint that we
/are/ concerned :-)
However, being mostly developers rather than system admins, and not knowing
anything about the details of how buildbot does its work (not to mention the
details of this or that particular buildslave and slave owner), makes us (at
least me) quite clueless when faced with a buildbot-not-working-as-expected
problem.

Regards

Antoine.

_______________________________________________
Python-Dev mailing list
Python-Dev [at] python
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: http://mail.python.org/mailman/options/python-dev/list-python-dev%40lists.gossamer-threads.com

ssteinerx at gmail

Oct 25, 2009, 7:37 AM

Post #12 of 60 (1457 views)
Permalink

Re: Possible language summit topic: buildbots [In reply to]

On Oct 25, 2009, at 10:05 AM, exarkun [at] twistedmatrix wrote:
> First, there are now a multitude of cloud hosting providers which
> will operate a slave machine for you. BuildBot has even begun to
> support this deployment use-case by allowing you to start up and
> shut down vms on demand to save on costs. Amazon's EC2 service is
> supported out of the box in the latest release.

I have been working to expand this support to Rackspace's Cloud
Servers as well.

S

_______________________________________________
Python-Dev mailing list
Python-Dev [at] python
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: http://mail.python.org/mailman/options/python-dev/list-python-dev%40lists.gossamer-threads.com

p.f.moore at gmail

Oct 25, 2009, 10:47 AM

Post #13 of 60 (1439 views)
Permalink

Re: Possible language summit topic: buildbots [In reply to]

2009/10/25 <exarkun [at] twistedmatrix>:
> Perhaps this is a significant portion of the problem. �Maintaining a build
> slave is remarkably simple and easy. �I maintain about half a dozen slaves
> and spend at most a few minutes a month operating them. Actually setting one
> up in the first place might take a bit longer, since it involves installing
> the necessary software and making sure everything's set up right, but the
> actual slave configuration itself is one command:
>
> �buildbot create-slave <path> <master address> <slave name> <slave password>
>
> Perhaps this will help dispel the idea that it is a serious undertaking to
> operate a slave.
>
> The real requirement which some people may find challenging is that the
> slave needs to operate on a host which is actually online almost all of the
> time. �If you don't such a machine, then there's little point offering to
> host a slave.

I have been seriously considering setting up one or more buildslaves
for a while now. However, my biggest issue is that they would be
running as VMs on my normal PC, which means that it's the issue of
keeping them continually online that hurts me.

If I could (say) just fire the slaves up for a set period, or fire
them up, have them do a build and report back, and then shut down,
that would make my life easier (regular activities rather than ongoing
sysadmin works better for me).

It sounds like a buildslave isn't really what I should be looking at.
Maybe Titus' push model pony-build project would make more sense for
me.

Paul.
_______________________________________________
Python-Dev mailing list
Python-Dev [at] python
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: http://mail.python.org/mailman/options/python-dev/list-python-dev%40lists.gossamer-threads.com

exarkun at twistedmatrix

Oct 25, 2009, 10:59 AM

Post #14 of 60 (1439 views)
Permalink

Re: Possible language summit topic: buildbots [In reply to]

On 05:47 pm, p.f.moore [at] gmail wrote:
>2009/10/25 <exarkun [at] twistedmatrix>:
>>Perhaps this is a significant portion of the problem. �Maintaining a
>>build
>>slave is remarkably simple and easy. �I maintain about half a dozen
>>slaves
>>and spend at most a few minutes a month operating them. Actually
>>setting one
>>up in the first place might take a bit longer, since it involves
>>installing
>>the necessary software and making sure everything's set up right, but
>>the
>>actual slave configuration itself is one command:
>>
>>�buildbot create-slave <path> <master address> <slave name> <slave
>>password>
>>
>>Perhaps this will help dispel the idea that it is a serious
>>undertaking to
>>operate a slave.
>>
>>The real requirement which some people may find challenging is that
>>the
>>slave needs to operate on a host which is actually online almost all
>>of the
>>time. �If you don't such a machine, then there's little point offering
>>to
>>host a slave.
>
>I have been seriously considering setting up one or more buildslaves
>for a while now. However, my biggest issue is that they would be
>running as VMs on my normal PC, which means that it's the issue of
>keeping them continually online that hurts me.
>
>If I could (say) just fire the slaves up for a set period, or fire
>them up, have them do a build and report back, and then shut down,
>that would make my life easier (regular activities rather than ongoing
>sysadmin works better for me).
>
>It sounds like a buildslave isn't really what I should be looking at.
>Maybe Titus' push model pony-build project would make more sense for
>me.

Maybe. I wonder if Titus' "push model" (I don't really understand this
term in this context) makes sense for continuous integration at all,
though. As a developer, I don't want to have access to build results
across multiple platforms when someone else feels like it. I want
access when *I* feel like it.

Anyway, BuildBot is actually perfectly capable of dealing with this. I
failed to separate my assumptions about how everyone would want to use
the system from what the system is actually capable of.

If you run a build slave and it's offline when a build is requested, the
build will be queued and run when the slave comes back online. So if
the CPython developers want to work this way (I wouldn't), then we don't
need pony-build; BuildBot will do just fine.

Jean-Paul

p.f.moore at gmail

Oct 25, 2009, 11:15 AM

Post #15 of 60 (1440 views)
Permalink

Re: Possible language summit topic: buildbots [In reply to]

2009/10/25 <exarkun [at] twistedmatrix>:
> If you run a build slave and it's offline when a build is requested, the
> build will be queued and run when the slave comes back online. �So if the
> CPython developers want to work this way (I wouldn't), then we don't need
> pony-build; BuildBot will do just fine.

OK, sounds useful. If I'm offline for a while, do multiple builds get
queued, or only the "last" one? If the former, I can imagine coming
back to a pretty huge load if the slave breaks while I'm on holiday
:-(

I should look all of this up somewhere. Is there a reference to
buildbot for slave maintainers? Are there any specifics for Python
slaves that I should refer to?

(From what I've been able to find, it seems to me that setting up a
slave first requires getting things sorted with the admins, which
sadly precludes experimenting to find things out - I can understand
why the python admins don't want people "playing" on the live buildbot
infrastructure, though :-))

Paul.
_______________________________________________
Python-Dev mailing list
Python-Dev [at] python
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: http://mail.python.org/mailman/options/python-dev/list-python-dev%40lists.gossamer-threads.com

martin at v

Oct 25, 2009, 11:32 AM

Post #16 of 60 (1440 views)
Permalink

Re: Possible language summit topic: buildbots [In reply to]

> I've been trying to get some feedback about firing up buildbots on Cloud
> Servers for a while now and haven't had much luck. I'd love to find a
> way of having buildbots come to life, report to the mother ship, do the
> build, then go away 'till next time they're required.

I'm not quite sure whom you have been trying to get feedback from, and
can't quite picture your proposed setup from above description.

In any case, ISTM that your approach isn't compatible with how buildbot
works today (not sure whether you are aware of that): a build slave
needs to stay connected all the time, so that the build master can
trigger a build when necessary.

So if your setup requires the slaves to shut down after a build, I don't
think this can possibly work.

Regards,
Martin
_______________________________________________
Python-Dev mailing list
Python-Dev [at] python
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: http://mail.python.org/mailman/options/python-dev/list-python-dev%40lists.gossamer-threads.com

ctb at msu

Oct 25, 2009, 11:38 AM

Post #17 of 60 (1449 views)
Permalink

Re: Possible language summit topic: buildbots [In reply to]

On Sun, Oct 25, 2009 at 07:32:52PM +0100, "Martin v. L?wis" wrote:
> > I've been trying to get some feedback about firing up buildbots on Cloud
> > Servers for a while now and haven't had much luck. I'd love to find a
> > way of having buildbots come to life, report to the mother ship, do the
> > build, then go away 'till next time they're required.
>
> I'm not quite sure whom you have been trying to get feedback from, and
> can't quite picture your proposed setup from above description.
>
> In any case, ISTM that your approach isn't compatible with how buildbot
> works today (not sure whether you are aware of that): a build slave
> needs to stay connected all the time, so that the build master can
> trigger a build when necessary.

Hi Martin,

it shouldn't be difficult to cobble together a build script that spins up a
buildslave on EC2 and runs the tests there; I wrote something similar a
few years ago for an infrequently connected home machine.

--titus
--
C. Titus Brown, ctb [at] msu
_______________________________________________
Python-Dev mailing list
Python-Dev [at] python
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: http://mail.python.org/mailman/options/python-dev/list-python-dev%40lists.gossamer-threads.com

exarkun at twistedmatrix

Oct 25, 2009, 11:45 AM

Post #18 of 60 (1440 views)
Permalink

Re: Possible language summit topic: buildbots [In reply to]

On 06:32 pm, martin [at] v wrote:
>>I've been trying to get some feedback about firing up buildbots on
>>Cloud
>>Servers for a while now and haven't had much luck. I'd love to find a
>>way of having buildbots come to life, report to the mother ship, do
>>the
>>build, then go away 'till next time they're required.
>
>I'm not quite sure whom you have been trying to get feedback from, and
>can't quite picture your proposed setup from above description.
>
>In any case, ISTM that your approach isn't compatible with how buildbot
>works today (not sure whether you are aware of that): a build slave
>needs to stay connected all the time, so that the build master can
>trigger a build when necessary.
>
>So if your setup requires the slaves to shut down after a build, I
>don't
>think this can possibly work.

This is supported in recent versions of BuildBot with a special kind of
slave:

http://djmitche.github.com/buildbot/docs/0.7.11/#On_002dDemand-
_0028_0022Latent_0022_0029-Buildslaves

Jean-Paul
_______________________________________________
Python-Dev mailing list
Python-Dev [at] python
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: http://mail.python.org/mailman/options/python-dev/list-python-dev%40lists.gossamer-threads.com

martin at v

Oct 25, 2009, 11:57 AM

Post #19 of 60 (1440 views)
Permalink

Re: Possible language summit topic: buildbots [In reply to]

Antoine Pitrou wrote:
>> For a), I think we can solve this only by redundancy, i.e. create more
>> build slaves, hoping that a sufficient number would be up at any point
>> in time.
>
> We are already doing this, aren't we?
> http://www.python.org/dev/buildbot/3.x/
>
> It doesn't seem to work very well, it's a bit like a Danaides vessel.

Both true. However, it seems that Mark is unhappy with the current set
of systems, so we probably need to do it again.

> Well, to be fair, buildbots breaking also happens much more frequently
> (perhaps one or two orders of magnitude) than the SVN server or the Web
> site going down. Maintaining them looks like a Sisyphean task, and nobody
> wants that.

It only looks so. It is like any server management task - it takes
constant effort. However, it is not Sisyphean (feeling Greek today,
ain't you :-); since you actually achieve something. It's not hard to
restart a buildbot when it has crashed, and it gives a warm feeling of
having achieved something.

> I don't know what kind of machines are the current slaves, but if they
> are 24/7 servers, isn't it a bit surprising that the slaves would go down
> so often? Is the buildbot software fragile?

Not really. It sometimes happens that the slaves don't reconnect after
a master restart, but more often, it is just a change on the slave side
that breaks it (such as a reboot done to the machine, and not having
the machine configured to restart the slave after the reboot).

> Does it require a lot of
> (maintenance, repair) work from the slave owners?

On Unix, not really. On Windows, there is still the issue that
sometimes, some error message pops up which you need to click away.
Over several builds, you may find that you have to click away dozens
of such messages. This could use some improvement.

Regards,
Martin
_______________________________________________
Python-Dev mailing list
Python-Dev [at] python
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: http://mail.python.org/mailman/options/python-dev/list-python-dev%40lists.gossamer-threads.com

martin at v

Oct 25, 2009, 12:06 PM

Post #20 of 60 (1440 views)
Permalink

Re: Possible language summit topic: buildbots [In reply to]

> As I have no specific experience maintaining any of the CPython build
> slaves, I can't speak to any maintenance issues which these slaves have
> encountered. I would expect that they are as minimal as the issues I
> have encountered maintaining slaves for other projects, but perhaps this
> is wrong. I do recall that there were some win32 issues (discussed on
> this list, I think) quite a while back, but I think those were resolved.
> I haven't heard of any other issues since then.

Only partially. One old issue was that previous builds would not
complete, keeping the executable files open, preventing further
runs. Buildbot is supposed to kill a build, but only kills the
parent process (as it is really difficult to kill the entire process
tree (*)). We work around this by explicitly killing any stale Python
processes at the beginning of a new build.

The remaining issue is the popups; if a process still has a popup,
you can't even terminate it properly. There are two kinds of popups:
system-generated ones, and CRT-generated ones. For the CRT ones, we
once had a way to turn them off, but I'm not sure whether that mechanism
might have been removed. For the system messages, there is a way to
turn them off in the parent process. David Bolen (IIRC) had developed
a patch, but I think this patch only runs on his system(s).

Regards,
Martin

(*) it may help if Buildbot would create a Win32 job object, and
then use TerminateJobObject. Contributions are welcome.
_______________________________________________
Python-Dev mailing list
Python-Dev [at] python
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: http://mail.python.org/mailman/options/python-dev/list-python-dev%40lists.gossamer-threads.com

brett at python

Oct 25, 2009, 12:12 PM

Post #21 of 60 (1442 views)
Permalink

Re: [TIP] Possible language summit topic: buildbots [In reply to]

On Sun, Oct 25, 2009 at 07:13, <exarkun [at] twistedmatrix> wrote:

> On 12:48 pm, ctb [at] msu wrote:
>
>>
>> [snip]
>>
>>
>> The most *exciting* part of pony-build, apart from the always-riveting
>> spectacle of "titus rediscovering problems that buildbot solved 5 years
>> ago",
>> is the loose coupling of recording server to the build slaves and build
>> reporters. My plan is to enable a simple and lightweight XML-RPC and/or
>> REST-ish interface for querying the recording server from scripts or other
>> Web
>> sites. This has Brett aquiver with anticipation, I gather -- no more
>> visual
>> inspection of buildbot waterfall pages ;)
>>
>
> BuildBot has an XML-RPC interface. So Brett can probably do what he wants
> with BuildBot right now.
>

Brett actually wants web hooks so pony-build will ping an App Engine web app
when there is more data, ala PubSubHubbub. Or hell, just have pony-build
have an Atom feed with updates and simply use PuSH. In other words I want to
be told when there is an update, not have to poll to find out.

-Brett

martin at v

Oct 25, 2009, 12:15 PM

Post #22 of 60 (1465 views)
Permalink

Re: Possible language summit topic: buildbots [In reply to]

> OK, sounds useful. If I'm offline for a while, do multiple builds get
> queued, or only the "last" one?

IIRC, it will only build the last one, then with a huge blame list.

> If the former, I can imagine coming
> back to a pretty huge load if the slave breaks while I'm on holiday
> :-(

If it's offline too often, I'm skeptical that it would be useful. If
you report breakage after a day, then it will be difficult to attribute
this to a specific commit. It is most useful to have continuous
integration if error reports are instantaneous.

> I should look all of this up somewhere. Is there a reference to
> buildbot for slave maintainers? Are there any specifics for Python
> slaves that I should refer to?

Hmm. I thought I had send you this before:

http://wiki.python.org/moin/BuildbotOnWindows

> (From what I've been able to find, it seems to me that setting up a
> slave first requires getting things sorted with the admins, which
> sadly precludes experimenting to find things out - I can understand
> why the python admins don't want people "playing" on the live buildbot
> infrastructure, though :-))

That's really not an issue. Feel free to play as much as you want, with
the live infrastructure. You can't break anything doing so (perhaps
except for spamming the mailing list with faulty reports). If you then
decide to withdraw your offer, that's fine as well (just make sure to
notify us instead of just silently taking the slave down).

Regards,
Martin
_______________________________________________
Python-Dev mailing list
Python-Dev [at] python
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: http://mail.python.org/mailman/options/python-dev/list-python-dev%40lists.gossamer-threads.com

ssteinerx at gmail

Oct 25, 2009, 12:19 PM

Post #23 of 60 (1436 views)
Permalink

Re: Possible language summit topic: buildbots [In reply to]

On Oct 25, 2009, at 2:32 PM, Martin v. L�wis wrote:

>> I've been trying to get some feedback about firing up buildbots on
>> Cloud
>> Servers for a while now and haven't had much luck. I'd love to
>> find a
>> way of having buildbots come to life, report to the mother ship, do
>> the
>> build, then go away 'till next time they're required.
>
> I'm not quite sure whom you have been trying to get feedback from, and
> can't quite picture your proposed setup from above description.

I posted a couple of messages on testing-in-python, and have sent
around some queries to others that I know are using buildbot type
setups with various tools/platforms, not necessarily Python.

> In any case, ISTM that your approach isn't compatible with how
> buildbot
> works today (not sure whether you are aware of that): a build slave
> needs to stay connected all the time, so that the build master can
> trigger a build when necessary.
>
> So if your setup requires the slaves to shut down after a build, I
> don't
> think this can possibly work.

It can't possibly work within the way the Python buildbot structure
currently works, as I understand it so far.

What I'm implementing is less of a 'continuous integration' tool like
you would use for something like Python itself, and more of a testing
tool for things that have to be installed on multiple versions of
Python, on multiple platforms.

I don't need to know that it works on every checkin, just every once
in a while I'd like to start from scratch and make sure everything
still works on all supported versions of Python on all the platforms I
test on and cloud servers are great for this since I'll usually only
need them for an hour or so.

S

_______________________________________________
Python-Dev mailing list
Python-Dev [at] python
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: http://mail.python.org/mailman/options/python-dev/list-python-dev%40lists.gossamer-threads.com

solipsis at pitrou

Oct 25, 2009, 12:23 PM

Post #24 of 60 (1437 views)
Permalink

Re: Possible language summit topic: buildbots [In reply to]

Martin v. Löwis <martin <at> v.loewis.de> writes:
>
> If it's offline too often, I'm skeptical that it would be useful. If
> you report breakage after a day, then it will be difficult to attribute
> this to a specific commit. It is most useful to have continuous
> integration if error reports are instantaneous.

Not only, but it's useful to have a stable set of buildbots, so that when a test
fails, you know whether it's a new problem caused by a recent revision, or
rather an already existing problem on this platform.

Regards

Antoine.

_______________________________________________
Python-Dev mailing list
Python-Dev [at] python
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: http://mail.python.org/mailman/options/python-dev/list-python-dev%40lists.gossamer-threads.com

martin at v

Oct 25, 2009, 12:24 PM

Post #25 of 60 (1435 views)
Permalink

Re: Possible language summit topic: buildbots [In reply to]

> I think that money can help in two ways in this case.
>
> First, there are now a multitude of cloud hosting providers which will
> operate a slave machine for you. BuildBot has even begun to support
> this deployment use-case by allowing you to start up and shut down vms
> on demand to save on costs. Amazon's EC2 service is supported out of
> the box in the latest release.

Here I'm skeptical. I think we can find people donating always-online
machines still; no need to throw donated money to Amazon.

> Second, there are a number of active BuildBot developers. One of them
> has even recently taken a contract from Mozilla to implement some non-
> trivial BuildBot enhancements. I think it very likely that he would
> consider taking such a contract from the PSF for whatever enhancements
> would help out the CPython buildbot.

That could indeed be interesting, assuming we had a clear requirement.
But then, most of us can "easily" fix things in buildbot themselves -
this is python-dev, after all.

>> This is something that only the slave admins can answer. I don't think
>> it's difficult - it's just that people are really unlikely to contribute
>> to the same thing over a period of five years at a steady rate. So we
>> need to make sure to find replacements when people drop out.
>
> This is a good argument for VMs.

Not really - you still would need somebody to manage them.

> It's certainly *possible* to chase an
> ever changing set of platforms, but it strikes me as something of a
> waste of time.

Hmm - can you really get "strange" operating systems "in the cloud"?
Some of the operating systems that we would like to test don't
even support VMs.

> To me, that raises the question of why people aren't more concerned with
> the status of the build system. Shouldn't developers care if the code
> they're writing works or not?

I think there are two issues here:
1. some developers actually *don't* care to much whether their code
works in all cases. If it fails on some strange platform they never
heard of (such as "Solaris", or "Windows"), they are not bothered by
the failure. Or, if they care, they still don't know what to do about
the failure.
2. the buildbots sometimes report false positives. Some tests fail in
a non-repeatable fashion, only on selected systems. So when you
change something, the tests break, and you cannot really see how
this could be possibly related to your change. Then you start
ignoring these reports - both the bogus ones, and the real ones.

Regards,
Martin
_______________________________________________
Python-Dev mailing list
Python-Dev [at] python
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: http://mail.python.org/mailman/options/python-dev/list-python-dev%40lists.gossamer-threads.com

1 2 3

View All

Index | Next | Previous | View Threaded

Interested in having your list archived? Contact Gossamer Threads