Mailing List Archive: Python: Dev

EC2 buildslaves

Index | Next | Previous | View Threaded

solipsis at pitrou

Oct 31, 2009, 1:13 PM

Post #1 of 10 (777 views)
Permalink

EC2 buildslaves

Martin v. LÃ¶wis <martin <at> v.loewis.de> writes:
>
> Not sure whether it's still relevant after the offers of individually
> donated hardware.

We'll see, indeed.

> However, if you want to look into this, feel free to
> set up EC2 slaves.

I only know to setup mainstream Linux distros though (Debian- or
Redhat-lookalikes :-)). I've just played a bit and, after the hassle of juggling
with a bunch of different keys and credentials, setting up an instance and
saving an image for future use is quite easy.

Regards

Antoine.

_______________________________________________
Python-Dev mailing list
Python-Dev [at] python
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: http://mail.python.org/mailman/options/python-dev/list-python-dev%40lists.gossamer-threads.com

exarkun at twistedmatrix

Nov 1, 2009, 9:54 AM

Post #2 of 10 (724 views)
Permalink

Re: EC2 buildslaves [In reply to]

On 31 Oct, 08:13 pm, solipsis [at] pitrou wrote:
>Martin v. Löwis <martin <at> v.loewis.de> writes:
>>
>>Not sure whether it's still relevant after the offers of individually
>>donated hardware.
>
>We'll see, indeed.
>>However, if you want to look into this, feel free to
>>set up EC2 slaves.
>
>I only know to setup mainstream Linux distros though (Debian- or
>Redhat-lookalikes :-)). I've just played a bit and, after the hassle of
>juggling
>with a bunch of different keys and credentials, setting up an instance
>and
>saving an image for future use is quite easy.

Starting with a mainstream distro doesn't seem like a bad idea. For
example, there isn't currently a 32bit Ubuntu (any version) slave. That
would be a nice gap to fill in, right?

Jean-Paul

solipsis at pitrou

Nov 2, 2009, 3:30 AM

Post #3 of 10 (720 views)
Permalink

Re: EC2 buildslaves [In reply to]

<exarkun <at> twistedmatrix.com> writes:
>
> Starting with a mainstream distro doesn't seem like a bad idea. For
> example, there isn't currently a 32bit Ubuntu (any version) slave. That
> would be a nice gap to fill in, right?

I've setup a buildslave on an EC2 Ubuntu Karmic instance here:
http://www.python.org/dev/buildbot/all/buildslaves/pitrou-ubuntu

However, since it's right now billed on my credit card, I'll shut it down in a
couple of days. I wonder how we can get the PSF to be billed instead of me, if
the PSF accepts to fund such an instance (which, given EC2 prices, is perhaps
not the best use of money?).

Regards

Antoine.

_______________________________________________
Python-Dev mailing list
Python-Dev [at] python
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: http://mail.python.org/mailman/options/python-dev/list-python-dev%40lists.gossamer-threads.com

martin at v

Nov 2, 2009, 4:31 AM

Post #4 of 10 (723 views)
Permalink

Re: EC2 buildslaves [In reply to]

> I've setup a buildslave on an EC2 Ubuntu Karmic instance here:
> http://www.python.org/dev/buildbot/all/buildslaves/pitrou-ubuntu
>
> However, since it's right now billed on my credit card, I'll shut it down in a
> couple of days. I wonder how we can get the PSF to be billed instead of me, if
> the PSF accepts to fund such an instance (which, given EC2 prices, is perhaps
> not the best use of money?).

Send a request to psf-board [at] python Such request should include:
- contact information (who'll be running this project)
- project purpose/description
- estimated expenses (in case of doubt, round up rather than
rounding down)
- a proposal of how payment should proceed. I'm not quite sure whether
it could be billed on the PSF credit card (you may ask Kurt Kaiser
in advance); traditionally, it worked best when we received invoices.
There will be a board meeting next Monday, so it might be useful to
have a proposal by then.

As for whether that's good use of the money, I'm skeptical as well.
I don't actually know what EC2 prices *are*, or what the current pricing
for a root server is (plus there had been various offers from people
donating hardware - from people who would be unable to also donate
time).

There was discussion that an EC2 instance can be turned on only when
needed, so we could try to set up something like that (the build master
could then trigger activation of the machine, IIUC). However, it might
be that the machine would have to be up most of the day, as there would
be one build going on always, anyway.

Regards,
Martin
_______________________________________________
Python-Dev mailing list
Python-Dev [at] python
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: http://mail.python.org/mailman/options/python-dev/list-python-dev%40lists.gossamer-threads.com

solipsis at pitrou

Nov 2, 2009, 4:41 AM

Post #5 of 10 (720 views)
Permalink

Re: EC2 buildslaves [In reply to]

Le lundi 02 novembre 2009 Ã 13:31 +0100, "Martin v. LÃ¶wis" a Ã©crit :
>
> There was discussion that an EC2 instance can be turned on only when
> needed, so we could try to set up something like that (the build master
> could then trigger activation of the machine, IIUC). However, it might
> be that the machine would have to be up most of the day, as there would
> be one build going on always, anyway.

Yes, I think that would be the case. We have frequent commits on each of
the 4 branches under test, with a test suite that takes quite a bit of
time to run in debug mode with -uall.

Moreover, a standard ("small", which also means cheapest) EC2 instance
apparently provides (based on a quick test) between 25% and 50% of the
power of a full CPU core, which makes builds longer. I thought a full
CPU core was provided, but it is not.

An always-on "small" EC2 instance is at least 500$ a year, with a small
storage cost added to that.

Therefore, I think EC2 buildslaves would be interesting under the
condition that they are donated rather than paid for. I don't know
whether anyone has contacts at Amazon.
(but then any donated piece of hardware would be good, not necessary an
EC2 instance)

I assume Jean-Paul made his original suggestion under the premise that
the EC2 instances would only be run when necessary, which is probably
very economical with Twisted's development model (few commits on trunk)
but not with ours.

Regards

Antoine.

_______________________________________________
Python-Dev mailing list
Python-Dev [at] python
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: http://mail.python.org/mailman/options/python-dev/list-python-dev%40lists.gossamer-threads.com

ssteinerx at gmail

Nov 2, 2009, 4:42 AM

Post #6 of 10 (720 views)
Permalink

Re: EC2 buildslaves [In reply to]

On Nov 2, 2009, at 6:30 AM, Antoine Pitrou wrote:

> <exarkun <at> twistedmatrix.com> writes:
>>
>> Starting with a mainstream distro doesn't seem like a bad idea. For
>> example, there isn't currently a 32bit Ubuntu (any version) slave.
>> That
>> would be a nice gap to fill in, right?
>
> I've setup a buildslave on an EC2 Ubuntu Karmic instance here:
> http://www.python.org/dev/buildbot/all/buildslaves/pitrou-ubuntu
>

If you could send me the script that you used to set it up, I could
give it a shot on RackSpace where it's cheaper (and I have a temporary
developer account).

S

_______________________________________________
Python-Dev mailing list
Python-Dev [at] python
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: http://mail.python.org/mailman/options/python-dev/list-python-dev%40lists.gossamer-threads.com

solipsis at pitrou

Nov 2, 2009, 4:49 AM

Post #7 of 10 (718 views)
Permalink

Re: EC2 buildslaves [In reply to]

Le lundi 02 novembre 2009 Ã 07:42 -0500, ssteinerX [at] gmail a Ã©crit :
>
> If you could send me the script that you used to set it up, I could
> give it a shot on RackSpace where it's cheaper (and I have a temporary
> developer account).

There's no need for a special script, really.
Install Python, buildbot, subversion (all packaged by your Linux
provider), then a couple of development libraries so that enough
extension modules get built: dev headers for zlib, bz2, openssl and
sqlite3 would be enough IMO.

When you have done that, go to http://wiki.python.org/moin/BuildBot and
follow the instructions at the end. Skip "install buildbot from source"
if you've installed it from your distro's package repo.
Also, you may have to replace "buildbot slave" with "buildbot
create-slave".

_______________________________________________
Python-Dev mailing list
Python-Dev [at] python
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: http://mail.python.org/mailman/options/python-dev/list-python-dev%40lists.gossamer-threads.com

martin at v

Nov 2, 2009, 4:59 AM

Post #8 of 10 (724 views)
Permalink

Re: EC2 buildslaves [In reply to]

> An always-on "small" EC2 instance is at least 500$ a year, with a small
> storage cost added to that.

OTOH, that isn't that expensive (compared to the other PSF expenses),
plus people keep donating money, so when we say what we use it for,
there may be a larger return than just the test results.

OTTH, the same could be achieved by buying a hosted server elsewhere.

Regards,
Martin

P.S. Perhaps this *is* the time for Steve to ask for "always-on" machines.
_______________________________________________
Python-Dev mailing list
Python-Dev [at] python
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: http://mail.python.org/mailman/options/python-dev/list-python-dev%40lists.gossamer-threads.com

solipsis at pitrou

Nov 2, 2009, 5:06 AM

Post #9 of 10 (726 views)
Permalink

Re: EC2 buildslaves [In reply to]

> OTOH, that isn't that expensive (compared to the other PSF expenses),
> plus people keep donating money, so when we say what we use it for,
> there may be a larger return than just the test results.
>
> OTTH, the same could be achieved by buying a hosted server elsewhere.

One advantage of a real hosted server is that we could have a bunch of
our own VMs on it, which is probably not possible (and perhaps not
allowed) on an EC2 instance (not to mention it could really be slow).

(I'm not volunteering to install and manage VMs, however; I don't think
I'm competent to do that)

_______________________________________________
Python-Dev mailing list
Python-Dev [at] python
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: http://mail.python.org/mailman/options/python-dev/list-python-dev%40lists.gossamer-threads.com

jnoller at gmail

Nov 2, 2009, 6:04 AM

Post #10 of 10 (719 views)
Permalink

Re: EC2 buildslaves [In reply to]

On Mon, Nov 2, 2009 at 8:06 AM, Antoine Pitrou <solipsis [at] pitrou> wrote:
>
>> OTOH, that isn't that expensive (compared to the other PSF expenses),
>> plus people keep donating money, so when we say what we use it for,
>> there may be a larger return than just the test results.
>>
>> OTTH, the same could be achieved by buying a hosted server elsewhere.
>
> One advantage of a real hosted server is that we could have a bunch of
> our own VMs on it, which is probably not possible (and perhaps not
> allowed) on an EC2 instance (not to mention it could really be slow).
>
> (I'm not volunteering to install and manage VMs, however; I don't think
> I'm competent to do that)

Now that the bulk of my pycon work is "completed" - I'll thread swap
to my VM host proposal and the moratorium PEP. So don't worry about
the VM host.

jesse
_______________________________________________
Python-Dev mailing list
Python-Dev [at] python
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: http://mail.python.org/mailman/options/python-dev/list-python-dev%40lists.gossamer-threads.com

Index | Next | Previous | View Threaded

Interested in having your list archived? Contact Gossamer Threads