Mailing List Archive: exim: dev

Vagrant&Chef and/or Exim Build Farm

Index | Next | Previous | View Threaded

tlyons at ivenue

Jul 9, 2013, 7:03 AM

Post #1 of 4 (94 views)
Permalink

Vagrant&Chef and/or Exim Build Farm

Phil P mentioned this in another thread:

> Todd is working on some Vagrant stuff that might change that and make it easier to benchmark?

Phil and I had spoken offline a while back about various ways of
making development better/easier and increasing confidence in commits.
We have ideas that complement each other in an effort to achieve
this.

1. Phil's idea is to use Vagrant (multi OS images) with Chef for
configuration control. Then you can get build results by building the
master branch across multiple platforms. I also envision the ability
for a dev to submit his/her own repo and branch(es) as a source, and
have it build those targets as well. Anybody with Vagrant/Chef
experience, I welcome advice and input on how to best implement this.
So far, I have a user named "farm", and am about to install a few
different OS images under vagrant. It's all new to me. I'm browsing
Vagrant/Chef HOWTOs now.

2. My thoughts were a little less extravagant. I was a member of the
Samba build farm for a few years. Participants' machines synced to
the master samba buildfarm server (multiple repos), built multiple
configs when new commits were found, ran samba torture (essentially
make test), and results were submitted back up to a master samba
buildfarm server, where it was parsed and made available for
display/analysis. I'd like to consider doing the same for Exim.
There are a couple of public projects that use a distributed build
farm, and we could simply model off of their systems.

The two ideas aren't mutually exclusive, but they do have common
portions which make it somewhat duplicative to do both. Comments,
feedback?

...Todd
--
The total budget at all receivers for solving senders' problems is $0.
If you want them to accept your mail and manage it the way you want,
send it the way the spec says to. --John Levine

--
## List details at https://lists.exim.org/mailman/listinfo/exim-dev Exim details at http://www.exim.org/ ##

jgh at wizmail

Jul 9, 2013, 7:17 AM

Post #2 of 4 (90 views)
Permalink

Re: Vagrant&Chef and/or Exim Build Farm [In reply to]

On 07/09/2013 03:03 PM, Todd Lyons wrote:
> 2. My thoughts were a little less extravagant. I was a member of the
> Samba build farm for a few years. Participants' machines synced to
> the master samba buildfarm server (multiple repos), built multiple
> configs when new commits were found, ran samba torture (essentially
> make test), and results were submitted back up to a master samba
> buildfarm server, where it was parsed and made available for
> display/analysis. I'd like to consider doing the same for Exim.
> There are a couple of public projects that use a distributed build
> farm, and we could simply model off of their systems.
>
> The two ideas aren't mutually exclusive, but they do have common
> portions which make it somewhat duplicative to do both. Comments,
> feedback?

The Samba build farm sounds somewhat like the Postgresql one:

https://wiki.postgresql.org/wiki/PostgreSQL_Buildfarm_Howto

--
Jeremy

--
## List details at https://lists.exim.org/mailman/listinfo/exim-dev Exim details at http://www.exim.org/ ##

tlyons at ivenue

Jul 9, 2013, 9:54 AM

Post #3 of 4 (84 views)
Permalink

Re: Vagrant&Chef and/or Exim Build Farm [In reply to]

On Tue, Jul 9, 2013 at 7:03 AM, Todd Lyons <tlyons [at] ivenue> wrote:
>
>> Todd is working on some Vagrant stuff that might change that and make it easier to benchmark?
> Phil and I had spoken offline a while back about various ways of
> making development better/easier and increasing confidence in commits.
> We have ideas that complement each other in an effort to achieve
> this.
>
> 1. Phil's idea is to use Vagrant (multi OS images) with Chef for
> configuration control. Then you can get build results by building the

Well this approach is kind of dead for a bit since the CPU in my Dell
2850 which I thought had VT extensions, does not in fact have them.
As a result, I can't boot 64 bit images in virtualbox, only 32 bit
images. Until I can do some kind of hardware upgrade to get CPU
virtual support so I can do 64 bit images, this approach is a
non-starter.

> 2. My thoughts were a little less extravagant. I was a member of the
> Samba build farm for a few years. Participants' machines synced to
> the master samba buildfarm server (multiple repos), built multiple
> configs when new commits were found, ran samba torture (essentially
> make test), and results were submitted back up to a master samba
> buildfarm server, where it was parsed and made available for
> display/analysis. I'd like to consider doing the same for Exim.
> There are a couple of public projects that use a distributed build
> farm, and we could simply model off of their systems.

I'm going to work on this approach for now. Short of inheriting a
Dell 1950 or R-210, I just won't be able to upgrade my webserver
again.

...Todd
--
The total budget at all receivers for solving senders' problems is $0.
If you want them to accept your mail and manage it the way you want,
send it the way the spec says to. --John Levine

--
## List details at https://lists.exim.org/mailman/listinfo/exim-dev Exim details at http://www.exim.org/ ##

tlyons at ivenue

Jul 11, 2013, 12:00 PM

Post #4 of 4 (71 views)
Permalink

Re: Vagrant&Chef and/or Exim Build Farm [In reply to]

On Tue, Jul 9, 2013 at 9:54 AM, Todd Lyons <tlyons [at] ivenue> wrote:
>>
>>> Todd is working on some Vagrant stuff that might change that and make it easier to benchmark?
>> 1. Phil's idea is to use Vagrant (multi OS images) with Chef for
>> configuration control. Then you can get build results by building the
> Well this approach is kind of dead for a bit since the CPU in my Dell
> 2850 which I thought had VT extensions, does not in fact have them.
> As a result, I can't boot 64 bit images in virtualbox, only 32 bit
> images. Until I can do some kind of hardware upgrade to get CPU
> virtual support so I can do 64 bit images, this approach is a
> non-starter.

The hardware gods smiled upon me and I now have a machine with VT
extensions, so 64 bit images now boot up. I migrated my webserver to
the newer machine (lower power, unfortunately also lower disk space).

So far I have setup a few basic Vagrant images:

[farm [at] mai ~]$ ls vagrant/
CentOS_5.9_64_Chef CentOS_6.4_64_Chef Debian_Squeeze_64_Chef
CentOS_6.4_32_Chef Debian_Squeeze_32_Chef FreeBSD_9.1_64_zfs

I also have several bare repos that I plan to process:

[farm [at] mai ~]$ ls exim/
exim.git exim-jgh.git exim-pdp.git exim-tlyons.git

There's a lot to do yet, but I'm taking baby steps. The next step in
this sequence is to get Chef recipes built for each of the OSes to
provide consistent build platforms.

I'm not firmly settled on how to structure the build management, but
my general approach will be to script something so that I can call it
from cron:
1. Loop through several repo/branch combinations and do 'git remote
update' and parse results of 'git status -uno'.
2. If any are updated, do a pull and flag the branch/repo name for
rebuilding, else exit.
3. Loop through the machine list, start the machine, do a git pull
against my local cloned repos, do the build (and make test?) for each
branch/repo.
4. Submit each set of results somewhere (TBD, probably something
version controlled so can look through history).

...Todd
--
The total budget at all receivers for solving senders' problems is $0.
If you want them to accept your mail and manage it the way you want,
send it the way the spec says to. --John Levine

--
## List details at https://lists.exim.org/mailman/listinfo/exim-dev Exim details at http://www.exim.org/ ##

Index | Next | Previous | View Threaded

Interested in having your list archived? Contact Gossamer Threads