Login | Register For Free | Help
Search for: (Advanced)

Mailing List Archive: MythTV: Dev

Best way to use SVN version for deployment?

 

 

MythTV dev RSS feed   Index | Next | Previous | View Threaded


dave at frascone

May 25, 2007, 8:31 AM

Post #1 of 11 (1108 views)
Permalink
Best way to use SVN version for deployment?

I typically build separately on my backend & frontend. I just do a snv
update, and then compile everything.

But, now I'm adding a separate backend, and it's time to upgrade svn (why
not?), so I thought I'd pose a question: How would you recommend doing the
svn build? I can think of a couple of possibilities:

1) Have separate build directories on each machine. Update one, make not of
the rev. Then update all others with -r rev, make everywhere and deploy.
2) Use one shared build directory. Update once, and install everywhere.
Not sure how well this would work, since my backends and frontends are a mix
of P4's and xeons.

What would be safest?

-Dave

--
David Frascone

I couldn't care less about apathy.


mtdean at thirdcontact

May 25, 2007, 8:57 AM

Post #2 of 11 (1064 views)
Permalink
Re: Best way to use SVN version for deployment? [In reply to]

On 05/25/2007 11:31 AM, David Frascone wrote:
> I typically build separately on my backend & frontend. I just do a
> snv update, and then compile everything.
>
> But, now I'm adding a separate backend, and it's time to upgrade svn
> (why not?), so I thought I'd pose a question: How would you recommend
> doing the svn build? I can think of a couple of possibilities:
>
> 1) Have separate build directories on each machine. Update one, make
> not of the rev. Then update all others with -r rev, make everywhere
> and deploy.

Better approach. Saves bandwidth. Create some directory which will
contain your source (I used MythTV/src :). Change to that directory.

Checkout a clean copy of source:
svn co http://svn.mythtv.org/svn/trunk/mythtv &&
svn co http://svn.mythtv.org/svn/trunk/mythplugins &&
svn co http://svn.mythtv.org/svn/trunk/myththemes

Then, use the following to update all to the same rev and tar it up:

( echo Last Updated: `date` | tee mythtv-update.log &&
exit $PIPESTATUS ) &&
( svn update mythtv mythplugins myththemes 2>&1 |
tee -a mythtv-update.log &&
exit $PIPESTATUS ) &&
tar cjf ../myth-`date +'%Y%m%d_%H%M%S'`.tar.bz2 *

That puts a file (i.e. myth-20070525_003429.tar.bz2) in the parent of
your source directory. The mythtv-update.log shows you the date/time
you updated (head -n 1), the changes since your last update, and the SVN
rev (tail -n 1)--all nicely bundled in a tarball. If you prefer to show
all the updates, change "tee mythtv-update.log" to "tee -a
mythtv-update.log").

Then, untar that tarball on your other machines. Only one svn up
required and guaranteed that all your systems have the same rev. It
also makes it easy for you to "clean up" your build directory (i.e. I
often add a bunch of files to my working copy while hacking and never
have to track down the extras when I'm done).

When it comes time to update again, "rm -rf *" in your src directory,
then untar the old tarball and re-run the update commands above. On
other machines, "rm -rf *" in your src directory and untar the new
tarball. This ensures you always have a pristine copy of the Myth trunk.

With this approach, you'll never have a merge conflict due to changed
files in your working copy. You'll never be caught off guard by a
"requires a make clean" or "requires a make distclean" commit (as you're
always starting clean). However, because of this, the biggest
recommendation I have for this approach is to use ccache. Otherwise,
you'll spend a lot of time recompiling everything every time.

I just maintain a patchset that I apply to the pristine copy I untar
each time before I build. See, also, quilt.

I'll probably chuck Myth into my subversion server one of these days as
a vendor branch, but I haven't gotten around to that, yet.

> 2) Use one shared build directory. Update once, and install
> everywhere. Not sure how well this would work, since my backends and
> frontends are a mix of P4's and xeons.
>
> What would be safest?

Safest is, by definition, to compile on each machine. I compile once on
my AMD Athlon XP 2400+ backend, then tar up the (compiled) src directory
structure and untar it on my Athlon XP 2000+ backend.
I--obviously--build a separate build on my Athlon X2 4800+ running in
64-bit mode. This works for me because my backends are basically
identical. When you have differences things get exponentially more
complicated.

This may not be the "best way" to use SVN, but it definitely works
well--especially for untrusting OC types like me that don't trust "svn
revert -R ." or even "svn up" depending on changes that have been made
to your wc.

Mike
_______________________________________________
mythtv-dev mailing list
mythtv-dev [at] mythtv
http://mythtv.org/cgi-bin/mailman/listinfo/mythtv-dev


mythtv at tsaiberspace

May 25, 2007, 12:29 PM

Post #3 of 11 (1061 views)
Permalink
Re: Best way to use SVN version for deployment? [In reply to]

On Fri, May 25, 2007 at 11:57:37AM -0400, Michael T. Dean wrote:
> When it comes time to update again, "rm -rf *" in your src directory,
> then untar the old tarball and re-run the update commands above. On
> other machines, "rm -rf *" in your src directory and untar the new
> tarball. This ensures you always have a pristine copy of the Myth trunk.

Instead of always redoing a complete "svn co", why not checkout/update
stuff in a pristine area, and "rm -fr; cp -r" that pristine area to a
new working-directory area? That still gets you your pristine build
area without wasting bandwidth/time with a complete checkout
everytime.

--Rob
_______________________________________________
mythtv-dev mailing list
mythtv-dev [at] mythtv
http://mythtv.org/cgi-bin/mailman/listinfo/mythtv-dev


dave at frascone

May 25, 2007, 12:39 PM

Post #4 of 11 (1063 views)
Permalink
Re: Best way to use SVN version for deployment? [In reply to]

On 5/25/07, Robert Tsai <mythtv [at] tsaiberspace> wrote:
>
> On Fri, May 25, 2007 at 11:57:37AM -0400, Michael T. Dean wrote:
> > When it comes time to update again, "rm -rf *" in your src directory,
> > then untar the old tarball and re-run the update commands above. On
> > other machines, "rm -rf *" in your src directory and untar the new
> > tarball. This ensures you always have a pristine copy of the Myth
> trunk.
>
> Instead of always redoing a complete "svn co", why not checkout/update
> stuff in a pristine area, and "rm -fr; cp -r" that pristine area to a
> new working-directory area? That still gets you your pristine build
> area without wasting bandwidth/time with a complete checkout
> everytime.


I ended up implementing a subset.

On my "svn" repo, I do: svn update; cd ..; tar -cvzf trunk.tgz trunk

Then on my "build" machines, I copy over the tar file, apply any necessary
patches, and rebuild




--
David Frascone

Oxymoron: Safe Sex.


mtdean at thirdcontact

May 25, 2007, 1:09 PM

Post #5 of 11 (1059 views)
Permalink
Re: Best way to use SVN version for deployment? [In reply to]

On 05/25/2007 03:29 PM, Robert Tsai wrote:
> On Fri, May 25, 2007 at 11:57:37AM -0400, Michael T. Dean wrote:
>
>> When it comes time to update again, "rm -rf *" in your src directory,
>> then untar the old tarball and re-run the update commands above. On
>> other machines, "rm -rf *" in your src directory and untar the new
>> tarball. This ensures you always have a pristine copy of the Myth trunk.
>>
> Instead of always redoing a complete "svn co", why not checkout/update
> stuff in a pristine area, and "rm -fr; cp -r" that pristine area to a
> new working-directory area? That still gets you your pristine build
> area without wasting bandwidth/time with a complete checkout
> everytime.

I did a complete checkout once 3 years ago (may have also done one when
we switched to Subversion--I don't remember, now). I've always had a
pristine copy--my tarball of the previous update.

When I said, "re-run the update commands above," I meant exactly
that--the /update/ commands. Not the checkout commands.

Also, I don't trust cp for recursive copies of source directories.
cpio, yes. tar, yes. cp, no.

Mike
_______________________________________________
mythtv-dev mailing list
mythtv-dev [at] mythtv
http://mythtv.org/cgi-bin/mailman/listinfo/mythtv-dev


memoryguy at gmail

May 25, 2007, 1:52 PM

Post #6 of 11 (1065 views)
Permalink
Re: Best way to use SVN version for deployment? [In reply to]

On 25/05/07, Michael T. Dean <mtdean [at] thirdcontact> wrote:
>
> Also, I don't trust cp for recursive copies of source directories.
> cpio, yes. tar, yes. cp, no.
>

Why's that? (getting off topic...)

I don't think I've ever had problems using cp -R to copy source
directories... takes a long time to complete, perhaps, but I don't
think I've ever had any problems...

--
aaron

"Oh oh oh. I'm incoherent with excitement. Please tell me what fascinating
bit of badger-spewtumly inconsequential trivia you will assail me with next."
-- Arthur Dent
_______________________________________________
mythtv-dev mailing list
mythtv-dev [at] mythtv
http://mythtv.org/cgi-bin/mailman/listinfo/mythtv-dev


mtdean at thirdcontact

May 25, 2007, 2:19 PM

Post #7 of 11 (1057 views)
Permalink
Re: Best way to use SVN version for deployment? [In reply to]

On 05/25/2007 04:52 PM, aaron wrote:
> On 25/05/07, Michael T. Dean <mtdean [at] thirdcontact> wrote:
>
>> Also, I don't trust cp for recursive copies of source directories.
>> cpio, yes. tar, yes. cp, no.
> Why's that? (getting off topic...)
>
> I don't think I've ever had problems using cp -R to copy source
> directories... takes a long time to complete, perhaps, but I don't
> think I've ever had any problems...

A lot of reasons--from permissions to symbolic and hard link handling
and portability (some options to work around some of the issues are only
available on some platforms). There are lots of discussions of the
issues out on the web. Search on "tar cp cpio" and then dig through all
the irrelevant garbage to find the interesting parts. :)

Basically, if you're using cp -R or cp -r, there's a good chance that
eventually, it will mess up your working copy--often in such a way that
an update "fails" silently. Then you're getting told to rm -r your
MythWeb directory or your mythplugins or ... and check out a new one
because your update didn't do what it should have. ;)

Of course, when you have files larger than 2GiB, you can't use tar/cpio
(because--except for "broken" versions that ignore the
specification--they cannot have large file support). Therefore, cpio
becomes my favorite. I can use a "find" with appropriate arguments to
prune any >= 2GiB files and "-print0" the rest and pipe it to cpio
-dpmv0. Then I can use the same find without the -prune and -print0 and
instead "-exec" a cp for the large files.

True, this would mess up hard links for large files, but there are few
times when I need to copy directory structures with large files Really,
the only time I can remember ever having done that is when moving my
Myth box from a single combined frontend/backend to 2 dedicated
backends, and I know I'm not using hard links for the Myth recordings,
which are the only large files I have. (Wow, I almost went back onto
the Myth topic. Would be close enough if this were the -users list.)

Mike
_______________________________________________
mythtv-dev mailing list
mythtv-dev [at] mythtv
http://mythtv.org/cgi-bin/mailman/listinfo/mythtv-dev


memoryguy at gmail

May 26, 2007, 6:07 AM

Post #8 of 11 (1038 views)
Permalink
Re: Best way to use SVN version for deployment? [In reply to]

On 25/05/07, Michael T. Dean <mtdean [at] thirdcontact> wrote:
> On 05/25/2007 04:52 PM, aaron wrote:
> >
> > Why's that? (getting off topic...)
> >
>
> A lot of reasons--from permissions to symbolic and hard link handling

<.. snip lots of good stuff ..>

Hm. I guess I've been lucky all these years... or I just haven't
copied source directories as often as I thought I had. :)

Thanks a lot for your explanation. Much appreciated. :)

--
aaron
(making a mental note to learn how to use cpio someday)
_______________________________________________
mythtv-dev mailing list
mythtv-dev [at] mythtv
http://mythtv.org/cgi-bin/mailman/listinfo/mythtv-dev


bjm at lvcm

May 26, 2007, 2:23 PM

Post #9 of 11 (1023 views)
Permalink
Re: Best way to use SVN version for deployment? [In reply to]

aaron wrote:
> On 25/05/07, Michael T. Dean <mtdean [at] thirdcontact> wrote:
>> On 05/25/2007 04:52 PM, aaron wrote:
>>> Why's that? (getting off topic...)
>>>
>> A lot of reasons--from permissions to symbolic and hard link handling
>
> <.. snip lots of good stuff ..>
>
> Hm. I guess I've been lucky all these years... or I just haven't
> copied source directories as often as I thought I had. :)

Your 'luck' will hold out for the foreseeable future. I always
use "cp -rp". There are no hard links or symlinks in the source
tree or as a result of compiling and you shouldn't be deleting
directories and reusing the inode to create a plain file while
the copy is in progress ;-). "cp" has correctly copied files on
UNIX based systems for about forty years as one of the most basic
commands. If cp should ever fail, it is a clear indication that
there is something seriously wrong with the system.

The key for source dirs is to be sure to use "p":

-p same as --preserve=mode,ownership,timestamps

--preserve[=ATTR_LIST]
preserve the specified attributes (default: mode,owner-
ship,timestamps), if possible additional attributes: links, all

for the timestamps to be preserved so that "make" can determine
what does or doesn't need to be recompiled. I normally checkout
a clean copy from svn and compile. Next,

$ mkdir foo
$ cd foo
$ cp -rp ../mythtv/ .
$ cd mythtv
$ patch -p0 < ../../somethingbrilliant.diff
$ make && make install

I have a different tree for each thing I work on and only need
to recompile for the files that change in that diff or files I
edit.

-- bjm


_______________________________________________
mythtv-dev mailing list
mythtv-dev [at] mythtv
http://mythtv.org/cgi-bin/mailman/listinfo/mythtv-dev


mythtv at colin

May 27, 2007, 2:22 PM

Post #10 of 11 (1007 views)
Permalink
Re: Best way to use SVN version for deployment? [In reply to]

Bruce Markey wrote:
> aaron wrote:
>> On 25/05/07, Michael T. Dean <mtdean [at] thirdcontact> wrote:
>>> On 05/25/2007 04:52 PM, aaron wrote:
>>>> Why's that? (getting off topic...)
>>>>
>>> A lot of reasons--from permissions to symbolic and hard link handling
>> <.. snip lots of good stuff ..>
>>
>> Hm. I guess I've been lucky all these years... or I just haven't
>> copied source directories as often as I thought I had. :)
>
> The key for source dirs is to be sure to use "p":
>
> -p same as --preserve=mode,ownership,timestamps
>

I've always used -a to copy larger trees - it's just the mutts nuts for
ensuring one copy is the same as the other.

-a, --archive
same as -dpR


Oh and if you are using tar try passing a "--exclude=.svn" to your
commands as this will reduce your tarball size (or rather it's extracted
size) but at least half.

Col


--

+------------------------+
| Colin Guthrie |
+------------------------+
| myth(at)colin.guthr.ie |
| http://colin.guthr.ie/ |
+------------------------+
_______________________________________________
mythtv-dev mailing list
mythtv-dev [at] mythtv
http://mythtv.org/cgi-bin/mailman/listinfo/mythtv-dev


jam at tigger

May 27, 2007, 6:12 PM

Post #11 of 11 (1003 views)
Permalink
Re: Best way to use SVN version for deployment? [In reply to]

On Monday 28 May 2007 00:00, mythtv-dev-request [at] mythtv wrote:
> > On 25/05/07, Michael T. Dean <mtdean [at] thirdcontact> wrote:
> >> On 05/25/2007 04:52 PM, aaron wrote:
> >>> Why's that?  (getting off topic...)
> >>
> >> A lot of reasons--from permissions to symbolic and hard link handling
> >
> > <.. snip lots of good stuff ..>
> >
> > Hm. I guess I've been lucky all these years... or I just haven't
> > copied source directories as often as I thought I had. :)
>
> Your 'luck' will hold out for the foreseeable future. I always
> use "cp -rp". There are no hard links or symlinks in the source
> tree or as a result of compiling and you shouldn't be deleting
> directories and reusing the inode to create a plain file while
> the copy is in progress ;-). "cp" has correctly copied files on
> UNIX based systems for about forty years as one of the most basic
> commands. If cp should ever fail, it is a clear indication that
> there is something seriously wrong with the system.
>
> The key for source dirs is to be sure to use "p":
>
>   -p     same as --preserve=mode,ownership,timestamps
>
>   --preserve[=ATTR_LIST]
>          preserve   the   specified   attributes   (default:  mode,owner-
>          ship,timestamps), if possible additional attributes: links, all
>
> for the timestamps to be preserved so that "make" can determine
> what does or doesn't need to be recompiled. I normally checkout
> a clean copy from svn and compile. Next,
>
> $ mkdir foo
> $ cd foo
> $ cp -rp ../mythtv/ .
> $ cd mythtv
> $ patch -p0 < ../../somethingbrilliant.diff
> $ make && make install
>
> I have a different tree for each thing I work on and only need
> to recompile for the files that change in that diff or files I
> edit.

I second that. Unix has just worked for 37 years <big grin>
cp -a bla just works
man cp says -a is equivalent to -dpPR

James
_______________________________________________
mythtv-dev mailing list
mythtv-dev [at] mythtv
http://mythtv.org/cgi-bin/mailman/listinfo/mythtv-dev

MythTV dev RSS feed   Index | Next | Previous | View Threaded
 
 


Interested in having your list archived? Contact Gossamer Threads
 
  Web Applications & Managed Hosting Powered by Gossamer Threads Inc.