Mailing List Archive: Python: Dev

Mercurial migration: progress report (PEP 385)

1 2 3 4

View All

Index | Next | Previous | View Threaded

brett at python

Jul 3, 2009, 3:02 PM

Post #26 of 83 (1344 views)
Permalink

Re: Mercurial migration: progress report (PEP 385) [In reply to]

On Fri, Jul 3, 2009 at 15:00, Dirkjan Ochtman <dirkjan [at] ochtman> wrote:

> On Fri, Jul 3, 2009 at 23:41, Brett Cannon<brett [at] python> wrote:
> > If we make it universal I say it should be '2.x' and '3.x'. The whole
> 'py'
> > prefix is redundant.
>
> Right, I was aiming for /python/2.x and /python/3.x as well.
>
> Actually, I currently have /cpython to also make CPython less special
> among it's peers, but that idea was met with some resistance on
> #python-dev.

Don't worry about doing that right now. When the stdlib gets separated out
we can revisit this. But for now just leave it as python.

-Brett

barry at python

Jul 3, 2009, 3:03 PM

Post #27 of 83 (1339 views)
Permalink

Re: Mercurial migration: progress report (PEP 385) [In reply to]

On Jul 3, 2009, at 5:00 PM, Martin v. Löwis wrote:

> I'm -1 on calling it "sys.revision", as this makes it difficult to
> tell what the actual versioning system was, and hence how the
> data should be interpreted. It will already be a problem for 2.6,
> when 2.6.3 will currently have a sys.subversion[2] of 'dd3ebf81af43',
> which will surely crash existing applications.

I can release a 2.6.3 right before the cut-over (well, just about any
time between now and August 1st). Should we just plan now for a 2.6.3
on say July 24th, with a release candidate on July 20th?

> I'm not sure what the motivation for a sys.revision is; it's
> probably similar to the desire of calling the machine code.python.org
> (instead of hg.python.org). It gives the illusion of being agnostic
> of the actual RCS being used. However, this is a complete illusion:
> anybody using it (either code.python.org, or sys.revision), *cannot*
> be agnostic of the specific technology.

Agreed. I originally chose code.python.org because I didn't want to
be biased (maybe I should have been :). +1 for hg.python.org. I'd
prefer to spell out sys.mercurial_revision.

-Barry

Attachments:

PGP.sig (0.81 KB)

martin at v

Jul 3, 2009, 3:09 PM

Post #28 of 83 (1342 views)
Permalink

Re: Mercurial migration: progress report (PEP 385) [In reply to]

> - First of all, I've got the basic conversion down, I've done it a few
> times now, with progressively better results. You can view some
> results at http://hg.python.org/, which has a preliminary cpython
> repository. *** The changeset hashes for that repo will change, so you
> won't be able to commit or pull from it in the future.***

I'd like to see as much infrastructure going "life" well in advance
of the switch over. Ideally, when we do switch, there should be no
surprises (i.e. all problems should be known, and, ideally, be
resolved).

> * Set up hg-ssh infra (should be easy)
> * Set up hooks (should be mostly straightforward)
> * Set up roundup integration (should be made easier by quick revision
> map hgweb extension)
> * Write docs

These are the ones that I was thinking of (and mentioned before, so
they got onto your list :-)

I would drop the roundup integration from the things that need to
be done pre-migration - there currently is no svn integration, so
not having it for hg is not a step backwards.

> * buildbots: both the regular and the community build masters must
> be notified. Fortunately buildbot includes support for hg. I've also
> implemented this for Mercurial itself, so I don't expect problems
> here.

This should also be setup before the switchover, with a few test
buildslaves (say, a Unix one and a Windows one).

> In a DVCS, the committers are not necessarily the same
> people who push, and so we can't check if the committer is a
> contributor. We could use a hook to check if the committer is a
> contributor if we keep a list of registered contributors.

In the first sentence, you say that it can't actually work - so I
think we should drop the test.

> I propose that the revision identifier will be the short version of
> hg's revision hash, for example 'dd3ebf81af43', augmented with '+'
> (instead of 'M') if the working directory from which it was built was
> modified. This mirrors the output of the hg id command, which is
> intended for this kind of usage.

I would like to see this well before the switch also. It could be
a patch (unified diff) stored in the tracker, or it could be an actual
branch that then needs to get merged with all active branches (IIUC).

Regards,
Martin
_______________________________________________
Python-Dev mailing list
Python-Dev [at] python
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: http://mail.python.org/mailman/options/python-dev/list-python-dev%40lists.gossamer-threads.com

martin at v

Jul 3, 2009, 3:15 PM

Post #29 of 83 (1342 views)
Permalink

Re: Mercurial migration: progress report (PEP 385) [In reply to]

Barry Warsaw wrote:
> On Jul 3, 2009, at 5:00 PM, Martin v. Löwis wrote:
>
>> I'm -1 on calling it "sys.revision", as this makes it difficult to
>> tell what the actual versioning system was, and hence how the
>> data should be interpreted. It will already be a problem for 2.6,
>> when 2.6.3 will currently have a sys.subversion[2] of 'dd3ebf81af43',
>> which will surely crash existing applications.
>
> I can release a 2.6.3 right before the cut-over (well, just about any
> time between now and August 1st). Should we just plan now for a 2.6.3
> on say July 24th, with a release candidate on July 20th?

I'm fine with that plan - but the original problem remains. We will
surely release 2.6.4 at some point, and it will have a different version
identification (based on hg rev ids).

So those existing applications (which are probably few) will then crash
for 2.6.4, unless we continue maintaining 2.6 in subversion, or just
arrange to fake sys.subversion somehow (e.g. freezing it on the last
subversion revision - which might still break applications that insist
on accessing the revision mentioned - not sure whether such applications
actually exist).

Regards,
Martin
_______________________________________________
Python-Dev mailing list
Python-Dev [at] python
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: http://mail.python.org/mailman/options/python-dev/list-python-dev%40lists.gossamer-threads.com

barry at python

Jul 3, 2009, 3:37 PM

Post #30 of 83 (1341 views)
Permalink

Re: Mercurial migration: progress report (PEP 385) [In reply to]

On Jul 3, 2009, at 6:15 PM, Martin v. Löwis wrote:

> I'm fine with that plan - but the original problem remains. We will
> surely release 2.6.4 at some point, and it will have a different
> version
> identification (based on hg rev ids).
>
> So those existing applications (which are probably few) will then
> crash
> for 2.6.4, unless we continue maintaining 2.6 in subversion, or just
> arrange to fake sys.subversion somehow (e.g. freezing it on the last
> subversion revision - which might still break applications that insist
> on accessing the revision mentioned - not sure whether such
> applications
> actually exist).

Doesn't Mercurial support an Subversion bridge? Would it be possible
to provide a /read-only/ copy of the hg branches for 2.4 (maybe), 2.5,
2.6, and 3.1? If so, then the release managers would simply have to
cut their releases from the svn copy instead of the hg master. /All/
other work would be done from the hg master. This shouldn't be too
much of a burden since it's done so rarely and would end with the EOL
of each of those branches.

It would mean maintaining the bridge until all currently released
versions are EOL.

If that's not possible or feasible, then given the documented
sys.subversion semantics, I think we should just freeze the tuple at
e.g. ('CPython', 'branches/release26-maint', None).

-Barry

Attachments:

PGP.sig (0.81 KB)

mhammond at skippinet

Jul 3, 2009, 4:30 PM

Post #31 of 83 (1349 views)
Permalink

Re: Mercurial migration: progress report (PEP 385) [In reply to]

On 4/07/2009 12:04 AM, Nick Coghlan wrote:

> However, I expect that would still be painful to work with for Windows
> developers, even if it prevented any line ending problems from making
> their way into the main repository. I believe that is where the
> win32text extensions can help. Looking at the Wiki page for win32text
> [1], I believe it would be a matter of configuring the extension to
> encode and decode all files with the extensions:
>
> *.py
> *.pyw
> *.h
> *.c
> *.in
> *.rst
> *.asdl
>
> That said, I don't see a way to tell win32text to also translate files
> which don't have an extension at all (e.g. NEWS or ACKS), and there
> doesn't seem to be a way to tell it to *skip* files matching certain
> patterns (if there was, we could just tell it to ignore extensions like
> .bat, .sln, .vcproj, .vsprops, .dps, .dsw, .wse, .ico, .bmp and convert
> everything else)

The big problem I have with win32text is that the rules are not
versioned, meaning we will rely on every (windows only?) developer
manually enabling the win32text extension, then manually adding these
rules to an unversioned hgrc file. Should the rules change over time,
we will need to communicate that to every developer. Contrast this
with, say, SVN, where the rule is a versioned property on the file.

Further, if a repo *already* has mixed line endings and the file doesn't
appear in the rules, win32text makes *lots* of noise regarding existing
line endings and to "review your rules before you commit." In effect,
it seems to me that win32text will only work effectively if *all*
developers using a repo are using it, or none are, or other steps have
been put in place to ensure \r\n line endings in the repo are otherwise
impossible. I've completely failed to use win32text on the mozilla hg
repo for exactly this reason.

Cheers,

Mark
_______________________________________________
Python-Dev mailing list
Python-Dev [at] python
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: http://mail.python.org/mailman/options/python-dev/list-python-dev%40lists.gossamer-threads.com

mhammond at skippinet

Jul 3, 2009, 4:37 PM

Post #32 of 83 (1337 views)
Permalink

Re: Mercurial migration: progress report (PEP 385) [In reply to]

On 3/07/2009 11:43 PM, Dirkjan Ochtman wrote:
> On Fri, Jul 3, 2009 at 15:31, Mark Hammond<mhammond [at] skippinet> wrote:
>> So we must work without effective EOL support? I fear we will end up like
>> the mozilla hg repo with some files in windows line endings and some with
>> linux. While my editing tools are good enough to preserve existing EOL
>> styles, I've found myself accidentally checking in new \r\n terminated files
>> in a repo which otherwise uses \n line endings. IMO, SVN's EOL support was
>> better than no EOL support.
>
> This is why we'll have hooks -- to prevent you from pushing changesets
> with inappropriate, to say the least, and, if you're willing to do a
> little bit of extra work, to prevent you from committing them.
>
>> This is exactly why I was asking for your advice - I can't work out how to
>> work effectively with win32text as it stands myself, so remain stuck
>> accidently checking in files with inappropriate line endings and stuck
>> working out how to move pywin32's CVS repo with abandoning the very
>> primitive EOL safety it offers...
>
> As long as the difference between \r\n- and \n-based files is clear
> and can be reasoned about, I don't see why having some of both (I'm
> assuming an overwhelming majority will have one, and only a few the
> other) is a big problem. But feel free to enlighten me!

I'm surprised it isn't obvious. Think about it this way: CVS had basic
EOL support and SVN gave slightly better support, and we leveraged this
support in both those tools. The end result is a very clean repo with
consistent line endings and a distinct lack of developer friction
between the platforms.

Your position seems to be that we *never* needed EOL conversion,
especially in SVN which offered decent hooks, and that having a source
repo with a mismatch and unpredictable line endings isn't really a
problem at all.

Even just at face value, it seems clear we have taken full advantage of
that feature for around 1.5 decades, so to suggest we never actually
needed it, or at least don't need it now for some reason, seems slightly
disingenuous...

Cheers,

Mark
_______________________________________________
Python-Dev mailing list
Python-Dev [at] python
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: http://mail.python.org/mailman/options/python-dev/list-python-dev%40lists.gossamer-threads.com

martin at v

Jul 3, 2009, 4:47 PM

Post #33 of 83 (1349 views)
Permalink

Re: Mercurial migration: progress report (PEP 385) [In reply to]

> Doesn't Mercurial support an Subversion bridge? Would it be possible to
> provide a /read-only/ copy of the hg branches for 2.4 (maybe), 2.5, 2.6,
> and 3.1? If so, then the release managers would simply have to cut
> their releases from the svn copy instead of the hg master. /All/ other
> work would be done from the hg master. This shouldn't be too much of a
> burden since it's done so rarely and would end with the EOL of each of
> those branches.
>
> It would mean maintaining the bridge until all currently released
> versions are EOL.

That sounds very complicated (but Dirkjan should correct me if I'm
wrong).

I have already mentally prepared to port welease to hg, to create
the missing 2.4 and 2.5 releases.

> If that's not possible or feasible, then given the documented
> sys.subversion semantics, I think we should just freeze the tuple at
> e.g. ('CPython', 'branches/release26-maint', None).

That would work for me (I had to re-read the documentation to see
that None is a valid, documented value for version, and allowed
if "the tree was exported").

Interesting to notice that sys.subversion didn't even last the
life time of one feature release :-)

Regards,
Martin
_______________________________________________
Python-Dev mailing list
Python-Dev [at] python
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: http://mail.python.org/mailman/options/python-dev/list-python-dev%40lists.gossamer-threads.com

mhammond at skippinet

Jul 3, 2009, 6:20 PM

Post #34 of 83 (1341 views)
Permalink

Re: Mercurial migration: progress report (PEP 385) [In reply to]

On 3/07/2009 11:43 PM, Dirkjan Ochtman wrote:

> As long as the difference between \r\n- and \n-based files is clear
> and can be reasoned about, I don't see why having some of both (I'm
> assuming an overwhelming majority will have one, and only a few the
> other) is a big problem. But feel free to enlighten me!

In that case, and given that I expect more Windows users to be clueless
about EOL issues than users of other operating systems, I propose we
check in all files initially with Windows line endings <wink>

Mark
_______________________________________________
Python-Dev mailing list
Python-Dev [at] python
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: http://mail.python.org/mailman/options/python-dev/list-python-dev%40lists.gossamer-threads.com

ncoghlan at gmail

Jul 3, 2009, 6:20 PM

Post #35 of 83 (1345 views)
Permalink

Re: Mercurial migration: progress report (PEP 385) [In reply to]

Mark Hammond wrote:
> On 4/07/2009 12:04 AM, Nick Coghlan wrote:
> The big problem I have with win32text is that the rules are not
> versioned, meaning we will rely on every (windows only?) developer
> manually enabling the win32text extension, then manually adding these
> rules to an unversioned hgrc file. Should the rules change over time,
> we will need to communicate that to every developer. Contrast this
> with, say, SVN, where the rule is a versioned property on the file.
>
> Further, if a repo *already* has mixed line endings and the file doesn't
> appear in the rules, win32text makes *lots* of noise regarding existing
> line endings and to "review your rules before you commit." In effect,
> it seems to me that win32text will only work effectively if *all*
> developers using a repo are using it, or none are, or other steps have
> been put in place to ensure \r\n line endings in the repo are otherwise
> impossible. I've completely failed to use win32text on the mozilla hg
> repo for exactly this reason.

I didn't realise Mercurial didn't have a way for a repository to provide
versioned extension settings, but in this case I think using our own
server side hook can deal with the problem (either adding it to the
existing whitespace hook that checks for tab/space sanity in Python
files, or else as a separate hook that just checks line endings).

Such a hook should be able to base the determination of which files it
checks for \r\n line endings on a normal versioned file in the
repository. For everything that *isn't* mentioned in that file, the hook
would check that it had \n line endings.

I believe that would address both of your concerns:
1. A server side hook means that the rules are enforced on everyone
regardless of whether they use win32text or not (e.g no Linux developers
corrupting a .dsp file by opening it and saving it with \n line endings)
2. The server side hooks rules will be versioned, allowing people to
copy them to their hgrc file so that win32text will correctly handle the
line endings for them.

Also, on another read of the win32text instructions, I believe the
configuration actually *does* support negative filters, it just doesn't
explain them very well (it only mentions them in the context of
overriding default settings in the config file for a specific
repository). I think a .hgrc file along the following lines might do
what we want (I'm not developing on Windows though, so I haven't
actually tested it):

=============
[extensions]
hgext.win32text=

[encode]
# Encode files that don't contain NUL characters.
** = cleverencode:
# But leave Windows specific and known binary files alone
**.bat = !
**.sln = !
**.vcproj = !
**.vsprops = !
**.dps = !
**.dsw = !
**.wse = !
**.ico = !
**.bmp = !

[decode]
# Decode files that don't contain NUL characters.
** = cleverdecode:
# But leave Windows specific and known binary files alone
**.bat = !
**.sln = !
**.vcproj = !
**.vsprops = !
**.dps = !
**.dsw = !
**.wse = !
**.ico = !
**.bmp = !
=============

Assuming we can find settings that do what we want then a sample HGRC
file could easily be provided in the Misc directory alongside things
like python-mode.el.

An alternative to the above would be to store *everything* (even the
Windows specific files) in the repository with \n line endings (still
enforced by a server side hook), and then *require* that Windows users
run win32text with settings like the following:

=============
[extensions]
hgext.win32text=

[encode]
# Encode files that don't contain NUL characters.
** = cleverencode:
# But leave known binary files alone
**.ico = !
**.bmp = !

[decode]
# Decode files that don't contain NUL characters.
** = cleverdecode:
# But leave known binary files alone
**.ico = !
**.bmp = !
=============

Cheers,
Nick.

--
Nick Coghlan | ncoghlan [at] gmail | Brisbane, Australia
---------------------------------------------------------------
_______________________________________________
Python-Dev mailing list
Python-Dev [at] python
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: http://mail.python.org/mailman/options/python-dev/list-python-dev%40lists.gossamer-threads.com

mhammond at skippinet

Jul 3, 2009, 6:33 PM

Post #36 of 83 (1337 views)
Permalink

Re: Mercurial migration: progress report (PEP 385) [In reply to]

On 4/07/2009 11:20 AM, Nick Coghlan wrote:
>
> I didn't realise Mercurial didn't have a way for a repository to provide
> versioned extension settings, but in this case I think using our own
> server side hook can deal with the problem (either adding it to the
> existing whitespace hook that checks for tab/space sanity in Python
> files, or else as a separate hook that just checks line endings).
>
> Such a hook should be able to base the determination of which files it
> checks for \r\n line endings on a normal versioned file in the
> repository. For everything that *isn't* mentioned in that file, the hook
> would check that it had \n line endings.
>
> I believe that would address both of your concerns:
> 1. A server side hook means that the rules are enforced on everyone
> regardless of whether they use win32text or not (e.g no Linux developers
> corrupting a .dsp file by opening it and saving it with \n line endings)
> 2. The server side hooks rules will be versioned, allowing people to
> copy them to their hgrc file so that win32text will correctly handle the
> line endings for them.

That sounds like it should work - presumably mention of this should go
into the PEP.

(While that is great for Python, it is a shame I probably can't use the
same strategy for other hg repos I care about as I don't have access to
server side hooks - but that if off-topic for this list.)

> Also, on another read of the win32text instructions, I believe the
> configuration actually *does* support negative filters, it just doesn't
> explain them very well (it only mentions them in the context of
> overriding default settings in the config file for a specific
> repository). I think a .hgrc file along the following lines might do
> what we want (I'm not developing on Windows though, so I haven't
> actually tested it):

This *appears* to be correct at first glance, but in practice it doesn't
interact well with wildcards specifications - particularly
'**=cleverencode'. I started a thread on the ML and submitted a patch a
few months ago to fix this and it was accepted, but sadly it seemed to
have dropped off the queue somewhere. The patch now conflicts, and I've
promised to resubmit it when I find time. But even with that in place
it doesn't address the more general problem that when only *some*
developers use the extension - mixed mode files are still very possible,
at which point win32text starts reporting enough spurious conflicts to
become unusable for me. Eg, doing a clean checkout of mozilla with
win32text enabled results in a working tree with hundreds of files
reporting every line in the file has changed.

It is worth noting that win32text also works on Linux, and that in
theory, win32text should be able to allow Linux users to work seamlessly
with a \r\n repo, in the same way it promises to allow Windows users to
work with \n repos. But in practice, it is simply never used by Linux
developers, even in the hg core dev team. Patches or comments regarding
win32text are always met with "but no one here ever uses it". If they
did, and some of the core hg team tried to experiment with win32text and
mixed line ending repos the problems would, I believe, become
self-evident...

Cheers,

Mark
_______________________________________________
Python-Dev mailing list
Python-Dev [at] python
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: http://mail.python.org/mailman/options/python-dev/list-python-dev%40lists.gossamer-threads.com

ncoghlan at gmail

Jul 3, 2009, 6:33 PM

Post #37 of 83 (1340 views)
Permalink

Re: Mercurial migration: progress report (PEP 385) [In reply to]

Brett Cannon wrote:
> On Fri, Jul 3, 2009 at 15:00, Dirkjan Ochtman <dirkjan [at] ochtman
> <mailto:dirkjan [at] ochtman>> wrote:
> Actually, I currently have /cpython to also make CPython less special
> among it's peers, but that idea was met with some resistance on
> #python-dev.
>
> Don't worry about doing that right now. When the stdlib gets separated
> out we can revisit this. But for now just leave it as python.

Yes, if we ever do that split we can leave the common code (stdlib, test
suite) under /python and move the CPython specific stuff to /cpython.

+1 on 2.x and 3.x for the main development branches (although that does
lead me to wonder if we should just make the maintenance branches 2.6,
3.1, etc).

Cheers,
Nick.

--
Nick Coghlan | ncoghlan [at] gmail | Brisbane, Australia
---------------------------------------------------------------
_______________________________________________
Python-Dev mailing list
Python-Dev [at] python
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: http://mail.python.org/mailman/options/python-dev/list-python-dev%40lists.gossamer-threads.com

ncoghlan at gmail

Jul 3, 2009, 6:37 PM

Post #38 of 83 (1342 views)
Permalink

Re: Mercurial migration: progress report (PEP 385) [In reply to]

Martin v. Löwis wrote:
> I would drop the roundup integration from the things that need to
> be done pre-migration - there currently is no svn integration, so
> not having it for hg is not a step backwards.

That's not quite true - the tracker turns text like "r64537" into a link
to the appropriate changeset on svn.python.org. It would be unfortunate
to lose that easy ability to indicate when a bug was fixed (or introduced).

This is more something on the roundup side though: recognising the
presence of a hg id in a comment and turning it into a link to
hg.python.org.

Cheers,
Nick.

--
Nick Coghlan | ncoghlan [at] gmail | Brisbane, Australia
---------------------------------------------------------------
_______________________________________________
Python-Dev mailing list
Python-Dev [at] python
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: http://mail.python.org/mailman/options/python-dev/list-python-dev%40lists.gossamer-threads.com

ncoghlan at gmail

Jul 3, 2009, 7:30 PM

Post #39 of 83 (1352 views)
Permalink

Re: Mercurial migration: progress report (PEP 385) [In reply to]

Mark Hammond wrote:
> This *appears* to be correct at first glance, but in practice it doesn't
> interact well with wildcards specifications - particularly
> '**=cleverencode'. I started a thread on the ML and submitted a patch a
> few months ago to fix this and it was accepted, but sadly it seemed to
> have dropped off the queue somewhere. The patch now conflicts, and I've
> promised to resubmit it when I find time. But even with that in place
> it doesn't address the more general problem that when only *some*
> developers use the extension - mixed mode files are still very possible,
> at which point win32text starts reporting enough spurious conflicts to
> become unusable for me. Eg, doing a clean checkout of mozilla with
> win32text enabled results in a working tree with hundreds of files
> reporting every line in the file has changed.

Ah, the scope of the issues begin to become clear...

While a server side hook should be able to deal with the mixed mode file
problem, I'm not sure what can be done about the problems with properly
configuring win32text.

If wildcard specifications don't interact properly with filtered
negative specifications then that would appear to rule out that
approach. The presence of text files without extensions like NEWS and
ACKS in the Python repository appears to rule out the use of positive
filters to select only the files that are stored in the repository with
\n line endings.

I spent some more time exploring the approach recommended on the
win32text documentation page as a possible way of handling the situation:
1. Store all text files (even Windows specific ones) in \n format in the
repository
2. Apply the win32text precommit hook to forbid the introduction of \r\n
line endings
3. Use the recommended settings from the win32text documentation:
===============================
[extensions]
hgext.win32text=

[encode]
# Encode files that don't contain NUL characters.
** = cleverencode:

[decode]
# Decode files that don't contain NUL characters.
** = cleverdecode:
===============================

This would be the equivalent of setting "svn:eol-style native" on every
file in the repository.

However, in running Martin's query (svn pg -R svn:eol-style .|grep CRLF)
over 2.x and 3.x checkouts, I found that \r\n line endings are currently
enforced for:
- .bat files under Tools/buildbot
- .dsp and .dsw files under PC/VC6
- Lib/email/test/data/msg_26.txt

I believe the rationale for the first two is to allow a source tarball
to be prepared on Linux but still be usable on Windows (e.g. see [1]).

I'm not clear on the rationale for the explicit CRLF line ending on the
email test message, but I would guess it is to ensure that the email
module can handle CRLF line endings correctly regardless of platform.

Only VC6 files appear on the list because later versions of Visual
Studio actually tolerate \n line endings in their project files.

Mercurial's heuristic handling of text vs binary and expected line
endings fails completely for these use cases.

I think there needs to be a solid answer in place for these use cases
before the actual migration to Mercurial takes place. A hand-waving "use
win32text" isn't enough - it needs to be "use win32text with these exact
settings" (with server side hook support to enforce the rules).

And since Mercurial doesn't even allow us to say "this is a binary file"
the way CVS used to I'm currently not seeing any way for that to happen
except for win32text to be updated to correctly handle wild cards in
combination with negative filters.

Cheers,
Nick.

[1] http://mail.python.org/pipermail/python-dev/2006-March/062225.html

--
Nick Coghlan | ncoghlan [at] gmail | Brisbane, Australia
---------------------------------------------------------------
_______________________________________________
Python-Dev mailing list
Python-Dev [at] python
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: http://mail.python.org/mailman/options/python-dev/list-python-dev%40lists.gossamer-threads.com

stephen at xemacs

Jul 3, 2009, 8:20 PM

Post #40 of 83 (1339 views)
Permalink

Re: Mercurial migration: progress report (PEP 385) [In reply to]

Dirkjan Ochtman writes:
> On Fri, Jul 3, 2009 at 15:29, Stephen J. Turnbull<stephen [at] xemacs> wrote:
> > I'll have to try them again now that 1.3 is out, but I found Mercurial
> > named branches fundamentally unsuited to release management.
>
> Can you explain why, please? It's not clear from what you say
> below.

Well, the main problem I had was one that you say has improved: the
various issues of confusion due to the presence of multiple active
heads in a single repository.

IME, Mercurial strongly encourages a non-branching style. Although I
can't fully explain in concrete terms what makes me feel that way,
it's certainly consistent with your own inclination to advise "subset
branches". Part of it comes from the fact that you can't have a
single revision on two branches. I would really like the node of a
release branch to be on both the branch and the mainline so that the
tag appears in the history of both, but that's not possible.

Another issue (which you say has improved) is the handling of multiple
heads. With unnamed heads, it's just too painful to leave them
hanging around, so you merge them immediately (in fact, all XEmacs
committers currently active use either "pull -u" or the fetch
extension).

With named branches, the additional heads do tend to hang around. I
found that named branches tended to get inadvertantly pushed, and
worse, they'd end up being the tip, which Mercurial treats specially.
In one case a completely private branch inadvertantly got pushed into
my "pristine" clone of the public repo, and from there into the public
repo, where (since it had been the tip in my private workspace) it
ended up as the tip in public. Embarrassing, to say the least.
Fortuntely, the branch was more or less ready to be pushed anyway, but
one of my colleagues ended up working on that branch without realizing
that he wasn't on (his) mainline any more, and wondered why some
previously done work suddenly disappeared. A good time was had by
nobody involved.

I don't know if that has been fixed in hg; the experience was painful
enough that my workflow adjusted immediately.

> > Ditto named branches. The problem is that (unless the internal
> > implementation has changed very recently) a Mercurial revision can be
> > on exactly one named branch (or on the trunk).
>
> That's still true.
>
> > Which defeats the purpose of having named branches, really. (I mean
> > the version control purpose; obviously it still can save disk space.)
>
> Why does it defeat the purpose? What, in your opinion, is the
> purpose?

I use named branches to collect a sequence of revisions as a named
object, for viewing and manipulation, as differentiated from some
other sequence, for *various* values of "some other sequence". The
problem is that suppose you have a branch A off the trunk, and you
then (several revisions down the line of development) branch B off A.
Now A meanders off and runs into "not ready for prime time" problems,
while B just swims along. Problem is, you can't easily find the
history of B relative to the trunk because much of its history (since
forking from the *trunk*) is labeled A.

> > Unless you're really short on space, though, that's not a big deal.
> > What would be more important to me (not that I matter for the purpose
> > of Python, but in XEmacs -- also a Mercurial shop -- I do :-) would be
> > the other way around: pulling an external branch into a named branch.
> > I have a feeling that working with such a repository with others would
> > be a little difficult.
>
> Can you give an example?

No, I haven't tried it. What concerns me is that I suspect that the
branch name will end up as part of the revision's internal identifier,
and that means that if you and I separately create the same named
branch we have to choose exactly the same name or the branches won't
be recognized as the same, resulting in the mother of all spurious
merge conflicts.

> > As others (MvL, I think) have commented, this isn't really relevant to
> > Python which generally has four mainlines going at once. I don't see
> > why the requirements are going to change with the shift to hg, and I
> > see no reason why hg won't handle the existing workflow just fine.
>
> It will handle it, for sure, but I think it would all go easier if we
> could work with stricter subset branches (and leave the effective
> cherrypicking for the occasional problem).

Sure, but what do you propose? That we nuke Python 3.1, 3.2, and 2.6?
They're all pretty divergent from 2.7 by now, as well as from each
other. Not to mention the "security branches" that are still around.
And individual developers are for sure going to do interesting things
in their private workspaces.

I see that George Brandl and Martin van Loewis seem to be accomodating
your viewpoint, but I don't get the impression that either you (as the
hg migration proponent) nor they (as core committers) realize how far
apart your assumptions are. You are talking about the Mercurial
project, which has *one* line of development. There are many such
projects; the ones I'm most familiar with are XEmacs and Scheme48,
which have adopted a subset branch approach different the Mercurial
project's (XEmacs is modeled on Scheme48). Works well enough (when
I'm wearing my release manager hat; it's a little constraining as a
developer used to the insane flexibility of git).

But Python is *not* such a project. The problem is not coordinating
concurrent development of a closeknit group of committers all working
on the same mainline. In Python, there are *four* mainlines with
rather different purposes, and a diverse group of developers, some who
work on only one line of development, others who work on several, and
a few who accept the role of coordinating across them.

_______________________________________________
Python-Dev mailing list
Python-Dev [at] python
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: http://mail.python.org/mailman/options/python-dev/list-python-dev%40lists.gossamer-threads.com

martin at v

Jul 3, 2009, 8:21 PM

Post #41 of 83 (1342 views)
Permalink

Re: Mercurial migration: progress report (PEP 385) [In reply to]

Nick Coghlan wrote:
> Martin v. Löwis wrote:
>> I would drop the roundup integration from the things that need to
>> be done pre-migration - there currently is no svn integration, so
>> not having it for hg is not a step backwards.
>
> That's not quite true - the tracker turns text like "r64537" into a link
> to the appropriate changeset on svn.python.org. It would be unfortunate
> to lose that easy ability to indicate when a bug was fixed (or introduced).
>
> This is more something on the roundup side though: recognising the
> presence of a hg id in a comment and turning it into a link to
> hg.python.org.

Ah, right. That must be done, of course (although I suppose there is
little hope to have the existing references continue to work).

Regards,
Martin
_______________________________________________
Python-Dev mailing list
Python-Dev [at] python
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: http://mail.python.org/mailman/options/python-dev/list-python-dev%40lists.gossamer-threads.com

steve at pearwood

Jul 3, 2009, 8:36 PM

Post #42 of 83 (1331 views)
Permalink

Re: Mercurial migration: progress report (PEP 385) [In reply to]

On Sat, 4 Jul 2009 04:22:57 am Terry Reedy wrote:

> I would very much like the 'k' dropped from the py3 name. It was a
> funny joke when py3 was vaporware, now it is excess baggage which
> only puzzles non-insiders and newcomers.

+1

Some day we'll be using Python3.7 and wondering what the "k" means.
Alpha, beta, .... kappa version?

--
Steven D'Aprano
_______________________________________________
Python-Dev mailing list
Python-Dev [at] python
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: http://mail.python.org/mailman/options/python-dev/list-python-dev%40lists.gossamer-threads.com

skippy.hammond at gmail

Jul 3, 2009, 9:03 PM

Post #43 of 83 (1331 views)
Permalink

Re: Mercurial migration: progress report (PEP 385) [In reply to]

On 4/07/2009 12:30 PM, Nick Coghlan wrote:
...
> I think there needs to be a solid answer in place for these use cases
> before the actual migration to Mercurial takes place. A hand-waving "use
> win32text" isn't enough - it needs to be "use win32text with these exact
> settings" (with server side hook support to enforce the rules).
>
> And since Mercurial doesn't even allow us to say "this is a binary file"
> the way CVS used to I'm currently not seeing any way for that to happen
> except for win32text to be updated to correctly handle wild cards in
> combination with negative filters.

I agree with your conclusion. My ruminating on this over the last few
months leaves me thinking this would involve:

* my older 'accepted but then lost' hg patch to allow an explicit 'none'
rule for a single file to override wildcards.

* win32text be enhanced to use a normal versioned file in the root of
the repo, much like hgingore, where a project can maintain project wide
rules.

* win32text be enhanced such that all python developers, regardless of
platform, are willing to use this extension, even if the majority of
files happen to use their native line ending (sauce for the goose is
sauce for the gander, and all that...)

* commit hooks be implemented to enforce this - but this should not be
necessary if the above was implemented and socially enforced.

Cheers,

Mark
>
> Cheers,
> Nick.
>
> [1] http://mail.python.org/pipermail/python-dev/2006-March/062225.html
>

_______________________________________________
Python-Dev mailing list
Python-Dev [at] python
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: http://mail.python.org/mailman/options/python-dev/list-python-dev%40lists.gossamer-threads.com

stephen at xemacs

Jul 3, 2009, 9:08 PM

Post #44 of 83 (1335 views)
Permalink

Re: Mercurial migration: progress report (PEP 385) [In reply to]

Nick Coghlan writes:

> I'm not clear on the rationale for the explicit CRLF line ending on the
> email test message, but I would guess it is to ensure that the email
> module can handle CRLF line endings correctly regardless of platform.

IIRC, that's just the standard for text email. Lines must be less
than 1000 characters long, should be no more than 78 characters long,
and are terminated by CRLF. As they say, "honored more in the breach
than the observance", but that's what the RFCs say.

_______________________________________________
Python-Dev mailing list
Python-Dev [at] python
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: http://mail.python.org/mailman/options/python-dev/list-python-dev%40lists.gossamer-threads.com

pje at telecommunity

Jul 3, 2009, 9:31 PM

Post #45 of 83 (1332 views)
Permalink

Re: Mercurial migration: progress report (PEP 385) [In reply to]

At 12:20 PM 7/4/2009 +0900, Stephen J. Turnbull wrote:
>IME, Mercurial strongly encourages a non-branching style. Although I
>can't fully explain in concrete terms what makes me feel that way,
>it's certainly consistent with your own inclination to advise "subset
>branches". Part of it comes from the fact that you can't have a
>single revision on two branches. I would really like the node of a
>release branch to be on both the branch and the mainline so that the
>tag appears in the history of both, but that's not possible.

I'm a total newbie to Mercurial - I've basically done little more
than the tutorial, in fact. So somebody correct me if what I'm about
to say is completely stupid.

Wouldn't the simple thing to do in Mercurial, just be to use
different repositories for long-lived branches? I mean, if you're
not merging them that much anyway, what's the point? Isn't it just
as easy to pull from another repository, as from another branch
within that repository?

_______________________________________________
Python-Dev mailing list
Python-Dev [at] python
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: http://mail.python.org/mailman/options/python-dev/list-python-dev%40lists.gossamer-threads.com

ncoghlan at gmail

Jul 3, 2009, 10:13 PM

Post #46 of 83 (1337 views)
Permalink

Re: Mercurial migration: progress report (PEP 385) [In reply to]

Martin v. Löwis wrote:
> Ah, right. That must be done, of course (although I suppose there is
> little hope to have the existing references continue to work).

I'd guess that the only way to keep those functional is to keep
svn.python.org around in read-only mode.

Although I'm not entirely sure about that - would it be possible to
build up a mapping from svn revision numbers to hg repository IDs as the
history information is being migrated? If that wasn't too difficult, it
might be worth doing.

Cheers,
Nick.

--
Nick Coghlan | ncoghlan [at] gmail | Brisbane, Australia
---------------------------------------------------------------
_______________________________________________
Python-Dev mailing list
Python-Dev [at] python
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: http://mail.python.org/mailman/options/python-dev/list-python-dev%40lists.gossamer-threads.com

stephen at xemacs

Jul 3, 2009, 10:50 PM

Post #47 of 83 (1328 views)
Permalink

Re: Mercurial migration: progress report (PEP 385) [In reply to]

P.J. Eby writes:

> Wouldn't the simple thing to do in Mercurial, just be to use
> different repositories for long-lived branches? I mean, if you're
> not merging them that much anyway, what's the point?

I basically agree with that, and so does Dirkjan, I think. I'm not
sure why he brought up the idea of using named branches.

> Isn't it just as easy to pull from another repository, as from
> another branch within that repository?

Not if the other repository is remote and offline, and pulling from a
remote does take more time, as do things like diffs against that
branch. The temptation is to leave the remote repos on the public
host, rather than put in the effort to make sure your local clones are
up to date. If you do, there are minor annoyances like that. So
people who actually work on more than one line of development learn
pretty soon to keep a local clone for each repo they're interested
in. Problem solved, IMO, but some people worry about the extra space
taken up.
_______________________________________________
Python-Dev mailing list
Python-Dev [at] python
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: http://mail.python.org/mailman/options/python-dev/list-python-dev%40lists.gossamer-threads.com

dirkjan at ochtman

Jul 4, 2009, 1:03 AM

Post #48 of 83 (1300 views)
Permalink

Re: Mercurial migration: progress report (PEP 385) [In reply to]

On Fri, Jul 3, 2009 at 20:04, Brett Cannon<brett [at] python> wrote:
> Fine by me as long as people realize that if anything is questionable then
> the switch will not happen. Getting this right takes precedence over any
> deadline. And obviously we will need to do at least one live conversion on
> python.org hardware to make sure everything will work smoothly.

I'm not sure I see the need to do a (live? what does that mean in this
context) on python.org hardware. Why exactly is that better than me
doing it on one of my boxes, as long as all the necessary tools and an
idea of how to do it are publically available (from the pymigr repo,
for example)?

> And will make the idea of splitting out the standard library and tests a
> reasonable thing to do.

In due time, yes.

> While I really like the idea of using named branches for each release so
> that there is a single py3k branch that contains all relevant history for
> every release, I think we should start simple and go with cloned branches.
> That way the workflow does not radically shift from what we do now for svn
> to start. Once the conversion is done and people are comfortable with hg we
> can then discuss moving towards a named branch approach.

I don't think the cloned branches is much simpler than the named
branches approach, in several ways. For example, populating the branch
part of a sys.whatever value is significantly harder. Also, if you
follow a useful tagging approach, doing cloned branches means that
release tags aren't available on trunk/main/default. That seems like a
step backwards.

> Sounds reasonable to me. We can just make a list and send it to
> python-committers to make final decisions of what should stick around.

This list exists and has been referenced in my email a few times.

> I don't use tags so I don't really care, but in the name of easy transition
> I say we don't change the naming scheme (although I have no issue dropping
> obscure tags).

The current proposal is to clean up old tags to agree with the current
naming scheme (and dropping obscure and partial tags).

> Something else that can go out to python-committers before the switch.

This should just be done ASAP, it helps with a smooth conversion process.

> I don't think there is a single project we host -- all two of them -- that
> have not said they want to convert. So I say convert everything and let's
> turn off the svn server by the end of the year.

I say we tackle each one as we go. I say doing a good conversion job
is valuable, and we should take as much time as we need (though not
more). You advocate something similar below for the Python conversion.

> Can we check these scripts into the repository itself? That way there is a
> chance of reuse as hg commands, e.g. ``hg pydev-ci`` as a replacement for
> ``make patchcheck``.

I'm not sure there's an easy way to make them into commands (although
I guess you could make an extension to that effect), but hooks would
be very easy.

> How about hg.python.org for the official branches and we keep
> code.python.org for personal branches of the developers like we have done
> with the bzr experiments?

I think that's just confusing. Most people seem to like hg.python.org,
and it's obvious that hg goes to hg.p.o. Dividing up the namespace
only makes it harder to find things.

> As I have said, we should change our workflow habits after the switch and
> people are comfortable with hg.

Well, not everyone agrees, and although I think it doesn't matter much
for the conversion itself, it may affect the branching strategy
discussion.

(sys.revision discussion elided because some others have already been
bikeshedding on it.)

Cheers,

Dirkjan
_______________________________________________
Python-Dev mailing list
Python-Dev [at] python
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: http://mail.python.org/mailman/options/python-dev/list-python-dev%40lists.gossamer-threads.com

dirkjan at ochtman

Jul 4, 2009, 1:05 AM

Post #49 of 83 (1297 views)
Permalink

Re: Mercurial migration: progress report (PEP 385) [In reply to]

On Sat, Jul 4, 2009 at 07:13, Nick Coghlan<ncoghlan [at] gmail> wrote:
> I'd guess that the only way to keep those functional is to keep
> svn.python.org around in read-only mode.

No, actually: the idea (I think I floated it in the PEP, as well), is
that I can write a simple extension for hgweb that knows the mapping
of SVN rev to hg rev and so can make
hg.python.org/python/2.x/rev/r32542 come out to the changeset that
resulted from converting that revision.

Cheers,

Dirkjan
_______________________________________________
Python-Dev mailing list
Python-Dev [at] python
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: http://mail.python.org/mailman/options/python-dev/list-python-dev%40lists.gossamer-threads.com

dirkjan at ochtman

Jul 4, 2009, 1:12 AM

Post #50 of 83 (1300 views)
Permalink

Re: Mercurial migration: progress report (PEP 385) [In reply to]

On Fri, Jul 3, 2009 at 17:17, Georg Brandl<g.brandl [at] gmx> wrote:
> Do you have a key to the second column in that file? E.g. the difference
> between "strip" and "discard" is not clear to me. "strip partial"?

strip == discard. strip = remove, merged should be obvious, keep-clone
means we'll keep the branch, in a separate clone, and keep-named means
we'll keep the branch as a named branch in one of the other clones.

> Why are there branch names starting with "../"?

Because some people in the history put their branch at
/python/branchname instead of /python/branches/branchname. That part
will get cleaned up.

[Regarding clone#branch syntax:]
> * Does it work with "hg pull" etc. too, afterwards?

Yes.

> * Does it support more than one branch?

You can only reference one branch at a time using the fragment specifier.

> I would just kill all feature branches unless someone indicates it is
> still used. Â There are very few active feature branches.
>
> (I guess in the case a branch gets killed erroneously it could still be
> re-created after the conversion?)

Re-creating it after the conversion could be done, I guess, but I'd
prefer to get it right, the first time. As the branch map says, I'm
mostly removing all the feature branches anyway.

> As long as it can send diffs...

It can.

> That gets very ugly as soon as you start pulling from repos that just
> fix a small typo or so.

Okay, so we do no automatic checking. We can always write a script
that walks the repository and finds all significant contributors.

Cheers,

Dirkjan
_______________________________________________
Python-Dev mailing list
Python-Dev [at] python
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: http://mail.python.org/mailman/options/python-dev/list-python-dev%40lists.gossamer-threads.com

1 2 3 4

View All

Index | Next | Previous | View Threaded

Interested in having your list archived? Contact Gossamer Threads