Login | Register For Free | Help
Search for: (Advanced)

Mailing List Archive: Zope: Coders

End-of-line translation problem

 

 

Zope coders RSS feed   Index | Next | Previous | View Threaded


jim at zope

Apr 30, 2004, 7:24 AM

Post #1 of 7 (790 views)
Permalink
End-of-line translation problem

(Sorry for the cross posting, but I think that all of the lists have an important
stake in this issue.)

In the text below, I respond to a report from Tim on the fact that, unlike CVS,
subversion doesn't do any end-of-line translation for text files. If someone
creates a file on a Unix-like system, it will have Unix line endings. If someone
creates a file on Windows, it will have windows line endings. Depending on tools
used, we could end up with files having mixed line endings. This has happened to
us in the past when people using SMB used windows editors to edit files checked
out and checked in using Linux CVS clients (using SMB). When this has happened,
people found it really annoying.

Subversion has a property that can be set on a file to cause end-of-line translation
to take place. I have argued that the setting of this property should be automated
and that the standard ways to do so with subversion are inadequate.

I started a thread on the subversion users mailing list explaining my
position and describing a work around that Tim suggested to me:

http://subversion.tigris.org/servlets/BrowseList?list=users&by=thread&from=184061

One of the subversion developers responded suggesting:

- File adds are rare

- Ad hoc mechanisms for setting the subversion eol-style property have worked
for the subversion project itself. (Text files that I have looked at in the
subversion repository do, indeed, have this property set.)

It could be argued that we should try subversion's ad hoc approach to
prove that, in fact, it doesn't work for us.

For myself, I'm inclined to do what Tim suggested,
and tell subversion to automatically set the eol-style for
all files to native. This will fail loudly when binary files
are checked in and I think that I can deal with that inconvenience. Note that
this is a big improvement over CVS as generally, binary files will automatically
*not* be treated as text files. (The danger being that the binary detection algorithm
will occasionally fail.)

Thoughts?

If we think that it's worth *trying* the subversion developers ad hoc approach, then I
think we could move forward with the transition sooner, perhaps the week after next.
(I still need to arrange for setting the property during conversion, but I think
I can manage that.)

Jim




Jim Fulton wrote:
> Tim Peters wrote:
>
>> [Jim Fulton]
>> ...
>>
>>> You will be able to do read-only anonymous checkouts like so:
>>>
>>> svn co svn://svn.zope.org/repos/main/<project>/trunk
>>>
>>> For example:
>>>
>>> svn co svn://svn.zope.org/repos/main/ZConfig/trunk
>>
>>
>>
>> FYI, I tried that on Windows (XP), and it worked fine.
>>
>> One glitch, which may be all over the place: some of the "text files"
>> got
>> checked out with Windows line ends, but most did not. For example, 14 of
>> the 19 *.txt files in ZConfig ended up with Windows line ends, but
>> none of
>> the 37 *.py files did.
>>
>> Ack, no, none of the checked-out .txt files did either. The .txt
>> files that
>> had Windows line ends were all created by svn for its own purposes
>> (README.txt files in .svn directories).
>>
>> I'm not sure what to do about this. Best I can tell from the docs so
>> far,
>> svn wants a
>>
>> svn:eol-style
>>
>> property added to every line-oriented file, with value
>>
>> native
>>
>> in order to get platform-sane line-end conversions. The doc's
>> explanation
>> of the effect of that matches my understanding of what CVS does for all
>> non-binary files, which is usually exactly right.
>>
>> I noticed that Fredrik Lundh complained about something similar here:
>>
>> http://effbot.org/zone/subversion.htm
>> ...
>> Properties are nice, but having to use three different commands
>> to check in a text file from Windows is pretty annoying.
>>
>> Looks like svn *expected* us to do this by setting enable-auto-props
>> during
>> the intial imports, with a bunch of [auto-props] settings in a config
>> file;
>> like
>>
>> """
>> [auto-props]
>> *.c = svn:eol-style=native
>> *.cpp = svn:eol-style=native
>> *.h = svn:eol-style=native
>> *.py = svn:eol-style=native
>> *.dsp = svn:eol-style=CRLF
>> *.dsw = svn:eol-style=CRLF
>> *.sh = svn:eol-style=native;svn:executable
>> *.txt = svn:eol-style=native
>> *.png = svn:mime-type=image/png
>> *.jpg = svn:mime-type=image/jpeg
>> """
>
>
> I found this to be so unbelievable, that I had to resoearch it myself.
> After looking this up in the book and expressing my amazement on the #svn
> channel (and recieving confirmation from svn developers there), I have to
> admit that you are right. I know better than to doubt you, but this is
> just so
> unbelievable, I couldn't help it.
>
>
>> I think we'll have to develop a standard set of config file settings like
>> that for committers to add to their personal svn configs --
>
>
> I don't think that this is practical. I think it will be very hard to
> communicate
> this to everyone. Plus, every time someone comes up with a new dang
> file suffix,
> everyone will have to update their config files.
>
> I think the "real" answer, the answer that the svn (and arch) developers
> believe
> in the heart of hearts is that windows users should be using tools that
> understand,
> well, understand and always produce Unix line endings.
>
> Is it practical to require windows users to use tools that understand
> and produce
> Unix line endings?
>
> > or can that be
>
>> done on the server side?
>
>
> I suppose it could. I think that a post-commit script could inspect new
> files and,
> for any new file that has no mine-type property, or has one with a text
> type,
> set the svn:eol-style proprty to native. It would have to do this in a
> separate
> transaction.
>
> Does anyone want to volunteer to write this script?
>
> We'll also need to fix cvs2svn to do something similar.
>
> Jim
>


--
Jim Fulton mailto:jim [at] zope Python Powered!
CTO (540) 361-1714 http://www.python.org
Zope Corporation http://www.zope.com http://www.zope.org

_______________________________________________
Zope-Coders mailing list
Zope-Coders [at] zope
http://mail.zope.org/mailman/listinfo/zope-coders


jim at zope

Apr 30, 2004, 8:22 AM

Post #2 of 7 (761 views)
Permalink
Re: End-of-line translation problem [In reply to]

Philipp von Weitershausen wrote:
> Jim,
>

...

> What's the ad-hoc approach? I could imagine a hook script...

Leave it up to users (developers) to set the propprty on files they add.

>> For myself, I'm inclined to do what Tim suggested, and tell
>> subversion to automatically set the eol-style for all files to
>> native. This will fail loudly when binary files are checked in and I
>> think that I can deal with that inconvenience.
>
>
> That does not make sense. You'd only want svn to set that property on
> text files anyway. So I don't understand what this "inconvenience" is.

This is explained in detail in the thread I gave a link to.

>> Note that this is a big improvement over CVS as generally, binary
>> files will automatically *not* be treated as text files. (The danger
>> being that the binary detection algorithm will occasionally fail.)
>
>
> I think that's very rare. It has never happened to me at least, and I
> have lots of binary stuff like OpenOffice files in my repository.

Yup

>>>> [auto-props]
>>>> *.c = svn:eol-style=native
>>>> *.cpp = svn:eol-style=native
>>>> *.h = svn:eol-style=native
>>>> *.py = svn:eol-style=native
>>>> *.dsp = svn:eol-style=CRLF
>>>> *.dsw = svn:eol-style=CRLF
>>>> *.sh = svn:eol-style=native;svn:executable
>>>> *.txt = svn:eol-style=native
>>>> *.png = svn:mime-type=image/png
>>>> *.jpg = svn:mime-type=image/jpeg
>
>
> I wouldn't mind such a configuration. I would really make it the
> responsbility of the committer. He/she may choose to use such settings
> or simply set svn:eol-style after every 'svn add' manually.

That's the position the subversion developers take. Here's mine:

[miscellany]
enable-auto-props = yes

[auto-props]
* = svn:eol-style=native

This sets all files not detected to be binary to native eol style.
If a file is detected as binary, I get an error, but it is still added,
however, if this was a multi-file add, only the files up to an including
the first binary file will be added. I have to repeat the add command
for the remaining files.

Jim

--
Jim Fulton mailto:jim [at] zope Python Powered!
CTO (540) 361-1714 http://www.python.org
Zope Corporation http://www.zope.com http://www.zope.org


_______________________________________________
Zope-Coders mailing list
Zope-Coders [at] zope
http://mail.zope.org/mailman/listinfo/zope-coders


jim at zope

Apr 30, 2004, 8:33 AM

Post #3 of 7 (761 views)
Permalink
Re: End-of-line translation problem [In reply to]

Jim Fulton wrote:
...
> http://subversion.tigris.org/servlets/BrowseList?list=users&by=thread&from=184061

Ken pointed out that the links on this page don't work. :( Sigh

Here are links that do work:

http://subversion.tigris.org/servlets/ReadMsg?list=users&msgNo=10439
http://subversion.tigris.org/servlets/ReadMsg?list=users&msgNo=10444
http://subversion.tigris.org/servlets/ReadMsg?list=users&msgNo=10445
http://subversion.tigris.org/servlets/ReadMsg?list=users&msgNo=10449

Jim

--
Jim Fulton mailto:jim [at] zope Python Powered!
CTO (540) 361-1714 http://www.python.org
Zope Corporation http://www.zope.com http://www.zope.org

_______________________________________________
Zope-Coders mailing list
Zope-Coders [at] zope
http://mail.zope.org/mailman/listinfo/zope-coders


barry at zope

Apr 30, 2004, 8:41 AM

Post #4 of 7 (759 views)
Permalink
Re: [ZODB-Dev] Re: End-of-line translation problem [In reply to]

On Fri, 2004-04-30 at 11:33, Jim Fulton wrote:
> Jim Fulton wrote:
> ...
> > http://subversion.tigris.org/servlets/BrowseList?list=users&by=thread&from=184061
>
> Ken pointed out that the links on this page don't work. :( Sigh

Neither do many of the "next in thread" and "previous in thread" links.

pipermail-doesn't-look-so-bad-now-ly y'rs,
-Barry



_______________________________________________
Zope-Coders mailing list
Zope-Coders [at] zope
http://mail.zope.org/mailman/listinfo/zope-coders


regebro at nuxeo

Apr 30, 2004, 9:25 AM

Post #5 of 7 (745 views)
Permalink
Re: End-of-line translation problem [In reply to]

Jim Fulton wrote:
> Thoughts?

Only that the Subversion people are wrong. Ad-hoc is not good enough. It
must be able to be configurable on the server.



_______________________________________________
Zope-Coders mailing list
Zope-Coders [at] zope
http://mail.zope.org/mailman/listinfo/zope-coders


fred at zope

Apr 30, 2004, 2:28 PM

Post #6 of 7 (759 views)
Permalink
Re: [Zope3-dev] Re: End-of-line translation problem [In reply to]

On Friday 30 April 2004 12:25 pm, Lennart Regebro wrote:
> Only that the Subversion people are wrong. Ad-hoc is not good enough. It
> must be able to be configurable on the server.

Another possible approach would be to write a script to use for adds instead
of the default client; it could set all the right attributes as part of the
add operation (do the "svn add" and then set the properties). The script
could have our initiali policy hardcoded, and could import the policy
configuration from the server (caching it and all that of course).

This would mean adds have to be done with the custom client though, which
could be a real problem for people who'd rather use TortoiseSVN.


-Fred

--
Fred L. Drake, Jr. <fred at zope.com>
PythonLabs at Zope Corporation


_______________________________________________
Zope-Coders mailing list
Zope-Coders [at] zope
http://mail.zope.org/mailman/listinfo/zope-coders


jim at zope

May 1, 2004, 7:08 AM

Post #7 of 7 (757 views)
Permalink
Re: End-of-line translation problem [In reply to]

Lennart Regebro wrote:
> Jim Fulton wrote:
>
>> Thoughts?
>
>
> Only that the Subversion people are wrong. Ad-hoc is not good enough. It
> must be able to be configurable on the server.

You may be right. This was certainly my initial reaction. I think, though,
that we should at least try handling this the normal way and see how it works.
If it doesn't work, we can try some server-side hacks.

Jim

--
Jim Fulton mailto:jim [at] zope Python Powered!
CTO (540) 361-1714 http://www.python.org
Zope Corporation http://www.zope.com http://www.zope.org

_______________________________________________
Zope-Coders mailing list
Zope-Coders [at] zope
http://mail.zope.org/mailman/listinfo/zope-coders

Zope coders RSS feed   Index | Next | Previous | View Threaded
 
 


Interested in having your list archived? Contact Gossamer Threads
 
  Web Applications & Managed Hosting Powered by Gossamer Threads Inc.