Login | Register For Free | Help
Search for: (Advanced)

Mailing List Archive: Python: Dev

deprecating .pyo and -O

 

 

Python dev RSS feed   Index | Next | Previous | View Threaded


solipsis at pitrou

Jun 14, 2012, 3:25 AM

Post #1 of 10 (272 views)
Permalink
deprecating .pyo and -O

On Wed, 13 Jun 2012 12:36:55 -0700
Ethan Furman <ethan [at] stoneleaf> wrote:
>
> Currently, the alternative to supporting this behavior is to either:
>
> 1) require the end-user to specify -O (major nuisance)
>
> or
>
> 2) have the distributor rename the .pyo file to .pyc
>
> I think 1 is a non-starter (non-finisher? ;) but I could live with 2 --
> after all, if someone is going to the effort of removing the .py file
> and moving the .pyo file into its place, renaming the .pyo to .pyc is
> trivial.
>
> So the question, then, is: is option 2 better than just supporting .pyo
> files without -O when they are all that is available?

Honestly, I think the best option would be to deprecate .pyo files as
well as the useless -O option. They only cause confusion without
providing any significant benefits.

(also, they ironically make Python installs bigger since both .pyc
and .pyo files have to be provided by system packages)

Regards

Antoine.


_______________________________________________
Python-Dev mailing list
Python-Dev [at] python
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: http://mail.python.org/mailman/options/python-dev/list-python-dev%40lists.gossamer-threads.com


flub at devork

Jun 14, 2012, 4:58 AM

Post #2 of 10 (264 views)
Permalink
Re: deprecating .pyo and -O [In reply to]

On 14 June 2012 11:25, Antoine Pitrou <solipsis [at] pitrou> wrote:
> Honestly, I think the best option would be to deprecate .pyo files as
> well as the useless -O option. They only cause confusion without
> providing any significant benefits.

+1

But what happens to __debug__ and assert statements? I think it
should be possible to always put assert statements inside a __debug__
block and then create -O a simple switch for setting __debug__ to
False. If desired a simple strip tool could then easily remove
__debug__ blocks and (unused) docstrings.

--
Debian GNU/Linux -- The Power of Freedom
www.debian.org | www.gnu.org | www.kernel.org
_______________________________________________
Python-Dev mailing list
Python-Dev [at] python
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: http://mail.python.org/mailman/options/python-dev/list-python-dev%40lists.gossamer-threads.com


solipsis at pitrou

Jun 14, 2012, 5:14 AM

Post #3 of 10 (265 views)
Permalink
Re: deprecating .pyo and -O [In reply to]

On Thu, 14 Jun 2012 12:58:16 +0100
Floris Bruynooghe <flub [at] devork> wrote:
> On 14 June 2012 11:25, Antoine Pitrou <solipsis [at] pitrou> wrote:
> > Honestly, I think the best option would be to deprecate .pyo files as
> > well as the useless -O option. They only cause confusion without
> > providing any significant benefits.
>
> +1
>
> But what happens to __debug__ and assert statements? I think it
> should be possible to always put assert statements inside a __debug__
> block and then create -O a simple switch for setting __debug__ to
> False. If desired a simple strip tool could then easily remove
> __debug__ blocks and (unused) docstrings.

I don't really see the point. In my experience there is no benefit to
removing assert statements in production mode. This is a C-specific
notion that doesn't really map very well to Python code. Do other
high-level languages have similar functionality?

Regards

Antoine.


_______________________________________________
Python-Dev mailing list
Python-Dev [at] python
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: http://mail.python.org/mailman/options/python-dev/list-python-dev%40lists.gossamer-threads.com


rdmurray at bitdance

Jun 14, 2012, 6:14 AM

Post #4 of 10 (262 views)
Permalink
Re: deprecating .pyo and -O [In reply to]

On Thu, 14 Jun 2012 14:14:54 +0200, Antoine Pitrou <solipsis [at] pitrou> wrote:
> On Thu, 14 Jun 2012 12:58:16 +0100
> Floris Bruynooghe <flub [at] devork> wrote:
> > On 14 June 2012 11:25, Antoine Pitrou <solipsis [at] pitrou> wrote:
> > > Honestly, I think the best option would be to deprecate .pyo files as
> > > well as the useless -O option. They only cause confusion without
> > > providing any significant benefits.
> >
> > +1
> >
> > But what happens to __debug__ and assert statements? I think it
> > should be possible to always put assert statements inside a __debug__
> > block and then create -O a simple switch for setting __debug__ to
> > False. If desired a simple strip tool could then easily remove
> > __debug__ blocks and (unused) docstrings.
>
> I don't really see the point. In my experience there is no benefit to
> removing assert statements in production mode. This is a C-specific
> notion that doesn't really map very well to Python code. Do other
> high-level languages have similar functionality?

What does matter though is the memory savings. I'm working with an
application where the difference between normal and -OO is around a 10%
savings (about 2MB) in program DATA size at startup, and that makes a
difference for an ap running in a memory constrained environment.

A docstring stripper would enable the bulk of that savings, but it is
still nice to be able to omit code (such as debug logging statements)
as well.

--David
_______________________________________________
Python-Dev mailing list
Python-Dev [at] python
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: http://mail.python.org/mailman/options/python-dev/list-python-dev%40lists.gossamer-threads.com


solipsis at pitrou

Jun 14, 2012, 6:23 AM

Post #5 of 10 (262 views)
Permalink
Re: deprecating .pyo and -O [In reply to]

On Thu, 14 Jun 2012 09:14:59 -0400
"R. David Murray" <rdmurray [at] bitdance> wrote:
>
> What does matter though is the memory savings. I'm working with an
> application where the difference between normal and -OO is around a 10%
> savings (about 2MB) in program DATA size at startup, and that makes a
> difference for an ap running in a memory constrained environment.
>
> A docstring stripper would enable the bulk of that savings,

Probably indeed.

> but it is
> still nice to be able to omit code (such as debug logging statements)
> as well.

But does that justify all the additional complication in the core
interpreter, as well as potential user confusion?

Regards

Antoine.


_______________________________________________
Python-Dev mailing list
Python-Dev [at] python
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: http://mail.python.org/mailman/options/python-dev/list-python-dev%40lists.gossamer-threads.com


steve at pearwood

Jun 14, 2012, 1:38 PM

Post #6 of 10 (258 views)
Permalink
Re: deprecating .pyo and -O [In reply to]

Floris Bruynooghe wrote:
> On 14 June 2012 11:25, Antoine Pitrou <solipsis [at] pitrou> wrote:
>> Honestly, I think the best option would be to deprecate .pyo files as
>> well as the useless -O option. They only cause confusion without
>> providing any significant benefits.
>
> +1
>
> But what happens to __debug__ and assert statements? I think it
> should be possible to always put assert statements inside a __debug__
> block and then create -O a simple switch for setting __debug__ to
> False. If desired a simple strip tool could then easily remove
> __debug__ blocks and (unused) docstrings.

So in other words, you want to keep the functionality of -O, but make it the
responsibility of the programmer to write an external tool to implement it?

Apart from the duplication of effort (everyone who wants to optimize their
code has to write their own source-code strip tool), that surely is only going
to decrease the reliability and usefulness of Python optimization, not
increase it.

-O may be under-utilized by programmers who don't need or want it, but that
doesn't mean it isn't useful to those who do want it. -1 on deprecation.



--
Steven
_______________________________________________
Python-Dev mailing list
Python-Dev [at] python
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: http://mail.python.org/mailman/options/python-dev/list-python-dev%40lists.gossamer-threads.com


solipsis at pitrou

Jun 14, 2012, 1:46 PM

Post #7 of 10 (259 views)
Permalink
Re: deprecating .pyo and -O [In reply to]

On Fri, 15 Jun 2012 06:38:45 +1000
Steven D'Aprano <steve [at] pearwood> wrote:
>
> Apart from the duplication of effort (everyone who wants to optimize their
> code has to write their own source-code strip tool),

Actually, it could be shipped with Python, or even done dynamically at
runtime (instead of relying on separate bytecode files).

Regards

Antoine.


_______________________________________________
Python-Dev mailing list
Python-Dev [at] python
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: http://mail.python.org/mailman/options/python-dev/list-python-dev%40lists.gossamer-threads.com


steve at pearwood

Jun 14, 2012, 1:54 PM

Post #8 of 10 (256 views)
Permalink
Re: deprecating .pyo and -O [In reply to]

Antoine Pitrou wrote:

> Do other high-level languages have similar functionality?


Parrot (does anyone actually use Parrot?) has a byte-code optimizer.

javac -O is supposed to emit optimized byte-code, but allegedly it is a no-op.

On the other hand, the Java ecosystem includes third-party Java compilers
which claim to be faster/better than Oracle's compiler, including emitting
much tighter byte-code.

There are also Java byte-code optimizers such as Proguard and Soot.

By default, Perl doesn't write byte-code to files. But when it does, there are
various "optimization back-ends" that you can use.

Until version 1.9, Ruby didn't even use byte-code at all.


--
Steven

_______________________________________________
Python-Dev mailing list
Python-Dev [at] python
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: http://mail.python.org/mailman/options/python-dev/list-python-dev%40lists.gossamer-threads.com


martin at v

Jun 14, 2012, 1:57 PM

Post #9 of 10 (258 views)
Permalink
Re: deprecating .pyo and -O [In reply to]

> I don't really see the point. In my experience there is no benefit to
> removing assert statements in production mode. This is a C-specific
> notion that doesn't really map very well to Python code. Do other
> high-level languages have similar functionality?

It's not at all C specific. C# also has it:

http://msdn.microsoft.com/en-us/library/ttcc4x86(v=vs.80).aspx

Java makes it a VM option (rather than a compiler option), but it's
still a flag to the VM (-enableassertions):

http://docs.oracle.com/javase/1.4.2/docs/tooldocs/windows/java.html

Delphi also has assertions that can be disabled at compile time.

Regards,
Martin
_______________________________________________
Python-Dev mailing list
Python-Dev [at] python
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: http://mail.python.org/mailman/options/python-dev/list-python-dev%40lists.gossamer-threads.com


greg at krypto

Jun 16, 2012, 5:04 PM

Post #10 of 10 (237 views)
Permalink
Re: deprecating .pyo and -O [In reply to]

On Thu, Jun 14, 2012 at 1:57 PM, "Martin v. Löwis" <martin [at] v>wrote:

> > I don't really see the point. In my experience there is no benefit to
> > removing assert statements in production mode. This is a C-specific
> > notion that doesn't really map very well to Python code. Do other
> > high-level languages have similar functionality?
>
> It's not at all C specific. C# also has it:
>
> http://msdn.microsoft.com/en-**us/library/ttcc4x86(v=vs.80).**aspx<http://msdn.microsoft.com/en-us/library/ttcc4x86(v=vs.80).aspx>
>
> Java makes it a VM option (rather than a compiler option), but it's
> still a flag to the VM (-enableassertions):
>
> http://docs.oracle.com/javase/**1.4.2/docs/tooldocs/windows/**java.html<http://docs.oracle.com/javase/1.4.2/docs/tooldocs/windows/java.html>
>
> Delphi also has assertions that can be disabled at compile time.
>
>
It may be a commonly supported feature but that doesn't mean it is a good
idea. :)

It seems to me that assertion laden code where the assertions are removed
before shipping it or running it in production is largely a concept from
the pre-unittesting era. One big issue with them in Python is that
enabling or disabling them is VM global rather than controlled on a per
module/library basis. That isn't true in C/C++ where you can control it on
a per file basis at compile time (NDEBUG).

As a developer I agree that it is very convenient to toss asserts into code
while you are iterating on writing it. But I regularly have people remove
assert statements during code reviews at work and, if the condition is
important, replacing them with actual if condition checks that handle the
problem or raise an appropriate module specific exception and documenting
that as part of their API. At this point, I wish Python just didn't
support assert as part of the language because it causes less headaches in
the end of code is just written without them begin with. Too late now.

In agreement with others: docstring memory consumption is a big deal, some
large Python apps at work strip them or use -O (I don't remember which
technique they're using today) before deployment. It does seem like
something we should provide a standard way of doing. Docstring data is not
needed within non-interactive code.

-gps

Python dev RSS feed   Index | Next | Previous | View Threaded
 
 


Interested in having your list archived? Contact Gossamer Threads
 
  Web Applications & Managed Hosting Powered by Gossamer Threads Inc.