Mailing List Archive: Maemo: Developers

maemo-optify, autobuilder & /opt

1 2 3 4

View All

Index | Next | Previous | View Threaded

andrew at bleb

Oct 28, 2009, 9:22 AM

Post #1 of 88 (1900 views)
Permalink

maemo-optify, autobuilder & /opt

Hi,

I've put together a suggested spec for the decision, taken at the
summit during the /opt BOF[1], that the auto-builder would run some
maemo-optify version during the build process (controlled by a control
file header):

http://talk.maemo.org/showthread.php?p=359996#post359996

I suggest the header is XS-Maemo-Optify, and has the following values:

none: no optification should be done, or considered, by the autobuilder.
manual: the application author will do optification manually. If the
package contains no entries under /opt it would be considered a
build failure.
auto: maemo-optify would be run if certain heuristics were met (e.g.
no entries in /opt, no Python dependency)
force: maemo-optify would always be run

Marius: are you taking ownership of talking to Ed Bartosh, and anyone
else, about this plan?

Thanks in advance,

Andrew

[1] http://lists.maemo.org/pipermail/maemo-developers/2009-October/021289.html

--
Andrew Flegg -- mailto:andrew [at] bleb | http://www.bleb.org/
_______________________________________________
maemo-developers mailing list
maemo-developers [at] maemo
https://lists.maemo.org/mailman/listinfo/maemo-developers

bundyo at gmail

Oct 28, 2009, 12:09 PM

Post #2 of 88 (1850 views)
Permalink

Re: maemo-optify, autobuilder & /opt [In reply to]

Wouldn't it be better to leave the none option out considering the lack of storage space in N900. It can be made available in Harmattan.

Regards:
Bundyo

bartosh at gmail

Oct 28, 2009, 12:10 PM

Post #3 of 88 (1851 views)
Permalink

Re: maemo-optify, autobuilder & /opt [In reply to]

2009/10/28 Andrew Flegg <andrew [at] bleb>:
> Hi,
>
> I've put together a suggested spec for the decision, taken at the
> summit during the /opt BOF[1], that the auto-builder would run some
> maemo-optify version during the build process (controlled by a control
> file header):
>
> http://talk.maemo.org/showthread.php?p=359996#post359996
>
> I suggest the header is XS-Maemo-Optify, and has the following values:
>
> none: no optification should be done, or considered, by the autobuilder.
> manual: the application author will do optification manually. If the
> package contains no entries under /opt it would be considered a
> build failure.
> auto: maemo-optify would be run if certain heuristics were met (e.g.
> no entries in /opt, no Python dependency)
> force: maemo-optify would always be run
>
> Marius: are you taking ownership of talking to Ed Bartosh, and anyone
> else, about this plan?
>
We can discuss it here.

Sorry, I seem to miss the whole point of this activity. Why do you
need to do that on autobuilder side? As far as I understood it's just
a matter of including maemo-optify as a build dependency and run it
in debian/rules, right? Why developer can't do this then? I don't see
much difference between setting XS-Maemo-Optify and changes I
mentioned above. In both cases developer should understand what
optification means.

BTW, when you want to have it done?
I'm going to vacation in a couple of weeks. Before that I was going to
finish implementation of multiple packages builds if I have time.

--
BR,
Ed
_______________________________________________
maemo-developers mailing list
maemo-developers [at] maemo
https://lists.maemo.org/mailman/listinfo/maemo-developers

andrew at bleb

Oct 28, 2009, 1:39 PM

Post #4 of 88 (1846 views)
Permalink

Re: maemo-optify, autobuilder & /opt [In reply to]

Ed wrote:
> >
> > I've put together a suggested spec for the decision, taken at the
> > summit during the /opt BOF[1], that the auto-builder would run some
> > maemo-optify version during the build process (controlled by a control
> > file header):
>
> Sorry, I seem to miss the whole point of this activity. Why do you
> need to do that on autobuilder side? As far as I understood it's just
> a matter of including maemo-optify as a build dependency and run it
> in debian/rules, right? Why developer can't do this then? I don't see
> much difference between setting XS-Maemo-Optify and changes I
> mentioned above. In both cases developer should understand what
> optification means.

The consensus at the BOF was that since it's something which SHOULD be done for all Maemo packages, but that this is Maemo specific, that it should be done at the autobuilder to ensure that as many packages are optified as possible.

By having "auto" be the default (i.e. the developer has NOT specified XS-Maemo-Optify) at some point in the future we can avoid the current problem where, even though the requirements are well understood more than half the user-facing applications in -testing aren't using /opt.

> BTW, when you want to have it done?
> I'm going to vacation in a couple of weeks. Before that I was going to
> finish implementation of multiple packages builds if I have time.

i don't know, it's not my baby :-) One would imagine, in a perfect world, that if this was being done then:

* We'd have a conclusion on the Python issue
and a comment from the Qt folks.
* XS-Maemo-Optify would be agreed and the builder
and maemo-optify(-deb) changes made with 'none'
as the default.
* We could opt our packages in for the testing
period, then change the default to "auto",
rebuild everything and not suffer any more
rootfs problems.

Given the small rootfs is the single largest problem which is going to impact reviews and opinion about the N900 we'd have it done ASAP. But then I no longer have to think of anybody else's priorities, that's somebody else's problem :)

Cheers,

Andrew

--
Andrew Flegg -- mailto:andrew [at] bleb | http://www.bleb.org/

_______________________________________________
maemo-developers mailing list
maemo-developers [at] maemo
https://lists.maemo.org/mailman/listinfo/maemo-developers

andrew at bleb

Oct 28, 2009, 1:45 PM

Post #5 of 88 (1853 views)
Permalink

Re: maemo-optify, autobuilder & /opt [In reply to]

Kamen wrote:
>
> Wouldn't it be better to leave the none option out considering the lack of
> storage space in N900. It can be made available in Harmattan.

The thought was that there might be a small, or niche, product which just wouldn't work with maemo-optify and changing it to use /opt would be impractical. By not being too dictatorial the developer isn't pushed into their own repo.

Similarly, if the bind mount is the best solution to the Python issue, Python modules might need to specify 'none' until auto's heuristic is updated to exclude them (Python modules would continue to install into /usr/lib/python2.5/site-packages under this solution, but the FS knows that that's some path under /opt really).

Does that make sense?

Cheers,

Andrew

--
Andrew Flegg -- mailto:andrew [at] bleb | http://www.bleb.org/

_______________________________________________
maemo-developers mailing list
maemo-developers [at] maemo
https://lists.maemo.org/mailman/listinfo/maemo-developers

Fred at Lefevere-Laoide

Oct 28, 2009, 3:32 PM

Post #6 of 88 (1848 views)
Permalink

Re: maemo-optify, autobuilder & /opt [In reply to]

I think it is good to be able to keep maemo-optify out of Build-Depends :
This way we can keep the same debian control file for Diablo and Fremantle.
I suppose the Diablo builder will only ignore the optify header ?

Fred

On Wed, Oct 28, 2009 at 8:10 PM, Ed Bartosh <bartosh [at] gmail> wrote:
> 2009/10/28 Andrew Flegg <andrew [at] bleb>:
>> Hi,
>>
>> I've put together a suggested spec for the decision, taken at the
>> summit during the /opt BOF[1], that the auto-builder would run some
>> maemo-optify version during the build process (controlled by a control
>> file header):
>>
>> http://talk.maemo.org/showthread.php?p=359996#post359996
>>
>> I suggest the header is XS-Maemo-Optify, and has the following values:
>>
>> none: no optification should be done, or considered, by the autobuilder.
>> manual: the application author will do optification manually. If the
>> package contains no entries under /opt it would be considered a
>> build failure.
>> auto: maemo-optify would be run if certain heuristics were met (e.g.
>> no entries in /opt, no Python dependency)
>> force: maemo-optify would always be run
>>
>> Marius: are you taking ownership of talking to Ed Bartosh, and anyone
>> else, about this plan?
>>
> We can discuss it here.
>
> Sorry, I seem to miss the whole point of this activity. Why do you
> need to do that on autobuilder side? As far as I understood it's just
> a matter of including maemo-optify as a build dependency and run it
> in debian/rules, right? Why developer can't do this then? I don't see
> much difference between setting XS-Maemo-Optify and changes I
> mentioned above. In both cases developer should understand what
> optification means.
>
> BTW, when you want to have it done?
> I'm going to vacation in a couple of weeks. Before that I was going to
> finish implementation of multiple packages builds if I have time.
>
> --
> BR,
> Ed
> _______________________________________________
> maemo-developers mailing list
> maemo-developers [at] maemo
> https://lists.maemo.org/mailman/listinfo/maemo-developers
>

--

Sent from La Rochelle, France
_______________________________________________
maemo-developers mailing list
maemo-developers [at] maemo
https://lists.maemo.org/mailman/listinfo/maemo-developers

g+770 at cobb

Oct 28, 2009, 3:39 PM

Post #7 of 88 (1842 views)
Permalink

Re: maemo-optify, autobuilder & /opt [In reply to]

On Wednesday 28 October 2009 20:39:57 Andrew Flegg wrote:
> Ed wrote:
> > BTW, when you want to have it done?
> > I'm going to vacation in a couple of weeks. Before that I was going to
> > finish implementation of multiple packages builds if I have time.
>
> i don't know, it's not my baby :-) One would imagine, in a perfect world,
> that if this was being done then:
>
> * We'd have a conclusion on the Python issue
> and a comment from the Qt folks.
> * XS-Maemo-Optify would be agreed and the builder
> and maemo-optify(-deb) changes made with 'none'
> as the default.
> * We could opt our packages in for the testing
> period, then change the default to "auto",
> rebuild everything and not suffer any more
> rootfs problems.

I think we should do the second item before Ed goes on holiday, even if it
means deferring the multiple package builds. We can then test it (setting
the header to auto in various packages) while Ed is away but there is minimal
danger of problems cropping up while he is away.

Ed, could we do that?

Graham
_______________________________________________
maemo-developers mailing list
maemo-developers [at] maemo
https://lists.maemo.org/mailman/listinfo/maemo-developers

bartosh at gmail

Oct 28, 2009, 3:50 PM

Post #8 of 88 (1848 views)
Permalink

Re: maemo-optify, autobuilder & /opt [In reply to]

2009/10/28 Andrew Flegg <andrew [at] bleb>:
> Ed wrote:
>> >
>> > I've put together a suggested spec for the decision, taken at the
>> > summit during the /opt BOF[1], that the auto-builder would run some
>> > maemo-optify version during the build process (controlled by a control
>> > file header):
>>
>> Sorry, I seem to miss the whole point of this activity. Why do you
>> need to do that on autobuilder side? As far as I understood it's just
>> a matter of including maemo-optify as a build dependency and run it
>> in debian/rules, right? Why developer can't do this then? I don't see
>> much difference between setting XS-Maemo-Optify and changes I
>> mentioned above. In both cases developer should understand what
>> optification means.
>
> The consensus at the BOF was that since it's something which SHOULD be done for all Maemo packages, but that this is Maemo specific, that it should be done at the autobuilder to ensure that as many packages are optified as possible.
>
> By having "auto" be the default (i.e. the developer has NOT specified XS-Maemo-Optify) at some point in the future we can avoid the current problem where, even though the requirements are well understood more than half the user-facing applications in -testing aren't using /opt.
>

Somehow I don't like the idea of doing anything with the package
without developer being aware of this. I'd rather implement check on
autobuilder side to insure that packages are optified. Developer can
use option XS-Maemo-Optify: none to disable optification if developer
doesn't want it.

Explicit is better than implicit. (C) Zen of Python :)

--
BR,
Ed
_______________________________________________
maemo-developers mailing list
maemo-developers [at] maemo
https://lists.maemo.org/mailman/listinfo/maemo-developers

g+770 at cobb

Oct 28, 2009, 4:42 PM

Post #9 of 88 (1835 views)
Permalink

Re: maemo-optify, autobuilder & /opt [In reply to]

On Wednesday 28 October 2009 22:50:25 Ed Bartosh wrote:
> Somehow I don't like the idea of doing anything with the package
> without developer being aware of this. I'd rather implement check on
> autobuilder side to insure that packages are optified. Developer can
> use option XS-Maemo-Optify: none to disable optification if developer
> doesn't want it.

Nobody likes doing something to the package automatically but, after a long
discussion at the BOF, we agreed that the alternatives were even worse [1].

In particular, there was a strong argument that the package should not have to
include anything (even a control field option) to cause optification to
happen. Packages which wanted to do their own optification or which had to
disable optification would have to include an option to stop optification.

If it makes you happier: rename the autobuilder as "autobuilder and optifier"!

We did agree that there had to be a way for developers to generate packages in
the scratchbox environment which had been optified in exactly the same way
the autobuilder would. And that we would update the wiki pages about
checking packages will build to include using that tool to test the
optification.

So, the consensus decision was that the solution would be that autobuilder
should automatically optify by default.

Graham

[1] It reminds me of the famous quote from Winston Churchill: "Democracy is
the worst form of government, except for all those other forms that have been
tried from time to time."
_______________________________________________
maemo-developers mailing list
maemo-developers [at] maemo
https://lists.maemo.org/mailman/listinfo/maemo-developers

bartosh at gmail

Oct 28, 2009, 10:52 PM

Post #10 of 88 (1836 views)
Permalink

Re: maemo-optify, autobuilder & /opt [In reply to]

2009/10/29 Graham Cobb <g+770 [at] cobb>:
> I think we should do the second item before Ed goes on holiday, even if it
> means deferring the multiple package builds. We can then test it (setting
> the header to auto in various packages) while Ed is away but there is minimal
> danger of problems cropping up while he is away.
>
> Ed, could we do that?
>
I think we can. I just want to understand why this way.

--
BR,
Ed
_______________________________________________
maemo-developers mailing list
maemo-developers [at] maemo
https://lists.maemo.org/mailman/listinfo/maemo-developers

bartosh at gmail

Oct 28, 2009, 11:07 PM

Post #11 of 88 (1840 views)
Permalink

Re: maemo-optify, autobuilder & /opt [In reply to]

2009/10/29 Graham Cobb <g+770 [at] cobb>:
> On Wednesday 28 October 2009 22:50:25 Ed Bartosh wrote:
>> Somehow I don't like the idea of doing anything with the package
>> without developer being aware of this. I'd rather implement check on
>> autobuilder side to insure that packages are optified. Developer can
>> use option XS-Maemo-Optify: none to disable optification if developer
>> doesn't want it.
>
> Nobody likes doing something to the package automatically but, after a long
> discussion at the BOF, we agreed that the alternatives were even worse [1].
>
Then let's find the way to do it better.
What I'm afraid of is that developers wouldn't like the approach to
change packages implicitly. It potentially can create repository mess
again. And I really don't want this to happen.

> In particular, there was a strong argument that the package should not have to
> include anything (even a control field option) to cause optification to
> happen. Packages which wanted to do their own optification or which had to
> disable optification would have to include an option to stop optification.
>
> If it makes you happier: rename the autobuilder as "autobuilder and optifier"!
>
No, it won't make me happier.

> We did agree that there had to be a way for developers to generate packages in
> the scratchbox environment which had been optified in exactly the same way
> the autobuilder would. And that we would update the wiki pages about
> checking packages will build to include using that tool to test the
> optification.
>
Would it be better to change the common part of developer environment
and autobuilder, for example somewhere in debian devkit? If
dpkg-buildpackage will produce optified packages they will be at least
the same everywhere.

> So, the consensus decision was that the solution would be that autobuilder
> should automatically optify by default.
>
I didn't even think it will go this way. This why I didn't participate
on discussions and BOF, sorry. Does it mean that I should shut up and
do what I'm told to do?

--
BR,
Ed
_______________________________________________
maemo-developers mailing list
maemo-developers [at] maemo
https://lists.maemo.org/mailman/listinfo/maemo-developers

mardy at users

Oct 29, 2009, 12:32 AM

Post #12 of 88 (1835 views)
Permalink

Re: maemo-optify, autobuilder & /opt [In reply to]

Graham Cobb wrote:
> So, the consensus decision was that the solution would be that autobuilder
> should automatically optify by default.

Sounds wrong to me. I agree with Ed, the default should be "manual": so, non
optified packages would fail to build, but fixing that would be as easy as
adding a "XS-Maemo-Optify: none".

Ciao,
Alberto

--
http://www.mardy.it <-- geek in un lingua international!
_______________________________________________
maemo-developers mailing list
maemo-developers [at] maemo
https://lists.maemo.org/mailman/listinfo/maemo-developers

andrew at bleb

Oct 29, 2009, 12:54 AM

Post #13 of 88 (1843 views)
Permalink

Re: maemo-optify, autobuilder & /opt [In reply to]

Ed wrote:
> 2009/10/29 Graham Cobb <g+770 [at] cobb>:
> >
> > Nobody likes doing something to the package automatically but, after a long
> > discussion at the BOF, we agreed that the alternatives were even worse [1].
> >
> Then let's find the way to do it better.
> What I'm afraid of is that developers wouldn't like the approach to
> change packages implicitly.

There were some very "senior" and well respected developers in the room, who package some of the leading Maemo applications.

> It potentially can create repository mess
> again. And I really don't want this to happen.

No-one does, however increasing the amount of work developers have to do to get into Extras because of Nokia's short-sightedness is also a demotivating factor which could lead to multiple repositories springing up.

> > In particular, there was a strong argument that the package should not have to
> > include anything (even a control field option) to cause optification to
> > happen. Packages which wanted to do their own optification or which had to
> > disable optification would have to include an option to stop optification.

And this is because /opt is basically a weirdness caused specific to Maemo packaging, and (with the move to Qt) the Maemo development community is increasingly realising the benefits of abstracting platform weirdness.

> Would it be better to change the common part of developer environment
> and autobuilder, for example somewhere in debian devkit? If
> dpkg-buildpackage will produce optified packages they will be at least
> the same everywhere.

Have you an estimate on the comparative costs of developing one vs. the other? This is an implementation detail of "make the autobuider" do it. Who owns the Debian devkit and do they want to do the work?

A "maemo-buildpackage" was mentioned in the BOF as a potential way of allowing developers to do what the auto-builder does. How hard would it be to develop this and get the autobuilder to call maemo- rather than dpkg-buildpackage?

However, there seem to be two arguments on your side:

1) Don't do anything, developers should modfy
debian/rules as they do now.
2) Make something in the build process do it,
rather than the autobuilder.

(2) is an internal implementation detail which isn't important externally: the consensus view could be tested by uploading a Diablo source package with no changes and having it auto-optified. Whether that's through a change to the devkit, autobuilder-specific code or the introduction of maemo-buildpackage is of little interest to the person doing the uploading :-)

> > So, the consensus decision was that the solution would be that autobuilder
> > should automatically optify by default.
>
> I didn't even think it will go this way. This why I didn't participate
> on discussions and BOF, sorry. Does it mean that I should shut up and
> do what I'm told to do?

There were a large number of stakeholders in the room, representing a variety of different views. It is unfortunate that you weren't there, meaning that the discussions have to be had again. It's disappointing that these comments weren't raised from the minutes of the BOF posted 3 weeks ago, but it's also disappointing that no-one's taken charge of driving this through and had spoken to you about enacting these changes.

Cheers,

Andrew

--
Andrew Flegg -- mailto:andrew [at] bleb | http://www.bleb.org/
_______________________________________________
maemo-developers mailing list
maemo-developers [at] maemo
https://lists.maemo.org/mailman/listinfo/maemo-developers

maemo at csipa

Oct 29, 2009, 1:08 AM

Post #14 of 88 (1831 views)
Permalink

Re: maemo-optify, autobuilder & /opt [In reply to]

On Thursday 29 October 2009 07:07:14 Ed Bartosh wrote:
> Then let's find the way to do it better.

I believe that was the stance on the problem since day 1 :)

> What I'm afraid of is that developers wouldn't like the approach to
> change packages implicitly.

Herein lies the root of the problem. We are beyond the phase of 'like'. We are
closer to damage control, in terms of a lesser evil and an imperative to make
the choice ASAP. The N900 release date approaches and we've seen that friendly
pokes for /opt-ification yield results very slowly. If the non-active
interference route is continued, we won't 'offend' fellow developers but the
ecosystem will suffer as the first users start hitting the 256MB NAND limits (or
not seeing packages at all as they don't make it to extras). Thus, I have no
doubt, a decision will have to be made and whatever it will be, someone will
be unhappy as a result of it. The question is which solution will hurt least
on the *platform* level.

>It potentially can create repository mess again. And I really don't want this
>to happen.

Nobody does. However, we're running out of time to find a better solution. And
if there is a potential for repository mess it has to happen before release.
Once end-users get to the devices, it's mostly game over, whatever solution
will be in place by that time is most likely the one we will be stuck with for
quite a while.
_______________________________________________
maemo-developers mailing list
maemo-developers [at] maemo
https://lists.maemo.org/mailman/listinfo/maemo-developers

andrew at bleb

Oct 29, 2009, 1:08 AM

Post #15 of 88 (1833 views)
Permalink

Re: maemo-optify, autobuilder & /opt [In reply to]

Alberto wrote:
> Graham Cobb wrote:
> > So, the consensus decision was that the solution would be that autobuilder
> > should automatically optify by default.
>
> Sounds wrong to me.

Can you elaborate? I'd like to be convinced (as I was during the BOF) rather than just whomever expresses the most feelings most often and loudest getting their way.

Assuming:
* Developers want to upload the same source package
for Diablo, Fremantle and Mer;
* /opt won't be necessary at some point in
the future, or on some devices such as those
running Mer;
* Nokia aren't going to implement a union FS or
just use the NAND for swap (and put the rootfs
on the eMMC) in the shortf (or even, probably,
medium-) term;
* Pretty much every package on Maemo should be using
/opt as much as possible;

...it was concluded that:

a) Modifying debian/rules is hacky and causes
forking between Diablo, Fremantle and Mer.
b) A control file field makes the most sense to
control the build process.
c) That the absence of the control field would
Do The Right Thing ((c) Ruby, Perl, Groovy)
which, on Fremantle, is optification in most cases.

Cheers,

Andrew

--
Andrew Flegg -- mailto:andrew [at] bleb | http://www.bleb.org/

_______________________________________________
maemo-developers mailing list
maemo-developers [at] maemo
https://lists.maemo.org/mailman/listinfo/maemo-developers

marius.vollmer at nokia

Oct 29, 2009, 1:23 AM

Post #16 of 88 (1832 views)
Permalink

Re: maemo-optify, autobuilder & /opt [In reply to]

ext Andrew Flegg <andrew [at] bleb> writes:

> A "maemo-buildpackage" was mentioned in the BOF as a potential way of
> allowing developers to do what the auto-builder does. How hard would
> it be to develop this and get the autobuilder to call maemo- rather
> than dpkg-buildpackage?

I'll give this a shot.
_______________________________________________
maemo-developers mailing list
maemo-developers [at] maemo
https://lists.maemo.org/mailman/listinfo/maemo-developers

bartosh at gmail

Oct 29, 2009, 1:24 AM

Post #17 of 88 (1835 views)
Permalink

Re: maemo-optify, autobuilder & /opt [In reply to]

2009/10/29 Andrew Flegg <andrew [at] bleb>:
> Ed wrote:
>> 2009/10/29 Graham Cobb <g+770 [at] cobb>:
>> >
>> > Nobody likes doing something to the package automatically but, after a long
>> > discussion at the BOF, we agreed that the alternatives were even worse [1].
>> >
>> Then let's find the way to do it better.
>> What I'm afraid of is that developers wouldn't like the approach to
>> change packages implicitly.
>
> There were some very "senior" and well respected developers in the room, who package some of the leading Maemo applications.
>
>> It potentially can create repository mess
>> again. And I really don't want this to happen.
>
> No-one does, however increasing the amount of work developers have to do to get into Extras because of Nokia's short-sightedness is also a demotivating factor which could lead to multiple repositories springing up.
>
True. However I'm not sure that making developers to do additional
work is worse than change package imlicitly, which can potentially
cause installation and runtime problems.

>> > In particular, there was a strong argument that the package should not have to
>> > include anything (even a control field option) to cause optification to
>> > happen.� Packages which wanted to do their own optification or which had to
>> > disable optification would have to include an option to stop optification.
>
> And this is because /opt is basically a weirdness caused specific to Maemo packaging, and (with the move to Qt) the Maemo development community is increasingly realising the benefits of abstracting platform weirdness.
>
>> Would it be better to change the common part of developer environment
>> and autobuilder, for example somewhere in debian devkit? If
>> dpkg-buildpackage will produce optified packages they will be at least
>> the same everywhere.
>
> Have you an estimate on the comparative costs of developing one vs. the other? This is an implementation detail of "make the autobuider" do it. Who owns the Debian devkit and do they want to do the work?
>
I'd say changing devkit would take twice more then modifying
autobuilder. Not a very big difference, considering that we can have
one solution to both problems. With modified devkit we can change both
developer and autobuider environments in one shot.

Devkit is a part of SDK, so SDK people own it.
I can help to modify it if we decide to go this way.

> A "maemo-buildpackage" was mentioned in the BOF as a potential way of allowing developers to do what the auto-builder does. How hard would it be to develop this and get the autobuilder to call maemo- rather than dpkg-buildpackage?
>
> However, there seem to be two arguments on your side:
>
> �1) Don't do anything, developers should modfy
> � � debian/rules as they do now.
> �2) Make something in the build process do it,
> � � rather than the autobuilder.
>
I like 1st better. Second is just a bit better alternative to your decision.

> (2) is an internal implementation detail which isn't important externally: the consensus view could be tested by uploading a Diablo source package with no changes and having it auto-optified. Whether that's through a change to the devkit, autobuilder-specific code or the introduction of maemo-buildpackage is of little interest to the person doing the uploading :-)
>
It is important. Instead of introducing new tool we just change the
existing. No additional work for developers in this case.

--
BR,
Ed
_______________________________________________
maemo-developers mailing list
maemo-developers [at] maemo
https://lists.maemo.org/mailman/listinfo/maemo-developers

g+770 at cobb

Oct 29, 2009, 3:06 AM

Post #18 of 88 (1824 views)
Permalink

Re: maemo-optify, autobuilder & /opt [In reply to]

On Thursday 29 October 2009 08:08:20 Attila Csipa wrote:
> On Thursday 29 October 2009 07:07:14 Ed Bartosh wrote:
> > Then let's find the way to do it better.
>
> I believe that was the stance on the problem since day 1 :)
>
> > What I'm afraid of is that developers wouldn't like the approach to
> > change packages implicitly.
>
> Herein lies the root of the problem. We are beyond the phase of 'like'. We
> are closer to damage control, in terms of a lesser evil and an imperative
> to make the choice ASAP.

That is the problem. We have to do something now to fix the problem.

Telling developers they have to add a field to request optification and
blocking them from Extras unless they have added it will just add to the
people saying "Nokia have closed down Extras and are not letting people put
their packages in -- so I have created my own, independent,
non-corporate-controlled repository for true freedom". Also, there are
developers who know very little about packaging and want to know even less --
adding another thing they have to do is likely to cause at least some of them
to give up and tell people to download their application from a website.

As a developer, of a set of packages with quite complex dependencies, I went
into the meeting with the same position as you: I do not want the autobuilder
messing with **my** package! But during the discussion I was convinced that
defaulting to auto-optifying was the right thing to do (with controls to turn
it off I need to). The position that really scared me was a proposal that
the optification should happen at installation time! At least with this
proposal I can look at the deb, see what is being put where, install it in
scratchbox and debug any problems.

I think if you had been there you would have been convinced that this is the
right option as well.

It is also very easy for us to back out of or replace with a different
solution if we need to.

Graham
_______________________________________________
maemo-developers mailing list
maemo-developers [at] maemo
https://lists.maemo.org/mailman/listinfo/maemo-developers

mardy at users

Oct 29, 2009, 3:56 AM

Post #19 of 88 (1827 views)
Permalink

Re: maemo-optify, autobuilder & /opt [In reply to]

Andrew Flegg wrote:
> Alberto wrote:
>> Graham Cobb wrote:
>>> So, the consensus decision was that the solution would be that autobuilder
>>> should automatically optify by default.
>> Sounds wrong to me.
>
> Can you elaborate? I'd like to be convinced (as I was during the BOF) rather than just whomever expresses the most feelings most often and loudest getting their way.

Sure :-)

> Assuming:
> * Developers want to upload the same source package
> for Diablo, Fremantle and Mer;

I have some doubts on this assumption. Fremantle is rather different then the
previous version, so the source package will be most likely a separate one
(especially for UI apps). At least this is going to be the case for maemo-mapper.

> * /opt won't be necessary at some point in
> the future, or on some devices such as those
> running Mer;
> * Nokia aren't going to implement a union FS or
> just use the NAND for swap (and put the rootfs
> on the eMMC) in the shortf (or even, probably,
> medium-) term;
> * Pretty much every package on Maemo should be using
> /opt as much as possible;

Ok, but then the maemo-optify will be doing different things in the various
build environments.

> ...it was concluded that:
>
> a) Modifying debian/rules is hacky and causes
> forking between Diablo, Fremantle and Mer.

Mmm... It might be hacky, true. But it does not cause any forking, if we provide
it for Diablo and Mer too.

> b) A control file field makes the most sense to
> control the build process.

Agreed.

> c) That the absence of the control field would
> Do The Right Thing ((c) Ruby, Perl, Groovy)
> which, on Fremantle, is optification in most cases.

Maybe. I don't have a strong opinion about that, but I wonder if the automatic
optification might introduce some subtle bugs that the developer might have
trouble to investigate, if he doesn't know that the build system did modify his
package.

Ciao,
Alberto

--
http://www.mardy.it <-- geek in un lingua international!
_______________________________________________
maemo-developers mailing list
maemo-developers [at] maemo
https://lists.maemo.org/mailman/listinfo/maemo-developers

marius.vollmer at nokia

Oct 29, 2009, 4:12 AM

Post #20 of 88 (1828 views)
Permalink

Re: maemo-optify, autobuilder & /opt [In reply to]

ext Alberto Mardegan <mardy [at] users> writes:

>> b) A control file field makes the most sense to
>> control the build process.
>
> Agreed.

I think dedicated files in debian/ are better, like the
debian/<package>.install files, etc.

Right now, I am just putting the mode into debian/optify, but I can
already see how this is going to be extended to control optification in
much more detail. Putting it all into debian/control is likely too
much. Also, the op\tification ugliness is easier to ignore when it is
in its own files.
_______________________________________________
maemo-developers mailing list
maemo-developers [at] maemo
https://lists.maemo.org/mailman/listinfo/maemo-developers

marius.vollmer at nokia

Oct 29, 2009, 5:02 AM

Post #21 of 88 (1826 views)
Permalink

Re: maemo-optify, autobuilder & /opt [In reply to]

ext Andrew Flegg <andrew [at] bleb> writes:

> I suggest the header is XS-Maemo-Optify, and has the following values:
>
> none: no optification should be done, or considered, by the autobuilder.
> manual: the application author will do optification manually. If the
> package contains no entries under /opt it would be considered a
> build failure.
> auto: maemo-optify would be run if certain heuristics were met (e.g.
> no entries in /opt, no Python dependency)
> force: maemo-optify would always be run

Thanks for the initiative, Andrew!

I have put maemo-optify 0.2 into extras-devel with the following
changes, motivated by your proposal:

* Added --auto option to maemo-optify-deb which will read debian/optify
and debian/files and does the right thing.

* Added maemo-optify-buildpackage, which invokes maemo-optify-deb in
auto mode.

More details in the README here:

http://maemo.gitorious.org/maemo-af/maemo-optify/blobs/master/README

Thus, only "none" and "auto" are really implemented right now, and the
mode is read from debian/optify instead of from debian/control.

(Proper man pages are still missing.)

Thus, the next step is to make sure that maemo-optify is installed in
the build environment and to use maemo-optify-buildpackage instead of
dpkg-buildpackage. (Or make the equivalent changes to dpkg-buildpackage
itself, which are quite small.)
_______________________________________________
maemo-developers mailing list
maemo-developers [at] maemo
https://lists.maemo.org/mailman/listinfo/maemo-developers

g+770 at cobb

Oct 29, 2009, 5:14 AM

Post #22 of 88 (1822 views)
Permalink

Re: maemo-optify, autobuilder & /opt [In reply to]

On Thursday 29 October 2009 11:12:45 Marius Vollmer wrote:
> ext Alberto Mardegan <mardy [at] users> writes:
> >> b) A control file field makes the most sense to
> >> control the build process.
> >
> > Agreed.
>
> I think dedicated files in debian/ are better, like the
> debian/<package>.install files, etc.

There is a difference. A debian/optify is fine for controlling how
maemo-optify works. A control field is more logical for controlling how the
auto-builder works: i.e. should it call maemo-optify at all, should it reject
the package because it claims to be optified but there are not files in /opt,
etc. If you are doing your own optification you shouldn't need a
debian/optify - just a control file field to tell the autobuilder not to run
maemo-optify at all.

So, I would not see the things Andrew defined going in to debian/optify. But
debian/optify would be a good place to put things like: what is the size
threshold for optification, what directories should be optified as a whole
directory, which files/directories should be left untouched, etc.

Graham
_______________________________________________
maemo-developers mailing list
maemo-developers [at] maemo
https://lists.maemo.org/mailman/listinfo/maemo-developers

bartosh at gmail

Oct 29, 2009, 5:23 AM

Post #23 of 88 (1821 views)
Permalink

Re: maemo-optify, autobuilder & /opt [In reply to]

2009/10/29 Marius Vollmer <marius.vollmer [at] nokia>:
> ext Andrew Flegg <andrew [at] bleb> writes:
>
>> I suggest the header is XS-Maemo-Optify, and has the following values:
>>
>> � none: � no optification should be done, or considered, by the autobuilder.
>> � manual: the application author will do optification manually. If the
>> � � � � � package contains no entries under /opt it would be considered a
>> � � � � � build failure.
>> � auto: � maemo-optify would be run if certain heuristics were met (e.g.
>> � � � � � no entries in /opt, no Python dependency)
>> � force: �maemo-optify would always be run
>
> Thanks for the initiative, Andrew!
>
> I have put maemo-optify 0.2 into extras-devel with the following
> changes, motivated by your proposal:
>
> �* Added --auto option to maemo-optify-deb which will read debian/optify
> � �and debian/files and does the right thing.
>
> �* Added maemo-optify-buildpackage, which invokes maemo-optify-deb in
> � �auto mode.
>
> More details in the README here:
>
> � �http://maemo.gitorious.org/maemo-af/maemo-optify/blobs/master/README
>
> Thus, only "none" and "auto" are really implemented right now, and the
> mode is read from debian/optify instead of from debian/control.
>
> (Proper man pages are still missing.)
>
> Thus, the next step is to make sure that maemo-optify is installed in
> the build environment and to use maemo-optify-buildpackage instead of
> dpkg-buildpackage. �(Or make the equivalent changes to dpkg-buildpackage
> itself, which are quite small.)
>
There are two ways to have maemo-optify in build environment - to add
it to the rootstrap and to put it into Build-Depends.
As I understood we don't want to ask developers to put it into
Build-Depends, so rootstrap should be changed.
I already hacked it one time when I removed /opt from there. I can do
it again, but I don't want to fork SDK rootstrap, so it would
be nice to somehow agree with SDK team to include these changes into
the next release.

I like the idea of making equivalent changes to dpkg-buildpackage. Why
we should introduce new tool?

--
BR,
Ed
_______________________________________________
maemo-developers mailing list
maemo-developers [at] maemo
https://lists.maemo.org/mailman/listinfo/maemo-developers

bartosh at gmail

Nov 1, 2009, 1:02 AM

Post #24 of 88 (1718 views)
Permalink

Re: maemo-optify, autobuilder & /opt [In reply to]

2009/10/29 Graham Cobb <g+770 [at] cobb>:
> On Wednesday 28 October 2009 22:50:25 Ed Bartosh wrote:
>> Somehow I don't like the idea of doing anything with the package
>> without developer being aware of this. I'd rather implement check on
>> autobuilder side to insure that packages are optified. Developer can
>> use option XS-Maemo-Optify: none to disable optification if developer
>> doesn't want it.
>
> Nobody likes doing something to the package automatically but, after a long
> discussion at the BOF, we agreed that the alternatives were even worse [1].
>
> In particular, there was a strong argument that the package should not have to
> include anything (even a control field option) to cause optification to
> happen. Packages which wanted to do their own optification or which had to
> disable optification would have to include an option to stop optification.
>
> If it makes you happier: rename the autobuilder as "autobuilder and optifier"!
>
> We did agree that there had to be a way for developers to generate packages in
> the scratchbox environment which had been optified in exactly the same way
> the autobuilder would. And that we would update the wiki pages about
> checking packages will build to include using that tool to test the
> optification.
>
> So, the consensus decision was that the solution would be that autobuilder
> should automatically optify by default.
>

So, what should we do?

My proposal is to make dpkg-buildpackage to call maemo-optify. With
this we can solve 2 problems - autobuilder will optify packages and
developers will have their packages automatically optified for their
local builds without using any extra tools.

--
BR,
Ed
_______________________________________________
maemo-developers mailing list
maemo-developers [at] maemo
https://lists.maemo.org/mailman/listinfo/maemo-developers

g+770 at cobb

Nov 1, 2009, 3:05 AM

Post #25 of 88 (1723 views)
Permalink

Re: maemo-optify, autobuilder & /opt [In reply to]

On Sunday 01 November 2009 09:02:34 Ed Bartosh wrote:
> So, what should we do?
>
> My proposal is to make dpkg-buildpackage to call maemo-optify. With
> this we can solve 2 problems - autobuilder will optify packages and
> developers will have their packages automatically optified for their
> local builds without using any extra tools.

I am happy to go with your recommendation. I guess there is a small downside
that we are forking dpkg-buildpackage but, as you say, the advantage is that
developers will not have to do anything locally.

Graham
_______________________________________________
maemo-developers mailing list
maemo-developers [at] maemo
https://lists.maemo.org/mailman/listinfo/maemo-developers

1 2 3 4

View All

Index | Next | Previous | View Threaded

Interested in having your list archived? Contact Gossamer Threads