Mailing List Archive: Maemo: Developers

Autobuilder repository priority ?

1 2

View All

Index | Next | Previous | View Threaded

maemo at csipa

Oct 31, 2009, 4:45 AM

Post #1 of 33 (1497 views)
Permalink

Autobuilder repository priority ?

I have a small issue with the autobuilder. The whole thing started out by
having a package that compiled nice in the SDK but not in the autobuilder due
to a versioning mismatch (in my case python-dbus, but it's a generic problem
as you'll see). After some snooping around, I realized the problem was that
the SDK had 0.83.0-1maemo1 in fremantle/tools/free (and that's what got used
when compiling in the SDK), however, the autobuilder insists on using
0.83.0-1maemo3 from fremantle/free (-devel). The problem is IMHO that
the repository priorities seem to be wrong. The autobuilder should be using
the highest version in the TOP PRIORITY repository that satisfies a dependency
to avoid breakage because of unstable stuff in -devel and because otherwise a
package can't be promoted without dragging other folks' packages with them.
Thoughts ?

Regards,
Attila
_______________________________________________
maemo-developers mailing list
maemo-developers [at] maemo
https://lists.maemo.org/mailman/listinfo/maemo-developers

bartosh at gmail

Oct 31, 2009, 6:49 AM

Post #2 of 33 (1458 views)
Permalink

Re: Autobuilder repository priority ? [In reply to]

2009/10/31 Attila Csipa <maemo [at] csipa>:
> I have a small issue with the autobuilder. The whole thing started out by
> having a package that compiled nice in the SDK but not in the autobuilder due
> to a versioning mismatch (in my case python-dbus, but it's a generic problem
> as you'll see). After some snooping around, I realized the problem was that
> the SDK had 0.83.0-1maemo1 in fremantle/tools/free (and that's what got used
> when compiling in the SDK), however, the autobuilder insists on using
> 0.83.0-1maemo3 from fremantle/free (-devel). The problem is IMHO that
> the repository priorities seem to be wrong. The autobuilder should be using
> the highest version in the TOP PRIORITY repository that satisfies a dependency
> to avoid breakage because of unstable stuff in -devel and because otherwise a
> package can't be promoted without dragging other folks' packages with them.
> Thoughts ?
>
The reason of this is that apt designed so that it will always install
higher possible version of package. It's actually rather good than
bad. Imagine what would happen if it would work another way. No new
python-dbus ever :)

The solution for this is to go to bugs.maemo.org and report your
problem to Pymaemo project and wait for a fix. Or, better, to provide
patch. You can also discuss this with pymaemo developers on their irc
channel or mailing list:
http://pymaemo.garage.maemo.org/getinvolved.html

--
BR,
Ed
_______________________________________________
maemo-developers mailing list
maemo-developers [at] maemo
https://lists.maemo.org/mailman/listinfo/maemo-developers

g+770 at cobb

Oct 31, 2009, 7:08 AM

Post #3 of 33 (1457 views)
Permalink

Re: Autobuilder repository priority ? [In reply to]

On Saturday 31 October 2009 11:45:54 Attila Csipa wrote:
>The problem is IMHO
> that the repository priorities seem to be wrong. The autobuilder should be
> using the highest version in the TOP PRIORITY repository that satisfies a
> dependency to avoid breakage because of unstable stuff in -devel and
> because otherwise a package can't be promoted without dragging other folks'
> packages with them. Thoughts ?

The current behaviour is by design, but that doesn't stop us reconsidering
whether it is still the best approach!

Part of the original intent of the current behaviour was to allow the
community to upgrade packages which are present in the SDK. Of course, since
Diablo (I think) Nokia has taken steps to prevent the community from
upgrading packages which are installed on the device but it still works for
things which are in the SDK but not on the device. However, I'm not sure if
that is still useful.

It is useful behaviour for things which are not in the SDK at all. When we
first created extras-devel one of the problems it was created to solve was
that a library maintainer would introduce a new version of a library without
applications being tested to make sure they still work with the new library
(and I think, back in the Maemo 1/2 days, that there were some actual problem
cases). So part of the point of extras-devel was to allow application
developers to pick up the latest libraries -- and all developers were
expected to use extras-devel in their ordinary scratchbox builds so they were
always building and testing against the latest libraries. At that time,
extras-devel was not intended to be a "just see if it compiles and the
package builds" type of repository -- it was expected that developers had
done some testing and thought that applications and, particularly, libraries
were ready for wider use, at least among the developer community. By the
way, at that time, most developers had a test repository to allow them to do
things like test installation before putting it in extras-devel.

When the autobuilder and, later, promotion were created it was deliberate that
dependencies would be promoted so that the latest version (the one the
application developer had tested with) would be promoted. The downside, of
course, is that the library developer may have found a bug in their new
library and not want it promoted!

It may be that, with a larger community, this needs to be re-thought. Let's
reconsider how we want the repositories (and related issues like promotions)
to work.

Graham
_______________________________________________
maemo-developers mailing list
maemo-developers [at] maemo
https://lists.maemo.org/mailman/listinfo/maemo-developers

maemo at csipa

Oct 31, 2009, 11:11 AM

Post #4 of 33 (1443 views)
Permalink

Re: Autobuilder repository priority ? [In reply to]

On Saturday 31 October 2009 14:49:24 Ed Bartosh wrote:
> > is IMHO that the repository priorities seem to be wrong. The autobuilder
> > should be using the highest version in the TOP PRIORITY repository that
> > satisfies a dependency to avoid breakage because of unstable stuff in
> > -devel and because otherwise a package can't be promoted without dragging
> > other folks' packages with them. Thoughts ?
>
> The reason of this is that apt designed so that it will always install
> higher possible version of package. It's actually rather good than
> bad. Imagine what would happen if it would work another way. No new
> python-dbus ever :)

This is not entirely correct (and definitely not recommended in our case of the
autobuilder). Apt uses priorities (both repository and package level
priorities can be defined, 'pinned') to decide which package to use to satisfy
a dependency, and only if the priorities are equal does it base the choice on
version number.

Why is the current (same-priority, version only) approach bad ?

Say, my app depends on python-dbus. I test it with version A, currently in the
SDK and it works just fine. I upload and promote my app. All is well. However,
later, the pymaemo team uploads a newer, experimental B version into -devel.
From that point on I am NO LONGER ABLE to reliably update my application nor
promote it (this actually happened to me, my stuff compiles in SDK, but not in
the autobuilder for this very reason).

But you can specify a fixed version dependency (the one in SDK and extras), you
say ? But THEN I get to the problem of no updates. If bugs get squashed in the
-devel version of the library I'm using, the user will never be able to
upgrade to it, as my package is insisting on the old version (and the package
manager wisely rather skips updates than removing already installed packages).

Ubuntu and Debian solve this problem by advising different repository
priorities. Stable has a highest priority, Testing is lower and Unstable is
lowest. If there is a package in Stable that satisfies my requirement, that's
the one that should be used. But then it will never get updated, you say ?
Wrong ! Remember what I said up there - if the priority is the same, the
version decides. So, the moment a new version of the dependency reaches
*Stable*, it will become the highest priority and will get updated. It's a
recommendation that dependencies should be REALISTICALLY listed. You should
not depend on a high version number just because it's new or even because it's
the one bundled on the system (!). You should depend on the minimal version
that provides the actual functionality you require.

Hope my problem is a bit more clear now.

Regards,
Attila

_______________________________________________
maemo-developers mailing list
maemo-developers [at] maemo
https://lists.maemo.org/mailman/listinfo/maemo-developers

bartosh at gmail

Oct 31, 2009, 3:35 PM

Post #5 of 33 (1446 views)
Permalink

Re: Autobuilder repository priority ? [In reply to]

2009/10/31 Attila Csipa <maemo [at] csipa>:
> On Saturday 31 October 2009 14:49:24 Ed Bartosh wrote:
>> > is IMHO that the repository priorities seem to be wrong. The autobuilder
>> > should be using the highest version in the TOP PRIORITY repository that
>> > satisfies a dependency to avoid breakage because of unstable stuff in
>> > -devel and because otherwise a package can't be promoted without dragging
>> > other folks' packages with them. Thoughts ?
>>
>> The reason of this is that apt designed so that it will always install
>> higher possible version of package. It's actually rather good than
>> bad. Imagine what would happen if it would work another way. No new
>> python-dbus ever :)
>
> This is not entirely correct (and definitely not recommended in our case of the
> autobuilder). Apt uses priorities (both repository and package level
> priorities can be defined, 'pinned') to decide which package to use to satisfy
> a dependency, and only if the priorities are equal does it base the choice on
> version number.
>
> Why is the current (same-priority, version only) approach bad ?
>
> Say, my app depends on python-dbus. I test it with version A, currently in the
> SDK and it works just fine. I upload and promote my app. All is well. However,
> later, the pymaemo team uploads a newer, experimental B version into -devel.
> From that point on I am NO LONGER ABLE to reliably update my application nor
> promote it (this actually happened to me, my stuff compiles in SDK, but not in
> the autobuilder for this very reason).
>
True.

However there can be another situation: SDK version of python-dbus is
buggy and you can't use it with your application. You would have to
wait until new fixed version of python-dbus reaches high priority repo
to be able to build your application with it. Not so good for
application developer, either. I don't want to say that current
uproach is the best, just want to show you another side of the
problem.

> But you can specify a fixed version dependency (the one in SDK and extras), you
> say ?
> But THEN I get to the problem of no updates. If bugs get squashed in the
> -devel version of the library I'm using, the user will never be able to
> upgrade to it, as my package is insisting on the old version (and the package
> manager wisely rather skips updates than removing already installed packages).
>
> Ubuntu and Debian solve this problem by advising different repository
> priorities. Stable has a highest priority, Testing is lower and Unstable is
> lowest. If there is a package in Stable that satisfies my requirement, that's
> the one that should be used.
Are you sure it works this way? I thought that packages are built with
dependencies from unstable in Debian, just like they're built against
extras-devel in Maemo.

> But then it will never get updated, you say ?
> Wrong ! Remember what I said up there - if the priority is the same, the
> version decides. So, the moment a new version of the dependency reaches
> *Stable*, it will become the highest priority and will get updated. It's a
> recommendation that dependencies should be REALISTICALLY listed. You should
> not depend on a high version number just because it's new or even because it's
> the one bundled on the system (!). You should depend on the minimal version
> that provides the actual functionality you require.
>
> Hope my problem is a bit more clear now.
>
It was clear from your previous mail.

Actually python-dbus is not very good example. It's not only SDK
package. I might be wrong but I think it's included into PyMaemo
releases and is delivered through Extras. Nokia included it into SDK,
but this is a special case.

SDK packages shouldn't be uploaded to extras-devel at all. There are
plans to implement checks on autobuilder side to prevent this.

--
BR,
Ed
_______________________________________________
maemo-developers mailing list
maemo-developers [at] maemo
https://lists.maemo.org/mailman/listinfo/maemo-developers

maemo at csipa

Oct 31, 2009, 6:20 PM

Post #6 of 33 (1441 views)
Permalink

Re: Autobuilder repository priority ? [In reply to]

On Saturday 31 October 2009 23:35:08 you wrote:
> > Ubuntu and Debian solve this problem by advising different repository
> > priorities. Stable has a highest priority, Testing is lower and Unstable
> > is lowest. If there is a package in Stable that satisfies my requirement,
> > that's the one that should be used.
>
> Are you sure it works this way? I thought that packages are built with
> dependencies from unstable in Debian, just like they're built against
> extras-devel in Maemo.

You're right you can't change pinning on the *builder* within the *same*
queue, that would make no sense. The problem is that the autobuilder already
uses a mix of repositories (as you said later, having packages from extras-*
overriding SDK stuff is not right) and that we can't skip the promotion process
(i.e. no security.debian.org that is handled differently). If I was unclear as
to the goal - I don't want to cross-reference repo contents (bad, bad idea). I
want to be able to issue updates to packages that have been already promoted
and got to the general public (obviously not typo fixes but stuff that is REALLY
important), and for that, we need a queue that has a different repository
layout (i.e. no extras-devel overriding extras).

> Actually python-dbus is not very good example. It's not only SDK
> package. I might be wrong but I think it's included into PyMaemo
> releases and is delivered through Extras. Nokia included it into SDK,
> but this is a special case.

I just ran into this problem personally with python-dbus, hence the mention,
but I did say the problem is more generic than that :)

Regards,
Attila
_______________________________________________
maemo-developers mailing list
maemo-developers [at] maemo
https://lists.maemo.org/mailman/listinfo/maemo-developers

bartosh at gmail

Nov 1, 2009, 12:20 AM

Post #7 of 33 (1427 views)
Permalink

Re: Autobuilder repository priority ? [In reply to]

2009/11/1 Attila Csipa <maemo [at] csipa>:
>>
>> Are you sure it works this way? I thought that packages are built with
>> dependencies from unstable in Debian, just like they're built against
>> extras-devel in Maemo.
>
> You're right you can't change pinning on the *builder* within the *same*
> queue, that would make no sense. The problem is that the autobuilder already
> uses a mix of repositories (as you said later, having packages from extras-*
> overriding SDK stuff is not right) and that we can't skip the promotion process
> (i.e. no security.debian.org that is handled differently). If I was unclear as
> to the goal - I don't want to cross-reference repo contents (bad, bad idea). I
> want to be able to issue updates to packages that have been already promoted
> and got to the general public (obviously not typo fixes but stuff that is REALLY
> important), and for that, we need a queue that has a different repository
> layout (i.e. no extras-devel overriding extras).
>
So, you propose to have one more queue, which would use only SDK? Or
only Extras? or both? Sorry, your proposal is still unclear to me and
I doubt it would be clear for other devs.

>> Actually python-dbus is not very good example. It's not only SDK
>> package. I might be wrong but I think it's included into PyMaemo
>> releases and is delivered through Extras. Nokia included it into SDK,
>> but this is a special case.
>
> I just ran into this problem personally with python-dbus, hence the mention,
> but I did say the problem is more generic than that :)
>
I agree that the problem exists. Let's find possible solutions for it.
I still think that the one I proposed is the fastest and simplest for now.

--
BR,
Ed
_______________________________________________
maemo-developers mailing list
maemo-developers [at] maemo
https://lists.maemo.org/mailman/listinfo/maemo-developers

bartosh at gmail

Nov 1, 2009, 12:23 AM

Post #8 of 33 (1428 views)
Permalink

Re: Autobuilder repository priority ? [In reply to]

maemo at csipa

Nov 1, 2009, 2:23 AM

Post #9 of 33 (1436 views)
Permalink

Re: Autobuilder repository priority ? [In reply to]

On Sunday 01 November 2009 08:23:13 Ed Bartosh wrote:
> You can try to use python2.5-dbus instead of python-dbus to work
> around the problem. python2.5-dbus is just a meta package, but it
> presents only in SDK, which might allow you to avoid installing
> python-dbus from extras-devel.

Been there, done that (actually, that was the original dependency from the
Diablo days :) ). That doesn't work as python2.5-dbus is essentially

Depends: python-dbus (= 0.83.0-1maemo1)

and when I put that through the autobuilder, It gives a

Python2.5-dbus: requires python-dbus (= 0.83.0-1maemo1), but it is not going
to be installed.

Most likely there is another package pulled in that has plain python-dbus as
dependency or has a set version to depend on, hence the conflict.

Note that the two versions (-1maemo1 and -1maemo3) have some level of
incompatibility (some paths changed) so I cannot compile with one and then use
with the other. I updated the package to use the version in -devel, but for
that reason now I cannot promote it to extras and testing. I've been told
python-dbus is going to be promoted to testing next week, but that only added
to my confusion - while solving this particular problem, will that mean that
there will be permanently packages in Extras overshadowing SDK/official packages
? I would expect (semi-)official packages shipped via SDK to be updated through
the same (SDK/Nokia) repositories (or a specific updates repo).

Regards,
Attila
_______________________________________________
maemo-developers mailing list
maemo-developers [at] maemo
https://lists.maemo.org/mailman/listinfo/maemo-developers

maemo at csipa

Nov 1, 2009, 2:51 AM

Post #10 of 33 (1423 views)
Permalink

Re: Autobuilder repository priority ? [In reply to]

On Sunday 01 November 2009 07:46:55 you wrote:
> So, you propose to have one more queue, which would use only SDK? Or
> only Extras? or both? Sorry, your proposal is still unclear to me and
> I doubt it would be clear for other devs.

First we need to decide on whether Extras packages can update packages from
the SDK/official repos. I'd say no. In that case, go the Debian way and have an
sdk-updates and extras-updates queue with a different QA procedure. Ideally,
these would function akin to the Debian security team (not sure we have the
manpower for that, though - ideas how to solve this welcome). See

http://people.debian.org/~skx/security/team.html and
http://people.debian.org/~skx/security/dsa.html

In case of overlapping SDK/Extras I can't think of a satisfactory solution as
then fixes for the latter would appear as fixes for the former, which is wrong
and dangerous. Another issue would be that you would not be getting SDK
updates if you have extras or extras-updates disabled, which is very
counterintuitive.
_______________________________________________
maemo-developers mailing list
maemo-developers [at] maemo
https://lists.maemo.org/mailman/listinfo/maemo-developers

bartosh at gmail

Nov 1, 2009, 9:05 AM

Post #11 of 33 (1421 views)
Permalink

Re: Autobuilder repository priority ? [In reply to]

2009/11/1 Attila Csipa <maemo [at] csipa>:
> On Sunday 01 November 2009 07:46:55 you wrote:
>> So, you propose to have one more queue, which would use only SDK? Or
>> only Extras? or both? Sorry, your proposal is still unclear to me and
>> I doubt it would be clear for other devs.
>
> First we need to decide on whether Extras packages can update packages from
> the SDK/official repos.
It depends on type of packages. SDK contains 2 parts: developer tools
& libs, which are not installed on the device and libs & apps, which
are installed on the device.
For first group updating through Extras is not good, but possible and
most of the time it would work faster than through SDK update
process(if it exists at all).
Packages from second group can break SSU if they they're considered as
a system packages. Actually, application manager doesn't allow to
install them, but if installed using dpkg they can break SSU.

Another concern here is that SDK and device firmware usually released
together, but packages in Extras are not synchronized with those
releases until they appear. It practically means that it's better to
avoid overlapping SDK/Extras as much as possible. However it's still
unclear to me what to do with components taken into SDK from projects
similar to PyMaemo with their own release circle and distributing
scheme.

> I'd say no. In that case, go the Debian way and have an
> sdk-updates and extras-updates queue with a different QA procedure.
Maemo isn't Debian. Debian has consistent repository one for
everything. Maemo has at least 3 different components(SDK, Extras,
device packages) and 2 different operation environments(scratchbox and
device). SDK has never been released through Extras, so separate queue
for SDK updates should be agreed with SDK team. I doubt they need it.

Idea of having separate queue for Extras updates sounds more promising
to me. Developers might become confused with all this set of
repositories, queues and processes, but the idea is good. I support
this.

>
> In case of overlapping SDK/Extras I can't think of a satisfactory solution as
> then fixes for the latter would appear as fixes for the former, which is wrong
> and dangerous. Another issue would be that you would not be getting SDK
> updates if you have extras or extras-updates disabled, which is very
> counterintuitive.
>
True. I also don't see any more or less acceptable solution for this.

--
BR,
Ed
_______________________________________________
maemo-developers mailing list
maemo-developers [at] maemo
https://lists.maemo.org/mailman/listinfo/maemo-developers

jeremiah at jeremiahfoster

Nov 1, 2009, 3:16 PM

Post #12 of 33 (1419 views)
Permalink

Re: Autobuilder repository priority ? [In reply to]

On Nov 1, 2009, at 18:05, Ed Bartosh wrote:

> 2009/11/1 Attila Csipa <maemo [at] csipa>:
>> On Sunday 01 November 2009 07:46:55 you wrote:
>>> So, you propose to have one more queue, which would use only SDK? Or
>>> only Extras? or both? Sorry, your proposal is still unclear to me
>>> and
>>> I doubt it would be clear for other devs.
>>
>> First we need to decide on whether Extras packages can update
>> packages from
>> the SDK/official repos.

I don't think any Extras packages should update official SDK repos.

> It depends on type of packages. SDK contains 2 parts: developer tools
> & libs, which are not installed on the device and libs & apps, which
> are installed on the device.

Can't we have a monolithic repo which is _identical_ to the device,
plus dev tools? This would allow developers to know in advance which
dependencies are on the device and which dependencies they will have
to pull in themselves.

> For first group updating through Extras is not good, but possible and
> most of the time it would work faster than through SDK update
> process(if it exists at all).
> Packages from second group can break SSU if they they're considered as
> a system packages. Actually, application manager doesn't allow to
> install them, but if installed using dpkg they can break SSU.
>
> Another concern here is that SDK and device firmware usually released
> together, but packages in Extras are not synchronized with those
> releases until they appear. It practically means that it's better to
> avoid overlapping SDK/Extras as much as possible. However it's still
> unclear to me what to do with components taken into SDK from projects
> similar to PyMaemo with their own release circle and distributing
> scheme.

I'd say that a lot of the Maemo release is still driven by corporate
Nokia goals which Maemo.org people and the community have to implement
the community tools to co-incide with that. This is really hard to do.
I think that those in charge of this system have done a great job, but
it would be much easier for them and for developers if more of the SDK
and device software could be maintained by the community. This is how
it is done in debian, and debian is a production quality operating
system. Doing it half-open, half-closed is not scaling well -
developers need greater insight into the dependency chain, the build
tools, and the device software - otherwise Nokia is asking them to
develop with one eye.

>
>> I'd say no. In that case, go the Debian way and have an
>> sdk-updates and extras-updates queue with a different QA procedure.
> Maemo isn't Debian. Debian has consistent repository one for
> everything. Maemo has at least 3 different components(SDK, Extras,
> device packages) and 2 different operation environments(scratchbox and
> device). SDK has never been released through Extras, so separate queue
> for SDK updates should be agreed with SDK team. I doubt they need it.

This is what makes things so hard for developers.
>
> Idea of having separate queue for Extras updates sounds more promising
> to me. Developers might become confused with all this set of
> repositories, queues and processes, but the idea is good. I support
> this.
>
>>
>> In case of overlapping SDK/Extras I can't think of a satisfactory
>> solution as
>> then fixes for the latter would appear as fixes for the former,
>> which is wrong
>> and dangerous. Another issue would be that you would not be getting
>> SDK
>> updates if you have extras or extras-updates disabled, which is very
>> counterintuitive.
>>
> True. I also don't see any more or less acceptable solution for this.

Having a single repo for each distro would be ideal, with _all_ the
software required to run the device in the repo. Then we can co-
ordinate the release as a single, monolithic repository. I know this
is wishful thinking because not everything is licensed to allow this,
but if we could get as much as possible in a single location, then we
make life easier for developers.

Jeremiah
_______________________________________________
maemo-developers mailing list
maemo-developers [at] maemo
https://lists.maemo.org/mailman/listinfo/maemo-developers

bartosh at gmail

Nov 2, 2009, 1:27 AM

Post #13 of 33 (1400 views)
Permalink

Re: Autobuilder repository priority ? [In reply to]

2009/11/2 Jeremiah Foster <jeremiah [at] jeremiahfoster>:
>
> On Nov 1, 2009, at 18:05, Ed Bartosh wrote:
>
>> 2009/11/1 Attila Csipa <maemo [at] csipa>:
>>> On Sunday 01 November 2009 07:46:55 you wrote:
>>>> So, you propose to have one more queue, which would use only SDK? Or
>>>> only Extras? or both? Sorry, your proposal is still unclear to me
>>>> and
>>>> I doubt it would be clear for other devs.
>>>
>>> First we need to decide on whether Extras packages can update
>>> packages from
>>> the SDK/official repos.
>
> I don't think any Extras packages should update official SDK repos.
>
>> It depends on type of packages. SDK contains 2 parts: developer tools
>> & libs, which are not installed on the device and libs & apps, which
>> are installed on the device.
>
> Can't we have a monolithic repo which is _identical_ to the device,
> plus dev tools? This would allow developers to know in advance which
> dependencies are on the device and which dependencies they will have
> to pull in themselves.
>
I don't know. Can we?

>> Idea of having separate queue for Extras updates sounds more promising
>> to me. Developers might become confused with all this set of
>> repositories, queues and processes, but the idea is good. I support
>> this.
>>
What about this? Can we have separate queue for updates? Any other ideas?

>>>
>>> In case of overlapping SDK/Extras I can't think of a satisfactory
>>> solution as
>>> then fixes for the latter would appear as fixes for the former,
>>> which is wrong
>>> and dangerous. Another issue would be that you would not be getting
>>> SDK
>>> updates if you have extras or extras-updates disabled, which is very
>>> counterintuitive.
>>>
>> True. I also don't see any more or less acceptable solution for this.
>
> Having a single repo for each distro would be ideal, with _all_ the
> software required to run the device in the repo. Then we can co-
> ordinate the release as a single, monolithic repository. I know this
> is wishful thinking because not everything is licensed to allow this,
> but if we could get as much as possible in a single location, then we
> make life easier for developers.
>
Sounds like a long-term plan.
Attila was asking us to solve problem with updates. And he proposed
good solution. Shell we implement it right now or you propose to wait
until we have Maemo distro and all this great things?

--
BR,
Ed
_______________________________________________
maemo-developers mailing list
maemo-developers [at] maemo
https://lists.maemo.org/mailman/listinfo/maemo-developers

jeremiah at jeremiahfoster

Nov 2, 2009, 1:52 PM

Post #14 of 33 (1394 views)
Permalink

Re: Autobuilder repository priority ? [In reply to]

On Nov 2, 2009, at 10:27, Ed Bartosh wrote:

> 2009/11/2 Jeremiah Foster <jeremiah [at] jeremiahfoster>:
>>
>> On Nov 1, 2009, at 18:05, Ed Bartosh wrote:
>
>>> Idea of having separate queue for Extras updates sounds more
>>> promising
>>> to me. Developers might become confused with all this set of
>>> repositories, queues and processes, but the idea is good. I support
>>> this.
>>>
> What about this? Can we have separate queue for updates? Any other
> ideas?

Can you explain how this might work and it's advantages? I am not
against it aside from the fact that I think another repo is confusing.
Is the proposal to create a repo called extras-updates which would
hold only updates to software that has already existed in the repos? I
don't see how that is different from just updating the existing
package with new software. If you want to pull in different version
numbers than what exists why would you need a separate repo?

I think this is adding complexity - and we are in danger of having too
much complexity already.

>>>>
>>>> In case of overlapping SDK/Extras I can't think of a satisfactory
>>>> solution as
>>>> then fixes for the latter would appear as fixes for the former,
>>>> which is wrong
>>>> and dangerous. Another issue would be that you would not be getting
>>>> SDK
>>>> updates if you have extras or extras-updates disabled, which is
>>>> very
>>>> counterintuitive.
>>>>
>>> True. I also don't see any more or less acceptable solution for
>>> this.
>>
>> Having a single repo for each distro would be ideal, with _all_ the
>> software required to run the device in the repo. Then we can co-
>> ordinate the release as a single, monolithic repository. I know this
>> is wishful thinking because not everything is licensed to allow this,
>> but if we could get as much as possible in a single location, then we
>> make life easier for developers.
>>
> Sounds like a long-term plan.

> Attila was asking us to solve problem with updates. And he proposed
> good solution. Shell we implement it right now or you propose to wait
> until we have Maemo distro and all this great things?

No, let's try and solve his proposal now. I am open to whatever people
think is best, I don't want to stand in the way of a new repository if
others think it is the right path to choose.

Jeremiah

_______________________________________________
maemo-developers mailing list
maemo-developers [at] maemo
https://lists.maemo.org/mailman/listinfo/maemo-developers

jeremiah at jeremiahfoster

Nov 2, 2009, 2:24 PM

Post #15 of 33 (1397 views)
Permalink

Re: Autobuilder repository priority ? [In reply to]

g+770 at cobb

Nov 2, 2009, 3:02 PM

Post #16 of 33 (1406 views)
Permalink

Re: Autobuilder repository priority ? [In reply to]

On Monday 02 November 2009 21:52:48 Jeremiah Foster wrote:
> On Nov 2, 2009, at 10:27, Ed Bartosh wrote:
> > 2009/11/2 Jeremiah Foster <jeremiah [at] jeremiahfoster>:
> >> On Nov 1, 2009, at 18:05, Ed Bartosh wrote:
> >>> Idea of having separate queue for Extras updates sounds more
> >>> promising
> >>> to me. Developers might become confused with all this set of
> >>> repositories, queues and processes, but the idea is good. I support
> >>> this.
> >
> > What about this? Can we have separate queue for updates? Any other
> > ideas?
>
> Can you explain how this might work and it's advantages? I am not
> against it aside from the fact that I think another repo is confusing.
> Is the proposal to create a repo called extras-updates which would
> hold only updates to software that has already existed in the repos? I
> don't see how that is different from just updating the existing
> package with new software. If you want to pull in different version
> numbers than what exists why would you need a separate repo?

Sorry, like Jeremiah I am now a bit confused. Can you, Ed, please summarise
what this proposal is? You mention separate queue but which queue are you
referring to? Does the suggestion involve additional repositories? And/or
does it involve differnet rules for which repositories will be in scope
during a build? Or what.

Sorry if everyone else understands this but I don't.

> I think this is adding complexity - and we are in danger of having too
> much complexity already.

Definitely a point worth bearing in mind!

Graham
_______________________________________________
maemo-developers mailing list
maemo-developers [at] maemo
https://lists.maemo.org/mailman/listinfo/maemo-developers

maemo at csipa

Nov 2, 2009, 4:17 PM

Post #17 of 33 (1397 views)
Permalink

Re: Autobuilder repository priority ? [In reply to]

A bit short on time, so could not reply to many good posts, but would not like
to drop out of the discussion...

On Monday 02 November 2009 23:24:12 Jeremiah Foster wrote:
> > Attila was asking us to solve problem with updates. And he proposed
> > good solution. Shell we implement it right now or you propose to wait
> > until we have Maemo distro and all this great things?
>
> No, let's try and solve his proposal now. I am open to whatever people
> think is best, I don't want to stand in the way of a new repository if
> others think it is the right path to choose.

Just to clarify things a bit - the additional repo was a suggested solution to
deeper problem(s) of overlapping repositories and fixed devel-testing-extras
promotion path. I'd much rather have those solved at the origin (cleanly
separated SDK/extras and ways to update them without one polluting the other),
but in case that is not possible (that's a LOT of policies/people/packages to
kick around), there's the repo suggestion. I'm not emotionally tied to extra
repos, better solutions are welcome :)

>Can you explain how this might work and it's advantages? I am not
>against it aside from the fact that I think another repo is confusing.
>is the proposal to create a repo called extras-updates which would
>hold only updates to software that has already existed in the repos? I
> don't see how that is different from just updating the existing
> package with new software. If you want to pull in different version
> numbers than what exists why would you need a separate repo?

The problem is that newer versions of dependencies from extras-devel can be
incompatible with those currently in extras (or plain broken), and that leads
to a dependency mess (you can't get it to extras as it won't build with the -
devel dependency, and if you change it to build, then you have to promote
other peoples dependencies as well). Remember that python2.5-dbus error I was
asking about a few days back ? Well, it's one of the packages that is able to
cause such fallout :) In plain English it means that currently applications
depending on python2.5-dbus in extras are cut off from being updated.

There is also the twist that we don't really have 'stable' in the Debian sense
of the repo. Extras is much more like 'testing' and Extras-testing is
(kindasorta) like NEW. Of course, I'm is saying we have to stick to Debian's
approach, especially as the Maemo SW lifecycle is quite a bit different, but we
do have to have mechanisms that can serve as a functional equivalents - for
(security) updates, QA, staging, etc.

_______________________________________________
maemo-developers mailing list
maemo-developers [at] maemo
https://lists.maemo.org/mailman/listinfo/maemo-developers

maemo at csipa

Nov 2, 2009, 5:26 PM

Post #18 of 33 (1390 views)
Permalink

Re: Autobuilder repository priority ? [In reply to]

On Tuesday 03 November 2009 00:02:31 Graham Cobb wrote:
> Sorry if everyone else understands this but I don't.

We have two (in my particular case slightly related) problems.

1) The fact that something builds in the pristine final SDK does not guarantee
that it will go through the autobuilder (as extras-devel might contain non-
backwards compatible updates to the SDK packages).

2) We seem to have an 'interesting' possibility, where, due to current repo
layout and promotion policies, you can get into a situation in which you are
not able to promote your own application update to Extras because of other
people's development activity in Extras-devel (especially nasty if you were to
make an important security fix).

We don't seem to have a final proposal (yet), looking for solutions, adding an
extra repo for updates/staging is just one of them.

Regards,
Attila

_______________________________________________
maemo-developers mailing list
maemo-developers [at] maemo
https://lists.maemo.org/mailman/listinfo/maemo-developers

bartosh at gmail

Nov 3, 2009, 12:25 AM

Post #19 of 33 (1384 views)
Permalink

Re: Autobuilder repository priority ? [In reply to]

2009/11/3 Graham Cobb <g+770 [at] cobb>:
> On Monday 02 November 2009 21:52:48 Jeremiah Foster wrote:
>> On Nov 2, 2009, at 10:27, Ed Bartosh wrote:
>> > 2009/11/2 Jeremiah Foster <jeremiah [at] jeremiahfoster>:
>> >> On Nov 1, 2009, at 18:05, Ed Bartosh wrote:
>> >>> Idea of having separate queue for Extras updates sounds more
>> >>> promising
>> >>> to me. Developers might become confused with all this set of
>> >>> repositories, queues and processes, but the idea is good. I support
>> >>> this.
>> >
>> > What about this? Can we have separate queue for updates? Any other
>> > ideas?
>>
>> Can you explain how this might work and it's advantages? I am not
>> against it aside from the fact that I think another repo is confusing.
>> Is the proposal to create a repo called extras-updates which would
>> hold only updates to software that has already existed in the repos? I
>> don't see how that is different from just updating the existing
>> package with new software. If you want to pull in different version
>> numbers than what exists why would you need a separate repo?
>
> Sorry, like Jeremiah I am now a bit confused. �Can you, Ed, please summarise
> what this proposal is? �You mention separate queue but which queue are you
> referring to? �Does the suggestion involve additional repositories? And/or
> does it involve differnet rules for which repositories will be in scope
> during a build? �Or what.
>
As I understood Attila he proposed to create separate autobulder
incoming queue for Extras updates.
If packages are uploaded to this queue they're built using only Extras
and SDK repos and put into extras-devel.
As a result they won't depend on unstable packages from Extras-devel
and can go to extras-testing and then to Extras faster.

As I already mentioned I'm also a bit afraid of extra complexity and
possible confusion, but I think this idea at least worth to be
discussed.

--
BR,
Ed
_______________________________________________
maemo-developers mailing list
maemo-developers [at] maemo
https://lists.maemo.org/mailman/listinfo/maemo-developers

jeremiah at jeremiahfoster

Nov 3, 2009, 5:08 AM

Post #20 of 33 (1372 views)
Permalink

Re: Autobuilder repository priority ? [In reply to]

On Nov 3, 2009, at 1:17, Attila Csipa wrote:

> A bit short on time, so could not reply to many good posts, but
> would not like
> to drop out of the discussion...

This is a very important discussion I feel, you have raised some
important issues and there has been some insightful feedback from
people like Ed Bartosh who really know the build toolchain well.

>
> On Monday 02 November 2009 23:24:12 Jeremiah Foster wrote:
>>> Attila was asking us to solve problem with updates. And he proposed
>>> good solution. Shell we implement it right now or you propose to
>>> wait
>>> until we have Maemo distro and all this great things?
>>
>> No, let's try and solve his proposal now. I am open to whatever
>> people
>> think is best, I don't want to stand in the way of a new repository
>> if
>> others think it is the right path to choose.
>
> Just to clarify things a bit - the additional repo was a suggested
> solution to
> deeper problem(s) of overlapping repositories and fixed devel-
> testing-extras
> promotion path. I'd much rather have those solved at the origin
> (cleanly
> separated SDK/extras and ways to update them without one polluting
> the other),

Yes. As I see it, this might be the core of the problem: overlapping
repositories preventing package promotion. For example, if python2.5-
dbus is a 'depends', you cannot build the package for Extras.

Am I understanding this correctly?

Can you use another dependency to get the same functionality?
Can you declare in your control file that a specific version of
python2.5-dbus be used?
Can we change the conflict that is preventing python2.5-dbus from
being used?

My hunch is that there may be a way to solve this specific case with
the tools we now have. If we can't we have to look at how we fix it
and like Graham says we may be able to find our own path and maybe an
update repository is the way to go.

Jeremiah
_______________________________________________
maemo-developers mailing list
maemo-developers [at] maemo
https://lists.maemo.org/mailman/listinfo/maemo-developers

jeremiah at jeremiahfoster

Nov 3, 2009, 5:17 AM

Post #21 of 33 (1363 views)
Permalink

Re: Autobuilder repository priority ? [In reply to]

On Nov 3, 2009, at 2:26, Attila Csipa wrote:
>
> 1) The fact that something builds in the pristine final SDK does not
> guarantee
> that it will go through the autobuilder (as extras-devel might
> contain non-
> backwards compatible updates to the SDK packages).

So the SDK and Extras-devel need to be in synch, they need to be
identical. Couldn't this be solved by having the SDK team update the
SDK repos? Instead of having a new repo?

>
> 2) We seem to have an 'interesting' possibility, where, due to
> current repo
> layout and promotion policies, you can get into a situation in which
> you are
> not able to promote your own application update to Extras because of
> other
> people's development activity in Extras-devel (especially nasty if
> you were to
> make an important security fix).

One solution to this might be to make Extras-devel more static by not
allowing developers to ship updated libs and dependencies but instead
forcing them to develop against existing libraries. When a library is
outdated, the maintainer of that lib (which I suspect is often Nokia
or Pymaemo for example) needs to upgrade that library.

Pros: Developers know what is in Extras-devel and they can be certain
that their software will build if they develop against it
The repository is in a 'known state' and easier to manage for users
and maemo.org
Greater opportunity to create stable repositories with fewer bugs in
libraries and few bugs in packages

Cons: Developers can't use the version of the libraries they want,
with the functionality they want
Newer software takes longer to get to users
There may be incentive for developers to create their own repos
for shipping software

At this point, I think the cons outweigh the pros, how do other people
feel?

Jeremiah
_______________________________________________
maemo-developers mailing list
maemo-developers [at] maemo
https://lists.maemo.org/mailman/listinfo/maemo-developers

jeremiah at jeremiahfoster

Nov 3, 2009, 5:23 AM

Post #22 of 33 (1363 views)
Permalink

Re: Autobuilder repository priority ? [In reply to]

On Nov 3, 2009, at 9:25, Ed Bartosh wrote:

>> You mention separate queue but which queue are you
>> referring to? Does the suggestion involve additional repositories?
>> And/or
>> does it involve differnet rules for which repositories will be in
>> scope
>> during a build? Or what.
>>
> As I understood Attila he proposed to create separate autobulder
> incoming queue for Extras updates.

Ah, a separate build queue.

> If packages are uploaded to this queue they're built using only Extras
> and SDK repos and put into extras-devel.

So no extra repository needed?

> As a result they won't depend on unstable packages from Extras-devel
> and can go to extras-testing and then to Extras faster.

That sounds like an elegant solution.

> As I already mentioned I'm also a bit afraid of extra complexity and
> possible confusion, but I think this idea at least worth to be
> discussed.

I completely agree. :)

Jeremiah
_______________________________________________
maemo-developers mailing list
maemo-developers [at] maemo
https://lists.maemo.org/mailman/listinfo/maemo-developers

anderson.lizardo at openbossa

Nov 3, 2009, 7:30 AM

Post #23 of 33 (1360 views)
Permalink

Re: Autobuilder repository priority ? [In reply to]

On Sat, Oct 31, 2009 at 7:45 AM, Attila Csipa <maemo [at] csipa> wrote:
> I have a small issue with the autobuilder. The whole thing started out by
> having a package that compiled nice in the SDK but not in the autobuilder due
> to a versioning mismatch (in my case python-dbus, but it's a generic problem
> as you'll see). After some snooping around, I realized the problem was that
> the SDK had 0.83.0-1maemo1 in fremantle/tools/free (and that's what got used
> when compiling in the SDK), however, the autobuilder insists on using
> 0.83.0-1maemo3 from fremantle/free (-devel). The problem is IMHO that
> the repository priorities seem to be wrong. The autobuilder should be using
> the highest version in the TOP PRIORITY repository that satisfies a dependency
> to avoid breakage because of unstable stuff in -devel and because otherwise a
> package can't be promoted without dragging other folks' packages with them.
> Thoughts ?

I'm interested to know what problems you are having with python-dbus.
Can you please fill a bug report under
https://bugs.maemo.org/enter_bug.cgi?product=PyMaemo ?

As you might know, Python is not officially supported on Maemo. The
PyMaemo team provides most of the Python components used by Python
development on Maemo.

Some Python packages happen to be present on the SDK, but that does
not mean that they are officially supported. They are there to satisfy
dependencies for some *SDK* tools, which are not present on the N900
device.

Also note that the python packages that are on the SDK were packaged
by the PyMaemo team (the -1maemo1 version was packaged by myself and
other PyMaemo team members, as you can see on the debian/changelog).
The Maemo developers just took the packages and integrated into the
SDK to satisfy the dependencies mentioned below.

But the PyMaemo team is still responsible for providing upgrades and
fixes for these packages through the extras-devel/extras-testing
repositories, and the user applications that use packages like
python-dbus, when promoted, will automatically promote the
dependencies. It *seems* to be the correct way of handling the
promotion for packages not under user/* sections, like all PyMaemo
components.

And last but not the least, the python-dbus version you are referring
(-1maemo3) is not supposed to be "unstable". We had no bug reports
about it, so if you are having problems with it, please report a bug
so we can fix it ASAP.

Thanks,
--
Anderson Lizardo
OpenBossa Labs - INdT
Manaus - Brazil
_______________________________________________
maemo-developers mailing list
maemo-developers [at] maemo
https://lists.maemo.org/mailman/listinfo/maemo-developers

anderson.lizardo at openbossa

Nov 3, 2009, 7:35 AM

Post #24 of 33 (1368 views)
Permalink

Re: Autobuilder repository priority ? [In reply to]

On Tue, Nov 3, 2009 at 11:30 AM, Anderson Lizardo
<anderson.lizardo [at] openbossa> wrote:
> But the PyMaemo team is still responsible for providing upgrades and
> fixes for these packages through the extras-devel/extras-testing
> repositories, and the user applications that use packages like
> python-dbus, when promoted, will automatically promote the
> dependencies. It *seems* to be the correct way of handling the
> promotion for packages not under user/* sections, like all PyMaemo
> components.

That also means you MUST have at least the extras-testing repository
enabled on your scratchbox targets for any Python development under
Maemo, otherwise you will use very outdated versions of some Python
packages, which lack important fixes.

Regards,
--
Anderson Lizardo
OpenBossa Labs - INdT
Manaus - Brazil
_______________________________________________
maemo-developers mailing list
maemo-developers [at] maemo
https://lists.maemo.org/mailman/listinfo/maemo-developers

henrik.hedberg at innologies

Nov 3, 2009, 8:03 AM

Post #25 of 33 (1368 views)
Permalink

Re: Autobuilder repository priority ? [In reply to]

Anderson Lizardo wrote:

> But the PyMaemo team is still responsible for providing upgrades and
> fixes for these packages through the extras-devel/extras-testing
> repositories, and the user applications that use packages like
> python-dbus, when promoted, will automatically promote the
> dependencies. It *seems* to be the correct way of handling the
> promotion for packages not under user/* sections, like all PyMaemo
> components.

Why is that?

You do not feel scary that you cannot push, for example, a security
fix for your components? Let's say that I am using one application
(user/* package) that depends on python. For some reason it is not
maintained anymore, and thus not updated. How do I get new versions of
python libraries?

Another thing to consider is that should _every_ application (user/*
package) that depends on python be updated to just have a new version
number in their dependencies when a new version of python libraries is
released? (May be not, if they are working with the earlier version too,
but what about those security fixes, for example.) There will be a lot
of unnecessary megabytes to download just for that.

For Microfeed backend (libraries and applications that are not
visible to user directly) I decided to create one user/* package that
depends on the latest library packages. Applications using the backend
are encouraged to depend on that package. When a library, for example,
is updated, there will be a new version of the user/* package that pulls
the library package.

How do you see that option?

BR,

Henrik

--
Henrik Hedberg - http://www.henrikhedberg.net/
_______________________________________________
maemo-developers mailing list
maemo-developers [at] maemo
https://lists.maemo.org/mailman/listinfo/maemo-developers

1 2

View All

Index | Next | Previous | View Threaded

Interested in having your list archived? Contact Gossamer Threads