chrism at plope
May 12, 2009, 8:20 AM
Post #11 of 41
(1829 views)
Permalink
|
On 5/12/09 4:44 AM, Patrick Gerken wrote:
> Hi,
>
> I agree with the package inflation. It makes people nervous of the
> complexity.
> But putting everything back into a pile of mud sounds like going back
> from where we came, and I this situation makes people even more nervous
> when they want to look into zope.
> As far as I know, the big dependency clean up is not finished yet, no?
> Maybe it might be better to do that first. With a clean dependency
> graph, its easier to see, which set of packages could be merged into one
> package.
I don't think there will ever be a point where it's "finished"; at least not in
any time frame in which Zope is still relevant at the end. Sure, we can keep
the current setuptools distributions and keep pulling apart their respective
dependencies forever, but by the time it's all over, it just won't matter
anyway; folks will be happily using "Django 3000" or "Pylons 4", which will have
recreated all the features we teased out.
- C
> About the configurability of all the subpackages, I remember a proposal
> long time ago to reduce the number of zcml directives. This time I am in
> favor of inventing some zcml directives with some care. But before doing
> that, it might be better to write two or free easy tutorials about how
> to write custom zcml directives and grokkers. That would result in
> people writing more custom zcml directives and grokkers and get the hang
> of it. With the a much bigger userbase of writers of custom zcml
> directives and grokkers, the community as a whole will get a much better
> idea of how to use zcml, and I think gaining that community sense first
> will result in some high quality new general usable zcml directives. At
> the moment, and with my limited view, it looks like most zope ppl really
> don't know how to write custom zcml directives.
>
> Just my 0.02€
>
> Best regards,
>
> Patrick
>
>
>
> On Tue, May 12, 2009 at 09:08, Chris McDonough <chrism [at] plope
> <mailto:chrism [at] plope>> wrote:
>
> I realize now that I've neglected to give sufficient reasoning for
> why less
> granular packaging would be a good thing.
>
> I've noticed that there's a common theme in Zope development,
> software, and
> packages that I can only describe as "power law development" or
> "turtles all the
> way down". It's a bit of an antipattern, unfortunately.
>
> I'll provide an example by way of Zope-3-the-appserver. In an
> application that
> uses Zope-3-the-appserver, many individual subframeworks will be
> used. For
> example, there is a traversal subframework, a security subframework, a
> cataloging subframework, and so on.
>
> Each of these subframeworks acts as a logical unit, and through the
> magic of the
> component architecture, each can be replaced wholesale by
> registering some
> adapter. However, each of these subframeworks tend to also have
> settings that
> can be configured. For example, individual traversal steps for
> certain types of
> objects can be overridden by registering an adapter that
> *configures* the
> subframework. In the case of Zope 3, we have a traversal situation
> where the
> larger traversal subframework can either be replaced wholesale via
> an adapter
> registration or extended piecemeal via other adapter registrations.
>
> The problem is that the mechanism to *replace* the subframework is
> the same as
> the mechanism to *configure* it (both are done via adapter registration,
> sometimes even in the same file). This is theoretically fine. But
> in reality,
> it's tremendously hard for someone just walking up to a complex
> system like Zope
> 3 to discern adapter registrations that replace subsystems from
> those which
> merely configure subsystems. An inability to discern the difference
> leads to
> situations where people just "don't get the joke" and try to wiggle
> wires to
> configure-to-death a existing subsystem that's clearly suboptimal
> for their use
> case instead of just replacing it wholesale with a much simpler
> custom policy.
> They just don't know it was engineered to be replaced. So they keep
> adding more
> configuration code to the existing subframework to handle various 1%
> edge cases.
> Often this code makes the subframework tremendously complex, and the
> subframework grows inappropriate dependencies along the way.
> *Sometimes* the
> situation gets so confusing for a new user, they just quit and go
> use something
> else.
>
> This is a pattern that happens over and over again in Zope
> development. In my
> personal opinion, the original error was trying to make the subframework
> configurable at all. Instead, the subframework should be
> replaceable easily,
> but it should itself be relatively rigid. At very least, for
> subframeworks that
> really do require extra configuration (should be very few), this
> configuration
> should be done via highly specialized ZCML directives (or grokkers),
> as opposed
> to some very general adapter registration that can't be easily
> discerned from
> other adapter registrations by a newbie.
>
> If the subframeworks were more rigid (but replaceable), the *intent*
> of the
> subframework author could be more easily discerned, and fewer people
> would fall
> into the trap of adding more configuration code to a subframework
> instead of
> just replacing it entirely. And fewer people would just walk away
> in frustration.
>
> What does this have to do with packaging? Well, currently, there's
> a dizzying
> number of packages that make up "the ZTK" (nee "Zope 3"). Each of these
> packages is a pure peer of all others in a PyPI listing with no real
> way to get
> a sense of their relative importance other than performing a linear
> audit. Even
> if a user *does* do a linear search of all of them, it's still awful
> hard to
> discern for some new user which ones are "important", and which ones
> just happen
> to exist by some inequity of history without trying to install it.
> The user
> needs to gain some holistic knowledge of the system in order to
> discern the
> important bits from these historical inequities.
>
> Most new users understandably just walk away from *all* Zope
> packages before
> they gain this knowledge; it's just too hard for them to tell the
> difference
> between the truly important and reusable bits and the stuff that
> just happens to
> be packaged up and released but which is useless outside of some
> highly specific
> context. In effect, we just don't communicate *intent* very
> effectively in our
> current packaging structure.
>
> In my opinion, this is why a lot of Python developers who are
> otherwise very
> smart have given up on trying to use Zope packages. The time
> required to figure
> out which ones are useful and which ones aren't is just too high.
> It's way
> "easier" for them to write them all off wholesale and just write
> what they need
> from scratch or use somebody else's software. Good developers tend
> to like
> small, useful libraries and frameworks.
>
> We can ameliorate the situation in a few ways:
>
> - We can reduce the number of distributions.
>
> - We can make each current Zope package distribution independently
> useful.
>
> My suggestion is that we do the former first in order to communicate
> *current
> intent* (the "state of reality right now"). We can do the latter
> after this in
> pieces, hopefully aided by some new developers we've picked up by
> making it
> easier to find useful stuff.
>
> Once we deflate our current set of packages down to a reasonable
> number, the
> packages listed in PyPI will immediately start to reflect "the state
> of reality
> right now". As a result, we'll hopefully be able to get some new
> blood in the
> form of new developers that use the smaller bits outside Zope to
> help us tease
> the truly independent pieces out of the larger pile. If we do this,
> at no time
> after the deflation will PyPI listings ever as badly advertise "the
> state of
> reality" as it is advertised right now, and the community will
> hopefully again
> start to grow.
>
> - C
>
>
> On 5/11/09 11:11 AM, Martijn Faassen wrote:
> > Hey,
> >
> > Chris McDonough wrote:
> >
> > > Instead, I have argued for promoting packages that have some
> life of
> > > their own (independent of the rest of the pile) into
> subprojects that
> > > have their own release cycles.
> >
> > > Then outside projects such as Plone and Grok could depend on
> > > independent versions of such packages, giving them slightly more
> > > flexibility than requiring a "version of the ZTK".
> >
> > We already have that flexibility today. To me, the utility of a
> release
> > of version numbers in the ZTK does not at all exclude the
> potential to
> > evolve the packages to more independent sub-projects.
> >
> >> Given that this suggestion has been met with skepticism, let me
> try another
> >> tact.
> >
> > I think that's an inaccurate description of the response you got. I'm
> > quite positive about trying to give as many packages as possible
> a life
> > of their own. I don't think you got anyone else arguing against this
> > point of view.
> >
> > I'm also quite positive that some packages are:
> >
> > * useful as independently distributed packages
> >
> > * only make sense in a Zope 3 or a Grok or a Zope 2 context, i.e.
> they
> > depend on a significant set of Zope packages.
> >
> > I'd like to get out of this paradigm:
> >
> > * the Zope packages are independent sub-projects.
> >
> > * therefore we cannot distribute a list of versions that work
> best together.
> >
> > And this one:
> >
> > * if we distribute a KGS of anything
> >
> > * packages in that anything aren't independently reusable
> automatically
> > and should be merged into a ball.
> >
> > I'd also like to get out of the following paradigm:
> >
> > * the Zope packages are not independently reusable yet
> >
> > * therefore we should distribute them all together
> >
> > We're in a grey area. Some package are here, some packages are there,
> > some are in between. Some packages build on other packages, but have
> > clear dependency structures. Some don't have clear dependency
> > structures. Some have better documentation and better focus than
> others.
> >
> > If there is to be a merging of code together, then I propose we
> continue
> > the project where the ZMI code is merged into one or a few
> packages. We
> > can also investigate merging 2 or 3 packages together where it
> seems to
> > make sense, or simply moving code between packages (some code
> needs to
> > go down the dependency list, some up).
> >
> >> Instead of thinking about it this way, could we think about it as
> >> *deflating* the current set of zope.* packages that do not
> currently have a
> >> meaningful life of their own into a single setuptools
> distribution named "ZTK".
> >> This package would include most everything in zope.app*, plus
> things like
> >> zope.server, zope.publisher, and other things that just aren't
> useful outside of
> >> Zope-the-appserver, or which currently just depend on "too much".
> >
> > -1
> >
> > This would make it a lot harder to:
> >
> > * clean up dependency relationships with the goal of creating more
> > reusable code. We'd all hide them in a sumo ball again.
> >
> > * get rid of bits we *don't want anymore*. If I need anything in
> a sumo
> > package I'd need *all* of it.
> >
> > * override individual packages with newer versions
> >
> > * we've done a lot of refactoring recently trying to separate the UI
> > from packages. This is done by creating a *new* package, leaving
> the old
> > package behind. We can do this, test this and release this
> > package-by-package now.
> >
> >> Over time, we'd tease out the dependencies of packages that live
> in the ZTK
> >> distribution, making them useful outside without any dependency
> on the ZTK. The
> >> names of these packages could be arbitrary (they wouldn't need
> to retain their
> >> old "zope.*" names). Some backwards dependency shims would be
> added to the ZTK
> >> to prevent old imports from breaking, and the ZTK distribution
> would then just
> >> have a dependency on the thing that got broken out.
> >
> > I don't like the attempt to redefine what the ZTK means to a
> giant ball
> > of Python packages. That's implying that, say, zope.component is
> *not*
> > in the ZTK. That's wrong.
> >
> > Why generate a whole lot of work for ourselves getting from where
> we are
> > now to here? We've learned how to work with the current situation in
> > 2007 already.
> >
> >> I'm thinking that this would simplify things greatly for people
> who want to be
> >> consumers of "truly independent" Zope packages. There'd be
> exactly N of them
> >> available for download (where N is much less than 100, more like
> 20), plus the
> >> ZTK, which would have the rest of the pile in it over time.
> >
> > I don't see why a big package would "simplify things greatly" for
> you or
> > anyone else.
> >
> >> If someone wanted
> >> to use a forked version of a package that lived in the ZTK
> distribution, they'd
> >> either do so by teasing out the dependencies and making it
> "truly independent"
> >> or they'd just reroll a custom version of the entire ZTK
> distribution.
> >
> > And that's easier than the current situation how? Are you really
> > proposing we destroy the dependency information we've already
> teased out
> > and then make people do the work again?
> >
> >> Does this make any sense?
> >
> > Not a lot in my book.
> >
> > I think an important reason why there's so much awareness of
> dependency
> > issues in the Zope world now (and effort spent on it) is precisely
> > because we released our separate packages and can see the dependency
> > information clearly.
> >
> > Regards,
> >
> > Martijn
> >
> > _______________________________________________
> > Zope-Dev maillist - Zope-Dev [at] zope <mailto:Zope-Dev [at] zope>
> > http://mail.zope.org/mailman/listinfo/zope-dev
> > ** No cross posts or HTML encoding! **
> > (Related lists -
> > http://mail.zope.org/mailman/listinfo/zope-announce
> > http://mail.zope.org/mailman/listinfo/zope )
> >
>
> _______________________________________________
> Zope-Dev maillist - Zope-Dev [at] zope <mailto:Zope-Dev [at] zope>
> http://mail.zope.org/mailman/listinfo/zope-dev
> ** No cross posts or HTML encoding! **
> (Related lists -
> http://mail.zope.org/mailman/listinfo/zope-announce
> http://mail.zope.org/mailman/listinfo/zope )
>
>
_______________________________________________
Zope-Dev maillist - Zope-Dev [at] zope
http://mail.zope.org/mailman/listinfo/zope-dev
** No cross posts or HTML encoding! **
(Related lists -
http://mail.zope.org/mailman/listinfo/zope-announce
http://mail.zope.org/mailman/listinfo/zope )
|
1
