Login | Register For Free | Help
Search for: (Advanced)

Mailing List Archive: Wikipedia: Wikitech

Abstract schema work

 

 

Wikipedia wikitech RSS feed   Index | Next | Previous | View Threaded


greg at endpoint

Apr 18, 2012, 9:09 AM

Post #1 of 16 (404 views)
Permalink
Abstract schema work

I'm jumping back into the abstract schema project, and was
wondering what the workflow is for something like this. Should
I simply create a new branch and have everybody (assuming other
people have an interest) collaborate on that branch until we
are ready to start involving gerrit? Obviously, this will
be a large, all-at-once patch once finished.

Also, I'm still not clear on which mailing list would be more
approriate for discussion of a feature like this. The descriptions
of mediawiki-l and wikitech-l both say "features and development".
I lean towards this list (wikitech) due to the higher traffic.
Thanks.

--
Greg Sabino Mullane greg [at] endpoint
End Point Corporation
PGP Key: 0x14964AC8


innocentkiller at gmail

Apr 18, 2012, 9:35 AM

Post #2 of 16 (399 views)
Permalink
Re: Abstract schema work [In reply to]

On Wed, Apr 18, 2012 at 12:09 PM, Greg Sabino Mullane <greg [at] endpoint> wrote:
> I'm jumping back into the abstract schema project, and was
> wondering what the workflow is for something like this. Should
> I simply create a new branch and have everybody (assuming other
> people have an interest) collaborate on that branch until we
> are ready to start involving gerrit? Obviously, this will
> be a large, all-at-once patch once finished.
>

I think this is a perfect use for a feature branch for people
to collaborate on. I'm certainly interested in this project and
I know I'm not alone.

> Also, I'm still not clear on which mailing list would be more
> approriate for discussion of a feature like this. The descriptions
> of mediawiki-l and wikitech-l both say "features and development".
> I lean towards this list (wikitech) due to the higher traffic.
>

I think we should probably work on mediawiki.org (subpage of
RfC?). This is something that needs to be thought out properly,
and I think on-wiki would be more productive than on-list.

-Chad

_______________________________________________
Wikitech-l mailing list
Wikitech-l [at] lists
https://lists.wikimedia.org/mailman/listinfo/wikitech-l


Platonides at gmail

Apr 18, 2012, 9:36 AM

Post #3 of 16 (400 views)
Permalink
Re: Abstract schema work [In reply to]

On 18/04/12 18:09, Greg Sabino Mullane wrote:
> I'm jumping back into the abstract schema project, and was
> wondering what the workflow is for something like this. Should
> I simply create a new branch and have everybody (assuming other
> people have an interest) collaborate on that branch until we
> are ready to start involving gerrit? Obviously, this will
> be a large, all-at-once patch once finished.

It took me a while to remember that idea of replacing the sql files with
an abstract schema.
Hopefully this will help subsequent readers.



> Also, I'm still not clear on which mailing list would be more
> approriate for discussion of a feature like this. The descriptions
> of mediawiki-l and wikitech-l both say "features and development".
> I lean towards this list (wikitech) due to the higher traffic.
> Thanks.

wikitech-l is the right one for proposing new features.


_______________________________________________
Wikitech-l mailing list
Wikitech-l [at] lists
https://lists.wikimedia.org/mailman/listinfo/wikitech-l


krinklemail at gmail

Apr 18, 2012, 6:37 PM

Post #4 of 16 (392 views)
Permalink
Re: Abstract schema work [In reply to]

On Apr 18, 2012, at 6:09 PM, Greg Sabino Mullane wrote:

> I'm jumping back into the abstract schema project, and was
> wondering what the workflow is for something like this. Should
> I simply create a new branch and have everybody (assuming other
> people have an interest) collaborate on that branch until we
> are ready to start involving gerrit? Obviously, this will
> be a large, all-at-once patch once finished.

I'd say put it in Gerrit from the start (in a branch) so that everyone
can check it out and send suggestions (either as a commit or through the
feedback channels on the mailing list, wiki or Gerrit comments).

Gerrit reviews are also enabled for branches, so you don't have to worry
much about clashing with others, a commit to the branch on gerrit will
not end up in the actual branch until it is reviewed.

When it is getting closer to perfection you could push a patchset of the
entire branch as a squashed-commit on the master branch to gerrit for
review. Even then there is still plenty of room for people to
cherry-pick that onto their local repositories for testing, and submit
revisions of the patch set to Gerrit etc., until it is approved.


>
> Also, I'm still not clear on which mailing list would be more
> approriate for discussion of a feature like this. The descriptions
> of mediawiki-l and wikitech-l both say "features and development".
> I lean towards this list (wikitech) due to the higher traffic.
> Thanks.
>

Starting an RFC page [1] is a good way to centralize the concept for
further collaboration and feedback. Discussion sometimes continues on
wikitech-l, and sometimes on the wiki talk page of the RFC.

-- Krinkle

[1] https://www.mediawiki.org/wiki/RFC
_______________________________________________
Wikitech-l mailing list
Wikitech-l [at] lists
https://lists.wikimedia.org/mailman/listinfo/wikitech-l


freak at drajv

Apr 19, 2012, 3:32 PM

Post #5 of 16 (439 views)
Permalink
Re: Abstract schema work [In reply to]

count me in :D

On 18. 04. 2012 18:09, Greg Sabino Mullane wrote:
> I'm jumping back into the abstract schema project, and was
> wondering what the workflow is for something like this. Should
> I simply create a new branch and have everybody (assuming other
> people have an interest) collaborate on that branch until we
> are ready to start involving gerrit? Obviously, this will
> be a large, all-at-once patch once finished.
>
> Also, I'm still not clear on which mailing list would be more
> approriate for discussion of a feature like this. The descriptions
> of mediawiki-l and wikitech-l both say "features and development".
> I lean towards this list (wikitech) due to the higher traffic.
> Thanks.
>
>
>
> _______________________________________________
> Wikitech-l mailing list
> Wikitech-l [at] lists
> https://lists.wikimedia.org/mailman/listinfo/wikitech-l

_______________________________________________
Wikitech-l mailing list
Wikitech-l [at] lists
https://lists.wikimedia.org/mailman/listinfo/wikitech-l


greg at endpoint

Apr 20, 2012, 2:27 PM

Post #6 of 16 (436 views)
Permalink
Re: Abstract schema work [In reply to]

Chad wrote:
> I think we should probably work on mediawiki.org (subpage of
> RfC?). This is something that needs to be thought out properly,
> and I think on-wiki would be more productive than on-list.

Okay. I will create an RFC page. Keep in mind that my time is limited
and I've only just started to write the proof of concept for it,
so please don't hold up version 12.0 to wait for me, ha ha ha!

Platonides wrote:
> It took me a while to remember that idea of replacing the
> sql files with an abstract schema.

Right. I've been thinking about this a lot, and the big picture plan
is to remove all the tables.sql files and create a single text file,
mediawiki.schema, which will have a generic schema, very similar to
the existing main tables.sql, but with a stricter syntax and more
generic definitions. This will be parsed and put into a php structure.
Each database will walk through this structure to build it's own
'create' statements, using common methods and attributes whenever
possible. Similarly, we can use this for updates as well by simply
walking through and adding missing tables, rather than having to
create patch files or other shenanigans. Most of the above is written:
the tricky part is actually figuring out exactly what the intent of
each column definition in the current tables.sql is! There is a lack
of consistency and some questionable choices. This will also be a great
time to introduce some standards, e.g. naming of indexes.

Krinkle wrote:
> I'd say put it in Gerrit from the start (in a branch) so that everyone
> can check it out and send suggestions (either as a commit or through the
> feedback channels on the mailing list, wiki or Gerrit comments).
>
> Gerrit reviews are also enabled for branches, so you don't have to worry
> much about clashing with others, a commit to the branch on gerrit will
> not end up in the actual branch until it is reviewed.

I don't think this is a good idea: the having to wait for a reviewer before
your changes show up elsewhere is great for the core code, but not so
great for fast-n-furious early prototyping of a big feature. Perhaps we
can add it to gerrit once everyone thinks we have a mostly stable
prototype?

Thanks for all the feedback. I will post here when I add a branch and/or
create an RFC page.

--
Greg Sabino Mullane greg [at] endpoint
End Point Corporation
PGP Key: 0x14964AC8


innocentkiller at gmail

Apr 20, 2012, 2:29 PM

Post #7 of 16 (385 views)
Permalink
Re: Abstract schema work [In reply to]

On Fri, Apr 20, 2012 at 5:27 PM, Greg Sabino Mullane <greg [at] endpoint> wrote:
> Krinkle wrote:
>> I'd say put it in Gerrit from the start (in a branch) so that everyone
>> can check it out and send suggestions (either as a commit or through the
>> feedback channels on the mailing list, wiki or Gerrit comments).
>>
>> Gerrit reviews are also enabled for branches, so you don't have to worry
>> much about clashing with others, a commit to the branch on gerrit will
>> not end up in the actual branch until it is reviewed.
>
> I don't think this is a good idea: the having to wait for a reviewer before
> your changes show up elsewhere is great for the core code, but not so
> great for fast-n-furious early prototyping of a big feature. Perhaps we
> can add it to gerrit once everyone thinks we have a mostly stable
> prototype?
>

We can enable direct pushing on branches as well, that's
not a big deal :)

-Chad

_______________________________________________
Wikitech-l mailing list
Wikitech-l [at] lists
https://lists.wikimedia.org/mailman/listinfo/wikitech-l


Platonides at gmail

Apr 20, 2012, 3:06 PM

Post #8 of 16 (391 views)
Permalink
Re: Abstract schema work [In reply to]

On 20/04/12 23:27, Greg Sabino Mullane wrote:
> Right. I've been thinking about this a lot, and the big picture plan
> is to remove all the tables.sql files and create a single text file,
> mediawiki.schema, which will have a generic schema, very similar to
> the existing main tables.sql, but with a stricter syntax and more
> generic definitions. This will be parsed and put into a php structure.
> Each database will walk through this structure to build it's own
> 'create' statements, using common methods and attributes whenever
> possible. Similarly, we can use this for updates as well by simply
> walking through and adding missing tables, rather than having to
> create patch files or other shenanigans. Most of the above is written:
> the tricky part is actually figuring out exactly what the intent of
> each column definition in the current tables.sql is! There is a lack
> of consistency and some questionable choices. This will also be a great
> time to introduce some standards, e.g. naming of indexes.

Note we will also need a way to generate a SQL from that (eg. for
manually creating a new database with SQLAdmin).

Personally, I prefer viewing the code (SQL) which is really used, but I
won't stop you from making the perfect abstraction.


_______________________________________________
Wikitech-l mailing list
Wikitech-l [at] lists
https://lists.wikimedia.org/mailman/listinfo/wikitech-l


greg at endpoint

Apr 22, 2012, 3:36 PM

Post #9 of 16 (382 views)
Permalink
Re: Abstract schema work [In reply to]

Platonides wrote:
> Note we will also need a way to generate a SQL from that (eg.
> for manually creating a new database with SQLAdmin).
>
> Personally, I prefer viewing the code (SQL) which is really
> used, but I won't stop you from making the perfect abstraction.

Well, the very first task of the new system will be to generate
a tables.sql and make sure it matches the current one. Perhaps we
can even keep them around, but make them "read only", for the benefit
of circumstances like the above (as well as making it easy to read
from the perspective of each database-in-question).

The only question in my mind at the moment is do we also copy all
the comments to each tables.sql (see maintenance/tables.sql), no
comments (see oracle|postgres/tables.sql) or something in the middle
(see mssql/tables.sql)?

--
Greg Sabino Mullane greg [at] endpoint
End Point Corporation
PGP Key: 0x14964AC8


hashar+wmf at free

Apr 23, 2012, 1:18 AM

Post #10 of 16 (381 views)
Permalink
Re: Abstract schema work [In reply to]

Le 18/04/12 18:09, Greg Sabino Mullane wrote:
> I'm jumping back into the abstract schema project

Are we reinventing the wheel once again or is that a project to select
and integrate an existing tool?

--
Antoine "hashar" Musso


_______________________________________________
Wikitech-l mailing list
Wikitech-l [at] lists
https://lists.wikimedia.org/mailman/listinfo/wikitech-l


freak at drajv

Apr 23, 2012, 4:05 AM

Post #11 of 16 (379 views)
Permalink
Re: Abstract schema work [In reply to]

I think it would be about time to start practicing what we preach (like,
that MW is a great knowlagebase tool) and move all these comments to
mw.org database manual pages.
It's handy to have all those comments in the code, but to clutter the
schema abstraction code over those comments is just silly.


LP, Jure

On 23. 04. 2012 00:36, Greg Sabino Mullane wrote:
> Platonides wrote:
>> Note we will also need a way to generate a SQL from that (eg.
>> for manually creating a new database with SQLAdmin).
>>
>> Personally, I prefer viewing the code (SQL) which is really
>> used, but I won't stop you from making the perfect abstraction.
> Well, the very first task of the new system will be to generate
> a tables.sql and make sure it matches the current one. Perhaps we
> can even keep them around, but make them "read only", for the benefit
> of circumstances like the above (as well as making it easy to read
> from the perspective of each database-in-question).
>
> The only question in my mind at the moment is do we also copy all
> the comments to each tables.sql (see maintenance/tables.sql), no
> comments (see oracle|postgres/tables.sql) or something in the middle
> (see mssql/tables.sql)?
>
>
>
> _______________________________________________
> Wikitech-l mailing list
> Wikitech-l [at] lists
> https://lists.wikimedia.org/mailman/listinfo/wikitech-l

_______________________________________________
Wikitech-l mailing list
Wikitech-l [at] lists
https://lists.wikimedia.org/mailman/listinfo/wikitech-l


greg at endpoint

Apr 23, 2012, 7:47 AM

Post #12 of 16 (379 views)
Permalink
Re: Abstract schema work [In reply to]

Antoine "hashar" Musso wrote:

> Are we reinventing the wheel once again or is that a project
> to select and integrate an existing tool?

I'm open to using something existing: did you have something
specific in mind? I didn't see anything out there that would
be less trouble to customize than rolling our own.

--
Greg Sabino Mullane greg [at] endpoint
End Point Corporation
PGP Key: 0x14964AC8


jpostlethwaite at wikimedia

Apr 23, 2012, 10:24 AM

Post #13 of 16 (379 views)
Permalink
Re: Abstract schema work [In reply to]

Doctrine does versioning of databases in PHP:

http://www.doctrine-project.org/

On Mon, Apr 23, 2012 at 7:47 AM, Greg Sabino Mullane <greg [at] endpoint>wrote:

> Antoine "hashar" Musso wrote:
>
> > Are we reinventing the wheel once again or is that a project
> > to select and integrate an existing tool?
>
> I'm open to using something existing: did you have something
> specific in mind? I didn't see anything out there that would
> be less trouble to customize than rolling our own.
>
> --
> Greg Sabino Mullane greg [at] endpoint
> End Point Corporation
> PGP Key: 0x14964AC8
>
> _______________________________________________
> Wikitech-l mailing list
> Wikitech-l [at] lists
> https://lists.wikimedia.org/mailman/listinfo/wikitech-l
>



--
Jeremy Postlethwaite
jpostlethwaite [at] wikimedia
415-839-6885 x6790
Backend Software Developer
Wikimedia Foundation <http://wikimediafoundation.org/>
_______________________________________________
Wikitech-l mailing list
Wikitech-l [at] lists
https://lists.wikimedia.org/mailman/listinfo/wikitech-l


greg at endpoint

Apr 25, 2012, 11:25 AM

Post #14 of 16 (371 views)
Permalink
Re: Abstract schema work [In reply to]

Freako F. Freakolowsky wrote:
> I think it would be about time to start practicing what we preach (like,
> that MW is a great knowlagebase tool) and move all these comments to
> mw.org database manual pages.
> It's handy to have all those comments in the code, but to clutter the
> schema abstraction code over those comments is just silly.

You want them out of the code entirely, or only in once place (e.g.
the mediawiki.schema file)? I'm all for the latter, especially as we
can also automate the updating of the MW pages once we have a sane,
machine-parseable schema in place.


Jeremy Postlethwaite wrote:
>> I'm open to using something existing: did you have something
>> specific in mind? I didn't see anything out there that would
>> be less trouble to customize than rolling our own.
>
> Doctrine does versioning of databases in PHP:
>
> http://www.doctrine-project.org/

Yeah, it is interesting, but severe overkill for what we are
looking at here. I'm also loath to add another dependency, which
I'm sure at the end of the day we are going to have to
[monkey] patch anyway.

--
Greg Sabino Mullane greg [at] endpoint
End Point Corporation
PGP Key: 0x14964AC8


innocentkiller at gmail

Apr 25, 2012, 11:32 AM

Post #15 of 16 (371 views)
Permalink
Re: Abstract schema work [In reply to]

On Sun, Apr 22, 2012 at 6:36 PM, Greg Sabino Mullane <greg [at] endpoint> wrote:
> Platonides wrote:
>> Note we will also need a way to generate a SQL from that (eg.
>> for manually creating a new database with SQLAdmin).
>>
>> Personally, I prefer viewing the code (SQL) which is really
>> used, but I won't stop you from making the perfect abstraction.
>
> Well, the very first task of the new system will be to generate
> a tables.sql and make sure it matches the current one. Perhaps we
> can even keep them around, but make them "read only", for the benefit
> of circumstances like the above (as well as making it easy to read
> from the perspective of each database-in-question).
>

That part is pretty easy, and I actually got as much done when
I worked on the feature before. It's actually really trivial to come
up with some kind of abstract representation of the tables (I used
PHP arrays) that class-specific implementations can then use to
make a working tables.sql.

The hard part is coming up with a "this is what I have, this is what
I want, so how do I get from A -> B?" Not having a clear solution
to that is what made me end up shelving the project the first time.

> The only question in my mind at the moment is do we also copy all
> the comments to each tables.sql (see maintenance/tables.sql), no
> comments (see oracle|postgres/tables.sql) or something in the middle
> (see mssql/tables.sql)?
>

If we're auto-generating a schema, I think putting comments
there are pretty useless. They should be documented at the
abstract level and on mediawiki.org.

-Chad

_______________________________________________
Wikitech-l mailing list
Wikitech-l [at] lists
https://lists.wikimedia.org/mailman/listinfo/wikitech-l


freak at drajv

Apr 25, 2012, 10:53 PM

Post #16 of 16 (363 views)
Permalink
Re: Abstract schema work [In reply to]

On 25. 04. 2012 20:25, Greg Sabino Mullane wrote:
> Freako F. Freakolowsky wrote:
>> I think it would be about time to start practicing what we preach (like,
>> that MW is a great knowlagebase tool) and move all these comments to
>> mw.org database manual pages.
>> It's handy to have all those comments in the code, but to clutter the
>> schema abstraction code over those comments is just silly.
> You want them out of the code entirely, or only in once place (e.g.
> the mediawiki.schema file)? I'm all for the latter, especially as we
> can also automate the updating of the MW pages once we have a sane,
> machine-parseable schema in place.
>
I wouldn't mind them being gone entirely from the source or at least
them being in a separate file, like a doc special file for tables.sql.
That way we can use the same base code for the creation of tables.sql
files for each DB type as it's one less thing we have to make
machine-readable. Keeping them in tables.sql file just complicates things.

My personal favorite option would be something like a i18n file for the
db schema, that would map to [TYPE][OBJECT] or [TYPE][OBJECT][FIELD].
That way we don't complicate the abstract schema project, have code that
can automate generation of mw pages for manual and can even have
Siebrand translate it.

LP, jure

_______________________________________________
Wikitech-l mailing list
Wikitech-l [at] lists
https://lists.wikimedia.org/mailman/listinfo/wikitech-l

Wikipedia wikitech RSS feed   Index | Next | Previous | View Threaded
 
 


Interested in having your list archived? Contact Gossamer Threads
 
  Web Applications & Managed Hosting Powered by Gossamer Threads Inc.