Mailing List Archive: Wikipedia: Wikitech

On templates and programming languages

1 2 3 4 5

View All

Index | Next | Previous | View Threaded

Simetrical+wikilist at gmail

Jun 30, 2009, 1:42 PM

Post #26 of 116 (1325 views)
Permalink

Re: On templates and programming languages [In reply to]

On Tue, Jun 30, 2009 at 4:25 PM, Brion Vibber<brion [at] wikimedia> wrote:
> IMO by the time you've implemented your whitelisting parser you might as
> well just interpret it rather than eval()ing.

I don't think so. You'd only have to do the whitelisting once, on
page save. After that you could just execute with no extra overhead.
Even better, you could write it to a file and include() the file; this
would be a huge win if you have an opcode cache. Of course, parsing
PHP within PHP should be much easier than parsing another language
within PHP: just use token_get_all() to do most of the work.

> (And of course, eval()
> might be disabled on the server. :)

Does anyone actually do this? It would break a lot of major web apps,
surely. If anyone does do this, it would still work if you could
write to a file and then include it.

> Looping constructs are also extremely valuable -- at a minimum in a
> foreach() kind of way.

Right, but we could live without them in an initial version. They
could be added later.

> It's not about "Wikipedia content", but about being able to grab things
> you see on another wiki and use or adapt them to your own needs. We get
> lots of questions from people trying to grab some particular template
> off Wikipedia to use on their own site for their own needs.

Sure. The point still holds. Some third parties would be unable to
use Wikipedia templates, yes. But given the tangle of dependencies
the major ones have, and how complicated they are, I'm guessing most
small third-party wikis don't bother in the end anyway. Requiring
exec() for full use of content is viable IMO.

> Considering the amount of trouble people have getting texvc working, I
> wouldn't want to force that on people just to use templates.

The problem with texvc is installing dependencies and compiling it. A
much better analogy is things like diff3 -- which we shell out to out
of the box, with zero configuration, if they exist and shelling out
works. (We'd probably want scripting off by default, of course, but
we could require just a single config line.)

> Python "comes with batteries included", which is to say it's got a huge
> standard library (most of which of course wouldn't be available in a
> restricted environment). Lua's bare interpreter of course wins in an
> embedded-shipping contest. :D

Yep, but that's a big advantage. It means Windows users don't have to
do any extra work. It also lets us ensure a specific version is
reliably available. Imagine Wikimedia using Python 2.6, and someone
trying to run that on some shared host running Fedora 8 or God knows
what, with Python 2.2 or something. (Someone actually came into
#mediawiki a few months ago for help and it turned out their VPS was
something like Fedora 7 or 8. And horribly overpriced at that!)

> Hmm... it might be interesting to experiment with something like this,
> if it can _really_ be compiled standalone. (Linux binary distribution is
> a hellhole of incompatible linked library versions!)

I hadn't thought of libraries, you're right. It should work pretty
reliably on Linux (and hopefully not be too much bigger) if it's
statically linked, though, right?

_______________________________________________
Wikitech-l mailing list
Wikitech-l [at] lists
https://lists.wikimedia.org/mailman/listinfo/wikitech-l

Simetrical+wikilist at gmail

Jun 30, 2009, 1:45 PM

Post #27 of 116 (1325 views)
Permalink

Re: On templates and programming languages [In reply to]

On Tue, Jun 30, 2009 at 4:33 PM, Marco
Schuster<marco [at] harddisk> wrote:
> Static compiling the stuff? How would this affect the binary size?

Hopefully not too badly if you use the right options. libc is huge,
but the linker should be able to throw out most of it if statically
linking, since Lua likely doesn't use most libc functions.

Alternatively, is the libc ABI stable enough that we could dynamically
link libc, and statically link everything else? The other libraries
required are very small.

> (And: is
> static linking working across different libc versions?)

Yes, it should work fine, AFAIK. If you statically link everything
you're just using the kernel ABIs, which are supposed to be very
stable (especially for reasonably common stuff).

> BTW, what about Mac OS / FreeBSD hosts?

Are there any shared webhosts you know of that run Mac or BSD? At
worst, they can fall into the same group as the no-exec() camp, able
to use Wikipedia content but not 100%.

_______________________________________________
Wikitech-l mailing list
Wikitech-l [at] lists
https://lists.wikimedia.org/mailman/listinfo/wikitech-l

jared.williams1 at ntlworld

Jun 30, 2009, 1:52 PM

Post #28 of 116 (1322 views)
Permalink

Re: On templates and programming languages [In reply to]

> -----Original Message-----
> From: wikitech-l-bounces [at] lists
> [mailto:wikitech-l-bounces [at] lists] On Behalf Of
> Aryeh Gregor
> Sent: 30 June 2009 20:56
> To: Wikimedia developers
> Subject: Re: [Wikitech-l] On templates and programming languages
>
> On Tue, Jun 30, 2009 at 12:16 PM, Brion
> Vibber<brion [at] wikimedia> wrote:
> > * PHP
> >
> > Advantage: Lots of webbish people have some experience with
> PHP or can
> > easily find references.
> >
> > Advantage: we're pretty much guaranteed to have a PHP interpreter
> > available. :)
> >
> > Disadvantage: PHP is difficult to lock down for secure execution.
>
> I think it would be easy to provide a very simple locked-down
> version, with most of the features gone. You could, for
> instance, only permit variable assignment, use of built-in
> operators, a small whitelist of functions, and conditionals.
> You could omit loops, function definitions, and abusable
> functions like str_repeat() (let alone exec(), eval(), etc.)
> from a first pass. This would still be vastly more powerful,
> more readable, and faster than ParserFunctions.

Pity there is not a method of locking down code execution to a single
namespace, (think ahead with php5.3)

namespace Template
{
function strlen($string) { return \strlen($string) * 2; }
function exec() { throw new \Exception(); }

class Template
{
function paint()
{
// Redirect \ namespace to Template, so \exec() is also
caught.

echo strlen('data');
}
}
}

Jared

_______________________________________________
Wikitech-l mailing list
Wikitech-l [at] lists
https://lists.wikimedia.org/mailman/listinfo/wikitech-l

marco at harddisk

Jun 30, 2009, 2:11 PM

Post #29 of 116 (1323 views)
Permalink

Re: On templates and programming languages [In reply to]

On Tue, Jun 30, 2009 at 10:45 PM, Aryeh Gregor <
Simetrical+wikilist [at] gmail <Simetrical%2Bwikilist [at] gmail>> wrote:

> Alternatively, is the libc ABI stable enough that we could dynamically
> link libc, and statically link everything else? The other libraries
> required are very small.

I wouldn't count on this... at least we should provide a dyn-linked version
for those wanting less storage/memory/whatever consumption.

How do statically compiled programs for x86 platforms behave on x64, btw?
And what about more "exotic" platforms like ARM (which can also be
multi-endian, IXP4xx is an example) / SPARC (Toolserver!!!) or PowerPC? Are
they actually supported by Lua?

>
> > BTW, what about Mac OS / FreeBSD hosts?
>
> Are there any shared webhosts you know of that run Mac or BSD? At
> worst, they can fall into the same group as the no-exec() camp, able
> to use Wikipedia content but not 100%.
>

The webhoster hosting our school's homepage does, for example... They host
all schools in Munich, and I think they're a bit security-paranoid. We don't
have any issues hosting a MediaWiki there, actually. (OK, we never imported
WP content.)

Marco
--
VMSoft GbR
Nabburger Str. 15
81737 München
Geschäftsführer: Marco Schuster, Volker Hemmert
http://vmsoft-gbr.de
_______________________________________________
Wikitech-l mailing list
Wikitech-l [at] lists
https://lists.wikimedia.org/mailman/listinfo/wikitech-l

Simetrical+wikilist at gmail

Jun 30, 2009, 2:16 PM

Post #30 of 116 (1325 views)
Permalink

Re: On templates and programming languages [In reply to]

On Tue, Jun 30, 2009 at 4:52 PM, Jared
Williams<jared.williams1 [at] ntlworld> wrote:
> Pity there is not a method of locking down code execution to a single
> namespace, (think ahead with php5.3)

This is implausible, but even if it happened it wouldn't stop trivial
DOSes like while (true);. We'd still need to validate the code if we
wanted to run it in-process.

_______________________________________________
Wikitech-l mailing list
Wikitech-l [at] lists
https://lists.wikimedia.org/mailman/listinfo/wikitech-l

Simetrical+wikilist at gmail

Jun 30, 2009, 2:20 PM

Post #31 of 116 (1325 views)
Permalink

Re: On templates and programming languages [In reply to]

On Tue, Jun 30, 2009 at 5:11 PM, Marco
Schuster<marco [at] harddisk> wrote:
> How do statically compiled programs for x86 platforms behave on x64, btw?

I'm pretty sure they work fine. Someone with more knowledge of Linux
binaries needs to comment on how we could best do this, though.

> And what about more "exotic" platforms like ARM (which can also be
> multi-endian, IXP4xx is an example) / SPARC (Toolserver!!!) or PowerPC? Are
> they actually supported by Lua?

Lua is designed to be extremely portable IIRC, across both
architectures and compilers.

> The webhoster hosting our school's homepage does, for example... They host
> all schools in Munich, and I think they're a bit security-paranoid.

That's not a shared host. They can easily install Lua themselves.

_______________________________________________
Wikitech-l mailing list
Wikitech-l [at] lists
https://lists.wikimedia.org/mailman/listinfo/wikitech-l

ssanbeg at ask

Jun 30, 2009, 2:29 PM

Post #32 of 116 (1322 views)
Permalink

Re: On templates and programming languages [In reply to]

On Tue, 30 Jun 2009 21:38:07 +0100, Thomas Dalton wrote:

> 2009/6/30 Michael Daly <michael.daly [at] kayakwiki>:

> How does that work with anonymous variables? Are all $[NUMBER] style
> names count as auto-declared?
>

They're not anonymous, they're just named sequentially. Most languages
should have some method of accessing/declaring those, i.e.

XSL:
<xsl:param name="1">default</xsl:param>
<xsl:value-of select="$1"/>

perl:
my $p=$ARG{1};
print $p;

etc...

If we do roll our own, it should have similar fucntionality.

_______________________________________________
Wikitech-l mailing list
Wikitech-l [at] lists
https://lists.wikimedia.org/mailman/listinfo/wikitech-l

thomas.dalton at gmail

Jun 30, 2009, 2:53 PM

Post #33 of 116 (1324 views)
Permalink

Re: On templates and programming languages [In reply to]

2009/6/30 Steve Sanbeg <ssanbeg [at] ask>:
> On Tue, 30 Jun 2009 21:38:07 +0100, Thomas Dalton wrote:
>
>> 2009/6/30 Michael Daly <michael.daly [at] kayakwiki>:
>
>> How does that work with anonymous variables? Are all $[NUMBER] style
>> names count as auto-declared?
>>
>
> They're not anonymous, they're just named sequentially.

They are anonymous when you call the template, though. The names are
determined by the order in the call rather than written explicitly.
They do need to be considered separately.

_______________________________________________
Wikitech-l mailing list
Wikitech-l [at] lists
https://lists.wikimedia.org/mailman/listinfo/wikitech-l

rarohde at gmail

Jun 30, 2009, 3:08 PM

Post #34 of 116 (1332 views)
Permalink

Re: On templates and programming languages [In reply to]

On Tue, Jun 30, 2009 at 12:56 PM, Aryeh
Gregor<Simetrical+wikilist [at] gmail> wrote:
> On Tue, Jun 30, 2009 at 12:16 PM, Brion Vibber<brion [at] wikimedia> wrote:
>> * PHP
>>
>> Advantage: Lots of webbish people have some experience with PHP or can
>> easily find references.
>>
>> Advantage: we're pretty much guaranteed to have a PHP interpreter
>> available. :)
>>
>> Disadvantage: PHP is difficult to lock down for secure execution.
>
> I think it would be easy to provide a very simple locked-down version,
> with most of the features gone. �You could, for instance, only permit
> variable assignment, use of built-in operators, a small whitelist of
> functions, and conditionals. �You could omit loops, function
> definitions, and abusable functions like str_repeat() (let alone
> exec(), eval(), etc.) from a first pass. �This would still be vastly
> more powerful, more readable, and faster than ParserFunctions.
>
> Hopefully, we could make this secure enough for your average
> shared-host website to run it by default with no special measures
> taken and without much risk. �Installations with more access and
> higher security requirements, like Wikimedia, could shell out to a
> process that's sandboxed on the OS level to be on the safe side. �I'd
> like to hear what Tim thinks about the possibility of securing PHP
> like this.
>
> Of course, PHP is evil, and supporting it sucks. �:( �But if we
> *really* *really* need to support users who can't shell out to other
> programs, I think it's the only real language that's a feasible
> solution.
>
>
> I'd encourage you to consider requiring exec() support for full use of
> Wikipedia templates, though. �Many really big shared hosts allow it,
> like 1and1.com. �Anyone big enough to include much Wikipedia content
> will likely be on at least a VPS anyway. �And if your host doesn't
> support exec(), then at *worst* you can still get the articles in a
> totally usable form -- just run Special:ExpandTemplates on all the
> article's templates. �You can then transclude those on a per-article
> basis; we could update Special:Export to make this easier. �The only
> problem in this case would be that you can't easily change the
> formatting of all the templates at once -- but such a small site would
> likely have few enough articles to do it by hand, if they even want
> to.
>
> I think saying that users without exec() support get to use Wikipedia
> content in a somewhat less usable form would be just fine, and it
> would *really* open up our options. �We could support basically any
> programming language in that case.
>
>> * Python
>>
>> Advantage: A Python interpreter will be present on most web servers,
>> though not necessarily all. (Windows-based servers especially.)
>>
>> Wash: Python is probably better known than Lua, but not as well as PHP
>> or JS.
>>
>> Disadvantage: Like PHP, Python is difficult to lock down securely.
>
> It doesn't matter whether it's present, does it? �If the user has
> exec() support, they could download a binary interpreter for *any*
> language to their webspace and run it from there regardless of whether
> the language is supported on the host. �So Python is on exactly the
> same level as Lua here.
>
> Much though I love Python, Lua looks like the better option. �First of
> all, it's *very* small. �sudo apt-get install lua50 on my machine uses
> up only 180 KB of disk space, and the package is 30 KB gzipped. �Our
> current tarballs are 10 MB; we could easily just chuck in Lua binaries
> for Linux x86-32 and Windows without even noticing the size increase,
> and allow users to enable it with one line in LocalSettings.php. �By
> contrast, python2.6 is around 10 MB uncompressed, 2.5 MB compressed.
> Perl is twice that size. �Windows users, or users with exec() allowed
> but open_basedir preventing access to /usr/bin, would have to obtain
> Python/Perl/etc. themselves.
>
> It looks to me like Lua would be a lot easier to sandbox. �It seems
> pretty simple to deny all I/O within the language itself, so you'd
> (hopefully) just need memory and CPU limits. �Both of those could be
> implemented on Linux with hard setrlimit() values plus nice. �Similar
> things exist on Windows, hopefully accessible by command line somehow.
> �If we're shipping binaries with MediaWiki, we could even hack the
> code if necessary, to use whatever sandboxing mechanisms the OS makes
> available, although hopefully that would be unneeded.
>
> I don't think we should fixate too much on how many people know the
> language. �It's not hard to pick up a new language if you already know
> one, and Lua has the reputation of being simple (although I haven't
> tried to learn it). �I think Lua is the best option here.
>
> _______________________________________________
> Wikitech-l mailing list
> Wikitech-l [at] lists
> https://lists.wikimedia.org/mailman/listinfo/wikitech-l
>

In addition to resource limits, any scheme better make sure what's
passed into the programming language and what's passed out makes
sense. For example, you shouldn't have it generating raw HTML and
probably shouldn't let it mess with strip markers. Some of this may
be automatic depending how it's integrated into the parser. One would
probably also want to limit the size of an allowed output (e.g. don't
let it send 5 MB to the user). Depending on the integration there may
be other control sequences that one needs to catch when it returns as
well.

On a separate point, one of the limitations of stand-alone type
sandboxes is that it would make it harder for the code to call other
template pages. One of the few virtues of the current template code
is that it is relatively modular, with more complex templates being
built out of less complex ones. If this programming language is meant
to replace that then it would also need to be able to reference the
results of other template pages. One solution is to pre-expand those
sections (similar to what is done now, I believe), but that can get
rather delicate once one has programming constructs like variable
assignments, looping, and recursion since the template parameters
won't necessarily be fixed at the Preprocessor stage.

-Robert Rohde

_______________________________________________
Wikitech-l mailing list
Wikitech-l [at] lists
https://lists.wikimedia.org/mailman/listinfo/wikitech-l

jared.williams1 at ntlworld

Jun 30, 2009, 3:12 PM

Post #35 of 116 (1323 views)
Permalink

Re: On templates and programming languages [In reply to]

> -----Original Message-----
> From: wikitech-l-bounces [at] lists
> [mailto:wikitech-l-bounces [at] lists] On Behalf Of
> Aryeh Gregor
> Sent: 30 June 2009 22:16
> To: Wikimedia developers
> Subject: Re: [Wikitech-l] On templates and programming languages
>
> On Tue, Jun 30, 2009 at 4:52 PM, Jared
> Williams<jared.williams1 [at] ntlworld> wrote:
> > Pity there is not a method of locking down code execution
> to a single
> > namespace, (think ahead with php5.3)
>
> This is implausible, but even if it happened it wouldn't stop
> trivial DOSes like while (true);. We'd still need to
> validate the code if we wanted to run it in-process.
>

Yeah, would also need time & mem use restrictions.

Jared

_______________________________________________
Wikitech-l mailing list
Wikitech-l [at] lists
https://lists.wikimedia.org/mailman/listinfo/wikitech-l

removed at example

Jun 30, 2009, 4:01 PM

Post #36 of 116 (1324 views)
Permalink

Re: On templates and programming languages [In reply to]

On Tue, Jun 30, 2009 at 11:20 AM, Robert Rohde <rarohde [at] gmail> wrote:
>
> However,
> given the nastiness of template syntax, I would expect no end of wiki
> authors willing to help convert the commonly used stuff.
>
> -Robert Rohde

I was curious just how terrible of a task conversion can be expected
to be. This is just a heuristic I came up with..

# Simple English parser functions
$ bunzip2 -c simplewiki-20090623-pages-articles.xml.bz2 | grep -o '{{#' | wc -l
22,211

# Simple English templates
$ bunzip2 -c simplewiki-20090623-pages-articles.xml.bz2 | grep -o '{{' | wc -l
416,126 - 22,211 = 393,915

# English parser functions
$� bunzip2 -c enwiki-20090618-pages-articles.xml.bz2 | grep -o '{{#' | wc -l
430,980

# English templates
$ bunzip2 -c enwiki-20090618-pages-articles.xml.bz2 | grep -o '{{' | wc -l
44,928,358 - 430,980 = 44,497,378

_______________________________________________
Wikitech-l mailing list
Wikitech-l [at] lists
https://lists.wikimedia.org/mailman/listinfo/wikitech-l

agarrett at wikimedia

Jun 30, 2009, 4:16 PM

Post #37 of 116 (1323 views)
Permalink

Re: On templates and programming languages [In reply to]

On 30/06/2009, at 9:42 PM, Aryeh Gregor wrote:

> On Tue, Jun 30, 2009 at 4:25 PM, Brion Vibber<brion [at] wikimedia>
> wrote:
>> IMO by the time you've implemented your whitelisting parser you
>> might as
>> well just interpret it rather than eval()ing.
>
> I don't think so. You'd only have to do the whitelisting once, on
> page save. After that you could just execute with no extra overhead.

That's just scary. We'd definitely want to do the validation as close
as possible to the actual eval()ing, to minimise backdoors like
Special:Import et al.

--
Andrew Garrett
Contract Developer, Wikimedia Foundation
agarrett [at] wikimedia
http://werdn.us

_______________________________________________
Wikitech-l mailing list
Wikitech-l [at] lists
https://lists.wikimedia.org/mailman/listinfo/wikitech-l

huskyr at gmail

Jun 30, 2009, 4:24 PM

Post #38 of 116 (1321 views)
Permalink

Re: On templates and programming languages [In reply to]

I would opt for Javascript.

PHP and Python are intended for large and complex applications and
come with a huge standard library people probably expect to be
available. Security concerns are a problem too, so a subset would
probably be necessary So, in essence you get a crippled-down language
that isn't really useful for templates.

Making our own language, either by 'fixing' the template language or
by inventing something new would only mean we introduce a new language
that'll be specific to our own platform and nobody knows outside of
Mediawiki developers.

XSLT is not meant to be written or read by humans. It's a
Turing-complete language stuffed into horrendous XML statements. Let's
not go down that road.

That leaves us to Lua and Javascript, which are both small and
efficient languages meant to solve tasks like this. Remember, i'm
talking about 'core' Javascript here, not with all DOM methods and
stuff. If you strip that all out (take a look at the 1.5. core
reference at Mozilla.com:
https://developer.mozilla.org/en/Core_JavaScript_1.5_Reference) you
get a pretty nice and simple language that isn't very large. Both
would require a new parser and/or installed compilers on the
server-side. Compared to the disadvantages of other options, that
seems like a pretty small loss for a great win.

Javascript is a widely understood and implemented language, with lots
of efforts to get it even faster in modern browsers. Every Wikipedia
user has a copy of it implemented in their browser and can start
experimenting without the need for installing a compiler or a web
server. Many people program in Javascript, so you have a huge
potential number of people who could start programming Mediawiki
templates. And it's already closely tied to the web, so you don't have
to invent new ways of dealing with web-specific stuff.

So, let's choose Javascript as our new template programming language.

Regards,
-- Hay

On Tue, Jun 30, 2009 at 6:16 PM, Brion Vibber<brion [at] wikimedia> wrote:
> As many folks have noted, our current templating system works ok for
> simple things, but doesn't scale well -- even moderately complex
> conditionals or text-munging will quickly turn your template source into
> what appears to be line noise.
>
> And we all thought Perl was bad! ;)
>
> There's been talk of Lua as an embedded templating language for a while,
> and there's even an extension implementation.
>
> One advantage of Lua over other languages is that its implementation is
> optimized for use as an embedded language, and it looks kind of pretty.
>
> An _inherent_ disadvantage is that it's a fairly rarely-used language,
> so still requires special learning on potential template programmers' part.
>
> An _implementation_ disadvantage is that it currently is dependent on an
> external Lua binary installation -- something that probably won't be
> present on third-party installs, meaning Lua templates couldn't be
> easily copied to non-Wikimedia wikis.
>
>
> There are perhaps three primary alternative contenders that don't
> involve making up our own scripting language (something I'd dearly like
> to avoid):
>
> * PHP
>
> Advantage: Lots of webbish people have some experience with PHP or can
> easily find references.
>
> Advantage: we're pretty much guaranteed to have a PHP interpreter
> available. :)
>
> Disadvantage: PHP is difficult to lock down for secure execution.
>
>
> * JavaScript
>
> Advantage: Even more folks have been exposed to JavaScript programming,
> including Wikipedia power-users.
>
> Disadvantage: Server-side interpreter not guaranteed to be present. Like
> Lua, would either restrict our portability or would require an
> interpreter reimplementation. :P
>
>
> * Python
>
> Advantage: A Python interpreter will be present on most web servers,
> though not necessarily all. (Windows-based servers especially.)
>
> Wash: Python is probably better known than Lua, but not as well as PHP
> or JS.
>
> Disadvantage: Like PHP, Python is difficult to lock down securely.
>
>
> Any thoughts? Does anybody happen to have a PHP implementation of a Lua
> or JavaScript interpreter? ;)
>
> -- brion
>
> _______________________________________________
> Wikitech-l mailing list
> Wikitech-l [at] lists
> https://lists.wikimedia.org/mailman/listinfo/wikitech-l
>

_______________________________________________
Wikitech-l mailing list
Wikitech-l [at] lists
https://lists.wikimedia.org/mailman/listinfo/wikitech-l

tparscal at wikimedia

Jun 30, 2009, 4:44 PM

Post #39 of 116 (1320 views)
Permalink

Re: On templates and programming languages [In reply to]

I personally agree entirely. Now we just need to revive J4P5
(http://j4p5.sourceforge.net) :)

- Trevor

On 6/30/09 4:24 PM, Hay (Husky) wrote:
> I would opt for Javascript.
>
> PHP and Python are intended for large and complex applications and
> come with a huge standard library people probably expect to be
> available. Security concerns are a problem too, so a subset would
> probably be necessary So, in essence you get a crippled-down language
> that isn't really useful for templates.
>
> Making our own language, either by 'fixing' the template language or
> by inventing something new would only mean we introduce a new language
> that'll be specific to our own platform and nobody knows outside of
> Mediawiki developers.
>
> XSLT is not meant to be written or read by humans. It's a
> Turing-complete language stuffed into horrendous XML statements. Let's
> not go down that road.
>
> That leaves us to Lua and Javascript, which are both small and
> efficient languages meant to solve tasks like this. Remember, i'm
> talking about 'core' Javascript here, not with all DOM methods and
> stuff. If you strip that all out (take a look at the 1.5. core
> reference at Mozilla.com:
> https://developer.mozilla.org/en/Core_JavaScript_1.5_Reference) you
> get a pretty nice and simple language that isn't very large. Both
> would require a new parser and/or installed compilers on the
> server-side. Compared to the disadvantages of other options, that
> seems like a pretty small loss for a great win.
>
> Javascript is a widely understood and implemented language, with lots
> of efforts to get it even faster in modern browsers. Every Wikipedia
> user has a copy of it implemented in their browser and can start
> experimenting without the need for installing a compiler or a web
> server. Many people program in Javascript, so you have a huge
> potential number of people who could start programming Mediawiki
> templates. And it's already closely tied to the web, so you don't have
> to invent new ways of dealing with web-specific stuff.
>
> So, let's choose Javascript as our new template programming language.
>
> Regards,
> -- Hay
>
> On Tue, Jun 30, 2009 at 6:16 PM, Brion Vibber<brion [at] wikimedia> wrote:
>
>> As many folks have noted, our current templating system works ok for
>> simple things, but doesn't scale well -- even moderately complex
>> conditionals or text-munging will quickly turn your template source into
>> what appears to be line noise.
>>
>> And we all thought Perl was bad! ;)
>>
>> There's been talk of Lua as an embedded templating language for a while,
>> and there's even an extension implementation.
>>
>> One advantage of Lua over other languages is that its implementation is
>> optimized for use as an embedded language, and it looks kind of pretty.
>>
>> An _inherent_ disadvantage is that it's a fairly rarely-used language,
>> so still requires special learning on potential template programmers' part.
>>
>> An _implementation_ disadvantage is that it currently is dependent on an
>> external Lua binary installation -- something that probably won't be
>> present on third-party installs, meaning Lua templates couldn't be
>> easily copied to non-Wikimedia wikis.
>>
>>
>> There are perhaps three primary alternative contenders that don't
>> involve making up our own scripting language (something I'd dearly like
>> to avoid):
>>
>> * PHP
>>
>> Advantage: Lots of webbish people have some experience with PHP or can
>> easily find references.
>>
>> Advantage: we're pretty much guaranteed to have a PHP interpreter
>> available. :)
>>
>> Disadvantage: PHP is difficult to lock down for secure execution.
>>
>>
>> * JavaScript
>>
>> Advantage: Even more folks have been exposed to JavaScript programming,
>> including Wikipedia power-users.
>>
>> Disadvantage: Server-side interpreter not guaranteed to be present. Like
>> Lua, would either restrict our portability or would require an
>> interpreter reimplementation. :P
>>
>>
>> * Python
>>
>> Advantage: A Python interpreter will be present on most web servers,
>> though not necessarily all. (Windows-based servers especially.)
>>
>> Wash: Python is probably better known than Lua, but not as well as PHP
>> or JS.
>>
>> Disadvantage: Like PHP, Python is difficult to lock down securely.
>>
>>
>> Any thoughts? Does anybody happen to have a PHP implementation of a Lua
>> or JavaScript interpreter? ;)
>>
>> -- brion
>>
>> _______________________________________________
>> Wikitech-l mailing list
>> Wikitech-l [at] lists
>> https://lists.wikimedia.org/mailman/listinfo/wikitech-l
>>
>>
>
> _______________________________________________
> Wikitech-l mailing list
> Wikitech-l [at] lists
> https://lists.wikimedia.org/mailman/listinfo/wikitech-l
>

_______________________________________________
Wikitech-l mailing list
Wikitech-l [at] lists
https://lists.wikimedia.org/mailman/listinfo/wikitech-l

rarohde at gmail

Jun 30, 2009, 5:09 PM

Post #40 of 116 (1329 views)
Permalink

Re: On templates and programming languages [In reply to]

On Tue, Jun 30, 2009 at 4:01 PM, Brian<Brian.Mingus [at] colorado> wrote:
> On Tue, Jun 30, 2009 at 11:20 AM, Robert Rohde <rarohde [at] gmail> wrote:
>>
>> However,
>> given the nastiness of template syntax, I would expect no end of wiki
>> authors willing to help convert the commonly used stuff.
>>
>> -Robert Rohde
>
> I was curious just how terrible of a task conversion can be expected
> to be. This is just a heuristic I came up with..
>
> # Simple English parser functions
> $ bunzip2 -c simplewiki-20090623-pages-articles.xml.bz2 | grep -o '{{#' | wc -l
> 22,211
>
> # Simple English templates
> $ bunzip2 -c simplewiki-20090623-pages-articles.xml.bz2 | grep -o '{{' | wc -l
> 416,126 - 22,211 = 393,915
>
> # English parser functions
> $� bunzip2 -c enwiki-20090618-pages-articles.xml.bz2 | grep -o '{{#' | wc -l
> 430,980
>
> # English templates
> $ �bunzip2 -c enwiki-20090618-pages-articles.xml.bz2 | grep -o '{{' | wc -l
> 44,928,358 - 430,980 = 44,497,378

I assume we are primarily talking about replacing template code and
not template calls, per se.

In other words, I assume things like "{{fact}}" and "{{msg | foo is
bar }}" will be be basically unchanged on the article side but
rewritten on the implementation side in Template: space. If that is
correct, it would be more useful to simply ask how large Template:
space is rather than counting all the template calls.

-Robert Rohde

_______________________________________________
Wikitech-l mailing list
Wikitech-l [at] lists
https://lists.wikimedia.org/mailman/listinfo/wikitech-l

removed at example

Jun 30, 2009, 5:28 PM

Post #41 of 116 (1320 views)
Permalink

Re: On templates and programming languages [In reply to]

On Tue, Jun 30, 2009 at 6:09 PM, Robert Rohde<rarohde [at] gmail> wrote:
> In other words, I assume things like "{{fact}}" and "{{msg | foo is
> bar }}" will be be basically unchanged on the article side but
> rewritten on the implementation side in Template: space. �If that is
> correct, it would be more useful to simply ask how large Template:
> space is rather than counting all the template calls.
>
> -Robert Rohde

Mixing the new language with existing wikicode? With a new language I
would like to see the old language go out the door. The end of double
braces.

_______________________________________________
Wikitech-l mailing list
Wikitech-l [at] lists
https://lists.wikimedia.org/mailman/listinfo/wikitech-l

brion at wikimedia

Jun 30, 2009, 5:33 PM

Post #42 of 116 (1318 views)
Permalink

Re: On templates and programming languages [In reply to]

Aryeh Gregor wrote:
> On Tue, Jun 30, 2009 at 4:25 PM, Brion Vibber<brion [at] wikimedia> wrote:
>> It's not about "Wikipedia content", but about being able to grab things
>> you see on another wiki and use or adapt them to your own needs. We get
>> lots of questions from people trying to grab some particular template
>> off Wikipedia to use on their own site for their own needs.
>
> Sure. The point still holds. Some third parties would be unable to
> use Wikipedia templates, yes. But given the tangle of dependencies
> the major ones have, and how complicated they are, I'm guessing most
> small third-party wikis don't bother in the end anyway.

That's why we want to fix it! :)

It *should* be fairly trivial to fetch a template/plugin sort of thing
off of one wiki and put it on another. Consider this as one of our goals
for next-gen templating.

-- brion

_______________________________________________
Wikitech-l mailing list
Wikitech-l [at] lists
https://lists.wikimedia.org/mailman/listinfo/wikitech-l

thomas.dalton at gmail

Jun 30, 2009, 5:34 PM

Post #43 of 116 (1320 views)
Permalink

Re: On templates and programming languages [In reply to]

2009/7/1 Brian <Brian.Mingus [at] colorado>:
> On Tue, Jun 30, 2009 at 6:09 PM, Robert Rohde<rarohde [at] gmail> wrote:
>> In other words, I assume things like "{{fact}}" and "{{msg | foo is
>> bar }}" will be be basically unchanged on the article side but
>> rewritten on the implementation side in Template: space. If that is
>> correct, it would be more useful to simply ask how large Template:
>> space is rather than counting all the template calls.
>>
>> -Robert Rohde
>
> Mixing the new language with existing wikicode? With a new language I
> would like to see the old language go out the door. The end of double
> braces.

What would you replace them with? The wikitext used by regular editors
should be as simple as possible, we don't want to require PHP or
Javascript to be used by anyone wanting to add an infobox to an
article.

_______________________________________________
Wikitech-l mailing list
Wikitech-l [at] lists
https://lists.wikimedia.org/mailman/listinfo/wikitech-l

innocentkiller at gmail

Jun 30, 2009, 5:34 PM

Post #44 of 116 (1320 views)
Permalink

Re: On templates and programming languages [In reply to]

On Tue, Jun 30, 2009 at 8:28 PM, Brian<Brian.Mingus [at] colorado> wrote:
> On Tue, Jun 30, 2009 at 6:09 PM, Robert Rohde<rarohde [at] gmail> wrote:
>> In other words, I assume things like "{{fact}}" and "{{msg | foo is
>> bar }}" will be be basically unchanged on the article side but
>> rewritten on the implementation side in Template: space. If that is
>> correct, it would be more useful to simply ask how large Template:
>> space is rather than counting all the template calls.
>>
>> -Robert Rohde
>
> Mixing the new language with existing wikicode? With a new language I
> would like to see the old language go out the door. The end of double
> braces.
>
> _______________________________________________
> Wikitech-l mailing list
> Wikitech-l [at] lists
> https://lists.wikimedia.org/mailman/listinfo/wikitech-l
>

Unless we plan on trying to mass-convert not only years of old revisions
but change years-old behavior that millions of users have come to expect?
I would expect _any_ change to keep {{sometemplate}} always working,
even if the mechanics behind it change.

-Chad

_______________________________________________
Wikitech-l mailing list
Wikitech-l [at] lists
https://lists.wikimedia.org/mailman/listinfo/wikitech-l

brion at wikimedia

Jun 30, 2009, 5:37 PM

Post #45 of 116 (1317 views)
Permalink

Re: On templates and programming languages [In reply to]

Andrew Garrett wrote:
> On 30/06/2009, at 9:42 PM, Aryeh Gregor wrote:
>
>> On Tue, Jun 30, 2009 at 4:25 PM, Brion Vibber<brion [at] wikimedia>
>> wrote:
>>> IMO by the time you've implemented your whitelisting parser you
>>> might as
>>> well just interpret it rather than eval()ing.
>> I don't think so. You'd only have to do the whitelisting once, on
>> page save. After that you could just execute with no extra overhead.
>
> That's just scary. We'd definitely want to do the validation as close
> as possible to the actual eval()ing, to minimise backdoors like
> Special:Import et al.

Executing PHP from apache-writable files saved on disk is also a
security danger.

The original implementation of the MonoBook skin used the TAL templating
language, which was compiled into executable PHP at runtime and stored
in /tmp so it could be cached for the next view.

In addition to difficulties with hosts which had misconfigured /tmp
directories, we found that people sharing their hosts with
poorly-secured WordPress installations would end up finding their wikis
hacked -- worms exploiting vulnerabilities in other PHP apps would hop
around the system modifying any .php files they could write to...
including the cached PHPTAL templates.

-- brion

_______________________________________________
Wikitech-l mailing list
Wikitech-l [at] lists
https://lists.wikimedia.org/mailman/listinfo/wikitech-l

removed at example

Jun 30, 2009, 5:41 PM

Post #46 of 116 (1319 views)
Permalink

Re: On templates and programming languages [In reply to]

On Tue, Jun 30, 2009 at 6:34 PM, Thomas Dalton<thomas.dalton [at] gmail> wrote:
> What would you replace them with? The wikitext used by regular editors
> should be as simple as possible, we don't want to require PHP or
> Javascript to be used by anyone wanting to add an infobox to an
> article.

There is nothing in the OP that indicates that we are keeping the
current template code or even that it would be desirable. Whatever
facilities the language we choose has for including other files and
passing arguments to functions is 100% sufficient.

_______________________________________________
Wikitech-l mailing list
Wikitech-l [at] lists
https://lists.wikimedia.org/mailman/listinfo/wikitech-l

thomas.dalton at gmail

Jun 30, 2009, 5:43 PM

Post #47 of 116 (1322 views)
Permalink

Re: On templates and programming languages [In reply to]

2009/7/1 Brian <Brian.Mingus [at] colorado>:
> On Tue, Jun 30, 2009 at 6:34 PM, Thomas Dalton<thomas.dalton [at] gmail> wrote:
>> What would you replace them with? The wikitext used by regular editors
>> should be as simple as possible, we don't want to require PHP or
>> Javascript to be used by anyone wanting to add an infobox to an
>> article.
>
> There is nothing in the OP that indicates that we are keeping the
> current template code or even that it would be desirable. Whatever
> facilities the language we choose has for including other files and
> passing arguments to functions is 100% sufficient.

There is no proposal to replace wikitext with PHP (it wouldn't even
work, PHP isn't a markup language, ditto Javascript, Python, etc.),
the proposal is to replace the template code, ie. the code on the
template pages.

_______________________________________________
Wikitech-l mailing list
Wikitech-l [at] lists
https://lists.wikimedia.org/mailman/listinfo/wikitech-l

removed at example

Jun 30, 2009, 5:56 PM

Post #48 of 116 (1318 views)
Permalink

Re: On templates and programming languages [In reply to]

On Tue, Jun 30, 2009 at 6:43 PM, Thomas Dalton<thomas.dalton [at] gmail> wrote:
> There is no proposal to replace wikitext with PHP (it wouldn't even
> work, PHP isn't a markup language, ditto Javascript, Python, etc.),
> the proposal is to replace the template code, ie. the code on the
> template pages.

The OP does not say it is a recommendation to replace ParserFunctions,
it says, "our current templating system." In my mind that absolutely
includes the use of templates in the article namespace.

There are lots of usability improvements that can be made to the
templating system. First and foremost the new system should allow
advanced wiki users to perform programmatic operations on article data
without the requirement that the data in the article be made
unreadable.

If we only focus our efforts on making the template namespace more
complicated by giving it a more advanced programming language and we
leave the article namespace as it is then we have not even touched the
usability issue. We have just made it worse.

I do of course have some specific ideas about how to achieve this
goal, but I'm kind of in "shock and awe" that it's not seen as the
main reason for improving the template system!

_______________________________________________
Wikitech-l mailing list
Wikitech-l [at] lists
https://lists.wikimedia.org/mailman/listinfo/wikitech-l

tparscal at wikimedia

Jun 30, 2009, 6:02 PM

Post #49 of 116 (1328 views)
Permalink

Re: On templates and programming languages [In reply to]

Seems like JSON syntax is pretty simple and could be a big improvement
to how templates are currently invoked.

Bottom line, a well defined syntax like JavaScript is going to be more
user friendly than a syntax which is only defined by the behavior of a
parser with standardization at all.

- Trevor

Sent from my iPod

On Jun 30, 2009, at 5:34 PM, Thomas Dalton <thomas.dalton [at] gmail>
wrote:

> 2009/7/1 Brian <Brian.Mingus [at] colorado>:
>> On Tue, Jun 30, 2009 at 6:09 PM, Robert Rohde<rarohde [at] gmail>
>> wrote:
>>> In other words, I assume things like "{{fact}}" and "{{msg | foo is
>>> bar }}" will be be basically unchanged on the article side but
>>> rewritten on the implementation side in Template: space. If that is
>>> correct, it would be more useful to simply ask how large Template:
>>> space is rather than counting all the template calls.
>>>
>>> -Robert Rohde
>>
>> Mixing the new language with existing wikicode? With a new language I
>> would like to see the old language go out the door. The end of double
>> braces.
>
> What would you replace them with? The wikitext used by regular editors
> should be as simple as possible, we don't want to require PHP or
> Javascript to be used by anyone wanting to add an infobox to an
> article.
>
> _______________________________________________
> Wikitech-l mailing list
> Wikitech-l [at] lists
> https://lists.wikimedia.org/mailman/listinfo/wikitech-l

_______________________________________________
Wikitech-l mailing list
Wikitech-l [at] lists
https://lists.wikimedia.org/mailman/listinfo/wikitech-l

randomcoder1 at gmail

Jun 30, 2009, 6:21 PM

Post #50 of 116 (1325 views)
Permalink

Re: On templates and programming languages [In reply to]

http://jtemplates.tpython.com/ ? :)

Trevor Parscal wrote:
> Seems like JSON syntax is pretty simple and could be a big improvement
> to how templates are currently invoked.
>
> Bottom line, a well defined syntax like JavaScript is going to be more
> user friendly than a syntax which is only defined by the behavior of a
> parser with standardization at all.
>
> - Trevor
>
> Sent from my iPod
>
> On Jun 30, 2009, at 5:34 PM, Thomas Dalton <thomas.dalton [at] gmail>
> wrote:
>
>
>> 2009/7/1 Brian <Brian.Mingus [at] colorado>:
>>
>>> On Tue, Jun 30, 2009 at 6:09 PM, Robert Rohde<rarohde [at] gmail>
>>> wrote:
>>>
>>>> In other words, I assume things like "{{fact}}" and "{{msg | foo is
>>>> bar }}" will be be basically unchanged on the article side but
>>>> rewritten on the implementation side in Template: space. If that is
>>>> correct, it would be more useful to simply ask how large Template:
>>>> space is rather than counting all the template calls.
>>>>
>>>> -Robert Rohde
>>>>
>>> Mixing the new language with existing wikicode? With a new language I
>>> would like to see the old language go out the door. The end of double
>>> braces.
>>>
>> What would you replace them with? The wikitext used by regular editors
>> should be as simple as possible, we don't want to require PHP or
>> Javascript to be used by anyone wanting to add an infobox to an
>> article.
>>
>> _______________________________________________
>> Wikitech-l mailing list
>> Wikitech-l [at] lists
>> https://lists.wikimedia.org/mailman/listinfo/wikitech-l
>>
>
> _______________________________________________
> Wikitech-l mailing list
> Wikitech-l [at] lists
> https://lists.wikimedia.org/mailman/listinfo/wikitech-l
>

_______________________________________________
Wikitech-l mailing list
Wikitech-l [at] lists
https://lists.wikimedia.org/mailman/listinfo/wikitech-l

1 2 3 4 5

View All

Index | Next | Previous | View Threaded

Interested in having your list archived? Contact Gossamer Threads