Login | Register For Free | Help
Search for: (Advanced)

Mailing List Archive: Wikipedia: Wikitech

On templates and programming languages

 

 

First page Previous page 1 2 3 4 5 Next page Last page  View All Wikipedia wikitech RSS feed   Index | Next | Previous | View Threaded


bryan.tongminh at gmail

Jul 3, 2009, 3:00 AM

Post #101 of 116 (1009 views)
Permalink
Re: On templates and programming languages [In reply to]

On Thu, Jul 2, 2009 at 7:32 PM, Brion Vibber<brion [at] wikimedia> wrote:
>
> I'd really _like_ to be able to avoid having to require external
> executables either, if it can be managed, but that's harder since it
> means having a pure PHP implementation of the scripting language. (ouch!)
>
> -- brion
>
We could always have a default implementation in PHP, and optionally
provide the same functionality but faster in a C module.


Bryan

_______________________________________________
Wikitech-l mailing list
Wikitech-l [at] lists
https://lists.wikimedia.org/mailman/listinfo/wikitech-l


jared.williams1 at ntlworld

Jul 3, 2009, 3:41 AM

Post #102 of 116 (1011 views)
Permalink
Re: On templates and programming languages [In reply to]

> -----Original Message-----
> From: wikitech-l-bounces [at] lists
> [mailto:wikitech-l-bounces [at] lists] On Behalf Of
> Dmitriy Sintsov
> Sent: 03 July 2009 07:48
> To: Wikimedia developers
> Subject: Re: [Wikitech-l] On templates and programming languages
>
> * Brion Vibber <brion [at] wikimedia> [Thu, 02 Jul 2009
> 10:18:14 -0700]:
> > Aryeh Gregor wrote:
> > > I was assuming it would just return wikitext, and that would be
> > > integrated into the page and parsed, following all limits on
> wikitext
> > > (including size) -- just as with current parser functions.
> >
> > That's one simple way to implement, but we may wish to consider
> working
> > with a document tree structure instead to help future-proof
> it against
> > future syntax changes (or dropping out the wiki syntax entirely).
> Things
> > to consider... :)
> >
> SLAX http://code.google.com/p/libslax/ (provided by Gregory
> Maxwell) looks like really good thing for document tree
> manipulation and as the people have pointed out, XSLT is
> simle to limit (lock the recursion down). It's compact and
> more easily readable comparing to "normal" xslt.
> I remember that PHP has some standard module for XSLT
> transformations, I wonder whether it's simple to convert
> SLAX->XSLT then use PHP XSLT transformation.
> Dmitriy
>

Think something like ESI language (http://www.w3.org/TR/esi-lang
but without the HTTP requests) would be preferable to XSLT.
If was going the XML route.

Jared


_______________________________________________
Wikitech-l mailing list
Wikitech-l [at] lists
https://lists.wikimedia.org/mailman/listinfo/wikitech-l


agarrett at wikimedia

Jul 3, 2009, 5:03 AM

Post #103 of 116 (1008 views)
Permalink
Re: On templates and programming languages [In reply to]

On 03/07/2009, at 11:00 AM, Bryan Tong Minh wrote:

> On Thu, Jul 2, 2009 at 7:32 PM, Brion Vibber<brion [at] wikimedia>
> wrote:
>>
>> I'd really _like_ to be able to avoid having to require external
>> executables either, if it can be managed, but that's harder since it
>> means having a pure PHP implementation of the scripting language.
>> (ouch!)
>>
>> -- brion
>>
> We could always have a default implementation in PHP, and optionally
> provide the same functionality but faster in a C module.


Writing an interpreter for a language is not trivial. Writing it in C,
and then porting to PHP is even worse.

Even if you could find somebody to write it, it would have to be
reviewed, as well.

--
Andrew Garrett
Contract Developer, Wikimedia Foundation
agarrett [at] wikimedia
http://werdn.us




_______________________________________________
Wikitech-l mailing list
Wikitech-l [at] lists
https://lists.wikimedia.org/mailman/listinfo/wikitech-l


questpc at rambler

Jul 3, 2009, 7:56 AM

Post #104 of 116 (1008 views)
Permalink
Re: On templates and programming languages [In reply to]

* Andrew Garrett <agarrett [at] wikimedia> [Fri, 3 Jul 2009 13:03:03
+0100]:
>
> Writing an interpreter for a language is not trivial. Writing it in C,
> and then porting to PHP is even worse.
>
Many of languages "resemble" syntax of C language (curly braces,
plusplus/minusminus and so on), but with "lousy" typing (numerical
strings transparently mixed with numbers, floats "mixed" with integers).
Add a dollar sign prefix to variable names to JS code and lots of
_simple_ JS code would be really similar to PHP. I wonder whether that
helps to translate?
BTW, one of LOGO interpreter distributions (an educational functional
language) has a partial (incomplete but working) interpreter of Pascal
language just in about 20-30KB! It's amazing! it seems that writing
interpreter with another interpreter is much easier than by using
low-level language like C. I am not sure whether Logo is suitable,
though - I am not expert in translation in any way.
Dmitriy

_______________________________________________
Wikitech-l mailing list
Wikitech-l [at] lists
https://lists.wikimedia.org/mailman/listinfo/wikitech-l


Simetrical+wikilist at gmail

Jul 3, 2009, 9:54 AM

Post #105 of 116 (1007 views)
Permalink
Re: On templates and programming languages [In reply to]

On Fri, Jul 3, 2009 at 3:13 AM, Tim Starling<tstarling [at] wikimedia> wrote:
> Loops are essential for readable code. There is no problem with
> allowing loops in conjunction with time limits, that we don't have
> already with complex templates. In fact, time limits for complex
> templates would be an improvement over the system of expansion limits
> we have at the moment.

But time limits are inconsistent. Whether a template hits the limit
might depend on whether it happens to be running on an Apache with a
Pentium IV, an Opteron, a Xeon, . . .

> Recursion can give a long running time even if the depth is limited.
> By calling the function multiple times from its own body, you can have
> exponential time order in the recursion depth.

You can also have exponential time with loops.

_______________________________________________
Wikitech-l mailing list
Wikitech-l [at] lists
https://lists.wikimedia.org/mailman/listinfo/wikitech-l


Platonides at gmail

Jul 3, 2009, 10:00 AM

Post #106 of 116 (1007 views)
Permalink
Re: On templates and programming languages [In reply to]

Tim Starling wrote:
>> * PHP
>>
>> Advantage: Lots of webbish people have some experience with PHP or can
>> easily find references.
>>
>> Advantage: we're pretty much guaranteed to have a PHP interpreter
>> available. :)
>>
>> Disadvantage: PHP is difficult to lock down for secure execution.
>
> PHP can be secured against arbitrary execution using token_get_all(),
> there's a proof-of-principle validator of this kind in the master
> switch script project. But there are problems with attempting a
> single-process PHP-in-PHP sandbox:
>
> * The poor support for signals in PHP makes it difficult to limit the
> execution time of a script snippet. Ticks only occur at the end of
> each statement, so you can defeat them by making a single statement
> that runs forever.

Inject a check_limits() call into each looping structure. If it detects
the script has been running for more than $maxTime, timeout it.
Can you defeat that?


> * Apart from blacklisting function definition, there is no way to
> protect against infinite recursion, which exhausts the process stack
> and causes a segfault.

Also inject the same call into functions.


> * Memory limits are implemented on a per-request basis, and there's no
> way to recover from exceeding the memory limit, the request is just
> killed.

Call memory_get_usage() before and also inside check_limits() to check
script abides inside memory limits. Abort if it gets near php memory
limit (I'd expect the script's memory to be much lower than php's).
However, that check is much easier to bypass.



_______________________________________________
Wikitech-l mailing list
Wikitech-l [at] lists
https://lists.wikimedia.org/mailman/listinfo/wikitech-l


sergey.chernyshev at gmail

Jul 3, 2009, 10:27 AM

Post #107 of 116 (1015 views)
Permalink
Re: On templates and programming languages [In reply to]

I think you're confusing simple logic of ParserFunctions in the template
with a full scripting language like PHP.

That's why I proposed to look at something simplified like Smarty or alike.

Thank you,

Sergey


--
Sergey Chernyshev
http://www.sergeychernyshev.com/


On Tue, Jun 30, 2009 at 11:46 PM, Aryeh Gregor <
Simetrical+wikilist [at] gmail <Simetrical%2Bwikilist [at] gmail>> wrote:

> On Tue, Jun 30, 2009 at 10:45 PM, Sergey
> Chernyshev<sergey.chernyshev [at] gmail> wrote:
> > I don't know about scripting languages for the templating, it might be an
> > overkill.
>
> People are using ParserFunctions as a scripting language already.
> That's not feasibly going to be removed at this point. So the only
> way to go is to replace it with a better scripting language, which is
> what we're talking about.
>
> _______________________________________________
> Wikitech-l mailing list
> Wikitech-l [at] lists
> https://lists.wikimedia.org/mailman/listinfo/wikitech-l
>
_______________________________________________
Wikitech-l mailing list
Wikitech-l [at] lists
https://lists.wikimedia.org/mailman/listinfo/wikitech-l


petr.kadlec at gmail

Jul 3, 2009, 10:37 AM

Post #108 of 116 (1008 views)
Permalink
Re: On templates and programming languages [In reply to]

2009/7/3 Aryeh Gregor <Simetrical+wikilist [at] gmail>:
> But time limits are inconsistent.  Whether a template hits the limit
> might depend on whether it happens to be running on an Apache with a
> Pentium IV, an Opteron, a Xeon, . . .

And they might depend on the server load at the time, which is
especially problematic if the script produces page rendering output,
which gets cached. A temporary server overload might cause caching of
broken pages (which render fine otherwise). Some invariant measure
(like instruction count etc.) would be great (and much more
complicated, unless we would use our own interpreter of the respective
scripting language).

-- [[cs:User:Mormegil | Petr Kadlec]]

_______________________________________________
Wikitech-l mailing list
Wikitech-l [at] lists
https://lists.wikimedia.org/mailman/listinfo/wikitech-l


Simetrical+wikilist at gmail

Jul 3, 2009, 12:15 PM

Post #109 of 116 (1008 views)
Permalink
Re: On templates and programming languages [In reply to]

On Fri, Jul 3, 2009 at 1:27 PM, Sergey
Chernyshev<sergey.chernyshev [at] gmail> wrote:
> I think you're confusing simple logic of ParserFunctions in the template
> with a full scripting language like PHP.

In what way is the logic of ParserFunctions "simple"? If you ignore
the limitations on parse length, it's Turing-complete.

> That's why I proposed to look at something simplified like Smarty or alike.

Hmm. Smarty looks interesting, at a quick glance. I suspect it's not
designed to be secure against DoS, so it would need some kind of
sandboxing. Hopefully less than some of the other solutions we're
contemplating, though! I'd think it might serve okay, if we wrote
enough custom functions to replace the existing ParserFunctions. I'm
not sure.

_______________________________________________
Wikitech-l mailing list
Wikitech-l [at] lists
https://lists.wikimedia.org/mailman/listinfo/wikitech-l


Simetrical+wikilist at gmail

Jul 3, 2009, 12:16 PM

Post #110 of 116 (1010 views)
Permalink
Re: On templates and programming languages [In reply to]

On Fri, Jul 3, 2009 at 1:37 PM, Petr Kadlec<petr.kadlec [at] gmail> wrote:
> And they might depend on the server load at the time

Probably not too much, if you count user+system instead of real time.
But yes, that could be an issue too (more context switches, etc.).

_______________________________________________
Wikitech-l mailing list
Wikitech-l [at] lists
https://lists.wikimedia.org/mailman/listinfo/wikitech-l


questpc at rambler

Jul 3, 2009, 11:37 PM

Post #111 of 116 (992 views)
Permalink
Re: On templates and programming languages [In reply to]

* Aryeh Gregor <Simetrical+wikilist [at] gmail> [Fri, 3 Jul 2009 15:15:48
-0400]:
> Hmm. Smarty looks interesting, at a quick glance. I suspect it's not
> designed to be secure against DoS, so it would need some kind of
> sandboxing. Hopefully less than some of the other solutions we're
> contemplating, though! I'd think it might serve okay, if we wrote
> enough custom functions to replace the existing ParserFunctions. I'm
> not sure.
>
http://en.wikipedia.org/wiki/Template_engine_%28web%29#Comparison

There are some template engines implemented in multiple languages.

This one (from the list above) combines DOM-like manipulations via PHP
and Javascript:
http://code.google.com/p/querytemplates/
Something more closer to XSLT, yet simplier and has loops.
Dmitriy

_______________________________________________
Wikitech-l mailing list
Wikitech-l [at] lists
https://lists.wikimedia.org/mailman/listinfo/wikitech-l


ssanbeg at ask

Jul 6, 2009, 9:46 AM

Post #112 of 116 (954 views)
Permalink
Re: On templates and programming languages [In reply to]

On Fri, 03 Jul 2009 17:13:45 +1000, Tim Starling wrote:

> Steve Sanbeg wrote:
>> I'd assume we want locked down. Loops would be hard in any locked-down
>> environment; I don't recall seeing any recommendation in this thread on
>> how that wold be done. Recursion is much simpler, just track the depth,
>> and throw an exception if it goes to deep; emacs lisp already uses this
>> mechanism.
>
> Loops are essential for readable code. There is no problem with
> allowing loops in conjunction with time limits, that we don't have
> already with complex templates. In fact, time limits for complex
> templates would be an improvement over the system of expansion limits
> we have at the moment.
>

In some cases they would be helpful; i.e. to loop over all of the template
arguments instead of those horrible #switch things they use now. But
letting people run out the clock with arbitrarily complex loops seems
messy.

On the one hand, anyone can easily write code to take up the
maximum alloted time, and stuff as many into a page as they could, to
either prevent the page from rendering at all, or cause the system to stop
executing code before it gets to the parts that are supposed to be there.

On the other hand, it could make templates fail unpredictably; with a
seemingly small change having just enough affect on execution time for the
template to fail, at least some of the time.


> Recursion can give a long running time even if the depth is limited.
> By calling the function multiple times from its own body, you can have
> exponential time order in the recursion depth.
>

All those calls still end up on the same stack; even if it could be a tree
in theory, the stack only grows one way, and execution time would only be
linear.

I found some documentation on the example I'd thought of emulating, which
may clarify a little:

http://www.delorie.com/gnu/docs/elisp-manual-21/elisp_123.html

This variable defines the maximum depth allowed in calls to eval, apply,
and funcall before an error is signaled (with error message "Lisp nesting
exceeds max-lisp-eval-depth"). This limit, with the associated error when
it is exceeded, is one way that Lisp avoids infinite recursion on an
ill-defined function.

The depth limit counts internal uses of eval, apply, and funcall, such as
for calling the functions mentioned in Lisp expressions, and recursive
evaluation of function call arguments and function body forms, as well as
explicit calls in Lisp code.



_______________________________________________
Wikitech-l mailing list
Wikitech-l [at] lists
https://lists.wikimedia.org/mailman/listinfo/wikitech-l


tstarling at wikimedia

Jul 8, 2009, 4:44 AM

Post #113 of 116 (922 views)
Permalink
Re: On templates and programming languages [In reply to]

Aryeh Gregor wrote:
> On Fri, Jul 3, 2009 at 3:13 AM, Tim Starling<tstarling [at] wikimedia> wrote:
>> Loops are essential for readable code. There is no problem with
>> allowing loops in conjunction with time limits, that we don't have
>> already with complex templates. In fact, time limits for complex
>> templates would be an improvement over the system of expansion limits
>> we have at the moment.
>
> But time limits are inconsistent. Whether a template hits the limit
> might depend on whether it happens to be running on an Apache with a
> Pentium IV, an Opteron, a Xeon, . . .

That's the reason I went with expansion limits when I wrote the code.
But I think it was the wrong choice, because the code is complex and
there are lots of ways to run over the 30s time limit set in php.ini,
or to exceed the memory limit, even with the expansion limits in
place. It's hard to find all the potential performance problems during
code review, especially when new parser functions are constantly added.

I didn't say either method was perfect, just that time limits are better.

>> Recursion can give a long running time even if the depth is limited.
>> By calling the function multiple times from its own body, you can have
>> exponential time order in the recursion depth.
>
> You can also have exponential time with loops.

Without the time limit, the worst case running time for a JavaScript
script is infinity with finite input, so the time order is O(∞). With
the time limit, it's O(1). That's the whole point, a time limit lets
you ignore algorithmic complexities.

If you measure script execution times, instead of trying to guess them
in advance, then you can concentrate developer effort on quotas,
access control, profiling tools, etc., which I think are more
tractable problems than analysing the performance every possible thing
the parser can do and limiting it in advance.

-- Tim Starling


_______________________________________________
Wikitech-l mailing list
Wikitech-l [at] lists
https://lists.wikimedia.org/mailman/listinfo/wikitech-l


oscar.vives at gmail

Jul 8, 2009, 5:35 AM

Post #114 of 116 (914 views)
Permalink
Re: On templates and programming languages [In reply to]

Another idea thrown against a wall:

can template scripts be "pre-calculated" ?
I think most people talk about script interpreted "on-demand". But what If
scripts are updated every 100 uses, and in a async way. That way a script
that take 10 min to finish is not a problem. It will be updated every 10
min (or every 1 hour, if the servers want that).

I mean, have "outdated" templates, that only updated wen the server can, and
not everytime the server use that template.


note: I have no idea if this message is more noise to make the raise/signal
worst. Sorry if is that.



--
--
ℱin del ℳensaje.
_______________________________________________
Wikitech-l mailing list
Wikitech-l [at] lists
https://lists.wikimedia.org/mailman/listinfo/wikitech-l


tstarling at wikimedia

Jul 8, 2009, 8:37 PM

Post #115 of 116 (908 views)
Permalink
Re: On templates and programming languages [In reply to]

Steve Sanbeg wrote:
> On Fri, 03 Jul 2009 17:13:45 +1000, Tim Starling wrote:
>> Recursion can give a long running time even if the depth is limited.
>> By calling the function multiple times from its own body, you can have
>> exponential time order in the recursion depth.
>>
>
> All those calls still end up on the same stack; even if it could be a tree
> in theory, the stack only grows one way, and execution time would only be
> linear.

That's an interesting theory.

> I found some documentation on the example I'd thought of emulating, which
> may clarify a little:
>
> http://www.delorie.com/gnu/docs/elisp-manual-21/elisp_123.html

I thought I would try it.

(defun pow5 (n)
(if (= n 0)
1
(+
(pow5 (1- n))
(pow5 (1- n))
(pow5 (1- n))
(pow5 (1- n))
(pow5 (1- n))
)
)
)

It calculates 5 to the power of n by adding 1+1+1+1+1... I found that
with a stack depth limit of 25, I was able to calculate 5^6 = 15625.
That's plainly not an O(N) execution time in stack depth.

-- Tim Starling


_______________________________________________
Wikitech-l mailing list
Wikitech-l [at] lists
https://lists.wikimedia.org/mailman/listinfo/wikitech-l


Platonides at gmail

Jul 9, 2009, 12:36 AM

Post #116 of 116 (912 views)
Permalink
Re: On templates and programming languages [In reply to]

Tei wrote:
> Another idea thrown against a wall:
>
> can template scripts be "pre-calculated" ?
> I think most people talk about script interpreted "on-demand". But what If
> scripts are updated every 100 uses, and in a async way. That way a script
> that take 10 min to finish is not a problem. It will be updated every 10
> min (or every 1 hour, if the servers want that).
>
> I mean, have "outdated" templates, that only updated wen the server can, and
> not everytime the server use that template.
>
>
> note: I have no idea if this message is more noise to make the raise/signal
> worst. Sorry if is that.

That's pretty much what mediawiki does.
The template is only parsed when article is modified. With the template
calculating the age of the subject you need to calculate it every time
(or cache the output for each template with each set of parameters).
When you modify the template, if i has many uses, the pages using it
will used the cached version of the page until the job queue reaches to
update it.


_______________________________________________
Wikitech-l mailing list
Wikitech-l [at] lists
https://lists.wikimedia.org/mailman/listinfo/wikitech-l

First page Previous page 1 2 3 4 5 Next page Last page  View All Wikipedia wikitech RSS feed   Index | Next | Previous | View Threaded
 
 


Interested in having your list archived? Contact Gossamer Threads
 
  Web Applications & Managed Hosting Powered by Gossamer Threads Inc.