Mailing List Archive: Perl: porters

[PATCH] Functional substitution (s///f)

1 2 3

View All

Index | Next | Previous | View Threaded

david at porkrind

Nov 22, 2009, 2:37 PM

Post #1 of 51 (863 views)
Permalink

[PATCH] Functional substitution (s///f)

Hi,

I wrote a patch to scratch an itch I've had over the years. The patch
adds a new flag to s/// ("f") to make it functional instead of destructive.
So you can do this:

$a = s/aaa/bbb/f;

Which makes s/// return the replacement instead of changing $_ (or whatever
=~ bound it to).

In the past I've tried to encapsulate this in some sort of library but
the different flags on the s/// make the interface really ugly. This patch
keeps the beauty, terseness, and functionality of the the original s///
syntax and because it's a new flag it shouldn't affect old code.

There is no documentation changed in the patch because I wanted to see if
there was any interest first.

So, I'm looking for comments and hopefully an indication of what it takes
to get this into perl. Be gentle, it's my first foray into the perl
internals and so I've probably done something brain-dead :-).

-David

Attachments:

functional-subst.patch (5.26 KB)

zefram at fysh

Nov 23, 2009, 8:31 AM

Post #2 of 51 (822 views)
Permalink

Re: [PATCH] Functional substitution (s///f) [In reply to]

David Caldwell wrote:
>adds a new flag to s/// ("f") to make it functional instead of
>destructive.

+1 for the concept.

> So, I'm looking for comments and hopefully an indication of what it
>takes to get this into perl.

Unfortunately you just missed the feature cutoff for 5.12, so this is a
bad time for a feature patch. But basically the process is what you've
just done: mailing a patch to perl5-porters. The patch would have to
include documentation to be fully acceptable.

-zefram

jesse at fsck

Nov 23, 2009, 8:38 AM

Post #3 of 51 (820 views)
Permalink

Re: [PATCH] Functional substitution (s///f) [In reply to]

On Mon, Nov 23, 2009 at 04:31:23PM +0000, Zefram wrote:
> David Caldwell wrote:
> >adds a new flag to s/// ("f") to make it functional instead of
> >destructive.
>
> +1 for the concept.
>
> > So, I'm looking for comments and hopefully an indication of what it
> >takes to get this into perl.
>
> Unfortunately you just missed the feature cutoff for 5.12, so this is a
> bad time for a feature patch. But basically the process is what you've
> just done: mailing a patch to perl5-porters. The patch would have to
> include documentation to be fully acceptable.

It would also probably want tests that include some more advanced/insane
regex features and tests that "prove" what it should or shouldn't be
doing with captures.

>
> -zefram
>

--

ikegami at adaelis

Nov 23, 2009, 9:13 AM

Post #4 of 51 (822 views)
Permalink

Re: [PATCH] Functional substitution (s///f) [In reply to]

On Sun, Nov 22, 2009 at 5:37 PM, David Caldwell <david [at] porkrind> wrote:

> [I wrote a patch to add] a new flag to s/// ("f") to make it functional
> instead of destructive.

> In the past I've tried to encapsulate this in some sort of library but the
> different flags on the s/// make the interface really ugly.
>

use List::MoreUtils qw( apply );
my $bar = apply { s/// } $foo;

use Algorithm::Loops qw( Filter );
my $bar = Filter { s/// } $foo;

Both work on lists as well (like map).

gisle at activestate

Nov 23, 2009, 11:27 AM

Post #5 of 51 (820 views)
Permalink

Re: [PATCH] Functional substitution (s///f) [In reply to]

On Nov 22, 2009, at 23:37 , David Caldwell wrote:

> I wrote a patch to scratch an itch I've had over the years. The patch adds a new flag to s/// ("f") to make it functional instead of destructive. So you can do this:
>
> $a = s/aaa/bbb/f;

So instead of

($a = $b) =~ s/aaa/bbb/;

We can now do:

$a = $b =~ s/aaa/bbb/f;

Not obviously a big win for me; but there are probably other use cases where this is more useful. For instance:

print HTML "<pre>" . $text =~ s/&/&/fg =~ s/</</fg . "</pre>";

This is kind of cool; but it's not really obvious to me which substitution would happen first here. It's quite unreadable as well :-(

Another one is:

@foo = map s/foo/bar/f, @bar;

which I often encounter in some form and I always found harder than it ought to be in perl.

--Gisle

gisle at activestate

Nov 23, 2009, 11:40 AM

Post #6 of 51 (820 views)
Permalink

Re: [PATCH] Functional substitution (s///f) [In reply to]

Another thought; if we do this for s/// then we should do it for tr/// as well.

--Gisle

david at porkrind

Nov 23, 2009, 11:55 AM

Post #7 of 51 (814 views)
Permalink

Re: [PATCH] Functional substitution (s///f) [In reply to]

On 11/23/09 8:27 PM +0100 Gisle Aas wrote:

> On Nov 22, 2009, at 23:37 , David Caldwell wrote:
>
>> I wrote a patch to scratch an itch I've had over the years. The patch
>> adds a new flag to s/// ("f") to make it functional instead of
>> destructive. So you can do this:
>
> $a = $b =~ s/aaa/bbb/f;
>
> Not obviously a big win for me; but there are probably other use cases
> where this is more useful.

I typically find myself wanting it in function calls, or big map
expressions where creating a new variable becomes unsightly.

some_function($a, $b,"string", $c =~ s/x/y/f);

> For instance:
>
> print HTML "<pre>" . $text =~ s/&/&/fg =~ s/</</fg . "</pre>";
>
> This is kind of cool; but it's not really obvious to me which
> substitution would happen first here. It's quite unreadable as well :-(

That is genious, I hadn't thought of chaining them. I don't know--I think
it's not bad, readability-wise, especially if you consider =~ to be like a
shell pipe. Lining it up vertically and aligning the =~ might make it look
nicer too, especially if there were more chained together.

I added that as a test because I wasn't sure how the bind operator would
bind in that case, but it did "the right thing":

$a = 'david';
$b = $a =~ s/david/sucks/f =~ s/sucks/rules/f;
ok( $a eq 'david' && $b eq 'rules' );

-David

davidnicol at gmail

Nov 23, 2009, 11:55 AM

Post #8 of 51 (820 views)
Permalink

Re: [PATCH] Functional substitution (s///f) [In reply to]

On Mon, Nov 23, 2009 at 1:40 PM, Gisle Aas <gisle [at] activestate> wrote:
> Another thought; if we do this for s/// then we should do it for tr/// as well.
>
> --Gisle

Eeek, please no. The precedence of the binding operator is tricky
enough as is without making it subject to change based on one of those
embarassingly placed postfix flags.

If you want a function suitable for use in C<map> that invokes s///,
please just write one

sub substf($$$;$){
my ($val, $pat, $repl, $flags) = @_;
(eval "sub {\$[0]=~s/\$_[1]/\$_[2]/$flags; shift}" ) -> ($val,
$pat, $repl)
$val;
}

or a set of them each with different flags

sub substf_with_flags::AUTOLOAD{

... left as an exercise ...

}

after which, you could write

@foo = map subst_with_flags::gi('foo','bar'), @bar;

--
"In the case of an infinite collection, the question of the existence
of a choice function is problematic"

david at porkrind

Nov 23, 2009, 12:08 PM

Post #9 of 51 (814 views)
Permalink

Re: [PATCH] Functional substitution (s///f) [In reply to]

On 11/23/09 1:55 PM -0600 David Nicol wrote:

> On Mon, Nov 23, 2009 at 1:40 PM, Gisle Aas <gisle [at] activestate> wrote:
>> Another thought; if we do this for s/// then we should do it for tr///
>> as well.
>>
>> --Gisle
>
> Eeek, please no. The precedence of the binding operator is tricky
> enough as is without making it subject to change based on one of those
> embarassingly placed postfix flags.

The patch doesn't change the precedence of the binding operator... It
already has "the correct" precedence.

-David

david at porkrind

Nov 23, 2009, 12:16 PM

Post #10 of 51 (814 views)
Permalink

Re: [PATCH] Functional substitution (s///f) [In reply to]

On 11/23/09 11:38 AM -0500 jesse wrote:
> On Mon, Nov 23, 2009 at 04:31:23PM +0000, Zefram wrote:
>> David Caldwell wrote:
>> > So, I'm looking for comments and hopefully an indication of what it
>> > takes to get this into perl.
>>
>> Unfortunately you just missed the feature cutoff for 5.12, so this is a
>> bad time for a feature patch. But basically the process is what you've
>> just done: mailing a patch to perl5-porters. The patch would have to
>> include documentation to be fully acceptable.
>
> It would also probably want tests that include some more advanced/insane
> regex features and tests that "prove" what it should or shouldn't be
> doing with captures.

I didn't include any tests for captures because the patch didn't touch
anything that would affect captures. It basically copies the target scalar
before anything is done and operates on the copy instead of the original.
Then it changes the return value to return the target scalar instead of the
number of matches. All the middle subst regexp stuff is untouched.

I'm completely open to writing more/better tests. I don't see how testing
the captures adds anything, though. I'm happy to be enilightened,
preferably with one of the lighter weight clue sticks.

-David

abigail at abigail

Nov 23, 2009, 1:14 PM

Post #11 of 51 (819 views)
Permalink

Re: [PATCH] Functional substitution (s///f) [In reply to]

On Mon, Nov 23, 2009 at 08:40:43PM +0100, Gisle Aas wrote:
> Another thought; if we do this for s/// then we should do it for tr/// as well.

And m//f could return $&. (Got to have some meaning for m//f).

Abigail

ikegami at adaelis

Nov 23, 2009, 1:17 PM

Post #12 of 51 (818 views)
Permalink

Re: [PATCH] Functional substitution (s///f) [In reply to]

On Mon, Nov 23, 2009 at 2:27 PM, Gisle Aas <gisle [at] activestate> wrote:

> there are probably other use cases where this is more useful. For
> instance:
>
> print HTML "<pre>" . $text =~ s/&/&/fg =~ s/</</fg . "</pre>";
>
> This is kind of cool; but it's not really obvious to me which substitution
> would happen first here. It's quite unreadable as well :-(
>

=~ is left associative like most ops*, so left to right. And you know how to
make it clear: Add parens.

* -- The assignment ops and exponentiation are right-associative.

gisle at activestate

Nov 23, 2009, 1:32 PM

Post #13 of 51 (819 views)
Permalink

Re: [PATCH] Functional substitution (s///f) [In reply to]

On Nov 23, 2009, at 22:17 , Eric Brine wrote:

> On Mon, Nov 23, 2009 at 2:27 PM, Gisle Aas <gisle [at] activestate> wrote:
> there are probably other use cases where this is more useful. For instance:
>
> print HTML "<pre>" . $text =~ s/&/&/fg =~ s/</</fg . "</pre>";
>
> This is kind of cool; but it's not really obvious to me which substitution would happen first here. It's quite unreadable as well :-(
>
> =~ is left associative like most ops*, so left to right. And you know how to make it clear: Add parens.
>
> * -- The assignment ops and exponentiation are right-associative.

Yeah. It's more obvious when I think about it a bit more. Since that has really never been a reason to chain =~ before I didn't have to really care about the associativeness of '=~' while it's very clear to be how '=' works; and they do look similar.

How about:

$a = $b !~ s/aaa/bbb/f;

Would that construct ever be useful?

--Gisle

gisle at activestate

Nov 23, 2009, 1:45 PM

Post #14 of 51 (819 views)
Permalink

Re: [PATCH] Functional substitution (s///f) [In reply to]

On Nov 23, 2009, at 20:55 , David Nicol wrote:

> On Mon, Nov 23, 2009 at 1:40 PM, Gisle Aas <gisle [at] activestate> wrote:
>> Another thought; if we do this for s/// then we should do it for tr/// as well.
>>
>> --Gisle
>
> Eeek, please no. The precedence of the binding operator is tricky
> enough as is without making it subject to change based on one of those
> embarassingly placed postfix flags.

I do agree that there is a mismatch between the significance of /f flag and how easy it is to spot a single letter flag at the end of the expression.

> If you want a function suitable for use in C<map> that invokes s///,
> please just write one

The point is that I do _not_ want to write a function :-)

--Gisle

davidnicol at gmail

Nov 23, 2009, 1:53 PM

Post #15 of 51 (817 views)
Permalink

Re: [PATCH] Functional substitution (s///f) [In reply to]

On Mon, Nov 23, 2009 at 3:45 PM, Gisle Aas <gisle [at] activestate> wrote:
>> If you want a function suitable for use in C<map> that invokes s///,
>> please just write one
>
> The point is that I do _not_ want to write a function :-)
>
> --Gisle

then upload it to CPAN.

although doing it right might imply some prototyping syntax for
binding-op quote preferences... okay I'll go away, please continue
adding bizarre new stuff.

so the /f modifier means the return value is no longer a boolean of if
it matched or not, but the thing that would have normally been
assigned to the bound l-value, which is unchanged, so you can use
subst on read-only strings without copying them to temps first.
Right?

ikegami at adaelis

Nov 23, 2009, 1:55 PM

Post #16 of 51 (819 views)
Permalink

Re: [PATCH] Functional substitution (s///f) [In reply to]

On Mon, Nov 23, 2009 at 4:32 PM, Gisle Aas <gisle [at] activestate> wrote:

> How about:
>
> $a = $b !~ s/aaa/bbb/f;
>
> Would that construct ever be useful?
>

[. Gisle: My previous message sent itself while I was typing it up. Ignore it
]

That gets compiled as

$a = !( $b =~ s/aaa/bbb/f );

You could do

$num = <>;
if ( $num !~ s/\n//f ) { ... }

as an alternative to

chomp( $num = <> );
if ( !$num ) { ... }

but I wouldn't recommend it.

david at kineticode

Nov 23, 2009, 2:36 PM

Post #17 of 51 (815 views)
Permalink

Re: [PATCH] Functional substitution (s///f) [In reply to]

On Nov 24, 2009, at 6:14 AM, Abigail wrote:

> And m//f could return $&. (Got to have some meaning for m//f).

I hereby dub this the �MoFo� operator.

;-p

Best,

David

david at porkrind

Nov 24, 2009, 2:09 AM

Post #18 of 51 (812 views)
Permalink

Re: [PATCH] Functional substitution (s///f) [In reply to]

On 11/22/09 2:37 PM -0800 David Caldwell wrote:

> Hi,
>
> I wrote a patch to scratch an itch I've had over the years. The patch
> adds a new flag to s/// ("f") to make it functional instead of
> destructive. So you can do this:
>
> $a = s/aaa/bbb/f;

I've attached an updated patch that:

* Adds a new warning when s///f happens in void context.
* Adds documentation.
* Updates various debug code so it knows about the /f flag.
and, most importantly:
* Adds some new 'f' words to B::Deparse.

Or you can find it here:

<http://github.com/caldwell/perl/commit/0279277f34e6ed1ec591c23df7a7fc2285179fa3>

-David

ps. I looked at tr/// to see if I could do something similar but it's
harder because there are no bits left in op_private.

Attachments:

functional-subst-2.patch (10.9 KB)

davidnicol at gmail

Nov 24, 2009, 9:57 AM

Post #19 of 51 (808 views)
Permalink

Re: [PATCH] Functional substitution (s///f) [In reply to]

>> print HTML "<pre>" . $text =~ s/&/&/fg =~ s/</</fg . "</pre>";

for that, you want HTML::Entities::Interpolate from CPAN, which lets
you spell that

print HTML "<pre>$Entitize{$text}</pre>";

> That is genious, I hadn't thought of chaining them.

so the /f flag turns a substitution expression $target =~ s/pat/repl/xyz
into

do { (my $tmp = $target) =~ s/pat/repl/xyz; $tmp }

do blocks are unmodifiable even when they return lvalues (at least in 5.8.8)

should do blocks be usable as lvalues?

pagaltzis at gmx

Nov 24, 2009, 12:35 PM

Post #20 of 51 (809 views)
Permalink

Re: [PATCH] Functional substitution (s///f) [In reply to]

Hi David,

* David Caldwell <david [at] porkrind> [2009-11-23 17:20]:
> I wrote a patch to scratch an itch I've had over the years.
> The patch adds a new flag to s/// ("f") to make it functional
> instead of destructive.

may I make just one stroke of paint on that shed, please? I’d
like the flag to be something else, and more prominent and
different. I am thinking either /C (for “copy”) or maybe /R (for
“return”).

> So you can do this:
>
> $a = s/aaa/bbb/f;
>
> Which makes s/// return the replacement instead of changing $_
> (or whatever =~ bound it to).

Thanks and ++ for this. It’s something I’ve wanted for as along
as I’ve been writing Perl.

(To the others who’ve replied with “just write a function” or
“here, use one of these modules”: sure, but the resulting code
is just ugly. Not very ugly, but enough to be off-putting.)

Regards,
--
Aristotle Pagaltzis // <http://plasmasturm.org/>

hv at crypt

Nov 24, 2009, 2:37 PM

Post #21 of 51 (806 views)
Permalink

Re: [PATCH] Functional substitution (s///f) [In reply to]

David Caldwell <david [at] porkrind> wrote:
: I wrote a patch to scratch an itch I've had over the years. The patch
:adds a new flag to s/// ("f") to make it functional instead of destructive.
:
:So you can do this:
:
: $a =3D s/aaa/bbb/f;
:
:Which makes s/// return the replacement instead of changing $_ (or whatever
:=~ bound it to).

I really like this, I'd love to see it in perl.

I'll try to make time to look through the patch itself in the next couple
of days.

Hugo

jim.cromie at gmail

Nov 25, 2009, 8:28 AM

Post #22 of 51 (796 views)
Permalink

Re: [PATCH] Functional substitution (s///f) [In reply to]

On Tue, Nov 24, 2009 at 3:37 PM, <hv [at] crypt> wrote:
> David Caldwell <david [at] porkrind> wrote:
> : �I wrote a patch to scratch an itch I've had over the years. The patch
> :adds a new flag to s/// ("f") to make it functional instead of destructive.
> :
> :So you can do this:
> :
> : � $a =3D s/aaa/bbb/f;
> :
> :Which makes s/// return the replacement instead of changing $_ (or whatever
> :=~ bound it to).
>
> I really like this, I'd love to see it in perl.
>
> I'll try to make time to look through the patch itself in the next couple
> of days.
>
> Hugo
>

perhaps s///f should be spelled
m///

since the match doesnt change the string,
it makes more mnemonic sense than substitute /// fake

1 while ( $buf =~ m/ ... / process_matches($1,$2,$3) /eg ) ;

I wonder whether the tokenizer / parser could be readily
adapted to this ?

abigail at abigail

Nov 25, 2009, 12:49 PM

Post #23 of 51 (798 views)
Permalink

Re: [PATCH] Functional substitution (s///f) [In reply to]

On Wed, Nov 25, 2009 at 09:28:45AM -0700, Jim Cromie wrote:
> On Tue, Nov 24, 2009 at 3:37 PM, <hv [at] crypt> wrote:
> > David Caldwell <david [at] porkrind> wrote:
> > : �I wrote a patch to scratch an itch I've had over the years. The patch
> > :adds a new flag to s/// ("f") to make it functional instead of destructive.
> > :
> > :So you can do this:
> > :
> > : � $a =3D s/aaa/bbb/f;
> > :
> > :Which makes s/// return the replacement instead of changing $_ (or whatever
> > :=~ bound it to).
> >
> > I really like this, I'd love to see it in perl.
> >
> > I'll try to make time to look through the patch itself in the next couple
> > of days.
> >
> > Hugo
> >
>
>
> perhaps s///f should be spelled
> m///
>
> since the match doesnt change the string,
> it makes more mnemonic sense than substitute /// fake
>
>
> 1 while ( $buf =~ m/ ... / process_matches($1,$2,$3) /eg ) ;
>
>
> I wonder whether the tokenizer / parser could be readily
> adapted to this ?

Consider:

($foo) = m/pattern/m; # Note the /m;

Is that modifying $_, returning the modified $_ in $foo, or just matching
against $_, setting $foo to $1?

Abigail

nick at ccl4

Nov 26, 2009, 3:41 AM

Post #24 of 51 (786 views)
Permalink

Re: [PATCH] Functional substitution (s///f) [In reply to]

On Mon, Nov 23, 2009 at 12:16:59PM -0800, David Caldwell wrote:
> On 11/23/09 11:38 AM -0500 jesse wrote:
> >On Mon, Nov 23, 2009 at 04:31:23PM +0000, Zefram wrote:
> >>David Caldwell wrote:
> >>> So, I'm looking for comments and hopefully an indication of what it
> >>> takes to get this into perl.
> >>
> >>Unfortunately you just missed the feature cutoff for 5.12, so this is a
> >>bad time for a feature patch. But basically the process is what you've
> >>just done: mailing a patch to perl5-porters. The patch would have to
> >>include documentation to be fully acceptable.
> >
> >It would also probably want tests that include some more advanced/insane
> >regex features and tests that "prove" what it should or shouldn't be
> >doing with captures.
>
> I didn't include any tests for captures because the patch didn't touch
> anything that would affect captures. It basically copies the target scalar
> before anything is done and operates on the copy instead of the original.
> Then it changes the return value to return the target scalar instead of the
> number of matches. All the middle subst regexp stuff is untouched.

Right. But what I think is important is to make sure that there isn't any
part of the behaviour that *relies* on this implementation. Because it would
be useful in future to have the possibility to optimise the implementation.

Right now there is an explicit, up-front copy, which then gets shuffled around
in-place every time a length-changing substitution is made. It would be nice
to be able to change to having the matching part of the regexp engine walk
the original (read-only), only copying chunks across that are unchanged, and
writing substitutions out directly. But we'd only have the freedom to do this
if we have all the corner cases covered now, to make sure that there isn't
a way to "see through" this and spot the difference, at the point when /f
is first introduced.

However, I can't (yet) find a way to "spot" this, as it seems that

1: the variable that one is matching isn't "changed" until the end of the
match, so that actions such as C<length> on it don't alter midway:

$ ./perl -lw
$_ = 'abcbcdcde';
s/c/2 ** length $_/ge;
print $_;
__END__
ab512b512d512de

2: variables such as C<$'>, which *would* differ depending on the
implementation above, can still be given the existing behaviour, by making
them (continue to) track the remainder of the original.

$ ./perl -lw
$_ = 'abcbcdcde';
s/c/2 ** length $'/ge;
print $_;
__END__
ab64b16d4de

Nicholas Clark

demerphq at gmail

May 20, 2010, 4:55 AM

Post #25 of 51 (512 views)
Permalink

Re: [PATCH] Functional substitution (s///f) [In reply to]

On 20 May 2010 13:49, �var Arnfj�r� Bjarmason <avarab [at] gmail> wrote:
> On Tue, May 18, 2010 at 17:55, David Caldwell <david [at] porkrind> wrote:
>> Hello,
>>
>> �Back in November of 2009 I submitted this patch but it wasn't considered
>> because Perl 5.11 was feature frozen. Now that 5.12 is out I would like to
>> submit it again.
>
> I like the functionality, but see the "PATCH: [perl #58182] partial,
> "The Unicode Bug". Add unicode semantics for \s, \w" for issues with
> adding new flags that are being currently discussed.

I like it too.

And repeat �var's recommendation, and also would like to offer the
thought, that "make it functional" didn't quite convey the intent as
well as you might have wanted.

While now i realize the intent, I'm wondering if there is a better
phrase. (Sorry for bikeshedding).

yves

--
perl -Mre=debug -e "/just|another|perl|hacker/"

1 2 3

View All

Index | Next | Previous | View Threaded

Interested in having your list archived? Contact Gossamer Threads