Mailing List Archive: Wikipedia: Wikitech

"known to fail" switch added to parserTests

Index | Next | Previous | View Threaded

dnessett at yahoo

Jul 20, 2009, 8:09 AM

Post #1 of 20 (1343 views)
Permalink

"known to fail" switch added to parserTests

I have modified parserTests to take a "known to fail" switch so those tests that have always failed now pass. It was pretty simple. It only required 3 changes to parserTests.inc and some editing on parserTests.txt. I added an option for each test called flipresult. When this option is specified, the test succeeds when it fails and vice versa.

I have tested the modified parserTest on 1.16a running over a 1.14 schema database. However, I have run into a problem attempting to install the latest trunk revision so I can test against it. Specifically, I added a database called wikitestdb so I would have a production and test wiki. However, when I checked out the latest trunk revision, ran the install script and update.php, and then accessed http://<wiki path>/index.php the installation gets into a infinite redirect loop. When I attempted to debug this (using netbeans 6.7 and Xdebug) the redirect doesn't appear. That is, Main_Page is rendered and displayed. The only difference between the two URLs are the first uses http://<wiki path>/index.php (which redirects to http://<wiki path>/index.php/Main_Page), while the debug session specifies http://localhost/MediawikiTest/Latest%20Trunk%20Version/phase3/index.php?XDEBUG_SESSION_START=netbeans-xdebug.

I need some help figuring out what is happening. I imagine using this list for that purpose would be inappropriate. So, if someone would volunteer to help me (email me at the from address in this email), I can get the parserTest changes tested against the newest revision. I can then open a bug (or use an already open bug) and attach the patch and edited parserTests.txt file to it.

_______________________________________________
Wikitech-l mailing list
Wikitech-l [at] lists
https://lists.wikimedia.org/mailman/listinfo/wikitech-l

dnessett at yahoo

Jul 20, 2009, 11:01 AM

Post #2 of 20 (1292 views)
Permalink

Re: "known to fail" switch added to parserTests [In reply to]

Correction: When I tried to create a patch for parserTests.inc and parserTest.txt I discovered the directory I was working from was not under SVN control. So, I created a new working copy of 52088 and when I ran the tests, they reported a databased selection error. So, I cannot claim the tests work for 1.16a under a 1.14 schema. I am now checking out REL1_14 and will run the modified parserTest under it.

--- On Mon, 7/20/09, dan nessett <dnessett [at] yahoo> wrote:

> From: dan nessett <dnessett [at] yahoo>
> Subject: [Wikitech-l] "known to fail" switch added to parserTests
> To: wikitech-l [at] lists
> Date: Monday, July 20, 2009, 8:09 AM
>
> I have modified parserTests to take a "known to fail"
> switch so those tests that have always failed now pass. It
> was pretty simple. It only required 3 changes to
> parserTests.inc and some editing on parserTests.txt. I added
> an option for each test called flipresult. When this option
> is specified, the test succeeds when it fails and vice
> versa.
>
> I have tested the modified parserTest on 1.16a running over
> a 1.14 schema database. However, I have run into a problem
> attempting to install the latest trunk revision so I can
> test against it. Specifically, I added a database called
> wikitestdb so I would have a production and test wiki.
> However, when I checked out the latest trunk revision, ran
> the install script and update.php, and then accessed
> http://<wiki path>/index.php the installation gets
> into a infinite redirect loop. When I attempted to debug
> this (using netbeans 6.7 and Xdebug) the redirect doesn't
> appear. That is, Main_Page is rendered and displayed. The
> only difference between the two URLs are the first uses
> http://<wiki path>/index.php (which redirects to
> http://<wiki path>/index.php/Main_Page), while the
> debug session specifies http://localhost/MediawikiTest/Latest%20Trunk%20Version/phase3/index.php?XDEBUG_SESSION_START=netbeans-xdebug.
>
> I need some help figuring out what is happening. I imagine
> using this list for that purpose would be inappropriate. So,
> if someone would volunteer to help me (email me at the from
> address in this email), I can get the parserTest changes
> tested against the newest revision. I can then open a bug
> (or use an already open bug) and attach the patch and edited
> parserTests.txt file to it.
>
>
>
>
> _______________________________________________
> Wikitech-l mailing list
> Wikitech-l [at] lists
> https://lists.wikimedia.org/mailman/listinfo/wikitech-l
>

_______________________________________________
Wikitech-l mailing list
Wikitech-l [at] lists
https://lists.wikimedia.org/mailman/listinfo/wikitech-l

dnessett at yahoo

Jul 20, 2009, 2:03 PM

Post #3 of 20 (1304 views)
Permalink

Re: "known to fail" switch added to parserTests [In reply to]

OK. I think I am now in sync. I checked out 1.14.1 (from the branches arm of r53568) and merged the local changes to parserTests.inc and parserTests.txt. Out of the box, 1.14.1 generates 40 parser test failures. With the code modifications, it generates 24 (i.e., 14 fewer). I have the patches for 1.14.1 and can attach them to an appropriate bug or create a new bug and attach them there.

I also have the patches to r53551, which I can't test because, as I stated previously, my testwiki gets into a redirect loop. No one has stepped forward to help, so I can either provide these for someone else to test or wander around trying to figure out why the redirect loop occurs on my own. One further bit of info on this - when I get to the main page using the debugger, any local link I attempt to navigate to leads me back to the main page, which dies because of the redirect loop. This suggests something is stuck in the redirect logic and everything (including the main page) is redirecting back to the main page.

If someone could let me know how to proceed with the patches, that would help.

--- On Mon, 7/20/09, dan nessett <dnessett [at] yahoo> wrote:

> From: dan nessett <dnessett [at] yahoo>
> Subject: Re: [Wikitech-l] "known to fail" switch added to parserTests
> To: "Wikimedia developers" <wikitech-l [at] lists>
> Date: Monday, July 20, 2009, 11:01 AM
>
> Correction: When I tried to create a patch for
> parserTests.inc and parserTest.txt I discovered the
> directory I was working from was not under SVN control. So,
> I created a new working copy of 52088 and when I ran the
> tests, they reported a databased selection error. So, I
> cannot claim the tests work for 1.16a under a 1.14 schema. I
> am now checking out REL1_14 and will run the modified
> parserTest under it.
>
> --- On Mon, 7/20/09, dan nessett <dnessett [at] yahoo>
> wrote:
>
> > From: dan nessett <dnessett [at] yahoo>
> > Subject: [Wikitech-l] "known to fail" switch added to
> parserTests
> > To: wikitech-l [at] lists
> > Date: Monday, July 20, 2009, 8:09 AM
> >
> > I have modified parserTests to take a "known to fail"
> > switch so those tests that have always failed now
> pass. It
> > was pretty simple. It only required 3 changes to
> > parserTests.inc and some editing on parserTests.txt. I
> added
> > an option for each test called flipresult. When this
> option
> > is specified, the test succeeds when it fails and
> vice
> > versa.
> >
> > I have tested the modified parserTest on 1.16a running
> over
> > a 1.14 schema database. However, I have run into a
> problem
> > attempting to install the latest trunk revision so I
> can
> > test against it. Specifically, I added a database
> called
> > wikitestdb so I would have a production and test
> wiki.
> > However, when I checked out the latest trunk revision,
> ran
> > the install script and update.php, and then accessed
> > http://<wiki path>/index.php the installation
> gets
> > into a infinite redirect loop. When I attempted to
> debug
> > this (using netbeans 6.7 and Xdebug) the redirect
> doesn't
> > appear. That is, Main_Page is rendered and displayed.
> The
> > only difference between the two URLs are the first
> uses
> > http://<wiki path>/index.php (which redirects
> to
> > http://<wiki path>/index.php/Main_Page), while
> the
> > debug session specifies http://localhost/MediawikiTest/Latest%20Trunk%20Version/phase3/index.php?XDEBUG_SESSION_START=netbeans-xdebug.
> >
> > I need some help figuring out what is happening. I
> imagine
> > using this list for that purpose would be
> inappropriate. So,
> > if someone would volunteer to help me (email me at the
> from
> > address in this email), I can get the parserTest
> changes
> > tested against the newest revision. I can then open a
> bug
> > (or use an already open bug) and attach the patch and
> edited
> > parserTests.txt file to it.
> >
> >
> >
> >
> > _______________________________________________
> > Wikitech-l mailing list
> > Wikitech-l [at] lists
> > https://lists.wikimedia.org/mailman/listinfo/wikitech-l
> >
>
>
>
>
> _______________________________________________
> Wikitech-l mailing list
> Wikitech-l [at] lists
> https://lists.wikimedia.org/mailman/listinfo/wikitech-l
>

_______________________________________________
Wikitech-l mailing list
Wikitech-l [at] lists
https://lists.wikimedia.org/mailman/listinfo/wikitech-l

gerard.meijssen at gmail

Jul 21, 2009, 12:19 AM

Post #4 of 20 (1290 views)
Permalink

Re: "known to fail" switch added to parserTests [In reply to]

Hoi,
Suppose that someone fixes a test that has been always failing... one of
those "known to fail". It makes no difference right ?? Giving them the
status of pass is imho dead wrong because they should not fail in the first
place.. now a status of KNOWN TO FAIL makes sense.
Thanks,
GeardM

2009/7/20 dan nessett <dnessett [at] yahoo>

>
> I have modified parserTests to take a "known to fail" switch so those tests
> that have always failed now pass. It was pretty simple. It only required 3
> changes to parserTests.inc and some editing on parserTests.txt. I added an
> option for each test called flipresult. When this option is specified, the
> test succeeds when it fails and vice versa.
>
> I have tested the modified parserTest on 1.16a running over a 1.14 schema
> database. However, I have run into a problem attempting to install the
> latest trunk revision so I can test against it. Specifically, I added a
> database called wikitestdb so I would have a production and test wiki.
> However, when I checked out the latest trunk revision, ran the install
> script and update.php, and then accessed http://<wiki path>/index.php the
> installation gets into a infinite redirect loop. When I attempted to debug
> this (using netbeans 6.7 and Xdebug) the redirect doesn't appear. That is,
> Main_Page is rendered and displayed. The only difference between the two
> URLs are the first uses http://<wiki path>/index.php (which redirects to
> http://<wiki path>/index.php/Main_Page), while the debug session specifies
>
> http://localhost/MediawikiTest/Latest%20Trunk%20Version/phase3/index.php?XDEBUG_SESSION_START=netbeans-xdebug
> .
>
> I need some help figuring out what is happening. I imagine using this list
> for that purpose would be inappropriate. So, if someone would volunteer to
> help me (email me at the from address in this email), I can get the
> parserTest changes tested against the newest revision. I can then open a bug
> (or use an already open bug) and attach the patch and edited parserTests.txt
> file to it.
>
>
>
>
> _______________________________________________
> Wikitech-l mailing list
> Wikitech-l [at] lists
> https://lists.wikimedia.org/mailman/listinfo/wikitech-l
>
_______________________________________________
Wikitech-l mailing list
Wikitech-l [at] lists
https://lists.wikimedia.org/mailman/listinfo/wikitech-l

Simetrical+wikilist at gmail

Jul 21, 2009, 2:44 AM

Post #5 of 20 (1291 views)
Permalink

Re: "known to fail" switch added to parserTests [In reply to]

On Tue, Jul 21, 2009 at 7:19 AM, Gerard
Meijssen<gerard.meijssen [at] gmail> wrote:
> Suppose that someone fixes a test that has been always failing... one of
> those "known to fail". It makes no difference right ??

Difference in what sense? It means we have one less failing test
reported, presumably.

> Giving them the
> status of pass is imho dead wrong because they should not fail in the first
> place.. now a status of KNOWN TO FAIL makes sense.

The known-failing tests have never passed. They're a wishlist. None
of them are likely to be fixed in the foreseeable future. I'd be fine
with just removing them, but Brion has been against it in the past.

_______________________________________________
Wikitech-l mailing list
Wikitech-l [at] lists
https://lists.wikimedia.org/mailman/listinfo/wikitech-l

gerard.meijssen at gmail

Jul 21, 2009, 2:56 AM

Post #6 of 20 (1288 views)
Permalink

Re: "known to fail" switch added to parserTests [In reply to]

Hoi,
There is no point having a perfect score when it is actually a lie. It seems
to me that Brion is against the removal of these tests because he wants them
to pass. Having a third state of "known to fail" makes sense, just changing
them to pass makes it necessary to add a "citation needed" because it is
just not true.
Thanks,
GerardM

2009/7/21 Aryeh Gregor
<Simetrical+wikilist [at] gmail<Simetrical%2Bwikilist [at] gmail>
>

> On Tue, Jul 21, 2009 at 7:19 AM, Gerard
> Meijssen<gerard.meijssen [at] gmail> wrote:
> > Suppose that someone fixes a test that has been always failing... one of
> > those "known to fail". It makes no difference right ??
>
> Difference in what sense? It means we have one less failing test
> reported, presumably.
>
> > Giving them the
> > status of pass is imho dead wrong because they should not fail in the
> first
> > place.. now a status of KNOWN TO FAIL makes sense.
>
> The known-failing tests have never passed. They're a wishlist. None
> of them are likely to be fixed in the foreseeable future. I'd be fine
> with just removing them, but Brion has been against it in the past.
>
> _______________________________________________
> Wikitech-l mailing list
> Wikitech-l [at] lists
> https://lists.wikimedia.org/mailman/listinfo/wikitech-l
>
_______________________________________________
Wikitech-l mailing list
Wikitech-l [at] lists
https://lists.wikimedia.org/mailman/listinfo/wikitech-l

Simetrical+wikilist at gmail

Jul 21, 2009, 4:09 AM

Post #7 of 20 (1307 views)
Permalink

Re: "known to fail" switch added to parserTests [In reply to]

On Tue, Jul 21, 2009 at 9:56 AM, Gerard
Meijssen<gerard.meijssen [at] gmail> wrote:
> There is no point having a perfect score when it is actually a lie. It seems
> to me that Brion is against the removal of these tests because he wants them
> to pass. Having a third state of "known to fail" makes sense, just changing
> them to pass makes it necessary to add a "citation needed" because it is
> just not true.

Nobody's changing them to pass.

_______________________________________________
Wikitech-l mailing list
Wikitech-l [at] lists
https://lists.wikimedia.org/mailman/listinfo/wikitech-l

ktc at ktchan

Jul 21, 2009, 4:43 AM

Post #8 of 20 (1291 views)
Permalink

Re: "known to fail" switch added to parserTests [In reply to]

Aryeh Gregor wrote:
> On Tue, Jul 21, 2009 at 9:56 AM, Gerard
> Meijssen<gerard.meijssen [at] gmail> wrote:
>> There is no point having a perfect score when it is actually a lie. It seems
>> to me that Brion is against the removal of these tests because he wants them
>> to pass. Having a third state of "known to fail" makes sense, just changing
>> them to pass makes it necessary to add a "citation needed" because it is
>> just not true.
>
> Nobody's changing them to pass.

I can understand where Gerard got the impression:

"I have modified parserTests to take a "known to fail" switch so those
tests that have always failed now pass." - dan nessett 2009-07-20 16:09

KTC

--
Experience is a good school but the fees are high.
- Heinrich Heine

Attachments:

PGP.sig (0.19 KB)

innocentkiller at gmail

Jul 21, 2009, 4:46 AM

Post #9 of 20 (1289 views)
Permalink

Re: "known to fail" switch added to parserTests [In reply to]

On Tue, Jul 21, 2009 at 7:43 AM, Kwan Ting Chan<ktc [at] ktchan> wrote:
> Aryeh Gregor wrote:
>>
>> On Tue, Jul 21, 2009 at 9:56 AM, Gerard
>> Meijssen<gerard.meijssen [at] gmail> wrote:
>>>
>>> There is no point having a perfect score when it is actually a lie. It
>>> seems
>>> to me that Brion is against the removal of these tests because he wants
>>> them
>>> to pass. Having a third state of "known to fail" makes sense, just
>>> changing
>>> them to pass makes it necessary to add a "citation needed" because it is
>>> just not true.
>>
>> Nobody's changing them to pass.
>
> I can understand where Gerard got the impression:
>
> "I have modified parserTests to take a "known to fail" switch so those tests
> that have always failed now pass." - dan nessett 2009-07-20 16:09
>
> KTC
>
> --
> Experience is a good school but the fees are high.
> Â Â - Heinrich Heine
>
> _______________________________________________
> Wikitech-l mailing list
> Wikitech-l [at] lists
> https://lists.wikimedia.org/mailman/listinfo/wikitech-l
>

That being said, a patch against 1.14.x is of no use to anyone.

-Chad

_______________________________________________
Wikitech-l mailing list
Wikitech-l [at] lists
https://lists.wikimedia.org/mailman/listinfo/wikitech-l

gerard.meijssen at gmail

Jul 21, 2009, 7:21 AM

Post #10 of 20 (1286 views)
Permalink

Re: "known to fail" switch added to parserTests [In reply to]

Hoi,
That is exactly the problem. You report that they pass and in reality they
still fail. Someone should change them to pass ie fix the software.
Thanks,
GerardM

2009/7/21 Aryeh Gregor
<Simetrical+wikilist [at] gmail<Simetrical%2Bwikilist [at] gmail>
>

> On Tue, Jul 21, 2009 at 9:56 AM, Gerard
> Meijssen<gerard.meijssen [at] gmail> wrote:
> > There is no point having a perfect score when it is actually a lie. It
> seems
> > to me that Brion is against the removal of these tests because he wants
> them
> > to pass. Having a third state of "known to fail" makes sense, just
> changing
> > them to pass makes it necessary to add a "citation needed" because it is
> > just not true.
>
> Nobody's changing them to pass.
>
> _______________________________________________
> Wikitech-l mailing list
> Wikitech-l [at] lists
> https://lists.wikimedia.org/mailman/listinfo/wikitech-l
>
_______________________________________________
Wikitech-l mailing list
Wikitech-l [at] lists
https://lists.wikimedia.org/mailman/listinfo/wikitech-l

magnusmanske at googlemail

Jul 21, 2009, 7:31 AM

Post #11 of 20 (1286 views)
Permalink

Re: "known to fail" switch added to parserTests [In reply to]

On Tue, Jul 21, 2009 at 3:21 PM, Gerard
Meijssen<gerard.meijssen [at] gmail> wrote:
> Hoi,
> That is exactly the problem. You report that they pass and in reality they
> still fail. Someone should change them to pass ie fix the software.

I think the appropriate English reply in this context would be "No
s**t, Sherlock"...

_______________________________________________
Wikitech-l mailing list
Wikitech-l [at] lists
https://lists.wikimedia.org/mailman/listinfo/wikitech-l

dnessett at yahoo

Jul 21, 2009, 7:45 AM

Post #12 of 20 (1287 views)
Permalink

Re: "known to fail" switch added to parserTests [In reply to]

The change I made was to add a "flipresult" option that simply turns a success into a failure and a failure into a success. This is what I understood I was asked to do. On the plus side, this approach also allows the addition of parser tests that are supposed to fail (not just have always failed). On the negative side it does hide problems that perhaps should remain in the open.

I just looked at the code and it shouldn't be hard to add a knowntofail option that acts like flipresult and then add a new category of test result that specifies how many known to fail results occurred. However, one issue is whether known to fail should count against success/failure (when computing the percentage of tests that failed) or whether these results should not count toward either.

Would someone tell me where the redirect from index.php to index.php/Main_Page occurs in the page processing flow?

--- On Tue, 7/21/09, Kwan Ting Chan <ktc [at] ktchan> wrote:

> From: Kwan Ting Chan <ktc [at] ktchan>
> Subject: Re: [Wikitech-l] "known to fail" switch added to parserTests
> To: "Wikimedia developers" <wikitech-l [at] lists>
> Date: Tuesday, July 21, 2009, 4:43 AM
> Aryeh Gregor wrote:
> > On Tue, Jul 21, 2009 at 9:56 AM, Gerard
> > Meijssen<gerard.meijssen [at] gmail>
> wrote:
> >> There is no point having a perfect score when it
> is actually a lie. It seems
> >> to me that Brion is against the removal of these
> tests because he wants them
> >> to pass. Having a third state of "known to fail"
> makes sense, just changing
> >> them to pass makes it necessary to add a "citation
> needed" because it is
> >> just not true.
> >
> > Nobody's changing them to pass.
>
> I can understand where Gerard got the impression:
>
> "I have modified parserTests to take a "known to fail"
> switch so those tests that have always failed now pass." -
> dan nessett 2009-07-20 16:09
>
> KTC
>
> -- Experience is a good school but the fees are high.
> - Heinrich Heine
>
> -----Inline Attachment Follows-----
>
> _______________________________________________
> Wikitech-l mailing list
> Wikitech-l [at] lists
> https://lists.wikimedia.org/mailman/listinfo/wikitech-l

_______________________________________________
Wikitech-l mailing list
Wikitech-l [at] lists
https://lists.wikimedia.org/mailman/listinfo/wikitech-l

stephen.bain at gmail

Jul 21, 2009, 8:49 AM

Post #13 of 20 (1297 views)
Permalink

Re: "known to fail" switch added to parserTests [In reply to]

On Wed, Jul 22, 2009 at 12:45 AM, dan nessett<dnessett [at] yahoo> wrote:
>
> I just looked at the code and it shouldn't be hard to add a knowntofail option that acts like flipresult and then add a new category of test result that specifies how many known to fail results occurred. However, one issue is whether known to fail should count against success/failure (when computing the percentage of tests that failed) or whether these results should not count toward either.

Having a third possible result would be more informative.

You could report two scores, one including known-to-fail tests and one
excluding them. Reporting both or just one of these scores could be
configurable by a command line option.

--
Stephen Bain
stephen.bain [at] gmail

_______________________________________________
Wikitech-l mailing list
Wikitech-l [at] lists
https://lists.wikimedia.org/mailman/listinfo/wikitech-l

Simetrical+wikilist at gmail

Jul 21, 2009, 9:51 AM

Post #14 of 20 (1286 views)
Permalink

Re: "known to fail" switch added to parserTests [In reply to]

On Tue, Jul 21, 2009 at 10:45 AM, dan nessett<dnessett [at] yahoo> wrote:
> The change I made was to add a "flipresult" option that simply turns a success into a failure and a failure into a success. This is what I understood I was asked to do. On the plus side, this approach also allows the addition of parser tests that are supposed to fail (not just have always failed). On the negative side it does hide problems that perhaps should remain in the open.

This isn't a good idea, no. The important thing is if someone runs
parserTests.php, they should be able to easily tell whether there are
any regressions in their working copy. But if we're going to keep the
known-to-fail tests at all, it doesn't make a lot of sense to report
them as passing when they're actually failing . . . if we do that we
may as well just drop them.

> I just looked at the code and it shouldn't be hard to add a knowntofail option that acts like flipresult and then add a new category of test result that specifies how many known to fail results occurred. However, one issue is whether known to fail should count against success/failure (when computing the percentage of tests that failed) or whether these results should not count toward either.

It should be made clear that some tests are failing, but that they are
not regressions.

> Would someone tell me where the redirect from index.php to index.php/Main_Page occurs in the page processing flow?

I don't know offhand.

_______________________________________________
Wikitech-l mailing list
Wikitech-l [at] lists
https://lists.wikimedia.org/mailman/listinfo/wikitech-l

removed at example

Jul 21, 2009, 9:57 AM

Post #15 of 20 (1296 views)
Permalink

Re: "known to fail" switch added to parserTests [In reply to]

On Tue, Jul 21, 2009 at 10:51 AM, Aryeh Gregor <
Simetrical+wikilist [at] gmail <Simetrical%2Bwikilist [at] gmail>> wrote:

> But if we're going to keep the known-to-fail tests at all, it doesn't make
> a lot of sense to report
> them as passing when they're actually failing . . . if we do that we may as
> well just drop them.

The tests have never passed - they should be commented out for usability
reasons. And ideally there would be a post-commit hook that runs the parser
tests and e-mails the committer letting them know they have just broken the
software!
_______________________________________________
Wikitech-l mailing list
Wikitech-l [at] lists
https://lists.wikimedia.org/mailman/listinfo/wikitech-l

Simetrical+wikilist at gmail

Jul 21, 2009, 10:10 AM

Post #16 of 20 (1286 views)
Permalink

Re: "known to fail" switch added to parserTests [In reply to]

On Tue, Jul 21, 2009 at 12:57 PM, Brian<Brian.Mingus [at] colorado> wrote:
> The tests have never passed - they should be commented out for usability
> reasons.

Well, I have no objections, but apparently that's not acceptable. An
"expected fail" flag would be about as usable.

> And ideally there would be a post-commit hook that runs the parser
> tests and e-mails the committer letting them know they have just broken the
> software!

Yes, that would be nice.

_______________________________________________
Wikitech-l mailing list
Wikitech-l [at] lists
https://lists.wikimedia.org/mailman/listinfo/wikitech-l

dnessett at yahoo

Jul 21, 2009, 11:05 AM

Post #17 of 20 (1292 views)
Permalink

Re: "known to fail" switch added to parserTests [In reply to]

Not sure the post-commit hook running the parser is a good idea. The software could have been broken by a previous committer. From that point on parserTests will report errors until the problem is fixed, so committers will just learn to ignore the message.

--- On Tue, 7/21/09, Brian <Brian.Mingus [at] colorado> wrote:

> From: Brian <Brian.Mingus [at] colorado>
> Subject: Re: [Wikitech-l] "known to fail" switch added to parserTests
> To: "Wikimedia developers" <wikitech-l [at] lists>
> Date: Tuesday, July 21, 2009, 9:57 AM
> On Tue, Jul 21, 2009 at 10:51 AM,
> Aryeh Gregor <
> Simetrical+wikilist [at] gmail
> <Simetrical%2Bwikilist [at] gmail>>
> wrote:

... And ideally there would be a post-commit hook that
> runs the parser
> tests and e-mails the committer letting them know they have
> just broken the
> software!
> _______________________________________________
> Wikitech-l mailing list
> Wikitech-l [at] lists
> https://lists.wikimedia.org/mailman/listinfo/wikitech-l
>

_______________________________________________
Wikitech-l mailing list
Wikitech-l [at] lists
https://lists.wikimedia.org/mailman/listinfo/wikitech-l

removed at example

Jul 21, 2009, 11:10 AM

Post #18 of 20 (1283 views)
Permalink

Re: "known to fail" switch added to parserTests [In reply to]

On Tue, Jul 21, 2009 at 12:05 PM, dan nessett <dnessett [at] yahoo> wrote:

>
> Not sure the post-commit hook running the parser is a good idea. The
> software could have been broken by a previous committer. From that point on
> parserTests will report errors until the problem is fixed, so committers
> will just learn to ignore the message.
>

Right, well, a pre-commit hook that rejects all commits which break the
software. Or a memory of what commits broke which tests and a conditional.
_______________________________________________
Wikitech-l mailing list
Wikitech-l [at] lists
https://lists.wikimedia.org/mailman/listinfo/wikitech-l

dnessett at yahoo

Jul 21, 2009, 11:22 AM

Post #19 of 20 (1284 views)
Permalink

Re: "known to fail" switch added to parserTests [In reply to]

Better ideas. Another possibility is every 24hrs to run parser tests (and any other regression tests that might exist) against all revisions committed into trunk since the last run. Post the results and keep track of the number of bugs each committer has introduced into the code base for the past running 6 month period. Post the names of committers and the number of bugs they have introduced on a "hall of shame" page ordering the list by number of bugs.

Sometimes social pressure can be a very effective behavior modifier.

--- On Tue, 7/21/09, Brian <Brian.Mingus [at] colorado> wrote:

> From: Brian <Brian.Mingus [at] colorado>
> Subject: Re: [Wikitech-l] "known to fail" switch added to parserTests
> To: "Wikimedia developers" <wikitech-l [at] lists>
> Date: Tuesday, July 21, 2009, 11:10 AM
>
> Right, well, a pre-commit hook that rejects all commits
> which break the
> software. Or a memory of what commits broke which tests and
> a conditional.
> _______________________________________________
> Wikitech-l mailing list
> Wikitech-l [at] lists
> https://lists.wikimedia.org/mailman/listinfo/wikitech-l
>

_______________________________________________
Wikitech-l mailing list
Wikitech-l [at] lists
https://lists.wikimedia.org/mailman/listinfo/wikitech-l

Simetrical+wikilist at gmail

Jul 21, 2009, 3:07 PM

Post #20 of 20 (1279 views)
Permalink

Re: "known to fail" switch added to parserTests [In reply to]

On Tue, Jul 21, 2009 at 6:05 PM, dan nessett<dnessett [at] yahoo> wrote:
> Not sure the post-commit hook running the parser is a good idea. The software could have been broken by a previous committer. From that point on parserTests will report errors until the problem is fixed, so committers will just learn to ignore the message.

It can use --record and --compare.

_______________________________________________
Wikitech-l mailing list
Wikitech-l [at] lists
https://lists.wikimedia.org/mailman/listinfo/wikitech-l

Index | Next | Previous | View Threaded

Interested in having your list archived? Contact Gossamer Threads