h.m.brand at xs4all
Apr 5, 2012, 12:38 AM
Post #9 of 11
(128 views)
Permalink
|
|
Re: clearer smoke signals (was Re: [perl #106538] Carp is missing a dot)
[In reply to]
|
|
On Wed, 4 Apr 2012 21:02:07 -0400 (EDT), George Greer
<perl [at] greerga> wrote:
> (due to an ADSL outage this is copy/pasted from the web archives)
>
> > On Wed, Feb 29, 2012 at 10:46:43AM -0500, George Greer wrote:
> >
> > > What's on your wishlist?
> >
> > I fear that this isn't complete, as I think I've forgotten something.
> >
> > It's partly that (as best I can tell) the code you're running locally
> > has diverged from the "upstream" code, so it's unclear whether bugs
> > fixes and other improvements are getting made in more than one place,
> > which is a duplication of effort.
>
> My Test::Smoke customizations are:
>
> 1. Run 'make minitest' in addition to other tests.
If others find that useful too, we should make that optional.
> (Although I never did investigate how to get the report matrix to add
> that as a column...)
That should have been solved in the new setup
> 2. I moved the "user_note" to the very top of the reports instead of the
> bottom. (Currently used for URL to reports but also soon a disclaimer
> about my Win32 VM's propensity to have timing issues.)
That is in a template in the new version, moving it to other places is
easy, but maybe not even necessary in the new database face. Our aim is
to not send reports to a list at all (you can still send mail to
yourself to get a copy).
> 3. Also allow ".config" suffix on configurations. (Previously "_config".)
Abe?
> 4. Get version (what is between the brackets in smoke email subject line)
> from $ENV{TEST_SMOKE_BRANCH}, if present, instead of repeating Perl
> version. (This is what makes the "Smoke [blead]" in my reports.)
As no mails will be send, this is moot
> > In particular, I'd like everyone else to run your code, because of a
> > couple of minor things:
> >
> > * the subject using the branch name is terser
> > Smoke [blead] v5.15.9-20-g15d94df
> > vs the tautological
> > Smoke [5.15.9] v5.15.9-20-g15d94df
> > Also those 1 or 2 characters can make a difference when the most
> > important bit is actually the detail of PASS(...) or FAIL(...),
> > which can fall off the right
>
> I thought that was rather silly too. I didn't add that to Test::Smoke
> in the best way though (via parsing ".patch") since my "smoke-me" script
> already adds the branch name as an environment variable and so that was
> fastest for tuit use.
>
> > * the smoke-me code mails me directly if the branch fails
>
> That's a function of the 'smoke-me' script, not Test::Smoke, although I
> agree it is a necessity for a 'smoke-me' service to do so. It's an
> unconditional email though, not just failures.
>
> My configuration for that is:
>
> driver/smoke-me_clang_quick.config.template:
> 'to' => 'smokers-reports [at] perl,%COMMITTER_EMAIL%',
>
> and the 'smoke-me' script does some variable replacements before making
> the .config file for Test::Smoke.
>
> (For those who may not know, the smoke-me script:
> https://github.com/greerga/smoke-me/
> )
>
> > * I can get the logs
In the new setup, logs are being sent to the database is the final
status was not "PASS". This is configurable.
> True, that's necessary for any 'smoke-me' service, Although the dashboard
> Tony Cook has is even better. (http://perl.develop-help.com/reports/)
>
> I wonder if he has that on github...
>
> > but also I'd like a couple of visibility bugs in your setup to be fixed:
> >
> > X X O X X X O X -Uusenm -Duseithreads -Dmad
> > | | | | | +- LC_ALL = en_US.utf8 -DDEBUGGING
> > | | | | +--- PERLIO = perlio -DDEBUGGING
> > | | | +----- PERLIO = stdio -DDEBUGGING
> > | | +------- LC_ALL = en_US.utf8
> > | +--------- PERLIO = perlio
> > +----------- PERLIO = stdio
> >
> > 8 results vs 6 annotations, or 4 results vs 2 annotations:
> >
> > O F F F
> > O F F F -Duseithreads
> > | +--------- -DDEBUGGING
> > +----------- no debugging
>
> Yes, my adding 'minitest' to all runs did that. I do need to fix that.
>
> > after which I guess that there are more general skimming issues with the
> > smoke output. I'm familiar with it, and I find it easy to read, but
> > others are not paying attention to the smoke output because they
> > perceive it as impenetrable noise. It would be useful force those people
> > to explain what they find most obnoxious about it, fix that, iterate
> > until they run out of complaints.
>
> I agree. My biggest wishlist for Test::Smoke is (automatically) keeping
> track of when a test first started failing so it can differentiate between
The new setup registers start time of every smoke-configuration subset
and the duration thereof
> "failure" and "known failure", and while we're at it "sporadic failure"
> would be nice. I find the Test::Smoke code...dense...but then I've mostly
Sporadic failures should now be detectable, as we store failures per
test file, so you could make a trendline for e.g. op/read.t
> only had time for drive-by changes to it and not been able to sit down and
> follow its flow.
>
> > First off, I'm not sure whether the line "Summary: PASS" (or FAIL...)
> > should be the first line, with 2 (or 3) blank lines beneath it. But I'm
> > not the target for such improvements - really the monthly release
> > managers are the people whose input we should be getting.
>
> There was a paucity of replies to this particular thread so we may never
> know.
We will be on #smoke in irc.perl.org to discuss wishes (when possible)
--
H.Merijn Brand http://tux.nl Perl Monger http://amsterdam.pm.org/
using perl5.00307 .. 5.14 porting perl5 on HP-UX, AIX, and openSUSE
http://mirrors.develooper.com/hpux/ http://www.test-smoke.org/
http://qa.perl.org http://www.goldmark.org/jeff/stupid-disclaimers/
|
