Login | Register For Free | Help
Search for: (Advanced)

Mailing List Archive: Perl: porters

Win32 assertions/crashes in leftover scope checks [was: RE: Black smoke on Win32 (caused by in-place array reverse changes?)]

 

 

Perl porters RSS feed   Index | Next | Previous | View Threaded


SteveHay at planit

Nov 12, 2009, 3:00 AM

Post #1 of 12 (414 views)
Permalink
Win32 assertions/crashes in leftover scope checks [was: RE: Black smoke on Win32 (caused by in-place array reverse changes?)]

Steve Hay wrote on 2009-11-11:
> Sorry, I didn't look hard enough. Last night's smoke looked much
> noisier than the previous one and I thought that all these test
failures
> were new. But actually the previous one failed the same tests in one
> configuration only, which I hadn't noticed:
>
> http://www.nntp.perl.org/group/perl.daily-
> build.reports/2009/11/msg75973 .html
>
> In fact, the last clean smoke (allowing for BCC's currently normaly
> failures) was fdabea7a33d4a48e0ef4d209acdec9d6d4fde9b1:
>
> http://www.nntp.perl.org/group/perl.daily-
> build.reports/2009/11/msg75912 .html
>
> So it's nothing to do with your changes. Apologies.


The crashes and assertions were introduced by the change that added
those assertions, 3d22c4f05aed968c6c562e40be30222328d66f6b. All tests
pass at the previous patchlevel,
394f6a098b9fc7255c17e6d9614c6c9fff4793ea.


gerard at ggoossen

Nov 12, 2009, 5:28 AM

Post #2 of 12 (399 views)
Permalink
Re: Win32 assertions/crashes in leftover scope checks [was: RE: Black smoke on Win32 (caused by in-place array reverse changes?)] [In reply to]

On Thu, Nov 12, 2009 at 11:00:03AM -0000, Steve Hay wrote:
> Steve Hay wrote on 2009-11-11:
> > Sorry, I didn't look hard enough. Last night's smoke looked much
> > noisier than the previous one and I thought that all these test
> failures
> > were new. But actually the previous one failed the same tests in one
> > configuration only, which I hadn't noticed:
> >
> > http://www.nntp.perl.org/group/perl.daily-
> > build.reports/2009/11/msg75973 .html
> >
> > In fact, the last clean smoke (allowing for BCC's currently normaly
> > failures) was fdabea7a33d4a48e0ef4d209acdec9d6d4fde9b1:
> >
> > http://www.nntp.perl.org/group/perl.daily-
> > build.reports/2009/11/msg75912 .html
> >
> > So it's nothing to do with your changes. Apologies.
>
>
> The crashes and assertions were introduced by the change that added
> those assertions, 3d22c4f05aed968c6c562e40be30222328d66f6b. All tests
> pass at the previous patchlevel,
> 394f6a098b9fc7255c17e6d9614c6c9fff4793ea.

That means somewhere the scopes are properly popped. I don't have any Win32
machines, could you try to pinpoint it down a bit and run a crashing program
using the "-Dl" flag.

Gerard


SteveHay at planit

Nov 12, 2009, 7:24 AM

Post #3 of 12 (400 views)
Permalink
RE: Win32 assertions/crashes in leftover scope checks [was: RE: Black smoke on Win32 (caused by in-place array reverse changes?)] [In reply to]

Gerard Goossen wrote on 2009-11-12:
> On Thu, Nov 12, 2009 at 11:00:03AM -0000, Steve Hay wrote:
>> The crashes and assertions were introduced by the change that added
>> those assertions, 3d22c4f05aed968c6c562e40be30222328d66f6b. All tests
>> pass at the previous patchlevel,
>> 394f6a098b9fc7255c17e6d9614c6c9fff4793ea.
> That means somewhere the scopes are properly popped. I don't have any
> Win32 machines, could you try to pinpoint it down a bit and run a
> crashing program using the "-Dl" flag.
>


C:\Temp>type crash.t
my $pid = fork;
die "Can't fork: $!" unless defined $pid;
exit unless $pid;
wait;

C:\Temp>\gitclients\perl\perl -Dl crash.t
(crash.t:0) ENTER scope 2 at op.c:5994
(crash.t:-1) LEAVE scope 2 at op.c:6029
(crash.t:0) ENTER scope 2 at ..\perly.c:353
(crash.t:2) ENTER scope 3 at ..\toke.c:1709
(crash.t:2) LEAVE scope 3 at ..\toke.c:1824
(crash.t:4) ENTER scope 3 at op.c:8440
(crash.t:2) ENTER scope 4 at op.c:8440
(crash.t:3) ENTER scope 5 at op.c:8440
(crash.t:4) LEAVE scope 5 at op.c:9064
(crash.t:3) LEAVE scope 4 at op.c:9064
(crash.t:2) LEAVE scope 3 at op.c:9064
(crash.t:4) LEAVE scope 2 at ..\perly.c:687
(crash.t:0) LEAVE scope 1 at perl.c:2178
(crash.t:0) ENTER scope 1 at perl.c:2189
(crash.t:0) popping jumplevel was 140fb2c, now 374388
(crash.t:0) Setting up jumplevel 140fb3c, was 374388

EXECUTING...

(crash.t:0) Entering new RUNOPS level
(crash.t:0) ENTER scope 2 at ..\pp_hot.c:1767
Entering block 0, type BLOCK
(crash.t:1) Setting up jumplevel 291ff44, was 1832d58
(crash.t:1) Entering new RUNOPS level
Unwinding block 0, type BLOCK
(crash.t:3) popping jumplevel was 291ff44, now 1832d58
Assertion failed: PL_scopestack_ix == 1, file perl.c, line 543

This application has requested the Runtime to terminate it in an unusual
way.
Please contact the application's support team for more information.


gerard at ggoossen

Nov 12, 2009, 7:58 AM

Post #4 of 12 (400 views)
Permalink
Re: Win32 assertions/crashes in leftover scope checks [was: RE: Black smoke on Win32 (caused by in-place array reverse changes?)] [In reply to]

On Thu, Nov 12, 2009 at 03:24:33PM -0000, Steve Hay wrote:
> [...]
> (crash.t:3) popping jumplevel was 291ff44, now 1832d58

This one should is probably missing cleaning the stack. I just send a patch
adding line information to the "popping jumplevel", could you run it again
with the patch applied, so I can find the corresponding code.
Thanks.

Gerard.


SteveHay at planit

Nov 12, 2009, 8:33 AM

Post #5 of 12 (403 views)
Permalink
RE: Win32 assertions/crashes in leftover scope checks [was: RE: Black smoke on Win32 (caused by in-place array reverse changes?)] [In reply to]

Gerard Goossen wrote on 2009-11-12:
> On Thu, Nov 12, 2009 at 03:24:33PM -0000, Steve Hay wrote:
>> [...]
>> (crash.t:3) popping jumplevel was 291ff44, now 1832d58
> This one should is probably missing cleaning the stack. I just send a
> patch adding line information to the "popping jumplevel", could you
run
> it again with the patch applied, so I can find the corresponding code.
> Thanks.
>


C:\Temp>\gitclients\perl\perl -Dl crash.t
(crash.t:0) ENTER scope 2 at op.c:5994
(crash.t:-1) LEAVE scope 2 at op.c:6029
(crash.t:0) ENTER scope 2 at ..\perly.c:353
(crash.t:2) ENTER scope 3 at ..\toke.c:1709
(crash.t:2) LEAVE scope 3 at ..\toke.c:1824
(crash.t:4) ENTER scope 3 at op.c:8440
(crash.t:2) ENTER scope 4 at op.c:8440
(crash.t:3) ENTER scope 5 at op.c:8440
(crash.t:4) LEAVE scope 5 at op.c:9064
(crash.t:3) LEAVE scope 4 at op.c:9064
(crash.t:2) LEAVE scope 3 at op.c:9064
(crash.t:4) LEAVE scope 2 at ..\perly.c:687
(crash.t:0) LEAVE scope 1 at perl.c:2178
(crash.t:0) ENTER scope 1 at perl.c:2189
(crash.t:0) popping jumplevel was 140fb2c, now 3743f8 at perl.c:1632
(crash.t:0) Setting up jumplevel 140fb3c, was 3743f8 at perl.c:2220

EXECUTING...

(crash.t:0) Entering new RUNOPS level
(crash.t:0) ENTER "block" scope 2 at ..\pp_hot.c:1767
Entering block 0, type BLOCK
(crash.t:1) Setting up jumplevel 291ff44, was 1832ad0 at
c:\gitclients\perl\win32\perlhost.h:1768
(crash.t:1) Entering new RUNOPS level
Unwinding block 0, type BLOCK
(crash.t:3) popping jumplevel was 291ff44, now 1832ad0 at
c:\gitclients\perl\win32\perlhost.h:1795
Assertion failed: PL_scopestack_ix == 1, file perl.c, line 543

This application has requested the Runtime to terminate it in an unusual
way.
Please contact the application's support team for more information.


SteveHay at planit

Nov 13, 2009, 2:50 AM

Post #6 of 12 (397 views)
Permalink
RE: Win32 assertions/crashes in leftover scope checks [was: RE: Black smoke on Win32 (caused by in-place array reverse changes?)] [In reply to]

Steve Hay wrote on 2009-11-12:
> Gerard Goossen wrote on 2009-11-12:
>> On Thu, Nov 12, 2009 at 03:24:33PM -0000, Steve Hay wrote:
>>> [...]
>>> (crash.t:3) popping jumplevel was 291ff44, now 1832d58
>>> This one should is probably missing cleaning the stack. I just send
> a
>> patch adding line information to the "popping jumplevel", could you
run
>> it again with the patch applied, so I can find the corresponding
code.
>> Thanks.
>>
>
>
> C:\Temp>\gitclients\perl\perl -Dl crash.t (crash.t:0) ENTER scope
2
> at op.c:5994 (crash.t:-1) LEAVE scope 2 at op.c:6029 (crash.t:0)

> ENTER scope 2 at ..\perly.c:353 (crash.t:2) ENTER scope 3 at
> ..\toke.c:1709 (crash.t:2) LEAVE scope 3 at ..\toke.c:1824
> (crash.t:4) ENTER scope 3 at op.c:8440 (crash.t:2) ENTER scope
4
> at op.c:8440 (crash.t:3) ENTER scope 5 at op.c:8440 (crash.t:4)

> LEAVE scope 5 at op.c:9064 (crash.t:3) LEAVE scope 4 at op.c:9064
> (crash.t:2) LEAVE scope 3 at op.c:9064 (crash.t:4) LEAVE scope
2
> at ..\perly.c:687 (crash.t:0) LEAVE scope 1 at perl.c:2178
> (crash.t:0) ENTER scope 1 at perl.c:2189 (crash.t:0) popping
> jumplevel was 140fb2c, now 3743f8 at perl.c:1632 (crash.t:0)
Setting
> up jumplevel 140fb3c, was 3743f8 at perl.c:2220
>
> EXECUTING...
>
> (crash.t:0) Entering new RUNOPS level (crash.t:0) ENTER
"block"
> scope 2 at ..\pp_hot.c:1767 Entering block 0, type BLOCK (crash.t:1)

> Setting up jumplevel 291ff44, was 1832ad0 at
> c:\gitclients\perl\win32\perlhost.h:1768 (crash.t:1) Entering new
> RUNOPS level Unwinding block 0, type BLOCK (crash.t:3) popping
> jumplevel was 291ff44, now 1832ad0 at
> c:\gitclients\perl\win32\perlhost.h:1795 Assertion failed:
> PL_scopestack_ix == 1, file perl.c, line 543
>
> This application has requested the Runtime to terminate it in an
unusual
> way. Please contact the application's support team for more
information.


Following cbdd53315d32c50fe69953522ff12b867c3e7d66 and
f90117a9a237d776b28f3a1207f17a7eed835355, that program now runs fine,
but there is still one test failure: op/fork.t test 17:

C:\gitclients\perl\t>.\perl harness -v op/fork.t
op/fork.t ..
1..21
ok 1
ok 2
ok 3
ok 4
ok 5
ok 6
ok 7
ok 8
ok 9
ok 10
ok 11
ok 12
ok 13
ok 14
ok 15
ok 16
PROG:
BEGIN {
$| = 1;
fork and exit;
print "inner\n";
}
# XXX In emulated fork(), the child will not execute anything after
# the BEGIN block, due to difficulties in recreating the parse stacks
# and restarting yyparse() midstream in the child. This can potentially
# be overcome by treating what's after the BEGIN{} as a brand new parse.
#print "outer\n"
EXPECTED:
inner
GOT:
inner
A s s e r t i o n f a i l e d : P L _ s c o p e s t a c k _ i x =
= 1 , f i l e p e r l . c , l i n e 5 4 3

This application has requested the Runtime to terminate it in an unusual
way.
Please contact the application's support team for more information.
not ok 17
ok 18
ok 19
ok 20
ok 21
Failed 1/21 subtests

Test Summary Report
-------------------
op/fork.t (Wstat: 0 Tests: 21 Failed: 1)
Failed test: 17
Files=1, Tests=21, 11 wallclock secs ( 0.05 usr + 0.00 sys = 0.05 CPU)
Result: FAIL


gerard at ggoossen

Nov 13, 2009, 5:18 AM

Post #7 of 12 (400 views)
Permalink
Re: Win32 assertions/crashes in leftover scope checks [was: RE: Black smoke on Win32 (caused by in-place array reverse changes?)] [In reply to]

On Fri, Nov 13, 2009 at 10:50:03AM -0000, Steve Hay wrote:
> [...]
> PROG:
> BEGIN {
> $| = 1;
> fork and exit;
> print "inner\n";
> }
> # XXX In emulated fork(), the child will not execute anything after
> # the BEGIN block, due to difficulties in recreating the parse stacks
> # and restarting yyparse() midstream in the child. This can potentially
> # be overcome by treating what's after the BEGIN{} as a brand new parse.
> #print "outer\n"
> EXPECTED:
> inner
> GOT:
> inner
> A s s e r t i o n f a i l e d : P L _ s c o p e s t a c k _ i x =
> = 1 , f i l e p e r l . c , l i n e 5 4 3

This assertion failure is caused by the same problem that causes the
execution to stop after the BEGIN block (it more or less detects that
there were things left unexecuted). We could simply throw away what
was left of the stack, but I think it might be better to given an
error or a warning that execution was aborted due to limitations of
the emulated fork.


Gerard


gerard at ggoossen

Nov 16, 2009, 1:45 AM

Post #8 of 12 (379 views)
Permalink
Re: Win32 assertions/crashes in leftover scope checks [was: RE: Black smoke on Win32 (caused by in-place array reverse changes?)] [In reply to]

On Fri, Nov 13, 2009 at 02:18:51PM +0100, Gerard Goossen wrote:
> On Fri, Nov 13, 2009 at 10:50:03AM -0000, Steve Hay wrote:
> > [...]
> > PROG:
> > BEGIN {
> > $| = 1;
> > fork and exit;
> > print "inner\n";
> > }
> > # XXX In emulated fork(), the child will not execute anything after
> > # the BEGIN block, due to difficulties in recreating the parse stacks
> > # and restarting yyparse() midstream in the child. This can potentially
> > # be overcome by treating what's after the BEGIN{} as a brand new parse.
> > #print "outer\n"
> > EXPECTED:
> > inner
> > GOT:
> > inner
> > A s s e r t i o n f a i l e d : P L _ s c o p e s t a c k _ i x =
> > = 1 , f i l e p e r l . c , l i n e 5 4 3
>
> This assertion failure is caused by the same problem that causes the
> execution to stop after the BEGIN block (it more or less detects that
> there were things left unexecuted). We could simply throw away what
> was left of the stack, but I think it might be better to given an
> error or a warning that execution was aborted due to limitations of
> the emulated fork.

With the attach patch, an error is given when the execution is "finished" after
the BEGIN block.
The patch is not tested, and the fork.t test still need to be adjusted for the
new error.

Gerard
Attachments: 0001-Croak-on-aborted-execution-with-pseudo-fork.patch (0.68 KB)


jand at activestate

Nov 16, 2009, 3:44 PM

Post #9 of 12 (377 views)
Permalink
RE: Win32 assertions/crashes in leftover scope checks [was: RE: Black smoke on Win32 (caused by in-place array reverse changes?)] [In reply to]

On Fri, 13 Nov 2009, Gerard Goossen wrote:
> On Fri, Nov 13, 2009 at 10:50:03AM -0000, Steve Hay wrote:
> > [...]
> > PROG:
> > BEGIN {
> > $| = 1;
> > fork and exit;
> > print "inner\n";
> > }
> > # XXX In emulated fork(), the child will not execute anything after
> > # the BEGIN block, due to difficulties in recreating the parse stacks
> > # and restarting yyparse() midstream in the child. This can potentially
> > # be overcome by treating what's after the BEGIN{} as a brand new parse.
> > #print "outer\n"
> > EXPECTED:
> > inner
> > GOT:
> > inner
> > A s s e r t i o n f a i l e d : P L _ s c o p e s t a c k _ i x =
> > = 1 , f i l e p e r l . c , l i n e 5 4 3
>
> This assertion failure is caused by the same problem that causes the
> execution to stop after the BEGIN block (it more or less detects that
> there were things left unexecuted). We could simply throw away what
> was left of the stack, but I think it might be better to given an
> error or a warning that execution was aborted due to limitations of
> the emulated fork.

I think I would prefer to just throw away anything still left on the
stack. The current behavior (exiting once the BEGIN block has executed)
is a documented limitation of the fork() emulation (pod/perlfork.pod).

Cheers,
-Jan


jand at activestate

Dec 2, 2009, 1:39 AM

Post #10 of 12 (281 views)
Permalink
RE: Win32 assertions/crashes in leftover scope checks [was: RE: Black smoke on Win32 (caused by in-place array reverse changes?)] [In reply to]

On Mon, 16 Nov 2009, Gerard Goossen wrote:
> On Fri, Nov 13, 2009 at 02:18:51PM +0100, Gerard Goossen wrote:
> > On Fri, Nov 13, 2009 at 10:50:03AM -0000, Steve Hay wrote:
> > > [...]
> > > PROG:
> > > BEGIN {
> > > $| = 1;
> > > fork and exit;
> > > print "inner\n";
> > > }
> > > # XXX In emulated fork(), the child will not execute anything after
> > > # the BEGIN block, due to difficulties in recreating the parse stacks
> > > # and restarting yyparse() midstream in the child. This can potentially
> > > # be overcome by treating what's after the BEGIN{} as a brand new parse.
> > > #print "outer\n"
> > > EXPECTED:
> > > inner
> > > GOT:
> > > inner
> > > A s s e r t i o n f a i l e d : P L _ s c o p e s t a c k _ i x =
> > > = 1 , f i l e p e r l . c , l i n e 5 4 3
> >
> > This assertion failure is caused by the same problem that causes the
> > execution to stop after the BEGIN block (it more or less detects that
> > there were things left unexecuted). We could simply throw away what
> > was left of the stack, but I think it might be better to given an
> > error or a warning that execution was aborted due to limitations of
> > the emulated fork.
>
> With the attach patch, an error is given when the execution is
> "finished" after the BEGIN block. The patch is not tested, and the
> fork.t test still need to be adjusted for the new error.

Commit adab9969 pops all remaining scopes instead of die()ing with an error.
This means no change to op/fork.t is necessary, which I find preferable, as
the current behavior is already documented (as a limitation) in perlfork.pod.

Cheers,
-Jan


gerard at ggoossen

Dec 8, 2009, 4:13 AM

Post #11 of 12 (254 views)
Permalink
Re: Win32 assertions/crashes in leftover scope checks [was: RE: Black smoke on Win32 (caused by in-place array reverse changes?)] [In reply to]

On Wed, Dec 02, 2009 at 01:39:19AM -0800, Jan Dubois wrote:
> On Mon, 16 Nov 2009, Gerard Goossen wrote:
> > On Fri, Nov 13, 2009 at 02:18:51PM +0100, Gerard Goossen wrote:
> > > On Fri, Nov 13, 2009 at 10:50:03AM -0000, Steve Hay wrote:
> > > > [...]
> > > > PROG:
> > > > BEGIN {
> > > > $| = 1;
> > > > fork and exit;
> > > > print "inner\n";
> > > > }
> > > > # XXX In emulated fork(), the child will not execute anything after
> > > > # the BEGIN block, due to difficulties in recreating the parse stacks
> > > > # and restarting yyparse() midstream in the child. This can potentially
> > > > # be overcome by treating what's after the BEGIN{} as a brand new parse.
> > > > #print "outer\n"
> > > > EXPECTED:
> > > > inner
> > > > GOT:
> > > > inner
> > > > A s s e r t i o n f a i l e d : P L _ s c o p e s t a c k _ i x =
> > > > = 1 , f i l e p e r l . c , l i n e 5 4 3
> > >
> > > This assertion failure is caused by the same problem that causes the
> > > execution to stop after the BEGIN block (it more or less detects that
> > > there were things left unexecuted). We could simply throw away what
> > > was left of the stack, but I think it might be better to given an
> > > error or a warning that execution was aborted due to limitations of
> > > the emulated fork.
> >
> > With the attach patch, an error is given when the execution is
> > "finished" after the BEGIN block. The patch is not tested, and the
> > fork.t test still need to be adjusted for the new error.
>
> Commit adab9969 pops all remaining scopes instead of die()ing with an error.
> This means no change to op/fork.t is necessary, which I find preferable, as
> the current behavior is already documented (as a limitation) in perlfork.pod.

Doing this should be fine, but with commit
1cb985b013ea71b82afbc114ed06f94d451f5e04 you changed it to throw them away,
with a comment about "failed because some of the memory was allocated from
a different pool", which I think must be wrong: when cloning the
interpreter everything should be copied, including all the scopes. I
suspect there is a bug in the cloning code which is only hidden by not
leaving the scopes.
Preventing the scopes to be cleaned only here is not a good idea, because
there are other places where they are cleaned, like for example when an
exit occured and the code exists using the "case 2" in the same function.
Also iirc if the BEGIN block was called from inside some kind of eval
execution will continue there.

Gerard Goossen


jand at activestate

Dec 8, 2009, 9:14 AM

Post #12 of 12 (250 views)
Permalink
RE: Win32 assertions/crashes in leftover scope checks [was: RE: Black smoke on Win32 (caused by in-place array reverse changes?)] [In reply to]

On Tue, 08 Dec 2009, Gerard Goossen wrote:
> On Wed, Dec 02, 2009 at 01:39:19AM -0800, Jan Dubois wrote:
> > Commit adab9969 pops all remaining scopes instead of die()ing with an error.
> > This means no change to op/fork.t is necessary, which I find preferable, as
> > the current behavior is already documented (as a limitation) in perlfork.pod.
>
> Doing this should be fine, but with commit
> 1cb985b013ea71b82afbc114ed06f94d451f5e04 you changed it to throw them away,
> with a comment about "failed because some of the memory was allocated from
> a different pool", which I think must be wrong: when cloning the
> interpreter everything should be copied, including all the scopes. I
> suspect there is a bug in the cloning code which is only hidden by not
> leaving the scopes.

Yes, I agree that this is hiding an issue elsewhere. Note though that
this is not a regression against previous releases; the additional
frames were always discarded before.

The motivation to change anything at all was purely to get rid of the
errors thrown in the new assertions in perl_destruct() to get clean
smokes for the upcoming 5.12 release.

The first patch did seem to work ok under -DDEBUGGING but moved the
problem to the non-DEBUGGING builds which would now die with a "free
from wrong pool" message. So short of finding and fixing the underlying
issue there were only 2 workarounds left (that I could think of): either
comment out the assertions for Windows builds, or discard the additional
scopes and push down our one remaining scope to the root of the stack. I
preferred the latter because it keeps the hack in one place.

I now get completely clean builds/tests on Windows for both DEBUGGING
and non-DEBUGGING, and while there may be some hidden problem left,
it is at least not one that is triggered by the current set of tests
(and may very well not be visible at the Perl level at all, beyond
what is already documented as limitations in perlfork.pod).

Now that I think about it once more, it seems possible that the "bad
scope" is just PL_scopestack[0] because the "free from wrong pool" still
happened when I simply set PL_scopestack_ix to 1 without moving the top
scope entry down. The error then originated from
CopFILE_free(&PL_compiling) in perl_destruct() and not from the scope
cleanup. Does this give you an idea where the bug might be?

> Preventing the scopes to be cleaned only here is not a good idea, because
> there are other places where they are cleaned, like for example when an
> exit occured and the code exists using the "case 2" in the same function.
> Also iirc if the BEGIN block was called from inside some kind of eval
> execution will continue there.

I have to admit that I personally don't find the fork() emulation that
useful to begin with, so looking into failure modes when fork() is called
from within an eval() inside a BEGIN block is pretty low on my list of
things I would look into, given my rather limited supply of tuits.

Cheers,
-Jan

Perl porters RSS feed   Index | Next | Previous | View Threaded
 
 


Interested in having your list archived? Contact Gossamer Threads
 
  Web Applications & Managed Hosting Powered by Gossamer Threads Inc.