Mailing List Archive: Request Tracker: Users

Disappearing messages?

Index | Next | Previous | View Threaded

aos at insync

Jul 21, 2000, 12:50 PM

Post #1 of 12 (768 views)
Permalink

Disappearing messages?

Has anyone else experienced disappearing message syndrome with RT? We're
running the latest release version of RT (and loving it) but lately
messages have started disappearing! It's really weird.

Sometimes a message will make it as far the web tool (it gets into the
transaction history), but never makes it to the customer (I find no
matching sendmail log entries either). Other times a message doesn't even
make it into the web tool, but seems to disappear into thin air.

The worst part is I can't find any consistency. If we resend the exact
same message, it'll usually work the second time. I haven't been able to
do a lot of investigation of the problem due to this lack of consistency.
I plan to dig a little deeper, but thought I'd check with the list to see
if anyone else has experienced something similar or has any tips for how
to go about diagnosing the problem.

-Andrew
--
Andrew O. Smith - <aos [at] insync>
Sysadmin, Insync Internet Services
Houston, Texas, USA

jesse at fsck

Jul 21, 2000, 12:55 PM

Post #2 of 12 (746 views)
Permalink

Re: Disappearing messages? [In reply to]

That's no good. Most of the time I've seen this it's been sendmail related.
What version of sendmail are you running? What are the relevant portions
of your config.pm. Are you sure that sendmail's runnin in queue mode or
is it attempting a single delivery and then falling over?

Is it only losing mail sent by your admin staff or is it losing
customer-generated mail?

-j

On Fri, Jul 21, 2000 at 02:50:36PM -0500, Andrew wrote:
> Has anyone else experienced disappearing message syndrome with RT? We're
> running the latest release version of RT (and loving it) but lately
> messages have started disappearing! It's really weird.
>
> Sometimes a message will make it as far the web tool (it gets into the
> transaction history), but never makes it to the customer (I find no
> matching sendmail log entries either). Other times a message doesn't even
> make it into the web tool, but seems to disappear into thin air.
>
> The worst part is I can't find any consistency. If we resend the exact
> same message, it'll usually work the second time. I haven't been able to
> do a lot of investigation of the problem due to this lack of consistency.
> I plan to dig a little deeper, but thought I'd check with the list to see
> if anyone else has experienced something similar or has any tips for how
> to go about diagnosing the problem.
>
> -Andrew
> --
> Andrew O. Smith - <aos [at] insync>
> Sysadmin, Insync Internet Services
> Houston, Texas, USA
>
>
>
> _______________________________________________
> rt-users mailing list
> rt-users [at] lists
> http://lists.fsck.com/mailman/listinfo/rt-users
>

--
jesse reed vincent --- root [at] eruditorum --- jesse [at] fsck
pgp keyprint: 50 41 9C 03 D0 BC BC C8 2C B9 77 26 6F E1 EB 91
-------------------------------------------------------------
"Bother," said Pooh, "Eeyore, ready two photon torpedoes and lock
phasers on the Heffalump, Piglet, meet me in transporter room three"

aos at insync

Jul 21, 2000, 1:08 PM

Post #3 of 12 (753 views)
Permalink

Re: Disappearing messages? [In reply to]

On Fri, 21 Jul 2000, Jesse wrote:

> That's no good. Most of the time I've seen this it's been sendmail
> related. What version of sendmail are you running?

8.9.3, with another SMTP server set as smarthost, so all non-local mail is
dropped off on the other server for handling. Very little is done
locally.

> What are the relevant portions of your config.pm.

I guess you mean the options related to mail handling, which are:

$mailprog = "/usr/lib/sendmail";
$mail_options = "-oi -t -ODeliveryMode=b -OErrorMode=m";

> Are you sure that sendmail's runnin in queue mode or is it attempting
> a single delivery and then falling over?

sendmail definitely appears to be running properly. One thing I failed to
mention in my previous note which you may find even more interesting is
that we sometimes see messages that contain no body, but DO have the
footer. Here's what a recent one looks like:

Date: Fri, 21 Jul 2000 14:21:34 -0500 (CDT)
From: Melanie Sohns <support [at] insync>
Subject: [insync.net #1067] (support)

-------------------------------------------- Managed by Request Tracker

That's the whole thing. No idea what Melanie said. Doesn't appear in the
web tool either. She is a staff member, so she does have an RT account if
that makes any difference.

Here's where you're going to go, "Oh, you didn't me THAT before": This
copy of RT has been slightly modified to make it behave a little
differently as far as mail goes.

I didn't like the fact that RT put everybody's addresses in the To header.
With a reasonably large support team (about 20 people) it starts to get
really cluttered at the top of the mail, plus people tend to "reply to
all" instead of just to the RT alias, so we end up with duplicate mails. I
modified the RT source to change from To to Bcc to make all the addresses
disappear. I wouldn't think this could be the source of my problem,
though, as I'm not doing anything to the body of the messages, so I don't
see how I could be trashing the entire body as in the above example.

I'll gladly send you my source diffs (privately--no need to subject the
whole lost) if you'd like to take a glance at what I've done. It's very
little, changed only two or three things.

I've considered just dropping in the unmodified version of RT to see if I
see the same problem, but again figured I'd check here before doing that.

> Is it only losing mail sent by your admin staff or is it losing
> customer-generated mail?

I'm only aware of it losing staff mail, but then again, it's the kind of
problem that I may not notice if it's coming from customers too. I
haven't seen any trashed-body messages from customers though.

-Andrew
--
Andrew O. Smith - <aos [at] insync>
Sysadmin, Insync Internet Services
Houston, Texas, USA

tobiasb at tobiasb

Jul 22, 2000, 6:32 AM

Post #4 of 12 (754 views)
Permalink

Re: Disappearing messages? [In reply to]

> That's no good. Most of the time I've seen this it's been sendmail related.

I think I have inserted a hack that makes RT warn or die if it can't send
mail for some reason. How do you process error messages? In RT1, one
option is to use Tie::STDERR, you might read the POD or you might put this
line at the top of rtmux.pl:

use Tie::STDERR my [at] email

It might also be an idea to insert some logging right before RT is to send
emails, if it doesn't log when it should, it's RT itself it's something
wrong with.

Be aware that to avoid loops, RT1 blatantly ignore mails from
mailer-deamon and mails which have the line:

Precedence: bulk

or

RT-Loop-Control: your-rt-tag

in the headers.

--
Spell checkers are for wimps
(please send feedback on all typos)

jdfalk at mail-abuse

Jul 22, 2000, 11:04 AM

Post #5 of 12 (753 views)
Permalink

Re: Disappearing messages? [In reply to]

On 07/22/00, Tobias Brox <tobiasb [at] tobiasb> wrote:

> Be aware that to avoid loops, RT1 blatantly ignore mails from
> mailer-deamon and mails which have the line:
>
> Precedence: bulk
>
> or
>
> RT-Loop-Control: your-rt-tag
>
> in the headers.

By "blatantly ignore," Tobias means that it sends 'em to
/dev/null without logging anything. I modified out this bit of
code pretty quickly; MAPS gets a lot of important, non-bounce
mail from postmaster addresses.

--
J.D. Falk "Laughter is the sound
Product Manager that knowledge makes when it's born."
Mail Abuse Prevention System LLC -- The Cluetrain Manifesto

tobiasb at tobiasb

Jul 22, 2000, 11:21 AM

Post #6 of 12 (748 views)
Permalink

Re: Disappearing messages? [In reply to]

> By "blatantly ignore," Tobias means that it sends 'em to
> /dev/null without logging anything.

This is handled a lot better in RT2; a warning/error is logged, and the
transaction is saved (though it doesn't send out mail to the interessted
parties), so it's possible to see it through the web view or the command
line tool.

--
Spell checkers are for wimps
(please send feedback on all typos)

jesse at fsck

Jul 22, 2000, 3:09 PM

Post #7 of 12 (750 views)
Permalink

Re: Disappearing messages? [In reply to]

Actually, I couldn't find the code that does the Precedence check in RT1.

I think the "right" thing for RT2 to do long term is that if there's mail that
it's 'afraid' to handle, it should send it to rt-owner [at] sit (configurable,
of course) which should be explicitly set to a human....

On Sat, Jul 22, 2000 at 11:04:33AM -0700, J.D. Falk wrote:
> On 07/22/00, Tobias Brox <tobiasb [at] tobiasb> wrote:
>
> > Be aware that to avoid loops, RT1 blatantly ignore mails from
> > mailer-deamon and mails which have the line:
> >
> > Precedence: bulk
> >
> > or
> >
> > RT-Loop-Control: your-rt-tag
> >
> > in the headers.
>
> By "blatantly ignore," Tobias means that it sends 'em to
> /dev/null without logging anything. I modified out this bit of
> code pretty quickly; MAPS gets a lot of important, non-bounce
> mail from postmaster addresses.
>
> --
> J.D. Falk "Laughter is the sound
> Product Manager that knowledge makes when it's born."
> Mail Abuse Prevention System LLC -- The Cluetrain Manifesto
>
>
> _______________________________________________
> rt-users mailing list
> rt-users [at] lists
> http://lists.fsck.com/mailman/listinfo/rt-users
>

--
jesse reed vincent --- root [at] eruditorum --- jesse [at] fsck
pgp keyprint: 50 41 9C 03 D0 BC BC C8 2C B9 77 26 6F E1 EB 91
-------------------------------------------------------------
I think co-ordinating 1000 prima donnas living all over the world will be as
easy as herding cats..." -- Andy Tanenbaum on the linux development model, 1992

tobiasb at tobiasb

Jul 22, 2000, 3:30 PM

Post #8 of 12 (750 views)
Permalink

Re: Disappearing messages? [In reply to]

> I think the "right" thing for RT2 to do long term is that if there's
> mail that it's 'afraid' to handle, it should send it to rt-owner [at] sit
> (configurable, of course) which should be explicitly set to a human....

The right thing is to log it as an alarm message, and the right
thing of the local RT administrator is to configure RT to log such errors
to mail (or eventually even a beeper). The right thing is also to record
the transaction, but avoid sending mail about it. That's how RT2 works
today, and I think it's good enough.

Well, I must admit that I've done at least one dirty hack to get this
working, but if we're to find some better way to do it, we should discuss
it privately or at the rt-devel list. I think the logic itself is as
perfect as it can be.

--
Spell checkers are for wimps
(please send feedback on all typos)

jesse at fsck

Jul 22, 2000, 7:18 PM

Post #9 of 12 (751 views)
Permalink

Re: Disappearing messages? [In reply to]

[followups to rt-devel, please]

On Sun, Jul 23, 2000 at 12:30:37AM +0200, Tobias Brox wrote:
> > I think the "right" thing for RT2 to do long term is that if there's
> > mail that it's 'afraid' to handle, it should send it to rt-owner [at] sit
> > (configurable, of course) which should be explicitly set to a human....
>
> The right thing is to log it as an alarm message, and the right
> thing of the local RT administrator is to configure RT to log such errors
> to mail (or eventually even a beeper). The right thing is also to record
> the transaction, but avoid sending mail about it. That's how RT2 works
> today, and I think it's good enough.
>

It is not always right to log the transaction. In _most_ cases I've dealt with, that would be the wrong thing to do. Which means that it should be configurable
if it exists at all.

--
jesse reed vincent --- root [at] eruditorum --- jesse [at] fsck
pgp keyprint: 50 41 9C 03 D0 BC BC C8 2C B9 77 26 6F E1 EB 91
-------------------------------------------------------------
"Bother," said Pooh, "Eeyore, ready two photon torpedoes and lock
phasers on the Heffalump, Piglet, meet me in transporter room three"

aos at insync

Jul 22, 2000, 7:57 PM

Post #10 of 12 (753 views)
Permalink

Re: Disappearing messages? [In reply to]

On Sat, 22 Jul 2000, Tobias Brox wrote:

> use Tie::STDERR my [at] email

Neat. I'll be sure to turn that on.

As Jesse suggested, I put bone-stock RT back, but I'm still receiving
blank messages. I found something in the logs today that I'd never
noticed before, and I suspect we have our culprit:

Jul 22 16:22:33 ops sendmail[633]: QAA00632: to=|"/usr/local/rt/bin/stripmime support correspond", delay=00:00:03, xdelay=00:00:02, mailer=prog, stat=unknown mailer error 2
Jul 22 16:22:33 ops sendmail[633]: QAA00632: QAA00633: DSN: unknown mailer error 2

Aha. That showed up in the logs just moments before RT fired off the
blank mail, so it could be stripmime is barfing on something.

I'm going to whip up a little wrapper for stripmime to capture stderr and
see where it's dying.

In the meantime, anyone seen this kind of error from stripmime before?

-Andrew
--
Andrew O. Smith - <aos [at] insync>
Sysadmin, Insync Internet Services
Houston, Texas, USA

jesse at fsck

Jul 22, 2000, 9:57 PM

Post #11 of 12 (750 views)
Permalink

Re: Disappearing messages? [In reply to]

Oh. stripmime could indeed be doing that. Try pulling it out of the mix.

On Sat, Jul 22, 2000 at 09:57:03PM -0500, Andrew wrote:
> On Sat, 22 Jul 2000, Tobias Brox wrote:
>
> > use Tie::STDERR my [at] email
>
> Neat. I'll be sure to turn that on.
>
> As Jesse suggested, I put bone-stock RT back, but I'm still receiving
> blank messages. I found something in the logs today that I'd never
> noticed before, and I suspect we have our culprit:
>
> Jul 22 16:22:33 ops sendmail[633]: QAA00632: to=|"/usr/local/rt/bin/stripmime support correspond", delay=00:00:03, xdelay=00:00:02, mailer=prog, stat=unknown mailer error 2
> Jul 22 16:22:33 ops sendmail[633]: QAA00632: QAA00633: DSN: unknown mailer error 2
>
> Aha. That showed up in the logs just moments before RT fired off the
> blank mail, so it could be stripmime is barfing on something.
>
> I'm going to whip up a little wrapper for stripmime to capture stderr and
> see where it's dying.
>
> In the meantime, anyone seen this kind of error from stripmime before?
>
> -Andrew
> --
> Andrew O. Smith - <aos [at] insync>
> Sysadmin, Insync Internet Services
> Houston, Texas, USA
>
>
>
>
>
> _______________________________________________
> rt-users mailing list
> rt-users [at] lists
> http://lists.fsck.com/mailman/listinfo/rt-users
>

--
jesse reed vincent --- root [at] eruditorum --- jesse [at] fsck
pgp keyprint: 50 41 9C 03 D0 BC BC C8 2C B9 77 26 6F E1 EB 91
-------------------------------------------------------------
I think co-ordinating 1000 prima donnas living all over the world will be as
easy as herding cats..." -- Andy Tanenbaum on the linux development model, 1992

aos at insync

Jul 22, 2000, 10:45 PM

Post #12 of 12 (751 views)
Permalink

Re: Disappearing messages? [In reply to]

On Sun, 23 Jul 2000, Jesse wrote:

> Oh. stripmime could indeed be doing that. Try pulling it out of the mix.

For the moment, I'm just capturing stripmime's stderr to a file, and I'm
copying all incoming mail to a regular spool. When it happens again, I
should have both stripmime's output and a copy of the offending mail for
further testing.

I'll let y'all know what I find...

-Andrew
--
Andrew O. Smith - <aos [at] insync>
Sysadmin, Insync Internet Services
Houston, Texas, USA

Index | Next | Previous | View Threaded

Interested in having your list archived? Contact Gossamer Threads