Mailing List Archive: Qmail: users

Weirdness on receiving large messages

Index | Next | Previous | View Threaded

de5 at sws5

Aug 25, 2010, 1:43 PM

Post #1 of 9 (1396 views)
Permalink

Weirdness on receiving large messages

Hey guys! It's been so long since I've posted here that I'm not sure I
remember how to do it.

What I'm seeing is that small messages come in normally, but larger
ones are somehow throttling down to receiving exactly 4096 bytes once
per minute until they eventually time out.

Here's my run file:

----
#!/bin/sh
QMAILDUID=`id -u qmaild`
NOFILESGID=`id -g qmaild`
MAXSMTPD=`cat /var/qmail/control/concurrencyincoming`
exec /usr/local/bin/softlimit -m 50000000 \
/usr/local/bin/tcpserver -v -R -H -x /etc/tcp.smtp.cdb -c "$MAXSMTPD" \
-u "$QMAILDUID" -g "$NOFILESGID" 0 smtp /usr/local/bin/rblsmtpd -r zen.spamhaus.org \
/var/qmail/bin/qmail-smtpd 2>&1
----

Anyone seen anything like this? Any ideas on how to debug it? I'm
seeing it from multiple senders, including another qmail system, so it
sure looks like the problem is on my end.

-Dave

scott.brynen at visioncritical

Aug 25, 2010, 2:40 PM

Post #2 of 9 (1342 views)
Permalink

RE: Weirdness on receiving large messages [In reply to]

I would look into some non-qmail issues; is there a firewall on the outside that's doing some throttling (things like MailChannels);
Or perhaps you're running into a [Cisco] switch which has auto [mis]negotiated the wrong duplex for the link.

-----Original Message-----
From: Dave Sill [mailto:de5 [at] sws5]
Sent: Wednesday, August 25, 2010 1:43 PM
To: qmail [at] list
Subject: Weirdness on receiving large messages

Hey guys! It's been so long since I've posted here that I'm not sure I
remember how to do it.

What I'm seeing is that small messages come in normally, but larger
ones are somehow throttling down to receiving exactly 4096 bytes once
per minute until they eventually time out.

Here's my run file:

----
#!/bin/sh
QMAILDUID=`id -u qmaild`
NOFILESGID=`id -g qmaild`
MAXSMTPD=`cat /var/qmail/control/concurrencyincoming`
exec /usr/local/bin/softlimit -m 50000000 \
/usr/local/bin/tcpserver -v -R -H -x /etc/tcp.smtp.cdb -c "$MAXSMTPD" \
-u "$QMAILDUID" -g "$NOFILESGID" 0 smtp /usr/local/bin/rblsmtpd -r zen.spamhaus.org \
/var/qmail/bin/qmail-smtpd 2>&1
----

Anyone seen anything like this? Any ideas on how to debug it? I'm
seeing it from multiple senders, including another qmail system, so it
sure looks like the problem is on my end.

-Dave

amb-sendok-1285374565.jjnmghmkcpfpceaelgfm at bradfo

Aug 25, 2010, 5:29 PM

Post #3 of 9 (1343 views)
Permalink

Re: Weirdness on receiving large messages [In reply to]

Thus said Dave Sill on Wed, 25 Aug 2010 16:43:08 EDT:

> What I'm seeing is that small messages come in normally, but larger
> ones are somehow throttling down to receiving exactly 4096 bytes once
> per minute until they eventually time out.

Has anything recently changed? Is this a new behavior on an otherwise
old and functioning system, or is this behavior new with a new
installation? It certainly seems that this is beyond qmail's bailiwick,
but it probably can't be ruled out entirely yet. Have you watched
recordio (or even use tcpdump to sniff the traffic)? Does the data come
across the wire in 4096 byte chunks or larger? What does netstat show?
Does it show a recv-q?

Andy

ar-djblists at acrconsulting

Aug 25, 2010, 5:37 PM

Post #4 of 9 (1341 views)
Permalink

Re: Weirdness on receiving large messages [In reply to]

On Wednesday 25 August 2010 21:43:08 Dave Sill wrote:
> Hey guys! It's been so long since I've posted here that I'm not sure I
> remember how to do it.

Good to have you back...

> What I'm seeing is that small messages come in normally, but larger
> ones are somehow throttling down to receiving exactly 4096 bytes once
> per minute until they eventually time out.
>
> Here's my run file:
>
> ----
> #!/bin/sh
> QMAILDUID=`id -u qmaild`
> NOFILESGID=`id -g qmaild`
> MAXSMTPD=`cat /var/qmail/control/concurrencyincoming`
> exec /usr/local/bin/softlimit -m 50000000 \
> /usr/local/bin/tcpserver -v -R -H -x /etc/tcp.smtp.cdb -c
> "$MAXSMTPD" \ -u "$QMAILDUID" -g "$NOFILESGID" 0 smtp
> /usr/local/bin/rblsmtpd -r zen.spamhaus.org \ /var/qmail/bin/qmail-smtpd
> 2>&1
> ----
>
> Anyone seen anything like this? Any ideas on how to debug it? I'm
> seeing it from multiple senders, including another qmail system, so it
> sure looks like the problem is on my end.

I'd agree with that suspicion; I like Scott's mention of external influences
like firewalls; I'd be interested to think about "What's changed?", assuming
that you didn't have this problem with the system (say) 3 months ago. In that
case, have you changed anything about the server or the environment it's in
that might help pinpoint the cause? Also, you say you're seeing this from
multiple senders, but are you seeing this from _all_ senders or is some mail
delivered okay? If some mail is delivered okay, could it be that mail from
particular IP addresses is always delivered okay or always fails?

cheers,

Andrew.
--
====================================================================
* Custom email solutions * Systems Administration * Networking
http://www.acrconsulting.co.uk/email/qmail.html
====================================================================

strange at cultural

Aug 25, 2010, 7:58 PM

Post #5 of 9 (1352 views)
Permalink

Re: Weirdness on receiving large messages [In reply to]

On Wed, 25 Aug 2010, Dave Sill wrote:
> What I'm seeing is that small messages come in normally, but larger
> ones are somehow throttling down to receiving exactly 4096 bytes once
> per minute until they eventually time out.

Long time for me to post, too, Dave. Hi.

That sounds like a classic fragmentation issue - possibly an MTU/MSS
problem. You on DSL by chance?

Lower the qmail server's MSS to 1436 and see if that helps (or changes
things, which will at least be a clue). If there is a new firewall, it
could be the cause. You may be blocking or ignoring MSS negotiation and
path MTU discovery on the FW or the mail server. (For example, see the
description of a similar problem at:
http://www.cisco.com/en/US/tech/tk175/tk15/technologies_tech_note09186a0080093bc7.shtml).

Hope that helps.

-M

de5 at sws5

Aug 26, 2010, 7:32 AM

Post #6 of 9 (1347 views)
Permalink

Re: Weirdness on receiving large messages [In reply to]

Scott Brynen <scott.brynen [at] visioncritical> wrote:
> I would look into some non-qmail issues; is there a firewall on the
> outside that's doing some throttling (things like MailChannels);
> Or perhaps you're running into a [Cisco] switch which has auto
> [mis]negotiated the wrong duplex for the link.

Thanks. Yes, upon further examination, it seems to be a general
problem with sending stuff to my system, not limited to SMTP. I scp'd
a 1 MB file and it did sort of the same thing, but different: a small
burst, then a slow trickle for a few minutes, then a big burst to
finish the transfer. Scp'ing in the other direction finished
immediately. Oddly, the slow scp reported success immediately but
didn't exit until the transfer was complete.

Anyway, I've asked the n/w folks here to look into it.

-Dave

de5 at sws5

Aug 26, 2010, 7:45 AM

Post #7 of 9 (1340 views)
Permalink

Re: Weirdness on receiving large messages [In reply to]

Andy Bradford <amb-sendok-1285374565.jjnmghmkcpfpceaelgfm [at] bradfords> wrote:
>
> Has anything recently changed?

Undoubtedly. :-) *I* haven't changed my n/w configuration recently,
but I have no idea what ORNL's networking folks are doing behind the
scenes.

> Is this a new behavior on an otherwise
> old and functioning system, or is this behavior new with a new
> installation?

New behavior on an old and previously functioning system.

> It certainly seems that this is beyond qmail's bailiwick,

Agreed.

> but it probably can't be ruled out entirely yet. Have you watched
> recordio (or even use tcpdump to sniff the traffic)? Does the data come
> across the wire in 4096 byte chunks or larger?

Here's an abbreviated snippet of one that died before it finished:

...
2010-08-25 16:12:22.129458500 24406 < cyzafSyLzJFyN2zdaudOry+H8w2v9u/Xz8LZSDi/Vuepaxr+6/Z6WaRs6YfBevPq7bPQe9pLrgKz^M
2010-08-25 16:12:22.129474500 24406 < Epl7SbRz0upBwVYgTGlB+lE4v1A2aRWcvPqdUzOCsnfDv3TnDsMvjXbozrlB1TfXj+
...
2010-08-25 16:13:22.130272500 24406 < oECLzIqQkTvZSmv37t2hSORwIrObEG1sBqjK9nKsVs/xIBIgARIgARIgARIgARIgARIgARIgARIg^M
2010-08-25 16:13:22.130288500 24406 < ARIgARKIAQIFXmRWIrISg1VeZrUJoPp/teGfIRDHg8g8efJkfPbZZ7/5dunS5ed2GO3hv1eQidg7^M
2010-08-25 16:13:22.130289500 24406 < 7YAMaAO0AdoAbYA2Q+
2010-08-25 16:13:22.130290500 24406 < [EOF]
2010-08-25 16:13:22.130816500 tcpserver: end 24406 status 256
2010-08-25 16:13:22.130818500 tcpserver: status: 4/30
2010-08-25 16:13:22.131293500 24406 > [EOF]

You can see that the 4k bursts are almost exactly 60 seconds apart.

> What does netstat show? Does it show a recv-q?

tcp 0 0 sws5.ornl.gov:smtp emroute3.ornl.gov:40243 ESTABLISHED
tcp 0 0 sws5.ornl.gov:smtp sws1.ornl.gov:49096 ESTABLISHED
tcp 0 0 sws5.ornl.gov:smtp sws1.ornl.gov:46261 ESTABLISHED

-Dave

de5 at sws5

Aug 26, 2010, 7:50 AM

Post #8 of 9 (1348 views)
Permalink

Re: Weirdness on receiving large messages [In reply to]

Mike Scher <strange [at] cultural> wrote:
> That sounds like a classic fragmentation issue - possibly an MTU/MSS
> problem. You on DSL by chance?

Good to hear from you, Mike. Nope, 100 Mbit Ethernet.

> Lower the qmail server's MSS to 1436 and see if that helps (or changes
> things, which will at least be a clue).

I didn't see any change, but I don't know that what I did worked. I
did:

iptables -A FORWARD -p tcp --tcp-flags SYN,RST SYN -j TCPMSS --set-mss 1436

I'm running Fedora 12.

> Hope that helps.

Thanks. It was worth a try, but I'm punting it to the real networking
experts here.

-Dave

strange at cultural

Aug 26, 2010, 7:52 PM

Post #9 of 9 (1354 views)
Permalink

Re: Weirdness on receiving large messages [In reply to]

On Thu, 26 Aug 2010, Dave Sill wrote:
> Good to hear from you, Mike. Nope, 100 Mbit Ethernet.

...

> Thanks. It was worth a try, but I'm punting it to the real networking
> experts here.

Yeah, I'd bet on problems related to blocking ICMP path MTU discovery,
though TCP RFC1323 extensions could be at play, too.

http://www.znep.com/~marcs/mtu/

Keep us posted!

-M

Index | Next | Previous | View Threaded

Interested in having your list archived? Contact Gossamer Threads