Mailing List Archive: SPF: Help

python policyd-spf hangs when Domain_Whitelist is used

Index | Next | Previous | View Threaded

hybiepoo at gmail

Dec 1, 2008, 9:37 PM

Post #1 of 10 (2767 views)
Permalink

python policyd-spf hangs when Domain_Whitelist is used

Firstly, the reason I'm having to whitelist a domain is because python
policyd times out when checking an email sent from aol. Not sure why
this is - possibly because of a very long SPF record?
Because of this, I switched to perl policyd only to find out that it
times out when checking an email sent from citigroup.com

My guess with the timeout on citigroup is because they have
underscores in the hostnames of their SPF records:
;; ANSWER SECTION:
citigroup.com. 3347 IN TXT "v=spf1
a:1._spf.citigroup.com a:2._spf.citigroup.com
redirect=ext1._spf.citigroup.com"

Anyway, I decided to try switching back to python policyd-spf and
using the Domain_Whitelist to allow aol.com (even though I really
don't want to) through without SPF checking.

After setting Domain_Whitelist = aol.com, running a manual test of
policyd-spf seems to take anywhere up to 10 minutes to respond. Any
idea why this might be?

Regards,
Jeff

-------------------------------------------
Sender Policy Framework: http://www.openspf.org
Modify Your Subscription: http://www.listbox.com/member/
Archives: https://www.listbox.com/member/archive/1020/=now
RSS Feed: https://www.listbox.com/member/archive/rss/1020/
Powered by Listbox: http://www.listbox.com

scott at kitterman

Dec 1, 2008, 11:46 PM

Post #2 of 10 (2654 views)
Permalink

Re: python policyd-spf hangs when Domain_Whitelist is used [In reply to]

On Tuesday 02 December 2008 00:37, Jeff Mills wrote:
> Firstly, the reason I'm having to whitelist a domain is because python
> policyd times out when checking an email sent from aol. Not sure why
> this is - possibly because of a very long SPF record?

aol.com has a record big enough that it won't generally fit in a UDP packet
and so it has to fall back to TCP. This is legal, but not recommended by RFC
4408.

Some things to check:

Make sure you have the current pyspf (python-spf) and pydns (python-dns). You
need pyspf 2.0.5 (or 2.0.4-4 in Debian/Ubuntu) and pydns at least 2.3.1 (and
you really want 2.3.3 for security reasons). Earlier versions did not
support TCP fallback.

If those are current, then consider using tcpdump or wireshark to capture the
DNS traffic. Some firewalls block DNS on TCP by default. If it's just AOL,
then I'm reasonably certain it's something to do with TCP fallback and not
the Python policy server.

Scott K

-------------------------------------------
Sender Policy Framework: http://www.openspf.org
Modify Your Subscription: http://www.listbox.com/member/
Archives: https://www.listbox.com/member/archive/1020/=now
RSS Feed: https://www.listbox.com/member/archive/rss/1020/
Powered by Listbox: http://www.listbox.com

scott at kitterman

Dec 1, 2008, 11:53 PM

Post #3 of 10 (2669 views)
Permalink

Re: python policyd-spf hangs when Domain_Whitelist is used [In reply to]

On Tuesday 02 December 2008 00:37, Jeff Mills wrote:
> Because of this, I switched to perl policyd only to find out that it
> times out when checking an email sent from citigroup.com
>
> My guess with the timeout on citigroup is because they have
> underscores in the hostnames of their SPF records:
> ;; ANSWER SECTION:
> citigroup.com. 3347 IN TXT "v=spf1
> a:1._spf.citigroup.com a:2._spf.citigroup.com
> redirect=ext1._spf.citigroup.com"

Works here with the Perl policy server. My prime suspect is a firewall that's
trying to help you because it thinks the underscores aren't valid (they are
valid domain names, but not valid hostnames). Once again, I suggest
wireshark or tcpdump to capture some data off the wire.

Scott K

-------------------------------------------
Sender Policy Framework: http://www.openspf.org
Modify Your Subscription: http://www.listbox.com/member/
Archives: https://www.listbox.com/member/archive/1020/=now
RSS Feed: https://www.listbox.com/member/archive/rss/1020/
Powered by Listbox: http://www.listbox.com

hybiepoo at gmail

Dec 2, 2008, 1:52 PM

Post #4 of 10 (2662 views)
Permalink

Re: python policyd-spf hangs when Domain_Whitelist is used [In reply to]

On Tue, Dec 2, 2008 at 6:46 PM, Scott Kitterman <scott [at] kitterman> wrote:
> On Tuesday 02 December 2008 00:37, Jeff Mills wrote:
>> Firstly, the reason I'm having to whitelist a domain is because python
>> policyd times out when checking an email sent from aol. Not sure why
>> this is - possibly because of a very long SPF record?
>
> aol.com has a record big enough that it won't generally fit in a UDP packet
> and so it has to fall back to TCP. This is legal, but not recommended by RFC
> 4408.
>
> Some things to check:
>
> Make sure you have the current pyspf (python-spf) and pydns (python-dns). You
> need pyspf 2.0.5 (or 2.0.4-4 in Debian/Ubuntu) and pydns at least 2.3.1 (and
> you really want 2.3.3 for security reasons). Earlier versions did not
> support TCP fallback.
>
> If those are current, then consider using tcpdump or wireshark to capture the
> DNS traffic. Some firewalls block DNS on TCP by default. If it's just AOL,
> then I'm reasonably certain it's something to do with TCP fallback and not
> the Python policy server.
>
> Scott K
>

Thanks Scott,
Looks like I have the right versions installed, so unfortunately its
time to go digging.

[ Searching for package 'pydns' in all categories among: ]
* installed packages
[I--] [ ~] dev-python/pydns-2.3.3 (0)

[ Searching for package 'pyspf' in all categories among: ]
* installed packages
[I--] [ ~] dev-python/pyspf-2.0.5 (0)

--
-----
regards,
Jeff Mills
http://www.australianpipebands.com
http://www.rightlanehogs.com

-------------------------------------------
Sender Policy Framework: http://www.openspf.org
Modify Your Subscription: http://www.listbox.com/member/
Archives: https://www.listbox.com/member/archive/1020/=now
RSS Feed: https://www.listbox.com/member/archive/rss/1020/
Powered by Listbox: http://www.listbox.com

scott at kitterman

Dec 2, 2008, 2:07 PM

Post #5 of 10 (2663 views)
Permalink

Re: python policyd-spf hangs when Domain_Whitelist is used [In reply to]

On Wed, 3 Dec 2008 08:52:32 +1100 "Jeff Mills" <hybiepoo [at] gmail> wrote:
>On Tue, Dec 2, 2008 at 6:46 PM, Scott Kitterman <scott [at] kitterman>
wrote:
>> On Tuesday 02 December 2008 00:37, Jeff Mills wrote:
>>> Firstly, the reason I'm having to whitelist a domain is because python
>>> policyd times out when checking an email sent from aol. Not sure why
>>> this is - possibly because of a very long SPF record?
>>
>> aol.com has a record big enough that it won't generally fit in a UDP
packet
>> and so it has to fall back to TCP. This is legal, but not recommended
by RFC
>> 4408.
>>
>> Some things to check:
>>
>> Make sure you have the current pyspf (python-spf) and pydns
(python-dns). You
>> need pyspf 2.0.5 (or 2.0.4-4 in Debian/Ubuntu) and pydns at least 2.3.1
(and
>> you really want 2.3.3 for security reasons). Earlier versions did not
>> support TCP fallback.
>>
>> If those are current, then consider using tcpdump or wireshark to
capture the
>> DNS traffic. Some firewalls block DNS on TCP by default. If it's just
AOL,
>> then I'm reasonably certain it's something to do with TCP fallback and
not
>> the Python policy server.
>>
>> Scott K
>>
>
>Thanks Scott,
>Looks like I have the right versions installed, so unfortunately its
>time to go digging.
>
>[ Searching for package 'pydns' in all categories among: ]
> * installed packages
>[I--] [ ~] dev-python/pydns-2.3.3 (0)
>
>[ Searching for package 'pyspf' in all categories among: ]
> * installed packages
>[I--] [ ~] dev-python/pyspf-2.0.5 (0)

I assume you looked in your mail logs and there are no errors of note?

If you can capture the attempt with tcpdump or wireshark, I'll have a look.
Send it off list if you prefer.

At this point my money is on router/firewall confguration.

Scott K

-------------------------------------------
Sender Policy Framework: http://www.openspf.org
Modify Your Subscription: http://www.listbox.com/member/
Archives: https://www.listbox.com/member/archive/1020/=now
RSS Feed: https://www.listbox.com/member/archive/rss/1020/
Powered by Listbox: http://www.listbox.com

hybiepoo at gmail

Dec 2, 2008, 3:25 PM

Post #6 of 10 (2662 views)
Permalink

Re: python policyd-spf hangs when Domain_Whitelist is used [In reply to]

>
> I assume you looked in your mail logs and there are no errors of note?
>
> If you can capture the attempt with tcpdump or wireshark, I'll have a look.
> Send it off list if you prefer.
>
> At this point my money is on router/firewall confguration.
>
> Scott K
>
>

The mail logs show nothing except for the spf timeout.
I just did some more manual testing.
If I use the following, the query can take up to 10 mins:
request=smtpd_access_policy
protocol_state=RCPT
protocol_name=SMTP
helo_name=mail.citigroup.com
queue_id=8045F2AB23
instance=12345.6789
sender=someuser [at] dinersclub
recipient=someuser [at] somedomain
client_address=192.193.221.103
client_name=dinersclub.com.au

10:13:00.510856 IP 10.100.2.240.63370 > exchange.domain: 60215+ TXT?
mail.citigroup.com. (36)
10:13:00.512634 IP 10.100.2.240.56945 > exchange.domain: 3832+ A?
mail.citigroup.com. (36)
10:13:19.815489 IP 10.100.2.240.15441 > exchange.domain: 7567+ TXT?
mail.citigroup.com. (36)
10:13:19.816997 IP 10.100.2.240.2577 > exchange.domain: 17004+ A?
mail.citigroup.com. (36)
10:15:00.944531 IP 10.100.2.240.61705 > exchange.domain: 28011+ TXT?
mail.citigroup.com. (36)
10:15:00.953546 IP 10.100.2.240.24226 > exchange.domain: 63835+ A?
mail.citigroup.com. (36)

If I change helo_name to citigroup.com instead of mail.citigroup.com,
the response is immediate

proxy2 millsj # policyd-spf
request=smtpd_access_policy
protocol_state=RCPT
protocol_name=SMTP
helo_name=citigroup.com
queue_id=8045F2AB23
instance=12345.6789
sender=someuser [at] dinersclub
recipient=someuser [at] somedomain
client_address=192.193.221.103
client_name=dinersclub.com.au

action=prepend Received-SPF: Pass (sender SPF authorized)
identity=helo; client-ip=192.193.221.103; helo=citigroup.com;
envelope-from=someuser [at] dinersclub;
receiver=someuser [at] somedomain

10:00:43.768132 IP 10.100.2.240.21745 > exchange.domain: 2978+ TXT?
citigroup.com. (31)
10:00:43.769410 IP 10.100.2.240.5208 > exchange.domain: 62770+ A?
1._spf.citigroup.com. (38)
10:00:43.769942 IP exchange.domain > 10.100.2.240.5208: 62770 18/0/0 A
smtp8.citigroup.com[|domain]

In the tcpdump I'm only requesting port 53.
Should I be looking at more data than that?

> -------------------------------------------
> Sender Policy Framework: http://www.openspf.org
> Modify Your Subscription: http://www.listbox.com/member/
> Archives: https://www.listbox.com/member/archive/1020/=now
> RSS Feed: https://www.listbox.com/member/archive/rss/1020/
> Powered by Listbox: http://www.listbox.com
>

--
-----
regards,
Jeff Mills
http://www.australianpipebands.com
http://www.rightlanehogs.com

-------------------------------------------
Sender Policy Framework: http://www.openspf.org
Modify Your Subscription: http://www.listbox.com/member/
Archives: https://www.listbox.com/member/archive/1020/=now
RSS Feed: https://www.listbox.com/member/archive/rss/1020/
Powered by Listbox: http://www.listbox.com

hybiepoo at gmail

Dec 2, 2008, 3:52 PM

Post #7 of 10 (2662 views)
Permalink

Re: python policyd-spf hangs when Domain_Whitelist is used [In reply to]

>
> I assume you looked in your mail logs and there are no errors of note?
>
> If you can capture the attempt with tcpdump or wireshark, I'll have a look.
> Send it off list if you prefer.
>
> At this point my money is on router/firewall confguration.
>
> Scott K
>

Ok, well I solved the issue by removing our internal DNS servers from
my mail host.
I'm now querying external servers and both of my issues went away.

So either Microsoft DNS has issues with SPF, or our PIX is blocking something.
I have dns-guard enabled on the firewall as well as a dns packet size
limit of 4096.

>
> -------------------------------------------
> Sender Policy Framework: http://www.openspf.org
> Modify Your Subscription: http://www.listbox.com/member/
> Archives: https://www.listbox.com/member/archive/1020/=now
> RSS Feed: https://www.listbox.com/member/archive/rss/1020/
> Powered by Listbox: http://www.listbox.com
>

--
-----
regards,
Jeff Mills
http://www.australianpipebands.com
http://www.rightlanehogs.com

-------------------------------------------
Sender Policy Framework: http://www.openspf.org
Modify Your Subscription: http://www.listbox.com/member/
Archives: https://www.listbox.com/member/archive/1020/=now
RSS Feed: https://www.listbox.com/member/archive/rss/1020/
Powered by Listbox: http://www.listbox.com

scott at kitterman

Dec 2, 2008, 4:12 PM

Post #8 of 10 (2648 views)
Permalink

Re: python policyd-spf hangs when Domain_Whitelist is used [In reply to]

On Tuesday 02 December 2008 18:52, Jeff Mills wrote:
> > I assume you looked in your mail logs and there are no errors of note?
> >
> > If you can capture the attempt with tcpdump or wireshark, I'll have a
> > look. Send it off list if you prefer.
> >
> > At this point my money is on router/firewall confguration.
> >
> > Scott K
>
> Ok, well I solved the issue by removing our internal DNS servers from
> my mail host.
> I'm now querying external servers and both of my issues went away.
>
> So either Microsoft DNS has issues with SPF, or our PIX is blocking
> something. I have dns-guard enabled on the firewall as well as a dns packet
> size limit of 4096.

Glad to hear it.

It's hard for me to pick. Generally if there's PIX involved, I'm going to
blame it. If you're up for a test it'd be worthwhile to switch back and then
disable dns-guard. If that's the problem, I'd guess Cisco would be
interested in a bug report. Because of their interest in DKIM, I expect
they'd actually be responsive on fixing TXT record related issues.

What platform are you running the policy server on and did you get it through
your distro/OS packaging system or did your install from source?

Scott K

-------------------------------------------
Sender Policy Framework: http://www.openspf.org
Modify Your Subscription: http://www.listbox.com/member/
Archives: https://www.listbox.com/member/archive/1020/=now
RSS Feed: https://www.listbox.com/member/archive/rss/1020/
Powered by Listbox: http://www.listbox.com

hybiepoo at gmail

Dec 2, 2008, 4:50 PM

Post #9 of 10 (2660 views)
Permalink

Re: python policyd-spf hangs when Domain_Whitelist is used [In reply to]

>
> Glad to hear it.
>
> It's hard for me to pick. Generally if there's PIX involved, I'm going to
> blame it. If you're up for a test it'd be worthwhile to switch back and then
> disable dns-guard. If that's the problem, I'd guess Cisco would be
> interested in a bug report. Because of their interest in DKIM, I expect
> they'd actually be responsive on fixing TXT record related issues.
>
> What platform are you running the policy server on and did you get it through
> your distro/OS packaging system or did your install from source?
>
> Scott K
>

Ok, so I installed pypolicyd on another two servers.
The first one was on the inside network, and it queried the internal
DNS servers fine.
This still supported the theory of the firewall blocking the queries.
I installed pypolicyd on another host in the DMZ and set it to query
the internal DNS servers.
It also had no problems, so I'm not sure this is a firewall issue.

I'm running Gentoo and I have installed via the Gentoo Portage System,
but I have also tried installing pypolicyd-spf from source.

--
-----
regards,
Jeff Mills
http://www.australianpipebands.com
http://www.rightlanehogs.com

-------------------------------------------
Sender Policy Framework: http://www.openspf.org
Modify Your Subscription: http://www.listbox.com/member/
Archives: https://www.listbox.com/member/archive/1020/=now
RSS Feed: https://www.listbox.com/member/archive/rss/1020/
Powered by Listbox: http://www.listbox.com

hybiepoo at gmail

Dec 2, 2008, 7:25 PM

Post #10 of 10 (2655 views)
Permalink

Re: python policyd-spf hangs when Domain_Whitelist is used [In reply to]

On Wed, Dec 3, 2008 at 11:12 AM, Scott Kitterman <scott [at] kitterman> wrote:
> On Tuesday 02 December 2008 18:52, Jeff Mills wrote:
>> > I assume you looked in your mail logs and there are no errors of note?
>> >
>> > If you can capture the attempt with tcpdump or wireshark, I'll have a
>> > look. Send it off list if you prefer.
>> >
>> > At this point my money is on router/firewall confguration.
>> >
>> > Scott K
>>
>> Ok, well I solved the issue by removing our internal DNS servers from
>> my mail host.
>> I'm now querying external servers and both of my issues went away.
>>
>> So either Microsoft DNS has issues with SPF, or our PIX is blocking
>> something. I have dns-guard enabled on the firewall as well as a dns packet
>> size limit of 4096.
>
> Glad to hear it.
>
> It's hard for me to pick. Generally if there's PIX involved, I'm going to
> blame it. If you're up for a test it'd be worthwhile to switch back and then
> disable dns-guard. If that's the problem, I'd guess Cisco would be
> interested in a bug report. Because of their interest in DKIM, I expect
> they'd actually be responsive on fixing TXT record related issues.
>
> What platform are you running the policy server on and did you get it through
> your distro/OS packaging system or did your install from source?
>
> Scott K
>
>

One final update.
The firewall was only allowing UDP DNS requests for the host in
question. TCP DNS requests were not allowed.
Fixed!

Thanks for your help Scott.

> -------------------------------------------
> Sender Policy Framework: http://www.openspf.org
> Modify Your Subscription: http://www.listbox.com/member/
> Archives: https://www.listbox.com/member/archive/1020/=now
> RSS Feed: https://www.listbox.com/member/archive/rss/1020/
> Powered by Listbox: http://www.listbox.com
>

--
-----
regards,
Jeff Mills
http://www.australianpipebands.com
http://www.rightlanehogs.com

-------------------------------------------
Sender Policy Framework: http://www.openspf.org
Modify Your Subscription: http://www.listbox.com/member/
Archives: https://www.listbox.com/member/archive/1020/=now
RSS Feed: https://www.listbox.com/member/archive/rss/1020/
Powered by Listbox: http://www.listbox.com

Index | Next | Previous | View Threaded

Interested in having your list archived? Contact Gossamer Threads