Login | Register For Free | Help
Search for: (Advanced)

Mailing List Archive: exim: users

Potential logic error in retry handling for IPv4+IPv6 hosts

 

 

First page Previous page 1 2 Next page Last page  View All exim users RSS feed   Index | Next | Previous | View Threaded


fw at deneb

Dec 1, 2005, 6:17 AM

Post #1 of 29 (2396 views)
Permalink
Potential logic error in retry handling for IPv4+IPv6 hosts

I'm currently facing a mail delivery problem. The sending MTA is Exim
4.50. The host has no real IPv6 connectivity, but a working IPv6
stack, and the DNS resolver returns AAAA records.

Mail to domains which are handled by MX hosts which have both an A and
AAAA record bounces sporadically with the error message "retry time
not reached for any host after a long failure period". The
interesting thing is that the bounce itself gets through, after
delivery of the original message has failed. "exim -bt" shows that
Exim gives indeed priority to the AAAA record.

Does this problem sound familiar?

I don't have sufficient permissions to check the retry database or the
Exim logs, I'm afraid. (The problem may not be related to IPv6 at
all, maybe it's just a corrupt retry database.)

--
## List details at http://www.exim.org/mailman/listinfo/exim-users
## Exim details at http://www.exim.org/
## Please use the Wiki with this list - http://www.exim.org/eximwiki/


msherman at projectile

Dec 1, 2005, 7:16 AM

Post #2 of 29 (2370 views)
Permalink
Re: Potential logic error in retry handling for IPv4+IPv6 hosts [In reply to]

Florian Weimer wrote:
> I'm currently facing a mail delivery problem. The sending MTA is Exim
> 4.50. The host has no real IPv6 connectivity, but a working IPv6
> stack, and the DNS resolver returns AAAA records.
>
> Mail to domains which are handled by MX hosts which have both an A and
> AAAA record bounces sporadically with the error message "retry time
> not reached for any host after a long failure period". The
> interesting thing is that the bounce itself gets through, after
> delivery of the original message has failed. "exim -bt" shows that
> Exim gives indeed priority to the AAAA record.
>
> Does this problem sound familiar?

Very familiar. It happened to me in June[0], and we were just
discussing it again earlier this week[1].

[0]
http://www.exim.org/mail-archives/exim-users/Week-of-Mon-20050613/msg00002.html
[1]
http://www.exim.org/mail-archives/exim-users/Week-of-Mon-20051128/msg00073.html

Philip's conclusion to the second thread[2], just yesterday in fact, was
"do... nothing until somebody else hits a problem." Timing is
everything. :)

[2]
http://www.exim.org/mail-archives/exim-users/Week-of-Mon-20051128/msg00173.html

Anyway, while Philip will probably have some questions for you to help
track down and debug the problem, there is an easy workaround in your
host has no real IPv6 connectivity:

http://www.exim.org/mail-archives/exim-users/Week-of-Mon-20051128/msg00043.html

- Marc

--
## List details at http://www.exim.org/mailman/listinfo/exim-users
## Exim details at http://www.exim.org/
## Please use the Wiki with this list - http://www.exim.org/eximwiki/


aba at not

Dec 1, 2005, 7:41 AM

Post #3 of 29 (2368 views)
Permalink
Re: Potential logic error in retry handling for IPv4+IPv6 hosts [In reply to]

* Marc Sherman (msherman[at]projectile.ca) [051201 16:17]:
> Anyway, while Philip will probably have some questions for you to help
> track down and debug the problem, there is an easy workaround in your
> host has no real IPv6 connectivity:
>
> http://www.exim.org/mail-archives/exim-users/Week-of-Mon-20051128/msg00043.html

Unfortunatly, current status of investiation is that this problem
happens on *all* hosts running Debian sarge w/o real ipv6 connectivity. :(

Some patch to exim would be appreciated for that reason, as that's
easier to upgrade then the config that the users might have edited.


Cheers,
Andi
--
http://home.arcor.de/andreas-barth/

--
## List details at http://www.exim.org/mailman/listinfo/exim-users
## Exim details at http://www.exim.org/
## Please use the Wiki with this list - http://www.exim.org/eximwiki/


fw at deneb

Dec 1, 2005, 7:44 AM

Post #4 of 29 (2354 views)
Permalink
Re: Potential logic error in retry handling for IPv4+IPv6 hosts [In reply to]

* Marc Sherman:

>> Does this problem sound familiar?
>
> Very familiar. It happened to me in June[0], and we were just
> discussing it again earlier this week[1].

Ah, the good old morphogenetic field.

> Anyway, while Philip will probably have some questions for you to help
> track down and debug the problem, there is an easy workaround in your
> host has no real IPv6 connectivity:
>
> http://www.exim.org/mail-archives/exim-users/Week-of-Mon-20051128/msg00043.html

I'm the one who has working IPv6 connectivity, it's other hosts I'm
worried about. 8-/

Note that the failing domains I've seen so far have an MX record
pointing to an A/AAAA combination, and it seems that Exim somehow
applies the AAAA retry hint to the A host, possibly if the A host is
temporarily unreachable as well.

--
## List details at http://www.exim.org/mailman/listinfo/exim-users
## Exim details at http://www.exim.org/
## Please use the Wiki with this list - http://www.exim.org/eximwiki/


mh+exim-users at zugschlus

Dec 1, 2005, 8:30 AM

Post #5 of 29 (2356 views)
Permalink
Re: Potential logic error in retry handling for IPv4+IPv6 hosts [In reply to]

On Thu, 01 Dec 2005 10:16:00 -0500, Marc Sherman
<msherman[at]projectile.ca> wrote:
>Florian Weimer wrote:
>> Does this problem sound familiar?
>
>Very familiar. It happened to me in June[0], and we were just
>discussing it again earlier this week[1].

>[1]
>http://www.exim.org/mail-archives/exim-users/Week-of-Mon-20051128/msg00073.html

Actually, I started that thread because Florian asked me.

Greetings
Marc

--
-------------------------------------- !! No courtesy copies, please !! -----
Marc Haber | " Questions are the | Mailadresse im Header
Mannheim, Germany | Beginning of Wisdom " | http://www.zugschlus.de/
Nordisch by Nature | Lt. Worf, TNG "Rightful Heir" | Fon: *49 621 72739834

--
## List details at http://www.exim.org/mailman/listinfo/exim-users
## Exim details at http://www.exim.org/
## Please use the Wiki with this list - http://www.exim.org/eximwiki/


msherman at projectile

Dec 1, 2005, 8:42 AM

Post #6 of 29 (2373 views)
Permalink
Re: Potential logic error in retry handling for IPv4+IPv6 hosts [In reply to]

Marc Haber wrote:
>
> Actually, I started that thread because Florian asked me.

Ah, slightly less of a coincidence than I thought. Still, it's
interesting that we both had the same retry bug; at least we now know it
wasn't something specific to my machine. Though it could well be
something specific to Debian (which I'm running, too).

- Marc

--
## List details at http://www.exim.org/mailman/listinfo/exim-users
## Exim details at http://www.exim.org/
## Please use the Wiki with this list - http://www.exim.org/eximwiki/


ph10 at cus

Dec 1, 2005, 9:02 AM

Post #7 of 29 (2354 views)
Permalink
Re: Potential logic error in retry handling for IPv4+IPv6 hosts [In reply to]

On Thu, 1 Dec 2005, Marc Sherman wrote:

> Ah, slightly less of a coincidence than I thought. Still, it's
> interesting that we both had the same retry bug; at least we now know it
> wasn't something specific to my machine. Though it could well be
> something specific to Debian (which I'm running, too).

If someone can produce debug information for this problem, it would be
very helpful. I am not so good at debugging just by eyeballing code,
though I will try to take a look next week some time.

--
Philip Hazel University of Cambridge Computing Service,
ph10[at]cus.cam.ac.uk Cambridge, England. Phone: +44 1223 334714.
Get the Exim 4 book: http://www.uit.co.uk/exim-book

--
## List details at http://www.exim.org/mailman/listinfo/exim-users
## Exim details at http://www.exim.org/
## Please use the Wiki with this list - http://www.exim.org/eximwiki/


adam00f at ducksburg

Dec 1, 2005, 11:24 AM

Post #8 of 29 (2365 views)
Permalink
Re: Potential logic error in retry handling for IPv4+IPv6 hosts [In reply to]

In the "Re: [exim] Exim4 & IPV4" thread you (Marc Sherman) said [1] I
should have both
dns_ipv4_lookup = *
(which I recently added to the main config) and
ignore_target_hosts = ... : ::::/0
(which I already had in the routers) for an IPv4-only setup. Later in the
thread you said [2] "...three config settings required to disable ipv6
support (if you count local_interfaces)..."

The spec (Ch.13) says "If your host has only one interface, and you want
all its IP addresses to be treated in the same way, and you are using
only the standard SMTP port, you should not need to take any special
action."

So am I right in continuing not to use this setting, although the default
includes IPv6? Or would I gain anything from setting it specifically to
the machine's lo and eth1 IPv4 addresses?

[1]
http://www.exim.org/mail-archives/exim-users/Week-of-Mon-20051128/msg00103.html
[2]
http://www.exim.org/mail-archives/exim-users/Week-of-Mon-20051128/msg00089.html

Thanks,
ADam

--
## List details at http://www.exim.org/mailman/listinfo/exim-users
## Exim details at http://www.exim.org/
## Please use the Wiki with this list - http://www.exim.org/eximwiki/


fw at deneb

Dec 1, 2005, 1:42 PM

Post #9 of 29 (2359 views)
Permalink
Re: Potential logic error in retry handling for IPv4+IPv6 hosts [In reply to]

* Philip Hazel:

> If someone can produce debug information for this problem, it would be
> very helpful. I am not so good at debugging just by eyeballing code,
> though I will try to take a look next week some time.

Here's a partially redacted log of a delivery attempt. As far as I
can tell, no delivery attempt for the fw[at]deneb.enyo.de recipient over
IPv4 is attempted.

2005-11-30 02:01:31 1EhMuA-0003QO-Se <= owner[at]packages.qa.debian.org U=qa P=local S=7043 id=E1EhMte-00076g-38[at]costa.debian.org
2005-11-30 02:01:32 1EhMuA-0003QO-Se mail.EXAMPLE.COM [192.0.2.25]: Connection refused
2005-11-30 02:01:32 1EhMuA-0003QO-Se == USER[at]EXAMPLE.COM R=dnslookup T=remote_smtp defer (111): Connection refused
2005-11-30 02:01:33 1EhMuA-0003QO-Se => USER[at]gmail.com R=dnslookup T=remote_smtp H=gmail-smtp-in.l.google.com [64.233.185.27]
[...]
2005-11-30 02:01:37 1EhMuA-0003QO-Se mail.enyo.de [2001:14b0:202:1::a7]: No route to host
2005-11-30 02:01:37 1EhMuA-0003QO-Se == fw[at]deneb.enyo.de R=dnslookup T=remote_smtp defer (113): No route to host
2005-11-30 02:01:37 1EhMuA-0003QO-Se ** fw[at]deneb.enyo.de: retry timeout exceeded
2005-11-30 02:01:37 1EhMuA-0003QO-Se ** USER[at]EXAMPLE.COM: retry timeout exceeded
2005-11-30 02:01:37 1EhMuH-0003SO-ME <= <> R=1EhMuA-0003QO-Se U=Debian-exim P=local S=8007
2005-11-30 02:01:37 1EhMuA-0003QO-Se Completed

(mail.EXAMPLE.COM hasn't got an AAAA record.)

The problem is probably not related to multiple recipients; I've seen
instances where only a single recipient has been given.

Unfortunately, I haven't found a way to trigger the problem. I only
know it occurs from time to time. 8-(

--
## List details at http://www.exim.org/mailman/listinfo/exim-users
## Exim details at http://www.exim.org/
## Please use the Wiki with this list - http://www.exim.org/eximwiki/


ph10 at cus

Dec 2, 2005, 1:20 AM

Post #10 of 29 (2354 views)
Permalink
Re: Potential logic error in retry handling for IPv4+IPv6 hosts [In reply to]

On Thu, 1 Dec 2005, Florian Weimer wrote:

> Here's a partially redacted log of a delivery attempt. As far as I
> can tell, no delivery attempt for the fw[at]deneb.enyo.de recipient over
> IPv4 is attempted.

Thanks. That's a start, anyway. It may be that deneb.enyo.de's retry
time hasn't come.

> Unfortunately, I haven't found a way to trigger the problem. I only
> know it occurs from time to time. 8-(

Now that I have a pattern, I can try to build an environment that
simulates it, and try various things.

--
Philip Hazel University of Cambridge Computing Service,
ph10[at]cus.cam.ac.uk Cambridge, England. Phone: +44 1223 334714.
Get the Exim 4 book: http://www.uit.co.uk/exim-book

--
## List details at http://www.exim.org/mailman/listinfo/exim-users
## Exim details at http://www.exim.org/
## Please use the Wiki with this list - http://www.exim.org/eximwiki/


ph10 at cus

Dec 5, 2005, 6:27 AM

Post #11 of 29 (2364 views)
Permalink
Re: Potential logic error in retry handling for IPv4+IPv6 hosts [In reply to]

On Thu, 1 Dec 2005, Florian Weimer wrote:

> Here's a partially redacted log of a delivery attempt. As far as I
> can tell, no delivery attempt for the fw[at]deneb.enyo.de recipient over
> IPv4 is attempted.

Well, I've spent some time trying to set up a test situation like this,
but needless to say, I cannot reproduce the effect.

> Unfortunately, I haven't found a way to trigger the problem. I only
> know it occurs from time to time. 8-(

I think we are stuck until there is more evidence.

Philip

--
Philip Hazel University of Cambridge Computing Service,
ph10[at]cus.cam.ac.uk Cambridge, England. Phone: +44 1223 334714.

--
## List details at http://www.exim.org/mailman/listinfo/exim-users
## Exim details at http://www.exim.org/
## Please use the Wiki with this list - http://www.exim.org/eximwiki/


ph10 at cus

Dec 6, 2005, 2:33 AM

Post #12 of 29 (2358 views)
Permalink
Re: Potential logic error in retry handling for IPv4+IPv6 hosts [In reply to]

On Thu, 1 Dec 2005, Marc Sherman wrote:

> > I'm currently facing a mail delivery problem. The sending MTA is Exim
> > 4.50. The host has no real IPv6 connectivity, but a working IPv6
> > stack, and the DNS resolver returns AAAA records.

OK, I got the message about disabling IPv6 at run time. The patch below,
which I think is still small enough for the list, adds the following
feature:

PH/01 There is a new global option called disable_ipv6, which does exactly what
its name implies. If set true, even if the Exim binary has IPv6 support,
no IPv6 activities take place. AAAA records are never looked up for
host names given in manual routing data or elsewhere. AAAA records that
are received from the DNS as additional data for MX records are ignored.
Any IPv6 addresses that are listed in local_interfaces, manualroute route
data, etc. are also ignored. If IP literals are enabled, the ipliteral
router declines to handle IPv6 literal addresses.

Please test this patch to see if I have got it right. The patch also
fixes a bug in the ipliteral router, which caused it not to recognize
literal IPv6 domains prefixed by "ipv6:".

Philip

--
Philip Hazel University of Cambridge Computing Service,
ph10[at]cus.cam.ac.uk Cambridge, England. Phone: +44 1223 334714.



*** exim-4.60/src/globals.h Mon Nov 28 10:57:32 2005
--- globals.h Mon Dec 5 14:55:51 2005
***************
*** 271,276 ****
--- 271,277 ----
extern int demime_ok; /* Nonzero if message has been demimed */
extern uschar *demime_reason; /* Reason for broken MIME container */
#endif
+ extern BOOL disable_ipv6; /* Don't do any IPv6 things */
extern BOOL disable_logging; /* Disables log writing when TRUE */

#ifdef EXPERIMENTAL_DOMAINKEYS
*** exim-4.60/src/globals.c Mon Nov 28 10:57:32 2005
--- globals.c Mon Dec 5 14:55:52 2005
***************
*** 482,487 ****
--- 482,488 ----
int demime_ok = 0;
uschar *demime_reason = NULL;
#endif
+ BOOL disable_ipv6 = FALSE;
BOOL disable_logging = FALSE;

#ifdef EXPERIMENTAL_DOMAINKEYS
*** exim-4.60/src/readconf.c Mon Nov 28 10:57:32 2005
--- readconf.c Mon Dec 5 15:01:12 2005
***************
*** 194,199 ****
--- 194,200 ----
{ "deliver_drop_privilege", opt_bool, &deliver_drop_privilege },
{ "deliver_queue_load_max", opt_fixed, &deliver_queue_load_max },
{ "delivery_date_remove", opt_bool, &delivery_date_remove },
+ { "disable_ipv6", opt_bool, &disable_ipv6 },
{ "dns_again_means_nonexist", opt_stringptr, &dns_again_means_nonexist },
{ "dns_check_names_pattern", opt_stringptr, &check_dns_names_pattern },
{ "dns_csa_search_limit", opt_int, &dns_csa_search_limit },
***************
*** 2832,2840 ****
struct hostent *hostdata;

#if HAVE_IPV6
! if (dns_ipv4_lookup == NULL ||
match_isinlist(hostname, &dns_ipv4_lookup, 0, NULL, NULL, MCL_DOMAIN,
! TRUE, NULL) != OK)
af = AF_INET6;
#else
af = AF_INET;
--- 2833,2841 ----
struct hostent *hostdata;

#if HAVE_IPV6
! if (!disable_ipv6 && (dns_ipv4_lookup == NULL ||
match_isinlist(hostname, &dns_ipv4_lookup, 0, NULL, NULL, MCL_DOMAIN,
! TRUE, NULL) != OK))
af = AF_INET6;
#else
af = AF_INET;
*** exim-4.60/src/host.c Mon Nov 28 10:57:32 2005
--- host.c Tue Dec 6 09:48:32 2005
***************
*** 774,783 ****

while ((s = string_nextinlist(&list, &sep, buffer, sizeof(buffer))) != NULL)
{
int port = host_address_extract_port(s); /* Leaves just the IP address */
! if (string_is_ip_address(s, NULL) == 0)
log_write(0, LOG_MAIN|LOG_PANIC_DIE, "Malformed IP address \"%s\" in %s",
s, name);

/* This use of strcpy() is OK because we have checked that s is a valid IP
address above. The field in the ip_address_item is large enough to hold an
--- 774,788 ----

while ((s = string_nextinlist(&list, &sep, buffer, sizeof(buffer))) != NULL)
{
+ int ipv;
int port = host_address_extract_port(s); /* Leaves just the IP address */
! if ((ipv = string_is_ip_address(s, NULL)) == 0)
log_write(0, LOG_MAIN|LOG_PANIC_DIE, "Malformed IP address \"%s\" in %s",
s, name);
+
+ /* Skip IPv6 addresses if IPv6 is disabled. */
+
+ if (disable_ipv6 && ipv == 6) continue;

/* This use of strcpy() is OK because we have checked that s is a valid IP
address above. The field in the ip_address_item is large enough to hold an
***************
*** 1965,1981 ****
return HOST_FIND_AGAIN;
}

! /* In an IPv6 world, we need to scan for both kinds of address, so go round the
! loop twice. Note that we have ensured that AF_INET6 is defined even in an IPv4
! world, which makes for slightly tidier code. However, if dns_ipv4_lookup
! matches the domain, we also just do IPv4 lookups here (except when testing
! standalone). */

#if HAVE_IPV6
#ifndef STAND_ALONE
! if (dns_ipv4_lookup != NULL &&
match_isinlist(host->name, &dns_ipv4_lookup, 0, NULL, NULL, MCL_DOMAIN,
! TRUE, NULL) == OK)
{ af = AF_INET; times = 1; }
else
#endif /* STAND_ALONE */
--- 1970,1986 ----
return HOST_FIND_AGAIN;
}

! /* In an IPv6 world, unless IPv6 has been disabled, we need to scan for both
! kinds of address, so go round the loop twice. Note that we have ensured that
! AF_INET6 is defined even in an IPv4 world, which makes for slightly tidier
! code. However, if dns_ipv4_lookup matches the domain, we also just do IPv4
! lookups here (except when testing standalone). */

#if HAVE_IPV6
#ifndef STAND_ALONE
! if (disable_ipv6 || (dns_ipv4_lookup != NULL &&
match_isinlist(host->name, &dns_ipv4_lookup, 0, NULL, NULL, MCL_DOMAIN,
! TRUE, NULL) == OK))
{ af = AF_INET; times = 1; }
else
#endif /* STAND_ALONE */
***************
*** 2249,2266 ****
return HOST_FOUND;
}

! /* On an IPv6 system, go round the loop up to three times, looking for A6 and
! AAAA records the first two times. However, unless doing standalone testing, we
! force an IPv4 lookup if the domain matches dns_ipv4_lookup is set. Since A6
! records look like being abandoned, support them only if explicitly configured
! to do so. On an IPv4 system, go round the loop once only, looking only for A
! records. */

#if HAVE_IPV6
#ifndef STAND_ALONE
! if (dns_ipv4_lookup != NULL &&
match_isinlist(host->name, &dns_ipv4_lookup, 0, NULL, NULL, MCL_DOMAIN,
! TRUE, NULL) == OK)
i = 0; /* look up A records only */
else
#endif /* STAND_ALONE */
--- 2254,2271 ----
return HOST_FOUND;
}

! /* On an IPv6 system, unless IPv6 is disabled, go round the loop up to three
! times, looking for A6 and AAAA records the first two times. However, unless
! doing standalone testing, we force an IPv4 lookup if the domain matches
! dns_ipv4_lookup is set. Since A6 records look like being abandoned, support
! them only if explicitly configured to do so. On an IPv4 system, go round the
! loop once only, looking only for A records. */

#if HAVE_IPV6
#ifndef STAND_ALONE
! if (disable_ipv6 || (dns_ipv4_lookup != NULL &&
match_isinlist(host->name, &dns_ipv4_lookup, 0, NULL, NULL, MCL_DOMAIN,
! TRUE, NULL) == OK))
i = 0; /* look up A records only */
else
#endif /* STAND_ALONE */
***************
*** 2893,2902 ****

if (rr->type != T_A
#if HAVE_IPV6
! && rr->type != T_AAAA
! #ifdef SUPPORT_A6
! && rr->type != T_A6
! #endif
#endif
) continue;

--- 2898,2911 ----

if (rr->type != T_A
#if HAVE_IPV6
! && ( disable_ipv6 ||
! (
! rr->type != T_AAAA
! #ifdef SUPPORT_A6
! && rr->type != T_A6
! #endif
! )
! )
#endif
) continue;

*** exim-4.60/src/routers/ipliteral.c Mon Nov 28 10:57:32 2005
--- routers/ipliteral.c Tue Dec 6 09:45:10 2005
***************
*** 102,109 ****
*/
host_item *h;
uschar *domain = addr->domain;
int len = Ustrlen(domain);
! int rc;

addr_new = addr_new; /* Keep picky compilers happy */
addr_succeed = addr_succeed;
--- 102,110 ----
*/
host_item *h;
uschar *domain = addr->domain;
+ uschar *ip;
int len = Ustrlen(domain);
! int rc, ipv;

addr_new = addr_new; /* Keep picky compilers happy */
addr_succeed = addr_succeed;
***************
*** 111,124 ****
DEBUG(D_route) debug_printf("%s router called for %s: domain = %s\n",
rblock->name, addr->address, addr->domain);

! /* Check that the domain is an IP address enclosed in square brackets. If
! not, the router declines. Otherwise route to the single IP address, setting the
! host name to "(unnamed)". */

if (domain[0] != '[' || domain[len-1] != ']') return DECLINE;
domain[len-1] = 0; /* temporarily */

! if (string_is_ip_address(domain+1, NULL) == 0)
{
domain[len-1] = ']';
return DECLINE;
--- 112,131 ----
DEBUG(D_route) debug_printf("%s router called for %s: domain = %s\n",
rblock->name, addr->address, addr->domain);

! /* Check that the domain is an IP address enclosed in square brackets. Remember
! to allow for the "official" form of IPv6 addresses. If not, the router
! declines. Otherwise route to the single IP address, setting the host name to
! "(unnamed)". */

if (domain[0] != '[' || domain[len-1] != ']') return DECLINE;
domain[len-1] = 0; /* temporarily */

! ip = domain + 1;
! if (strncmpic(ip, US"IPV6:", 5) == 0 || strncmpic(ip, US"IPV4:", 5) == 0)
! ip += 5;
!
! ipv = string_is_ip_address(ip, NULL);
! if (ipv == 0 || (disable_ipv6 && ipv == 6))
{
domain[len-1] = ']';
return DECLINE;
***************
*** 128,137 ****
but if it is set, it should probably work. */

if (verify_check_this_host(&(rblock->ignore_target_hosts), NULL, domain,
! domain + 1, NULL) == OK)
{
DEBUG(D_route)
! debug_printf("%s is in ignore_target_hosts\n", domain+1);
addr->message = US"IP literal host explicitly ignored";
domain[len-1] = ']';
return DECLINE;
--- 135,144 ----
but if it is set, it should probably work. */

if (verify_check_this_host(&(rblock->ignore_target_hosts), NULL, domain,
! ip, NULL) == OK)
{
DEBUG(D_route)
! debug_printf("%s is in ignore_target_hosts\n", ip);
addr->message = US"IP literal host explicitly ignored";
domain[len-1] = ']';
return DECLINE;
***************
*** 142,148 ****
h = store_get(sizeof(host_item));

h->next = NULL;
! h->address = string_copy(domain+1);
h->port = PORT_NONE;
domain[len-1] = ']'; /* restore */
h->name = string_copy(domain);
--- 149,155 ----
h = store_get(sizeof(host_item));

h->next = NULL;
! h->address = string_copy(ip);
h->port = PORT_NONE;
domain[len-1] = ']'; /* restore */
h->name = string_copy(domain);

--
## List details at http://www.exim.org/mailman/listinfo/exim-users
## Exim details at http://www.exim.org/
## Please use the Wiki with this list - http://www.exim.org/eximwiki/


mh+exim-users at zugschlus

Dec 17, 2005, 6:27 AM

Post #13 of 29 (2338 views)
Permalink
Re: Potential logic error in retry handling for IPv4+IPv6 hosts [In reply to]

On Mon, 5 Dec 2005 14:27:43 +0000 (GMT), Philip Hazel
<ph10[at]cus.cam.ac.uk> wrote:
>I think we are stuck until there is more evidence.

This is actually an issue with how exim handles DNS answers. Just
imagine that the A record for a target host name expires in the
resolver's cache some time earlier than the AAAA record. When exim now
queries for the MX record, the resolver returns the data which it
still has cached, which is the AAAA record, in the additional section.

Exim will believe the information from the additional section, and try
delivering there.

Here is a script which can be used to reproduce the issue. I believe
this is independent of whether the host actually has ipv6
connectivity. The script should be run only once at a time against the
same resolving DNS server. The domain brokenv6.zugschlus.de and the
host name mailgate.brokenv6.zugschlus.de have been especially
configured for this demonstration with a TTL of 120 seconds, and
nobody[at]brokenv6.zugschlus.de is available for tests - messages to that
address are accepted and blackholed.

#!/bin/bash

withecho() {
echo $@
$@
}

echo have the prepared DNS entries expire from the cache TTL 120
withecho sleep 180

echo pull A record into cache
withecho dig mailgate.brokenv6.zugschlus.de A > /dev/null

echo have records expiration time deviate
withecho sleep 60

echo output 1, should show A and AAAA in ADDITIONAL SECTION
withecho dig brokenv6.zugschlus.de MX

echo exim will deliver message to v4 and v6
withecho exim -bt nobody[at]brokenv6.zugschlus.de

echo have A record expire
withecho sleep 65

echo output 2, should show only AAAA record in ADDITIONAL SECTION
withecho dig brokenv6.zugschlus.de MX

echo exim will now only try delivery to v6
withecho exim -bt nobody[at]brokenv6.zugschlus.de

Here is the script output, edited to the relevant parts:

|[19/516]mh[at]ivanova:~/enyo$ ./reproduce
|have the prepared DNS entries expire from the cache TTL 120
|sleep 180
|pull A record into cache
|have records expiration time deviate
|sleep 60
|output 1, should show A and AAAA in ADDITIONAL SECTION
|dig brokenv6.zugschlus.de MX
|
|; <<>> DiG 9.3.1 <<>> brokenv6.zugschlus.de MX
|;; global options: printcmd
|;; Got answer:
|;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 46049
|;; flags: qr rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 5, ADDITIONAL: 4
|
|;; ANSWER SECTION:
|brokenv6.zugschlus.de. 120 IN MX 10 mailgate.brokenv6.zugschlus.de.
|
|;; ADDITIONAL SECTION:
|mailgate.brokenv6.zugschlus.de. 59 IN A 217.151.83.1
|mailgate.brokenv6.zugschlus.de. 120 IN AAAA 2001:14b0:202:f::1:19
|
|exim will deliver message to v4 and v6
|exim -bt nobody[at]brokenv6.zugschlus.de
|R: dnslookup for nobody[at]brokenv6.zugschlus.de
|nobody[at]brokenv6.zugschlus.de
| router = dnslookup, transport = remote_smtp
| host mailgate.brokenv6.zugschlus.de [2001:14b0:202:f::1:19] MX=10
| host mailgate.brokenv6.zugschlus.de [217.151.83.1] MX=10
|have A record expire
|sleep 65
|output 2, should show only AAAA record in ADDITIONAL SECTION
|dig brokenv6.zugschlus.de MX
|
|; <<>> DiG 9.3.1 <<>> brokenv6.zugschlus.de MX
|;; global options: printcmd
|;; Got answer:
|;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 64098
|;; flags: qr rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 5, ADDITIONAL: 3
|
|;; ANSWER SECTION:
|brokenv6.zugschlus.de. 55 IN MX 10 mailgate.brokenv6.zugschlus.de.
|
|;; ADDITIONAL SECTION:
|mailgate.brokenv6.zugschlus.de. 55 IN AAAA 2001:14b0:202:f::1:19
|
|;; Query time: 1 msec
|;; SERVER: 81.169.148.34#53(81.169.148.34)
|;; WHEN: Sat Dec 17 15:20:17 2005
|;; MSG SIZE rcvd: 238
|
|exim will now only try delivery to v6
|exim -bt nobody[at]brokenv6.zugschlus.de
|R: dnslookup for nobody[at]brokenv6.zugschlus.de
|nobody[at]brokenv6.zugschlus.de
| router = dnslookup, transport = remote_smtp
| host mailgate.brokenv6.zugschlus.de [2001:14b0:202:f::1:19] MX=10
|[20/516]mh[at]ivanova:~/enyo$

If the v6 host is never reachable, as it is for a host that doesn't
have ipv6 connectivity, this leads to messages being flagged as
undeliverable.

Thanks to Florian for discussion on IRC which led to this explanation
of things happening.

Greetings
Marc

--
-------------------------------------- !! No courtesy copies, please !! -----
Marc Haber | " Questions are the | Mailadresse im Header
Mannheim, Germany | Beginning of Wisdom " | http://www.zugschlus.de/
Nordisch by Nature | Lt. Worf, TNG "Rightful Heir" | Fon: *49 621 72739834

--
## List details at http://www.exim.org/mailman/listinfo/exim-users
## Exim details at http://www.exim.org/
## Please use the Wiki with this list - http://www.exim.org/eximwiki/


fw at deneb

Dec 17, 2005, 7:37 AM

Post #14 of 29 (2321 views)
Permalink
Re: Potential logic error in retry handling for IPv4+IPv6 hosts [In reply to]

* Marc Haber:

> Here is a script which can be used to reproduce the issue. I believe
> this is independent of whether the host actually has ipv6
> connectivity.

If there's v6 connectivity, delivery over the AAAA record will work,
so this bug is not really visible.

(Just to clarify, the bug is not related to retry handling at all. It
was just a red herring, I'm afraid.)

--
## List details at http://www.exim.org/mailman/listinfo/exim-users
## Exim details at http://www.exim.org/
## Please use the Wiki with this list - http://www.exim.org/eximwiki/


msherman at projectile

Dec 17, 2005, 7:56 AM

Post #15 of 29 (2334 views)
Permalink
Re: Potential logic error in retry handling for IPv4+IPv6 hosts [In reply to]

Marc Haber wrote:
> On Mon, 5 Dec 2005 14:27:43 +0000 (GMT), Philip Hazel
> <ph10[at]cus.cam.ac.uk> wrote:
>> I think we are stuck until there is more evidence.
>
> This is actually an issue with how exim handles DNS answers. Just
> imagine that the A record for a target host name expires in the
> resolver's cache some time earlier than the AAAA record. When exim now
> queries for the MX record, the resolver returns the data which it
> still has cached, which is the AAAA record, in the additional section.
>
> Exim will believe the information from the additional section, and try
> delivering there.

Great debugging, Marc. So in my case earlier this year, I probably
actually caused the retry failure by setting "dns_ipv4_lookup = *"; it
caused host verification to only query for A records, so when I received
mail from the exim lists, it would populate the cache with just the A
records, but when I sent mail, both A and AAAA records would show up,
causing the A and AAAA records to expire at different times.

What's the fix for this? Have exim always explicitly query DNS for A if
the additional section only returns AAAA (or vice versa)? Or should the
additional section not be trusted at all?

- Marc

--
## List details at http://www.exim.org/mailman/listinfo/exim-users
## Exim details at http://www.exim.org/
## Please use the Wiki with this list - http://www.exim.org/eximwiki/


mh+exim-users at zugschlus

Dec 17, 2005, 8:20 AM

Post #16 of 29 (2327 views)
Permalink
Re: Potential logic error in retry handling for IPv4+IPv6 hosts [In reply to]

On Sat, 17 Dec 2005 10:56:48 -0500, Marc Sherman
<msherman[at]projectile.ca> wrote:
>What's the fix for this? Have exim always explicitly query DNS for A if
>the additional section only returns AAAA (or vice versa)? Or should the
>additional section not be trusted at all?

I think that if none of the hosts listed in the additional section is
reachable, exim should explicitly query for AAAA and A to make sure to
catch even the hosts that are not listed in the additional section.

This is, btw, not an ipv6 issue exclusively, it might happen in
ipv4-only setups as well. See Debian Bug #342619 for another example.

Greetings
Marc

--
-------------------------------------- !! No courtesy copies, please !! -----
Marc Haber | " Questions are the | Mailadresse im Header
Mannheim, Germany | Beginning of Wisdom " | http://www.zugschlus.de/
Nordisch by Nature | Lt. Worf, TNG "Rightful Heir" | Fon: *49 621 72739834

--
## List details at http://www.exim.org/mailman/listinfo/exim-users
## Exim details at http://www.exim.org/
## Please use the Wiki with this list - http://www.exim.org/eximwiki/


fw at deneb

Dec 17, 2005, 8:21 AM

Post #17 of 29 (2325 views)
Permalink
Re: Potential logic error in retry handling for IPv4+IPv6 hosts [In reply to]

* Marc Sherman:

> What's the fix for this? Have exim always explicitly query DNS for A if
> the additional section only returns AAAA (or vice versa)?

This is one approach.

> Or should the additional section not be trusted at all?

This is probably the safest approach, and it's also necessary to
comply with the letter of the SMTP RFC. The resolver can drop as many
data as it wishes from the additional section.

It would increase the number of DNS queries by quite a bit, though.

--
## List details at http://www.exim.org/mailman/listinfo/exim-users
## Exim details at http://www.exim.org/
## Please use the Wiki with this list - http://www.exim.org/eximwiki/


fw at deneb

Dec 17, 2005, 8:32 AM

Post #18 of 29 (2322 views)
Permalink
Re: Potential logic error in retry handling for IPv4+IPv6 hosts [In reply to]

* Marc Haber:

> This is, btw, not an ipv6 issue exclusively, it might happen in
> ipv4-only setups as well. See Debian Bug #342619 for another example.

I'm not sure if it's the same bug, and I wouldn't be surprised if the
behavior was deliberate in that case (after all, the whole "long
failure period" business is there to generate immediate bounces, so
that users won't have to wait for five days until they are told about
their mistake).

In the example in the bug report, we there are two A RRs:

mailrelay.direct-adsl.nl. 86400 IN A 195.121.6.12
mailrelay.direct-adsl.nl. 86400 IN A 195.121.6.56

But resolvers MUST cache the whole set of records and expire them at
the same time. If the resolver fails to do this properly and provides
a wrong view on DNS, there is no workaround on Exim's side.

--
## List details at http://www.exim.org/mailman/listinfo/exim-users
## Exim details at http://www.exim.org/
## Please use the Wiki with this list - http://www.exim.org/eximwiki/


jeroen at wolffelaar

Dec 18, 2005, 6:33 PM

Post #19 of 29 (2340 views)
Permalink
Re: Potential logic error in retry handling for IPv4+IPv6 hosts [In reply to]

(Please cc me on all replies, I'm not subscribed)

On Sat, Dec 17, 2005 at 05:32:15PM +0100, Florian Weimer wrote:
> * Marc Haber:
>
> > This is, btw, not an ipv6 issue exclusively, it might happen in
> > ipv4-only setups as well. See Debian Bug #342619 for another example.
>
> I'm not sure if it's the same bug, and I wouldn't be surprised if the
> behavior was deliberate in that case (after all, the whole "long
> failure period" business is there to generate immediate bounces, so
> that users won't have to wait for five days until they are told about
> their mistake).
>
> In the example in the bug report, we there are two A RRs:
>
> mailrelay.direct-adsl.nl. 86400 IN A 195.121.6.12
> mailrelay.direct-adsl.nl. 86400 IN A 195.121.6.56
>
> But resolvers MUST cache the whole set of records and expire them at
> the same time. If the resolver fails to do this properly and provides
> a wrong view on DNS, there is no workaround on Exim's side.

The DNS setup changed in the meanwhile, because of, as I now know, a wrong
guess at the cause of the failure. The old setup had MX's from multiple
different zone's, and the one MX that had a long failure was also served
the DNS server of the mail server, but the secundary MX's to which the mail
should have been delivered to, was only remotely DNS-served -- so the DNS
server in question at times only had the broken MX cached (well, was
authoritive for it, even), and only that was in the additional section --
the IP addresses of the working MX's were out of the cache.

The only solution seems to me to actively query for all A (and AAAA)
records of all MX's before determining that no MX's are available for
delivery -- the additional DNS section is not to be trusted to ever give an
exhaustive list of IP addresses to try. I'd even say that it needs to
happen at every delivery attempt after delivery is found to be unsuccesful
to all the MX's in the additional section, because it can happen that some
MX's are more often in it than others. And you don't want delivery to fail
just because at the ultimate attempt all MX's happen to be down -- but some
were up in the past 4 days.

Note that RFC 974, MAIL ROUTING AND THE DOMAIN SYSTEM from 1986 (predating IPv6
by 12 years) explicitely warns against wrong handling of the DNS additional
section for MX queries:

| The incomplete data problem also requires some care when handling
| domain queries. If the answer section of a query is incomplete
| critical MX RRs may be left out. This may result in mail looping, or
| in a message being mistakenly labelled undeliverable. As a result,
| mailers may only accept responses from the domain system which have
| complete answer sections. Note that this entire problem can be
| avoided by only using virtual circuits for queries, but since this
| situation is likely to be very rare and datagrams are the preferred
| way to interact with the domain system, implementors should probably
| just ensure that their mailer will repeat a query with virtual
| circuits should the truncation bit ever be set.

Even though the RFC only mentions problems regarding to DNS datagram
truncation, and does not mention the issue that is more relevant here,
incomplete answer due to unbalanced caching -- a problem which only can
happen with MX's out of multiple zone's or with IPv6 (or explicitely
unbalanced TTL's in one zone, but I consider *that* very unlikely).

--Jeroen

--
Jeroen van Wolffelaar
jeroen[at]wolffelaar.nl
http://jeroen.A-Eskwadraat.nl

--
## List details at http://www.exim.org/mailman/listinfo/exim-users
## Exim details at http://www.exim.org/
## Please use the Wiki with this list - http://www.exim.org/eximwiki/


jeroen at wolffelaar

Dec 18, 2005, 6:37 PM

Post #20 of 29 (2317 views)
Permalink
Re: Potential logic error in retry handling for IPv4+IPv6 hosts [In reply to]

On Sat, Dec 17, 2005 at 05:21:20PM +0100, Florian Weimer wrote:
> * Marc Sherman:
> > Or should the additional section not be trusted at all?
>
> This is probably the safest approach, and it's also necessary to
> comply with the letter of the SMTP RFC. The resolver can drop as many
> data as it wishes from the additional section.
>
> It would increase the number of DNS queries by quite a bit, though.

Not so much if first all of the additional section is tried, and extra
queries only been done of all of those fail -- a more rare situation I'd
say. There are more queries though when a mail is failing for a while, but
in that case, it's every time for the same domain, so the local DNS server
has the relevant answers cached.

--Jeroen

--
Jeroen van Wolffelaar
jeroen[at]wolffelaar.nl
http://jeroen.A-Eskwadraat.nl

--
## List details at http://www.exim.org/mailman/listinfo/exim-users
## Exim details at http://www.exim.org/
## Please use the Wiki with this list - http://www.exim.org/eximwiki/


fw at deneb

Dec 19, 2005, 1:43 AM

Post #21 of 29 (2332 views)
Permalink
Re: Potential logic error in retry handling for IPv4+IPv6 hosts [In reply to]

* Jeroen van Wolffelaar:

> On Sat, Dec 17, 2005 at 05:21:20PM +0100, Florian Weimer wrote:
>> * Marc Sherman:
>> > Or should the additional section not be trusted at all?
>>
>> This is probably the safest approach, and it's also necessary to
>> comply with the letter of the SMTP RFC. The resolver can drop as many
>> data as it wishes from the additional section.
>>
>> It would increase the number of DNS queries by quite a bit, though.
>
> Not so much if first all of the additional section is tried, and extra
> queries only been done of all of those fail

This would require a significant change in how Exim handles routing
(part of the routing would run after the a delivery attempt has been
performed). I doubt it's worth the complexity.

--
## List details at http://www.exim.org/mailman/listinfo/exim-users
## Exim details at http://www.exim.org/
## Please use the Wiki with this list - http://www.exim.org/eximwiki/


ph10 at cus

Dec 19, 2005, 2:20 AM

Post #22 of 29 (2339 views)
Permalink
Re: Potential logic error in retry handling for IPv4+IPv6 hosts [In reply to]

On Sat, 17 Dec 2005, Marc Haber wrote:

> This is actually an issue with how exim handles DNS answers. Just
> imagine that the A record for a target host name expires in the
> resolver's cache some time earlier than the AAAA record. When exim now
> queries for the MX record, the resolver returns the data which it
> still has cached, which is the AAAA record, in the additional section.
>
> Exim will believe the information from the additional section, and try
> delivering there.

Aarrgghh!! Yes, that does explain it. Thank you for tracking this one
down.

On Sat, 17 Dec 2005, Marc Sherman wrote:

> What's the fix for this? Have exim always explicitly query DNS for A if
> the additional section only returns AAAA (or vice versa)? Or should the
> additional section not be trusted at all?

I'm beginning to think that the additional section should not be trusted
at all.

On Sat, 17 Dec 2005, Marc Haber wrote:

> I think that if none of the hosts listed in the additional section is
> reachable, exim should explicitly query for AAAA and A to make sure to
> catch even the hosts that are not listed in the additional section.

Given the current design of Exim, this is not possible. It does the DNS
queries at routing time, and doesn't know whether the hosts are
reachable until transport time.

(1) A possibility, I suppose, would be to believe the additional section
for the very first time a message is tried, and to do the additional DNS
lookups for any subsequent delivery attempts. No, even that would be bad
if the additional section hosts were long-time dead, because Exim would
then bounce the message.

(2) Try again: Exim could see if any of the additional section hosts
have a retry record in Exim's database, and if so, try the full DNS
lookup. But that would put extra load on the hints databases. It might
not be any better than just doing the full A/AAAA lookup.

(3) I suppose a modified (1) would be never to bounce a message for "all
hosts timed out" on the first delivery attempt. Then always do the full
lookup on the subsequent attempts. That's the best I can come up with
for now.

> This is, btw, not an ipv6 issue exclusively, it might happen in
> ipv4-only setups as well.

Indeed.

On Sat, 17 Dec 2005, Florian Weimer wrote:

> It would increase the number of DNS queries by quite a bit, though.

Yes. Do we have any idea if this would be a serious problem?

On Mon, 19 Dec 2005, Jeroen van Wolffelaar wrote:

> Not so much if first all of the additional section is tried, and extra
> queries only been done of all of those fail -- a more rare situation I'd
> say.

That's true, but Exim's design does not lend itself to implementing
that, other than by the first-time/next-time idea I outlined above.

On Mon, 19 Dec 2005, Florian Weimer wrote:

> This would require a significant change in how Exim handles routing
> (part of the routing would run after the a delivery attempt has been
> performed). I doubt it's worth the complexity.

I agree.

--
Philip Hazel University of Cambridge Computing Service,
ph10[at]cus.cam.ac.uk Cambridge, England. Phone: +44 1223 334714.
Get the Exim 4 book: http://www.uit.co.uk/exim-book

--
## List details at http://www.exim.org/mailman/listinfo/exim-users
## Exim details at http://www.exim.org/
## Please use the Wiki with this list - http://www.exim.org/eximwiki/


fw at deneb

Dec 19, 2005, 2:46 AM

Post #23 of 29 (2331 views)
Permalink
Re: Potential logic error in retry handling for IPv4+IPv6 hosts [In reply to]

* Philip Hazel:

>> It would increase the number of DNS queries by quite a bit, though.
>
> Yes. Do we have any idea if this would be a serious problem?

If the server handles a significant mail volume, it's a good idea to
install a local caching resolver anyway. The additional queries would
be answered from cache most of the time, so it shouldn't significantly
increase load (or add delay to routing).

--
## List details at http://www.exim.org/mailman/listinfo/exim-users
## Exim details at http://www.exim.org/
## Please use the Wiki with this list - http://www.exim.org/eximwiki/


mh+exim-users at zugschlus

Dec 30, 2005, 12:53 PM

Post #24 of 29 (2303 views)
Permalink
Re: Potential logic error in retry handling for IPv4+IPv6 hosts [In reply to]

On Mon, 19 Dec 2005 10:20:01 +0000 (GMT), Philip Hazel
<ph10[at]cus.cam.ac.uk> wrote:
>On Sat, 17 Dec 2005, Marc Haber wrote:
>> This is actually an issue with how exim handles DNS answers. Just
>> imagine that the A record for a target host name expires in the
>> resolver's cache some time earlier than the AAAA record. When exim now
>> queries for the MX record, the resolver returns the data which it
>> still has cached, which is the AAAA record, in the additional section.
>>
>> Exim will believe the information from the additional section, and try
>> delivering there.
>
>Aarrgghh!! Yes, that does explain it. Thank you for tracking this one
>down.

Have you already decided on how to fix this in exim? As this is a bad
bug, we'd need to backport the fix to exim 4.50 as well for inclusion
in the next Debian point release. I'd greatly appreciate a patch.

Greetings
Marc

--
-------------------------------------- !! No courtesy copies, please !! -----
Marc Haber | " Questions are the | Mailadresse im Header
Mannheim, Germany | Beginning of Wisdom " | http://www.zugschlus.de/
Nordisch by Nature | Lt. Worf, TNG "Rightful Heir" | Fon: *49 621 72739834

--
## List details at http://www.exim.org/mailman/listinfo/exim-users
## Exim details at http://www.exim.org/
## Please use the Wiki with this list - http://www.exim.org/eximwiki/


ph10 at cus

Jan 1, 2006, 3:38 AM

Post #25 of 29 (2301 views)
Permalink
Re: Potential logic error in retry handling for IPv4+IPv6 hosts [In reply to]

On Fri, 30 Dec 2005, Marc Haber wrote:

> >> Exim will believe the information from the additional section, and try
> >> delivering there.
> >
> >Aarrgghh!! Yes, that does explain it. Thank you for tracking this one
> >down.
>
> Have you already decided on how to fix this in exim? As this is a bad
> bug, we'd need to backport the fix to exim 4.50 as well for inclusion
> in the next Debian point release. I'd greatly appreciate a patch.

It seems that the only sensible thing to do is to ignore the additional
section completely, and that is what I plan to do. I have not decided
whether to be more elaborate than that or not, but I suspect that it is
not worth doing anything. Busy systems should have "nearby" DNS servers
so that the additional queries will be cheap; for unbusy systems it
won't matter much.

I have not looked at the source, but the patch should be trivial; just
delete (or skip) the code that processes the additional section.

I do not expect to actually look at this for one or two or three weeks.
I have been on vacation over Christmas and New Year, and this week is a
short one in which I want to do documentation work. Thereafter I am
planning on a PCRE blitz, which will probably last most of January.

--
Philip Hazel University of Cambridge Computing Service
Get the Exim 4 book: http://www.uit.co.uk/exim-book

--
## List details at http://www.exim.org/mailman/listinfo/exim-users
## Exim details at http://www.exim.org/
## Please use the Wiki with this list - http://www.exim.org/eximwiki/

First page Previous page 1 2 Next page Last page  View All exim users RSS feed   Index | Next | Previous | View Threaded
 
 


Interested in having your list archived? Contact lists@gossamer-threads.com
 
  Web Applications & Managed Hosting Powered by Gossamer Threads Inc.