Mailing List Archive: nsp: ipv6

Broken DNS client resolvers (Was: Dealing with filtered 6to4 clients)

Index | Next | Previous | View Threaded

jeroen at unfix

Oct 27, 2009, 5:39 AM

Post #1 of 14 (1867 views)
Permalink

Broken DNS client resolvers (Was: Dealing with filtered 6to4 clients)

Martin List-Petersen wrote:
> Geert Hendrickx wrote:
>> On Tue, Oct 27, 2009 at 12:10:35PM +0000, Martin List-Petersen wrote:
>>> Martin List-Petersen wrote:
>>>> I wouldn't encourage that, but if your eyeball networks are that
>>>> paranoid, that's a way, how they can be in control. They could then
>>>> choose not to provide AAAA records to 6to4- and teredo- clients.
>>>>
>>>> Anyhow .. that's a hack and not to be encouraged, really.
>>> Arghh .. me not thinking today. Obviously they can't know, what the
>>> client has, but they could whitelist known good deployments then.
>>
>> Or, on your side, you could not serve AAAA records to (the DNS chaches of)
>> the problematic network(s)?
>
> That is the better approach alright.

Which doesn't help you much.

The problem with broken clients is that they are broken and that you do
not have control over them (which is good from one point ;)

As an excellent example look at:
https://bugs.edge.launchpad.net/ubuntu/+source/linux/+bug/417757

In short: host has IPv6 enabled, application does a getaddrinfo(), which
means it will ask for AAAA and then A from the resolvers. The DNS
resolver though sees a DNS query for an AAAA record and does "eeehmm
dunno, go away" and then just drops the request. The DNS client thus has
to time out, as that is the only option it has. The client then send a
request for an A record and gets a direct response.

Users solution: disable IPv6

Real solution: fix/replace the broken DNS server (eg change to OpenDNS*)
using the upstream DNS servers directly instead of the one in the DSL
modem generally is already a good enough fix as you bypass the problem.
Installing a local DNS recursor is another good option.

Still... it is an end-user issue which the network admin has no control
over and also can't easily check, nor does it actually require any IPv6
packets to flow over the network at all, just an active IPv6 stack, some
OSs are a little 'smart' about this, but one can't be completely smart.

Oh and yes, it also applies to Windows and every other OS...

Greets,
Jeroen

* = also broken in various ways:
https://lists.dns-oarc.net/pipermail/dns-operations/2009-July/004217.html

Attachments:

signature.asc (0.19 KB)

tore at linpro

Oct 27, 2009, 6:01 AM

Post #2 of 14 (1819 views)
Permalink

Re: Broken DNS client resolvers (Was: Dealing with filtered 6to4 clients) [In reply to]

Hi,

* Jeroen Massar

> In short: host has IPv6 enabled, application does a getaddrinfo(), which
> means it will ask for AAAA and then A from the resolvers. The DNS
> resolver though sees a DNS query for an AAAA record and does "eeehmm
> dunno, go away" and then just drops the request. The DNS client thus has
> to time out, as that is the only option it has. The client then send a
> request for an A record and gets a direct response.

Well, this is a different problem than the one I've been describing.
I'm sure this issue is causing its share of the client loss I see on the
dualstack site too, but I believe it is smaller in scope than the
problem I've been asking about. From what I can tell "my" problem is
responsible for more than half of the client loss on the dualstack site.

"Your" problem is incredibly hard to do anything about short of copying
Google's approach, as the problem is probably in the users' SOHO routers
most of the time and it would be impossible to contact them all and get
them to do anything about it. Unless of course it is a common problem
with a certain type of CPE device distributed by a certain ISP, of
course, but that does not appear to be the case here.

"My" problem is confined to two specific eyeball networks here in Norway
which I think it is more likely to do be able to do something about
somehow. If I can do that, and cut the client loss on the dualstack
site in half (now it varies between 0.12% to 0.18% from day to day) it
is more likely that my customers will be inclined to ignore the rest of
the lost clients (inluding the ones caused by "your" problem) and
proceed with dualstacking their content anyway.

By the way - the sites that have been used for the IPv6 testing have a
Norwegian reader mass (they're all in Norwegian). So my breakage
numbers and reasons are probably very specific to Norway.

Best regards,
--
Tore Anderson
Redpill Linpro AS - http://www.redpill-linpro.com/
Tel: +47 21 54 41 27

jeroen at unfix

Oct 27, 2009, 6:16 AM

Post #3 of 14 (1814 views)
Permalink

Re: Broken DNS client resolvers (Was: Dealing with filtered 6to4 clients) [In reply to]

Tore Anderson wrote:
> Hi,
>
> * Jeroen Massar
>
>> In short: host has IPv6 enabled, application does a getaddrinfo(), which
>> means it will ask for AAAA and then A from the resolvers. The DNS
>> resolver though sees a DNS query for an AAAA record and does "eeehmm
>> dunno, go away" and then just drops the request. The DNS client thus has
>> to time out, as that is the only option it has. The client then send a
>> request for an A record and gets a direct response.
>
> Well, this is a different problem than the one I've been describing.

Guess why I changed the subject ;)

> I'm sure this issue is causing its share of the client loss I see on the
> dualstack site too, but I believe it is smaller in scope than the
> problem I've been asking about. From what I can tell "my" problem is
> responsible for more than half of the client loss on the dualstack site.

try googling for "ubuntu ipv6 disable", you'll get an idea :)
or for that matter "disable ipv6" which also gives you the results for
Windows.

Googling here returns about 173.000 results along with:

Searches related to: disable ipv6
disable ipv6 linux disable ipv6 ubuntu disable ipv6 xp
disable ipv6 centos disable ipv6 fedora disable ipv6 debian disable ipv6
windows xp disable ipv6 redhat

I guess that tells enough on how wide-spread this issue is.

> "Your" problem is incredibly hard to do anything about short of copying
> Google's approach, as the problem is probably in the users' SOHO routers

Nope, that won't help. It does not matter if there are AAAA records in
DNS, it is all about having the client query for them and the resolver
(the one in the CPE) which drops the requests.

> most of the time and it would be impossible to contact them all and get
> them to do anything about it. Unless of course it is a common problem
> with a certain type of CPE device distributed by a certain ISP, of
> course, but that does not appear to be the case here.

The problem there is that even older versions of dnsmasq had this issue
and that is used in in a lot of CPEs.

> "My" problem is confined to two specific eyeball networks here in Norway
> which I think it is more likely to do be able to do something about
> somehow.

In that case not returning AAAA records for those would work and should
not be too much overhead. Best solution in this case though is to
convince the networks to fix their filtering issue, that is that they
don't filter.

Greets,
Jeroen

Attachments:

signature.asc (0.19 KB)

tore at linpro

Oct 27, 2009, 6:58 AM

Post #4 of 14 (1821 views)
Permalink

Re: Broken DNS client resolvers (Was: Dealing with filtered 6to4 clients) [In reply to]

* Jeroen Massar

> try googling for "ubuntu ipv6 disable", you'll get an idea :)

Oh, by the way, googling for "Tore Anderson" returns 11.400 results.
I'm pretty sure that does _not_ give me an idea of how many name
brothers I have.... ;-) (Sorry, couldn't resist.)

> Nope, that won't help. It does not matter if there are AAAA records in
> DNS, it is all about having the client query for them and the resolver
> (the one in the CPE) which drops the requests.

In that case no additional breakage is introduced by dualstacking the
sites in question, so it's not really something I (or my customers) care
too much about really. My problem is that if dualstacking www.mycust.no
makes it unavailable for a number of users (who will still be able
access the singlestacked www.competitor-of-mycust.no just fine), then my
customer is not likely to want any quad-A records anywhere near his
site. Depending on how many users gets problems, of course.

> In that case not returning AAAA records for those would work and should
> not be too much overhead. Best solution in this case though is to
> convince the networks to fix their filtering issue, that is that they
> don't filter.

Yup, blacklisting those operators would be the inverse of the Google
approach, and I think it will work well enough - take out a large chunk
of the brokenness while not being too conservative about it either (like
Google). I think that is probably the best approach for now, at least I
intend to have a chat with our hostmasters about how to go about
implementing it.

Convincing the problematic networks to allow proto-41 is of course a
possibility, but I can understand their reluctance - I mean, they have
probably a valid reason to want to filter inbound connections to, say,
1.2.3.4 port 22, 25, 80, 139, or whatever, but if they allow proto-41
then it will be embarrassingly easy for an external attacker to sidestep
the filter by connecting to 2002:0102:0304::0102:0304 on the given port
instead.

Best regards,
--
Tore Anderson
Redpill Linpro AS - http://www.redpill-linpro.com/
Tel: +47 21 54 41 27

remi at remlab

Oct 27, 2009, 7:20 AM

Post #5 of 14 (1820 views)
Permalink

Re: Broken DNS client resolvers (Was: Dealing with filtered 6to4 clients) [In reply to]

On Tue, 27 Oct 2009 13:39:42 +0100, Jeroen Massar <jeroen [at] unfix> wrote:
> As an excellent example look at:
> https://bugs.edge.launchpad.net/ubuntu/+source/linux/+bug/417757

Hmm well, I suspect the problem is rather along the line of glibc depending
on the little-known and hence under-used AI_ADDRCONFIG flag for correct
RFC3484 operation. With that flag, it will only look quad-A records up if
there is one (non-link-local) IPv6 address configured - in this case, it is
reasonable to assume DNS works correctly, especially as Linux distros don't
do "automatic" 6to4 setup by default.

Whether it's a glibc or a many-applications bug is debatable.

--
Rémi Denis-Courmont

jeroen at unfix

Oct 27, 2009, 7:29 AM

Post #6 of 14 (1816 views)
Permalink

Re: Broken DNS client resolvers (Was: Dealing with filtered 6to4 clients) [In reply to]

Rémi Denis-Courmont wrote:
> On Tue, 27 Oct 2009 13:39:42 +0100, Jeroen Massar <jeroen [at] unfix> wrote:
>> As an excellent example look at:
>> https://bugs.edge.launchpad.net/ubuntu/+source/linux/+bug/417757
>
> Hmm well, I suspect the problem is rather along the line of glibc depending
> on the little-known and hence under-used AI_ADDRCONFIG flag for correct
> RFC3484 operation. With that flag, it will only look quad-A records up if
> there is one (non-link-local) IPv6 address configured - in this case, it is
> reasonable to assume DNS works correctly, especially as Linux distros don't
> do "automatic" 6to4 setup by default.
>
> Whether it's a glibc or a many-applications bug is debatable.

*WHICH IS NOT THE ISSUE*

(it is another one, but that generally doesn't hit anyone, as without an
IPv6 default route your packets directly return anyway with an UNREACH
when they are sent outbound thus it does not hurt in most cases)

dig @<broken dns resolver> www.microsoft.com AAAA

and the dns resolver will never ever reply that that query.

This means that your resolver will time out.

It is irrespective of having actualy IPv6 connectivity or not. It does
depend on the OS making a decision if it needs to query an AAAA record
or not which could indeed then be based on flags passed to getaddrinfo
AI_ADDRCONFIG.

Note though that AI_ADDRCONFIG stated per the man page on Debian:
8<-----------
If hints.ai_flags includes the AI_ADDRCONFIG flag, then IPv4
addresses are returned in the list pointed to by
result only if the local system has at least one IPv4 address
configured, and IPv6 addresses are only returned if
the local system has at least one IPv6 address configured.
------------>

In other words, 6to4, Teredo etc and you are bust.
Also note that those are the defaults on Windows Vista and Seven...

And even if you have proper IPv6 connectivity, this DNS querying bug can
still hit you straight in the face. (Heck I had it at home with an old
WRT box but I never noticed it till I once didn't have my VPN open and
thus started using the local DNS server which was broken...)

(And gee, see why so many other people are confused about this....)

Greets,
Jeroen

Attachments:

signature.asc (0.19 KB)

remi at remlab

Oct 27, 2009, 8:03 AM

Post #7 of 14 (1823 views)
Permalink

Re: Broken DNS client resolvers (Was: Dealing with filtered 6to4 clients) [In reply to]

On Tue, 27 Oct 2009 15:29:42 +0100, Jeroen Massar <jeroen [at] unfix> wrote:
>> Whether it's a glibc or a many-applications bug is debatable.
>
> *WHICH IS NOT THE ISSUE*

It is the issue.

(...)
> In other words, 6to4, Teredo etc and you are bust.
> Also note that those are the defaults on Windows Vista and Seven...

To my knowledge, _none_ of the common Linux distros enable 6to4 or Teredo
automatically by default. Of course, if they did, then they'd have to
provide resolver hacks such as those done by Microsoft. _Then_ you can
think of running the A and AAA queries in parallel, and timing out the AAAA
query quickly after the A response. But it is currently a non-issue on
_Linux_, which is the system the bug refers to.

--
Rémi Denis-Courmont

jeroen at unfix

Oct 27, 2009, 9:02 AM

Post #8 of 14 (1819 views)
Permalink

Re: Broken DNS client resolvers (Was: Dealing with filtered 6to4 clients) [In reply to]

Rémi Denis-Courmont wrote:
> On Tue, 27 Oct 2009 15:29:42 +0100, Jeroen Massar <jeroen [at] unfix> wrote:
>>> Whether it's a glibc or a many-applications bug is debatable.
>> *WHICH IS NOT THE ISSUE*
>
> It is the issue.

I would say, explain then to the Ubuntu folks how to properly resolve
it, I am sure they will love you for it. (And it would save again some
people on blocking IPv6 on their boxes, then again, their box, their
problem)

Yes, I can see that the ADDRCONF flag can be useful for this, as it
avoids querying AAAA records in the first place, but that should not be
done on a per-application level. That is a decision to be made by the
resolver library which should be smart about that, link-local addresses
can't be stuffed in a AAAA address anyway and if you don't have
connectivity then there is not much to be done.

> (...)
>> In other words, 6to4, Teredo etc and you are bust.
>> Also note that those are the defaults on Windows Vista and Seven...
>
> To my knowledge, _none_ of the common Linux distros enable 6to4 or Teredo
> automatically by default.

If you have IPv6 enabled in the kernel, which is the default, and
somebody runs a "rogue" RA it gets enabled already (then you generally
also get nice broken routes in addition ;)

There are enough people who also magically tend to configure all kinds
of things wrong or install magic tools they don't need, especially when
they hear that "IPv6 will give them access to free warez". uTorrent is
an example of that, which enables Teredo, but there are also other tools
which do so.

> Of course, if they did, then they'd have to
> provide resolver hacks such as those done by Microsoft. _Then_ you can
> think of running the A and AAA queries in parallel, and timing out the AAAA
> query quickly after the A response.

Which is what current glibc's (2.9 series) already do in most cases, but
these also have some smarter algorithms to determine when and when not
to do IPv6 queries.

An application should not be forced to one or the other though, maybe
the user wants to connect to that server on the link-local network, that
was the whole point of the dentist-problem. As such, for instance
Firefox should be able to do that too. (with for instance mDNS for
resolving in that case, and yep, again something annoying called avahi
is a semi-default, good that there are ways to block packages from ever
installing)

> But it is currently a non-issue on
> _Linux_, which is the system the bug refers to.

If it is such a "non-issue", why are there so many people complaining
about it and then disabling IPv6? While if they specify eg the opendns
nameservers in their resolv.conf everything works fine!? :)

Greets,
Jeroen

Attachments:

signature.asc (0.19 KB)

swmike at swm

Oct 27, 2009, 9:12 AM

Post #9 of 14 (1828 views)
Permalink

Re: Broken DNS client resolvers (Was: Dealing with filtered 6to4 clients) [In reply to]

On Tue, 27 Oct 2009, Jeroen Massar wrote:

> I would say, explain then to the Ubuntu folks how to properly resolve
> it, I am sure they will love you for it.

There is a bug reported that when you disassociate/reassociate wlan, it
doesn't drop the IPv6 address between (so after a few movements between
wlan:s, you'll have a bunch of IPv6 addresses from different /64s), the
response from the ubuntu people was that this needed a feature
enhancement, it wasn't a bug. The writing from the person in question
seemed to indicate that this wasn't seen as an issue.

So no, there is very little IPv6 love from Ubuntu.

<https://bugs.launchpad.net/ubuntu/+source/network-manager/+bug/215497>

--
Mikael Abrahamsson email: swmike [at] swm

jeroen at unfix

Oct 27, 2009, 9:28 AM

Post #10 of 14 (1825 views)
Permalink

Re: Broken DNS client resolvers (Was: Dealing with filtered 6to4 clients) [In reply to]

Mikael Abrahamsson wrote:
> On Tue, 27 Oct 2009, Jeroen Massar wrote:
>
>> I would say, explain then to the Ubuntu folks how to properly resolve
>> it, I am sure they will love you for it.
>
> There is a bug reported that when you disassociate/reassociate wlan, it
> doesn't drop the IPv6 address between (so after a few movements between
> wlan:s, you'll have a bunch of IPv6 addresses from different /64s), the
> response from the ubuntu people was that this needed a feature
> enhancement, it wasn't a bug. The writing from the person in question
> seemed to indicate that this wasn't seen as an issue.

Had not seen that one, but the below might be a work-around for it:

In /etc/network/interfaces at the interfaces with the issues, add:
pre-down ip -6 addr flush dev <name>
pre-down ip -6 ro flush dev <name>

(Maybe a post-down, but I guess that if the interface is marked 'down'
one can't flush anymore, I noticed that once)

Can't test it, as I don't use Linux on my laptop (Windows XP FTW! etc :)
due to the crap wireless support on that platform. Oh and something with
having already loads of Linux boxes to SSH/VNC/X into if I need a
specific thing from such a box....

> So no, there is very little IPv6 love from Ubuntu.

That was the impression I am getting too, that even while they are at
least shipping IPv6-per-default-enabled, though I guess that might also
be to match features with other distros.

Greets,
Jeroen

Attachments:

signature.asc (0.19 KB)

remi at remlab

Oct 27, 2009, 9:37 AM

Post #11 of 14 (1822 views)
Permalink

Re: Broken DNS client resolvers (Was: Dealing with filtered 6to4 clients) [In reply to]

On Tue, 27 Oct 2009 17:02:23 +0100, Jeroen Massar <jeroen [at] unfix> wrote:
> Yes, I can see that the ADDRCONF flag can be useful for this, as it
> avoids querying AAAA records in the first place, but that should not be
> done on a per-application level. That is a decision to be made by the
> resolver library which should be smart about that, link-local addresses
> can't be stuffed in a AAAA address anyway and if you don't have
> connectivity then there is not much to be done.

I guess glibc does standard lawyering. There are cases where an application
wantd to query AAAA regardless of the system configuration. Those cases are
corner cases. glibc follows the standard to the letter and resolves
everything by default.

And so, glibc adds a (non-standard) flag if you want to use getaddrinfo for
policy. In other words, even though this is the most common case, it is not
the default. I am not going to get into a fight with Ulrich Drepper on
this, as it would most likely get us nowhere.

Ubuntu can force AI_ADDRCONFIG into their glibc if they see fit.

> If you have IPv6 enabled in the kernel, which is the default, and
> somebody runs a "rogue" RA it gets enabled already (then you generally
> also get nice broken routes in addition ;)

Well then you are really screwed anyway, just like if someone runs a rogue
DHCPv4 at the moment you boot your computer.

> There are enough people who also magically tend to configure all kinds
> of things wrong or install magic tools they don't need, especially when
> they hear that "IPv6 will give them access to free warez". uTorrent is
> an example of that, which enables Teredo, but there are also other tools
> which do so.

AFAIK, that case is worked around in glibc 2.10:
* DNS IPv4-IPv6 parallel lookup now deals better with broken DNS
servers (the case, e.g., for some people using the built-in DNS
server in ADSL modems/routers). There is a once-per-process timeout
in case of a broken server. To avoid it, users can run nscd or put
'options single-request' in /etc/resolv.conf.
Implemented by Ulrich Drepper.

...which builds on top of an earlier glibc 2.9 enhancement:
* Unified lookup for getaddrinfo: IPv4 and IPv6 addresses are now looked
up at the same time. Implemented by Ulrich Drepper.

>> Of course, if they did, then they'd have to
>> provide resolver hacks such as those done by Microsoft. _Then_ you can
>> think of running the A and AAA queries in parallel, and timing out the
>> AAAA query quickly after the A response.
>
> Which is what current glibc's (2.9 series) already do in most cases, but
> these also have some smarter algorithms to determine when and when not
> to do IPv6 queries.

2.9 is not current - anymore.

Anyway, I fail to see what else can be done. There is no way to determine
that AAAA are broken without trying. Worse, this hack might cause false
positives.

>> But it is currently a non-issue on
>> _Linux_, which is the system the bug refers to.
>
> If it is such a "non-issue", why are there so many people complaining
> about it and then disabling IPv6? While if they specify eg the opendns
> nameservers in their resolv.conf everything works fine!? :)

Automatically brought up 6to4 and Teredo are non-issues because there are
NO SUCH THINGS on Linux. The point is, if glibc stops querying AAAA
pointlessly, then the issue is solved. There is no need to hacks for
automatic tunneling on a platform that does not have automatic tunneling.

--
Rémi Denis-Courmont

charles at thewybles

Oct 27, 2009, 10:51 AM

Post #12 of 14 (1815 views)
Permalink

Re: Broken DNS client resolvers (Was: Dealing with filtered 6to4 clients) [In reply to]

On Oct 27, 2009, at 9:37 AM, R�mi Denis-Courmont wrote:
>>
>
> AFAIK, that case is worked around in glibc 2.10:
> * DNS IPv4-IPv6 parallel lookup now deals better with broken DNS
> servers (the case, e.g., for some people using the built-in DNS
> server in ADSL modems/routers). There is a once-per-process timeout
> in case of a broken server. To avoid it, users can run nscd or put
> 'options single-request' in /etc/resolv.conf.
> Implemented by Ulrich Drepper.

I would recommend not running nscd. It causes all manner of issues and
weirdness.

Haven't played with the single-request in /etc/resolv.conf so can't
comment on that.

ipv6-ops+phil at spodhuis

Oct 27, 2009, 8:45 PM

Post #13 of 14 (1818 views)
Permalink

Re: Broken DNS client resolvers (Was: Dealing with filtered 6to4 clients) [In reply to]

On 2009-10-27 at 17:37 +0100, Rémi Denis-Courmont wrote:
>
> On Tue, 27 Oct 2009 17:02:23 +0100, Jeroen Massar <jeroen [at] unfix> wrote:
> > Yes, I can see that the ADDRCONF flag can be useful for this, as it
> > avoids querying AAAA records in the first place, but that should not be
> > done on a per-application level. That is a decision to be made by the
> > resolver library which should be smart about that, link-local addresses
> > can't be stuffed in a AAAA address anyway and if you don't have
> > connectivity then there is not much to be done.
>
> I guess glibc does standard lawyering. There are cases where an application
> wantd to query AAAA regardless of the system configuration. Those cases are
> corner cases. glibc follows the standard to the letter and resolves
> everything by default.
>
> And so, glibc adds a (non-standard) flag if you want to use getaddrinfo for
> policy. In other words, even though this is the most common case, it is not
> the default. I am not going to get into a fight with Ulrich Drepper on
> this, as it would most likely get us nowhere.
>
> Ubuntu can force AI_ADDRCONFIG into their glibc if they see fit.

Non-standard? Really?

Well, it's true that the RFC for "Basic Socket Interface Extensions for
IPv6" is non-standard, it's "Informational", but that's because it's a
host API and more of a guideline of how to implement things. But
AI_ADDRCONFIG is in RFC 3493 (from 2003).

It was even in RFC 2553, from 1999. It was not in RFC 2133, from 1997.

So AI_ADDRCONFIG was only introduced ten years ago, in an Informational
RFC.

Fortunately, we have the Single Unix Specification. SUSv3 includes
AI_ADDRCONFIG. That makes it standard. SUSv2 did not include
getaddrinfo(), AFAICT.

So please, stop being rude about people who aren't here to defend
themselves from such false statements.

-Phil

remi at remlab

Oct 28, 2009, 12:54 AM

Post #14 of 14 (1815 views)
Permalink

Re: Broken DNS client resolvers (Was: Dealing with filtered 6to4 clients) [In reply to]

On Tue, 27 Oct 2009 20:45:06 -0700, Phil Pennock
<ipv6-ops+phil [at] spodhuis> wrote:
>> Ubuntu can force AI_ADDRCONFIG into their glibc if they see fit.
>
> Non-standard? Really?
>
> Well, it's true that the RFC for "Basic Socket Interface Extensions for
> IPv6" is non-standard, it's "Informational", but that's because it's a
> host API and more of a guideline of how to implement things. But
> AI_ADDRCONFIG is in RFC 3493 (from 2003).

Well, then maybe it is standard. In that case, the blame is indeed on the
applications rather than on glibc. But still most (not all)
getaddrinfo-aware applications ignore this flag. This probably has
something to do with the fact that it did not exist in original getaddrinfo
spec, and that some major operating systems or older version thereof don't
have it...

--
Rémi Denis-Courmont

Index | Next | Previous | View Threaded

Interested in having your list archived? Contact Gossamer Threads