Mailing List Archive: Linux: Kernel

[PATCH] x86: fix gart iommu using for amd 64 bit system

Index | Next | Previous | View Threaded

yinghai at kernel

Nov 21, 2009, 8:24 PM

Post #1 of 15 (305 views)
Permalink

[PATCH] x86: fix gart iommu using for amd 64 bit system

after
|commit 75f1cdf1dda92cae037ec848ae63690d91913eac
|Author: FUJITA Tomonori <fujita.tomonori [at] lab>
|Date: Tue Nov 10 19:46:20 2009 +0900
|
| x86: Handle HW IOMMU initialization failure gracefully
|
| If HW IOMMU initialization fails (Intel VT-d often does this,
| typically due to BIOS bugs), we fall back to nommu. It doesn't
| work for the majority since nowadays we have more than 4GB
| memory so we must use swiotlb instead of nommu.
...

amd 64 systems that
1. do not have AGP
2. do not have IOMMU
3. mem > 4g
4. BIOS do not allocate correct gart in NB.
will leave them to use SWIOTLB forcely.

also in pci_iommu_alloc()
pci_swiotlb_init is stealing the preallocate range that is for
gart_iommu_hole workaround.

so restore the sequence...

will get back
[ 0.000000] Your BIOS doesn't leave a aperture memory hole
[ 0.000000] Please enable the IOMMU option in the BIOS setup
[ 0.000000] This costs you 64 MB of RAM
[ 0.000000] Mapping aperture over 65536 KB of RAM @ 20000000

Signed-off-by: Yinghai Lu <yinghai [at] kernel>

---
arch/x86/kernel/aperture_64.c | 2 +-
arch/x86/kernel/pci-dma.c | 5 +++--
2 files changed, 4 insertions(+), 3 deletions(-)

Index: linux-2.6/arch/x86/kernel/aperture_64.c
===================================================================
--- linux-2.6.orig/arch/x86/kernel/aperture_64.c
+++ linux-2.6/arch/x86/kernel/aperture_64.c
@@ -458,7 +458,7 @@ out:

if (aper_alloc) {
/* Got the aperture from the AGP bridge */
- } else if (!valid_agp) {
+ } else if (swiotlb && !valid_agp) {
/* Do nothing */
} else if ((!no_iommu && max_pfn > MAX_DMA32_PFN) ||
force_iommu ||
Index: linux-2.6/arch/x86/kernel/pci-dma.c
===================================================================
--- linux-2.6.orig/arch/x86/kernel/pci-dma.c
+++ linux-2.6/arch/x86/kernel/pci-dma.c
@@ -124,11 +124,12 @@ void __init pci_iommu_alloc(void)
/* free the range so iommu could get some range less than 4G */
dma32_free_bootmem();
#endif
+ if (!swiotlb_force)
+ gart_iommu_hole_init();
+
if (pci_swiotlb_init())
return;

- gart_iommu_hole_init();
-
detect_calgary();

detect_intel_iommu();
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo [at] vger
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/

fujita.tomonori at lab

Nov 24, 2009, 12:46 AM

Post #2 of 15 (272 views)
Permalink

Re: [PATCH] x86: fix gart iommu using for amd 64 bit system [In reply to]

On Sat, 21 Nov 2009 20:24:52 -0800
Yinghai Lu <yinghai [at] kernel> wrote:

> amd 64 systems that
> 1. do not have AGP
> 2. do not have IOMMU
> 3. mem > 4g
> 4. BIOS do not allocate correct gart in NB.
> will leave them to use SWIOTLB forcely.

Sorry, I forgot about this GART workaround.

> Index: linux-2.6/arch/x86/kernel/pci-dma.c
> ===================================================================
> --- linux-2.6.orig/arch/x86/kernel/pci-dma.c
> +++ linux-2.6/arch/x86/kernel/pci-dma.c
> @@ -124,11 +124,12 @@ void __init pci_iommu_alloc(void)
> /* free the range so iommu could get some range less than 4G */
> dma32_free_bootmem();
> #endif
> + if (!swiotlb_force)
> + gart_iommu_hole_init();
> +
> if (pci_swiotlb_init())
> return;
>
> - gart_iommu_hole_init();
> -
> detect_calgary();
>
> detect_intel_iommu();

I prefer to detect all the IOMMU drivers in a consistent way;
detecting only GART before swioltb doesn't look nice.

As we did before, we could detect all the IOMMU driver before
swiotlb. However, I think that it's better to simply change
pci_swiotlb_init() not to steal the preallocate GART workaround memory.

btw, initializing swiotlb before IOMMU detection is useful to GART
too? If GART can't allocate the workaround memory, then the kernel
panic now. We can use swiotlb instead in that case?

=
From: FUJITA Tomonori <fujita.tomonori [at] lab>
Subject: [PATCH] x86: stop swiotlb stealing the GART workaround memory

swiotlb wrongly uses the GART workaround memory (for bad BIOS) that
dma32_reserv_bootmem allocates. We need to initialize swiotlb before
dma32_free_bootmem().

Signed-off-by: FUJITA Tomonori <fujita.tomonori [at] lab>
---
arch/x86/kernel/pci-dma.c | 6 +++++-
1 files changed, 5 insertions(+), 1 deletions(-)

diff --git a/arch/x86/kernel/pci-dma.c b/arch/x86/kernel/pci-dma.c
index afcc58b..26fe2cd 100644
--- a/arch/x86/kernel/pci-dma.c
+++ b/arch/x86/kernel/pci-dma.c
@@ -120,11 +120,15 @@ static void __init dma32_free_bootmem(void)

void __init pci_iommu_alloc(void)
{
+ int use_swiotlb;
+
+ use_swiotlb = pci_swiotlb_init();
+
#ifdef CONFIG_X86_64
/* free the range so iommu could get some range less than 4G */
dma32_free_bootmem();
#endif
- if (pci_swiotlb_init())
+ if (use_swiotlb)
return;

gart_iommu_hole_init();
--
1.5.6.5

--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo [at] vger
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/

yinghai at kernel

Nov 24, 2009, 1:19 AM

Post #3 of 15 (270 views)
Permalink

Re: [PATCH] x86: fix gart iommu using for amd 64 bit system [In reply to]

FUJITA Tomonori wrote:
> On Sat, 21 Nov 2009 20:24:52 -0800
> Yinghai Lu <yinghai [at] kernel> wrote:
>
>> amd 64 systems that
>> 1. do not have AGP
>> 2. do not have IOMMU
>> 3. mem > 4g
>> 4. BIOS do not allocate correct gart in NB.
>> will leave them to use SWIOTLB forcely.
>
> Sorry, I forgot about this GART workaround.
>
>
>> Index: linux-2.6/arch/x86/kernel/pci-dma.c
>> ===================================================================
>> --- linux-2.6.orig/arch/x86/kernel/pci-dma.c
>> +++ linux-2.6/arch/x86/kernel/pci-dma.c
>> @@ -124,11 +124,12 @@ void __init pci_iommu_alloc(void)
>> /* free the range so iommu could get some range less than 4G */
>> dma32_free_bootmem();
>> #endif
>> + if (!swiotlb_force)
>> + gart_iommu_hole_init();
>> +
>> if (pci_swiotlb_init())
>> return;
>>
>> - gart_iommu_hole_init();
>> -
>> detect_calgary();
>>
>> detect_intel_iommu();
>
> I prefer to detect all the IOMMU drivers in a consistent way;
> detecting only GART before swioltb doesn't look nice.
>
> As we did before, we could detect all the IOMMU driver before
> swiotlb. However, I think that it's better to simply change
> pci_swiotlb_init() not to steal the preallocate GART workaround memory.
>
> btw, initializing swiotlb before IOMMU detection is useful to GART
> too? If GART can't allocate the workaround memory, then the kernel
> panic now. We can use swiotlb instead in that case?
>
> =
> From: FUJITA Tomonori <fujita.tomonori [at] lab>
> Subject: [PATCH] x86: stop swiotlb stealing the GART workaround memory
>
> swiotlb wrongly uses the GART workaround memory (for bad BIOS) that
> dma32_reserv_bootmem allocates. We need to initialize swiotlb before
> dma32_free_bootmem().
>
> Signed-off-by: FUJITA Tomonori <fujita.tomonori [at] lab>
> ---
> arch/x86/kernel/pci-dma.c | 6 +++++-
> 1 files changed, 5 insertions(+), 1 deletions(-)
>
> diff --git a/arch/x86/kernel/pci-dma.c b/arch/x86/kernel/pci-dma.c
> index afcc58b..26fe2cd 100644
> --- a/arch/x86/kernel/pci-dma.c
> +++ b/arch/x86/kernel/pci-dma.c
> @@ -120,11 +120,15 @@ static void __init dma32_free_bootmem(void)
>
> void __init pci_iommu_alloc(void)
> {
> + int use_swiotlb;
> +
> + use_swiotlb = pci_swiotlb_init();
> +
> #ifdef CONFIG_X86_64
> /* free the range so iommu could get some range less than 4G */
> dma32_free_bootmem();
> #endif
> - if (pci_swiotlb_init())
> + if (use_swiotlb)
> return;
>
> gart_iommu_hole_init();

pci_swiotlb_init need be called after dma32_free_bootmem
otherwise it could fail for system with lots of memory and numa=off

so dma32_free_bootmem will work for gart_iommu_hole_init and pci_swiotlb_init..
make sure they can get RAM < 4g.

also in gart_iommu_hole_init we need check swiotlb and !valid_agp...
- } else if (!valid_agp) {
+ } else if (swiotlb && !valid_agp) {

otherwise the the workaround will be skipped...

YH
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo [at] vger
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/

fujita.tomonori at lab

Nov 24, 2009, 1:35 AM

Post #4 of 15 (271 views)
Permalink

Re: [PATCH] x86: fix gart iommu using for amd 64 bit system [In reply to]

On Tue, 24 Nov 2009 01:19:21 -0800
Yinghai Lu <yinghai [at] kernel> wrote:

> FUJITA Tomonori wrote:
> > On Sat, 21 Nov 2009 20:24:52 -0800
> > Yinghai Lu <yinghai [at] kernel> wrote:
> >
> >> amd 64 systems that
> >> 1. do not have AGP
> >> 2. do not have IOMMU
> >> 3. mem > 4g
> >> 4. BIOS do not allocate correct gart in NB.
> >> will leave them to use SWIOTLB forcely.
> >
> > Sorry, I forgot about this GART workaround.
> >
> >
> >> Index: linux-2.6/arch/x86/kernel/pci-dma.c
> >> ===================================================================
> >> --- linux-2.6.orig/arch/x86/kernel/pci-dma.c
> >> +++ linux-2.6/arch/x86/kernel/pci-dma.c
> >> @@ -124,11 +124,12 @@ void __init pci_iommu_alloc(void)
> >> /* free the range so iommu could get some range less than 4G */
> >> dma32_free_bootmem();
> >> #endif
> >> + if (!swiotlb_force)
> >> + gart_iommu_hole_init();
> >> +
> >> if (pci_swiotlb_init())
> >> return;
> >>
> >> - gart_iommu_hole_init();
> >> -
> >> detect_calgary();
> >>
> >> detect_intel_iommu();
> >
> > I prefer to detect all the IOMMU drivers in a consistent way;
> > detecting only GART before swioltb doesn't look nice.
> >
> > As we did before, we could detect all the IOMMU driver before
> > swiotlb. However, I think that it's better to simply change
> > pci_swiotlb_init() not to steal the preallocate GART workaround memory.
> >
> > btw, initializing swiotlb before IOMMU detection is useful to GART
> > too? If GART can't allocate the workaround memory, then the kernel
> > panic now. We can use swiotlb instead in that case?
> >
> > =
> > From: FUJITA Tomonori <fujita.tomonori [at] lab>
> > Subject: [PATCH] x86: stop swiotlb stealing the GART workaround memory
> >
> > swiotlb wrongly uses the GART workaround memory (for bad BIOS) that
> > dma32_reserv_bootmem allocates. We need to initialize swiotlb before
> > dma32_free_bootmem().
> >
> > Signed-off-by: FUJITA Tomonori <fujita.tomonori [at] lab>
> > ---
> > arch/x86/kernel/pci-dma.c | 6 +++++-
> > 1 files changed, 5 insertions(+), 1 deletions(-)
> >
> > diff --git a/arch/x86/kernel/pci-dma.c b/arch/x86/kernel/pci-dma.c
> > index afcc58b..26fe2cd 100644
> > --- a/arch/x86/kernel/pci-dma.c
> > +++ b/arch/x86/kernel/pci-dma.c
> > @@ -120,11 +120,15 @@ static void __init dma32_free_bootmem(void)
> >
> > void __init pci_iommu_alloc(void)
> > {
> > + int use_swiotlb;
> > +
> > + use_swiotlb = pci_swiotlb_init();
> > +
> > #ifdef CONFIG_X86_64
> > /* free the range so iommu could get some range less than 4G */
> > dma32_free_bootmem();
> > #endif
> > - if (pci_swiotlb_init())
> > + if (use_swiotlb)
> > return;
> >
> > gart_iommu_hole_init();
>
> pci_swiotlb_init need be called after dma32_free_bootmem
> otherwise it could fail for system with lots of memory and numa=off

Why? swiotlb needs just 64MB (and some) in DMA32.

swiotlb had been worked fine without that hack until 2.6.26?

Only broken GART needs that hack (and I think that it's better to move
that hack to GART code).

> so dma32_free_bootmem will work for gart_iommu_hole_init and pci_swiotlb_init..
> make sure they can get RAM < 4g.
>
> also in gart_iommu_hole_init we need check swiotlb and !valid_agp...
> - } else if (!valid_agp) {
> + } else if (swiotlb && !valid_agp) {
>
> otherwise the the workaround will be skipped...
>
> YH
> --
> To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
> the body of a message to majordomo [at] vger
> More majordomo info at http://vger.kernel.org/majordomo-info.html
> Please read the FAQ at http://www.tux.org/lkml/
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo [at] vger
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/

yinghai at kernel

Nov 24, 2009, 1:48 AM

Post #5 of 15 (270 views)
Permalink

Re: [PATCH] x86: fix gart iommu using for amd 64 bit system [In reply to]

FUJITA Tomonori wrote:
> On Tue, 24 Nov 2009 01:19:21 -0800
> Yinghai Lu <yinghai [at] kernel> wrote:
>
>> FUJITA Tomonori wrote:
>>> On Sat, 21 Nov 2009 20:24:52 -0800
>>> Yinghai Lu <yinghai [at] kernel> wrote:
>>>
>>>> amd 64 systems that
>>>> 1. do not have AGP
>>>> 2. do not have IOMMU
>>>> 3. mem > 4g
>>>> 4. BIOS do not allocate correct gart in NB.
>>>> will leave them to use SWIOTLB forcely.
>>> Sorry, I forgot about this GART workaround.
>>>
>>>
>>>> Index: linux-2.6/arch/x86/kernel/pci-dma.c
>>>> ===================================================================
>>>> --- linux-2.6.orig/arch/x86/kernel/pci-dma.c
>>>> +++ linux-2.6/arch/x86/kernel/pci-dma.c
>>>> @@ -124,11 +124,12 @@ void __init pci_iommu_alloc(void)
>>>> /* free the range so iommu could get some range less than 4G */
>>>> dma32_free_bootmem();
>>>> #endif
>>>> + if (!swiotlb_force)
>>>> + gart_iommu_hole_init();
>>>> +
>>>> if (pci_swiotlb_init())
>>>> return;
>>>>
>>>> - gart_iommu_hole_init();
>>>> -
>>>> detect_calgary();
>>>>
>>>> detect_intel_iommu();
>>> I prefer to detect all the IOMMU drivers in a consistent way;
>>> detecting only GART before swioltb doesn't look nice.
>>>
>>> As we did before, we could detect all the IOMMU driver before
>>> swiotlb. However, I think that it's better to simply change
>>> pci_swiotlb_init() not to steal the preallocate GART workaround memory.
>>>
>>> btw, initializing swiotlb before IOMMU detection is useful to GART
>>> too? If GART can't allocate the workaround memory, then the kernel
>>> panic now. We can use swiotlb instead in that case?
>>>
>>> =
>>> From: FUJITA Tomonori <fujita.tomonori [at] lab>
>>> Subject: [PATCH] x86: stop swiotlb stealing the GART workaround memory
>>>
>>> swiotlb wrongly uses the GART workaround memory (for bad BIOS) that
>>> dma32_reserv_bootmem allocates. We need to initialize swiotlb before
>>> dma32_free_bootmem().
>>>
>>> Signed-off-by: FUJITA Tomonori <fujita.tomonori [at] lab>
>>> ---
>>> arch/x86/kernel/pci-dma.c | 6 +++++-
>>> 1 files changed, 5 insertions(+), 1 deletions(-)
>>>
>>> diff --git a/arch/x86/kernel/pci-dma.c b/arch/x86/kernel/pci-dma.c
>>> index afcc58b..26fe2cd 100644
>>> --- a/arch/x86/kernel/pci-dma.c
>>> +++ b/arch/x86/kernel/pci-dma.c
>>> @@ -120,11 +120,15 @@ static void __init dma32_free_bootmem(void)
>>>
>>> void __init pci_iommu_alloc(void)
>>> {
>>> + int use_swiotlb;
>>> +
>>> + use_swiotlb = pci_swiotlb_init();
>>> +
>>> #ifdef CONFIG_X86_64
>>> /* free the range so iommu could get some range less than 4G */
>>> dma32_free_bootmem();
>>> #endif
>>> - if (pci_swiotlb_init())
>>> + if (use_swiotlb)
>>> return;
>>>
>>> gart_iommu_hole_init();
>> pci_swiotlb_init need be called after dma32_free_bootmem
>> otherwise it could fail for system with lots of memory and numa=off
>
> Why? swiotlb needs just 64MB (and some) in DMA32.
>
> swiotlb had been worked fine without that hack until 2.6.26?
>
> Only broken GART needs that hack (and I think that it's better to move
> that hack to GART code).

gart iommu workaround will allocate 64M by default.

fail config: like system with 512g and sparse mem, and vmmap.

YH
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo [at] vger
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/

fujita.tomonori at lab

Nov 24, 2009, 2:31 AM

Post #6 of 15 (269 views)
Permalink

Re: [PATCH] x86: fix gart iommu using for amd 64 bit system [In reply to]

On Tue, 24 Nov 2009 01:48:54 -0800
Yinghai Lu <yinghai [at] kernel> wrote:

> FUJITA Tomonori wrote:
> > On Tue, 24 Nov 2009 01:19:21 -0800
> > Yinghai Lu <yinghai [at] kernel> wrote:
> >
> >> FUJITA Tomonori wrote:
> >>> On Sat, 21 Nov 2009 20:24:52 -0800
> >>> Yinghai Lu <yinghai [at] kernel> wrote:
> >>>
> >>>> amd 64 systems that
> >>>> 1. do not have AGP
> >>>> 2. do not have IOMMU
> >>>> 3. mem > 4g
> >>>> 4. BIOS do not allocate correct gart in NB.
> >>>> will leave them to use SWIOTLB forcely.
> >>> Sorry, I forgot about this GART workaround.
> >>>
> >>>
> >>>> Index: linux-2.6/arch/x86/kernel/pci-dma.c
> >>>> ===================================================================
> >>>> --- linux-2.6.orig/arch/x86/kernel/pci-dma.c
> >>>> +++ linux-2.6/arch/x86/kernel/pci-dma.c
> >>>> @@ -124,11 +124,12 @@ void __init pci_iommu_alloc(void)
> >>>> /* free the range so iommu could get some range less than 4G */
> >>>> dma32_free_bootmem();
> >>>> #endif
> >>>> + if (!swiotlb_force)
> >>>> + gart_iommu_hole_init();
> >>>> +
> >>>> if (pci_swiotlb_init())
> >>>> return;
> >>>>
> >>>> - gart_iommu_hole_init();
> >>>> -
> >>>> detect_calgary();
> >>>>
> >>>> detect_intel_iommu();
> >>> I prefer to detect all the IOMMU drivers in a consistent way;
> >>> detecting only GART before swioltb doesn't look nice.
> >>>
> >>> As we did before, we could detect all the IOMMU driver before
> >>> swiotlb. However, I think that it's better to simply change
> >>> pci_swiotlb_init() not to steal the preallocate GART workaround memory.
> >>>
> >>> btw, initializing swiotlb before IOMMU detection is useful to GART
> >>> too? If GART can't allocate the workaround memory, then the kernel
> >>> panic now. We can use swiotlb instead in that case?
> >>>
> >>> =
> >>> From: FUJITA Tomonori <fujita.tomonori [at] lab>
> >>> Subject: [PATCH] x86: stop swiotlb stealing the GART workaround memory
> >>>
> >>> swiotlb wrongly uses the GART workaround memory (for bad BIOS) that
> >>> dma32_reserv_bootmem allocates. We need to initialize swiotlb before
> >>> dma32_free_bootmem().
> >>>
> >>> Signed-off-by: FUJITA Tomonori <fujita.tomonori [at] lab>
> >>> ---
> >>> arch/x86/kernel/pci-dma.c | 6 +++++-
> >>> 1 files changed, 5 insertions(+), 1 deletions(-)
> >>>
> >>> diff --git a/arch/x86/kernel/pci-dma.c b/arch/x86/kernel/pci-dma.c
> >>> index afcc58b..26fe2cd 100644
> >>> --- a/arch/x86/kernel/pci-dma.c
> >>> +++ b/arch/x86/kernel/pci-dma.c
> >>> @@ -120,11 +120,15 @@ static void __init dma32_free_bootmem(void)
> >>>
> >>> void __init pci_iommu_alloc(void)
> >>> {
> >>> + int use_swiotlb;
> >>> +
> >>> + use_swiotlb = pci_swiotlb_init();
> >>> +
> >>> #ifdef CONFIG_X86_64
> >>> /* free the range so iommu could get some range less than 4G */
> >>> dma32_free_bootmem();
> >>> #endif
> >>> - if (pci_swiotlb_init())
> >>> + if (use_swiotlb)
> >>> return;
> >>>
> >>> gart_iommu_hole_init();
> >> pci_swiotlb_init need be called after dma32_free_bootmem
> >> otherwise it could fail for system with lots of memory and numa=off
> >
> > Why? swiotlb needs just 64MB (and some) in DMA32.
> >
> > swiotlb had been worked fine without that hack until 2.6.26?
> >
> > Only broken GART needs that hack (and I think that it's better to move
> > that hack to GART code).
>
> gart iommu workaround will allocate 64M by default.
>
> fail config: like system with 512g and sparse mem, and vmmap.

Well, some systems could fail but they can use swiotlb. If users
configures their systems not to use swioltb by boot option, it's fair
that systems don't work.

We are talking about a rare broken GART BIOS issue. I think that it's
much better to remove this hack completely. As VT-d does, we can use
swiotlb instead of this GART specific hack. I think that that's saner
approach to handle broken IOMMU hardware.
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo [at] vger
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/

yinghai at kernel

Nov 24, 2009, 2:42 AM

Post #7 of 15 (269 views)
Permalink

Re: [PATCH] x86: fix gart iommu using for amd 64 bit system [In reply to]

FUJITA Tomonori wrote:
> On Tue, 24 Nov 2009 01:48:54 -0800
> Yinghai Lu <yinghai [at] kernel> wrote:
>
>> FUJITA Tomonori wrote:
>>> On Tue, 24 Nov 2009 01:19:21 -0800
>>> Yinghai Lu <yinghai [at] kernel> wrote:
>>>
>>>> FUJITA Tomonori wrote:
>>>>> On Sat, 21 Nov 2009 20:24:52 -0800
>>>>> Yinghai Lu <yinghai [at] kernel> wrote:
>>>>>
>>>>>> amd 64 systems that
>>>>>> 1. do not have AGP
>>>>>> 2. do not have IOMMU
>>>>>> 3. mem > 4g
>>>>>> 4. BIOS do not allocate correct gart in NB.
>>>>>> will leave them to use SWIOTLB forcely.
>>>>> Sorry, I forgot about this GART workaround.
>>>>>
>>>>>
>>>>>> Index: linux-2.6/arch/x86/kernel/pci-dma.c
>>>>>> ===================================================================
>>>>>> --- linux-2.6.orig/arch/x86/kernel/pci-dma.c
>>>>>> +++ linux-2.6/arch/x86/kernel/pci-dma.c
>>>>>> @@ -124,11 +124,12 @@ void __init pci_iommu_alloc(void)
>>>>>> /* free the range so iommu could get some range less than 4G */
>>>>>> dma32_free_bootmem();
>>>>>> #endif
>>>>>> + if (!swiotlb_force)
>>>>>> + gart_iommu_hole_init();
>>>>>> +
>>>>>> if (pci_swiotlb_init())
>>>>>> return;
>>>>>>
>>>>>> - gart_iommu_hole_init();
>>>>>> -
>>>>>> detect_calgary();
>>>>>>
>>>>>> detect_intel_iommu();
>>>>> I prefer to detect all the IOMMU drivers in a consistent way;
>>>>> detecting only GART before swioltb doesn't look nice.
>>>>>
>>>>> As we did before, we could detect all the IOMMU driver before
>>>>> swiotlb. However, I think that it's better to simply change
>>>>> pci_swiotlb_init() not to steal the preallocate GART workaround memory.
>>>>>
>>>>> btw, initializing swiotlb before IOMMU detection is useful to GART
>>>>> too? If GART can't allocate the workaround memory, then the kernel
>>>>> panic now. We can use swiotlb instead in that case?
>>>>>
>>>>> =
>>>>> From: FUJITA Tomonori <fujita.tomonori [at] lab>
>>>>> Subject: [PATCH] x86: stop swiotlb stealing the GART workaround memory
>>>>>
>>>>> swiotlb wrongly uses the GART workaround memory (for bad BIOS) that
>>>>> dma32_reserv_bootmem allocates. We need to initialize swiotlb before
>>>>> dma32_free_bootmem().
>>>>>
>>>>> Signed-off-by: FUJITA Tomonori <fujita.tomonori [at] lab>
>>>>> ---
>>>>> arch/x86/kernel/pci-dma.c | 6 +++++-
>>>>> 1 files changed, 5 insertions(+), 1 deletions(-)
>>>>>
>>>>> diff --git a/arch/x86/kernel/pci-dma.c b/arch/x86/kernel/pci-dma.c
>>>>> index afcc58b..26fe2cd 100644
>>>>> --- a/arch/x86/kernel/pci-dma.c
>>>>> +++ b/arch/x86/kernel/pci-dma.c
>>>>> @@ -120,11 +120,15 @@ static void __init dma32_free_bootmem(void)
>>>>>
>>>>> void __init pci_iommu_alloc(void)
>>>>> {
>>>>> + int use_swiotlb;
>>>>> +
>>>>> + use_swiotlb = pci_swiotlb_init();
>>>>> +
>>>>> #ifdef CONFIG_X86_64
>>>>> /* free the range so iommu could get some range less than 4G */
>>>>> dma32_free_bootmem();
>>>>> #endif
>>>>> - if (pci_swiotlb_init())
>>>>> + if (use_swiotlb)
>>>>> return;
>>>>>
>>>>> gart_iommu_hole_init();
>>>> pci_swiotlb_init need be called after dma32_free_bootmem
>>>> otherwise it could fail for system with lots of memory and numa=off
>>> Why? swiotlb needs just 64MB (and some) in DMA32.
>>>
>>> swiotlb had been worked fine without that hack until 2.6.26?
>>>
>>> Only broken GART needs that hack (and I think that it's better to move
>>> that hack to GART code).
>> gart iommu workaround will allocate 64M by default.
>>
>> fail config: like system with 512g and sparse mem, and vmmap.
>
> Well, some systems could fail but they can use swiotlb. If users
> configures their systems not to use swioltb by boot option, it's fair
> that systems don't work.
i mean when 512g and numa = off, even swiotlb could have chance not to get that 64M ram
>
> We are talking about a rare broken GART BIOS issue. I think that it's
> much better to remove this hack completely. As VT-d does, we can use
> swiotlb instead of this GART specific hack. I think that that's saner
> approach to handle broken IOMMU hardware.

your second patch could break swiotlb...

at GART iommu workaround : it is not rare. a lot of systems have this problem.
and that workaround works well for several years.
don't think user will like to use SWIOTLB instead of swiotlb.

that gart iommu hardware is not broken, just those BIOS guys just forget to program it.

here intel vt-d is different, it seems can not make it work just program some hardware register...

please check my v2 patch, to see if it breaks your setup.

YH

--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo [at] vger
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/

fujita.tomonori at lab

Nov 24, 2009, 5:50 AM

Post #8 of 15 (266 views)
Permalink

Re: [PATCH] x86: fix gart iommu using for amd 64 bit system [In reply to]

On Tue, 24 Nov 2009 02:42:26 -0800
Yinghai Lu <yinghai [at] kernel> wrote:

> at GART iommu workaround : it is not rare. a lot of systems have this problem.
> and that workaround works well for several years.

swiotlb has worked too.

> don't think user will like to use SWIOTLB instead of swiotlb.

I doubt that users care about a way to fix their problems.

> that gart iommu hardware is not broken, just those BIOS guys just forget to program it.
>
> here intel vt-d is different, it seems can not make it work just program some hardware register...

Because BIOS is broken, IIRC. Both cases are pretty similar.

> please check my v2 patch, to see if it breaks your setup.

As I said, adding another trick only for GART is not good. And using
force_iommu in pci-dma.c is confusing since force_iommu should be used
only in X86_64.

The following patch works for you?

diff --git a/arch/x86/include/asm/swiotlb.h b/arch/x86/include/asm/swiotlb.h
index 87ffcb1..8085277 100644
--- a/arch/x86/include/asm/swiotlb.h
+++ b/arch/x86/include/asm/swiotlb.h
@@ -5,13 +5,17 @@

#ifdef CONFIG_SWIOTLB
extern int swiotlb;
-extern int pci_swiotlb_init(void);
+extern int __init pci_swiotlb_detect(void);
+extern void __init pci_swiotlb_init(void);
#else
#define swiotlb 0
-static inline int pci_swiotlb_init(void)
+static inline int pci_swiotlb_detect(void)
{
return 0;
}
+static inline void pci_swiotlb_init(void)
+{
+}
#endif

static inline void dma_mark_clean(void *addr, size_t size) {}
diff --git a/arch/x86/kernel/aperture_64.c b/arch/x86/kernel/aperture_64.c
index e0dfb68..750dd8d 100644
--- a/arch/x86/kernel/aperture_64.c
+++ b/arch/x86/kernel/aperture_64.c
@@ -401,6 +401,7 @@ void __init gart_iommu_hole_init(void)

iommu_detected = 1;
gart_iommu_aperture = 1;
+ swiotlb = 0;
x86_init.iommu.iommu_init = gart_iommu_init;

aper_order = (read_pci_config(bus, slot, 3, AMD64_GARTAPERTURECTL) >> 1) & 7;
@@ -458,7 +459,7 @@ out:

if (aper_alloc) {
/* Got the aperture from the AGP bridge */
- } else if (!valid_agp) {
+ } else if (swiotlb && !valid_agp) {
/* Do nothing */
} else if ((!no_iommu && max_pfn > MAX_DMA32_PFN) ||
force_iommu ||
diff --git a/arch/x86/kernel/pci-dma.c b/arch/x86/kernel/pci-dma.c
index afcc58b..75e14e2 100644
--- a/arch/x86/kernel/pci-dma.c
+++ b/arch/x86/kernel/pci-dma.c
@@ -124,8 +124,8 @@ void __init pci_iommu_alloc(void)
/* free the range so iommu could get some range less than 4G */
dma32_free_bootmem();
#endif
- if (pci_swiotlb_init())
- return;
+ if (pci_swiotlb_detect())
+ goto out;

gart_iommu_hole_init();

@@ -135,6 +135,8 @@ void __init pci_iommu_alloc(void)

/* needs to be called after gart_iommu_hole_init */
amd_iommu_detect();
+out:
+ pci_swiotlb_init();
}

void *dma_generic_alloc_coherent(struct device *dev, size_t size,
diff --git a/arch/x86/kernel/pci-gart_64.c b/arch/x86/kernel/pci-gart_64.c
index e6a0d40..2358409 100644
--- a/arch/x86/kernel/pci-gart_64.c
+++ b/arch/x86/kernel/pci-gart_64.c
@@ -847,7 +847,6 @@ int __init gart_iommu_init(void)
flush_gart();
dma_ops = &gart_dma_ops;
x86_platform.iommu_shutdown = gart_iommu_shutdown;
- swiotlb = 0;

return 0;
}
diff --git a/arch/x86/kernel/pci-swiotlb.c b/arch/x86/kernel/pci-swiotlb.c
index e36e71d..6971ba5 100644
--- a/arch/x86/kernel/pci-swiotlb.c
+++ b/arch/x86/kernel/pci-swiotlb.c
@@ -43,12 +43,12 @@ static struct dma_map_ops swiotlb_dma_ops = {
};

/*
- * pci_swiotlb_init - initialize swiotlb if necessary
+ * pci_swiotlb_init - set swiotlb to 1 if necessary
*
* This returns non-zero if we are forced to use swiotlb (by the boot
* option).
*/
-int __init pci_swiotlb_init(void)
+int __init pci_swiotlb_detect(void)
{
/* don't initialize swiotlb if iommu=off (no_iommu=1) */
#ifdef CONFIG_X86_64
@@ -58,10 +58,13 @@ int __init pci_swiotlb_init(void)
if (swiotlb_force)
swiotlb = 1;

+ return swiotlb_force;
+}
+
+void __init pci_swiotlb_init(void)
+{
if (swiotlb) {
swiotlb_init(0);
dma_ops = &swiotlb_dma_ops;
}
-
- return swiotlb_force;
}
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo [at] vger
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/

mingo at elte

Nov 24, 2009, 7:26 AM

Post #9 of 15 (258 views)
Permalink

Re: [PATCH] x86: fix gart iommu using for amd 64 bit system [In reply to]

* FUJITA Tomonori <fujita.tomonori [at] lab> wrote:

> > don't think user will like to use SWIOTLB instead of swiotlb.
>
> I doubt that users care about a way to fix their problems.

As far as swiotlb vs. GART goes, arguably a non-bouncing hw based
solution (GART) is superior to a bouncing one (swiotlb), right?

ngo
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo [at] vger
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/

yinghai at kernel

Nov 24, 2009, 10:28 AM

Post #10 of 15 (256 views)
Permalink

Re: [PATCH] x86: fix gart iommu using for amd 64 bit system [In reply to]

Ingo Molnar wrote:
> * FUJITA Tomonori <fujita.tomonori [at] lab> wrote:
>
>>> don't think user will like to use SWIOTLB instead of swiotlb.
>> I doubt that users care about a way to fix their problems.
>
> As far as swiotlb vs. GART goes, arguably a non-bouncing hw based
> solution (GART) is superior to a bouncing one (swiotlb), right?
>

yes, should use GART if possible.

YH
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo [at] vger
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/

yinghai at kernel

Nov 24, 2009, 10:48 AM

Post #11 of 15 (254 views)
Permalink

Re: [PATCH] x86: fix gart iommu using for amd 64 bit system [In reply to]

FUJITA Tomonori wrote:
> On Tue, 24 Nov 2009 02:42:26 -0800
> Yinghai Lu <yinghai [at] kernel> wrote:
>
>> at GART iommu workaround : it is not rare. a lot of systems have this problem.
>> and that workaround works well for several years.
>
> swiotlb has worked too.
>
>
>> don't think user will like to use SWIOTLB instead of swiotlb.
>
> I doubt that users care about a way to fix their problems.
>
>
>> that gart iommu hardware is not broken, just those BIOS guys just forget to program it.
>>
>> here intel vt-d is different, it seems can not make it work just program some hardware register...
>
> Because BIOS is broken, IIRC. Both cases are pretty similar.
>
>> please check my v2 patch, to see if it breaks your setup.
>
> As I said, adding another trick only for GART is not good. And using
> force_iommu in pci-dma.c is confusing since force_iommu should be used
> only in X86_64.
>
> The following patch works for you?
>
>
> diff --git a/arch/x86/include/asm/swiotlb.h b/arch/x86/include/asm/swiotlb.h
> index 87ffcb1..8085277 100644
> --- a/arch/x86/include/asm/swiotlb.h
> +++ b/arch/x86/include/asm/swiotlb.h
> @@ -5,13 +5,17 @@
>
> #ifdef CONFIG_SWIOTLB
> extern int swiotlb;
> -extern int pci_swiotlb_init(void);
> +extern int __init pci_swiotlb_detect(void);
> +extern void __init pci_swiotlb_init(void);
> #else
> #define swiotlb 0
> -static inline int pci_swiotlb_init(void)
> +static inline int pci_swiotlb_detect(void)
> {
> return 0;
> }
> +static inline void pci_swiotlb_init(void)
> +{
> +}
> #endif
>
> static inline void dma_mark_clean(void *addr, size_t size) {}
> diff --git a/arch/x86/kernel/aperture_64.c b/arch/x86/kernel/aperture_64.c
> index e0dfb68..750dd8d 100644
> --- a/arch/x86/kernel/aperture_64.c
> +++ b/arch/x86/kernel/aperture_64.c
> @@ -401,6 +401,7 @@ void __init gart_iommu_hole_init(void)
>
> iommu_detected = 1;
> gart_iommu_aperture = 1;
> + swiotlb = 0;
> x86_init.iommu.iommu_init = gart_iommu_init;
>
> aper_order = (read_pci_config(bus, slot, 3, AMD64_GARTAPERTURECTL) >> 1) & 7;
> @@ -458,7 +459,7 @@ out:
>
> if (aper_alloc) {
> /* Got the aperture from the AGP bridge */
> - } else if (!valid_agp) {
> + } else if (swiotlb && !valid_agp) {
> /* Do nothing */

still have some problem here.
because you set swiotlb to 0 before. it will break
iommu=soft

when amd 64bit system, ram > 4g, and no AGP bridge, and BIOS doesn't set gart.
if the user set iommu=soft.
old way, kernel will use swiotlb.

so instead set swiotlb before, it seems you should restore
iommu_detected
gart_iommu_aperture
swiotlb = 0;
x86_init.iommu.iommu_init
to make swiotlb happy later.

> } else if ((!no_iommu && max_pfn > MAX_DMA32_PFN) ||
> force_iommu ||
> diff --git a/arch/x86/kernel/pci-dma.c b/arch/x86/kernel/pci-dma.c
> index afcc58b..75e14e2 100644
> --- a/arch/x86/kernel/pci-dma.c
> +++ b/arch/x86/kernel/pci-dma.c
> @@ -124,8 +124,8 @@ void __init pci_iommu_alloc(void)
> /* free the range so iommu could get some range less than 4G */
> dma32_free_bootmem();
> #endif
> - if (pci_swiotlb_init())
> - return;
> + if (pci_swiotlb_detect())
> + goto out;
>
> gart_iommu_hole_init();
>
> @@ -135,6 +135,8 @@ void __init pci_iommu_alloc(void)
>
> /* needs to be called after gart_iommu_hole_init */
> amd_iommu_detect();
> +out:
> + pci_swiotlb_init();
> }
>
> void *dma_generic_alloc_coherent(struct device *dev, size_t size,
> diff --git a/arch/x86/kernel/pci-gart_64.c b/arch/x86/kernel/pci-gart_64.c
> index e6a0d40..2358409 100644
> --- a/arch/x86/kernel/pci-gart_64.c
> +++ b/arch/x86/kernel/pci-gart_64.c
> @@ -847,7 +847,6 @@ int __init gart_iommu_init(void)
> flush_gart();
> dma_ops = &gart_dma_ops;
> x86_platform.iommu_shutdown = gart_iommu_shutdown;
> - swiotlb = 0;
>
> return 0;
> }
> diff --git a/arch/x86/kernel/pci-swiotlb.c b/arch/x86/kernel/pci-swiotlb.c
> index e36e71d..6971ba5 100644
> --- a/arch/x86/kernel/pci-swiotlb.c
> +++ b/arch/x86/kernel/pci-swiotlb.c
> @@ -43,12 +43,12 @@ static struct dma_map_ops swiotlb_dma_ops = {
> };
>
> /*
> - * pci_swiotlb_init - initialize swiotlb if necessary
> + * pci_swiotlb_init - set swiotlb to 1 if necessary

pci_swiotlb_detect ?

YH
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo [at] vger
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/

fujita.tomonori at lab

Nov 24, 2009, 3:48 PM

Post #12 of 15 (256 views)
Permalink

Re: [PATCH] x86: fix gart iommu using for amd 64 bit system [In reply to]

On Tue, 24 Nov 2009 10:48:32 -0800
Yinghai Lu <yinghai [at] kernel> wrote:

> > if (aper_alloc) {
> > /* Got the aperture from the AGP bridge */
> > - } else if (!valid_agp) {
> > + } else if (swiotlb && !valid_agp) {
> > /* Do nothing */
>
> still have some problem here.
> because you set swiotlb to 0 before. it will break
> iommu=soft

Oops, thanks. However, it's not GART specific issue.

'iommu=soft' is already broken on all the configurations. It should be
fixed separately. I've just sent a fix.

> > void *dma_generic_alloc_coherent(struct device *dev, size_t size,
> > diff --git a/arch/x86/kernel/pci-gart_64.c b/arch/x86/kernel/pci-gart_64.c
> > index e6a0d40..2358409 100644
> > --- a/arch/x86/kernel/pci-gart_64.c
> > +++ b/arch/x86/kernel/pci-gart_64.c
> > @@ -847,7 +847,6 @@ int __init gart_iommu_init(void)
> > flush_gart();
> > dma_ops = &gart_dma_ops;
> > x86_platform.iommu_shutdown = gart_iommu_shutdown;
> > - swiotlb = 0;
> >
> > return 0;
> > }
> > diff --git a/arch/x86/kernel/pci-swiotlb.c b/arch/x86/kernel/pci-swiotlb.c
> > index e36e71d..6971ba5 100644
> > --- a/arch/x86/kernel/pci-swiotlb.c
> > +++ b/arch/x86/kernel/pci-swiotlb.c
> > @@ -43,12 +43,12 @@ static struct dma_map_ops swiotlb_dma_ops = {
> > };
> >
> > /*
> > - * pci_swiotlb_init - initialize swiotlb if necessary
> > + * pci_swiotlb_init - set swiotlb to 1 if necessary
>
> pci_swiotlb_detect ?

I'll fix in the next version.
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo [at] vger
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/

fujita.tomonori at lab

Nov 24, 2009, 3:57 PM

Post #13 of 15 (256 views)
Permalink

Re: [PATCH] x86: fix gart iommu using for amd 64 bit system [In reply to]

On Tue, 24 Nov 2009 16:26:45 +0100
Ingo Molnar <mingo [at] elte> wrote:

>
> * FUJITA Tomonori <fujita.tomonori [at] lab> wrote:
>
> > > don't think user will like to use SWIOTLB instead of swiotlb.
> >
> > I doubt that users care about a way to fix their problems.
>
> As far as swiotlb vs. GART goes, arguably a non-bouncing hw based
> solution (GART) is superior to a bouncing one (swiotlb), right?

Yeah, but GART is not a real IOMMU that always does address
translation. If users want performance, they need 64bit DMA capable
devices to skip GART. With such devices, GART and SWIOTLB are same.

So this hacky GART workaround is useful for only people who use non
64bit DMA devices and broken BIOS with GART. IMO, it's not worth
having the workaround.
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo [at] vger
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/

mingo at elte

Nov 24, 2009, 11:25 PM

Post #14 of 15 (249 views)
Permalink

Re: [PATCH] x86: fix gart iommu using for amd 64 bit system [In reply to]

* FUJITA Tomonori <fujita.tomonori [at] lab> wrote:

> On Tue, 24 Nov 2009 16:26:45 +0100
> Ingo Molnar <mingo [at] elte> wrote:
>
> >
> > * FUJITA Tomonori <fujita.tomonori [at] lab> wrote:
> >
> > > > don't think user will like to use SWIOTLB instead of swiotlb.
> > >
> > > I doubt that users care about a way to fix their problems.
> >
> > As far as swiotlb vs. GART goes, arguably a non-bouncing hw based
> > solution (GART) is superior to a bouncing one (swiotlb), right?
>
> Yeah, but GART is not a real IOMMU that always does address
> translation. If users want performance, they need 64bit DMA capable
> devices to skip GART. With such devices, GART and SWIOTLB are same.
>
> So this hacky GART workaround is useful for only people who use non
> 64bit DMA devices and broken BIOS with GART. IMO, it's not worth
> having the workaround.

Well, what Yinghai is saying that for that (nonzero) subset of people
this change is a regression, and his patch fixes that. We dont do
regressions so this needs to be addressed.

Thanks,

Ingo
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo [at] vger
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/

fujita.tomonori at lab

Nov 24, 2009, 11:56 PM

Post #15 of 15 (247 views)
Permalink

Re: [PATCH] x86: fix gart iommu using for amd 64 bit system [In reply to]

On Wed, 25 Nov 2009 08:25:04 +0100
Ingo Molnar <mingo [at] elte> wrote:

>
> * FUJITA Tomonori <fujita.tomonori [at] lab> wrote:
>
> > On Tue, 24 Nov 2009 16:26:45 +0100
> > Ingo Molnar <mingo [at] elte> wrote:
> >
> > >
> > > * FUJITA Tomonori <fujita.tomonori [at] lab> wrote:
> > >
> > > > > don't think user will like to use SWIOTLB instead of swiotlb.
> > > >
> > > > I doubt that users care about a way to fix their problems.
> > >
> > > As far as swiotlb vs. GART goes, arguably a non-bouncing hw based
> > > solution (GART) is superior to a bouncing one (swiotlb), right?
> >
> > Yeah, but GART is not a real IOMMU that always does address
> > translation. If users want performance, they need 64bit DMA capable
> > devices to skip GART. With such devices, GART and SWIOTLB are same.
> >
> > So this hacky GART workaround is useful for only people who use non
> > 64bit DMA devices and broken BIOS with GART. IMO, it's not worth
> > having the workaround.
>
> Well, what Yinghai is saying that for that (nonzero) subset of people
> this change is a regression, and his patch fixes that. We dont do
> regressions so this needs to be addressed.

OK, I guess that I have to wait until nobody cares about GART.

Anyway, my patch fixes his issue without any regression:

http://marc.info/?l=linux-kernel&m=125907069703595&w=2

I'll post the second version of it after you merge the following fix:

http://marc.info/?l=linux-kernel&m=125910650500523&w=2

Thanks,
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo [at] vger
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/

Index | Next | Previous | View Threaded

Interested in having your list archived? Contact Gossamer Threads