Mailing List Archive: Linux: Kernel

[PATCH 01/34] AMD IOMMU: add Kconfig entry

1 2

View All

Index | Next | Previous | View Threaded

joerg.roedel at amd

Jun 26, 2008, 12:27 PM

Post #1 of 29 (324 views)
Permalink

[PATCH 01/34] AMD IOMMU: add Kconfig entry

This patch adds the Kconfig entry for the AMD IOMMU driver.

Signed-off-by: Joerg Roedel <joerg.roedel[at]amd.com>
---
arch/x86/Kconfig | 7 +++++++
1 files changed, 7 insertions(+), 0 deletions(-)

diff --git a/arch/x86/Kconfig b/arch/x86/Kconfig
index e0edaaa..5a82f18 100644
--- a/arch/x86/Kconfig
+++ b/arch/x86/Kconfig
@@ -549,6 +549,13 @@ config CALGARY_IOMMU_ENABLED_BY_DEFAULT
Calgary anyway, pass 'iommu=calgary' on the kernel command line.
If unsure, say Y.

+config AMD_IOMMU
+ bool "AMD IOMMU support"
+ select SWIOTL
+ depends on X86_64 && PCI
+ help
+ Select this to get support for AMD IOMMU hardware in your system.
+
# need this always selected by IOMMU for the VIA workaround
config SWIOTLB
bool
--
1.5.3.7

--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo[at]vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/

bunk at kernel

Jun 27, 2008, 7:25 AM

Post #2 of 29 (313 views)
Permalink

Re: [PATCH 01/34] AMD IOMMU: add Kconfig entry [In reply to]

On Thu, Jun 26, 2008 at 09:27:37PM +0200, Joerg Roedel wrote:
> This patch adds the Kconfig entry for the AMD IOMMU driver.
>
> Signed-off-by: Joerg Roedel <joerg.roedel[at]amd.com>
> ---
> arch/x86/Kconfig | 7 +++++++
> 1 files changed, 7 insertions(+), 0 deletions(-)
>
> diff --git a/arch/x86/Kconfig b/arch/x86/Kconfig
> index e0edaaa..5a82f18 100644
> --- a/arch/x86/Kconfig
> +++ b/arch/x86/Kconfig
> @@ -549,6 +549,13 @@ config CALGARY_IOMMU_ENABLED_BY_DEFAULT
> Calgary anyway, pass 'iommu=calgary' on the kernel command line.
> If unsure, say Y.
>
> +config AMD_IOMMU
> + bool "AMD IOMMU support"
> + select SWIOTL
> + depends on X86_64 && PCI
> + help
> + Select this to get support for AMD IOMMU hardware in your system.
> +
>...

This tells neither what as IOMMU is nor whether my system actually has
one.

Please shortly describe in the help text what an IOMMU is and which
AMD systems have an IOMMU.

The goal is that a system administrator building a kernel for his system
gets enough information for deciding whether to enable or disable this
option.

Thanks
Adrian

--

"Is there not promise of rain?" Ling Tan asked suddenly out
of the darkness. There had been need of rain for many days.
"Only a promise," Lao Er said.
Pearl S. Buck - Dragon Seed

--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo[at]vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/

andi at firstfloor

Jun 27, 2008, 7:47 AM

Post #3 of 29 (313 views)
Permalink

Re: [PATCH 01/34] AMD IOMMU: add Kconfig entry [In reply to]

Adrian Bunk <bunk[at]kernel.org> writes:
>
> This tells neither what as IOMMU is nor whether my system actually has
> one.
>
> Please shortly describe in the help text what an IOMMU is and which
> AMD systems have an IOMMU.
>
> The goal is that a system administrator building a kernel for his system
> gets enough information for deciding whether to enable or disable this
> option.

Also it should ideally describe that there is a trade off between
reliability and performance with IOMMU enabled.

-Andi
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo[at]vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/

muli at il

Jun 27, 2008, 9:39 AM

Post #4 of 29 (310 views)
Permalink

Re: [PATCH 01/34] AMD IOMMU: add Kconfig entry [In reply to]

On Fri, Jun 27, 2008 at 04:47:29PM +0200, Andi Kleen wrote:

> Also it should ideally describe that there is a trade off between
> reliability and performance with IOMMU enabled.

I agree that there's a performance tradeoff with current generation
hardware and ingerfaces, but it wouldn't be fair to single out a
single IOMMU implementation ;=)

config DMAR
bool "Support for DMA Remapping Devices (EXPERIMENTAL)"
depends on X86_64 && PCI_MSI && ACPI && EXPERIMENTAL
help
DMA remapping (DMAR) devices support enables independent address
translations for Direct Memory Access (DMA) from devices.
These DMA remapping devices are reported via ACPI tables
and include PCI device scope covered by these DMA
remapping devices.

Cheers,
Muli
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo[at]vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/

joro at 8bytes

Jun 27, 2008, 9:40 AM

Post #5 of 29 (310 views)
Permalink

Re: [PATCH 01/34] AMD IOMMU: add Kconfig entry [In reply to]

On Fri, Jun 27, 2008 at 05:25:58PM +0300, Adrian Bunk wrote:
> On Thu, Jun 26, 2008 at 09:27:37PM +0200, Joerg Roedel wrote:
> > This patch adds the Kconfig entry for the AMD IOMMU driver.
> >
> > Signed-off-by: Joerg Roedel <joerg.roedel[at]amd.com>
> > ---
> > arch/x86/Kconfig | 7 +++++++
> > 1 files changed, 7 insertions(+), 0 deletions(-)
> >
> > diff --git a/arch/x86/Kconfig b/arch/x86/Kconfig
> > index e0edaaa..5a82f18 100644
> > --- a/arch/x86/Kconfig
> > +++ b/arch/x86/Kconfig
> > @@ -549,6 +549,13 @@ config CALGARY_IOMMU_ENABLED_BY_DEFAULT
> > Calgary anyway, pass 'iommu=calgary' on the kernel command line.
> > If unsure, say Y.
> >
> > +config AMD_IOMMU
> > + bool "AMD IOMMU support"
> > + select SWIOTL
> > + depends on X86_64 && PCI
> > + help
> > + Select this to get support for AMD IOMMU hardware in your system.
> > +
> >...
>
> This tells neither what as IOMMU is nor whether my system actually has
> one.
>
> Please shortly describe in the help text what an IOMMU is and which
> AMD systems have an IOMMU.
>
> The goal is that a system administrator building a kernel for his system
> gets enough information for deciding whether to enable or disable this
> option.

Ok, I will provide a new help text as an update to this patch series.

Joerg
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo[at]vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/

joro at 8bytes

Jun 27, 2008, 9:54 AM

Post #6 of 29 (310 views)
Permalink

Re: [PATCH 01/34] AMD IOMMU: add Kconfig entry [In reply to]

On Fri, Jun 27, 2008 at 12:39:45PM -0400, Muli Ben-Yehuda wrote:
> On Fri, Jun 27, 2008 at 04:47:29PM +0200, Andi Kleen wrote:
>
> > Also it should ideally describe that there is a trade off between
> > reliability and performance with IOMMU enabled.
>
> I agree that there's a performance tradeoff with current generation
> hardware and ingerfaces, but it wouldn't be fair to single out a
> single IOMMU implementation ;=)

True. At least for the case without device isolation I have some
optimizations in mind which will minimize the performance tradeoff. I
hope to have them ready for 2.6.28 :)

Joerg.

--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo[at]vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/

muli at il

Jun 27, 2008, 9:59 AM

Post #7 of 29 (310 views)
Permalink

Re: [PATCH 01/34] AMD IOMMU: add Kconfig entry [In reply to]

On Fri, Jun 27, 2008 at 06:54:30PM +0200, Joerg Roedel wrote:

> True. At least for the case without device isolation I have some
> optimizations in mind which will minimize the performance
> tradeoff. I hope to have them ready for 2.6.28 :)

Do you mean the case where you have a single I/O address space which
is shared by all devices?

Cheers,
Muli
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo[at]vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/

joro at 8bytes

Jun 27, 2008, 10:05 AM

Post #8 of 29 (310 views)
Permalink

Re: [PATCH 01/34] AMD IOMMU: add Kconfig entry [In reply to]

On Fri, Jun 27, 2008 at 12:59:47PM -0400, Muli Ben-Yehuda wrote:
> On Fri, Jun 27, 2008 at 06:54:30PM +0200, Joerg Roedel wrote:
>
> > True. At least for the case without device isolation I have some
> > optimizations in mind which will minimize the performance
> > tradeoff. I hope to have them ready for 2.6.28 :)
>
> Do you mean the case where you have a single I/O address space which
> is shared by all devices?

Yes. I think this will be the case used most when IOMMU is used for
virtualization and to handle devices with limited DMA address ranges. In
this case there is a lot to optimize.

Joerg

--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo[at]vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/

muli at il

Jun 27, 2008, 10:12 AM

Post #9 of 29 (310 views)
Permalink

Re: [PATCH 01/34] AMD IOMMU: add Kconfig entry [In reply to]

On Fri, Jun 27, 2008 at 07:05:46PM +0200, Joerg Roedel wrote:
> On Fri, Jun 27, 2008 at 12:59:47PM -0400, Muli Ben-Yehuda wrote:
> > On Fri, Jun 27, 2008 at 06:54:30PM +0200, Joerg Roedel wrote:
> >
> > > True. At least for the case without device isolation I have some
> > > optimizations in mind which will minimize the performance
> > > tradeoff. I hope to have them ready for 2.6.28 :)
> >
> > Do you mean the case where you have a single I/O address space which
> > is shared by all devices?
>
> Yes. I think this will be the case used most when IOMMU is used for
> virtualization

Could you elaborate on what you mean here? I assume you're thinking
one I/O address space for the host, and one I/O address space per
guest with assigned devices?

> and to handle devices with limited DMA address ranges.

I'd be pretty surprised if you'll find such devices on machines which
will have AMD's IOMMU...

> In this case there is a lot to optimize.

Looking forward to seeing what you have in mind :-)

Cheers,
Muli

--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo[at]vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/

joro at 8bytes

Jun 27, 2008, 10:20 AM

Post #10 of 29 (300 views)
Permalink

Re: [PATCH 01/34] AMD IOMMU: add Kconfig entry [In reply to]

On Fri, Jun 27, 2008 at 01:12:01PM -0400, Muli Ben-Yehuda wrote:
> On Fri, Jun 27, 2008 at 07:05:46PM +0200, Joerg Roedel wrote:
> > On Fri, Jun 27, 2008 at 12:59:47PM -0400, Muli Ben-Yehuda wrote:
> > > On Fri, Jun 27, 2008 at 06:54:30PM +0200, Joerg Roedel wrote:
> > >
> > > > True. At least for the case without device isolation I have some
> > > > optimizations in mind which will minimize the performance
> > > > tradeoff. I hope to have them ready for 2.6.28 :)
> > >
> > > Do you mean the case where you have a single I/O address space which
> > > is shared by all devices?
> >
> > Yes. I think this will be the case used most when IOMMU is used for
> > virtualization
>
> Could you elaborate on what you mean here? I assume you're thinking
> one I/O address space for the host, and one I/O address space per
> guest with assigned devices?

I think we can create an address space which almost direct-maps the
physical memory and let some room free for the aperture at the beginning
(say 64MB). If a mapping request arrives the code looks if it has to do
mapping (physical address of memory to map is in the first 64MB or
not in the device address range). If this is not the case it simply
returns the physical address as dma_addr. otherwise it does the
expensive mapping. This way we could minimize the default overhead which
we will get with an IOMMU and still use it for virtualization and as a
GART replacement.

> > and to handle devices with limited DMA address ranges.
>
> I'd be pretty surprised if you'll find such devices on machines which
> will have AMD's IOMMU...

Think of 32bit PCI devices in a host with more than 4GB memory :)

Cheers,

Joerg
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo[at]vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/

muli at il

Jun 27, 2008, 10:31 AM

Post #11 of 29 (301 views)
Permalink

Re: [PATCH 01/34] AMD IOMMU: add Kconfig entry [In reply to]

On Fri, Jun 27, 2008 at 07:20:30PM +0200, Joerg Roedel wrote:

> > Could you elaborate on what you mean here? I assume you're
> > thinking one I/O address space for the host, and one I/O address
> > space per guest with assigned devices?
>
> I think we can create an address space which almost direct-maps the
> physical memory and let some room free for the aperture at the
> beginning (say 64MB). If a mapping request arrives the code looks if
> it has to do mapping (physical address of memory to map is in the
> first 64MB or not in the device address range). If this is not the
> case it simply returns the physical address as dma_addr. otherwise
> it does the expensive mapping. This way we could minimize the
> default overhead which we will get with an IOMMU and still use it
> for virtualization and as a GART replacement.

What you are suggesting is an "almost-direct-map" approach for the
host I/O address space, which provides no protection from mis-behaving
host drivers. If we could avoid needing a GART replacement (see below
for why I think we could), you could simply avoid enabling translation
for host devices and be done with it.

In my humble opinion it's more interesting to try and figure out how
to get protection from mis-behaving host drivers while still keeping
performance as close as possible to native.

> > > and to handle devices with limited DMA address ranges.
> >
> > I'd be pretty surprised if you'll find such devices on machines which
> > will have AMD's IOMMU...
>
> Think of 32bit PCI devices in a host with more than 4GB memory :)

I am thinking of them and I'd be surprised if you'd find any in such
machines. Certainly I assume none of the on-board devices will have
this ancient limitation. But hey, it could happen ;-)

Cheers,
Muli
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo[at]vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/

joro at 8bytes

Jun 27, 2008, 10:40 AM

Post #12 of 29 (300 views)
Permalink

Re: [PATCH 01/34] AMD IOMMU: add Kconfig entry [In reply to]

On Fri, Jun 27, 2008 at 01:31:00PM -0400, Muli Ben-Yehuda wrote:
> On Fri, Jun 27, 2008 at 07:20:30PM +0200, Joerg Roedel wrote:
>
> > > Could you elaborate on what you mean here? I assume you're
> > > thinking one I/O address space for the host, and one I/O address
> > > space per guest with assigned devices?
> >
> > I think we can create an address space which almost direct-maps the
> > physical memory and let some room free for the aperture at the
> > beginning (say 64MB). If a mapping request arrives the code looks if
> > it has to do mapping (physical address of memory to map is in the
> > first 64MB or not in the device address range). If this is not the
> > case it simply returns the physical address as dma_addr. otherwise
> > it does the expensive mapping. This way we could minimize the
> > default overhead which we will get with an IOMMU and still use it
> > for virtualization and as a GART replacement.
>
> What you are suggesting is an "almost-direct-map" approach for the
> host I/O address space, which provides no protection from mis-behaving
> host drivers. If we could avoid needing a GART replacement (see below
> for why I think we could), you could simply avoid enabling translation
> for host devices and be done with it.

Yes. As I said, this is for the non-isolating case. For the isolation
case (which is needed for protection) it is harder to optimize. But
there I think about some sort of lazy IOMMU TLB flushing. The flushing
and 'wait for the flush to finish' is the most expensive part in the
mapping and unmapping code path. But this needs some experiments.

> In my humble opinion it's more interesting to try and figure out how
> to get protection from mis-behaving host drivers while still keeping
> performance as close as possible to native.

True. But I also see IOMMU as an device usable to pass devices to
virtualization guests. In this case you don't necessarily want device
isolation in the host (for devices only the host uses). So the
optimization for the non-isolation case is also important imho.

> > > > and to handle devices with limited DMA address ranges.
> > >
> > > I'd be pretty surprised if you'll find such devices on machines which
> > > will have AMD's IOMMU...
> >
> > Think of 32bit PCI devices in a host with more than 4GB memory :)
>
> I am thinking of them and I'd be surprised if you'd find any in such
> machines. Certainly I assume none of the on-board devices will have
> this ancient limitation. But hey, it could happen ;-)

The IOMMU machine under my desk has a 32bit PCI slot with a card in it
:-)

Cheers,

Joerg

--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo[at]vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/

muli at il

Jun 27, 2008, 10:44 AM

Post #13 of 29 (301 views)
Permalink

Re: [PATCH 01/34] AMD IOMMU: add Kconfig entry [In reply to]

On Fri, Jun 27, 2008 at 07:40:35PM +0200, Joerg Roedel wrote:

> Yes. As I said, this is for the non-isolating case. For the
> isolation case (which is needed for protection) it is harder to
> optimize. But there I think about some sort of lazy IOMMU TLB
> flushing. The flushing and 'wait for the flush to finish' is the
> most expensive part in the mapping and unmapping code path. But this
> needs some experiments.

Agreed. The paper I just posted to the iommu mailing list has some
ideas and experimental data.

> > I am thinking of them and I'd be surprised if you'd find any in
> > such machines. Certainly I assume none of the on-board devices
> > will have this ancient limitation. But hey, it could happen ;-)
>
> The IOMMU machine under my desk has a 32bit PCI slot with a card in
> it :-)

Touche :-)

Cheers,
Muli
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo[at]vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/

joro at 8bytes

Jun 27, 2008, 10:52 AM

Post #14 of 29 (301 views)
Permalink

Re: [PATCH 01/34] AMD IOMMU: add Kconfig entry [In reply to]

On Fri, Jun 27, 2008 at 01:44:15PM -0400, Muli Ben-Yehuda wrote:
> On Fri, Jun 27, 2008 at 07:40:35PM +0200, Joerg Roedel wrote:
>
> > Yes. As I said, this is for the non-isolating case. For the
> > isolation case (which is needed for protection) it is harder to
> > optimize. But there I think about some sort of lazy IOMMU TLB
> > flushing. The flushing and 'wait for the flush to finish' is the
> > most expensive part in the mapping and unmapping code path. But this
> > needs some experiments.
>
> Agreed. The paper I just posted to the iommu mailing list has some
> ideas and experimental data.

Cool. I wasn't aware if that paper. I will read it next week. Thanks for
the pointer to it.

Joerg

--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo[at]vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/

andi at firstfloor

Jun 27, 2008, 11:54 AM

Post #15 of 29 (301 views)
Permalink

Re: [PATCH 01/34] AMD IOMMU: add Kconfig entry [In reply to]

Joerg Roedel <joro[at]8bytes.org> writes:

> On Fri, Jun 27, 2008 at 12:39:45PM -0400, Muli Ben-Yehuda wrote:
>> On Fri, Jun 27, 2008 at 04:47:29PM +0200, Andi Kleen wrote:
>>
>> > Also it should ideally describe that there is a trade off between
>> > reliability and performance with IOMMU enabled.
>>
>> I agree that there's a performance tradeoff with current generation
>> hardware and ingerfaces, but it wouldn't be fair to single out a
>> single IOMMU implementation ;=)
>
> True. At least for the case without device isolation I have some
> optimizations in mind which will minimize the performance tradeoff. I
> hope to have them ready for 2.6.28 :)

Not sure that would be very useful. Outside of virtualization device
isolation is the main feature of an IOMMU on a native kernel.

-Andi
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo[at]vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/

leo.duran at amd

Jun 27, 2008, 1:39 PM

Post #16 of 29 (300 views)
Permalink

RE: [PATCH 01/34] AMD IOMMU: add Kconfig entry [In reply to]

On Fri, Jun 27, 2008 at 12:06 PM, Joerg Roedel wrote

> On Fri, Jun 27, 2008 at 12:59:47PM -0400, Muli Ben-Yehuda wrote:
> > On Fri, Jun 27, 2008 at 06:54:30PM +0200, Joerg Roedel wrote:
> >
> > > True. At least for the case without device isolation I have some
> > > optimizations in mind which will minimize the performance
> > > tradeoff. I hope to have them ready for 2.6.28 :)
> >
> > Do you mean the case where you have a single I/O address space which
> > is shared by all devices?
>
> Yes. I think this will be the case used most when IOMMU is used for
> virtualization and to handle devices with limited DMA address ranges.
> In
> this case there is a lot to optimize.
>

With the AMD IOMMU there are basically two options with regards to
(untranslated) DMA handling:
1) IOMMU will translate using page tables, which can be set on per
device-table-entry basis; sharing page tables between devices could be
considered a 'resource usage optimization', with the caveat of not being
to provide protection for devices sharing the page tables.

2) IOMMU will not translate if the exclusion range has been enabled, and
the DMA address falls inside that range.
The exclusion range can be enabled for specific devices, or for all
devices... Enabling the exclusion range can be considered a 'performance
optimization' (no table-walks), with the caveat of not being able to
provide protection for devices sharing the exclusion range (BTW, there's
a single exclusion address range per IOMMU).

Leo.

--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo[at]vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/

muli at il

Jun 27, 2008, 3:29 PM

Post #17 of 29 (290 views)
Permalink

Re: [PATCH 01/34] AMD IOMMU: add Kconfig entry [In reply to]

On Fri, Jun 27, 2008 at 03:39:31PM -0500, Duran, Leo wrote:

> 2) IOMMU will not translate if the exclusion range has been enabled, and
> the DMA address falls inside that range.
> The exclusion range can be enabled for specific devices, or for all
> devices... Enabling the exclusion range can be considered a 'performance
> optimization' (no table-walks), with the caveat of not being able to
> provide protection for devices sharing the exclusion range (BTW, there's
> a single exclusion address range per IOMMU).

So, if I understand this correctly, could we implement Joerg's
"almost-direct-map" by having 0-64MB translated for host-owned
devices, and then 64MB-end excluded (again, for host-owned devices
only)? If yes, it should provide a small boost in performance (may or
may not be measurable) over having 64MB-end be an identity
translation.

Cheers,
Muli
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo[at]vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/

leo.duran at amd

Jun 27, 2008, 3:47 PM

Post #18 of 29 (290 views)
Permalink

RE: [PATCH 01/34] AMD IOMMU: add Kconfig entry [In reply to]

On Friday, June 27, 2008 5:30 PM, Muli Ben-Yehuda wrote:

> > 2) IOMMU will not translate if the exclusion range has been enabled,
> and
> > the DMA address falls inside that range.
> > The exclusion range can be enabled for specific devices, or for all
> > devices... Enabling the exclusion range can be considered a
> 'performance
> > optimization' (no table-walks), with the caveat of not being able to
> > provide protection for devices sharing the exclusion range (BTW,
> there's
> > a single exclusion address range per IOMMU).
>
> So, if I understand this correctly, could we implement Joerg's
> "almost-direct-map" by having 0-64MB translated for host-owned
> devices, and then 64MB-end excluded (again, for host-owned devices
> only)? If yes, it should provide a small boost in performance (may or
> may not be measurable) over having 64MB-end be an identity
> translation.
>
> Cheers,
> Muli

Hi Muli,

Yes, you could set an exclusion range for addresses about the virtual
address space (or 'translation aperture').
I played with that in the context of "dma.ops" (not surprisingly, the
exclusion range was pretty busy!).

Again, keep in mind: no protection in the exclusion range (sort of
defeating the purpose of having an IOMMU in the first place), and only
ONE exclusion range per IOMMU.

Leo.

--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo[at]vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/

joro at 8bytes

Jun 28, 2008, 3:52 AM

Post #19 of 29 (273 views)
Permalink

Re: [PATCH 01/34] AMD IOMMU: add Kconfig entry [In reply to]

On Fri, Jun 27, 2008 at 08:54:59PM +0200, Andi Kleen wrote:
> Joerg Roedel <joro[at]8bytes.org> writes:
>
> > On Fri, Jun 27, 2008 at 12:39:45PM -0400, Muli Ben-Yehuda wrote:
> >> On Fri, Jun 27, 2008 at 04:47:29PM +0200, Andi Kleen wrote:
> >>
> >> > Also it should ideally describe that there is a trade off between
> >> > reliability and performance with IOMMU enabled.
> >>
> >> I agree that there's a performance tradeoff with current generation
> >> hardware and ingerfaces, but it wouldn't be fair to single out a
> >> single IOMMU implementation ;=)
> >
> > True. At least for the case without device isolation I have some
> > optimizations in mind which will minimize the performance tradeoff. I
> > hope to have them ready for 2.6.28 :)
>
> Not sure that would be very useful. Outside of virtualization device
> isolation is the main feature of an IOMMU on a native kernel.

I agree that device isolation is one of the a main usage scenarios for
an IOMMU. But if somebody want to use it for virtualization and doesn't
need device isolation this optimization would be usefull.

Joerg

--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo[at]vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/

joro at 8bytes

Jun 28, 2008, 4:04 AM

Post #20 of 29 (273 views)
Permalink

Re: [PATCH 01/34] AMD IOMMU: add Kconfig entry [In reply to]

On Fri, Jun 27, 2008 at 05:47:56PM -0500, Duran, Leo wrote:
> On Friday, June 27, 2008 5:30 PM, Muli Ben-Yehuda wrote:
>
> > > 2) IOMMU will not translate if the exclusion range has been enabled,
> > and
> > > the DMA address falls inside that range.
> > > The exclusion range can be enabled for specific devices, or for all
> > > devices... Enabling the exclusion range can be considered a
> > 'performance
> > > optimization' (no table-walks), with the caveat of not being able to
> > > provide protection for devices sharing the exclusion range (BTW,
> > there's
> > > a single exclusion address range per IOMMU).
> >
> > So, if I understand this correctly, could we implement Joerg's
> > "almost-direct-map" by having 0-64MB translated for host-owned
> > devices, and then 64MB-end excluded (again, for host-owned devices
> > only)? If yes, it should provide a small boost in performance (may or
> > may not be measurable) over having 64MB-end be an identity
> > translation.
> >
> > Cheers,
> > Muli
>
> Hi Muli,
>
> Yes, you could set an exclusion range for addresses about the virtual
> address space (or 'translation aperture').
> I played with that in the context of "dma.ops" (not surprisingly, the
> exclusion range was pretty busy!).
>
> Again, keep in mind: no protection in the exclusion range (sort of
> defeating the purpose of having an IOMMU in the first place), and only
> ONE exclusion range per IOMMU.

Yes, there is only one exclusion range per IOMMU. The problem is that an
exclusion range from 64MB to the end may not be possible because there is
an exclusion range already configured in the ACPI table. On my System for
example the exclusion range is somewhere in the first megabyte of RAM.
In this case the direct mapping using page tables is needed. If there is
no exclusion range defined in ACPI this idea would work of course.

Joerg

--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo[at]vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/

leo.duran at amd

Jun 28, 2008, 7:40 AM

Post #21 of 29 (267 views)
Permalink

RE: [PATCH 01/34] AMD IOMMU: add Kconfig entry [In reply to]

On Saturday, June 28, 2008 6:04 AM, Joerg Roedel wrote:

> Yes, there is only one exclusion range per IOMMU. The problem is that
> an
> exclusion range from 64MB to the end may not be possible because there
> is
> an exclusion range already configured in the ACPI table. On my System
> for
> example the exclusion range is somewhere in the first megabyte of RAM.
> In this case the direct mapping using page tables is needed. If there
> is
> no exclusion range defined in ACPI this idea would work of course.
>
> Joerg
>

Direct 1:1 mappings have a couple of issues:
1) They don't provide a performance optimization (i.e., table-walk still
required)
2) Your aperture would have to be large enough so that virtual==physical
(i.e., lots of memory for page-tables)

Leo.

--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo[at]vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/

leo.duran at amd

Jun 28, 2008, 7:58 AM

Post #22 of 29 (266 views)
Permalink

RE: [PATCH 01/34] AMD IOMMU: add Kconfig entry [In reply to]

On Saturday, June 28, 2008 6:04 AM, Joerg Roedel wrote:

> Yes, there is only one exclusion range per IOMMU. The problem is that
> an exclusion range from 64MB to the end may not be possible because
> there is an exclusion range already configured in the ACPI table. On
> my System for example the exclusion range is somewhere in the first
> megabyte of RAM.
> In this case the direct mapping using page tables is needed. If there
> is no exclusion range defined in ACPI this idea would work of course.
>
> Joerg
>

It would OK to have ONE all inclusive exclusion range (e.g., 64MB to
'end' if you want).
That is, any pre-defined areas would simply fall inside the larger
range.
And, if the ACPI table calls for exclusion of a range inside the
aperture (e.g., inside 64MB),
that range would have to be 'reserved', and set for 1:1 mapping.

Leo.

--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo[at]vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/

joro at 8bytes

Jun 28, 2008, 9:27 AM

Post #23 of 29 (266 views)
Permalink

Re: [PATCH 01/34] AMD IOMMU: add Kconfig entry [In reply to]

On Sat, Jun 28, 2008 at 09:40:57AM -0500, Duran, Leo wrote:
> On Saturday, June 28, 2008 6:04 AM, Joerg Roedel wrote:
>
> > Yes, there is only one exclusion range per IOMMU. The problem is that
> > an
> > exclusion range from 64MB to the end may not be possible because there
> > is
> > an exclusion range already configured in the ACPI table. On my System
> > for
> > example the exclusion range is somewhere in the first megabyte of RAM.
> > In this case the direct mapping using page tables is needed. If there
> > is
> > no exclusion range defined in ACPI this idea would work of course.
> >
> > Joerg
> >
>
> Direct 1:1 mappings have a couple of issues:
> 1) They don't provide a performance optimization (i.e., table-walk still
> required)

True. But the table walk is nothing which could be optimized in
software. The optimization here is that we don't need to map/unmap on
each dma_ops call. Mapping/Unmapping requires TLB flushing and
completion_wait. This is expensive for the CPU. But direct mapping
exclusion ranges inside the aperture and mark the area from 64mb to the
end as the exclusion range as you suggested could be a real option.

> 2) Your aperture would have to be large enough so that virtual==physical
> (i.e., lots of memory for page-tables)

The AMD IOMMU supports multiple page sizes from 4kb up to 4GB. I don't
think that we need too much memory for the page tables.

Joerg

--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo[at]vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/

amit.shah at qumranet

Jul 1, 2008, 10:45 PM

Post #24 of 29 (253 views)
Permalink

Re: [PATCH 01/34] AMD IOMMU: add Kconfig entry [In reply to]

On Friday 27 June 2008 23:10:35 Joerg Roedel wrote:
> On Fri, Jun 27, 2008 at 01:31:00PM -0400, Muli Ben-Yehuda wrote:

> > > > > and to handle devices with limited DMA address ranges.
> > > >
> > > > I'd be pretty surprised if you'll find such devices on machines which
> > > > will have AMD's IOMMU...
> > >
> > > Think of 32bit PCI devices in a host with more than 4GB memory :)
> >
> > I am thinking of them and I'd be surprised if you'd find any in such
> > machines. Certainly I assume none of the on-board devices will have
> > this ancient limitation. But hey, it could happen ;-)
>
> The IOMMU machine under my desk has a 32bit PCI slot with a card in it

Also, remember the "odd-ball devices" we had in our "why is PCI passthrough
needed" slide? People might want to upgrade to newer servers and still retain
their old PCI devices and assign them to guests.

Amit
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo[at]vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/

alan at lxorguk

Jul 2, 2008, 1:12 AM

Post #25 of 29 (250 views)
Permalink

Re: [PATCH 01/34] AMD IOMMU: add Kconfig entry [In reply to]

> > > > Think of 32bit PCI devices in a host with more than 4GB memory :)
> > >
> > > I am thinking of them and I'd be surprised if you'd find any in such
> > > machines. Certainly I assume none of the on-board devices will have
> > > this ancient limitation. But hey, it could happen ;-)
> >
> > The IOMMU machine under my desk has a 32bit PCI slot with a card in it
>
> Also, remember the "odd-ball devices" we had in our "why is PCI passthrough
> needed" slide? People might want to upgrade to newer servers and still retain
> their old PCI devices and assign them to guests.

Nothing odd ball at all.

All SFF ATA controllers are 32bit (thats the PATA port on most PCs
still), even the Intel ICH controllers are 32bit in SFF mode (which is
still how most PCs come with it configured). Most PC hardware is still
32bit. A few bits are 31 or 28 bit just to ruin the party - 31bit being
quite common as old Windows had a 2/2GB split.
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo[at]vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/

1 2

View All

Index | Next | Previous | View Threaded

Interested in having your list archived? Contact lists@gossamer-threads.com