Login | Register For Free | Help
Search for: (Advanced)

Mailing List Archive: Lucene: Java-User

Memory question

 

 

Lucene java-user RSS feed   Index | Next | Previous | View Threaded


chris.bamford at talktalk

May 15, 2012, 7:47 AM

Post #1 of 13 (884 views)
Permalink
Memory question

Hi

Can anyone tell me what happens to the memory when Lucene opens an index? Is it loaded into the JVM's heap or is it mapped into virtual memory outside of it?
I am running on Linux and if I use pmap on the PID of my JVM, I can see lots of entries for index cfs files.

Does this mean that indexes are mapped into non-heap memory? If so, how can I monitor the space my process is using if I cache open IndexSearchers?

The details are:

Sun 64-bit JVM on Linux.
Lucene 3.6 running in 2.3 compatibility mode (as we are in the in the process of a migration to 3.6)

Thanks,

- Chris


LFechner at hubwoo

May 15, 2012, 7:51 AM

Post #2 of 13 (869 views)
Permalink
RE: Memory question [In reply to]

Hi,


By design memory outside the JVM heap space should not be accessible for
java applications.

Why you might see is the disc cache of the Linux storage subsystem.


Best Regards

Lutz

-----Original Message-----
From: Chris Bamford [mailto:chris.bamford [at] talktalk]
Sent: Dienstag, 15. Mai 2012 09:47
To: java-user [at] lucene
Subject: Memory question

Hi

Can anyone tell me what happens to the memory when Lucene opens an
index? Is it loaded into the JVM's heap or is it mapped into virtual
memory outside of it?
I am running on Linux and if I use pmap on the PID of my JVM, I can see
lots of entries for index cfs files.

Does this mean that indexes are mapped into non-heap memory? If so, how
can I monitor the space my process is using if I cache open
IndexSearchers?

The details are:

Sun 64-bit JVM on Linux.
Lucene 3.6 running in 2.3 compatibility mode (as we are in the in the
process of a migration to 3.6)

Thanks,

- Chris


---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe [at] lucene
For additional commands, e-mail: java-user-help [at] lucene


ian.lea at gmail

May 15, 2012, 8:19 AM

Post #3 of 13 (869 views)
Permalink
Re: Memory question [In reply to]

In versions from 3.3 onwards MMapDirectory is the default on 64-bit
linux. Not sure exactly what that means wrt your questions, but may
well be relevant.


--
Ian.


On Tue, May 15, 2012 at 3:51 PM, Lutz Fechner <LFechner [at] hubwoo> wrote:
> Hi,
>
>
> By design memory outside the JVM heap space should not be accessible for
> java applications.
>
> Why you might see is the disc cache of the Linux storage subsystem.
>
>
> Best Regards
>
> Lutz
>
> -----Original Message-----
> From: Chris Bamford [mailto:chris.bamford [at] talktalk]
> Sent: Dienstag, 15. Mai 2012 09:47
> To: java-user [at] lucene
> Subject: Memory question
>
> Hi
>
> Can anyone tell me what happens to the memory when Lucene opens an
> index?  Is it loaded into the JVM's heap or is it mapped into virtual
> memory outside of it?
> I am running on Linux and if I use pmap on the PID of my JVM, I can see
> lots of entries for index cfs files.
>
> Does this mean that indexes are mapped into non-heap memory?  If so, how
> can I monitor the space my process is using if I cache open
> IndexSearchers?
>
> The details are:
>
> Sun 64-bit JVM on Linux.
> Lucene 3.6 running in 2.3 compatibility mode (as we are in the in the
> process of a migration to 3.6)
>
> Thanks,
>
> - Chris
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe [at] lucene
> For additional commands, e-mail: java-user-help [at] lucene
>

---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe [at] lucene
For additional commands, e-mail: java-user-help [at] lucene


uwe at thetaphi

May 15, 2012, 10:10 AM

Post #4 of 13 (870 views)
Permalink
RE: Memory question [In reply to]

It mmaps the files into virtual memory if it runs on a 64 bit JVM. Because
of that you see the mmapped CFS files. This is outside Java Heap and is all
*virtual* no RAM is explicitely occupied except the O/S cache.

-----
Uwe Schindler
H.-H.-Meier-Allee 63, D-28213 Bremen
http://www.thetaphi.de
eMail: uwe [at] thetaphi


> -----Original Message-----
> From: Chris Bamford [mailto:chris.bamford [at] talktalk]
> Sent: Tuesday, May 15, 2012 4:47 PM
> To: java-user [at] lucene
> Subject: Memory question
>
> Hi
>
> Can anyone tell me what happens to the memory when Lucene opens an index?
> Is it loaded into the JVM's heap or is it mapped into virtual memory
outside of
> it?
> I am running on Linux and if I use pmap on the PID of my JVM, I can see
lots of
> entries for index cfs files.
>
> Does this mean that indexes are mapped into non-heap memory? If so, how
> can I monitor the space my process is using if I cache open
IndexSearchers?
>
> The details are:
>
> Sun 64-bit JVM on Linux.
> Lucene 3.6 running in 2.3 compatibility mode (as we are in the in the
process of
> a migration to 3.6)
>
> Thanks,
>
> - Chris


---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe [at] lucene
For additional commands, e-mail: java-user-help [at] lucene


chris.bamford at talktalk

May 15, 2012, 12:38 PM

Post #5 of 13 (875 views)
Permalink
Re: RE: Memory question [In reply to]

Thanks Uwe.

What I'd like to understand is the implications of this on a server which opens a large number of indexes over a long period. Will this non-heap memory continue to grow? Will gc be effective at spotting it and releasing it via references in the heap?

I had an instance yesterday where a server swapped itself to a standstill and had to be restarted. The load average was through the roof and I am trying to understand why. One of my recent changes is updating from 2.3 to 3.6, so naturally I am keen to know the impact of the mmap stuff which is now standard under the covers.

My server caches indexsearchers and then closes them based on how full the heap is getting. My worry is that if the bulk of the memory is being allocated outside the Jvm, how can I make sensible decisions?

Thanks for any pointers / info.

Chris



-----Original Message-----
From: uwe [at] thetaphi
To: java-user [at] lucene
Sent: Tue, 15 May 2012 18:10
Subject: RE: Memory question



It mmaps the files into virtual memory if it runs on a 64 bit JVM. Because
of that you see the mmapped CFS files. This is outside Java Heap and is all
*virtual* no RAM is explicitely occupied except the O/S cache.

-----
Uwe Schindler
H.-H.-Meier-Allee 63, D-28213 Bremen
http://www.thetaphi.de
eMail: uwe [at] thetaphi

> -----Original Message-----
> From: Chris Bamford [mailto:chris.bamford [at] talktalk]
> Sent: Tuesday, May 15, 2012 4:47 PM
> To: java-user [at] lucene
> Subject: Memory question
>
> Hi
>
> Can anyone tell me what happens to the memory when Lucene opens an index?
> Is it loaded into the JVM's heap or is it mapped into virtual memory
outside of
> it?
> I am running on Linux and if I use pmap on the PID of my JVM, I can see
lots of
> entries for index cfs files.
>
> Does this mean that indexes are mapped into non-heap memory? If so, how
> can I monitor the space my process is using if I cache open
IndexSearchers?
>
> The details are:
>
> Sun 64-bit JVM on Linux.
> Lucene 3.6 running in 2.3 compatibility mode (as we are in the in the
process of
> a migration to 3.6)
>
> Thanks,
>
> - Chris

---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe [at] lucene
For additional commands, e-mail: java-user-help [at] lucene


john.nader at cengage

May 15, 2012, 1:12 PM

Post #6 of 13 (877 views)
Permalink
Re: Memory question [In reply to]

We've encountered this issue and came up with a fairly good approach to
address it.

We are on Lucene 3.0.2 with Java 1.6.0_29. Our indices are about 35GB in
size. Our JVM runs at 20GB of heap, with about 12GB of steady usage. Our
server has 32GB total.

What would happen in our case is that the linux would page in more and
more of the memory mapped index files into memory, forcing idle portions
of the JVM heap to be swapped out. This was not an issue until our CMS GC
kicked in. This would force swapping in of all JVM memory to collect
unused references. I/O wait would shoot up and performance would suffer.
Yes, even CMS can kill performance is you are swapping. The tell-tale
sign was a spike in inbound swap at the start of CMS.

In our case, we addressed the situation using iCMS, which is Incremental
CMS. This takes the mark phase (and sweep too?) and does it continuously
with a configurable duty cycle. The result was that swapping was smoothed
out to be a small stead drag on the system instead of a hard spike. There
was an small loss in performance, but a big gain in stability.

This tuning may be an option for you. BTW, pmap will give you statistic
on total file size and how much is resident. The java heap shows up in
pmap as well on linux, so you can determine how much of that is in memory
as well.

John



On 5/15/12 3:38 PM, "Chris Bamford" <chris.bamford [at] talktalk> wrote:

>Thanks Uwe.
>
>What I'd like to understand is the implications of this on a server which
>opens a large number of indexes over a long period. Will this non-heap
>memory continue to grow? Will gc be effective at spotting it and
>releasing it via references in the heap?
>
> I had an instance yesterday where a server swapped itself to a
>standstill and had to be restarted. The load average was through the roof
>and I am trying to understand why. One of my recent changes is updating
>from 2.3 to 3.6, so naturally I am keen to know the impact of the mmap
>stuff which is now standard under the covers.
>
>My server caches indexsearchers and then closes them based on how full
>the heap is getting. My worry is that if the bulk of the memory is being
>allocated outside the Jvm, how can I make sensible decisions?
>
>Thanks for any pointers / info.
>
>Chris
>
>
>
>-----Original Message-----
>From: uwe [at] thetaphi
>To: java-user [at] lucene
>Sent: Tue, 15 May 2012 18:10
>Subject: RE: Memory question
>
>
>
>It mmaps the files into virtual memory if it runs on a 64 bit JVM. Because
>of that you see the mmapped CFS files. This is outside Java Heap and is
>all
>*virtual* no RAM is explicitely occupied except the O/S cache.
>
>-----
>Uwe Schindler
>H.-H.-Meier-Allee 63, D-28213 Bremen
>http://www.thetaphi.de
>eMail: uwe [at] thetaphi
>
>> -----Original Message-----
>> From: Chris Bamford [mailto:chris.bamford [at] talktalk]
>> Sent: Tuesday, May 15, 2012 4:47 PM
>> To: java-user [at] lucene
>> Subject: Memory question
>>
>> Hi
>>
>> Can anyone tell me what happens to the memory when Lucene opens an
>>index?
>> Is it loaded into the JVM's heap or is it mapped into virtual memory
>outside of
>> it?
>> I am running on Linux and if I use pmap on the PID of my JVM, I can see
>lots of
>> entries for index cfs files.
>>
>> Does this mean that indexes are mapped into non-heap memory? If so, how
>> can I monitor the space my process is using if I cache open
>IndexSearchers?
>>
>> The details are:
>>
>> Sun 64-bit JVM on Linux.
>> Lucene 3.6 running in 2.3 compatibility mode (as we are in the in the
>process of
>> a migration to 3.6)
>>
>> Thanks,
>>
>> - Chris
>
>---------------------------------------------------------------------
>To unsubscribe, e-mail: java-user-unsubscribe [at] lucene
>For additional commands, e-mail: java-user-help [at] lucene
>


---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe [at] lucene
For additional commands, e-mail: java-user-help [at] lucene


chris.bamford at talktalk

May 15, 2012, 2:38 PM

Post #7 of 13 (868 views)
Permalink
Re: Memory question [In reply to]

Hi John,

Very interesting, thanks for the detailed explanation. It certainly sounds like the same symptoms!
Can I please clarify a couple of things ?

- I have googled CMS / iCMS as I wasn't familiar with those acronyms (apart from 'code management system' and that didn't sound right!)
and am I right in thinking that it is some sort of monitoring code pulled into your server via a jar? (I'm confused why it would have its' own
GC cycle...)
- So are you suggesting I play with my own JVM's (Sun/Oracle) parameters to achieve a similar effect ?

Thanks again,

- Chris






-----Original Message-----
From: Nader, John P <john.nader [at] cengage>
To: java-user [at] lucene <java-user [at] lucene>
Sent: Tue, 15 May 2012 21:12
Subject: Re: Memory question


We've encountered this issue and came up with a fairly good approach to
address it.

We are on Lucene 3.0.2 with Java 1.6.0_29. Our indices are about 35GB in
size. Our JVM runs at 20GB of heap, with about 12GB of steady usage. Our
server has 32GB total.

What would happen in our case is that the linux would page in more and
more of the memory mapped index files into memory, forcing idle portions
of the JVM heap to be swapped out. This was not an issue until our CMS GC
kicked in. This would force swapping in of all JVM memory to collect
unused references. I/O wait would shoot up and performance would suffer.
Yes, even CMS can kill performance is you are swapping. The tell-tale
sign was a spike in inbound swap at the start of CMS.

In our case, we addressed the situation using iCMS, which is Incremental
CMS. This takes the mark phase (and sweep too?) and does it continuously
with a configurable duty cycle. The result was that swapping was smoothed
out to be a small stead drag on the system instead of a hard spike. There
was an small loss in performance, but a big gain in stability.

This tuning may be an option for you. BTW, pmap will give you statistic
on total file size and how much is resident. The java heap shows up in
pmap as well on linux, so you can determine how much of that is in memory
as well.

John



On 5/15/12 3:38 PM, "Chris Bamford" <chris.bamford [at] talktalk> wrote:

>Thanks Uwe.
>
>What I'd like to understand is the implications of this on a server which
>opens a large number of indexes over a long period. Will this non-heap
>memory continue to grow? Will gc be effective at spotting it and
>releasing it via references in the heap?
>
> I had an instance yesterday where a server swapped itself to a
>standstill and had to be restarted. The load average was through the roof
>and I am trying to understand why. One of my recent changes is updating
>from 2.3 to 3.6, so naturally I am keen to know the impact of the mmap
>stuff which is now standard under the covers.
>
>My server caches indexsearchers and then closes them based on how full
>the heap is getting. My worry is that if the bulk of the memory is being
>allocated outside the Jvm, how can I make sensible decisions?
>
>Thanks for any pointers / info.
>
>Chris
>
>
>
>-----Original Message-----
>From: uwe [at] thetaphi
>To: java-user [at] lucene
>Sent: Tue, 15 May 2012 18:10
>Subject: RE: Memory question
>
>
>
>It mmaps the files into virtual memory if it runs on a 64 bit JVM. Because
>of that you see the mmapped CFS files. This is outside Java Heap and is
>all
>*virtual* no RAM is explicitely occupied except the O/S cache.
>
>-----
>Uwe Schindler
>H.-H.-Meier-Allee 63, D-28213 Bremen
>http://www.thetaphi.de
>eMail: uwe [at] thetaphi
>
>> -----Original Message-----
>> From: Chris Bamford [mailto:chris.bamford [at] talktalk]
>> Sent: Tuesday, May 15, 2012 4:47 PM
>> To: java-user [at] lucene
>> Subject: Memory question
>>
>> Hi
>>
>> Can anyone tell me what happens to the memory when Lucene opens an
>>index?
>> Is it loaded into the JVM's heap or is it mapped into virtual memory
>outside of
>> it?
>> I am running on Linux and if I use pmap on the PID of my JVM, I can see
>lots of
>> entries for index cfs files.
>>
>> Does this mean that indexes are mapped into non-heap memory? If so, how
>> can I monitor the space my process is using if I cache open
>IndexSearchers?
>>
>> The details are:
>>
>> Sun 64-bit JVM on Linux.
>> Lucene 3.6 running in 2.3 compatibility mode (as we are in the in the
>process of
>> a migration to 3.6)
>>
>> Thanks,
>>
>> - Chris
>
>---------------------------------------------------------------------
>To unsubscribe, e-mail: java-user-unsubscribe [at] lucene
>For additional commands, e-mail: java-user-help [at] lucene
>


---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe [at] lucene
For additional commands, e-mail: java-user-help [at] lucene


LFechner at hubwoo

May 15, 2012, 3:32 PM

Post #8 of 13 (875 views)
Permalink
RE: Memory question [In reply to]

CMS is the concurrent mark sweep garbage collector. Instead of waiting
for the memory to fill up and being collected for memory to be freed up
again, it runs concurrently while the app threads are running. Usually
the JVM would call a full stop the world collection. All threads would
be on hold until GC is finished. CMS would prevent lengthy stops from
happening in trade for constant micro stops.

This usually is a good option for apps that are sensitive (from a user
experience) from hanging during GC time.

See http://docs.oracle.com/javase/6/docs/technotes/guides/vm/cms-6.html

Best Regards

Lutz

-----Original Message-----
From: Chris Bamford [mailto:chris.bamford [at] talktalk]
Sent: Dienstag, 15. Mai 2012 16:38
To: java-user [at] lucene
Subject: Re: Memory question


Hi John,

Very interesting, thanks for the detailed explanation. It certainly
sounds like the same symptoms!
Can I please clarify a couple of things ?

- I have googled CMS / iCMS as I wasn't familiar with those acronyms
(apart from 'code management system' and that didn't sound right!)
and am I right in thinking that it is some sort of monitoring code
pulled into your server via a jar? (I'm confused why it would have its'
own
GC cycle...)
- So are you suggesting I play with my own JVM's (Sun/Oracle) parameters
to achieve a similar effect ?

Thanks again,

- Chris






-----Original Message-----
From: Nader, John P <john.nader [at] cengage>
To: java-user [at] lucene <java-user [at] lucene>
Sent: Tue, 15 May 2012 21:12
Subject: Re: Memory question


We've encountered this issue and came up with a fairly good approach to
address it.

We are on Lucene 3.0.2 with Java 1.6.0_29. Our indices are about 35GB
in
size. Our JVM runs at 20GB of heap, with about 12GB of steady usage.
Our
server has 32GB total.

What would happen in our case is that the linux would page in more and
more of the memory mapped index files into memory, forcing idle portions
of the JVM heap to be swapped out. This was not an issue until our CMS
GC
kicked in. This would force swapping in of all JVM memory to collect
unused references. I/O wait would shoot up and performance would
suffer.
Yes, even CMS can kill performance is you are swapping. The tell-tale
sign was a spike in inbound swap at the start of CMS.

In our case, we addressed the situation using iCMS, which is Incremental
CMS. This takes the mark phase (and sweep too?) and does it
continuously
with a configurable duty cycle. The result was that swapping was
smoothed
out to be a small stead drag on the system instead of a hard spike.
There
was an small loss in performance, but a big gain in stability.

This tuning may be an option for you. BTW, pmap will give you statistic
on total file size and how much is resident. The java heap shows up in
pmap as well on linux, so you can determine how much of that is in
memory
as well.

John



On 5/15/12 3:38 PM, "Chris Bamford" <chris.bamford [at] talktalk> wrote:

>Thanks Uwe.
>
>What I'd like to understand is the implications of this on a server
which
>opens a large number of indexes over a long period. Will this non-heap
>memory continue to grow? Will gc be effective at spotting it and
>releasing it via references in the heap?
>
> I had an instance yesterday where a server swapped itself to a
>standstill and had to be restarted. The load average was through the
roof
>and I am trying to understand why. One of my recent changes is updating
>from 2.3 to 3.6, so naturally I am keen to know the impact of the mmap
>stuff which is now standard under the covers.
>
>My server caches indexsearchers and then closes them based on how full
>the heap is getting. My worry is that if the bulk of the memory is
being
>allocated outside the Jvm, how can I make sensible decisions?
>
>Thanks for any pointers / info.
>
>Chris
>
>
>
>-----Original Message-----
>From: uwe [at] thetaphi
>To: java-user [at] lucene
>Sent: Tue, 15 May 2012 18:10
>Subject: RE: Memory question
>
>
>
>It mmaps the files into virtual memory if it runs on a 64 bit JVM.
Because
>of that you see the mmapped CFS files. This is outside Java Heap and is
>all
>*virtual* no RAM is explicitely occupied except the O/S cache.
>
>-----
>Uwe Schindler
>H.-H.-Meier-Allee 63, D-28213 Bremen
>http://www.thetaphi.de
>eMail: uwe [at] thetaphi
>
>> -----Original Message-----
>> From: Chris Bamford [mailto:chris.bamford [at] talktalk]
>> Sent: Tuesday, May 15, 2012 4:47 PM
>> To: java-user [at] lucene
>> Subject: Memory question
>>
>> Hi
>>
>> Can anyone tell me what happens to the memory when Lucene opens an
>>index?
>> Is it loaded into the JVM's heap or is it mapped into virtual memory
>outside of
>> it?
>> I am running on Linux and if I use pmap on the PID of my JVM, I can
see
>lots of
>> entries for index cfs files.
>>
>> Does this mean that indexes are mapped into non-heap memory? If so,
how
>> can I monitor the space my process is using if I cache open
>IndexSearchers?
>>
>> The details are:
>>
>> Sun 64-bit JVM on Linux.
>> Lucene 3.6 running in 2.3 compatibility mode (as we are in the in the
>process of
>> a migration to 3.6)
>>
>> Thanks,
>>
>> - Chris
>
>---------------------------------------------------------------------
>To unsubscribe, e-mail: java-user-unsubscribe [at] lucene
>For additional commands, e-mail: java-user-help [at] lucene
>


---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe [at] lucene
For additional commands, e-mail: java-user-help [at] lucene





---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe [at] lucene
For additional commands, e-mail: java-user-help [at] lucene


sean.bridges at gmail

May 15, 2012, 10:00 PM

Post #9 of 13 (866 views)
Permalink
Re: Memory question [In reply to]

Another solution is to set -Xmx == -Xms, and make a native call to
mlockall on application startup, this will prevent the jvm from
swapping.

Using JNA, calling mlockall is not very hard, you can see how
cassandra does it here,

https://github.com/apache/cassandra/blob/cassandra-1.1.0/src/java/org/apache/cassandra/utils/CLibrary.java

Sean

On Tue, May 15, 2012 at 1:12 PM, Nader, John P <john.nader [at] cengage> wrote:
> We've encountered this issue and came up with a fairly good approach to
> address it.
>
> We are on Lucene 3.0.2 with Java 1.6.0_29.  Our indices are about 35GB in
> size.  Our JVM runs at 20GB of heap, with about 12GB of steady usage.  Our
> server has 32GB total.
>
> What would happen in our case is that the linux would page in more and
> more of the memory mapped index files into memory, forcing idle portions
> of the JVM heap to be swapped out.  This was not an issue until our CMS GC
> kicked in.  This would force swapping in of all JVM memory to collect
> unused references.  I/O wait would shoot up and performance would suffer.
> Yes, even CMS can kill performance is you are swapping.  The tell-tale
> sign was a spike in inbound swap at the start of CMS.
>
> In our case, we addressed the situation using iCMS, which is Incremental
> CMS.  This takes the mark phase (and sweep too?) and does it continuously
> with a configurable duty cycle.  The result was that swapping was smoothed
> out to be a small stead drag on the system instead of a hard spike.  There
> was an small loss in performance, but a big gain in stability.
>
> This tuning may be an option for you.  BTW, pmap will give you statistic
> on total file size and how much is resident.  The java heap shows up in
> pmap as well on linux, so you can determine how much of that is in memory
> as well.
>
> John
>
>
>
> On 5/15/12 3:38 PM, "Chris Bamford" <chris.bamford [at] talktalk> wrote:
>
>>Thanks Uwe.
>>
>>What I'd like to understand is the implications of this on a server which
>>opens a large number of indexes over a long period. Will this non-heap
>>memory continue to grow? Will gc be effective at spotting it and
>>releasing it via references in the heap?
>>
>> I had an instance yesterday where a server swapped itself to a
>>standstill and had to be restarted. The load average was through the roof
>>and I am trying to understand why. One of my recent changes is updating
>>from 2.3 to 3.6, so naturally I am keen to know the impact of the mmap
>>stuff which is now standard under the covers.
>>
>>My server caches indexsearchers and then closes them based on how full
>>the heap is getting. My worry is that if the bulk of the memory is being
>>allocated outside the Jvm, how can I make sensible decisions?
>>
>>Thanks for any pointers / info.
>>
>>Chris
>>
>>
>>
>>-----Original Message-----
>>From: uwe [at] thetaphi
>>To: java-user [at] lucene
>>Sent: Tue, 15 May 2012 18:10
>>Subject: RE: Memory question
>>
>>
>>
>>It mmaps the files into virtual memory if it runs on a 64 bit JVM. Because
>>of that you see the mmapped CFS files. This is outside Java Heap and is
>>all
>>*virtual* no RAM is explicitely occupied except the O/S cache.
>>
>>-----
>>Uwe Schindler
>>H.-H.-Meier-Allee 63, D-28213 Bremen
>>http://www.thetaphi.de
>>eMail: uwe [at] thetaphi
>>
>>> -----Original Message-----
>>> From: Chris Bamford [mailto:chris.bamford [at] talktalk]
>>> Sent: Tuesday, May 15, 2012 4:47 PM
>>> To: java-user [at] lucene
>>> Subject: Memory question
>>>
>>> Hi
>>>
>>> Can anyone tell me what happens to the memory when Lucene opens an
>>>index?
>>> Is it loaded into the JVM's heap or is it mapped into virtual memory
>>outside of
>>> it?
>>> I am running on Linux and if I use pmap on the PID of my JVM, I can see
>>lots of
>>> entries for index cfs files.
>>>
>>> Does this mean that indexes are mapped into non-heap memory?  If so, how
>>> can I monitor the space my process is using if I cache open
>>IndexSearchers?
>>>
>>> The details are:
>>>
>>> Sun 64-bit JVM on Linux.
>>> Lucene 3.6 running in 2.3 compatibility mode (as we are in the in the
>>process of
>>> a migration to 3.6)
>>>
>>> Thanks,
>>>
>>> - Chris
>>
>>---------------------------------------------------------------------
>>To unsubscribe, e-mail: java-user-unsubscribe [at] lucene
>>For additional commands, e-mail: java-user-help [at] lucene
>>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe [at] lucene
> For additional commands, e-mail: java-user-help [at] lucene
>

---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe [at] lucene
For additional commands, e-mail: java-user-help [at] lucene


lucene_list at iconparc

May 16, 2012, 12:11 AM

Post #10 of 13 (864 views)
Permalink
Re: Memory question [In reply to]

Another option to consider is to *decrease* the JVM maximum heap size.
This in effect leaves more memory for swapped in mmio pages and
decreases the GC effort, which might increase system performance and
stability.

Regards,
Christoph

Am 15.05.2012 21:38, schrieb Chris Bamford:
> Thanks Uwe.
>
> What I'd like to understand is the implications of this on a server which opens a large number of indexes over a long period. Will this non-heap memory continue to grow? Will gc be effective at spotting it and releasing it via references in the heap?
>
> I had an instance yesterday where a server swapped itself to a standstill and had to be restarted. The load average was through the roof and I am trying to understand why. One of my recent changes is updating from 2.3 to 3.6, so naturally I am keen to know the impact of the mmap stuff which is now standard under the covers.
>
> My server caches indexsearchers and then closes them based on how full the heap is getting. My worry is that if the bulk of the memory is being allocated outside the Jvm, how can I make sensible decisions?
>
> Thanks for any pointers / info.
>
> Chris
>
>
>
> -----Original Message-----
> From: uwe [at] thetaphi
> To: java-user [at] lucene
> Sent: Tue, 15 May 2012 18:10
> Subject: RE: Memory question
>
>
>
> It mmaps the files into virtual memory if it runs on a 64 bit JVM. Because
> of that you see the mmapped CFS files. This is outside Java Heap and is all
> *virtual* no RAM is explicitely occupied except the O/S cache.
>
> -----
> Uwe Schindler
> H.-H.-Meier-Allee 63, D-28213 Bremen
> http://www.thetaphi.de
> eMail: uwe [at] thetaphi
>
>> -----Original Message-----
>> From: Chris Bamford [mailto:chris.bamford [at] talktalk]
>> Sent: Tuesday, May 15, 2012 4:47 PM
>> To: java-user [at] lucene
>> Subject: Memory question
>>
>> Hi
>>
>> Can anyone tell me what happens to the memory when Lucene opens an index?
>> Is it loaded into the JVM's heap or is it mapped into virtual memory
> outside of
>> it?
>> I am running on Linux and if I use pmap on the PID of my JVM, I can see
> lots of
>> entries for index cfs files.
>>
>> Does this mean that indexes are mapped into non-heap memory? If so, how
>> can I monitor the space my process is using if I cache open
> IndexSearchers?
>> The details are:
>>
>> Sun 64-bit JVM on Linux.
>> Lucene 3.6 running in 2.3 compatibility mode (as we are in the in the
> process of
>> a migration to 3.6)
>>
>> Thanks,
>>
>> - Chris
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe [at] lucene
> For additional commands, e-mail: java-user-help [at] lucene
>
>


---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe [at] lucene
For additional commands, e-mail: java-user-help [at] lucene


john.nader at cengage

May 16, 2012, 8:27 AM

Post #11 of 13 (859 views)
Permalink
Re: Memory question [In reply to]

Another good link is
http://www.oracle.com/technetwork/java/javase/gc-tuning-6-140523.html,
which also includes details on iCMS, which is the Incremental Mode for CMS.

On 5/15/12 6:32 PM, "Lutz Fechner" <LFechner [at] hubwoo> wrote:

>CMS is the concurrent mark sweep garbage collector. Instead of waiting
>for the memory to fill up and being collected for memory to be freed up
>again, it runs concurrently while the app threads are running. Usually
>the JVM would call a full stop the world collection. All threads would
>be on hold until GC is finished. CMS would prevent lengthy stops from
>happening in trade for constant micro stops.
>
>This usually is a good option for apps that are sensitive (from a user
>experience) from hanging during GC time.
>
>See http://docs.oracle.com/javase/6/docs/technotes/guides/vm/cms-6.html
>
>Best Regards
>
>Lutz
>
>-----Original Message-----
>From: Chris Bamford [mailto:chris.bamford [at] talktalk]
>Sent: Dienstag, 15. Mai 2012 16:38
>To: java-user [at] lucene
>Subject: Re: Memory question
>
>
> Hi John,
>
>Very interesting, thanks for the detailed explanation. It certainly
>sounds like the same symptoms!
>Can I please clarify a couple of things ?
>
>- I have googled CMS / iCMS as I wasn't familiar with those acronyms
>(apart from 'code management system' and that didn't sound right!)
> and am I right in thinking that it is some sort of monitoring code
>pulled into your server via a jar? (I'm confused why it would have its'
>own
> GC cycle...)
>- So are you suggesting I play with my own JVM's (Sun/Oracle) parameters
>to achieve a similar effect ?
>
>Thanks again,
>
>- Chris
>
>
>
>
>
>
>-----Original Message-----
>From: Nader, John P <john.nader [at] cengage>
>To: java-user [at] lucene <java-user [at] lucene>
>Sent: Tue, 15 May 2012 21:12
>Subject: Re: Memory question
>
>
>We've encountered this issue and came up with a fairly good approach to
>address it.
>
>We are on Lucene 3.0.2 with Java 1.6.0_29. Our indices are about 35GB
>in
>size. Our JVM runs at 20GB of heap, with about 12GB of steady usage.
>Our
>server has 32GB total.
>
>What would happen in our case is that the linux would page in more and
>more of the memory mapped index files into memory, forcing idle portions
>of the JVM heap to be swapped out. This was not an issue until our CMS
>GC
>kicked in. This would force swapping in of all JVM memory to collect
>unused references. I/O wait would shoot up and performance would
>suffer.
>Yes, even CMS can kill performance is you are swapping. The tell-tale
>sign was a spike in inbound swap at the start of CMS.
>
>In our case, we addressed the situation using iCMS, which is Incremental
>CMS. This takes the mark phase (and sweep too?) and does it
>continuously
>with a configurable duty cycle. The result was that swapping was
>smoothed
>out to be a small stead drag on the system instead of a hard spike.
>There
>was an small loss in performance, but a big gain in stability.
>
>This tuning may be an option for you. BTW, pmap will give you statistic
>on total file size and how much is resident. The java heap shows up in
>pmap as well on linux, so you can determine how much of that is in
>memory
>as well.
>
>John
>
>
>
>On 5/15/12 3:38 PM, "Chris Bamford" <chris.bamford [at] talktalk> wrote:
>
>>Thanks Uwe.
>>
>>What I'd like to understand is the implications of this on a server
>which
>>opens a large number of indexes over a long period. Will this non-heap
>>memory continue to grow? Will gc be effective at spotting it and
>>releasing it via references in the heap?
>>
>> I had an instance yesterday where a server swapped itself to a
>>standstill and had to be restarted. The load average was through the
>roof
>>and I am trying to understand why. One of my recent changes is updating
>>from 2.3 to 3.6, so naturally I am keen to know the impact of the mmap
>>stuff which is now standard under the covers.
>>
>>My server caches indexsearchers and then closes them based on how full
>>the heap is getting. My worry is that if the bulk of the memory is
>being
>>allocated outside the Jvm, how can I make sensible decisions?
>>
>>Thanks for any pointers / info.
>>
>>Chris
>>
>>
>>
>>-----Original Message-----
>>From: uwe [at] thetaphi
>>To: java-user [at] lucene
>>Sent: Tue, 15 May 2012 18:10
>>Subject: RE: Memory question
>>
>>
>>
>>It mmaps the files into virtual memory if it runs on a 64 bit JVM.
>Because
>>of that you see the mmapped CFS files. This is outside Java Heap and is
>>all
>>*virtual* no RAM is explicitely occupied except the O/S cache.
>>
>>-----
>>Uwe Schindler
>>H.-H.-Meier-Allee 63, D-28213 Bremen
>>http://www.thetaphi.de
>>eMail: uwe [at] thetaphi
>>
>>> -----Original Message-----
>>> From: Chris Bamford [mailto:chris.bamford [at] talktalk]
>>> Sent: Tuesday, May 15, 2012 4:47 PM
>>> To: java-user [at] lucene
>>> Subject: Memory question
>>>
>>> Hi
>>>
>>> Can anyone tell me what happens to the memory when Lucene opens an
>>>index?
>>> Is it loaded into the JVM's heap or is it mapped into virtual memory
>>outside of
>>> it?
>>> I am running on Linux and if I use pmap on the PID of my JVM, I can
>see
>>lots of
>>> entries for index cfs files.
>>>
>>> Does this mean that indexes are mapped into non-heap memory? If so,
>how
>>> can I monitor the space my process is using if I cache open
>>IndexSearchers?
>>>
>>> The details are:
>>>
>>> Sun 64-bit JVM on Linux.
>>> Lucene 3.6 running in 2.3 compatibility mode (as we are in the in the
>>process of
>>> a migration to 3.6)
>>>
>>> Thanks,
>>>
>>> - Chris
>>
>>---------------------------------------------------------------------
>>To unsubscribe, e-mail: java-user-unsubscribe [at] lucene
>>For additional commands, e-mail: java-user-help [at] lucene
>>
>
>
>---------------------------------------------------------------------
>To unsubscribe, e-mail: java-user-unsubscribe [at] lucene
>For additional commands, e-mail: java-user-help [at] lucene
>
>
>
>
>
>---------------------------------------------------------------------
>To unsubscribe, e-mail: java-user-unsubscribe [at] lucene
>For additional commands, e-mail: java-user-help [at] lucene
>


---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe [at] lucene
For additional commands, e-mail: java-user-help [at] lucene


chris.bamford at talktalk

May 16, 2012, 9:00 AM

Post #12 of 13 (860 views)
Permalink
Re: Memory question [In reply to]

Thanks everyone. Looks like I have lots of reading to do :-)





-----Original Message-----
From: Nader, John P <john.nader [at] cengage>
To: java-user [at] lucene <java-user [at] lucene>
Sent: Wed, 16 May 2012 16:27
Subject: Re: Memory question


Another good link is
http://www.oracle.com/technetwork/java/javase/gc-tuning-6-140523.html,
which also includes details on iCMS, which is the Incremental Mode for CMS.

On 5/15/12 6:32 PM, "Lutz Fechner" <LFechner [at] hubwoo> wrote:

>CMS is the concurrent mark sweep garbage collector. Instead of waiting
>for the memory to fill up and being collected for memory to be freed up
>again, it runs concurrently while the app threads are running. Usually
>the JVM would call a full stop the world collection. All threads would
>be on hold until GC is finished. CMS would prevent lengthy stops from
>happening in trade for constant micro stops.
>
>This usually is a good option for apps that are sensitive (from a user
>experience) from hanging during GC time.
>
>See http://docs.oracle.com/javase/6/docs/technotes/guides/vm/cms-6.html
>
>Best Regards
>
>Lutz
>
>-----Original Message-----
>From: Chris Bamford [mailto:chris.bamford [at] talktalk]
>Sent: Dienstag, 15. Mai 2012 16:38
>To: java-user [at] lucene
>Subject: Re: Memory question
>
>
> Hi John,
>
>Very interesting, thanks for the detailed explanation. It certainly
>sounds like the same symptoms!
>Can I please clarify a couple of things ?
>
>- I have googled CMS / iCMS as I wasn't familiar with those acronyms
>(apart from 'code management system' and that didn't sound right!)
> and am I right in thinking that it is some sort of monitoring code
>pulled into your server via a jar? (I'm confused why it would have its'
>own
> GC cycle...)
>- So are you suggesting I play with my own JVM's (Sun/Oracle) parameters
>to achieve a similar effect ?
>
>Thanks again,
>
>- Chris
>
>
>
>
>
>
>-----Original Message-----
>From: Nader, John P <john.nader [at] cengage>
>To: java-user [at] lucene <java-user [at] lucene>
>Sent: Tue, 15 May 2012 21:12
>Subject: Re: Memory question
>
>
>We've encountered this issue and came up with a fairly good approach to
>address it.
>
>We are on Lucene 3.0.2 with Java 1.6.0_29. Our indices are about 35GB
>in
>size. Our JVM runs at 20GB of heap, with about 12GB of steady usage.
>Our
>server has 32GB total.
>
>What would happen in our case is that the linux would page in more and
>more of the memory mapped index files into memory, forcing idle portions
>of the JVM heap to be swapped out. This was not an issue until our CMS
>GC
>kicked in. This would force swapping in of all JVM memory to collect
>unused references. I/O wait would shoot up and performance would
>suffer.
>Yes, even CMS can kill performance is you are swapping. The tell-tale
>sign was a spike in inbound swap at the start of CMS.
>
>In our case, we addressed the situation using iCMS, which is Incremental
>CMS. This takes the mark phase (and sweep too?) and does it
>continuously
>with a configurable duty cycle. The result was that swapping was
>smoothed
>out to be a small stead drag on the system instead of a hard spike.
>There
>was an small loss in performance, but a big gain in stability.
>
>This tuning may be an option for you. BTW, pmap will give you statistic
>on total file size and how much is resident. The java heap shows up in
>pmap as well on linux, so you can determine how much of that is in
>memory
>as well.
>
>John
>
>
>
>On 5/15/12 3:38 PM, "Chris Bamford" <chris.bamford [at] talktalk> wrote:
>
>>Thanks Uwe.
>>
>>What I'd like to understand is the implications of this on a server
>which
>>opens a large number of indexes over a long period. Will this non-heap
>>memory continue to grow? Will gc be effective at spotting it and
>>releasing it via references in the heap?
>>
>> I had an instance yesterday where a server swapped itself to a
>>standstill and had to be restarted. The load average was through the
>roof
>>and I am trying to understand why. One of my recent changes is updating
>>from 2.3 to 3.6, so naturally I am keen to know the impact of the mmap
>>stuff which is now standard under the covers.
>>
>>My server caches indexsearchers and then closes them based on how full
>>the heap is getting. My worry is that if the bulk of the memory is
>being
>>allocated outside the Jvm, how can I make sensible decisions?
>>
>>Thanks for any pointers / info.
>>
>>Chris
>>
>>
>>
>>-----Original Message-----
>>From: uwe [at] thetaphi
>>To: java-user [at] lucene
>>Sent: Tue, 15 May 2012 18:10
>>Subject: RE: Memory question
>>
>>
>>
>>It mmaps the files into virtual memory if it runs on a 64 bit JVM.
>Because
>>of that you see the mmapped CFS files. This is outside Java Heap and is
>>all
>>*virtual* no RAM is explicitely occupied except the O/S cache.
>>
>>-----
>>Uwe Schindler
>>H.-H.-Meier-Allee 63, D-28213 Bremen
>>http://www.thetaphi.de
>>eMail: uwe [at] thetaphi
>>
>>> -----Original Message-----
>>> From: Chris Bamford [mailto:chris.bamford [at] talktalk]
>>> Sent: Tuesday, May 15, 2012 4:47 PM
>>> To: java-user [at] lucene
>>> Subject: Memory question
>>>
>>> Hi
>>>
>>> Can anyone tell me what happens to the memory when Lucene opens an
>>>index?
>>> Is it loaded into the JVM's heap or is it mapped into virtual memory
>>outside of
>>> it?
>>> I am running on Linux and if I use pmap on the PID of my JVM, I can
>see
>>lots of
>>> entries for index cfs files.
>>>
>>> Does this mean that indexes are mapped into non-heap memory? If so,
>how
>>> can I monitor the space my process is using if I cache open
>>IndexSearchers?
>>>
>>> The details are:
>>>
>>> Sun 64-bit JVM on Linux.
>>> Lucene 3.6 running in 2.3 compatibility mode (as we are in the in the
>>process of
>>> a migration to 3.6)
>>>
>>> Thanks,
>>>
>>> - Chris
>>
>>---------------------------------------------------------------------
>>To unsubscribe, e-mail: java-user-unsubscribe [at] lucene
>>For additional commands, e-mail: java-user-help [at] lucene
>>
>
>
>---------------------------------------------------------------------
>To unsubscribe, e-mail: java-user-unsubscribe [at] lucene
>For additional commands, e-mail: java-user-help [at] lucene
>
>
>
>
>
>---------------------------------------------------------------------
>To unsubscribe, e-mail: java-user-unsubscribe [at] lucene
>For additional commands, e-mail: java-user-help [at] lucene
>


---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe [at] lucene
For additional commands, e-mail: java-user-help [at] lucene


chris at bammers

May 21, 2012, 9:49 AM

Post #13 of 13 (830 views)
Permalink
Re: Memory question [In reply to]

This is a progress update on the issue:

I have tried several things and they all gave improvements. In order of magnitude they are

1) Reduced heap space from 6GB to 3GB.
This on it's own has so far been the biggest win as swapping almost completely stopped after this step.

2) Began limiting the number of cached IndexSearchers. This is because previously with non-mmapped indexes sitting in the heap, I evicted cached items based on heap size calculations - obviously, this is no longer the way to go. Now i just count them - which involves trial and error, but I have arrived at a magic number which seems to have brought swapping completely under control (even under extreme load)

3) Implemented CMS gc behaviour in my JVM which seems to keep the heap pruned nicely without affecting throughput.
Your link to iCMS John is timely as I want to try that next. Will report my findings -- thanks again.

So far so good - cheers to everyone for your valuable suggestions and insight.

Chris

---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe [at] lucene
For additional commands, e-mail: java-user-help [at] lucene

Lucene java-user RSS feed   Index | Next | Previous | View Threaded
 
 


Interested in having your list archived? Contact Gossamer Threads
 
  Web Applications & Managed Hosting Powered by Gossamer Threads Inc.