Login | Register For Free | Help
Search for: (Advanced)

Mailing List Archive: Lucene: Java-User

Websphere and Dark Matter

 

 

Lucene java-user RSS feed   Index | Next | Previous | View Threaded


rollo.dupre at i-nexus

Jan 16, 2007, 8:03 AM

Post #1 of 7 (4820 views)
Permalink
Websphere and Dark Matter

Hi,

Has anyone encountered significant amounts of Websphere Dark Matter
generation when using Lucene?

We have a scenario where a web search app using Lucene causes
Websphere 5.1 allocated memory to grow but not shrink. JProfiler shows
the heap shrinks back ok, leaving the JVM with over 1GB allocated to
the jvm but only 400MB in use. Websphere does not perform a level 2
garbage collection and there is significant dark matter in the
allocated memory.

Thanks in advance for any help.

Rol.
--
Rollo du Pre
Software Architect

www.i-nexus.com

NOTICE: The information contained in this e-mail, including
any attachments, is confidential. If you have received this
in error you should not disclose, copy, circulate or in any
way use the information contained herein. This e-mail may
be legally privileged and unauthorised use may be unlawful.
If you have received this e-mail in error, please telephone
us on +44 (0)870 011 0622 immediately so we can arrange for
its return. Thank you.

In addition, the opinions expressed in this e-mail are those
of the employee and for the intended recipient(s) only. They
do not necessarily represent the opinions of i-solutions
Global Ltd.

---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe [at] lucene
For additional commands, e-mail: java-user-help [at] lucene


jch at scalix

Jan 16, 2007, 8:40 AM

Post #2 of 7 (4624 views)
Permalink
Re: Websphere and Dark Matter [In reply to]

Rollo du Pre wrote:
> We have a scenario where a web search app using Lucene causes
> Websphere 5.1 allocated memory to grow but not shrink. JProfiler shows
> the heap shrinks back ok, leaving the JVM with over 1GB allocated to
> the jvm but only 400MB in use. Websphere does not perform a level 2
> garbage collection and there is significant dark matter in the
> allocated memory.
Are you expecting the process to shrink? They don't normally.

jch

---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe [at] lucene
For additional commands, e-mail: java-user-help [at] lucene


rollo.dupre at i-nexus

Jan 16, 2007, 9:15 AM

Post #3 of 7 (4631 views)
Permalink
Re: Websphere and Dark Matter [In reply to]

Hi John,

I was hoping it would, yes. Does websphere not release memory back to
the OS when it not longer needs it? I'm concerned that if the memory
spiked for some reason (indexing a large document) then that would
hamper the rest of the OS because it'd hold on to far more memory than
is needed.

Thanks,

Rol.

John Haxby wrote:
> Rollo du Pre wrote:
>> We have a scenario where a web search app using Lucene causes
>> Websphere 5.1 allocated memory to grow but not shrink. JProfiler shows
>> the heap shrinks back ok, leaving the JVM with over 1GB allocated to
>> the jvm but only 400MB in use. Websphere does not perform a level 2
>> garbage collection and there is significant dark matter in the
>> allocated memory.
> Are you expecting the process to shrink? They don't normally.
>
> jch
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe [at] lucene
> For additional commands, e-mail: java-user-help [at] lucene
>
>
>

--
Rollo du Pre
Software Architect
rollo.dupre [at] i-nexus

www.i-nexus.com

NOTICE: The information contained in this e-mail, including
any attachments, is confidential. If you have received this
in error you should not disclose, copy, circulate or in any
way use the information contained herein. This e-mail may
be legally privileged and unauthorised use may be unlawful.
If you have received this e-mail in error, please telephone
us on +44 (0)870 011 0622 immediately so we can arrange for
its return. Thank you.

In addition, the opinions expressed in this e-mail are those
of the employee and for the intended recipient(s) only. They
do not necessarily represent the opinions of i-solutions
Global Ltd.

---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe [at] lucene
For additional commands, e-mail: java-user-help [at] lucene


nyh at math

Jan 16, 2007, 11:46 PM

Post #4 of 7 (4611 views)
Permalink
Re: Websphere and Dark Matter [In reply to]

On Tue, Jan 16, 2007, Rollo du Pre wrote about "Re: Websphere and Dark Matter":
> I was hoping it would, yes. Does websphere not release memory back to
> the OS when it not longer needs it? I'm concerned that if the memory
> spiked for some reason (indexing a large document) then that would
> hamper the rest of the OS because it'd hold on to far more memory than
> is needed.

This is a well known, may I say infamous, Java issue. Java could, in theory,
easily shrink its heap as soon as it needs less memory, because in Java's GC
model, memory can be moved around so fragmentation is not a problem. But
unfortunately, the JVM's heap rarely does shrink by default: once the JVM's
heap grows, it rarely ever shrinks back.

It turns out that in Sun's JVM (at least in the 1.4 version which I tried),
you can actually control and improve the shrinking behavior. There's an option
"-XX:MaxHeapFreeRatio=" (see [1]) which basically means how much of the Heap
should be free before it is physically shrunk. Another option
"-XX:MinHeapFreeRatio=" specifies how much of the heap to leave free after
the shrinking.

The default values are 70 and 40 respectively, which means that the JVM only
shrinks its memory use after 70% (!!!) of the heap is free, and it only shrinks
to the point that 40% of the heap is still free. A few numbers to put this
waste into perspective: If your JVM's heap grew to 300 MB, and now only
requires 100 MB, it will NOT shrink (because only 66% is free). If later
you need just 90 MB, the heap will finally shrink, but only to 150 MB (so
that 40% still remains free).

One important thing you should note if you decide to lower these percentages,
is that much of the JVM's excellent GC performance relies on its "generation"
approach and the availability of a lot of free heap (so that by the time the
heap is exausted, most of the new stuff in it is already garbage). So to
keep performance - not just minimal memory use - you might want to tune other
GC parameters as well. A good combination I once used is this:

-XX:NewRatio=2 -XX:MinHeapFreeRatio=20 -XX:MaxHeapFreeRatio=30

But your milage may vary.

[1] http://java.sun.com/javase/technologies/hotspot/vmoptions.jsp

--
Nadav Har'El | Wednesday, Jan 17 2007, 27 Tevet 5767
IBM Haifa Research Lab |-----------------------------------------
|Unix is user friendly - it's just picky
http://nadav.harel.org.il |about its friends.

---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe [at] lucene
For additional commands, e-mail: java-user-help [at] lucene


rollo.dupre at i-nexus

Jan 18, 2007, 2:49 AM

Post #5 of 7 (4633 views)
Permalink
Re: Websphere and Dark Matter [In reply to]

Thanks for this information, it was very useful.

Rol.

Nadav Har'El wrote:
> On Tue, Jan 16, 2007, Rollo du Pre wrote about "Re: Websphere and Dark Matter":
>> I was hoping it would, yes. Does websphere not release memory back to
>> the OS when it not longer needs it? I'm concerned that if the memory
>> spiked for some reason (indexing a large document) then that would
>> hamper the rest of the OS because it'd hold on to far more memory than
>> is needed.
>
> This is a well known, may I say infamous, Java issue. Java could, in theory,
> easily shrink its heap as soon as it needs less memory, because in Java's GC
> model, memory can be moved around so fragmentation is not a problem. But
> unfortunately, the JVM's heap rarely does shrink by default: once the JVM's
> heap grows, it rarely ever shrinks back.
>
> It turns out that in Sun's JVM (at least in the 1.4 version which I tried),
> you can actually control and improve the shrinking behavior. There's an option
> "-XX:MaxHeapFreeRatio=" (see [1]) which basically means how much of the Heap
> should be free before it is physically shrunk. Another option
> "-XX:MinHeapFreeRatio=" specifies how much of the heap to leave free after
> the shrinking.
>
> The default values are 70 and 40 respectively, which means that the JVM only
> shrinks its memory use after 70% (!!!) of the heap is free, and it only shrinks
> to the point that 40% of the heap is still free. A few numbers to put this
> waste into perspective: If your JVM's heap grew to 300 MB, and now only
> requires 100 MB, it will NOT shrink (because only 66% is free). If later
> you need just 90 MB, the heap will finally shrink, but only to 150 MB (so
> that 40% still remains free).
>
> One important thing you should note if you decide to lower these percentages,
> is that much of the JVM's excellent GC performance relies on its "generation"
> approach and the availability of a lot of free heap (so that by the time the
> heap is exausted, most of the new stuff in it is already garbage). So to
> keep performance - not just minimal memory use - you might want to tune other
> GC parameters as well. A good combination I once used is this:
>
> -XX:NewRatio=2 -XX:MinHeapFreeRatio=20 -XX:MaxHeapFreeRatio=30
>
> But your milage may vary.
>
> [1] http://java.sun.com/javase/technologies/hotspot/vmoptions.jsp
>

--
Rollo du Pre
Software Architect
rollo.dupre [at] i-nexus

www.i-nexus.com

NOTICE: The information contained in this e-mail, including
any attachments, is confidential. If you have received this
in error you should not disclose, copy, circulate or in any
way use the information contained herein. This e-mail may
be legally privileged and unauthorised use may be unlawful.
If you have received this e-mail in error, please telephone
us on +44 (0)870 011 0622 immediately so we can arrange for
its return. Thank you.

In addition, the opinions expressed in this e-mail are those
of the employee and for the intended recipient(s) only. They
do not necessarily represent the opinions of i-solutions
Global Ltd.

---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe [at] lucene
For additional commands, e-mail: java-user-help [at] lucene


jch at scalix

Jan 22, 2007, 7:52 AM

Post #6 of 7 (4611 views)
Permalink
Re: Websphere and Dark Matter [In reply to]

Nadav Har'El wrote:
> On Tue, Jan 16, 2007, Rollo du Pre wrote about "Re: Websphere and Dark Matter":
>
>> I was hoping it would, yes. Does websphere not release memory back to
>> the OS when it not longer needs it? I'm concerned that if the memory
>> spiked for some reason (indexing a large document) then that would
>> hamper the rest of the OS because it'd hold on to far more memory than
>> is needed.
>>
>
> This is a well known, may I say infamous, Java issue. Java could, in theory,
> easily shrink its heap as soon as it needs less memory, because in Java's GC
> model, memory can be moved around so fragmentation is not a problem. But
> unfortunately, the JVM's heap rarely does shrink by default: once the JVM's
> heap grows, it rarely ever shrinks back.
>
Are you implying that the process memory shrinks, that memory is
returned to the kernel? I didn't read the page you referenced that way.

I know that if I allocate memory by memory mapping anonymous regions
with Linux/Unix I can give it back, but is that the technique that JVMs use?

It's not generally a problem though. Provided you have a compacting
garbage collector (and the Sun Java GC is one) then the unused memory
will just get paged out. It may be a different story on windows and it's
certainly a different story on an embedded platform, but releasing
memory to the kernel under Linux is not generally necessary or desirable.

jch

---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe [at] lucene
For additional commands, e-mail: java-user-help [at] lucene


nyh at math

Jan 23, 2007, 5:03 AM

Post #7 of 7 (4605 views)
Permalink
Re: Websphere and Dark Matter [In reply to]

On Mon, Jan 22, 2007, John Haxby wrote about "Re: Websphere and Dark Matter":
> Nadav Har'El wrote:
> Are you implying that the process memory shrinks, that memory is
> returned to the kernel? I didn't read the page you referenced that way.
> I know that if I allocate memory by memory mapping anonymous regions
> with Linux/Unix I can give it back, but is that the technique that JVMs use?

Indeed - I've seen this actually happen in Windows (I didn't check on Linux).
Once more than 70% of the heap is free (which is the default setting on Sun's
JVM 1.4) the heap is shrunk and the memory use of the process (according to
Windows "Task Manager") is actually reduced.

You could easily verify whether this happens on Linux as well. Just write
a trivial program that allocates a large, say, 100 MB, object, sleeps for
a few seconds (or runs "ps"), and then "forgets" about this object and runs
System.gc() (perhaps a few times), and then runs ps again and see that the
process shrunk. You might want to use the "-verbose:gc" JVM option to see
some debugging messages about what the GC's do, and how large Java thinks its
heap is. I did not check this myself, so if you have any conclusions, please
report it back to this list - I think it's interesting for many people here.

Anyway, theoretically, Java should have no problem to shrink its memory use
because of the fact that it can (and does) move objects around, so memory
fragmentation is not a problem like it is in many other languages. This is
also possible technically to do in Linux (there are several ways to do this,
including anonymous maps). If Java was Open-Source, I could find out how they
did it (if at all), but last time I checked, it wasn't (I heard this is
changing, though).

> It's not generally a problem though. Provided you have a compacting
> garbage collector (and the Sun Java GC is one) then the unused memory
> will just get paged out. It may be a different story on windows and it's
> certainly a different story on an embedded platform, but releasing
> memory to the kernel under Linux is not generally necessary or desirable.

Believe it or not, this is exactly what every GC expert told me when I
asked this question (how can I force the JVM to reduce its memory use when
it doesn't need all of it): "don't worry about it". This is "conventional
wisdom" second only to "never run System.gc()". But this answer ("virtual
memory means you don't care about the process not shrinking") ignores two
very important issues (not just on embedded platforms, like you suggested):

1. If the JVM heap itself doesn't shrink (it's not just that the memory isn't
returned to the system), Java will continue to fill the entire huge heap,
allowing itself to gather more garbage before a GC is necessary. Since Java
goes over all this memory filling it with garbage all the time, none of
this memory can ever be paged out and "forgotten".

For example, with the default setting (of 70%), it is possible that you
need just 100 MB of memory, but the JVM will keep a 300 MB heap, and
continuously go over the entire 200 MB free heap filling it with soon-to-
become garbage objects.

2. The thinking behind your statement (which, like I said, was shared by most
GC experts I asked) is usually rooted on the idea that the machine is
running one huge JVM. If that is the case, then indeed, nobody cares if
when the JVM's memory needs fluctuates, whether the heep stays large,
whether it gets swapped to disk, or the JVM gets smaller.
But think about a machine running dozens of separate JVMs: this was *NOT*
the intention of the designers of Java, but it is what happens in reality
on modern desktops (and sometimes servers): you run several independent
programs written in Java. In this case, if every one of these programs
takes up the maximum size that it ever needed, even if this waste is just
in the swap space, we can be in trouble. Swap space is "cheap", but not
cheap enough that you don't mind that each Java program you run takes up
hundreds of megabytes of it more than it actually needs.

The use-case that led me to these insights was a desktop search program
written in Java. This program had large peak memory needs (when parsing
large files, and when building the index), at the order of 150 MB. But then,
the JVM did not shrink, and continued to waste 150 MB of memory on each
desktop. This was unacceptable, even if (like you said), when you don't
perform a search the entire 150 MB gets swapped out. It was unacceptable for
several reasons: First, this is not the only program running on the machine,
so what if each one wasted 150 MB of swap space? Second, when you do use
the search, the entire 150 MB heap gets used and swapped in, and worse -
swaps out 150 MB of other useful memory (slowing your work in other
applications).

--
Nadav Har'El | Tuesday, Jan 23 2007, 4 Shevat 5767
IBM Haifa Research Lab |-----------------------------------------
|"Do you want to restart Windows now or
http://nadav.harel.org.il |wait for the next crash?"

---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe [at] lucene
For additional commands, e-mail: java-user-help [at] lucene

Lucene java-user RSS feed   Index | Next | Previous | View Threaded
 
 


Interested in having your list archived? Contact Gossamer Threads
 
  Web Applications & Managed Hosting Powered by Gossamer Threads Inc.