Mailing List Archive: Lucene: Java-User

Throughput doesn't increase when using more concurrent threads

1 2 3

View All

Index | Next | Previous | View Threaded

shoren at gmail

Nov 21, 2005, 7:23 AM

Post #1 of 62 (3638 views)
Permalink

Throughput doesn't increase when using more concurrent threads

Hi,

I tried stressing Lucene in a controlled environment: one static
IndexSearcher for an index that doesn't change, and in same process I create
a number of Threads that call this Searcher concurrently for a limited time.
I expected the number of successful queries to increase when using more
threads, but this is not the case. From 1 thread to 10 I see 25% increase,
but from 10 threads to 100 there is no change, only the average response
time increases. same goes for 200 threads. I tried RAMDirectory and
FSDirectory, and the behavior is the same.
I Extract the first 100 results from the Hits object, but on RAMDirectory
this should be insignificant, right?

I tested this in version 1.4.3 and 1.9rc1, and they are both the same in
this aspect. 1.9rc1 is faster, but does not benefit from multi threading.

Did anyone see other behaviour?

Will it be better to dedicate a searcher for each thread (maybe
http://java.sun.com/j2se/1.4.2/docs/api/java/lang/ThreadLocal.html)?

Thanks,
Oren Shir

me at gekkokid

Nov 21, 2005, 7:41 AM

Post #2 of 62 (3524 views)
Permalink

Re: Throughput doesn't increase when using more concurrent threads [In reply to]

Oren Shir wrote:
> I tested this in version 1.4.3 and 1.9rc1, and they are both the same in
> this aspect. 1.9rc1 is faster, but does not benefit from multi threading.

some newbie questions i have,
does 1.4.3 benefit from multi-threading?
is 1.9 the version in the source repository?

_gk

---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe [at] lucene
For additional commands, e-mail: java-user-help [at] lucene

yseeley at gmail

Nov 21, 2005, 7:43 AM

Post #3 of 62 (3544 views)
Permalink

Re: Throughput doesn't increase when using more concurrent threads [In reply to]

This is expected behavior: you are probably quickly becoming CPU bound
(which isn't a bad thing). More threads only help when some threads
are waiting on IO, or if you actually have a lot of CPUs in the box.

-Yonik
Now hiring -- http://forms.cnet.com/slink?231706

On 11/21/05, Oren Shir <shoren [at] gmail> wrote:
> Hi,
>
> I tried stressing Lucene in a controlled environment: one static
> IndexSearcher for an index that doesn't change, and in same process I create
> a number of Threads that call this Searcher concurrently for a limited time.
> I expected the number of successful queries to increase when using more
> threads, but this is not the case. From 1 thread to 10 I see 25% increase,
> but from 10 threads to 100 there is no change, only the average response
> time increases. same goes for 200 threads. I tried RAMDirectory and
> FSDirectory, and the behavior is the same.
> I Extract the first 100 results from the Hits object, but on RAMDirectory
> this should be insignificant, right?
>
> I tested this in version 1.4.3 and 1.9rc1, and they are both the same in
> this aspect. 1.9rc1 is faster, but does not benefit from multi threading.
>
> Did anyone see other behaviour?
>
> Will it be better to dedicate a searcher for each thread (maybe
> http://java.sun.com/j2se/1.4.2/docs/api/java/lang/ThreadLocal.html)?
>
>
> Thanks,
> Oren Shir
>
>

---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe [at] lucene
For additional commands, e-mail: java-user-help [at] lucene

shoren at gmail

Nov 21, 2005, 8:01 AM

Post #4 of 62 (3541 views)
Permalink

Re: Throughput doesn't increase when using more concurrent threads [In reply to]

gekkokid,

does 1.4.3 benefit from multi-threading?
>

Sorry for not being clear. My tests show that both version does not benefit
from multi threading, but it is possible that I'm CPU bound, as Yonik kindly
reminded me.

is 1.9 the version in the source repository?

1.9 is the version in source repository.

It is rather sad if 10 threads reach the CPU limit. I'll check it and get
back to you.

Thanks,
Oren Shir

yseeley at gmail

Nov 21, 2005, 8:07 AM

Post #5 of 62 (3531 views)
Permalink

Re: Throughput doesn't increase when using more concurrent threads [In reply to]

On 11/21/05, Oren Shir <shoren [at] gmail> wrote:
> It is rather sad if 10 threads reach the CPU limit. I'll check it and get
> back to you.

It's about performance and throughput though, not about number of
threads it takes to reach saturation.

In a 2 CPU box, I would say that the ideal situation is where it only
takes two threads to reach 100% CPU utilization. Normally it takes
more because of some kind of IO (disk or network).

-Yonik
Now hiring -- http://forms.cnet.com/slink?231706

---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe [at] lucene
For additional commands, e-mail: java-user-help [at] lucene

jbooth at net-temps

Nov 21, 2005, 8:35 AM

Post #6 of 62 (3537 views)
Permalink

RE: Throughput doesn't increase when using more concurrent threads [In reply to]

I had a similar problem with threading, the problem turned out to be that in
the back end of the FSDirectory class I believe it was, there was a
synchronized block on the actual RandomAccessFile resource when reading a
block of data from it... high-concurrency situations caused threads to stack
up in front of this synchronized block and our CPU time wound up being spent
thrashing between blocked threads instead of doing anything useful.

Making multiple IndexSearchers and FSDirectories didn't help because in the
back end, lucene consults a singleton HashMap of some kind (don't remember
implementation) that maintained a single FSDirectory for any given index
being accessed from the JVM... multiple calls to FSDirectory.getDirectory
actually return the same FSDirectory object with synchronization at the same
point.

Our solution was to subclass FSDirectory to skip this lookup step, and then
create several distinct FSDirectories reading the same index. This was in a
read only context so it worked out for us, although memory consumption was
higher... we put a pooling class in front of the whole thing and added a
refresh method to re-instantiate the IndexSearchers and associated
FSDirectories when the index was modified... we also needed some
catch(FileNotFoundException) blocks to hack around the case where the index
was being modified while an IndexSearcher was trying to search.

-----Original Message-----
From: Yonik Seeley [mailto:yseeley [at] gmail]
Sent: Monday, November 21, 2005 11:08 AM
To: java-user [at] lucene; shoren [at] alum
Subject: Re: Throughput doesn't increase when using more concurrent
threads

On 11/21/05, Oren Shir <shoren [at] gmail> wrote:
> It is rather sad if 10 threads reach the CPU limit. I'll check it and get
> back to you.

It's about performance and throughput though, not about number of
threads it takes to reach saturation.

In a 2 CPU box, I would say that the ideal situation is where it only
takes two threads to reach 100% CPU utilization. Normally it takes
more because of some kind of IO (disk or network).

-Yonik
Now hiring -- http://forms.cnet.com/slink?231706

---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe [at] lucene
For additional commands, e-mail: java-user-help [at] lucene

---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe [at] lucene
For additional commands, e-mail: java-user-help [at] lucene

shoren at gmail

Nov 21, 2005, 8:51 AM

Post #7 of 62 (3543 views)
Permalink

Re: Throughput doesn't increase when using more concurrent threads [In reply to]

Thanks Jay Booth,

I thought as much. I just verified that I'm not reaching 100% CPU, and I
found out that when using RAMDirectory and 100 threads the CPU usage is 60%,
avarage request time 40 times more that one thread, but number of requests
the same. I think I'll have to do somthing like you suggested. Any thoughts
about adding this functionality to Lucene core?

Thanks again,
Oren Shir

On 11/21/05, Jay Booth <jbooth [at] net-temps> wrote:
>
> I had a similar problem with threading, the problem turned out to be that
> in
> the back end of the FSDirectory class I believe it was, there was a
> synchronized block on the actual RandomAccessFile resource when reading a
> block of data from it... high-concurrency situations caused threads to
> stack
> up in front of this synchronized block and our CPU time wound up being
> spent
> thrashing between blocked threads instead of doing anything useful.
>
> Making multiple IndexSearchers and FSDirectories didn't help because in
> the
> back end, lucene consults a singleton HashMap of some kind (don't remember
> implementation) that maintained a single FSDirectory for any given index
> being accessed from the JVM... multiple calls to FSDirectory.getDirectory
> actually return the same FSDirectory object with synchronization at the
> same
> point.
>
> Our solution was to subclass FSDirectory to skip this lookup step, and
> then
> create several distinct FSDirectories reading the same index. This was in
> a
> read only context so it worked out for us, although memory consumption was
> higher... we put a pooling class in front of the whole thing and added a
> refresh method to re-instantiate the IndexSearchers and associated
> FSDirectories when the index was modified... we also needed some
> catch(FileNotFoundException) blocks to hack around the case where the
> index
> was being modified while an IndexSearcher was trying to search.
>
> -----Original Message-----
> From: Yonik Seeley [mailto:yseeley [at] gmail]
> Sent: Monday, November 21, 2005 11:08 AM
> To: java-user [at] lucene; shoren [at] alum
> Subject: Re: Throughput doesn't increase when using more concurrent
> threads
>
>
> On 11/21/05, Oren Shir <shoren [at] gmail> wrote:
> > It is rather sad if 10 threads reach the CPU limit. I'll check it and
> get
> > back to you.
>
> It's about performance and throughput though, not about number of
> threads it takes to reach saturation.
>
> In a 2 CPU box, I would say that the ideal situation is where it only
> takes two threads to reach 100% CPU utilization. Normally it takes
> more because of some kind of IO (disk or network).
>
> -Yonik
> Now hiring -- http://forms.cnet.com/slink?231706
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe [at] lucene
> For additional commands, e-mail: java-user-help [at] lucene
>
>
>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe [at] lucene
> For additional commands, e-mail: java-user-help [at] lucene
>
>

cutting at apache

Nov 21, 2005, 9:43 AM

Post #8 of 62 (3532 views)
Permalink

Re: Throughput doesn't increase when using more concurrent threads [In reply to]

Jay Booth wrote:
> I had a similar problem with threading, the problem turned out to be that in
> the back end of the FSDirectory class I believe it was, there was a
> synchronized block on the actual RandomAccessFile resource when reading a
> block of data from it... high-concurrency situations caused threads to stack
> up in front of this synchronized block and our CPU time wound up being spent
> thrashing between blocked threads instead of doing anything useful.

This is correct. In Lucene, multiple streams per file are created by
cloning, and all clones of an FSDirectory input stream share a
RandomAccessFile and must synchronize input from it. MmapDirectory does
not have this limitation. If your indexes are less than a few GB or you
are using 64-bit hardware, then MmapDirectory should work well for you.
Otherwise it would be simple to write an nio-based Directory that does
not use mmap that is also unsynchronized. Such a contribution would be
welcome.

> Making multiple IndexSearchers and FSDirectories didn't help because in the
> back end, lucene consults a singleton HashMap of some kind (don't remember
> implementation) that maintained a single FSDirectory for any given index
> being accessed from the JVM... multiple calls to FSDirectory.getDirectory
> actually return the same FSDirectory object with synchronization at the same
> point.

This does not make sense to me. FSDirectory does keep a cache of
FSDirectory instances, but i/o should not be synchronized on these. One
should be able to open multiple input streams on the same file from an
FSDirectory. But this would not be a great solution, since file handle
limits would soon become a problem.

Doug

---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe [at] lucene
For additional commands, e-mail: java-user-help [at] lucene

shoren at gmail

Nov 22, 2005, 6:33 AM

Post #9 of 62 (3535 views)
Permalink

Re: Throughput doesn't increase when using more concurrent threads [In reply to]

Hi,

There are two sunchronization points: on the stream and on the reader. Using
different FSDirectoriy and IndexReaders should solve this. I'll let you know
once I code it. Right now I'm checking if making my Documents store less
data will move the bottleneck to some other place.

Thanks again,
Oren Shir

On 11/21/05, Doug Cutting <cutting [at] apache> wrote:
>
> Jay Booth wrote:
> > I had a similar problem with threading, the problem turned out to be
> that in
> > the back end of the FSDirectory class I believe it was, there was a
> > synchronized block on the actual RandomAccessFile resource when reading
> a
> > block of data from it... high-concurrency situations caused threads to
> stack
> > up in front of this synchronized block and our CPU time wound up being
> spent
> > thrashing between blocked threads instead of doing anything useful.
>
> This is correct. In Lucene, multiple streams per file are created by
> cloning, and all clones of an FSDirectory input stream share a
> RandomAccessFile and must synchronize input from it. MmapDirectory does
> not have this limitation. If your indexes are less than a few GB or you
> are using 64-bit hardware, then MmapDirectory should work well for you.
> Otherwise it would be simple to write an nio-based Directory that does
> not use mmap that is also unsynchronized. Such a contribution would be
> welcome.
>
> > Making multiple IndexSearchers and FSDirectories didn't help because in
> the
> > back end, lucene consults a singleton HashMap of some kind (don't
> remember
> > implementation) that maintained a single FSDirectory for any given index
> > being accessed from the JVM... multiple calls to
> FSDirectory.getDirectory
> > actually return the same FSDirectory object with synchronization at the
> same
> > point.
>
> This does not make sense to me. FSDirectory does keep a cache of
> FSDirectory instances, but i/o should not be synchronized on these. One
> should be able to open multiple input streams on the same file from an
> FSDirectory. But this would not be a great solution, since file handle
> limits would soon become a problem.
>
> Doug
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe [at] lucene
> For additional commands, e-mail: java-user-help [at] lucene
>
>

peterlkeegan at gmail

Jan 25, 2006, 10:50 AM

Post #10 of 62 (3450 views)
Permalink

Re: Throughput doesn't increase when using more concurrent threads [In reply to]

This is just fyi - in my stress tests on a 8-cpu box (that's 8 real cpus),
the maximum throughput occurred with just 4 query threads. The query
throughput decreased with fewer than 4 or greater than 4 query threads. The
entire index was most likely in the file system cache, too. Periodic
snapshots of stack traces showed most threads blocked in the synchronization
in: FSIndexInput.readInternal(), when the thread count exceeded 4.

Peter

On 11/22/05, Oren Shir <shoren [at] gmail> wrote:
>
> Hi,
>
> There are two sunchronization points: on the stream and on the reader.
> Using
> different FSDirectoriy and IndexReaders should solve this. I'll let you
> know
> once I code it. Right now I'm checking if making my Documents store less
> data will move the bottleneck to some other place.
>
> Thanks again,
> Oren Shir
>
> On 11/21/05, Doug Cutting <cutting [at] apache> wrote:
> >
> > Jay Booth wrote:
> > > I had a similar problem with threading, the problem turned out to be
> > that in
> > > the back end of the FSDirectory class I believe it was, there was a
> > > synchronized block on the actual RandomAccessFile resource when
> reading
> > a
> > > block of data from it... high-concurrency situations caused threads to
> > stack
> > > up in front of this synchronized block and our CPU time wound up being
> > spent
> > > thrashing between blocked threads instead of doing anything useful.
> >
> > This is correct. In Lucene, multiple streams per file are created by
> > cloning, and all clones of an FSDirectory input stream share a
> > RandomAccessFile and must synchronize input from it. MmapDirectory does
> > not have this limitation. If your indexes are less than a few GB or you
> > are using 64-bit hardware, then MmapDirectory should work well for you.
> > Otherwise it would be simple to write an nio-based Directory that does
> > not use mmap that is also unsynchronized. Such a contribution would be
> > welcome.
> >
> > > Making multiple IndexSearchers and FSDirectories didn't help because
> in
> > the
> > > back end, lucene consults a singleton HashMap of some kind (don't
> > remember
> > > implementation) that maintained a single FSDirectory for any given
> index
> > > being accessed from the JVM... multiple calls to
> > FSDirectory.getDirectory
> > > actually return the same FSDirectory object with synchronization at
> the
> > same
> > > point.
> >
> > This does not make sense to me. FSDirectory does keep a cache of
> > FSDirectory instances, but i/o should not be synchronized on these. One
> > should be able to open multiple input streams on the same file from an
> > FSDirectory. But this would not be a great solution, since file handle
> > limits would soon become a problem.
> >
> > Doug
> >
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: java-user-unsubscribe [at] lucene
> > For additional commands, e-mail: java-user-help [at] lucene
> >
> >
>
>

yseeley at gmail

Jan 25, 2006, 11:07 AM

Post #11 of 62 (3453 views)
Permalink

Re: Throughput doesn't increase when using more concurrent threads [In reply to]

Thanks Peter, that's useful info.

Just out of curiosity, what kind of box is this? what CPUs?

-Yonik

On 1/25/06, Peter Keegan <peterlkeegan [at] gmail> wrote:
> This is just fyi - in my stress tests on a 8-cpu box (that's 8 real cpus),
> the maximum throughput occurred with just 4 query threads. The query
> throughput decreased with fewer than 4 or greater than 4 query threads. The
> entire index was most likely in the file system cache, too. Periodic
> snapshots of stack traces showed most threads blocked in the synchronization
> in: FSIndexInput.readInternal(), when the thread count exceeded 4.
>
> Peter
>
>
> On 11/22/05, Oren Shir <shoren [at] gmail> wrote:
> >
> > Hi,
> >
> > There are two sunchronization points: on the stream and on the reader.
> > Using
> > different FSDirectoriy and IndexReaders should solve this. I'll let you
> > know
> > once I code it. Right now I'm checking if making my Documents store less
> > data will move the bottleneck to some other place.
> >
> > Thanks again,
> > Oren Shir
> >
> > On 11/21/05, Doug Cutting <cutting [at] apache> wrote:
> > >
> > > Jay Booth wrote:
> > > > I had a similar problem with threading, the problem turned out to be
> > > that in
> > > > the back end of the FSDirectory class I believe it was, there was a
> > > > synchronized block on the actual RandomAccessFile resource when
> > reading
> > > a
> > > > block of data from it... high-concurrency situations caused threads to
> > > stack
> > > > up in front of this synchronized block and our CPU time wound up being
> > > spent
> > > > thrashing between blocked threads instead of doing anything useful.
> > >
> > > This is correct. In Lucene, multiple streams per file are created by
> > > cloning, and all clones of an FSDirectory input stream share a
> > > RandomAccessFile and must synchronize input from it. MmapDirectory does
> > > not have this limitation. If your indexes are less than a few GB or you
> > > are using 64-bit hardware, then MmapDirectory should work well for you.
> > > Otherwise it would be simple to write an nio-based Directory that does
> > > not use mmap that is also unsynchronized. Such a contribution would be
> > > welcome.
> > >
> > > > Making multiple IndexSearchers and FSDirectories didn't help because
> > in
> > > the
> > > > back end, lucene consults a singleton HashMap of some kind (don't
> > > remember
> > > > implementation) that maintained a single FSDirectory for any given
> > index
> > > > being accessed from the JVM... multiple calls to
> > > FSDirectory.getDirectory
> > > > actually return the same FSDirectory object with synchronization at
> > the
> > > same
> > > > point.
> > >
> > > This does not make sense to me. FSDirectory does keep a cache of
> > > FSDirectory instances, but i/o should not be synchronized on these. One
> > > should be able to open multiple input streams on the same file from an
> > > FSDirectory. But this would not be a great solution, since file handle
> > > limits would soon become a problem.
> > >
> > > Doug
> > >
> > > ---------------------------------------------------------------------
> > > To unsubscribe, e-mail: java-user-unsubscribe [at] lucene
> > > For additional commands, e-mail: java-user-help [at] lucene

---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe [at] lucene
For additional commands, e-mail: java-user-help [at] lucene

cutting at apache

Jan 25, 2006, 11:14 AM

Post #12 of 62 (3430 views)
Permalink

Re: Throughput doesn't increase when using more concurrent threads [In reply to]

Peter Keegan wrote:
> This is just fyi - in my stress tests on a 8-cpu box (that's 8 real cpus),
> the maximum throughput occurred with just 4 query threads. The query
> throughput decreased with fewer than 4 or greater than 4 query threads. The
> entire index was most likely in the file system cache, too. Periodic
> snapshots of stack traces showed most threads blocked in the synchronization
> in: FSIndexInput.readInternal(), when the thread count exceeded 4.

Was this with a compound or non-compound format index? The non-compound
should fare slightly better, since there are more file handles per
index. Did you try using MMapDirectory? This should have no i/o
concurrency limits, but, on 32-bit systems, only works with indexes less
than a few GB.

Doug

---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe [at] lucene
For additional commands, e-mail: java-user-help [at] lucene

peterlkeegan at gmail

Jan 25, 2006, 11:48 AM

Post #13 of 62 (3440 views)
Permalink

Re: Throughput doesn't increase when using more concurrent threads [In reply to]

It's a 3GHz Intel box with Xeon processors, 64GB ram :)

Peter

On 1/25/06, Yonik Seeley <yseeley [at] gmail> wrote:
>
> Thanks Peter, that's useful info.
>
> Just out of curiosity, what kind of box is this? what CPUs?
>
> -Yonik
>
> On 1/25/06, Peter Keegan <peterlkeegan [at] gmail> wrote:
> > This is just fyi - in my stress tests on a 8-cpu box (that's 8 real
> cpus),
> > the maximum throughput occurred with just 4 query threads. The query
> > throughput decreased with fewer than 4 or greater than 4 query threads.
> The
> > entire index was most likely in the file system cache, too. Periodic
> > snapshots of stack traces showed most threads blocked in the
> synchronization
> > in: FSIndexInput.readInternal(), when the thread count exceeded 4.
> >
> > Peter
> >
> >
> > On 11/22/05, Oren Shir <shoren [at] gmail> wrote:
> > >
> > > Hi,
> > >
> > > There are two sunchronization points: on the stream and on the reader.
> > > Using
> > > different FSDirectoriy and IndexReaders should solve this. I'll let
> you
> > > know
> > > once I code it. Right now I'm checking if making my Documents store
> less
> > > data will move the bottleneck to some other place.
> > >
> > > Thanks again,
> > > Oren Shir
> > >
> > > On 11/21/05, Doug Cutting <cutting [at] apache> wrote:
> > > >
> > > > Jay Booth wrote:
> > > > > I had a similar problem with threading, the problem turned out to
> be
> > > > that in
> > > > > the back end of the FSDirectory class I believe it was, there was
> a
> > > > > synchronized block on the actual RandomAccessFile resource when
> > > reading
> > > > a
> > > > > block of data from it... high-concurrency situations caused
> threads to
> > > > stack
> > > > > up in front of this synchronized block and our CPU time wound up
> being
> > > > spent
> > > > > thrashing between blocked threads instead of doing anything
> useful.
> > > >
> > > > This is correct. In Lucene, multiple streams per file are created by
> > > > cloning, and all clones of an FSDirectory input stream share a
> > > > RandomAccessFile and must synchronize input from it. MmapDirectory
> does
> > > > not have this limitation. If your indexes are less than a few GB or
> you
> > > > are using 64-bit hardware, then MmapDirectory should work well for
> you.
> > > > Otherwise it would be simple to write an nio-based Directory that
> does
> > > > not use mmap that is also unsynchronized. Such a contribution would
> be
> > > > welcome.
> > > >
> > > > > Making multiple IndexSearchers and FSDirectories didn't help
> because
> > > in
> > > > the
> > > > > back end, lucene consults a singleton HashMap of some kind (don't
> > > > remember
> > > > > implementation) that maintained a single FSDirectory for any given
> > > index
> > > > > being accessed from the JVM... multiple calls to
> > > > FSDirectory.getDirectory
> > > > > actually return the same FSDirectory object with synchronization
> at
> > > the
> > > > same
> > > > > point.
> > > >
> > > > This does not make sense to me. FSDirectory does keep a cache of
> > > > FSDirectory instances, but i/o should not be synchronized on these.
> One
> > > > should be able to open multiple input streams on the same file from
> an
> > > > FSDirectory. But this would not be a great solution, since file
> handle
> > > > limits would soon become a problem.
> > > >
> > > > Doug
> > > >
> > > >
> ---------------------------------------------------------------------
> > > > To unsubscribe, e-mail: java-user-unsubscribe [at] lucene
> > > > For additional commands, e-mail: java-user-help [at] lucene
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe [at] lucene
> For additional commands, e-mail: java-user-help [at] lucene
>
>

peterlkeegan at gmail

Jan 25, 2006, 11:51 AM

Post #14 of 62 (3442 views)
Permalink

Re: Throughput doesn't increase when using more concurrent threads [In reply to]

The index is non-compound format and optimized. Yes, I did try
MMapDirectory, but the index is too big - 3.5 GB (1.3GB is term vectors)

Peter

On 1/25/06, Doug Cutting <cutting [at] apache> wrote:
>
> Peter Keegan wrote:
> > This is just fyi - in my stress tests on a 8-cpu box (that's 8 real
> cpus),
> > the maximum throughput occurred with just 4 query threads. The query
> > throughput decreased with fewer than 4 or greater than 4 query threads.
> The
> > entire index was most likely in the file system cache, too. Periodic
> > snapshots of stack traces showed most threads blocked in the
> synchronization
> > in: FSIndexInput.readInternal(), when the thread count exceeded 4.
>
> Was this with a compound or non-compound format index? The non-compound
> should fare slightly better, since there are more file handles per
> index. Did you try using MMapDirectory? This should have no i/o
> concurrency limits, but, on 32-bit systems, only works with indexes less
> than a few GB.
>
> Doug
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe [at] lucene
> For additional commands, e-mail: java-user-help [at] lucene
>
>

yseeley at gmail

Jan 25, 2006, 12:06 PM

Post #15 of 62 (3439 views)
Permalink

Re: Throughput doesn't increase when using more concurrent threads [In reply to]

On 1/25/06, Peter Keegan <peterlkeegan [at] gmail> wrote:
> It's a 3GHz Intel box with Xeon processors, 64GB ram :)

Nice!

Xeon processors are normally hyperthreaded. On a linux box, if you
cat /proc/cpuinfo, you will see 8 processors for a 4 physical CPU
system. Are you positive you have 8 physical Xeon processors?

-Yonik

---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe [at] lucene
For additional commands, e-mail: java-user-help [at] lucene

peterlkeegan at gmail

Jan 25, 2006, 12:20 PM

Post #16 of 62 (3437 views)
Permalink

Re: Throughput doesn't increase when using more concurrent threads [In reply to]

Yes, it's hyperthreaded (16 cpus show up in task manager - the box is
running 2003). I plan to turn off hyperthreading to see if it has any
effect.

Peter

On 1/25/06, Yonik Seeley <yseeley [at] gmail> wrote:
>
> On 1/25/06, Peter Keegan <peterlkeegan [at] gmail> wrote:
> > It's a 3GHz Intel box with Xeon processors, 64GB ram :)
>
> Nice!
>
> Xeon processors are normally hyperthreaded. On a linux box, if you
> cat /proc/cpuinfo, you will see 8 processors for a 4 physical CPU
> system. Are you positive you have 8 physical Xeon processors?
>
> -Yonik
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe [at] lucene
> For additional commands, e-mail: java-user-help [at] lucene
>
>

paul.elschot at xs4all

Jan 25, 2006, 11:59 PM

Post #17 of 62 (3437 views)
Permalink

Re: Throughput doesn't increase when using more concurrent threads [In reply to]

On Wednesday 25 January 2006 20:51, Peter Keegan wrote:
> The index is non-compound format and optimized. Yes, I did try
> MMapDirectory, but the index is too big - 3.5 GB (1.3GB is term vectors)
>
> Peter
>
You could also give this a try:

http://issues.apache.org/jira/browse/LUCENE-283

Regards,
Paul Elschot

---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe [at] lucene
For additional commands, e-mail: java-user-help [at] lucene

saturnism at gmail

Jan 26, 2006, 12:43 AM

Post #18 of 62 (3436 views)
Permalink

Re: Throughput doesn't increase when using more concurrent threads [In reply to]

Speaking of NioFSDirectory, I thought there was one posted a while
ago, is this something that can be used?
http://issues.apache.org/jira/browse/LUCENE-414

ray,

On 11/22/05, Doug Cutting <cutting [at] apache> wrote:
> Jay Booth wrote:
> > I had a similar problem with threading, the problem turned out to be that in
> > the back end of the FSDirectory class I believe it was, there was a
> > synchronized block on the actual RandomAccessFile resource when reading a
> > block of data from it... high-concurrency situations caused threads to stack
> > up in front of this synchronized block and our CPU time wound up being spent
> > thrashing between blocked threads instead of doing anything useful.
>
> This is correct. In Lucene, multiple streams per file are created by
> cloning, and all clones of an FSDirectory input stream share a
> RandomAccessFile and must synchronize input from it. MmapDirectory does
> not have this limitation. If your indexes are less than a few GB or you
> are using 64-bit hardware, then MmapDirectory should work well for you.
> Otherwise it would be simple to write an nio-based Directory that does
> not use mmap that is also unsynchronized. Such a contribution would be
> welcome.
>
> > Making multiple IndexSearchers and FSDirectories didn't help because in the
> > back end, lucene consults a singleton HashMap of some kind (don't remember
> > implementation) that maintained a single FSDirectory for any given index
> > being accessed from the JVM... multiple calls to FSDirectory.getDirectory
> > actually return the same FSDirectory object with synchronization at the same
> > point.
>
> This does not make sense to me. FSDirectory does keep a cache of
> FSDirectory instances, but i/o should not be synchronized on these. One
> should be able to open multiple input streams on the same file from an
> FSDirectory. But this would not be a great solution, since file handle
> limits would soon become a problem.
>
> Doug
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe [at] lucene
> For additional commands, e-mail: java-user-help [at] lucene
>
>

peterlkeegan at gmail

Jan 26, 2006, 7:49 AM

Post #19 of 62 (3445 views)
Permalink

Re: Throughput doesn't increase when using more concurrent threads [In reply to]

Paul,

I tried this but it ran out of memory trying to read the 500Mb .fdt file. I
tried various values for MAX_BBUF, but it still ran out of memory (I'm using
-Xmx1600M, which is the jvm's maximum value (v1.5)) I'll give
NioFSDirectory a try.

Thanks,
Peter

On 1/26/06, Paul Elschot <paul.elschot [at] xs4all> wrote:
>
> On Wednesday 25 January 2006 20:51, Peter Keegan wrote:
> > The index is non-compound format and optimized. Yes, I did try
> > MMapDirectory, but the index is too big - 3.5 GB (1.3GB is term vectors)
> >
> > Peter
> >
> You could also give this a try:
>
> http://issues.apache.org/jira/browse/LUCENE-283
>
> Regards,
> Paul Elschot
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe [at] lucene
> For additional commands, e-mail: java-user-help [at] lucene
>
>

peterlkeegan at gmail

Jan 26, 2006, 7:56 AM

Post #20 of 62 (3442 views)
Permalink

Re: Throughput doesn't increase when using more concurrent threads [In reply to]

Ray,

The throughput is worse with NioFSDIrectory than with the FSDIrectory
(patched and unpatched). The bottleneck still seems to be synchronization,
this time in NioFile.getChannel (7 of the 8 threads were blocked there
during one snapshot). I tried this with 4 and 8 channels.

The throughput with the patched FSDirectory was about the same as before the
patch.

Thanks,
Peter

On 1/26/06, Ray Tsang <saturnism [at] gmail> wrote:
>
> Speaking of NioFSDirectory, I thought there was one posted a while
> ago, is this something that can be used?
> http://issues.apache.org/jira/browse/LUCENE-414
>
> ray,
>
> On 11/22/05, Doug Cutting <cutting [at] apache> wrote:
> > Jay Booth wrote:
> > > I had a similar problem with threading, the problem turned out to be
> that in
> > > the back end of the FSDirectory class I believe it was, there was a
> > > synchronized block on the actual RandomAccessFile resource when
> reading a
> > > block of data from it... high-concurrency situations caused threads to
> stack
> > > up in front of this synchronized block and our CPU time wound up being
> spent
> > > thrashing between blocked threads instead of doing anything useful.
> >
> > This is correct. In Lucene, multiple streams per file are created by
> > cloning, and all clones of an FSDirectory input stream share a
> > RandomAccessFile and must synchronize input from it. MmapDirectory does
> > not have this limitation. If your indexes are less than a few GB or you
> > are using 64-bit hardware, then MmapDirectory should work well for you.
> > Otherwise it would be simple to write an nio-based Directory that does
> > not use mmap that is also unsynchronized. Such a contribution would be
> > welcome.
> >
> > > Making multiple IndexSearchers and FSDirectories didn't help because
> in the
> > > back end, lucene consults a singleton HashMap of some kind (don't
> remember
> > > implementation) that maintained a single FSDirectory for any given
> index
> > > being accessed from the JVM... multiple calls to
> FSDirectory.getDirectory
> > > actually return the same FSDirectory object with synchronization at
> the same
> > > point.
> >
> > This does not make sense to me. FSDirectory does keep a cache of
> > FSDirectory instances, but i/o should not be synchronized on these. One
> > should be able to open multiple input streams on the same file from an
> > FSDirectory. But this would not be a great solution, since file handle
> > limits would soon become a problem.
> >
> > Doug
> >
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: java-user-unsubscribe [at] lucene
> > For additional commands, e-mail: java-user-help [at] lucene
> >
> >
>

yseeley at gmail

Jan 26, 2006, 7:58 AM

Post #21 of 62 (3434 views)
Permalink

Re: Throughput doesn't increase when using more concurrent threads [In reply to]

Hmmm, can you run the 64 bit version of Windows (and hence a 64 bit JVM?)
We're running with heap sizes up to 8GB (RH Linux 64 bit, Opterons,
Sun Java 1.5)

-Yonik

On 1/26/06, Peter Keegan <peterlkeegan [at] gmail> wrote:
> Paul,
>
> I tried this but it ran out of memory trying to read the 500Mb .fdt file. I
> tried various values for MAX_BBUF, but it still ran out of memory (I'm using
> -Xmx1600M, which is the jvm's maximum value (v1.5)) I'll give
> NioFSDirectory a try.
>
> Thanks,
> Peter
>
>
> On 1/26/06, Paul Elschot <paul.elschot [at] xs4all> wrote:
> >
> > On Wednesday 25 January 2006 20:51, Peter Keegan wrote:
> > > The index is non-compound format and optimized. Yes, I did try
> > > MMapDirectory, but the index is too big - 3.5 GB (1.3GB is term vectors)
> > >
> > > Peter
> > >
> > You could also give this a try:
> >
> > http://issues.apache.org/jira/browse/LUCENE-283
> >
> > Regards,
> > Paul Elschot
> >
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: java-user-unsubscribe [at] lucene
> > For additional commands, e-mail: java-user-help [at] lucene
> >
> >
>
>

---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe [at] lucene
For additional commands, e-mail: java-user-help [at] lucene

peterlkeegan at gmail

Jan 26, 2006, 9:53 AM

Post #22 of 62 (3449 views)
Permalink

Re: Throughput doesn't increase when using more concurrent threads [In reply to]

I'd love to try this, but I'm not aware of any 64-bit jvms for Windows on
Intel. If you know of any, please let me know. Linux may be an option, too.

btw, I'm getting a sustained rate of 135 queries/sec with 4 threads, which
is pretty impressive. Another way around the concurrency limit is to run
multiple jvms. The throughput of each is less, but the aggregate throughput
is higher.

Peter

On 1/26/06, Yonik Seeley <yseeley [at] gmail> wrote:
>
> Hmmm, can you run the 64 bit version of Windows (and hence a 64 bit JVM?)
> We're running with heap sizes up to 8GB (RH Linux 64 bit, Opterons,
> Sun Java 1.5)
>
> -Yonik
>
> On 1/26/06, Peter Keegan <peterlkeegan [at] gmail> wrote:
> > Paul,
> >
> > I tried this but it ran out of memory trying to read the 500Mb .fdt
> file. I
> > tried various values for MAX_BBUF, but it still ran out of memory (I'm
> using
> > -Xmx1600M, which is the jvm's maximum value (v1.5)) I'll give
> > NioFSDirectory a try.
> >
> > Thanks,
> > Peter
> >
> >
> > On 1/26/06, Paul Elschot <paul.elschot [at] xs4all> wrote:
> > >
> > > On Wednesday 25 January 2006 20:51, Peter Keegan wrote:
> > > > The index is non-compound format and optimized. Yes, I did try
> > > > MMapDirectory, but the index is too big - 3.5 GB (1.3GB is term
> vectors)
> > > >
> > > > Peter
> > > >
> > > You could also give this a try:
> > >
> > > http://issues.apache.org/jira/browse/LUCENE-283
> > >
> > > Regards,
> > > Paul Elschot
> > >
> > > ---------------------------------------------------------------------
> > > To unsubscribe, e-mail: java-user-unsubscribe [at] lucene
> > > For additional commands, e-mail: java-user-help [at] lucene
> > >
> > >
> >
> >
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe [at] lucene
> For additional commands, e-mail: java-user-help [at] lucene
>
>

yseeley at gmail

Jan 26, 2006, 10:17 AM

Post #23 of 62 (3451 views)
Permalink

Re: Throughput doesn't increase when using more concurrent threads [In reply to]

BEA Jrockit supports both AMD64 and Intel's EM64T (basically renamed AMD64)
http://www.bea.com/framework.jsp?CNT=index.htm&FP=/content/products/jrockit/

and Sun's Java 1.5 for "Windows AMD64 Platform"
They advertize AMD64, presumably because that's what there servers
use, but it should work on Intel's x86_64 (EM64T) also. The release
notes have the following:
"With the release, J2SE support for Windows 64-bit has progressed from
release candidate to final release. This version runs on AMD64/EM64T
64-bit mode machines with Windows Server 2003 x64 Editions."

Of course, if the platform is up to you, I'd choose Linux :-)

-Yonik

On 1/26/06, Peter Keegan <peterlkeegan [at] gmail> wrote:
> I'd love to try this, but I'm not aware of any 64-bit jvms for Windows on
> Intel. If you know of any, please let me know. Linux may be an option, too.
>
> btw, I'm getting a sustained rate of 135 queries/sec with 4 threads, which
> is pretty impressive. Another way around the concurrency limit is to run
> multiple jvms. The throughput of each is less, but the aggregate throughput
> is higher.
>
> Peter
>
>
> On 1/26/06, Yonik Seeley <yseeley [at] gmail> wrote:
> >
> > Hmmm, can you run the 64 bit version of Windows (and hence a 64 bit JVM?)
> > We're running with heap sizes up to 8GB (RH Linux 64 bit, Opterons,
> > Sun Java 1.5)
> >
> > -Yonik
> >
> > On 1/26/06, Peter Keegan <peterlkeegan [at] gmail> wrote:
> > > Paul,
> > >
> > > I tried this but it ran out of memory trying to read the 500Mb .fdt
> > file. I
> > > tried various values for MAX_BBUF, but it still ran out of memory (I'm
> > using
> > > -Xmx1600M, which is the jvm's maximum value (v1.5)) I'll give
> > > NioFSDirectory a try.
> > >
> > > Thanks,
> > > Peter
> > >
> > >
> > > On 1/26/06, Paul Elschot <paul.elschot [at] xs4all> wrote:
> > > >
> > > > On Wednesday 25 January 2006 20:51, Peter Keegan wrote:
> > > > > The index is non-compound format and optimized. Yes, I did try
> > > > > MMapDirectory, but the index is too big - 3.5 GB (1.3GB is term
> > vectors)
> > > > >
> > > > > Peter
> > > > >
> > > > You could also give this a try:
> > > >
> > > > http://issues.apache.org/jira/browse/LUCENE-283
> > > >
> > > > Regards,
> > > > Paul Elschot
> > > >
> > > > ---------------------------------------------------------------------
> > > > To unsubscribe, e-mail: java-user-unsubscribe [at] lucene
> > > > For additional commands, e-mail: java-user-help [at] lucene
> > > >
> > > >
> > >
> > >
> >
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: java-user-unsubscribe [at] lucene
> > For additional commands, e-mail: java-user-help [at] lucene
> >
> >
>
>

---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe [at] lucene
For additional commands, e-mail: java-user-help [at] lucene

cutting at apache

Jan 26, 2006, 11:55 AM

Post #24 of 62 (3441 views)
Permalink

Re: Throughput doesn't increase when using more concurrent threads [In reply to]

Peter Keegan wrote:
> The throughput is worse with NioFSDIrectory than with the FSDIrectory
> (patched and unpatched). The bottleneck still seems to be synchronization,
> this time in NioFile.getChannel (7 of the 8 threads were blocked there
> during one snapshot). I tried this with 4 and 8 channels.

Another thing to try might be to increase IndexOutputStream.BUFFER_SIZE,
to say, 8192. Note that you need to 'ant clean compile', since other
files that reference this constant need to be recompiled. Does that
help any?

A 64-bit JVM with NioDirectory would really be optimal for this.
Anything else either requires opening lots more file handles, or
synchronizing on a few file handles, neither of which is attractive. We
could try to keep a pool of file handles for highly contended files, but
that's tricky to implement.

Doug

---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe [at] lucene
For additional commands, e-mail: java-user-help [at] lucene

cutting at apache

Jan 26, 2006, 11:59 AM

Post #25 of 62 (3438 views)
Permalink

Re: Throughput doesn't increase when using more concurrent threads [In reply to]

Doug Cutting wrote:
> A 64-bit JVM with NioDirectory would really be optimal for this.

Oops. I meant MMapDirectory, not NioDirectory.

Doug

---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe [at] lucene
For additional commands, e-mail: java-user-help [at] lucene

1 2 3

View All

Index | Next | Previous | View Threaded

Interested in having your list archived? Contact Gossamer Threads