Mailing List Archive: Lucene: Java-Dev

[jira] [Commented] (SOLR-3658) SolrCmdDistributor can briefly create spikes of threads in the thousands.

Index | Next | Previous | View Threaded

jira at apache

Jul 20, 2012, 1:13 PM

Post #1 of 7 (115 views)
Permalink

[jira] [Commented] (SOLR-3658) SolrCmdDistributor can briefly create spikes of threads in the thousands.

[ https://issues.apache.org/jira/browse/SOLR-3658?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13419514#comment-13419514 ]

Mark Miller commented on SOLR-3658:
-----------------------------------

SolrCmdDIstributor is using a thread executor that has no bounds on the number of threads it can create. Commit or busy activity on a replica can cause a small backup that leads to spikes of threads on the leader. They tend to spike up and then come back down. Since each thread eats memory, this can be very costly or cause an OOME: cant create new native thread.

One possible workaround is to lower the thread stack size - on many 64-bit platforms it defaults to 1mb...-Xss=128kb might be enough and might make the situation better. It's not a solution though.

As a fix, after discussing with Yonik, I've been working on limiting the number of threads to 8 x the number of hosts in the cluster as a start. Perhaps it will be overridable with config.

> SolrCmdDistributor can briefly create spikes of threads in the thousands.
> -------------------------------------------------------------------------
>
> Key: SOLR-3658
> URL: https://issues.apache.org/jira/browse/SOLR-3658
> Project: Solr
> Issue Type: Bug
> Components: SolrCloud
> Reporter: Mark Miller
> Assignee: Mark Miller
> Fix For: 4.0, 5.0
>
>
> see mailing list http://markmail.org/thread/yy5b7g6g7733wgcp

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe [at] lucene
For additional commands, e-mail: dev-help [at] lucene

jira at apache

Jul 21, 2012, 12:33 AM

Post #2 of 7 (111 views)
Permalink

[jira] [Commented] (SOLR-3658) SolrCmdDistributor can briefly create spikes of threads in the thousands. [In reply to]

[ https://issues.apache.org/jira/browse/SOLR-3658?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13419755#comment-13419755 ]

Yonik Seeley commented on SOLR-3658:
------------------------------------

Other thing to keep in mind memory-wise is that each thread is associated with a number of outgoing update requests and other associated state, which in a realistic system probably take up more mem than the thread stack.

> SolrCmdDistributor can briefly create spikes of threads in the thousands.
> -------------------------------------------------------------------------
>
> Key: SOLR-3658
> URL: https://issues.apache.org/jira/browse/SOLR-3658
> Project: Solr
> Issue Type: Bug
> Components: SolrCloud
> Reporter: Mark Miller
> Assignee: Mark Miller
> Fix For: 4.0, 5.0
>
> Attachments: SOLR-3658.patch
>
>
> see mailing list http://markmail.org/thread/yy5b7g6g7733wgcp

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe [at] lucene
For additional commands, e-mail: dev-help [at] lucene

jira at apache

Jul 21, 2012, 10:12 AM

Post #3 of 7 (115 views)
Permalink

[jira] [Commented] (SOLR-3658) SolrCmdDistributor can briefly create spikes of threads in the thousands. [In reply to]

[ https://issues.apache.org/jira/browse/SOLR-3658?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13419881#comment-13419881 ]

Mark Miller commented on SOLR-3658:
-----------------------------------

Yeah, I'm not sure of the nitty gritty - but a couple sites I found were people where experimenting with just firing up threads to see how many they could create (that where not really doing anything), they could fire up way more before running out of RAM if they used a low Xss.

> SolrCmdDistributor can briefly create spikes of threads in the thousands.
> -------------------------------------------------------------------------
>
> Key: SOLR-3658
> URL: https://issues.apache.org/jira/browse/SOLR-3658
> Project: Solr
> Issue Type: Bug
> Components: SolrCloud
> Reporter: Mark Miller
> Assignee: Mark Miller
> Fix For: 4.0, 5.0
>
> Attachments: SOLR-3658.patch
>
>
> see mailing list http://markmail.org/thread/yy5b7g6g7733wgcp

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe [at] lucene
For additional commands, e-mail: dev-help [at] lucene

jira at apache

Jul 21, 2012, 10:21 AM

Post #4 of 7 (110 views)
Permalink

[jira] [Commented] (SOLR-3658) SolrCmdDistributor can briefly create spikes of threads in the thousands. [In reply to]

[ https://issues.apache.org/jira/browse/SOLR-3658?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13419883#comment-13419883 ]

Mark Miller commented on SOLR-3658:
-----------------------------------

(although i don't know if they were just using 32-bit or what)

> SolrCmdDistributor can briefly create spikes of threads in the thousands.
> -------------------------------------------------------------------------
>
> Key: SOLR-3658
> URL: https://issues.apache.org/jira/browse/SOLR-3658
> Project: Solr
> Issue Type: Bug
> Components: SolrCloud
> Reporter: Mark Miller
> Assignee: Mark Miller
> Fix For: 4.0, 5.0
>
> Attachments: SOLR-3658.patch
>
>
> see mailing list http://markmail.org/thread/yy5b7g6g7733wgcp

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe [at] lucene
For additional commands, e-mail: dev-help [at] lucene

jira at apache

Jul 22, 2012, 9:32 AM

Post #5 of 7 (113 views)
Permalink

[jira] [Commented] (SOLR-3658) SolrCmdDistributor can briefly create spikes of threads in the thousands. [In reply to]

[ https://issues.apache.org/jira/browse/SOLR-3658?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13420231#comment-13420231 ]

Mark Miller commented on SOLR-3658:
-----------------------------------

I'm going to commit this as a start - it seems to test out fine so far - we can change or tweak as necessary after.

> SolrCmdDistributor can briefly create spikes of threads in the thousands.
> -------------------------------------------------------------------------
>
> Key: SOLR-3658
> URL: https://issues.apache.org/jira/browse/SOLR-3658
> Project: Solr
> Issue Type: Bug
> Components: SolrCloud
> Reporter: Mark Miller
> Assignee: Mark Miller
> Fix For: 4.0, 5.0
>
> Attachments: SOLR-3658.patch
>
>
> see mailing list http://markmail.org/thread/yy5b7g6g7733wgcp

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe [at] lucene
For additional commands, e-mail: dev-help [at] lucene

jira at apache

Jul 22, 2012, 10:17 AM

Post #6 of 7 (114 views)
Permalink

[jira] [Commented] (SOLR-3658) SolrCmdDistributor can briefly create spikes of threads in the thousands. [In reply to]

[ https://issues.apache.org/jira/browse/SOLR-3658?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13420244#comment-13420244 ]

Mark Miller commented on SOLR-3658:
-----------------------------------

I committed what we have so far.

> SolrCmdDistributor can briefly create spikes of threads in the thousands.
> -------------------------------------------------------------------------
>
> Key: SOLR-3658
> URL: https://issues.apache.org/jira/browse/SOLR-3658
> Project: Solr
> Issue Type: Bug
> Components: SolrCloud
> Reporter: Mark Miller
> Assignee: Mark Miller
> Fix For: 4.0, 5.0
>
> Attachments: SOLR-3658.patch
>
>
> see mailing list http://markmail.org/thread/yy5b7g6g7733wgcp

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe [at] lucene
For additional commands, e-mail: dev-help [at] lucene

jira at apache

Aug 2, 2012, 3:01 PM

Post #7 of 7 (87 views)
Permalink

[jira] [Commented] (SOLR-3658) SolrCmdDistributor can briefly create spikes of threads in the thousands. [In reply to]

[ https://issues.apache.org/jira/browse/SOLR-3658?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13427659#comment-13427659 ]

Mark Miller commented on SOLR-3658:
-----------------------------------

There were some real problems with my previous solution - it somewhat worked accidentally - but I think really damaged performance probably.

I just committed a new approach that has tested out nicely so far.

> SolrCmdDistributor can briefly create spikes of threads in the thousands.
> -------------------------------------------------------------------------
>
> Key: SOLR-3658
> URL: https://issues.apache.org/jira/browse/SOLR-3658
> Project: Solr
> Issue Type: Bug
> Components: SolrCloud
> Reporter: Mark Miller
> Assignee: Mark Miller
> Fix For: 4.0, 5.0
>
> Attachments: SOLR-3658.patch
>
>
> see mailing list http://markmail.org/thread/yy5b7g6g7733wgcp

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe [at] lucene
For additional commands, e-mail: dev-help [at] lucene

Index | Next | Previous | View Threaded

Interested in having your list archived? Contact Gossamer Threads