Mailing List Archive: RSyslog: users

support for arbitrary number of open files

Index | Next | Previous | View Threaded

theinric at redhat

Nov 16, 2009, 8:24 AM

Post #1 of 6 (491 views)
Permalink

support for arbitrary number of open files

Hi all,

currently the total number of files (and tcp connections) that can be
open simultaneously is limited by the select() system call. Ideally this
would be changed to *poll(), but that can take some time.

Until that happens, this patch[1] tries to remove the limitations of
select() by enlarging the bit mask that is used for storing file
descriptor information and redefining the macros that process it.
This modification is inspired by Bind's use of select().
It is rather a workaround and may not be entirely portable.

The actual changes to the code aren't big, but they are in many places
so sufficient testing is needed. Allocating and freeing fd_sets in some
frequently called functions may decrease performance. This can be dealt
with but would require more pervasive changes.

Thoughts?

Thanks,
Tomas

[1] - http://people.redhat.com/pvrabec/rsyslog-4.4.2-unlimited-select.patch

_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com

rgerhards at hq

Nov 16, 2009, 9:08 AM

Post #2 of 6 (469 views)
Permalink

Re: support for arbitrary number of open files [In reply to]

> -----Original Message-----
> From: rsyslog-bounces [at] lists [mailto:rsyslog-
> bounces [at] lists] On Behalf Of Tomas Heinrich
> Sent: Monday, November 16, 2009 5:25 PM
> To: rsyslog-users
> Subject: [rsyslog] support for arbitrary number of open files
>
> Hi all,
>
> currently the total number of files (and tcp connections) that can be
> open simultaneously is limited by the select() system call. Ideally
> this
> would be changed to *poll(), but that can take some time.

For imudp, this change already happened (in the current devel). The others
are on my radar. TCP is probably the most problematic, I need to redo the
driver level. I wonder if I should split out gssapi while doing so, because
that would simplify a lot of the calling conventions (at the price of some
code duplication).

For real-world scenarios, I think tcp is probably the only real issue, it is
hard to believe that e.g. imuxsock will have an issue with that ;)

> Until that happens, this patch[1] tries to remove the limitations of
> select() by enlarging the bit mask that is used for storing file
> descriptor information and redefining the macros that process it.
> This modification is inspired by Bind's use of select().
> It is rather a workaround and may not be entirely portable.
>
> The actual changes to the code aren't big, but they are in many places
> so sufficient testing is needed. Allocating and freeing fd_sets in
> some
> frequently called functions may decrease performance. This can be dealt
> with but would require more pervasive changes.

I need to have a close look at the patch. I consider it useful, but I agree
that it must be very carefully checked. I've had a quick look and my
understanding is that it is enabled by conditional compilation. That would
ease many of my concerns, as we could disable it by default and only those
that actually need it are at risk.

>
> Thoughts?

I'd also appreciate feedback from other list members.

Thanks,
Rainer

>
> Thanks,
> Tomas
>
> [1] - http://people.redhat.com/pvrabec/rsyslog-4.4.2-unlimited-
> select.patch
>
> _______________________________________________
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com

rgerhards at hq

Nov 17, 2009, 1:00 AM

Post #3 of 6 (468 views)
Permalink

Re: support for arbitrary number of open files [In reply to]

Tomas,

I am currently working on integration of this patch. What puzzles me is the
call to "howmany()". I don't find any doc on it, nor an implementation inside
the patch. Could you elaborate what it does and where it stems from?

Also, I think there is a segfault in gss-misc, because the glbl interface is
never aquired (the will result in a NULL-pointer dereference). I also need to
change the glbl interface definitions, FdSetSize must always be present, else
the interface is no longer well-defined. I will post the completely
integrated patch when I am done.

But feedback on howmany() would be most appreciated, because I currently do
not know exactly how to handle it.

Thanks,
Rainer

> -----Original Message-----
> From: rsyslog-bounces [at] lists [mailto:rsyslog-
> bounces [at] lists] On Behalf Of Tomas Heinrich
> Sent: Monday, November 16, 2009 5:25 PM
> To: rsyslog-users
> Subject: [rsyslog] support for arbitrary number of open files
>
> Hi all,
>
> currently the total number of files (and tcp connections) that can be
> open simultaneously is limited by the select() system call. Ideally
> this
> would be changed to *poll(), but that can take some time.
>
> Until that happens, this patch[1] tries to remove the limitations of
> select() by enlarging the bit mask that is used for storing file
> descriptor information and redefining the macros that process it.
> This modification is inspired by Bind's use of select().
> It is rather a workaround and may not be entirely portable.
>
> The actual changes to the code aren't big, but they are in many places
> so sufficient testing is needed. Allocating and freeing fd_sets in
> some
> frequently called functions may decrease performance. This can be dealt
> with but would require more pervasive changes.
>
> Thoughts?
>
> Thanks,
> Tomas
>
> [1] - http://people.redhat.com/pvrabec/rsyslog-4.4.2-unlimited-
> select.patch
>
> _______________________________________________
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com

rgerhards at hq

Nov 17, 2009, 1:49 AM

Post #4 of 6 (459 views)
Permalink

Re: support for arbitrary number of open files [In reply to]

OK, I have finally integrated the patch, I could work around my missing
knowledge of howmany() [but still would appreciate some more info on it].

As of rsyslog policy, new features are never integrated into stable or beta
builds. As such, it is available in the v4-devel and master branches. Note
that I made a number of changes, you will probably like to re-check for your
use case. I ran the testbench successfully in both modes and did some manual
tests with the new functionality disabled.

The v5-branch does NOT include the patch for imudp. As you said, it is a
(good ;)) work-around and imudp already supports epoll(), so there is no need
for this workaround. I plan to gradually remove that feature in v5 a I work
towards supporting epoll in all inputs.

The master branch will probably be released tomorrow, v4-devel as need arises
(but it is available from git right now).

Thanks again, I think this is a very useful addition.

Rainer

> -----Original Message-----
> From: rsyslog-bounces [at] lists [mailto:rsyslog-
> bounces [at] lists] On Behalf Of Rainer Gerhards
> Sent: Tuesday, November 17, 2009 10:01 AM
> To: rsyslog-users
> Subject: Re: [rsyslog] support for arbitrary number of open files
>
> Tomas,
>
> I am currently working on integration of this patch. What puzzles me is
> the
> call to "howmany()". I don't find any doc on it, nor an implementation
> inside
> the patch. Could you elaborate what it does and where it stems from?
>
> Also, I think there is a segfault in gss-misc, because the glbl
> interface is
> never aquired (the will result in a NULL-pointer dereference). I also
> need to
> change the glbl interface definitions, FdSetSize must always be
> present, else
> the interface is no longer well-defined. I will post the completely
> integrated patch when I am done.
>
> But feedback on howmany() would be most appreciated, because I
> currently do
> not know exactly how to handle it.
>
> Thanks,
> Rainer
>
> > -----Original Message-----
> > From: rsyslog-bounces [at] lists [mailto:rsyslog-
> > bounces [at] lists] On Behalf Of Tomas Heinrich
> > Sent: Monday, November 16, 2009 5:25 PM
> > To: rsyslog-users
> > Subject: [rsyslog] support for arbitrary number of open files
> >
> > Hi all,
> >
> > currently the total number of files (and tcp connections) that can be
> > open simultaneously is limited by the select() system call. Ideally
> > this
> > would be changed to *poll(), but that can take some time.
> >
> > Until that happens, this patch[1] tries to remove the limitations of
> > select() by enlarging the bit mask that is used for storing file
> > descriptor information and redefining the macros that process it.
> > This modification is inspired by Bind's use of select().
> > It is rather a workaround and may not be entirely portable.
> >
> > The actual changes to the code aren't big, but they are in many
> places
> > so sufficient testing is needed. Allocating and freeing fd_sets in
> > some
> > frequently called functions may decrease performance. This can be
> dealt
> > with but would require more pervasive changes.
> >
> > Thoughts?
> >
> > Thanks,
> > Tomas
> >
> > [1] - http://people.redhat.com/pvrabec/rsyslog-4.4.2-unlimited-
> > select.patch
> >
> > _______________________________________________
> > rsyslog mailing list
> > http://lists.adiscon.net/mailman/listinfo/rsyslog
> > http://www.rsyslog.com
> _______________________________________________
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com

theinric at redhat

Nov 18, 2009, 5:18 AM

Post #5 of 6 (456 views)
Permalink

Re: support for arbitrary number of open files [In reply to]

Rainer,

thanks for integrating it and thanks for fixing the things I've overlooked.

I've skimmed through the commit logs and the changes look good, I'll run
some more tests with the new sources.

Regarding howmany(), it is defined here in my environment:
/usr/include/sys/param.h:
# define howmany(x, y) (((x) + ((y) - 1)) / (y))
It computes the number of 'y's needed to store 'x'.

Tomas

On 11/17/2009 10:49 AM, Rainer Gerhards wrote:
> OK, I have finally integrated the patch, I could work around my missing
> knowledge of howmany() [but still would appreciate some more info on it].
>
> As of rsyslog policy, new features are never integrated into stable or beta
> builds. As such, it is available in the v4-devel and master branches. Note
> that I made a number of changes, you will probably like to re-check for your
> use case. I ran the testbench successfully in both modes and did some manual
> tests with the new functionality disabled.
>
> The v5-branch does NOT include the patch for imudp. As you said, it is a
> (good ;)) work-around and imudp already supports epoll(), so there is no need
> for this workaround. I plan to gradually remove that feature in v5 a I work
> towards supporting epoll in all inputs.
>
> The master branch will probably be released tomorrow, v4-devel as need arises
> (but it is available from git right now).
>
> Thanks again, I think this is a very useful addition.
>
> Rainer
>
>> -----Original Message-----
>> From: rsyslog-bounces [at] lists [mailto:rsyslog-
>> bounces [at] lists] On Behalf Of Rainer Gerhards
>> Sent: Tuesday, November 17, 2009 10:01 AM
>> To: rsyslog-users
>> Subject: Re: [rsyslog] support for arbitrary number of open files
>>
>> Tomas,
>>
>> I am currently working on integration of this patch. What puzzles me is
>> the
>> call to "howmany()". I don't find any doc on it, nor an implementation
>> inside
>> the patch. Could you elaborate what it does and where it stems from?
>>
>> Also, I think there is a segfault in gss-misc, because the glbl
>> interface is
>> never aquired (the will result in a NULL-pointer dereference). I also
>> need to
>> change the glbl interface definitions, FdSetSize must always be
>> present, else
>> the interface is no longer well-defined. I will post the completely
>> integrated patch when I am done.
>>
>> But feedback on howmany() would be most appreciated, because I
>> currently do
>> not know exactly how to handle it.
>>
>> Thanks,
>> Rainer
>>
>>> -----Original Message-----
>>> From: rsyslog-bounces [at] lists [mailto:rsyslog-
>>> bounces [at] lists] On Behalf Of Tomas Heinrich
>>> Sent: Monday, November 16, 2009 5:25 PM
>>> To: rsyslog-users
>>> Subject: [rsyslog] support for arbitrary number of open files
>>>
>>> Hi all,
>>>
>>> currently the total number of files (and tcp connections) that can be
>>> open simultaneously is limited by the select() system call. Ideally
>>> this
>>> would be changed to *poll(), but that can take some time.
>>>
>>> Until that happens, this patch[1] tries to remove the limitations of
>>> select() by enlarging the bit mask that is used for storing file
>>> descriptor information and redefining the macros that process it.
>>> This modification is inspired by Bind's use of select().
>>> It is rather a workaround and may not be entirely portable.
>>>
>>> The actual changes to the code aren't big, but they are in many
>> places
>>> so sufficient testing is needed. Allocating and freeing fd_sets in
>>> some
>>> frequently called functions may decrease performance. This can be
>> dealt
>>> with but would require more pervasive changes.
>>>
>>> Thoughts?
>>>
>>> Thanks,
>>> Tomas
>>>
>>> [1] - http://people.redhat.com/pvrabec/rsyslog-4.4.2-unlimited-
>>> select.patch
>>>
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com

rgerhards at hq

Nov 18, 2009, 5:35 AM

Post #6 of 6 (453 views)
Permalink

Re: support for arbitrary number of open files [In reply to]

> -----Original Message-----
> From: rsyslog-bounces [at] lists [mailto:rsyslog-
> bounces [at] lists] On Behalf Of Tomas Heinrich
> Sent: Wednesday, November 18, 2009 2:19 PM
> To: rsyslog-users
> Subject: Re: [rsyslog] support for arbitrary number of open files
>
> Rainer,
>
> thanks for integrating it and thanks for fixing the things I've
> overlooked.
>
> I've skimmed through the commit logs and the changes look good, I'll
> run
> some more tests with the new sources.

excellent - pls let me know if you see some issues. As a side-note, I've also
used this as a reminder to finally look at epoll() for the plain tcp driver,
probably the most prominent case, also performance wise. I hope I am able
continue work on it, so we should have it in the not so distant future (in
v5).

>
> Regarding howmany(), it is defined here in my environment:
> /usr/include/sys/param.h:
> # define howmany(x, y) (((x) + ((y) - 1)) / (y))
> It computes the number of 'y's needed to store 'x'.
>

Ahhh! I should have done a grep ;) But somehow did not expect it there...

Thanks,
Rainer

> Tomas
>
> On 11/17/2009 10:49 AM, Rainer Gerhards wrote:
> > OK, I have finally integrated the patch, I could work around my
> missing
> > knowledge of howmany() [but still would appreciate some more info on
> it].
> >
> > As of rsyslog policy, new features are never integrated into stable
> or beta
> > builds. As such, it is available in the v4-devel and master branches.
> Note
> > that I made a number of changes, you will probably like to re-check
> for your
> > use case. I ran the testbench successfully in both modes and did some
> manual
> > tests with the new functionality disabled.
> >
> > The v5-branch does NOT include the patch for imudp. As you said, it
> is a
> > (good ;)) work-around and imudp already supports epoll(), so there is
> no need
> > for this workaround. I plan to gradually remove that feature in v5 a
> I work
> > towards supporting epoll in all inputs.
> >
> > The master branch will probably be released tomorrow, v4-devel as
> need arises
> > (but it is available from git right now).
> >
> > Thanks again, I think this is a very useful addition.
> >
> > Rainer
> >
> >> -----Original Message-----
> >> From: rsyslog-bounces [at] lists [mailto:rsyslog-
> >> bounces [at] lists] On Behalf Of Rainer Gerhards
> >> Sent: Tuesday, November 17, 2009 10:01 AM
> >> To: rsyslog-users
> >> Subject: Re: [rsyslog] support for arbitrary number of open files
> >>
> >> Tomas,
> >>
> >> I am currently working on integration of this patch. What puzzles me
> is
> >> the
> >> call to "howmany()". I don't find any doc on it, nor an
> implementation
> >> inside
> >> the patch. Could you elaborate what it does and where it stems from?
> >>
> >> Also, I think there is a segfault in gss-misc, because the glbl
> >> interface is
> >> never aquired (the will result in a NULL-pointer dereference). I
> also
> >> need to
> >> change the glbl interface definitions, FdSetSize must always be
> >> present, else
> >> the interface is no longer well-defined. I will post the completely
> >> integrated patch when I am done.
> >>
> >> But feedback on howmany() would be most appreciated, because I
> >> currently do
> >> not know exactly how to handle it.
> >>
> >> Thanks,
> >> Rainer
> >>
> >>> -----Original Message-----
> >>> From: rsyslog-bounces [at] lists [mailto:rsyslog-
> >>> bounces [at] lists] On Behalf Of Tomas Heinrich
> >>> Sent: Monday, November 16, 2009 5:25 PM
> >>> To: rsyslog-users
> >>> Subject: [rsyslog] support for arbitrary number of open files
> >>>
> >>> Hi all,
> >>>
> >>> currently the total number of files (and tcp connections) that can
> be
> >>> open simultaneously is limited by the select() system call. Ideally
> >>> this
> >>> would be changed to *poll(), but that can take some time.
> >>>
> >>> Until that happens, this patch[1] tries to remove the limitations
> of
> >>> select() by enlarging the bit mask that is used for storing file
> >>> descriptor information and redefining the macros that process it.
> >>> This modification is inspired by Bind's use of select().
> >>> It is rather a workaround and may not be entirely portable.
> >>>
> >>> The actual changes to the code aren't big, but they are in many
> >> places
> >>> so sufficient testing is needed. Allocating and freeing fd_sets in
> >>> some
> >>> frequently called functions may decrease performance. This can be
> >> dealt
> >>> with but would require more pervasive changes.
> >>>
> >>> Thoughts?
> >>>
> >>> Thanks,
> >>> Tomas
> >>>
> >>> [1] - http://people.redhat.com/pvrabec/rsyslog-4.4.2-unlimited-
> >>> select.patch
> >>>
> _______________________________________________
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com

Index | Next | Previous | View Threaded

Interested in having your list archived? Contact Gossamer Threads