Mailing List Archive: MythTV: Users

0.25->0.26: problems with high mem consumption and oom kils

Index | Next | Previous | View Threaded

warpme at o2

Sep 2, 2012, 2:51 AM

Post #1 of 7 (492 views)
Permalink

0.25->0.26: problems with high mem consumption and oom kils

Hi,

Over week ago I moved from 0.25.2 to 0.26.
I have headless BE (no X running) + 3x diskless FE (ION1, 2G RAM).

Last week I was trying to understand what is driver significantly higher
mem usage by 0.26.
With 0.26 I have average 300-900MB for mythbackend process as avg 950MB
for mythfrontned.
mem issue is significant problem as it quickly decreasing WAF because
frequent oom kills in FE (triggered when mythfrontend RSS crosses approx
1.1G).

For mythfrontend I can't find any specific usage pattern leading to so
high RSS, but during week of operation (and periodic monitoring RSS) I
got impression RSS is high but stable except some moments when it
"explodes" and is finished by oom killer.
Unfortunately I cant find what is trigger to this. It seems to happen in
UI operations but also in watching movie or pictures.

For mythbackend, RSS is quite low after process start (230-300MB) but
sometimes goes to 800-900MB and stays on that level.
This increase is sporadic and I observed it is correlated with starting
high number of concurrent recordings - but not always.

I'll be really greatful if somebody hints me about most effective way to
diagnose problem.

Currently I'm quite desperate as those oom kills are really frustrating
(i.e. we have evening with friends, wathing holidays pictures, and
expierienced 3 oom kills; yesterday during normal BluRay movie watch
also oom :-(

-br

Attachments:

warpme.vcf (83 B)

mythtv at hbuus

Sep 2, 2012, 9:42 AM

Post #2 of 7 (463 views)
Permalink

Re: 0.25->0.26: problems with high mem consumption and oom kils [In reply to]

On 9/2/2012 5:51 AM, Warpme wrote:
> Hi,
>
> Over week ago I moved from 0.25.2 to 0.26.
> I have headless BE (no X running) + 3x diskless FE (ION1, 2G RAM).
>
> Last week I was trying to understand what is driver significantly higher mem usage by 0.26.
> With 0.26 I have average 300-900MB for mythbackend process as avg 950MB for mythfrontned.
> mem issue is significant problem as it quickly decreasing WAF because frequent oom kills in FE (triggered when mythfrontend RSS crosses approx 1.1G).
>
> For mythfrontend I can't find any specific usage pattern leading to so high RSS, but during week of operation (and periodic monitoring RSS) I got impression RSS is high but stable except some moments when it "explodes" and is finished by oom killer.
> Unfortunately I cant find what is trigger to this. It seems to happen in UI operations but also in watching movie or pictures.
>
> For mythbackend, RSS is quite low after process start (230-300MB) but sometimes goes to 800-900MB and stays on that level.
> This increase is sporadic and I observed it is correlated with starting high number of concurrent recordings - but not always.
>
> I'll be really greatful if somebody hints me about most effective way to diagnose problem.
>
> Currently I'm quite desperate as those oom kills are really frustrating (i.e. we have evening with friends, wathing holidays pictures, and expierienced 3 oom kills; yesterday during normal BluRay movie watch also oom :-(
>
> -br

Why bother trying to diagnose it? Why not add swap space? You don't need to have a swap partition, you can add swap using a regular file. If you're running ubuntu, this should help:
https://help.ubuntu.com/community/SwapFaq

The other option is to add more RAM.

HTH,
Helen
_______________________________________________
mythtv-users mailing list
mythtv-users [at] mythtv
http://www.mythtv.org/mailman/listinfo/mythtv-users

warpme at o2

Sep 4, 2012, 8:46 AM

Post #3 of 7 (444 views)
Permalink

Re: 0.25->0.26: problems with high mem consumption and oom kils [In reply to]

On 9/2/12 6:42 PM, faginbagin wrote:
>
> Why bother trying to diagnose it? Why not add swap space? You don't need to have a swap partition, you can add swap using a regular file. If you're running ubuntu, this should help:
> https://help.ubuntu.com/community/SwapFaq
>
> The other option is to add more RAM.
>
> HTH,
> Helen
> _______________________________________________
>
>
Sure. Adding swap is option - but I would consider this as last resort.
I have PXE booted diskless FEs using ram disks.
I have this not by accident: diskless means not costs/power/phy
space/management issues related to FEs HDD.
RAM disks means fastest execution speed (your mass storage has
practically RAM speed).
Price for this is FE RAM, as on FE EVERYTHING (code & data) is mapped to
RAM.
With 0.25 & 2G RAM (512M allocated to VIDEO) I had average 300-500MB
free RAM on FE.
This is not a case with 0.26.
I can always buy more RAM, but from economic point of view I want to
understand return on this investment (especially when I have 3
FEs to upgrade).
Also, before spending money into old technology (DDR2) I want to be sure
it is unavoidable...
BE seems to be little different story: 800-900MB for it seems to be
little too high, especially compared with 250-300MB I usually observed
in 0.25

-br

Attachments:

warpme.vcf (83 B)

tylernt at gmail

Sep 4, 2012, 10:34 AM

Post #4 of 7 (440 views)
Permalink

Re: 0.25->0.26: problems with high mem consumption and oom kils [In reply to]

>> Why bother trying to diagnose it? Why not add swap space?
>> The other option is to add more RAM.

If it's a memory leak, it'll still crash with 32GB of RAM installed...
just takes a little longer.

> BE seems to be little different story: 800-900MB for it seems to be little
> too high, especially compared with 250-300MB I usually observed in 0.25

Agreed. Have you considered enabling verbose logging and submitting a
bug report?
_______________________________________________
mythtv-users mailing list
mythtv-users [at] mythtv
http://www.mythtv.org/mailman/listinfo/mythtv-users

warpme at o2

Sep 4, 2012, 12:05 PM

Post #5 of 7 (441 views)
Permalink

Re: 0.25->0.26: problems with high mem consumption and oom kils [In reply to]

On 9/4/12 7:34 PM, Tyler T wrote:
>>> Why bother trying to diagnose it? Why not add swap space?
>>> The other option is to add more RAM.
> If it's a memory leak, it'll still crash with 32GB of RAM installed...
> just takes a little longer.
>
>> BE seems to be little different story: 800-900MB for it seems to be little
>> too high, especially compared with 250-300MB I usually observed in 0.25
> Agreed. Have you considered enabling verbose logging and submitting a
> bug report?
> _______________________________________________
> mythtv-users mailing list
> mythtv-users [at] mythtv
> http://www.mythtv.org/mailman/listinfo/mythtv-users
>
Thx for replay.
In fact I launch detailed logging for to be sure there is no error from
my side. Probability is low as 0.26 was compiled/installed on exactly
the same environment like 0.25, so I really doubt it is environment issue.
Next I observe RSS as function on time. It has stepped growth but after
reaching given level is practically constant in time. This hints me that
issue is probably related to particular code paths.
I.e. in BE it seems to almost directly related to number of running
recorders AND recording concurrency.
I can't resist to feeling that it is result of switching logging
messaging to zeromq - but this is only speculation.
What is more interesting - before upgrade to 0.26, I had 0.25 BE with
all major changes in EIT/recorders backported from 0.26->0.25.
It as working really nicely weeks without issues. So in recording
infrastructure I can fairly say 0.25 with 0.26's recording subsystem
don't have RSS issue. On the other hand, 0.26 BE RSS issue is directly
proportional to running recorders+concurrency...

I post to ML because I want to find most efficient way to diagnose this
type of problems. Software tools are constantly improved and probably I
missed some nice tools/techniques to diagnose memory consumption in
Linux...
Having limited maintenance time windows on my production BE + somehow
limited time I can devote to this, I prefer hints from knowledgeable ppl
instead learning by try-error

-br

Attachments:

warpme.vcf (83 B)

mtdean at thirdcontact

Sep 5, 2012, 6:00 AM

Post #6 of 7 (432 views)
Permalink

Re: 0.25->0.26: problems with high mem consumption and oom kils [In reply to]

On 09/04/2012 01:34 PM, Tyler T wrote:
>>> Why bother trying to diagnose it? Why not add swap space?
>>> The other option is to add more RAM.
> If it's a memory leak, it'll still crash with 32GB of RAM installed...
> just takes a little longer.

Remember, though, that memory use is not necessarily a memory leak

>> BE seems to be little different story: 800-900MB for it seems to be little
>> too high, especially compared with 250-300MB I usually observed in 0.25
> Agreed. Have you considered enabling verbose logging and submitting a
> bug report?

which is why you'll need much more than just MythTV logs to file a
memory leak bug (we don't actually add in log messages that say, "this
will leak memory," since we fix known leaks :). You'd need to run
valgrind and submit a valgrind log that shows a real memory leak (or
point out flawed code that results in a leak).

Mike
_______________________________________________
mythtv-users mailing list
mythtv-users [at] mythtv
http://www.mythtv.org/mailman/listinfo/mythtv-users

udovdh at xs4all

Oct 6, 2012, 2:05 AM

Post #7 of 7 (307 views)
Permalink

Re: 0.25->0.26: problems with high mem consumption and oom kils [In reply to]

On 2012-09-05 15:00, Michael T. Dean wrote:
>> If it's a memory leak, it'll still crash with 32GB of RAM installed...
>> just takes a little longer.
>
> Remember, though, that memory use is not necessarily a memory leak

Did you forget about our discussion?
The backend keeps on allocating RSS at a steady pace for weeks on end.

>> Agreed. Have you considered enabling verbose logging and submitting a
>> bug report?
>
> which is why you'll need much more than just MythTV logs to file a
> memory leak bug (we don't actually add in log messages that say, "this
> will leak memory," since we fix known leaks :). You'd need to run
> valgrind and submit a valgrind log that shows a real memory leak (or
> point out flawed code that results in a leak).

Did you forget about what I did?
The results were thought to be insignificant and are still not
completely fixed.
Now if other people can also point out leaks, even after my input and
subsequent fixes, then would that not be a nice time to implement some
valgrind testing before releasing code?

Kind regards,
Udo
_______________________________________________
mythtv-users mailing list
mythtv-users [at] mythtv
http://www.mythtv.org/mailman/listinfo/mythtv-users

Index | Next | Previous | View Threaded

Interested in having your list archived? Contact Gossamer Threads