Login | Register For Free | Help
Search for: (Advanced)

Mailing List Archive: Linux: Kernel

Kernel cached memory

 

 

Linux kernel RSS feed   Index | Next | Previous | View Threaded


ashleyz at alchip

Jul 22, 2005, 2:46 AM

Post #1 of 17 (999 views)
Permalink
Kernel cached memory

Hi all
I've a server with 2 Operton 64bit CPU and 12G memory, and this server
is used to run applications which will comsume huge memory,
the problem is: when this aplications exits, the free memory of the server
is still very low(accroding to the output of "top"), and
from the output of command "free", I can see that many GB memory was cached
by kernel. Does anyone know how to free the kernel cached
memory? thanks in advance.

Sincerely,
Ashley
------------------------------------------------------
AlChip Technologies, Limited
26F, Huamin Empire Plaza, Suite B-H,
726 West YanAn Road,
Shanghai, China 200050
Tel: 86-21-5237-8800 ext.220
Fax: 86-21-5237-8880
http://www.alchip.com
------------------------------------------------------


-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo [at] vger
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/


diegocg at gmail

Jul 22, 2005, 4:31 AM

Post #2 of 17 (974 views)
Permalink
Re: Kernel cached memory [In reply to]

El Fri, 22 Jul 2005 17:46:58 +0800,
"Ashley" <ashleyz [at] alchip> escribió:

> from the output of command "free", I can see that many GB memory was cached
> by kernel. Does anyone know how to free the kernel cached
> memory? thanks in advance.


You don't want that. Kernel will free cached memory when apps need it. When
there's a lot of free memory, linux always tries to use it for cache - because
caches speed up things, and it's silly to to leave memory free if you can
use it for caching.
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo [at] vger
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/


erik at harddisk-recovery

Jul 22, 2005, 5:57 AM

Post #3 of 17 (976 views)
Permalink
Re: Kernel cached memory [In reply to]

On Fri, Jul 22, 2005 at 05:46:58PM +0800, Ashley wrote:
> I've a server with 2 Operton 64bit CPU and 12G memory, and this server
> is used to run applications which will comsume huge memory,
> the problem is: when this aplications exits, the free memory of the server
> is still very low(accroding to the output of "top"), and
> from the output of command "free", I can see that many GB memory was cached
> by kernel. Does anyone know how to free the kernel cached
> memory? thanks in advance.

Free memory is bad, it means the memory doesn't have a proper use.
Cached memory will be freed automatically when the kernel needs memory
for other (more important) things.


Erik

--
+-- Erik Mouw -- www.harddisk-recovery.com -- +31 70 370 12 90 --
| Lab address: Delftechpark 26, 2628 XH, Delft, The Netherlands
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo [at] vger
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/


lgb at lgb

Jul 22, 2005, 6:25 AM

Post #4 of 17 (978 views)
Permalink
Re: Kernel cached memory [In reply to]

On Fri, Jul 22, 2005 at 05:46:58PM +0800, Ashley wrote:
> I've a server with 2 Operton 64bit CPU and 12G memory, and this server
> is used to run applications which will comsume huge memory,
> the problem is: when this aplications exits, the free memory of the server
> is still very low(accroding to the output of "top"), and
> from the output of command "free", I can see that many GB memory was cached
> by kernel. Does anyone know how to free the kernel cached
> memory? thanks in advance.

It's a very - very - very old and bad logic (at least nowdays) from the
stone age to free up memory. Memory is for using ... If you have memory why
don't you want to use? For an actively used system, memory should be used as
much as possible to maximize the performance. The only problem when kernel
does not want to "give back" used memory for eg caching for an application
though but it's another problem ...

Anyway, want to have 'free memory' is a thing like having dozens of cars
in your garage which don't want to be used ...

--
- Gábor
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo [at] vger
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/


rlrevell at joe-job

Jul 22, 2005, 10:58 AM

Post #5 of 17 (971 views)
Permalink
Re: Kernel cached memory [In reply to]

On Fri, 2005-07-22 at 15:25 +0200, Gábor Lénárt wrote:
> Anyway, want to have 'free memory' is a thing like having dozens of cars
> in your garage which don't want to be used ...
>

Really? I thought it was good to leave some memory free to speed up
application startup, so we don't have to evict a bunch of pages first.

Lee

-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo [at] vger
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/


john at illhostit

Jul 22, 2005, 2:43 PM

Post #6 of 17 (977 views)
Permalink
RE: Kernel cached memory [In reply to]

Wouldn't having (practically) all your memory used for cache slow down
starting a new program? First it would have to free up that space, and then
put stuff in that space, taking potentially twice as long. I think there
should be a system call for freeing cached memory, for those that do want to
do it.

-----Original Message-----
From: linux-kernel-owner [at] vger
[mailto:linux-kernel-owner [at] vger]On Behalf Of Erik Mouw
Sent: Friday, July 22, 2005 5:58 AM
To: Ashley
Cc: linux-kernel [at] vger
Subject: Re: Kernel cached memory


On Fri, Jul 22, 2005 at 05:46:58PM +0800, Ashley wrote:
> I've a server with 2 Operton 64bit CPU and 12G memory, and this server
> is used to run applications which will comsume huge memory,
> the problem is: when this aplications exits, the free memory of the
server
> is still very low(accroding to the output of "top"), and
> from the output of command "free", I can see that many GB memory was
cached
> by kernel. Does anyone know how to free the kernel cached
> memory? thanks in advance.

Free memory is bad, it means the memory doesn't have a proper use.
Cached memory will be freed automatically when the kernel needs memory
for other (more important) things.


Erik

--
+-- Erik Mouw -- www.harddisk-recovery.com -- +31 70 370 12 90 --
| Lab address: Delftechpark 26, 2628 XH, Delft, The Netherlands
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo [at] vger
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/

-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo [at] vger
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/


oliver at neukum

Jul 23, 2005, 3:50 AM

Post #7 of 17 (973 views)
Permalink
Re: Kernel cached memory [In reply to]

Am Freitag, 22. Juli 2005 19:58 schrieb Lee Revell:
> On Fri, 2005-07-22 at 15:25 +0200, Gábor Lénárt wrote:
> > Anyway, want to have 'free memory' is a thing like having dozens of cars
> > in your garage which don't want to be used ...
> >
>
> Really? I thought it was good to leave some memory free to speed up
> application startup, so we don't have to evict a bunch of pages first.

Eviction is cheap when the pages are clean.

Regards
Oliver
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo [at] vger
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/


vda at ilport

Jul 23, 2005, 5:31 AM

Post #8 of 17 (973 views)
Permalink
Re: Kernel cached memory [In reply to]

On Saturday 23 July 2005 00:43, John Pearson wrote:
> Wouldn't having (practically) all your memory used for cache slow down
> starting a new program? First it would have to free up that space, and then
> put stuff in that space, taking potentially twice as long. I think there
> should be a system call for freeing cached memory, for those that do want to
> do it.

I think this one is good enough:

#include <stdlib.h>

int main() {
void *p;
unsigned size = 1<<20;
unsigned long total=0;
while(size) {
p = malloc(size);
if(!p) size>>=1;
else {
memset(p, 0x77, size);
total+=size;
printf("Allocated %9u bytes, %12lu total\n",size,total);
}
}
return 0;
}

You may want to adapt it so that it takes an argument now many megabytes
to eat before it dies.
--
vda

-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo [at] vger
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/


davidsen at tmr

Jul 25, 2005, 9:47 AM

Post #9 of 17 (972 views)
Permalink
Re: Kernel cached memory [In reply to]

Gábor Lénárt wrote:
> On Fri, Jul 22, 2005 at 05:46:58PM +0800, Ashley wrote:
>
>> I've a server with 2 Operton 64bit CPU and 12G memory, and this server
>>is used to run applications which will comsume huge memory,
>>the problem is: when this aplications exits, the free memory of the server
>>is still very low(accroding to the output of "top"), and
>>from the output of command "free", I can see that many GB memory was cached
>>by kernel. Does anyone know how to free the kernel cached
>>memory? thanks in advance.
>
>
> It's a very - very - very old and bad logic (at least nowdays) from the
> stone age to free up memory.

It's very Microsoft to claim that the OS always knows best, and not let
the user tune the system the way they want it tuned. And if that means
to leave a bunch of free memory for absolute fastest availability, the
admin should have that option.

> Memory is for using ... If you have memory why
> don't you want to use? For an actively used system, memory should be used as
> much as possible to maximize the performance. The only problem when kernel
> does not want to "give back" used memory for eg caching for an application
> though but it's another problem ...

No, that's the problem he's trying to address.
>
> Anyway, want to have 'free memory' is a thing like having dozens of cars
> in your garage which don't want to be used ...
>
Which wait to be used when needed, rather than after someone cleans a
bunch of old junk out of them and scurries around the garage looking for
the right car to let you use.

Just because default operation works well for you, kindly don't try to
convice us that there are no cases when the default operation is NOT
optimal. And IMHO Linux is *way* too willing to evicy clean pages of my
programs to use as disk buffer, so that when system memory is full I pay
the overhead of TWO disk i/o's, one to finally write the data to the
disk and one to read my program back in. If free software is about
choice, I wish there was more in the area of how memory is used.

--
-bill davidsen (davidsen [at] tmr)
"The secret to procrastination is to put things off until the
last possible moment - but no longer" -me
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo [at] vger
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/


diegocg at gmail

Jul 25, 2005, 10:03 AM

Post #10 of 17 (975 views)
Permalink
Re: Kernel cached memory [In reply to]

El Mon, 25 Jul 2005 12:47:50 -0400,
Bill Davidsen <davidsen [at] tmr> escribió:

> Just because default operation works well for you, kindly don't try to
> convice us that there are no cases when the default operation is NOT
> optimal. And IMHO Linux is *way* too willing to evicy clean pages of my
> programs to use as disk buffer, so that when system memory is full I pay
> the overhead of TWO disk i/o's, one to finally write the data to the
> disk and one to read my program back in. If free software is about
> choice, I wish there was more in the area of how memory is used.

You must be more specific here; cached memory isn't always dirty. Most
of the times the pages used for cache are clean and can be evicted
quite fast and of course without writting anything to the disk.

If you want to tune the balance between "pages used for programs" vs
"pages used for cache" that's another problem, but I can't find a
reason why kernel shouldn't cache things when free memory is
available. And certainly that's not what Ashley was asking.
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo [at] vger
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/


ornati at fastwebnet

Jul 25, 2005, 10:07 AM

Post #11 of 17 (972 views)
Permalink
Re: Kernel cached memory [In reply to]

On Mon, 25 Jul 2005 12:47:50 -0400
Bill Davidsen <davidsen [at] tmr> wrote:

> And IMHO Linux is *way* too willing to evicy clean pages of my
> programs to use as disk buffer, so that when system memory is full I
> pay the overhead of TWO disk i/o's, one to finally write the data to
> the disk and one to read my program back in. If free software is
> about choice, I wish there was more in the area of how memory is
> used.

isn't this tuned enough by "/proc/sys/vm/swappiness" ?

--
Paolo Ornati
Linux 2.6.13-rc3 on x86_64
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo [at] vger
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/


davidsen at tmr

Jul 25, 2005, 11:02 AM

Post #12 of 17 (969 views)
Permalink
Re: Kernel cached memory [In reply to]

Paolo Ornati wrote:

>On Mon, 25 Jul 2005 12:47:50 -0400
>Bill Davidsen <davidsen [at] tmr> wrote:
>
>
>
>>And IMHO Linux is *way* too willing to evicy clean pages of my
>>programs to use as disk buffer, so that when system memory is full I
>>pay the overhead of TWO disk i/o's, one to finally write the data to
>>the disk and one to read my program back in. If free software is
>>about choice, I wish there was more in the area of how memory is
>>used.
>>
>>
>
>isn't this tuned enough by "/proc/sys/vm/swappiness" ?
>
>
>
Let me generate some data points for discussion. But the general answer
is no, I just want to try to have some numbers to discuss. I also want
to go back to some 2.4.xx-aa kernels, Andrea had some very nice things
in his bdflush code, and he was kind enough to explain to me how to tune
them so I avoided the worst case events.

--
bill davidsen <davidsen [at] tmr>
CTO TMR Associates, Inc
Doing interesting things with small computers since 1979

-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo [at] vger
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/


76306.1226 at compuserve

Jul 25, 2005, 5:35 PM

Post #13 of 17 (972 views)
Permalink
Re: Kernel cached memory [In reply to]

On Mon, 25 Jul 2005 at 12:47:50 -0400, Bill Davidsen wrote:

> > It's a very - very - very old and bad logic (at least nowdays) from the
> > stone age to free up memory.
>
> It's very Microsoft to claim that the OS always knows best, and not let
> the user tune the system the way they want it tuned.

Ironically, Microsoft offers a choice here.

In the registry (NT and W2K at least, don't know about XP:)

HKLM\SYSTEM\CurrentControlSet\Session Manager\Memory Management
LargeSystemCache : REG_DWORD
0: prefer application code over cached data
1: prefer cached data

Also:

DisablePagingExecutive : REG_DWORD
1: don't allow kernel code to be paged out

IOPageLockLimit : REG_DWORD
controls the amount of memory that can be locked for I/O


__
Chuck
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo [at] vger
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/


hancockr at shaw

Jul 25, 2005, 10:03 PM

Post #14 of 17 (971 views)
Permalink
Re: Kernel cached memory [In reply to]

John Pearson wrote:
> Wouldn't having (practically) all your memory used for cache slow down
> starting a new program? First it would have to free up that space, and then
> put stuff in that space, taking potentially twice as long.

If the cache pages are clean (not been modified since they were read
from the disk), then evicting that data will not take very long. If the
program you are just starting is not in the cache, then the time taken
to load it from disk will dwarf the time needed to evict cached pages.
And there's also the possibility that the cache contains the data you
are loading, which definitely will speed things up..

--
Robert Hancock Saskatoon, SK, Canada
To email, remove "nospam" from hancockr [at] nospamshaw
Home Page: http://www.roberthancock.com/

-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo [at] vger
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/


davidsen at tmr

Jul 26, 2005, 8:00 AM

Post #15 of 17 (971 views)
Permalink
Re: Kernel cached memory [In reply to]

Robert Hancock wrote:
> John Pearson wrote:
>
>> Wouldn't having (practically) all your memory used for cache slow down
>> starting a new program? First it would have to free up that space, and
>> then
>> put stuff in that space, taking potentially twice as long.
>
>
> If the cache pages are clean (not been modified since they were read
> from the disk), then evicting that data will not take very long. If the
> program you are just starting is not in the cache, then the time taken
> to load it from disk will dwarf the time needed to evict cached pages.
> And there's also the possibility that the cache contains the data you
> are loading, which definitely will speed things up..
>
The problem lies with data write evicting program pages in many cases.
You are right that they don't need to be written, but they do need to be
read back, so when I unhide a window there's a large delay while the
underlying program is read back in. If the pages are dirty, then there's
the delay while they are written.

It's exactly the benefit from having cached pages which is lost.

I would love more control in this area, but short of maintaining a patch
I don't see it happening.

--
-bill davidsen (davidsen [at] tmr)
"The secret to procrastination is to put things off until the
last possible moment - but no longer" -me
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo [at] vger
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/


lgb at lgb

Aug 1, 2005, 3:38 AM

Post #16 of 17 (971 views)
Permalink
Re: Kernel cached memory [In reply to]

On Mon, Jul 25, 2005 at 12:47:50PM -0400, Bill Davidsen wrote:
> Gábor Lénárt wrote:
> >On Fri, Jul 22, 2005 at 05:46:58PM +0800, Ashley wrote:
> >
> >> I've a server with 2 Operton 64bit CPU and 12G memory, and this server
> >>is used to run applications which will comsume huge memory,
> >>the problem is: when this aplications exits, the free memory of the
> >>server is still very low(accroding to the output of "top"), and
> >>from the output of command "free", I can see that many GB memory was
> >>cached by kernel. Does anyone know how to free the kernel cached
> >>memory? thanks in advance.
> >
> >
> >It's a very - very - very old and bad logic (at least nowdays) from the
> >stone age to free up memory.
>
> It's very Microsoft to claim that the OS always knows best, and not let
> the user tune the system the way they want it tuned. And if that means
> to leave a bunch of free memory for absolute fastest availability, the
> admin should have that option.

Sure, sorry if my comment can be treated in this way ... I mean surprising
amount of people I've met criticised Linux (well, some years ago when DOS
was popular) that he/she want to see that 'free memory' field reported eg by
'top' should be the maximum all the time ... I mean this way: this is the
behaviour which is quite wrong, I've written about this.

Sure, because of my not too good English, I may have missed the real meaning
of the mail, sorry about it!

--
- Gábor
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo [at] vger
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/


davidsen at tmr

Aug 1, 2005, 2:08 PM

Post #17 of 17 (970 views)
Permalink
Re: Kernel cached memory [In reply to]

Gábor Lénárt wrote:

>On Mon, Jul 25, 2005 at 12:47:50PM -0400, Bill Davidsen wrote:
>
>
>>Gábor Lénárt wrote:
>>
>>
>>>On Fri, Jul 22, 2005 at 05:46:58PM +0800, Ashley wrote:
>>>
>>>
>>>
>>>> I've a server with 2 Operton 64bit CPU and 12G memory, and this server
>>>>is used to run applications which will comsume huge memory,
>>>>the problem is: when this aplications exits, the free memory of the
>>>>server is still very low(accroding to the output of "top"), and
>>>>
>>>>
>>>>from the output of command "free", I can see that many GB memory was
>>>
>>>
>>>>cached by kernel. Does anyone know how to free the kernel cached
>>>>memory? thanks in advance.
>>>>
>>>>
>>>It's a very - very - very old and bad logic (at least nowdays) from the
>>>stone age to free up memory.
>>>
>>>
>>It's very Microsoft to claim that the OS always knows best, and not let
>>the user tune the system the way they want it tuned. And if that means
>>to leave a bunch of free memory for absolute fastest availability, the
>>admin should have that option.
>>
>>
>
>Sure, sorry if my comment can be treated in this way ... I mean surprising
>amount of people I've met criticised Linux (well, some years ago when DOS
>was popular) that he/she want to see that 'free memory' field reported eg by
>'top' should be the maximum all the time ... I mean this way: this is the
>behaviour which is quite wrong, I've written about this.
>
>Sure, because of my not too good English, I may have missed the real meaning
>of the mail, sorry about it!
>
Well, I thought I understood "from the stone age" but I may have taken
it slightly too literally. But I really would like to have more control
over Linux memory use, because it does cause bad behaviour at times. If
I have 4GB of RAM, I'd like to set 200MB or so aside for programs, and
never page out the window I'm going to uncover later. Likewise when I
write a DVD image, I would like to avoid buffering a few GB without i/o
and then driving the disk totally busy while it gets written out (after
it has pushed out things I will use again).

The old 2.4.x-aa kernels had some tunables to make the kernel aggressive
about writing pages to disk quickly, and I haven't been able to match
that behaviour without patches in 2.6. I may be missing a tunable, but
swappiness doesn't seem to be the one I want. I have a patch I'm playing
with, but it's not ready for prime time, and is probably counter to the
current philosophy of memory management.

Thanks for clarifying.

--
bill davidsen <davidsen [at] tmr>
CTO TMR Associates, Inc
Doing interesting things with small computers since 1979

-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo [at] vger
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/

Linux kernel RSS feed   Index | Next | Previous | View Threaded
 
 


Interested in having your list archived? Contact Gossamer Threads
 
  Web Applications & Managed Hosting Powered by Gossamer Threads Inc.