Mailing List Archive: Linux Virtual Server: Users

[lvs-users] ipvs-dr and ip_vs_conn

Index | Next | Previous | View Threaded

calculator at land

Nov 1, 2009, 1:10 AM

Post #1 of 7 (308 views)
Permalink

[lvs-users] ipvs-dr and ip_vs_conn

Hello.
Similar problem post on
http://lists.graemef.net/pipermail/lvs-users/2005-May/013820.html
I use ipvsadm-1.24-8.1 with heartbeat-ldirectord-2.1.3-3.el5 running on
Centos 5 (2.6.18)
With //debug_level 8 I dont have "delayed" messages and have like:
...
Nov 1 07:54:50 lb1-1 kernel: [ 121.636866] IPVS: ip_vs_sh_schedule():
Scheduling...
Nov 1 07:54:50 lb1-1 kernel: [ 121.636868] IPVS: SH: source IP address
10.11.11.213 --> server 10.104.4.102:80
Nov 1 07:54:50 lb1-1 kernel: [ 121.636875] IPVS: Bind-dest TCP
c:10.11.11.213:62498 v:10.11.11.222:80 d:10.104.4.102:80 fwd:R s:0
conn->flags:183 con
n->refcnt:1 dest->refcnt:4068
Nov 1 07:54:50 lb1-1 kernel: [ 121.636879] IPVS: Schedule fwd:R
c:10.11.11.213:62498 v:10.11.11.222:80 d:10.104.4.102:80 conn->flags:1C3
conn->refcnt
:2
Nov 1 07:54:50 lb1-1 kernel: [ 121.636882] IPVS: TCP input [S...]
10.104.4.102:80->10.11.11.213:62498 state: NONE->SYN_RECV conn->refcnt:2
Nov 1 07:54:50 lb1-1 kernel: [ 121.638061] IPVS: TCP input [..A.]
10.104.4.102:80->10.11.11.213:62497 state: SYN_RECV->ESTABLISHED
conn->refcnt:2
Nov 1 07:54:50 lb1-1 kernel: [ 121.638724] IPVS: TCP input [..A.]
10.104.4.102:80->10.11.11.213:62498 state: SYN_RECV->ESTABLISHED
conn->refcnt:2
Nov 1 07:54:50 lb1-1 kernel: [ 121.640768] IPVS: TCP input [.FA.]
10.104.4.102:80->10.11.11.213:62327 state: ESTABLISHED->FIN_WAIT
conn->refcnt:2
Nov 1 07:54:50 lb1-1 kernel: [ 121.640873] IPVS: TCP input [.FA.]
10.104.4.102:80->10.11.11.213:62465 state: ESTABLISHED->FIN_WAIT
conn->refcnt:2
Nov 1 07:54:50 lb1-1 kernel: [ 121.641675] IPVS: ip_vs_sh_schedule():
Scheduling...
...
//

In .config:
CONFIG_SMP=y
CONFIG_PREEMPT_NONE=y

this version does'nt exist string "__ip_vs_conn_put" in net/ipv4/ipvs/ip_vs_xmit.c
And my similar questions: What's wrong? Or, what am I not understanding? :-)

excuse me for my english

_______________________________________________
Please read the documentation before posting - it's available at:
http://www.linuxvirtualserver.org/

LinuxVirtualServer.org mailing list - lvs-users[at]LinuxVirtualServer.org
Send requests to lvs-users-request[at]LinuxVirtualServer.org
or go to http://lists.graemef.net/mailman/listinfo/lvs-users

graeme at graemef

Nov 2, 2009, 9:02 AM

Post #2 of 7 (293 views)
Permalink

Re: [lvs-users] ipvs-dr and ip_vs_conn [In reply to]

Hi there

On Sun, 2009-11-01 at 11:10 +0300, calculator[at]land.ru wrote:
> Similar problem post on
> http://lists.graemef.net/pipermail/lvs-users/2005-May/013820.html

Do you mean you have a director which is running out of memory?

> I use ipvsadm-1.24-8.1 with heartbeat-ldirectord-2.1.3-3.el5 running on
> Centos 5 (2.6.18)

That's reasonably recent.

> this version does'nt exist string "__ip_vs_conn_put" in net/ipv4/ipvs/ip_vs_xmit.c

That's because it was removed in 2.6.12 (I think).

> And my similar questions: What's wrong? Or, what am I not understanding? :-)

I think it would more accurate to say that we don't understand the
problem - can you explain it a little more clearly?

Graeme

_______________________________________________
Please read the documentation before posting - it's available at:
http://www.linuxvirtualserver.org/

LinuxVirtualServer.org mailing list - lvs-users[at]LinuxVirtualServer.org
Send requests to lvs-users-request[at]LinuxVirtualServer.org
or go to http://lists.graemef.net/mailman/listinfo/lvs-users

calculator at land

Nov 3, 2009, 12:54 AM

Post #3 of 7 (286 views)
Permalink

Re: [lvs-users] ipvs-dr and ip_vs_conn [In reply to]

Graeme Fowler:
> Hi there
>
> On Sun, 2009-11-01 at 11:10 +0300, calculator[at]land.ru wrote:
>
>> Similar problem post on
>> http://lists.graemef.net/pipermail/lvs-users/2005-May/013820.html
>>
>
> Do you mean you have a director which is running out of memory?
>

No, my mistake.
Records from /proc/net/ip_vs_conn are always removed and it's ok,
but with default settings, sometimes i have "IPVS: ip_vs_conn_new: no memory available." with 4G memory...

I don't understand: why if i use ipvs-dr scheme with sh
sheduler there are records in /proc/net/ip_vs_conn?

Does this is so because we do not want to drop "not full openen"/"not
full closed" connection so we renew sh sheduler _routing table_?

And one more question. If I set 'ipvsadm --set 1 1 1' will it influence on
whole ipvs (are packets which does not make whole cicle in 1 second
droped?), or will it influence only on count of records in ip_vs_conn?

>
>> I use ipvsadm-1.24-8.1 with heartbeat-ldirectord-2.1.3-3.el5 running on
>> Centos 5 (2.6.18)
>>
>
> That's reasonably recent.
>
>
>> this version does'nt exist string "__ip_vs_conn_put" in net/ipv4/ipvs/ip_vs_xmit.c
>>
>
> That's because it was removed in 2.6.12 (I think).
>
>
>> And my similar questions: What's wrong? Or, what am I not understanding? :-)
>>
>
> I think it would more accurate to say that we don't understand the
> problem - can you explain it a little more clearly?
>
> Graeme
>
>
> _______________________________________________
> Please read the documentation before posting - it's available at:
> http://www.linuxvirtualserver.org/
>
> LinuxVirtualServer.org mailing list - lvs-users[at]LinuxVirtualServer.org
> Send requests to lvs-users-request[at]LinuxVirtualServer.org
> or go to http://lists.graemef.net/mailman/listinfo/lvs-users
>
>
>
>
>

_______________________________________________
Please read the documentation before posting - it's available at:
http://www.linuxvirtualserver.org/

LinuxVirtualServer.org mailing list - lvs-users[at]LinuxVirtualServer.org
Send requests to lvs-users-request[at]LinuxVirtualServer.org
or go to http://lists.graemef.net/mailman/listinfo/lvs-users

horms at verge

Nov 3, 2009, 9:42 PM

Post #4 of 7 (280 views)
Permalink

Re: [lvs-users] ipvs-dr and ip_vs_conn [In reply to]

On Tue, Nov 03, 2009 at 11:54:07AM +0300, calculator[at]land.ru wrote:
>
> Graeme Fowler:
> > Hi there
> >
> > On Sun, 2009-11-01 at 11:10 +0300, calculator[at]land.ru wrote:
> >
> >> Similar problem post on
> >> http://lists.graemef.net/pipermail/lvs-users/2005-May/013820.html
> >>
> >
> > Do you mean you have a director which is running out of memory?
> >
>
> No, my mistake.
> Records from /proc/net/ip_vs_conn are always removed and it's ok,
> but with default settings, sometimes i have "IPVS: ip_vs_conn_new: no memory available." with 4G memory...
>
>
> I don't understand: why if i use ipvs-dr scheme with sh
> sheduler there are records in /proc/net/ip_vs_conn?
>
> Does this is so because we do not want to drop "not full openen"/"not
> full closed" connection so we renew sh sheduler _routing table_?

I guess that technically it ought to be possible to do without
/proc/net/ip_vs_conn entries when the sh-scheduler is in use.
But from an implementation point of view this is quite
a departure from the way LVS works internally. That is,
the scheduler is called to choose a real-server for new connections
and an entry is created in /proc/net/ip_vs_conn which is used
for the rest of the life of the connection.

> And one more question. If I set 'ipvsadm --set 1 1 1' will it influence on
> whole ipvs (are packets which does not make whole cicle in 1 second
> droped?), or will it influence only on count of records in ip_vs_conn?

I doubt that timeouts that short will have any affect on the system
as there are other, per-connection-state timeouts which are significantly
larger. I imagine that the smallest workable timeout for TCP would
be around 2 minutes.

Are you sure that you are exhausting ~4bytes of memory
through the entries in /proc/net/ip_vs_conn? To put this
in perspective, this implies that you have in the
order of 30 million connections in /proc/net/ip_vs_conn.

_______________________________________________
Please read the documentation before posting - it's available at:
http://www.linuxvirtualserver.org/

LinuxVirtualServer.org mailing list - lvs-users[at]LinuxVirtualServer.org
Send requests to lvs-users-request[at]LinuxVirtualServer.org
or go to http://lists.graemef.net/mailman/listinfo/lvs-users

calculator at land

Nov 5, 2009, 12:22 PM

Post #5 of 7 (272 views)
Permalink

Re: [lvs-users] ipvs-dr and ip_vs_conn [In reply to]

Strange, can not see previous my answer. Try again.
> On Tue, Nov 03, 2009 at 11:54:07AM +0300, calculator[at]land.ru wrote:
> >
> > Graeme Fowler:
> > > Hi there
> > >
> > > On Sun, 2009-11-01 at 11:10 +0300, calculator[at]land.ru wrote:
> > >
> > >> Similar problem post on
> > >> http://lists.graemef.net/pipermail/lvs-users/2005-May/013820.html
> > >>
> > >
> > > Do you mean you have a director which is running out of memory?
> > >
> >
> > No, my mistake.
> > Records from /proc/net/ip_vs_conn are always removed and it's ok,
> > but with default settings, sometimes i have "IPVS: ip_vs_conn_new:
> no memory available." with 4G memory...
> >
> >
> > I don't understand: why if i use ipvs-dr scheme with sh
> > sheduler there are records in /proc/net/ip_vs_conn?
> >
> > Does this is so because we do not want to drop "not full openen"/"not
> > full closed" connection so we renew sh sheduler _routing table_?
>
> I guess that technically it ought to be possible to do without
> /proc/net/ip_vs_conn entries when the sh-scheduler is in use.
> But from an implementation point of view this is quite
> a departure from the way LVS works internally. That is,
> the scheduler is called to choose a real-server for new connections
> and an entry is created in /proc/net/ip_vs_conn which is used
> for the rest of the life of the connection.
>
> > And one more question. If I set 'ipvsadm --set 1 1 1' will it
> influence on
> > whole ipvs (are packets which does not make whole cicle in 1 second
> > droped?), or will it influence only on count of records in ip_vs_conn?
>
> I doubt that timeouts that short will have any affect on the system
> as there are other, per-connection-state timeouts which are significantly
> larger. I imagine that the smallest workable timeout for TCP would
> be around 2 minutes.
Am i right, with lvs-dr+sh good timeouts important on real server? Not
in ip_vs_conn.
Does packages always delived to real server in order to sh _route
table_? Or not?
simple test:
LVS ~ # ipvsadm --set 1 1 1
http_test ~ # telnet host 80
...
(sleep 5 sec)
GET / HTTP/1.0
HTTP/1.1 302 Moved Temporarily
...
Connection closed by foreign host.

LVS ~ # cat /proc/net/ip_vs_conn
Pro FromIP FPrt ToIP TPrt DestIP DPrt State Expires
TCP D949C8D6 A77D D949C8DE 0050 0A680466 0050 ESTABLISHED 0
(sleep 5 sec)
LVS ~ # cat /proc/net/ip_vs_conn
Pro FromIP FPrt ToIP TPrt DestIP DPrt State Expires
TCP D949C8D6 A77D D949C8DE 0050 0A680466 0050 CLOSE 8

All ok.
>
> Are you sure that you are exhausting ~4bytes of memory
> through the entries in /proc/net/ip_vs_conn? To put this
> in perspective, this implies that you have in the
> order of 30 million connections in /proc/net/ip_vs_conn.
I have stat of current memory usage:

IP Virtual Server version 1.2.1 (size=262144)
Prot LocalAddress:Port CPS InPPS OutPPS InBPS OutBPS
-> RemoteAddress:Port
TCP XXX.XXX.XXX.XXX:80 17446 92686 0 13826264 0
...
slabtop:

OBJS ACTIVE USE OBJ SIZE SLABS OBJ/SLAB CACHE SIZE NAME
2041710 1976868 96% 0.12K 68057 30 272228K ip_vs_conn
150300 149100 99% 0.25K 10020 15 40080K ip_dst_cache
8482 8408 99% 2.00K 4241 2 16964K size-2048
65726 65689 99% 0.06K 1114 59 4456K inet_peer_cache
13720 8647 63% 0.19K 686 20 2744K skbuff_head_cache

sometime server goes down when CPS like ~45k:

1 Time(s): Oct 29 14:16:36 lb1-2 kernel: [ 565.355342] IPVS:
ip_vs_conn_new: no memory available.
1 Time(s): Oct 29 14:23:49 lb1-2 kernel: [ 997.509759] printk: 3365
messages suppressed.
1 Time(s): Oct 29 14:23:49 lb1-2 kernel: [ 997.509774] heartbeat invoked
oom-killer: gfp_mask=0xd0, order=0, oomkilladj=0
1 Time(s): Oct 29 14:23:49 lb1-2 kernel: [ 997.509798] [<c045489e>]
out_of_memory+0x1ae/0x1e0
1 Time(s): Oct 29 14:23:50 lb1-2 kernel: [ 997.509818] [<c04561a6>]
__alloc_pages+0x276/0x2e0
1 Time(s): Oct 29 14:23:50 lb1-2 kernel: [ 997.509834] [<c045622e>]
__get_free_pages+0x1e/0x40
1 Time(s): Oct 29 14:23:51 lb1-2 kernel: [ 997.509848] [<c04a68f8>]
proc_file_read+0x98/0x280
1 Time(s): Oct 29 14:23:51 lb1-2 kernel: [ 997.509863] [<c0635527>]
sys_recv+0x37/0x40
1 Time(s): Oct 29 14:23:51 lb1-2 kernel: [ 997.509877] [<c0473661>]
vfs_read+0xa1/0x160
1 Time(s): Oct 29 14:23:51 lb1-2 kernel: [ 997.509968] [<c04729bc>]
vfs_llseek+0x3c/0x50
1 Time(s): Oct 29 14:23:51 lb1-2 kernel: [ 997.510056] [<c04a6860>]
proc_file_read+0x0/0x280
1 Time(s): Oct 29 14:23:51 lb1-2 kernel: [ 997.510181] [<c0473bd1>]
sys_read+0x41/0x70
1 Time(s): Oct 29 14:23:51 lb1-2 kernel: [ 997.510270] [<c040525d>]
sysenter_past_esp+0x56/0x79
1 Time(s): Oct 29 14:23:51 lb1-2 kernel: [ 997.510362]
=======================
1 Time(s): Oct 29 14:23:51 lb1-2 kernel: [ 997.510446] Mem-info:
1 Time(s): Oct 29 14:23:51 lb1-2 kernel: [ 997.510448] DMA per-cpu:
1 Time(s): Oct 29 14:23:51 lb1-2 kernel: [ 997.510533] cpu 0 hot: high
0, batch 1 used:0
1 Time(s): Oct 29 14:23:51 lb1-2 kernel: [ 997.510618] cpu 0 cold: high
0, batch 1 used:0
1 Time(s): Oct 29 14:23:51 lb1-2 kernel: [ 997.510704] cpu 1 hot: high
0, batch 1 used:0
1 Time(s): Oct 29 14:23:51 lb1-2 kernel: [ 997.510789] cpu 1 cold: high
0, batch 1 used:0
1 Time(s): Oct 29 14:23:52 lb1-2 kernel: [ 997.510875] DMA32 per-cpu: empty
1 Time(s): Oct 29 14:23:52 lb1-2 kernel: [ 997.510960] Normal per-cpu:
1 Time(s): Oct 29 14:23:52 lb1-2 kernel: [ 997.511046] cpu 0 hot: high
186, batch 31 used:32
1 Time(s): Oct 29 14:23:52 lb1-2 kernel: [ 997.511178] cpu 0 cold: high
62, batch 15 used:51
1 Time(s): Oct 29 14:23:52 lb1-2 kernel: [ 997.511266] cpu 1 hot: high
186, batch 31 used:167
1 Time(s): Oct 29 14:23:52 lb1-2 kernel: [ 997.511350] cpu 1 cold: high
62, batch 15 used:54
1 Time(s): Oct 29 14:23:52 lb1-2 kernel: [ 997.511434] HighMem per-cpu:
1 Time(s): Oct 29 14:23:52 lb1-2 kernel: [ 997.511516] cpu 0 hot: high
186, batch 31 used:21
1 Time(s): Oct 29 14:23:52 lb1-2 kernel: [ 997.511600] cpu 0 cold: high
62, batch 15 used:7
1 Time(s): Oct 29 14:23:52 lb1-2 kernel: [ 997.511684] cpu 1 hot: high
186, batch 31 used:140
1 Time(s): Oct 29 14:23:52 lb1-2 kernel: [ 997.511768] cpu 1 cold: high
62, batch 15 used:10
1 Time(s): Oct 29 14:23:52 lb1-2 kernel: [ 997.511853] Free pages:
2142380kB (2137456kB HighMem)
1 Time(s): Oct 29 14:23:52 lb1-2 kernel: [ 997.511940] Active:17594
inactive:2652 dirty:0 writeback:38 unstable:0 free:535595 slab:219589
mapped-file:2869 mapped-anon:14553 pagetables:596
1 Time(s): Oct 29 14:23:52 lb1-2 kernel: [ 997.512151] DMA free:3548kB
min:68kB low:84kB high:100kB active:4kB inactive:0kB present:16384kB
pages_scanned:6 all_unreclaimable? yes
1 Time(s): Oct 29 14:23:52 lb1-2 kernel: [ 997.512322] lowmem_reserve[]:
0 0 880 3055
1 Time(s): Oct 29 14:23:52 lb1-2 kernel: [ 997.512415] DMA32 free:0kB
min:0kB low:0kB high:0kB active:0kB inactive:0kB present:0kB
pages_scanned:0 all_unreclaimable? no
1 Time(s): Oct 29 14:23:52 lb1-2 kernel: [ 997.512583] lowmem_reserve[]:
0 0 880 3055
1 Time(s): Oct 29 14:23:53 lb1-2 kernel: [ 997.512677] Normal
free:1376kB min:3756kB low:4692kB high:5632kB active:112kB
inactive:180kB present:901120kB pages_scanned:265 all_unreclaimable? no
1 Time(s): Oct 29 14:23:53 lb1-2 kernel: [ 997.512850] lowmem_reserve[]:
0 0 0 17400
1 Time(s): Oct 29 14:23:53 lb1-2 kernel: [ 997.512943] HighMem
free:2137456kB min:512kB low:2832kB high:5156kB active:70224kB
inactive:10464kB present:2227200kB pages_scanned:0 all_unreclaimable? no
1 Time(s): Oct 29 14:23:53 lb1-2 kernel: [ 997.513136] lowmem_reserve[]:
0 0 0 0
1 Time(s): Oct 29 14:23:53 lb1-2 kernel: [ 997.513228] DMA: 1*4kB 1*8kB
1*16kB 0*32kB 1*64kB 1*128kB 1*256kB 0*512kB 1*1024kB 1*2048kB 0*4096kB
= 3548kB
1 Time(s): Oct 29 14:23:53 lb1-2 kernel: [ 997.513417] DMA32: empty
1 Time(s): Oct 29 14:23:53 lb1-2 kernel: [ 997.513498] Normal: 0*4kB
4*8kB 0*16kB 2*32kB 0*64kB 0*128kB 1*256kB 0*512kB 1*1024kB 0*2048kB
0*4096kB = 1376kB
1 Time(s): Oct 29 14:23:53 lb1-2 kernel: [ 997.513686] HighMem: 182*4kB
59*8kB 78*16kB 105*32kB 39*64kB 14*128kB 2*256kB 2*512kB 0*1024kB
0*2048kB 519*4096kB = 2137456kB
1 Time(s): Oct 29 14:23:53 lb1-2 kernel: [ 997.513878] Swap cache: add
0, delete 0, find 0/0, race 0+0
1 Time(s): Oct 29 14:23:53 lb1-2 kernel: [ 997.513964] Free swap = 0kB
1 Time(s): Oct 29 14:23:54 lb1-2 kernel: [ 997.514045] Total swap = 0kB
1 Time(s): Oct 29 14:23:54 lb1-2 kernel: [ 997.514163] Free swap: 0kB
1 Time(s): Oct 29 14:23:54 lb1-2 kernel: [ 997.520251] 786176 pages of RAM
1 Time(s): Oct 29 14:23:54 lb1-2 kernel: [ 997.520253] 556800 pages of
HIGHMEM
1 Time(s): Oct 29 14:23:54 lb1-2 kernel: [ 997.520254] 8061 reserved pages
1 Time(s): Oct 29 14:23:54 lb1-2 kernel: [ 997.520255] 26709 pages shared
1 Time(s): Oct 29 14:23:54 lb1-2 kernel: [ 997.520256] 0 pages swap cached
1 Time(s): Oct 29 14:23:54 lb1-2 kernel: [ 997.520257] 0 pages dirty
1 Time(s): Oct 29 14:23:54 lb1-2 kernel: [ 997.520258] 38 pages writeback
1 Time(s): Oct 29 14:23:54 lb1-2 kernel: [ 997.520259] 2869 pages mapped
1 Time(s): Oct 29 14:23:54 lb1-2 kernel: [ 997.520260] 219589 pages slab
1 Time(s): Oct 29 14:23:54 lb1-2 kernel: [ 997.520261] 596 pages pagetables
1 Time(s): Oct 29 14:23:54 lb1-2 kernel: [ 997.520298] Out of memory:
Killed process 2291 (bash).

Maybe, there is missconfiguration kernel?

_______________________________________________
Please read the documentation before posting - it's available at:
http://www.linuxvirtualserver.org/

LinuxVirtualServer.org mailing list - lvs-users[at]LinuxVirtualServer.org
Send requests to lvs-users-request[at]LinuxVirtualServer.org
or go to http://lists.graemef.net/mailman/listinfo/lvs-users

horms at verge

Nov 5, 2009, 4:00 PM

Post #6 of 7 (271 views)
Permalink

Re: [lvs-users] ipvs-dr and ip_vs_conn [In reply to]

On Thu, Nov 05, 2009 at 11:22:54PM +0300, calculator[at]land.ru wrote:
> Strange, can not see previous my answer. Try again.
> > On Tue, Nov 03, 2009 at 11:54:07AM +0300, calculator[at]land.ru wrote:
> > >
> > > Graeme Fowler:
> > > > Hi there
> > > >
> > > > On Sun, 2009-11-01 at 11:10 +0300, calculator[at]land.ru wrote:
> > > >
> > > >> Similar problem post on
> > > >> http://lists.graemef.net/pipermail/lvs-users/2005-May/013820.html
> > > >>
> > > >
> > > > Do you mean you have a director which is running out of memory?
> > > >
> > >
> > > No, my mistake.
> > > Records from /proc/net/ip_vs_conn are always removed and it's ok,
> > > but with default settings, sometimes i have "IPVS: ip_vs_conn_new:
> > no memory available." with 4G memory...
> > >
> > >
> > > I don't understand: why if i use ipvs-dr scheme with sh
> > > sheduler there are records in /proc/net/ip_vs_conn?
> > >
> > > Does this is so because we do not want to drop "not full openen"/"not
> > > full closed" connection so we renew sh sheduler _routing table_?
> >
> > I guess that technically it ought to be possible to do without
> > /proc/net/ip_vs_conn entries when the sh-scheduler is in use.
> > But from an implementation point of view this is quite
> > a departure from the way LVS works internally. That is,
> > the scheduler is called to choose a real-server for new connections
> > and an entry is created in /proc/net/ip_vs_conn which is used
> > for the rest of the life of the connection.
> >
> > > And one more question. If I set 'ipvsadm --set 1 1 1' will it
> > influence on
> > > whole ipvs (are packets which does not make whole cicle in 1 second
> > > droped?), or will it influence only on count of records in ip_vs_conn?
> >
> > I doubt that timeouts that short will have any affect on the system
> > as there are other, per-connection-state timeouts which are significantly
> > larger. I imagine that the smallest workable timeout for TCP would
> > be around 2 minutes.
> Am i right, with lvs-dr+sh good timeouts important on real server? Not
> in ip_vs_conn.
> Does packages always delived to real server in order to sh _route
> table_? Or not?

I'm not sure what you are asking.

> simple test:
> LVS ~ # ipvsadm --set 1 1 1
> http_test ~ # telnet host 80
> ...
> (sleep 5 sec)
> GET / HTTP/1.0
> HTTP/1.1 302 Moved Temporarily
> ...
> Connection closed by foreign host.
>
> LVS ~ # cat /proc/net/ip_vs_conn
> Pro FromIP FPrt ToIP TPrt DestIP DPrt State Expires
> TCP D949C8D6 A77D D949C8DE 0050 0A680466 0050 ESTABLISHED 0
> (sleep 5 sec)
> LVS ~ # cat /proc/net/ip_vs_conn
> Pro FromIP FPrt ToIP TPrt DestIP DPrt State Expires
> TCP D949C8D6 A77D D949C8DE 0050 0A680466 0050 CLOSE 8
>
> All ok.

Do you notice any difference in memory usage when you set time-outs to 1 1 1?
Have you observed what state the entries in ip_vs_conn are in? Profiling
that may help in working out how to tune things - or why there are so many
entries.

> > Are you sure that you are exhausting ~4bytes of memory
> > through the entries in /proc/net/ip_vs_conn? To put this
> > in perspective, this implies that you have in the
> > order of 30 million connections in /proc/net/ip_vs_conn.
> I have stat of current memory usage:
>
> IP Virtual Server version 1.2.1 (size=262144)
> Prot LocalAddress:Port CPS InPPS OutPPS InBPS OutBPS
> -> RemoteAddress:Port
> TCP XXX.XXX.XXX.XXX:80 17446 92686 0 13826264 0
> ...
> slabtop:
>
> OBJS ACTIVE USE OBJ SIZE SLABS OBJ/SLAB CACHE SIZE NAME
> 2041710 1976868 96% 0.12K 68057 30 272228K ip_vs_conn
> 150300 149100 99% 0.25K 10020 15 40080K ip_dst_cache
> 8482 8408 99% 2.00K 4241 2 16964K size-2048
> 65726 65689 99% 0.06K 1114 59 4456K inet_peer_cache
> 13720 8647 63% 0.19K 686 20 2744K skbuff_head_cache

Unless I'm reading things wrong, it looks like you have about 272Mbytes
worth of ip_vs_conn entries, or about 2Million entires. I must say
that I am surprised that the number is that high. It implies
that you are routing 2Million simultaneous connections through LVS.

> sometime server goes down when CPS like ~45k:

I'm still a little surprised that you're running out of memory,
the numbers above don't seem to add up to anything close to 4Gb.
Perhaps the kernel only has access to a portion of that memory?

> 1 Time(s): Oct 29 14:16:36 lb1-2 kernel: [ 565.355342] IPVS:
> ip_vs_conn_new: no memory available.
> 1 Time(s): Oct 29 14:23:49 lb1-2 kernel: [ 997.509759] printk: 3365
> messages suppressed.
> 1 Time(s): Oct 29 14:23:49 lb1-2 kernel: [ 997.509774] heartbeat invoked
> oom-killer: gfp_mask=0xd0, order=0, oomkilladj=0
> 1 Time(s): Oct 29 14:23:49 lb1-2 kernel: [ 997.509798] [<c045489e>]
> out_of_memory+0x1ae/0x1e0
> 1 Time(s): Oct 29 14:23:50 lb1-2 kernel: [ 997.509818] [<c04561a6>]
> __alloc_pages+0x276/0x2e0
> 1 Time(s): Oct 29 14:23:50 lb1-2 kernel: [ 997.509834] [<c045622e>]
> __get_free_pages+0x1e/0x40
> 1 Time(s): Oct 29 14:23:51 lb1-2 kernel: [ 997.509848] [<c04a68f8>]
> proc_file_read+0x98/0x280
> 1 Time(s): Oct 29 14:23:51 lb1-2 kernel: [ 997.509863] [<c0635527>]
> sys_recv+0x37/0x40
> 1 Time(s): Oct 29 14:23:51 lb1-2 kernel: [ 997.509877] [<c0473661>]
> vfs_read+0xa1/0x160
> 1 Time(s): Oct 29 14:23:51 lb1-2 kernel: [ 997.509968] [<c04729bc>]
> vfs_llseek+0x3c/0x50
> 1 Time(s): Oct 29 14:23:51 lb1-2 kernel: [ 997.510056] [<c04a6860>]
> proc_file_read+0x0/0x280
> 1 Time(s): Oct 29 14:23:51 lb1-2 kernel: [ 997.510181] [<c0473bd1>]
> sys_read+0x41/0x70
> 1 Time(s): Oct 29 14:23:51 lb1-2 kernel: [ 997.510270] [<c040525d>]
> sysenter_past_esp+0x56/0x79
> 1 Time(s): Oct 29 14:23:51 lb1-2 kernel: [ 997.510362]
> =======================
> 1 Time(s): Oct 29 14:23:51 lb1-2 kernel: [ 997.510446] Mem-info:
> 1 Time(s): Oct 29 14:23:51 lb1-2 kernel: [ 997.510448] DMA per-cpu:
> 1 Time(s): Oct 29 14:23:51 lb1-2 kernel: [ 997.510533] cpu 0 hot: high
> 0, batch 1 used:0
> 1 Time(s): Oct 29 14:23:51 lb1-2 kernel: [ 997.510618] cpu 0 cold: high
> 0, batch 1 used:0
> 1 Time(s): Oct 29 14:23:51 lb1-2 kernel: [ 997.510704] cpu 1 hot: high
> 0, batch 1 used:0
> 1 Time(s): Oct 29 14:23:51 lb1-2 kernel: [ 997.510789] cpu 1 cold: high
> 0, batch 1 used:0
> 1 Time(s): Oct 29 14:23:52 lb1-2 kernel: [ 997.510875] DMA32 per-cpu: empty
> 1 Time(s): Oct 29 14:23:52 lb1-2 kernel: [ 997.510960] Normal per-cpu:
> 1 Time(s): Oct 29 14:23:52 lb1-2 kernel: [ 997.511046] cpu 0 hot: high
> 186, batch 31 used:32
> 1 Time(s): Oct 29 14:23:52 lb1-2 kernel: [ 997.511178] cpu 0 cold: high
> 62, batch 15 used:51
> 1 Time(s): Oct 29 14:23:52 lb1-2 kernel: [ 997.511266] cpu 1 hot: high
> 186, batch 31 used:167
> 1 Time(s): Oct 29 14:23:52 lb1-2 kernel: [ 997.511350] cpu 1 cold: high
> 62, batch 15 used:54
> 1 Time(s): Oct 29 14:23:52 lb1-2 kernel: [ 997.511434] HighMem per-cpu:
> 1 Time(s): Oct 29 14:23:52 lb1-2 kernel: [ 997.511516] cpu 0 hot: high
> 186, batch 31 used:21
> 1 Time(s): Oct 29 14:23:52 lb1-2 kernel: [ 997.511600] cpu 0 cold: high
> 62, batch 15 used:7
> 1 Time(s): Oct 29 14:23:52 lb1-2 kernel: [ 997.511684] cpu 1 hot: high
> 186, batch 31 used:140
> 1 Time(s): Oct 29 14:23:52 lb1-2 kernel: [ 997.511768] cpu 1 cold: high
> 62, batch 15 used:10
> 1 Time(s): Oct 29 14:23:52 lb1-2 kernel: [ 997.511853] Free pages:
> 2142380kB (2137456kB HighMem)
> 1 Time(s): Oct 29 14:23:52 lb1-2 kernel: [ 997.511940] Active:17594
> inactive:2652 dirty:0 writeback:38 unstable:0 free:535595 slab:219589
> mapped-file:2869 mapped-anon:14553 pagetables:596
> 1 Time(s): Oct 29 14:23:52 lb1-2 kernel: [ 997.512151] DMA free:3548kB
> min:68kB low:84kB high:100kB active:4kB inactive:0kB present:16384kB
> pages_scanned:6 all_unreclaimable? yes
> 1 Time(s): Oct 29 14:23:52 lb1-2 kernel: [ 997.512322] lowmem_reserve[]:
> 0 0 880 3055
> 1 Time(s): Oct 29 14:23:52 lb1-2 kernel: [ 997.512415] DMA32 free:0kB
> min:0kB low:0kB high:0kB active:0kB inactive:0kB present:0kB
> pages_scanned:0 all_unreclaimable? no
> 1 Time(s): Oct 29 14:23:52 lb1-2 kernel: [ 997.512583] lowmem_reserve[]:
> 0 0 880 3055
> 1 Time(s): Oct 29 14:23:53 lb1-2 kernel: [ 997.512677] Normal
> free:1376kB min:3756kB low:4692kB high:5632kB active:112kB
> inactive:180kB present:901120kB pages_scanned:265 all_unreclaimable? no
> 1 Time(s): Oct 29 14:23:53 lb1-2 kernel: [ 997.512850] lowmem_reserve[]:
> 0 0 0 17400
> 1 Time(s): Oct 29 14:23:53 lb1-2 kernel: [ 997.512943] HighMem
> free:2137456kB min:512kB low:2832kB high:5156kB active:70224kB
> inactive:10464kB present:2227200kB pages_scanned:0 all_unreclaimable? no
> 1 Time(s): Oct 29 14:23:53 lb1-2 kernel: [ 997.513136] lowmem_reserve[]:
> 0 0 0 0
> 1 Time(s): Oct 29 14:23:53 lb1-2 kernel: [ 997.513228] DMA: 1*4kB 1*8kB
> 1*16kB 0*32kB 1*64kB 1*128kB 1*256kB 0*512kB 1*1024kB 1*2048kB 0*4096kB
> = 3548kB
> 1 Time(s): Oct 29 14:23:53 lb1-2 kernel: [ 997.513417] DMA32: empty
> 1 Time(s): Oct 29 14:23:53 lb1-2 kernel: [ 997.513498] Normal: 0*4kB
> 4*8kB 0*16kB 2*32kB 0*64kB 0*128kB 1*256kB 0*512kB 1*1024kB 0*2048kB
> 0*4096kB = 1376kB
> 1 Time(s): Oct 29 14:23:53 lb1-2 kernel: [ 997.513686] HighMem: 182*4kB
> 59*8kB 78*16kB 105*32kB 39*64kB 14*128kB 2*256kB 2*512kB 0*1024kB
> 0*2048kB 519*4096kB = 2137456kB
> 1 Time(s): Oct 29 14:23:53 lb1-2 kernel: [ 997.513878] Swap cache: add
> 0, delete 0, find 0/0, race 0+0
> 1 Time(s): Oct 29 14:23:53 lb1-2 kernel: [ 997.513964] Free swap = 0kB
> 1 Time(s): Oct 29 14:23:54 lb1-2 kernel: [ 997.514045] Total swap = 0kB
> 1 Time(s): Oct 29 14:23:54 lb1-2 kernel: [ 997.514163] Free swap: 0kB
> 1 Time(s): Oct 29 14:23:54 lb1-2 kernel: [ 997.520251] 786176 pages of RAM
> 1 Time(s): Oct 29 14:23:54 lb1-2 kernel: [ 997.520253] 556800 pages of
> HIGHMEM
> 1 Time(s): Oct 29 14:23:54 lb1-2 kernel: [ 997.520254] 8061 reserved pages
> 1 Time(s): Oct 29 14:23:54 lb1-2 kernel: [ 997.520255] 26709 pages shared
> 1 Time(s): Oct 29 14:23:54 lb1-2 kernel: [ 997.520256] 0 pages swap cached
> 1 Time(s): Oct 29 14:23:54 lb1-2 kernel: [ 997.520257] 0 pages dirty
> 1 Time(s): Oct 29 14:23:54 lb1-2 kernel: [ 997.520258] 38 pages writeback
> 1 Time(s): Oct 29 14:23:54 lb1-2 kernel: [ 997.520259] 2869 pages mapped
> 1 Time(s): Oct 29 14:23:54 lb1-2 kernel: [ 997.520260] 219589 pages slab
> 1 Time(s): Oct 29 14:23:54 lb1-2 kernel: [ 997.520261] 596 pages pagetables
> 1 Time(s): Oct 29 14:23:54 lb1-2 kernel: [ 997.520298] Out of memory:
> Killed process 2291 (bash).
>
> Maybe, there is missconfiguration kernel?
>
> _______________________________________________
> Please read the documentation before posting - it's available at:
> http://www.linuxvirtualserver.org/
>
> LinuxVirtualServer.org mailing list - lvs-users[at]LinuxVirtualServer.org
> Send requests to lvs-users-request[at]LinuxVirtualServer.org
> or go to http://lists.graemef.net/mailman/listinfo/lvs-users

_______________________________________________
Please read the documentation before posting - it's available at:
http://www.linuxvirtualserver.org/

LinuxVirtualServer.org mailing list - lvs-users[at]LinuxVirtualServer.org
Send requests to lvs-users-request[at]LinuxVirtualServer.org
or go to http://lists.graemef.net/mailman/listinfo/lvs-users

calculator at land

Nov 6, 2009, 5:59 AM

Post #7 of 7 (268 views)
Permalink

Re: [lvs-users] ipvs-dr and ip_vs_conn [In reply to]

Simon Horman wrote:
> On Thu, Nov 05, 2009 at 11:22:54PM +0300, calculator[at]land.ru wrote:
>
>> Strange, can not see previous my answer. Try again.
>>
>>> On Tue, Nov 03, 2009 at 11:54:07AM +0300, calculator[at]land.ru wrote:
>>>
>>>> Graeme Fowler:
>>>>
>>>>> Hi there
>>>>>
>>>>> On Sun, 2009-11-01 at 11:10 +0300, calculator[at]land.ru wrote:
>>>>>
>>>>>
>>>>>> Similar problem post on
>>>>>> http://lists.graemef.net/pipermail/lvs-users/2005-May/013820.html
>>>>>>
>>>>>>
>>>>> Do you mean you have a director which is running out of memory?
>>>>>
>>>>>
>>>> No, my mistake.
>>>> Records from /proc/net/ip_vs_conn are always removed and it's ok,
>>>> but with default settings, sometimes i have "IPVS: ip_vs_conn_new: no memory available." with 4G memory...
>>>> I don't understand: why if i use ipvs-dr scheme with sh
>>>> sheduler there are records in /proc/net/ip_vs_conn?
>>>>
>>>> Does this is so because we do not want to drop "not full openen"/"not
>>>> full closed" connection so we renew sh sheduler _routing table_?
>>>>
>>> I guess that technically it ought to be possible to do without
>>> /proc/net/ip_vs_conn entries when the sh-scheduler is in use.
>>> But from an implementation point of view this is quite
>>> a departure from the way LVS works internally. That is,
>>> the scheduler is called to choose a real-server for new connections
>>> and an entry is created in /proc/net/ip_vs_conn which is used
>>> for the rest of the life of the connection.
>>>
>>>
>>>> And one more question. If I set 'ipvsadm --set 1 1 1' will it influence on whole ipvs (are packets which does not make whole cicle in 1 second
>>>> droped?), or will it influence only on count of records in ip_vs_conn?
>>>>
>>> I doubt that timeouts that short will have any affect on the system
>>> as there are other, per-connection-state timeouts which are significantly
>>> larger. I imagine that the smallest workable timeout for TCP would
>>> be around 2 minutes.
>>>
>> Am i right, with lvs-dr+sh good timeouts important on real server? Not
>> in ip_vs_conn.
>> Does packages always delived to real server in order to sh _route
>> table_? Or not?
>>
>
> I'm not sure what you are asking.
>
>
>> simple test:
>> LVS ~ # ipvsadm --set 1 1 1
>> http_test ~ # telnet host 80
>> ...
>> (sleep 5 sec)
>> GET / HTTP/1.0
>> HTTP/1.1 302 Moved Temporarily
>> ...
>> Connection closed by foreign host.
>>
>> LVS ~ # cat /proc/net/ip_vs_conn
>> Pro FromIP FPrt ToIP TPrt DestIP DPrt State Expires
>> TCP D949C8D6 A77D D949C8DE 0050 0A680466 0050 ESTABLISHED 0
>> (sleep 5 sec)
>> LVS ~ # cat /proc/net/ip_vs_conn
>> Pro FromIP FPrt ToIP TPrt DestIP DPrt State Expires
>> TCP D949C8D6 A77D D949C8DE 0050 0A680466 0050 CLOSE 8
>>
>> All ok.
>>
>
> Do you notice any difference in memory usage when you set time-outs to 1 1 1?
>
Yes.
> Have you observed what state the entries in ip_vs_conn are in? Profiling
> that may help in working out how to tune things - or why there are so many
> entries.
>
>
On test station, with sets 1 1 1 i have ~1000 ip_vs_conn OBJECTS after 2
minutes test. with default settings: ~55000
Profiling is good, but my competition in profiling is not good :-(
>>> Are you sure that you are exhausting ~4bytes of memory
>>> through the entries in /proc/net/ip_vs_conn? To put this
>>> in perspective, this implies that you have in the
>>> order of 30 million connections in /proc/net/ip_vs_conn.
>>>
>> I have stat of current memory usage:
>>
>> IP Virtual Server version 1.2.1 (size=262144)
>> Prot LocalAddress:Port CPS InPPS OutPPS InBPS OutBPS
>> -> RemoteAddress:Port
>> TCP XXX.XXX.XXX.XXX:80 17446 92686 0 13826264 0
>> ...
>> slabtop:
>>
>> OBJS ACTIVE USE OBJ SIZE SLABS OBJ/SLAB CACHE SIZE NAME
>> 2041710 1976868 96% 0.12K 68057 30 272228K ip_vs_conn
>> 150300 149100 99% 0.25K 10020 15 40080K ip_dst_cache
>> 8482 8408 99% 2.00K 4241 2 16964K size-2048
>> 65726 65689 99% 0.06K 1114 59 4456K inet_peer_cache
>> 13720 8647 63% 0.19K 686 20 2744K skbuff_head_cache
>>
>
> Unless I'm reading things wrong, it looks like you have about 272Mbytes
> worth of ip_vs_conn entries, or about 2Million entires. I must say
> that I am surprised that the number is that high. It implies
> that you are routing 2Million simultaneous connections through LVS.
>
This depends from big expires(like 900) on some connections.
>
>> sometime server goes down when CPS like ~45k:
>>
>
> I'm still a little surprised that you're running out of memory,
> the numbers above don't seem to add up to anything close to 4Gb.
> Perhaps the kernel only has access to a portion of that memory?
>
Maybe. How can i test this?
>
>> 1 Time(s): Oct 29 14:16:36 lb1-2 kernel: [ 565.355342] IPVS:
>> ip_vs_conn_new: no memory available.
>> 1 Time(s): Oct 29 14:23:49 lb1-2 kernel: [ 997.509759] printk: 3365
>> messages suppressed.
>> 1 Time(s): Oct 29 14:23:49 lb1-2 kernel: [ 997.509774] heartbeat invoked
>> oom-killer: gfp_mask=0xd0, order=0, oomkilladj=0
>> 1 Time(s): Oct 29 14:23:49 lb1-2 kernel: [ 997.509798] [<c045489e>]
>> out_of_memory+0x1ae/0x1e0
>> 1 Time(s): Oct 29 14:23:50 lb1-2 kernel: [ 997.509818] [<c04561a6>]
>> __alloc_pages+0x276/0x2e0
>> 1 Time(s): Oct 29 14:23:50 lb1-2 kernel: [ 997.509834] [<c045622e>]
>> __get_free_pages+0x1e/0x40
>> 1 Time(s): Oct 29 14:23:51 lb1-2 kernel: [ 997.509848] [<c04a68f8>]
>> proc_file_read+0x98/0x280
>> 1 Time(s): Oct 29 14:23:51 lb1-2 kernel: [ 997.509863] [<c0635527>]
>> sys_recv+0x37/0x40
>> 1 Time(s): Oct 29 14:23:51 lb1-2 kernel: [ 997.509877] [<c0473661>]
>> vfs_read+0xa1/0x160
>> 1 Time(s): Oct 29 14:23:51 lb1-2 kernel: [ 997.509968] [<c04729bc>]
>> vfs_llseek+0x3c/0x50
>> 1 Time(s): Oct 29 14:23:51 lb1-2 kernel: [ 997.510056] [<c04a6860>]
>> proc_file_read+0x0/0x280
>> 1 Time(s): Oct 29 14:23:51 lb1-2 kernel: [ 997.510181] [<c0473bd1>]
>> sys_read+0x41/0x70
>> 1 Time(s): Oct 29 14:23:51 lb1-2 kernel: [ 997.510270] [<c040525d>]
>> sysenter_past_esp+0x56/0x79
>> 1 Time(s): Oct 29 14:23:51 lb1-2 kernel: [ 997.510362]
>> =======================
>> 1 Time(s): Oct 29 14:23:51 lb1-2 kernel: [ 997.510446] Mem-info:
>> 1 Time(s): Oct 29 14:23:51 lb1-2 kernel: [ 997.510448] DMA per-cpu:
>> 1 Time(s): Oct 29 14:23:51 lb1-2 kernel: [ 997.510533] cpu 0 hot: high
>> 0, batch 1 used:0
>> 1 Time(s): Oct 29 14:23:51 lb1-2 kernel: [ 997.510618] cpu 0 cold: high
>> 0, batch 1 used:0
>> 1 Time(s): Oct 29 14:23:51 lb1-2 kernel: [ 997.510704] cpu 1 hot: high
>> 0, batch 1 used:0
>> 1 Time(s): Oct 29 14:23:51 lb1-2 kernel: [ 997.510789] cpu 1 cold: high
>> 0, batch 1 used:0
>> 1 Time(s): Oct 29 14:23:52 lb1-2 kernel: [ 997.510875] DMA32 per-cpu: empty
>> 1 Time(s): Oct 29 14:23:52 lb1-2 kernel: [ 997.510960] Normal per-cpu:
>> 1 Time(s): Oct 29 14:23:52 lb1-2 kernel: [ 997.511046] cpu 0 hot: high
>> 186, batch 31 used:32
>> 1 Time(s): Oct 29 14:23:52 lb1-2 kernel: [ 997.511178] cpu 0 cold: high
>> 62, batch 15 used:51
>> 1 Time(s): Oct 29 14:23:52 lb1-2 kernel: [ 997.511266] cpu 1 hot: high
>> 186, batch 31 used:167
>> 1 Time(s): Oct 29 14:23:52 lb1-2 kernel: [ 997.511350] cpu 1 cold: high
>> 62, batch 15 used:54
>> 1 Time(s): Oct 29 14:23:52 lb1-2 kernel: [ 997.511434] HighMem per-cpu:
>> 1 Time(s): Oct 29 14:23:52 lb1-2 kernel: [ 997.511516] cpu 0 hot: high
>> 186, batch 31 used:21
>> 1 Time(s): Oct 29 14:23:52 lb1-2 kernel: [ 997.511600] cpu 0 cold: high
>> 62, batch 15 used:7
>> 1 Time(s): Oct 29 14:23:52 lb1-2 kernel: [ 997.511684] cpu 1 hot: high
>> 186, batch 31 used:140
>> 1 Time(s): Oct 29 14:23:52 lb1-2 kernel: [ 997.511768] cpu 1 cold: high
>> 62, batch 15 used:10
>> 1 Time(s): Oct 29 14:23:52 lb1-2 kernel: [ 997.511853] Free pages:
>> 2142380kB (2137456kB HighMem)
>> 1 Time(s): Oct 29 14:23:52 lb1-2 kernel: [ 997.511940] Active:17594
>> inactive:2652 dirty:0 writeback:38 unstable:0 free:535595 slab:219589
>> mapped-file:2869 mapped-anon:14553 pagetables:596
>> 1 Time(s): Oct 29 14:23:52 lb1-2 kernel: [ 997.512151] DMA free:3548kB
>> min:68kB low:84kB high:100kB active:4kB inactive:0kB present:16384kB
>> pages_scanned:6 all_unreclaimable? yes
>> 1 Time(s): Oct 29 14:23:52 lb1-2 kernel: [ 997.512322] lowmem_reserve[]:
>> 0 0 880 3055
>> 1 Time(s): Oct 29 14:23:52 lb1-2 kernel: [ 997.512415] DMA32 free:0kB
>> min:0kB low:0kB high:0kB active:0kB inactive:0kB present:0kB
>> pages_scanned:0 all_unreclaimable? no
>> 1 Time(s): Oct 29 14:23:52 lb1-2 kernel: [ 997.512583] lowmem_reserve[]:
>> 0 0 880 3055
>> 1 Time(s): Oct 29 14:23:53 lb1-2 kernel: [ 997.512677] Normal
>> free:1376kB min:3756kB low:4692kB high:5632kB active:112kB
>> inactive:180kB present:901120kB pages_scanned:265 all_unreclaimable? no
>> 1 Time(s): Oct 29 14:23:53 lb1-2 kernel: [ 997.512850] lowmem_reserve[]:
>> 0 0 0 17400
>> 1 Time(s): Oct 29 14:23:53 lb1-2 kernel: [ 997.512943] HighMem
>> free:2137456kB min:512kB low:2832kB high:5156kB active:70224kB
>> inactive:10464kB present:2227200kB pages_scanned:0 all_unreclaimable? no
>> 1 Time(s): Oct 29 14:23:53 lb1-2 kernel: [ 997.513136] lowmem_reserve[]:
>> 0 0 0 0
>> 1 Time(s): Oct 29 14:23:53 lb1-2 kernel: [ 997.513228] DMA: 1*4kB 1*8kB
>> 1*16kB 0*32kB 1*64kB 1*128kB 1*256kB 0*512kB 1*1024kB 1*2048kB 0*4096kB
>> = 3548kB
>> 1 Time(s): Oct 29 14:23:53 lb1-2 kernel: [ 997.513417] DMA32: empty
>> 1 Time(s): Oct 29 14:23:53 lb1-2 kernel: [ 997.513498] Normal: 0*4kB
>> 4*8kB 0*16kB 2*32kB 0*64kB 0*128kB 1*256kB 0*512kB 1*1024kB 0*2048kB
>> 0*4096kB = 1376kB
>> 1 Time(s): Oct 29 14:23:53 lb1-2 kernel: [ 997.513686] HighMem: 182*4kB
>> 59*8kB 78*16kB 105*32kB 39*64kB 14*128kB 2*256kB 2*512kB 0*1024kB
>> 0*2048kB 519*4096kB = 2137456kB
>> 1 Time(s): Oct 29 14:23:53 lb1-2 kernel: [ 997.513878] Swap cache: add
>> 0, delete 0, find 0/0, race 0+0
>> 1 Time(s): Oct 29 14:23:53 lb1-2 kernel: [ 997.513964] Free swap = 0kB
>> 1 Time(s): Oct 29 14:23:54 lb1-2 kernel: [ 997.514045] Total swap = 0kB
>> 1 Time(s): Oct 29 14:23:54 lb1-2 kernel: [ 997.514163] Free swap: 0kB
>> 1 Time(s): Oct 29 14:23:54 lb1-2 kernel: [ 997.520251] 786176 pages of RAM
>> 1 Time(s): Oct 29 14:23:54 lb1-2 kernel: [ 997.520253] 556800 pages of
>> HIGHMEM
>> 1 Time(s): Oct 29 14:23:54 lb1-2 kernel: [ 997.520254] 8061 reserved pages
>> 1 Time(s): Oct 29 14:23:54 lb1-2 kernel: [ 997.520255] 26709 pages shared
>> 1 Time(s): Oct 29 14:23:54 lb1-2 kernel: [ 997.520256] 0 pages swap cached
>> 1 Time(s): Oct 29 14:23:54 lb1-2 kernel: [ 997.520257] 0 pages dirty
>> 1 Time(s): Oct 29 14:23:54 lb1-2 kernel: [ 997.520258] 38 pages writeback
>> 1 Time(s): Oct 29 14:23:54 lb1-2 kernel: [ 997.520259] 2869 pages mapped
>> 1 Time(s): Oct 29 14:23:54 lb1-2 kernel: [ 997.520260] 219589 pages slab
>> 1 Time(s): Oct 29 14:23:54 lb1-2 kernel: [ 997.520261] 596 pages pagetables
>> 1 Time(s): Oct 29 14:23:54 lb1-2 kernel: [ 997.520298] Out of memory:
>> Killed process 2291 (bash).
>>
>> Maybe, there is missconfiguration kernel?
>>
>> _______________________________________________
>> Please read the documentation before posting - it's available at:
>> http://www.linuxvirtualserver.org/
>>
>> LinuxVirtualServer.org mailing list - lvs-users[at]LinuxVirtualServer.org
>> Send requests to lvs-users-request[at]LinuxVirtualServer.org
>> or go to http://lists.graemef.net/mailman/listinfo/lvs-users
>>
>
> _______________________________________________
> Please read the documentation before posting - it's available at:
> http://www.linuxvirtualserver.org/
>
> LinuxVirtualServer.org mailing list - lvs-users[at]LinuxVirtualServer.org
> Send requests to lvs-users-request[at]LinuxVirtualServer.org
> or go to http://lists.graemef.net/mailman/listinfo/lvs-users
>
>
>
>
>

_______________________________________________
Please read the documentation before posting - it's available at:
http://www.linuxvirtualserver.org/

LinuxVirtualServer.org mailing list - lvs-users[at]LinuxVirtualServer.org
Send requests to lvs-users-request[at]LinuxVirtualServer.org
or go to http://lists.graemef.net/mailman/listinfo/lvs-users

Index | Next | Previous | View Threaded

Interested in having your list archived? Contact lists@gossamer-threads.com