Login | Register For Free | Help
Search for: (Advanced)

Mailing List Archive: Netapp: toasters

High CPU utilisation

 

 

Netapp toasters RSS feed   Index | Next | Previous | View Threaded


MDAVIES at generalatlantic

Nov 6, 2008, 1:09 AM

Post #1 of 7 (11095 views)
Permalink
High CPU utilisation

Hi all,

We are having a problem with a FAS270 that appears to be having high cpu utilisation issues.

To be honest I have looked at sysstat and statit and I cannot see the problem. I would appreciate if someone could take a look at the below outputs and let me know what I'm missing.

Thanks

Matt

 ukfiler*> sysstat -i 3
 CPU    NFS   CIFS  iSCSI      Net kB/s     Disk kB/s     iSCSI  kB/s    Cache
                               in   out     read  write      in   out      age
 85%     18      0     26     203    13      661      0      91     0      24
 68%      9      1     50     283    23      692      0     221     9      24
 92%      7      4     25     144    11     1643   2443      90     3      24
 78%     12      1     74     345    21      618      0     198     0      24
 67%      8      0     45     288    18      722      0     205     5      24
 96%     13      1     32     236    14     1365   1708     151     1      24
 81%      8      4     39     222    16      843    867     167     4      24
 85%      8      0     33     443    10     1212   1456     118     0      24
 99%     16      2     43     370    21      912      0     168     4      24
 99%     14     16     23     219    26     1023      0      98     0      24
 99%     12      0     36     585    39     2452   3781     465    18      24
ukfiler*>

Hostname: ukfiler  ID: 0084264545  Memory: 1022 MB
  NetApp Release 7.2.3: Thu Jul  5 10:06:16 PDT 2007
    <6O>
  Start time: Wed Nov  5 22:45:03 GMT 2008

                       CPU Statistics
      64.822501 time (seconds)       100 %
      52.221602 system time           81 %
       0.511694 rupt time              1 %   (161581 rupts x 3 usec/rupt)
      51.709908 non-rupt system time  80 %
      77.423400 idle time            119 %

       4.011144 time in CP             6 %   100 %
       0.052061 rupt time in CP                1 %   (12144 rupts x 4 usec/rupt)

                       Multiprocessor Statistics (per second)
                          cpu0       cpu1      total
sk switches            6581.74    7585.82   14167.57
hard switches          1485.87    1560.68    3046.55
domain switches          72.75      65.01     137.76
CP rupts                125.45      61.89     187.34
nonCP rupts            1350.03     955.30    2305.33
IPI rupts                18.23      17.32      35.56

CP rupt usec            565.04     238.10     803.13
nonCP rupt usec        4342.69    2747.95    7090.64
idle                 632999.86  561391.13 1194390.83
kahuna               334624.38  409797.24  744421.77
network                6697.98    8506.20   15204.18
storage                4193.90    5622.94    9816.84
exempt                 3941.24    4036.10    7977.34
raid                   7872.95    4627.33   12500.27
target                 4409.87    2761.70    7171.57
netcache                  0.00       0.00       0.00
netcache2                 0.00       0.00       0.00
cifs                    352.07     271.33     623.39
wafl_exempt               0.00       0.00       0.00

      49.342358 seconds with one or more CPUs active   ( 76%)

      46.750469 seconds with one CPU active            ( 72%)
       2.591888 seconds with both CPUs active          (  4%)

                       Domain Utilization By Exempt (per second)
      0.00 idle                              0.00 kahuna
      0.00 network                           0.00 storage
      0.00 exempt                            0.00 raid
      0.00 target                            0.00 netcache
      0.00 netcache2                         0.00 cifs
      0.00 wafl_exempt


                       CSMP Domain Switches (per second)
   From\To       idle     kahuna    network    storage     exempt       raid     target   netcache  netcache2       cifs wafl_exem
pt
      idle       0.00       0.00       0.00       0.00       0.00       0.00       0.00       0.00       0.00       0.00       0.0
0
    kahuna       0.00       0.00      27.09       2.65       0.00       5.32       1.87       0.00       0.00      18.79       0.0
0
   network       0.00      27.09       0.00       0.00       0.00       0.00       0.00       0.00       0.00       0.20       0.0
0
   storage       0.00       2.65       0.00       0.00       0.00      12.96       0.00       0.00       0.00       0.00       0.0
0
    exempt       0.00       0.00       0.00       0.00       0.00       0.00       0.00       0.00       0.00       0.00       0.0
0
      raid       0.00       5.32       0.00      12.96       0.00       0.00       0.00       0.00       0.00       0.00       0.0
0
    target       0.00       1.87       0.00       0.00       0.00       0.00       0.00       0.00       0.00       0.00       0.0
0
  netcache       0.00       0.00       0.00       0.00       0.00       0.00       0.00       0.00       0.00       0.00       0.0
0
 netcache2       0.00       0.00       0.00       0.00       0.00       0.00       0.00       0.00       0.00       0.00       0.0
0
      cifs       0.00      18.79       0.20       0.00       0.00       0.00       0.00       0.00       0.00       0.00       0.0
0
wafl_exempt       0.00       0.00       0.00       0.00       0.00       0.00       0.00       0.00       0.00       0.00       0.
00

                       Miscellaneous Statistics (per second)
   3046.55 hard context switches            10.38 NFS operations
      1.42 CIFS operations                   0.00 HTTP operations
      0.00 NetCache URLs                     0.00 streaming packets
   1240.42 network KB received             100.32 network KB transmitted
    655.39 disk KB read                   1146.02 disk KB written
    648.90 NVRAM KB written                  0.00 nolog KB written
      8.27 WAFL bufs given to clients        0.00 checksum cache hits  (   0%)
      1.23 no checksum - partial buffer      0.00 FCP operations
     74.48 iSCSI operations

                       WAFL Statistics (per second)
     15.94 name cache hits      (  95%)      0.86 name cache misses    (   5%)
 529973.33 buf hash hits        ( 100%)   1710.32 buf hash misses      (   0%)
    516.44 inode cache hits     ( 100%)      0.06 inode cache misses   (   0%)
 262496.95 buf cache hits       ( 100%)     19.65 buf cache misses     (   0%)
     30.17 blocks read                      67.00 blocks read-ahead
     58.10 chains read-ahead                 0.99 dummy reads
     14.25 blocks speculative read-ahead    226.31 blocks written
      2.27 stripes written                   0.00 blocks over-written
      0.09 wafl_timer generated CP           0.00 snapshot generated CP
      0.00 wafl_avail_bufs generated CP      0.00 dirty_blk_cnt generated CP
      0.00 full NV-log generated CP          0.00 back-to-back CP
      0.00 flush generated CP                0.00 sync generated CP
      0.00 wafl_avail_vbufs generated CP      0.00 deferred back-to-back CP
      0.00 low datavecs generated CP       304.72 non-restart messages
      4.98 IOWAIT suspends                 133872 buffers

                       RAID Statistics (per second)
     40.91 xors                              0.00 long dispatches [0]
      0.00 long consumed [0]                 0.00 long consumed hipri [0]
      0.00 long low priority [0]             0.00 long high priority [0]
      0.00 long monitor tics [0]             0.00 long monitor clears [0]
      0.00 long dispatches [1]               0.00 long consumed [1]
      0.00 long consumed hipri [1]           0.00 long low priority [1]
      0.00 long high priority [1]            0.00 long monitor tics [1]
      0.00 long monitor clears [1]             18 max batch
      2.11 blocked mode xor                 19.68 timed mode xor
      0.05 fast adjustments                  0.05 slow adjustments
         0 avg batch start                      0 avg stripe/msec
      2.64 tetrises written                  0.00 master tetrises
      0.00 slave tetrises                   29.70 stripes written
     11.22 partial stripes                  18.48 full stripes
    226.68 blocks written                   31.56 blocks read
      1.13 1 blocks per stripe size 6        0.29 2 blocks per stripe size 6
      0.17 3 blocks per stripe size 6        0.35 4 blocks per stripe size 6
      1.80 5 blocks per stripe size 6       15.63 6 blocks per stripe size 6
      1.11 1 blocks per stripe size 16       0.31 2 blocks per stripe size 16
      0.32 3 blocks per stripe size 16       0.19 4 blocks per stripe size 16
      0.09 5 blocks per stripe size 16       0.20 6 blocks per stripe size 16
      0.05 7 blocks per stripe size 16       0.20 8 blocks per stripe size 16
      0.12 9 blocks per stripe size 16       0.23 10 blocks per stripe size 16
      0.23 11 blocks per stripe size 16      0.56 12 blocks per stripe size 16
      0.83 13 blocks per stripe size 16      1.26 14 blocks per stripe size 16
      1.76 15 blocks per stripe size 16      2.85 16 blocks per stripe size 16

                       Network Interface Statistics (per second)
iface    side      bytes    packets multicasts     errors collisions  pkt drops
e0a      recv  114768.96      92.82       0.00       0.00                  0.00
         xmit     193.85       1.43       0.05       0.00       0.00
e0b      recv  521347.10     383.35       0.00       0.00                  0.00
         xmit   53015.11     280.33       0.14       0.00       0.00
vh       recv       0.00       0.00       0.00       0.00                  0.00
         xmit       0.00       0.00       0.00       0.00       0.00
mtrunk1  recv  636116.06     476.18       0.00       0.00                  0.00
         xmit   53208.96     281.77       0.25       0.00       0.00
mtrunk1-1 recv     588.41       5.26       2.10       0.00                  0.00
         xmit     665.54       2.92       0.08       0.00       0.00
mtrunk1-21 recv  261620.02     204.87       0.00       0.00                  0.00
         xmit   22000.14     120.08       0.05       0.00       0.00
mtrunk1-22 recv       0.00       0.00       0.00       0.00                  0.00
         xmit       5.28       0.05       0.05       0.00       0.00
mtrunk1-23 recv  371871.00     264.20       0.02       0.00                  0.00
         xmit   26855.71     156.13       0.05       0.00       0.00

                       Disk Statistics (per second)
        ut% is the percent of time the disk was busy.
        xfers is the number of data-transfer commands issued per second.
        xfers = ureads + writes + cpreads + greads + gwrites
        chain is the average number of 4K blocks per command.
        usecs is the average disk round-trip time per 4K block.

disk             ut%  xfers  ureads--chain-usecs writes--chain-usecs cpreads-chain-usecs greads--chain-usecs gwrites-chain-usecs
/aggr2/plex0/rg0:
0b.45              1   1.97    0.19   1.00 12333   1.10   9.42   988   0.69   5.07   768   0.00   ....     .   0.00   ....     .
0b.44              2   2.16    0.19   1.00 38167   1.28   8.35   925   0.69   5.07   838   0.00   ....     .   0.00   ....     .
0b.43              5   6.03    4.29   1.04 14107   1.13   6.84  1489   0.62   2.85  2430   0.00   ....     .   0.00   ....     .
0b.42              4   5.20    3.72   1.05 15565   0.88   8.00  1197   0.60   5.03   939   0.00   ....     .   0.00   ....     .
0b.41              4   5.09    3.76   1.01 14919   0.88   8.98  1184   0.45   4.03  1325   0.00   ....     .   0.00   ....     .
0b.40              5   5.28    4.09   1.08 13686   0.74   9.42  1210   0.45   5.28  1477   0.00   ....     .   0.00   ....     .
0b.39              5   5.54    4.18   1.08 15908   0.83   9.63  1142   0.52   4.44  1788   0.00   ....     .   0.00   ....     .
0b.38              4   5.45    3.90   1.00 17528   0.91   8.51  1327   0.63   3.56  1349   0.00   ....     .   0.00   ....     .
0b.37              4   5.29    3.72   1.07 16442   0.97   7.95  1359   0.60   4.23  1509   0.00   ....     .   0.00   ....     .
0b.36              4   5.01    3.50   1.01 13113   0.89   8.66  1371   0.62   4.18  1413   0.00   ....     .   0.00   ....     .
0b.32              5   5.54    4.03   1.36 11093   0.91   8.22  1433   0.60   3.92  1255   0.00   ....     .   0.00   ....     .
0b.17              4   5.06    3.76   1.01 13752   0.83   9.11  1268   0.46   4.60  1254   0.00   ....     .   0.00   ....     .
0b.33              5   5.51    3.90   1.02 14456   0.88   7.89  1553   0.73   4.02  1280   0.00   ....     .   0.00   ....     .
0b.18              4   5.51    3.79   1.19 12478   0.97   7.76  1591   0.74   3.71  1472   0.00   ....     .   0.00   ....     .
0b.34              5   5.51    3.89   1.12 13188   0.89   8.95  1301   0.73   3.62  1653   0.00   ....     .   0.00   ....     .
0b.19              4   5.04    3.56   1.07 11947   0.83   8.94  1308   0.65   3.50  2238   0.00   ....     .   0.00   ....     .
0b.35              4   4.74    3.33   1.14 12020   0.80   9.21  1353   0.60   4.26  1416   0.00   ....     .   0.00   ....     .
0b.20              4   5.32    3.89   1.01 13341   0.76   9.27  1352   0.68   3.86  1729   0.00   ....     .   0.00   ....     .
/aggr1/plex0/rg0:
0b.24              2   2.39    0.19   1.00 13667   1.73  11.25   655   0.48   5.06  1140   0.00   ....     .   0.00   ....     .
0b.29              5   7.30    4.97   1.26 17742   1.71  10.48   666   0.62   3.45  1543   0.00   ....     .   0.00   ....     .
0b.26              4   6.34    4.37   1.28 13500   1.42  12.46   595   0.56   3.97  1427   0.00   ....     .   0.00   ....     .
0b.28              5   7.05    4.95   1.14 19218   1.50  11.82   695   0.60   3.44  1642   0.00   ....     .   0.00   ....     .
0b.27              7   6.20    4.26   1.05 19422   1.45  12.28   659   0.49   4.56  1000   0.00   ....     .   0.00   ....     .
0b.22              5   6.90    4.81   1.21 18814   1.53  11.51   835   0.56   3.81  1591   0.00   ....     .   0.00   ....     .
0b.21              4   6.57    4.37   1.11 18990   1.51  11.73   843   0.69   3.80  1655   0.00   ....     .   0.00   ....     .
0b.23              2   2.36    0.19   1.00  9667   1.70  11.42   677   0.48   5.06  1178   0.00   ....     .   0.00   ....     .

Aggregate statistics:
Minimum            1   1.97    0.19                0.74                0.45                0.00                0.00
Mean               4   5.15    3.44                1.11                0.59                0.00                0.00
Maximum            7   7.30    4.97                1.73                0.74                0.00                0.00

Spares and other disks:
0b.25              0   0.00    0.00   ....     .   0.00   ....     .   0.00   ....     .   0.00   ....     .   0.00   ....     .

Spares and other disks:
0b.16              0   0.00    0.00   ....     .   0.00   ....     .   0.00   ....     .   0.00   ....     .   0.00   ....     .

                       FCP Statistics (per second)
      0.00 FCP Bytes recv                    0.00 FCP Bytes sent
      0.00 FCP ops

                       iSCSI Statistics (per second)
 516537.68 iSCSI Bytes recv              22866.13 iSCSI Bytes xmit
     74.48 iSCSI ops

                       Interrupt Statistics (per second)
   1604.06 Clock (IRQ 5)                    35.51 IPI (IRQ 8)
     35.51 IPI (IRQ 9)                     160.84 FCAL (IRQ 23)
    259.37 SB MAC (IRQ 25)                  41.22 SB MAC (IRQ 26)
     35.56 IPI                            2172.07 total

                       Data Mover Statistics (per second)
    627.05 total dma transfer KB           408.55 dma descriptors
     82.58 dma desc wait count             753.86 total dma transfer KB
    523.52 dma descriptors                 112.63 dma desc wait count
      0.00 total dma transfer KB             0.00 dma descriptors
      0.00 dma desc wait count               0.00 total dma transfer KB
      0.00 dma descriptors                   0.00 dma desc wait count

____________________________
Matt Davies
Director of International IT Operations
General Atlantic
83 Pall Mall
London
SW1Y 5ES

Tel: +44 207 484 3203
Fax: +44 207 484 2803
Mobile: +44 777 559 4265
____________________________




_____________________________________________________________
This e-mail (including all attachments) is confidential and may be privileged.
It is for the exclusive use of the addressee only. If you are not the addressee,
you are hereby notified that any dissemination of this communication is strictly
prohibited. If you have received this communication in error, please erase all
copies of the message and its attachments and notify us immediately at
help [at] generalatlantic <mailto:help [at] generalatlantic>. Thank You.


kheal at hotmail

Nov 6, 2008, 1:41 AM

Post #2 of 7 (10902 views)
Permalink
RE: High CPU utilisation [In reply to]

Hi Matt

I have seen several issues with FAS270s and high cpu utilisation related to several bugs in pre-7.2.4 releases, especially with the filer sat in background processes, cpu spikes and nfs perf issues. Troubleshooting these bugs is a pain, and the solution is to upgrade anyway, so I would be inclined to upgrade to 7.2.6 first and see if the issue still persists.

cheers
Kenneth

> Subject: High CPU utilisation
> Date: Thu, 6 Nov 2008 09:09:41 +0000
> From: MDAVIES [at] generalatlantic
> To: toasters [at] mathworks
>
> Hi all,
>
> We are having a problem with a FAS270 that appears to be having high cpu utilisation issues.
>
> To be honest I have looked at sysstat and statit and I cannot see the problem. I would appreciate if someone could take a look at the below outputs and let me know what I'm missing.
>
> Thanks
>
> Matt

_________________________________________________________________
Express yourself instantly with MSN Messenger! Download today it's FREE!
http://messenger.msn.click-url.com/go/onm00200471ave/direct/01/


MDAVIES at generalatlantic

Nov 6, 2008, 1:41 AM

Post #3 of 7 (11127 views)
Permalink
RE: High CPU utilisation [In reply to]

Nils,

1, yes it is both a source and destination for snapmirror, there is no
snapvault

2, see below.

ukfiler*> wafl scan status
Volume CIFS_UK:
Scan id Type of scan progress
3 active bitmap rearrangement fbn 1605 of 2230 w/
max_chain_len 3
191730 container block reclamation block 2042 of 2230
Volume root:
Scan id Type of scan progress
5 active bitmap rearrangement fbn 63 of 107 w/
max_chain_len 3
191636 container block reclamation block 64 of 107
Volume UKEXCHDB:
Scan id Type of scan progress
8 active bitmap rearrangement fbn 138 of 713 w/
max_chain_len 3
Volume UKEXCHLOGS:
Scan id Type of scan progress
10 active bitmap rearrangement fbn 369 of 401 w/
max_chain_len 3
191608 container block reclamation block 391 of 401
Volume UKVMWARE:
Scan id Type of scan progress
12 active bitmap rearrangement fbn 1538 of 4461 w/
max_chain_len 3
Volume DUEXCHLOGS:
Scan id Type of scan progress
191713 container block reclamation block 879 of 892
Volume UKVIRTUALCENTRE:
Scan id Type of scan progress
16 active bitmap rearrangement fbn 197 of 1115 w/
max_chain_len 3
Volume UKVMWARE1:
Scan id Type of scan progress
18 active bitmap rearrangement fbn 4215 of 4461 w/
max_chain_len 3
Volume UKVMWARENFS:
Scan id Type of scan progress
21 active bitmap rearrangement fbn 1019 of 1784 w/
max_chain_len 3
Volume TESTUKEXCHLOGS:
Scan id Type of scan progress
191708 container block reclamation block 393 of 401
Volume UKEXCH3DB:
Scan id Type of scan progress
152073 active bitmap rearrangement fbn 548 of 571 w/
max_chain_len 3
191729 container block reclamation block 356 of 571
Volume vol1:
Scan id Type of scan progress
191630 container block reclamation block 0 of 0
Volume UKEXCH3LOGS:
Scan id Type of scan progress
152075 active bitmap rearrangement fbn 29 of 321 w/
max_chain_len 3
191728 container block reclamation block 195 of 321

Thanks

Matt

-----Original Message-----
From: Nils Vogels [mailto:bacardicoke [at] gmail]
Sent: 06 November 2008 09:29
To: Davies,Matt
Subject: Re: High CPU utilisation

Couple of questions ..

1) Do you have snapvault/snapmirror running?
2) What does "wafl scan status" tell you?

Greetings,

Nils

On Thu, Nov 6, 2008 at 10:09 AM, Davies,Matt
<MDAVIES [at] generalatlantic> wrote:
> Hi all,
>
> We are having a problem with a FAS270 that appears to be having high
cpu utilisation issues.
>
> To be honest I have looked at sysstat and statit and I cannot see the
problem. I would appreciate if someone could take a look at the below
outputs and let me know what I'm missing.
>
> Thanks
>
> Matt
>
> ukfiler*> sysstat -i 3
> CPU NFS CIFS iSCSI Net kB/s Disk kB/s iSCSI kB/s
Cache
> in out read write in out
age
> 85% 18 0 26 203 13 661 0 91 0
24
> 68% 9 1 50 283 23 692 0 221 9
24
> 92% 7 4 25 144 11 1643 2443 90 3
24
> 78% 12 1 74 345 21 618 0 198 0
24
> 67% 8 0 45 288 18 722 0 205 5
24
> 96% 13 1 32 236 14 1365 1708 151 1
24
> 81% 8 4 39 222 16 843 867 167 4
24
> 85% 8 0 33 443 10 1212 1456 118 0
24
> 99% 16 2 43 370 21 912 0 168 4
24
> 99% 14 16 23 219 26 1023 0 98 0
24
> 99% 12 0 36 585 39 2452 3781 465 18
24
> ukfiler*>
>
> Hostname: ukfiler ID: 0084264545 Memory: 1022 MB
> NetApp Release 7.2.3: Thu Jul 5 10:06:16 PDT 2007
> <6O>
> Start time: Wed Nov 5 22:45:03 GMT 2008
>
> CPU Statistics
> 64.822501 time (seconds) 100 %
> 52.221602 system time 81 %
> 0.511694 rupt time 1 % (161581 rupts x 3
usec/rupt)
> 51.709908 non-rupt system time 80 %
> 77.423400 idle time 119 %
>
> 4.011144 time in CP 6 % 100 %
> 0.052061 rupt time in CP 1 % (12144 rupts x 4
usec/rupt)
>
> Multiprocessor Statistics (per second)
> cpu0 cpu1 total
> sk switches 6581.74 7585.82 14167.57
> hard switches 1485.87 1560.68 3046.55
> domain switches 72.75 65.01 137.76
> CP rupts 125.45 61.89 187.34
> nonCP rupts 1350.03 955.30 2305.33
> IPI rupts 18.23 17.32 35.56
>
> CP rupt usec 565.04 238.10 803.13
> nonCP rupt usec 4342.69 2747.95 7090.64
> idle 632999.86 561391.13 1194390.83
> kahuna 334624.38 409797.24 744421.77
> network 6697.98 8506.20 15204.18
> storage 4193.90 5622.94 9816.84
> exempt 3941.24 4036.10 7977.34
> raid 7872.95 4627.33 12500.27
> target 4409.87 2761.70 7171.57
> netcache 0.00 0.00 0.00
> netcache2 0.00 0.00 0.00
> cifs 352.07 271.33 623.39
> wafl_exempt 0.00 0.00 0.00
>
> 49.342358 seconds with one or more CPUs active ( 76%)
>
> 46.750469 seconds with one CPU active ( 72%)
> 2.591888 seconds with both CPUs active ( 4%)
>
> Domain Utilization By Exempt (per second)
> 0.00 idle 0.00 kahuna
> 0.00 network 0.00 storage
> 0.00 exempt 0.00 raid
> 0.00 target 0.00 netcache
> 0.00 netcache2 0.00 cifs
> 0.00 wafl_exempt
>
>
> CSMP Domain Switches (per second)
> From\To idle kahuna network storage exempt
raid target netcache netcache2 cifs wafl_exem
> pt
> idle 0.00 0.00 0.00 0.00 0.00
0.00 0.00 0.00 0.00 0.00 0.0
> 0
> kahuna 0.00 0.00 27.09 2.65 0.00
5.32 1.87 0.00 0.00 18.79 0.0
> 0
> network 0.00 27.09 0.00 0.00 0.00
0.00 0.00 0.00 0.00 0.20 0.0
> 0
> storage 0.00 2.65 0.00 0.00 0.00
12.96 0.00 0.00 0.00 0.00 0.0
> 0
> exempt 0.00 0.00 0.00 0.00 0.00
0.00 0.00 0.00 0.00 0.00 0.0
> 0
> raid 0.00 5.32 0.00 12.96 0.00
0.00 0.00 0.00 0.00 0.00 0.0
> 0
> target 0.00 1.87 0.00 0.00 0.00
0.00 0.00 0.00 0.00 0.00 0.0
> 0
> netcache 0.00 0.00 0.00 0.00 0.00
0.00 0.00 0.00 0.00 0.00 0.0
> 0
> netcache2 0.00 0.00 0.00 0.00 0.00
0.00 0.00 0.00 0.00 0.00 0.0
> 0
> cifs 0.00 18.79 0.20 0.00 0.00
0.00 0.00 0.00 0.00 0.00 0.0
> 0
> wafl_exempt 0.00 0.00 0.00 0.00 0.00
0.00 0.00 0.00 0.00 0.00 0.
> 00
>
> Miscellaneous Statistics (per second)
> 3046.55 hard context switches 10.38 NFS operations
> 1.42 CIFS operations 0.00 HTTP operations
> 0.00 NetCache URLs 0.00 streaming packets
> 1240.42 network KB received 100.32 network KB
transmitted
> 655.39 disk KB read 1146.02 disk KB written
> 648.90 NVRAM KB written 0.00 nolog KB written
> 8.27 WAFL bufs given to clients 0.00 checksum cache hits
( 0%)
> 1.23 no checksum - partial buffer 0.00 FCP operations
> 74.48 iSCSI operations
>
> WAFL Statistics (per second)
> 15.94 name cache hits ( 95%) 0.86 name cache misses
( 5%)
> 529973.33 buf hash hits ( 100%) 1710.32 buf hash misses
( 0%)
> 516.44 inode cache hits ( 100%) 0.06 inode cache misses
( 0%)
> 262496.95 buf cache hits ( 100%) 19.65 buf cache misses
( 0%)
> 30.17 blocks read 67.00 blocks read-ahead
> 58.10 chains read-ahead 0.99 dummy reads
> 14.25 blocks speculative read-ahead 226.31 blocks written
> 2.27 stripes written 0.00 blocks over-written
> 0.09 wafl_timer generated CP 0.00 snapshot generated
CP
> 0.00 wafl_avail_bufs generated CP 0.00 dirty_blk_cnt
generated CP
> 0.00 full NV-log generated CP 0.00 back-to-back CP
> 0.00 flush generated CP 0.00 sync generated CP
> 0.00 wafl_avail_vbufs generated CP 0.00 deferred
back-to-back CP
> 0.00 low datavecs generated CP 304.72 non-restart messages
> 4.98 IOWAIT suspends 133872 buffers
>
> RAID Statistics (per second)
> 40.91 xors 0.00 long dispatches [0]
> 0.00 long consumed [0] 0.00 long consumed hipri
[0]
> 0.00 long low priority [0] 0.00 long high priority
[0]
> 0.00 long monitor tics [0] 0.00 long monitor clears
[0]
> 0.00 long dispatches [1] 0.00 long consumed [1]
> 0.00 long consumed hipri [1] 0.00 long low priority
[1]
> 0.00 long high priority [1] 0.00 long monitor tics
[1]
> 0.00 long monitor clears [1] 18 max batch
> 2.11 blocked mode xor 19.68 timed mode xor
> 0.05 fast adjustments 0.05 slow adjustments
> 0 avg batch start 0 avg stripe/msec
> 2.64 tetrises written 0.00 master tetrises
> 0.00 slave tetrises 29.70 stripes written
> 11.22 partial stripes 18.48 full stripes
> 226.68 blocks written 31.56 blocks read
> 1.13 1 blocks per stripe size 6 0.29 2 blocks per stripe
size 6
> 0.17 3 blocks per stripe size 6 0.35 4 blocks per stripe
size 6
> 1.80 5 blocks per stripe size 6 15.63 6 blocks per stripe
size 6
> 1.11 1 blocks per stripe size 16 0.31 2 blocks per stripe
size 16
> 0.32 3 blocks per stripe size 16 0.19 4 blocks per stripe
size 16
> 0.09 5 blocks per stripe size 16 0.20 6 blocks per stripe
size 16
> 0.05 7 blocks per stripe size 16 0.20 8 blocks per stripe
size 16
> 0.12 9 blocks per stripe size 16 0.23 10 blocks per stripe
size 16
> 0.23 11 blocks per stripe size 16 0.56 12 blocks per stripe
size 16
> 0.83 13 blocks per stripe size 16 1.26 14 blocks per stripe
size 16
> 1.76 15 blocks per stripe size 16 2.85 16 blocks per stripe
size 16
>
> Network Interface Statistics (per second)
> iface side bytes packets multicasts errors collisions
pkt drops
> e0a recv 114768.96 92.82 0.00 0.00
0.00
> xmit 193.85 1.43 0.05 0.00 0.00
> e0b recv 521347.10 383.35 0.00 0.00
0.00
> xmit 53015.11 280.33 0.14 0.00 0.00
> vh recv 0.00 0.00 0.00 0.00
0.00
> xmit 0.00 0.00 0.00 0.00 0.00
> mtrunk1 recv 636116.06 476.18 0.00 0.00
0.00
> xmit 53208.96 281.77 0.25 0.00 0.00
> mtrunk1-1 recv 588.41 5.26 2.10 0.00
0.00
> xmit 665.54 2.92 0.08 0.00 0.00
> mtrunk1-21 recv 261620.02 204.87 0.00 0.00
0.00
> xmit 22000.14 120.08 0.05 0.00 0.00
> mtrunk1-22 recv 0.00 0.00 0.00 0.00
0.00
> xmit 5.28 0.05 0.05 0.00 0.00
> mtrunk1-23 recv 371871.00 264.20 0.02 0.00
0.00
> xmit 26855.71 156.13 0.05 0.00 0.00
>
> Disk Statistics (per second)
> ut% is the percent of time the disk was busy.
> xfers is the number of data-transfer commands issued per
second.
> xfers = ureads + writes + cpreads + greads + gwrites
> chain is the average number of 4K blocks per command.
> usecs is the average disk round-trip time per 4K block.
>
> disk ut% xfers ureads--chain-usecs writes--chain-usecs
cpreads-chain-usecs greads--chain-usecs gwrites-chain-usecs
> /aggr2/plex0/rg0:
> 0b.45 1 1.97 0.19 1.00 12333 1.10 9.42 988
0.69 5.07 768 0.00 .... . 0.00 .... .
> 0b.44 2 2.16 0.19 1.00 38167 1.28 8.35 925
0.69 5.07 838 0.00 .... . 0.00 .... .
> 0b.43 5 6.03 4.29 1.04 14107 1.13 6.84 1489
0.62 2.85 2430 0.00 .... . 0.00 .... .
> 0b.42 4 5.20 3.72 1.05 15565 0.88 8.00 1197
0.60 5.03 939 0.00 .... . 0.00 .... .
> 0b.41 4 5.09 3.76 1.01 14919 0.88 8.98 1184
0.45 4.03 1325 0.00 .... . 0.00 .... .
> 0b.40 5 5.28 4.09 1.08 13686 0.74 9.42 1210
0.45 5.28 1477 0.00 .... . 0.00 .... .
> 0b.39 5 5.54 4.18 1.08 15908 0.83 9.63 1142
0.52 4.44 1788 0.00 .... . 0.00 .... .
> 0b.38 4 5.45 3.90 1.00 17528 0.91 8.51 1327
0.63 3.56 1349 0.00 .... . 0.00 .... .
> 0b.37 4 5.29 3.72 1.07 16442 0.97 7.95 1359
0.60 4.23 1509 0.00 .... . 0.00 .... .
> 0b.36 4 5.01 3.50 1.01 13113 0.89 8.66 1371
0.62 4.18 1413 0.00 .... . 0.00 .... .
> 0b.32 5 5.54 4.03 1.36 11093 0.91 8.22 1433
0.60 3.92 1255 0.00 .... . 0.00 .... .
> 0b.17 4 5.06 3.76 1.01 13752 0.83 9.11 1268
0.46 4.60 1254 0.00 .... . 0.00 .... .
> 0b.33 5 5.51 3.90 1.02 14456 0.88 7.89 1553
0.73 4.02 1280 0.00 .... . 0.00 .... .
> 0b.18 4 5.51 3.79 1.19 12478 0.97 7.76 1591
0.74 3.71 1472 0.00 .... . 0.00 .... .
> 0b.34 5 5.51 3.89 1.12 13188 0.89 8.95 1301
0.73 3.62 1653 0.00 .... . 0.00 .... .
> 0b.19 4 5.04 3.56 1.07 11947 0.83 8.94 1308
0.65 3.50 2238 0.00 .... . 0.00 .... .
> 0b.35 4 4.74 3.33 1.14 12020 0.80 9.21 1353
0.60 4.26 1416 0.00 .... . 0.00 .... .
> 0b.20 4 5.32 3.89 1.01 13341 0.76 9.27 1352
0.68 3.86 1729 0.00 .... . 0.00 .... .
> /aggr1/plex0/rg0:
> 0b.24 2 2.39 0.19 1.00 13667 1.73 11.25 655
0.48 5.06 1140 0.00 .... . 0.00 .... .
> 0b.29 5 7.30 4.97 1.26 17742 1.71 10.48 666
0.62 3.45 1543 0.00 .... . 0.00 .... .
> 0b.26 4 6.34 4.37 1.28 13500 1.42 12.46 595
0.56 3.97 1427 0.00 .... . 0.00 .... .
> 0b.28 5 7.05 4.95 1.14 19218 1.50 11.82 695
0.60 3.44 1642 0.00 .... . 0.00 .... .
> 0b.27 7 6.20 4.26 1.05 19422 1.45 12.28 659
0.49 4.56 1000 0.00 .... . 0.00 .... .
> 0b.22 5 6.90 4.81 1.21 18814 1.53 11.51 835
0.56 3.81 1591 0.00 .... . 0.00 .... .
> 0b.21 4 6.57 4.37 1.11 18990 1.51 11.73 843
0.69 3.80 1655 0.00 .... . 0.00 .... .
> 0b.23 2 2.36 0.19 1.00 9667 1.70 11.42 677
0.48 5.06 1178 0.00 .... . 0.00 .... .
>
> Aggregate statistics:
> Minimum 1 1.97 0.19 0.74
0.45 0.00 0.00
> Mean 4 5.15 3.44 1.11
0.59 0.00 0.00
> Maximum 7 7.30 4.97 1.73
0.74 0.00 0.00
>
> Spares and other disks:
> 0b.25 0 0.00 0.00 .... . 0.00 .... .
0.00 .... . 0.00 .... . 0.00 .... .
>
> Spares and other disks:
> 0b.16 0 0.00 0.00 .... . 0.00 .... .
0.00 .... . 0.00 .... . 0.00 .... .
>
> FCP Statistics (per second)
> 0.00 FCP Bytes recv 0.00 FCP Bytes sent
> 0.00 FCP ops
>
> iSCSI Statistics (per second)
> 516537.68 iSCSI Bytes recv 22866.13 iSCSI Bytes xmit
> 74.48 iSCSI ops
>
> Interrupt Statistics (per second)
> 1604.06 Clock (IRQ 5) 35.51 IPI (IRQ 8)
> 35.51 IPI (IRQ 9) 160.84 FCAL (IRQ 23)
> 259.37 SB MAC (IRQ 25) 41.22 SB MAC (IRQ 26)
> 35.56 IPI 2172.07 total
>
> Data Mover Statistics (per second)
> 627.05 total dma transfer KB 408.55 dma descriptors
> 82.58 dma desc wait count 753.86 total dma transfer
KB
> 523.52 dma descriptors 112.63 dma desc wait count
> 0.00 total dma transfer KB 0.00 dma descriptors
> 0.00 dma desc wait count 0.00 total dma transfer
KB
> 0.00 dma descriptors 0.00 dma desc wait count
>
> ____________________________
> Matt Davies
> Director of International IT Operations
> General Atlantic
> 83 Pall Mall
> London
> SW1Y 5ES
>
> Tel: +44 207 484 3203
> Fax: +44 207 484 2803
> Mobile: +44 777 559 4265
> ____________________________
>
>
>
>
> _____________________________________________________________
> This e-mail (including all attachments) is confidential and may be
privileged.
> It is for the exclusive use of the addressee only. If you are not the
addressee,
> you are hereby notified that any dissemination of this communication
is strictly
> prohibited. If you have received this communication in error, please
erase all
> copies of the message and its attachments and notify us immediately at
> help [at] generalatlantic <mailto:help [at] generalatlantic>. Thank You.
>
>



--
Simple guidelines to happiness:
Work like you don't need the money,
Love like your heart has never been broken and
Dance like no one can see you.


Michael.Kappe at izb

Nov 6, 2008, 2:48 AM

Post #4 of 7 (10917 views)
Permalink
AW: High CPU utilisation [In reply to]

Hi,

if you have Snapmirror running - try "wafl scan status" (priv set advanced) - lock for deswizzling

https://now.netapp.com/Knowledgebase/solutionarea.asp?id=kb16746

http://now.netapp.com/NOW/cgi-bin/bol?Type=Detail&Display=263215

-----Ursprüngliche Nachricht-----
Von: owner-toasters [at] mathworks [mailto:owner-toasters [at] mathworks] Im Auftrag von Davies,Matt
Gesendet: Donnerstag, 6. November 2008 10:10
An: toasters [at] mathworks
Betreff: High CPU utilisation

Hi all,

We are having a problem with a FAS270 that appears to be having high cpu utilisation issues.

To be honest I have looked at sysstat and statit and I cannot see the problem. I would appreciate if someone could take a look at the below outputs and let me know what I'm missing.

Thanks

Matt

ukfiler*> sysstat -i 3
CPU NFS CIFS iSCSI Net kB/s Disk kB/s iSCSI kB/s Cache
in out read write in out age
85% 18 0 26 203 13 661 0 91 0 24
68% 9 1 50 283 23 692 0 221 9 24
92% 7 4 25 144 11 1643 2443 90 3 24
78% 12 1 74 345 21 618 0 198 0 24
67% 8 0 45 288 18 722 0 205 5 24
96% 13 1 32 236 14 1365 1708 151 1 24
81% 8 4 39 222 16 843 867 167 4 24
85% 8 0 33 443 10 1212 1456 118 0 24
99% 16 2 43 370 21 912 0 168 4 24
99% 14 16 23 219 26 1023 0 98 0 24
99% 12 0 36 585 39 2452 3781 465 18 24
ukfiler*>

Hostname: ukfiler ID: 0084264545 Memory: 1022 MB
NetApp Release 7.2.3: Thu Jul 5 10:06:16 PDT 2007
<6O>
Start time: Wed Nov 5 22:45:03 GMT 2008

CPU Statistics
64.822501 time (seconds) 100 %
52.221602 system time 81 %
0.511694 rupt time 1 % (161581 rupts x 3 usec/rupt)
51.709908 non-rupt system time 80 %
77.423400 idle time 119 %

4.011144 time in CP 6 % 100 %
0.052061 rupt time in CP 1 % (12144 rupts x 4 usec/rupt)

Multiprocessor Statistics (per second)
cpu0 cpu1 total
sk switches 6581.74 7585.82 14167.57
hard switches 1485.87 1560.68 3046.55
domain switches 72.75 65.01 137.76
CP rupts 125.45 61.89 187.34
nonCP rupts 1350.03 955.30 2305.33
IPI rupts 18.23 17.32 35.56

CP rupt usec 565.04 238.10 803.13
nonCP rupt usec 4342.69 2747.95 7090.64
idle 632999.86 561391.13 1194390.83
kahuna 334624.38 409797.24 744421.77
network 6697.98 8506.20 15204.18
storage 4193.90 5622.94 9816.84
exempt 3941.24 4036.10 7977.34
raid 7872.95 4627.33 12500.27
target 4409.87 2761.70 7171.57
netcache 0.00 0.00 0.00
netcache2 0.00 0.00 0.00
cifs 352.07 271.33 623.39
wafl_exempt 0.00 0.00 0.00

49.342358 seconds with one or more CPUs active ( 76%)

46.750469 seconds with one CPU active ( 72%)
2.591888 seconds with both CPUs active ( 4%)

Domain Utilization By Exempt (per second)
0.00 idle 0.00 kahuna
0.00 network 0.00 storage
0.00 exempt 0.00 raid
0.00 target 0.00 netcache
0.00 netcache2 0.00 cifs
0.00 wafl_exempt


CSMP Domain Switches (per second)
From\To idle kahuna network storage exempt raid target netcache netcache2 cifs wafl_exem
pt
idle 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.0
0
kahuna 0.00 0.00 27.09 2.65 0.00 5.32 1.87 0.00 0.00 18.79 0.0
0
network 0.00 27.09 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.20 0.0
0
storage 0.00 2.65 0.00 0.00 0.00 12.96 0.00 0.00 0.00 0.00 0.0
0
exempt 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.0
0
raid 0.00 5.32 0.00 12.96 0.00 0.00 0.00 0.00 0.00 0.00 0.0
0
target 0.00 1.87 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.0
0
netcache 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.0
0
netcache2 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.0
0
cifs 0.00 18.79 0.20 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.0
0
wafl_exempt 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.
00

Miscellaneous Statistics (per second)
3046.55 hard context switches 10.38 NFS operations
1.42 CIFS operations 0.00 HTTP operations
0.00 NetCache URLs 0.00 streaming packets
1240.42 network KB received 100.32 network KB transmitted
655.39 disk KB read 1146.02 disk KB written
648.90 NVRAM KB written 0.00 nolog KB written
8.27 WAFL bufs given to clients 0.00 checksum cache hits ( 0%)
1.23 no checksum - partial buffer 0.00 FCP operations
74.48 iSCSI operations

WAFL Statistics (per second)
15.94 name cache hits ( 95%) 0.86 name cache misses ( 5%)
529973.33 buf hash hits ( 100%) 1710.32 buf hash misses ( 0%)
516.44 inode cache hits ( 100%) 0.06 inode cache misses ( 0%)
262496.95 buf cache hits ( 100%) 19.65 buf cache misses ( 0%)
30.17 blocks read 67.00 blocks read-ahead
58.10 chains read-ahead 0.99 dummy reads
14.25 blocks speculative read-ahead 226.31 blocks written
2.27 stripes written 0.00 blocks over-written
0.09 wafl_timer generated CP 0.00 snapshot generated CP
0.00 wafl_avail_bufs generated CP 0.00 dirty_blk_cnt generated CP
0.00 full NV-log generated CP 0.00 back-to-back CP
0.00 flush generated CP 0.00 sync generated CP
0.00 wafl_avail_vbufs generated CP 0.00 deferred back-to-back CP
0.00 low datavecs generated CP 304.72 non-restart messages
4.98 IOWAIT suspends 133872 buffers

RAID Statistics (per second)
40.91 xors 0.00 long dispatches [0]
0.00 long consumed [0] 0.00 long consumed hipri [0]
0.00 long low priority [0] 0.00 long high priority [0]
0.00 long monitor tics [0] 0.00 long monitor clears [0]
0.00 long dispatches [1] 0.00 long consumed [1]
0.00 long consumed hipri [1] 0.00 long low priority [1]
0.00 long high priority [1] 0.00 long monitor tics [1]
0.00 long monitor clears [1] 18 max batch
2.11 blocked mode xor 19.68 timed mode xor
0.05 fast adjustments 0.05 slow adjustments
0 avg batch start 0 avg stripe/msec
2.64 tetrises written 0.00 master tetrises
0.00 slave tetrises 29.70 stripes written
11.22 partial stripes 18.48 full stripes
226.68 blocks written 31.56 blocks read
1.13 1 blocks per stripe size 6 0.29 2 blocks per stripe size 6
0.17 3 blocks per stripe size 6 0.35 4 blocks per stripe size 6
1.80 5 blocks per stripe size 6 15.63 6 blocks per stripe size 6
1.11 1 blocks per stripe size 16 0.31 2 blocks per stripe size 16
0.32 3 blocks per stripe size 16 0.19 4 blocks per stripe size 16
0.09 5 blocks per stripe size 16 0.20 6 blocks per stripe size 16
0.05 7 blocks per stripe size 16 0.20 8 blocks per stripe size 16
0.12 9 blocks per stripe size 16 0.23 10 blocks per stripe size 16
0.23 11 blocks per stripe size 16 0.56 12 blocks per stripe size 16
0.83 13 blocks per stripe size 16 1.26 14 blocks per stripe size 16
1.76 15 blocks per stripe size 16 2.85 16 blocks per stripe size 16

Network Interface Statistics (per second)
iface side bytes packets multicasts errors collisions pkt drops
e0a recv 114768.96 92.82 0.00 0.00 0.00
xmit 193.85 1.43 0.05 0.00 0.00
e0b recv 521347.10 383.35 0.00 0.00 0.00
xmit 53015.11 280.33 0.14 0.00 0.00
vh recv 0.00 0.00 0.00 0.00 0.00
xmit 0.00 0.00 0.00 0.00 0.00
mtrunk1 recv 636116.06 476.18 0.00 0.00 0.00
xmit 53208.96 281.77 0.25 0.00 0.00
mtrunk1-1 recv 588.41 5.26 2.10 0.00 0.00
xmit 665.54 2.92 0.08 0.00 0.00
mtrunk1-21 recv 261620.02 204.87 0.00 0.00 0.00
xmit 22000.14 120.08 0.05 0.00 0.00
mtrunk1-22 recv 0.00 0.00 0.00 0.00 0.00
xmit 5.28 0.05 0.05 0.00 0.00
mtrunk1-23 recv 371871.00 264.20 0.02 0.00 0.00
xmit 26855.71 156.13 0.05 0.00 0.00

Disk Statistics (per second)
ut% is the percent of time the disk was busy.
xfers is the number of data-transfer commands issued per second.
xfers = ureads + writes + cpreads + greads + gwrites
chain is the average number of 4K blocks per command.
usecs is the average disk round-trip time per 4K block.

disk ut% xfers ureads--chain-usecs writes--chain-usecs cpreads-chain-usecs greads--chain-usecs gwrites-chain-usecs
/aggr2/plex0/rg0:
0b.45 1 1.97 0.19 1.00 12333 1.10 9.42 988 0.69 5.07 768 0.00 .... . 0.00 .... .
0b.44 2 2.16 0.19 1.00 38167 1.28 8.35 925 0.69 5.07 838 0.00 .... . 0.00 .... .
0b.43 5 6.03 4.29 1.04 14107 1.13 6.84 1489 0.62 2.85 2430 0.00 .... . 0.00 .... .
0b.42 4 5.20 3.72 1.05 15565 0.88 8.00 1197 0.60 5.03 939 0.00 .... . 0.00 .... .
0b.41 4 5.09 3.76 1.01 14919 0.88 8.98 1184 0.45 4.03 1325 0.00 .... . 0.00 .... .
0b.40 5 5.28 4.09 1.08 13686 0.74 9.42 1210 0.45 5.28 1477 0.00 .... . 0.00 .... .
0b.39 5 5.54 4.18 1.08 15908 0.83 9.63 1142 0.52 4.44 1788 0.00 .... . 0.00 .... .
0b.38 4 5.45 3.90 1.00 17528 0.91 8.51 1327 0.63 3.56 1349 0.00 .... . 0.00 .... .
0b.37 4 5.29 3.72 1.07 16442 0.97 7.95 1359 0.60 4.23 1509 0.00 .... . 0.00 .... .
0b.36 4 5.01 3.50 1.01 13113 0.89 8.66 1371 0.62 4.18 1413 0.00 .... . 0.00 .... .
0b.32 5 5.54 4.03 1.36 11093 0.91 8.22 1433 0.60 3.92 1255 0.00 .... . 0.00 .... .
0b.17 4 5.06 3.76 1.01 13752 0.83 9.11 1268 0.46 4.60 1254 0.00 .... . 0.00 .... .
0b.33 5 5.51 3.90 1.02 14456 0.88 7.89 1553 0.73 4.02 1280 0.00 .... . 0.00 .... .
0b.18 4 5.51 3.79 1.19 12478 0.97 7.76 1591 0.74 3.71 1472 0.00 .... . 0.00 .... .
0b.34 5 5.51 3.89 1.12 13188 0.89 8.95 1301 0.73 3.62 1653 0.00 .... . 0.00 .... .
0b.19 4 5.04 3.56 1.07 11947 0.83 8.94 1308 0.65 3.50 2238 0.00 .... . 0.00 .... .
0b.35 4 4.74 3.33 1.14 12020 0.80 9.21 1353 0.60 4.26 1416 0.00 .... . 0.00 .... .
0b.20 4 5.32 3.89 1.01 13341 0.76 9.27 1352 0.68 3.86 1729 0.00 .... . 0.00 .... .
/aggr1/plex0/rg0:
0b.24 2 2.39 0.19 1.00 13667 1.73 11.25 655 0.48 5.06 1140 0.00 .... . 0.00 .... .
0b.29 5 7.30 4.97 1.26 17742 1.71 10.48 666 0.62 3.45 1543 0.00 .... . 0.00 .... .
0b.26 4 6.34 4.37 1.28 13500 1.42 12.46 595 0.56 3.97 1427 0.00 .... . 0.00 .... .
0b.28 5 7.05 4.95 1.14 19218 1.50 11.82 695 0.60 3.44 1642 0.00 .... . 0.00 .... .
0b.27 7 6.20 4.26 1.05 19422 1.45 12.28 659 0.49 4.56 1000 0.00 .... . 0.00 .... .
0b.22 5 6.90 4.81 1.21 18814 1.53 11.51 835 0.56 3.81 1591 0.00 .... . 0.00 .... .
0b.21 4 6.57 4.37 1.11 18990 1.51 11.73 843 0.69 3.80 1655 0.00 .... . 0.00 .... .
0b.23 2 2.36 0.19 1.00 9667 1.70 11.42 677 0.48 5.06 1178 0.00 .... . 0.00 .... .

Aggregate statistics:
Minimum 1 1.97 0.19 0.74 0.45 0.00 0.00
Mean 4 5.15 3.44 1.11 0.59 0.00 0.00
Maximum 7 7.30 4.97 1.73 0.74 0.00 0.00

Spares and other disks:
0b.25 0 0.00 0.00 .... . 0.00 .... . 0.00 .... . 0.00 .... . 0.00 .... .

Spares and other disks:
0b.16 0 0.00 0.00 .... . 0.00 .... . 0.00 .... . 0.00 .... . 0.00 .... .

FCP Statistics (per second)
0.00 FCP Bytes recv 0.00 FCP Bytes sent
0.00 FCP ops

iSCSI Statistics (per second)
516537.68 iSCSI Bytes recv 22866.13 iSCSI Bytes xmit
74.48 iSCSI ops

Interrupt Statistics (per second)
1604.06 Clock (IRQ 5) 35.51 IPI (IRQ 8)
35.51 IPI (IRQ 9) 160.84 FCAL (IRQ 23)
259.37 SB MAC (IRQ 25) 41.22 SB MAC (IRQ 26)
35.56 IPI 2172.07 total

Data Mover Statistics (per second)
627.05 total dma transfer KB 408.55 dma descriptors
82.58 dma desc wait count 753.86 total dma transfer KB
523.52 dma descriptors 112.63 dma desc wait count
0.00 total dma transfer KB 0.00 dma descriptors
0.00 dma desc wait count 0.00 total dma transfer KB
0.00 dma descriptors 0.00 dma desc wait count

____________________________
Matt Davies
Director of International IT Operations
General Atlantic
83 Pall Mall
London
SW1Y 5ES

Tel: +44 207 484 3203
Fax: +44 207 484 2803
Mobile: +44 777 559 4265
____________________________




_____________________________________________________________
This e-mail (including all attachments) is confidential and may be privileged.
It is for the exclusive use of the addressee only. If you are not the addressee, you are hereby notified that any dissemination of this communication is strictly prohibited. If you have received this communication in error, please erase all copies of the message and its attachments and notify us immediately at help [at] generalatlantic <mailto:help [at] generalatlantic>. Thank You.


Darren.Sykes at csr

Nov 6, 2008, 3:01 AM

Post #5 of 7 (10896 views)
Permalink
RE: High CPU utilisation [In reply to]

I'd agree; we had high CPU utilization on a pre 7.2.4 release on a 960
cluster due to background tasks not being given the correct priority and
thus affecting clients.


-----Original Message-----
From: owner-toasters [at] mathworks [mailto:owner-toasters [at] mathworks]
On Behalf Of Kenneth Heal
Sent: 06 November 2008 09:42
To: mdavies [at] generalatlantic; toasters [at] mathworks
Subject: RE: High CPU utilisation


Hi Matt

I have seen several issues with FAS270s and high cpu utilisation related
to several bugs in pre-7.2.4 releases, especially with the filer sat in
background processes, cpu spikes and nfs perf issues. Troubleshooting
these bugs is a pain, and the solution is to upgrade anyway, so I would
be inclined to upgrade to 7.2.6 first and see if the issue still
persists.

cheers
Kenneth

> Subject: High CPU utilisation
> Date: Thu, 6 Nov 2008 09:09:41 +0000
> From: MDAVIES [at] generalatlantic
> To: toasters [at] mathworks
>
> Hi all,
>
> We are having a problem with a FAS270 that appears to be having high
cpu utilisation issues.
>
> To be honest I have looked at sysstat and statit and I cannot see the
problem. I would appreciate if someone could take a look at the below
outputs and let me know what I'm missing.
>
> Thanks
>
> Matt

_________________________________________________________________
Express yourself instantly with MSN Messenger! Download today it's FREE!
http://messenger.msn.click-url.com/go/onm00200471ave/direct/01/


To report this email as spam click
https://www.mailcontrol.com/sr/CVCY6ygGE7DTndxI!oX7Us8RWzDHXrou56581NVR9
y+0LES29YJo8kWK7nyITuN9F4Cg!TqFUW1ICd5VsZuQpQ== .


ljason at netapp

Nov 6, 2008, 10:51 AM

Post #6 of 7 (10876 views)
Permalink
Re: High CPU utilisation [In reply to]

Given that your FAS270 is on 7.2.3, one likely culprit is Bug# 250630
which is first fixed in 7.2.3P6. This issue is specific to the FAS200
series systems.

http://now.netapp.com/NOW/cgi-bin/bol?Type=Detail&Display=250630

Once you are on a release with the fix for this in place, the
workaround should be removed.

-jbl

On Nov 6, 2008, at 4:09 AM, Davies,Matt wrote:

> Hi all,
>
> We are having a problem with a FAS270 that appears to be having high
> cpu utilisation issues.
>
> To be honest I have looked at sysstat and statit and I cannot see
> the problem. I would appreciate if someone could take a look at the
> below outputs and let me know what I'm missing.
>
> Thanks
>
> Matt
>
> ukfiler*> sysstat -i 3
> CPU NFS CIFS iSCSI Net kB/s Disk kB/s iSCSI kB/
> s Cache
> in out read write in
> out age
> 85% 18 0 26 203 13 661 0 91
> 0 24
> 68% 9 1 50 283 23 692 0 221
> 9 24
> 92% 7 4 25 144 11 1643 2443 90
> 3 24
> 78% 12 1 74 345 21 618 0 198
> 0 24
> 67% 8 0 45 288 18 722 0 205
> 5 24
> 96% 13 1 32 236 14 1365 1708 151
> 1 24
> 81% 8 4 39 222 16 843 867 167
> 4 24
> 85% 8 0 33 443 10 1212 1456 118
> 0 24
> 99% 16 2 43 370 21 912 0 168
> 4 24
> 99% 14 16 23 219 26 1023 0 98
> 0 24
> 99% 12 0 36 585 39 2452 3781 465
> 18 24
> ukfiler*>
>
> Hostname: ukfiler ID: 0084264545 Memory: 1022 MB
> NetApp Release 7.2.3: Thu Jul 5 10:06:16 PDT 2007
> <6O>
> Start time: Wed Nov 5 22:45:03 GMT 2008
>
> CPU Statistics
> 64.822501 time (seconds) 100 %
> 52.221602 system time 81 %
> 0.511694 rupt time 1 % (161581 rupts x 3 usec/
> rupt)
> 51.709908 non-rupt system time 80 %
> 77.423400 idle time 119 %
>
> 4.011144 time in CP 6 % 100 %
> 0.052061 rupt time in CP 1 % (12144 rupts x
> 4 usec/rupt)
>
> Multiprocessor Statistics (per second)
> cpu0 cpu1 total
> sk switches 6581.74 7585.82 14167.57
> hard switches 1485.87 1560.68 3046.55
> domain switches 72.75 65.01 137.76
> CP rupts 125.45 61.89 187.34
> nonCP rupts 1350.03 955.30 2305.33
> IPI rupts 18.23 17.32 35.56
>
> CP rupt usec 565.04 238.10 803.13
> nonCP rupt usec 4342.69 2747.95 7090.64
> idle 632999.86 561391.13 1194390.83
> kahuna 334624.38 409797.24 744421.77
> network 6697.98 8506.20 15204.18
> storage 4193.90 5622.94 9816.84
> exempt 3941.24 4036.10 7977.34
> raid 7872.95 4627.33 12500.27
> target 4409.87 2761.70 7171.57
> netcache 0.00 0.00 0.00
> netcache2 0.00 0.00 0.00
> cifs 352.07 271.33 623.39
> wafl_exempt 0.00 0.00 0.00
>
> 49.342358 seconds with one or more CPUs active ( 76%)
>
> 46.750469 seconds with one CPU active ( 72%)
> 2.591888 seconds with both CPUs active ( 4%)
>
> Domain Utilization By Exempt (per second)
> 0.00 idle 0.00 kahuna
> 0.00 network 0.00 storage
> 0.00 exempt 0.00 raid
> 0.00 target 0.00 netcache
> 0.00 netcache2 0.00 cifs
> 0.00 wafl_exempt
>
>
> CSMP Domain Switches (per second)
> From\To idle kahuna network storage
> exempt raid target netcache netcache2 cifs
> wafl_exem
> pt
> idle 0.00 0.00 0.00 0.00
> 0.00 0.00 0.00 0.00 0.00 0.00 0.0
> 0
> kahuna 0.00 0.00 27.09 2.65
> 0.00 5.32 1.87 0.00 0.00 18.79 0.0
> 0
> network 0.00 27.09 0.00 0.00
> 0.00 0.00 0.00 0.00 0.00 0.20 0.0
> 0
> storage 0.00 2.65 0.00 0.00
> 0.00 12.96 0.00 0.00 0.00 0.00 0.0
> 0
> exempt 0.00 0.00 0.00 0.00
> 0.00 0.00 0.00 0.00 0.00 0.00 0.0
> 0
> raid 0.00 5.32 0.00 12.96
> 0.00 0.00 0.00 0.00 0.00 0.00 0.0
> 0
> target 0.00 1.87 0.00 0.00
> 0.00 0.00 0.00 0.00 0.00 0.00 0.0
> 0
> netcache 0.00 0.00 0.00 0.00
> 0.00 0.00 0.00 0.00 0.00 0.00 0.0
> 0
> netcache2 0.00 0.00 0.00 0.00
> 0.00 0.00 0.00 0.00 0.00 0.00 0.0
> 0
> cifs 0.00 18.79 0.20 0.00
> 0.00 0.00 0.00 0.00 0.00 0.00 0.0
> 0
> wafl_exempt 0.00 0.00 0.00 0.00
> 0.00 0.00 0.00 0.00 0.00 0.00 0.
> 00
>
> Miscellaneous Statistics (per second)
> 3046.55 hard context switches 10.38 NFS operations
> 1.42 CIFS operations 0.00 HTTP operations
> 0.00 NetCache URLs 0.00 streaming packets
> 1240.42 network KB received 100.32 network KB
> transmitted
> 655.39 disk KB read 1146.02 disk KB written
> 648.90 NVRAM KB written 0.00 nolog KB written
> 8.27 WAFL bufs given to clients 0.00 checksum cache
> hits ( 0%)
> 1.23 no checksum - partial buffer 0.00 FCP operations
> 74.48 iSCSI operations
>
> WAFL Statistics (per second)
> 15.94 name cache hits ( 95%) 0.86 name cache
> misses ( 5%)
> 529973.33 buf hash hits ( 100%) 1710.32 buf hash
> misses ( 0%)
> 516.44 inode cache hits ( 100%) 0.06 inode cache
> misses ( 0%)
> 262496.95 buf cache hits ( 100%) 19.65 buf cache
> misses ( 0%)
> 30.17 blocks read 67.00 blocks read-ahead
> 58.10 chains read-ahead 0.99 dummy reads
> 14.25 blocks speculative read-ahead 226.31 blocks written
> 2.27 stripes written 0.00 blocks over-written
> 0.09 wafl_timer generated CP 0.00 snapshot generated
> CP
> 0.00 wafl_avail_bufs generated CP 0.00 dirty_blk_cnt
> generated CP
> 0.00 full NV-log generated CP 0.00 back-to-back CP
> 0.00 flush generated CP 0.00 sync generated CP
> 0.00 wafl_avail_vbufs generated CP 0.00 deferred back-to-
> back CP
> 0.00 low datavecs generated CP 304.72 non-restart messages
> 4.98 IOWAIT suspends 133872 buffers
>
> RAID Statistics (per second)
> 40.91 xors 0.00 long dispatches [0]
> 0.00 long consumed [0] 0.00 long consumed
> hipri [0]
> 0.00 long low priority [0] 0.00 long high priority
> [0]
> 0.00 long monitor tics [0] 0.00 long monitor
> clears [0]
> 0.00 long dispatches [1] 0.00 long consumed [1]
> 0.00 long consumed hipri [1] 0.00 long low priority
> [1]
> 0.00 long high priority [1] 0.00 long monitor tics
> [1]
> 0.00 long monitor clears [1] 18 max batch
> 2.11 blocked mode xor 19.68 timed mode xor
> 0.05 fast adjustments 0.05 slow adjustments
> 0 avg batch start 0 avg stripe/msec
> 2.64 tetrises written 0.00 master tetrises
> 0.00 slave tetrises 29.70 stripes written
> 11.22 partial stripes 18.48 full stripes
> 226.68 blocks written 31.56 blocks read
> 1.13 1 blocks per stripe size 6 0.29 2 blocks per
> stripe size 6
> 0.17 3 blocks per stripe size 6 0.35 4 blocks per
> stripe size 6
> 1.80 5 blocks per stripe size 6 15.63 6 blocks per
> stripe size 6
> 1.11 1 blocks per stripe size 16 0.31 2 blocks per
> stripe size 16
> 0.32 3 blocks per stripe size 16 0.19 4 blocks per
> stripe size 16
> 0.09 5 blocks per stripe size 16 0.20 6 blocks per
> stripe size 16
> 0.05 7 blocks per stripe size 16 0.20 8 blocks per
> stripe size 16
> 0.12 9 blocks per stripe size 16 0.23 10 blocks per
> stripe size 16
> 0.23 11 blocks per stripe size 16 0.56 12 blocks per
> stripe size 16
> 0.83 13 blocks per stripe size 16 1.26 14 blocks per
> stripe size 16
> 1.76 15 blocks per stripe size 16 2.85 16 blocks per
> stripe size 16
>
> Network Interface Statistics (per second)
> iface side bytes packets multicasts errors
> collisions pkt drops
> e0a recv 114768.96 92.82 0.00
> 0.00 0.00
> xmit 193.85 1.43 0.05 0.00 0.00
> e0b recv 521347.10 383.35 0.00
> 0.00 0.00
> xmit 53015.11 280.33 0.14 0.00 0.00
> vh recv 0.00 0.00 0.00
> 0.00 0.00
> xmit 0.00 0.00 0.00 0.00 0.00
> mtrunk1 recv 636116.06 476.18 0.00
> 0.00 0.00
> xmit 53208.96 281.77 0.25 0.00 0.00
> mtrunk1-1 recv 588.41 5.26 2.10
> 0.00 0.00
> xmit 665.54 2.92 0.08 0.00 0.00
> mtrunk1-21 recv 261620.02 204.87 0.00
> 0.00 0.00
> xmit 22000.14 120.08 0.05 0.00 0.00
> mtrunk1-22 recv 0.00 0.00 0.00
> 0.00 0.00
> xmit 5.28 0.05 0.05 0.00 0.00
> mtrunk1-23 recv 371871.00 264.20 0.02
> 0.00 0.00
> xmit 26855.71 156.13 0.05 0.00 0.00
>
> Disk Statistics (per second)
> ut% is the percent of time the disk was busy.
> xfers is the number of data-transfer commands issued per
> second.
> xfers = ureads + writes + cpreads + greads + gwrites
> chain is the average number of 4K blocks per command.
> usecs is the average disk round-trip time per 4K block.
>
> disk ut% xfers ureads--chain-usecs writes--chain-usecs
> cpreads-chain-usecs greads--chain-usecs gwrites-chain-usecs
> /aggr2/plex0/rg0:
> 0b.45 1 1.97 0.19 1.00 12333 1.10 9.42
> 988 0.69 5.07 768 0.00 .... . 0.00 .... .
> 0b.44 2 2.16 0.19 1.00 38167 1.28 8.35
> 925 0.69 5.07 838 0.00 .... . 0.00 .... .
> 0b.43 5 6.03 4.29 1.04 14107 1.13 6.84
> 1489 0.62 2.85 2430 0.00 .... . 0.00 .... .
> 0b.42 4 5.20 3.72 1.05 15565 0.88 8.00
> 1197 0.60 5.03 939 0.00 .... . 0.00 .... .
> 0b.41 4 5.09 3.76 1.01 14919 0.88 8.98
> 1184 0.45 4.03 1325 0.00 .... . 0.00 .... .
> 0b.40 5 5.28 4.09 1.08 13686 0.74 9.42
> 1210 0.45 5.28 1477 0.00 .... . 0.00 .... .
> 0b.39 5 5.54 4.18 1.08 15908 0.83 9.63
> 1142 0.52 4.44 1788 0.00 .... . 0.00 .... .
> 0b.38 4 5.45 3.90 1.00 17528 0.91 8.51
> 1327 0.63 3.56 1349 0.00 .... . 0.00 .... .
> 0b.37 4 5.29 3.72 1.07 16442 0.97 7.95
> 1359 0.60 4.23 1509 0.00 .... . 0.00 .... .
> 0b.36 4 5.01 3.50 1.01 13113 0.89 8.66
> 1371 0.62 4.18 1413 0.00 .... . 0.00 .... .
> 0b.32 5 5.54 4.03 1.36 11093 0.91 8.22
> 1433 0.60 3.92 1255 0.00 .... . 0.00 .... .
> 0b.17 4 5.06 3.76 1.01 13752 0.83 9.11
> 1268 0.46 4.60 1254 0.00 .... . 0.00 .... .
> 0b.33 5 5.51 3.90 1.02 14456 0.88 7.89
> 1553 0.73 4.02 1280 0.00 .... . 0.00 .... .
> 0b.18 4 5.51 3.79 1.19 12478 0.97 7.76
> 1591 0.74 3.71 1472 0.00 .... . 0.00 .... .
> 0b.34 5 5.51 3.89 1.12 13188 0.89 8.95
> 1301 0.73 3.62 1653 0.00 .... . 0.00 .... .
> 0b.19 4 5.04 3.56 1.07 11947 0.83 8.94
> 1308 0.65 3.50 2238 0.00 .... . 0.00 .... .
> 0b.35 4 4.74 3.33 1.14 12020 0.80 9.21
> 1353 0.60 4.26 1416 0.00 .... . 0.00 .... .
> 0b.20 4 5.32 3.89 1.01 13341 0.76 9.27
> 1352 0.68 3.86 1729 0.00 .... . 0.00 .... .
> /aggr1/plex0/rg0:
> 0b.24 2 2.39 0.19 1.00 13667 1.73 11.25
> 655 0.48 5.06 1140 0.00 .... . 0.00 .... .
> 0b.29 5 7.30 4.97 1.26 17742 1.71 10.48
> 666 0.62 3.45 1543 0.00 .... . 0.00 .... .
> 0b.26 4 6.34 4.37 1.28 13500 1.42 12.46
> 595 0.56 3.97 1427 0.00 .... . 0.00 .... .
> 0b.28 5 7.05 4.95 1.14 19218 1.50 11.82
> 695 0.60 3.44 1642 0.00 .... . 0.00 .... .
> 0b.27 7 6.20 4.26 1.05 19422 1.45 12.28
> 659 0.49 4.56 1000 0.00 .... . 0.00 .... .
> 0b.22 5 6.90 4.81 1.21 18814 1.53 11.51
> 835 0.56 3.81 1591 0.00 .... . 0.00 .... .
> 0b.21 4 6.57 4.37 1.11 18990 1.51 11.73
> 843 0.69 3.80 1655 0.00 .... . 0.00 .... .
> 0b.23 2 2.36 0.19 1.00 9667 1.70 11.42
> 677 0.48 5.06 1178 0.00 .... . 0.00 .... .
>
> Aggregate statistics:
> Minimum 1 1.97 0.19
> 0.74 0.45 0.00 0.00
> Mean 4 5.15 3.44
> 1.11 0.59 0.00 0.00
> Maximum 7 7.30 4.97
> 1.73 0.74 0.00 0.00
>
> Spares and other disks:
> 0b.25 0 0.00 0.00 .... .
> 0.00 .... . 0.00 .... . 0.00 .... .
> 0.00 .... .
>
> Spares and other disks:
> 0b.16 0 0.00 0.00 .... .
> 0.00 .... . 0.00 .... . 0.00 .... .
> 0.00 .... .
>
> FCP Statistics (per second)
> 0.00 FCP Bytes recv 0.00 FCP Bytes sent
> 0.00 FCP ops
>
> iSCSI Statistics (per second)
> 516537.68 iSCSI Bytes recv 22866.13 iSCSI Bytes xmit
> 74.48 iSCSI ops
>
> Interrupt Statistics (per second)
> 1604.06 Clock (IRQ 5) 35.51 IPI (IRQ 8)
> 35.51 IPI (IRQ 9) 160.84 FCAL (IRQ 23)
> 259.37 SB MAC (IRQ 25) 41.22 SB MAC (IRQ 26)
> 35.56 IPI 2172.07 total
>
> Data Mover Statistics (per second)
> 627.05 total dma transfer KB 408.55 dma descriptors
> 82.58 dma desc wait count 753.86 total dma transfer
> KB
> 523.52 dma descriptors 112.63 dma desc wait count
> 0.00 total dma transfer KB 0.00 dma descriptors
> 0.00 dma desc wait count 0.00 total dma transfer
> KB
> 0.00 dma descriptors 0.00 dma desc wait count
>
> ____________________________
> Matt Davies
> Director of International IT Operations
> General Atlantic
> 83 Pall Mall
> London
> SW1Y 5ES
>
> Tel: +44 207 484 3203
> Fax: +44 207 484 2803
> Mobile: +44 777 559 4265
> ____________________________
>
>
>
>
> _____________________________________________________________
> This e-mail (including all attachments) is confidential and may be
> privileged.
> It is for the exclusive use of the addressee only. If you are not
> the addressee,
> you are hereby notified that any dissemination of this communication
> is strictly
> prohibited. If you have received this communication in error, please
> erase all
> copies of the message and its attachments and notify us immediately at
> help [at] generalatlantic <mailto:help [at] generalatlantic>. Thank You.
>


MDAVIES at generalatlantic

Nov 6, 2008, 11:53 PM

Post #7 of 7 (10875 views)
Permalink
RE: High CPU utilisation [In reply to]

Jeff,

The clients are experiencing some increase in latency, so I will
investigate further, and see the problems occur during times of high
utilisation.

Thanks

Matt
-----Original Message-----
From: Jeff Mohler [mailto:speedtoys.racing [at] gmail]
Sent: 06 November 2008 19:35
To: Davies,Matt
Cc: toasters [at] mathworks
Subject: Re: High CPU utilisation

Matt:

Are you experiencing client latency issues?

A filer with a "high cpu state" is just that..and many have nothing to
do with client protocol latency.

CPU's doing what it's doing. It's not a windows/unix box.

I dont see from this statit, that you have any actual performance
issue, just "high cpu" which could be innumerable low priority tasks
unique to your platform and OS.



_____________________________________________________________
This e-mail (including all attachments) is confidential and may be privileged.
It is for the exclusive use of the addressee only. If you are not the addressee,
you are hereby notified that any dissemination of this communication is strictly
prohibited. If you have received this communication in error, please erase all
copies of the message and its attachments and notify us immediately at
help [at] generalatlantic <mailto:help [at] generalatlantic>. Thank You.

Netapp toasters RSS feed   Index | Next | Previous | View Threaded
 
 


Interested in having your list archived? Contact Gossamer Threads
 
  Web Applications & Managed Hosting Powered by Gossamer Threads Inc.