Mailing List Archive: Linux-HA: Pacemaker

Resource capacity limit

1 2

View All

Index | Next | Previous | View Threaded

ygao at novell

Oct 30, 2009, 4:41 AM

Post #1 of 44 (1455 views)
Permalink

Resource capacity limit

Hi Andrew and Lars,
The attachment is the first try to implement "Resource capacity limit"
which is issued by Lars from:
https://fate.novell.com/303384

Description:
We need a mechanism for the PE to take resource weight into account to
prevent nodes from being overloaded.

Resources would require certain minimal values for node attributes
(this is available right now); however, they would also "consume" them,
reducing the value of the node attributes for further resource placement.
(This could be a special flag in the rsc_location rule, for example.)
If a node does not have enough capacity available, it is not considered.
..

User case:
Xen guests have memory requirements; nodes cannot host more guests than
the node has physical memory installed.

Configuration example:

node yingying \
attributes capacity="100"
primitive dummy0 ocf:heartbeat:Dummy \
meta weight="90" priority="2"
primitive dummy1 ocf:heartbeat:Dummy \
meta weight="60" priority="1"
..
property $id="cib-bootstrap-options" \
limit-capacity="true"
..

Because dummy0 has the higher priority, it'll be running on node "yingying".
While this node only has "10" (100-90) capacity remaining now, so dummy1 cannot
be running on this node. If there's no other node where it can be running on,
dummy1 will be stopped.

If we don't want to enable capacity limit. We could set property
"limit-capacity" to "false", or default it.

What do you think about the way it's implemented? Did I do it right?

I also noticed a likely similar planned feature described in
http://clusterlabs.org/wiki/Planned_Features

"Implement adaptive service placement (based on the RAM, CPU etc.
required by the service and made available by the nodes) "

Indeed, this try only supports single kind of capacity, and it's not
adaptive... Do you already have a thorough consideration about this
feature?
Any comments or suggestions are appreciated. Thanks!

Regards,
Yan
--
ygao [at] novell
Software Engineer
China Server Team, OPS Engineering

Novell, Inc.
Making IT Work As One™

Attachments:

pacemaker-capacity-limit.patch (4.83 KB)

lmb at suse

Oct 30, 2009, 5:20 AM

Post #2 of 44 (1424 views)
Permalink

Re: Resource capacity limit [In reply to]

On 2009-10-30T19:41:35, Yan Gao <ygao [at] novell> wrote:

Hi Yan Gao,

excellent!

Before reviewing the code, lets review the interface/configuration
though.

> User case:
> Xen guests have memory requirements; nodes cannot host more guests than
> the node has physical memory installed.
>
>
> Configuration example:
>
> node yingying \
> attributes capacity="100"
> primitive dummy0 ocf:heartbeat:Dummy \
> meta weight="90" priority="2"
> primitive dummy1 ocf:heartbeat:Dummy \
> meta weight="60" priority="1"
> ..
> property $id="cib-bootstrap-options" \
> limit-capacity="true"

First, I would prefer not to contaminate the regular node attribute
namespace; the word "capacity" might already be used. Second, the
"weight" is just one dimension, which is somewhat difficult.

I'd propose to introduce a new XML element, "resource_utilization" (name
to be decided ;-) containing a "nvset", and which can be used in a node
element or a resource primitive.

This creates a new namespace, avoiding clashes, and distinguishes the
utilization parameters from the other various attributes.

Further, it trivially allows for several user-defined metrics.

node hex-0 \
utilization memory="4096" cpu="8"
...
primitive dummy0 ocf:heartbeat:Dummy \
meta priority="2"
utilization memory="2048" cpu="2"
primitive dummy1 ocf:heartbeat:Dummy \
utilization memory="3012"
primitive dummy2 ocf:heartbeat:Dummy \
utilization cpu="6"

dummy0 + dummy2 could both be placed on hex-0, or dummy1+dummy2, but not
dummy0 + dummy1.

"Placement allowed where none of the utilization parameters would become
negative." (ie, iterate over the utilization attributes specified for
the resource.)

> If we don't want to enable capacity limit. We could set property
> "limit-capacity" to "false", or default it.

Right, a cluster property to globally disable/enable this is a very good
idea.

> I also noticed a likely similar planned feature described in
> http://clusterlabs.org/wiki/Planned_Features
>
> "Implement adaptive service placement (based on the RAM, CPU etc.
> required by the service and made available by the nodes) "
>
> Indeed, this try only supports single kind of capacity, and it's not
> adaptive... Do you already have a thorough consideration about this
> feature?

I think this is a two phase feature for the PE: The first phase is what
you propose - make sure we do not overload any given node, basically
implementing hard limits.

The second phase would be for the PE to actually try to "optimize"
placement, and try to solve the constraints imposed by the utilization
versus capacity scores to a) place as many resources as possible
successfully, and b) to either spread them thinly (load distribution) or
condensed (load concentration, think power savings by being able to put
some nodes to sleep).

The first phase should, IMHO, be quite easy to implement. The second one
is significantly more difficult, and we'd need to pull in an
optimization library to solve this for us. It's conceivable that for
this to happen, we'd need to disable the normal "rsc_location" rules
altogether because they'd interfere badly. (And interesting to note that
the rsc_collocation constraints can be mapped into this scheme and
entirely handled by this solver.)

There is the "adaptive" bit, of course, where the utilization of the
resources and the nodes is automatically determined and adjusted based
on utilization monitoring. This is even more challenging and frequently
considered a research problem.

In summary, I think phase one is urgently needed; thankfully, it is
straightforward to solve too, and the admin can influence placement with
priorities and scoring sufficiently to avoid resources being offlined
due to resource collisions too frequently.

Phase two is a "solved problem" from an algorithmic point of view, but
implementing it is probably not quite as trivial. I'd welcome to see
this happening too.

Adaptive placement ... anyone who wants to write a master or phd thesis
around? ;-)

Best,
Lars

--
Architect Storage/HA, OPS Engineering, Novell, Inc.
SUSE LINUX Products GmbH, GF: Markus Rex, HRB 16746 (AG N�rnberg)
"Experience is the name everyone gives to their mistakes." -- Oscar Wilde

_______________________________________________
Pacemaker mailing list
Pacemaker [at] oss
http://oss.clusterlabs.org/mailman/listinfo/pacemaker

r.bhatia at ipax

Oct 30, 2009, 5:40 AM

Post #3 of 44 (1416 views)
Permalink

Re: Resource capacity limit [In reply to]

hi,

On 10/30/2009 01:20 PM, Lars Marowsky-Bree wrote:
> I think this is a two phase feature for the PE: The first phase is what
> you propose - make sure we do not overload any given node, basically
> implementing hard limits.
>
> The second phase would be for the PE to actually try to "optimize"
> placement, and try to solve the constraints imposed by the utilization
> versus capacity scores to a) place as many resources as possible
> successfully, and b) to either spread them thinly (load distribution) or
> condensed (load concentration, think power savings by being able to put
> some nodes to sleep).

i just want to let you know that i think that this is a marvelous
addition to pacemaker!

cheers,
raoul
--
____________________________________________________________________
DI (FH) Raoul Bhatia M.Sc. email. r.bhatia [at] ipax
Technischer Leiter

IPAX - Aloy Bhatia Hava OEG web. http://www.ipax.at
Barawitzkagasse 10/2/2/11 email. office [at] ipax
1190 Wien tel. +43 1 3670030
FN 277995t HG Wien fax. +43 1 3670030 15
____________________________________________________________________

_______________________________________________
Pacemaker mailing list
Pacemaker [at] oss
http://oss.clusterlabs.org/mailman/listinfo/pacemaker

ygao at novell

Nov 1, 2009, 11:09 PM

Post #4 of 44 (1415 views)
Permalink

Re: Resource capacity limit [In reply to]

Hi Lars,
Thanks for the great suggestions!

Lars Marowsky-Bree wrote:
> On 2009-10-30T19:41:35, Yan Gao <ygao [at] novell> wrote:
>>
>> Configuration example:
>>
>> node yingying \
>> attributes capacity="100"
>> primitive dummy0 ocf:heartbeat:Dummy \
>> meta weight="90" priority="2"
>> primitive dummy1 ocf:heartbeat:Dummy \
>> meta weight="60" priority="1"
>> ..
>> property $id="cib-bootstrap-options" \
>> limit-capacity="true"
>
> First, I would prefer not to contaminate the regular node attribute
> namespace; the word "capacity" might already be used. Second, the
> "weight" is just one dimension, which is somewhat difficult.
>
> I'd propose to introduce a new XML element, "resource_utilization" (name
> to be decided ;-) containing a "nvset", and which can be used in a node
> element or a resource primitive.
>
> This creates a new namespace, avoiding clashes, and distinguishes the
> utilization parameters from the other various attributes.
>
> Further, it trivially allows for several user-defined metrics.
Right, great idea! I'll try to implement it if Andrew is OK with that either:)

>
> node hex-0 \
> utilization memory="4096" cpu="8"
> ...
> primitive dummy0 ocf:heartbeat:Dummy \
> meta priority="2"
> utilization memory="2048" cpu="2"
> primitive dummy1 ocf:heartbeat:Dummy \
> utilization memory="3012"
> primitive dummy2 ocf:heartbeat:Dummy \
> utilization cpu="6"
>
> dummy0 + dummy2 could both be placed on hex-0, or dummy1+dummy2, but not
> dummy0 + dummy1.
>
> "Placement allowed where none of the utilization parameters would become
> negative." (ie, iterate over the utilization attributes specified for
> the resource.)
>
>> I also noticed a likely similar planned feature described in
>> http://clusterlabs.org/wiki/Planned_Features
>>
>> "Implement adaptive service placement (based on the RAM, CPU etc.
>> required by the service and made available by the nodes) "
>>
>> Indeed, this try only supports single kind of capacity, and it's not
>> adaptive... Do you already have a thorough consideration about this
>> feature?
>
> I think this is a two phase feature for the PE: The first phase is what
> you propose - make sure we do not overload any given node, basically
> implementing hard limits.
>
> The second phase would be for the PE to actually try to "optimize"
> placement, and try to solve the constraints imposed by the utilization
> versus capacity scores to a) place as many resources as possible
> successfully, and b) to either spread them thinly (load distribution) or
> condensed (load concentration, think power savings by being able to put
> some nodes to sleep).
>
> The first phase should, IMHO, be quite easy to implement. The second one
> is significantly more difficult, and we'd need to pull in an
> optimization library to solve this for us. It's conceivable that for
> this to happen, we'd need to disable the normal "rsc_location" rules
> altogether because they'd interfere badly. (And interesting to note that
> the rsc_collocation constraints can be mapped into this scheme and
> entirely handled by this solver.)
>
> There is the "adaptive" bit, of course, where the utilization of the
> resources and the nodes is automatically determined and adjusted based
> on utilization monitoring. This is even more challenging and frequently
> considered a research problem.
>
>
> In summary, I think phase one is urgently needed; thankfully, it is
> straightforward to solve too, and the admin can influence placement with
> priorities and scoring sufficiently to avoid resources being offlined
> due to resource collisions too frequently.
>
> Phase two is a "solved problem" from an algorithmic point of view, but
> implementing it is probably not quite as trivial. I'd welcome to see
> this happening too.
>
Thanks for the information!

Best regards,
Yan

--
ygao [at] novell
Software Engineer
China Server Team, OPS Engineering

Novell, Inc.
Making IT Work As One™

_______________________________________________
Pacemaker mailing list
Pacemaker [at] oss
http://oss.clusterlabs.org/mailman/listinfo/pacemaker

ygao at novell

Nov 3, 2009, 3:15 AM

Post #5 of 44 (1403 views)
Permalink

Re: Resource capacity limit [In reply to]

Hi again,

Yan Gao wrote:
> Hi Lars,
> Thanks for the great suggestions!
>
> Lars Marowsky-Bree wrote:
>> On 2009-10-30T19:41:35, Yan Gao <ygao [at] novell> wrote:
>>> Configuration example:
>>>
>>> node yingying \
>>> attributes capacity="100"
>>> primitive dummy0 ocf:heartbeat:Dummy \
>>> meta weight="90" priority="2"
>>> primitive dummy1 ocf:heartbeat:Dummy \
>>> meta weight="60" priority="1"
>>> ..
>>> property $id="cib-bootstrap-options" \
>>> limit-capacity="true"
>> First, I would prefer not to contaminate the regular node attribute
>> namespace; the word "capacity" might already be used. Second, the
>> "weight" is just one dimension, which is somewhat difficult.
>>
>> I'd propose to introduce a new XML element, "resource_utilization" (name
>> to be decided ;-) containing a "nvset", and which can be used in a node
>> element or a resource primitive.
>>
>> This creates a new namespace, avoiding clashes, and distinguishes the
>> utilization parameters from the other various attributes.
>>
>> Further, it trivially allows for several user-defined metrics.
> Right, great idea! I'll try to implement it if Andrew is OK with that either:)
>
Done and attached.

XML sample:
..
<nodes>
<node id="yingying" type="normal" uname="yingying">
<utilization id="nodes-yingying-utilization">
<nvpair id="nodes-yingying-memory" name="memory" value="150"/>
<nvpair id="nodes-yingying-cpu" name="cpu" value="1"/>
</utilization>
</node>
</nodes>
..
<resources>
<primitive class="ocf" id="Dummy0" provider="heartbeat" type="Dummy">
<utilization id="Dummy0-utilization">
<nvpair id="Dummy0-utilization-memory" name="memory" value="90"/>
<nvpair id="Dummy0-utilization-cpu" name="cpu" value="1"/>
</utilization>
</primitive>
<primitive class="ocf" id="Dummy1" provider="heartbeat" type="Dummy">
<utilization id="Dummy1-utilization">
<nvpair id="Dummy1-utilization-memory" name="memory" value="60"/>
<nvpair id="Dummy1-utilization-cpu" name="cpu" value="1"/>
</utilization>
</primitive>
</resources>
..

Please kindly review it...
Any suggestions are appreciated!

Thanks,
Yan
--
ygao [at] novell
Software Engineer
China Server Team, OPS Engineering

Novell, Inc.
Making IT Work As One™

Attachments:

pacemaker-utilization-limit.patch (8.55 KB)

lmb at suse

Nov 4, 2009, 8:41 AM

Post #6 of 44 (1402 views)
Permalink

Re: Resource capacity limit [In reply to]

On 2009-11-03T19:15:59, Yan Gao <ygao [at] novell> wrote:

> XML sample:
> ..
> <nodes>
> <node id="yingying" type="normal" uname="yingying">
> <utilization id="nodes-yingying-utilization">
> <nvpair id="nodes-yingying-memory" name="memory" value="150"/>
> <nvpair id="nodes-yingying-cpu" name="cpu" value="1"/>
> </utilization>
> </node>
> </nodes>
> ..
> <resources>
> <primitive class="ocf" id="Dummy0" provider="heartbeat" type="Dummy">
> <utilization id="Dummy0-utilization">
> <nvpair id="Dummy0-utilization-memory" name="memory" value="90"/>
> <nvpair id="Dummy0-utilization-cpu" name="cpu" value="1"/>
> </utilization>
> </primitive>
> <primitive class="ocf" id="Dummy1" provider="heartbeat" type="Dummy">
> <utilization id="Dummy1-utilization">
> <nvpair id="Dummy1-utilization-memory" name="memory" value="60"/>
> <nvpair id="Dummy1-utilization-cpu" name="cpu" value="1"/>
> </utilization>
> </primitive>
> </resources>
> ..
>
> Please kindly review it...
> Any suggestions are appreciated!

I think this is exactly what we need. Great job!

Code looks good too.

Andrew?

For added kicks, which may be something Andrew can add more readily, I
wonder if utilization should also be subject to time-based evaluation.
Think of a database needing more horsepower on weekdays - but perhaps
that is something that should wait until dynamic load balancing
happens.

Regards,
Lars

--
Architect Storage/HA, OPS Engineering, Novell, Inc.
SUSE LINUX Products GmbH, GF: Markus Rex, HRB 16746 (AG N�rnberg)
"Experience is the name everyone gives to their mistakes." -- Oscar Wilde

_______________________________________________
Pacemaker mailing list
Pacemaker [at] oss
http://oss.clusterlabs.org/mailman/listinfo/pacemaker

ygao at novell

Nov 4, 2009, 11:49 PM

Post #7 of 44 (1400 views)
Permalink

Re: Resource capacity limit [In reply to]

Hi,

Lars Marowsky-Bree wrote:
> On 2009-11-03T19:15:59, Yan Gao <ygao [at] novell> wrote:
>
>> XML sample:
>> ..
>> <nodes>
>> <node id="yingying" type="normal" uname="yingying">
>> <utilization id="nodes-yingying-utilization">
>> <nvpair id="nodes-yingying-memory" name="memory" value="150"/>
>> <nvpair id="nodes-yingying-cpu" name="cpu" value="1"/>
>> </utilization>
>> </node>
>> </nodes>
>> ..
>> <resources>
>> <primitive class="ocf" id="Dummy0" provider="heartbeat" type="Dummy">
>> <utilization id="Dummy0-utilization">
>> <nvpair id="Dummy0-utilization-memory" name="memory" value="90"/>
>> <nvpair id="Dummy0-utilization-cpu" name="cpu" value="1"/>
>> </utilization>
>> </primitive>
>> <primitive class="ocf" id="Dummy1" provider="heartbeat" type="Dummy">
>> <utilization id="Dummy1-utilization">
>> <nvpair id="Dummy1-utilization-memory" name="memory" value="60"/>
>> <nvpair id="Dummy1-utilization-cpu" name="cpu" value="1"/>
>> </utilization>
>> </primitive>
>> </resources>
>> ..
>>
>> Please kindly review it...
>> Any suggestions are appreciated!
>
> I think this is exactly what we need. Great job!
>
> Code looks good too.
>
> Andrew?
>
>
> For added kicks, which may be something Andrew can add more readily, I
> wonder if utilization should also be subject to time-based evaluation.
> Think of a database needing more horsepower on weekdays - but perhaps
> that is something that should wait until dynamic load balancing
> happens.
As the "unpack_instance_attributes()" function which is exploited already
considers the rules of attributes sets. So we could add rules contain
date_expressions for an "utilization" to achieve that. And yes, I've tested
it.

Thanks,
Yan
--
ygao [at] novell
Software Engineer
China Server Team, OPS Engineering

Novell, Inc.
Making IT Work As One™

_______________________________________________
Pacemaker mailing list
Pacemaker [at] oss
http://oss.clusterlabs.org/mailman/listinfo/pacemaker

andrew at beekhof

Nov 5, 2009, 5:14 AM

Post #8 of 44 (1405 views)
Permalink

Re: Resource capacity limit [In reply to]

On Tue, Nov 3, 2009 at 12:15 PM, Yan Gao <ygao [at] novell> wrote:
> Hi again,
>
> Yan Gao wrote:
>> Hi Lars,
>> Thanks for the great suggestions!
>>
>> Lars Marowsky-Bree wrote:
>>> On 2009-10-30T19:41:35, Yan Gao <ygao [at] novell> wrote:
>>>> Configuration example:
>>>>
>>>> node yingying \
>>>> � � attributes capacity="100"
>>>> primitive dummy0 ocf:heartbeat:Dummy \
>>>> � � meta weight="90" priority="2"
>>>> primitive dummy1 ocf:heartbeat:Dummy \
>>>> � � meta weight="60" priority="1"
>>>> ..
>>>> property $id="cib-bootstrap-options" \
>>>> � � limit-capacity="true"
>>> First, I would prefer not to contaminate the regular node attribute
>>> namespace; the word "capacity" might already be used. Second, the
>>> "weight" is just one dimension, which is somewhat difficult.
>>>
>>> I'd propose to introduce a new XML element, "resource_utilization" (name
>>> to be decided ;-) containing a "nvset", and which can be used in a node
>>> element or a resource primitive.
>>>
>>> This creates a new namespace, avoiding clashes, and distinguishes the
>>> utilization parameters from the other various attributes.
>>>
>>> Further, it trivially allows for several user-defined metrics.
>> Right, great idea! I'll try to implement it if Andrew is OK with that either:)
>>
> Done and attached.
>
> XML sample:
> ..
> � �<nodes>
> � � �<node id="yingying" type="normal" uname="yingying">
> � � � �<utilization id="nodes-yingying-utilization">
> � � � � �<nvpair id="nodes-yingying-memory" name="memory" value="150"/>
> � � � � �<nvpair id="nodes-yingying-cpu" name="cpu" value="1"/>
> � � � �</utilization>
> � � �</node>
> � �</nodes>
> ..
> � �<resources>
> � � �<primitive class="ocf" id="Dummy0" provider="heartbeat" type="Dummy">
> � � � �<utilization id="Dummy0-utilization">
> � � � � �<nvpair id="Dummy0-utilization-memory" name="memory" value="90"/>
> � � � � �<nvpair id="Dummy0-utilization-cpu" name="cpu" value="1"/>
> � � � �</utilization>
> � � �</primitive>
> � � �<primitive class="ocf" id="Dummy1" provider="heartbeat" type="Dummy">
> � � � �<utilization id="Dummy1-utilization">
> � � � � �<nvpair id="Dummy1-utilization-memory" name="memory" value="60"/>
> � � � � �<nvpair id="Dummy1-utilization-cpu" name="cpu" value="1"/>
> � � � �</utilization>
> � � �</primitive>
> � �</resources>
> ..
>
> Please kindly review it...
> Any suggestions are appreciated!

Whats the behavior if a node has either no utilization block or no
value for that attribute?
Can a resource with a utilization block still be placed there?

/me goes off to read the patch

_______________________________________________
Pacemaker mailing list
Pacemaker [at] oss
http://oss.clusterlabs.org/mailman/listinfo/pacemaker

andrew at beekhof

Nov 5, 2009, 5:45 AM

Post #9 of 44 (1397 views)
Permalink

Re: Resource capacity limit [In reply to]

On Wed, Nov 4, 2009 at 5:41 PM, Lars Marowsky-Bree <lmb [at] suse> wrote:
> On 2009-11-03T19:15:59, Yan Gao <ygao [at] novell> wrote:
>
>> XML sample:
>> ..
>> � � <nodes>
>> � � � <node id="yingying" type="normal" uname="yingying">
>> � � � � <utilization id="nodes-yingying-utilization">
>> � � � � � <nvpair id="nodes-yingying-memory" name="memory" value="150"/>
>> � � � � � <nvpair id="nodes-yingying-cpu" name="cpu" value="1"/>
>> � � � � </utilization>
>> � � � </node>
>> � � </nodes>
>> ..
>> � � <resources>
>> � � � <primitive class="ocf" id="Dummy0" provider="heartbeat" type="Dummy">
>> � � � � <utilization id="Dummy0-utilization">
>> � � � � � <nvpair id="Dummy0-utilization-memory" name="memory" value="90"/>
>> � � � � � <nvpair id="Dummy0-utilization-cpu" name="cpu" value="1"/>
>> � � � � </utilization>
>> � � � </primitive>
>> � � � <primitive class="ocf" id="Dummy1" provider="heartbeat" type="Dummy">
>> � � � � <utilization id="Dummy1-utilization">
>> � � � � � <nvpair id="Dummy1-utilization-memory" name="memory" value="60"/>
>> � � � � � <nvpair id="Dummy1-utilization-cpu" name="cpu" value="1"/>
>> � � � � </utilization>
>> � � � </primitive>
>> � � </resources>
>> ..
>>
>> Please kindly review it...
>> Any suggestions are appreciated!
>
> I think this is exactly what we need. Great job!
>
> Code looks good too.
>
> Andrew?

Four things...

Do we still need the limit-utilization option?
I guess it might be nice to be able to turn it off globally... was
that the intention here?

The next one is minor, there should at least be a debug message when
we filter out a node in native_color()
Thats the sort of thing thats going to mess with people :-)

There also needs to be some PE regression tests for this (and be sure
to run the existing ones to make sure they don't break).

Lastly, I would really like to defer this for 1.2
I know I've bent the rules a bit for 1.0 in the past, but its really
late in the game now.

Which reminds me, I need to get devel sorted out...

_______________________________________________
Pacemaker mailing list
Pacemaker [at] oss
http://oss.clusterlabs.org/mailman/listinfo/pacemaker

misch at multinet

Nov 5, 2009, 6:03 AM

Post #10 of 44 (1397 views)
Permalink

Re: Resource capacity limit [In reply to]

Am Donnerstag, 5. November 2009 14:45:36 schrieb Andrew Beekhof:
(...)
> Lastly, I would really like to defer this for 1.2
> I know I've bent the rules a bit for 1.0 in the past, but its really
> late in the game now.

Good idea. Make 1.0 STABLE and develop new things in 1.1

Greetings,

--
Dr. Michael Schwartzkopff
MultiNET Services GmbH
Addresse: Bretonischer Ring 7; 85630 Grasbrunn; Germany
Tel: +49 - 89 - 45 69 11 0
Fax: +49 - 89 - 45 69 11 21
mob: +49 - 174 - 343 28 75

mail: misch [at] multinet
web: www.multinet.de

Sitz der Gesellschaft: 85630 Grasbrunn
Registergericht: Amtsgericht M�nchen HRB 114375
Gesch�ftsf�hrer: G�nter Jurgeneit, Hubert Martens

---

PGP Fingerprint: F919 3919 FF12 ED5A 2801 DEA6 AA77 57A4 EDD8 979B
Skype: misch42

_______________________________________________
Pacemaker mailing list
Pacemaker [at] oss
http://oss.clusterlabs.org/mailman/listinfo/pacemaker

dejanmm at fastmail

Nov 5, 2009, 8:25 AM

Post #11 of 44 (1397 views)
Permalink

Re: Resource capacity limit [In reply to]

Hi,

On Thu, Nov 05, 2009 at 02:45:36PM +0100, Andrew Beekhof wrote:
> On Wed, Nov 4, 2009 at 5:41 PM, Lars Marowsky-Bree <lmb [at] suse> wrote:
> > On 2009-11-03T19:15:59, Yan Gao <ygao [at] novell> wrote:
> >
> >> XML sample:
> >> ..
> >> � � <nodes>
> >> � � � <node id="yingying" type="normal" uname="yingying">
> >> � � � � <utilization id="nodes-yingying-utilization">
> >> � � � � � <nvpair id="nodes-yingying-memory" name="memory" value="150"/>
> >> � � � � � <nvpair id="nodes-yingying-cpu" name="cpu" value="1"/>
> >> � � � � </utilization>
> >> � � � </node>
> >> � � </nodes>
> >> ..
> >> � � <resources>
> >> � � � <primitive class="ocf" id="Dummy0" provider="heartbeat" type="Dummy">
> >> � � � � <utilization id="Dummy0-utilization">
> >> � � � � � <nvpair id="Dummy0-utilization-memory" name="memory" value="90"/>
> >> � � � � � <nvpair id="Dummy0-utilization-cpu" name="cpu" value="1"/>
> >> � � � � </utilization>
> >> � � � </primitive>
> >> � � � <primitive class="ocf" id="Dummy1" provider="heartbeat" type="Dummy">
> >> � � � � <utilization id="Dummy1-utilization">
> >> � � � � � <nvpair id="Dummy1-utilization-memory" name="memory" value="60"/>
> >> � � � � � <nvpair id="Dummy1-utilization-cpu" name="cpu" value="1"/>
> >> � � � � </utilization>
> >> � � � </primitive>
> >> � � </resources>
> >> ..
> >>
> >> Please kindly review it...
> >> Any suggestions are appreciated!
> >
> > I think this is exactly what we need. Great job!
> >
> > Code looks good too.
> >
> > Andrew?
>
> Four things...
>
> Do we still need the limit-utilization option?
> I guess it might be nice to be able to turn it off globally... was
> that the intention here?
>
> The next one is minor, there should at least be a debug message when
> we filter out a node in native_color()
> Thats the sort of thing thats going to mess with people :-)
>
> There also needs to be some PE regression tests for this (and be sure
> to run the existing ones to make sure they don't break).
>
> Lastly, I would really like to defer this for 1.2

Agreed.

> I know I've bent the rules a bit for 1.0 in the past, but its really
> late in the game now.
>
> Which reminds me, I need to get devel sorted out...

While you're at it, perhaps it would be good to rethink the
release policy. For me in particular it would be great to know at
least one week in advance when there'll be a release. For the
general public as well, since they'll have a chance to do some
testing of the new code. I know that you do test before
releasing, but the more people test in various environments, the
more bugs found. Also, it may be good to introduce and announce
the feature freeze point, after which only bug fixes will be
accepted.

Thanks,

Dejan

> _______________________________________________
> Pacemaker mailing list
> Pacemaker [at] oss
> http://oss.clusterlabs.org/mailman/listinfo/pacemaker

_______________________________________________
Pacemaker mailing list
Pacemaker [at] oss
http://oss.clusterlabs.org/mailman/listinfo/pacemaker

andrew at beekhof

Nov 5, 2009, 10:28 AM

Post #12 of 44 (1397 views)
Permalink

Re: Resource capacity limit [In reply to]

On Thu, Nov 5, 2009 at 5:25 PM, Dejan Muhamedagic <dejanmm [at] fastmail> wrote:
> Hi,
>> Which reminds me, I need to get devel sorted out...
>
> While you're at it, perhaps it would be good to rethink the
> release policy. For me in particular it would be great to know at
> least one week in advance when there'll be a release. For the
> general public as well, since they'll have a chance to do some
> testing of the new code.

The idea is that people can always pull from stable-1.0, in theory it
should never be broken.
If you're in the middle of stuff, keep it locally until you're done.

Generally though, I start testing on the 15th of every month.
I thought I said that somewhere... I know the releases page indicates
the month (if its delayed as it was due to my move).

But I'm thinking of moving to a bi-monthly cycle. Thoughts?

> I know that you do test before
> releasing, but the more people test in various environments, the
> more bugs found. Also, it may be good to introduce and announce
> the feature freeze point, after which only bug fixes will be
> accepted.

Well in theory that point is x.y.0
I've been turning a blind eye to your changes in the shell because its
still very immature (I don't mean that negatively, its just new code).

Though I've a history of allowing isolated, non-invasive features if
we've not yet planned the next stable series (basically what happened
for the node health stuff from IBM). Its a case-by-case thing, but
I'd agree that we could do with documenting this.

_______________________________________________
Pacemaker mailing list
Pacemaker [at] oss
http://oss.clusterlabs.org/mailman/listinfo/pacemaker

misch at multinet

Nov 5, 2009, 11:50 AM

Post #13 of 44 (1398 views)
Permalink

Resource capacity limit [In reply to]

Hi,

on the list was a discussion about resource capacity limits. Yan Gao also
implemented it.

As far as I understood the discussion the solution is to attach nodes and
resources capacity limits. Resources are distributed on the nodes of a cluster
according to its capacatiy needs. They would be migrated or shutdown if the
capacity limits on the node are not met.

My question is: Can the capacity figures of the resources be made dynamically?

So i.e. for every monitor operation the CRM updates the capacity usage figures
of the resource. So the cluster could react dynamically oin the actual
capacity of a resource.

Greetings,

Michael.

_______________________________________________
Pacemaker mailing list
Pacemaker [at] oss
http://oss.clusterlabs.org/mailman/listinfo/pacemaker

andrew at beekhof

Nov 5, 2009, 12:37 PM

Post #14 of 44 (1398 views)
Permalink

Re: Resource capacity limit [In reply to]

On Thu, Nov 5, 2009 at 8:50 PM, Michael Schwartzkopff <misch [at] multinet> wrote:
> Hi,
>
> on the list was a discussion about resource capacity limits. Yan Gao also
> implemented it.
>
> As far as I understood the discussion the solution is to attach nodes and
> resources capacity limits. Resources are distributed on the nodes of a cluster
> according to its capacatiy needs. They would be migrated or shutdown if the
> capacity limits on the node are not met.
>
> My question is: Can the capacity figures of the resources be made dynamically?

You can, but you probably don't want to.
For example, free RAM and CPU load are two things that absolutely make
no sense to include in such calculations.

Consider how it works:

- The node starts and reports 2Gb of RAM
- We place a service there that reserves 512Mb
- The cluster knows there is 1.5Gb remaining
- We place two more services there that also reserve 512Mb each

If the amount of RAM at the beginning was the amount free, then when
you updated it to be 512Mb the PE would run and stop two of the
resources!

You always want to feed the cluster the total amount of RAM installed
which, at most, you'd query when the cluster starts on that node.
>
> So i.e. for every monitor operation the CRM updates the capacity usage figures
> of the resource. So the cluster could react dynamically oin the actual
> capacity of a resource.
>
> Greetings,
>
> Michael.
>
> _______________________________________________
> Pacemaker mailing list
> Pacemaker [at] oss
> http://oss.clusterlabs.org/mailman/listinfo/pacemaker
>

_______________________________________________
Pacemaker mailing list
Pacemaker [at] oss
http://oss.clusterlabs.org/mailman/listinfo/pacemaker

ygao at novell

Nov 5, 2009, 11:22 PM

Post #15 of 44 (1395 views)
Permalink

Re: Resource capacity limit [In reply to]

Hi Andrew,
Thanks for your reply!

Andrew Beekhof wrote:
> On Wed, Nov 4, 2009 at 5:41 PM, Lars Marowsky-Bree <lmb [at] suse> wrote:
>> On 2009-11-03T19:15:59, Yan Gao <ygao [at] novell> wrote:
>>
>>> XML sample:
>>> ..
>>> <nodes>
>>> <node id="yingying" type="normal" uname="yingying">
>>> <utilization id="nodes-yingying-utilization">
>>> <nvpair id="nodes-yingying-memory" name="memory" value="150"/>
>>> <nvpair id="nodes-yingying-cpu" name="cpu" value="1"/>
>>> </utilization>
>>> </node>
>>> </nodes>
>>> ..
>>> <resources>
>>> <primitive class="ocf" id="Dummy0" provider="heartbeat" type="Dummy">
>>> <utilization id="Dummy0-utilization">
>>> <nvpair id="Dummy0-utilization-memory" name="memory" value="90"/>
>>> <nvpair id="Dummy0-utilization-cpu" name="cpu" value="1"/>
>>> </utilization>
>>> </primitive>
>>> <primitive class="ocf" id="Dummy1" provider="heartbeat" type="Dummy">
>>> <utilization id="Dummy1-utilization">
>>> <nvpair id="Dummy1-utilization-memory" name="memory" value="60"/>
>>> <nvpair id="Dummy1-utilization-cpu" name="cpu" value="1"/>
>>> </utilization>
>>> </primitive>
>>> </resources>
>>> ..
>>>
>>> Please kindly review it...
>>> Any suggestions are appreciated!
>> I think this is exactly what we need. Great job!
>>
>> Code looks good too.
>>
>> Andrew?
>
> Four things...
>
> Do we still need the limit-utilization option?
> I guess it might be nice to be able to turn it off globally... was
> that the intention here?
Sorry, missed it in the sample, while it has been implemented in the codes:-)
Yes, it's "limit-utilization" property, and default to "false".
So the working XML sample should be :
..
<crm_config>
<cluster_property_set id="cib-bootstrap-options">
...
<nvpair id="cib-bootstrap-options-limit-utilization" name="limit-utilization" value="true"/>
</cluster_property_set>
</crm_config>
<nodes>
<node id="yingying" type="normal" uname="yingying">
<utilization id="nodes-yingying-utilization">
<nvpair id="nodes-yingying-memory" name="memory" value="150"/>
<nvpair id="nodes-yingying-cpu" name="cpu" value="1"/>
</utilization>
</node>
</nodes>...
<resources>
<primitive class="ocf" id="Dummy0" provider="heartbeat" type="Dummy">
<utilization id="Dummy0-utilization">
<nvpair id="Dummy0-utilization-memory" name="memory" value="90"/>
<nvpair id="Dummy0-utilization-cpu" name="cpu" value="1"/>
</utilization>
</primitive>
<primitive class="ocf" id="Dummy1" provider="heartbeat" type="Dummy">
<utilization id="Dummy1-utilization">
<nvpair id="Dummy1-utilization-memory" name="memory" value="60"/>
<nvpair id="Dummy1-utilization-cpu" name="cpu" value="1"/>
</utilization>
</primitive>
</resources>
..

>
> The next one is minor, there should at least be a debug message when
> we filter out a node in native_color()
> Thats the sort of thing thats going to mess with people :-)
Indeed :-) Added one and attached the revised patch.
>
> There also needs to be some PE regression tests for this (and be sure
> to run the existing ones to make sure they don't break).
Right.

>
> Lastly, I would really like to defer this for 1.2
Agree too.

> I know I've bent the rules a bit for 1.0 in the past, but its really
> late in the game now.
>
> Which reminds me, I need to get devel sorted out...
:-)

Thanks again!

Best regards,
Yan
--
ygao [at] novell
Software Engineer
China Server Team, OPS Engineering

Novell, Inc.
Making IT Work As One™

Attachments:

pacemaker-utilization-limit.patch (8.67 KB)

ygao at novell

Nov 5, 2009, 11:27 PM

Post #16 of 44 (1401 views)
Permalink

Re: Resource capacity limit [In reply to]

Hi Andrew,

Andrew Beekhof wrote:
> On Tue, Nov 3, 2009 at 12:15 PM, Yan Gao <ygao [at] novell> wrote:
>> Hi again,
>>
>> Yan Gao wrote:
>>
>> XML sample:
>> ..
>> <nodes>
>> <node id="yingying" type="normal" uname="yingying">
>> <utilization id="nodes-yingying-utilization">
>> <nvpair id="nodes-yingying-memory" name="memory" value="150"/>
>> <nvpair id="nodes-yingying-cpu" name="cpu" value="1"/>
>> </utilization>
>> </node>
>> </nodes>
>> ..
>> <resources>
>> <primitive class="ocf" id="Dummy0" provider="heartbeat" type="Dummy">
>> <utilization id="Dummy0-utilization">
>> <nvpair id="Dummy0-utilization-memory" name="memory" value="90"/>
>> <nvpair id="Dummy0-utilization-cpu" name="cpu" value="1"/>
>> </utilization>
>> </primitive>
>> <primitive class="ocf" id="Dummy1" provider="heartbeat" type="Dummy">
>> <utilization id="Dummy1-utilization">
>> <nvpair id="Dummy1-utilization-memory" name="memory" value="60"/>
>> <nvpair id="Dummy1-utilization-cpu" name="cpu" value="1"/>
>> </utilization>
>> </primitive>
>> </resources>
>> ..
>>
>> Please kindly review it...
>> Any suggestions are appreciated!
>
> Whats the behavior if a node has either no utilization block or no
> value for that attribute?
If so, it would be regarded that the node has no any capacity or no that
specific capacity.

> Can a resource with a utilization block still be placed there?
No, unless the utilization block is blank. As long as any attribute is set
in resource utilization, which means the resource requests some kind of
capacity, while a node has no that capacity, the node would not be considered.

A interesting case is, if a resource has no utilization block, it would be regarded
that the resource doesn't consume any capacity. so it could be placed on any node,
even the node has no utilization block (no any capacity).

Do you think the behavior is reasonable?

Thanks,
Yan
--
ygao [at] novell
Software Engineer
China Server Team, OPS Engineering

Novell, Inc.
Making IT Work As One™

_______________________________________________
Pacemaker mailing list
Pacemaker [at] oss
http://oss.clusterlabs.org/mailman/listinfo/pacemaker

misch at multinet

Nov 5, 2009, 11:31 PM

Post #17 of 44 (1395 views)
Permalink

Re: Resource capacity limit [In reply to]

Am Donnerstag, 5. November 2009 21:37:23 schrieb Andrew Beekhof:
> On Thu, Nov 5, 2009 at 8:50 PM, Michael Schwartzkopff <misch [at] multinet>
wrote:
> > Hi,
> >
> > on the list was a discussion about resource capacity limits. Yan Gao also
> > implemented it.
> >
> > As far as I understood the discussion the solution is to attach nodes and
> > resources capacity limits. Resources are distributed on the nodes of a
> > cluster according to its capacatiy needs. They would be migrated or
> > shutdown if the capacity limits on the node are not met.
> >
> > My question is: Can the capacity figures of the resources be made
> > dynamically?
>
> You can, but you probably don't want to.
> For example, free RAM and CPU load are two things that absolutely make
> no sense to include in such calculations.
>
> Consider how it works:
>
> - The node starts and reports 2Gb of RAM
> - We place a service there that reserves 512Mb
> - The cluster knows there is 1.5Gb remaining
> - We place two more services there that also reserve 512Mb each
>
> If the amount of RAM at the beginning was the amount free, then when
> you updated it to be 512Mb the PE would run and stop two of the
> resources!

Stop. I do not want to make the capacity of the nodes dynamically but the
actual resource consumption of the resources (i.e the database).

It sometimes happens that resource (i.e. RAM) consumption of resources varies
and I wanted to make that number dynamical. So after some kind of damping the
node that started swapping out the resources could migrate resources to a node
with free capacities.

> You always want to feed the cluster the total amount of RAM installed
> which, at most, you'd query when the cluster starts on that node.

The amount of capacity of any resource (RAM, CPU, ...) of a node should be
fixed. That makes sense because a node does not get more resources if switched
on.

Greetings,

Michael.

_______________________________________________
Pacemaker mailing list
Pacemaker [at] oss
http://oss.clusterlabs.org/mailman/listinfo/pacemaker

andrew at beekhof

Nov 5, 2009, 11:49 PM

Post #18 of 44 (1394 views)
Permalink

Re: Resource capacity limit [In reply to]

On Fri, Nov 6, 2009 at 8:31 AM, Michael Schwartzkopff <misch [at] multinet> wrote:
> Am Donnerstag, 5. November 2009 21:37:23 schrieb Andrew Beekhof:
>> On Thu, Nov 5, 2009 at 8:50 PM, Michael Schwartzkopff <misch [at] multinet>
> wrote:
>> > Hi,
>> >
>> > on the list was a discussion about resource capacity limits. Yan Gao also
>> > implemented it.
>> >
>> > As far as I understood the discussion the solution is to attach nodes and
>> > resources capacity limits. Resources are distributed on the nodes of a
>> > cluster according to its capacatiy needs. They would be migrated or
>> > shutdown if the capacity limits on the node are not met.
>> >
>> > My question is: Can the capacity figures of the resources be made
>> > dynamically?
>>
>> You can, but you probably don't want to.
>> For example, free RAM and CPU load are two things that absolutely make
>> no sense to include in such calculations.
>>
>> Consider how it works:
>>
>> - The node starts and reports 2Gb of RAM
>> - We place a service there that reserves 512Mb
>> - The cluster knows there is 1.5Gb remaining
>> - We place two more services there that also reserve 512Mb each
>>
>> If the amount of RAM at the beginning was the amount free, then when
>> you updated it to be 512Mb the PE would run and stop two of the
>> resources!
>
> Stop. I do not want to make the capacity of the nodes dynamically but the
> actual resource consumption of the resources (i.e the database).

Ah, ok, that makes more sense.
Well like any part of the configuration it can be changed at any time.
What we don't have though is a nice cli tool like crm_attribute for doing so.

>
> It sometimes happens that resource (i.e. RAM) consumption of resources varies
> and I wanted to make that number dynamical. So after some kind of damping the
> node that started swapping out the resources could migrate resources to a node
> with free capacities.
>
>> You always want to feed the cluster the total amount of RAM installed
>> which, at most, you'd query when the cluster starts on that node.
>
> The amount of capacity of any resource (RAM, CPU, ...) of a node should be
> fixed. That makes sense because a node does not get more resources if switched
> on.
>
> Greetings,
>
> Michael.
>
>
> _______________________________________________
> Pacemaker mailing list
> Pacemaker [at] oss
> http://oss.clusterlabs.org/mailman/listinfo/pacemaker
>

_______________________________________________
Pacemaker mailing list
Pacemaker [at] oss
http://oss.clusterlabs.org/mailman/listinfo/pacemaker

dejanmm at fastmail

Nov 6, 2009, 12:49 AM

Post #19 of 44 (1394 views)
Permalink

Re: Resource capacity limit [In reply to]

Hi,

On Thu, Nov 05, 2009 at 07:28:16PM +0100, Andrew Beekhof wrote:
> On Thu, Nov 5, 2009 at 5:25 PM, Dejan Muhamedagic <dejanmm [at] fastmail> wrote:
> > Hi,
> >> Which reminds me, I need to get devel sorted out...
> >
> > While you're at it, perhaps it would be good to rethink the
> > release policy. For me in particular it would be great to know at
> > least one week in advance when there'll be a release. For the
> > general public as well, since they'll have a chance to do some
> > testing of the new code.
>
> The idea is that people can always pull from stable-1.0, in theory it
> should never be broken.
> If you're in the middle of stuff, keep it locally until you're done.
>
> Generally though, I start testing on the 15th of every month.
> I thought I said that somewhere... I know the releases page indicates
> the month (if its delayed as it was due to my move).
>
> But I'm thinking of moving to a bi-monthly cycle. Thoughts?

Agreed. In principle, somewhat slower release process should
result in better releases.

> > I know that you do test before
> > releasing, but the more people test in various environments, the
> > more bugs found. Also, it may be good to introduce and announce
> > the feature freeze point, after which only bug fixes will be
> > accepted.
>
> Well in theory that point is x.y.0
> I've been turning a blind eye to your changes in the shell because its
> still very immature (I don't mean that negatively, its just new code).

That should of course change as soon as the shell supports all
CIB constructs (which is not far away). But we all understood
that it made no sense to keep those changes out :)

Thanks,

Dejan

> Though I've a history of allowing isolated, non-invasive features if
> we've not yet planned the next stable series (basically what happened
> for the node health stuff from IBM). Its a case-by-case thing, but
> I'd agree that we could do with documenting this.
>
> _______________________________________________
> Pacemaker mailing list
> Pacemaker [at] oss
> http://oss.clusterlabs.org/mailman/listinfo/pacemaker

_______________________________________________
Pacemaker mailing list
Pacemaker [at] oss
http://oss.clusterlabs.org/mailman/listinfo/pacemaker

andrew at beekhof

Nov 6, 2009, 3:28 AM

Post #20 of 44 (1396 views)
Permalink

Re: Resource capacity limit [In reply to]

On Fri, Nov 6, 2009 at 9:49 AM, Dejan Muhamedagic <dejanmm [at] fastmail> wrote:
> Hi,
>
> On Thu, Nov 05, 2009 at 07:28:16PM +0100, Andrew Beekhof wrote:
>> But I'm thinking of moving to a bi-monthly cycle. �Thoughts?
>
> Agreed. In principle, somewhat slower release process should
> result in better releases.

Actually thats not the rationale behind it at all.
I don't believe additional time would have improved the quality.
CTS passed, regression tests passed and no-one tests until its released...
At least releasing often means that we see the next set of bugs sooner.

The main reason is that if one week out of every 4 is consumed by
testing, and another one for waiting for the build service, then thats
not much time to actually get any work done in between.

Thankfully the build service is out of the equation now so it takes at
most a day to rebuild.
But still, I think we're at the point where Pacemaker is stable enough
that we don't need releases so often.

>> Well in theory that point is x.y.0
>> I've been turning a blind eye to your changes in the shell because its
>> still very immature (I don't mean that negatively, its just new code).
>
> That should of course change as soon as the shell supports all
> CIB constructs (which is not far away). But we all understood
> that it made no sense to keep those changes out :)

Exactly. And we've basically been learning about what works and what
doesn't as feedback comes in.
I think, when I write this up, that we should retain the ability to
designate some pieces as "experimental" and that those can continue to
be updated.

_______________________________________________
Pacemaker mailing list
Pacemaker [at] oss
http://oss.clusterlabs.org/mailman/listinfo/pacemaker

r.bhatia at ipax

Nov 6, 2009, 3:44 AM

Post #21 of 44 (1398 views)
Permalink

Re: Resource capacity limit [In reply to]

On 11/06/2009 12:28 PM, Andrew Beekhof wrote:
> Actually thats not the rationale behind it at all.
> I don't believe additional time would have improved the quality.
> CTS passed, regression tests passed and no-one tests until its released...
not too true - when there have been issues in the past, i used to
"test" and even deploy interim builds to avoid my cluster freaking out
:)

> At least releasing often means that we see the next set of bugs sooner.
agreed

> The main reason is that if one week out of every 4 is consumed by
> testing, and another one for waiting for the build service, then thats
> not much time to actually get any work done in between.
>
> Thankfully the build service is out of the equation now so it takes at
> most a day to rebuild.
> But still, I think we're at the point where Pacemaker is stable enough
> that we don't need releases so often.
agreed for the core parts. but the cli might be worth updating more
often.

cheers,
raoul
--
____________________________________________________________________
DI (FH) Raoul Bhatia M.Sc. email. r.bhatia [at] ipax
Technischer Leiter

IPAX - Aloy Bhatia Hava OEG web. http://www.ipax.at
Barawitzkagasse 10/2/2/11 email. office [at] ipax
1190 Wien tel. +43 1 3670030
FN 277995t HG Wien fax. +43 1 3670030 15
____________________________________________________________________

_______________________________________________
Pacemaker mailing list
Pacemaker [at] oss
http://oss.clusterlabs.org/mailman/listinfo/pacemaker

andrew at beekhof

Nov 6, 2009, 3:45 AM

Post #22 of 44 (1392 views)
Permalink

Re: Resource capacity limit [In reply to]

On Fri, Nov 6, 2009 at 8:27 AM, Yan Gao <ygao [at] novell> wrote:
> Hi Andrew,
>
> Andrew Beekhof wrote:

>> Whats the behavior if a node has either no utilization block or no
>> value for that attribute?
> If so, it would be regarded that the node has no any capacity or no that
> specific capacity.
>
>> Can a resource with a utilization block still be placed there?
> No, unless the utilization block is blank. As long as any attribute is set
> in resource utilization, which means the resource requests some kind of
> capacity, while a node has no that capacity, the node would not be considered.
>
> A interesting case is, if a resource has no utilization block, it would be regarded
> that the resource doesn't consume any capacity. so it could be placed on any node,
> even the node has no utilization block (no any capacity).
>
> Do you think the behavior is reasonable?

I think so.

I've been thinking about this more and while this will work, I think
we can make it better.

I'd like to see the while-block from native_color() be a function that
is called from native_assign_node().
And instead of a limit-utilization option, we'd have
placement-strategy=(default|utilization|minimal)

Default ::= what we do now
Utilization ::= what you've implemented
Minimal ::= what you've implemented _without_ the load balancing we
currently do.

(Maybe the names could be improved, but hopefully you get the idea).

The last one is interesting because it allows us to concentrate
services on the minimum number of required nodes (and potentially
power some of the others down).

_______________________________________________
Pacemaker mailing list
Pacemaker [at] oss
http://oss.clusterlabs.org/mailman/listinfo/pacemaker

ygao at novell

Nov 8, 2009, 8:53 PM

Post #23 of 44 (1363 views)
Permalink

Re: Resource capacity limit [In reply to]

Hi,
Andrew Beekhof wrote:
>
> I've been thinking about this more and while this will work, I think
> we can make it better.
>
> I'd like to see the while-block from native_color() be a function that
> is called from native_assign_node().
Okay.

> And instead of a limit-utilization option, we'd have
> placement-strategy=(default|utilization|minimal)
>
> Default ::= what we do now
> Utilization ::= what you've implemented
> Minimal ::= what you've implemented _without_ the load balancing we
> currently do.
>
> (Maybe the names could be improved, but hopefully you get the idea).
Great! There would be more policy options for users.

>
> The last one is interesting because it allows us to concentrate
> services on the minimum number of required nodes (and potentially
> power some of the others down).
Right. I'll look it into.

Thanks,
Yan

--
ygao [at] novell
Software Engineer
China Server Team, OPS Engineering

Novell, Inc.
Making IT Work As One™

_______________________________________________
Pacemaker mailing list
Pacemaker [at] oss
http://oss.clusterlabs.org/mailman/listinfo/pacemaker

misch at multinet

Nov 9, 2009, 1:52 AM

Post #24 of 44 (1362 views)
Permalink

Re: Resource capacity limit [In reply to]

Am Freitag, 6. November 2009 08:49:06 schrieb Andrew Beekhof:
> On Fri, Nov 6, 2009 at 8:31 AM, Michael Schwartzkopff <misch [at] multinet>
wrote:
> > Am Donnerstag, 5. November 2009 21:37:23 schrieb Andrew Beekhof:
> >> On Thu, Nov 5, 2009 at 8:50 PM, Michael Schwartzkopff
> >> <misch [at] multinet>
> >
> > wrote:
> >> > Hi,
> >> >
> >> > on the list was a discussion about resource capacity limits. Yan Gao
> >> > also implemented it.
> >> >
> >> > As far as I understood the discussion the solution is to attach nodes
> >> > and resources capacity limits. Resources are distributed on the nodes
> >> > of a cluster according to its capacatiy needs. They would be migrated
> >> > or shutdown if the capacity limits on the node are not met.
> >> >
> >> > My question is: Can the capacity figures of the resources be made
> >> > dynamically?
> >>
> >> You can, but you probably don't want to.
> >> For example, free RAM and CPU load are two things that absolutely make
> >> no sense to include in such calculations.
> >>
> >> Consider how it works:
> >>
> >> - The node starts and reports 2Gb of RAM
> >> - We place a service there that reserves 512Mb
> >> - The cluster knows there is 1.5Gb remaining
> >> - We place two more services there that also reserve 512Mb each
> >>
> >> If the amount of RAM at the beginning was the amount free, then when
> >> you updated it to be 512Mb the PE would run and stop two of the
> >> resources!
> >
> > Stop. I do not want to make the capacity of the nodes dynamically but the
> > actual resource consumption of the resources (i.e the database).
>
> Ah, ok, that makes more sense.
> Well like any part of the configuration it can be changed at any time.
> What we don't have though is a nice cli tool like crm_attribute for doing
> so.

I know the crm_attribute command. Of course someone can always do somw
scripting to acchieve the aim I described above.

I just think it would be cool solution to make the cluster itself doing the
work if condfigured to do so. So the CRM (or the RAs) should have the abaility
to monitor the resource consumption of resources dynamically. This automatism
would make the live of admins much easier and they would not be forced to do
the scripting by their own.

Greetings,

--
Dr. Michael Schwartzkopff
MultiNET Services GmbH
Addresse: Bretonischer Ring 7; 85630 Grasbrunn; Germany
Tel: +49 - 89 - 45 69 11 0
Fax: +49 - 89 - 45 69 11 21
mob: +49 - 174 - 343 28 75

mail: misch [at] multinet
web: www.multinet.de

Sitz der Gesellschaft: 85630 Grasbrunn
Registergericht: Amtsgericht M�nchen HRB 114375
Gesch�ftsf�hrer: G�nter Jurgeneit, Hubert Martens

---

PGP Fingerprint: F919 3919 FF12 ED5A 2801 DEA6 AA77 57A4 EDD8 979B
Skype: misch42

_______________________________________________
Pacemaker mailing list
Pacemaker [at] oss
http://oss.clusterlabs.org/mailman/listinfo/pacemaker

lmb at suse

Nov 11, 2009, 4:36 AM

Post #25 of 44 (1354 views)
Permalink

Re: Resource capacity limit [In reply to]

On 2009-11-05T14:45:36, Andrew Beekhof <andrew [at] beekhof> wrote:

> Lastly, I would really like to defer this for 1.2
> I know I've bent the rules a bit for 1.0 in the past, but its really
> late in the game now.

Personally, I think the Linux kernel model works really well. ie, no
"major releases" any more, but bugfixes and features alike get merged
over time and constantly.

With increasing coverage of the regression tests, the existing
functionality is protected; which is really the important bit. This
encourages a smooth forward transition.

There's a point in having a devel tree (similar to linux-next) before
merging back major features into the trunk, but I don't really subscribe
to the major version flow. That just means that there's a lot of testing
that needs to happen at once, which means more things slip through than
with incremental testing. In my experience, major updates make them a
royal PITA for users.

Just my few euro cents ;-)

Regards,
Lars

--
Architect Storage/HA, OPS Engineering, Novell, Inc.
SUSE LINUX Products GmbH, GF: Markus Rex, HRB 16746 (AG N�rnberg)
"Experience is the name everyone gives to their mistakes." -- Oscar Wilde

_______________________________________________
Pacemaker mailing list
Pacemaker [at] oss
http://oss.clusterlabs.org/mailman/listinfo/pacemaker

1 2

View All

Index | Next | Previous | View Threaded

Interested in having your list archived? Contact Gossamer Threads