Policies are highly reusable as their inputs, thresholds and targets are customizable. Config key details for each policy can be found in the Catalog in the Brooklyn UI.
HA/DR and Scaling Policies
An AutoScaler policy can take any sensor as a metric, have its watermarks tuned live, and target any resizable entity - be it an application server managing how many instances it handles, or a tier managing global capacity.
e.g. if the average request per second across a cluster of Tomcat servers goes over the high watermark, it will resize the cluster to bring the average back to within the watermarks.
brooklyn.policies: - type: org.apache.brooklyn.policy.autoscaling.AutoScalerPolicy brooklyn.config: metric: webapp.reqs.perSec.perNode metricUpperBound: 3 metricLowerBound: 1 resizeUpStabilizationDelay: 2s resizeDownStabilizationDelay: 1m maxPoolSize: 3
Attaches to a SoftwareProcess or to anything Startable which emits
ha.entityFailed on failure
(or other configurable sensor), and invokes
restart on that failure.
If there is a subsequent failure within a configurable time interval or if the restart fails,
this gives up and emits
ha.entityFailed.restart for other policies to act upon or for manual intervention.
brooklyn.policies: - type: org.apache.brooklyn.policy.ha.ServiceRestarter brooklyn.config: failOnRecurringFailuresInThisDuration: 5m
The ServiceReplacer attaches to a DynamicCluster and replaces a failed member in response to
ha.entityFailed (or other configurable sensor) as typically emitted by the ServiceFailureDetector enricher.
The introduction to policies shows a worked example of this policy in use.
The ServiceFailureDetector enricher detects problems and fires an
ha.entityFailed (or other configurable sensor)
for use by ServiceRestarter and ServiceReplacer.
The introduction to policies shows a worked
example of this in use.
The SshMachineFailureDetector is an HA policy for monitoring an SshMachine, emitting an event if the connection is lost/restored.
The ConnectionFailureDetector is an HA policy for monitoring an HTTP connection, emitting an event if the connection is lost/restored.
Primary Election / Failover Policies
There are a collection of policies, enrichers, and effectors to assist with common failover scenarios and more generally anything which requires the election and re-election of a primary member.
These can be used for:
- Nominating one child among many to be noted as a primary via a sensor (simply add the
- Allowing preferences for such children to be specified (via
- Causing the primary to change if the current primary goes down or away
demoteeffectors to be invoked on the appropriate nodes when the primary is elected/changed (with the parent reset to
STARTINGwhile this occurs)
- Mirroring sensors and optionally effectors from the primary to the parent
A simple example is as follows, deploying two
item entities with one designated as primary
main.uri sensor published at the root. If "Preferred Item" fails, "Failover Item"
will be become the primary. Any
demote effector on "Preferred Item" and any
on "Failover Item" will be invoked on failover.
brooklyn.policies: - type: org.apache.brooklyn.policy.failover.ElectPrimaryPolicy brooklyn.config: # `best` will cause failback to occur automatically when possible; could use `failover` instead primary.selection.mode: best propagate.primary.sensors: [ main.uri ] brooklyn.enrichers: - # this enricher will cause the parent to report as failed if there is no primary type: org.apache.brooklyn.policy.failover.PrimaryRunningEnricher services: - type: item name: Preferred Item brooklyn.config: ha.primary.weight: 1 - type: item name: Failover Item
The ElectPrimaryPolicy acts to keep exactly one of its children or members as primary, promoting and demoting them when required.
A simple use case is where we have two children, call them North and South, and we wish for North to be primary. If North fails, however, we want to promote and fail over to South. This can be done by:
- adding this policy at the parent
- optionally defining
promoteon North and South (if action is required there to promote it)
- observing the
primarysensor to see which is primary
- optionally setting
propagate.primary.sensor: main.urito publish
main.urifrom whichever of North or South is active
- optionally setting
primary.selection.mode: bestto switch back to North if it comes back online
The policy works by listening for service-up changes in the target pool (children or members) and listening for
ha.primary.weight sensor values from those elements. On any change, it invokes an effector to perform the primary election. By default, the effector invoked is
electPrimary, but this can be changed with the
primary.election.effector config key. If this effector does not exist, the policy will add a default behaviour using
ElectPrimaryEffector. Details of the election are described in that effector, but to summarize, it will find an appropriate primary from the target pool and publish a sensor indicating who the new primary is. Optionally it can invoke
demote on the relevant entities.
primary.* parameters accepted by that effector can be defined on the policy and will be passed to the effector, along with an
event parameter indicating the sensor which triggered the election.
quorum.running is set on the entity, both will be set to a constant 1.
This effector will scan candidates among children or members to determine which should be noted as "primary".
The primary is selected from service-up candidates based on a numeric weight as a sensor or config on the candidates (
ha.primary.weight, unless overridden), with higher weights being preferred and negative indicating not permitted.
In the case of ties, or a new candidate emerging with a weight higher than a current healthy primary, behaviour can be configured with
If there is a primary and it is unchanged, the effector will end.
If a new primary is detected, the effector will:
set the local entity to the STARTING state
clear any "primary-election" problem
set service up true
cancel any other ongoing promote calls, and if there is an ongoing demote call on the entity being promoted, cancel that also
set the local entity to the RUNNING state
If no primary can be found, the effector will:
add a "primary-election" problem so that service state logic, if applicable, will know that the entity is unhealthy
demote any old primary
set service up false
if the local entity is expected to be RUNNING, it will set actual state to ON_FIRE
if the local entity has no expectation, it will set actual state to STOPPED
More details of behaviour in edge conditions can be seen and set via the parameters on this effector.
primary.target.mode: where should the policy look for primary candidates; one of 'children', 'members', or 'auto' (members if it has members and no children)
primary.selection.mode: under what circumstances should the primary change:
failoverto change only if an existing primary is unhealthy,
bestto change so one with the highest weight is always selected, or
strictto act as
bestbut fail if several advertise the highest weight (for use when the weight sensor is updated by the nodes and should tell us unambiguously who was elected)
primary.stopped.wait.timeout: if the highest-ranking primary is stopped (but not failed), the effector will wait this long for it to be starting before picking a less highly-weighted primary; default 3s, typically long enough to avoid races where multiple children are started concurrently but they complete extremely quickly and one completes before a better one starts
primary.starting.wait.timeout: if the highest-ranking primary is starting, the effector will wait this long for it to be running before picking a less highly-weighted primary (or in the case of
strictbefore failing if there are ties); default 5m, typically long enough to avoid races where multiple children are started and a sub-optimal one comes online before the best one
primary.sensor.name: name to publish, defaulting to
primary.weight.name: config key or sensor to scan from candidate nodes to determine who should be primary
primary.promote.effector.name: effector to invoke on promotion, default
promoteand with no error if not present (but if set explicitly it will cause an error if not present)
primary.demote.effector.name: effector to invoke on demotion, default
demoteand with no error if not present (but if set explicitly it will cause an error if not present)
This adds service not-up and problems entries if the primary is not running, so that the parent will only be up/healthy if there is a healthy primary.
This allows selected sensors from the primary to be available at the parent. As the primary changes, the indicated sensors will be updated to reflect the values from the new primaries.
PeriodicEffectorPolicy calls an effector with a set of arguments at a specified time and date. The policy monitors the
sensor configured by
start.sensor and will only start when this is set to
true. The default sensor checked is
so that the policy will not execute the effector until the entity is started. The following example calls a
to resize a cluster up to 10 members at 8am and then down to 1 member at 6pm.
- type: org.apache.brooklyn.policy.action.PeriodicEffectorPolicy brooklyn.config: effector: resize args: desiredSize: 10 period: 1 day time: 08:00:00 - type: org.apache.brooklyn.policy.action.PeriodicEffectorPolicy brooklyn.config: effector: resize args: desiredSize: 1 period: 1 day time: 18:00:00
ScheduledEffectorPolicy calls an effector at a specific time. The policy monitors the sensor configured by
and will only execute the effector at the specified time if this is set to
There are two modes of operation, one based solely on policy configuration where the effector will execute at the time set
time key or after the duration set using the
wait key, or by monitoring sensors. The policy monitors the
scheduler.invoke.now sensor and will execute the effector immediately when its value changes to
scheduler.invoke.at sensor changes, it will set a time in the future when the effector should be executed.
The following example calls a
backup effector every night at midnight.
- type: org.apache.brooklyn.policy.action.ScheduledEffectorPolicy brooklyn.config: effector: backup time: 00:00:00
The LoadBalancingPolicy is attached to a pool of "containers", each of which can host one or more migratable "items". The policy monitors the workrates of the items and effects migrations in an attempt to ensure that the containers are all sufficiently utilized without any of them being overloaded.
Lifecycle and User Management Policies
The ConditionalSuspendPolicy will suspend and resume a target policy based on configured suspend and resume sensors.
The CreateUserPolicy Attaches to an Entity and monitors for the addition of a location to that entity, the policy then adds a new user to the VM with a randomly generated password, with the SSH connection details set on the entity as the createuser.vm.user.credentials sensor.