Copyright © 2015-2016 Open Baton. Licensed under Apache v2 License.
The Open Baton Fault Management System (openbaton-fms) is an external component of the NFVO Open Baton. It manages the alarms coming from the VIM and executes actions through the NFVO.
The openbaton-fms is implemented as a Spring Boot application.
It runs as an external component and communicate with the NFVO via Open Baton's SDK and RabbitMQ.
Before starting this component you need to ensure that the technical requirements are met and proceed with the installation guide.
- Preconfigured Open Baton environment (NFVO, VNFMs, VIM drivers)
- Running Zabbix server
- Preconfigured and running Zabbix plugin (see the doc of Zabbix plugin)
- Mysql server installed and running
There are two options available for the installation of the openbaton-fms. Installation based on the Debian package or on the source code (which is suggested for development).
When using the Debian package you need to add the apt-repository of Open Baton to your local environment with the following command if not yet done:
wget -O - http://get.openbaton.org/keys/openbaton.public.key | apt-key add -
echo "deb http://get.openbaton.org/repos/openbaton/<dist>/release <dist> main" >> /etc/apt/sources.listReplace <dist> with trusty, xenial or jessie depending on the distribution you are using.
Once you added the repo to your environment you should update the list of repos by executing:
apt-get updateNow you can install the openbaton-fms by executing:
apt-get install openbaton-fmsNote: During the installation you will be prompted for entering the IP address of the host of the Zabbix Plugin, make sure this IP can be reached by the Zabbix Server host.
After the installation, the openbaton-fms will be already configured and running.
The latest stable version of the Open Baton FM System can be cloned from this repository by executing the following command:
git clone https://github.com/openbaton/openbaton-fms.gitOnce this is done, go inside the cloned folder and install the project as done below:
./gradlew installDistThe installation from the source code requires manual configuration before running the openbaton-fms, which is explained in the following section.
This chapter describes what needs to be done before starting the Open Baton FM System.
The configuration file must be copied to /etc/openbaton/openbaton-fms.properties by executing the following command from inside the repository folder:
cp src/main/resources/application.properties /etc/openbaton/openbaton-fms.propertiesIn the following sections, we will refer to this configuration file as openbaton-fms.properties.
In order to create the database be sure you have installed mysql server as already mentioned in the requirements section. You need root access to mysql-server in order to create a new database called faultmanagement. Once you access into mysql, execute the following operation:
create database faultmanagement;Once the database has been created, you should create a user which will be used by the FM system to access and store data on the database. If you decide to use the root user you can skip this step, but you need to modify the openbaton-fms.properties file accordingly as defined in the next section.
By default username and password are set with the following values in the openbaton-fms.properties properties file (see next section if you plan to use a different user and password):
- username=fmsuser
- password=changeme
Grant the access to the database "faultmanagement", to the user, running the following command:
GRANT ALL PRIVILEGES ON faultmanagement.* TO fmsuser@'%' IDENTIFIED BY 'changeme';(optional) Modify openbaton-fms.properties file in order to use different credentials for the database
In order to use different credentials, you need to modify the following properties:
# DB properties
spring.datasource.username=fmsuser
spring.datasource.password=changemeIn case your database is running remotely, you can specify a different host, instead of localhost, in the following property (be careful to have port 3306 open and accessible from remote):
spring.datasource.url=jdbc:mysql://localhost:3306/faultmanagementYou should update this file in order to make it work with your NFVO instance. Change the Open Baton related properties section:
################################################
####### Open Baton Related properties ##########
################################################
nfvo.ip=localhost
nfvo.port=8080
nfvo-usr=admin
nfvo-pwd=openbaton
nfvo.ssl.enabled=falseThe openbaton-fms authenticates to the NFVO through a service key which has to be set in the properties. You need to obtain the service key from the dashboard of the NFVO,
in particular you can set the service key with the following instructions:
- Go to Admin->Services
- Click on "Enable a new Service"
- Input the name "fms"
- Click on Role
- Select "*" in the Project checkbox
- Click Save
- Open the downloaded file
- Copy the service key in the
openbaton-fms.properties:
fms.service.key=<SERVICE KEY>How to start the Open Baton FM System depends of the way you installed this component.
If you installed the Open Baton FM System with the Debian packages you can start it with the following command:
openbaton-fms startFor stopping it you can just type:
openbaton-fms stopIf you are using the source code you can start the Open Baton FM System easily with the following command from inside the repository folder:
cd build/install/openbaton-fms
./bin/openbaton-fms startFor stopping you can use:
./bin/openbaton-fms stopNote: Since openbaton-fms subscribes to specific events towards the NFVO, you should take care about that the NFVO is already running when starting the openbaton-fms.
Open Baton FMS is a rule-driven tool. The rules define when to generate an alarm and how to react. The rule for generating the alarm is called fault management policy (see the next section). The rule for defining how to react upon alarms is a Drools Rule. Once such rules are in place, Open Baton FM follows the following workflow.
The actions are listed below:
| ACTION | DESCRIPTION |
|---|---|
| Heal | The VNFM executes the scripts in the Heal lifecycle event (in the VNFD). The message contains the cause of the fault, which can be used in the scripts. |
| Switch to stanby VNFC (Stateless) | If the VDU requires redoundancy active-passive, there will be a component VNFC* in standby mode. This action consists in: activate the VNFC*, route all signalling and data flow(s) for VNFC to VNFC*, deactivate VNFC |
The fault management policy needs to be present in the VNFD, in particular in the VDU. This is an example of fault management policy:
"fault_management_policy":[
{
"name":"web server not available",
"isVNFAlarm": true,
"criteria":[
{
"parameter_ref":"net.tcp.listen[80]",
"function":"last()",
"vnfc_selector":"at_least_one",
"comparison_operator":"=",
"threshold":"0"
}
],
"period":5,
"severity":"CRITICAL"
}
]The parameter isVNFAlarm=true tells the openbaton-fms that the alarm is of type VNF, and at default it will execute the HEAL action.
Description of the fault management policy:
| Property | Description |
|---|---|
| name | The name of the fault management policy. |
| isVNFAlarm | if the alarm is of type VNF |
| criteria | The criteria defines a threshold on a monitoring paramenter. When the threshold is crossed an alarm is fired |
| period | The criteria is checked every "period" seconds |
| severity | severity of the alarm |
Description of the criteria:
| Property | Description |
|---|---|
| parameter_ref | Reference to a monitoring parameter in the VDU. (see below how to define monitoring parameters) |
| function | The function to apply to the parameter. ( last(0) means the last value available of the parameter). Since currently only Zabbix is supported, look at the Zabbix documentation for the all available functions. |
| vnfc_selector | select if the criteria is met when all VNFC components cross the thresold (all) or at least one (at_least_one) |
| comparison_operator | comparison operator for the threshold |
| threshold | value of the threshold to compare against the parameter_ref value |
In order to refer a monitoring parameter with the property parameter_ref, it needs to be present in the vdu:
"monitoring_parameter":[
"agent.ping",
"net.tcp.listen[5001]",
"system.cpu.load[all,avg5]",
"vfs.file.regmatch[/var/log/app.log,Exception]"
]You can specify every parameter available for the Zabbix Agent.
The openbaton-fms as soon as it gets an alarm from the VIM, it checks if the alarm is referred to a VNF ("isVNFAlarm": true) and it sends the Heal VNF message to the NFVO which forwards it to the respective VNFM.
The VNFM will then execute, in the failed VNFC, the scripts in the HEAL lifecycle event.
Here an example of the heal script you can use:
#!/bin/bash
case "$cause" in
("serviceDown")
echo "Apache is down, let's try to restart it..."
service apache2 restart
if [ $? -ne 0 ]; then
echo "ERROR: the Apache service is not started"
exit 1
fi
echo "The Apache service is running again!"
;;
*) echo "The cause $cause is unknown"
exit 2
;;
esacThe variable $cause is specified in the Drools rule. In our case is "serviceDown" and we try to restart the Apache server.
The Open Baton FM is a rule-based system. Such rules are specified in Drools language and processed by the Drools engine in the Open Baton FM. An example rule is the following:
rule "Save a VNFAlarm"
when
vnfAlarm : VNFAlarm()
then
VNFAlarm alarm = vnfAlarmRepository.save(vnfAlarm);
logger.debug("Saved VnfAlarm: "+alarm);
end
This rule saves a VNFAlarm in the database. The following rule executes the HEAL action once a VNFAlarm is received.
rule "Got a critical VNF Alarm and execute the HEAL action"
when
vnfAlarm : VNFAlarm( alarmState == AlarmState.FIRED, perceivedSeverity == PerceivedSeverity.CRITICAL)
then
//Get the vnfr
VirtualNetworkFunctionRecord vnfr = nfvoRequestorWrapper.getVirtualNetworkFunctionRecord(vnfAlarm.getVnfrId());
//Get the vnfc failed (assuming only one vnfc is failed)
VNFCInstance vnfcInstance = nsrManager.getVNFCInstanceFromVnfr(vnfr,vnfAlarm.getVnfcIds().iterator().next());
logger.info("(VNF LAYER) A CRITICAL alarm is received by the vnfc: "+vnfcInstance.getHostname());
//Get the vdu of the failed VNFC
VirtualDeploymentUnit vdu = nfvoRequestorWrapper.getVDU(vnfr,vnfcInstance.getId());
logger.info("Heal fired!");
highAvailabilityManager.executeHeal("serviceDown",vnfr.getParent_ns_id(),vnfr.getId(),vdu.getId(),vnfcInstance.getId());
//Insert a new recovery action
RecoveryAction recoveryAction= new RecoveryAction(RecoveryActionType.HEAL,vnfr.getEndpoint(),"");
recoveryAction.setStatus(RecoveryActionStatus.IN_PROGRESS);
insert(recoveryAction);
end
The Switch to Standby action can be performed by the openbaton-fms once a VNFC in standby is present in the VNF. It consists in switching the service from the failing VNFC to the standby VNFC automatically.
In order to have a VNFC in standby, such information must be included in the VNFD, in particular in the VDU, as the following:
"high_availability":{
"resiliencyLevel":"ACTIVE_STANDBY_STATELESS",
"redundancyScheme":"1:N"
}
This information will be processed by the openbaton-fms which will create a VNFC instance in standby.
Then in a Drools rule this action can be called as following:
highAvailabilityManager.switchToRedundantVNFC(failedVnfcInstance,vnfr,vdu);
You can follow the tutorial at this page: http://openbaton.github.io/documentation/fms-sipp-tutorial/
Issues and bug reports should be posted to the GitHub Issue Tracker of this project
OpenBaton is an open source project providing a comprehensive implementation of the ETSI Management and Orchestration (MANO) specification.
Open Baton is a ETSI NFV MANO compliant framework. Open Baton was part of the OpenSDNCore (www.opensdncore.org) project started almost three years ago by Fraunhofer FOKUS with the objective of providing a compliant implementation of the ETSI NFV specification.
Open Baton is easily extensible. It integrates with OpenStack, and provides a plugin mechanism for supporting additional VIM types. It supports Network Service management either using a generic VNFM or interoperating with VNF-specific VNFM. It uses different mechanisms (REST or PUB/SUB) for interoperating with the VNFMs. It integrates with additional components for the runtime management of a Network Service. For instance, it provides autoscaling and fault management based on monitoring information coming from the the monitoring system available at the NFVI level.
The Source Code of the other Open Baton projects can be found here and the documentation can be found here .
Check the Open Baton Website Follow us on Twitter @openbaton.
Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.
You can contribute to the Open Baton community through bug-reports, bug-fixes, new code or new documentation. For contributing to the Open Baton community, drop a post to the [openbaton-mail][Open Baton Public Mailing List] providing full information about your contribution and its value. In your contributions, you must comply with the following guidelines
- You must specify the specific contents of your contribution either through a detailed bug description, through a pull-request or through a patch.
- You must specify the licensing restrictions of the code you contribute.
- For newly created code to be incorporated in the Open Baton code-base, you must accept Open Baton to own the code copyright, so that its open source nature is guaranteed.
- You must justify appropriately the need and value of your contribution. The Open Baton project has no obligations in relation to accepting contributions from third parties.
- The Open Baton project leaders have the right of asking for further explanations, tests or validations of any code contributed to the community before it being incorporated into the Open Baton code-base. You must be ready to addressing all these kind of concerns before having your code approved.
The Open Baton project provides community support through the Open Baton Public Mailing List and through StackOverflow using the tags openbaton.