This puppet module is for configuring L2 and L3 network interfaces (e.g 802.1q vlans, access ports, NIC-bonding, assigning IP addresses, DHCP, and interfaces without IP addresses).
It works with standard Linux interfaces as well as with Open vSwitch interfaces. At the moment, it supports CentOS/RHEL 6.3+ and Ubuntu 12.04+. The module should have the same behavior across both operating systems.
WARNING!!! This is l23network v1.1, which contains some incompatibilities with earlier versions. Be careful.
Place this module at /etc/puppet/modules/l23network or in the directory where your puppet modules are stored.
Include the l23network module and initialize it. It is recommended to do this during an early puppet stage (before the main stage). E.g.
# Network configuration
stage { 'netconfig':
before => Stage['main'],
}
class { 'l23network':
use_ovs => true,
use_lnx => true,
stage => 'netconfig'
}
The 'l23network' class has the following parameters and default values:
class { 'l23network':
use_ovs => false,
use_lnx => true,
install_ovs => $use_ovs,
install_brtool => $use_lnx,
modprobe_bridge => $use_lnx,
install_bondtool => $use_lnx,
modprobe_bonding => $use_lnx,
install_vlantool => $use_lnx,
modprobe_8021q => $use_lnx,
install_ethtool => $use_lnx,
ovs_module_name => undef,
use_ovs_dkms_datapath_module => undef,
ovs_datapath_package_name => undef,
ovs_common_package_name => undef,
}
If you plan to use Open vSwitch you can enable it as follows:
class { 'l23network':
use_ovs => true,
}
The following L2 concepts are used:
- bridge - a virtual ethernet L2 switch. You can plug ports into a bridge.
- port - an interface that you plug into the bridge. A port may be a virtual or a physical interface.
These can be used as parameterized classes:
l23network::l2::bridge { 'br-mgmt': }
l23network::l2::port { 'eth0': bridge => 'br-mgmt' }
l23network::l2::bridge { 'br-ex': provider => ovs }
l23network::l2::port { 'eth1': bridge => 'br-ex' }
l23network::l2::port { 've0': bridge => 'br-ex' }
l23network::l2::port { 've1': bridge => 'br-ex' }
The following sections describe their usage in more detail.
This resource is for configuring bridges:
l23network::l2::bridge { 'br1':
ensure => present,
stp => true, # or false
vendor_specific => {
.....
},
provider => lnx,
}
Optional parameters:
- stp - enable/disable STP for the bridge
- bpdu_forward - enable/disable BPDU forwarding on the bridge
- bridge_id - bridge_id for the STP protocol
- vendor_specific - vendor_specific hash (see below)
- delay_while_up - delay, in seconds, after the interface comes up, which happens every time the nodes boots
This resource is for configuring ports with L2 options. To configure L3 options, use the l23network::l3::ifconfig resource.
l23network::l2::port { 'eth1':
mtu => 9000, # MTU value, unchanged if absent
onboot => true, # Whether port has UP state after setup or node boot
ethtool => {
.....
},
vendor_specific => {
.....
},
provider => lnx,
}
l23network::l2::port { 'eth1.101':
ensure => present,
bridge => 'br1', # port can be a member of a bridge
# If no value is given this property remains
# unchanged. If 'absent' is given the port will
# be excluded from any bridges
onboot => true,
delay_while_up => 10,
provider => lnx,
}
Alternative VLAN definition (not recommended for 'lnx' provider)
l23network::l2::port { 'vlan77':
vlan_id => 77,
vlan_dev => eth1,
provider => lnx,
}
This is a special type of port for bonding two or more interfaces. A detailed description of bonding is available here. If you plan to use LACP, we highly recommend not using OVS. We also recommend not inserting native linux bonds into OVS bridges. This case works, but leads to many complications when troubleshooting.
l23network::l2::bond { 'bond0':
interfaces => ['eth1', 'eth2'],
bridge => 'br0', # only required for OVS provider
mtu => 9000,
onboot => true,
bond_properties => { # bond configuration properties (see bonding.txt)
mode => '803.1ad',
lacp_rate => 'slow',
xmit_hash_policy => 'encap3+4'
},
interface_properties => { # config properties for included interfaces
ethtool => {
.....
},
},
vendor_specific => {
.....
},
delay_while_up => 45,
provider => lnx,
}
mode and xmit_hash_policy parameters have some differences depending on whether the provider is lnx or ovs:
For lnx provider mode can be:
- balance-rr (default)
- active-backup
- balance-xor
- broadcast
- 802.3ad
- balance-tlb
- balance-alb
For 802.3ad (LACP) with balance-xor, balance-tlb or balance-alb xmit_hash_policy should be defined as one of:
- layer2 (default)
- layer2+3
- layer3+4
- encap2+3
- encap3+4
For ovs provider mode can be:
- active-backup
- balance-slb (default)
- balance-tcp
If mode is balance-tcp, lacp should be set to 'active' or 'passive'.
The xmit_hash_policy parameter is not used for ovs bonds at all.
When the bond is created, it will also create the ports for the bond slaves. These ports will be created with the properties specified by the interface_properties parameter. For further flexibility, these ports can be created using the l23network::l2::port resource. In this case, do not use the interface_properties parameter.
When the mtu parameter is set on a bonded interface, the MTU will also be assigned to slave interfaces automatically.
It is recommended to use the delay_while_up parameter when configuring LACP bonds, because such bonds may take some time to settle.
For some providers (e.g. ovs), the bridge parameter is obligatory.
This resource is a patchcord for connecting two bridges. One architecture limitation is that two bridges can only be connected by one patchcord. The name for the patchcord interfaces is calculated automatically and cannot be changed.
The ovs provider can connect OVS-to-OVS, OVS-to-LNX and LNX-to-LNX bridges. You should always create the bridges before using this resource.
l23network::l2::patch { 'patch__br0--br1':
bridges => ['br0','br1'],
vendor_specific => {
.....
},
}
Each low-level patchcord resource l2::patch has its name in the following format: 'bridge__%bridge1%--%bridge2%', with the bridges in alphabetical order for all providers.
This resource also contains a 'bridges' property which is an array of the two bridge names. The order of the names depends on the provider implementation. For example, the ovs provider bridge names are listed in alphabetical order for OVS-to-OVS connections, and ovs-bridge is always listed first for OVS-to-LNX bridges.
Each L2::patch instance contains a read-only 'jacks' property, which is an array of two names of jacks, 'inserted' into each bridge. This property has the same ordering style as the 'bridges' property for each provider.
If a patchcord connects two different types of bridges, the 'cross' property will be set to 'true'.
To manage offloading and other ethtool options for any resources, it is possible to use the ethtool parameter. This is like a hash of hashes. Keys of the external hash map to section names from the ethtool manual. Each section name maps to an internal hash. Internal hashes are option to value mappings, where option names correspond to ethtool output option naming. For example, you can see list of offloading options by executing 'ethtool -k eth0'.
Ethtool options are pre-defined and stateful.
All implemented sections and options are listed below.
ethtool => {
offload => {
rx-checksumming => true or false,
tx-checksumming => true or false,
scatter-gather => true or false,
tcp-segmentation-offload => true or false,
udp-fragmentation-offload => true or false,
generic-segmentation-offload => true or false,
generic-receive-offload => true or false,
large-receive-offload => true or false,
rx-vlan-offload => true or false,
tx-vlan-offload => true or false,
ntuple-filters => true or false,
receive-hashing => true or false,
rx-fcs => true or false,
rx-all => true or false,
highdma => true or false,
rx-vlan-filter => true or false,
fcoe-mtu => true or false,
l2-fwd-offload => true or false,
loopback => true or false,
tx-nocache-copy => true or false,
tx-gso-robust => true or false,
tx-fcoe-segmentation => true or false,
tx-gre-segmentation => true or false,
tx-ipip-segmentation => true or false,
tx-sit-segmentation => true or false,
tx-udp_tnl-segmentation => true or false,
tx-mpls-segmentation => true or false,
tx-vlan-stag-hw-insert => true or false,
rx-vlan-stag-hw-parse => true or false,
rx-vlan-stag-filter => true or false,
},
settings => {
duplex => 'half',
mdix => off
}
}
This resource is for configuring IP addresses on an interface. Only L3 options. For configuring L2 options, use the L2 resources.
l23network::l3::ifconfig { 'eth1.101':
ensure => present,
ipaddr => ['192.168.10.3/24', '10.20.30.40/25'],
gateway => '192.168.10.1',
gateway_metric => 10, # different ifconfig resources should not have
# gateways with same metric
vendor_specific => {
.....
},
}
The option ipaddr may contain an array of IP addresses (even to configure a single IP address), 'dhcp', or 'none'. CIDR-notation is required for the IP address.
DHCP or address-less interfaces are configured as follows:
l23network::l3::ifconfig { 'eth2': ipaddr => 'dhcp' }
l23network::l3::ifconfig { 'eth3': ipaddr => 'none' }
The default gateway can be configured as follows:
l23network::l3::ifconfig { 'eth1':
ipaddr => ['192.168.2.5/24'],
gateway => '192.168.2.1',
gateway_metric => 10,
}
If gateway_metric is omitted, the gateway will be configured without a metric.
network_scheme is a hierarchical-based scheme to define a network topology for a host. In the following examples the yaml format is used.
The main idea is as follows:
- When we have an undeployed server we have a number of NICs. NICs, managed by puppet should be listed in the interfaces section.
- The result of our network configuration process is a network topology on the host with interfaces that are assigned IP addresses (or not). These are the endpoints.
- Interfaces become endpoints by successive transformations. Transformations are explained in the following document: Transformations. How they work.
Example of a typical network scheme:
---
network_scheme:
version: 1.1
provider: lnx
interfaces:
eth1:
mtu: 7777
eth2:
mtu: 9000
transformations:
- action: add-br
name: br1
- action: add-port
name: eth1
bridge: br1
- action: add-br
name: br-mgmt
- action: add-port
name: eth1.101
bridge: br-mgmt
- action: add-br
name: br-ex
- action: add-port
name: eth1.102
bridge: br-ex
- action: add-br
name: br-storage
- action: add-port
name: eth1.103
bridge: br-storage
- action: add-br
name: br-prv
provider: ovs
- action: add-port
name: eth2
bridge: br-prv
provider: ovs
endpoints:
br-mgmt:
IP:
- 192.168.101.3/24
gateway: 192.168.101.1
gateway-metric: 100
routes:
- net: 192.168.210.0/24
via: 192.168.101.1
- net: 192.168.211.0/24
via: 192.168.101.1
- net: 192.168.212.0/24
via: 192.168.101.1
br-ex:
gateway: 192.168.102.1
IP:
- 192.168.102.3/24
br-storage:
IP:
- 192.168.103.3/24
br-prv:
IP: none
roles:
management: br-mgmt
ceph: br-mgmt
private: br-prv
fw-admin: br1
ex: br-ex
floating: br-ex
storage: br-storage
Example of typical network scheme with bonding and offloads disabled:
---
network_scheme:
version: 1.1
provider: lnx
interfaces:
eth1:
mtu: 9000
eth2:
eth3:
transformations:
- action: add-br
name: br1
- action: add-port
bridge: br1
name: eth1
ethtool:
offload:
tcp-segmentation-offload: off
udp-fragmentation-offload: off
generic-segmentation-offload: off
generic-receive-offload: off
large-receive-offload: off
- action: add-br
name: br2
- action: add-bond
name: bond23
bridge: br2
interfaces:
- eth2
- eth3
mtu: 9000
interface_properties:
ethtool:
offload:
tcp-segmentation-offload: off
udp-fragmentation-offload: off
bond_properties:
mode: balance-rr
xmit_hash_policy: encap3+4
updelay: 10
downdelay: 40
use_carrier: 0
- action: add-br
name: br-mgmt
- action: add-port
name: bond23.101
bridge: br-mgmt
- action: add-br
name: br-ex
- action: add-port
name: bond23.102
bridge: br-ex
- action: add-br
name: br-storage
- action: add-port
name: bond23.103
bridge: br-storage
endpoints:
br-mgmt:
IP:
- 192.168.101.3/24
gateway: 192.168.101.1
gateway-metric: 100
br-ex:
gateway: 192.168.102.1
IP:
- 192.168.102.3/24
br-storage:
IP:
- 192.168.103.3/24
roles:
fw-admin: br1
ex: br-ex
management: br-mgmt
storage: br-storage
The vendor_specific parameter is a hash, empty by default, required only for plug-ins. It allows plugin developers to use custom code for adding non-standard parameters. As a result of inheriting and extending the puppet type (not the provider), this a non-trivial task. Plugin developers may pass any data structures using this hash and its subhashes. All data from this hash is passed to the provider transparently.
For debug purposes you can use following puppet calls to get prefetchable properties for existing resources. Please note that bridges and bonds in linux are ports too, and are present in the l2_port output with the corresponding flags (if_type).
# puppet resource -vd --trace l23_stored_config
# puppet resource -vd --trace l2_port
# puppet resource -vd --trace l2_bridge
# puppet resource -vd --trace l2_bond
# puppet resource -vd --trace l3_ifconfig
# puppet resource -vd --trace l3_route
These commands may fail before the initial configuration run by L23network because some kernel modules are not loaded or some command-line tools are not installed.
Each L23network resource has an interface through the puppet 'define' resource. These defines contain some simple logic, including the define provider for low-level resources and a call to two low-level resources:
- l23_stored_config - for modifying OS config files
- low-level resource for live-configuring the resource (e.g. l2_bridge)
This resource manages the interface configuration files directly. Each possible parameter should be described in the resource type.
This resource allows us to avoid using ERB templates, because in some cases (e.g. bridge and port with the same name and IP address) we need to modify the same config file content three times.
l23_stored_config { 'br1':
onboot => true,
method => manual,
mtu => 1500,
ethtool => {
.....
},
provider => lnx_ubuntu,
}
The location of the configuration files is defined inside the provider for the corresponding operating system and provider. The provider name for l23_stored_config also depends on the operating system, the operating system version, and the specific network provider (native linux, ovs, etc...)
- CentOS 6 and 7
- RedHat 7
- OracleLinux 7
- Ubuntu
When I started working on this module I was inspired by https://github.com/ekarlso/puppet-vswitch. Endre, big thanks...