Sysadmins – because even developers need heroes

sysadminBeing a developer fiddling with the server where your application is running on and *b00m* something breaks on upgrade. Apache doesn’t come up again, your Varnish cache becomes wonky and everything’s broken. If you don’t have your Icinga monitoring system in place already, one of your sysadmins will certainly approach you why you did break the production system. He/she will monitor your ssh session and actions triggered and logged to your Logstash environment, also looking into fancy Grafana dashboards marking your development actions as “website reachability none”. Working with you (or without you when it is 3am in the morning and developers sleep already) and fixing your mess, re-establishing the service. The day after you’ll learn to coordinate better and add maintenance tickets and downtimes before doing any harm.

That story is just one of many where some of us are familiar with, or we may just recognize our sysadmins ranting all the way about their new storage system which just does not work the way it was designed. Or, nailing down the network latency problem inside your WiFi at the company while you keep working on your development or consulting area. Apart from the Linux and Unix bits, sysadmins generally have fun with Windows (group policies, exchange, etc) and save us plenty of time all over.

Head over to your sysadmins and say thanks – and don’t break anything today, it is Sysadmin appreciation day! :-)

Me being an Icinga developer for >6 years gives a special shout-out to the NETWAYS sysadmins running the Icinga server infrastructure and ensuring that {web,dev,git,wiki,…} keep running safe and stable – THANK YOU!

PS: While writing this blog post, Blerim sent me a preview to the new upcoming Icinga project infrastructure monitoring (now Graphite Web, later on fancy Grafana). Just awesome!



Vagrant boxes revamped

vagrant-logoWe’ve found Vagrant pretty useful for our demo cases – the first draft of the Icinga 2 Cluster happened to be a NETWAYS CeBit demo and evolved ever since. icinga-vagrant was added is official Icinga project similar to what we have with our Docker demos and gained more attraction by testers and developers.

The good thing about Vagrant using different providers is – you can even run these demos on Windows. You’ll need some prerequisites to make it work on Windows though – Git with SSH and Ruby. On Linux the Vagrant packages already requires them. Additionally you’ll need at least the following versions:

  • Vagrant >= 1.6.5
  • VirtualBox >= 4.2.16

Our default provider is VirtualBox where these boxes are regularly tested on. The underlying provisioner is using Puppet modules to setup the Vagrant environment. This installs for example Apache, MySQL, Plugins, Icinga 2 and Icinga Web 2 in the icinga2x box. The Icinga packages are pulled from our snapshot repositories allowing developers and testers fetching the latest and greatest – and also help finding bugs soon enough :-)

Currently there are four different Vagrant environments available for testing:

  • icinga2x: Standalone Icinga 2 box with user interfaces: Icinga Web 2, Icinga Web 1.x and Classic UI 1.x
  • icinga2x-cluster: Icinga 2 Cluster as Master-Checker setup. Includes Icinga Web 2 as user interface.
  • icinga2x-graylog2: Special Graylog demo environment with standalone Icinga 2
  • icinga1x: Standalone Icinga 1.x Core with Icinga Web 1.x and Classic UI 1.x. Includes Jasper-Reporting.

Note that icinga1x will stick with CentOS6 due to the Reporting integration (unless someone sends a patch). The other boxes have been upgraded recently to use CentOS 7.1 as base box.

There were several changes in the past months, refactoring the required git modules from submodules to subtree and also updating the demo configuration. All icinga2x boxes also share common host-only IP addresses for easier access. Once puppet-icinga2 and puppet-icingaweb2 become ready, they’ll be included inside the provisioning mechanism replacing our own local modules. In order to keep it simple, you may just download the latest release tarball instead of cloning the git repository.

Start away with the Icinga 2 box:

$ git clone
$ cd icinga-vagrant
$ cd icinga2x
$ vagrant up

Open your browser and navigate to the host-only ip addresses (Login: icingaadmin/icinga)


If you encounter problems, please open an issue at – meanwhile enjoy the Vagrant show! :-)

Icinga 2 bugfix release v2.3.8

2.3.7 last week introduced a problem with DB IDO PostgreSQL when fixing a separate issue. Next to that we’ve also backported some more fixes from our development tree for 2.4 into v2.3.8.

Check the Changelog below while preparing your update.


What’s New in Version 2.3.8


  • Bugfixes


  • Bug 9554: Don’t allow “ignore where” for groups when there’s no “assign where”
  • Bug 9634: DB IDO: Do not update endpointstatus table on config updates
  • Bug 9637: Wrong parameter for CheckCommand “ping-common-windows”
  • Bug 9665: Escaping does not work for OpenTSDB perfdata plugin
  • Bug 9666: checkcommand disk does not check free inode – check_disk

Bugfix releases: Icinga 2 v2.3.7 and Icinga v1.13.3

This time we’ll release two Icinga Core bugfix releases – Icinga 2 v2.3.7 and Icinga v1.13.3.

Package updates are available soon, meanwhile check the Changelog below.


What’s New in Version 2.3.7


  • Bugfixes


  • Feature 9610: Enhance troubleshooting ssl errors & cluster replay log


  • Bug 9406: Selective cluster reconnecting breaks client communication
  • Bug 9535: Config parser ignores “ignore” in template definition
  • Bug 9584: Incorrect return value for the macro() function
  • Bug 9585: Wrong formatting in DB IDO extensions docs
  • Bug 9586: DB IDO: endpoint* tables are cleared on reload causing constraint violations
  • Bug 9621: Assertion failed in icinga::ScriptUtils::Intersection
  • Bug 9622: Missing lock in ScriptUtils::Union

What’s New in Version 1.13.3


  • Bugfixes


  • Bug 7337: Only use SCHEDULE_HOST_DOWNTIME command for Icinga 2.x
  • Bug 8130: Wrong values for percent_* when using hostgroup in availability report
  • Bug 9020: Solaris package behaves badly upon uninstall
  • Bug 9106: Icinga no longer sending acknowledgement notifications
  • Bug 9240: invalid JSON for flapping threshold configuration

Icinga 2 bugfix release v2.3.6

While updating the documentation and Icinga Template Library definitions, we’ve also tackled a more severe problem with OpenSSL on SLES11 SP3 with this bugfix release v2.3.6.

icinga2_sles11_opensslSLES11 uses the old 0.9.8j release causing trouble when verifying the SSL certificates generated from ‘node wizard’ commands (see #9549 for a detailed analysis). The problem became even more weird when debugging it, so we decided to go for the only safe solution – link against openssl1 from the Security Module repository.

The package update for SLES11 requires openssl1, please ensure enabling the repository beforehand. You can use this small check script “check_icinga2_openssl” (shown in the screenshot).

Other than that, we’ve also fixed some bugs found inside the Windows plugins and NSClient++ integration. Whilst backporting a stability fix for the cluster from our 2.4 development tree there’s also more verbose logging for unauthenticated clients and cluster troubleshooting available.

Package updates should be around soon, meanwhile keep cool and check the Changelog below!


What’s New in Version 2.3.6


  • Require openssl1 on sles11sp3 from Security Module repository
    • Bug in SLES 11’s OpenSSL version 0.9.8j preventing verification of generated certificates.
    • Re-create these certificates with 2.3.6 linking against openssl1 (cli command or CSR auto-signing).
  • ITL: Add ldap, ntp_peer, mongodb and elasticsearch CheckCommand definitions
  • Bugfixes


  • Feature 6714: add pagerduty notification documentation
  • Feature 9172: Add “ldap” CheckCommand for “check_ldap” plugin
  • Feature 9191: Add “mongodb” CheckCommand definition
  • Feature 9415: Add elasticsearch checkcommand to itl
  • Feature 9416: snmpv3 CheckCommand: Add possibility to set securityLevel
  • Feature 9451: Merge documentation fixes from GitHub
  • Feature 9523: Add ntp_peer CheckCommand
  • Feature 9562: Add new options for ntp_time CheckCommand
  • Feature 9578: new options for smtp CheckCommand


  • Bug 9205: port empty when using icinga2 node wizard
  • Bug 9253: Incorrect variable name in the ITL
  • Bug 9303: Missing ‘snmp_is_cisco’ in Manubulon snmp-memory command definition
  • Bug 9436: Functions can’t be specified as command arguments
  • Bug 9450: node setup: indent accept_config and accept_commands
  • Bug 9452: Wrong file reference in
  • Bug 9456: Windows client w/ command_endpoint broken with $nscp_path$ and NscpPath detection
  • Bug 9463: Incorrect check_ping.exe parameter in the ITL
  • Bug 9476: Documentation for checks in an HA zone is wrong
  • Bug 9481: Fix stability issues in the TlsStream/Stream classes
  • Bug 9489: Add log message for discarded cluster events (e.g. from unauthenticated clients)
  • Bug 9490: Missing openssl verify in cluster troubleshooting docs
  • Bug 9513: itl/plugins-contrib.d/*.conf should point to PluginContribDir
  • Bug 9522: wrong default port documentated for nrpe
  • Bug 9549: Generated certificates cannot be verified w/ openssl 0.9.8j on SLES 11
  • Bug 9558: mysql-devel is not available in sles11sp3
  • Bug 9563: Update getting started for Debian Jessie