Icinga @ CeBit 2015

cebitStarting early on Sunday morning meeting in the NETWAYS office in Nuremberg, the first shift moved to Hannover reaching out for CeBit 2015 preparations. The Open Source Park is located in hall 6, and we were excited to have our booth directly attached to the Open Source Forum where all the cool presentations took place as well.

Monday morning should’ve started quiet and calm, but it wasn’t – users and visitors stopping by and looking at the fancy Icinga Web 2 demo. Listening to our stories on how Icinga 2‘s apply-services-based-on-patterns works and all the benefits with and around it. The first day already was overwhelming and so did it go on the second day on Tuesday. So much #monitoringlove at CeBit is really incredible, and we certainly enjoyed showing everything in detail :-)

We also had the pleasure giving two radio interviews – make sure to join the Open Source Magazin CeBit special on Monday, 23.3.2015 and the Midnight Gaming CeBit Podcast (German only).

Bernd joined later in the evening, having prepared his talk in the Open Source Forum on Wednesday, Enterprise Open Source Monitoring with Icinga 2. Never seen so many people attending a talk in the Open Source Forum and the presentation really was a blast. After the talk the Icinga booth was a bit crowded ;-) The second shift with Eric, Johannes and co-workers from NETWAYS already joined us and so Marius and myself where leaving Wednesday, tired but happy :-)

While our feedback was overwhelming, it did not stop yet. People coming over and saying “Thanks for Icinga, we love it” is the best feedback there is! And rumors do tell the party on Wednesday evening was great too, and Thursday was a bit heavy. But still, following the Icinga social media streams the appreciation is massive and really keeps us going forward!

See you next year, hopefully :-)

Follow Icinga on Twitter, Facebook & Google+ – some impressions below.


Bugfix release Icinga 2 v2.3.2

Thanks for your kind feedback on the initial v2.3.0 release – we’ve found a few bug reports and already implemented fixes. There’s a regression with the disk plugin check command in 2.3.0 – warning and critical thresholds now require the percent sign explicitly. The documentation has been fixed – make sure to update your configuration accordingly.

icinga2_2.3.1Cluster heartbeat does not disconnect clients on slow bandwidth, process groups are killed on timeouts instead of only the parent process and the String type provides a new method called ‘contains‘.

Update: There was a problem with the fix in #8687 which is why we’re now at v2.3.2.

Upgrade now and enjoy Icinga 2 v2.3.1 v2.3.2!


Please note that this version fixes the default thresholds for the disk check which were inadvertently broken in 2.3.0; if you’re using percent-based custom thresholds you will need to add the ‘%’ sign to your custom attributes.


  • Feature 8659: Implement String#contains
  • Bug 8540: Kill signal sent only to check process, not whole process group
  • Bug 8657: Missing program name in ‘icinga2 –version’
  • Bug 8658: Fix check_disk thresholds: make sure partitions are the last arguments
  • Bug 8672: Api heartbeat message response time problem
  • Bug 8673: Fix check_disk default thresholds and document the change of unit
  • Bug 8679: Config validation fail because of unexpected new-line
  • Bug 8680: Update documentation for DB IDO HA Run-Once
  • Bug 8683: Make sure that the /var/log/icinga2/crash directory exists
  • Bug 8684: Fix formatting for the GDB stacktrace
  • Bug 8687: Crash in Dependency::Stop
  • Bug 8691: Debian packages do not create /var/log/icinga2/crash

Icinga 2 v2.3.0 released

You may have heard it already – 2.3 adds lots of new features, for example object attribute accessors at runtime accompanied by functions, loops, conditionals and much more. Bringing you Icinga 2 v2.3.0 also means: 660 Git commits since 2.2.0, 94 features & 127 bug fixes.

While upgrading your Icinga 2 installation, you can test-drive the new language features in the new live console online on icinga.org. Grab a coffee, check additional feature details below, switch to the Changelog and once your upgrade has finished, get to work with the all new shiny Icinga 2 v2.3.0. The online documentation is currently undergoing changes, things to note: live search and removable tables of content tab.

Conditional statements

icinga2_2.3_conditions_icingaweb2This was frequently asked in the past: “How can I inherit values from the host to the service, and leave it to a default value if not set?” Consider it done with if-then-else-if-else conditions inside the Icinga 2 configuration language. The example shows a fallback to the host’s FQDN if its address6 attribute has not been set – cool, isn’t it?


Functions – what for?

You can now define your own functions including the return keyword. That includes locally scoped variables identified by the var keyword and anonymous lambda functions too. We’ve thought about functions and their use cases for Icinga 2. One thing we came up with is the Boolean return value for set_if inside command arguments – not only a macro string value, but also nearly any condition. Same applies for command argument values. The short way of assigning return values is putting them in double curly brackets {{ …. }}.


Loops, loops, loops, …

You’ve seen for loops already inside the fancy apply for rules introduced with 2.2. Using while and for loops including break and continue keywords has been experimental for Icinga Web 2’s Vagrant Box quite a while. In real-life scenarios you would use them in combination with functions and if-then-else conditions iterating over arrays and dictionaries defined in custom attributes.


Object attribute accessors for clustered checks

icinga2_2.3_object_runtime_attributes_cluster_bp_icingaweb2Digging up the old problem with so-called “on-demand macros” in Icinga 1.x and migration issues we’ve tackled this one differently: Instead of obfuscating the macro parser once again, the problem is solved differently – you’d want to access objects and their run-time attributes. Most commonly, get_host(NodeName).state for the old-fashioned check_cluster plugin. And many, many objects and attributes more …

That syntax could be used for cluster checks and business processes inside Icinga 2 – we’ll tackle the dummy check problem sooner or later, promise!


Time dependent thresholds

Right before the feature freeze one of our colleagues approached us with the request of time-dependent check threshold values. We’ve already had if-conditions and object accessor functions thus far, and so added the is_inside run-time attribute to the get_time_period() function. That way you can set thresholds depending on the current time of the day.

Type methods

Defined an array, but need it in a sorted manner? Remove a dictionary item inherited from a template? Split a string into parts? Not an issue anymore.

console CLI command

icinga_org_live_consoleTest all the language features inside the Icinga 2 console. Install rlwrap to keep history and line continuation and test-drive your new configuration before putting into the files.

Misc features

From OpenTSDB support to ignoring soft states in dependencies. Additional ITL plugin check commands (interfacetable, IPMI, webinject, vmware_esx, local ‘nscp client’ commands for the Windows agent). Livestatus header support, improved performance and additional bygroup tables. Improved cluster stability and scalability using fewer threads for socket I/O and SNI TLS support. ‘icinga2 troubleshoot’ cli command for better community support … check the Changelog below and in the documentation for more details.

PS: I’ve uploaded the configuration samples made for this blog post into the Vagrant boxes.


2.3.0 Changelog

  • Improved configuration validation
    • Unnecessary escapes are no longer permitted (e.g. \’)
    • Dashes are no longer permitted in identifier names (as their semantics are ambiguous)
    • Unused values are detected (e.g. { “-M” })
    • Validation for time ranges has been improved
    • Additional validation rules for some object types (Notification and User)
  • New language features
    • Implement a separate type for Boolean values
    • Support for user-defined functions
    • Support for conditional statements (if/else)
    • Support for ‘for’ and ‘while’ loops
    • Support for local variables using the ‘var’ keyword
    • New operators: % (modulo), ^ (xor), – (unary minus) and + (unary plus)
    • Implemented prototype-based methods for most built-in types (e.g. [ 3, 2 ].sort())
    • Explicit access to local and global variables using the ‘locals’ and ‘globals’ keywords
    • Changed the order in which filters are evaluated for apply rules with ‘for’
    • Make type objects accessible as global variables
    • Support for using functions in custom attributes
    • Access objects and their runtime attributes in functions (e.g. get_host(NodeName).state)
  • ITL improvements
    • Additional check commands were added to the ITL
    • Additional arguments for existing check commands
  • CLI improvements
    • Add the ‘icinga2 console’ CLI command which can be used to test expressions
    • Add the ‘icinga2 troubleshoot’ CLI command for collecting troubleshooting information
    • Performance improvements for the ‘icinga2 node update-config’ CLI command
    • Implement argument auto-completion for short options (e.g. daemon -c)
    • ‘node setup’ and ‘node wizard’ create backups for existing certificate files
  • Add ignore_soft_states option for Dependency object configuration
  • Fewer threads are used for socket I/O
  • Flapping detection for hosts and services is disabled by default
  • Added support for OpenTSDB
  • New Livestatus tables: hostsbygroup, servicesbygroup, servicesbyhostgroup
  • Include GDB backtrace in crash reports
  • Various documentation improvements
  • Solved a number of issues where cluster instances would not reconnect after intermittent connection problems
  • A lot of other, minor changes
  • DB IDO schema upgrade to 1.13.0 required!

Find the detailed Changelog in the “What’s new” section in the documentation!

Team Icinga welcomes …

It’s been a while that we’ve written about new team members, but you may have seen new faces on community events and also names inside the git commits recently. Icinga is a community project after all, but some of us want to do more. Getting access to Icinga’s infrastructure, working on existing tasks and finally tagging a release being proud of your work is enjoyable & fun at once. Contact us if you feel the same!


Team Core


Carlos CesarioCarlos Cesario | Core Developer
A sys admin of 8+ years, Carlos’ joins us from Brazil to bring much virtualisation and monitoring experience. The man behind Icinga 2 Vim and Nano syntax highlighting, Carlos has been an avid Icinga user since Icinga 1.2. On board since June 2014.

tkroneTobias von der Krone | Core Developer, QA & Support
Tobias is a long-term Icinga user managing distributed setups with Icinga 2 clusters & Saltstack formulas at ProfitBricks. Whilst learning C++ he takes care of QA, documentation and community support. On the team since January 2015.


Team Web


jmeyerJohannes Meyer | Icinga Web 2 Developer
Johannes started working on Icinga Web 1.x and is now responsible for Icinga Web 2 development. One of his babies is the Icinga Web 2 Setup Wizard paving the way for all the exciting modules and views he’s also been building. On the team since June 2013.

mjentschMatthias Jentsch | Icinga Web 2 Developer
Using the newest web technologies & frameworks, Matthias has proven his expertise in many ways. He’s the main author of NETRP and contributes to Icinga Web 2 development with a love of charts for reporting & auth backend integration. On the team since June 2013.

afuhrAlexander Fuhr | Icinga Web 2 Developer
With more than 10 years of web development experience, Alexander knows how to deal with ExtJS in Icinga Web 1.x and LConf. His main contribution area is the all new shiny Icinga Web 2 and its responsiveness. On the team since August 2014.


Team Packages and Tools


vgagliardiValentino Gagliardi | Ansible Role Developer
A freelance system administrator, in love with monitoring and logging, Valentino uses Icinga to monitor hundreds of Linux servers. IT automation fanatic, is responsible for Icinga 2 Ansible role. Onboard since June 2014.

nchappellNick Chappell | Puppet Module Developer
Nick is located in Portland, OR and working as a sysadmin. Icinga is his first open source project and so he was brave and chose a hard one: Nick is the creator and maintainer of the official puppet-icinga2 Puppet module.  Onboard since August 2014.

bahlersBernd Ahlers | Graylog2 Vagrant Maintainer
Bernd is a Graylog developer and is located in Hamburg. During his journey of log processing and connecting monitoring cores like Icinga 2, he recently joined the Icinga team helping in maintaining the Vagrant boxes, specifically the icinga2x-graylog2 demo box. Onboard since October 2014.

vkhatriVirender Khatri | Chef Cookbook Developer
Virender works as Senior DevOps Engineer at BSB and has 7+ years of experience with AWS, Open Source technologies and automation tooling. He is the lead maintainer of the official Icinga 2 Chef Cookbook paving the way for config management tool intergration. On the team since January 2015.

tdvylderTom de Vylder | Puppet Module Developer
Tom is working at Inuits and is an experienced Puppet developerincluding officially approved modules. He’s putting his effort into thePuppet module for Icinga Web 2, but also ensures that the Icinga 2 module passes Q&A and unit tests. On the team since January 2015.

dgoetzDirk Götz | RPM Packager & SELinux Master
Dirk is working as Senior Consultant at NETWAYS. His main tasks are monitoring with and around Icinga as well as configuration management with Puppet. He’s an RHCE and helps mastering the Icinga RPMs especially focusing on SELinux. On the team since February 2015.


Thank You!


Every little or big contribution counts – but if there is no time left for Icinga, and other things in life become more important, it’s truly understandable.

You know Jannis as the brain behind Icinga Mobile whilst developing Icinga Web 1.x and its modules. He also kicked off the development of Icinga Web 2 together with Tom. Thank you Jannis for remarkable 5 years making Icinga great! All the best for your future – salute to him on Twitter, if you like :-)

From FOSDEM to CfgMgmtCamp and beyond

fosdem_2015_logoIcinga team members are spread all around the globe, and so are the events we’re attending or organizing :-)

We’ve never been to FOSDEM, so it was really nice to catch up with community members and speakers knowing Icinga very well, as well as talking about the hot stuff with Icinga 2 and Icinga Web 2. Bruxelles & Waffels, Gin & Beer, and also meeting with the Inuits guys talking Puppet.

FOSDEM took place on Saturday & Sunday where we mostly attended the configuration management dev room, next to distributions (systemd, etc) and visiting the booths. Numbers said 3000 to 5000 users, and certain rooms were just full any time – FOSDEM was renamed to “Fulldem” on Twitter. Sunday evening we joined the Foreman developers and contributors for dinner, moving to Ghent afterwards.

cfgmgmtcamp_logoWith nearly ~500 registered users CfgMgmtCamp has a full house, kicked off by the keynote by Mitchell Hashimoto (creator of Vagrant), providing a vision where configuration management will head. Next up was a panel on the future of configuration management with the brains behind Chef, Ansible, Saltstack and Puppet – awesome! We were moving further into the Foreman track with Katello as content life-cycle management system, and a nifty talk on how to deploy CoreOS with Foreman.

Last but not least – James Turnbull on Monitoring Sucks. While certain aspects are definitely true such as using the configuration management tool to discover and register services, we do not agree on a bad monolithic model behind Icinga 2.

Second day started with a great talk by Jez Humble from Chef on Lean Configuration Management. Next up: Writing Foreman plugins, using autodiscovery and even further – there certainly were many tracks and talks to choose from, Puppet, Ansible, Chef, Salt, Juju, Foreman, … and the ones who discovered the already developed modules for integrating Icinga2 with Puppet, Ansible & Chef doing their home exercises already (have a Saltstack recipe? Contact us!).

Afterall, CfgMgmtCamp was an awesome event – big thank you to the organizers! #cfgmgmtcamp – see you 2016 :)

PS: Keep monitoring all the things! ;-)