Connect and monitor feature flag changes in DataDog
A updated version of this blog post is available here.
Nowadays, thanks to modern continuous delivery tools, many software development processes make it easy to deliver multiple releases per week. It is often common for several scheduled releases per week to be released. The usual practice is that during the release process, DevOps guys keep their eyes on the monitoring dashboard, and if they find any anomalies, they roll back the version.
But what happens if someone changes a feature flag's value (release toggles) in the production environment?
Release Toggles allow incomplete and un-tested code paths to be shipped to production as latent code which may never be turned on. https://martinfowler.com/articles/feature-toggles.html
DevOps are notified that a certain threshold has been reached for the production environment and that when they open a related metric they see something similar:
This is a common dashboard view shown in the metrics of my Pizza Order Application (request per minute).
"What happened around 01:15? Isn't that a glimpse of a ramp-up? how strange! Should we do anything about it? Scale-out/up or do we have a bug on our hands?"
Use Datadog - ConfigCat integration
Wouldn't it be nice to notify your team members when a feature flag is switched on (or off)? With Datadog ConfigCat Integration we can use an unique notification process based on certain Events.
This post is not a full cover guide on how to connect your ConfigCat product to Datadog, please follow these instructions to set up it.
Every ConfigCat event is tagged with the following names: Product, Config, and Environment. The source of this event is configcat. These tags help you to separate your monitoring events and alerts.
Option 1 - Send Notification with Triggered Monitors
This option uses Datadog Alerts to notify the related channels about ConfigCat's settings change.
Here is a quick sample set up to send a notification to @devops if ConfigCat's Change Settings events count is above 1 in the last 5 minutes.
Option 2 - Timeseries
This solution will not send any notification that just creates a powerful Timeseries diagram. The Timeseries widget can add more context to your Event-Overlay graph.
I added the following search query to my previous Timeseries widget:
The red bar shows that some related event(s) happened.
You can improve your release monitoring dashboard and alerting system with a minor change. It's cool, isn't it? :)
Happy feature flagging!
If you're interested in DevOps automation, take a look at what our friends at Bunnyshell are building.