Using Variation IDs in ConfigCat for A/B Testing

The ability to make good decisions is often the leading factor in the success of a business. Yet, it is becoming increasingly difficult for companies to decide on what ideas to develop and content to optimize for users with certainty that it will perform as predicted.

Feature flagging is a vital technique that enables businesses to perform controlled A/B test experiments to gauge and analyze the impact of their decisions. A/B testing can effectively improve a business's overall performance and boost conversion rates by comparing and contrasting multiple implementations based on their performance with real users.

When done right, A/B testing can be a powerful way to improve the most relevant metrics in your business. In this guide, we'll look at how you can feed A/B test results to an analytics tool and visualize these results using Variation IDs in ConfigCat.

What is a Variation ID?

The purpose of A/B testing is to determine which variant of a web page or UI component has more conversions or performs better with visitors, i.e., more clicks, orders, or registrations. For example, experimenting with two colors of a "Buy Now" button implementation to see which one drives more sales.

You'd typically do this by showing the Control version (blue color) to 50% of users, while the other 50% are shown variant B (red color). To measure each variant's impact or conversion rate in your analytics tool, you'll need a way to identify, track, and distinguish each variant from the other.

The variation ID in ConfigCat is a unique ID generated for each feature flag's evaluated values set in ConfigCat's dashboard (true/false, text, or number). It identifies a line on the dashboard, i.e., a targeting rule, a percentage rule, or a default value.

If two targeting rules have the same value (e.g., both are true), their respective variation ID will differ. This way, you can simply check which specific targeting/percentage rule was matched for the specific userObject you pass to the SDK.

Why Would You Use Variation IDs?

There are two main reasons why you would need the variation ID:

• If you have lots of targeting rules that are evaluated to the same value (e.g., both of them are true) and don't want to only track the evaluated value (true/false, etc.). But instead, wish to track why the specific userObject that was evaluated to true, i.e., which targeting/percentage rule was matching for the userObject, you can use the variation ID.
• Sometimes analytic tools limit the values you can send to them. So if you have a long text as a feature flag value, you can not send that to the analytics tool. However, you can send this shortened hashed variation ID in place of the flag's value.

How are the Variation IDs Generated?

If you are curious how these variation IDs are generated, it is done by basically hashing the feature flag's key and the specific line's value (targeting rule, percentage rule, or default value).

Below is the C# code for how they are generated:

private static string GetVariationId(string settingKey, string value)
{
    return Sha1(settingKey + "_" + value).Substring(0, 8);
}

How to Send Variation IDs to Your Analytics Tool?

Just as you would send specific user information to your analytics tool, you can append the variation ID when sending events to your analytics tool. So if you are sending events to the analytics tool, you should call the getValueDetails method (or something similar) on the ConfigCat SDK and enrich your analytics data with the unique variation ID provided by the returned evaluation details object.

For this demonstration, we will be making use of a React demo application - the source code can be found on Github.

In this example application, we will be performing A/B test experimentation with two color variants (blue and red) of a “Buy Now” button implementation to see which one leads to better conversion with users (better sales). Both buttons are set to conditionally render based on the value of the associated feature flag on ConfigCat. We will also be using the Amplitude Analytics service to analyze the performance of the buttons to get a measurable overview of their respective conversion with users.

So let’s head to our ConfigCat dashboard and create a feature flag for these buttons with the targeting rule for each set to 50%. The variation ID for each target rule can be accessed by clicking the chart icon right next to the feature flag toggles.

info

For the chart icons to show up, you need to turn on the "Show VariationIDs to support A/B testing" feature in the Product's preferences.

Clicking on any of these icons will reveal the unique variation ID for the value of that specific feature flag’s target rule, as shown below.

To use the Variation ID for A/B testing, our application will need to get these IDs from the ConfigCat SDK using either the getValueDetailsAsync() or getAllValueDetailsAsync() methods and then pass them to the Amplitude analytics tool. We can filter for that specific Variation ID in the analytics tool to identify the variation.

So let’s do just that:

import { useEffect, useState } from 'react';
import * as configcat from 'configcat-js';
import amplitude from 'amplitude-js';

function App() {
  const [flagValue, setFlagValue] = useState('');
  const [variationID, setVariationID] = useState('');

  // handle click event
  const handleClick = () => {
    // send click event to amplitude
    amplitude.getInstance().logEvent('Buy Now Clicked');
  };

  useEffect(() => {
    // initialize ConfigCat client
    const configCatClient = configcat.getClient(
      'fK7ZCApWbkaDu14njPKZQw/s9XWupU5K0KRp_9PvkU02g',
    );

    // initialize Amplitude client
    const amplitudeInstance = amplitude
      .getInstance()
      .init('b7be4afb11b7945664f096cf2321e38f');

    // Mimic unique user for this demo
    const userID = 'User123';
    const userObject = { identifier: userID };

    // Send user ID to amplitude
    amplitude.getInstance(amplitudeInstance).setUserId(userID);

    // Get status and unique variation ID of feature flag from ConfigCat
    configCatClient
      .getValueDetailsAsync('salesflag', false, userObject)
      .then((details) => {
        setFlagValue(details.value);
        setVariationID(details.variationId);
      });

    // identify variation ID and send it to amplitude as part of the user property
    let identifyID = new amplitude.Identify().set('Variation ID', variationID);
    amplitude.getInstance().identify(identifyID);
  }, [variationID, flagValue]);

  return (
    <div className="App App-header">
      <h2>Hello World!</h2>
      {flagValue && <button onClick={handleClick}>Buy Now (blue)</button>}

      {!flagValue && <button onClick={handleClick}>Buy Now (red)</button>}

      <p>{variationID}</p>
    </div>
  );
}

export default App;

To reiterate the code above:

• We start by installing both the Amplitude and ConfigCat SDKs, after which we import both packages into our application.
• Next, we create states for the expected feature flag value and associated Variation ID.
• An event handler function is set up to log any click event in our buttons to Amplitude as a “Buy Now Clicked” event.
• Then we initialize both ConfigCat and Amplitude using their respective secret keys.
• Afterward, we create a random user ID to mimic a real user that will be the active user and a userObject to aid the ConfigCat targeting rule.
• We check the status of the feature flag and the current variation ID using the getValueDetailsAsync() method and the “salesflag” key, then store both in their respective states.
• Finally, we set the variation ID as a property of Amplitude event user property and upload it to Amplitude.
• The last bit of the puzzle is that our component conditionally renders a different button based on the targeted user and associated flags value.

How to Match Your Analytics Tool Events and VIDs in CC?

You can see the variation ID on the ConfigCat dashboard for a specific feature flag value line, and then you can filter for this value in your analytics tool.

Amplitude's user properties give you insight into the user when an action or event is triggered in the app, such as information about the device, browser, language, etc. Additionally, it is possible to set custom user properties using the Amplitude Identify object, which is exactly what we did by attaching the Variation ID to the user properties.

We can now filter for each Variation ID on Amplitude as they are now part of the logged user properties. So let's head over to Amplitude's analytics dashboard >> New >> Analysis >> Event Chart >> and "Filter" by User Properties.

Result after filtering the events by the variation ID:

As you can see, the variation IDs enrich our analytics tool with unique data for each target rule, giving more context to the experiment.

Conclusion

Using A/B testing, you can improve a website or application metric by conducting controlled experiments. To do this, you must instrument your A/B test trials with the appropriate data to compare the results of each experiment group in your application. ConfigCat's Variation IDs enrich your analytics tool with insights into your customers' behavior which can be very helpful when analyzing your A/B test experiments. We recommend using the Variation ID to associate a user with a given experiment variation and to segment your analysis when tracking the results of A/B tests in your analytics tool.

Visit ConfigCat's Facebook, Twitter, and LinkedIn for more details.