Skip to main content

A/B Testing in Ruby Using Feature Flags

· 9 min read
Chavez Harris
Inspiration does exist, but it must find you writing code.

Conducting an A/B test experiment is a popular approach to determining which variation of a particular feature works best for your users. This experiment can shape the success of any product or business. Conducting these tests can be tricky and time-consuming, but with the ideal tools, it doesn't have to be.

If you're a Ruby developer, you're in luck. I'll show you how to perform an A/B test experiment in Ruby using ConfigCat feature flags and how to record the results with Amplitude's data analytics platform. Let's get started!

A/B Testing in Ruby Using Feature Flags cover

Getting familiar with the concepts

Let's take a look at the concepts we'll be working with.

What is A/B testing?

Imagine you're a founder at a new tech startup looking to encourage users to download your mobile app via a landing page. Your team presents you with two designs: one has a white call-to-action button (Variation A), and the other has a red button (Variation B). However, you're not sure which design to choose. This is where A/B testing comes in handy.

An A/B test compares two variations to determine which one performs better. In this case, you want to know which landing page leads to more user downloads. To do this, you'll deploy both variations of the landing page to different user groups, track the downloads from each group, and see which design wins.

A/B variations of the landing page

To deploy both variations of the landing page, you'll need to create a feature flag.

How do feature flags work?

Feature flags work like switches that toggle features on or off. We'll use a feature flag to show each group a different variation of the landing page: one group will see Variation A, and the other will see Variation B. The benefit of using feature flags is that you can do this without redeploying the app, saving you time.

To use feature flags, you'll need a feature flag provider. For this demo, we'll use ConfigCat's Ruby SDK to set up a feature flag for the landing page.

The final question is: how do you track both tests to determine which variation of the landing page is performing best? This is where Amplitude's data analytics platform becomes essential.

What is Amplitude?

Amplitude is a data analytics platform that helps you track user behavior and analyze results. We'll use Amplitude to monitor downloads influenced by each landing page variation and these results. While the two designs will look similar, we'll set up an event tracker for each button: Variation A's button will trigger an event labeled WHITE_CTA_BUTTON_CLICKED, and Variation B's button will trigger RED_CTA_BUTTON_CLICKED.

Before we dive into the code, let's briefly review the resources we'll need to get started.

Prerequisites

Here are the things you'll need to follow along:

Live Demo

Now that we've got all the essentials lined up, let's proceed with a hands-on demo of an A/B test experiment.

A/B variations of the landing page

When feature flags are coupled with user segmentation, each variation, to end-users, will appear as if it's released from two separate code bases. More on segmentation later. Here's a snippet of the code that makes this possible, using an if-else statement to show one of the buttons and hide the other based on the value of the feature flag:

<div class="cta-button-wrapper">
<% if @is_my_feature_flag_enabled %>
<button class="button red-button" id="redCTAButton" onclick="trackRedCTAButtonClick()">Download Now</button>
<% else %>
<button class="button white-button" id="whiteCTAButton" onclick="trackWhiteCTAButtonClick()">Download Now</button>
<% end %>
</div>

Sample App

To follow along, clone the companion sample app from here. As we explore the next sections, I'll highlight the code segments that need updating.

Setting up the landing page

  1. Install the Gemfile dependencies by executing the following command in the terminal from the root of the repo:
bundle install
  1. Create a .env file in the root directory with these environment variables. Don't forget to replace the placeholders with your actual values:
AMPLITUDE_API_KEY="YOUR-AMPLITUDE-API-KEY-GOES-HERE"
CONFIGCAT_SDK_KEY="YOUR-CONFIGCAT-SDK-KEY-GOES-HERE"
  1. Run the app with:
ruby app.rb

If all goes well, you should be able to see the landing page at http://localhost:4567/

Adding the feature flag

  1. In your ConfigCat dashboard, create a feature flag with the following details:
  • Name: my-feature-flag
  • Key: myfeatureflag
  • Hint: Switches between white or red CTA button
  1. Copy your ConfigCat SDK key by clicking the View SDK Key button in the top right corner of the dashboard or by opening the following URL in a new tab: https://app.configcat.com/sdkkey and paste it in the .env file.

  2. Earlier, we touched on segmentation, and here's how it comes into play: To set up the two user groups, we'll create two user segments on the feature flag. Click the + IF button and then choose Add rule with segment condition from the dropdown menu:

Add rule with segment condition

Navigate to Manage segments:

Navigating to Manage segments

Create two user segments with the following details:

Details for creating segments

Using the user segments, users in Hungary (Group A) see Variation A (white CTA button), and those in France (Group B) get Variation B (red CTA button). To make it happen, let's update the code to include the feature flag.

Integrating the feature flag into the Ruby code

Update the code to include the feature flag. Here is how the code in app.rb should look like:

# app.rb
require 'configcat'

# ...

# ConfigCat client initialization
configcat_client = ConfigCat.get(
ENV['CONFIGCAT_SDK_KEY']
)

get '/' do
# Create a variable to store the value of the feature flag
@is_my_feature_flag_enabled = configcat_client.get_value('myfeatureflag', false)
erb :index
end

You can see the full code here.

User Object

To help ConfigCat determine which group should see a specific variation, the user's country must be known. This can be achieved by including a country attribute in the User Object. Let's add a user object to the code:

# app.rb

# ...

# Create a user object to identify the user
user_object = ConfigCat::User.new(
'7b8c03a6-502d-4d6b-8d67-fc5e1a2b9a94',
email: 'john@example',
country: 'France',
)

get '/' do
# Create a variable to store the value of the feature flag
@is_my_feature_flag_enabled = configcat_client.get_value('myfeatureflag', false, user_object)
erb :index
end

Did it work?

Tweaking the country in the user object is a quick way to ensure our feature flag and user segment are functioning as intended.

But wait! We also want to track the clicks on each button. Here's how to set up Amplitude.

Tracking the results with Amplitude

  1. Create a free account on Amplitude

  2. In your Amplitude dashboard, navigate to your Organization settings by clicking on the gear icon at the top right corner of the page, click on the Projects in the left sidebar, then click Create Project button to create a project:

Creating a new project in Amplitude
  1. Select the data source and SDK to use. In this case, we'll be using the JavaScript SDK.
Creating a new project in Amplitude
  1. To view the click events from each button on the landing page, we'll need to create a chart to visualize the clicks from each button. Click the Create button on the top left corner and select Chart -> Segmentation:
Creating a new segmentation chart in Amplitude
  1. Copy your Amplitude API key from the project settings page and paste it into the .env file.

  2. Add the following script tags to the head of the index.erb file to initialize Amplitude with your Amplitude API key.

 <head>
<!-- ... -->
<script type="text/javascript" src="https://cdn.amplitude.com/libs/amplitude-7.2.1-min.gz.js"></script>
<script type="text/javascript">
amplitude.getInstance().init("<%= ENV['AMPLITUDE_API_KEY'] %>");
</script>
</head>
  1. Before the closing body tag, add the following script for tracking the clicks from each button:
<body>
<script type="text/javascript">
function trackWhiteCTAButtonClick() {
amplitude.getInstance().logEvent('WHITE_CTA_BUTTON_CLICKED');
}
function trackRedCTAButtonClick() {
amplitude.getInstance().logEvent('RED_CTA_BUTTON_CLICKED');
}
</script>
</body>

From the above, you can see that the white button will send an event called WHITE_CTA_BUTTON_CLICKED, and the red button will send an event called RED_CTA_BUTTON_CLICKED. This is how we'll be able to differentiate the clicks from each button.

The complete code for index.erb can be found here.

Viewing the results in Amplitude

Set the user object's country to Hungary and click the white button five times. Do the same for France and the red button. If everything works, you should see both events logged in Amplitude.

It may take a few minutes for the events to show up, so make sure to select the correct options in the left sidebar to view the events, as shown below:

A/B comparison on bar chart

The results will indicate which button performed better. If the red button has a higher conversation rate, it's a strong signal that Variation B is the better option and should be deployed to all users.

Conclusion

In this tutorial, we learned how to implement A/B testing in Ruby using ConfigCat's feature flagging service and Amplitude's data analytics platform. We learned how to create a feature flag in ConfigCat, how to segment users into groups, how to integrate the feature flag into the Ruby code, and how to track the results with Amplitude. We also learned how to use the results to make a decision on which variation of the landing page to deploy to all users.

If you're feeling curious and want to explore this on your own, I'd suggest signing up for a free account on ConfigCat and Amplitude. ConfigCat hooks you up with a generous free tier, letting you dive into feature flags without spending a dime. Created by developers for developers with ❤️.

For more awesome content, keep up with ConfigCat on X, Facebook, LinkedIn, and GitHub.