Parse Push Experiments: A/B Testing Best Practices and the Screencast

We recently launched Push Experiments, which lets you conduct A/B tests on push notification campaigns to identify the most engaging message variant. With Push Experiments, we wanted to make it easy to run successful A/B tests. In this post, we’ll discuss some of the statistical techniques we’re using behind the scenes. And, catch a screencast showing you the ins and outs of the new tool below.

Parse Push Experiments in Action

Ready to try Parse Push Experiments?  Watch the screencast here.  Or, read on for A/B Testing best practices.

 

What makes an A/B test successful?

We can say that an A/B test succeeds whenever we get a precise, correct answer to the question that originally motivated us to run the test. In other words, a good platform for A/B testing should try to prevent two kinds of failure:

(1): We should rarely get a result that leaves the answer to our question in doubt.
(2): We should rarely get an answer that seems precise, but is actually incorrect.

Parse Push Experiments uses three strategies to prevent these two kinds of failure:

  1. Encourage developers to ask precise questions that can be answered unambiguously.
  2. Prevent developers from reaching wrong conclusions by always reporting results along with a margin of error.
  3. Ensure that most A/B tests will give a precise answer by suggesting the minimum number of users that must be included in an A/B test in order to reasonably expect accurate results.

Step 1: Asking Precise Questions

Here’s one of the most important things you can do while running A/B tests: Commit to the metric you’re testing before gathering any data. Instead of asking questions like, “Is A better than B?”, the Push Experiments platform encourages you to ask a much more precise question: “Does A have a higher open rate than B?”

The distinction between these two questions may seem trivial, but asking the more precise question prevents a common pitfall that can occur in A/B testing. If you allow yourself to choose metrics post hoc, it’s almost always possible to find a metric that makes A look better than B. By committing up front to using open rates as the definitive metric of success, you can rest assured that Push Experiments will produce precise answers.

Step 2: Acknowledging Margins of Error

Once you’ve chosen the question you’d like to answer, you can start gathering data. But the data you get might not be entirely representative of the range of results you’d get if you repeated the same test multiple times. For example, you might find that A seems to be better than B in 25% of your tests, but that B seems to be better than A in the other 75%.

As such, when reporting the difference between the A and B groups (we’ll call this difference the lift), it’s important to emphasize the potential for variability in future results by supplementing the raw result with a margin of error. If you have an A/B test that has a lift of +1% and a margin of error that is between -1% and +3%, you should report that your A/B test’s results were inconclusive. If you simply reported a +1% change, your results would be misleading and might set up unrealistic expectations about the success of your push strategy in the future. By reporting a range of values that should contain the true answer to your question (this range is what a statistician would call a 95% confidence interval), you can help to ensure that anyone reading a report about your A/B test will not reach premature conclusions.

At Parse, we determine margins of error for open rate data using a calculation called the Agresti-Caffo method. When you’re working with push notification open rates, the Agresti-Caffo method produces much more reliable margins of error than naive methods like normal approximations.

In addition to automatically calculating margins of error using the Agresti-Caffo method, the Push Experiments platform only reports results after it’s become clear that either A offers a lift over B or that B offers a lift over A — helping to further protect you from reaching premature conclusions. Until there’s enough data to determine a clear winner, the Push Experiments dashboard will report that there’s still uncertainty on whether A or B is more successful.

Step 3: Choosing the Right Sample Size

Given that the Push Experiments platform will always report results with a margin of error, you’ll want to try to make that margin smaller in order to draw definite conclusions from more of your tests. For example, if you think that your A group will show a lift of 1% over your B group, you’ll want to make sure you gather enough data to ensure your margin of error will be smaller than 1%.

The process of picking a sample size that ensures that your margin of error will be small enough to justify a definite conclusion is called power analysis. The Push Experiments platform automatically performs a power analysis for your A/B test based on the historical open rates for your previous push notifications. To simplify the process, we provide suggested sample sizes based on the assumption that you’ll be trying to measure lifts at least as large as 1% with your A/B tests.

Only running A/B tests with carefully chosen sample sizes makes it much more likely that your A/B tests will succeed. If you select a sample size that’s much smaller than the size we suggest, you should expect that many of your A/B tests will lead to inconclusive results.

Putting It All Together

We believe the combination of precise questions, clean statistics and careful choice of sample size is essential for running a successful A/B test. You can achieve that with Parse Push Experiments, and we hope this look into the statistical methods behind our platform will help you do it.

 

johnmyleswhite
November 20, 2014

Let the Browser Handle Your UI Logic for You

At Parse, we’re in the middle of refreshing our frontend stack, and much of that has meant reevaluating the way we write browser-bound code. We’ve begun developing new custom inputs such as date pickers, dropdowns, or toggles in a way that leverages default browser behaviors to let us write less JavaScript (and reduce opportunities for bugs). Nearly every web developer has seen or implemented a date input that looked like a calendar, and chances are it contained plenty of logic to ensure that only one day could be selected at a time. This is the exact type of logic we’re trying to let the browser handle as we rebuild our interfaces.

It all starts with a simple checkbox

Let’s consider the scripting that might go into implementing a checkbox-style switch. When we click a switch that’s “off,” we change its state to “on” and get some visual feedback. We’d probably store its state in a boolean variable somewhere, add a click handler that updates the boolean, and then add or remove CSS classes based upon its current value. From a logic standpoint, it’s a duplication of something the browser can already do with <input type="checkbox">, but we put up with it for years because it let us have pretty switches.

Then the technique of using checked state to style checkboxes hit the mainstream. Using a combination of new CSS selectors and older browser behavior, it was possible to have visual elements that were tied to invisible inputs without writing a single line of JS. We use this same technique for many of our basic toggle switches today, but we’ve also taken it to new extremes with many of our UI components.

When building components, examine their core behavior

When we build new UI components for Parse.com, we follow a simple strategy: use the most similar browser element as a starting point. This goes beyond the obvious cases — such as using an anchor tag for something the user clicks — and looks at the inner behavior of a complex element. Remember the date picker we discussed earlier? The core interaction relies on being able to select a single day out of the month and visually represent that selection. Your browser already knows how to handle this: it’s behaviorally identical to a set of radio buttons! In fact, you’d be amazed by how many complex elements boil down to the radio button’s select-one-of-many behavior. They’re at the core of our own date picker, our fully styled dropdown menus, and a number of other switches and toggles. Knowing that the browser will ensure a single element is selected at any time allows us to eliminate a concern from our client logic.

uie_blog_2

Simultaneously, we avoid having to synchronize our data with our visualization, because a single interaction within the browser updates both. Along those same lines, at Parse we refrain from storing any data within a component that could be reasonably derived from its inner elements. For our specialized numeric inputs, we get the value by parsing the contents of an inner text input on demand, rather than returning an in-memory value that we update with an onChange handler. This guarantees that our JS component and its visual appearance never get out of sync.

Enough talk, let’s see a demo

My favorite implementation of this technique is found in our Hosting product. It’s a file tree that features complex interactions with absolutely no JavaScript.

uie_blog_3

When I designed the tree, I broke it down into its core behaviors: it has folders that open and close to display contents, and an overall set of files that can be individually selected. If you’re following along, it should make sense that the folders are backed by checkboxes while the files themselves are a single set of radio buttons.

<!-- Files are a set of radio buttons with the same name field -->
<input type="radio" name="hosted_files" id="f_myfile_js" value="MyFile.js">
<label for="f_myfile_js">MyFile.js</label>

<input type="radio" name="hosted_files" id="f_another_js" value="Another.js">
<label for="f_another_js">Another.js</label>

<!-- Folders are checkboxes that toggle the visibility of divs -->
<input type="checkbox" id="f_myfolder">
<label for="f_myfolder">My Folder</label>
<div class="dir_wrapper">
  Folder contents...
</div>

In CSS, we simply provide default styles for labels, and additional styles for when they follow a checked input.

label {
  color: #ffffff;
}
input[type=radio]:checked + label {
  color: #5298fc;
}

For folders, we also toggle the display property of a .dir_wrapper element that comes after a selected checkbox.

.dir_wrapper {
  display: none;
}

input[type=checkbox]:checked + label + .dir_wrapper {
  display: block;
}

With these styles, I don’t need to worry about having numerous click handlers to open or close folders, and a single change handler on the radio buttons can tell me when a new file is selected.

You can examine a slightly modified version of our file tree at this CodePen.

So what?

As web developers, we tend to lean a lot on JavaScript to build fancy interfaces and effects. I know I certainly do. But the next time you set out to build something new, look at its core behaviors and ask yourself if there is some pre-written, standardized part of the web browser that can handle some of the work for you. Yes, there is a bit of a tradeoff with markup instead of code, but I’d rather write code to generate HTML than deal with synchronizing data and visualization, handling a plethora of clicks, taps, and keyboard inputs, and ultimately replicating behavior that’s already embedded in the browser.

With these methods, you can build custom inputs that play nicely with the existing web. They fire native change events, they don’t risk side-effects in your view logic, and in many cases they even improve screen reader accessibility. If you take a look at the HTML5 spec for radio buttons, you’ll see that it only talks about behavior, not appearance. It’s no long stretch to reuse that behavior in new and creative ways, and I look forward to seeing how others employ it for interface construction.

Andrew Imm
November 10, 2014

Parse Developer Day Video Series: A New Kind of Checkout (Stripe)

This is a guest post from Christian Anderson, Software Engineer at Stripe.

Users abandon purchases for many reasons: poor design of the checkout form, the hassle of having to punch in their card details, or even just the fact that they’re confronted with an unfamiliar checkout experience. With Stripe Checkout, Stripe tackles these problems so you can focus on your core business.

This video from Parse Developer Day shows viewers how to use Checkout and Parse’s Stripe integration to craft a great purchasing experience for your users, including a demo of the powerful Checkout feature.

Courtney Witmer
February 12, 2014

Parse Developer Day Video Series: Monetize Your Windows Apps

The following is a guest post from Sanjeev Dwivedi, Technical Evangelist at Microsoft.

Windows 8 and Windows Phone 8 offer a fast growing platform for developers to build apps and start monetizing. Apps you build can be made available to all types of devices, from phones to tablets and Windows 8 PCs, giving you a wide reach of users, from casual to enterprise environment. The opportunity is even larger if you publish your apps to the more than 190 markets where Windows and Windows Phone Stores are available. The Stores also support the largest options of payment instruments, to help more consumers purchase your apps or in-app purchase: we support credit card, debit card, PayPal, mobile operator billing, Alipay (China) and others depending on the market. No other store enables you to publish to all three screens, and is available in so many different markets, with as many payment instruments.

On the technical side, Windows and Windows Phone offer developers a choice on whatever technology they know and are familiar with to develop apps. On both Windows 8 and Windows Phone, you can use C++ or C# to develop your applications, and Windows also offers HTML5/JavaScript – each of these languages have full access to all of the API. So, you can develop apps using technologies that you are familiar with and love.

If you want to learn more about the Windows and Windows Phone opportunity, tune in to the video, below.

Courtney Witmer
February 5, 2014

Parse Developer Day Video Series: iOS and Android Apps with Xamarin & C#

The following is a guest post from Craig Dunn, Developer Evangelist at Xamarin.

Parse provides amazing back-end services for your mobile apps – and with Xamarin you can quickly and easily build those apps for all the important mobile platforms using C# and sharing lots of code.

This talk shows how easy it is to take a cross platform (iOS and Android) app written with Xamarin and implement Parse Data to take advantage of remote storage and sharing data between devices. Parse and Xamarin support the latest C# features like async-await so your code can be short and sweet.

Tune in to this Parse Developer Day video to see how easily Xamarin apps can integrate with Parse, bringing advanced functionality quickly and easily to iOS and Android with C#.

Courtney Witmer
January 29, 2014

Parse Developer Day Video Series – Email: Driving Repeat Usage for Your App

This is a guest post from Nick Quinlan, Developer Evangelist at SendGrid.

Once you’ve built your app, you’re faced with the challenge of getting users to use it. This is even more difficult when you consider that 26% of users never re-open an app.

This talk discusses using email as a way to ensure repeat usage of your app. It looks at different kinds of user engagement emails, examines their strengths and weaknesses, and demonstrates how to send them using Parse Cloud Code.

By adopting the techniques described in this talk and using other engagement techniques, like Push Notifications, and SMS you’ll help ensure users stick with your app after downloading.

Courtney Witmer
January 22, 2014

Parse Developer Day Video Series: Building a Call Center with Parse and Twilio

The introduction of this blog post is a guest post from Twilio Developer Evangelist Kevin Whinnery.

API platforms like Parse and Twilio are changing the way developers build applications. By providing high-level building blocks for tasks like data persistence, user management, messaging, and VoIP communication, developers can rapidly assemble new software solutions when an opportunity arises. With these capabilities in hand, developers can focus on building unique value for their users in surprising ways. They can use software to solve problems that were previously out of their reach.

One problem space that is now open to software people is the call center, a key business component that at one time was a black box of arcane telecom technologies. In this video, we’ll see how you can use Twilio and Parse to quickly build a custom call center application entirely in the cloud. Using the Twilio Cloud Code module and Parse platform, you can create VoIP apps in the browser for support agents, accept and route inbound calls, and send text messages all with a few lines of JavaScript code.

This is just one example of the new breed of applications that are now easily created by software people. We can’t wait to see what yours looks like.

Courtney Witmer
January 15, 2014

Parse Developer Day Video Series: Relationships in Parse

Unlike many human relationships, relationships in Parse aren’t complicated. In this talk from Parse Developer Day, Parse Developer Advocate Prashant Sridharan breaks down object relationships in Parse.

Objects can have relationships with other objects. Whether they are one-to-one, one-to-many, or even many-to-many, there are a number of really easy ways to implement these relationships. This video takes a look at the different scenarios for which you will want to create relationships, shows you how to implement each of them in Parse, and then discusses some of the circumstances for when you would want to choose one over the other, and the tradeoffs in doing so.

Courtney Witmer
January 8, 2014

Parse Developer Day Video Series: Advanced Cloud Code

Getting started with Cloud Code is easy, but writing optimal Cloud Code takes practice. You may not even be aware of all the powerful features that are available.

This talk takes a deep dive into some of the advanced features you can use to get the most out of Cloud Code. You’ll learn how to host your website on Parse, starting from static files and building up to fully dynamic server-side web apps. We’ll cover the entire structure of a Parse-powered Express app. Examples of Jade and EJS templates will demonstrate how to build dynamic views. See how Express middleware modules can give you more control over exactly how your website requests are processed, and take a look at how you can use Express routing to organize your app.

We’ll also dig into some new Cloud Modules that Parse has released. The Parse User Session Module makes it super easy to let users log in to your website, while the Parse Image Cloud Module lets you crop or scale images in Cloud Code, whether from a beforeSave handler or from a custom HTTP endpoint.

So if you want to get the most out of Cloud Code, or just want to learn more about how Express web apps work, tune in to our latest Parse Developer Day video.

Bryan Klimt
December 18, 2013

Introducing Anywall Android

AnywallWe’re excited to announce the latest Android SDK sample in our arsenal: Anywall. This is a fun, social geolocation app that was previously only available on iOS. It has now been released for Android and is available for download on Google Play.

The accompanying Anywall tutorial walks you through the key Parse features used in the app. You’ll learn how to use ParseUser for user management, ParseObject and ParseGeoPoint to store Anywall content, and ParseQueryAdapter to display the content in a list view. We also walk through how the Google Play services Maps API is integrated into Anywall to provide the user’s location as well as display the content in a map.

We hope you enjoy playing with Anywall and that the tutorial helps you build awesome location-aware Android apps using Parse.

Christine Abernathy
December 12, 2013