Playbooks Solution for A/B Testing
How to A/B Test Features to Put Out Your Best Content
Setting up an effective sales strategy can be challenging if you don’t employ a mechanism to test and compare effectiveness. If you’re considering making updates to email templates, Plays, or even phone call talk tracks, try A/B Testing to have data that proves or disproves your hypothesis. Make methodical, purposeful changes to your sales process.
What is A/B Testing?
A/B Testing is comparing the effectiveness of two or more processes or features.
For example, in Playbooks you could compare:
- Two Plays with different wait times
- Email templates with different messaging
- Play Step order (send swag on step 3 vs step 5)
- Play Step types (starting with an email first vs a phone call)
- Phone call cue cards with different call to action
- Two different pre-recorded voicemails
To do this, records must be randomly assigned to the two or more different elements being compared. The simplest way to do this is to set records to be identified by Salesforce as either being record A or record B. Once the records are labeled A or B, Robots then assign records to the Play with the elements being compared.
Setting up A/B Testing
The first step of A/B Testing is to auto-number records. This can be done by either using the existing Salesforce record ID or creating a new custom field that auto numbers new records as they’re created.
To Create a Custom Auto-Numbered Field in Salesforce Lightning
- Click on Setup > Object Manager > Leads> Fields and Relationships.
- Click the New button.
- Select the Auto Number button.
- Click Next.
- Enter a field label of ‘AB_Record_ID’ and enter a Starting Number.
NOTE: Salesforce automatically generates the Field Name. - Optionally, click the box to Generate Auto Number for existing records.
- Click Next.
- Specify visibility for relevant security profiles and click Next.
- Click Save.
The next step is to create a formula field to label each record A or B based on whether that custom field is odd or even.
To Create a Formula Field in Salesforce Lightning
- Click on Setup > Object Manager > Leads> Fields and Relationships.
- Click the New button.
- Select the Formula button.
- Click Next.
- Enter a Field Label (i.e. AB_Test).
Note: This field is used by the A/B Test Robots to enroll records into different Plays. - Select Formula Return Type Text button.
- Click Next.
- Insert one of the formulas from the formula section below.
- Click Next.
- Specify visibility for relevant security profiles and click Next.
- Click Save.
For your A/B Test to work correctly, you need to add this formula to the formula field. Review the code below and click the button to copy the code.
This is the code for the A/B Test. AB_Record_ID__c is the name of the new custom ID field created previously. If your field has a different name, be sure to update this formula accordingly.
IF (MOD(VALUE(AB_Record_ID__c) ,2) =0, “A”, “B”)
This code is used for testing three elements against one another or the A/B/C Test. This test can be used for something like testing email send method. AB_Record_ID__c is the name of the new custom ID field created previously. If your field has a different name, be sure to update this formula accordingly.
(IF(MOD(VALUE(Lead_Number__c) ,3)=0,"A",
(IF(MOD(VALUE(Lead_Number__c) ,2)=0,"B",
"C"))))
Set up Plays to compare whatever elements you are working with (email template, Play, or phone call talk track). Then label these Plays to indicate which is which (i.e. Inbound A/B Test – Play A and Inbound A/B Test – Play B). Be sure to only change one thing between the two Plays. Having too many changes between the Plays will muddle the data making it impossible to determine which change is responsible for changing the outcome.
Finally, create 2 Robots to enroll the records into Plays that have the different elements being tested, depending on whether the records are labeled A or B.
To Create A/B Test Robots
- Name the Robot to indicate which test group it corresponds to (i.e. Inbound A/B Test A).
- Create the Robot to enroll records into the Play based on the CRM custom field you created earlier (i.e. AB Test).
- Repeat this step for Robot B.
Playbooks Governance
More and more customers are following the Playbooks best practice of creating a governance or Play Committee to discuss how Plays can be improved, listen to user feedback, and develop better strategy. A/B testing can provide these types of groups and other Sales Operations teams with necessary data to make decisions.
Before implementing a new strategy, test your theory with a small group of varied test users to make sure it will have a positive impact. Review the results with the Play Committee and determine next steps.
A/B Test Ideas
You may already have some questions in mind, but here are some ideas to help you start thinking about how A/B testing can be employed in your current strategy.
Play Steps
- Does an email or a phone call work better as an introduction to a new customer?
- Is there a better/best time to send swag?
- Do customers lose interest and stop responding to emails after multiple emails in a row?
Emails
- Does a personalized video result in more responses?
- Is a formal overview of the company better than a casual introduction?
- Is a longer or shorter subject line better?
- What is the best way to send an email: SmartSend, manual schedule, or Send Now (A/B/C Test)?
Cue Cards / Talk Tracks
- Which call to action has better results?
- Does a detailed Cue Card help reps stick with the messaging vs. general bullet points?
- Should the script be longer or shorter?
- Does Pre-Recorded VM A get more returned calls that B?
Proper Testing Best Practices
A/B Testing can be a powerful tool for refining engagement strategy, but it must be approached properly to qualify that success measurements were properly attained. This article has noted a few times that you should isolate your test to one element, but what other considerations will have an impact on the measurement of success? Generally speaking, keep tests as simple as possible by only comparing two things that are directly related to each other (ex: two Plays for the same campaign).
Sample size is a very important factor in A/B testing. Don’t jump the gun and declare victory after 10 phone calls or emails. To determine how large of a sample your test should encompass, consider how much total exposure your Play or email will cover. Will the Play be active for a few months or in perpetuity? Is the talk track specific for a subset of your customers (i.e. industry, title) or all customers? Other things to balance when determining sample size are risk tolerance, cost, and timing. How much time do you have to test an invite email before the actual event? And also think about the prospect/customer’s timetable; is it the end of the month/quarter? There isn’t one right answer for how large your sample size should be. But consulting with your Play Committee, Sales Operations team, and internal analytics team (if you have one) is advised.
Statistical significance should also be considered in high impact testing and will help you to understand how much probable chance could be swaying your results (this is also where an internal analytics team would be helpful). Statistical significance is expressed as a “p-value”; the lower the p-value, the less likely your results are swayed by chance. There are many resources online to help you learn more about statistical significance, but just note that random chance could be swaying your results one way or another. For a simple, low impact A/B test don’t get stuck in analysis paralysis.
Measuring Success of A/B Testing
Once you’ve completed everything in this article, how do you figure out which Play is most successful? There are several reports in Salesforce that can help clarify which Play was more successful. When a Playbooks user achieves the goal of a Play, they need to mark the Play as successful (easy two-click process). Otherwise, it will be difficult to see which Play was more effective.
If the particular A/B Test you’re running involves emails, you may need a couple of different reports to get deeper insight into how valuable email changes are. Reports for emails highlight link clicks, email opens, and email replies, which are all signals of customer engagement. This can provide insight into the effectiveness of different content strategies, and help teams optimize the likelihood of contact with prospects and customers.
