A / B Test Scenario
To configure your Campaign to use an A / B Test scenario:
Search for the desired Regular One-off email Campaign (see Search for an Item for more details on the available search methods).
Click on the Campaign name. The classic Campaign screen is displayed and populated with the details of the selected Campaign.
In the Tool Ribbon, click Options > Audience > Split Into Testing Cells. The system displays a warning message. Please note that splitting a Campaign into cells is a permanent action that can't be undone. Confirm the action by clicking split into cells.
The system adds a new section to the Campaign screen named "Cells & Splits." In addition, the system adds a new component to the user interface called the Cells Tree. This component is displayed within a collapsible / expandable pane in between the System Tray and the Workspace. When collapsed, the component appears as a gray vertical bar. If you hover your mouse cursor over this bar, the system expands the Cells Tree component; you can collapse it again by clicking that same vertical bar. You can also expand this component by clicking Edit > Actions > Cells Tree in the Tool Ribbon.
The "Cell Code" field within the "Cells & Splits" section is automatically populated with a default cell code. Optionally, enter a custom value in this field.
Place a check mark in the "Allow sub-cells for" check box, then select "A/B Test" from the drop-down menu. The system displays additional options on the screen for configuring your A/B Test scenario.
Split Options
The next step in the process is to determine how you want to split your Campaign Audience, and to create the different testing cells.
Within the "Cells & Splits" section, from the "Split Options" drop-down menu, select the split method you want to use:
Split by Amounts -- Select this option if you want to create the sub-cells based on a specific record quantity (in increments of 500).
Split by Percentages -- Select this option if you want to create the sub-cells based on a percentage of the Campaign Audience.
Split by Field Values -- Select this option if you want to create the sub-cells based on the unique values in a field (or fields).
Split by Filters -- Select this option if you want to create the sub-cells based on inclusion in a Filter.
Note: You can't use different split methods within the same parent cell.
The "Order By" drop-down menu allows you to determine how you want to order the recipients in the Campaign Audience. For an A / B Test, you likely want to keep the default option of "Random," so that you don't inadvertently skew your test results. From this menu, select either "Random," or select a specific field. If you select a field by which to sort, select a sort order -- either "Ascending" or "Descending."
If you selected "Amounts," "Percentages," or "Filters" as the split method -- in the "Add New Fields" text field, enter the number of sub-cells that you want to create (not including the "Remainder" cell, which is created by default). If you selected "Field Values" as the split method -- in the "Add by Field" drop-down menu, select the field on the Campaign source table that should be used to generate the different sub-cells.
Optionally, if you want one of your sub-cells to be designated as a Control Cell, place a check mark in the "Include Control" check box.
Click the add button (plus-sign icon). The system populates the cell list with the number of sub-cells that you indicated (or, if you selected "Field Values," based on the number of unique values in the selected field). If you included a Control, this cell is marked with a test tube icon. The Remainder cell is created by default, and is marked with a star icon.
Note: When creating the hierarchy of cells for an A / B Test scenario, the platform allows you to create only one layer of "child" cells beneath the main "parent" Campaign. In an A / B Test, you can't further divide those "child" cells into "grandchild" cells. Unlike a Split Cells Campaign (which can create any number of levels), an A / B Test Campaign can have only a single level of child cells beneath the parent Campaign.
Each sub-cell is given a default Cell Code. Optionally, edit this value to provide a custom Cell Code. The values for the different Cell Codes must be unique.
Depending on which "Split Option" you selected above in step 1, you next need to define your individual sub-cells:
Split by AmountsSplit by Amounts
Determine the record quantity for each sub-cell. Best practices for testing are to make each sub-cell the same size, but you can optionally make the sub-cells different sizes if you want.
Same size: By default, the system "locks" the sub-cell sizes to keep them all uniform. Enter the desired record quantity in the "Size" field. The system populates this same value into the "Amount" field in every sub-cell.
Different sizes: Click the lock icon next to the "Size" field. This icon toggles off the "lock" feature, and enables the "Amount" fields on each sub-cell. Manually enter the desired record quantity into each sub-cell.
Note: If you added a Control Cell, you have to manually enter the desired record quantity for the Control group.
Split by PercentagesSplit by Percentages
Determine the percentages for each sub-cell. Best practices for testing are to make each sub-cell the same percentage, but you can optionally make the sub-cells different percentages if you want.
Same percentage: By default, the system "locks" the sub-cell percentages to keep them all uniform. Enter the desired percentage in the "Size" field. The system populates this same value into the "Percentage" field in every sub-cell.
Different percentages: Click the lock icon next to the "Size" field. This icon toggles off the "lock" feature, and enables the "Percentage" fields on each sub-cell. Manually enter the desired percentage into each sub-cell.
Note: If you added a Control Cell, you have to manually enter the desired percentage for the Control group.
Split by Field ValuesSplit by Field Values
Optionally, make any edits to the "Field Values" for the different sub-cells that were automatically created by the system.
Note: If you added a Control Cell, you have to manually enter the desired Field Value for the Control group.
Optionally, you can pick another field from the "Add by Field" drop-down menu, and click the add button to add more sub-cells.
Split by FilterSplit by Filter
For each sub-cell (and the Control Cell, if you created one), click the browse icon (magnifying glass icon). A "Browse Filters" pop-up window is displayed.
Browse to and select the desired Filter.
Optionally, you can further limit a sub-cell by defining a maximum record quantity. If you don't enter a quantity, then the system will populate the cell with the entire Filter result set.
Same size: By default, the system "locks" the sub-cell sizes to keep them all uniform. Enter the desired record quantity in the "Size" field. The system populates this same value into the "Amount" field in every sub-cell.
Different sizes: Click the lock icon next to the "Size" field. This icon toggles off the "lock" feature, and enables the "Amount" fields on each sub-cell. Manually enter the desired record quantity into each sub-cell.
Optionally, if you want to rearrange the sub-cells, click on the gray box to the left of the sub-cell row, and drag-and-drop the row into its desired new position.
Optionally, if you need to delete a sub-cell, click the delete button ("X" icon) to the right of the sub-cell, then click Edit > Actions > Save in the Tool Ribbon (you can't delete the Remainder sub-cell).
Note: The drop-down menu in the "Channel" column has "Email" selected, and is disabled. A / B Testing is available only in the email Channel, so this option can't be modified.
Test Conditions
The next step is to select what you're testing (that is, the Campaign element or elements that are different between each of the sub-cells), and how to determine the winner.
Within the "Cells & Splits" section, from the "Elements to test" drop-down menu, select one of the following options:
None
Subject line
Content
Any -- This option presents you with a list of all the possible Campaign elements to test, grouped into different categories. Place a check mark next to one or more elements. For A/B Testing, the possible elements that can be tested are:
General Settings |
|
Email Components |
|
Response Options |
|
|
|
|
|
From the "Determine and send winner" drop-down menu, select one of the following deployment options:
Manually -- the system won't determine a winner, and won't automatically deploy any messages to the Remainder sub-cell.
Automatically -- the system will determine a winner based on the business rules you provide below; in addition, the system will automatically send out messages to the Remainder sub-cell, using the "winner" variant, as soon as the winner has been determined.
If you selected "Automatically," you need to determine the business rules for defining the "winner." From the "By" drop-down menu, select one of the following winning conditions:
Unique Reads (includes "opens" and "clicks")
Unique Clickers (includes "clicks" only)
By default, the system will identify the winner as the sub-cell with the most activity of the type you selected in step 3. Optionally, you can define a threshold that states the winning sub-cell must surpass the other sub-cells by a certain percentage. To define this threshold, select "at least" from the "with improvement of" drop-down menu, then enter a percentage amount.
Define the testing period. Once this testing period is finished, the system will begin the process of identifying the winner. The testing period options are:
To define a testing window, click after a time of. Enter a duration in hours or days. This window begins once all test emails have been sent, and ends once the specific duration has elapsed.
To enter a specific date / time, click start on. Enter the date and time when the winner calculation process should begin.
Define the tiebreaker rules in the event that no sub-cell can be determined to be the winner. From the "If no winner (tie) send" drop-down menu, select one of the following options:
Random leader -- The system randomly selects one of the tied sub-cells to be the winner.
None -- The system will not break the tie; instead, you'll manually select the sub-cell to be the winner.
Select a specific sub-cell to be the winner in the event of a tie.
Optionally, if you want the system to send out a notification email when the A / B Test is completed, enter one or more email addresses in the "When test complete alert" field.
Sub-Cells
After you've defined all of the testing options as described above, the final step is to actually populate the different Campaign variants in each of the different sub-cells.
Note: It's important you keep track of WHAT you're editing -- either the top-level Campaign, or one of the sub-cells. If you're viewing a sub-cell, the name at the top of the Campaign screen uses the following naming convention to help you stay oriented: <Campaign Name>_<Sub-cell Name>.
In the Tool Ribbon, click Edit > Actions > Cells Tree to expand the Cells Tree panel.
Within the Cells Tree panel, click on the desired sub-cell. The system displays a stripped-down version of the Campaign details screen that shows ONLY the Campaign element (or elements) that you selected to test. For example, if you selected "Subject Line" from the "Elements to test" drop-down menu, then the Campaign details screen for a sub-cell will show only the Subject Line text field.
Note: If you select the Control Cell, you won't be able to edit anything, as the Control group doesn't actually receive any Campaign messages.
Enter or select the desired options for this sub-cell. You can also edit the Cell Code from this screen.
Repeat steps 1 through 3 to define the Campaign elements for the remaining sub-cells.
After you've defined all of the cells and sub-cells, your Campaign can now go through the regular life cycle of Proofing, Auditing, and Launching. You can send proofs from any cell (which sends proofs for only that cell), or from the top-level Campaign (which sends proofs to all cells). Auditing and Launching, however, must occur from the top-level Campaign, and not from a cell.