Introduction

Optimization is the data-driven process of discovering and quantifying the most effective experience for your customers and prospects, both offline and online. This process of discovery is ongoing as more data and results present themselves. Like analytics, optimization is finding a permanent home in organizations that want to present the most effective experience to their consumers and prospects.

Optimization requires significant resources from analysts, content developers, project managers, and to varying degrees, IT. Organizations must also put robust processes in place to support these efforts. Optimization requires test intake and output processes, coordination with content developers, operationalizing test winners, and incorporating learnings across the organization.

Organizations that have invested in resources and established optimization processes have found it incredibly worthwhile both in net learnings and bottom-line results. The most successful organizations see tens of millions of dollars from their test winners.

However, organizations that have chosen the wrong testing platform have learned:

• Selecting a tool that can only facilitate simple testslimits where and to which audience you can execute a test.
• Choosing a tool that relies solely on the vendor to execute tests prevents you from instilling a culture of optimization and increases the time for executing tests.
• Not being able to integrate the testing solution with the in-house analytics solution undermines the visibility of test performance and limits the test analysis.
• Internal as well as external data points are valuable and essential to connecting to and learning from the consumer base.

Building an optimization program takes time. Talented analysts, inspired marketers, data-driving content developers, and a testing tool are key to making this happen. An organization needs to select the testing tool that both helps it get up and running with testing and sets it up for long-term success with operational best practices.

To assist in this process, Adobe has collaborated with Bryan Hawkins, Partner at Web Analytics Mystified, to identify the core components in selecting a testing solution.

Organizational considerations

When selecting an optimization solution for your organization, the following considerations are important.

• Who in the organization will be managing and operating the solution on an ongoing basis?
  • Does this group have the technical knowhow to execute a multichannel approach to optimization?
  • How will this group be enabled to work with other key stakeholders to facilitate execution, test intake, and communication of test results?
• Which goals does the organization want to achieve with the optimization solution?
  • What are the organization’s goals for facilitating or creating a culture of optimization?
• Will testing be limited to key landing pages or are there plans to incorporate tests across channels, such as within email, display, and mobile properties?
  • Have you considered automated approaches to testing versus rolling out marketer-driven tests?
  • Is there a need to integrate the testing solution with analytics or other data sets for analysis and enriched targeting?
• What are the key business goals impacted by iterative testing and targeting?
  • Retail—Increase purchases, revenue per visitor, average order value, cross-sell, upsell, lines of credit
  • Financial and insurance services—Targeting of content based on known visitor attributes
  • Travel—Conversion rates, cross-sell, upsell, credit options
  • Media—Page and video consumption, registrations, repeat visits
  • B2B high tech—Converting unknown visitors to known via form registrations and integrations with demand-generation platforms, enabling lead scoring and customer resource management (CRM) cross-channel optimization

After mapping out the key business goals, evaluate how the testing solution can address these goals. It is helpful to have the optimization solution provider show you what other companies have done, which also gives you insight for how it can be done internally.

Key capabilities to consider

Not all testing platforms are the same. Your optimization solution should provide different tools or capabilities that can work together or independently based on your organization’s short- and long-term needs and goals.

Running tests on your site requires different capabilities than running tests within email. Leveraging automated algorithms for testing product recommendations requires different capabilities than running a multivariate test (MVT). The ability to leverage the right mix of the capabilities described here will determine the long-term viability and success of an optimization program.

Types of tests

A/B tests

A/B tests are the most straightforward type of test one can use—you’re simply comparing one version (A) to one or more other versions (B).

Redbox makes extensive use of A/B tests to optimize site design and content. In one A/B test, the team discovered that changing button text from “find” to a more specific call to action related to “pickup,” yielded significant results. “Customers preferred the new wording and we saw a significant lift in engagement to the new program,” says Mike DiMiele, senior analyst for web and mobile eCommerce

Lenovo, the leading PC manufacturer of unit shipments worldwide, leverages A/B testing for their strategic marketing efforts with tremendous success. Lenovo tasked its digital marketing team with delivering an annual eight-fold return on its investment in the testing program. Within the first two quarters, the team exceeded the goal with a 12-fold return, with a 14% increase in RPV and conversion.

Multivariate tests

This test type allows you to test multiple elements and alternatives. It provides rich reporting around element interaction effects and element contribution reports that enable analysts to better understand which content, not just the experience, is impacting visitors.

Auto-optimizing

Some testing solutions offer auto-optimizing testing capabilities. This approach allows for real-time shifts in test variants while collecting test results. Similar to an A/B test in that you set up a discrete set of experiences, but instead of distributing visitors equally across the test experiences, automation changes traffic allocation based on how the different test experiences are doing. For instance, if you use this approach with three different experiences, and one experience is doing quite well, more of the traffic going forward will receive that experience.

This approach is helpful when running a test within email. If you’re sending an email blast to 100,000 subscribers, the first 5 or 10 thousand subscribers that opened it could provide enough signal to shift traffic so that the subscribers that hadn’t opened the email yet will be served the best performing experience.

Mutual exclusivity

As an organization develops its optimization program and gets to the point where it is running more than 10 tests at any given point in time, it needs to make sure that visitors don’t cross into different tests. Maintaining participation in a single test allows for the cleanest test results possible and the ability to attribute which visitor experience was successful. Not having the ability to keep visitors in mutually exclusive campaigns can create heartache for analysts as they try to interpret test results.

Organizations typically don’t consider the mutual exclusivity component when evaluating testing solutions, but they soon come across the issue three to six months down the road if they are executing a good amount of tests.

Testing different channels

Most organizations start their optimization efforts on their main website, where the majority of visitors spend the most time. While there are countless opportunities to find wins and learn what visitors respond to on this channel, there is great value in running tests across additional channels.

Email

Some organizations are doing some fascinating things optimizing email, such as testing content within emails as the subscriber opens them or using email platforms as a conduit to pass data to the testing platform. If email plays a significant role in your acquisition and customer communications, explore how the testing solution can help you.

Mobile

Mobile traffic is growing at an amazing rate and, in some cases, overtaking standard website traffic. You need to make sure that your testing solution provides optimization efforts for this channel. Mobile sites present optimization opportunities for testing layout, navigational elements, and imagery. Businesses that have adopted optimization strategies across channels are taking test learnings from the desktop site and testing them on their mobile site as part of their iterative approach to testing.

Display

Display advertising is another channel to incorporate into your testing efforts. Typically, organizations use traffic from display ads as a key segment for website tests, or they target content on the website from these visitors. Some organizations even do testing within the display ads based on data collected on their website.

Internal search and recommendations

Internal search and onsite recommendations have become powerful areas to optimize. Organizations that use software for these services see tremendous value from leveraging rich data for testing algorithms and the presentation of internal search result pages and recommendations.

Skullcandy, a leading audio brand, uses automated recommendations to create ideal presentations and to test algorithms. It had tremendous success, leading to a 30% increase in upsell revenues through iterative testing of recommendations. Skullcandy reduced administration of its website recommendation process from 10 hours to 30 minutes per week by moving from a manual recommendations system to an automated version.

If you have an interest or opportunity to incorporate dynamic recommendations or internal search into your testing efforts, it’s important to consider solutions that not only integrate well with your testing solution, but also with analytics and other internal data sources. You want a solution that offers reporting or visibility into how the solution is doing and what insights or learnings have been found.

Targeting capabilities

Targeting or segmenting subsets of visitors for analysis is the most efficient way to get as much value from your tests and testing efforts. Targeting can make or break organization success, so be wary of a testing solution that does not offer the ability to run targeted tests. Organizations that have embraced optimization would argue, “If you are not targeting tests, you truly aren’t testing.” Targeted tests offer some of the quickest wins in terms of results, while laying the groundwork for personalization and learning what your customers and prospects want.

Varying degrees of targeting options are available on the market today. Basic targeting options include environmental variables, such as browser, operating system, time of day, and day of week. Building on that, there are geographic variables, such as country, state, city, and zip code based off of reverse IP lookup. The more advanced targeting options include real-time behavioral targeting, which allows you to immediately run a campaign or test targeted to previous purchasers. For example, if a visitor made a purchase and then returns to the home page where the targeted test to previous purchasers was running, the visitor becomes a member of that campaign. This capability is essential to any business that has a strategic testing roadmap.

Additional options include the ability to target tests based on offline or internal CRM data. This can be powerful if you are a financial institution or a B2B company that wants to use internal data points, such as account types or sales-related information, for test targeting or segmenting.

If your organization makes extensive use of third-party data for display or analytics purposes, being able to use the data for onsite testing is a key consideration. A great example of this is DemandBase, which provides a reverse IP lookup to identify if someone is accessing the site via a corporate network. If you are in the B2B space and want to target content or tests to specific organizations or industries, this is a great way to accomplish that goal.

When selecting a testing solution to support these efforts, you need to evaluate just what is involved to integrate the data sets and which marketing technologies are in place. Also check what would be needed in terms of support and cost to maintain the integration. Find out which internal and external datasets you would like to make use of, and talk to internal stakeholders to see which targeting capabilities are going to be needed. Then evaluate how the different vendors can support those needs.

Analysis capabilities

Getting tests up and ready is only half the battle. Having robust tools that can analyze the results is critical to any company’s success in testing and optimization. Consider who is going to be consuming the test results in the organization and which mechanism they are currently using to consume data.

At a minimum, users of the testing solution should be able to understand how a test impacted multiple success events on the site. This understanding should be coupled with easy interpretation of statistical confidence and confidence intervals to allow for a correct interpretation around lift and the impact of the different test variants. In case you’re not familiar with these terms, statistical confidence indicates how likely a given success event was not due to chance. Confidence intervals represent the range where the true value can be found at a given confidence level. For example, let’s say you have 90% statistical confidence and the confidence interval is 3%–6%.If you run this same test multiple times, the expected range would fall between 3%–6%.

Reporting

The reporting capabilities should allow for easy understanding of how segments of traffic responded to the test treatment. Advancing the culture of optimization in an organization usually involves incorporating some targeted tests. The quickest way to do that is by seeing how different segments respond to different treatments.

Segmentation analysis also gives a closer look at the results. For example, lets say you run an A/B test and the overall results looked flat. But you might still have a winner: There could be one segment that responded with a 10% lift to experience A and another segment that responded well to experience B. Although the overall results look flat, two segments in fact show positive results.

If one of the major components of an organization’s success events is revenue, it is essential that the testing solution calculate extreme orders. Typically, extreme orders are purchases that are more than three standard deviations from the average order amount. For example, if the average order amount on a site is $100, a purchase of $12,000 would be outside of the three standard deviation window. If an organization doesn’t have the ability to recognize this, it can misinterpret the test results, leading to risky decisions.

Another key component of analysis is graph reports. Reports are key to optimization success. For instance, if test results show a significant win in a week’s worth of data, some would be quick to call a winner. But reviewing a graph report of results over time could show that there was a spike in the last day of the test while the rest of the time that it was running, the results were negatively impacting things. Successful optimization programs make extensive use of graphs for this type of analysis as well as for segmentation breakdowns, hour-by-hour results, and plotting multiple success metrics.

At a minimum, the reporting component of the optimization solution should have:

• Customization capabilities, such as graph customization based on counting methodology, success event mapping, and segmentation analysis
• The ability to share the tests results with key business stakeholders
• Easy, high-level consumption capabilities so that users can quickly understand what is going on as well as drill into advanced analysis

Segmentation

Testing solutions need to enable and support visitor profiles, an anonymous visitor ID that maps to visitor attributes. For example, a visitor attribute could contain information such as time since last visit, pages consumed, and purchased history. The visitor profile is one of the most valuable components of a testing solution because it enables targeting, data mapping, and segmentation.

Segmentation is the ability to define subsets of visitors into profile for analysis. For example, if you run a test on your home page and define segments such as paid search, repeat visitors, visitors from email, or direct visitors, you can use these segments to better understand how they responded to the different variants of the tests. Segmentation is the quickest way to advance general testing efforts into targeted testing.

Here is an example that highlights how important segmentation can be to a business. The following screen shot shows the average revenue per visitor for all test traffic as a whole. Experience C is the most effective experience, with a 4.65% lift in revenue per visitor.

But digging a bit deeper, we find hidden gems in this aggregate test result. Through segmentation analysis, you can see that those visitors who arrived to the site via Google (a segment of the total population) responded better to experience D with a 32.06% increase in revenue per visitor.

Scenarios like this happen all the time, which is why you want a tool that enables segmentation analysis and customization around segments. Organizations that establish strategic best practices around inclusion and evaluation of test segments set themselves up for substantial success and learnings. In the above example, a successful approach would be to push experience C to all traffic, except for those visitors that arrive from Google, who would be pushed experience D. Then you can use the new winners as the test control for iterative tests. Testing tools that support segmentation allow you to make the right decision while making it easier to mine test results for situations like this in which visitor segments respond differently to different test variations.

Work flow and collaboration

Setting up a test that changes visitor experiences can often be intimidating to business and marketing departments. HTML, JavaScript, cascading style sheets, and Document Object Model manipulation can be tough concepts to understand, so you want a testing solution that makes things easy and seamless so that it actually gets used.

You want a tool that makes it clear exactly who, what, when, and why you are launching a test as part of the creation process. There are many best practices when it comes to setting up tests and rolling them out, such as how to select visitors that the test is targeted for, selecting and creating the test experiences, inclusion of any and all success events, and key visitor segments for analysis. All this can be daunting, and one simple mistake might severely damage visitor experiences, post-test analysis, and the adoption of testing throughout the organization. Consider how solutions support these practices and how they can serve as guardrails for new marketers or different business units as they incorporate testing into their part of the organization.

Integration capabilities

While capability evaluation is important, you must also consider what is needed to make use of the capabilities. Implementation can make or break an organization’s use of a testing solution. All too often, an organization has licensed a testing solution only to have it sit on a shelf because of organizational challenges, often from IT, to get the solution in place. The team responsible for implementing the solution should be involved and sign off before a tool is purchased. Although this might lengthen the evaluation period, it pays off in dividends after the solution is purchased.

A mandatory requirement for any testing solution is the ability to integrate with the analytics solution that is currently in place in the organization. This integration has many benefits, but the two that organizations find the most valuable are:

• Visibility of tests and tests results across the organization
• Analysis of test results impact on success metrics not captured in the testing solution but available in the analytics engine

Integrating your testing solution with other key marketing toolsets in place, such as your email management platform, demand-generation platform, and third-party data solutions, allows you to make use of the data that these other solutions provide. You want to select a solution that is relatively easy to get in place, whether it is a single line of code, on premises, or hosted. It is also crucial that the tool integrate with your CMS, especially if all the test content comes from it.

If your organization makes use of tag management solutions (TMS), evaluate how the testing solution works with the system in place. A TMS makes data available for the testing solution so much easier, allows for easy and custom deployment, and can make data from the testing solution available for other marketing platforms.

APIs

APIs facilitate programmatic creation of tests, result data extraction, and augmenting visitor profile attribute. If your organization has IT heavily involved, it might be worth exploring how the testing solution makes use of APIs. Using APIs is crucial for augmenting visitor profiles managed by the testing solution with internal data attributes as well as third-party data. If you need the testing platform to be flexible and open for data to be easily ingested or extracted, API functionality should weigh high on your list of prioritized capabilities.

Requirementschecklist

Because choosing a testing solution is a substantial and important initiative, a requirement document or checklist should be used to help evaluate which solution and partner is right for you. Begin by mapping the key capabilities that allow your organization to be successful. Then align key stakeholders and get their collaboration for creating a detailed list. After you have collected the requirements, the key decision-makers decide what is a must have, a nice to have, or optional. During the evaluation process, record whether each requirement is supported by the vender as part of their core offering or an optional component that requires customization. And record what is required in terms of the implementation effort to support each requirement. After you have gathered this information, your organization can make a sound decision on which testing solution best meets its short- and long-term optimization goals.

If your organization is new to testing, you want a tool that enables and incorporates best practices in terms of setup and execution. Also consider what, if any, tactical and strategic support comes with the testing solution license. This initial support ensures that you get testing off on the right foot. Organizations that try to do it initially on their own often fail, rather than using valuable cycles to build and enrich a culture of optimization.

For organizations that are just starting off with testing or have limited experience using a testing solution, these capability requirements are essential:

• Work flow and collaboration best practices enablement
• Range of test type capabilities
• Comprehensive reporting interface
• Segmentation
• Targeting
• Real-time reporting
• Rich and robust targeting capabilities
• Customizable graphical analysis

For organizations that have a considerable level of experience with testing or are currently using a testing solution that is not achieving the expected value, the following are recommended in addition to the requirements above:

Integration capabilities, especially with the analytics platform

• Automation of test content
• APIs
• Mutual exclusivity
• Ability to execute tests across channels

Conclusion

To stay competitive and build relationships, organizations need to present the most effective content to their customers and prospects. Choosing the right testing solution and partner is the key to instilling a culture of optimization. When you have this culture, ROI, engagement, conversion, and customer loyalty are maximized.

As part of the organizational self-reflection around testing, consider your needs to support testing. You’re not just choosing a testing solution; you’re also choosing a partner that you are going to rely on for execution and best practices while supporting your desire to promote a data-driven optimization culture. The testing tool should help your organization go down the path of optimization and not stand in its way. Too much is at stake by choosing the wrong testing solution. The key requirements outlined in this white paper can help you get off on the right foot toward success.

Next Steps

Adobe, the sponsor of this paper, has built a Value Navigator tool hat can help organizations determine where the potential impact and conversion increase from testing lie across the respective areas of their site. Walking through the steps in the tool delivers a personalized recommendation of what you should use to increase your conversion rate.

About the author

Brian Hawkins, partner at Web Analytics Demystified, has worked in the optimization space since 2006 when he started at Offermatica, which is now Adobe Target. While working in a variety of roles at Offermatica, Omniture, and Adobe in both a technical and strategic consultative capacity, he has helped scores of companies get value out of their testing platforms and instill best practices that help organizations build a culture of optimization.