Etsy tried a test in which it omitted shipping costs from sellers’ listing pages, instead requiring shoppers to place items in their shopping carts in order to see shipping. After running the test for 2 and a half weeks, Etsy shared information about why it ran the test and the results.
Many sellers were unhappy that Etsy would hide shipping costs from shoppers, fearing it would make them look as though they were deceiving buyers. In a long announcement, Eric Fixler, Director of Shipping Programs at Etsy, apologized for “any angst and confusion brought about by the amount of testing we’re doing” as part of an acknowledgement that Etsy was running a number of tests in addition to the shipping-cost test.
But on Monday, an Etsy moderator announced Etsy was continuing with another round of tests related to the way shipping costs are shown to buyers. “This next round has incorporated the feedback that you offered regarding the first test.”
In Fixler’s announcement on Friday, he explained the challenges of communicating shipping costs to buyers and said, “We want to provide better, simpler and faster options for pricing shipping. The current model works well for many of you, and we’ll continue to support it, but the way that “item plus with another item” is embedded in the presentation of shipping costs to buyers makes it difficult for us to provide other alternatives for sellers, and it’s also not always the clearest way to explain shipping prices to buyers.”
Fixler said in order to understand the right way forward, “it was important to understand and measure the direct effects of removing shipping prices from the listing page entirely.”
So what did the company learn from its first round of testing? “Conversion was not affected by the changes to listing page. As you might reasonably expect, we saw a measurable, but not very large, increase in the number of items added to the cart, and a corresponding increase in the number of items added to the cart but not purchased. These changes essentially cancelled each other out.”
And on the downside, Etsy found that people in the test group contacted sellers at a rate 1.5% higher than people not in the test group. “This is a concern for us, as it is for you, and we’ll be watching this number closely as we evaluate future changes to the display of shipping prices. We’re also running further analysis on our test data to understand better how the changes may have generated different reactions among different sets of Etsy members.”
With many sellers complaining about the duration of the test, Fixler said, “For these sorts of tests, we need to track many sessions in order to be sure that our analysis is meaningful. We only show the test versions to a minority of people, so it can take some time for the data to accumulate. That said, we monitor the interim data constantly over the course of the test. If we see that a test is negatively impacting shop revenue at any point, we will end the test immediately. We can shorten the duration of the tests by including more members, but we try to strike a balance between the length of the test and the number of people in it to minimize disruptions to you.”
About six hours after announcing phase two of the shipping-cost test on Monday, Etsy announced the test was no longer active due to an issue. Without getting specific, Etsy said, “If you were affected by this issue, we are sorry for any inconvenience it may have caused you.”