
Etsy sellers aren't the only ones who have concerns about testing the company runs on its platform. Two techies at the company recently raised concerns about it on Etsy's Code as Craft blog.
Etsy data scientists Stephane Shao and Clare Burke were surprisingly candid about the potential for unintended negative consequence of running so many tests on its platform - which Etsy calls "experiments."
Etsy's widespread use of experimentation empowers its teams, they said, but it also means that "many different teams end up experimenting concurrently on shared pages or closely related products."
And that can render isolated results from individual experiments misleading, they said - and worse, it can lead to "negative interactions" where two or more tests, each with the potential to have a positive impact on their own, have a lessor or even negative impact when tested together.
The authors of the blog post used a hypothetical example involving the navigation bar on Etsy that features a black shopping cart icon on a white background. If one team ran a test to turn the white background to orange, and another team independently ran a test where it turned the shopping cart from black to orange, the shopping cart would no longer be visible to shoppers (the orange shopping cart wouldn't be seen against the orange background):
Over the years, sellers on eBay have also expressed concerns about testing the company does on its marketplace. The tests can be disconcerting, since marketplaces general don't disclose when they are running tests.
Testing can also waste sellers' (and buyers') time as they try to figure out why they're suddenly seeing something different. (The old, "is it a glitch, a test, or a permanent change?" routine.)
"Our migration to Google Cloud has enabled us to shift more than 15% of our engineering headcount from daily infrastructure management to improving the customer experience. The combination of increased brain power and significant compute power has dramatically sped up our innovation pipeline. In fact, Etsy's experiment velocity was up 115% in 2019; this process leads to meaningful insights that help us meet ever-evolving buyer and seller needs, ultimately delivering a more personalized, seamless experience."
Now, Shao and Burke are looking at how to look at Etsy experiments more holistically, which you can read about on the
Etsy Code as Craft blog.
While some sellers deplore encountering constant testing, some appreciate that marketplaces at least test changes for unintended consequences before they roll them out sitewide.
Have you noticed any differences in the level and nature of testing Etsy and eBay have been doing on their marketplaces? What are the best and worst tests you recall on either platform? What would you like Etsy techies to know about the impact of testing on your business?