|Tue June 8 2021 12:21:43|
Etsy Runs Hundreds of Tests on Its Marketplace Every Day
By: Ina Steiner
Remember when Etsy used to share information with buyers and sellers about tests it was running on its site? That transparency was enormously helpful to sellers especially in cases where they encountered something new so they could check to see if it was a glitch or an intentional change the company was testing.
(As online sellers can attest, the latter - tests - sometimes leads to the former - glitches.)
Etsy runs hundreds of tests every day, it recently revealed - but they are run on "limited online traffic."
"Our desire for fast development naturally calls for ways to gain insights as early as possible in the lifetime of each experiment, without sacrificing the scientific validity of our conclusions," company engineers explained on Etsy's tech blog, Code As Craft
The good news - the post was about how Etsy engineers are using an approach that can shorten the duration of experiments by up to 20%.
Implementing changes without testing them first wouldn't make sense, but keeping sellers in the dark about tests can also be frustrating and can waste sellers' time when they try to determine why something is suddenly working differently.
Four years ago, Etsy told sellers
, "We know it can be frustrating to be surprised by a change in your shop when you're part of a test, but announcing experiments before they happen can bias the results. If you knew we were testing a new tool in Shop Stats, for example, you might use the tool differently, impacting the results of the experiment."
It's interesting to read the comments Etsy sellers left
when we wrote about that 2017 post about its experiments. Months later, Etsy changed the way it shared information about the experiments it runs on its site. (The link no longer goes to that announcement, but you can find in our coverage
Would you like to see Etsy and other marketplaces bring more transparency to testing? How do you weigh the benefits of testing before launching a change versus running tests without any notification or explanation?