It can feel like marketplaces are incessantly running tests to make tweaks or test new features, which can prove disruptive to sellers' routines. And because marketplaces rarely disclose the existence of a test, never mind the nature of a given test, it has gotten to the point sellers can't differentiate between tests and glitches.
So when Etsy engineer Theresa Eades wrote about tests in a post on the Etsy Code as Craft blog, we thought sellers would be interested in what she shared.
First, she disclosed some examples of issues her team encountered in two prior tests (which Etsy calls "experiments"):
"Two years ago, when my team worked on adding video to the listing page, we had to restart the experiment because we forgot to include specific event logging for our feature.
"Last year we implemented the key functionality for an experiment with the "add to cart" button, but neglected to get accessibility and text strings translated early, costing us a delay till the translations could complete."
She then explained how she created a standard checklist to help reduce such pitfalls when conducting product experiments.
Here are some of the more interesting questions on her checklist:
- Is there anything in my experiment that could degrade performance of the site?
- What populations should be included or excluded from the experiment?
- When should users see this feature? (Which pages, signed in/signed out, mobile, desktop, etc.)
- Where/when should bucketing occur?
- Will the experiment conflict with any other experiments? Do the experiments need to run exclusively?
- What countries should the experiment run in (can impact translations)?
Take a look
at the others and let us know what you think of tests on Etsy and elsewhere. What problems do you encounter, and what changes would you like Etsy and other marketplaces to implement to reduce disruption to your daily tasks?