Questions Marketplaces Should Ask before Running Tests

by: Snapped

“Two years ago….we forgot…”

“Last year…neglected to…”

“ She then explained how she created a standard checklist..”

That’s the problem, right there.  Experiment first, THEN ask questions?  Why wasn’t such a ‘checklist’ developed and mandated before the first hack was unleashed?  It has to come to being as a ‘personal’ and ‘evolving’ tool years after the fact?  Then blogged about as if it were some new ‘discovery’?

It’s quite evident this - and too many other - foundational elements in complex interfaced system’s SW T&E are sorely lacking for all these platforms.  One doesn’t need a 25+ year experience base in DoD systems integration and SW T&E (since before ‘computer science’ existed as a discipline) to see that.

SW testing has three levels that must be examined and proved operational BEFORE the new version code is dumped on the unsuspecting user, and even then as a ‘test’ only, prior to official release and reliance.

Start with specification testing.  Independent of any ‘utility’ assessment, does the code DO what it was SPECIFIED to do when the button is pushed?  (Example - does the number 6 change to a number 9 as designed?). That should be exhausted in a lab setting, disconnected from any operational interface.  And it should be done for each ‘box’ n the system chain impacted independently, be it by code changed, or due to a changed interface message as result of code change.  

Next comes functional testing.  With the code processing as specified, is the JOB (result) what was specified?  Is the man/machine interface supported and viable?  Are the data within tolerances?  Any smoke or seizure issues?  Is that new number 9 displayed in the correct place for the correct time in the correct user application?

Then comes integration testing.  Again, per specification, if I push the button on box A, does box C respond correctly without melting down something unintended in Box B, arrayed between them?  When I change the 6 to a 9, does it scramble the other digits in the database somehow?

And ALL of this should be done on specially designed TEST platforms, identical as possible to real world conditions, but not the actual real world systems who’s ‘results’ can wreak havoc on lives and livelihoods.  

Software testing is a science.  It’s not a ‘craft’ to be conducted as an afterthought to coding.  Not hunt and peck and see what happens so one can get better at it.  Results should be predicted and predictable, and if not, stop THERE and understand it, then fix it before proceeding.

And THEN, only THEN, can the new version be tried ‘in the field’.  But even so, all users should be specifically alerted, set population limited to ‘test conditions’, and data still collected, so any further ‘glitches’ can be corrected BEFORE an official release.  That’s the ‘beta’ testing most are familiar with.

Moreover, strict version control must be maintained.  That’s called configuration management, (what works with what, and what doesn’t), and it seems to be a foreign concept to these guys.  Especially critical if there are user ‘classes’ or ‘types’ or ‘systems’ that are to be ‘excluded’ from access to the changed versions by operational convenience or necessity.  It’s also critical to maintain in case the grand experiment goes south, so the previous ‘working’ version(s) can be quickly re-established.

The OP author has it right to establish and rely on a checklist.  And, though not mentioned, specific and established (certified) test PROCEDURES.  But that should exist BEFORE the first test ever occurs, not be developed from them, after the fact.  And just as an aside, it can’t be both ‘personal’ and ‘standard’.  

Clearly, adequate SW T&E costs money.  Much cheaper to toss it out there and THEN (try to) fix (what SEEMS to be) the problems - but as it has been clearly demonstrated, not only does that make finding and fixing the actual problem more difficult and takes more time, there’s no guarantee the fix is adequately tested either.  Sound familiar?

And what we are left with is Swiss cheesed code, constantly glitching, constantly being “worked hard” on by cluelessly disconnected “teams”.  

One last thought for the toolbox.  Don’t just test what ‘should’ work.  Test what shouldn’t work too, to be sure not only it doesn’t, but doesn’t cause some other issue.  

And this is by no means limited to an ‘Etsy’ problem either.  

by: shut1968

A long long time ago Ebay use to do beta testing before making live changes to the site, there would be almost like a second site with the changes and sellers would get invited to test it out before hand for any bugs.. then when it went live for everyone if there was an issue you got credits etc etc for any down time... somewhere along the way they just said screw sellers, make the changes and fix it as you go.. I personally believe it should be illegal for any Marketplace selling site to do changes without beta testing first since people's business and livelihood are not affected.

I can remember one site about 15 years ago. can't think of the name of it now, it was a decent selling site back then, "offer" or something like that did an update that crashed the site and wiped most of it's database in mid November, right before the holiday rush and didn't get it back up and running right for 6 months at least.. screwed all their sellers and was never the same from that point forward.

by: LDWCallsOut

Are you thinking of iOffer?  They used to dominate the counterfeit luxury brand market online, and then Etsy took that over.

by: The End

Ya, Ya, Ya,...Blah, Blah Blah.....
I was there when Etsy first started.....around 2008.
It was sleek and functional.
I brought the action.
And I could make a decent living.
The new "humanoids" ruined it.
It's useless as good steady income now.
The new owners crippled Etsy beyond repair.
And I firmly believe they meant to do it.

by: Bubbles

Like most web platforms - they throw new stuff at the wall to see if sticks - then they TRY to fix the problems it has created when it doesn't - usually creating even more problems that never get fixed.

They waste so much time, money & resources making changes this way instead of having a duplicate site to experiment with and get it working perfectly BEFORE applying it to the main website.

by: spooky

last month the eBay app for iphone became unusable to list an item. One glitch after another. I contacted eBay Twitter and told them about it. After a week I contacted same again and they said there is a ticket on it and they are aware and working on it. I knew what it was and told them - they had made "upgrades" attempting to have the seller buy high visibility add ons at extra cost but those links had made the app unstable. It took 5 weeks to correct. Multi billion dollar company can not get the most simple basic things to work!  

by: Queen of the Jungle

Until the geniuses at Etsy can fix the search function, they shouldn't bother with anything else.

All of their engineers and tech people are politically correct hires who "check all the boxes" when it comes to race, sex, political affiliation.  People who work for Etsy are not there because of merit or skill.

by: LDWCallsOut

WJW, Queen of the Jungle!  Your post nailed it.  

About Etsy's politically selected hires, that does seem to be the way employees are selected.  It's not only the tech people.  I remember more than a decade ago an Etsy staffer (anyone remember Hey Michelle?) posting in the Forums that she felt bad when processing reports of reselling that would get items removed or shops closed because that could interfere with a reseller's ability to earn a living!  Like, who do you think you're working for, sweetness?  

Last week I was in an Etsy ''experiment'' with search results pages.  At the top of the first page of results were two new lines of smaller thumbnails.  The top one was headed ''most popular'' for (my search term), and the second on was ''priced from $50 to $100'' for (my search term).  I am using a larger screen desktop, and the experiment pushed the actual search results down enough on the page so that no real results were completely visible above the bottom of the screen! without scrolling down.  Who dreams up this kind of feature?

Even worse, the items shown in the 2 experimental lines didn't match my search terms at all!  When you are searching for handbags, what do pet items (for dogs only), items for babies and pre-school age children, wedding items, and crafting supplies fit in?  And when I tried searching for other kinds of items, the experiment was always off in a similar way.

After Etsy started including description text keywords in their algorithm for search results, search has gotten much worse -- more than I would have thought possible.  With the holiday shopping season just weeks away now, it might be a good idea for Etsy to clean up the mess they made with reading descriptions for keywords rather than try to force shoppers to look at more dog, baby, toddler, wedding, and DIY crafting supply listings.  

by: Sierra

@Snapped - I agree with what I could understand in your comment. YES, testing should be done on a mirrored server, as it used to be, but I'm told that's become incredibly expensive ... so that's apparently why many/most sites now test on the live site. I **totally** agree with your comment about ''cluelessly disconnected teams.'' For YEARS, I've had a feeling that Etsy's internal teams don't communicate well, if at all.

@TheEnd - Etsy started in June 2005. I opened my Etsy shop in August 2006 and have been there ever since. As a public company, aren't the shareholders the ''owners''? Did you mean the new management (Josh, in particular, who isn't so new anymore)? Etsy started downhill in October 2013 when ''production partners'' (then called ''outside manufacturers'') was officially announced. I hope someone can get Etsy cleaned up and make it a better version of what it was originally intended to be, but I'm not holding my breath on that.

@LDW - I remember ''HeyMichelle!'' Several years ago, Josh spoke about his ''cushion to couch'' strategy; this is about selling higher-priced goods and is apparently a strategy to make lower-priced imports sink farther down in search. As for seeing irrelevant listings in Etsy search, they're just copying Amazon. Again.