Hi Everyone,
We’re looking into automated testing of our OEM product and was wondering if anyone else had any success with this and what you’ve used to do it?
We very briefly tried a couple of the WinAppDriver type implementations (Winium and Appium) but found that we just couldn’t get them to locate the buttons on the ribbon or any of our forms. I can’t be certain whether the problem lay with them or us - it was only a brief investigation.
We’re about to look into these three for starters as they’re more graphical and potentially don’t require developer involvement (at least that’s what it looks like at first glance):
Thanks in advance.
Paul
Paul,
I have not looked into those but if you happen to test them, please let us know how they work.
Since ones we tried out in the past didn’t really work, I would be interested to find out if you found one that was successful.
Thanks
Shawn Golden
Microvellum
1 Like
Hi Shawn,
My findings are as follows.
Katalon - unfortunately, this one was just way too slow to be useable. It took around a minute to read the interface to figure out what buttons and elements were available and then took around two minutes to perform a button click. Shame 'cos it looks like quite a comprehensive bit of software. Maybe testing of Web applications would be a more positive experience but the speed of this one just made it impractical.
Telerik Test Studio - this one was a lot quicker but I found it to be quite flakey. It would often just ignore clicks or it would record the click but then not actually perform the click on the button meaning that it required clicking the button again to continue with the recording of the test but then you had to go back later and manually delete the duplicate click. I also found that if it failed to record a click it would then subsequently not record anything else requiring the test recording to be abandoned. I did manage to get it to click one of our ribbon buttons and to run one of our commands on a drawing but there was then no way to verify if the drawing was then in the expected state.
Smartbear Test Complete - this one came out head and shoulders above the other two. Easy to record tests, clicks were always recorded. Tests can be reused as steps in other tests. Additional steps can be recorded on the end of an existing test or even in the middle. The most important thing about Test Complete was that it allows for taking an image at certain points in the test which it then compares against when the test is run. This is the only option available to verify that the button click has produced the desired result in the drawing. For all three pieces of software, the elements in the drawing were invisible to them so a visual comparison is really the only way to verify that the test has succeeded. It does mean that minimal, test-specific drawings would need to be set up so that it was easy to verify visually but it’s better than nothing.
I think relying on the images would make the tests quite brittle and I did have some false failures with the image comparisons but I think that might be resolvable with some training or more time (apparently you can create masks on the images to look at certain bits only but I ran out of allotted time to get too deep into that).
All three had all the CI/CD capabilities and there was a lot in each that I didn’t even look at. I was focusing on proving the viability.
To be clear, I’m not affiliated with Smartbear 
. Just sharing my experience.
Paul
1 Like
Thanks for sharing!
Shawn
Microvellum