How to apply artificial intelligence technology to automated testing?

Artificial intelligence is the most advanced technology in the region, the program code, and data analysis for automatic visualization, rely on the integration of software and hardware to imitate some tedious or simple work of human beings.

This article is also about the discussion of artificial intelligence technology, the theme is: how to apply artificial intelligence technology to automatic testing?

Add AI to the test, and testing automation becomes possible. Leave boring, repetitive work to AI so that testers can spend more time thinking about other tests.

Gil Tayar, an evangelist and senior architect of Applitools, introduced us to how to apply AI technology to automated testing.

InfoQ caught up with Tayar to ask him about the major challenges facing testing today, what are the six levels of automated testing, what kind of progress has the software industry made in these six levels, how to automate visual testing, how we are applying machine learning techniques in testing, and what impact AI will have on the work of testers.

What are the main challenges facing testing today?

Tayar: Unfortunately, testing hasn’t gone mainstream yet. I remember back in the 80’s and 90’s, the industry as a whole didn’t take QA as a career or manually testing software seriously. Not only that there was no automated testing, but no testing at all! Well, times are different now, testers regularly test the software before delivering the product and many are starting to use automated testing.
But in today’s “Internet Age” this is not enough. As a result of agile methods, we need to deploy faster and faster, which is actually a good thing. This is a good thing because more and more developers are testing their software instead of waiting for specialized testers to test their software.

However, developers testing their software is not yet mainstream. Most developers don’t write their tests to check their code; they choose to test manually or wait for testers to test and make sure their software is working properly.

This is the main challenge facing testing, how to get developers to write their own tests. Unfortunately, AI doesn’t play much of a role in this, AI is just a tool to use when testing, but if you don’t test yourself, AI is useless. That’s just the way it is, AI can only be used as an aid to testing from now on and in the future, if you don’t want to test yourself, AI can’t automatically do it for you.

Developers have to think about testing and that’s the main challenge we face in testing.

There are six levels of automated testing. What are they?

Tayar: The concept comes from the five levels of automated driving. They describe how AI can help us with testing.

There is no automation at all, you need to write your tests!

Driver assistance: AI can look at the page and help you write the assertions. You still have to write the code that “drives” the application, but the AI can check the page and make sure the expectations on the page are correct.

Partial automation: While it’s nice to be able to tell the difference between the actual page and the expected value, the second level of AI requires a deeper understanding. For example, if all the pages have the same changes, the AI needs to recognize that these are the same pages and show us the changes. Further, the AI needs to look at the layout and content of the page and categorize each change as either a content change or a layout change. This can be very helpful if we’re testing responsive websites, where the content should be the same even if the layout has changed slightly. This is the level where tools like Applitools Eyes come in.

Conditional automation: in the second tier, problems and changes detected in the software still need to be reviewed by a human. Layer 2 AI can help us analyze the changes, but it can’t just look at the page to determine if the page is correct or not, it needs to be compared to the expectations to determine that. But Layer 3 AI can do this and more because it uses machine learning techniques. For example, tier 3 AI can look at a page visually and determine if the design is over the top based on standard design rules such as alignment, spaces, color and font usage, and layout rules. the AI can also look at the content of the page and determine if the content makes sense based on previous views of the same page, without human intervention. We can’t get to that level yet, but we’re working toward it.

Highly automated: until now, all AI has just been automating the checks. Humans, despite using automated software, still need to manually launch the test and click on links, whereas AI at the fourth level can automatically launch the test itself. the AI will understand how to launch the test itself by observing the behavior of the real user who launches the application. The AI in this tier can write tests and can test pages through checkpoints. But that’s not the end of it; it will also need to observe human behavior and occasionally need to follow tester instructions.

Fully automated: I must admit, this level is a bit scary. The AI at this level can “communicate” with the product manager, understand the product’s standards, and write tests on its own, without human help.

What kind of progress has the software industry made in these six tiers?

Tayar: Most companies are definitely at the second tier, to work toward the third tier. I believe it will take a while to reach Tier 4, but we can get there one day. As for reaching the fifth tier, I highly doubt it, but the future is immeasurable! (In the original article, the allusion to the destruction of the Second Temple by the ancient Jews is used as a metaphor for the future being immeasurable.)

How do you automate visualization testing?

Tayar: Actually, a few years ago, even developers couldn’t automate visualization testing, let alone use AI! Screenshotting apps and checking them against expectations was impossible due to the high rate of false positives from pixel-based algorithms.

But with the development of computer vision algorithms and the use of modern Artificial Intelligence (AI) techniques, writing visual tests has become a reality. One finds a lack of visualization tests in testing. Thanks to visual testing, we can now test every aspect of the software.

So how do we use AI techniques to avoid false positives, and can AI do a better job than algorithms that simply compare pixels? The answer involves more than just one technology, it requires combining different algorithms, one algorithm to solve an accuracy problem, and combining the results of different algorithms through a decision tree (sometimes applying the results of one algorithm to another) to determine the final result. But what algorithm to use? An example of this is the segmentation algorithm, which tries to determine the text and image parts of a picture. This is not a simple problem, as we need to figure out that emojis are text, and the text in the image is part of the image and not new text. This is where deep learning comes in, and thanks to the deep learning techniques used in our code, our accuracy increased from 88% to 96%.