Components

• 1: Configuration
• 1.1: AutomatedWebTest
• 1.2: WebConfig
• 1.3: WebDriverProvider
• 1.4: Browser Config Profiles
• 2: UI Elements
• 2.1: Locators
• 2.2: WebFinder
• 3: Commands
• 3.1: Web Commander
• 3.2: Web Inspector
• 3.3: Logging
• 3.4: Extensibility
• 3.5: Browser Commands
• 4: Conditions & Expectations
• 4.1: Validations
• 4.2: Waits
• 5: Test Results
• 5.1: Console Logs
• 5.2: HTML Reports

1 - Configuration

1.1 - AutomatedWebTest

The AutomatedWebTest is the base class for all test classes in a Qadenz-powered test project. This class is responsible for gathering parameter values from the TestNG Suite XML file, configuring and starting a WebDriver instance on a Selenium Grid for each test method, stopping the WebDriver after each test, and invoking the reporters after the Suite has completed.

Classes that hold @Test methods must extend this class in order for tests to run using Qadenz configurations.

The Execution Cycle

Before the Suite Begins

The first task performed after launching a Suite execution is to capture a timestamp and save as the suiteStartDate value on the WebConfig. This will be used later by the reporter to calculate the duration of the execution. Next, the Suite-level parameters are retrieved from the ITestContext, which were specified by the user on the TestNG Suite XML file. The gridHost is validated and saved.

Before Each Test

Repeated before each test method, the Test-level parameters are retrieved from the ITestContext, which are the browser, browserVersion, browserConfigProfile, platform, timeout, appUrl, retryInterceptedClicks. These parameters are validated and saved to the WebConfig.

Once all the parameters are processed, the WebDriver can be launched and execution of a test method can begin. The CapabilityProvider is invoked to configure the browser session. The CapabilityProvider calls values on the WebConfig and attempts to load any Browser Config Profiles declared in one of the browser configuration JSON files. This process chooses a browser, determines if a specific version of the browser is required if a version has been declared, ensures that the test will be run on the appropriate OS if one has been specified, and applies any arguments provided on the JSON file if a matching Browser Config Profile was found.

These options are then used to configure and initialize a WebDriver instance, which is in turn set on the WebDriverProvider.

Finally, the browser window is maximized, and the appUrl value is loaded.

After Each Test

At the end of each test method, the WebDriver is stopped.

After the Suite Ends

The final task of the execution cycle is to capture a second timestamp as the suiteEndDate on the WebConfig.

Reporting

The default TestNG reporters are not disabled by default, so the standard HTML & XML reports will be generated, along with the emailable-report.html. Next, the TestReporter is invoked which generates the Qadenz Reports in JSON and HTML formats.

1.2 - WebConfig

WebConfig stores information that Qadenz uses at various points throughout the execution cycle. Primarily, these are the values given as parameters on the TestNG Suite XML file. While Qadenz calls these values during the setup and reporting phases they are made available on the chance these values are needed for any project-specific configuration.

Since these fields are all static, there really is no need to extend this class. Teams are encouraged to follow this pattern, though, and create a ProjectConfig class of their own should the need arise to track specific values or project-specific Suite parameters are in play.

1.3 - WebDriverProvider

The WebDriverProvider stores the WebDriver instance and makes it available to components where it is needed. Since the test classes work on an inheritance hierarchy, and with how TestNG manages threads, the WebDriver instance is required to be kept on a ThreadLocal<WebDriver> object. Rather than forcing testers to pass the WebDriver around to various library components, Qadenz has chosen to make the ThreadLocal static, and allow components that need the WebDriver to call a centralized location. On the (hopefully) rare occurrence where the WebDriver is needed directly, it can be invoked with the following call.

WebDriver webDriver = WebDriverProvider.getWebDriver();

1.4 - Browser Config Profiles

In some cases, extra configuration arguments need to be passed to the browser Options instance for WebDriver configuration. A common case for this is instructing a browser to run in headless mode. Qadenz provides a simple means to set these configurations, and allows for multiple “profiles” of these configurations to be set on the TestNG Suite XML file. For example, if a tester wishes to run in headless mode on the Selenium Grid, but run in a native browser when executing locally, the switch is as simple as altering a parameter value.

Configurations are stored in JSON files within the project’s resources directory. To create a new configuration, within the resources directory, add a new folder called config, and create a new JSON file. This file must be named {browser}-config.json, where {browser} is the name of the browser being configured. A separate file must be created for each browser requiring configuration.

This example configures Chrome to run in headless mode.

[
    {
        "profile": "headless",
        "args": [
            "--headless",
            "--window-size=1920,1080",
            "--disable-gpu"
        ]
    }
]

The JSON format is an array of configuration objects. Each object holds profile and args fields. The profile field will be matched to the browserConfigProfile parameter on the Suite XML. The args is an array of individual arguments to be passed to the WebDriver configuration.

2 - UI Elements

2.1 - Locators

The Locator is the central component of UI modeling with Qadenz. It is a key ingredient in an approach that seeks to improve the UI modeling by avoiding the PageFactory class and @FindBy annotation entirely. It’s design sets out to accomplish several things. First, the Locator is a clean wrapper for both an element’s selector and a display friendly name. Second, the Locator is a vehicle for parameterization of element selectors, leading to much more efficient UI models. And finally, the Locator carries attributes that assist with validations and element inspections.

In short, Qadenz is quite happy to follow Simon Stewart’s advice, and find a better way.

Basics of a Locator

The Locator object simply carries the name of an element, the element’s selector.

public Locator(String name, String selector)

An overloaded constructor allows one Locator instance to be passed to another, allowing a child relationship to be defined and to reduce repetitious selector segments. This approach will append the selector of the child to the selector of the parent, allowing the parent to be used as a reference point for one or more child elements.

public Locator(String name, Locator parent, String selector)

Display-friendly Element Names

Selenium does not readily (nor should it, being simply the tool that automates browsers) provide a meaningful context-friendly reference to element names, which can complicate debugging and troubleshooting when problems arise. Without additional logging in the test project, testers are generally only provided references to elements expressed as the given selectors. This has a very strong potential to slow down the resolution process as testers must first translate the selector in their stack trace to an actual element on the UI, which can then establish a point of orientation within the progression of test steps.

Locator signInButton = new Locator("Sign In Button", ".btn-signIn");

By requiring a name value to be given in the Locator constructor, Qadenz refers to the display-friendly name of an element as the primary identifier in all logging and reporting output. This eliminates the time needed to perform any lookup or cross-referencing of selectors to elements in relation to test steps. By reviewing the default logging output or report content, the point at which a problem appears in a test is clearly marked and quickly identified.

CSS Selectors

Many teams will choose either a default standard selector strategy (ID, CSS, XPath, etc.), or at least set an order of prioritization for the varying types. Qadenz has chosen CSS selectors for the superior performance, ease of use, and better browser compatibility when compared to XPath. The addition of the SizzleJS library provides a number of pseudo-classes that make element selection more flexible and accurate.

The end result is a powerful and straightforward selector strategy that supports full parameterization capabilities. This also leads to a singular defined approach that an entire team can adopt and consistently apply across the entire project.

Parameterization

Parameterization is a simple and effective means to reducing repeated code, and increasing the reusability of each component.

Selector Parameters

Using list of search results as an example, testers would be forced to create a separate @FindBy annotated WebElement instance for each result element needed for a test. Alternately, a @FindBy could be used to initialize a single List<WebElement>, but additional logic would be necessary to identify a specific result element needed for a test step. In either case, the result is many extra lines of code.

Using a parameterized Locator (coupled with the benefits of Sizzle CSS Selectors), testers will be able to define a single Locator instance for a generic search result, and rely upon the parameterization to direct the test to choose the appropriate element instance.

public Locator searchResultLink(String name) {
    return new Locator(name + " Search Result Link", ".search-result:contains(" + name + ")");
}

In this example, we also have a benefit of passing the parameter to the name field on the Locator, which increases clarity in the logs and reports by providing the exact instance of the element against which the interaction takes place.

Parent Locators

The Locator can hold an optional instance of another Locator. This is intended to allow abstraction of selector segments by combining the selector of the parent Locator with that of the current Locator as a single selector value. This can be helpful in reducing repeated selector segments when creating Locators for closely related UI Elements.

Consider an e-commerce application wherein a list of catalog items are presented on the UI. Each item card contains the item name text, a ‘Cost’ value, a ‘Quantity’ field, and an ‘Add to Cart’ button.

As a very simple HTML representation:

<div id="item-list-section" class="grid">
    <div class="item-row even">
        <div class="item-card">
            <div class="item-name">ACME Rocket Powered Roller Skates</div>
            <div class="item-cost">$99.99</div>
            <div class="item-qty input-field">
                <input type="text"></div>
            <div class="item-add action-button">
                <button type="submit">Add to Cart</button></div>
        </div>
    </div>
<div>

The selector for the ‘Add to Cart’ button could be:

#item-list-section .item-card:contains(ACME Rocket Powered Roller Skates) .item-add button

While mapping the other elements on an item card, however, it would be discovered that the item card selector itself is repeated on each of the child elements. In this situation, a parent Locator could be created to abstract the repeated selector segments, especially if the abstracted selector can stand as it’s own element mapping.

With a parent Locator the resulting element mappings for the item card, ‘Quantity’ field, and ‘Add to Cart’ button could be:

public Locator itemCard(String itemName) {
    return new Locator(itemName + " Item Card", "#item-list-section .item-card:contains(" + itemName + ")");
}
public Locator itemQuantityField(String itemName) {
    return new Locator(itemName + " Quantity Field", itemCard(itemName), " .item-qty input");
}
public Locator itemAddToCartButton(String itemName) {
    return new Locator(itemName + " Add to Cart Button", itemCard(itemName), " .item-add button");
}

In the above example, the itemCard() Locator could have the added benefit of not only being an abstracted selector, but could also serve as the target element of a verification of a given item card element to be visible on the page.

Parent Locators are also not limited to a single layer. Locators can be passed as parent references as many times as needed.

Definable Element State Attributes

There are some cases where the UI under test experiences conditions where traditional element state inspections are unreliable due to styling, DOM structure, UI framework in use, or perhaps even unconventional UI development. This can sometimes produce incorrect results on inspections like visibility, selected, or enabled states. To mitigate this issue, the Locator allows three additional fields to be set to assist evaluating these element states.

These fields are available as setter methods on the Locator class, and each will define an HTML attribute and expected value that will be added to the corresponding element inspection methods on WebInspector. Each of these methods are overloaded to allow a tester to define either an attribute/value combo, or just an attribute name in the case of empty or boolean attributes.

When a WebInspector method that supports custom attribute checks runs, the attribute check will be appended to any default checks made to determine the given state of an element.

Disabled Elements

The WebInspector.getEnabledStateOfElement() method checks <input> elements to determine whether the element is enabled for user input. This method presumes the element to be enabled, and performs each check in an attempt to prove the element is disabled.

Consider an example where a form is present on the UI that contains a checkbox that is only enabled for input under specific conditions. The form UI is heavily stylized, and the UI developer has chosen to create checkboxes using CSS with no underlying <input> element. Calls to WebElement.isEnabled() are not reliably returning an accurate result due to the UI design.

The tester has identified the CSS class that renders the checkbox inoperable, and will configure the Locator to provide this information to WebInspector to yield accurate inspections.

Locator iAgreeCheckbox = new Locator("I Agree Checkbox", "#i-agree")
        .setDisabledByAttribute("class", "checkbox-disabled");

To specify a custom attribute that determines the element as disabled, the setDisabledByAttribute() method on the Locator must be called.

Hidden Elements

The WebInspector.getVisibilityOfElement() method checks for elements that match the provided selector, dimensions of the element, and standard W3C defined means of rendering elements invisible. This method presumes the element to be visible, and performs each of the checks to attempt to prove the element is in fact hidden until a check proves the element hidden (and returns a result accordingly), or no additional checks can be made (in which case the element is determined to indeed be visible).

Consider an example of a “Confirm” button that is present on a form page, but is hidden from view until the form input has been completed by the user. Unfortunately, the UI developers have taken an unconventional approach to hiding this element, and normal visibility inspections are determining the element is visible. The tester has identified the CSS class that hides the element, and is able to pass this information to the WebInspector for a more accurate result.

Locator confirmButton = new Locator("Confirm Button", ".customButton-confirm")
        .setHiddenByAttribute("class", "invisible");

To specify a custom attribute that defines the element as hidden, the setHiddenByAttribute() method must be called.

Selected Elements

The WebInspector.getSelectedStateOfElement() method checks element such as checkboxes, options in a <select> menu, and radio buttons to determine whether the element is selected. This method presumes the element to be unselected and performs each check in an attempt to prove the element to be selected.

Revisiting the example above for the disabled checkbox, the UI developer has also created an animated interaction when the checkbox is selected. Calls to WebElement.isSelected() are not reliably returning an accurate result.

The tester has identified the resulting CSS class responsible for rendering the checkbox as checked, and will configure the Locator as required.

Locator iAgreeCheckbox = new Locator("I Agree Checkbox", "#i-agree")
        .setSelectedByAttribute("class", "checkbox-checked");

To specify a custom attribute that defines the element as selected, the setSelectedByAttribute() method must be called.

Fluent Design

The setters on the Locator for each of the element state attributes all return a self-reference. This allows the setter calls to be chained together. In the examples above where the “I Agree” checkbox has both disabled-by and selected-by attributes defined, the Locator can be instantiated and both configurations can be made in one chained method call.

Locator iAgreeCheckbox = new Locator("I Agree Checkbox", "#i-agree")
        .setDisabledByAttribute("class", "checkbox-disabled")
        .setSelectedByAttribute("class", "checkbox-checked");

The LocatorGroup

The LocatorGroup allows multiple Locator instances to be combined together on a List, and acted upon as a group. This is commonly applied to verify the visibility of UI component, or a set of default UI elements. Instead of passing multiple individual Conditions to a .verify() or .check() validation, a LocatorGroup can be verified with a single Condition call.

For example, a simple authentication form has several basic elements, the ‘Username’ field, the ‘Password’ field, a ‘Remember Me’ checkbox, and a ‘Sign In’ button. Each element would be mapped as an individual Locator instance for the purposes of input, but these same elements could also be included in a LocatorGroup, should the need arise to verify each element to be visible as part of the form.

Locator usernameField = new Locator("Username Field", "#username");
Locator passwordField = new Locator("Password Field", "#password");
Locator rememberMeCheckbox = new Locator("Remember Me Checkbox", "#remember-me");
Locator signInButton = new Locator("Sign In Button", "#sign-in");

LocatorGroup signInForm = new LocatorGroup("Sign In Form",
        usernameField, passwordField, rememberMeCheckbox, signInButton);

By combining individual Locators onto a LocatorGroup instance, testers are able to identify and refer to collections of elements as the UI Components they represent.

2.2 - WebFinder

WebFinder initializes WebElements. This is not a class that will typically be invoked directly in day-to-day use of Qadenz, but it’s a class worth familiarity in terms of how Qadenz initializes WebElements for use in Commands classes, especially if the need to create custom commands should arise.

On-the-fly Element Initialization

When Locator objects are passed to a Command, the WebFinder is called to initialize a WebElement using the CSS selector from the Locator. This process is performed for each interaction, inspection, and validation method, immediately prior to the WebElement being acted upon. The intent of this approach is to mitigate situations that might produce a StaleElementReferenceException. Additionally, when errors are encountered while initializing or interacting with a WebElement, the problem can be captured and presented in the logs/reports far more clearly.

To Wait, or Not To Wait…

The WebFinder is the only class where the ExpectedConditions class is used within Qadenz. Methods are avaialable to initialize either a singe WebElement or a List<WebElement> using no explicit wait at all (immediate initialization), with a wait for the element(s) to become visible, the element(s) to become present on the DOM, or a wait for the element(s) to become clickable.

These initializer methods will attempt to initialize the given element every 500 milliseconds up to the limit of time expressed by the timeout value. If the attempt reaches the timeout, an appropriate exception will be thrown and the test will be stopped.

In the case of WebCommander, methods that perform a click action will initialize the target WebElement after it becomes clickable, and all other interactions will initialize the WebElement after it becomes visible.

On the WebInspector, methods wait until the target element is visible prior to acting, with the exception of methods that interact with multiple instances of a Locator. In these cases, the List<WebElement> is initialized with no wait.

3 - Commands

Commands are a collection of classes that either act upon elements, interrogate elements, or control the browser under test. The methods on these classes represent steps in a test case. Commands can be used directly inside tests, or can be wrapped on Page Objects to create a contextually friendly reference to actions performed on the UI under test.

Methods on the Commands classes drive the supporting functionality that makes Qadenz useful. In addition to performing the surface actions, Commands methods handle WebElement initialization, catching and logging exceptions, and capturing screenshots.

Commands classes cover default Selenium interactions and cover some additional common actions, but are designed to be extensible so that specific needs of an automation project can be met.

3.1 - Web Commander

The WebCommander performs actions against WebElements. This class also extends the abstract class Commands, which is built to be agnostic of any automation tooling. From this class, functionality such as validations, time-based waits, and log wrappers are available to inheriting classes. This design was intended as a future-proofing measure should Qadenz at some point expand to include other underlying tool sets.

Each method on the WebCommander includes a number of activities beyond simply performing WebElement interactions. The workflow for these methods is as follows:

1. Log the action taking place and the name of the target element.
2. Initialize a WebElement using the provided Locator instance.
3. Perform the WebElement or Actions command.
4. Catch and log any exceptions that are thrown.
5. If an exception is caught, capture a screenshot of the UI under test.
6. Throw the exception to stop execution of the test.

All WebCommander commands include an Explicit Wait during WebElement initialization. Commands that involve a Click action include a wait for the clickability of the target element. All other commands include a wait for the visibility of the target element to be true.

Basic Element Commands

The 4 basic Selenium WebElement Interactions are covered by the WebCommander. These are the click(), sendKeys(), clear(), and select() functions.

Clicks

The primary click() method includes a fallback intended for flexibility against tricky DOM configurations. On the chance that an element click would be intercepted by another element and an ElementClickInterceptedException Exception is caught, the click() method may reattempt the click using Actions.click(). This behavior is configurable and is enabled by default. To configure, simply add a parameter to the TestNG Suite XML file.

<parameter name="retryInterceptedClicks" value="false" />

It should be noted that the ElementClickInterceptedException can be a symptom of an element selector that needs to be optimized a little further. Before relying on the fallback, it is highly recommended to address the Locator first.

An overloaded click() method is also available that allows point-precision clicks on an element. Two int arguments represent X and Y offsets that allow the click to be placed precisely on an element. Since this method wraps the Actions class, the ElementClickInterceptedException fallback is not included.

Inputs

The enterText() method retains the flexibility of the underlying WebElement.sendKeys() method to accept both characters (Strings) and enumerated Keys. The logging in this method is configured to accurately represent both in a simple manner on the resulting reports.

The clearAndEnterText() method combines clearing a field and sending input as a convenience wrapper to save a second method call.

Selects

As the method name would imply, the select() method wraps the Select API to interact with menus constructed on the DOM as <select> elements. The WebCommander currently selects and deselects options only by visible text as a common use pattern. To gain access to other forms of interaction (by Index or by Value), the best solution is to extend WebCommander and create custom commands as needed. (More on this topic below)

WebDriver Actions

The Actions API is an extremely versatile interface for simulating keyboard, mouse, pen, and wheel actions. The Builder pattern invovled with the Actions API would make wrapping various combinations extremely difficult and extremely limited in actual value in a consuming project. While there are some more common and straightforward action sequences that are provided on the WebCommander, the Actions API is largely intended to build sequences custom to the needs of the UI under test.

The WebCommander currently provides Actions wrapping for a point-precise click() on an element, a doubleClick() on an element, a controlClick() series on multiple elements, and a hover() on an element.

More complex usage of the Actions API can be (and should be) wrapped as custom commands on a Project-level WebCommander sub-class. More on this in the ‘Extensibility’ section below.

Other Functions

In addition to WebElement interactions, the WebCommander provides additional functionality that helps to interact with the rendered DOM.

Frames

Switching between frames is one such function. The methods involved simply wrap the WebDriver.switchTo().defaultContent() and WebDriver.switchTo().frame(). To move focus to a child frame, simple invoke the focusOnFrame(Locator locator) method with a Locator instance that maps of the Frame node on the DOM. Invoking the focusOnDefaultContent() method will return focus to the primary frame on the page.

Waits

Inevitably, tests will need to work around some timing and synchronization issues. Implementing a time-based wait such as Thread.sleep() can work in a pinch, but best practices suggest something more flexible and performant. The pause(Condition) method was designed to function as an Explicit Wait, but uses Qadenz’s Conditions and Expectations to express the type and criteria of the wait. As with Explicit Waits, the Condition will be evaluated repeatedly until either the Condition is satisfied, or a timeout occurs, at which point the test will be stopped.

Screenshots

While failed assertions and caught exceptions will trigger the capturing of a screenshot, there are other occasions where a team might need visual confirmation of the UI state at a given point in a test. This can also provide some additional insights while troubleshooting a troublesome test.

Invoking the captureScreenshot() method will save an image of the visible UI and embed the image into the final HTML report.

3.2 - Web Inspector

The WebInspector works alongside the WebCommander, but instead of acting upon UI elements, this class interrogates them for information. This includes retrieving inner text values, attribute values, element state, and instance counts. Most of the methods on WebInspector are used by Conditions and Expectations as part of evaluative logic for verifications and waits, but can be useful through the course of a test for retrieving data from the UI under test.

Each method on the WebInspector includes a number of activities beyond simply performing WebElement inspections. The workflow for these methods is as follows:

1. Log the action taking place and the name of the target element.
2. Initialize a WebElement using the provided Locator instance.
3. Retrieve the required information from the WebElement.
4. Catch and log any exceptions that are thrown.
5. If an exception is caught, capture a screenshot of the UI under test.
6. Throw the exception to stop execution of the test.

Element Attributes and Properties

The values assigned to attributes on elements can be retrieved via the getAttributeOfElement() method by passing the name of the attribute to be evaluated. Invoking getAttributeOfElements() will return the value for all instances of the matching Locator on a List<String>.

Similarly, the value of a CSS property can be retrieved from an element via the getCssPropertyOfElement() method.

Element States

Element states can be evaluated and will return a boolean value based on the result.

Enabled

The getEnabledStateOfElement() method evaluates if an element is enabled for interaction, and returns true if the element is in fact enabled. This is useful for determining if elements such as <input> elements or <button> elements are enabled. The state is determined by first invoking the WebElement.isEnabled() method. If the WebElement evaluates as enabled, a second check is performed against user-defined attributes on the element via the Locator instance.

This method presumes an element is enabled until one of the evaluations proves the element is disabled.

Selected

The getSelectedStateOfElement() method evaluates if an element is selected, and returns true if the element is in fact selected. This is useful for determining if elements such as checkboxes or radio buttons are selected. The state is determined by first invoking the WebElement.isSelected() method. If the WebElement evaluates as selected, a second check is performed against user-defined attributes on the element via the Locator instance.

This method presumes an element is selected until one of the evaluations proves the element is not selected.

Visiblity

The getVisibilityOfElement() method evaluates if an element is visible on the UI, and returns true if the element is in fact visible. The state is determined by first finding all matching DOM nodes for the element selector on the given Locator. If more than zero matches are found, the method will then evaluate the dimensions of the first matching node. If the element has a height and width greater than zero, the method will then evaluate the styling and attributes of the element. The element will be checked for display:none; and then visibility:hidden;, and finally a check for the hidden attribute. If these checks result in no match, the method will then evaluate user-defined attributes on the element via the Locator instance. Additionally, the method will catch a StaleElementReferenceException throughout this series of evaluations and instantly return a false result.

This method presumes an element is visible until one of the evaluations proves the element is hidden.

Element Text

The WebInspector offers methods that retrieve text of UI elements in a variety of ways and a variety of formats. In addition to returning String values, methods exist to convert and return element text values as Temporals or Numbers as well. This allows a test to be designed where the UI data can be interacted with in a more flexible way, enjoying the ability to perform operations against, comparisons, and validations with the capabilities of these secondary object types.

General text inspection methods allow retrieval of basic element text, a List of text values from all matching nodes of the given Locator, the currently selected <option> value on a <select> menu, or the List of <option> values on a <select> menu.

Element Text vs Direct Element Text

The WebElement.getText() method returns the visible inner text of the given element along with the text of any child elements. In most use cases, this is perfectly fine. Some situations exist, however, when the DOM is constructed in such a way where a target element has child elements that also contain visible inner text that a tester may wish to ignore or avoid for an inspection or validation. For these scenarios, WebInspector employs a concept of ‘direct text of element’. Methods with directTextOfElement in the name will retrieve element text, but prior to returning the text value to the calling method, will filter the visible inner text values of any child elements relative to the given target element.

Element Text as a non-String Object

The WebInspector includes the means to examine the text of an element, and with the aid of a formatter, parse and convert the text value into a Number or Temporal.

A NumberFormat can be used to drive the converstion of a text value to either an Integer or a Double value. A DateTimeFormatter can be used to drive the conversion to either a LocalDate, LocalDateTime, or a LocalTime object. By using these methods, the UI data can be used in mathematical operations directly without requiring the need to build any parsing or conversion logic into the test.

Element Instances

There are also methods on WebInspector that focus specifically on all instances of an element. The getCountOfElement() simply returns the number of instances on the DOM of a matching Locator. This can drive mathematical operations, looping logic, or precision validations of element groups. The getInstanceOfElementText() and getInstanceOfElementAttribute() will examine all instances of a matching Locator and return the index of the first node that contains a matching text or attribute value. This capability can provide an extra convenience in dynamically parameterizing Locator selector values based on the state if the UI.

3.3 - Logging

The WebCommander and WebInspector can be instantiated and used either from the tests directly, or from the UI Modeling layer, depending on the design of the test project. Both WebCommander and WebInspector have overloaded constructors that enable different types of logging to take place, and will both directly impact how the logs are presented on the report output.

The WebCommander constructor is used as an example, but the WebInspector shares this same pattern:

private Logger LOG;

public WebCommander() {
    super();
    LOG = LoggerFactory.getLogger(WebCommander.class);
}

public WebCommander(Class<?> logger) {
    super(logger);
    LOG = LoggerFactory.getLogger(logger);
}

The no-args constructor is generic and can be used regardless of how the classes are consumed. This constructor assigns the WebCommander.class as the Logger, and all logging output for method calls on this class will be shown to originate from the WebCommander class. If the UI of the application under test is very simple, or the team simply does not require a level of detail in the logging that ties actions to specific UI Models, then this constructor will provide an ideal configuration.

09:55:00.939 | INFO | WebCommander | Entering text [admin@qadenz.dev] into element [Username Field].
09:55:01.308 | INFO | WebCommander | Entering text [Test123$] into element [Password Field].
09:55:01.472 | INFO | WebCommander | Clicking element [Sign In Button].
09:55:02.583 | INFO | Commands | Verifying Condition - Visibility of element [Qadenz Logo Image] is TRUE.
09:55:02.998 | INFO | Commands | Result - PASS

The overloaded constructor requires a Class<?> argument, and allows for another class reference to be injected as the logger for the WebCommander instance. If WebCommander is being instantiated from a Page Object, and the Page Object class is passed to the constructor, the logs and reporting output will be shown to originate from the Page Object itself, resulting in a greater level of detail in the logs and reports. By using the class injection for the logger, commands will be logged in the context of the page where the command was executed.

09:55:00.939 | INFO | LoginPage | Entering text [admin@qadenz.dev] into element [Username Field].
09:55:01.308 | INFO | LoginPage | Entering text [Test123$] into element [Password Field].
09:55:01.472 | INFO | LoginPage | Clicking element [Sign In Button].
09:55:02.583 | INFO | UserProfilePage | Verifying Condition - Visibility of element [Company Logo Image] is TRUE.
09:55:02.998 | INFO | UserProfilePage | Result - PASS

3.4 - Extensibility

Both the WebCommander and WebInspector classes are designed to be extended within automation projects to enable custom commands to be created as needed by the UI under test.

Starting at the class-level, the class will obviously need to extend either WebCommander or WebInspector. Then, a Logger instance and constructors need to be added.

import org.slf4j.Logger;
import org.slf4j.LoggerFactory;

public class AcmeWebCommander extends WebCommander {

    private Logger LOG;

    public AcmeWebCommander() {
        super();
        LOG = LoggerFactory.getLogger(AcmeWebCommander.class);
    }

    public AcmeWebCommander(Class<?> logger) {
        super(logger);
        LOG = LoggerFactory.getLogger(logger);
    }
}

Then, the anatomy of the command method is as follows:

public void doSomething(Locator locator) {
    LOG.info("Doing something with element [{}].", locator.getName());
    try {
        WebElement webElement = webFinder.findWhenVisible(locator);

        // add logic as needed to complete the custom command
    }
    catch (Exception exception) {
        LOG.error("Error doing something :: {}: {}", exception.getClass().getSimpleName(), exception.getMessage());
        screenshot.capture();

        throw exception;
    }
}

The first operation in a command method is to log the command being executed (at the INFO level). Qadenz uses Logback, which provides the {} placeholder for additional values to be inserted. It is recommended to utilize this where possible to convey an appropriate level of detail in the logs and subsequent reports.

Next, the try block will initialize a WebElement using the inherited WebFinder instance, then perform any necessary actions. If the method is on a WebInspector sub-class, the return should take place within the try block.

The catch block should be set to catch the appropriate Exception, though a wide net may be cast by simply catching all descendents of Expception. A multi-catch or finally could be employed here if the use case is appropriate. Within the catch block, the Exception will be logged (at the ERROR level) and, if appropriate, a screenshot captured using the inherited Screenshot instance.

Finally, re-throw the Exception. Throwing a new instance of RunTimeException with a context-friendly message would be an appropriate alternative to allowing the Exception to surface to the test level, requiring a try/catch in the test itself or an Exception to be added to the method signature. This is obviously a preferential decision to be made within the automation project. Qadenz has simply been designed to avoid the need to do this.

Once complete, instantiating the new commands class in either the UI Models or directly in the test method will make functionality available for use.

3.5 - Browser Commands

The Browser class manages the browser under test. This includes activities within the browser, but outside the rendered DOM of the application. Actions such as navigation, alert handling, cookie management, and switching between browser windows, are all handled by the Browser. The methods on the Browser class are static, and are able to be called from anywhere within the scope of the test, be it from the UI Modeling layer or directly from the test itself.

4 - Conditions & Expectations

The pairing of Conditions and Expectations becomes the core evaluative logic of Qadenz. These classes are used to determine if the state of the UI under test meets a given criteria.

The goal of the Condition/Expectation pairing is to provide a unified structure for making evaluations throughout a testing project, rather than mixing Selenium ExpectedConditions calls for Explicit Waits, and various syntax patterns for test assertions. The resulting code is simple to read and quickly understand, and is easily maintained as the application under test evolves.

The evaluative logic behind Conditions and Expectations aims to answer the questions, “What is the current state of the UI?” and “What do I expect to see?”. Conveyed in terms familiar to testers, “What is the ACTUAL outcome?” and “What is the EXPECTED outcome?”.

A Condition describes a specific criteria to be evaluated on the UI. This could be the visibility of one or more elements, or the text shown in an element, for example. The Condition is used to establish the “actual outcome” portion of the evaluation. An Expectation, then, is required for each Condition. The Expectation will describe the “expected outcome” portion of the evaluation.

How does it work?

Each Condition uses WebDriver commands to retrieve data from, or information about, elements on the UI. Each Expectation invokes a Hamcrest Matcher that is used for the evaluation, which is passed to the Condition. If the the value retrieved by the Condition matches the value given on the Expectation, the Condition result will return TRUE.

Logging of evaluations is achieved by combining a description of the evaluation on the Condition with a description of the expected outcome on the Expectation. If an evaluation should fail and the Condition result return FALSE, additional information will be provided on the logs to illustrate the cause of the failure. This typically equates to capturing the “actual” value that did not meet the “expected” value.

4.1 - Validations

Unit testing frameworks such as TestNG or JUnit include assertion functionality as a core component, and are relatively simple to use. Being open-ended frameworks, however, individual users may tend to express very similar validations in a variety of different assertions. This leads to inconsistent coding patterns, and more difficult maintenance of test code.

Using Conditions and Expectations allows a team to ensure all contributors are following the same pattern for validations.

Conditions.textOfElement(greetingText, Expectations.isEqualTo("Hello World!");

That said, Qadenz does employ a single TestNG assertion, the assertTrue() method, as a means of validating a Condition. The result() of a Condition is a simple representation of whether the state of the UI under test meets expectation. If the output of the Condition evaluation matches the Expectation, result() will return true.

By passing this result to the assertTrue() method, Qadenz is ensuring that a passing result depends on the Condition evaluation meeting the Expectation. If not, the validation will fail.

Assertion Types

The concept of Hard Assertions and Soft Assertions are not new in the test automation world. Qadenz implements both concepts by way of the verify() and check() methods.

verify() represents a Hard Assertion. If the validation fails, the test will be marked as failed and execution will be stopped.

check() represents a Soft Assertion. If the validation fails, the test will be marked as failed, but execution will be allowed to continue until a call to Assertions.flush() is made, which will stop execution of the test if any failures have been encountered.

The verify() or check() methods are available on the Commands Hierarchy and are callable on any descendant class of Commands. The mechanics of using these validations are the same, with the only difference being an additional step with check() required to call Assertions.flush() in order to handle any failed Soft Assertions and stop execution.

Grouped Conditions

Validations in Qadenz are further enhanced with the ability to evaluate multiple Conditions as a group. In scenarios where a single UI action can trigger multiple verification points in a test, a tester may have to express multiple assert statements to ensure necessary coverage. If, for example, the first assertion were to fail, the remaining assertions would remain unchecked until either the UI under test is fixed, or the test scenario is executed manually.

Using Qadenz, a tester is able to execute these same validations in one call to verify() or check(), and will receive results for each Condition evaluation regardless of individual results. If again, the first validation fails, Qadenz will perform handling tasks on the failure, then proceed to evaluate each of the other Conditions that were passed. In the case of a verify() with multiple Conditions where one or more have failed, halting of test execution will be delayed until all Conditions have been evaluated, which will ensure that the test step is completed in its entirety.

In the example below, a user has added an item to the shopping cart, and the next step will verify a snackbar notification is displayed with a confirmation message, the item quantity is shown on the shopping cart icon, and the ‘Checkout Now’ button is enabled.

commander.verify(
        Conditions.textOfElement(snackBarNotification, Expectations.isEqualTo("Items added successfully!")),
        Conditions.textOfElement(quantityInCartIndicator, Expectations.isEqualTo("1")),
        Conditions.enabledStateOfElement(checkOutNowButton, Expectations.isTrue()));

By grouping these verifications together, even if one (or more) Conditions fail, all will be evaluated and reported individually.

Managing Soft Assertions

The check() methods works alongside the static Assertions.flush() method to delay execution stoppages in the event of failed validations. As calls to check() are made and executed through the course of a test, the Assertions class tracks whether any failures have been encountered. When the call to Assertions.flush() is made, this tracker is checked. If any failures are present, execution will be stopped. If no failures are found, execution continues.

Since the tracker is live for the entire duration of a test, there is no limit to how many calls to Assertions.flush() can be made throughout a test. It is possible then, to create a series of “checkpoints” in longer tests whenever it is deemed sensible to stop a test if failures have been found. This is especially convenient for smoke to end-to-end tests where a focus on completion of test is important for a full accounting of key validation points.

Please note, however, that at least one call to Assertions.flush() is required in tests where only check() validations are made. If no call is made, the test will be allowed to continue to completion, and individual steps will be reported as failed (if validations have indeed failed), but the test as a whole will be reported as passing. Since the Qadenz reporter is integrated with TestNG, the AssertionError thrown by the flush() method in the event of individual failures is required to mark the test itself as failed.

One additional design consideration must be made when mixing verify() and check() validations within the same test. When a check() validation is made, and is followed by a verify() validation prior to calling Assertions.flush(), if the verify() validation fails, the test will be stopped at the failed verify() validation.

Screenshots

Qadenz validations are built to capture screenshots whenever a Condition evaluation fails. If screenshots are desired for validation failures, no special action need be taken. Should screenshots not be needed for a validation, disabling is easy with the overloaded verify() and check() methods.

Adding a call to Screenshot.SKIP as the first argument in either verify() or check() will disable screenshots from being captured if the evaluations for any accompanying Conditions fail.

verify(Screenshot.SKIP, Conditions.visibilityOfElement(locator, Expectations.isTrue());

A boolean could also be passed to achieve the same outcome. The Screenshot.SKIP value is intended as a means to keep the resulting code easily readable at a glance.

4.2 - Waits

Selenium provides both Implicit and Explicit Wait types, and Java provides the Thread.sleep(). While all technically valid, each have their own advantages and disadvantages. Qadenz does not implement the WebDriver Implicit Wait. The Implicit Wait can serve as a basic catch-all wait approach in simple projects, but the flexibility is limited, and more importantly, it tends to not pair well with Explicit Waits. Using both in conjunction can cause some very unexpected side effects.

Qadenz opts for the Explicit Wait as the primary UI synchronization approach, and pairs this concept with Conditions and Expectations to define the criteria for the syncronization.

What about ExpectedConditions?

The ExpectedConditions class is well known among automation engineers and provides a wide variety of wait-conditions to handle timing and synchronization. In his 2017 Selenium State of the Union, Simon Stewart calls ExpectedConditions a “useful dumping ground for functionality” that “brutally violates this attempt to be concise”. While there is no denying the usefulness of a class such as ExpectedConditions, it could also be said that the method options aren’t always intuitive for choosing an ideal fit. In that same talk, Mr. Stewart uses the original intent and the evolution of ExpectedConditions as an example of why it’s important that developers not punish themselves too harshly for code written in the past.

Conditions and Expectations are implemented in Qadenz for waits with the intent of making the invocations of wait-conditions much more concise and exact, thus enhancing the readability (and maintainability) in test code, as well as improving the clarity and precision of logging output that is captured and presented on the final reports.

Invoking a Wait

Invoking a Wait is as simple as calling the pause() command, and passing an appropriate Condition/Expectation pairing.

If an application under test displays a confirmation banner when an item is added to a shopping cart that blocks access to the navigation menu, for example, the test would benefit from pausing execution until the banner confirmation disappears after a few seconds.

commander.pause(Conditions.visibilityOfElement(addItemConfirmation, Expectations.isFalse()));

Unlike the validation functionality that consumes the Condition and Expectation pairing, the pause() method does not provide the ability to pass multiple Conditions as a group for a wait. This is done in the mindset that waits should be used as sparingly as possible to avoid unnecessary lenthening of execution times.

I want my MTV ExpectedConditions

While Qadenz does implement certain functionality in an opinionated manner, there is no reason to prevent access to the underlying tools for use in a customized solution within a consuming test project. By extending the WebCommander, it would be very possible to create an instance of the WebDriverWait and pass an ExpectedCondition to execute a wait. The recommended practice would be to use this approach only if a suitable Condition does not exist for the needed wait.

5 - Test Results

Logs tell a very important story during the lifecycle of any software application, and a testing library is no different. In fact, it could easily be argued that logging in a testing context is doubly important in order to capture all the details of what happened during the execution cycle. It’s the step by step details that tell us what a test was doing when something goes wrong, which in turn helps us testers to more quickly understand and recreate these failure scenarios so that system defects can be reported earlier in the testing cycle.

Qadenz uses two primary loggers to tell the story of an execution run. The Suite Logger monitors major activity with the execution run itself and captures events outside of the individual tests. This includes setup and tear-down activities, configuration details, the starting and stopping of tests, and errors that are encountered during the reporting phase. The Test Logger captures the step by step details of each test that runs as part of a Suite. This includes every action, inspection, validation, and any errors that are encountered along the way.

Qadenz uses Logback to handle all logging within the library. These logs are presented on the console during a Suite execution, and are also used to generate content for the Qadenz HTML Reporter. Each log event will contain a timestamp, the logging level, the name of the logger, and the log message.

Logs are made available during and after an execution run in a variety of formats. The console logs will show the output of both the Suite and Test loggers in real time. The JSON report will show the compiled Test log output, combined with Base64 encoded screenshots, and sorted into Pass/Fail/Stop result sets. The HTML report will show the same data as the JSON report, but in an all-inclusive and sharable HTML file.

5.1 - Console Logs

The console logs will show output from both the Suite Logger and the Test Logger. These will be present when running locally either from Maven or within an IDE like IntelliJ IDEA, as well as in build systems like Jenkins or TeamCity. The console logs are the “raw” log output, showing all events in chronological order, regardless of origin. This means events reported inside test methods that are run in parallel will be shown in the same output. To facilitate better tracing of these events, the console logger is configured to add a thread identifier to each event.

>>> Add log output example

The Suite Logger output can be identified by Thread: main, and the Test Logger output for each test can be identified by a numbered idntifier, such as Thread: 1, based on how many tests are executed in parallel.

The console logs are beneficial for local debugging of tests, providing a fast and easy view into the events and any errors encountered during the execution of a problematic test.

5.2 - HTML Reports

The Qadenz HTML Reporter is the primary view of test results. This report was originally inspired by the TestNG emailable-report.html, but sought to make some key improvements to both the level of detail in the report, and general user experience.

The TestNG emailable report is sufficient for a count and listing of tests that passed and failed, and some basic insight on any exceptions that were caught during a test, but is lacking for any additional detail. There are other libraries that generate HTML reports, but can be cumbersome to use and may introduce additional code management overhead in order to integrate with tests and UI models.

Qadenz solves this problem by leveraging the Logback loggers to generate the detailed content for each test, and integrating directly with TestNG to generate and calculate results.

Report Output

The Qadenz HTML report header shows the Suite Name as given on the Suite XML file, the date and time the execution run was launched, test counts (total, passed, failed, stopped, skipped), and an overall duration for the run.

Next, the classes and methods invoked by each <test> node on the Suite XML will be present in blocks. The name given in the <test> node will be present, followed by an expandable list of test classes. Failed tests will be listed first, followed by Stopped tests, then Skipped, followed by Passed.

Expanding a class entry on the list will expose a list of methods from the same class that were included in the run and that fall within the current result section. The same class may be listed in multiple sections depending on the final outcome of the included tests. If a @DataProvider has been used with the tests, the parameters from the Data Provider will be listed next to the name of the test method.

Next, expanding a test entry will expose the individual logs for the test. The report will show the start time and duration for each test method, followed by the detailed logging output from the test. In Failed and Stopped tests, each failure and error will be accompanied by a “View Screenshot” link. Clicking this link will open the screenshot image in a modal for viewing.

Failed vs Stopped Tests

Default TestNG behavior is to classify any test that throws any exception as a failed test. Qadenz introduces a new result type called “Stopped” to allow some granularity in how test results can be sorted and analyzed.

Qadenz considers a “Failed” test to be a test that contains one or more failed validations, and considers a “Stopped” test to be a test that has thrown any other exception. This includes timing/sync issues, missing elements, etc.

By limiting the “Failed” category to validation failures, this allows teams to focus their post-execution results analysis on the tests that could potentially be more likely to produce a defect. This does not preclude Stopped tests from being so due to system defects, nor does it preclude Failed tests from being so due to issues in the tests. This is simply a means to draw attention specifically to tests that have failed due to validations that did not meed expectations, and those that have failed due to other reasons.

Screenshots

The Qadenz HTML report is completely inclusive of all detailed reporting content and screenshots. The ease of distribution is the same as the TestNG emailable report.

Screenshots in the Qadenz library are captured and immediately encoded to a Base64 String, which is then added to the report. This allows the images to be embedded directly into the report and to be shared without having to work with cumbersome zip files or deal with broken image links.