Ultralight can help you automate two different parts of your Verification/Validation process - Test Case documentation and Test Reporting. You can see an example project for both Test Case documentation and Test Reporting here: https://github.com/ultralightlabs/test-project
Test Case Documentation
With Ultralight, either using the GitHub Action (https://github.com/ultralightlabs/ultralight-github-action) or using the Ultralight CLI (see npx ultralight-core report --help ), we can help you automate your test documentation.
Test case documentation can be used to sync your test documentation in code (GitHub/GitLab etc) to Verification/Validation design elements in Ultralight.
There are two major formats we can ingest - a YAML file and Cucumber Features.
YAML
ultralight-test-id: string (e.g. VER-1)
title: string
method: string (optional. e.g. Unit Test)
description: string
steps:
- step: string
acceptance_criteria: string
- step: string
acceptance_criteria: string
Cucumber Features
You can automatically sync Cucumber Features to Verification/Validation in Ultralight. All you need to add is a tag, @ul-test:<Ultralight Test ID> to your Cucumber feature. See example below:
@ul-test:VER-1
Feature: Guess the word
Scenario: Maker starts a game
When the Maker starts a game
Then the Maker waits for a Breaker to join
You can also automatically link Verification to System Requirements and Subsystem Requirements, and Validation to User Needs using the @ul-links tag. See usage below:
@ul-test:VER-1 @ul-links:SR-4,SSR-2
Feature: Guess the word
Scenario: Maker starts a game
When the Maker starts a game
Then the Maker waits for a Breaker to join
Test Reports
By specifying the ID of the Verification/Validation in the test case, you can report to a Verification or Validation in Ultralight. This is one-to-many - you can report multiple Test Executions to the same Verification or Validation, and Ultralight will collect all the reports to present a unified view.
You can currently report tests to Ultralight by generating a JUnit report, and providing the Ultralight Verification/Validation ID in the test case name. JUnit is supported by most testing frameworks.
Pytest Example
See https://github.com/ultralightlabs/test-project/blob/main/python/test.py for an example using Pytest.
Playwright Example
See https://github.com/ultralightlabs/test-project/blob/main/next/e2e/pagesTest.spec.ts for an example using Playwright