AIO Tests for Playwright Reporter

AIO Tests for Jira is a Jira-native test management app covering the entire QA lifecycle with Test Cases, Cycles, Reports, Automation, Dashboards, providing a one stop shop for all your testing needs.

With it's Playwright reporter, AIO Tests simplifies reporting of results from the automated Playwright tests to AIO Tests for Jira.

How does the AIO Tests Reporter work

By hooking into Playwright events, the AIO Tests Reporter reports results in the onExit event, ater all the tests are run.
The reporter can create a new cycle for the executions or reuse existing cycles, based on the configuration done in playwright.config.ts.
It can also upload attachments for failed executions.
Retries can either be reported as new runs or used to update the existing execution run.

How to get started?

npm install aiotests-playwright-reporter

Use

Mapping automated Playwright tests to AIO Tests

The AIO Tests Case key can be added to tags of Playwright cases or they can be added to the test title with @ key to mark it as a Playwright tag.
If there are multiple case keys added as tags, then the result of one test will be updated to multiple cases.

✒ Please note that the case key can appear anywhere in the description

1. Mapping single case

   test('should clear text input field when an item is added',{tag:'@NVTES-TC-1'}, async ({ page }) => {
   //Or test('@NVTES-TC-1 : should clear text input field when an item is added', async ({ page }) => {
    const newTodo = page.getByPlaceholder('What needs to be done?');
    await newTodo.fill(TODO_ITEMS[0]);
    await newTodo.press('Enter');
    await expect(newTodo).toBeEmpty();
    await checkNumberOfTodosInLocalStorage(page, 1);
  });

2. Mapping to multiple cases

   test('should clear text input field when an item is added',{tag: ['@NVTES-TC-4','@NVTES-TC-5']}, async ({ page }) => {
    const newTodo = page.getByPlaceholder('What needs to be done?');
    await newTodo.fill(TODO_ITEMS[0]);
    await newTodo.press('Enter');
    await expect(newTodo).toBeEmpty();
    await checkNumberOfTodosInLocalStorage(page, 1);
  });

Setup

const aioConfigDetails = {configDetailedBelow};

module.exports = defineConfig({
....
reporter: [['aiotests-playwright-reporter',{ aioConfig: aioConfigDetails}]],

Cloud - Authentication

For Jira Cloud (eg. https://yourco.atlassian.net/..), the "cloud" property needs to be set in the config for authentication.

1. Local runs : The API Key generated from AIO Tests, needs to be set as "apiKey" value.
2. CI/CD: For CI runs, you can set the AIO_API_KEY environment variable to pass it externally as a SECRET.

  
    const aioConfigDetails = {
      "enableReporting": true,
      "cloud": {
        "apiKey": "<your API KEY>"
      },
      "jiraProjectId": "SCRUM",
      "cycleDetails": {
        "createNewCycle": true, //possible values "true","false","CREATE_IF_ABSENT", true, false
        "cycleName": "Playwright Nightly Run ", //used if createNewCycle set to true or "CREATE_IF_ABSENT"
        "cycleKey": "SERV-CY-2", //used if createNewCycle set to false or "false"
        "folder": ["Cloud","Smoke Test Nightly"], //array of folder hierarchy
        "tasks": ["SCRUM-1","SCRUM-2],
        "customFields": [ //optional
          {
            "name": "Reviewed? [Boolean CF]",
            "value": "Yes",
          },
          {
            "name": "Set (Single Select CF)",
            "value": {"value":"P1"}
          },
          {
            "name": "Teams (Multi Select CF)",
            "value": [{"value":"TeamAlpha"},{"value":"Zeta"}]
          },
          {
            "name": "NumberCF", "value": 0
          },
          {
            "name": "TextValue CF", "value": "This can be a long note",
          },
          {
            "name": "Reviewed Date", "value": "2024-08-29T04:38:36.437Z",
          },
          {
            "name": "SME [User CF]", "value": "<accountid of user>"
          }
        ],
      },
       "runDetails": { //optional
        "customFieldsToUpdate": [
          {
            "operationType": "ADD_TO_EXISTING",
            "name": "StepOwner",
            "value": [{"value": "Val2"},{"value": "Val1"}]
          },
          {
            "name": "Env", "value": {"value":"UAT"}
          },
          {
            "name": "Browser", "value": {"value":"$project"}
          }
        ]
      },
      "addNewRun": true,
      "addAttachmentToFailedCases": true,
      "createNewRunForRetries": true,
      "debugMode": false,
      "parallelBuild":{ //optional
        "masterBuild": true,
        "waitForSeconds": 10 //defaults to 2 seconds
      }
    }
  }
}

Jira Server/Data Center - Authentication

For Jira Hosted or DataCenter versions, the "hosted" property needs to be set in the for authentication as shown below.
The "jiraUrl" needs to be specified with the base url of the hosted Jira instance.

Authentication is supported either by providing Jira username and password or by using the Jira PAT. More information can be found on Server Authentication here

1. Local runs : For local runs, either "jiraUsername" + "jiraPassword" can be set or one can simply set the "jiraPAT" value.
2. CI/CD: For CI runs, you can set the JIRA_USERNAME and JIRA_PASSWORD or JIRA_PATenvironment variable to pass it externally as a SECRET.

{
    "enableReporting": true,
    "hosted" : {
      "jiraUrl": "https://jira.yourco.com",
      "jiraPAT": "PAT from Jira Tokens | JIRA_PAT as environment variable",
      "jiraUsername": "Jira Username. If PAT is specified, then username is not required",
      "jiraPassword": "Jira password, required if authentication is through username/password"
    },
    "jiraProjectId": "SERV",
    "cycleDetails": {
      "createNewCycle": true, //possible values "true","false","CREATE_IF_ABSENT", true, false
      "cycleName": "Playwright Nightly Run ", //used if createNewCycle set to true or "CREATE_IF_ABSENT"
      "cycleKey": "SERV-CY-2", //used if createNewCycle set to false or "false"
      "folder": ["Server","Smoke Test Nightly"],//array of folder hierarchy
      "tasks": ["SERV-1","SERV-2],
      "customFields": [ //optional
        {
          "name": "Reviewed? [Boolean CF]",
          "value": "Yes",
        },
        {
          "name": "Set (Single Select CF)",
          "value": {"value":"P1"}
        },
        {
          "name": "Teams (Multi Select CF)",
          "value": [{"value":"TeamAlpha"},{"value":"Zeta"}]
        },
        {
          "name": "NumberCF", "value": 0
        },
        {
          "name": "TextValue CF", "value": "This can be a long note",
        },
        {
          "name": "Reviewed Date", "value": "2024-08-29T04:38:36.437Z",
        },
        {
          "name": "SME [User CF]", "value": "<accountid of user>"
        }
      ],
    },
    "runDetails": { //optional
      "customFieldsToUpdate": [
        {
          "operationType": "ADD_TO_EXISTING",
          "name": "StepOwner",
          "value": [{"value": "Val2"},{"value": "Val1"}]
        },
        {
          "name": "Env", "value": {"value":"UAT"}
        },
        {
          "name": "Browser", "value": {"value":"$project"}
        }
      ]
    },
    "addNewRun": true,
    "addAttachmentToFailedCases": false,
    "createNewRunForRetries": false,
    "addTestBodyToComments": true,
    "debugMode": false,
    "parallelBuild":{ //optional
      "masterBuild": true,
      "waitForSeconds": 10 //defaults to 2 seconds
    }
}

Configurable values

Create New Cycle options

• createNewCycle = true or "true", uses the cycleName and cycleDetails value to generate new cycle
• createNewCycle = false or "false", uses the cycleKey value to find an existing cycle and updates the cycle. If cycle is not found, an error is thrown
• createNewCycle = "CREATE_IF_ABSENT" uses the cycleName to search for an existing cycle with an exact match. If cycle is found, then the cycle is updated. If it is not found, then a new cycle is created using cycleName and cycleDetails

Setting browser or project name in run custom fields

Since Playwright supports running cases on multiple browsers, there may be a need to capture the browser details where the run has executed in AIO Tests. The browser value is set as the project name in playwright config file. The project name can be set in the custom fields in AIO as below:

1. Setup a run custom field in AIO (Project Settings -> Custom Fields)
2. In the aioConfig, specify it in the runDetails.customFieldsToUpdate property as below:
   (the example below assumes the run custom field is named "Browser" and the values if defined as a single select list, match the text used in the project name in playwright config)

...
"runDetails": { 
      "customFieldsToUpdate": [
        {
          "name": "Browser", "value": {"value":"$project"}
        }
      ]
    }
    ...

Parallel builds

If multiple builds are being triggered in parallel, the parallelBuild setting can be used to specify the masterBuild.
One of the parallel builds can be configured to set masterBuild as true. This build should have createNewCycle either set as true or CREATE_IF_ABSENT.
The other builds running in parallel can have masterBuild set to false, which would imply, they would wait for the masterBuild to run the cycle creation code, waiting for waitForSeconds (defaults to 2 seconds), before trying to find the cycle.

If multiple builds are not being run in parallel, the parallelBuild value can be ignored.

Logging

AIO Tests Reporter logs can be seen in the run logs as below for successful updates

*************** AIO Tests Reporter AIO Tests Playwright Reporter - Reporting results - test 2 ***************
11:38:10 AM : Debug Mode is set to true
*************** AIO Tests Reporter : Determining cycle to update ***************
11:38:17 AM : Reporting results to cycle : NVTES-CY-828
*************** AIO Tests Reporter : Reporting results ***************
11:38:17 AM : ***** Updating failed cases *****
11:38:17 AM : Posting results to /project/NVTES/testcycle/NVTES-CY-828/testcase/NVTES-TC-4/testrun?createNewRun=true
11:38:20 AM : Successfully reported NVTES-TC-4 as Failed with runID 11362947.
11:38:22 AM : ***** Updating all passed cases *****
11:38:22 AM : Reporting results in bulk: /project/NVTES/testcycle/NVTES-CY-828/bulk/testrun/update?createNewRun=true
11:38:23 AM : Successfully reported 2 passed cases for browser: chromium.
*************** AIO Tests Reporter : AIO Tests Playwright Reporter - Reporting results completed ***************

Errors received while updating will appear in a similar way

*************** AIO Tests Reporter  Reporting results ***************
***** Updating all passed cases *****
Successfully reported 0 passed cases for browser: chromium.
 - Failures in reporting passed cases:
 - 120801: aio.jira.tcms.exceptions.InvalidDataException: The selected version for Case NVTES-TC-20 is in Draft status. This status is not eligible for addition to a cycle.
 - 120802: aio.jira.tcms.exceptions.InvalidDataException: The selected version for Case NVTES-TC-21 is in Draft status. This status is not eligible for addition to a cycle.
Successfully reported 0 passed cases for browser: firefox.
 - Failures in reporting passed cases:
 - 120801: aio.jira.tcms.exceptions.InvalidDataException: The selected version for Case NVTES-TC-20 is in Draft status. This status is not eligible for addition to a cycle.
 - 120802: aio.jira.tcms.exceptions.InvalidDataException: The selected version for Case NVTES-TC-21 is in Draft status. This status is not eligible for addition to a cycle.
*************** AIO Tests Reporter : AIO Tests Playwright Reporter - Reporting results completed ***************

To troubleshoot an issue, debugMode can be set to true in config, to increase logging verbosity and to see detailed response errors

Queries/Suggestions?

For any queries, suggestions or issues, please feel free to reach out @ help@aiotests.com

AIO Tests Overview

AIO Tests Automation