Playwright AI Reporter

Transform test failures into actionable insights with AI-powered analysis and auto-healing

Playwright AI Reporter is an enterprise-grade, production-ready test reporter that combines artificial intelligence with comprehensive test automation workflows. Built on a flexible provider-based architecture, it automatically analyzes test failures, creates detailed bug reports, generates fix suggestions, and can even submit auto-healing pull requests—all while integrating seamlessly with your existing development tools.

🎯 Why Choose Playwright AI Reporter?

• 🧠 AI-Powered Analysis - Multiple AI providers (Azure OpenAI, Anthropic Claude, Google Gemini, Mistral, OpenAI) analyze failures and suggest intelligent fixes
• 🔌 Plug & Play Architecture - Swap bug trackers, databases, AI providers, and notification systems without code changes
• 🚀 Auto-Healing Tests - Automatically generate and submit PRs with AI-suggested fixes for flaky or failing tests
• 📊 Enterprise Integration - Native support for GitHub, Azure DevOps, Jira, MySQL, SQLite, SMTP, and more
• 🎨 Rich Reporting - Colorized console output, comprehensive metrics, historical analysis, and build integration
• ⚡ Production-Ready - TypeScript, fully tested, extensive documentation, CI/CD workflows included

💡 Perfect for - CI/CD pipelines • Enterprise test automation • Multi-team projects • Flaky test management • Test debugging at scale

📖 Table of Contents

• Features
• Architecture
• Quick Start
• Installation
• Configuration
• Provider Support
• Usage Examples
• Output Examples
• FAQs
• Documentation
• Contributing
• License

✨ Features

🎨 Smart Reporting & Visualization

• ✅ Colorized console output (Passed, Failed, Retries, Skipped)
• 📊 Comprehensive test metrics and statistics
• 🎯 Slowest test identification and ranking
• ⏱️ Average test duration analysis
• 📈 Test history tracking and comparison
• 🏗️ CI/CD integration with build information
• 🌐 Interactive HTML Report: Self-contained HTML dashboard with charts, test details, and AI fix suggestions

🤖 AI-Powered Intelligence

• 🧠 Multi-AI Provider Support: Azure OpenAI, OpenAI, Anthropic (Claude), Google AI (Gemini), Mistral AI
• 🔧 Automatic Fix Suggestions: AI analyzes failures and suggests fixes
• 📝 Context-Aware Analysis: Includes test code, error details, and stack traces
• 🎓 Smart Categorization: Automatic error categorization (Timeout, Selector, Network, etc.)
• 💡 Best Practices: Suggestions follow Playwright best practices

🐛 Automated Bug Management

• 🎫 Multi-Platform Bug Creation: GitHub Issues, Azure DevOps Work Items, Jira Tickets
• 📋 Rich Bug Details: Test info, error details, AI suggestions, environment data
• 🏷️ Smart Labeling: Automatic labels and priority assignment
• 🔗 Integrated Tracking: Links bugs to test runs and failures

🔄 Auto-Healing & PR Generation

• 🚀 Automatic PR Creation: Generate PRs with AI-suggested fixes (set generatePR: true)
• 🌿 Branch Management: Auto-create topic branches (autofix/test-name-timestamp)
• 💾 Smart Commits: Commit fixes to topic branch with detailed messages
• 📤 Push & PR: Push changes and create pull request from topic to base branch
• 📝 Rich PR Descriptions: Include error analysis, test details, and fix rationale
• 🎯 Draft PRs: Created as drafts for mandatory code review
• 🏷️ Auto Labels: auto-fix, test-failure, ai-generated labels
• ⚙️ Platform Support: GitHub and Azure DevOps

💾 Database & Analytics

• 📊 Test Run Tracking: Store complete test run metadata (environment, branch, commit, totals, duration)
• 📈 Result History: Track individual test results over time with full details
• 🔍 Failure Analysis: Query and analyze failure patterns with indexed searches
• 🏗️ Build Integration: Link results to CI/CD builds with metadata
• 💾 Multi-Database: SQLite (file-based), MySQL, PostgreSQL support
• 🗃️ Schema: 2 tables (test_runs, test_results) with 4 performance indexes

🏗️ Architecture

System Overview

graph TB
    subgraph "Playwright Test Runner"
        Tests[Test Execution]
    end

    subgraph "AI Test Reporter"
        Reporter[Reporter Core]
        Registry[Provider Registry]
        Workflow[Test Workflow Engine]
    end

    subgraph "AI Providers"
        Azure[Azure OpenAI]
        OpenAI[OpenAI]
        Anthropic[Anthropic Claude]
        Google[Google Gemini]
        Mistral[Mistral AI]
    end

    subgraph "Bug Trackers"
        GitHub[GitHub Issues]
        ADO[Azure DevOps]
        Jira[Jira]
    end

    subgraph "Databases"
        SQLite[SQLite]
        MySQL[MySQL]
        Postgres[PostgreSQL]
    end

    subgraph "Notifications"
        Email[Email/SMTP]
        Slack[Slack]
        Teams[MS Teams]
    end

    subgraph "PR Providers"
        GHPR[GitHub PRs]
        ADOPR[Azure Repos PRs]
    end

    Tests --> Reporter
    Reporter --> Registry
    Registry --> Workflow

    Workflow --> Azure
    Workflow --> OpenAI
    Workflow --> Anthropic
    Workflow --> Google
    Workflow --> Mistral

    Workflow --> GitHub
    Workflow --> ADO
    Workflow --> Jira

    Workflow --> SQLite
    Workflow --> MySQL
    Workflow --> Postgres

    Workflow --> Email
    Workflow --> Slack
    Workflow --> Teams

    Workflow --> GHPR
    Workflow --> ADOPR

    style Reporter fill:#4CAF50
    style Registry fill:#2196F3
    style Workflow fill:#FF9800

Provider Architecture

The reporter uses a provider-based architecture for maximum flexibility:

graph LR
    subgraph "Application Layer"
        Reporter[Test Reporter]
        Utils[Utilities]
    end

    subgraph "Provider Registry"
        Registry[Provider Registry<br/>Singleton Manager]
    end

    subgraph "Factory Layer"
        AIFactory[AI Factory]
        BugFactory[Bug Tracker Factory]
        DBFactory[Database Factory]
        NotifyFactory[Notification Factory]
        PRFactory[PR Factory]
    end

    subgraph "Provider Interfaces"
        IAI[IAIProvider]
        IBug[IBugTrackerProvider]
        IDB[IDatabaseProvider]
        INotify[INotificationProvider]
        IPR[IPRProvider]
    end

    subgraph "Concrete Implementations"
        AzureAI[Azure OpenAI]
        OpenAI[OpenAI]
        GitHubBug[GitHub Issues]
        SQLite[SQLite]
        EmailNotify[Email]
        GitHubPR[GitHub PRs]
    end

    Reporter --> Registry
    Utils --> Registry

    Registry --> AIFactory
    Registry --> BugFactory
    Registry --> DBFactory
    Registry --> NotifyFactory
    Registry --> PRFactory

    AIFactory --> IAI
    BugFactory --> IBug
    DBFactory --> IDB
    NotifyFactory --> INotify
    PRFactory --> IPR

    IAI --> AzureAI
    IAI --> OpenAI
    IBug --> GitHubBug
    IDB --> SQLite
    INotify --> EmailNotify
    IPR --> GitHubPR

    style Registry fill:#FF6B6B
    style IAI fill:#4ECDC4
    style IBug fill:#4ECDC4
    style IDB fill:#4ECDC4
    style INotify fill:#4ECDC4
    style IPR fill:#4ECDC4

Test Failure Workflow

sequenceDiagram
    participant PT as Playwright Test
    participant R as Reporter
    participant AI as AI Provider
    participant BT as Bug Tracker
    participant PR as PR Provider
    participant DB as Database
    participant N as Notification

    PT->>R: Test Failed
    R->>R: Categorize Error
    R->>R: Extract Test Code

    R->>AI: Generate Fix Suggestion
    AI-->>R: AI Analysis & Fix

    par Parallel Operations
        R->>BT: Create Bug/Issue
        BT-->>R: Bug Created
    and
        R->>DB: Save Test Result
        DB-->>R: Result Saved
    end

    alt Auto-Healing Enabled
        R->>PR: Create Fix PR
        PR-->>R: PR Created
    end

    R->>N: Send Notification
    N-->>R: Notification Sent

    R->>PT: Report Complete

Core Components

Design Principles

• 🔌 Provider Independence - Not locked into any single service
• 🏭 Factory Pattern - Clean, standardized provider creation
• ⚡ Lazy Initialization - Resources loaded only when needed
• 🛡️ Type Safety - Full TypeScript support
• 🧪 Testable - Easy mocking for unit tests
• 📦 Modular - Import only what you need

⚡ Quick Start

Prerequisites

• Node.js 18 or higher
• Playwright 1.51 or higher
• An AI provider API key (Azure OpenAI, OpenAI, Anthropic, Google AI, or Mistral)

1. Installation

# Install the reporter
npm install playwright-ai-reporter --save-dev

# Install optional peer dependencies based on your needs
npm install @azure/identity        # For Azure OpenAI with Managed Identity
npm install @octokit/rest          # For GitHub integration
npm install azure-devops-node-api  # For Azure DevOps integration
npm install mysql2                 # For MySQL database
npm install nodemailer            # For email notifications

2. Choose a Configuration Stack

Copy one of the pre-configured environment files from the examples:

# GitHub + Mistral AI + SQLite
cp examples/env-configs/.env.github-stack .env

# OR Azure DevOps + Azure OpenAI + MySQL
cp examples/env-configs/.env.azure-stack .env

# OR Jira + OpenAI + SQLite
cp examples/env-configs/.env.openai-jira .env

# OR Claude AI only (minimal setup)
cp examples/env-configs/.env.anthropic-minimal .env

3. Configure Your Credentials

Edit your .env file with your API keys and settings:

# AI Provider (choose one)
AI_PROVIDER=mistral
MISTRAL_API_KEY=your-api-key-here

# Bug Tracker (optional)
BUG_TRACKER_PROVIDER=github
GITHUB_TOKEN=ghp_your_personal_access_token
GITHUB_OWNER=your-org
GITHUB_REPO=your-repo

# Database (optional)
DATABASE_PROVIDER=sqlite
SQLITE_DATABASE_PATH=./data/test-results.db

# PR Provider (optional - for auto-PR generation)
PR_PROVIDER=github
BASE_BRANCH=main

4. Add to Playwright Config

Update your playwright.config.ts:

import {defineConfig} from '@playwright/test';

export default defineConfig({
    reporter: [
        ['list'],
        [
            'playwright-ai-reporter',
            {
                // Test thresholds
                slowTestThreshold: 3,
                maxSlowTestsToShow: 5,

                // Output
                outputDir: './test-results',
                showStackTrace: true,

                // AI & Automation features
                generateFix: true, // Generate AI fix suggestions
                createBug: false, // Auto-create bugs for failures
                generatePR: false, // Auto-create PRs with fixes
                publishToDB: false, // Save to database
                sendEmail: false, // Send email notifications
            },
        ],
    ],
});

5. Validate Setup

# From the examples folder
cd examples
npm install
npm run validate:config  # Check configuration

6. Run Your Tests

# From the examples folder
npm test

That's it! The reporter will now analyze failures, generate AI-powered fix suggestions, and optionally create bugs, PRs, or store results in a database based on your configuration.

🚀 Installation

NPM

npm install playwright-ai-reporter --save-dev

Yarn

yarn add -D playwright-ai-reporter

PNPM

pnpm add -D playwright-ai-reporter

⚙️ Configuration

Reporter Configuration Options

Feature Workflows

Generate Fix Only (Default)

{ generateFix: true, createBug: false, generatePR: false, publishToDB: false }
// → Creates AI fix suggestions in test-results/fixes/

Create Bugs for Failures

{ generateFix: false, createBug: true, generatePR: false, publishToDB: false }
// → Creates bugs in GitHub/Jira/Azure DevOps for each failure

Generate Fix + Auto PR

{ generateFix: true, createBug: false, generatePR: true, publishToDB: false }
// → Creates fix files + topic branch + draft PR with fixes

Full Stack (All Features)

{ generateFix: true, createBug: true, generatePR: true, publishToDB: true, sendEmail: true }
// → AI fixes + bug tracking + PRs + database logging + email notifications

Workflow: AI Fix → Topic Branch → PR

1. Test Fails → AI analyzes failure and generates fix suggestion
2. Generate Fix (generateFix: true) → Creates fix files in test-results/fixes/
3. Create Branch (generatePR: true) → Creates topic branch autofix/test-name-{timestamp}
4. Commit Changes → Commits AI fix to topic branch with detailed message
5. Create PR → Opens draft PR from topic branch → base branch with:
   • Error details and AI analysis
   • Labels: auto-fix, test-failure, ai-generated
   • Links to commit and test details
6. Review & Merge → Team reviews draft PR before merging

🔌 Provider Support

AI Providers

Bug Trackers

Databases

Notifications

PR Providers

💻 Usage Examples

Basic Usage (Environment Variables)

// playwright.config.ts
import {defineConfig} from '@playwright/test';

export default defineConfig({
    reporter: [
        [
            'playwright-ai-reporter',
            {
                generateFix: true,
                categorizeFailures: true,
                slowTestThreshold: 3,
                maxSlowTestsToShow: 5,
            },
        ],
    ],
});

Using Provider Registry

import {ProviderRegistry} from 'playwright-ai-reporter';

// Initialize providers
await ProviderRegistry.initialize({
    ai: {type: 'openai'},
    bugTracker: {type: 'github'},
    database: {type: 'sqlite'},
});

// Get providers
const ai = await ProviderRegistry.getAIProvider();
const bugTracker = await ProviderRegistry.getBugTrackerProvider();
const db = await ProviderRegistry.getDatabaseProvider();

Direct Factory Usage

import {AIProviderFactory} from 'playwright-ai-reporter';

// Create specific provider
const provider = await AIProviderFactory.createProvider('anthropic');

// Generate completion
const response = await provider.generateCompletion([
    {role: 'system', content: 'You are a test engineer.'},
    {role: 'user', content: 'Analyze this test failure...'},
]);

console.log(response.content);

Complete Workflow Example

import {ReporterWorkflow} from 'playwright-ai-reporter';

// Initialize
await ReporterWorkflow.initialize();

// Process test failure
await ReporterWorkflow.processTestFailure(failure, sourceCode);

// Save test run
const runId = await ReporterWorkflow.saveTestRun(summary);

// Send notifications
await ReporterWorkflow.sendNotification(summary, failures);

// Cleanup
await ReporterWorkflow.cleanup();

For more examples, check the examples folder.

📋 Output Examples

Successful Run

🚀 Starting test run: 3 tests using 2 workers
✅ Login test passed in 1.23s
✅ API integration test passed in 2.45s
⚠️ Payment test was skipped

✅ All 3 tests passed | 1 skipped | ⏱ Total: 3.68s

🖥️ Running locally

Additional Metrics:
- Average passed test time: 1.84s
- Slowest test took: 2.45s
- Top 3 slowest tests:
  1. API integration test: 2.45s
  2. Login test: 1.23s

⚠️ Warning: 1 test was skipped.
   Please ensure to test the skipped scenarios manually before deployment.

Failed Run with AI Fix Suggestions

🚀 Starting test run: 3 tests using 2 workers
✅ Login test passed in 1.23s
❌ API test failed in 2.45s
🔄 Retry attempt for "API test" (failed) in 2.50s
⚠️ Payment test was skipped

❌ 1 of 3 tests failed | 1 passed | 1 skipped | ⏱ Total: 6.18s

🤖 Generating AI-powered fix suggestions...
Generating fix suggestion for: API test
✅ Fix suggestion generated:
  - Prompt: ./test-results/prompts/api-test.md
  - Fix: ./test-results/fixes/fix-api-test.md

🔄 Generating pull request with fix...
   Creating topic branch: autofix/api-test-2025-12-30T10-30-45
   ✅ Branch created successfully
   Committing changes to autofix/api-test-2025-12-30T10-30-45
   ✅ Changes committed: a1b2c3d
   Creating pull request: autofix/api-test-2025-12-30T10-30-45 → main
✅ Pull request created successfully:
   PR #42: https://github.com/yourorg/yourrepo/pull/42
   Branch: autofix/api-test-2025-12-30T10-30-45 → main
   Status: open (draft)

AI fix suggestion generation complete

Additional Metrics:
- Average passed test time: 1.23s
- Slowest test took: 1.23s

Test Failures:
--- Failure #1 ---
  Test: API test
  Category: NetworkError
  Error: Connection timeout
  Stack Trace:
    at Connection.connect (/src/api/connection.ts:45:7)

HTML Test Health Report

After each test run, a self-contained HTML report is automatically generated:

📊 Generating self-contained test health report...
  ✓ Loaded testSummary.json (25 tests)
  ✓ Loaded testFailures.json (2 failures)
  ✓ Loaded HTML template
  ✓ Loaded CSS
  ✓ Loaded JavaScript
  ✓ Generated standalone report: E:\project\test-results\test-health-report.html

╔══════════════════════════════════════════════════════════════╗
║                    📊 Test Health Report                     ║
╠══════════════════════════════════════════════════════════════╣
║  Open in browser: file:///E:/project/test-results/test-health-report.html
║                                                              ║
║  Or run: npx playwright show-report (if using Playwright)   ║
╚══════════════════════════════════════════════════════════════╝

The HTML report includes:

• 📈 Interactive Charts: Test results overview, failure categories, duration analysis
• 📋 Test Details Grid: Searchable/filterable list of all tests with status badges
• 🔍 Failed Tests Section: Detailed error messages with AI fix suggestions
• ⚠️ Flaky Tests Analysis: Pattern detection and stability recommendations
• 🐌 Slowest Tests: Performance analysis and optimization targets
• 📎 Artifact Links: Screenshots, videos, and trace files for failed tests

❓ Frequently Asked Questions (FAQs)

General

Installation & Setup

AI & Fix Suggestions

Auto-Healing & PRs

test(
    'critical test',
    {
        annotation: {type: 'no-auto-heal', description: 'Manual review required'},
    },
    async ({page}) => {
        // Test code
    },
);

Integration & Providers

import {IAIProvider} from 'playwright-ai-reporter';

export class CustomAIProvider implements IAIProvider {
    async generateCompletion(messages) {
        // Your implementation
    }
}

Database & Analytics

import {ProviderRegistry} from 'playwright-ai-reporter';

const db = await ProviderRegistry.getDatabaseProvider();
const results = await db.query('SELECT * FROM test_results WHERE status = ? AND timestamp > ?', ['failed', oneWeekAgo]);

Troubleshooting

npm run validate:config

{
  reporters: [
    ['playwright-ai-reporter', {
      publishToDB: false,  // Disable database - no sqlite3 needed!
      // ... other options
    }]
  ]
}

Advanced Usage

import type {IAIProvider, IBugTrackerProvider, IDatabaseProvider} from 'playwright-ai-reporter';

📚 Documentation

📖 Complete Documentation Hub →

Quick Links

• Quick Start Guide - Get started in 5 minutes
• Environment Configuration - Complete setup guide with sample configurations
• Provider Documentation - Detailed provider documentation and usage
• Architecture & Design - System architecture and design decisions
• Implementation Details - Technical implementation overview
• API Reference - Complete API documentation
• Troubleshooting Guide - Common issues and solutions

Examples

• Environment Config Examples - Pre-configured .env files for different stacks
• Test Examples - Sample test files demonstrating reporter usage
• Workflow Examples - Code examples for common workflows

💡 New here? Start with the Quick Start Guide and ENV_CONFIG_GUIDE

🤝 Contributing

We welcome contributions! Here's how you can help:

1. Fork the repository
2. Create your feature branch: git checkout -b feature/amazing-feature
3. Make your changes and commit: git commit -m 'Add amazing feature'
4. Push to your fork: git push origin feature/amazing-feature
5. Open a Pull Request

Please ensure your PR:

• Follows the existing code style
• Includes appropriate tests
• Updates documentation as needed
• Describes the changes made

Adding New Providers

See docs/PROVIDERS.md for instructions on adding new provider implementations.

🌟 Why Choose This Reporter?

✅ Provider Independence - Not locked into any single service
✅ Enterprise Ready - Azure integration, managed identity, MySQL support
✅ Cost Optimized - Choose the most cost-effective AI provider
✅ Flexible - Use only the features you need
✅ Extensible - Easy to add new providers
✅ Type Safe - Full TypeScript support
✅ Production Tested - Battle-tested in real-world scenarios
✅ Well Documented - Comprehensive docs and examples
✅ Active Development - Regular updates and improvements
✅ Open Source - MIT licensed, community-driven

🎯 Use Cases

Enterprise CI/CD Pipeline

• Automatic bug creation in Azure DevOps for test failures
• Store historical test data in MySQL for trend analysis
• Email notifications to QA team
• Azure OpenAI for fix suggestions with enterprise security

Open Source Projects

• GitHub Issues for bug tracking
• SQLite for lightweight data storage
• GitHub PRs for automated fixes
• OpenAI or Anthropic for AI suggestions

Startup/Agile Teams

• Slack notifications for instant alerts
• Jira integration for sprint planning
• Google AI (Gemini) for cost-effective analysis
• Quick iteration with auto-healing

📝 License

This project is licensed under the MIT License - see the LICENSE file for details.

🙏 Acknowledgments

• Built with ❤️ for the Playwright community
• Inspired by the need for better test reporting and automatic debugging in CI/CD pipelines
• Multi-provider AI support: Azure OpenAI, OpenAI, Anthropic, Google AI, Mistral AI
• Thanks to all contributors who help make this reporter better

📞 Support

• 📖 Documentation
• 🐛 Issue Tracker
• 💬 Discussions
• 📧 Email Support
• 🐦 Twitter

Made with ❤️ by Deepak Kamboj for the Playwright community

⭐ Star us on GitHub if you find this useful!