Hyly E2E Test Automation Plugin

Transform your requirements into automated Playwright tests with AI-powered intelligence.

The Hyly E2E Plugin is a comprehensive test automation solution that converts requirements from Notion into fully automated, production-ready Playwright tests through an intelligent 6-stage pipeline.

Features

🚀 6-Stage Test Automation Pipeline

Complete end-to-end automation from requirements to execution:

1. Requirements Extraction - Pull requirements from Notion pages
2. Test Case Generation - Generate comprehensive test cases with AI
3. Notion Test Registration - Register test cases in Notion database
4. Playwright Script Generation - Create production-ready test scripts
5. Script Review & Auto-Fix - Review scripts with automatic improvements
6. Test Execution & Reporting - Run tests and report results to Notion

🎯 Intelligent Selector Discovery

• Live Application Inspection - Use Chrome DevTools to inspect real DOM elements
• Semantic Selectors - Prioritize accessibility-based selectors (getByRole, getByText)
• Pattern Learning - Learn from existing test patterns
• Best Practice Recommendations - Get selector alternatives and usage examples

🐛 Test Debugging Assistant

• Failure Analysis - Analyze test results, errors, and stack traces
• Root Cause Identification - Determine why tests fail
• Specific Fixes - Get exact code changes with file paths and line numbers
• Selector Improvements - Recommendations based on reference files

💡 Context-Aware Script Review

• Hints System - Use domain-specific hints for intelligent review
• Auto-Fix Loop - Automatically request improvements from script generator
• Quality Scoring - Score scripts on best practices and patterns
• Iterative Improvement - Re-review after fixes (max 2 iterations)

📝 Notion Integration

• Requirements Extraction - Parse Notion pages for user stories and acceptance criteria
• Test Case Registry - Track test cases in Notion database
• Results Reporting - Report test execution results back to Notion

Installation

Prerequisites

1. Node.js >= 18.0.0
2. Playwright >= 1.40.0
3. Chrome DevTools MCP Server (for selector-finder skill)
4. Notion API Access (optional, for Notion integration)

Install the Plugin

Option 1: Install from GitHub (Recommended)

/plugin install hyly-e2e-plugin@hyly-ai

Option 2: Manual Installation

1. Clone or download this repository
2. Copy the hyly-e2e-plugin folder to your project
3. Install via local path:

/plugin install ./hyly-e2e-plugin

Configuration

Environment Variables

Create a .env file in your project root with the following variables:

# Notion Integration (Required for pipeline stages 1, 3, 6)
NOTION_API_KEY=your_notion_api_key_here
NOTION_DATABASE_ID=your_notion_database_id_here

# Test Credentials (Optional - for your application)
TEST_EMAIL=your_test_email@example.com
TEST_PASSWORD=your_test_password_here

Notion Setup

1. Create Notion Integration

1. Go to https://www.notion.so/my-integrations
2. Click "New integration"
3. Name it (e.g., "E2E Test Automation")
4. Copy the API key

2. Create Test Case Database

Create a Notion database with these properties:

• Name (Title) - Test case name
• Status (Select) - Passed/Failed/Pending
• Test ID (Text) - Unique test identifier
• Priority (Select) - High/Medium/Low
• Tags (Multi-select) - Test categories
• Date (Date) - Last execution date
• Duration (Text) - Execution time
• Passed/Failed/Skipped (Number) - Test statistics

3. Share Database with Integration

1. Open your database in Notion
2. Click "Share" button
3. Search for your integration name
4. Click "Invite"

4. Extract Database ID

From the database URL: https://www.notion.so/{DATABASE_ID}?v=... Copy the DATABASE_ID portion.

Chrome DevTools MCP Server Setup

The selector-finder skill requires the Chrome DevTools MCP server:

1. Install the MCP server (if not already installed)
2. Configure in your Claude settings
3. Ensure Chrome DevTools MCP is running

Usage

Running the Complete Pipeline

The pipeline orchestrator manages all 6 stages:

# Invoke the pipeline orchestrator
"Run the test automation pipeline"

# Provide Notion URL when prompted
https://www.notion.so/your-requirements-page

The orchestrator will:

1. Extract requirements from Notion
2. Generate test cases (with human approval)
3. Register test cases to Notion (with human approval)
4. Generate Playwright scripts (with human approval)
5. Review scripts with auto-fix (with human approval)
6. Execute tests and report results

Using Individual Skills

Selector Finder Skill

Find reliable Playwright selectors:

# Invoke the selector-finder skill
"Find selector for the Add button on the dashboard page"

The skill will:

• Inspect the live application using Chrome DevTools
• Compare with existing selector patterns
• Recommend the best selector with alternatives
• Provide page object examples

Test Debugger Skill

Debug failed tests:

# Invoke the test-debugger skill
"Debug the failing dashboard export test"

The skill will:

• Analyze test results and error traces
• Identify the root cause
• Provide specific fixes with line numbers
• Suggest improvements based on best practices

Running Individual Pipeline Stages

You can run specific stages of the pipeline:

Stage 1: Extract Requirements

"Extract requirements from this Notion page: https://notion.so/..."

Stage 2: Generate Test Cases

"Generate test cases from the requirements file"

Stage 4: Generate Playwright Scripts

"Generate Playwright scripts from the registration summary"

Stage 5: Review Scripts

"Review the generated Playwright scripts"

When asked for hints, provide:

• master-hints.md - Comprehensive rules from all patterns
• default-hints.md - General Playwright best practices
• login-test-hints.md - Authentication patterns
• dashboard-widget-hints.md - Dashboard/widget specific
• Or any custom hint file from your project

Plugin Components

Skills (Agent Skills)

selector-finder

Finds and validates reliable Playwright selectors through live application inspection.

When to use:

• Finding selectors for new elements
• Replacing fragile selectors
• Learning best practices for selector usage
• Troubleshooting selector issues

Capabilities:

• Live DOM inspection with Chrome DevTools
• Semantic selector prioritization
• Pattern matching against proven selectors
• Complete usage examples and alternatives

test-debugger

Debugs failed Playwright tests with specific fix recommendations.

When to use:

• Test failures need investigation
• Understanding why tests break
• Getting specific code fixes
• Learning from test failures

Capabilities:

• Test result analysis
• Error trace interpretation
• Root cause identification
• Fix recommendations with line numbers

Agents (Subagents)

pipeline-orchestrator

Coordinates the entire 6-stage pipeline with human approval gates.

Responsibilities:

• Manage communication between all agents
• Track pipeline state and progress
• Handle human approval checkpoints
• Validate agent outputs for quality
• Request improvements when needed

notion-requirements-extractor (Stage 1)

Extracts and structures requirements from Notion pages.

Outputs:

• requirements.json - Structured requirements data
• requirements-summary.md - Human-readable summary

test-case-generator (Stage 2)

Generates comprehensive test cases from requirements.

Outputs:

• test-plan.json - Test cases with detailed steps

notion-test-registrar (Stage 3)

Registers test cases in Notion database.

Outputs:

• registration-summary.json - Registration status with Notion page IDs

playwright-test-generator (Stage 4)

Generates production-ready Playwright test scripts.

Outputs:

• Multiple .spec.ts files in tests directory
• generation-summary.json - Generation metadata

playwright-script-reviewer (Stage 5)

Reviews generated scripts with context-aware analysis and auto-fix.

Outputs:

• review-summary.json - Review results with scores
• Improved scripts (if auto-fix triggered)

test-execution-reporter (Stage 6)

Executes tests and reports results to Notion.

Outputs:

• execution-summary.json - Test execution results
• Notion database updates with test outcomes

Hints System

Hint files provide context-aware review rules:

• master-hints.md - Comprehensive rules from all test patterns
• default-hints.md - General Playwright best practices
• login-test-hints.md - Authentication and login patterns
• dashboard-widget-hints.md - Dashboard and widget testing
• export-test-hints.md - Export functionality rules
• filter-hints.md - Filter component patterns
• favorites-hints.md - Favorites feature patterns
• help-center-hints.md - Help center testing
• hayley-navigation-hints.md - Navigation app specifics
• pdf-template-*.md - PDF template testing rules
• table-explorer-*.md - Data table testing patterns
• regression-test-hints.md - Regression testing standards

Project Structure

hyly-e2e-plugin/
├── .claude-plugin/
│   └── plugin.json           # Plugin manifest
├── skills/
│   ├── selector-finder/
│   │   └── SKILL.md         # Selector discovery skill
│   └── test-debugger/
│       └── SKILL.md         # Test debugging skill
├── agents/
│   ├── 00-pipeline-orchestrator.md
│   ├── 01-notion-requirements-extractor.md
│   ├── 02-test-case-generator.md
│   ├── 03-notion-test-registrar.md
│   ├── 04-playwright-test-generator.md
│   ├── 05-playwright-script-reviewer.md
│   └── 06-test-execution-reporter.md
├── hints/
│   ├── master-hints.md
│   ├── default-hints.md
│   └── ... (14 hint files total)
├── scripts/
│   ├── stage1-notion-requirements-reader.js
│   ├── stage3-notion-test-registration.js
│   └── stage6-test-execution-reporting.js
└── README.md

Best Practices

Selector Priority (ALWAYS Follow This Order)

1. Semantic Selectors (BEST)

   page.getByRole('button', { name: 'Add' })
   page.getByText('Dashboard', { exact: true })

2. Data Test IDs (GOOD)

   page.getByTestId('add-widget-button')

3. ID Selectors (ACCEPTABLE)

   page.locator('#email-input')

4. Class Selectors (USE WITH CAUTION)

   page.locator('.dashboard-card')

5. Complex Selectors (LAST RESORT)

   page.locator('div > ul > li:nth-child(3)')

Test Structure Requirements

Always include in your tests:

import { test, expect } from '@playwright/test';
import { LoginPage } from './pages/login.page';
import { DashboardPage } from './pages/dashboard.page';

test.describe('Feature Name', () => {
  // Set timeout for slow operations
  test.setTimeout(10 * 60 * 1000);

  test('Test description', async ({ page }) => {
    // Use page objects
    const loginPage = new LoginPage(page);
    const dashboardPage = new DashboardPage(page);

    await test.step('Step 1: Login', async () => {
      await loginPage.navigate();
      await loginPage.login();
    });

    await test.step('Step 2: Navigate', async () => {
      await dashboardPage.navigate();
    });

    // Add assertions
    await expect(page).toHaveTitle(/Expected Title/);
  });
});

Common Patterns to Avoid

❌ DON'T:

• Use waitForLoadState('networkidle') - causes browser closure
• Hardcode credentials - use environment variables
• Use nth() or position-based selectors without good reason
• Create tests without page objects
• Skip waitFor() before interactions

✅ DO:

• Use semantic selectors (getByRole, getByText)
• Use test.step() for organization
• Add proper timeouts for slow operations
• Use page object model
• Add waitFor() before interactions
• Use environment variables for config

Troubleshooting

Pipeline Issues

Issue: Stage 1 not extracting requirements properly Solution: Ensure NOTION_API_KEY is set and Notion page is shared with integration

Issue: Scripts fail to generate in Stage 4 Solution: Check that registration-summary.json has valid test case data

Issue: Tests fail after generation Solution: Run Stage 5 review with appropriate hint file, let auto-fix improve scripts

Skill Issues

Issue: Selector-finder can't inspect application Solution: Ensure Chrome DevTools MCP server is running and configured

Issue: Test-debugger can't find test results Solution: Check that test-results directory exists with recent test output

General Issues

Issue: Plugin not recognized after installation Solution: Restart Claude Code CLI after plugin installation

Issue: Environment variables not loading Solution: Ensure .env file is in project root and properly formatted

Contributing

To adapt this plugin for your project:

1. Customize Hints - Create project-specific hint files
2. Update Scripts - Modify stage scripts for your workflow
3. Add Selectors - Document your application's selector patterns
4. Extend Skills - Add new skills for project-specific needs

Support

For issues, questions, or contributions:

• GitHub Issues: Report an issue
• Documentation: View full docs
• Website: https://hyly.ai

License

MIT License - See LICENSE file for details

Acknowledgments

Built with:

• Playwright - Web testing framework
• Claude Code - AI-powered development assistant
• Notion API - Knowledge management integration

Made with ❤️ by Hyly.ai

Automate smarter, test better, ship faster.