generate-robotstxt
Awesome generator robots.txt
Found 62 results for robots-txt
Awesome generator robots.txt
A webpack plugin to output a robots.txt file
Generate a robots.txt for Astro
VibeSEO — Auditoria de SEO para quem cria com IA. Meta tags, sitemap, Open Graph e visibilidade no Google.
See your site the way search engines and AI see it.
Content use policy signal parsing for PEAC (robots.txt, tdmrep.json, Content-Usage)
CLI validator for ai-attribution.txt files. The AIACTA open standard for AI content attribution and publisher rights (AIACTA Proposal 4, §5.7)
Check if AI agents can discover, understand, and use your product
Lighthouse for AI SEO. Audit any webpage for AI search readiness. 7 categories, 30+ factors, research-backed scoring. Deterministic, engine-agnostic, zero API keys.
AI Readiness toolkit for Astro — six JSON-LD components + dist/llms.txt, dist/agents.md, dist/.well-known/mcp.json. dist/llms-full.txt and dist/robots.txt on the roadmap.
Framework-agnostic SEO analysis engines, types, and utilities
Answer Engine Optimization for the modern web. Make your site discoverable by AI crawlers and LLMs.
MCP server for GEO (Generative Engine Optimization) analysis — check any domain's AI-readiness
Agentic Engine Optimization toolkit — audit and test how well AI agents can find, parse, and use your website
Generate a robots.txt file for your Astro site
Framework-agnostic agent readiness primitives: robots.txt, sitemap, RFC 8288/9309/9727, agent-skills.
Framework-agnostic llms.txt, llms-full.txt, A2A Agent Cards, JSON-LD, markdown mirrors, headers, AI crawler controls, and validation
MCP server for PageSpeed & SEO audits. Zero config required - works out of the box.
Astro components and integration for managing page head, metadata, social sharing, icons, and SEO outputs.
Build-time llms.txt, llms-full.txt, A2A Agent Cards, JSON-LD, markdown mirrors, headers, AI crawler controls, and validation for Next.js
Framework-agnostic SEO utilities, metadata builders, structured data helpers, and React components for SSR applications
Generate AI Agent Experience (AX) files — companion to ax-audit
SEO optimization package for JavaScript/React/Next.js/Vue apps. Automatic meta tags, JSON-LD schema, OpenGraph, Twitter cards, sitemap generation, and SEO analysis. Perfect for AI code generation.
Make your site crawlable by AI agents in minutes
Build-time llms.txt, llms-full.txt, A2A Agent Cards, JSON-LD, markdown mirrors, headers, AI crawler controls, and validation for Astro
Promise based parser for robots.txt files.
Build-time llms.txt, llms-full.txt, A2A Agent Cards, JSON-LD, markdown mirrors, headers, AI crawler controls, and validation for Vite
SEO plugin for Momentum CMS — meta tags, sitemap, robots.txt, analysis
A fast, modular, SEO-optimized static site generator that minifies CSS, JS, and HTML for improved performance. It also supports JSON-LD, sitemap generation, and more, making it ideal for production-ready websites.
Next.js integration for GEO AI — middleware, route handler and static file generation for llms.txt / llms-full.txt
AI-powered GEO (Generative Engine Optimization) tool — generates llms.txt, robots.txt rules, and Schema markup to make websites discoverable by ChatGPT, Claude, Perplexity, and Gemini.
AI crawler blocking — generate robots.txt, meta tags, and HTTP headers for 30+ AI bots
A really lightweight and simple robots-txt parser.
Zero-dependency TypeScript engine for AI Search Optimization (GEO) — generates llms.txt, manages AI crawlers, SEO signals, caching and more
Make any website visible to ChatGPT, Claude, Gemini & Perplexity. Free MCP tools for AI-readiness scoring, robots.txt/llms.txt generation, VibeTags, and AI bot analysis. No API key required.
AI agent and SEO optimization plugin for Avalon — JSON-LD, llms.txt, markdown negotiation, sitemaps
SEO toolkit MCP server — analyze meta tags, robots.txt, sitemaps, keyword density, readability, and heading structure
A powerful, configurable SEO module for Nuxt 4 with automatic OG image generation
CLI for generating and validating llms.txt / llms-full.txt — AI Search Optimization (GEO) for any Node.js project
MCP Server for ZERONOVA LAB tools — SEO audit, link checking, OGP validation, config file generation, and more for AI agents
Vite plugin for automatic sitemap generation from TanStack Router route tree
Developer mini-tools: YAML, OpenAPI, JSON, Base64, Cron, Regex, Robots.txt, PostgreSQL, MySQL, UUID, Uptime, Dockerfile, JWT, Hash, Timestamp, DNS
Audit any website's AI-readiness. Score 0-100 for GEO (Generative Engine Optimization) best practices.
Framework-agnostic GEO asset generator — llms.txt, JSON-LD, robots.txt, and sitemap for AI discoverability.
Parser, generator, and validator for the ai.txt web standard — universal AI policy declaration
Unified CLI and fix pipeline for GeoKit — audit, generate, convert, and fix your site's AI-readiness.
Standalone AEO health checks (29 checks) and AI mentions analysis. No external API needed for health checks.
NestJS integration for GEO AI — dynamic module, middleware, controller, guard, interceptor and decorators for llms.txt / AI Search Optimization
Content-Signal headers and markdown content negotiation for AI agents. Like helmet for agent-readiness.
SEO toolkit MCP server — analyze meta tags, robots.txt, sitemaps, keyword density, readability, and heading structure
SEO linter for your deployment pipeline
SEO Quick Audit MCP Server — audit any URL in seconds from your AI assistant
Parser, generator, and validator for the agents.txt web standard
Lightweight robots.txt parsing component without any external dependencies for Node.js.
Drop-in fetch replacement for AI agents — identity headers, policy enforcement, and payment flows
Express middleware for serving agents.txt — one line to make your site AI-agent ready
WebScribe — The web's scribe. Crawls, chunks and semantically indexes any HTTP documentation for AI copilot agents.
GEO (Generative Engine Optimization) site audit tool — check how well your website is optimized for AI search engines
CLI for the Tollway protocol — generate agent identities, fetch URLs with identity headers, inspect site policies
MCP server for comprehensive SEO audits - PageSpeed, Technical SEO, Sitemap/Robots, On-Page SEO, Security Headers, Mobile-Friendliness. Zero config required.
The ultimate open-source marketing stack for developers. Ship marketing as code.
The ultimate open-source marketing stack for developers. Ship marketing as code.