robots-parser
A specification compliant robots.txt parser with wildcard (*) matching support.
Found 2 results for robots-exclusion-standard
robots-txt-component
Lightweight robots.txt parsing component without any external dependencies for Node.js.
Found 2 results for robots-exclusion-standard
A specification compliant robots.txt parser with wildcard (*) matching support.
Lightweight robots.txt parsing component without any external dependencies for Node.js.