isbot
🤖/👨🦰 Recognise bots/crawlers/spiders using the user agent string.
Found 29 results for crawlers
@crawlee/utils
A set of shared utilities that can be used by crawlers
sitemapper
Parser for XML Sitemaps to be used with Robots.txt and web crawlers
robot-directives
Parse robot directives within HTML meta and/or HTTP headers.
sitemapper_mos
Parser for XML Sitemaps to be used with Robots.txt and web crawlers
@microlink/ua
A simple redis primitives to incr() and top() user agents
@qbbsh/sitemapper
Parser for XML Sitemaps to be used with Robots.txt and web crawlers
crawler-js
Opensource Framework Crawler in Node.js
bot-detector
It uses the user-agents.org xml file for detecting bots.
robots-txt-component
Lightweight robots.txt parsing component without any external dependencies for Node.js.
@context-labs/sitemapper
Parser for XML Sitemaps to be used with Robots.txt and web crawlers
@javivelasco/isbot
🤖 detect bots/crawlers/spiders via the user agent.
@wmod/sitemapper
Parser for XML Sitemaps to be used with Robots.txt and web crawlers
isbot-bundle
isbot Bundle UMD
hidemyemail
A jQuery plugin that helps you to hide your email on your page and prevent crawlers to get it!
@yeskiy/sitemapper
Parser for XML Sitemaps to be used with Robots.txt and web crawlers
@public-square/sitemapper
Parser for XML Sitemaps to be used with Robots.txt and web crawlers
@drorgl/sitemapper
Parser for XML Sitemaps to be used with Robots.txt and web crawlers
planisphere
A straightforward sitemap generator written in TypeScript.
@raing3/sitemapper
Parser for XML Sitemaps to be used with Robots.txt and web crawlers
@jarronb/site-mapper
Site mapper utilizing puppeteer to generate a xml sitemap
aliyu-crawlers
crawlers
@cabbiepete/sitemapper
Parser for XML Sitemaps to be used with Robots.txt and web crawlers
simplecrawling
Crawler made simple
adibenmat-sitemapper
Parser for XML Sitemaps to be used with Robots.txt and web crawlers
@myagizmaktav/sitemapper
Parser for XML Sitemaps to be used with Robots.txt and web crawlers
@mastixmc/sitemapper
Parser for XML Sitemaps to be used with Robots.txt and web crawlers. (Extended version by mastixmc)
@vladfrangu-dev/crawlee-utils
A set of shared utilities that can be used by crawlers
@xianyukeji/isbot
detects bots/crawlers/spiders via the user agent.