Package Exports
- scrappey-wrapper
- scrappey-wrapper/axios
- scrappey-wrapper/fetch
Readme
Scrappey - Web Scraping API Wrapper
The official Node.js wrapper for the Scrappey web scraping API. Bypass Cloudflare, Datadome, PerimeterX, and other antibot protections. Solve captchas automatically.
Features
- Antibot Bypass: Cloudflare, Datadome, PerimeterX, Kasada, Akamai, and more
- Captcha Solving: reCAPTCHA, hCaptcha, Turnstile, FunCaptcha automatic solving
- Browser Automation: Full browser actions (click, type, scroll, execute JS)
- Session Management: Persistent sessions with cookie and state management
- All HTTP Methods: GET, POST, PUT, DELETE, PATCH support
- Proxy Support: Built-in residential proxies with country selection
- Screenshots & Video: Capture screenshots and record browser sessions
- TypeScript Support: Full TypeScript declarations included
Pricing Comparison
Compare Scrappey with other web scraping services:
| Features | Scrappey | ZenRows | ScrapingBee | Scrapfly |
|---|---|---|---|---|
| Price per 1K Scrapes (JS Render + Residential Proxies) |
€1 | $25 | $25 | $187 |
| Concurrent Requests (Simultaneous scraping) |
200 | 10 | 5 | 5 |
| Browser Automation (Actions and interactions) |
30+ Actions | Basic | Basic | Basic |
| Billing Model (Payment flexibility) |
Pay-as-you-go | Monthly | Monthly | Monthly |
| Success Rate (Successful scrapes) |
95% | 95% | 95% | 95% |
Why Scrappey?
- 25x cheaper than competitors for JS rendering with residential proxies
- 20x more concurrent requests for faster scraping
- Most advanced browser automation with 30+ actions
- Flexible billing - pay only for what you use, no monthly commitments
- Same high success rate as premium competitors
How It Works
Scrappey provides a powerful web scraping API that handles all the complexity of bypassing antibot systems, solving captchas, and managing browser sessions. Here's how it works:
flowchart TB
subgraph YourApp[Your Application]
Code[Your Code]
Wrapper[Scrappey Wrapper]
end
subgraph ScrappeyAPI[Scrappey API]
RequestHandler[Request Handler]
AntibotBypass[Antibot Bypass Engine]
CaptchaSolver[Captcha Solver]
BrowserEngine[Browser Engine]
ProxyManager[Proxy Manager]
end
subgraph TargetSite[Target Website]
Cloudflare[Cloudflare Protection]
Datadome[Datadome Protection]
Captcha[CAPTCHA Challenge]
Website[Website Content]
end
Code -->|1. Send Request| Wrapper
Wrapper -->|2. API Request| RequestHandler
RequestHandler -->|3. Route Request| AntibotBypass
AntibotBypass -->|4. Bypass Protection| Cloudflare
AntibotBypass -->|4. Bypass Protection| Datadome
RequestHandler -->|5. Handle Captcha| CaptchaSolver
CaptchaSolver -->|6. Solve Automatically| Captcha
RequestHandler -->|7. Execute| BrowserEngine
BrowserEngine -->|8. Use Proxy| ProxyManager
ProxyManager -->|9. Residential IP| Website
Website -->|10. Return Content| BrowserEngine
BrowserEngine -->|11. Process Response| RequestHandler
RequestHandler -->|12. Return Data| Wrapper
Wrapper -->|13. Formatted Response| CodeRequest Flow
- Your Code → Send request through Scrappey wrapper
- Scrappey API → Receives and processes your request
- Antibot Bypass → Automatically bypasses Cloudflare, Datadome, PerimeterX, etc.
- Captcha Solver → Detects and solves CAPTCHAs automatically
- Browser Engine → Executes browser actions if needed (click, type, scroll)
- Proxy Manager → Routes through residential proxies for anonymity
- Target Website → Returns content successfully
- Response → Formatted data returned to your application
Use Cases
- E-commerce Scraping: Product data, prices, reviews
- Social Media: Profile data, posts, engagement metrics
- Search Engines: SERP data, rankings
- Real Estate: Property listings, prices, details
- News & Content: Articles, headlines, metadata
- API Testing: Test your own APIs with different IPs
- Data Aggregation: Collect data from multiple sources
Installation
npm install scrappey-wrapperQuick Start
const Scrappey = require('scrappey-wrapper');
const scrappey = new Scrappey('YOUR_API_KEY');
// Basic GET request
const response = await scrappey.get({
url: 'https://example.com'
});
console.log(response.solution.response); // HTML content
console.log(response.solution.statusCode); // 200Request Modes
Scrappey supports two request modes to optimize for your use case:
| Mode | Description | Cost | Best For |
|---|---|---|---|
browser |
Headless browser (default) | 1 + 0.2 balance/request | Complex pages, JS rendering, browser actions, antibot bypass |
request |
HTTP library with TLS | 0.2 balance/request | Simple requests, speed-critical, cost-sensitive applications |
Using Browser Mode (Default)
Browser mode uses a real headless browser, enabling full JavaScript execution, browser actions, and advanced antibot bypass:
const response = await scrappey.get({
url: 'https://example.com',
requestType: 'browser', // Default - can be omitted
browserActions: [
{ type: 'click', cssSelector: '#button' },
{ type: 'wait', wait: 1000 }
]
});Using Request Mode (Faster & Cheaper)
Request mode uses an HTTP library with TLS fingerprinting - much faster and cheaper for simple requests:
const response = await scrappey.get({
url: 'https://api.example.com/data',
requestType: 'request' // Use HTTP mode - 5x cheaper!
});When to use request mode:
- API calls and JSON endpoints
- Simple HTML pages without heavy JavaScript
- High-volume scraping where cost matters
- Speed-critical applications
When to use browser mode:
- Sites with JavaScript-rendered content
- Complex antibot systems (Cloudflare, Datadome, etc.)
- When you need browser actions (click, type, scroll)
- CAPTCHA solving
Drop-in Replacement for Axios/Fetch
New in v2.0.0: Use Scrappey as a drop-in replacement for axios or fetch! Simply change your import and all requests automatically go through Scrappey with antibot bypass, captcha solving, and proxy support.
Axios Drop-in
Replace axios with Scrappey - just change the import:
// Before
import axios from 'axios';
const response = await axios.get('https://example.com');
// After - just change the import!
import axios from 'scrappey-wrapper/axios';
axios.defaults.apiKey = 'YOUR_API_KEY';
const response = await axios.get('https://example.com');
// Automatically bypasses Cloudflare, solves captchas, etc.All axios methods work:
import axios from 'scrappey-wrapper/axios';
axios.defaults.apiKey = 'YOUR_API_KEY';
// GET request
const response = await axios.get('https://example.com', {
headers: { 'Authorization': 'Bearer token' },
params: { page: 1 }
});
// POST request
const response = await axios.post('https://api.example.com/submit', {
name: 'John',
email: 'john@example.com'
}, {
headers: { 'Content-Type': 'application/json' }
});
// All standard axios options work
const response = await axios.get('https://example.com', {
timeout: 5000,
proxy: 'http://proxy:port',
cookies: 'session=abc123',
responseType: 'json'
});
// Scrappey-specific options also work
const response = await axios.get('https://protected-site.com', {
cloudflareBypass: true,
premiumProxy: true,
automaticallySolveCaptchas: true
});Response format matches axios:
const response = await axios.get('https://example.com');
console.log(response.data); // Response body
console.log(response.status); // HTTP status code
console.log(response.statusText); // Status text
console.log(response.headers); // Response headers
console.log(response.scrappey); // Additional Scrappey data (verified, cookies, etc.)Fetch Drop-in
Replace fetch with Scrappey:
// Before
const response = await fetch('https://example.com');
const data = await response.json();
// After
import fetch from 'scrappey-wrapper/fetch';
fetch.configure({ apiKey: 'YOUR_API_KEY' });
const response = await fetch('https://example.com', {
cloudflareBypass: true
});
const data = await response.json();All fetch methods work:
import fetch from 'scrappey-wrapper/fetch';
fetch.configure({ apiKey: 'YOUR_API_KEY' });
// GET request
const response = await fetch('https://example.com', {
headers: { 'Authorization': 'Bearer token' }
});
// POST request
const response = await fetch('https://api.example.com/submit', {
method: 'POST',
headers: { 'Content-Type': 'application/json' },
body: JSON.stringify({ name: 'John', email: 'john@example.com' })
});
// Response methods
const text = await response.text();
const json = await response.json();
const blob = await response.blob();
const arrayBuffer = await response.arrayBuffer();Migration Guide
From Axios:
- Change import:
import axios from 'axios'→import axios from 'scrappey-wrapper/axios' - Set API key:
axios.defaults.apiKey = 'YOUR_API_KEY' - That's it! All your existing code works the same.
From Fetch:
- Change import:
import fetch from 'node-fetch'→import fetch from 'scrappey-wrapper/fetch' - Configure:
fetch.configure({ apiKey: 'YOUR_API_KEY' }) - All fetch calls now use Scrappey automatically.
Configuration
Axios:
import axios from 'scrappey-wrapper/axios';
// Set defaults
axios.defaults.apiKey = 'YOUR_API_KEY';
axios.defaults.cloudflareBypass = true;
axios.defaults.premiumProxy = true;
axios.defaults.timeout = 60000;
// Or create custom instance
const scrappeyAxios = axios.create({
apiKey: 'YOUR_API_KEY',
cloudflareBypass: true,
premiumProxy: true
});Fetch:
import fetch from 'scrappey-wrapper/fetch';
// Configure globally
fetch.configure({
apiKey: 'YOUR_API_KEY',
cloudflareBypass: true,
premiumProxy: true,
timeout: 60000
});Supported Options
All standard axios/fetch options are supported:
headers→customHeadersdata/body→postDataparams→ URL query stringtimeout→timeoutproxy→proxycookies→cookiesorcookiejarresponseType: 'json'→ UsesinnerTextfor JSON
Plus all Scrappey-specific options:
cloudflareBypass,datadomeBypass,kasadaBypassautomaticallySolveCaptchas,alwaysLoadbrowserActions,screenshot,videosession,premiumProxy,proxyCountry- And many more!
Session Management
Both adapters support Scrappey session management:
// Axios
import axios from 'scrappey-wrapper/axios';
axios.defaults.apiKey = 'YOUR_API_KEY';
const session = await axios.createSession();
const sessionId = session.session;
await axios.get('https://example.com', { session: sessionId });
await axios.destroySession(sessionId);
// Fetch
import fetch from 'scrappey-wrapper/fetch';
fetch.configure({ apiKey: 'YOUR_API_KEY' });
const session = await fetch.createSession();
const sessionId = session.session;
await fetch('https://example.com', { session: sessionId });
await fetch.destroySession(sessionId);API Reference
Constructor
const scrappey = new Scrappey(apiKey, options);| Parameter | Type | Description |
|---|---|---|
apiKey |
string |
Your Scrappey API key (required) |
options.baseUrl |
string |
Custom API base URL (optional) |
options.timeout |
number |
Default timeout in ms (default: 300000) |
HTTP Methods
GET Request
const response = await scrappey.get({
url: 'https://example.com',
session: 'optional-session-id',
// ... other options
});POST Request
const response = await scrappey.post({
url: 'https://api.example.com/submit',
postData: {
name: 'John Doe',
email: 'john@example.com'
},
customHeaders: {
'content-type': 'application/json'
}
});PUT, DELETE, PATCH
await scrappey.put({ url, postData, ...options });
await scrappey.delete({ url, ...options });
await scrappey.patch({ url, postData, ...options });Session Management
// Create a session
const session = await scrappey.createSession({
proxyCountry: 'UnitedStates',
// proxy: 'http://user:pass@ip:port'
});
const sessionId = session.session;
// Use the session
await scrappey.get({
url: 'https://example.com',
session: sessionId
});
// Check if session is active
const status = await scrappey.isSessionActive(sessionId);
// List all sessions
const sessions = await scrappey.listSessions(userId);
// Destroy session when done
await scrappey.destroySession(sessionId);Browser Actions
Execute browser automation actions:
const response = await scrappey.get({
url: 'https://example.com/login',
browserActions: [
{
type: 'wait_for_selector',
cssSelector: '#login-form'
},
{
type: 'type',
cssSelector: '#username',
text: 'myuser'
},
{
type: 'type',
cssSelector: '#password',
text: 'mypassword'
},
{
type: 'click',
cssSelector: '#submit',
waitForSelector: '.dashboard'
},
{
type: 'execute_js',
code: 'document.querySelector(".user-info").innerText'
}
]
});
// Access JS execution results
console.log(response.solution.javascriptReturn[0]);Available Actions
| Action | Description |
|---|---|
click |
Click on an element |
type |
Type text into an input |
goto |
Navigate to a URL |
wait |
Wait for milliseconds |
wait_for_selector |
Wait for element to appear |
wait_for_function |
Wait for JS condition |
wait_for_load_state |
Wait for page load state |
wait_for_cookie |
Wait for cookie to be set |
execute_js |
Execute JavaScript |
scroll |
Scroll to element or bottom |
hover |
Hover over element |
keyboard |
Press keyboard keys |
dropdown |
Select dropdown option |
switch_iframe |
Switch to iframe context |
set_viewport |
Set viewport size |
if |
Conditional actions |
while |
Loop actions |
solve_captcha |
Solve captcha |
Antibot Bypass
const response = await scrappey.get({
url: 'https://protected-site.com',
datadomeBypass: true,
kasadaBypass: true,
premiumProxy: true,
proxyCountry: 'UnitedStates'
});Captcha Solving
// Automatic captcha solving
const response = await scrappey.get({
url: 'https://example.com',
automaticallySolveCaptchas: true,
alwaysLoad: ['recaptcha', 'hcaptcha', 'turnstile']
});
// Manual captcha solving with browser action
const response = await scrappey.get({
url: 'https://example.com',
browserActions: [
{
type: 'solve_captcha',
captcha: 'turnstile',
captchaData: {
sitekey: '0x4AAAAAAA...',
cssSelector: '.cf-turnstile'
}
}
]
});Screenshots & Video
const response = await scrappey.get({
url: 'https://example.com',
screenshot: true,
screenshotWidth: 1920,
screenshotHeight: 1080,
video: true
});
// Base64 screenshot
const screenshot = response.solution.screenshot;
// Video URL
const videoUrl = response.solution.videoUrl;Data Extraction
const response = await scrappey.get({
url: 'https://example.com',
cssSelector: '.product-title',
innerText: true,
includeLinks: true,
includeImages: true,
regex: 'price: \\$([0-9.]+)'
});Request Interception
const response = await scrappey.get({
url: 'https://example.com',
interceptFetchRequest: '/api/data',
abortOnDetection: ['analytics', 'tracking', 'ads'],
whitelistedDomains: ['example.com', 'cdn.example.com'],
blackListedDomains: ['ads.com']
});
// Intercepted data
console.log(response.solution.interceptFetchRequestResponse);Configuration Options
| Option | Type | Description |
|---|---|---|
url |
string |
Target URL (required) |
session |
string |
Session ID for session reuse |
proxy |
string |
Proxy string (http://user:pass@ip:port) |
proxyCountry |
string |
Proxy country (e.g., "UnitedStates") |
premiumProxy |
boolean |
Use premium proxy pool |
mobileProxy |
boolean |
Use mobile proxy pool |
postData |
object/string |
POST/PUT/PATCH request data |
customHeaders |
object |
Custom HTTP headers |
cookies |
string |
Cookie string to set |
cookiejar |
array |
Cookie jar array |
localStorage |
object |
LocalStorage data to set |
browserActions |
array |
Browser actions to execute |
cloudflareBypass |
boolean |
Enable Cloudflare bypass |
datadomeBypass |
boolean |
Enable Datadome bypass |
kasadaBypass |
boolean |
Enable Kasada bypass |
automaticallySolveCaptchas |
boolean |
Auto-solve captchas |
cssSelector |
string |
Extract content by CSS selector |
innerText |
boolean |
Include page text content |
includeLinks |
boolean |
Include all page links |
includeImages |
boolean |
Include all page images |
screenshot |
boolean |
Capture screenshot |
video |
boolean |
Record video |
pdf |
boolean |
Generate PDF |
filter |
array |
Filter response fields |
timeout |
number |
Request timeout (ms) |
Response Structure
{
solution: {
verified: true, // Request verification status
response: '<html>...', // HTML content
statusCode: 200, // HTTP status code
currentUrl: 'https://...', // Final URL after redirects
userAgent: 'Mozilla/5.0...', // User agent used
cookies: [...], // Array of cookies
cookieString: '...', // Cookie string
responseHeaders: {...}, // Response headers
innerText: '...', // Page text content
screenshot: 'base64...', // Base64 screenshot
screenshotUrl: 'https://...', // Screenshot URL
videoUrl: 'https://...', // Video URL
javascriptReturn: [...], // JS execution results
},
timeElapsed: 1234, // Request time (ms)
data: 'success', // 'success' or 'error'
session: 'session-id', // Session ID
error: '...', // Error message (if failed)
}Error Handling
try {
const response = await scrappey.get({ url: 'https://example.com' });
if (response.data === 'error') {
console.error('Error:', response.error);
// Handle specific error codes
// See: https://wiki.scrappey.com/getting-started
}
} catch (error) {
// Network or request error
console.error('Request failed:', error.message);
}Examples
Multi-language examples are available in the examples directory:
- Node.js: examples/nodejs
- TypeScript: examples/typescript
- Python: examples/python
- Go: examples/go
- C#: examples/csharp
- PHP: examples/php
- Java: examples/java
- Ruby: examples/ruby
- Rust: examples/rust
- cURL: examples/curl
TypeScript
Full TypeScript support is included:
import Scrappey = require('scrappey-wrapper');
const scrappey = new Scrappey('API_KEY');
const response = await scrappey.get({
url: 'https://example.com',
browserActions: [
{
type: 'execute_js',
code: 'document.title'
}
]
});
// Fully typed response
console.log(response.solution.statusCode);Links
- Website: https://scrappey.com
- Documentation: https://wiki.scrappey.com
- Request Builder: https://app.scrappey.com/#/builder
- GitHub: https://github.com/pim97/scrappey.js
- NPM: https://www.npmjs.com/package/scrappey-wrapper
License
This project is licensed under the MIT License - see the LICENSE file for details.
Disclaimer
Please ensure that your web scraping activities comply with the website's terms of service and legal regulations. Scrappey is not responsible for any misuse or unethical use of the library. Use it responsibly and respect the website's policies.