urllib

Request HTTP URLs in a complex world — basic and digest authentication, redirections, cookies, timeout and more.

Install

npm install urllib --save

Usage

TypeScript and ESM

import { request } from 'urllib';

const { data, res } = await request('http://cnodejs.org/');
// result: { data: Buffer, res: Response }
console.log('status: %s, body size: %d, headers: %j', res.statusCode, data.length, res.headers);

CommonJS

const { request } = require('urllib');

const { data, res } = await request('http://cnodejs.org/');
// result: { data: Buffer, res: Response }
console.log('status: %s, body size: %d, headers: %j', res.statusCode, data.length, res.headers);

Global `response` event

You should create a urllib instance first.

import { HttpClient } from 'urllib';

const httpclient = new HttpClient();
httpclient.on('response', (info) => {
  error: err,
  ctx: args.ctx,
  req: {
    url: url,
    options: options,
    size: requestSize,
  },
  res: res
});

const { data, res } = await httpclient.request('https://nodejs.org');
console.log('status: %s, body size: %d, headers: %j', res.statusCode, data.length, res.headers);

API Doc

Method: `async request(url[, options])`

Arguments

url String | Object - The URL to request, either a String or a Object that return by url.parse.
options Object - Optional
- method String - Request method, defaults to GET. Could be GET, POST, DELETE or PUT. Alias 'type'.
- data Object - Data to be sent. Will be stringify automatically.
- content String | Buffer - Manually set the content of payload. If set, data will be ignored.
- stream stream.Readable - Stream to be pipe to the remote. If set, data and content will be ignored.
- writeStream stream.Writable - A writable stream to be piped by the response stream. Responding data will be write to this stream and callback will be called with data set null after finished writing.
- files {Array<ReadStream|Buffer|String> | Object | ReadStream | Buffer | String - The files will send with multipart/form-data format, base on formstream. If method not set, will use POST method by default.
- contentType String - Type of request data. Could be json (Notes: not use application/json here). If it's json, will auto set Content-Type: application/json header.
- dataType String - Type of response data. Could be text or json. If it's text, the callbacked data would be a String. If it's json, the data of callback would be a parsed JSON Object and will auto set Accept: application/json header. Default callbacked data would be a Buffer.
- fixJSONCtlChars Boolean - Fix the control characters (U+0000 through U+001F) before JSON parse response. Default is false.
- headers Object - Request headers.
- timeout Number | Array - Request timeout in milliseconds for connecting phase and response receiving phase. Defaults to exports.TIMEOUT, both are 5s. You can use timeout: 5000 to tell urllib use same timeout on two phase or set them seperately such as timeout: [3000, 5000], which will set connecting timeout to 3s and response 5s.
- auth String - username:password used in HTTP Basic Authorization.
- digestAuth String - username:password used in HTTP Digest Authorization.
- agent http.Agent - HTTP Agent object. Set false if you does not use agent.
- httpsAgent https.Agent - HTTPS Agent object. Set false if you does not use agent.
- ca String | Buffer | Array - An array of strings or Buffers of trusted certificates. If this is omitted several well known "root" CAs will be used, like VeriSign. These are used to authorize connections. Notes: This is necessary only if the server uses the self-signed certificate
- rejectUnauthorized Boolean - If true, the server certificate is verified against the list of supplied CAs. An 'error' event is emitted if verification fails. Default: true.
- pfx String | Buffer - A string or Buffer containing the private key, certificate and CA certs of the server in PFX or PKCS12 format.
- key String | Buffer - A string or Buffer containing the private key of the client in PEM format. Notes: This is necessary only if using the client certificate authentication
- cert String | Buffer - A string or Buffer containing the certificate key of the client in PEM format. Notes: This is necessary only if using the client certificate authentication
- passphrase String - A string of passphrase for the private key or pfx.
- ciphers String - A string describing the ciphers to use or exclude.
- secureProtocol String - The SSL method to use, e.g. SSLv3_method to force SSL version 3.
- followRedirect Boolean - follow HTTP 3xx responses as redirects. defaults to false.
- maxRedirects Number - The maximum number of redirects to follow, defaults to 10.
- formatRedirectUrl Function - Format the redirect url by your self. Default is url.resolve(from, to).
- beforeRequest Function - Before request hook, you can change every thing here.
- streaming Boolean - let you get the res object when request connected, default false. alias customResponse
- gzip Boolean - Accept gzip, br response content and auto decode it, default is false.
- timing Boolean - Enable timing or not, default is false.
- lookup Function - Custom DNS lookup function, default is dns.lookup. Require node >= 4.0.0(for http protocol) and node >=8(for https protocol)
- checkAddress Function: optional, check request address to protect from SSRF and similar attacks. It receive tow arguments(ip and family) and should return true or false to identified the address is legal or not. It rely on lookup and have the same version requirement.

Options: `options.data`

When making a request:

await request('https://example.com', {
  method: 'GET',
  data: {
    'a': 'hello',
    'b': 'world',
  }
});

For GET request, data will be stringify to query string, e.g. http://example.com/?a=hello&b=world.

For others like POST, PATCH or PUT request, in defaults, the data will be stringify into application/x-www-form-urlencoded format if content-type header is not set.

If content-type is application/json, the data will be JSON.stringify to JSON data format.

Options: `options.content`

options.content is useful when you wish to construct the request body by yourself, for example making a content-type: application/json request.

Notes that if you want to send a JSON body, you should stringify it yourself:

await request('https://example.com', {
  method: 'POST',
  headers: {
    'Content-Type': 'application/json'
  },
  content: JSON.stringify({
    a: 'hello',
    b: 'world',
  })
});

It would make a HTTP request like:

POST / HTTP/1.1
host: example.com
content-type: application/json

{
  "a": "hello",
  "b": "world"
}

This exmaple can use options.data with application/json content type:

await request('https://example.com', {
  method: 'POST',
  headers: {
    'content-type': 'application/json'
  },
  data: {
    a: 'hello',
    b: 'world',
  }
});

Options: `options.files`

Upload a file with a hello field.

await request('https://example.com/upload', {
  method: 'POST',
  files: __filename,
  data: {
    hello: 'hello urllib',
  },
});

Upload multi files with a hello field.

await request('https://example.com/upload', {
  method: 'POST',
  files: [
    __filename,
    fs.createReadStream(__filename),
    Buffer.from('mock file content'),
  ],
  data: {
    hello: 'hello urllib with multi files',
  },
});

Custom file field name with uploadfile.

await request('https://example.com/upload', {
  method: 'POST',
  files: {
    uploadfile: __filename,
  },
});

Options: `options.stream`

Uploads a file with formstream:

import formstream from 'formstream';

const form = formstream();
form.file('file', __filename);
form.field('hello', '你好urllib');

await request('https://example.com/upload', {
  method: 'POST',
  headers: form.headers(),
  stream: form,
});

Response Object

Response is normal object, it contains:

status or statusCode: response status code.
- -1 meaning some network error like ENOTFOUND
- -2 meaning ConnectionTimeoutError
headers: response http headers, default is {}
size: response size
aborted: response was aborted or not
rt: total request and response time in ms.
timing: timing object if timing enable.
remoteAddress: http server ip address
remotePort: http server ip port
socketHandledRequests: socket already handled request count
socketHandledResponses: socket already handled response count

Response: `res.aborted`

If the underlaying connection was terminated before response.end() was called, res.aborted should be true.

import { createServer } from 'http';

createServer((req, res) => {
  req.resume();
  req.on('end', () => {
    res.write('foo haha\n');
    setTimeout(() => {
      res.write('foo haha 2');
      setTimeout(() => {
        res.socket.end();
      }, 300);
    }, 200);
    return;
  });
}).listen(2022);

const { data, res } = await request('http://127.0.0.1:2022/socket.end');
assert.equal(data.toString(), 'foo haha\nfoo haha 2');
assert(res.aborted);

TODO

❎ Support Proxy
✅ Upload file like form upload
✅ Auto redirect handle
✅ https & self-signed certificate
✅ Connection timeout & Response timeout
✅ Support Digest access authentication

Contributors

_fengmk2	_dead-horse	_xingrz	_popomore	_JacksonTian	_ibigbug
_{greenkeeperio-bot}	_atian25	_paambaati	_denghongcai	_XadillaX	_alsotang
_leoner	_hyj1991	_isayme	_cyjake	_whxaxes	_chadxz
_danielwpz	_danielsss	_Jeff-Tian	_jedahan	_nick-ng	_rishavsharan
_willizm	_davidkhala	_aleafs	_Amunu	_{azure-pipelines[bot]}	_changzhiwin
_yuzhigang33	_fishbar	_gxcsoccer	_killagu	_mars-coder	_rockdai
_dickeylth	_aladdin-add