JSPM

json-to-jsonl

1.1.0
  • ESM via JSPM
  • ES Module Entrypoint
  • Export Map
  • Keywords
  • License
  • Repository URL
  • TypeScript Types
  • README
  • Created
  • Published
  • Downloads 620
  • Score
    100M100P100Q93773F
  • License MIT

Convert an array in a .json file to a newline-delimited .jsonl file.

Package Exports

  • json-to-jsonl

This package does not declare an exports field, so the exports above have been automatically detected and optimized by JSPM instead. If any package subpath is missing, it is recommended to post an issue to the original package (json-to-jsonl) to support the "exports" field. If that is not possible, create a JSPM override to customize the exports field for this package.

Readme

JSON to JSON Lines

You have a JSON file containing a very large array of objects you want to analyse. json-to-jsonl is a lightweight package to re-write the array to a JSON Lines file - one row per object.

JSON Lines Format

JSON Lines is a convenient format for storing structured data that may be processed one record at a time. It works well with unix-style text processing tools and shell pipelines. Here's how easy it is to create a document database (in MongoDB) from a JSON Lines file:

mongoimport --db my-db-name --collection my-collection-name --file /path/to/my-file.jsonl

Installation

$ npm install --save json-to-jsonl

Examples

// my-file-1.json
[
  { "a" : 1 },
  { "a" : 2 },
  { "a" : 3 }
]
// my-file-2.json
{
  "name" : "Some JSON Object",
  "longList" : [
    "one",
    "two",
    "three",
    "four"
  ]
}
// convert.js
const jsonTojsonl = require('json-to-jsonl')

try {
  // The array is top-level in my-file-1.json so don't have to specify getArray func:
  const response1 = jsonTojsonl('my-file-1.json')
  // { lines: 3, file: 'my-file-1.jsonl' }

  // The array is a value in the my-file-2.json object so we have to specify a getArray func:
  const response2 = jsonTojsonl('my-file-2.json', x => x.longList)
  // { lines: 4, file: 'my-file-2.jsonl' }
}
catch (e) {
  ...
}

Docs

jsonTojsonl(jsonFilename, getArray=function(x) {return x})

Writes a new file with same name (but .json extension replaced with .jsonl). The optional argument getArray allows you to define where the JSON array is (if it isn't top-level). Returns an object with lines and file properties (number of lines written and name of new file).

Memory Limit

If your JSON file is very big you might encounter a FATAL ERROR due to JavaScript heap out of memory. It might be necessary to increase the memory usage limit when running your script, e.g. to 4GB:

$ node --max-old-space-size=4096 convert.js