Package Exports

@wholebuzz/fs
@wholebuzz/fs/lib/fs
@wholebuzz/fs/lib/gcp
@wholebuzz/fs/lib/json

This package does not declare an exports field, so the exports above have been automatically detected and optimized by JSPM instead. If any package subpath is missing, it is recommended to post an issue to the original package (@wholebuzz/fs) to support the "exports" field. If that is not possible, create a JSPM override to customize the exports field for this package.

Readme

@wholebuzz/fs

File system abstraction with implementations for GCP GCS, AWS S3, Azure, SMB, HTTP, and Local file systems. Provides atomic primitives enabling multiple readers and writers.

LocalFileSystem employs content hashing to approximate GCS Object Versioning.
GoogleCloudFileSystem provides consistent parallel access paterns.
S3FileSystem provides basic file system primitives.
SMBFileSystem provides basic file system primitives.
HTTPFileSystem provides a basic HTTP file system.

Dependencies

The FileSystem implementations require peer dependencies:

AzureBlobStorageFileSystem: @azure/storage-blob and @azure/identity
AzureFileShareFileSystem: @azure/storage-file-share
GoogleCloudFileSystem: @google-cloud/storage
HTTPFileSystem: axios
LocalFileSystem: fs-ext
S3FileSystem: aws-sdk and s3-stream-upload
SMBFileSystem: @marsaud/smb2

History

The project started to support @wholebuzz/archive, a terabyte-scale archive for GCS. The focus has since expanded to include powering dbcp with a collection of file system implementations under a common interface. The atomic primitives are only available for Google Cloud Storage and local.

Example

import {
  AnyFileSystem,
  GoogleCloudFileSystem,
  HTTPFileSystem,
  LocalFileSystem,
  S3FileSystem
} from '@wholebuzz/fs'
import { readJSON, writeJSON } from '@whilebuzz/fs/lib/json'

const httpFileSystem = new HTTPFileSystem()
const fs = new AnyFileSystem([
  { urlPrefix: 'gs://', fs: new GoogleCloudFileSystem() },
  { urlPrefix: 's3://', fs: new S3FileSystem() },
  { urlPrefix: 'http://', fs: httpFileSystem },
  { urlPrefix: 'https://', fs: httpFileSystem },
  { urlPrefix: '', fs: new LocalFileSystem() },
])

await writeJSON(fs, 's3://bucket/file', { foo: 'bar' })
const foobar = await readJSON(fs, 's3://bucket/file')

CLI

node lib/cli.js ls .
node lib/cli.js --help

API Reference

Modules

Methods

appendToFile

▸ Abstract appendToFile(urlText: string, writeCallback: (stream: WritableStreamTree) => Promise<boolean>, createCallback?: (stream: WritableStreamTree) => Promise<boolean>, createOptions?: CreateOptions, appendOptions?: AppendOptions): Promise<null | FileStatus>

Appends to the file, safely. Either writeCallback or createCallback is called. For simple appends, the same paramter can be supplied for both writeCallback and createCallback.

Parameters

Name	Type	Description
`urlText`	string	The URL of the file to append to.
`writeCallback`	(`stream`: WritableStreamTree) => Promise<boolean>	Stream callback for appending to the file.
`createCallback?`	(`stream`: WritableStreamTree) => Promise<boolean>	Stream callback for initializing the file, if necessary.
`createOptions?`	CreateOptions	Initial metadata for initializing the file, if necessary.
`appendOptions?`	AppendOptions	-

Returns: Promise<null | FileStatus>

Defined in: fs.ts:158

copyFile

▸ Abstract copyFile(sourceUrlText: string, destUrlText: string): Promise<boolean>

Copies the file.

Parameters

Name	Type	Description
`sourceUrlText`	string	The URL of the source file to copy.
`destUrlText`	string	The destination URL to copy the file to.

Returns: Promise<boolean>

Defined in: fs.ts:133

createFile

▸ Abstract createFile(urlText: string, createCallback?: (stream: WritableStreamTree) => Promise<boolean>, createOptions?: CreateOptions): Promise<boolean>

Creates file, failing if the file already exists.

Parameters

Name	Type	Description
`urlText`	string	The URL of the file to create.
`createCallback?`	(`stream`: WritableStreamTree) => Promise<boolean>	Stream callback for initializing the file.
`createOptions?`	CreateOptions	Initial metadata.

Returns: Promise<boolean>

Defined in: fs.ts:110

ensureDirectory

▸ Abstract ensureDirectory(urlText: string, mask?: number): Promise<boolean>

Ensures the directory exists

Parameters

Name	Type	Description
`urlText`	string	The URL of the directory.
`mask?`	number	-

Returns: Promise<boolean>

Defined in: fs.ts:72

fileExists

▸ Abstract fileExists(urlText: string): Promise<boolean>

Returns true if the file exists.

Parameters

Name	Type	Description
`urlText`	string	The URL of the file to check whether exists.

Returns: Promise<boolean>

Defined in: fs.ts:78

getFileStatus

▸ Abstract getFileStatus(urlText: string, getVersion?: boolean): Promise<FileStatus>

Determines the file status. The file version is used to implement atomic mutations.

Parameters

Name	Type	Description
`urlText`	string	The URL of the file to retrieve the status for.
`getVersion?`	boolean	-

Returns: Promise<FileStatus>

Defined in: fs.ts:84

openReadableFile

▸ Abstract openReadableFile(url: string, version?: string | number): Promise<ReadableStreamTree>

Opens a file for reading.

optional version Fails if version doesn't match for GCS URLs.

Parameters

Name	Type	Description
`url`	string	The URL of the file to read from.
`version?`	string \| number	-

Returns: Promise<ReadableStreamTree>

Defined in: fs.ts:91

openWritableFile

▸ Abstract openWritableFile(url: string, version?: string | number, options?: CreateOptions): Promise<WritableStreamTree>

Opens a file for writing.

optional version Fails if version doesn't match for GCS URLs.

Parameters

Name	Type	Description
`url`	string	The URL of the file to write to.
`version?`	string \| number	-
`options?`	CreateOptions	-

Returns: Promise<WritableStreamTree>

Defined in: fs.ts:98

queueRemoveFile

▸ Abstract queueRemoveFile(urlText: string): Promise<boolean>

Queues deletion, e.g. after DaysSinceCustomTime.

Parameters

Name	Type	Description
`urlText`	string	The URL of the file to remove.

Returns: Promise<boolean>

Defined in: fs.ts:126

readDirectory

▸ Abstract readDirectory(urlText: string, prefix?: string): Promise<string[]>

Returns the URLs of the files in a directory.

Parameters

Name	Type	Description
`urlText`	string	The URL of the directory to list files in.
`prefix?`	string	-

Returns: Promise<string[]>

Defined in: fs.ts:66

removeFile

▸ Abstract removeFile(urlText: string): Promise<boolean>

Deletes the file.

Parameters

Name	Type	Description
`urlText`	string	The URL of the file to remove.

Returns: Promise<boolean>

Defined in: fs.ts:120

replaceFile

▸ Abstract replaceFile(urlText: string, writeCallback: (stream: WritableStreamTree) => Promise<boolean>, createOptions?: CreateOptions, version?: string | number): Promise<boolean>

Replaces the file, failing if the file version doesn't match.

Parameters

Name	Type	Description
`urlText`	string	The URL of the file to replace.
`writeCallback`	(`stream`: WritableStreamTree) => Promise<boolean>	Stream callback for replacing the file.
`createOptions?`	CreateOptions	Initial metadata for replaced file.
`version?`	string \| number	The version of the file to replace.

Returns: Promise<boolean>

Defined in: fs.ts:142 @wholebuzz/fs / Exports / json

Module: json

Variables

JSONStream

Functions

Variables

JSONStream

• Const JSONStream: any

Defined in: json.ts:19

Functions

mapLines

▸ mapLines<X>(stream: ReadableStreamTree, map: (x: string) => X): Promise<X[]>

Maps lines from [[stream]]. Used to implement readLines.

Type parameters

Name
`X`

Parameters

Name	Type	Description
`stream`	ReadableStreamTree	The stream to read lines from.
`map`	(`x`: string) => X	-

Returns: Promise<X[]>

Defined in: json.ts:148

mapLinesWithHeader

▸ mapLinesWithHeader<X, H>(stream: ReadableStreamTree, map: (x: string) => X, header?: (y: string) => H, ret?: X[]): Promise<[H | *undefined*, X[]]>

Parses lines (with header) from [[stream]]. Used to implement readLinesWithHeader.

Type parameters

Name
`X`
`H`

Parameters

Name	Type	Default value	Description
`stream`	ReadableStreamTree	-	The stream to read lines from.
`map`	(`x`: string) => X	-	-
`header?`	(`y`: string) => H	-	-
`ret`	X[]	[]	-

Returns: Promise<[H | *undefined*, X[]]>

Defined in: json.ts:158

parseJSON

▸ parseJSON(stream: ReadableStreamTree): Promise<unknown>

Parses JSON object from [[stream]]. Used to implement readJSON.

Parameters

Name	Type	Description
`stream`	ReadableStreamTree	The stream to read a JSON object from.

Returns: Promise<unknown>

Defined in: json.ts:186

parseJSONLines

▸ parseJSONLines(stream: ReadableStreamTree): Promise<unknown[]>

Parses JSON object from [[stream]]. Used to implement readJSON.

Parameters

Name	Type	Description
`stream`	ReadableStreamTree	The stream to read a JSON object from.

Returns: Promise<unknown[]>

Defined in: json.ts:203

parseLines

▸ parseLines(stream: ReadableStreamTree, callback: (x: string) => void): Promise<void>

Parameters

Name	Type
`stream`	ReadableStreamTree
`callback`	(`x`: string) => void

Returns: Promise<void>

Defined in: json.ts:132

pipeFilter

▸ pipeFilter(stream: ReadableStreamTree, filter: (x: any) => any): ReadableStreamTree

Create filter stream.

Parameters

Name	Type
`stream`	ReadableStreamTree
`filter`	(`x`: any) => any

Returns: ReadableStreamTree

Defined in: json.ts:280

pipeFromFilter

▸ pipeFromFilter(stream: WritableStreamTree, filter: (x: any) => any): WritableStreamTree

Create filter stream.

Parameters

Name	Type
`stream`	WritableStreamTree
`filter`	(`x`: any) => any

Returns: WritableStreamTree

Defined in: json.ts:292

pipeJSONFormatter

▸ pipeJSONFormatter(stream: WritableStreamTree, isArray: boolean): WritableStreamTree

Create JSON formatter stream.

Parameters

Name	Type	Description
`stream`	WritableStreamTree	-
`isArray`	boolean	Accept array objects or property tuples.

Returns: WritableStreamTree

Defined in: json.ts:258

pipeJSONLinesFormatter

▸ pipeJSONLinesFormatter(stream: WritableStreamTree): WritableStreamTree

Create JSON-lines formatter stream.

Parameters

Name	Type
`stream`	WritableStreamTree

Returns: WritableStreamTree

Defined in: json.ts:273

pipeJSONLinesParser

▸ pipeJSONLinesParser(stream: ReadableStreamTree): ReadableStreamTree

Create JSON parser stream.

Parameters

Name	Type
`stream`	ReadableStreamTree

Returns: ReadableStreamTree

Defined in: json.ts:250

pipeJSONParser

▸ pipeJSONParser(stream: ReadableStreamTree, isArray: boolean): ReadableStreamTree

Create JSON parser stream.

Parameters

Name	Type
`stream`	ReadableStreamTree
`isArray`	boolean

Returns: ReadableStreamTree

Defined in: json.ts:241

readJSON

▸ readJSON(fileSystem: FileSystem, url: string): Promise<unknown>

Reads a serialized JSON object or array from a file.

Parameters

Name	Type	Description
`fileSystem`	FileSystem	-
`url`	string	The URL of the file to parse a JSON object or array from.

Returns: Promise<unknown>

Defined in: json.ts:54

readJSONHashed

▸ readJSONHashed(fileSystem: FileSystem, url: string): Promise<[*unknown*, null | *string*]>

Reads a serialized JSON object from a file, and also hashes the file.

Parameters

Name	Type	Description
`fileSystem`	FileSystem	-
`url`	string	The URL of the file to parse a JSON object from.

Returns: Promise<[*unknown*, null | *string*]>

Defined in: json.ts:62

readJSONLines

▸ readJSONLines(fileSystem: FileSystem, url: string): Promise<unknown[]>

Reads a serialized JSON-lines array from a file.

Parameters

Name	Type	Description
`fileSystem`	FileSystem	-
`url`	string	The URL of the file to parse a JSON object or array from.

Returns: Promise<unknown[]>

Defined in: json.ts:72

readLines

▸ readLines<X>(fileSystem: FileSystem, url: string, map: (x: string) => X): Promise<X[]>

Reads every line from a file.

Type parameters

Name
`X`

Parameters

Name	Type	Description
`fileSystem`	FileSystem	-
`url`	string	The URL of the file to read lines from.
`map`	(`x`: string) => X	Callback called for each line.

Returns: Promise<X[]>

Defined in: json.ts:26

readLinesWithHeader

▸ readLinesWithHeader<X, H>(fileSystem: FileSystem, url: string, map: (x: string) => X, header?: (x: string) => H, ret?: X[]): Promise<[H | *undefined*, X[]]>

Reads every line from a file, treating the first line as a header.

Type parameters

Name
`X`
`H`

Parameters

Name	Type	Default value	Description
`fileSystem`	FileSystem	-	-
`url`	string	-	The URL of the file to read lines from.
`map`	(`x`: string) => X	-	Callback called for every line succeeding the header.
`header?`	(`x`: string) => H	-	Callback called for the first line.
`ret`	X[]	[]	-

Returns: Promise<[H | *undefined*, X[]]>

Defined in: json.ts:40

serializeJSON

▸ serializeJSON(stream: WritableStreamTree, obj: object | any[]): Promise<boolean>

Serializes JSON object to [[stream]]. Used to implement writeJSON.

Parameters

Name	Type	Description
`stream`	WritableStreamTree	The stream to write a JSON object to.
`obj`	object \| any[]	-

Returns: Promise<boolean>

Defined in: json.ts:219

serializeJSONLines

▸ serializeJSONLines(stream: WritableStreamTree, obj: any[]): Promise<boolean>

Serializes JSON object to [[stream]]. Used to implement writeJSONLines.

Parameters

Name	Type	Description
`stream`	WritableStreamTree	The stream to write a JSON object to.
`obj`	any[]	-

Returns: Promise<boolean>

Defined in: json.ts:234

writeContent

▸ writeContent(fileSystem: FileSystem, url: string, value: string): Promise<void>

Writes the string to a file.

Parameters

Name	Type	Description
`fileSystem`	FileSystem	-
`url`	string	The URL of the file to serialize the string to.
`value`	string	The string to serialize.

Returns: Promise<void>

Defined in: json.ts:81

writeJSON

▸ writeJSON(fileSystem: FileSystem, url: string, value: object | any[]): Promise<boolean>

Serializes object or array to a JSON file.

Parameters

Name	Type	Description
`fileSystem`	FileSystem	-
`url`	string	The URL of the file to serialize a JSON object or array to.
`value`	object \| any[]	The object or array to serialize.

Returns: Promise<boolean>

Defined in: json.ts:95

writeJSONLines

▸ writeJSONLines(fileSystem: FileSystem, url: string, obj: object[]): Promise<boolean>

Serializes array to a JSON Lines file.

Parameters

Name	Type	Description
`fileSystem`	FileSystem	-
`url`	string	The URL of the file to serialize a JSON array to.
`obj`	object[]	-

Returns: Promise<boolean>

Defined in: json.ts:104

writeShardedJSONLines

▸ writeShardedJSONLines(fileSystem: FileSystem, url: string, obj: object[], shards: number, shardFunction?: (x: object, modulus: number) => number): Promise<void>

Parameters

Name	Type
`fileSystem`	FileSystem
`url`	string
`obj`	object[]
`shards`	number
`shardFunction`	(`x`: object, `modulus`: number) => number

Returns: Promise<void>

Defined in: json.ts:108

@wholebuzz/fs

Package Exports

Readme

@wholebuzz/fs

Dependencies

History

Example

CLI

API Reference

Modules

Methods

appendToFile

Parameters

copyFile

Parameters

createFile

Parameters

ensureDirectory

Parameters

fileExists

Parameters

getFileStatus

Parameters

openReadableFile

Parameters

openWritableFile

Parameters

queueRemoveFile

Parameters

readDirectory

Parameters

removeFile

Parameters

replaceFile

Parameters

Module: json

Table of contents

Variables

Functions

Variables

JSONStream

Functions

mapLines

Type parameters

Parameters

mapLinesWithHeader

Type parameters

Parameters

parseJSON

Parameters

parseJSONLines

Parameters

parseLines

Parameters

pipeFilter

Parameters

pipeFromFilter

Parameters

pipeJSONFormatter

Parameters

pipeJSONLinesFormatter

Parameters

pipeJSONLinesParser

Parameters

pipeJSONParser

Parameters

readJSON

Parameters

readJSONHashed

Parameters

readJSONLines

Parameters

readLines

Type parameters

Parameters

readLinesWithHeader

Type parameters

Parameters

serializeJSON

Parameters