Package Exports

@wholebuzz/fs
@wholebuzz/fs/lib/fs
@wholebuzz/fs/lib/gcp
@wholebuzz/fs/lib/json

This package does not declare an exports field, so the exports above have been automatically detected and optimized by JSPM instead. If any package subpath is missing, it is recommended to post an issue to the original package (@wholebuzz/fs) to support the "exports" field. If that is not possible, create a JSPM override to customize the exports field for this package.

Readme

@wholebuzz/fs

FileSystem with atomic primitives enabling multiple readers and writers.

LocalFileSystem employs content hashing to approximate GCS Object Versioning.
GoogleCloudFileSystem provides consistent parallel access paterns.
S3FileSystem provides basic file system primitives.

Example

import { AnyFileSystem, GoogleCloudFileSystem, LocalFileSystem, S3FileSystem } from '@wholebuzz/fs'
import { readJSON, writeJSON } from '@whilebuzz/fs/lib/json'

const fs = new AnyFileSystem([
  { urlPrefix: 'gs://', fs: new GoogleCloudFileSystem() },
  { urlPrefix: 's3://', fs: new S3FileSystem() },
  { urlPrefix: '', fs: new LocalFileSystem() },
])

await writeJSON(fs, 's3://bucket/file', { foo: 'bar' })
const foobar = await readJSON(fs, 's3://bucket/file')

CLI

yarn build
node lib/cli.js ls .
node lib/cli.js --help

API Reference

Modules

Class: FileSystem

fs.FileSystem

File system interface for atomic primitives enabling multiple readers and writers.

Hierarchy

FileSystem

↳ AnyFileSystem

↳ LocalFileSystem

↳ GoogleCloudFileSystem

↳ S3FileSystem

Constructors

constructor

Methods

Constructors

constructor

+ new FileSystem(): FileSystem

Returns: FileSystem

Methods

appendToFile

▸ Abstract appendToFile(urlText: string, writeCallback: (stream: WritableStreamTree) => Promise<boolean>, createCallback?: (stream: WritableStreamTree) => Promise<boolean>, createOptions?: CreateOptions, appendOptions?: AppendOptions): Promise<null | FileStatus>

Appends to the file, safely. Either writeCallback or createCallback is called. For simple appends, the same paramter can be supplied for both writeCallback and createCallback.

Parameters

Name	Type	Description
`urlText`	string	The URL of the file to append to.
`writeCallback`	(`stream`: WritableStreamTree) => Promise<boolean>	Stream callback for appending to the file.
`createCallback?`	(`stream`: WritableStreamTree) => Promise<boolean>	Stream callback for initializing the file, if necessary.
`createOptions?`	CreateOptions	Initial metadata for initializing the file, if necessary.
`appendOptions?`	AppendOptions	-

Returns: Promise<null | FileStatus>

Defined in: fs.ts:158

copyFile

▸ Abstract copyFile(sourceUrlText: string, destUrlText: string): Promise<boolean>

Copies the file.

Parameters

Name	Type	Description
`sourceUrlText`	string	The URL of the source file to copy.
`destUrlText`	string	The destination URL to copy the file to.

Returns: Promise<boolean>

Defined in: fs.ts:133

createFile

▸ Abstract createFile(urlText: string, createCallback?: (stream: WritableStreamTree) => Promise<boolean>, createOptions?: CreateOptions): Promise<boolean>

Creates file, failing if the file already exists.

Parameters

Name	Type	Description
`urlText`	string	The URL of the file to create.
`createCallback?`	(`stream`: WritableStreamTree) => Promise<boolean>	Stream callback for initializing the file.
`createOptions?`	CreateOptions	Initial metadata.

Returns: Promise<boolean>

Defined in: fs.ts:110

ensureDirectory

▸ Abstract ensureDirectory(urlText: string, mask?: number): Promise<boolean>

Ensures the directory exists

Parameters

Name	Type	Description
`urlText`	string	The URL of the directory.
`mask?`	number	-

Returns: Promise<boolean>

Defined in: fs.ts:72

fileExists

▸ Abstract fileExists(urlText: string): Promise<boolean>

Returns true if the file exists.

Parameters

Name	Type	Description
`urlText`	string	The URL of the file to check whether exists.

Returns: Promise<boolean>

Defined in: fs.ts:78

getFileStatus

▸ Abstract getFileStatus(urlText: string, getVersion?: boolean): Promise<FileStatus>

Determines the file status. The file version is used to implement atomic mutations.

Parameters

Name	Type	Description
`urlText`	string	The URL of the file to retrieve the status for.
`getVersion?`	boolean	-

Returns: Promise<FileStatus>

Defined in: fs.ts:84

openReadableFile

▸ Abstract openReadableFile(url: string, version?: string | number): Promise<ReadableStreamTree>

Opens a file for reading.

optional version Fails if version doesn't match for GCS URLs.

Parameters

Name	Type	Description
`url`	string	The URL of the file to read from.
`version?`	string \| number	-

Returns: Promise<ReadableStreamTree>

Defined in: fs.ts:91

openWritableFile

▸ Abstract openWritableFile(url: string, version?: string | number, options?: CreateOptions): Promise<WritableStreamTree>

Opens a file for writing.

optional version Fails if version doesn't match for GCS URLs.

Parameters

Name	Type	Description
`url`	string	The URL of the file to write to.
`version?`	string \| number	-
`options?`	CreateOptions	-

Returns: Promise<WritableStreamTree>

Defined in: fs.ts:98

queueRemoveFile

▸ Abstract queueRemoveFile(urlText: string): Promise<boolean>

Queues deletion, e.g. after DaysSinceCustomTime.

Parameters

Name	Type	Description
`urlText`	string	The URL of the file to remove.

Returns: Promise<boolean>

Defined in: fs.ts:126

readDirectory

▸ Abstract readDirectory(urlText: string, prefix?: string): Promise<string[]>

Returns the URLs of the files in a directory.

Parameters

Name	Type	Description
`urlText`	string	The URL of the directory to list files in.
`prefix?`	string	-

Returns: Promise<string[]>

Defined in: fs.ts:66

removeFile

▸ Abstract removeFile(urlText: string): Promise<boolean>

Deletes the file.

Parameters

Name	Type	Description
`urlText`	string	The URL of the file to remove.

Returns: Promise<boolean>

Defined in: fs.ts:120

replaceFile

▸ Abstract replaceFile(urlText: string, writeCallback: (stream: WritableStreamTree) => Promise<boolean>, createOptions?: CreateOptions, version?: string | number): Promise<boolean>

Replaces the file, failing if the file version doesn't match.

Parameters

Name	Type	Description
`urlText`	string	The URL of the file to replace.
`writeCallback`	(`stream`: WritableStreamTree) => Promise<boolean>	Stream callback for replacing the file.
`createOptions?`	CreateOptions	Initial metadata for replaced file.
`version?`	string \| number	The version of the file to replace.

Returns: Promise<boolean>

Defined in: fs.ts:142

Module: json

Functions

mapLines

▸ mapLines<X>(stream: ReadableStreamTree, map: (x: string) => X): Promise<X[]>

Maps lines from [[stream]]. Used to implement readLines.

Type parameters

Name
`X`

Parameters

Name	Type	Description
`stream`	ReadableStreamTree	The stream to read lines from.
`map`	(`x`: string) => X	-

Returns: Promise<X[]>

Defined in: json.ts:151

mapLinesWithHeader

▸ mapLinesWithHeader<X, H>(stream: ReadableStreamTree, map: (x: string) => X, header?: (y: string) => H, ret?: X[]): Promise<[H | *undefined*, X[]]>

Parses lines (with header) from [[stream]]. Used to implement readLinesWithHeader.

Type parameters

Name
`X`
`H`

Parameters

Name	Type	Default value	Description
`stream`	ReadableStreamTree	-	The stream to read lines from.
`map`	(`x`: string) => X	-	-
`header?`	(`y`: string) => H	-	-
`ret`	X[]	[]	-

Returns: Promise<[H | *undefined*, X[]]>

Defined in: json.ts:161

parseJSON

▸ parseJSON(stream: ReadableStreamTree): Promise<unknown>

Parses JSON object from [[stream]]. Used to implement readJSON.

Parameters

Name	Type	Description
`stream`	ReadableStreamTree	The stream to read a JSON object from.

Returns: Promise<unknown>

Defined in: json.ts:189

parseLines

▸ parseLines(stream: ReadableStreamTree, callback: (x: string) => void): Promise<void>

Parameters

Name	Type
`stream`	ReadableStreamTree
`callback`	(`x`: string) => void

Returns: Promise<void>

Defined in: json.ts:132

readJSON

▸ readJSON(fileSystem: FileSystem, url: string): Promise<unknown>

Reads a serialized JSON object or array from a file.

Parameters

Name	Type	Description
`fileSystem`	FileSystem	-
`url`	string	The URL of the file to parse a JSON object or array from.

Returns: Promise<unknown>

Defined in: json.ts:43

readJSONHashed

▸ readJSONHashed(fileSystem: FileSystem, url: string): Promise<[*unknown*, null | *string*]>

Reads a serialized JSON object from a file, and also hashes the file.

Parameters

Name	Type	Description
`fileSystem`	FileSystem	-
`url`	string	The URL of the file to parse a JSON object from.

Returns: Promise<[*unknown*, null | *string*]>

Defined in: json.ts:51

readLines

▸ readLines<X>(fileSystem: FileSystem, url: string, map: (x: string) => X): Promise<X[]>

Reads every line from a file.

Type parameters

Name
`X`

Parameters

Name	Type	Description
`fileSystem`	FileSystem	-
`url`	string	The URL of the file to read lines from.
`map`	(`x`: string) => X	Callback called for each line.

Returns: Promise<X[]>

Defined in: json.ts:15

readLinesWithHeader

▸ readLinesWithHeader<X, H>(fileSystem: FileSystem, url: string, map: (x: string) => X, header?: (x: string) => H, ret?: X[]): Promise<[H | *undefined*, X[]]>

Reads every line from a file, treating the first line as a header.

Type parameters

Name
`X`
`H`

Parameters

Name	Type	Default value	Description
`fileSystem`	FileSystem	-	-
`url`	string	-	The URL of the file to read lines from.
`map`	(`x`: string) => X	-	Callback called for every line succeeding the header.
`header?`	(`x`: string) => H	-	Callback called for the first line.
`ret`	X[]	[]	-

Returns: Promise<[H | *undefined*, X[]]>

Defined in: json.ts:29

serializeJSON

▸ serializeJSON(stream: WritableStreamTree, obj: object | any[]): Promise<boolean>

Serializes JSON object to [[stream]]. Used to implement writeJSON.

Parameters

Name	Type	Description
`stream`	WritableStreamTree	The stream to write a JSON object to.
`obj`	object \| any[]	-

Returns: Promise<boolean>

Defined in: json.ts:210

writeContent

▸ writeContent(fileSystem: FileSystem, url: string, value: string): Promise<void>

Writes the string to a file.

Parameters

Name	Type	Description
`fileSystem`	FileSystem	-
`url`	string	The URL of the file to serialize the string to.
`value`	string	The string to serialize.

Returns: Promise<void>

Defined in: json.ts:62

writeJSON

▸ writeJSON(fileSystem: FileSystem, url: string, value: object | any[]): Promise<boolean>

Serializes object or array to a JSON file.

Parameters

Name	Type	Description
`fileSystem`	FileSystem	-
`url`	string	The URL of the file to serialize a JSON object or array to.
`value`	object \| any[]	The object or array to serialize.

Returns: Promise<boolean>

Defined in: json.ts:79

writeJSONLines

▸ writeJSONLines(fileSystem: FileSystem, url: string, obj: object[]): Promise<void>

Serializes array to a JSON Lines file.

Parameters

Name	Type	Description
`fileSystem`	FileSystem	-
`url`	string	The URL of the file to serialize a JSON array to.
`obj`	object[]	-

Returns: Promise<void>

Defined in: json.ts:88

writeShardedJSONLines

▸ writeShardedJSONLines(fileSystem: FileSystem, url: string, obj: object[], shards: number, shardFunction?: (x: object, modulus: number) => number): Promise<void>

Parameters

Name	Type
`fileSystem`	FileSystem
`url`	string
`obj`	object[]
`shards`	number
`shardFunction`	(`x`: object, `modulus`: number) => number

Returns: Promise<void>

Defined in: json.ts:108

@wholebuzz/fs

Package Exports

Readme

@wholebuzz/fs

Example

CLI

API Reference

Table of contents

Modules

Class: FileSystem

Hierarchy

Table of contents

Constructors

Methods

Constructors

constructor

Methods

appendToFile

Parameters

copyFile

Parameters

createFile

Parameters

ensureDirectory

Parameters

fileExists

Parameters

getFileStatus

Parameters

openReadableFile

Parameters

openWritableFile

Parameters

queueRemoveFile

Parameters

readDirectory

Parameters

removeFile

Parameters

replaceFile

Parameters

Module: json

Table of contents

Functions

Functions

mapLines

Type parameters

Parameters

mapLinesWithHeader

Type parameters

Parameters

parseJSON

Parameters

parseLines

Parameters

readJSON

Parameters

readJSONHashed

Parameters

readLines

Type parameters

Parameters

readLinesWithHeader

Type parameters

Parameters

serializeJSON

Parameters

writeContent

Parameters

writeJSON

Parameters

writeJSONLines

Parameters

writeShardedJSONLines

Parameters