@wholebuzz/fs

File system abstraction with implementations for GCP GCS, AWS S3, Azure, SMB, HTTP, and Local file systems. Provides atomic primitives enabling multiple readers and writers.

• LocalFileSystem employs content hashing to approximate GCS Object Versioning.
• GoogleCloudFileSystem provides consistent parallel access paterns.
• S3FileSystem provides basic file system primitives.
• SMBFileSystem provides basic file system primitives.
• HTTPFileSystem provides a basic HTTP file system.

Dependencies

The FileSystem implementations require peer dependencies:

• AzureBlobStorageFileSystem: @azure/storage-blob and @azure/identity
• AzureFileShareFileSystem: @azure/storage-file-share
• GoogleCloudFileSystem: @google-cloud/storage
• HTTPFileSystem: axios
• LocalFileSystem: fs-ext
• S3FileSystem: aws-sdk and s3-stream-upload
• SMBFileSystem: @marsaud/smb2

History

The project started to support @wholebuzz/archive, a terabyte-scale archive for GCS. The focus has since expanded to include powering dbcp with a collection of file system implementations under a common interface. The atomic primitives are only available for Google Cloud Storage and local.

Example

import {
  AnyFileSystem,
  GoogleCloudFileSystem,
  HTTPFileSystem,
  LocalFileSystem,
  S3FileSystem
} from '@wholebuzz/fs'
import { readJSON, writeJSON } from '@whilebuzz/fs/lib/json'

const httpFileSystem = new HTTPFileSystem()
const fs = new AnyFileSystem([
  { urlPrefix: 'gs://', fs: new GoogleCloudFileSystem() },
  { urlPrefix: 's3://', fs: new S3FileSystem() },
  { urlPrefix: 'http://', fs: httpFileSystem },
  { urlPrefix: 'https://', fs: httpFileSystem },
  { urlPrefix: '', fs: new LocalFileSystem() },
])

await writeJSON(fs, 's3://bucket/file', { foo: 'bar' })
const foobar = await readJSON(fs, 's3://bucket/file')

CLI

node lib/cli.js ls .
node lib/cli.js --help

API Reference

Modules

• azure-blob
• cli
• fs
• gcp
• http
• json
• lines
• local
• parquet
• s3
• smb
• stream
• tfrecord
• util

Methods

• appendToFile
• copyFile
• createFile
• ensureDirectory
• fileExists
• getFileStatus
• moveFile
• openReadableFile
• openWritableFile
• queueRemoveFile
• readDirectory
• removeDirectory
• removeFile
• replaceFile

Constructors

constructor

+ new FileSystem(): FileSystem

Returns: FileSystem

Methods

appendToFile

▸ Abstract appendToFile(urlText: string, writeCallback: (stream: WritableStreamTree) => Promise<boolean>, createCallback?: (stream: WritableStreamTree) => Promise<boolean>, createOptions?: CreateOptions, appendOptions?: AppendOptions): Promise<null | FileStatus>

Appends to the file, safely. Either writeCallback or createCallback is called. For simple appends, the same paramter can be supplied for both writeCallback and createCallback.

Parameters

Returns: Promise<null | FileStatus>

Defined in: src/fs.ts:192

copyFile

▸ Abstract copyFile(sourceUrlText: string, destUrlText: string): Promise<boolean>

Copies the file.

Parameters

Returns: Promise<boolean>

Defined in: src/fs.ts:161

createFile

▸ Abstract createFile(urlText: string, createCallback?: (stream: WritableStreamTree) => Promise<boolean>, options?: CreateOptions): Promise<boolean>

Creates file, failing if the file already exists.

Parameters

Returns: Promise<boolean>

Defined in: src/fs.ts:138

ensureDirectory

▸ Abstract ensureDirectory(urlText: string, options?: EnsureDirectoryOptions): Promise<boolean>

Ensures the directory exists

Parameters

Returns: Promise<boolean>

Defined in: src/fs.ts:92

fileExists

▸ Abstract fileExists(urlText: string): Promise<boolean>

Returns true if the file exists.

Parameters

Returns: Promise<boolean>

Defined in: src/fs.ts:104

getFileStatus

▸ Abstract getFileStatus(urlText: string, options?: GetFileStatusOptions): Promise<FileStatus>

Determines the file status. The file version is used to implement atomic mutations.

Parameters

Returns: Promise<FileStatus>

Defined in: src/fs.ts:110

moveFile

▸ Abstract moveFile(sourceUrlText: string, destUrlText: string): Promise<boolean>

Moves the file.

Parameters

Returns: Promise<boolean>

Defined in: src/fs.ts:168

openReadableFile

▸ Abstract openReadableFile(url: string, options?: OpenReadableFileOptions): Promise<ReadableStreamTree>

Opens a file for reading.

optional version Fails if version doesn't match for GCS URLs.

Parameters

Returns: Promise<ReadableStreamTree>

Defined in: src/fs.ts:117

openWritableFile

▸ Abstract openWritableFile(url: string, options?: OpenWritableFileOptions): Promise<WritableStreamTree>

Opens a file for writing.

optional version Fails if version doesn't match for GCS URLs.

Parameters

Returns: Promise<WritableStreamTree>

Defined in: src/fs.ts:127

queueRemoveFile

▸ Abstract queueRemoveFile(urlText: string): Promise<boolean>

Queues deletion, e.g. after DaysSinceCustomTime.

Parameters

Returns: Promise<boolean>

Defined in: src/fs.ts:154

readDirectory

▸ Abstract readDirectory(urlText: string, options?: ReadDirectoryOptions): Promise<DirectoryEntry[]>

Returns the URLs of the files in a directory.

Parameters

Returns: Promise<DirectoryEntry[]>

Defined in: src/fs.ts:86

removeDirectory

▸ Abstract removeDirectory(urlText: string): Promise<boolean>

Removes the directory

Parameters

Returns: Promise<boolean>

Defined in: src/fs.ts:98

removeFile

▸ Abstract removeFile(urlText: string): Promise<boolean>

Deletes the file.

Parameters

Returns: Promise<boolean>

Defined in: src/fs.ts:148

replaceFile

▸ Abstract replaceFile(urlText: string, writeCallback: (stream: WritableStreamTree) => Promise<boolean>, options?: ReplaceFileOptions): Promise<boolean>

Replaces the file, failing if the file version doesn't match.

Parameters

Returns: Promise<boolean>

Defined in: src/fs.ts:177 @wholebuzz/fs / Exports / json

Module: json

Table of contents

Variables

• JSONStream

Functions

• parseJSON
• parseJSONLines
• pipeJSONFormatter
• pipeJSONLinesFormatter
• pipeJSONLinesParser
• pipeJSONParser
• readJSON
• readJSONHashed
• readJSONLines
• serializeJSON
• serializeJSONLines
• writeJSON
• writeJSONLines
• writeShardedJSONLines

Variables

JSONStream

• Const JSONStream: any

Defined in: src/json.ts:12

Functions

parseJSON

▸ parseJSON(stream: ReadableStreamTree): Promise<unknown>

Parses JSON object from [[stream]]. Used to implement readJSON.

Parameters

Returns: Promise<unknown>

Defined in: src/json.ts:86

parseJSONLines

▸ parseJSONLines(stream: ReadableStreamTree): Promise<unknown[]>

Parses JSON object from [[stream]]. Used to implement readJSON.

Parameters

Returns: Promise<unknown[]>

Defined in: src/json.ts:103

pipeJSONFormatter

▸ pipeJSONFormatter(stream: WritableStreamTree, isArray: boolean): WritableStreamTree

Create JSON formatter stream.

Parameters

Returns: WritableStreamTree

Defined in: src/json.ts:158

pipeJSONLinesFormatter

▸ pipeJSONLinesFormatter(stream: WritableStreamTree): WritableStreamTree

Create JSON-lines formatter stream.

Parameters

Returns: WritableStreamTree

Defined in: src/json.ts:173

pipeJSONLinesParser

▸ pipeJSONLinesParser(stream: ReadableStreamTree): ReadableStreamTree

Create JSON parser stream.

Parameters

Returns: ReadableStreamTree

Defined in: src/json.ts:150

pipeJSONParser

▸ pipeJSONParser(stream: ReadableStreamTree, isArray: boolean): ReadableStreamTree

Create JSON parser stream.

Parameters

Returns: ReadableStreamTree

Defined in: src/json.ts:141

readJSON

▸ readJSON(fileSystem: FileSystem, url: string): Promise<unknown>

Reads a serialized JSON object or array from a file.

Parameters

Returns: Promise<unknown>

Defined in: src/json.ts:18

readJSONHashed

▸ readJSONHashed(fileSystem: FileSystem, url: string): Promise<[*unknown*, null | *string*]>

Reads a serialized JSON object from a file, and also hashes the file.

Parameters

Returns: Promise<[*unknown*, null | *string*]>

Defined in: src/json.ts:26

readJSONLines

▸ readJSONLines(fileSystem: FileSystem, url: string): Promise<unknown[]>

Reads a serialized JSON-lines array from a file.

Parameters

Returns: Promise<unknown[]>

Defined in: src/json.ts:36

serializeJSON

▸ serializeJSON(stream: WritableStreamTree, obj: object | any[]): Promise<boolean>

Serializes JSON object to [[stream]]. Used to implement writeJSON.

Parameters

Returns: Promise<boolean>

Defined in: src/json.ts:119

serializeJSONLines

▸ serializeJSONLines(stream: WritableStreamTree, obj: any[]): Promise<boolean>

Serializes JSON object to [[stream]]. Used to implement writeJSONLines.

Parameters

Returns: Promise<boolean>

Defined in: src/json.ts:134

writeJSON

▸ writeJSON(fileSystem: FileSystem, url: string, value: object | any[]): Promise<boolean>

Serializes object or array to a JSON file.

Parameters

Returns: Promise<boolean>

Defined in: src/json.ts:45

writeJSONLines

▸ writeJSONLines(fileSystem: FileSystem, url: string, obj: object[]): Promise<boolean>

Serializes array to a JSON Lines file.

Parameters

Returns: Promise<boolean>

Defined in: src/json.ts:54

writeShardedJSONLines

▸ writeShardedJSONLines(fileSystem: FileSystem, url: string, obj: object[], shards: number, shardFunction?: (x: object, modulus: number) => number): Promise<void>

Parameters

Returns: Promise<void>

Defined in: src/json.ts:58