s3-streamer

Stream uploads to S3 with automatic content-type detection, ETag generation, and metadata extraction. Built on AWS SDK v3 for optimal performance and smaller bundle sizes.

Features

• 🚀 Streaming uploads - Memory efficient streaming directly to S3
• 🔍 Automatic content-type detection - Uses magic bytes to detect file types
• 🏷️ Metadata extraction - Generates ETags, SHA hashes, and UUIDs for uploaded content
• 📦 AWS SDK v3 - Modern, modular SDK with better tree-shaking
• 🔧 MinIO compatible - Works with S3-compatible storage services
• 💪 TypeScript support - Full type definitions included

Installation

npm install @constructive-io/s3-streamer

Quick Start

Stream uploads to S3

import Streamer from '@constructive-io/s3-streamer';
const streamer = new Streamer(opts)
const readStream = createReadStream(filename);
const results = await streamer.upload({
    readStream,
    filename,
    bucket,
    key
});

Response Format

The upload methods return a detailed payload with upload results and file metadata:

{
  upload: {
    ETag: '"952fd44d14cee87882239b707231609d"',
    Location: 'http://localhost:9000/constructive/db1/assets/.gitignore',
    Key: 'db1/assets/.gitignore',
    Bucket: 'constructive'
  },
  magic: { 
    type: 'text/plain', 
    charset: 'us-ascii' 
  },
  contentType: 'text/plain',
  contents: {
    uuid: '278aee01-1404-5725-8f0e-7044c9c16397',
    sha: '7d65523f2a5afb69d76824dd1dfa62a34faa3197',
    etag: '952fd44d14cee87882239b707231609d'
  }
}

Response Fields

• upload: S3 upload response
  • ETag: S3 ETag of the uploaded object
  • Location: Full URL to the uploaded object
  • Key: S3 object key
  • Bucket: Bucket name where object was uploaded
• magic: File type detection results
  • type: MIME type detected from file content
  • charset: Character encoding (for text files)
• contentType: Final content-type used for upload
• contents: File metadata
  • uuid: Deterministic UUID based on file content
  • sha: SHA-1 hash of file content
  • etag: Computed ETag (matches S3 ETag for single-part uploads)

functional utils

If you don't want to use the Streamer class you can use the utils directly:

import { getClient, upload } from '@constructive-io/s3-streamer';
const client = getClient(opts)
const readStream = createReadStream(filename);
const results = await upload({
    client,
    readStream,
    filename,
    bucket,
    key
});

Configuration

AWS S3 Production

const streamer = new Streamer({
  defaultBucket: 'my-bucket',
  awsRegion: 'us-east-1',
  awsSecretKey: process.env.AWS_SECRET_ACCESS_KEY,
  awsAccessKey: process.env.AWS_ACCESS_KEY_ID
});

MinIO / S3-Compatible Storage

const streamer = new Streamer({
  defaultBucket: 'my-bucket',
  awsRegion: 'us-east-1',
  awsSecretKey: 'minio-secret',
  awsAccessKey: 'minio-access',
  minioEndpoint: 'http://localhost:9000'
});

API Reference

Streamer Class

Constructor Options

interface StreamerOptions {
  awsRegion: string;        // AWS region (e.g., 'us-east-1')
  awsSecretKey: string;     // AWS secret access key
  awsAccessKey: string;     // AWS access key ID
  minioEndpoint?: string;   // Optional: MinIO/S3-compatible endpoint
  defaultBucket: string;    // Default bucket for uploads
}

Methods

upload(params)

Uploads a file stream to S3 with automatic content-type detection and metadata extraction.

interface UploadParams {
  readStream: ReadStream;   // Node.js readable stream
  filename: string;         // Original filename (used for content-type detection)
  key: string;             // S3 object key (path in bucket)
  bucket?: string;         // Optional: Override default bucket
}

destroy()

Cleans up the S3 client connections. Should be called when done with the streamer instance.

streamer.destroy();

Functional API

If you prefer functional programming over classes:

import { getClient, upload } from '@constructive-io/s3-streamer';

// Create S3 client
const client = getClient({
  awsRegion: 'us-east-1',
  awsSecretKey: process.env.AWS_SECRET_ACCESS_KEY,
  awsAccessKey: process.env.AWS_ACCESS_KEY_ID,
  minioEndpoint: 'http://localhost:9000' // optional
});

// Upload file
const results = await upload({
  client,
  readStream: createReadStream('file.pdf'),
  filename: 'file.pdf',
  bucket: 'my-bucket',
  key: 'uploads/file.pdf'
});

// Clean up when done
client.destroy();

Education and Tutorials

1. 🚀 Quickstart: Getting Up and Running Get started with modular databases in minutes. Install prerequisites and deploy your first module.

2. 📦 Modular PostgreSQL Development with Database Packages Learn to organize PostgreSQL projects with pgpm workspaces and reusable database modules.

3. ✏️ Authoring Database Changes Master the workflow for adding, organizing, and managing database changes with pgpm.

4. 🧪 End-to-End PostgreSQL Testing with TypeScript Master end-to-end PostgreSQL testing with ephemeral databases, RLS testing, and CI/CD automation.

5. ⚡ Supabase Testing Use TypeScript-first tools to test Supabase projects with realistic RLS, policies, and auth contexts.

6. 💧 Drizzle ORM Testing Run full-stack tests with Drizzle ORM, including database setup, teardown, and RLS enforcement.

7. 🔧 Troubleshooting Common issues and solutions for pgpm, PostgreSQL, and testing.

Related Constructive Tooling

📦 Package Management

• pgpm: 🖥️ PostgreSQL Package Manager for modular Postgres development. Works with database workspaces, scaffolding, migrations, seeding, and installing database packages.

🧪 Testing

• pgsql-test: 📊 Isolated testing environments with per-test transaction rollbacks—ideal for integration tests, complex migrations, and RLS simulation.
• pgsql-seed: 🌱 PostgreSQL seeding utilities for CSV, JSON, SQL data loading, and pgpm deployment.
• supabase-test: 🧪 Supabase-native test harness preconfigured for the local Supabase stack—per-test rollbacks, JWT/role context helpers, and CI/GitHub Actions ready.
• graphile-test: 🔐 Authentication mocking for Graphile-focused test helpers and emulating row-level security contexts.
• pg-query-context: 🔒 Session context injection to add session-local context (e.g., SET LOCAL) into queries—ideal for setting role, jwt.claims, and other session settings.

🧠 Parsing & AST

• pgsql-parser: 🔄 SQL conversion engine that interprets and converts PostgreSQL syntax.
• libpg-query-node: 🌉 Node.js bindings for libpg_query, converting SQL into parse trees.
• pg-proto-parser: 📦 Protobuf parser for parsing PostgreSQL Protocol Buffers definitions to generate TypeScript interfaces, utility functions, and JSON mappings for enums.
• @pgsql/enums: 🏷️ TypeScript enums for PostgreSQL AST for safe and ergonomic parsing logic.
• @pgsql/types: 📝 Type definitions for PostgreSQL AST nodes in TypeScript.
• @pgsql/utils: 🛠️ AST utilities for constructing and transforming PostgreSQL syntax trees.

Credits

🛠 Built by the Constructive team — creators of modular Postgres tooling for secure, composable backends. If you like our work, contribute on GitHub.

Disclaimer

AS DESCRIBED IN THE LICENSES, THE SOFTWARE IS PROVIDED "AS IS", AT YOUR OWN RISK, AND WITHOUT WARRANTIES OF ANY KIND.

No developer or entity involved in creating this software will be liable for any claims or damages whatsoever associated with your use, inability to use, or your interaction with other users of the code, including any direct, indirect, incidental, special, exemplary, punitive or consequential damages, or loss of profits, cryptocurrencies, tokens, or anything else of value.