node_s3_file_buffer

About: API to locally buffer and upload JSON streams to S3 files S3 Uploader is provided but other implementations can be created Upload is based on size limit and time limits Multiple streams can be created with domain and namespace hierarchy

to run with S3 uploader AWS credentials must be setup to be used by the AWS SDK: check this for setting up the credentilas: https://docs.aws.amazon.com/amazonswf/latest/awsrbflowguide/set-up-creds.html

the JsonBufferStreamer class requires the following parameters to instantiate: uploader - a class that implements the - async upload({file_name, domain, namespace}) signature filesPath - location for the buffered filed, a directory onUpload - optional - callbeack for file upload notifications schedulerIntervalSec - if time based flush is required, this is used to indicate the time interval for the scheduler to implement the buffers flushing (based on the specific stream definitions). use 0 to prevent scheduled flush

API: open(domain, namespace, sizeLimit = 100000, timeLimit = 60) - opens a new stream with domain name, a namespace, the size limit that once passed will trigger a file upload the the timeLimit after which, if any data is in the stream it will be uploaded

append(domain, namespace, json) - Appends a Json to an existing stream, the file created will have an array of Jsons.

close(domain, namespace) - finalizes a Json stream, triggers a final flush of the stream to be uploaded.

flush() - used to flush all the buffers by uploading them using the uploader

shutDown() - used to initiate service shutdown, all files will be flushed and uploaded, all incoming append requests will be refused.