Storage

Node: also in Python · Go · Rust · Swift · Dart · Java and Kotlin · .NET · C++ · Objective-C

Remotes using storage services through the shared native storage library.

Concepts explains the remote and local store.

Each constructor returns an ordinary remote. Pass it to the same open call and close it the same way. The other reference pages apply to these remotes without change.

storage.fromS3

export function fromS3(options: S3Options): Remote;

export interface S3Options {
  /** The bucket holding the objects. Required. */
  bucket?: string | null;
  /**
   * The prefix within the bucket that this database lives under.
   * Required: one bucket holds many databases, one per namespace.
   */
  namespace?: string | null;
  /** The region, when the store needs one named. R2 wants `"auto"`. */
  region?: string | null;
  /** The endpoint, for a store that is not AWS. */
  endpoint?: string | null;
  /**
   * Address the bucket as a path under the endpoint rather than as a
   * subdomain of it, which is what most non-AWS stores and every local
   * one expect.
   */
  forcePathStyle?: boolean | null;
  /**
   * An explicit key pair. Given together with {@link secretAccessKey} or
   * not at all; leave both out to use the AWS credential chain.
   */
  accessKeyId?: string | null;
  secretAccessKey?: string | null;
  /** The session token that goes with temporary credentials. */
  sessionToken?: string | null;
  /** Concurrency and retention for the remote, as elsewhere in the SDK. */
  remote?: RemoteOptions | null;
}

A remote over an S3-compatible bucket.

Credentials come from the AWS chain unless accessKeyId and secretAccessKey are supplied. aws-config resolves the chain itself:

• Environment
• Shared configuration and credentials files
• SSO
• Web identity
• Container and instance metadata

A storage service other than AWS is named by endpoint. The AWS chain still supplies credentials unless accessKeyId and secretAccessKey are supplied, so supply the key pair that the service issues. Cloudflare R2 issues a pair with an account endpoint and requires region set to “auto”. MinIO and most self-hosted gateways require forcePathStyle. This addresses the bucket as a path segment instead of a subdomain.

• bucket: The bucket holding the objects.
• namespace: The database’s prefix. One bucket can hold many databases.
• region: The signing region. If absent, the AWS chain selects it.
• endpoint: The endpoint of a storage service other than AWS.
• forcePathStyle: Address the bucket as a path segment instead of a subdomain.
• accessKeyId: An explicit access key. Supply it together with secretAccessKey, or supply neither.
• secretAccessKey: The secret for accessKeyId.
• sessionToken: The session token for temporary credentials.
• remote: Remote concurrency and retention, as elsewhere in the SDK.

storage.fromGcs

export function fromGcs(options: GcsOptions): Remote;

export interface GcsOptions {
  /** The bucket holding the objects. Required. */
  bucket?: string | null;
  /**
   * The prefix within the bucket that this database lives under.
   * Required: one bucket holds many databases, one per namespace.
   */
  namespace?: string | null;
  /** The endpoint, for a store that is not Google's own. */
  endpoint?: string | null;
  /**
   * The service account key Google issues, as an object or as the text
   * of one. Leave it out to use Application Default Credentials.
   */
  serviceAccount?: Record<string, unknown> | string | null;
  /** Concurrency and retention for the remote, as elsewhere in the SDK. */
  remote?: RemoteOptions | null;
}

A remote over a Google Cloud Storage bucket.

Unless serviceAccount is supplied, google-cloud-auth resolves Application Default Credentials in this order:

• GOOGLE_APPLICATION_CREDENTIALS
• The gcloud user credentials file
• The metadata server

If the process has no such chain, pass Google’s service account JSON as an object or JSON string. A storage service other than Google’s is named by endpoint, including gateways and emulators.

• bucket: The bucket holding the objects.
• namespace: The database’s prefix. One bucket can hold many databases.
• endpoint: The endpoint of a storage service other than Google’s.
• serviceAccount: Google’s service account JSON, as an object or JSON string.
• remote: Remote concurrency and retention, as elsewhere in the SDK.

storage.fromAzure

export function fromAzure(options: AzureOptions): Remote;

export interface AzureOptions {
  /** The storage account holding the container. Required. */
  account?: string | null;
  /** The container holding the objects. Required. */
  container?: string | null;
  /**
   * The prefix within the container that this database lives under.
   * Required: one container holds many databases, one per namespace.
   */
  namespace?: string | null;
  /**
   * The endpoint, for a store that is not Azure's own. Reaches the
   * native library as `endpoint_url`; Azurite is one account under one
   * port, so its endpoint carries the account as a path segment.
   */
  endpoint?: string | null;
  /**
   * An explicit account key. Leave it out to use the Azure credential
   * chain.
   */
  accountKey?: string | null;
  /** Concurrency and retention for the remote, as elsewhere in the SDK. */
  remote?: RemoteOptions | null;
}

A remote over an Azure Blob Storage container.

Unless accountKey is supplied, azure-identity resolves the Azure credential chain itself:

• Environment
• Workload identity
• Managed identity
• Azure CLI
• Developer tools

Pass an account key for a storage service without such a chain. Azurite supplies the well-known devstoreaccount1 pair.

A storage service other than Azure’s is named by endpoint, passed to the library as endpoint_url. Azurite provides one account per port, so its endpoint includes the account as a path segment.

• account: The storage account holding the container.
• container: The container holding the objects.
• namespace: The database’s prefix. One container can hold many databases.
• endpoint: The endpoint of a storage service other than Azure’s.
• accountKey: An explicit account key.
• remote: Remote concurrency and retention, as elsewhere in the SDK.