Cuttlestore is a generic API for key-value stores. It allows you to support multiple key-value stores with zero additional effort, and makes it possible to switch between different stores at runtime.

Example

use cuttlestore::{Cuttlestore, PutOptions};
use serde::{Deserialize, Serialize};
use std::time::Duration;

#[derive(Debug, Serialize, Deserialize)]
struct SelfDestructingMessage {
    message: String,
}

#[tokio::main]
async fn main() {
    let store = Cuttlestore::new("filesystem://./example-store")
                                // or redis, sqlite, in-memory
        .await
        .unwrap();

    let mission = SelfDestructingMessage {
        message: "Your mission, should you choose to accept it, ...".to_string(),
    };
    store
        .put_with("impossible", &mission, PutOptions::ttl_secs(60))
        .await
        .unwrap();

    // Later

    let value: Option<SelfDestructingMessage> = store.get("impossible").await.unwrap();
    println!("Message says: {value:?}");
}

Supported Backends

Cuttlestore currently has support for:

Installing

Add Cuttlestore to your Cargo.toml:

cuttlestore = "0.2"

If you want to disable or enable some of the backends, disable the default features and pick the ones you want:

cuttlestore = { version = "0.2", default-features = false, features = [
  # Only leave redis and sqlite enabled
  "backend-redis",
  "backend-sqlite",
  # Remember to enable this or `logging-log`
  "logging-tracing",
] }

You're now ready to use Cuttlestore! See the example above, check the documentation, and find more examples in the repository.

Overview

• Pros: Cuttlestore is useful if
  • You want to allow end-users to pick which store to use without recompiling
  • You are looking for a simple API for a basic key-value store
• Cons: Avoid Cuttlestore if:
  • You need access to key-value store specific features
  • You only want to one key-value store, and don't care about switching

For example, if you are making a self-hostable web application, and you want to allow users to pick between using Redis and sqlite depending on their needs, you could use Cuttlestore. Cuttlestore supports both of these backends, and your users could input the connection string in your application settings to pick one of these backends. Users with large deployments could pick Redis, and small-scale users could pick sqlite so they don't have to deal with also deploying Redis.

Logging

The library can log errors with both tracing and log. tracing is enabled by default, but you can switch to log by enabling the feature:

cuttlestore = { version = "0.2", default-features = false, features = [
    "logging-log",
    # remember to enable the backends!
    "backend-redis",
    "backend-sqlite",
    "backend-filesystem",
    "backend-in-memory",
] }

Details of backends

Redis

Redis is generally the best option if you don't mind setting it up. It offers good performance and scalability as you can connect many app servers into the same Redis instance.

Cuttlestore has support for TLS, which you can activate by adding an s to the connection string like rediss://127.0.0.1. You can also change the port you are using by adding :port to the end, for example redis://127.0.0.1:5678.

Cuttlestore has support for ACLs as well. You can enable them by adding them to the connection string. For example, if your username is agent and password is 47, you can use the connection string redis://127.0.0.1?username=agent&password=47.

Sqlite

Cuttlestore can use an sqlite database as a key-value store when using this backend. The database and any tables are automatically created.

The sqlite database is configured to use write ahead logging, which means it may create some additional files next to the database file you configure in the connection string. The configuration is also set in a way that there is a small chance of losing the last few put or delete operations if a crash occurs, which is unfortunately required to bring the performance to a reasonable level.

Sqlite doesn't have built-in ttl support, so ttl is supported by periodically scanning the database and deleting expired entries on a best-effort basis. This scan uses a Tokio task, meaning it will run within your existing Tokio thread pool.

For sqlite, you can enable the feature backend-sqlite-native-tls or backend-sqlite-rustls to pick between native TLS or Rustls. backend-sqlite is equal to backend-sqlite-rustls.

Filesystem

Cuttlestore can be configured to use a folder as a key value store. When using this backend, the file names in the folder are the keys, and the values are stored using a binary encoding within the files.

The performance largely depends on your filesystem. Durability is similar to sqlite: there is a small risk of losing the latest few operations, but data corruption is not expected.

The ttl feature is supported by periodically scanning the database and deleting expired entries on a best-effort basis. This scan uses a Tokio task, meaning it will run within your existing Tokio thread pool.

DynamoDB

Cuttlestore can use Amazon DynamoDB as a backing store. The connection string takes the form dynamodb://<region>/<table>. The table is automatically created if it does not exist, using on-demand (PAY_PER_REQUEST) billing, and DynamoDB's native TTL feature is enabled on the live_until attribute.

For example, to connect to a cuttlestore table in us-east-1:

dynamodb://us-east-1/cuttlestore

You can supply credentials and a custom endpoint through query string parameters: endpoint, access_key, and secret_key. The endpoint is mostly useful for pointing at a local DynamoDB instance (such as the official amazon/dynamodb-local Docker image) for testing:

dynamodb://us-east-1/cuttlestore-test?endpoint=http://localhost:8000

If endpoint is set and no credentials are provided, dummy credentials are used (DynamoDB Local ignores them). When connecting to real AWS, omit the endpoint parameter and credentials will be sourced from the standard AWS chain (environment variables, shared profile, IAM role, etc.).

DynamoDB's native TTL deletes expired items on its own schedule, which can take up to 48 hours. Cuttlestore additionally filters expired items in get and scan so they are never returned to your application.

Note that enabling backend-dynamodb pulls in the AWS SDK (the DynamoDB client plus credential-resolution SDKs such as STS and SSO), which can add roughly 13 MB to your stripped release binary. For this reason backend-dynamodb is not part of the default feature set; enable it explicitly when you need it.

In-Memory

The in-memory backend is a multithreaded in-memory key-value store backed by dashmap.

The performance is the best, but everything is kept in-memory so there is no durability.

CouchDB

Cuttlestore can use an Apache CouchDB database as a key-value store. Each Cuttlestore key becomes a document in the database, and the values are stored as base64-encoded strings on those documents. The database is automatically created if it does not already exist.

The connection string follows the form couchdb://[user:pass@]host[:port]/database. To use HTTPS, use the couchdbs:// scheme instead. For example: couchdbs://admin:hunter2@couch.example.com/my-store.

CouchDB does not have built-in TTL support, so TTL is emulated by periodically scanning the database and deleting expired entries on a best-effort basis. This scan uses a Tokio task, meaning it will run within your existing Tokio thread pool.

For CouchDB, you can enable the feature backend-couchdb-native-tls or backend-couchdb-rustls to pick between native TLS or Rustls for the underlying HTTP client. backend-couchdb is equal to backend-couchdb-rustls.

SurrealDB

Cuttlestore can use SurrealDB as a backend. Each Cuttlestore key becomes a record in a single table, where the record id is the key and the bytes are stored in a value field. The connection is made over SurrealDB's WebSocket RPC protocol.

The connection string follows the form surrealdb://[user:pass@]host[:port]/namespace/database[?table=...]. To use WSS (TLS), use the surrealdbs:// scheme instead. For example: surrealdbs://root:root@db.example.com/cuttlestore/cache?table=sessions. The table query parameter is optional and defaults to cuttlestore.

SurrealDB does not have built-in per-record TTL, so TTL is emulated by periodically scanning the table and deleting expired entries on a best-effort basis. This scan uses a Tokio task, meaning it will run within your existing Tokio thread pool.

The SurrealDB backend is pure Rust (it uses tokio-tungstenite with rustls), so it builds without any extra C dependencies. It is opt-in because the surrealdb crate has a sizable dependency tree that adds roughly 10 MB to a stripped release binary.

TTL

The TTL (time to live) feature allows you to designate values that should only exist in the store for a limited amount of time. The values that run out of TTL will be expired and deleted from the store to save space.

store.put_with("impossible", &mission, PutOptions::ttl_secs(60))
// or
store.put_with("impossible", &mission, PutOptions::ttl(Duration::from_secs(60)))

Some backends have built-in support for TTLs (redis). For other backends, the TTL support is emulated by periodically running a Tokio task which scans the store and cleans up expired values. This task runs within your existing Tokio thread pool. You can configure how often this cleanup task runs using CuttlestoreBuilder, see the builder example.

Get and scan operations are guaranteed to never return expired values, but expired values are not necessarily deleted immediately.

Benchmarks

There are some benchmarks to compare the performance of the different backends. All the existing benchmarks use small keyspaces so the performance is not necessarily realistic.

The concurrent benchmarks show you the overall throughput of the backend, while the sequential benchmarks show you the average latency you can expect from each request.

In short, these benchmarks show a few things:

• Redis has relatively high latency, around 67 microseconds per operation. In comparison, the second slowest is sqlite with around 13 to 23 microseconds per operation.
• The filesystem backend offers the best performance both for throughput and latency, but there is a very large spread between the lows and highs. At worst case, it is slower than all other backends.
• In-memory offers incredible performance, but obviously not durable.

These benchmarks validate the suggestions listed earlier in the readme. Redis is a good option if you need scalability, and sqlite is good if scalability is not a concern. Filesystem can be an option if performance is not critical, but there is risk that it will not perform well for large key spaces.