bbolt

bbolt is a fork of Ben Johnson's Bolt key/value store. The purpose of this fork is to provide the Go community with an active maintenance and development target for Bolt; the goal is improved reliability and stability. bbolt includes bug fixes, performance enhancements, and features not found in Bolt while preserving backwards compatibility with the Bolt API.

Bolt is a pure Go key/value store inspired by Howard Chu's LMDB project. The goal of the project is to provide a simple, fast, and reliable database for projects that don't require a full database server such as Postgres or MySQL.

Since Bolt is meant to be used as such a low-level piece of functionality, simplicity is key. The API will be small and only focus on getting values and setting values. That's it.

Project Status

Bolt is stable, the API is fixed, and the file format is fixed. Full unit test coverage and randomized black box testing are used to ensure database consistency and thread safety. Bolt is currently used in high-load production environments serving databases as large as 1TB. Many companies such as Shopify and Heroku use Bolt-backed services every day.

Project versioning

bbolt uses semantic versioning. API should not change between patch and minor releases. New minor versions may add additional features to the API.

Table of Contents

• Getting Started
  • Installing
  • Opening a database
  • Transactions
  • Read-write transactions
  • Read-only transactions
  • Batch read-write transactions
  • Managing transactions manually
  • Using buckets
  • Using key/value pairs
  • Autoincrementing integer for the bucket
  • Iterating over keys
  • Prefix scans
  • Range scans
  • ForEach()
  • Nested buckets
  • Database backups
  • Statistics
  • Read-Only Mode
  • Mobile Use (iOS/Android)
• Resources
• Comparison with other databases
  • Postgres, MySQL, & other relational databases
  • LevelDB, RocksDB
  • LMDB
• Caveats & Limitations
• Reading the Source
• Known Issues
• Other Projects Using Bolt

Getting Started

Installing

To start using bbolt, install Go and run go get:

$ go get go.etcd.io/bbolt@latest

This will retrieve the library and update your go.mod and go.sum files.

To run the command line utility, execute:

$ go run go.etcd.io/bbolt/cmd/bbolt@latest

Run go install to install the bbolt command line utility into your $GOBIN path, which defaults to $GOPATH/bin or $HOME/go/bin if the GOPATH environment variable is not set.

$ go install go.etcd.io/bbolt/cmd/bbolt@latest

Importing bbolt

To use bbolt as an embedded key-value store, import as:

import bolt "go.etcd.io/bbolt"

db, err := bolt.Open(path, 0600, nil)
if err != nil {
  return err
}
defer db.Close()

Opening a database

The top-level object in Bolt is a DB. It is represented as a single file on your disk and represents a consistent snapshot of your data.

To open your database, simply use the bolt.Open() function:

package main

import (
    "log"

    bolt "go.etcd.io/bbolt"
)

func main() {
    // Open the my.db data file in your current directory.
    // It will be created if it doesn't exist.
    db, err := bolt.Open("my.db", 0600, nil)
    if err != nil {
        log.Fatal(err)
    }
    defer db.Close()

    ...
}

Please note that Bolt obtains a file lock on the data file so multiple processes cannot open the same database at the same time. Opening an already open Bolt database will cause it to hang until the other process closes it. To prevent an indefinite wait you can pass a timeout option to the Open() function:

db, err := bolt.Open("my.db", 0600, &bolt.Options{Timeout: 1 * time.Second})

Transactions

Bolt allows only one read-write transaction at a time but allows as many read-only transactions as you want at a time. Each transaction has a consistent view of the data as it existed when the transaction started.

Individual transactions and all objects created from them (e.g. buckets, keys) are not thread safe. To work with data in multiple goroutines you must start a transaction for each one or use locking to ensure only one goroutine accesses a transaction at a time. Creating transaction from the DB is thread safe.

Transactions should not depend on one another and generally shouldn't be opened simultaneously in the same goroutine. This can cause a deadlock as the read-write transaction needs to periodically re-map the data file but it cannot do so while any read-only transaction is open. Even a nested read-only transaction can cause a deadlock, as the child transaction can block the parent transaction from releasing its resources.

Read-write transactions

To start a read-write transaction, you can use the DB.Update() function:

err := db.Update(func(tx *bolt.Tx) error {
    ...
    return nil
})

Inside the closure, you have a consistent view of the database. You commit the transaction by returning nil at the end. You can also rollback the transaction at any point by returning an error. All database operations are allowed inside a read-write transaction.

Always check the return error as it will report any disk failures that can cause your transaction to not complete. If you return an error within your closure it will be passed through.

Read-only transactions

To start a read-only transaction, you can use the DB.View() function:

err := db.View(func(tx *bolt.Tx) error {
    ...
    return nil
})

You also get a consistent view of the database within this closure, however, no mutating operations are allowed within a read-only transaction. You can only retrieve buckets, retrieve values, and copy the database within a read-only transaction.

Batch read-write transactions

Each DB.Update() waits for disk to commit the writes. This overhead can be minimized by combining multiple updates with the DB.Batch() function:

err := db.Batch(func(tx *bolt.Tx) error {
    ...
    return nil
})

Concurrent Batch calls are opportunistically combined into larger transactions. Batch is only useful when there are multiple goroutines calling it.

The trade-off is that Batch can call the given function multiple times, if parts of the transaction fail. The function must be idempotent and side effects must take effect only after a successful return from DB.Batch().

For example: don't display messages from inside the function, instead set variables in the enclosing scope:

var id uint64
err := db.Batch(func(tx *bolt.Tx) error {
    // Find last key in bucket, decode as bigendian uint64, increment
    // by one, encode back to []byte, and add new key.
    ...
    id = newValue
    return nil
})
if err != nil {
    return ...
}
fmt.Println("Allocated ID %d", id)

Managing transactions manually

The DB.View() and DB.Update() functions are wrappers around the DB.Begin() function. These helper functions will start the transaction, execute a function, and then safely close your transaction if an error is returned. This is the recommended way to use Bolt transactions.

However, sometimes you may want to manually start and end your transactions. You can use the DB.Begin() function directly but please be sure to close the transaction.

// Start a writable transaction.
tx, err := db.Begin(true)
if err != nil {
    return err
}
defer tx.Rollback()

// Use the transaction...
_, err := tx.CreateBucket([]byte("MyBucket"))
if err != nil {
    return err
}

// Commit the transaction and check for error.
if err := tx.Commit(); err != nil {
    return err
}

The first argument to DB.Begin() is a boolean stating if the transaction should be writable.

Using buckets

Buckets are collections of key/value pairs within the database. All keys in a bucket must be unique. You can create a bucket using the Tx.CreateBucket() function:

db.Update(func(tx *bolt.Tx) error {
    b, err := tx.CreateBucket([]byte("MyBucket"))
    if err != nil {
        return fmt.Errorf("create bucket: %s", err)
    }
    return nil
})

You can retrieve an existing bucket using the Tx.Bucket() function:

db.Update(func(tx *bolt.Tx) error {
    b := tx.Bucket([]byte("MyBucket"))
    if b == nil {
        return errors.New("bucket does not exist")
    }
    return nil
})

You can also create a bucket only if it doesn't exist by using the Tx.CreateBucketIfNotExists() function. It's a common pattern to call this function for all your top-level buckets after you open your database so you can guarantee that they exist for future transactions.

To delete a bucket, simply call the Tx.DeleteBucket() function.

You can also iterate over all existing top-level buckets with Tx.ForEach():

db.View(func(tx *bolt.Tx) error {
    tx.ForEach(func(name []byte, b *bolt.Bucket) error {
        fmt.Println(string(name))
        return nil
    })
    return nil
})

Using key/value pairs

To save a key/value pair to a bucket, use the Bucket.Put() function:

db.Update(func(tx *bolt.Tx) error {
    b := tx.Bucket([]byte("MyBucket"))
    err := b.Put([]byte("answer"), []byte("42"))
    return err
})

This will set the value of the "answer" key to "42" in the MyBucket bucket. To retrieve this value, we can use the Bucket.Get() function:

db.View(func(tx *bolt.Tx) error {
    b := tx.Bucket([]byte("MyBucket"))
    v := b.Get([]byte("answer"))
    fmt.Printf("The answer is: %s\n", v)
    return nil
})

The Get() function does not return an error because its operation is guaranteed to work (unless there is some kind of system failure). If the key exists then it will return its byte slice value. If it doesn't exist then it will return nil. It's important to note that you can have a zero-length value set to a key which is different than the key not existing.

Use the Bucket.Delete() function to delete a key from the bucket:

db.Update(func (tx *bolt.Tx) error {
    b := tx.Bucket([]byte("MyBucket"))
    err := b.Delete([]byte("answer"))
    return err
})

This will delete the key answers from the bucket MyBucket.

Please note that values returned from Get() are only valid while the transaction is open. If you need to use a value outside of the transaction then you must use copy() to copy it to another byte slice.

Autoincrementing integer for the bucket

By using the NextSequence() function, you can let Bolt determine a sequence which can be used as the unique identifier for your key/value pairs. See the example below.

// CreateUser saves u to the store. The new user ID is set on u once the data is persisted.
func (s *Store) CreateUser(u *User) error {
    return s.db.Update(func(tx *bolt.Tx) error {
        // Retrieve the users bucket.
        // This should be created when the DB is first opened.
        b := tx.Bucket([]byte("users"))

        // Generate ID for the user.
        // This returns an error only if the Tx is closed or not writeable.
        // That can't happen in an Update() call so I ignore the error check.
        id, _ := b.NextSequence()
        u.ID = int(id)

        // Marshal user data into bytes.
        buf, err := json.Marshal(u)
        if err != nil {
            return err
        }

        // Persist bytes to users bucket.
        return b.Put(itob(u.ID), buf)
    })
}

// itob returns an 8-byte big endian representation of v.
func itob(v int) []byte {
    b := make([]byte, 8)
    binary.BigEndian.PutUint64(b, uint64(v))
    return b
}

type User struct {
    ID int
    ...
}

Iterating over keys

Bolt stores its keys in byte-sorted order within a bucket. This makes sequential iteration over these keys extremely fast. To iterate over keys we'll use a Cursor:

db.View(func(tx *bolt.Tx) error {
    // Assume bucket exists and has keys
    b := tx.Bucket([]byte("MyBucket"))

    c := b.Cursor()

    for k, v := c.First(); k != nil; k, v = c.Next() {
        fmt.Printf("key=%s, value=%s\n", k, v)
    }

    return nil
})

The cursor allows you to move to a specific point in the list of keys and move forward or backward through the keys one at a time.

The following functions are available on the cursor:

First()  Move to the first key.
Last()   Move to the last key.
Seek()   Move to a specific key.
Next()   Move to the next key.
Prev()   Move to the previous key.

Each of those functions has a return signature of (key []byte, value []byte). You must seek to a position using First(), Last(), or Seek() before calling Next() or Prev(). If you do not seek t