Quantum Resistant Email: How we use encrypted SQLite mailboxes to keep your email safe

Foreword

Important

Our email service is 100% open-source and privacy-focused through secure and encrypted SQLite mailboxes.

Until we launched IMAP support, we used MongoDB for our persistent data storage needs.

This technology is amazing and we still use it today – but in order to have encryption-at-rest with MongoDB you need to use a provider that offers MongoDB Enterprise, such as Digital Ocean or Mongo Atlas – or pay for an enterprise license (and subsequently have to work with sales team latency).

Our team at Forward Email needed a developer-friendly, scalable, reliable, and encrypted storage solution for IMAP mailboxes. As open-source developers, using a technology you need to pay a license fee in order to get the encryption-at-rest feature was against our principles – and so we experimented, researched, and developed a new solution from scratch to solve these needs.

Instead of using a shared database to store your mailboxes, we individually store and encrypt your mailboxes with your password (which only you have). Our email service is so secure that if you forget your password, then you lose your mailbox (and need to recover with offline backups or start over).

Keep reading as we take a deep dive below with a comparison of email service providers, how our service works, our technology stack, and more.

Email service provider comparison

We are the only 100% open-source and privacy-focused email service provider that stores individually encrypted SQLite mailboxes, offers unlimited domains, aliases, and users, and has outbound SMTP, IMAP, and POP3 support:

Unlike other email providers, you do not need to pay for storage on a per domain or alias basis with Forward Email. Storage is shared across your entire account – so if you have multiple custom domain names and multiple aliases on each, then we are the perfect solution for you. Note that you can still enforce storage limits if desired on a per domain or alias basis.

Read Email Service Comparison

How does it work

Using your email client such as Apple Mail, Thunderbird, Gmail, or Outlook – you connect to our secure IMAP servers using your username and password:
- Your username is your full alias with your domain such as hello@example.com.
- Your password is randomly generated and only displayed to you for 30 seconds when you click Generate Password from My Account Domains Aliases.
Once connected, your email client will send IMAP protocol commands to our IMAP server to keep your mailbox in sync. This includes writing and storing draft emails and other actions you might do (e.g. label an email as Important or flag an email as Spam/Junk Mail).
Mail exchange servers (commonly known as "MX" servers) receive new inbound email and store it to your mailbox. When this happens your email client will get notified and sync your mailbox. Our mail exchange servers can forward your email to one or more recipients (including webhooks), store your email for you in your encrypted IMAP storage with us, or both!

Tip
Interested in learning more? Read how to setup email forwarding, how our mail exchange service works, or view our guides.
Behind the scenes, our secure email storage design works in two ways to keep your mailboxes encrypted and only accessible by you:
- When new mail is received for you from a sender, our mail exchange servers write to an individual, temporary, and encrypted mailbox for you.
- When you connect to our IMAP server with your email client, your password is then encrypted in-memory and used to read and write to your mailbox. Your mailbox can only be read from and written to with this password. Keep in mind that since you are the only one with this password, only you can read and write to your mailbox when you are accessing it. The next time your email client attempts to poll for mail or syncs, your new messages will be transferred from this temporary mailbox and stored in your actual mailbox file using your supplied password. Note that this temporary mailbox is purged and deleted afterwards so that only your password protected mailbox has the messages.
- If you are connected to IMAP (e.g. using an email client such as Apple Mail or Thunderbird), then we do not need to write to temporary disk storage. Your in-memory encrypted IMAP password is instead fetched and used. In real-time, when a message is attempting to be delivered to you, we send a WebSocket request to all IMAP servers asking them if they have an active session for you (this is the fetch part), and then subsequently will pass along that encrypted in-memory password – so we don't need to write to a temporary mailbox, we can write to your actual encrypted mailbox using your encrypted password.
Backups of your encrypted mailboxes are made daily. You can also request a new backup at any time or download the latest backup from My Account Domains Aliases. If you decide to switch to another email service, then you can easily migrate, download, export, and purge your mailboxes and backups at anytime.

Technologies

Databases

We explored other possible database storage layers, however none satisfied our requirements as much as SQLite did:

Database	Encryption-at-rest	Sandboxed Mailboxes	License	Used Everywhere
SQLite ⭐	✅ Yes with SQLite3MultipleCiphers	✅	✅ Public Domain	✅
MongoDB	❌ "Available in MongoDB Enterprise only"	❌ Relational database	❌ AGPL and `SSPL-1.0`	❌
rqlite	❌ Network only	❌ Relational database	✅ `MIT`	❌
dqlite	❌ Untested and not yet supported?	❌ Untested and not yet supported?	✅ `LGPL-3.0-only`	❌
PostgreSQL	✅ Yes	❌ Relational database	✅ `PostgreSQL` (similar to `BSD` or `MIT`)	❌
MariaDB	✅ For InnoDB only	❌ Relational database	✅ `GPLv2` and `BUSL-1.1`	❌
CockroachDB	❌ Enterprise-only feature	❌ Relational database	❌ `BUSL-1.1` and others	❌

Here is a blog post that compares several SQLite database storage options in the table above.

Security

At all times we use encryption-at-rest (AES-256), encryption-in-transit (TLS), DNS over HTTPS ("DoH") using 🍊 Tangerine, and sqleet (ChaCha20-Poly1305) encryption on mailboxes. Additionally we use token-based two-factor authentication (as opposed to SMS which is suspectible to man-in-the-middle-attacks), rotated SSH keys with root access disabled, exclusive access to servers through restricted IP addresses, and more.

In the event of an evil maid attack or rogue employee from a third-party vendor, your mailbox can still only be opened with your generated password. Rest assured, we don't rely upon any third-party vendors other than our SOC Type 2 complaint server providers of Cloudflare, DataPacket, Digital Ocean, and Vultr.

Our goal is to have as few single point of failures as possible.

Mailboxes

tldr; Our IMAP servers use individually encrypted SQLite databases for each of your mailboxes.

SQLite is an extremely popular embedded database – it's currently running on your phone and computer – and used by nearly all major technologies.

For example, on our encrypted servers there's a SQLite database mailbox for linux@example.com, info@example.com, hello@example.com and so on – one for each as a .sqlite database file. We don't name the database files with the email address either – instead we use BSON ObjectID and unique UUID's generated which do not share who the mailbox belongs to or which email address it is under (e.g. 353a03f21e534321f5d6e267.sqlite).

Each of these databases are encrypted themselves using your password (which only you have) using sqleet (ChaCha20-Poly1305). This means that your mailboxes are individually encrypted, self-contained, sandboxed, and portable.

We have fine-tuned SQLite with the following PRAGMA:

`PRAGMA`	Purpose
`cipher=chacha20`	ChaCha20-Poly1305 SQLite database encryption. Reference `better-sqlite3-multiple-ciphers` under Projects for more insight.
`key="****************"`	This is your decrypted in-memory only password that gets passed through your email client's IMAP connection to our server. New database instances are created and closed for each read and write session (in order to ensure sandboxing and isolation).
`journal_model=WAL`	Write-ahead-log ("WAL") which boosts performance and allows concurrent read access.
`busy_timeout=5000`	Prevents write-lock errors while other writes are taking place.
`synchronous=NORMAL`	Increases durability of transactions without data corruption risk.
`foreign_keys=ON`	Enforces that foreign key references (e.g. a relation from one table to another) are enforced. By default this is not turned on in SQLite, but for validation and data integrity it should be enabled.
`encoding='UTF-8'`	Default encoding to use to ensure developer sanity.

All other defaults are from SQLite as specified from the official PRAGMA documentation.

Concurrency

tldr; We use WebSocket for concurrent reads and writes to your encrypted SQLite mailboxes.

Reads

Your email client on your phone may resolve imap.forwardemail.net to one of our Digital Ocean IP addresses – and your desktop client may resolve a separate IP from a different provider altogether.

Regardless of which IMAP server your email client connects to, we want the connection to read from your database in real-time with 100% accuracy. This is done through WebSockets.

Writes

Writing to your database is a bit different – since SQLite is an embedded database and your mailbox lives in a single file by default.

We had explored options such as litestream, rqlite, and dqlite below – however none of these satisfied our requirements.

To accomplish writes with write-ahead-logging ("WAL") enabled – we need to ensure that only one server ("Primary") is responsible for doing so. WAL drastically speeds up concurrency and allows one writer and multiple readers.

The Primary is running on the data servers with the mounted volumes containing the encrypted mailboxes. From a distribution standpoint, you could consider all the individual IMAP servers behind imap.forwardemail.net to be secondary servers ("Secondary").

We accomplish two-way communication with WebSockets:

Primary servers use an instance of ws's WebSocketServer server.
Secondary servers use an instance of ws's WebSocket client that is wrapped with websocket-as-promised and reconnecting-websocket. These two wrappers ensure that the WebSocket reconnects and can send and receive data for specific database writes.

Backups

tldr; Backups of your encrypted mailboxes are made daily. You can also instantly request a new backup or download the latest backup at anytime from My Account Domains Aliases.

For backups, we simply run the SQLite VACUUM INTO command every day during IMAP command processing, which leverages your encrypted password from an in-memory IMAP connection. Backups are stored if no existing backup is detected or if the SHA-256 hash has changed on the file as compared to the most recent backup.

Note that we use the VACUUM INTO command as opposed to the built-in backup command because if a page is modified during a backup command operation, then it has to start over. The VACUUM INTO command will take a snapshot. See these comments on GitHub and Hacker News for more insight.

Additionally we use VACUUM INTO as opposed to backup, because the backup command would leave the database unencrypted for a brief period until rekey is invoked (see this GitHub comment for insight).

The Secondary will instruct the Primary over the WebSocket connection to execute the backup – and the Primary will then receive the command to do so and will subsequently:

Connect to your encrypted mailbox.
Acquire a write lock.
Run a WAL checkpoint via wal_checkpoint(PASSIVE).
Run the VACUUM INTO SQLite command.
Ensure that the copied file can be opened with the encrypted password (safeguard/dummyproofing).
Upload it to Cloudflare R2 for storage (or your own provider if specified).

Remember that your mailboxes are encrypted – and while we have IP restrictions and other authentication measures in place for WebSocket communication – in the event of a bad actor, you can rest assured that unless the WebSocket payload has your IMAP password, it cannot open your database.

Only one backup is stored per mailbox at this time, but in the future we may offer point-in-time-recovery ("PITR").

Search

Our IMAP servers support the SEARCH command with complex queries, regular expressions, and more.

Fast search performance is thanks to FTS5 and sqlite-regex.

We store Date values in the SQLite mailboxes as ISO 8601 strings via Date.prototype.toISOString (with UTC timezone for equality comparisons to function properly).

Indices are also stored for all properties that are in search queries.

Projects

Here's a table outlining projects we use in our source code and development process (sorted alphabetically):

Project	Purpose
Ansible	DevOps automation platform for maintaing, scaling, and managing our entire fleet of servers with ease.
Bree	Job scheduler for Node.js and JavaScript with cron, dates, ms, later, and human-friendly support.
Cabin	Developer-friendly JavaScript and Node.js logging library with security and privacy in mind.
Lad	Node.js framework which powers our entire architecture and engineering design with MVC and more.
MongoDB	NoSQL database solution that we use for storing all other data outside of mailboxes (e.g. your account, settings, domains, and alias configurations).
Mongoose	MongoDB object document modeling ("ODM") which we use across our entire stack. We wrote special helpers that allow us to simply continue using Mongoose with SQLite 🎉
Node.js	Node.js is the open-source, cross-platform JavaScript runtime environment which runs all of our server processes.
Nodemailer	Node.js package for sending emails, creating connections, and more. We are an official sponsor of this project.
Redis	In-memory database for caching, publish/subscribe channels, and DNS over HTTPS requests.
SQLite3MultipleCiphers	Encryption extension for SQLite to allow entire database files to be encrypted (including the write-ahead-log ("WAL"), journal, rollback, …).
SQLiteStudio	Visual SQLite editor (which you could also use) to test, download, and view development mailboxes.
SQLite	Embedded database layer for scalable, self-contained, fast, and resilient IMAP storage.
Spam Scanner	Node.js anti-spam, email filtering, and phishing prevention tool (our alternative to Spam Assassin and rspamd).
Tangerine	DNS over HTTPS requests with Node.js and caching using Redis – which ensures global consistency and much more.
Thunderbird	Our development team uses this (and recommends this too) as the preferred email client to use with Forward Email.
UTM	Our development team uses this create virtual machines for iOS and macOS in order to test different email clients (in parallel) with our IMAP and SMTP servers.
Ubuntu	Modern open-source Linux-based server operating system which powers all of our infrastructure.
WildDuck	IMAP server library – see its notes on attachment de-duplication and IMAP protocol support.
better-sqlite3-multiple-ciphers	Fast and simple API library for Node.js to interact with SQLite3 programmatically.
email-templates	Developer-friendly email framework to create, preview, and send custom emails (e.g. account notifications and more).
json-sql-enhanced	SQL query builder using Mongo-style syntax. This saves our development team time since we can continue to write in Mongo-style across the entire stack with a database agnostic approach. It also helps to avoid SQL injection attacks by using query parameters.
knex-schema-inspector	SQL utility to extract information about existing database schema. This allows us to easily validate that all indices, tables, columns, constraints, and more are valid and are `1:1` with how they should be. We even wrote automated helpers to add new columns and indexes if changes are made to database schemas (with extremely detailed error alerting too).
knex	SQL query builder which we use only for database migrations and schema validation through `knex-schema-inspector`.
mandarin	Automatic i18n phrase translation with support for Markdown using Google Cloud Translation API.
mx-connect	Node.js package to resolve and establish connections with MX servers and handle errors.
pm2	Node.js production process manager with built-in load balancer (fine-tuned for performance).
smtp-server	SMTP server library – we use this for our mail exchange ("MX") and outbound SMTP servers.
ImapTest	Useful tool for testing IMAP servers against benchmarks and RFC specification IMAP protocol compatibility. This project was created by the Dovecot team (an active open-source IMAP and POP3 server from July 2002). We extensively tested our IMAP server with this tool.

You can find other projects we use in our source code on GitHub.

Providers

Provider	Purpose
Cloudflare	DNS provider, health checks, load balancers, and backup storage using Cloudflare R2.
Digital Ocean	Dedicated server hosting and managed databases.
Vultr	Dedicated server hosting.
DataPacket	Dedicated server hosting.

Thoughts

Principles

Forward Email is designed according to these principles:

Always be developer-friendly, security and privacy-focused, and transparent.
Adhere to MVC, Unix, KISS, DRY, YAGNI, Twelve Factor, Occam's razor, and dogfooding
Target the scrappy, bootstrapped, and ramen-profitable developer

Experiments

tldr; Ultimately using S3-compatible object storage and/or Virtual Tables are not technically feasible for performance reasons and prone to error due to memory limitations.

We have done a few experiments leading up to our final SQLite solution as discussed above.

One of these was to try using rclone and SQLite together with an S3-compatible storage layer.

That experiment led us to further understand and discover edge cases surrounding rclone, SQLite, and VFS usage:

If you enable --vfs-cache-mode writes flag with rclone, then reads will be OK, however writes will get cached.
- If you have multiple IMAP servers distributed globally, then the cache will be off across them unless you have a single writer and multiple listeners (e.g. a pub/sub approach).
- This is incredibly complex and adding any additional complexity like this will result in more single points of failure.
- S3-compatible storage providers do not support partial file changes – which means any change of the .sqlite file will result in a complete change and re-upload of the database.
- Other solutions like rsync exist, but they are not focused on write-ahead-log ("WAL") support – so we ended up reviewing Litestream. Fortunately our encryption usage already encrypts the WAL files for us, so we do not need to rely on Litestream for that. However we weren't yet confident in Litestream for production-use and have a few notes below on that.
- Using this option of --vfs-cache-mode writes (the only way to use SQLite over rclone for writes) will attempt to copy the entire database from scratch in-memory – handling one 10 GB mailbox is OK, however handling multiple mailboxes with exceedingly high storage will cause the IMAP servers to run into memory limitations and ENOMEM errors, segmentation faults, and data corruption.
If you attempt to use SQLite Virtual Tables (e.g. using s3db) in order to have data live on an S3-compatible storage layer, then you will run into several more issues:
- Read and writes will be extremely slow as S3 API endpoints will need to be hit with HTTP GET, PUT, HEAD, and POST methods.
- Development tests showed that exceeding 500K-1M+ records on fiber internet is still limited by the throughput of writing and reading to S3-compatible providers. For example, our developers ran for loops to do both sequential SQL INSERT statements and ones that bulk wrote large amounts of data. In both cases the performance was staggeringly slow.
- Virtual tables cannot have indexes, ALTER TABLE statements, and other limitations – which leads to delays upwards of 1-2 minutes or more depending on the amount of data.
- Objects were stored unencrypted and no native encryption support is readily available.
We also explored using sqlite-s3vfs which is similar conceptually and technically to the previous bullet point (so it has the same issues). A possibility would be to use a custom sqlite3 build wrapped with encryption such as wxSQLite3 (which we currently use in our solution above) through editing the setup file.
Another potential approach was to use the multiplex extension, however this has a limitation of 32 GB and would require complex building and development headaches.
ALTER TABLE statements are required (so this completely rules out using Virtual Tables). We need ALTER TABLE statements in order for our hook with knex-schema-inspector to work properly – which ensures that data is not corrupted and rows retrieved can be converted to valid documents according to our mongoose schema definitions (which includes constraint, variable type, and arbitrary data validation).
Almost all of the S3-compatible projects related to SQLite in the open-source community are in Python (and not JavaScript which we use for 100% of our stack).
Compression libraries such as sqlite-zstd (see comments) look promising, but may not yet be ready for production usage. Instead application-side compression on data types such as String, Object, Map, Array, Set, and Buffer is going to be a cleaner and easier approach (and is easier to migrate too, since we could store a Boolean flag or column – or even use PRAGMA user_version=1 for compression or user_version=0 for no compression as database metadata).
- Fortunately we already have attachment de-duplication implemented in our IMAP server storage – therefore every message with the same attachment won't keep a copy of the attachment – instead a single attachment is stored for multiple messages and threads in a mailbox (and a foreign reference is subsequently used).
The project Litestream, which is a SQLite replication and backup solution is very promising and we will most likely use it in the future.
- Not to discredit the author(s) – because we love their work and contributions to open-source for well over a decade now – however from real-world usage it appears that there may be a lot of headaches and potential data loss from usage.
Backup restoration needs to be frictionless and trivial. Using a solution such as MongoDB with mongodump and mongoexport is not only tedious, but time intensive and has configuration complexity.
- SQLite databases make it simple (it's a single file).
- We wanted to design a solution where users could take their mailbox and leave at any moment.
  - Simple Node.js commands to fs.unlink('mailbox.sqlite')) and it's permanently erased from disk storage.
  - We can similarly use an S3-compatible API with HTTP DELETE to easily remove snapshots and backups for users.
- SQLite was the simplest, fastest, and most cost-effective solution.

Lack of alternatives

To our knowledge, no other email services are designed this way nor are they open-source.

We think this might be due to existing email services having legacy technology in production with spaghetti code 🍝.

Most if not all of existing email service providers are either closed-source or advertise as open-source, but in reality only their front-end is open-source.

The most sensitive part of email (the actual storage/IMAP/SMTP interaction) is all done on the back-end (server), and not on the front-end (client).