Skip to main content

Encrypt a field with Python

By the end of this page you have a working PyMongo script that encrypts ssn and still searches it, while the cluster holds nothing but ciphertext.

Before you start​

Read Queryable Encryption first, because the limits on encrypted fields shape what this code can and cannot do. Everything in this table has to be in place before the script will run.

ItemRequirement
ClusterMongoDB Enterprise 7.0 or later
Encryption modeAutomatic Encryption
Application hostcrypt_shared (mongo_crypt_v1.so) installed, at a version equal to or newer than the server
PythonPython 3.9 or later
Key managementA local key file is enough for a trial. Production needs a KMS (AWS, Azure, GCP, or KMIP)
note

crypt_shared runs on the machine that runs your application, not on the MongoDB server. Download the Enterprise package for your operating system and server version from the MongoDB Download Center, then extract it locally, for example to /opt/crypt_shared/lib/mongo_crypt_v1.so.

Install the driver​

pip install 'pymongo[encryption]'

Set the imports and connection details​

Everything below goes into one file

From here down, every Python block belongs to a single file, qe_demo.py, appended in the order shown. Create it now, and run it at the end with python qe_demo.py.

Take the hosts, replica set name, and credentials from the cluster's connection details in the console.

import os
from urllib.parse import quote_plus

from bson.codec_options import CodecOptions
from pymongo import MongoClient, ReturnDocument
from pymongo.encryption import AutoEncryptionOpts, ClientEncryption

HOSTS = "<HOST_1>:27017,<HOST_2>:27017,<HOST_3>:27017"
REPLICA_SET = "<REPLICA_SET_NAME>"
USER = "<USERNAME>"
PASSWORD = os.environ["MONGO_PASSWORD"]          # never hard-code the password
URI = (f"mongodb://{quote_plus(USER)}:{quote_plus(PASSWORD)}@{HOSTS}"
       f"/admin?replicaSet={REPLICA_SET}")       # quote_plus escapes special characters

DB, COLL = "qe_demo", "users"
KEY_VAULT_NS = "encryption.__keyVault"           # where Data Encryption Keys are stored
CRYPT_SHARED_PATH = "/opt/crypt_shared/lib/mongo_crypt_v1.so"

Create the master key​

The key file is generated once and reused on every run. For production, replace this block with a KMS provider.

if not os.path.exists("master_key.bin"):
    with open("master_key.bin", "wb") as f:
        f.write(os.urandom(96))
with open("master_key.bin", "rb") as f:
    kms_providers = {"local": {"key": f.read()}}

plain_client = MongoClient(URI)
client_encryption = ClientEncryption(kms_providers, KEY_VAULT_NS, plain_client, CodecOptions())
danger

Data encrypted with a lost master key cannot be recovered. Back master_key.bin up somewhere separate from the database before you put real data behind it.

Declare the encrypted fields​

This example declares ssn with queryType: equality, which covers the most common case of looking a record up by an exact value. keyId stays None here. The driver fills it in when it creates the collection.

encrypted_fields = {
    "fields": [
        {"path": "ssn", "bsonType": "string", "keyId": None,
         "queries": {"queryType": "equality"}},
    ]
}

Create the encrypted collection​

The collection must not already exist. This call also creates the enxcol_.users.esc and enxcol_.users.ecoc metadata collections on the server.

_, encrypted_fields = client_encryption.create_encrypted_collection(
    plain_client[DB], COLL, encrypted_fields, "local"
)  # returns encrypted_fields with keyId filled in

Build the encrypting client​

This has to come after the collection exists, because the field map now carries the key IDs.

opts = AutoEncryptionOpts(
    kms_providers,
    KEY_VAULT_NS,
    encrypted_fields_map={f"{DB}.{COLL}": encrypted_fields},
    crypt_shared_lib_path=CRYPT_SHARED_PATH,
    crypt_shared_lib_required=True,
)
encrypted_client = MongoClient(URI, auto_encryption_opts=opts)
users = encrypted_client[DB][COLL]

Read and write as usual​

Inserts need no special handling. The driver encrypts ssn before it leaves the process.

users.insert_one({"name": "An", "ssn": "123-45-6789"})
users.insert_many([
    {"name": "Binh", "ssn": "987-65-4321"},
    {"name": "Chi", "ssn": "111-22-3333"},
])

Queries on ssn work because the field was declared with queryType: equality.

print(users.find_one({"ssn": "123-45-6789"}))                       # decrypted automatically
print(list(users.find({"ssn": {"$in": ["987-65-4321", "111-22-3333"]}})))
print(users.count_documents({"ssn": "123-45-6789"}))

Updates to an encrypted field accept $set and $unset only, and find_one_and_update must return the document as it was before the change.

users.update_one({"name": "Binh"}, {"$set": {"ssn": "000-00-0001"}})
users.find_one_and_update(
    {"ssn": "000-00-0001"}, {"$set": {"name": "Binh (updated)"}},
    return_document=ReturnDocument.BEFORE,
)

Aggregation can match on an encrypted field, and deletes are one document at a time.

print(list(users.aggregate([
    {"$match": {"ssn": "123-45-6789"}},
    {"$project": {"_id": 0, "name": 1, "ssn": 1}},
])))

users.delete_one({"ssn": "111-22-3333"})

Confirm the cluster holds only ciphertext​

plain_client has no encryption options, so it sees the stored form.

raw = plain_client[DB][COLL].find_one({"name": "An"})
print("ssn via plain client:", type(raw["ssn"]),
      "| has __safeContent__:", "__safeContent__" in raw)
print("plaintext search via plain client:",
      plain_client[DB][COLL].find_one({"ssn": "123-45-6789"}))     # None

Running it prints:

ssn via plain client: <class 'bson.binary.Binary'> | has __safeContent__: True
plaintext search via plain client: None

ssn comes back as Binary subtype 6, the document carries __safeContent__, and the plaintext search returns None.

Clean up​

Dropping the collection with the plain client would leave the metadata collections behind, so use the encrypted client.

encrypted_client[DB].drop_collection(COLL)

Next steps​