Skip to content
notes.vinhnq.com

Search

SecurityMid17 min read

Encoding, hashing, encryption and signatures: picking the right crypto tool as a backend engineer

Review 16 cards
On this page

Backend security series: 1. Encoding, hashing, encryption and signatures · 2. Storing passwords and encrypting personal data · 3. Web attacks and the OWASP Top 10 · 4. Authentication: sessions, cookies and JWT · 5. Authorization, SSO and OAuth2

Many security holes start with a sentence that uses the wrong word: “passwords are encrypted with MD5”, “the JWT is encrypted, so it’s safe”, “we store phone numbers in Base64”. After this post you’ll pick the right tool, know how far TLS reaches, and know how to keep secrets. The two case studies that apply these tools (storing passwords, encrypting personal data) are in part 2; the code uses only JDK 21 and has been run.

1. Four questions, four tools

Key idea: ask “what am I trying to guarantee?”; each answer maps to one tool.

Question Tool Key? Reversible?
Can another system read this format? Encoding (Base64, URL encoding, JSON, serialization) No Yes, by anyone
Has the data changed? (no attacker involved) Hash (SHA-256, BLAKE2) No No
Has the data been tampered with and did it come from whoever holds the secret? MAC (HMAC-SHA256/512) Symmetric key No
Same, but the verifier must not be able to forge signatures Digital signature (Ed25519, ECDSA, RSA-PSS) Key pair No
Can outsiders read the content? Encryption (AES-GCM, ChaCha20-Poly1305; RSA/ECDH for key exchange) Yes Yes, for the key holder

1.1. Encoding: a format change that protects nothing

Key idea: encoding turns data into a format another system can consume, with no key. Anyone who knows the algorithm (that is, everyone) can reverse it.

Like a fruit with three names in three regions: the name changes, the fruit doesn’t. Examples: Base64 to put bytes into JSON or a header, URL encoding for special characters in a query string, serialization to turn an object into bytes and back.

1.2. Hashing: a fixed-length fingerprint

Key idea: a hash maps any input to a fixed-length value, one way. An unkeyed hash only catches accidents; whoever can modify the data can recompute the hash.

SHA-256 always produces 32 bytes, whether the input is one character or 1 MB (verified). Plain hashes fit: a file checksum published over a trusted channel, de-duplicating by content, a hash index in a database, a cache key.

The three properties of a cryptographic hash and why MD5 and SHA-1 fail
  • One-way (preimage resistance): given a digest, you can’t find an input.
  • Second-preimage resistance: given one input, you can’t find a different input with the same digest.
  • Collision resistance: you can’t find any two inputs with the same digest.

Collisions always exist mathematically (infinite inputs, finite outputs); what matters is that nobody can find one in practice. MD5 and SHA-1 have lost that property: MD5 collisions take seconds, and in 2017 the SHAttered team published two different PDF files with the same SHA-1.

1.3. Encryption: secrecy, reversible for whoever holds the key

Key idea: encryption turns plaintext into ciphertext with a key; only the holder of the decryption key reads it back. Symmetric is for the data, asymmetric is for exchanging keys and signing.

  • Symmetric (AES, ChaCha20): one key encrypts and decrypts. Very fast (CPUs have AES-NI instructions, several GB/s).
  • Asymmetric (RSA, elliptic-curve schemes): public and private keys differ. Hundreds to thousands of times slower, not used for bulk data.

“Use AES-256” is only half the answer; the other half is the mode. Use AES-GCM or ChaCha20-Poly1305: authenticated encryption (AEAD), which hides the content and detects tampering. ECB leaks patterns (identical blocks give identical ciphertext), and CBC without a MAC is open to padding oracles. Cipher.getInstance("AES") defaults to AES/ECB/PKCS5Padding, so always spell out AES/GCM/NoPadding.

1.4. MACs and digital signatures: integrity plus authenticity

Key idea: to prove “not modified and from the right sender”, use a keyed hash (HMAC) when both sides share a secret, or a digital signature when one party signs and many verify.

The course’s case study: a merchant sends a payment request (nothing secret in it) to a gateway and needs integrity and authenticity. The request passes through the customer’s browser (the URL is editable), so HTTPS isn’t enough. Encryption is overkill and a plain hash falls short (whoever edits amount recomputes the hash).

The answer is HMAC: merchant and gateway share a secret, and the request carries HMAC-SHA512(payload, secret). Without the secret, nobody can produce a valid signature for a modified payload. Signing a VNPay-style URL is covered in the payment gateway post.

static String hmacHex(byte[] key, String data) throws Exception {
    Mac mac = Mac.getInstance("HmacSHA512");
    mac.init(new SecretKeySpec(key, "HmacSHA512"));
    return HexFormat.of().formatHex(mac.doFinal(data.getBytes(StandardCharsets.UTF_8)));
}

static boolean verifyHmac(byte[] key, String data, String sigHex) throws Exception {
    byte[] expected = HexFormat.of().parseHex(hmacHex(key, data));
    byte[] given;
    try { given = HexFormat.of().parseHex(sigHex); } catch (IllegalArgumentException e) { return false; }
    return MessageDigest.isEqual(expected, given);   // constant-time comparison
}
// original payload -> true, payload with a modified amount -> false (verified)

A digital signature uses a key pair: the signer holds the private key, anyone with the public key can verify, but only the signer can produce a signature. The flow: hash the data, sign it with the private key, send the signature along; the receiver recomputes the hash and verifies with the public key.

KeyPair kp = KeyPairGenerator.getInstance("Ed25519").generateKeyPair();
Signature signer = Signature.getInstance("Ed25519");
signer.initSign(kp.getPrivate());
signer.update(payload.getBytes(StandardCharsets.UTF_8));
byte[] sig = signer.sign();                       // 64 bytes

Signature verifier = Signature.getInstance("Ed25519");
verifier.initVerify(kp.getPublic());
verifier.update(payload.getBytes(StandardCharsets.UTF_8));
boolean ok = verifier.verify(sig);                // true (verified)
Signing is not “encrypting the hash”, and the corrected comparison table

1.5. The comparison table, corrected

The slide has an encryption-versus-hashing table. This version keeps the main idea and fixes a few cells:

Encryption Hash MAC Digital signature
Purpose Confidentiality Integrity (no adversary) Integrity + origin Integrity + origin + non-repudiation
Reversible Yes, with the key No No No
Key Symmetric or key pair None Shared secret Key pair
Output length Plaintext + nonce + tag Fixed Fixed Fixed per algorithm
Cost Symmetric very cheap, asymmetric expensive Cheap Cheap More expensive than HMAC
Examples National ID column, backup files, TLS Hash index, dedup, checksum Signing gateway requests, webhooks, JWT HS256 JWT RS256/ES256, TLS certificates, package signing

“Passwords” are deliberately missing from the table: storing passwords needs its own family of functions, see part 2.

2. TLS on one page

Key idea: TLS uses signatures and certificates to know who’s on the other end, ECDHE to agree on a session key, then AES-GCM to encrypt. It only protects each connection hop.

sequenceDiagram
    participant C as Client
    participant S as Server
    C->>S: ClientHello, client ephemeral ECDHE share
    S->>C: ServerHello, server ECDHE share, certificate, signature
    Note over C: Validate CA chain, host name, dates, then derive session key
    C->>S: Finished
    C->>S: Application data encrypted with AES-GCM

TLS doesn’t cover: data after the load balancer (TLS usually terminates there, and the next hop may be plain HTTP), data passing through the user’s browser (they can edit it), data sitting in logs. That’s why a payment request still needs an HMAC over HTTPS: not because HTTPS is “unsafe”, but because it doesn’t cover the hop through the client. Man-in-the-middle attacks, mTLS between backends and replay protection are in part 3.

The four steps of a TLS 1.3 handshake and Java 21’s crypto support
  1. The client sends a ClientHello with the algorithms it supports and an ephemeral ECDHE key share.
  2. The server returns its own ECDHE share, its certificate (the server’s public key, signed by a CA) and a signature proving it holds the matching private key.
  3. The client validates the certificate chain up to a root CA it trusts, checks the host name and the validity dates; both sides derive the same session key from ECDHE.
  4. Data flows encrypted with AES-GCM or ChaCha20-Poly1305 under the session key. TLS 1.3 does all of this in one round trip.

3. Secrets: keys, DB passwords, API keys

Key idea: if secrets leak into Git, everything above is pointless. Load secrets from outside the code, rotate them without downtime, and let each service see only its own.

  • Never commit secrets, not even to a private repo. Turn on secret scanning (gitleaks, GitHub secret scanning) in pre-commit hooks and CI. If one slips in, treat it as leaked: rotate first, clean Git history second.
  • Load from the environment or a secret manager: environment variables per 12-factor are the minimum; better is Vault, AWS Secrets Manager or Kubernetes Secrets (with etcd encryption on), loaded via Spring Cloud Vault or spring.config.import.
  • Rotate without downtime: accept the old and new key side by side for a while (for example, verify a webhook HMAC against both secrets); every ciphertext or token carries a key_id.
  • Least privilege: one identity per service, each reading only its own secrets; the KMS logs every decryption.
  • Mask logs: never log the Authorization header, cookies, secrets or personal data; mask in the logback layout rather than at every call site.

Exercises

Exercise 1: Pick the tool

For each scenario choose encoding, hash, MAC, digital signature, encryption or a password-hashing function, and justify briefly: (a) storing user passwords; (b) sending an image file inside JSON; (c) checking that a webhook really came from a partner; (d) storing a national ID number and occasionally showing it back to its owner; (e) an identity provider issuing tokens that 30 services verify; (f) detecting duplicate image uploads; (g) putting an order code in a URL.

Solution to Exercise 1
  • (a) Argon2id/bcrypt: one-way, slow, salted.
  • (b) Base64, it’s just a format.
  • (c) HMAC with a secret specific to that partner, constant-time comparison, plus a timestamp against replays.
  • (d) AES-GCM at the application layer with envelope encryption and AAD bound to the customer; if you need to search by ID number, add a blind index.
  • (e) A digital signature (RS256/ES256/EdDSA): the 30 services only need the public key via JWKS, and none of them can forge a token. With HS256 all 30 services hold the secret and any of them could mint tokens.
  • (f) SHA-256 of the content (no meaningful adversary; even if someone tries to craft collisions, SHA-256 holds).
  • (g) URL encoding; if you don’t want to reveal sequential order numbers, use a random code rather than “encrypting the ID”.

Remember

  • Explain why Base64 and a plain hash don’t protect data against an attacker.
  • Know when to pick an HMAC and when a digital signature, and why HMAC gives no non-repudiation.
  • Explain why sha256(secret + message) is wrong and why signature comparison must be constant-time.
  • Know which hops TLS covers and why a payment request still needs signing over HTTPS.
  • Describe how to rotate a secret without downtime using a key_id.

Interview questions

Junior

How do encoding, hashing and encryption differ?

Encoding changes the format, hashing makes a one-way fingerprint, encryption keeps things secret with a key.

Encoding has no key and anyone can reverse it (Base64, URL encoding), so it has no security value. Hashing produces a fixed-length value, used for integrity checks, hash indexes, dedup. Encryption needs a key and is reversible for whoever holds it. To prove origin you add a MAC or a digital signature.

Does Base64 protect data? Is a JWT payload secret?

No, anyone can read both.

Base64 is an encoding with no key, decoded with one command. An ordinary JWT payload (a JWS) is only Base64url-encoded; the signature prevents tampering, not reading. Don’t put sensitive data in the payload; if it must be hidden, encrypt it (JWE) or use an opaque reference token.

Mid

How is an HMAC different from a plain hash, and why not use sha256(secret + message)?

An HMAC is a keyed hash, so it also proves origin; concatenating the secret and the message yourself is open to length extension.

With a plain hash, whoever modifies the data simply recomputes it. With sha256(secret || message) on SHA-2, one valid signature lets an attacker compute a signature for an extended message without the secret. HMAC’s two-pass construction avoids that. Compare signatures in constant time.

When do you use an HMAC and when a digital signature?

HMAC when two parties trust each other and share a secret; a signature when one party issues and many verify.

HMAC is fast and simple, a good fit for signing payment gateway requests or webhooks; but the receiver can also produce signatures, so there’s no non-repudiation, and each partner needs its own secret. With a signature only the issuer holds the private key and the public key is published via JWKS, as with identity provider tokens or TLS certificates.

Is symmetric or asymmetric encryption more secure?

Neither family is inherently “more secure”; strength depends on key size and correct use (AES-256 is on par with or stronger than RSA-3072).

The differences are key distribution and speed: symmetric is fast but both sides must share a secret; asymmetric is slow but half the key can be public. In practice it’s hybrid: TLS agrees on a session key with ECDHE and then encrypts the data with AES-GCM.

Senior

Why does a payment request still need an HMAC when we already use HTTPS?

Because HTTPS protects each hop, while the request passes through the customer’s browser, where the user can edit the URL.

The request goes from the backend, through the browser, and only then to the gateway; TLS also terminates at load balancers and proxies. An HMAC over the payload gives end-to-end integrity and origin regardless of the path. It doesn’t stop replays: that needs a unique transaction reference, an expiry and a state machine on the receiving side.

What does TLS protect, and what doesn’t it?

TLS gives confidentiality and integrity on one hop and authenticates the server through a certificate; it doesn’t cover data after the termination point, in the browser, or at rest.

It also authenticates the client with mTLS. The termination point is the load balancer or proxy, and then come the logs. It’s worthless if the client disables certificate or host name validation: the connection is still encrypted, just with whoever sits in the middle. For end-to-end integrity, sign the payload (HMAC, digital signature).

How do you design rotation of a webhook signing secret or an encryption key without downtime?

Put a key_id on every signature, token and ciphertext, and run a transition period that accepts both the old and the new key.

  • The verifier accepts both keys, the signer switches to the new one, you wait until data signed with the old key expires (or re-encrypt it), then remove the old key.
  • Secrets live in a secret manager or KMS and can be loaded dynamically or reloaded; each service reads only its own.
  • If a secret leaks, rotate first and investigate second.
  • For large encrypted datasets, envelope encryption means rotating the KEK only re-encrypts the DEKs.

Useful phrases for interviews

  • “Base64 is an encoding, not encryption; anyone can reverse it.”
  • “A plain hash only protects integrity when there’s no attacker; once someone can tamper with the data, I need an HMAC or a signature.”
  • “We sign the payload with HMAC-SHA512 and compare signatures in constant time.”
  • “Symmetric versus asymmetric isn’t about strength, it’s about key distribution; in practice we use both, the way TLS does.”
  • “If there’s one issuer and many verifiers, I’d go for signatures and publish the public keys through JWKS.”
  • “TLS protects the hop, not the payload end to end, so anything that passes through the browser still needs to be signed.”
  • “Disabling certificate validation keeps the encryption but throws away the authentication, which is the whole point.”
  • “Every signature and ciphertext carries a key id, so we can rotate keys without downtime.”

Notes based on the Advanced Backend course (Ronin Engineer), rewritten in my own words.