Problem Statement
Applications handling sensitive documents — contracts, identification files, medical records, financial statements — often grant broad storage access to backend services, exposing credentials in configuration and lacking file-level access control. Direct object storage URLs are permanent and shareable without consent. The goal was a service providing fine-grained, time-limited, audited file access that any application could use without managing storage credentials or implementing access logic independently.
Key Challenges:
- Time-limited file access without exposing permanent storage credentials
- File-level access policies supporting owner, group, and role-based rules
- Tamper-evident audit log for compliance and forensic review
- Encryption at rest and in transit for sensitive document categories
- Scalable URL generation without becoming a performance bottleneck
System Architecture
The service sits between applications and the underlying object storage. Upload and download requests pass through an access policy check and are serviced via short-lived signed URLs generated by the service. Every access attempt is logged with full context regardless of outcome.
Signed URL Generation
Applications request a time-limited signed URL for a specific file and operation (read/write/delete). The service validates the requester's access policy and generates a pre-signed object storage URL valid for a configurable duration, never exposing raw credentials to the application.
Access Policy Engine
File-level policies define who can access each file: owner rules, group membership, role-based permissions, and tenant isolation. Policies are evaluated at URL generation time with deny-by-default semantics — no policy match means no access.
Encryption Layer
Files in designated sensitive categories are encrypted before storage using per-file keys managed by a key management service. Decryption occurs during download URL generation, ensuring encrypted storage without decryption complexity in application code.
Audit Logging
Every access event — URL generation, file upload, download, deletion, and access denial — is appended to an append-only audit log with requester identity, timestamp, file metadata, and outcome. The log is replicated and tamper-evident for compliance requirements.
Key Engineering Challenges
Credential Isolation
Challenge: Applications need file access without holding long-lived storage credentials that could be leaked or misused.
Solution: All storage credentials are held exclusively by the service. Applications receive only short-lived signed URLs anchored to specific files and operations, expiring after the configured window.
URL Revocation
Challenge: Once a signed URL is issued, it can be used until expiry — revoking access requires invalidating the URL before it expires.
Solution: URL token registry in Redis with revocation support. Signed URLs include a token ID that is checked against the revocation list on first use; revoked tokens are rejected even within their validity window.
Per-File Encryption Key Management
Challenge: Encrypting each file with a unique key requires secure key storage and efficient retrieval at download time.
Solution: Envelope encryption — file is encrypted with a data key; the data key is encrypted with a master key held in the key management service. Only the encrypted data key is stored with the file metadata.
Audit Log Integrity
Challenge: An audit log that can be deleted or modified is insufficient for compliance purposes.
Solution: Append-only log table with deletion denied at the database level, combined with periodic cryptographic checksum chaining linking each log entry to its predecessor — making tampering detectable.
Solutions Implemented
- Signed URL Service: Time-limited, operation-scoped URLs generated from storage credentials held exclusively by the service — never exposed to applications.
- Access Policy Engine: Deny-by-default file-level access evaluation supporting owner, group, role, and tenant-based rules.
- Envelope Encryption: Per-file data keys secured by a master key, enabling file-level encryption without credential management complexity in consuming applications.
- Revocable URL Tokens: Redis token registry enabling URL invalidation before natural expiry for immediate access revocation.
- Tamper-Evident Audit Log: Append-only access log with cryptographic chaining for compliance-grade integrity guarantees.
Outcome & Impact
Apps never hold storage credentials
Every attempt audited
Granular policy per document
Per-file key management