Frontend
Why You Shouldn’t Store Large Files in Your Database: Database vs. Amazon S3
Ammar Eyad Dev.to (EN Zone)
1 views
Storing files in a database instead of S3 introduces two massive architectural bottlenecks as your application scales:
1. CDN becomes much harder to use
One of the biggest advantages of S3 is how easily it works with a CDN such as Amazon CloudFront or Cloudflare.
Imagine you have millions of users requesting the same images.
With S3 and a CDN, the CDN can cache those images at locations around the world. Users can get the file from a nearby edge location instead of requesting it from your application every time.
This is exactly what we want for things like:
Images
Videos
PDFs
JavaScript
CSS
Other static content
Now imagine the image is stored as a BLOB inside your database.
If your images are locked in a database, your CDN is effectively blind to them. Every single image request must hit your web server, which must then execute a heavy database query to fetch the binary data, stream it back to the web server, and then serve it to the client. You are forcing your most expensive piece of infrastructure (the database) to act as a dumb file server, completely bypassing the speed and cost benefits of edge caching.
2. Database replication becomes heavier
Production databases usually have replicas for high availability.
If your database contains mostly structured data, replication is relatively straightforward.
But imagine your database contains hundreds of gigabytes or even terabytes of images and videos.
Now those large binary objects also become part of your database replication and backup workload.
For example, instead of replicating:
Users
Orders
Products
Payments
you might also be replicating:
500 GB of images
1 TB of videos
200 GB of documents
That can make replication, backups, and restores significantly heavier.
The same applies when you need to restore your database.
A database backup containing huge amounts of binary data can take much longer to restore than a database containing only structured application data.
3. Your database becomes unnecessarily large
Databases are extremely good at what they are designed for.
They are great for:
Relationships
Transactions
Queries
Indexes
Structured data
Consistency
But storing large files is a different problem.
Object storage is designed specifically for storing large objects.
So instead of making the database responsible for everything, we can separate the responsibilities.
The database stores:
User
Order
Payment
File metadata
File reference
S3 stores:
Images
Videos
PDFs
Documents
Each system does what it is good at.
What about the physical storage behind S3?
Another interesting question is:
Where does S3 actually store my file?
At the end of the day, the data has to live on physical storage hardware.
But as an S3 user, you don't need to know whether your particular object is sitting on an HDD, SSD, or which physical disk contains it.
AWS manages that infrastructure for you.
You interact with S3 as object storage:
my-bucket/users/123/profile.jpg
You don't interact with:
Disk 123
Sector 456
SSD 789
That abstraction is one of the main benefits of object storage.
What does the typical architecture look like?
A common approach is:
Application
|
+--- Database
| File metadata
|
+--- S3
Actual file
For uploads, we can even use a presigned URL so the client uploads the file directly to S3 instead of sending the entire file through our application server.
The database then only needs to know things like the file name, size, content type, and S3 key.
Is storing BLOBs in a database always wrong?
No.
There are cases where it can be perfectly reasonable.
For example, if the files are very small and the application is simple, storing them in the database might be easier and completely acceptable.
The important thing is to understand the trade-off.
For a large-scale system, I generally prefer this simple rule:
Database for structured data and metadata.
S3/object storage for large files.
It keeps the architecture cleaner, allows the file storage to scale independently, makes CDN integration easier, and avoids turning your primary database into a file server.
Read original: https://dev.to/ammar_eyad/why-you-shouldnt-store-large-files-in-your-database-database-vs-amazon-s3-hec
← Previous
Stop Paying Full Reboot Downtime: Practical systemd soft-reboot on Linux
Next →
Ask canvas for a WebP in Safari and you silently get a PNG
Related
I built a compiler so I could stop writing custom element boilerplate
Frontend
2
DEV Community
RepoRoad: A Cosy Lofi Drive That Puts Open Source on the Map
Frontend
0
DEV Community
Designing a 5-band parametric EQ from the biquad up, in MATLAB
Frontend
8
Dev.to (EN Zone)
WanderJournal: A digital travelling Journal
Frontend
4
Dev.to (EN Zone)
Comments0
No comments yet — be the first