Building a Production-Ready File Upload System with Node.js and AWS S3
Jul 26, 2026
Building a Production-Ready File Upload System with Node.js and AWS S3
A complete guide covering image uploads, multipart video uploads, orphan cleanup, tagging strategies, and frontend integration — built from real decisions.
Table of Contents
- The S3 Client Setup
- Choosing Your Upload Architecture
- Approach 2 — The Upload Route
- The Orphan Image Problem
- Deleting Images When a Blog is Deleted
- S3 Tagging — Let AWS Do the Cleanup
- Handling 2GB Video Uploads — Multipart Upload
- Frontend Integration
- Full Architecture Summary
1. The S3 Client Setup
Before anything else, you need a typed, safe S3 client. A common mistake is letting process.env values flow through as string | undefined — the AWS SDK will reject them at runtime. Fix this with a guard and proper type annotations.
import {
S3Client,
PutObjectCommand,
DeleteObjectCommand,
GetObjectCommand,
PutObjectTaggingCommand,
} from "@aws-sdk/client-s3"
import { getSignedUrl } from "@aws-sdk/s3-request-presigner"
import dotenv from 'dotenv'
dotenv.config()
const bucketName = process.env.AWS_BUCKET_NAME!
const region = process.env.AWS_BUCKET_REGION!
const accessKeyId = process.env.AWS_ACCESS_KEY!
const secretAccessKey = process.env.AWS_SECRET_ACCESS_KEY!
// Fail fast — surfaces misconfiguration at boot, not mid-request
if (!accessKeyId || !secretAccessKey) {
throw new Error("AWS credentials must be set in environment variables")
}
export const s3Client = new S3Client({
region,
credentials: { accessKeyId, secretAccessKey }
})
type FileStatus = 'PENDING' | 'COMPLETED' | 'DELETED'
export function uploadFile(fileBuffer: Buffer, fileName: string, mimetype: string) {
return s3Client.send(new PutObjectCommand({
Bucket: bucketName,
Body: fileBuffer,
Key: fileName,
ContentType: mimetype,
Tagging: 'status=PENDING' // Tag on upload — no extra API call needed
}))
}
export async function updateFileTag(fileName: string, status: FileStatus) {
return s3Client.send(new PutObjectTaggingCommand({
Bucket: bucketName,
Key: fileName,
Tagging: { TagSet: [{ Key: 'status', Value: status }] }
}))
}
export function deleteFile(fileName: string) {
return s3Client.send(new DeleteObjectCommand({
Bucket: bucketName,
Key: fileName,
}))
}
export async function getObjectSignedUrl(key: string) {
const command = new GetObjectCommand({ Bucket: bucketName, Key: key })
return getSignedUrl(s3Client, command, { expiresIn: 60 })
}Key fixes over a naive setup:
- Runtime guard on credentials — narrows
string | undefinedtostring fileBuffer: Buffer,fileName: string,mimetype: string— no implicitanyTagging: 'status=PENDING'baked intouploadFileso every upload is tracked from the first moment
2. Choosing Your Upload Architecture
When building a blog platform with image uploads, you have three main approaches:
Approach 1 — Presigned URL (Frontend uploads directly to S3)
─────────────────────────────────────────────────────────────
Frontend ──► GET /presigned-url ──► Backend
Backend ──► { url, signature, config }
Frontend ──► Upload directly to S3
Frontend ──► POST /create-blog { content, imageUrl }
Approach 2 — Upload First, Then Create
───────────────────────────────────────
Frontend ──► POST /upload ──► Backend ──► S3
Backend ──► { imageUrl }
Frontend ──► POST /create-blog { content, imageUrl }
Approach 3 — Single Request
────────────────────────────
Frontend ──► POST /create-blog { file + content } ──► Backend ──► S3Which to Use When?
| Scenario | Best Approach | Why |
|---|---|---|
| Fast blog upload | Approach 1 | File bypasses your server entirely |
| Secure upload | Approach 2 | Backend validates, scans, and controls before S3 |
| 2GB video upload | Approach 1 or Multipart | Routing 2GB through Express will time out |
| Simplest to build | Approach 3 | One request, one handler |
For a blog platform, Approach 2 hits the right balance — your server validates the file (type, size, malware), resizes it with Sharp, and only then sends it to S3. You stay in control.
3. Approach 2 — The Upload Route
Multer is required because Express cannot parse multipart/form-data natively. Without it, req.file is always undefined.
Express natively handles:
application/json → express.json()
application/x-www-urlencoded → express.urlencoded()
Express cannot handle:
multipart/form-data → needs Multer// src/routes/upload.ts
import { Router, Request, Response } from 'express'
import multer from 'multer'
import sharp from 'sharp'
import { v4 as uuidv4 } from 'uuid'
import { uploadFile, getObjectSignedUrl } from '../s3'
import { db } from '../db'
const router = Router()
const upload = multer({
storage: multer.memoryStorage(),
limits: { fileSize: 10 * 1024 * 1024 }, // 10MB max
fileFilter: (req, file, cb) => {
if (!file.mimetype.startsWith('image/')) {
return cb(new Error('Only image files are allowed'))
}
cb(null, true)
}
})
router.post('/image', upload.single('file'), async (req: Request, res: Response) => {
try {
const file = req.file
if (!file) return res.status(400).json({ message: 'No file provided' })
const ext = file.originalname.split('.').pop()
const imageKey = `images/${uuidv4()}-${Date.now()}.${ext}`
// Resize before upload
const fileBuffer = await sharp(file.buffer)
.resize({ height: 1920, width: 1080, fit: 'contain' })
.toBuffer()
// uploadFile sets tag to PENDING automatically
await uploadFile(fileBuffer, imageKey, file.mimetype)
const imageUrl = await getObjectSignedUrl(imageKey)
return res.status(200).json({ imageKey, imageUrl })
} catch (error) {
console.error('Image upload error:', error)
return res.status(500).json({ message: 'Failed to upload image' })
}
})
export default routerIn Postman: set Body → form-data, add a key namedfile, change its type to File, then select your image. The key name must matchupload.single('file').
4. The Orphan Image Problem
Here's a subtle but costly bug that bites every blog platform eventually:
Upload Image ──► ✅ Saved to S3
Create Blog ──► ❌ Failed (validation error, network blip, anything)
Result: Image sits in S3 forever, unlinked, costing money.
Second attempt: another image uploaded, another orphan if blog fails again.
Third attempt: blog succeeds — but 2 orphaned images remain.The Fix — Track Status in Your DB
When the image is uploaded, create a PENDING record in your database. Only promote it to LINKED when the blog is successfully created.
// POST /image — track as PENDING immediately
router.post('/image', upload.single('file'), async (req, res) => {
// ... upload to S3 ...
await db.image.create({
data: {
key: imageKey,
status: 'PENDING',
createdAt: new Date()
}
})
return res.json({ imageKey, imageUrl })
})
// POST /blog — link image on success
router.post('/blog', async (req: Request, res: Response) => {
const { title, content, imageKey } = req.body
const blog = await db.blog.create({ data: { title, content, imageKey } })
// Promote: image is now linked
await db.image.update({
where: { key: imageKey },
data: { status: 'LINKED', blogId: blog.id }
})
// Update S3 tag too
await updateFileTag(imageKey, 'COMPLETED')
return res.status(201).json(blog)
})Images stuck in PENDING for over 24 hours are orphans — clean them up automatically (more on this in Section 6).
5. Deleting Images When a Blog is Deleted
When a user deletes a blog, you have three options for the associated image:
Option A — Delete Immediately ❌
// Dangerous — if S3 delete fails, your DB record is already gone
await db.blog.delete({ where: { id } })
await deleteFile(blog.imageKey) // what if this throws?Option B — Soft Delete + Scheduled Cleanup ✅
Don't delete the image immediately. Mark the blog as deleted and let a background job handle the actual removal after a grace period. This also gives you a 30-day accidental-delete recovery window.
// DELETE /blog/:id — soft delete only
router.delete('/blog/:id', async (req: Request, res: Response) => {
const blog = await db.blog.findUnique({ where: { id: req.params.id } })
await db.blog.update({
where: { id: req.params.id },
data: { deletedAt: new Date(), status: 'DELETED' }
})
// Update S3 tag — lifecycle policy will handle actual deletion
await updateFileTag(blog!.imageKey, 'DELETED')
return res.json({ message: 'Blog deleted' })
})Option C — S3 Lifecycle Policy (Best — no cron needed)
Set rules directly in your S3 bucket. AWS runs them automatically every day.
6. S3 Tagging — Let AWS Do the Cleanup
This is the cleanest production approach. Instead of running your own cron jobs, you tag objects in S3 and let AWS Lifecycle Policies act on those tags.
The Tag Lifecycle
Image uploaded → tag: status=PENDING
Blog created → tag: status=COMPLETED
Blog deleted → tag: status=DELETEDLifecycle Rules
Rule 1 — Orphan Cleanup
Filter : tag status = PENDING
Action : Delete after 1 day
Catches : images uploaded but blog never created
Rule 2 — Deleted Blog Cleanup
Filter : tag status = DELETED
Action : Delete after 30 days
Gives : 30-day accidental-delete recovery window
Rule 3 — Archive Old Content (optional)
Filter : tag status = COMPLETED
Action : Move to S3 Glacier after 1 year
Saves : ~80% on storage cost for old blogsSet these via code:
import { PutBucketLifecycleConfigurationCommand } from '@aws-sdk/client-s3'
await s3Client.send(new PutBucketLifecycleConfigurationCommand({
Bucket: bucketName,
LifecycleConfiguration: {
Rules: [
{
ID: 'delete-pending-orphans',
Status: 'Enabled',
Filter: { Tag: { Key: 'status', Value: 'PENDING' } },
Expiration: { Days: 1 }
},
{
ID: 'delete-soft-deleted-blogs',
Status: 'Enabled',
Filter: { Tag: { Key: 'status', Value: 'DELETED' } },
Expiration: { Days: 30 }
},
{
ID: 'archive-completed-blogs',
Status: 'Enabled',
Filter: { Tag: { Key: 'status', Value: 'COMPLETED' } },
Transitions: [{ Days: 365, StorageClass: 'GLACIER' }]
}
]
}
}))Tagging vs Cron — Comparison
| S3 Lifecycle + Tags | Custom Cron Job | |
|---|---|---|
| Infra to maintain | ✅ None | ❌ Need a scheduler |
| Guaranteed cleanup | ✅ AWS manages it | ⚠️ Cron can fail silently |
| Granularity | ⚠️ Day-level only | ✅ Any interval |
| Accidental delete recovery | ✅ 30-day window | ✅ Possible |
| Cost | ✅ Free | ⚠️ Server cost |
Caveat: S3 Lifecycle runs once per day — a DELETED object won't disappear instantly, but within 24 hours of the 30-day mark. For a blog platform this is perfectly acceptable.7. Handling 2GB Video Uploads — Multipart Upload
Routing a 2GB file through Express will time out, consume all available memory, and block every other request on the server. The solution is S3 Multipart Upload — split the file into chunks on the client, send each chunk separately, and let S3 assemble them.
How It Works
Client Backend S3
│ │ │
├── POST /multipart/init ──► │
│ ├── CreateMultipartUpload ───►│
│ │◄── uploadId ───────────────┤
│◄── { uploadId, key } ───┤ │
│ │ │
├── POST /multipart/chunk ─► (chunk 1, 10MB) │
│ ├── UploadPart partNumber=1 ─►│
│ │◄── ETag 1 ─────────────────┤
│◄── { ETag, partNumber } ┤ │
│ │ │
│ ... repeat per chunk ... │
│ │ │
├── POST /multipart/complete► │
│ { uploadId, parts } ──► │
│ ├── CompleteMultipartUpload ─►│
│ │◄── final URL ──────────────┤
│◄── { videoKey } ────────┤ │Backend Routes
// src/routes/multipart.ts
import { Router, Request, Response } from 'express'
import multer from 'multer'
import {
CreateMultipartUploadCommand,
UploadPartCommand,
CompleteMultipartUploadCommand,
AbortMultipartUploadCommand
} from '@aws-sdk/client-s3'
import { v4 as uuidv4 } from 'uuid'
import { s3Client } from '../s3'
import { db } from '../db'
const router = Router()
const upload = multer({
storage: multer.memoryStorage(),
limits: { fileSize: 10 * 1024 * 1024 } // 10MB per chunk
})
const bucketName = process.env.AWS_BUCKET_NAME!
// Step 1: Initialise — get uploadId
router.post('/init', async (req: Request, res: Response) => {
const { fileName, fileType } = req.body
const videoKey = `videos/${uuidv4()}-${Date.now()}-${fileName}`
const { UploadId } = await s3Client.send(new CreateMultipartUploadCommand({
Bucket: bucketName,
Key: videoKey,
ContentType: fileType
}))
await db.video.create({
data: { key: videoKey, uploadId: UploadId!, status: 'PENDING' }
})
return res.json({ uploadId: UploadId, videoKey })
})
// Step 2: Upload one chunk at a time
router.post('/chunk', upload.single('chunk'), async (req: Request, res: Response) => {
const { uploadId, videoKey, partNumber } = req.body
const chunk = req.file
if (!chunk) return res.status(400).json({ message: 'No chunk provided' })
const { ETag } = await s3Client.send(new UploadPartCommand({
Bucket: bucketName,
Key: videoKey,
UploadId: uploadId,
PartNumber: parseInt(partNumber),
Body: chunk.buffer
}))
return res.json({ ETag, partNumber: parseInt(partNumber) })
})
// Step 3: Complete — S3 assembles all chunks
router.post('/complete', async (req: Request, res: Response) => {
const { uploadId, videoKey, parts } = req.body
await s3Client.send(new CompleteMultipartUploadCommand({
Bucket: bucketName,
Key: videoKey,
UploadId: uploadId,
MultipartUpload: { Parts: parts }
}))
await db.video.update({
where: { key: videoKey },
data: { status: 'UPLOADED' }
})
return res.json({ message: 'Video uploaded successfully', videoKey })
})
// Abort — MUST be called on cancel, or AWS bills for incomplete parts
router.post('/abort', async (req: Request, res: Response) => {
const { uploadId, videoKey } = req.body
await s3Client.send(new AbortMultipartUploadCommand({
Bucket: bucketName,
Key: videoKey,
UploadId: uploadId
}))
await db.video.delete({ where: { key: videoKey } })
return res.json({ message: 'Upload aborted' })
})
export default router8. Frontend Integration
The frontend needs to: slice the file into chunks, send them one by one, track progress, and handle pause/resume/cancel. A custom React hook keeps all this logic clean and reusable.
useVideoUpload.ts
import { useState, useRef, useCallback } from 'react'
const CHUNK_SIZE = 10 * 1024 * 1024 // 10MB — must match backend multer limit
interface UploadState {
progress: number
status: 'idle' | 'uploading' | 'paused' | 'completed' | 'error' | 'aborted'
error: string | null
videoKey: string | null
}
interface Part {
ETag: string
PartNumber: number
}
export function useVideoUpload() {
const [state, setState] = useState<UploadState>({
progress: 0, status: 'idle', error: null, videoKey: null
})
// Refs — persist across renders without triggering re-renders
const uploadIdRef = useRef<string | null>(null)
const videoKeyRef = useRef<string | null>(null)
const partsRef = useRef<Part[]>([])
const currentChunkRef = useRef<number>(0)
const isPausedRef = useRef<boolean>(false)
const isAbortedRef = useRef<boolean>(false)
const fileRef = useRef<File | null>(null)
const updateState = (partial: Partial<UploadState>) =>
setState(prev => ({ ...prev, ...partial }))
const initUpload = async (file: File) => {
const res = await fetch('/api/multipart/init', {
method: 'POST',
headers: { 'Content-Type': 'application/json' },
body: JSON.stringify({ fileName: file.name, fileType: file.type })
})
if (!res.ok) throw new Error('Failed to initialise upload')
return res.json()
}
const uploadChunk = async (chunk: Blob, partNumber: number, uploadId: string, videoKey: string): Promise<Part> => {
const formData = new FormData()
formData.append('chunk', chunk)
formData.append('uploadId', uploadId)
formData.append('videoKey', videoKey)
formData.append('partNumber', String(partNumber))
const res = await fetch('/api/multipart/chunk', { method: 'POST', body: formData })
if (!res.ok) throw new Error(`Failed to upload chunk ${partNumber}`)
return res.json()
}
const completeUpload = async (uploadId: string, videoKey: string, parts: Part[]) => {
const res = await fetch('/api/multipart/complete', {
method: 'POST',
headers: { 'Content-Type': 'application/json' },
body: JSON.stringify({ uploadId, videoKey, parts })
})
if (!res.ok) throw new Error('Failed to complete upload')
return res.json()
}
// Core chunk loop — startFrom supports resume
const uploadChunks = async (file: File, uploadId: string, videoKey: string, startFrom = 0) => {
const totalChunks = Math.ceil(file.size / CHUNK_SIZE)
for (let i = startFrom; i < totalChunks; i++) {
if (isPausedRef.current) { currentChunkRef.current = i; return }
if (isAbortedRef.current) return
const chunk = file.slice(i * CHUNK_SIZE, Math.min((i + 1) * CHUNK_SIZE, file.size))
const part = await uploadChunk(chunk, i + 1, uploadId, videoKey)
partsRef.current.push(part)
updateState({ progress: Math.round(((i + 1) / totalChunks) * 100) })
}
if (!isPausedRef.current && !isAbortedRef.current) {
// S3 requires parts sorted by PartNumber
const sorted = [...partsRef.current].sort((a, b) => a.PartNumber - b.PartNumber)
await completeUpload(uploadId, videoKey, sorted)
updateState({ status: 'completed', videoKey, progress: 100 })
}
}
const upload = useCallback(async (file: File) => {
try {
isPausedRef.current = false
isAbortedRef.current = false
partsRef.current = []
currentChunkRef.current = 0
fileRef.current = file
updateState({ status: 'uploading', progress: 0, error: null, videoKey: null })
const { uploadId, videoKey } = await initUpload(file)
uploadIdRef.current = uploadId
videoKeyRef.current = videoKey
await uploadChunks(file, uploadId, videoKey)
} catch (err) {
updateState({ status: 'error', error: err instanceof Error ? err.message : 'Upload failed' })
}
}, [])
const pause = useCallback(() => {
isPausedRef.current = true
updateState({ status: 'paused' })
}, [])
const resume = useCallback(async () => {
if (!fileRef.current || !uploadIdRef.current || !videoKeyRef.current) return
isPausedRef.current = false
isAbortedRef.current = false
updateState({ status: 'uploading' })
await uploadChunks(fileRef.current, uploadIdRef.current, videoKeyRef.current, currentChunkRef.current)
}, [])
const abort = useCallback(async () => {
if (!uploadIdRef.current || !videoKeyRef.current) return
isAbortedRef.current = true
await fetch('/api/multipart/abort', {
method: 'POST',
headers: { 'Content-Type': 'application/json' },
body: JSON.stringify({ uploadId: uploadIdRef.current, videoKey: videoKeyRef.current })
})
uploadIdRef.current = null
videoKeyRef.current = null
partsRef.current = []
currentChunkRef.current = 0
fileRef.current = null
updateState({ status: 'aborted', progress: 0 })
}, [])
return { state, upload, pause, resume, abort }
}VideoUpload.tsx
import React, { useRef } from 'react'
import { useVideoUpload } from '../hooks/useVideoUpload'
export function VideoUpload() {
const { state, upload, pause, resume, abort } = useVideoUpload()
const inputRef = useRef<HTMLInputElement>(null)
const handleFileChange = (e: React.ChangeEvent<HTMLInputElement>) => {
const file = e.target.files?.[0]
if (!file) return
if (!file.type.startsWith('video/')) { alert('Please select a video file'); return }
upload(file)
}
const handleAbort = async () => {
await abort()
if (inputRef.current) inputRef.current.value = ''
}
return (
<div>
<input
ref={inputRef}
type="file"
accept="video/*"
onChange={handleFileChange}
style={{ display: 'none' }}
disabled={['uploading', 'paused'].includes(state.status)}
/>
{['idle', 'error', 'aborted', 'completed'].includes(state.status) && (
<button onClick={() => inputRef.current?.click()}>
{state.status === 'completed' ? 'Upload another' : 'Select video'}
</button>
)}
{['uploading', 'paused'].includes(state.status) && (
<div>
{/* Progress bar */}
<div style={{ background: '#e5e7eb', borderRadius: 8, height: 12 }}>
<div style={{
background: state.status === 'paused' ? '#f59e0b' : '#3b82f6',
width: `${state.progress}%`,
height: '100%',
borderRadius: 8,
transition: 'width 0.3s ease'
}} />
</div>
<p>{state.status === 'paused' ? `Paused at ${state.progress}%` : `Uploading... ${state.progress}%`}</p>
<div style={{ display: 'flex', gap: 8 }}>
{state.status === 'uploading'
? <button onClick={pause}>Pause</button>
: <button onClick={resume}>Resume</button>
}
<button onClick={handleAbort}>Cancel</button>
</div>
</div>
)}
{state.status === 'completed' && <p>✅ Upload complete — key: {state.videoKey}</p>}
{state.status === 'error' && <p>❌ {state.error}</p>}
</div>
)
}Why refs instead of state for isPaused / isAborted?
The chunk loop runs asynchronously. If you stored pause/abort in React state, the loop would capture a stale closure and never see the updated value. Refs are mutable and always return the current value — exactly what a long-running loop needs.
9. Full Architecture Summary
┌─────────────────────────────────────────────────────────────────┐
│ IMAGE UPLOAD FLOW │
└─────────────────────────────────────────────────────────────────┘
Browser
│
├─── POST /upload/image (multipart/form-data)
│ │
│ Multer (parse multipart)
│ │
│ Sharp (resize to 1080×1920)
│ │
│ S3 uploadFile() ──────────────────► S3 Object
│ │ tag: status=PENDING
│ DB image.create(status: PENDING)
│ │
│ ◄── { imageKey, imageUrl }
│
├─── POST /blog { content, imageKey }
│ │
│ DB blog.create()
│ │
│ DB image.update(status: LINKED)
│ │
│ S3 updateFileTag(COMPLETED) ─────────► tag: status=COMPLETED
│ │
│ ◄── { blog }
│
└─── DELETE /blog/:id
│
DB blog.update(status: DELETED, deletedAt: now)
│
S3 updateFileTag(DELETED) ──────────► tag: status=DELETED
│ AWS deletes after 30 days
◄── { message: 'Blog deleted' }
┌─────────────────────────────────────────────────────────────────┐
│ 2GB VIDEO UPLOAD FLOW │
└─────────────────────────────────────────────────────────────────┘
Browser
│
├─── POST /multipart/init { fileName, fileType }
│ ◄── { uploadId, videoKey }
│
├─── file.slice(0, 10MB) → POST /multipart/chunk (part 1) → ETag 1
├─── file.slice(10MB, 20MB) → POST /multipart/chunk (part 2) → ETag 2
├─── file.slice(20MB, 30MB) → POST /multipart/chunk (part 3) → ETag 3
│ ... repeat for all chunks ...
│
├─── POST /multipart/complete { uploadId, parts: [{ETag, PartNumber}] }
│ └── S3 assembles all parts into final object ✅
│
└─── (on cancel) POST /multipart/abort
└── S3 cleans up incomplete parts — avoids billing ✅
┌─────────────────────────────────────────────────────────────────┐
│ S3 LIFECYCLE POLICY │
└─────────────────────────────────────────────────────────────────┘
tag: status=PENDING → delete after 1 day (orphan cleanup)
tag: status=DELETED → delete after 30 days (soft delete window)
tag: status=COMPLETED → glacier after 1 year (archive)Conclusion
Building a production file upload system involves a lot more than wiring up an S3 bucket. The decisions compound:
- Multer bridges
multipart/form-datatoreq.file - Sharp keeps your S3 storage lean by resizing before upload
- UUID + timestamp keys prevent collisions
- PENDING → LINKED tracking catches orphaned images before they accumulate
- Soft delete gives users a recovery window and keeps your S3 operations safe
- S3 tagging + Lifecycle Policies eliminates the need for cron jobs entirely
- Multipart upload is the only viable approach for large files — routing 2GB through Express is a guaranteed timeout
Each of these decisions was made for a reason. Understanding the why behind them means you can adapt them to whatever your platform needs next.