Frontend School

Scalable Image Gallery with CDN

System Design
medium

Scalable Image Gallery with CDN

Medium
System Design
CDN
Image Processing
Scalability
Backend for Frontend

Overview

Design a highly scalable and performant image gallery application capable of serving millions of images to a global user base. This problem focuses on the backend system architecture for efficient storage, processing, and delivery of user-uploaded images, leveraging Content Delivery Networks (CDNs) to ensure low latency and high availability for a frontend client.

What you need to do / Task

  • Design the core architecture for storing and serving images.
  • Detail how images are processed (e.g., resizing, thumbnail generation) before storage and delivery.
  • Explain the role and integration of a CDN for optimal image delivery.
  • Discuss caching strategies at various layers to minimize latency and improve performance.
  • Outline the API endpoints required for image management and retrieval.

Scenario

Users can upload their photos to a personal gallery. Once uploaded, these photos should be viewable quickly in various sizes (thumbnail, medium, full-resolution) by the user and potentially shared with others. The primary goal is a smooth, fast browsing experience, even with a vast number of images and concurrent global users.

Constraints

Constraints & assumptions

  • Users: 100M registered users, 10M daily active users.
  • Images: Each user uploads an average of 100 images. Average image size: 5MB. Total images: ~10 Billion. Total storage: ~50 PB.
  • Traffic: Read-heavy workload, with peak read requests around 100,000 QPS for image retrieval. Uploads are significantly less frequent (1,000 QPS).
  • Latency: Image load times (especially for thumbnails/previews) should be under 200ms globally.
  • Availability: High availability (99.99%) is required.
  • Cost: Optimize for storage and bandwidth costs, especially with CDN usage.
  • Image types: Assume common image formats (JPEG, PNG, WebP).
  • No video: Focus solely on static images.

Out of scope

  • Detailed user authentication and authorization mechanisms (assume a user ID is available).
  • Complex image editing features beyond basic resizing/cropping for display.
  • Real-time notifications or chat features.
  • Specific frontend UI component design (focus on backend infrastructure and APIs that serve the frontend).

What interviewers evaluate

  • Ability to design a scalable and resilient system for image storage and delivery.
  • Effective utilization and understanding of CDN principles for performance and cost optimization.
  • Thoughtful data modeling and API design for image metadata and content.
  • Identification and discussion of key trade-offs (e.g., consistency vs. availability, cost vs. performance).

Requirements

1.

Design a robust image ingestion pipeline that handles uploads, generates multiple image derivatives (e.g., thumbnail, medium, full-resolution), and stores them durably.

2.

Propose a storage solution for both original images and their derivatives, ensuring high availability and cost-effectiveness for petabytes of data.

3.

Detail the integration of a CDN to serve images globally, including strategies for caching, cache invalidation, and handling cache misses.

4.

Define API endpoints for uploading images and retrieving specific image versions by ID, considering parameters for resolution/format.

5.

Explain how the system scales to support 100M users, 10B images, and 100K QPS read requests, maintaining low latency.

6.

Discuss mechanisms to ensure high availability (99.99%) and fault tolerance across the image serving pipeline.

7.

Outline strategies for cost optimization related to storage and CDN bandwidth.

Design Brief

You are tasked with designing the core backend system that powers a global image gallery application. This system will enable frontend clients to upload, store, and efficiently view images. The critical requirements are high scalability, low latency for image retrieval, and cost efficiency for a global user base.

Consider the entire lifecycle of an image:

  1. Upload: A user uploads an image.
  2. Processing: The uploaded image needs to be processed to generate various derivatives (e.g., thumbnail, medium, large versions) suitable for different display contexts.
  3. Storage: Both the original image and its derivatives need to be stored durably and cost-effectively.
  4. Delivery: When a user requests an image (e.g., for their gallery feed, a full-screen view), the most appropriate version should be served quickly and efficiently, especially considering global users.

Your design should cover:

  • High-Level Architecture: Diagram the main components involved (e.g., client, API Gateway, services, storage, CDN).
  • Image Ingestion Workflow: Describe the steps from a user uploading an image to it being ready for serving, including asynchronous processing.
  • Storage Strategy: How would you store original images and their derivatives? What type of storage would you use, and why? Consider object storage, metadata storage.
  • Image Processing: How would you handle generating different image sizes/formats? Where would this processing occur, and how would it scale?
  • CDN Integration: Explain in detail how a CDN would be used to serve images. What are the key configurations and considerations (e.g., caching headers, invalidation, edge locations, custom domains)? How would you handle cache misses?
  • API Design: Propose key API endpoints for image upload and retrieval, including parameters for requesting specific image sizes.
  • Scalability & Reliability: How would your design handle the stated scale (10B images, 100K QPS reads)? What mechanisms would ensure high availability and fault tolerance?
  • Cost Optimization: Discuss strategies to minimize storage and bandwidth costs, particularly with CDN usage.
  • Error Handling: Briefly touch upon how you would handle failures during upload or processing.

Focus on the backend and infrastructure, assuming a frontend client consumes your APIs.

Loading editor...