Designing a Photo-Sharing Platform Inspired by Instagram

We're conceptualizing a platform akin to Instagram, focused on photo sharing. This project is at a medium difficulty level, drawing inspiration from services like Flickr and Picasa.

Conceptualizing Our Service

Our service is akin to Instagram, a renowned social media platform. It allows individuals to post and circulate their photos and videos. Users have the flexibility to control the visibility of their posts – either to the public or to a select group. Our service integrates with various social media channels, including Facebook, Twitter, Flickr, and Tumblr, for wider content dissemination.

Design Scope

For this endeavor, we aim to create a streamlined variant of Instagram. Key features will include photo sharing and the ability to follow other users. Each user's 'News Feed' will display a curated selection of top photos from the users they follow.

Requirements and Objectives of the System

We are outlining the essential requirements for our Instagram-inspired photo-sharing service:

Functional Requirements

1. Photo Management: Users must have the capability to upload, download, and view photos.
2. Search Functionality: The ability to search content using photo or video titles.
3. Social Interaction: Users should be able to follow other accounts.
4. News Feed Generation: The system should dynamically create and display a News Feed, showcasing top photos from followed accounts.

Non-functional Requirements

1. High Availability: The service should be accessible at all times.
2. Latency: The target latency for generating the News Feed is set at 200ms.
3. Consistency vs. Availability: While consistency is important, it can be compromised for improved availability. Delayed photo visibility is acceptable.
4. Reliability: It's crucial that no uploaded photos or videos are lost.

Out of Scope

The current scope excludes features like tagging photos, searching by tags, commenting on photos, user tagging in photos, and recommendations for following.

Capacity Estimation and Constraints Analysis

In this section, we reassess the capacity needs and limitations for our photo-sharing platform:

1. User Base and Activity: Let's hypothesize that our platform hosts 600 million registered users, with 1.2 million users actively engaging daily.
2. Photo Upload Volume: On average, the platform receives 2.5 million new photos each day, which translates to roughly 29 new photos every second.
3. Average Photo Size: We estimate the average size of a photo to be approximately 250KB.

Detailed Capacity Calculations

• Daily Storage Requirements: For the daily influx of photos, we calculate the storage needs as follows:
  • 2.5M photos/day * 250KB/photo = 625 GB/day
• Long-Term Storage Needs: Projecting over a decade, the storage requirement would be:
  • 625 GB/day _ 365 days/year _ 10 years ≈ 2281.25 TB

This revised estimate provides a comprehensive view of the storage capacity we need to plan for, ensuring robust scalability for the projected user base and photo upload volume.

High-Level System Design Overview

Our photo-sharing platform's design is centered around two main functionalities:

1. Photo Upload Functionality: This aspect necessitates a robust infrastructure for users to upload their photos seamlessly. It involves the implementation of reliable and efficient object storage servers, dedicated to handling the large volume of photo uploads.

2. Photo Viewing and Searching Functionality: To facilitate user engagement, the system must enable efficient viewing and searching of photos. This requires not only the storage servers for photo retrieval but also a sophisticated database system for managing metadata associated with each photo. The metadata storage will play a crucial role in optimizing search functions and enhancing user experience.

This high-level design ensures that our service efficiently manages both the storage and retrieval of photos, along with providing a seamless user interface for interaction with the platform.

Database Schema Design

Understanding the database schema early in the development process is crucial for grasping data flow and guiding data partitioning strategies.

Core Data Elements

Our system needs to efficiently manage data related to:

• Users: Information about the users of the platform.
• Photos: Details of the photos uploaded by users.
• User Connections: Data on the follow relationships between users.

Photo Table and Indexing

• The 'Photo' table will contain all the information related to each photo.
• An index on (PhotoID, CreationDate) is essential for quickly fetching the most recent photos.

Choosing the Database Type

• While an RDBMS like MySQL is suitable for our need for joins, scaling relational databases presents challenges. This necessitates exploring SQL vs. NoSQL databases.
• For storing the actual photos, we can use distributed file storage systems like HDFS or S3.

Database Tables and Structure

Users Table

• UserID (Primary Key): Unique identifier for each user.
• Username: The user's chosen username.
• Email: User's email address.
• Password: Hashed password for user account security.
• CreationDate: Date and time when the account was created.
• LastLogin: Timestamp of the last login activity.

Photos Table

• PhotoID (Primary Key): Unique identifier for each photo.
• UserID: Identifier of the user who uploaded the photo.
• PhotoLocation: URL or path where the photo is stored.
• CreationDate: Date and time when the photo was uploaded.
• UserLocation: Geographical location of the user at the time of upload.
• Description: Optional description of the photo.

Index on (PhotoID, CreationDate) for efficient retrieval of recent photos.

UserFollow Table

• UserID (Primary Key): Identifier of the user.
• FollowsUserID: Identifier of the user being followed.
• FollowDate: Timestamp when the follow action was initiated.

UserPhoto Table

• UserID (Primary Key): Identifier of the user.
• PhotoIDs: List of PhotoIDs owned by the user.

This table can be stored in a wide-column store like Cassandra for efficient access and scalability.

Metadata Storage and NoSQL Benefits

• We can leverage a distributed key-value store for the schema to benefit from the scalability of NoSQL.
• Metadata for each photo, including PhotoLocation, UserLocation, and CreationTimestamp, can be stored in a table where PhotoID is the key.

Managing User Relationships

• We need to track which user owns which photo and the follow list of each user.
• For this, a wide-column store like Cassandra is ideal.
• In the 'UserPhoto' table, 'UserID' acts as the key, with 'PhotoIDs' as values in different columns.
• A similar approach is used for the 'UserFollow' table.

Reliability and Data Retention in Cassandra

• Cassandra and similar key-value stores maintain multiple replicas for reliability.
• Deletes in these systems are not immediate. Data is retained for a specified period, allowing for potential undeleting, before permanent removal.

This schema design outlines a robust and scalable approach to managing the vast amount of data in our photo-sharing platform.

Data Size Estimation

Let's recalculate the storage requirements for each table over a span of 10 years with updated assumptions.

User Table

• Row Size Calculation:
  • UserID: 4 bytes
  • Name: 24 bytes (increased size for longer names)
  • Email: 36 bytes (allowing for longer email addresses)
  • DateOfBirth: 4 bytes
  • CreationDate: 4 bytes
  • LastLogin: 4 bytes
  • Total: 76 bytes per row
• Total Storage for 600 Million Users:
  • 600 million * 76 bytes ≈ 43.2GB

Photo Table

• Row Size Calculation:
  • PhotoID: 4 bytes
  • UserID: 4 bytes
  • PhotoPath: 260 bytes (increased for longer paths)
  • PhotoLatitude: 4 bytes
  • PhotoLongitude: 4 bytes
  • UserLatitude: 4 bytes
  • UserLongitude: 4 bytes
  • CreationDate: 4 bytes
  • Total: 288 bytes per row
• Daily Storage for 2.5 Million New Photos:
  • 2.5M * 288 bytes ≈ 0.675GB per day
• Storage for 10 Years:
  • 0.675GB/day _ 365 days/year _ 10 years ≈ 2.46TB

UserFollow Table

• Row Size:
  • 8 bytes per row
• Assuming Each User Follows 600 Others:
  • 600 million users _ 600 followers _ 8 bytes ≈ 2.88TB

Total Storage for All Tables for 10 Years

• Calculation:
  • User Table: 43.2GB
  • Photo Table: 2.46TB
  • UserFollow Table: 2.88TB
  • Total: ≈ 5.39TB

Component Design

Balancing Reads and Writes

• Challenge with Photo Uploads: Upload operations, which involve writing to disk, are inherently slower than reads. This can lead to a scenario where uploading users occupy all available connections, preventing read operations.
• Web Server Connection Limits: Assuming a web server limit of 500 concurrent connections, this cap restricts the number of simultaneous uploads or reads.
• Solution - Separating Services: To mitigate this, we propose dividing read and write operations into distinct services. This separation ensures that photo uploads do not monopolize system resources, allowing efficient handling of read requests.
• Independent Scaling and Optimization: By segregating photo read and write requests, we can scale and refine these operations independently, optimizing performance for each.

Ensuring Reliability and Redundancy

• Zero Tolerance for Data Loss: Our service mandates the storage of multiple copies of each file. This redundancy ensures file availability even if one storage server fails.
• Redundancy Across the System: This principle extends beyond file storage. To guarantee high system availability, we'll implement multiple replicas of each service. This strategy eliminates single points of failure.
• Failover Mechanisms: In addition to active service instances, we'll maintain redundant secondary copies in standby mode. These can take over in case the primary service instance encounters issues, ensuring uninterrupted service.
• Crisis Management through Redundancy: With redundant system components, we ensure a backup mechanism. In case of service failure, the system can automatically or manually failover to a functioning copy, maintaining operational continuity.

Ranking and News Feed Generation

Generating User's News Feed

• Objective: To assemble a News Feed, the system needs to gather the most recent, popular, and relevant photos from the followed accounts.
• Basic Approach: For each user, the top 100 photos are fetched. The server first retrieves the list of followed users, then gathers metadata for the latest 100 photos from each. Finally, a ranking algorithm sorts these photos based on criteria like recency and popularity.

Challenges and Efficiency Improvements

• Latency Issue: This method might result in higher latency due to multiple table queries and the need for sorting and merging results.
• Pre-Generating News Feed: To enhance efficiency, we can pre-generate News Feeds and store them in a 'UserNewsFeed' table. This approach speeds up the retrieval of the latest photos when a user requests their News Feed.
• Update Mechanism: Dedicated servers periodically update each user's News Feed in the 'UserNewsFeed' table, based on the latest information since the last generation.

Distribution Approaches for News Feed Contents

1. Pull Method:
   • Clients regularly request (pull) News Feed updates. Challenges include potential delays in showing new data and frequent empty responses if no new data is available.
2. Push Method:
   • Servers actively send (push) updates to users. This is efficient but can be challenging for users with many follows or celebrities with vast followers, due to frequent updates.
3. Hybrid Method:
   • Combining pull and push approaches based on user activity. High-follow users might pull updates, while others receive push updates. Alternatively, servers could push updates at a controlled frequency, prompting active users to pull updates regularly.

For an in-depth understanding of News Feed generation mechanisms, refer to the case study on 'Designing Facebook’s Newsfeed'.

Conclusion of Photo-Sharing Service Design

In summary, the design of our Instagram-inspired photo-sharing service encompasses a comprehensive approach to handling large-scale user interactions and data management. Key elements include:

• Scalable Architecture: Ensuring the system can handle millions of users and billions of photos through efficient data sharding and storage strategies.
• Efficient Data Handling: Utilizing partitioning based on both UserID and PhotoID to optimize data retrieval and storage distribution.
• Robust News Feed Algorithm: Implementing a sophisticated algorithm for the News Feed, considering factors like recency, popularity, and relevance of photos.
• Intelligent Caching and Load Balancing: Leveraging advanced caching mechanisms and global load balancing to enhance user experience and system performance.
• Focus on Reliability and Redundancy: Prioritizing data integrity and system availability by incorporating multiple data copies and failover strategies.

This design aims to provide a seamless, efficient, and engaging experience for users, mirroring the key functionalities of popular photo-sharing platforms like Instagram, while also addressing unique challenges and considerations for scalability and performance.