Prelude

To build a streaming platform, as discussed here - read it if you miss it! - you first need to know what’s behind a streaming. Have you ever notice that streaming from streaming platform constantly adapt to your network, trying to avoid buffering and slowing down of the stream itself? Today it’s pretty hard to have connection problems, but even only 10 years ago this was a problem for many of us. Moreover, have you ever tried to download a video from a streaming resource? From time to time it’s hard to do it, you only get some empty file or “small pieces”, even using downloader extensions…

Let’s speak about HLS.

HLS (HTTP Live Streaming) unveiled

HTTP Live Streaming (HLS) is a widely adopted protocol for delivering audio and video content over the internet. Developed by Apple Inc., HLS enables adaptive bitrate streaming, allowing content providers to deliver high-quality media to a wide range of devices and network conditions. Here’s a comprehensive explanation of how HLS works.

How it’s done

HLS breaks down the media content into smaller, sequential files called “segments.” These segments are typically in the form of MPEG-2 Transport Stream (TS) or fragmented MP4 files. A manifest file (M3U8) is used as an index of these files, or a playlist, containing the list the available segments and their URLs. The manifest file contains also metadata about the stream, such as duration, bitrate, and resolution, along with the URLs of the media segments. Here’s an example

#EXTM3U
#EXT-X-VERSION:3
#EXT-X-TARGETDURATION:10
#EXT-X-MEDIA-SEQUENCE:0

#EXTINF:10.0,
segment1.ts
#EXTINF:10.0,
segment2.ts
#EXTINF:10.0,
segment3.ts
#EXT-X-ENDLIST

Moreover, the segments are encoded at various bitrates and resolutions to accommodate different network conditions and device capabilities.

Before streaming, the media content undergoes encoding and transcoding to create multiple versions of each segment at different bitrates and resolutions. This process allows HLS to adaptively switch between different quality levels based on the available bandwidth and device capabilities. So much more that only security!

Adaptive Bitrate Streaming

One of the key advantages of HLS is its support for Adaptive Bitrate Streaming (ABR), which improves the viewer experience by adjusting the video quality in real-time based on network conditions. Here’s how ABR works within the HLS framework:

  • initial variant selection: when a client requests the manifest file, it initially selects a stream variant (bitrate and resolution) based on factors like available bandwidth and device capabilities.

  • segment download and playback: as the video plays, the client continuously monitors the network conditions. If the available bandwidth increases or decreases significantly, the client dynamically switches to a higher or lower bitrate variant to maintain a smooth playback experience. This switching happens seamlessly between segment downloads, ensuring uninterrupted playback.

AWS Media Convert Pipeline

The architecture is the reference architecture by AWS (VOD) service. It uses the following AWS services:

  • Amazon S3 (source): Stores the video content in a serverless object storage service.
  • AWS Elemental MediaConvert (job submit): Transcodes the video content into different formats and bitrates for optimal delivery to viewers on different devices and network conditions.
  • Amazon CloudWatch (job complete): Monitors the progress of the transcoding jobs and sends notifications when they are complete.
  • Amazon EventBridge (event trigger): Triggers the Lambda function to start the transcoding job when a new video is uploaded to S3.
  • AWS Lambda (job complete): Starts the transcoding job by calling the MediaConvert API.
  • Amazon S3 (destination): Stores the transcoded video content.
  • Amazon CloudFront (distribution): Delivers the transcoded video content to viewers with low latency and high availability.
  • Amazon Simple Notification Service (SNS) (notifications): Sends notifications to viewers when their transcoding job is complete.

How it Works

  1. An admin user (me) uploads a video to the Amazon S3 source bucket;
  2. The Amazon S3 event triggers a Lambda function;
  3. The Lambda function calls the MediaConvert API to start the transcoding job;
  4. MediaConvert transcodes the video content into different formats and bitrates;
  5. Amazon CloudWatch monitors the progress of the transcoding job: when the transcoding job is complete, Amazon EventBridge sends a notification to a Lambda function;
  6. The Lambda function publishes a message to the SNS topic;
  7. SNS sends notifications to admin (me) about the completion of their transcoding job with bucket reference

Viewers can watch the transcoded video content from the Amazon S3 destination bucket through CloudFront.

from aws_cdk import core
from aws_cdk import aws_s3 as s3
from aws_cdk import aws_mediaconvert as mediaconvert
from aws_cdk import aws_lambda as _lambda
from aws_cdk import aws_events as events
from aws_cdk import aws_cloudfront as cloudfront
from aws_cdk import aws_sns as sns

class VodStack(core.Stack):

    def __init__(self, scope: core.Construct, construct_id: str, **kwargs) -> None:
        super().__init__(scope, construct_id, **kwargs)

        # Create an S3 bucket for the source video content
        source_bucket = s3.Bucket(self, "SourceBucket")

        # Create an S3 bucket for the transcoded video content
        destination_bucket = s3.Bucket(self, "DestinationBucket")

        # Create a MediaConvert job template
        job_template = mediaconvert.JobTemplate(self, "JobTemplate",
            queue="your_queue_name",
            settings={
                "Inputs": [
                    mediaconvert.Input.file_input(
                        file_path="s3://" + source_bucket.bucket_name + "/{filename}"
                    )
                ],
                "Outputs": [
                    mediaconvert.Output.file_output(
                        preset="your_preset_name",
                        output_s3_bucket=destination_bucket
                    )
                ]
            }
        )

        # Create a Lambda function to start the transcoding job
        lambda_function = _lambda.Function(self, "LambdaFunction",
            runtime=_lambda.Runtime.PYTHON_3_9,
            code=_lambda.Code.asset("lambda_code"),
            handler="lambda_handler.main",
            environment={
                "MEDIA_CONVERT_JOB_TEMPLATE_ARN": job_template.arn
            }
        )

        # Create an event rule to trigger the Lambda function when a new video is uploaded to S3
        event_rule = events.Rule(self, "EventRule",
            event_pattern=events.EventPattern(
                source=["aws.s3"],
                event_type=["ObjectCreated"],
                bucket=[source_bucket.bucket_name]
            )
        )

        event_rule.add_target(lambda_function)

        # Create a CloudFront distribution to deliver the transcoded video content
        cloudfront_distribution = cloudfront.Distribution(self, "CloudFrontDistribution",
            default_cache_behavior=cloudfront.CacheBehavior(
                allowed_methods=["GET", "HEAD"],
                target=cloudfront.Origin(
                    domain_name=destination_bucket.domain_name
                )
            )
        )

        # Create an SNS topic to send notifications to viewers
        sns_topic = sns.Topic(self, "SNSTopic")

        # Output the CloudFront distribution URL
        core.CfnOutput(self, "CloudFrontDistributionURL",
            value=cloudfront_distribution.distribution_domain_name
        )

Conclusion

The AWS architecture described in this document provides a scalable, reliable, and cost-effective solution for delivering video content to viewers. The AWS CDK Python code example demonstrates how to create the resources described in the architecture. In the next article, we are gonna see how we can integrate safety to provide premium content only to authenticated user.

Stay tuned, and if you miss the previous article, you can find it here! Otherwise, move to the next article: I’m gonna show you how you can leverage Cloudfront Signed Cookies to safely deliver your premium contents. Enjoy the reading and let me know what you think in the comment section!