Selenium Fargate at scale

Hello, today we look deeper into deploying Selenium Grid with AWS CDK and Fargate. I find inspiration from this AWS blog post, I rewrote this in CDK 2 Python. I will explain each function, so you understand how they work together to make the infrastructure. We will use HTTP like in the old example. I have an advanced https version but I have also product code in it which I am using for product development, so unfortunatly I cannot provide this source. But I think the reader can easily make this http version scaled and with https!

selenium grid learnautomated testing

What You Need Before Starting

Before you start, make sure you have:

An AWS account with permissions to make repositories in ECR and send images.
Docker installed on your local machine.
AWS CLI installed on your local machine.

Step 2: Setting Up Environment Variables

First, you need to set up the environment variables for the script. You can change the script directly or make a separate .env file with these variables:

ECR_REPO_NAME_SEL_HUB="selenium-hub"
ECR_REPO_NAME_CHROME="chrome"
ECR_REPO_NAME_FIREFOX="firefox"
AWS_REGION="eu-west-2"
AWS_ACCOUNT="your-aws-account-id"

Note: Replace AWS_ACCOUNT with your actual AWS account ID.

Step 3: Running the Script

After setting up the environment variables, navigate to the directory containing the shell script and execute the following command:

./push-to-ecr.sh

To verify whether the images have been successfully pushed to ECR, visit the ECR console in the AWS Management Console. You should see the newly created repositories and the associated images.

Conclusion

Utilizing a shell script to create and push Docker images to AWS ECR is a time-saver. By adhering to this tutorial, you can effortlessly create and push Docker images to ECR. While this process can be integrated into any GitLab/GitHub pipeline, I opted to execute it manually in this instance, both as a cost-saving measure and for educational purposes.

The CDK2 Python Part

In the cdk2 we will create the infra as code in python, which also can be written as typescript or javascript. The essence is that it generates infra cloudformation like Terraform does, but then it is native like Bicep is for Azure.

How-> Here comes the very technical part, I think I lost already a lot of readers :-)

init Function

The init function acts as the constructor for the SeleniumStack class. It plays a pivotal role in initializing an array of AWS resources. These resources include:

VPC (Virtual Private Cloud): Provides an isolated virtual network, allowing you to control your own network environment.
Fargate Cluster: A serverless compute engine for containers.
ECR Repositories (Elastic Container Registry): A fully-managed Docker container registry that makes it easy for developers to store, manage, and deploy Docker container images.
IAM Roles: These are AWS Identity and Access Management (IAM) entities that define a set of permissions for making AWS service requests. After initializing these resources, the function proceeds to create the Selenium Hub service and node services for both Chrome and Firefox browsers.

create_selenium_hub_service Function

This function is dedicated to creating the Selenium Hub Fargate service. Here’s what it does:

Task Definition: Sets up a Fargate task definition that includes the Selenium Hub container.
Security Group: Establishes a security group to allow inbound access.
Log Group: Initiates a log group for logging purposes.
Application Load Balancer (ALB): Configures an ALB, which performs a health check and facilitates communication via HTTP.

create_selenium_node_service Function

This function crafts a Selenium Node Fargate service, specifically for a given browser—either Chrome or Firefox. The steps it follows are:

Task Definition: Creates a task definition with the relevant browser container.
Environment Variables: Configures environment variables essential for connecting to the Selenium Hub.
Log Group & Security Group: Similar to the hub service, it sets up a log group for logging and a security group for inbound access.
Fargate Service Registration: Finally, the function creates the Fargate service and registers it with the ECS cluster and Cloud Map for service discovery.

add_autoscaling_policy Function

This function is tasked with establishing an autoscaling policy for a specified Fargate service. The autoscaling is configured based on:

CPU and Memory Utilization: The service scales up or down, adjusting to the load.
Target Utilization Percentages and Cooldown Periods: These parameters are specified to control the scaling activities efficiently. In essence, this function ensures that the service adapts to the varying loads by adjusting its scale based on the utilization of resources and specified parameters.

AWS Cloud Map

In order to communicate with eachother hub, nodes etc you need to make use of cloudmap

AWS Cloud Map is a cloud resource discovery service that allows you to define custom names for your application resources, and it keeps track of the changing locations of these dynamically changing resources. This service simplifies the management of service discovery and helps in maintaining the health of the resources by allowing applications to discover services based on custom names and attributes.

code can be found in the repo:

https://github.com/learnautomatedtesting/cdk2seleniumpython