Effective CI/CD pipelines play a crucial role in streamlining software development processes. For meson-middleware – a Django based application, standardizing its workflow by integration with the organization's CI/CD setup on Azure was a necessary step. However, the existing CI/CD setup provided limited customizations, making it a challenge to meet the specific requirements of meson-middleware.
Available Capabilities on the Current CI/CD Platform:
Running Shell Scripts for Test Case Execution:
The current CI/CD platform allows running shell scripts and provide feedback through exit codes – where a zero-exit code indicates a successful execution.
Installing Required Python Libraries:
To support test case execution the CI/CD platform allows installation of necessary Python libraries.
Docker Image of the Current Code Changes on Amazon ECR:
This enabled the development team to access and utilize the latest version of Meson Middleware, incorporating any recent changes or updates.
Middleware requires running functional test cases with external dependencies like PostgreSQL and publishing the results for every GitHub pull request.
This article delves into the steps that meet those requirements, including running test cases with coverage and lint checks that necessitated a PSQL database connection.
To address the specific challenges of running functional test cases for meson-middleware, two essential components must be considered.
The first solution involves executing test cases on a remote machine, leveraging the Docker image stored on Amazon Elastic Container Registry (ECR).
The second solution revolves around hosting a dedicated test database to support the functional test cases.
AWS ECS Fargate is a great solution for small running tasks for a variety of reasons.
Firstly, Fargate provides a fully managed container service that eliminates the need to provision and manage infrastructure. This means that small tasks can run without worrying about the underlying infrastructure, allowing developers to focus on writing code instead.
Secondly, Fargate is highly scalable and can easily handle fluctuations in demand, which is important for small tasks that may not have consistent usage patterns.
Finally, Fargate is cost-effective since it only charges for the resources used by the container, making it a great choice for small tasks that don't require a lot of resources.
Overall, AWS ECS Fargate is a reliable and efficient solution for small running tasks that provides developers with the flexibility and scalability needed to deliver high-quality applications.
AWS Aurora Serverless offers a powerful and flexible database solution for meson-middleware.
With its on-demand capacity, automatic scaling, cost optimization, high availability and seamless integration with the AWS ecosystem, Aurora Serverless simplifies database management and ensures optimal performance.
By leveraging this serverless offering, meson-middleware can focus on delivering value to its users while enjoying the benefits of a scalable and cost-efficient database solution.
Here’s an overall architecture of the implementation. Each step is explained in detail below.
In modern software development, Continuous Integration and Continuous Deployment (CI/CD) pipelines are crucial for ensuring the quality and timely delivery of applications. meson-middleware has been successfully onboarded onto the CI system, enabling efficient build processes on BuildHUB (Jenkins). We will explore the step-by-step journey of a build triggered by a GitHub Pull Request, highlighting more on the various stages and tools involved in the running the functional test cases and lint checks.
The CI pipeline starts with performing sanity checks on the GitHub Pull Request. These checks verify the PR's validity, including factors like having a valid title, formatting conventions, and adherence to best practices. This initial validation helps maintain code quality and consistency.
Upon passing the PR sanity checks, the pipeline proceeds to execute the unit test cases on BuildHUB. The results of these tests are then compiled into a coverage report, which is uploaded to Amazon S3. This step ensures that the code changes introduced by the PR do not break the existing functionality and provides insights into the test coverage.
Next, the pipeline builds a Docker image using the updated codebase and relevant dependencies. This containerized application is then pushed to the Amazon Elastic Container Registry (ECR), a fully managed Docker container registry service.
In this phase, the pipeline leverages Boto3, the AWS SDK for Python, to interact with Amazon Elastic Container Service (ECS) and launch an Amazon Fargate task. The Fargate task runs within an isolated environment, eliminating the need for manual server management.
The following steps are involved as part of the task.
Running Lint Checks and Generating Reports
Within the Fargate task, lint checks are performed using pylint to enforce code quality standards and best practices. The resulting report is uploaded to Amazon S3 for further analysis and reference.
Running Functional/Integration Test Cases and Generating Coverage Reports
The Fargate task also executes functional and integration test cases to ensure the application functions as expected in its target environment. Similar to lint checks, the coverage report generated during these tests is uploaded to Amazon S3 for visibility and analysis.
To run the task on ECS, we need to go through the following steps using Boto3 on the client side (assuming a dedicated cluster is already setup on ECS):
a. Register a Task Definition: A task definition is registered, specifying the current Docker image to be used. This defines the resources, environment variables, and other configurations required to run the task.
response = client.register_task_definition(...)
b. Run Fargate Task: The Fargate task is initiated using the task definition created in the previous step. All the logs generated during the task's execution are captured in a dedicated log group in Amazon CloudWatch, facilitating easy monitoring and troubleshooting.
response = client.run_task(...)
c. Polling for Fargate Task Completion: The CI pipeline continually monitors the Fargate task's progress, periodically checking for completion. Once the task has finished running, the pipeline examines its exit code. A zero exit code indicates successful execution, while a non-zero exit code indicates an error or failure. Based on the outcome, the pipeline concludes the job accordingly.
response = client.describe_tasks(
cluster='string',
tasks=[
'string',
],
include=[
'TAGS',
]
)
By implementing the two key solutions of remote functional testing with the Docker image on ECR and hosting a dedicated test database, meson-middleware can achieve efficient and comprehensive testing.
Leveraging the Docker image on a remote machine ensures a production-like environment for accurate testing, while the test database provides an isolated and controlled environment for reliable data management.
These solutions collectively contribute to the development of a robust and high-quality application, empowering the meson-middleware team to deliver exceptional functionality and performance to their users.
Sign up with your email address to receive news and updates from InMobi Technology
