How to use a universal package manager for scheduled and containerized tasks


Scheduled tasks are essential for business continuity. They can be developed in many different languages ​​(including Java, C #, Go, Python, and R) and can be deployed to containers using Docker registries. These containers can then communicate with each other through Kubernetes.

But when working with scheduled tasks, how can you collect, track and manage the artifacts associated with them at all times and use their metadata for analysis and improvements? A universal package manager, also known as a universal repository manager, can help.

Here’s a step-by-step look at how these tools work and how you can use them with your DevOps authoring pipelines for both scheduled and containerized tasks.

1. Develop scheduled tasks

First, when developing scheduled tasks, you should manage your build artifacts and their metadata using a package manager that supports multiple languages ​​and continuous integration (CI) tools such as Jenkins. This is necessary to resolve dependencies from remote repositories and to deploy generated artifacts to local repositories.

For example, if you create your scheduled tasks using Golang, you can use a Go registry to troubleshoot artifacts and deploy the Go packages you create to your local repositories. The go.mod file is modified based on internal project dependencies when a Go module is built and when those Go packages or Go modules are released.

You can also publish a variety of settings, such as name and build number, which all contribute to artifact metadata.

A universal binary repository manager supports similar functionality for all platforms including Java (Maven and Gradle versions), C # (NuGet), Python (PyPI and Conda), R (CRAN), Helm, Go, etc.

Construction information can be used for a variety of metrics, including:

  • Measure the number of builds that have been sent to QA / release

  • Determine dependency variations in different versions

  • Identify the builds that make your project big

Figure 1: Getting Go Modules in Production with a Universal Package Manager.

2. Containerize a scheduled task and manage Docker information

After you develop a scheduled task, it should run at certain time intervals.

Imagine a job that runs hourly and puts some load on your production servers, with output files and logs that are backed up for years. When multiple such tasks are running simultaneously, your infrastructure is prone to major failures and load errors.

In this new era of containerized applications, you can move to containerization of scheduled tasks, as long as you have a way to manage the associated information. A package manager can provide the Docker images needed to run Docker containers in a continuous integration task. It can also store the information about these Docker images, in a layered format, as Docker artifacts. So you can use a Docker registry or perform the tasks of a scheduled job in Docker proxies without infrastructure.

The good thing is that these Docker registers can be used natively for the customer with a package manager, making them easier to understand. In addition, as with every building language, metadata for Docker repositories is stored. You can use it for analyzes, such as discovering the dynamic container or layer size for each scheduled task run, or comparing two builds to find differences in the artifacts each contains.

3. Run multiple Docker containers in a Kubernetes cluster and use a Helm charter registry

Consider a real-time environment in which these scheduled tasks are generally interrelated and you may have varying environments to perform each of them. For example, financial organizations typically create these tasks for complex reports that are quite interrelated and that may have different environmental, programming, or scripting requirements.

In such cases, multiple Docker containers with different operating systems or settings in memory, each working individually, can be combined to work together in a Kubernetes cluster.

A package manager in this case can function as a Kubernetes registry to store and manage information about these Kubernetes clusters. You can use Helm charts to model the cluster, and can also store and manage them in a package manager using a remote Helm repository.

After you deploy these charts, you can view their metadata in the Helm repository. This makes it even easier to manage the automation related to the task infrastructure, and you have everything in one place for easier access.

Accelerate your DevOps

Developing, managing, and analyzing scheduled tasks becomes much easier when you use containerized clusters with a repository manager. Use one to speed up the process of developing scheduled tasks the DevOps way.

These tools provide a simple and central solution for the maintenance and analysis of artefacts. And application developers and support engineers can monitor every detail using metadata stored and readily available in the Repository Manager.

Keep learning


Comments are closed.