Jobset

Example Workloads

Wed, 23 Jul 2025 00:00:00 +0000

PyTorch Example

Distributed Training of a CNN on the MNIST dataset using PyTorch and JobSet

Note: Machine learning container images can be quite large so it may take some time to pull the images.

TensorFlow Example

Distributed Training of a Handwritten Digit Classifier on the MNIST dataset using TensorFlow and JobSet

This example runs an example job for a single epoch. You can view the progress of your jobs via kubectl logs jobs/tensorflow-tensorflow-0.

Simple Examples

Wed, 23 Jul 2025 00:00:00 +0000

Here we have some simple examples demonstrating core JobSet features.

Success Policy demonstrates an example of utilizing successPolicy. Success Policy allows one to specify when to mark a JobSet as completed successfully. This example showcases how to use success policy to mark the JobSet as successful if the worker replicated job completes.
Exclusive Job Placement demonstrates how to configure a JobSet to have a 1:1 mapping between each child Job and a particular topology domain, such as a datacenter rack or zone. This means that all the pods belonging to a child job will be colocated in the same topology domain, while pods from other jobs will not be allowed to run within this domain. This gives the child job exclusive access to computer resources in this domain.

Development Environment Setup

Mon, 01 Jan 0001 00:00:00 +0000

Dependencies

go>=1.24.0
make
kubectl
git
docker
Kubernetes cluster running one of the last 3 Kubernetes minor versions
- kind allows you to run a local Kubernetes cluster using Docker containers

Building and deploying JobSet from source

Building the image

See Makefile targets for more information. In particular:

make image-build: Builds a JobSet image locally
make image-push: Builds a JobSet image locally AND pushes it to the registry

The make image-push hook will attempt to push the built image to the public us-central1-docker.pkg.dev/k8s-staging-images/jobset registry with an image tag determined by git describe. It is recommended to set your own GIT_TAG and IMAGE_REGISTRY environment variables to ensure that your latest changes are pushed to an image registry your cluster can access.

Failure Policy

Wed, 23 Jul 2025 00:00:00 +0000

JobSet provides failure policy API to control how your workload behaves in response to child Job failures.

The failurePolicy is defined by a set of rules. For any job failure, the rules are evaluated in order, and the first matching rule’s action is executed. If no rule matches, the default action is RestartJobSet, which counts towards the maxRestarts limit.

Failure Policy Actions

`FailJobSet`

This action immediately marks the entire JobSet as failed.

Volume Claim Policies

Tue, 13 Jan 2026 00:00:00 +0000

JobSet provides the VolumeClaimPolicies API to automatically create and manage shared PersistentVolumeClaims (PVCs) across multiple ReplicatedJobs within a JobSet. This enables stateful JobSets that require persistent storage for datasets, models, checkpoints, or intermediate results.

Basic Usage

To use VolumeClaimPolicies, define them in the volumeClaimPolicies field of your JobSet spec. Each policy can contain one or more PVC templates.

This example demonstrates creating shared PVCs with different retention policies:

In this example:

Prometheus Metrics

Mon, 14 Feb 2022 00:00:00 +0000

Prometheus Metrics

JobSet exposes prometheus metrics to monitor the health of the controller.

Installation Examples

The following example show how to install the Prometheus Operator for JobSet system.

JobSet controller health

Use the following metrics to monitor the health of the jobset controller:

Metric name	Type	Description	Labels
`controller_runtime_reconcile_errors_total`	Counter	The total number of reconciliation errors encountered by each controller.	`controller`: name of controller (i.e. use value `jobset` to obtain metrics for jobset controller)
`controller_runtime_reconcile_time_seconds`	Histogram	The latency of a reconciliation attempt in seconds.	`controller`: name of controller (i.e. use value `jobset` to obtain metrics for jobset controller)

JobSet metrics

Use the following metrics to monitor the health of the jobsets created by the jobset controller:

JobSet API

Mon, 01 Jan 0001 00:00:00 +0000

Resource Types

JobSet

`JobSet`

Appears in:

JobSet is the Schema for the jobsets API

Field	Description
`apiVersion` string	`jobset.x-k8s.io/v1alpha2`
`kind` string	`JobSet`
`spec` [Required] `JobSetSpec`	spec is the specification for jobset
`status` [Required] `JobSetStatus`	status is the status of the jobset

`Coordinator`

Appears in:

JobSetSpec

Coordinator defines which pod can be marked as the coordinator for the JobSet workload.

Field Description

Field	Description
`replicatedJob` [Required] `string`	replicatedJob is the name of the ReplicatedJob which contains the coordinator pod. Search Results Mon, 01 Jan 0001 00:00:00 +0000

replicatedJob [Required]
string

replicatedJob is the name of the ReplicatedJob which contains the coordinator pod.

Search Results

Mon, 01 Jan 0001 00:00:00 +0000