> For the complete documentation index, see [llms.txt](https://docs.kaisar.io/kaisar-network/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://docs.kaisar.io/kaisar-network/kaisar-ai-ops/deep-learning-platform/jobs.md).

# Jobs

Submit and manage training jobs on cluster resources.

![Jobs Overview](/files/hDnBvlQ76FEF0S8Ycpsx)

## Overview

The Jobs section allows you to submit, monitor, and manage ML training jobs that run on cluster nodes. Track job status, progress, and resource utilization in real-time.

## Jobs Dashboard

The dashboard displays key job metrics at a glance:

**Summary Cards**:

* **Total Jobs**: Total number of jobs in the system
* **Running**: Number of jobs currently running
* **Completed**: Number of jobs completed successfully
* **Failed**: Number of jobs that failed

## Job List View

The jobs table shows all submitted jobs with the following information:

**Columns**:

* **Job Name**: Job name and type (Training, Inference, etc.)
* **Status**: Current status with color-coded badges
* **Priority**: Job priority (High, Medium, Low)
* **Experiment**: Associated experiment name
* **Progress**: Progress bar with percentage
* **Created**: Job creation date
* **Actions**: Quick actions menu

**Filtering and Search**:

* Search by job name
* Filter by status, priority, or experiment

## Creating a Job

Navigate to **Deep Learning Platform** → **Jobs** → Click **Create**

![Create Job](/files/HQXeH8NXqlcg4Ww2afDy)

### Basic Information

**Job Name**\* (Required)

* Enter a descriptive name for the job
* Example: `BERT Fine-tuning - Sentiment Analysis`
* Helper text: "Enter a descriptive name for the job"

**Description**

* Detailed description of the job
* Example: "Fine-tuning BERT-base model for sentiment analysis on IMDB movie reviews"

**Job Type**\* (Required)

* Select from dropdown: Training, Inference, Hyperparameter Tuning, etc.
* Default: `Training`
* Helper text: "training"

**Priority**\* (Required)

* Select priority level: High, Medium, Low
* Default: `High`
* Helper text: "high"

### Configuration

**Epochs**

* Number of training epochs
* Example: `3`
* Helper text: "Number of training epochs"

**Batch Size**

* Training batch size
* Example: `16`
* Helper text: "Training batch size"

**Learning Rate**

* Initial learning rate
* Example: `0.00002`
* Helper text: "Initial learning rate"

**Optimizer**

* Select optimizer from dropdown
* Options: AdamW, Adam, SGD, etc.
* Default: `AdamW`
* Helper text: "adamw"

### Resources

**CPU Cores**

* Number of CPU cores required
* Example: `4`

**Memory (GB)**

* Memory allocation in GB
* Example: `16`

**GPU Count**

* Number of GPUs required
* Example: `0` (for CPU-only jobs)

### Actions

* **Cancel**: Discard and close
* **Create Job**: Submit the job to the queue

## Viewing Job Details

To view detailed information about a job:

1. Navigate to **Deep Learning Platform** → **Jobs**
2. Click on a job from the list
3. View comprehensive details in the modal dialog

![View Job Details](/files/LiIKhbu66TgHTRyE6N2l)

**Details Panel Sections**:

**Basic Information**:

* **Job Name**: e.g., "BERT Fine-tuning - Sentiment Analysis"
* **Description**: Full description of the job
* **Job Type**: Training, Inference, etc.
* **Priority**: High, Medium, Low

**Configuration**:

* **Epochs**: Number of training epochs (e.g., 3)
* **Batch Size**: Training batch size (e.g., 16)
* **Learning Rate**: Initial learning rate (e.g., 0.00002)
* **Optimizer**: Optimizer used (e.g., AdamW)

**Resources**:

* **CPU Cores**: Allocated CPU cores (e.g., 4)
* **Memory (GB)**: Allocated memory (e.g., 16)
* **GPU Count**: Number of GPUs (e.g., 0)

## Editing a Job

To update job configuration:

1. Open job details page
2. Click **Edit** button (or three-dot menu → Edit)
3. Modify editable fields in the Edit Job modal

![Edit Job Form](/files/q36QKAvrmpBsfESQCff1)

4. Click **Update Job** to save changes

> \[!NOTE] You can only edit jobs that are in Pending or Failed status. Running or Completed jobs cannot be edited.

**Editable Fields**:

* ✅ Job Name
* ✅ Description
* ✅ Priority (can change to expedite or delay)
* ✅ Configuration (epochs, batch size, learning rate, optimizer)
* ✅ Resources (CPU, memory, GPU)
* ❌ Job Type (cannot edit)
* ❌ Status (managed by system)

## Job Status

**Status Types**:

**Pending** (Orange):

* Job is queued, waiting for resources
* Will start when cluster resources become available

**Running** (Blue):

* Job is actively executing
* Resources are allocated
* Progress is being tracked

**Completed** (Green):

* Job finished successfully
* Results are available
* Resources have been released

**Failed** (Red):

* Job encountered an error
* Check logs for error details
* Can be restarted or debugged

## Managing Jobs

### Stopping a Running Job

To stop a job that's currently running:

1. Open job details or click actions menu
2. Click **Stop** button
3. Confirm action
4. Job will be terminated and resources released

> \[!WARNING] Stopping a job will lose all progress. Consider checkpointing your training jobs.

### Restarting a Failed Job

To restart a job that failed:

1. Open failed job details
2. Click **Restart** button
3. Job will be resubmitted to the queue
4. Monitor for success

### Deleting a Job

To remove a job:

1. Navigate to job details or list
2. Click **Delete** button
3. Confirm deletion

> \[!WARNING] Deleting a job will permanently remove:
>
> * Job configuration
> * Training logs
> * Checkpoints and outputs
> * This action cannot be undone!

**Before Deleting**:

* Download important logs
* Save model checkpoints
* Export results if needed

## Job Monitoring

### Real-time Progress

Monitor job progress in real-time:

* Progress bar shows completion percentage
* View live logs in job details
* Track resource utilization
* Monitor metrics and loss curves

### Job Logs

Access job logs:

1. Open job details
2. Navigate to **Logs** tab
3. View stdout/stderr output
4. Filter by log level
5. Download logs for offline analysis

### Resource Usage

Monitor resource consumption:

* CPU utilization
* Memory usage
* GPU utilization (if applicable)
* Network I/O
* Disk I/O

## Job Scheduling

**Priority-based Scheduling**:

* **High** priority jobs run first
* **Medium** priority jobs run when high-priority queue is empty
* **Low** priority jobs run when resources are available

**Resource Allocation**:

* Jobs are matched to suitable cluster nodes
* GPU jobs require GPU-enabled nodes
* CPU/Memory requirements must be met

**Queue Management**:

* View pending jobs in queue
* Estimate wait time based on current load
* Adjust priority if needed

## Best Practices

**Job Naming**:

* Use descriptive names: `bert-sentiment-imdb-v1`
* Include model, task, and version
* Keep names concise but informative

**Resource Requests**:

* Request only what you need
* Don't over-allocate resources
* Monitor actual usage and adjust

**Checkpointing**:

* Save checkpoints regularly
* Enable auto-save in training code
* Store checkpoints in persistent storage

**Error Handling**:

* Implement retry logic
* Log errors comprehensively
* Set up failure notifications

**Priority Usage**:

* Use High priority sparingly
* Reserve for urgent production jobs
* Most jobs should be Medium priority

## Next Steps

* Run jobs on [Cluster](/kaisar-network/kaisar-ai-ops/deep-learning-platform/cluster.md) nodes
* Link jobs to [Experiments](/kaisar-network/kaisar-ai-ops/deep-learning-platform/experiments.md)
* Monitor performance in [Analytics](/kaisar-network/kaisar-ai-ops/deep-learning-platform/analytics.md)
* Deploy successful models via [Deployments](/kaisar-network/kaisar-ai-ops/deep-learning-platform/deployments.md)