Crossplane Operations: End-User Guide

Overview

Crossplane Operations enable you to build day-two operational workflows using operation functions. Instead of just creating and managing resources, you can now automate operational tasks like rolling upgrades, database backups, configuration validation, and resource cleanup.

Operations work similarly to Crossplane composite resources (XRs) but run once to completion (like Kubernetes Jobs) rather than continuously reconciling desired state. Like XRs use composition functions, Operations use operation functions to execute their workflows. You can trigger operations manually, on a schedule, or when resources change.

flowchart TD
    A[User] --> B[Manual Operation]
    C[Cron Schedule] --> D[CronOperation]
    E[Resource Change] --> F[WatchOperation]
    
    B --> G[Creates Operation]
    D --> G
    F --> G
    
    G --> H[Function Pipeline]
    H --> I[Operation Complete]
    
    D -.->|"schedule: '0 2 * * *'"| D
    F -.->|watches| E

Enabling Operations

Operations are currently an alpha feature and must be explicitly enabled in your Crossplane installation.

Enable via Helm Installation

When installing Crossplane with Helm, add the --enable-operations flag to the args:

1helm upgrade --install crossplane crossplane-stable/crossplane \
2  --namespace crossplane-system \
3  --create-namespace \
4  --set args='{"--enable-operations"}'

Getting Started

Let’s create your first Operation to see how it works:

1. Install the function-dummy function (v0.4.1 or above supports operations):

1apiVersion: pkg.crossplane.io/v1
2kind: Function
3metadata:
4  name: function-dummy
5spec:
6  package: xpkg.crossplane.io/crossplane-contrib/function-dummy:v0.4.1

1kubectl apply -f function-dummy.yaml
2# Wait for it to be ready
3kubectl get function function-dummy -w

2. Create a simple Operation that runs a function:

 1apiVersion: ops.crossplane.io/v1alpha1
 2kind: Operation
 3metadata:
 4  name: my-first-operation
 5spec:
 6  mode: Pipeline
 7  pipeline:
 8  - step: create-configmap
 9    functionRef:
10      name: function-dummy
11    input:
12      apiVersion: dummy.fn.crossplane.io/v1beta1
13      kind: Response
14      # This is a YAML-serialized RunFunctionResponse. function-dummy will
15      # overlay the desired state on any that was passed into it.
16      response:
17        desired:
18          resources:
19            configmap:
20              resource:
21                apiVersion: v1
22                kind: ConfigMap
23                metadata:
24                  namespace: default
25                  name: hello-from-operations
26                data:
27                  message: "Hello from Operations!"
28        results:
29         - severity: SEVERITY_NORMAL
30           message: "I am doing an operate!"

3. Monitor the operation with kubectl:

 1# Watch the operation status
 2kubectl get operation my-first-operation -w
 3
 4# Check detailed status
 5kubectl describe operation my-first-operation
 6
 7# View events
 8kubectl get events --field-selector involvedObject.name=my-first-operation
 9
10# Check if the ConfigMap was created
11kubectl get configmap hello-from-operations -o yaml

4. Try a scheduled operation with CronOperation:

 1apiVersion: ops.crossplane.io/v1alpha1
 2kind: CronOperation
 3metadata:
 4  name: daily-hello
 5spec:
 6  schedule: "0 9 * * *"  # Daily at 9 AM
 7  operationTemplate:
 8    spec:
 9      mode: Pipeline
10      pipeline:
11      - step: hello
12        functionRef:
13          name: function-dummy
14        input:
15          apiVersion: dummy.fn.crossplane.io/v1beta1
16          kind: Response
17          response:
18            results:
19             - severity: SEVERITY_NORMAL
20               message: "Good morning from CronOperation!"

Core Concepts

Operation

An Operation runs a function pipeline once to completion. It’s the fundamental building block for operational workflows.

 1apiVersion: ops.crossplane.io/v1alpha1
 2kind: Operation
 3metadata:
 4  name: backup-database
 5spec:
 6  mode: Pipeline
 7  pipeline:
 8  - step: create-backup
 9    functionRef:
10      name: function-database-backup
11    input:
12      apiVersion: fn.crossplane.io/v1beta1
13      kind: DatabaseBackupInput
14      database: my-production-db
15      retentionDays: 30

Key Features:

• Runs once to completion with success/failure status
• Can mutate any Kubernetes resources using server-side apply
• Supports multi-step pipelines with function composition
• Provides detailed status and results from each pipeline step

CronOperation

A CronOperation creates Operation resources on a schedule, like Kubernetes CronJobs.

 1apiVersion: ops.crossplane.io/v1alpha1
 2kind: CronOperation
 3metadata:
 4  name: daily-backup
 5spec:
 6  schedule: "0 2 * * *"  # Daily at 2 AM
 7  concurrencyPolicy: Forbid
 8  successfulHistoryLimit: 5
 9  failedHistoryLimit: 3
10  operationTemplate:
11    spec:
12      mode: Pipeline
13      pipeline:
14      - step: backup
15        functionRef:
16          name: function-database-backup
17        input:
18          apiVersion: fn.crossplane.io/v1beta1
19          kind: DatabaseBackupInput
20          retentionDays: 7

Key Features:

• Standard cron scheduling syntax
• Configurable concurrency policies (Allow, Forbid, Replace)
• Automatic garbage collection of old Operations
• Tracks last successful execution and running operations

WatchOperation

A WatchOperation creates Operation resources when watched Kubernetes resources change.

 1apiVersion: ops.crossplane.io/v1alpha1
 2kind: WatchOperation
 3metadata:
 4  name: app-deployment-monitor
 5spec:
 6  watch:
 7    apiVersion: apps/v1
 8    kind: Deployment
 9    namespace: production
10    matchLabels:
11      app: my-app
12  concurrencyPolicy: Allow
13  operationTemplate:
14    spec:
15      mode: Pipeline
16      pipeline:
17      - step: validate-deployment
18        functionRef:
19          name: function-deployment-validator
20        input:
21          apiVersion: fn.crossplane.io/v1beta1
22          kind: DeploymentValidatorInput

Key Features:

• Watches any Kubernetes resource type
• Supports namespace and label filtering
• Automatically injects the changed resource into the operation
• Tracks count of watched resources and operation history

Operation Lifecycle

Understanding how Operations work internally helps you debug and monitor them effectively:

stateDiagram-v2
    [*] --> Created: User/CronOperation/WatchOperation creates Operation
    Created --> Validating: Operation controller reconciles
    Validating --> Invalid: Function capabilities missing
    Validating --> Running: All validations pass
    Invalid --> [*]: Operation fails
    
    Running --> PipelineStep1: Execute first function
    PipelineStep1 --> PipelineStep2: Pass context/state
    PipelineStep2 --> PipelineStepN: Continue pipeline
    PipelineStepN --> ApplyingResources: All functions complete
    
    ApplyingResources --> ResourcesApplied: Server-side apply with force
    ResourcesApplied --> Completed: Update Operation status
    
    Running --> Failed: Function returns fatal error
    PipelineStep1 --> Failed: Function error
    PipelineStep2 --> Failed: Function error
    PipelineStepN --> Failed: Function error
    ApplyingResources --> Failed: Apply failure
    
    Failed --> Retrying: Retry count < retryLimit
    Retrying --> Running: Exponential backoff
    Failed --> [*]: Max retries exceeded
    
    Completed --> [*]: Operation finished
    
    note right of ResourcesApplied
        Resources created with force ownership
        No owner references added
        No garbage collection
    end note

Common Use Cases

1. Scheduled Database Backups

 1apiVersion: ops.crossplane.io/v1alpha1
 2kind: CronOperation
 3metadata:
 4  name: postgres-backup
 5spec:
 6  schedule: "0 3 * * *"  # Daily at 3 AM
 7  operationTemplate:
 8    spec:
 9      mode: Pipeline
10      pipeline:
11      - step: backup
12        functionRef:
13          name: function-postgres-backup
14        input:
15          apiVersion: fn.crossplane.io/v1beta1
16          kind: PostgresBackupInput
17          instance: production-db
18          s3Bucket: db-backups

2. Rolling Cluster Upgrades

 1apiVersion: ops.crossplane.io/v1alpha1
 2kind: Operation
 3metadata:
 4  name: cluster-upgrade
 5spec:
 6  mode: Pipeline
 7  pipeline:
 8  - step: upgrade
 9    functionRef:
10      name: function-cluster-upgrade
11    input:
12      apiVersion: fn.crossplane.io/v1beta1
13      kind: ClusterUpgradeInput
14      targetVersion: "1.28"
15      batches: [0.25, 0.5, 1.0]  # Upgrade 25%, then 50%, then 100%
16      healthChecks:
17      - Synced
18      - Ready

3. Resource Change Reactions

 1apiVersion: ops.crossplane.io/v1alpha1
 2kind: WatchOperation
 3metadata:
 4  name: config-validator
 5spec:
 6  watch:
 7    apiVersion: v1
 8    kind: ConfigMap
 9    matchLabels:
10      validate: "true"
11  operationTemplate:
12    spec:
13      mode: Pipeline
14      pipeline:
15      - step: validate
16        functionRef:
17          name: function-config-validator
18        input:
19          apiVersion: fn.crossplane.io/v1beta1
20          kind: ConfigValidatorInput
21      - step: notify
22        functionRef:
23          name: function-slack-notifier
24        input:
25          apiVersion: fn.crossplane.io/v1beta1
26          kind: SlackNotifierInput

Working with Required Resources

Operations can declare required resources that functions need access to. This is especially useful for WatchOperations where you want to process the changed resource.

 1apiVersion: ops.crossplane.io/v1alpha1
 2kind: Operation
 3metadata:
 4  name: process-config
 5spec:
 6  mode: Pipeline
 7  pipeline:
 8  - step: process
 9    functionRef:
10      name: function-config-processor
11    input:
12      apiVersion: fn.crossplane.io/v1beta1
13      kind: ConfigProcessorInput
14    requirements:
15      requiredResources:
16      - requirementName: source-config
17        apiVersion: v1
18        kind: ConfigMap
19        name: app-config
20        namespace: default

For WatchOperations, the watched resource is automatically injected with the special requirement name ops.crossplane.io/watched-resource.

Configuration Options

Concurrency Policies

• Allow: Multiple operations can run simultaneously (default)
• Forbid: New operations won’t start if others are running
• Replace: New operations terminate running ones

History Limits

• successfulHistoryLimit: Number of successful operations to keep (default: 3)
• failedHistoryLimit: Number of failed operations to keep (default: 1)

Pause Operations

Pause any operation type using the standard Crossplane annotation:

1metadata:
2  annotations:
3    crossplane.io/paused: "true"

Status and Monitoring

All operation types provide rich status information:

 1status:
 2  conditions:
 3  - type: Synced
 4    status: "True"
 5    reason: ReconcileSuccess
 6  - type: Scheduling  # CronOperation only
 7    status: "True"
 8    reason: ScheduleActive
 9  - type: Watching    # WatchOperation only
10    status: "True"
11    reason: WatchActive
12  watchingResources: 12  # WatchOperation only
13  lastScheduleTime: "2024-01-15T10:00:00Z"
14  lastSuccessfulTime: "2024-01-15T10:02:30Z"
15  runningOperationRefs:
16  - name: daily-backup-1705305600

Events

Each operation type emits Kubernetes events for important lifecycle activities:

Operation Events:

• RunPipelineStep (Normal/Warning) - Function execution results and warnings
• FunctionInvocation (Warning) - Function invocation failures
• InvalidOutput (Warning) - Function output marshaling errors
• InvalidResource (Warning) - Resource application failures
• InvalidPipeline (Warning) - Function capability check failures
• BootstrapRequirements (Warning) - Required resource fetch failures
• MaxFailures (Warning) - Operation exceeded retry limit

CronOperation Events:

• CreateOperation (Warning) - Scheduled operation creation failures
• GarbageCollectOperations (Warning) - Garbage collection failures
• ReplaceRunningOperation (Warning) - Running operation deletion failures during replace policy
• InvalidSchedule (Warning) - Cron schedule parsing errors
• ListOperations (Warning) - Operation listing failures

WatchOperation Events:

• EstablishWatched (Warning) - Watch establishment failures
• TerminateWatched (Warning) - Watch termination failures
• GarbageCollectOperations (Warning) - Operation cleanup failures
• CreateOperation (Warning) - Operation creation failures (from watched controller)
• ReplaceRunningOperation (Warning) - Running operation deletion failures during replace policy (from watched controller)
• ListOperations (Warning) - Operation listing failures (from watched controller)

Prometheus Metrics

Operations expose comprehensive Prometheus metrics for monitoring function execution, controller management, and caching performance.

Function Execution Metrics

Operations use these metrics when executing operation functions:

Note: These metrics are named crossplane_composition_* for historical reasons but apply to both composition functions and operation functions. They may be renamed to crossplane_functions_* in a future release.

• crossplane_composition_run_function_request_total - Counter of function requests sent

  • Labels: function_name, function_package, grpc_target, grpc_method

• crossplane_composition_run_function_response_total - Counter of function responses received

  • Labels: function_name, function_package, grpc_target, grpc_method, grpc_code, result_severity

• crossplane_composition_run_function_seconds - Histogram of function execution latency

  • Labels: function_name, function_package, grpc_target, grpc_method, grpc_code, result_severity

Controller Engine Metrics

WatchOperations use these metrics when managing dynamic controllers:

• crossplane_engine_controllers_started_total - Counter of controllers started

  • Labels: controller (follows pattern watched/{watch-operation-name})

• crossplane_engine_controllers_stopped_total - Counter of controllers stopped

  • Labels: controller

• crossplane_engine_watches_started_total - Counter of watches started

  • Labels: controller, type (will be WatchOperation for WatchOperations)

• crossplane_engine_watches_stopped_total - Counter of watches stopped

  • Labels: controller, type

Function Response Cache Metrics

Operations use these metrics when function response caching is enabled:

• crossplane_composition_run_function_response_cache_hits_total - Counter of cache hits

  • Labels: function_name

• crossplane_composition_run_function_response_cache_misses_total - Counter of cache misses

  • Labels: function_name

• crossplane_composition_run_function_response_cache_errors_total - Counter of cache errors

  • Labels: function_name

• crossplane_composition_run_function_response_cache_writes_total - Counter of cache writes

  • Labels: function_name

• crossplane_composition_run_function_response_cache_deletes_total - Counter of cache deletes

  • Labels: function_name

• crossplane_composition_run_function_response_cache_bytes_written_total - Counter of bytes written to cache

  • Labels: function_name

• crossplane_composition_run_function_response_cache_bytes_deleted_total - Counter of bytes deleted from cache

  • Labels: function_name

• crossplane_composition_run_function_response_cache_read_seconds - Histogram of cache read latency

  • Labels: function_name

• crossplane_composition_run_function_response_cache_write_seconds - Histogram of cache write latency

  • Labels: function_name

Monitoring Recommendations

For Operations:

• Monitor function execution latency using the _seconds histogram
• Track function failure rates using result_severity labels (Fatal, Warning, Normal)
• Monitor cache hit ratios when caching is enabled

For WatchOperations:

• Monitor controller lifecycle using engine controller metrics
• Track watch establishment/termination using engine watch metrics
• Alert on controller or watch failures

For CronOperations:

• Monitor scheduled execution patterns through standard Operation metrics
• Track garbage collection effectiveness through controller logs and events

Higher-Level Metrics with Recording Rules

You can create higher-level operational metrics using Prometheus recording rules. These precomputed metrics provide better performance for dashboards and alerting:

Function Success Rate:

1groups:
2- name: crossplane-operations
3  rules:
4  - record: crossplane:operation_function_success_rate5m
5    expr: |
6      sum(rate(crossplane_composition_run_function_response_total{result_severity="Normal"}[5m])) by (function_name)
7      /
8      sum(rate(crossplane_composition_run_function_response_total[5m])) by (function_name)      

Function Average Latency:

1  - record: crossplane:operation_function_latency_avg5m
2    expr: |
3      sum(rate(crossplane_composition_run_function_seconds_sum[5m])) by (function_name)
4      /
5      sum(rate(crossplane_composition_run_function_seconds_count[5m])) by (function_name)      

Operation Throughput by Type:

1  - record: crossplane:operation_throughput_by_type5m
2    expr: |
3      sum(rate(crossplane_composition_run_function_request_total[5m])) by (function_name)      

WatchOperation Controller Health:

1  - record: crossplane:watchoperation_controller_health
2    expr: |
3      sum(rate(crossplane_engine_controllers_started_total[5m])) by (controller)
4      -
5      sum(rate(crossplane_engine_controllers_stopped_total[5m])) by (controller)      

Function Cache Hit Rate:

1  - record: crossplane:function_cache_hit_rate5m
2    expr: |
3      sum(rate(crossplane_composition_run_function_response_cache_hits_total[5m])) by (function_name)
4      /
5      (
6        sum(rate(crossplane_composition_run_function_response_cache_hits_total[5m])) by (function_name)
7        +
8        sum(rate(crossplane_composition_run_function_response_cache_misses_total[5m])) by (function_name)
9      )      

Operation Error Rate by Severity:

1  - record: crossplane:operation_error_rate_by_severity5m
2    expr: |
3      sum(rate(crossplane_composition_run_function_response_total{result_severity!="Normal"}[5m])) by (function_name, result_severity)
4      /
5      sum(rate(crossplane_composition_run_function_response_total[5m])) by (function_name)      

WatchOperation Activity Level:

1  - record: crossplane:watchoperation_activity_level5m
2    expr: |
3      sum(rate(crossplane_engine_watches_started_total{type="WatchOperation"}[5m])) by (controller)      

These recording rules enable you to:

• Create dashboards showing operation success rates and latencies
• Set up alerting on function failure rates or cache miss rates
• Monitor WatchOperation controller stability
• Track operational throughput and performance trends
• Identify problematic functions or operations quickly

WatchOperation Workflow

sequenceDiagram
    participant U as User
    participant WO as WatchOperation
    participant WC as Watched Controller
    participant K8s as Kubernetes API
    participant Op as Operation
    participant Fn as Function

    U->>WO: Creates WatchOperation
    WO->>WC: Starts watched controller
    WC->>K8s: Watches specified resources
    
    loop Resource Changes
        K8s-->>WC: Resource change event
        WC->>WC: Check concurrency policy
        alt Allow/No running ops
            WC->>Op: Create new Operation
            WC->>Op: Inject watched resource
            Op->>Fn: Execute function pipeline
            Fn->>K8s: Apply resources (force ownership)
            Fn-->>Op: Return results
            Op->>Op: Update status
        else Forbid/Running ops exist
            WC->>WC: Skip operation creation
        else Replace/Running ops exist
            WC->>K8s: Delete running operations
            WC->>Op: Create new Operation
        end
    end
    
    WO->>WO: Garbage collect old operations

Best Practices

1. Use descriptive names for operations that include their purpose
2. Set appropriate history limits to balance debugging and resource usage
3. Choose concurrency policies carefully based on your operational needs
4. Monitor operation status through standard Kubernetes tooling
5. Test operations thoroughly before deploying to production
6. Use required resources to make function dependencies explicit
7. Implement proper error handling in your operation functions

Package Distribution

Operations can be packaged and distributed using Crossplane Configurations:

1apiVersion: meta.pkg.crossplane.io/v1
2kind: Configuration
3metadata:
4  name: database-operations
5spec:
6  dependsOn:
7  - provider: xpkg.upbound.io/crossplane-contrib/provider-sql
8    version: ">=v0.5.0"

This allows you to version and distribute operational workflows alongside your platform configurations.

Function Development

Operations use operation functions that are built like composition functions but designed for operational workflows. Operation functions use the same function SDK as composition functions. Key differences:

• Functions receive required resources instead of observed composite resources
• Functions can return output in the response for status tracking
• Functions should focus on operational logic rather than resource composition

Function Capabilities

Crossplane validates that functions have the appropriate capabilities for their intended use. Functions must explicitly declare the operation capability to be used in Operations. Functions must declare the composition capability to be used in Compositions. Functions without any declared capabilities default to having the composition capability for backward compatibility.

Example operation function package metadata (crossplane.yaml):

1apiVersion: meta.pkg.crossplane.io/v1
2kind: Function
3metadata:
4  name: function-database-backup
5spec:
6  capabilities:
7  - operation
8  crossplane:
9    version: ">=v1.18.0"

Example FunctionRevision status showing capabilities:

1apiVersion: pkg.crossplane.io/v1
2kind: FunctionRevision
3metadata:
4  name: function-database-backup-abc123
5status:
6  capabilities:
7  - operation
8  # ... other status fields

Important: You must use a Crossplane CLI (crank) build with capability support when building function packages. Older versions of the CLI will silently drop the capabilities field during package build time, causing capability validation to fail at runtime.

How Operation Functions Work

Operation functions use the same gRPC interface as composition functions, including the same RunFunctionRequest and RunFunctionResponse messages. However, there are important differences in how Operations use these messages:

Key Differences from Composition Functions:

1. No Observed Resources: Crossplane never sends observed resources to operation functions. The observed.resources field in RunFunctionRequest will be empty.

2. Required Resources Only: Operation functions only receive resources they explicitly request via requirements.resources in RunFunctionResponse, or that are specified in the pipeline step’s requirements field.

3. Resource Creation: Functions can create or update arbitrary Kubernetes resources by returning them in desired.resources in RunFunctionResponse.

4. No Owner References: Crossplane doesn’t establish ownership relationships between Operations and the resources they create - no owner references are added automatically.

5. No Garbage Collection: Operation functions cannot delete resources, including resources they previously created. Resources created by operations are not garbage collected when the operation completes. This is a known limitation of the alpha implementation.

6. Function Output: Unlike composition functions, operation functions can return output in RunFunctionResponse.output which is stored in the Operation’s status for tracking and debugging.

7. Force Ownership: Operations use server-side apply with force ownership when applying resources, which means they can take ownership of fields from other controllers.

Important Considerations:

• Take care with WatchOperations: Since operations force ownership when applying resources, ensure that WatchOperations don’t conflict with other controllers managing the same resources.

• Connection Details and Conditions: Fields like connection_details and conditions are ignored by Operations as they’re composition-specific.

• Ready State: The ready field is also ignored by Operations since they run to completion rather than maintaining desired state.

Function Pipeline Comparison

flowchart TD
    subgraph "Composition Functions"
        A1[Composite Resource] --> B1[Function Pipeline]
        C1[Observed Resources] --> B1
        B1 --> D1[Desired Resources]
        D1 --> E1[Crossplane Controller]
        E1 --> F1[Creates/Updates Resources]
        F1 --> G1[Owner References Added]
        G1 --> H1[Garbage Collection]
    end
    
    subgraph "Operation Functions"
        A2[Operation] --> B2[Function Pipeline]
        C2[Required Resources Only] --> B2
        B2 --> D2[Arbitrary Resources]
        D2 --> E2[Operation Controller]
        E2 --> F2[Force Apply Resources]
        F2 --> G2[No Owner References]
        G2 --> H2[No Garbage Collection]
    end

API Reference

Operation

The Operation type defines a pipeline of functions that runs once to completion.

spec

spec.pipeline[*]

spec.pipeline[].credentials[]

spec.pipeline[*].requirements

spec.pipeline[].requirements.requiredResources[]

status

status.pipeline[*]

status.appliedResourceRefs[*]

Operation Status Conditions

Operation Annotations

CronOperation

The CronOperation type creates Operation resources on a cron schedule.

spec

status

CronOperation Status Conditions

CronOperation Annotations

CronOperation Labels (on created Operations)

WatchOperation

The WatchOperation type creates Operation resources when watched Kubernetes resources change.

spec

spec.watch

status

WatchOperation Status Conditions

WatchOperation Annotations

WatchOperation Labels (on created Operations)

WatchOperation Annotations (on created Operations)

Common Types

ConcurrencyPolicy

spec.operationTemplate

status.runningOperationRefs[*]