Manage Databricks clusters for running Spark jobs including creating, starting, editing, listing, terminating, and deleting clusters.
openapi: 3.1.0
info:
title: Azure Databricks REST API
description: >-
Core REST API for managing Azure Databricks workspaces, clusters, jobs,
and workspace objects programmatically. This specification covers the
Clusters API (2.0), Jobs API (2.1), and Workspace API (2.0) endpoints
available on Azure Databricks instances.
version: 2.1.0
contact:
name: Azure Databricks Support
url: https://learn.microsoft.com/answers/tags/166/azure-databricks
license:
name: Microsoft Azure Terms
url: https://azure.microsoft.com/en-us/support/legal/
x-apiReferenceDocumentation: https://docs.databricks.com/api/azure/workspace/introduction
servers:
- url: https://{databricks_instance}.azuredatabricks.net/api
description: Azure Databricks workspace instance
variables:
databricks_instance:
default: adb-1234567890123456.1
description: >-
The unique identifier for your Azure Databricks workspace instance.
Found in the workspace URL.
security:
- BearerAuth: []
- AzureADToken: []
tags:
- name: Clusters
description: >-
Manage Databricks clusters for running Apache Spark workloads. Create,
start, restart, resize, terminate, and permanently delete clusters.
externalDocs:
description: Clusters API reference
url: https://docs.databricks.com/api/azure/workspace/clusters
- name: Jobs
description: >-
Create and manage jobs to run notebooks, JARs, Spark submit, Python
scripts, and pipeline tasks on Databricks clusters. Schedule recurring
jobs, trigger runs, and monitor execution.
externalDocs:
description: Jobs API reference
url: https://docs.databricks.com/api/azure/workspace/jobs
- name: Workspace
description: >-
Manage workspace objects such as notebooks, folders, files, and repos.
Import, export, list, and delete workspace objects.
externalDocs:
description: Workspace API reference
url: https://docs.databricks.com/api/azure/workspace/workspace
paths:
/2.0/clusters/create:
post:
operationId: createCluster
summary: Azure Databricks Create a New Cluster
description: >-
Creates a new Apache Spark cluster. Returns the ID of the newly created
cluster. The cluster starts in a PENDING state and transitions to
RUNNING when ready. Optionally, you can attach libraries to the cluster
after creation.
tags:
- Clusters
requestBody:
required: true
content:
application/json:
schema:
$ref: '#/components/schemas/CreateClusterRequest'
examples:
CreateclusterRequestExample:
summary: Default createCluster request
x-microcks-default: true
value:
cluster_name: example_value
spark_version: example_value
node_type_id: '500123'
driver_node_type_id: '500123'
num_workers: 10
autoscale:
min_workers: 10
max_workers: 10
spark_conf: example_value
azure_attributes:
first_on_demand: 10
availability: SPOT_AZURE
spot_bid_max_price: 42.5
ssh_public_keys:
- example_value
custom_tags: example_value
cluster_log_conf:
dbfs:
destination: example_value
s3:
destination: example_value
region: example_value
endpoint: example_value
init_scripts:
- workspace: {}
volumes: {}
dbfs: {}
abfss: {}
spark_env_vars: example_value
enable_elastic_disk: true
instance_pool_id: '500123'
policy_id: '500123'
enable_local_disk_encryption: true
runtime_engine: STANDARD
data_security_mode: NONE
single_user_name: example_value
responses:
'200':
description: Cluster created successfully
content:
application/json:
schema:
type: object
properties:
cluster_id:
type: string
description: Canonical identifier for the newly created cluster
examples:
- "1234-567890-abcde123"
examples:
Createcluster200Example:
summary: Default createCluster 200 response
x-microcks-default: true
value:
cluster_id: '500123'
'400':
$ref: '#/components/responses/BadRequest'
'401':
$ref: '#/components/responses/Unauthorized'
'403':
$ref: '#/components/responses/Forbidden'
'429':
$ref: '#/components/responses/TooManyRequests'
'500':
$ref: '#/components/responses/InternalError'
x-microcks-operation:
delay: 0
dispatcher: FALLBACK
/2.0/clusters/edit:
post:
operationId: editCluster
summary: Azure Databricks Edit a Cluster
description: >-
Edits the configuration of an existing cluster. The cluster must be in
a RUNNING or TERMINATED state. If the cluster is running, it will be
restarted with the new configuration.
tags:
- Clusters
requestBody:
required: true
content:
application/json:
schema:
allOf:
- type: object
required:
- cluster_id
properties:
cluster_id:
type: string
description: ID of the cluster to edit
- $ref: '#/components/schemas/CreateClusterRequest'
examples:
EditclusterRequestExample:
summary: Default editCluster request
x-microcks-default: true
value: {}
responses:
'200':
description: Cluster edited successfully
content:
application/json:
schema:
type: object
examples:
Editcluster200Example:
summary: Default editCluster 200 response
x-microcks-default: true
value: {}
'400':
$ref: '#/components/responses/BadRequest'
'401':
$ref: '#/components/responses/Unauthorized'
'403':
$ref: '#/components/responses/Forbidden'
'429':
$ref: '#/components/responses/TooManyRequests'
'500':
$ref: '#/components/responses/InternalError'
x-microcks-operation:
delay: 0
dispatcher: FALLBACK
/2.0/clusters/start:
post:
operationId: startCluster
summary: Azure Databricks Start a Terminated Cluster
description: >-
Starts a terminated cluster given its ID. Works only for clusters in a
TERMINATED state. Uses the last specified cluster configuration.
tags:
- Clusters
requestBody:
required: true
content:
application/json:
schema:
type: object
required:
- cluster_id
properties:
cluster_id:
type: string
description: ID of the cluster to start
examples:
StartclusterRequestExample:
summary: Default startCluster request
x-microcks-default: true
value:
cluster_id: '500123'
responses:
'200':
description: Cluster start initiated successfully
content:
application/json:
schema:
type: object
examples:
Startcluster200Example:
summary: Default startCluster 200 response
x-microcks-default: true
value: {}
'400':
$ref: '#/components/responses/BadRequest'
'401':
$ref: '#/components/responses/Unauthorized'
'403':
$ref: '#/components/responses/Forbidden'
'429':
$ref: '#/components/responses/TooManyRequests'
'500':
$ref: '#/components/responses/InternalError'
x-microcks-operation:
delay: 0
dispatcher: FALLBACK
/2.0/clusters/restart:
post:
operationId: restartCluster
summary: Azure Databricks Restart a Running Cluster
description: >-
Restarts a running cluster given its ID. The cluster must be in a
RUNNING state.
tags:
- Clusters
requestBody:
required: true
content:
application/json:
schema:
type: object
required:
- cluster_id
properties:
cluster_id:
type: string
description: ID of the cluster to restart
examples:
RestartclusterRequestExample:
summary: Default restartCluster request
x-microcks-default: true
value:
cluster_id: '500123'
responses:
'200':
description: Cluster restart initiated successfully
content:
application/json:
schema:
type: object
examples:
Restartcluster200Example:
summary: Default restartCluster 200 response
x-microcks-default: true
value: {}
'400':
$ref: '#/components/responses/BadRequest'
'401':
$ref: '#/components/responses/Unauthorized'
'403':
$ref: '#/components/responses/Forbidden'
'429':
$ref: '#/components/responses/TooManyRequests'
'500':
$ref: '#/components/responses/InternalError'
x-microcks-operation:
delay: 0
dispatcher: FALLBACK
/2.0/clusters/delete:
post:
operationId: terminateCluster
summary: Azure Databricks Terminate a Cluster
description: >-
Terminates a cluster given its ID. The cluster is removed from the
running state but its configuration is preserved so it can be restarted.
Use permanent-delete to fully remove a cluster.
tags:
- Clusters
requestBody:
required: true
content:
application/json:
schema:
type: object
required:
- cluster_id
properties:
cluster_id:
type: string
description: ID of the cluster to terminate
examples:
TerminateclusterRequestExample:
summary: Default terminateCluster request
x-microcks-default: true
value:
cluster_id: '500123'
responses:
'200':
description: Cluster termination initiated successfully
content:
application/json:
schema:
type: object
examples:
Terminatecluster200Example:
summary: Default terminateCluster 200 response
x-microcks-default: true
value: {}
'400':
$ref: '#/components/responses/BadRequest'
'401':
$ref: '#/components/responses/Unauthorized'
'403':
$ref: '#/components/responses/Forbidden'
'429':
$ref: '#/components/responses/TooManyRequests'
'500':
$ref: '#/components/responses/InternalError'
x-microcks-operation:
delay: 0
dispatcher: FALLBACK
/2.0/clusters/permanent-delete:
post:
operationId: permanentDeleteCluster
summary: Azure Databricks Permanently Delete a Cluster
description: >-
Permanently deletes a Spark cluster. If the cluster is running, it is
terminated and its resources freed. The cluster is removed permanently
and cannot be restarted.
tags:
- Clusters
requestBody:
required: true
content:
application/json:
schema:
type: object
required:
- cluster_id
properties:
cluster_id:
type: string
description: ID of the cluster to permanently delete
examples:
PermanentdeleteclusterRequestExample:
summary: Default permanentDeleteCluster request
x-microcks-default: true
value:
cluster_id: '500123'
responses:
'200':
description: Cluster permanently deleted
content:
application/json:
schema:
type: object
examples:
Permanentdeletecluster200Example:
summary: Default permanentDeleteCluster 200 response
x-microcks-default: true
value: {}
'400':
$ref: '#/components/responses/BadRequest'
'401':
$ref: '#/components/responses/Unauthorized'
'403':
$ref: '#/components/responses/Forbidden'
'429':
$ref: '#/components/responses/TooManyRequests'
'500':
$ref: '#/components/responses/InternalError'
x-microcks-operation:
delay: 0
dispatcher: FALLBACK
/2.0/clusters/get:
get:
operationId: getCluster
summary: Azure Databricks Get Cluster Information
description: >-
Retrieves the information for a cluster given its identifier. Returns
the current state, configuration, and metadata for the cluster.
tags:
- Clusters
parameters:
- name: cluster_id
in: query
required: true
description: The cluster about which to retrieve information
schema:
type: string
example: '500123'
responses:
'200':
description: Cluster information retrieved successfully
content:
application/json:
schema:
$ref: '#/components/schemas/ClusterInfo'
examples:
Getcluster200Example:
summary: Default getCluster 200 response
x-microcks-default: true
value:
cluster_id: '500123'
cluster_name: example_value
spark_version: example_value
node_type_id: '500123'
driver_node_type_id: '500123'
num_workers: 10
autoscale:
min_workers: 10
max_workers: 10
state: PENDING
state_message: example_value
creator_user_name: example_value
start_time: 10
terminated_time: 10
last_state_loss_time: 10
last_activity_time: 10
autotermination_minutes: 10
cluster_source: UI
spark_conf: example_value
azure_attributes:
first_on_demand: 10
availability: SPOT_AZURE
spot_bid_max_price: 42.5
custom_tags: example_value
cluster_log_conf:
dbfs:
destination: example_value
s3:
destination: example_value
region: example_value
endpoint: example_value
init_scripts:
- workspace: {}
volumes: {}
dbfs: {}
abfss: {}
spark_env_vars: example_value
enable_elastic_disk: true
instance_pool_id: '500123'
policy_id: '500123'
default_tags: example_value
cluster_log_status:
last_attempted: 10
last_exception: example_value
termination_reason:
code: example_value
type: example_value
parameters: example_value
data_security_mode: example_value
single_user_name: example_value
runtime_engine: example_value
disk_spec:
disk_count: 10
disk_size: 10
disk_type:
azure_disk_volume_type: example_value
executors:
- private_ip: example_value
public_dns: example_value
node_id: '500123'
instance_id: '500123'
start_timestamp: 10
host_private_ip: example_value
driver:
private_ip: example_value
public_dns: example_value
node_id: '500123'
instance_id: '500123'
start_timestamp: 10
host_private_ip: example_value
jdbc_port: 10
cluster_memory_mb: 10
cluster_cores: 42.5
'400':
$ref: '#/components/responses/BadRequest'
'401':
$ref: '#/components/responses/Unauthorized'
'403':
$ref: '#/components/responses/Forbidden'
'404':
$ref: '#/components/responses/NotFound'
'429':
$ref: '#/components/responses/TooManyRequests'
'500':
$ref: '#/components/responses/InternalError'
x-microcks-operation:
delay: 0
dispatcher: FALLBACK
/2.0/clusters/list:
get:
operationId: listClusters
summary: Azure Databricks List All Clusters
description: >-
Returns information about all pinned and active clusters, and up to 200
of the most recently terminated all-purpose clusters in the past 30
days, and up to 30 of the most recently terminated job clusters in the
past 30 days.
tags:
- Clusters
parameters:
- name: can_use_client
in: query
required: false
description: >-
Filter clusters based on what type of client can use the cluster.
Possible values are NOTEBOOKS and JOBS.
schema:
type: string
example: example_value
responses:
'200':
description: List of clusters retrieved successfully
content:
application/json:
schema:
type: object
properties:
clusters:
type: array
description: List of cluster information objects
items:
$ref: '#/components/schemas/ClusterInfo'
examples:
Listclusters200Example:
summary: Default listClusters 200 response
x-microcks-default: true
value:
clusters:
- cluster_id: '500123'
cluster_name: example_value
spark_version: example_value
node_type_id: '500123'
driver_node_type_id: '500123'
num_workers: 10
state: PENDING
state_message: example_value
creator_user_name: example_value
start_time: 10
terminated_time: 10
last_state_loss_time: 10
last_activity_time: 10
autotermination_minutes: 10
cluster_source: UI
spark_conf: example_value
custom_tags: example_value
init_scripts:
- {}
spark_env_vars: example_value
enable_elastic_disk: true
instance_pool_id: '500123'
policy_id: '500123'
default_tags: example_value
cluster_log_status:
last_attempted: 10
last_exception: example_value
termination_reason:
code: example_value
type: example_value
parameters: example_value
data_security_mode: example_value
single_user_name: example_value
runtime_engine: example_value
disk_spec:
disk_count: 10
disk_size: 10
disk_type: {}
executors:
- {}
jdbc_port: 10
cluster_memory_mb: 10
cluster_cores: 42.5
'401':
$ref: '#/components/responses/Unauthorized'
'403':
$ref: '#/components/responses/Forbidden'
'429':
$ref: '#/components/responses/TooManyRequests'
'500':
$ref: '#/components/responses/InternalError'
x-microcks-operation:
delay: 0
dispatcher: FALLBACK
/2.0/clusters/pin:
post:
operationId: pinCluster
summary: Azure Databricks Pin a Cluster
description: >-
Pins a cluster to ensure it is always returned by the list clusters
API. Pinning a cluster that is already pinned has no effect.
tags:
- Clusters
requestBody:
required: true
content:
application/json:
schema:
type: object
required:
- cluster_id
properties:
cluster_id:
type: string
description: ID of the cluster to pin
examples:
PinclusterRequestExample:
summary: Default pinCluster request
x-microcks-default: true
value:
cluster_id: '500123'
responses:
'200':
description: Cluster pinned successfully
content:
application/json:
schema:
type: object
examples:
Pincluster200Example:
summary: Default pinCluster 200 response
x-microcks-default: true
value: {}
'400':
$ref: '#/components/responses/BadRequest'
'401':
$ref: '#/components/responses/Unauthorized'
'403':
$ref: '#/components/responses/Forbidden'
'429':
$ref: '#/components/responses/TooManyRequests'
'500':
$ref: '#/components/responses/InternalError'
x-microcks-operation:
delay: 0
dispatcher: FALLBACK
/2.0/clusters/unpin:
post:
operationId: unpinCluster
summary: Azure Databricks Unpin a Cluster
description: >-
Unpins a cluster. Unpinning a cluster that is not pinned has no effect.
tags:
- Clusters
requestBody:
required: true
content:
application/json:
schema:
type: object
required:
- cluster_id
properties:
cluster_id:
type: string
description: ID of the cluster to unpin
examples:
UnpinclusterRequestExample:
summary: Default unpinCluster request
x-microcks-default: true
value:
cluster_id: '500123'
responses:
'200':
description: Cluster unpinned successfully
content:
application/json:
schema:
type: object
examples:
Unpincluster200Example:
summary: Default unpinCluster 200 response
x-microcks-default: true
value: {}
'400':
$ref: '#/components/responses/BadRequest'
'401':
$ref: '#/components/responses/Unauthorized'
'403':
$ref: '#/components/responses/Forbidden'
'429':
$ref: '#/components/responses/TooManyRequests'
'500':
$ref: '#/components/responses/InternalError'
x-microcks-operation:
delay: 0
dispatcher: FALLBACK
/2.0/clusters/events:
post:
operationId: listClusterEvents
summary: Azure Databricks List Cluster Events
description: >-
Retrieves a list of events about the activity of a cluster. Events are
returned in reverse chronological order. This endpoint allows paginating
through cluster events using the next_page field.
tags:
- Clusters
requestBody:
required: true
content:
application/json:
schema:
type: object
required:
- cluster_id
properties:
cluster_id:
type: string
description: ID of the cluster to retrieve events about
start_time:
type: integer
format: int64
description: >-
Start timestamp in epoch milliseconds. If empty, returns
events starting from the beginning of time.
end_time:
type: integer
format: int64
description: >-
End timestamp in epoch milliseconds. If empty, returns
events up to the current time.
order:
type: string
enum:
- DESC
- ASC
description: Sort order by timestamp. Default is DESC.
event_types:
type: array
items:
type: string
enum:
- CREATING
- DID_NOT_EXPAND_DISK
- EXPANDED_DISK
- FAILED_TO_EXPAND_DISK
- INIT_SCRIPTS_STARTED
- INIT_SCRIPTS_FINISHED
- STARTING
- RESTARTING
- TERMINATING
- EDITED
- RUNNING
- RESIZING
- UPSIZE_COMPLETED
- NODES_LOST
- DRIVER_HEALTHY
- DRIVER_NOT_RESPONDING
- DRIVER_UNAVAILABLE
- SPARK_EXCEPTION
- PINNED
- UNPINNED
description: Filter by event types
offset:
type: integer
format: int64
description: Offset for pagination
limit:
type: integer
format: int64
description: >-
Maximum number of events to return. Default and maximum
value is 50.
examples:
ListclustereventsRequestExample:
summary: Default listClusterEvents request
x-microcks-default: true
value:
cluster_id: '500123'
start_time: 10
end_time: 10
order: DESC
event_types:
- CREATING
offset: 10
limit: 10
responses:
'200':
description: Cluster events retrieved successfully
content:
application/json:
schema:
type: object
properties:
events:
type: array
items:
$ref: '#/components/schemas/ClusterEvent'
next_page:
type: object
description: Parameters for fetching the next page of events
total_count:
type: integer
format: int64
description: Total number of events matching the filter
examples:
Listclusterevents200Example:
summary: Default listClusterEvents 200 response
x-microcks-default: true
value:
events:
- cluster_id: '500123'
timestamp: 10
type: example_value
details: example_value
next_page: example_value
total_count: 10
'400':
$ref: '#/components/responses/BadRequest'
'401':
$ref: '#/components/responses/Unauthorized'
'403':
$ref: '#/components/responses/Forbidden'
'429':
$ref: '#/components/responses/TooManyRequests'
'500':
$ref: '#/components/responses/InternalError'
x-microcks-operation:
delay: 0
dispatcher: FALLBACK
/2.0/clusters/spark-versions:
get:
operationId: listSparkVersions
summary: Azure Databricks List Available Spark Versions
description: >-
Returns the list of available Databricks Runtime versions. These
versions can be used to launch clusters.
tags:
- Clusters
responses:
'200':
description: Spark versions retrieved successfully
content:
application/json:
schema:
type: object
properties:
versions:
type: array
items:
type: object
properties:
key:
type: string
description: Databricks Runtime version key
name:
type: string
description: Human-readable name for the version
examples:
Listsparkversions200Example:
summary: Default listSparkVersions 200 response
x-microcks-default: true
value:
versions:
- key: example_value
name: Example Title
'401':
$ref: '#/components/responses/Unauthorized'
'429':
$ref: '#/components/responses/TooManyRequests'
'500':
$ref: '#/components/responses/InternalError'
x-microcks-operation:
delay: 0
dispatcher: FALLBACK
/2.0/clusters/list-node-types:
get:
operationId: listNodeTypes
summary: Azure Databricks List Available Node Types
description: >-
Returns a list of supported Azure VM node types. These node types can
be used to launch clusters.
tags:
- Clusters
responses:
'200':
description: Node types retrieved successfully
content:
application/json:
schema:
type: object
properties:
node_types:
type: array
items:
$ref: '#/components/schemas/NodeType'
examples:
Listnodetypes200Example:
summary: Default listNodeTypes 200 response
x-microcks-default: true
value:
node_types:
- node_type_id: '500123'
memory_mb: 10
num_cores: 42.5
description: A sample description.
# --- truncated at 32 KB (129 KB total) ---
# Full source: https://raw.githubusercontent.com/api-evangelist/azure-databricks/refs/heads/main/openapi/azure-databricks-openapi.yml