terraform-provider-databricks

Auto-generated from databricks/terraform-provider-databricks by Mutable.ai Auto WikiRevise

The Databricks Terraform Provider is a powerful tool that allows engineers to manage Databricks resources, such as clusters, notebooks, jobs, SQL objects, Unity Catalog, and more, using Terraform. This provider simplifies the deployment and management of Databricks infrastructure by enabling infrastructure as code (IaC) workflows.

The most important functionality of this provider is centered around the management of Databricks Catalog, Clusters, Jobs, MLflow, Managed Workspace Service (MWS), Cluster Policies, Repositories, SCIM, Secrets, SQL, Storage, and Workspace. Each of these areas is implemented in a separate directory within the terraform-provider-databricks repository.

The catalog directory provides comprehensive functionality for managing various Databricks Catalog entities, including the ability to create, read, update, and delete catalogs, schemas, tables, views, shares, connections, and permissions. The UnityCatalogPermissionsAPI struct and the corresponding data sources and resources handle the complex task of managing Databricks Catalog permissions.

The clusters directory focuses on the management of Databricks clusters, including creating, resizing, editing, starting, and terminating clusters. It also provides data sources for retrieving information about node types, Spark versions, and available cluster zones, as well as a resource for managing the installation and uninstallation of libraries on clusters.

The jobs directory handles the management of Databricks jobs, including creating, reading, updating, and deleting jobs, as well as managing the lifecycle of job runs. The JobsAPI struct and the JobSettings struct encapsulate the necessary functionality for interacting with the Databricks Jobs API.

The mlflow directory provides resources and data sources for managing MLflow experiments, models, and webhooks within a Databricks workspace. This allows users to integrate their machine learning workflows with the Databricks platform.

The mws directory focuses on the management of the Databricks Managed Workspace Service (MWS), including the creation and management of credentials, storage configurations, networks, permission assignments, private access settings, VPC endpoints, and workspaces. The WorkspacesAPI struct is the main entry point for interacting with the MWS API.

The policies directory handles the management of Databricks cluster policies, including the ability to create, read, update, and delete policies, as well as the differentiation between built-in and custom policy families.

The repos directory provides resources for managing Databricks Repositories, including Git credentials and the creation, reading, updating, and deletion of repositories.

The scim directory implements the management of Databricks SCIM (System for Cross-domain Identity Management) entities, such as users, groups, and service principals, including the assignment of entitlements.

The secrets directory handles the management of Databricks secrets and secret scopes, allowing users to create, read, update, and delete secrets, as well as manage access control lists (ACLs) for secret scopes.

The sql directory focuses on the management of Databricks SQL-related resources, including warehouses, alerts, dashboards, endpoints, global configurations, queries, visualizations, and widgets.

The storage directory provides functionality for managing storage mounts for various cloud storage providers, as well as the ability to create, read, and delete files within the Databricks Distributed File System (DBFS).

Finally, the workspace directory handles the management of Databricks workspace resources, such as directories, notebooks, and global init scripts.

The Databricks Terraform Provider relies on the Databricks SDK for Go to interact with the Databricks REST API, and it utilizes the common.Resource struct and related utility functions to provide a consistent interface for managing the various Databricks resources. The code is well-structured, with clear separation of concerns and comprehensive test suites to ensure the reliability and correctness of the provider's functionality.

References: catalog

The catalog directory contains the implementation of the Databricks Catalog functionality for the Terraform provider. This includes the ability to manage various Databricks Catalog entities, such as catalogs, schemas, tables, views, shares, connections, and permissions.

References: catalog/permissions

The …/permissions directory contains the implementation of the permissions management functionality for the Databricks Catalog in the Terraform provider. This includes the ability to retrieve, update, and wait for updates to permissions for various Databricks Catalog entities.

References: catalog

The catalog directory contains the implementation of the data sources for retrieving information about various Databricks Catalog entities, such as catalogs, schemas, tables, views, shares, connections, and storage credentials.

References: catalog

The catalog directory contains the implementation of the resources for managing various Databricks Catalog entities, such as catalogs, connections, external locations, grants, and Lakehouse Monitors.

References: clusters

The clusters directory contains the implementation of the Databricks cluster management functionality in the Terraform provider. This includes the ability to create, resize, edit, start, terminate, and list Databricks clusters, as well as manage cluster policies and instance pools.

References: clusters/clusters_api.go, clusters/resource_cluster.go

The clusters_api.go file in the clusters directory provides the core functionality for managing Databricks clusters. This includes the ability to create, resize, edit, start, terminate, and list clusters.

References: clusters/data_cluster.go, clusters/data_node_type.go, clusters/data_spark_version.go, clusters/data_zones.go

The terraform-provider-databricks/clusters package provides several data sources for retrieving information about Databricks clusters, node types, Spark versions, and available cluster zones.

References: clusters/resource_library.go

The resource_library.go file in the clusters package defines a Terraform resource for managing Databricks libraries. The main functionality of this file includes:

References: jobs

The jobs directory contains the implementation of the Databricks jobs-related functionality in the Terraform provider. This includes the following key components:

References: jobs/data_job.go, jobs/data_jobs.go

The data_job.go file in the jobs directory defines a Terraform data source for retrieving information about a specific Databricks job. The DataSourceJob() function is responsible for creating this data source.

References: jobs/resource_job.go, jobs/resource_job_test.go

The terraform-provider-databricks/jobs/resource_job.go file defines the Terraform resource for managing Databricks jobs. It provides functionality for creating, reading, updating, and deleting jobs, as well as managing the lifecycle of job runs.

References: terraform-provider-databricks

The JobSettings struct and the JobsAPI interface provide the underlying functionality for managing Databricks jobs, including settings for tasks, clusters, parameters, triggers, and notifications.

References: mlflow

The mlflow directory contains the implementation of various Terraform resources and data sources related to MLflow in the Databricks platform. The key components in this directory are:

References: mlflow/resource_mlflow_experiment.go

The ResourceMlflowExperiment() function in the …/resource_mlflow_experiment.go file defines the Terraform resource for managing MLflow experiments in a Databricks workspace. This resource allows you to create, read, update, and delete MLflow experiments using Terraform.

References: mlflow/data_mlflow_model.go, mlflow/resource_mlflow_model.go

The resource_mlflow_model.go file in the mlflow package defines a Terraform resource for managing MLflow models in a Databricks workspace. This resource supports the following key operations:

References: mlflow/resource_mlflow_webhook.go

The file …/resource_mlflow_webhook.go defines a Terraform resource for managing MLflow webhooks in a Databricks workspace. This resource allows creating, reading, updating, and deleting webhooks, which are used to trigger actions based on events in the MLflow Model Registry.

References: mws

The mws directory contains the implementation of the Databricks Managed Workspace Service (MWS) resources for the Terraform provider. It provides functionality to manage various MWS entities, such as credentials, storage configurations, networks, permission assignments, private access settings, VPC endpoints, and workspaces.

References: mws/data_mws_credentials.go, mws/resource_mws_credentials.go

The data_mws_credentials.go file in the mws package defines a Terraform data source for retrieving a list of Databricks Managed Workspace Service (MWS) credentials associated with a specific Databricks account.

References: mws/resource_mws_storage_configurations.go

The ResourceMwsStorageConfigurations function in the …/resource_mws_storage_configurations.go file is responsible for managing the lifecycle of Databricks Managed Workspace Service (MWS) storage configurations. This includes creating, reading, and deleting storage configurations.

References: mws/resource_mws_networks.go

The terraform-provider-databricks/mws/resource_mws_networks.go file defines the functionality for managing Databricks Managed Workspace (MWS) networks. It provides an API for creating, reading, deleting, and listing MWS networks.

References: mws/resource_mws_permission_assignment.go

The key functionality for managing permission assignments to principals (users or groups) for Databricks workspaces is implemented in the …/resource_mws_permission_assignment.go file.

References: mws/resource_mws_private_access_settings.go

The ResourceMwsPrivateAccessSettings() function in …/resource_mws_private_access_settings.go defines the Terraform resource for managing the private access settings for a Databricks account.

References: mws/resource_mws_vpc_endpoint.go

The VPCEndpointAPI struct in the …/resource_mws_vpc_endpoint.go file provides the functionality for managing Databricks Managed Workspace Service (MWS) VPC endpoints. This includes creating, reading, deleting, and listing VPC endpoints associated with a Databricks MWS account.

References: mws/data_mws_workspaces.go, mws/resource_mws_workspaces.go

The terraform-provider-databricks/mws/resource_mws_workspaces.go file provides a Terraform resource for managing Databricks Managed Workspace Service (MWS) workspaces. This resource allows users to create, read, update, and delete MWS workspaces, as well as manage workspace-specific Personal Access Tokens (PATs).

References: policies

The policies directory contains the implementation of the Databricks cluster policy management functionality in the Terraform provider for Databricks. This includes the following key components:

References: policies/data_cluster_policy.go, policies/resource_cluster_policy.go

The resource_cluster_policy.go file in the policies package provides the implementation for the databricks_cluster_policy Terraform resource, which allows users to manage Databricks cluster policies. The key functionality includes:

References: policies/data_cluster_policy.go

The …/data_cluster_policy.go file provides a Terraform data source for retrieving information about Databricks cluster policies. This data source allows users to fetch details of a specific cluster policy, as well as list all available cluster policies.

References: policies/resource_cluster_policy.go

The resource_cluster_policy.go file in the policies package defines the Terraform resource for managing Databricks cluster policies. This resource allows users to create, read, update, and delete cluster policies in a Databricks workspace, while handling the differences between built-in and custom policy families.

References: repos

The repos directory contains the implementation of the Databricks Repositories functionality in the Terraform provider for Databricks. This includes the management of Git credentials and repositories within the Databricks workspace.

References: repos/resource_git_credential.go, repos/resource_git_credential_test.go

The main functionality of the file …/resource_git_credential.go is defined in the ResourceGitCredential() function. This function returns a common.Resource object that defines the schema and CRUD operations for a Terraform resource that manages Git credentials in the Databricks workspace.

References: repos/resource_repo.go, repos/resource_repo_test.go

The terraform-provider-databricks/repos/resource_repo.go file defines the functionality for managing Databricks repositories using Terraform. The main components in this file are:

References: scim

The scim directory contains the implementation of the SCIM (System for Cross-domain Identity Management) functionality for the Databricks Terraform provider. This includes the management of users, groups, and service principals within a Databricks workspace.

References: scim/users.go

The users.go file in the scim package provides an API for managing users in the Databricks platform. The main functionality includes:

References: scim/groups.go

The groups.go file in the scim package provides an API for managing SCIM (System for Cross-domain Identity Management) groups in a Databricks workspace. The GroupsAPI struct encapsulates the functionality for interacting with the Databricks SCIM API to create, read, update, and delete SCIM groups.

References: scim/resource_service_principal.go

The terraform-provider-databricks/scim/resource_service_principal.go file provides the implementation for managing service principals within a Databricks workspace using the Terraform provider.

References: terraform-provider-databricks

The scim directory provides the implementation of the SCIM (System for Cross-domain Identity Management) functionality in the Databricks Terraform provider. This includes the management of entitlements for Databricks groups, users, and service principals.

References: terraform-provider-databricks

The scim directory contains the implementation of the Databricks SCIM (System for Cross-domain Identity Management) data sources in the Terraform provider. These data sources allow retrieving information about various SCIM entities, such as users and groups.

References: secrets

The secrets directory contains the implementation of Terraform resources for managing secrets and secret scopes in the Databricks platform.

References: secrets/resource_secret.go

The resource_secret.go file in the secrets directory contains the implementation of the Terraform resource for managing secrets in the Databricks workspace.

References: secrets/resource_secret_scope.go

The terraform-provider-databricks/secrets/resource_secret_scope.go file defines the functionality for managing secret scopes in the Databricks platform using Terraform. The key components are:

References: secrets/resource_secret_acl.go

The resource_secret_acl.go file in the secrets package defines a Terraform resource for managing access control lists (ACLs) for Databricks secret scopes. This resource allows users to create, read, and delete ACL entries that grant permissions to principals (users or groups) for specific secret scopes.

References: sql

The sql directory provides a comprehensive set of functionality for managing SQL-related resources in the Databricks platform using Terraform. This includes resources for managing SQL queries, dashboards, visualizations, and widgets.

References: sql/data_sql_warehouse.go, sql/data_sql_warehouses.go

The data_sql_warehouse.go and data_sql_warehouses.go files in the sql directory provide data sources for retrieving information about Databricks SQL warehouses.

References: sql/resource_sql_alerts.go

The resource_sql_alerts.go file in the sql package defines the functionality for managing SQL alerts in the Databricks platform using Terraform. The key components are:

References: sql/resource_sql_dashboard.go

The …/resource_sql_dashboard.go file provides the implementation for managing Databricks SQL dashboards using Terraform resources. The key components are:

References: sql/resource_sql_endpoint.go

The terraform-provider-databricks/sql/resource_sql_endpoint.go file provides the implementation for managing SQL endpoints (warehouses) in the Databricks platform using Terraform. It allows users to create, read, update, and delete SQL endpoints, as well as configure various settings such as cluster size, auto-stop minutes, and custom tags.

References: sql/resource_sql_global_config.go

The terraform-provider-databricks/sql/resource_sql_global_config.go file defines the functionality for managing the global SQL configuration in the Databricks Terraform provider. The key components are:

References: sql/resource_sql_query.go

The resource_sql_query.go file in the sql package provides the implementation for managing SQL queries in the Databricks platform using Terraform. The key functionality includes:

References: sql/resource_sql_visualization.go

The resource_sql_visualization.go file in the sql package defines the functionality for managing SQL visualizations in the Databricks platform using Terraform.

References: sql/resource_sql_widget.go

The terraform-provider-databricks/sql/resource_sql_widget.go file defines the functionality for managing SQL widgets in the Databricks platform using Terraform. The key components are:

References: storage

The storage directory provides functionality for managing storage-related resources in the Databricks platform using Terraform. The key components are:

References: storage/mounts.go, storage/s3.go, storage/adls_gen2_mount.go, storage/adls_gen1_mount.go, storage/azure_blob_mount.go, storage/gs.go

The terraform-provider-databricks/storage package provides functionality for managing storage mounts in Databricks, supporting various cloud storage providers such as Azure Data Lake Storage (ADLS) Gen1 and Gen2, Amazon S3, Google Cloud Storage (GCS), and Azure Blob Storage.

References: storage/data_dbfs_file.go, storage/data_dbfs_file_paths.go, storage/resource_dbfs_file.go, storage/resource_file.go

The terraform-provider-databricks/storage package provides functionality for managing files within the Databricks Distributed File System (DBFS). This includes creating, reading, and deleting DBFS files.

References: storage/dbfs.go, storage/generic_mounts.go, storage/mounts.go

The dbfs.go file in the storage package provides a comprehensive API for interacting with the Databricks File System (DBFS). The main functionality includes:

References: workspace

The workspace directory contains the implementation of the Databricks provider for Terraform, focusing on the management of various workspace entities. The key functionality includes:

References: workspace/resource_directory.go

The terraform-provider-databricks/workspace/resource_directory.go file defines a Terraform resource for managing directories in the Databricks workspace. This resource allows you to create, read, update, and delete directories, with various attributes and options to control the behavior.

References: workspace/resource_notebook.go, workspace/data_notebook.go, workspace/data_notebook_paths.go

The terraform-provider-databricks/workspace/resource_notebook.go file defines the functionality for managing Databricks notebooks using the Terraform provider. The core of this implementation is the NotebooksAPI struct, which provides methods for interacting with the Databricks workspace API to create, read, export, list, and delete notebooks.

References: workspace/resource_global_init_script.go

The resource_global_init_script.go file in the workspace directory defines a Terraform resource for managing global init scripts in the Databricks workspace. This resource allows users to create, read, update, and delete global init scripts, which are scripts that run during the initialization of Databricks clusters.

References: workspace/resource_workspace_conf.go

The …/resource_workspace_conf.go file provides the implementation for managing the overall configuration of the Databricks workspace, including settings and preferences.

References: workspace/resource_workspace_file.go, workspace/file_resource.go

The main functionality for managing files in the Databricks workspace is implemented in the …/resource_workspace_file.go file. The ResourceWorkspaceFile() function defines the Terraform resource for managing files, including the schema and CRUD operations.

References: workspace/data_directory.go, workspace/data_notebook.go, workspace/data_notebook_paths.go

The Databricks Terraform provider includes several data sources for retrieving information about various workspace entities, such as directories, notebooks, and notebook paths.