Skip to main content

Upgrade a Databricks workspaces to Unity Catalog

This page gives an overview of how to upgrade a non-Unity Catalog workspace to Unity Catalog. It also gives instructions for migrating off of the legacy workspace-local Hive metastore.

Overview of upgrade steps​

To upgrade to Unity Catalog, you must:

  1. Provision identities (users, groups, and service principals) directly to your Databricks account, if you aren't doing so already. Turn off any workspace-level identity provisioning.
  2. Convert any workspace-local groups to account-level groups. Unity Catalog centralizes identity management at the account level.
  3. Attach the workspace to a Unity Catalog metastore. If no metastore exists for your workspace region, an account admin must create one.
  4. Upgrade tables and views managed in Hive metastore to Unity Catalog.
  5. Grant account-level users, groups, or service principals access to the upgraded tables.
  6. Update queries and jobs to reference the new Unity Catalog tables instead of the old Hive metastore tables.
  7. Disable the Hive metastore. See Disable access to the Hive metastore used by your Databricks workspace.

UCX, a Databricks Labs project, provides tools that help you upgrade your non-Unity-Catalog workspace to Unity Catalog. UCX is a good choice for larger-scale migrations. See Use the UCX utilities to upgrade your workspace to Unity Catalog.

Before you begin​

Before you begin, you should familiarize yourself with the basic Unity Catalog concepts, including metastores and managed storage. See What is Unity Catalog?.

You should also confirm that you meet the following requirements:

  • For most setup steps, you must be a Databricks account admin. For any task that follows for which there are other permission requirements, they are listed in the task-specific documentation.
  • Your Databricks account must be on the Premium plan or above.

Provision users, groups, and service principals to your account​

Unity Catalog references account-level identities. Before you attach a metastore to your workspace, you should do the following:

  • If you are using SCIM to provision users, groups, and service principals from your IdP to your workspace, turn it off and set up provisioning to your Databricks account instead. See Assign users to Databricks and Identities.

  • Update any automation that has been configured to manage users, groups, and service principals, such as SCIM provisioning connectors and Terraform automation, so that they refer to account endpoints instead of workspace endpoints. See Account-level and workspace-level SCIM provisioning.

Convert workspace-local groups to account-level groups​

See Migrate workspace-local groups to account groups.

Attach your workspace to a metastore​

If your workspace is not enabled for Unity Catalog (attached to a metastore), the next step depends on whether or not you already have a Unity Catalog metastore defined for your workspace region:

  • If your account already has a Unity Catalog metastore defined for your workspace region, you can simply attach your workspace to the existing metastore. Go to Enable a workspace for Unity Catalog.
  • If there is no Unity Catalog metastore defined for your workspace's region, you must create a metastore and then attach the workspace. Go to Go to Create a Unity Catalog metastore.

Upgrade tables in your Hive metastore to Unity Catalog tables​

If your workspace was in service before it was enabled for Unity Catalog, it has a Hive metastore that likely contains data that you want to continue to use. Databricks recommends that you upgrade the tables managed by the Hive metastore to the Unity Catalog metastore.

You can upgrade gradually by federating your Hive metastore. See the next section.

You can upgrade tables directly using the instructions in Upgrade a Databricks workspaces to Unity Catalog.

(Optional) Federate your Hive metastore to keep working with it​

If your workspace has a Hive metastore that contains data that you want to continue to use, and you choose not to follow the recommendation to upgrade all the tables managed by the Hive metastore to the Unity Catalog metastore, you can continue to work with data in the Hive metastore by federating it as a foreign catalog in Unity Catalog. See Hive metastore federation: enable Unity Catalog to govern tables registered in a Hive metastore.

Grant access to upgraded or federated tables​

Grant account-level users, groups, or service principals access to the new tables. See Manage privileges in Unity Catalog.

Update queries and jobs to work with your upgraded tables and paths to data​

While you are transitioning from the workspace-local Hive metastore to Unity Catalog, you can continue to use queries and jobs that reference the data registered in the Hive metastore, using Hive metastore federation (recommended) or the syntax described in Work with the legacy Hive metastore alongside Unity Catalog. However, eventually you should update all queries and jobs to use Unity Catalog tables and syntax.

Likewise, update queries and jobs that use path-based access to files to use Unity Catalog volumes instead.

For detailed recommendations, see Update jobs when you upgrade legacy workspaces to Unity Catalog.

Last updated on