What is the Use of Azure Service Bus in Databricks?

The Databricks control plane automatically creates a service bus to collect metrics and other data about your Databricks cluster's activity. This data is used to generate insights into the performance and health of your Databricks cluster. The Databricks cluster in the data plane does not need to create its own service bus because it can use the service bus that is created in the control plane. This allows the Databricks cluster to focus on its primary task of processing data. The service bus in the control plane is also used to collect data for Databricks Autoscaling. Databricks Autoscaling uses this data to automatically scale your Databricks cluster up or down based on demand.

We can view the data collected by the service bus in the Databricks UI. To do this, go to the Clusters page and click on the cluster that you want to view. Then, click on the Metrics tab.