Data Engineering · 3 min read

Databricks Lifts Cluster Rate Limits 7.5× by Switching to Adaptive Concurrency

Databricks raised default cluster-creation rate limits 7.5× by replacing fixed request caps with adaptive concurrency limits, cutting rate-limit errors over 99.99% and deleting 100+ manual overrides.

By Grace Kim636 words

Key takeaways

  1. Databricks raised default cluster-creation request-rate limits 7.5× using adaptive concurrency limits.
  2. Cluster-creation rate-limit errors fell more than 99.99% in the reported rollout scope.
  3. Over a hundred customer-specific rate-limit overrides were removed; no new manual overrides added since.
  4. The system spans provisioning workflows across AWS, Azure and Google Cloud for Databricks classic compute.
How we raised Databricks cluster rate limits by 7.5×
How we raised Databricks cluster rate limits by 7.5× — geralt / pixabay

Databricks raised default cluster-creation request-rate limits by 7.5×, and the trick wasn't buying more capacity — it was swapping fixed request-rate limits for adaptive concurrency control on the provisioning workflows behind classic compute clusters.

The engineering team — Heqing Ya, Tan Zhang, Ankit Batra and Abhishek Rai of Databricks' Compute Platform — reported the results from the initial rollout: a 7.5× increase in default cluster-creation rate limits, more than 99.99% fewer rate-limit errors in the reported scope, and removal of over a hundred customer-specific rate-limit overrides. Since the raise, few customers have hit the new ceilings, and engineers have added no new manual overrides.

Why did fixed rate limits fail here?

The core problem is duration. A CreateCluster API call takes milliseconds; the workflow behind it — coordinating services and provisioning VMs across AWS, Azure and Google Cloud — takes minutes. Capping requests per second at the front door says nothing about work already in flight.

Worse, one request might provision a few VMs while the next provisions many. During a shared cloud-provider slowdown, incoming request rates stay flat while in-flight work balloons and capacity to finish it shrinks. Then the retry storms start: long-running workflows retry failing dependency calls, and since those retries come from work already accepted, the API rate limiter never sees them. Overload persists even after traffic normalizes.

The old regime also demanded manual babysitting. Engineers lowered limits by hand to shed load during incidents, then raised them again after recovery. Conservative caps rejected customers despite spare capacity; generous ones kept accepting work while dependencies gasped.

What does adaptive concurrency actually do?

The new system gives each active provisioning workflow a slot, held from start to finish — including time spent waiting on cloud providers. The controller adjusts the slot count using congestion signals such as provisioning latency and critical thread-pool saturation. Congested means fewer new starts; recovered means cautiously more. Slots don't hold threads, and one slot doesn't imply a fixed resource cost, since a single workflow can provision many VMs.

Requests that find all slots occupied wait in a bounded queue rather than getting an instant error. Queues come with their own rules, the authors note: bound size and wait time, make progress visible, and expire work that can no longer finish usefully. "A queue cannot turn sustained overload into capacity; an unbounded one can turn a brief disruption into hours of delay."

Fair scheduling matters too. Delaying a hundred requests across a hundred customers looks identical, in aggregate, to delaying one customer's hundred requests. Only one of those keeps customers. The system defines tenant shares under contention while letting tenants use otherwise idle capacity where policy allows.

The fixed API request-rate limit survives as a final guardrail against exceptional incoming bursts — and as a boundary Databricks can actually communicate to customers. Adaptive limits stay internal. A successful API response, the team is careful to say, does not guarantee an immediate cluster launch.

What should data teams take from this?

If you operate long-running workflows behind thin APIs, request-rate limiting is the wrong model of capacity. Databricks' checklist is transferable:

  • Cap concurrent operations, not just arrivals; each holds a slot until done.
  • Pair concurrency limits with retry backoff and attempt caps, or recovering dependencies will keep getting hammered.
  • Track system health, per-tenant demand and scheduling decisions separately — high in-flight counts alone don't prove a customer misbehaved.
  • Count degradation-driven throttling in reliability metrics, and queue-wait time in launch latency.

Open questions remain: weighted slots for operations with very different resource costs, and where to place control in multi-bottleneck workflows so one slow dependency doesn't stall unrelated work. The 7.5× figure suggests the first version was worth shipping anyway.

Databricks Blog (Source)

Price tags

Share this article:

More from Grace Kim

Grace Kim

Show full bio

Market editor covering companies, deals and strategy in Analytics Digest is a professional daily at Analytics Digest.

18 articles

Read also

« Previous articleNext article »