Skip to content

lely

One plan for your whole Databricks deploy. The bundle and everything around it — the steps before, the steps after — reviewed before anything runs, and taken down again when you say so.

Alpha

Everything described here is built, and has run for real a few times: on a workspace, and through GitHub's workflows. That is a first proof, not a track record — what has been tried says exactly what was and wasn't. Try it on a development target first.

databricks bundle deploy deploys the bundle. A real deploy is more than that: the schema that has to exist first, a model version looked up and handed to the bundle, the job that has to run once after, the table that has to change. The bundle has a plan and a deploy. The things around it have a shell script, and a script has no dry run. lely replaces that script.

lely plan · target dev · https://dbc-example.cloud.databricks.com as jane@example.com

  model  ./ops/steps.py:LatestModel
    → version = 14
  app  bundle
    model_version = 14  ← model.version
    + jobs.bar
    ± pipelines.foo  destructive
        replaced: storage (immutable)
    ▶ uploads the bundle's files
  notify  ./ops/steps.py:Notify
    ⏸ waiting for app.resources.jobs.bar.id
  backfill  bundle.run
    bundle  ← app
    ▶ runs jobs.backfill

Plan: 2 changes · 2 runs · 1 destructive · 1 waiting

Your first plan What may run On GitHub

What it gives you

  • The whole deploy in one plan — every step, in the order it runs, with what it takes from the steps above it and what it would change.
  • What one step hands the next is written down — ${steps.<name>.<output>}, in the step's own options, and checked before anything runs.
  • One rule for consent — nothing that changes a workspace runs unasked, and nothing destructive without --allow-destructive. A reviewed plan file runs exactly what was reviewed, or refuses.
  • It goes both ways — lely destroy plans the teardown, from the bottom up, and asks.
  • No state — whatever lely needs to know, it asks the system it manages: for a bundle, the Databricks CLI.
  • Where plans get read — as one comment on the pull request, kept current; on the run's page; and as a page of its own you can open or pass on.

Where it fits

lely is one of the Kostavo tools: small tools for the ugly gaps on Databricks, one gap each. Its gap is the deploy that is more than a deploy: the steps before and after databricks bundle deploy, kept in a script with no dry run. Each works alone. Why the tools exist has the rules they keep, and the README says when not to use it and when you no longer need it.

lely borrows Terraform's words — plan, apply, destroy — because everyone knows what they promise. The job is not the same: lely manages no resource itself and remembers nothing. It owns the order, the review and the consent.

Named after Cornelis Lely (1854–1929), the engineer who designed the Zuiderzee Works. Community project, not affiliated with or endorsed by Databricks.