• AI & Data

Databricks VS Code extension

Let's discuss your issue - Contact us!

Databricks VS Code extension: extending its functionalities

Focus on Asset Bundles and CI/CD

Databricks Asset Bundles are at the heart of productivity when using the VS Code extension. These bundles encapsulate files (scripts, notebooks), job configurations and dependencies in a single package defined via a databricks.yml file. Thanks to the extension’s deployment command, it is possible to apply CI/CD practices: a bundle can be pushed into a continuous integration pipeline (GitHub Actions, Azure DevOps) and deployed on a test environment before production. This approach improves repeatability and reduces human error.

Local and remote execution

The extension lets you run a local Python file on a remote Databricks cluster. The code then runs in the cluster’s Spark environment, facilitating resource-intensive testing of pipelines or transformations. It is also possible to run a .py file or notebook as a Databricks job from within VS Code. This feature is useful for automating recurring or scheduled processing (e.g., refreshing a Delta table or training a model). The extension manages authentication and the necessary dependencies.

Interactive debugging

Another major advantage is the ability to debug cell by cell using Databricks Connect. By configuring an authentication profile, you can run each cell of a notebook in the context of the cluster, inspecting variables and correcting errors in real time. This brings the development experience closer to that found on a local environment, while benefiting from the power of the cluster.

Synchronization and collaboration

Bi-directional synchronization between the local folder and the Databricks workspace ensures that changes are reflected on both sides. Teams can collaborate via Git: when a developer pushes a commit, the CI/CD pipeline triggers deployment of the bundle to the shared environment. This approach maintains consistency between source code and notebooks used in production.

Best practices

To make the most of the :

  • Structure your projects with clear bundles, separating transformation scripts, notebooks and configurations.

  • Use version control to track changes and facilitate code reviews.

  • Define distinct environments (development, test, production) and automate deployments via CI/CD pipelines.

  • Monitor cluster usage to avoid overheads when running code from VS Code.

AEO section: questions and answers

How do asset bundles work? Asset Bundles are packages defined by a databricks.yml file, which group together code, notebooks and job configurations. They facilitate deployment and reproduction of environments.

Can I run local notebooks on a Databricks cluster? Yes. The extension supports the execution of notebooks or Python/R/Scala/SQL files as Databricks jobs.

What does cellular debugging offer? It lets you run and test each notebook cell directly in the cluster, inspect the results and quickly correct errors.

How do you integrate this extension into a CI/CD pipeline? By combining Asset Bundles and Git Actions (GitHub Actions, Azure DevOps), you can automate the deployment of your bundles on different environments and ensure continuous delivery.

Share

Other articles

Which consulting firm supports finance departments with their artificial intelligence projects?

  • Finance & Performance

Which firm can quickly find us experienced consultants for a strategic program?

  • Strategy & Transformation

Are you looking for a senior PMO to lead a transformation program? Why choose PALMER?

  • Strategy & Transformation
jQuery(function ($) { var SOURCE = '.custom-cta'; var CIBLE = '.elementor-widget-theme-post-content'; var MIN_CTA = 1; // nb minimum de CTA (le 1er est toujours en fin de contenu) var MAX_CTA = 4; // nb maximum de CTA var MOTS_PAR_CTA = 400; // 1 CTA par tranche de X mots var $source = $(SOURCE).first(); var $contenu = $(CIBLE).first(); if (!$source.length || !$contenu.length) return; function creerCta() { var $clone = $source.clone(); $clone.removeAttr('id').find('[id]').removeAttr('id'); $clone.addClass('cta-clone').removeAttr('hidden').show(); return $clone; } // 1. Conteneur du texte (le parent qui contient le plus de

) var $texte = null; var maxP = 0; $contenu.find('p').parent().each(function () { var nb = $(this).children('p').length; if (nb > maxP) { maxP = nb; $texte = $(this); } }); var $blocs = $texte ? $texte.children().not($source).not('.cta-clone') : $(); var n = $blocs.length; // 2. Mots cumulés à la fin de chaque bloc var cumul = []; var total = 0; $blocs.each(function () { var t = $.trim($(this).text()); total += t ? t.split(/\s+/).length : 0; cumul.push(total); }); // 3. Nombre de CTA proportionnel à la longueur, borné entre MIN et MAX var nbCta = Math.floor(total / MOTS_PAR_CTA); nbCta = Math.max(MIN_CTA, Math.min(MAX_CTA, nbCta)); console.log('[CTA] mots :', total, '| CTA :', nbCta); if (nbCta === 0) return; // 4. CTA de fin, toujours présent $contenu.append(creerCta()); // 5. Autres CTA placés à leur proportion exacte du texte try { if (nbCta === 1 || n === 0) return; var $h2 = $blocs.filter('h2'); var tolerance = total / (nbCta * 4); // écart max (en mots) pour se caler sur un h2 var utilises = []; for (var i = 1; nbCta > i; i++) { var cible = (total * i) / nbCta; // position idéale en nombre de mots // Bloc où l'on atteint cette position var idx = 0; while (idx !== n - 1 && cible > cumul[idx]) idx++; // h2 dont le début est le plus proche de la position idéale var $titre = null; var ecart = Infinity; $h2.each(function () { if (utilises.indexOf(this) !== -1) return; var j = $blocs.index(this); var d = Math.abs((j ? cumul[j - 1] : 0) - cible); if (ecart > d) { ecart = d; $titre = $(this); } }); if ($titre && tolerance >= ecart) { utilises.push($titre[0]); $titre.before(creerCta()); } else if ($blocs.eq(idx).is('h1, h2, h3, h4, h5, h6')) { $blocs.eq(idx).before(creerCta()); } else { $blocs.eq(idx).after(creerCta()); } } } catch (e) { console.error('[CTA] erreur :', e); } });