skills(pipelines): port DLT migration guide from a-d-k (phase 1)#85
Draft
jamesbroadhead wants to merge 1 commit into
Draft
skills(pipelines): port DLT migration guide from a-d-k (phase 1)#85jamesbroadhead wants to merge 1 commit into
jamesbroadhead wants to merge 1 commit into
Conversation
Phase 1 of d-a-s #73's TODO #5 — port a-d-k's databricks-spark-declarative-pipelines content into stable skills/databricks-pipelines/. Adds references/dlt-migration.md covering both migration paths (DLT Python → SDP Python via the modern pyspark.pipelines API, and DLT Python → SDP SQL) with side-by-side conversions for decorators, reads, expectations, CDC/SCD, and partitioning → liquid clustering. Source clean — no MCP-tool refs to strip, no docs.databricks.com URLs to rewrite. SKILL.md updates: - bump version to 0.2.0 - new "Migrating from DLT" section pointing at the reference Subsequent phases (separate commits) port the remaining a-d-k content: workflow A/B/C decision matrix (project initialization), per-language performance reference, language-selection rules. Co-authored-by: Isaac
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Phase 1 of d-a-s #73's TODO #5 — port
databricks-spark-declarative-pipelinescontent fromdatabricks-solutions/ai-dev-kitinto stableskills/databricks-pipelines/.Stable's
databricks-pipelinescovers feature reference (decision tree, common traps, format options, per-feature × per-language refs). a-d-k's version covers scaffolding, migration, and language-selection — different surface; non-trivial overlap. Plan is to add high-value a-d-k content asreferences/files, not rewrite SKILL.md.This first commit ports the DLT → SDP migration guide, the cleanest piece: no MCP-tool refs to strip, no doc-URL prefixes to rewrite, no API/SDK divergence vs current main.
Changes
skills/databricks-pipelines/references/dlt-migration.md— both migration paths (DLT Python → SDP Python viapyspark.pipelines, DLT Python → SDP SQL) with side-by-side conversions for decorators, reads, expectations, CDC/SCD, and partitioning → liquid clustering.versionto0.2.0; new "Migrating from DLT" section pointing at the new reference.Follow-up phases (this branch, separate commits)
databricks pipelines initfor standalone projects vs DAB integration vs MCP workflows (from a-d-k's1-project-initialization.md).python/andsql/subdirectories not already covered by stable's existing per-feature references.All follow-up content will land here as commits, then this PR ships once the dev-side surface is complete.
Test plan
python3 scripts/skills.py generateclean.python3 scripts/skills.py validatepasses.@lennartkats-db/@camielstee-dbper CODEOWNERS).This pull request and its description were written by Claude.