feat: apps init speed improvements#4678
Merged
MarioCadenas merged 8 commits intomainfrom Mar 16, 2026
Merged
Conversation
MarioCadenas
temporarily deployed
to
test-trigger-is
GitHub Actions
Inactive
Collaborator
|
Commit: d5c0893
18 interesting tests: 9 SKIP, 6 RECOVERED, 3 flaky
Top 22 slowest tests (at least 2 minutes):
|
MarioCadenas
force-pushed
the
init-speed-improvements
branch
from
bcc9026 to
1fb43f8
Compare
MarioCadenas
temporarily deployed
to
test-trigger-is
GitHub Actions
Inactive
MarioCadenas
force-pushed
the
init-speed-improvements
branch
from
1fb43f8 to
7599847
Compare
MarioCadenas
temporarily deployed
to
test-trigger-is
GitHub Actions
Inactive
MarioCadenas
force-pushed
the
init-speed-improvements
branch
from
7599847 to
257f2d6
Compare
MarioCadenas
temporarily deployed
to
test-trigger-is
GitHub Actions
Inactive
pkosiec
reviewed
Mar 12, 2026
Member
pkosiec
left a comment
pkosiec
left a comment
There was a problem hiding this comment.
It's so fast now 🚀 awesome work! Just a few comments
MarioCadenas
temporarily deployed
to
test-trigger-is
GitHub Actions
Inactive
MarioCadenas
force-pushed
the
init-speed-improvements
branch
from
72d5b8f to
f608c2d
Compare
MarioCadenas
temporarily deployed
to
test-trigger-is
GitHub Actions
Inactive
MarioCadenas
temporarily deployed
to
test-trigger-is
GitHub Actions
Inactive
MarioCadenas
temporarily deployed
to
test-trigger-is
GitHub Actions
Inactive
MarioCadenas
temporarily deployed
to
test-trigger-is
GitHub Actions
Inactive
pkosiec
approved these changes
Mar 16, 2026
Member
pkosiec
left a comment
pkosiec
left a comment
There was a problem hiding this comment.
Works well 👌 Just small comment about the "load more" labels
Member
There was a problem hiding this comment.
Do we need them though? If we have Paged alternatives?
1. Dead List* functions (~200 LOC) — listers.go
Nine old List* functions (ListJobs, ListServingEndpoints, ListCatalogs, ListConnections, ListGenieSpaces, ListVectorSearchIndexes,
ListDatabaseInstances, ListPostgresProjects, ListExperiments) are now dead code, replaced by NewPaged* counterparts. Also capResults
and maxListResults become dead. ~200 lines to remove.
| branchForClone = gitRef | ||
| subdirForClone = appkitTemplateDir | ||
| } | ||
| templateCh := resolveTemplateAsync(ctx, templateSrc, branchForClone, subdirForClone) |
Member
There was a problem hiding this comment.
3. Temp directory leak on early cancellation — init.go:744
If the user cancels during the name prompt, resolveTemplateAsync may have cloned a repo to a temp directory. Since awaitTemplate is
never called, cleanup is never invoked. Add a drain in the runCreate defer:
defer func() {
select {
case res := <-templateCh:
if res.cleanup != nil { res.cleanup() }
default:
}
}()
atilafassina
approved these changes
Mar 16, 2026
atilafassina
left a comment
atilafassina
left a comment
There was a problem hiding this comment.
🏆 agree the cleanup would be great to have. But not a blocker.
MarioCadenas
temporarily deployed
to
test-trigger-is
GitHub Actions
Inactive
MarioCadenas
temporarily deployed
to
test-trigger-is
GitHub Actions
Inactive
MarioCadenas
temporarily deployed
to
test-trigger-is
GitHub Actions
Inactive
rauchy
pushed a commit
that referenced
this pull request
Mar 17, 2026
## Changes Speed up `databricks apps init` by parallelizing three slow I/O operations (template cloning, npm install, resource list fetching) that previously ran sequentially, so they overlap with user interaction time. ### Main changes **Background template cloning** (`cmd/apps/init.go`): Template cloning now starts in a background goroutine immediately when the command runs, while the user is typing the project name. `resolveTemplateAsync` returns a channel; `awaitTemplate` either returns instantly (if already done) or shows a spinner for the remaining wait. This replaces the previous blocking `resolveTemplate` call that happened after the name prompt. **Background npm install** (`cmd/apps/init.go`): For Node.js templates, `startBackgroundNpmInstall` copies `package.json`/`package-lock.json` into the destination and kicks off `npm ci` while the user answers plugin/resource prompts. The result is awaited before template files are written to avoid concurrent writes. The Node.js initializer now skips redundant installs when `node_modules` already exists. **Prefetched resource lists** (`libs/apps/prompt/prefetch.go`, `libs/apps/prompt/prompt.go`): `PrefetchResources` fires background goroutines to fetch the first page of every resource type the manifest might need (warehouses, jobs, endpoints, experiments, Genie spaces, etc.) before the user selects plugins. A `ResourceCache` in the context stores `PagedFetcher` instances, so pickers render instantly when reached. Resource prompts now use `promptFromPagedFetcher` with "Load more..." and "Enter manually" / server-side search fallbacks, replacing the old `promptForResourceFromLister` that fetched everything in one blocking call. **Paged resource picker UX** (`libs/apps/prompt/prompt.go`, `libs/apps/prompt/listers.go`): All resource prompts switched from loading everything upfront to incremental paging (200 items/page, 10,000 cap). When capped, users can enter a name/ID manually or trigger a server-side search (currently supported for Jobs). Prompt titles now include the plugin display name for context (e.g. "Select SQL Warehouse for Analytics"). ### Supporting infrastructure - **`libs/apps/prompt/paged.go`** — `PagedFetcher` struct with `WaitForFirstPage`, `LoadMore`, `IsDone`, and a generic `collectN` helper for SDK iterators. - **`libs/apps/prompt/cache.go`** — Thread-safe `ResourceCache` stored in `context.Context` via `ContextWithCache`/`CacheFromContext`. - **`libs/apps/prompt/prefetch.go`** — Maps resource types to paged constructors and launches background prefetch goroutines for both single-step and multi-step (catalog/instance/project) resources. - **`libs/apps/prompt/listers.go`** — Added `NewPaged*` constructors for all resource types and a `SearchJobs` server-side search function. ### Other changes - Brand colors extracted into package-level `colorRed`/`colorGray`/`colorYellow`/`colorOrange` variables for consistency. - Filter hint changed from "type to filter" to "/ to filter" and explicit `.Filtering(true)` removed. - Volume IDs changed from `catalog.schema.volume` to `/Volumes/catalog/schema/volume`. - `manifest.Resource` gained a `PluginDisplayName` field (JSON-excluded) set during collection. - `manifest.Manifest` gained a `GetPluginNames` helper. - Checkmark output (`✔`/`✓`) consolidated into `prompt.PrintDone` across `deploy_bundle.go`, `import.go`, and `init.go`. - Existing `List*` functions now consistently apply `capResults` and use `min(len, maxListResults)` for initial slice capacity. ## Why <!-- Why are these changes needed? Provide the context that the reviewer might be missing. For example, were there any decisions behind the change that are not reflected in the code itself? --> ## Tests <!-- How have you tested the changes? --> <!-- If your PR needs to be included in the release notes for next release, add a separate entry in NEXT_CHANGELOG.md as part of your PR. --> --------- Co-authored-by: MarioCadenas <MarioCadenas@users.noreply.github.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
4 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Changes
Speed up
databricks apps initby parallelizing three slow I/O operations (template cloning, npm install, resource list fetching) that previously ran sequentially, so they overlap with user interaction time.Main changes
Background template cloning (
cmd/apps/init.go):Template cloning now starts in a background goroutine immediately when the command runs, while the user is typing the project name.
resolveTemplateAsyncreturns a channel;awaitTemplateeither returns instantly (if already done) or shows a spinner for the remaining wait. This replaces the previous blockingresolveTemplatecall that happened after the name prompt.Background npm install (
cmd/apps/init.go):For Node.js templates,
startBackgroundNpmInstallcopiespackage.json/package-lock.jsoninto the destination and kicks offnpm ciwhile the user answers plugin/resource prompts. The result is awaited before template files are written to avoid concurrent writes. The Node.js initializer now skips redundant installs whennode_modulesalready exists.Prefetched resource lists (
libs/apps/prompt/prefetch.go,libs/apps/prompt/prompt.go):PrefetchResourcesfires background goroutines to fetch the first page of every resource type the manifest might need (warehouses, jobs, endpoints, experiments, Genie spaces, etc.) before the user selects plugins. AResourceCachein the context storesPagedFetcherinstances, so pickers render instantly when reached. Resource prompts now usepromptFromPagedFetcherwith "Load more..." and "Enter manually" / server-side search fallbacks, replacing the oldpromptForResourceFromListerthat fetched everything in one blocking call.Paged resource picker UX (
libs/apps/prompt/prompt.go,libs/apps/prompt/listers.go):All resource prompts switched from loading everything upfront to incremental paging (200 items/page, 10,000 cap). When capped, users can enter a name/ID manually or trigger a server-side search (currently supported for Jobs). Prompt titles now include the plugin display name for context (e.g. "Select SQL Warehouse for Analytics").
Supporting infrastructure
libs/apps/prompt/paged.go—PagedFetcherstruct withWaitForFirstPage,LoadMore,IsDone, and a genericcollectNhelper for SDK iterators.libs/apps/prompt/cache.go— Thread-safeResourceCachestored incontext.ContextviaContextWithCache/CacheFromContext.libs/apps/prompt/prefetch.go— Maps resource types to paged constructors and launches background prefetch goroutines for both single-step and multi-step (catalog/instance/project) resources.libs/apps/prompt/listers.go— AddedNewPaged*constructors for all resource types and aSearchJobsserver-side search function.Other changes
colorRed/colorGray/colorYellow/colorOrangevariables for consistency..Filtering(true)removed.catalog.schema.volumeto/Volumes/catalog/schema/volume.manifest.Resourcegained aPluginDisplayNamefield (JSON-excluded) set during collection.manifest.Manifestgained aGetPluginNameshelper.✔/✓) consolidated intoprompt.PrintDoneacrossdeploy_bundle.go,import.go, andinit.go.List*functions now consistently applycapResultsand usemin(len, maxListResults)for initial slice capacity.Why
Tests