Closed
Conversation
Adds a new GenerateTemplate RPC endpoint that serves as the single source of truth for YAML format generation. Replaces frontend imperative YAML builders with a centralized backend handler. - Add EnvVarName field to PropertySpec for custom env var name mapping - Populate EnvVarName values across 14 driver specs (S3, GCS, Azure, ClickHouse, Postgres, BigQuery, Snowflake, Redshift, MotherDuck, Athena, MySQL, Druid, Pinot, Salesforce) - Define GenerateTemplate proto messages with validation annotations - Implement handler: auth check, driver validation, property validation, DuckDB rewrite for object/file stores and sqlite - Add comprehensive YAML renderers for connectors and models with precise formatting via yaml.Node tree construction - Implement env var conflict resolution with _1, _2 suffix appending - Add 54 passing test cases covering all drivers, error scenarios, and edge cases Frontend migration to use this RPC will follow as a separate step. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Replace client-side YAML builders with backend RPC for single source of truth. - New helper: `generateTemplate()` + `mergeEnvVars()` in generate-template.ts - Updated submission paths: submitAddConnectorForm, submitAddSourceForm, saveConnectorAnyway - Updated preview: computeYamlPreview now calls RPC (async, 150ms debounced) - HTTPS connector path preserved (backend doesn't support headers array yet) - Removed dead code: compileSourceYAML, prepareSourceFormData, maybeRewriteToDuckDb - Backend now handles DuckDB rewrite and env var naming (conflict resolution) - Tests: added mergeEnvVars (9 tests) and normalizeConnectorError (11 tests) - Net: 234 lines removed (~27% reduction in submission/preview code) Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
…ector headers
Add HTTPS headers support to the backend GenerateTemplate RPC, enabling removal
of all client-side YAML generation code. The HTTPS connector now uses the same
RPC path as all other connectors.
Backend changes:
- Add `headers` property to HTTPS driver ConfigProperties
- Implement header rendering with sensitive value extraction to .env
- Support Authorization header scheme prefixes (Bearer, Basic, Token, Bot)
- Use `connector.{driver}.{key}` naming convention for header env vars
- Add 4 comprehensive test cases for header handling
Frontend changes:
- Convert key-value UI arrays to maps before RPC submission
- Remove HTTPS-specific YAML preview and submission code
- All connectors now use GenerateTemplate RPC exclusively
- Remove 13 dead functions (compileConnectorYAML, updateDotEnvWithSecrets, etc)
- Consolidate env var naming to backend-only responsibility
Schema cleanup:
- Remove 30 dead `x-env-var-name` annotations (frontend no longer reads them)
- Single source of truth for env var names: backend PropertySpec.EnvVarName
Results:
- -1,317 lines of dead code removed
- +221 lines of backend implementation
- 2,055 frontend tests pass, 62 backend tests pass (4 new HTTPS tests)
The mapping from property Key to env var name doesn't follow a mechanical pattern. Well-known names are shared across drivers (AWS_ACCESS_KEY_ID), some add infixes (AZURE_STORAGE_*), and others diverge entirely (GCS key_id -> GCP_ACCESS_KEY_ID). Each driver must specify explicitly. Also remove personal docs from .gitignore.
- Fix gofmt alignment in connectors.go PropertySpec struct - Remove unused connectorName param from renderDuckDBModelYAML - Use const instead of let for newEnvBlob and connectorYamlBlob
…o — Core types: Template, File, ProcessedProp registry.go — Registry with //go:embed loading of YAML definitions, List(), Get(), ListByTags(), LookupByDriver() render.go — Rendering engine: pre-processes properties (secret extraction, empty filtering, DuckDB derivation), renders Go text/template with [[ ]] delimiters funcmap.go — Template functions: renderProps, indent, quote env.go — Extracted: ResolveEnvVarName, ReadEnvKeys headers.go — Extracted: IsSensitiveHeaderKey, SplitAuthSchemePrefix, HeaderKeyToEnvSegment, ResolveHeaderEnvVarName (with fixed per-call regex compilation bug) duckdb.go — Extracted: BuildDuckDBQuery, matchesExt (with fixed false positive bug for paths like parquet-archive/readme.txt) 32 template definitions (runtime/templates/definitions/) 17 connector templates: s3, gcs, azure, https, postgres, bigquery, snowflake, mysql, athena, redshift, salesforce, clickhouse, duckdb, motherduck, druid, pinot, starrocks 7 DuckDB model templates: s3-duckdb, gcs-duckdb, azure-duckdb, https-duckdb, local-file-duckdb, sqlite-duckdb 8 warehouse model templates: clickhouse-model, postgres-model, mysql-model, bigquery-model, snowflake-model, athena-model, redshift-model, salesforce-model 1 new template: iceberg-duckdb (the motivating use case) Proto definitions Template, TemplateFile, ListTemplatesRequest/Response, GenerateFileRequest/Response, GeneratedFile messages ListTemplates and GenerateFile RPCs on RuntimeService Frontend client auto-generated via Orval Server handlers (runtime/server/templates.go) ListTemplates — returns templates filtered by tags GenerateFile — renders template, optionally writes files + merges .env Tests (4 test files, 32+ test cases) Registry: loading, lookup, tag filtering, sorted output, all definitions valid Render: S3 connector, S3-DuckDB model, Snowflake warehouse model, Redshift no-dev, Iceberg-DuckDB, env var conflicts, empty filtering, output filtering, local file, SQLite Env: explicit name, fallback, conflict, double conflict DuckDB: query building for all formats, false positive fix Headers: sensitive detection, auth scheme splitting, env segment naming Bug fixes from PRD containsExt false positive fixed (basename suffix check) headerKeyToEnvSegment regex compiled at package level Render errors propagated (not silently empty)
…nnector, rewire front end to new templates
…s to put files into local_files in tmp/
Contributor
Author
|
All three fixes are done: Backslash escaping (templates.go:176-177) — Now escapes backslashes before double quotes, and includes \ in the ContainsAny check so values with backslashes also get quoted. Cached funcMap (funcmap.go:11-25) — Replaced funcMap() function with a package-level sharedFuncMap var. All entries are stateless functions so sharing is safe. Empty array edge case (render.go:188-191) — When kvArrayToMap returns nil (empty/malformed array), we now continue instead of falling through to fmt.Sprintf which would render "[]". Skipped issue 1 (ListTemplates auth) since the request has no instance_id and returns only static template metadata. |
Contributor
Author
|
tagging both as this has a lot of both side changes. I took Cyrus' first changes along with the Notion doc and compiled the changes here. In terms of large PR, this is huge but a lot of it is removing .ts config and creating the JSON config in templated (see description for #s) Let me know what yall think about the implementation ! |
Resolves conflicts: keeps template-driven approach from branch, regenerates proto/runtime clients for GenerateFile API, updates generate-template.ts to use v2 client. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
These were dropped during merge conflict resolution but are still imported by AddDataModal.svelte. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
The auto-merge kept the branch's slimmer version, but saveAiConnector.ts (from main) depends on compileConnectorYAML, updateDotEnvWithSecrets, and other functions that were only on main's side. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Missing exports (getSchemaSecretKeys, getSchemaStringKeys, etc.) needed by saveAiConnector.ts. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Add AI_CONNECTORS and updateRillYAMLWithAiConnector imports that were dropped during merge conflict resolution. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Contributor
Author
|
going to split this into smaller PRs based on the new flows that we have today; lots of changes |
This was referenced Apr 28, 2026
Draft
8 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Move connector schemas, SQL generation, and YAML templating from hand-written TypeScript into declarative JSON template definitions served by a new Go
runtime/templates/package.Excluding auto-generated code (proto, Orval), JSON definitions, and deleted TypeScript schemas, the actual implementation is ~2,100 lines of new Go + TypeScript across 46 files — replacing ~3,400 lines of removed TypeScript.
What changed
runtime/templates/Go package (7 files, ~1,100 lines): registry withembed.FSloading, property processor that extracts env var refs for secrets,text/templaterenderer with[[ ]]delimiters, and template helper functions (duckdbSQL,propVal,default,azureContainer,azureBlobPath,clickhouseURLSuffix)ListTemplatesreturns all definitions with schemas;GenerateFilerenders templates server-side, optionally writing files and merging.envAddDataModalfetches schemas fromListTemplatesAPI instead of importing TypeScript schema files; form rendering, validation, and submission unchanged; icon resolution uses template-levelicon/small_iconfieldsschemas/*.ts),sourceUtils.ts(SQL generation),connector-icon-mapping.ts,icons.ts— all replaced by JSON definitionsurl()table function support with auto-detected format parameter andheaders()syntax for authenticated endpoints_reason— ClickHouse'sfile()/sqlite()require server-sideuser_filesdirectory, unusable in Rill CloudnormalizeOlapForTemplatechecks bothconnectorsandprojectConnectorsto resolve arbitrarily-named OLAP connectors (e.g.clickhouse_1)Code review fixes
.envinjection hardening:appendEnvVarnow strips newlines and quotes values containing spaces/special characters to prevent env var injectionmanagedskip replaced: genericx-omit-if-defaultschema annotation replaces the ClickHouse-specifickey == "managed"check inrender.go; removed deadtoBoolhelperGenerateTemplate,GenerateFile, andwriteRenderedFilesnow log warnings instead of silently swallowing errorsdefaultValsafety: added doc comment warning against pipeline syntax (only positional[[ default (expr) "fallback" ]]is safe)display_nameacross OLAP engines (Postgres,Azure);x-categoryvalues use camelCase (sourceOnly); added missingnamefield pattern validation tohttps-duckdb.jsonmanageddefault: changed fromfalsetotrueto match theallOfconditional that enforcesconst: truefor rill-managedgetSchemaSecretKeys,getSchemaStringKeys,getBackendConnectorNamefromschema-utils.tsLine count breakdown
QA: Clickhouse, with verification workaround (Aditya will work on proper changes here)
https://ui.rilldata.com/demo/ch-managed-sources/-/status
Some of the error messages are quite confusing as it always complains about manually setting the columns when thats not the actual issue. will likely need a follow up PR that helps with error messaging.
Checklist:
Developed in collaboration with Claude Code