Skip to content

Flyte 2#6583

Draft
EngHabu wants to merge 246 commits intomainfrom
v2
Draft

Flyte 2#6583
EngHabu wants to merge 246 commits intomainfrom
v2

Conversation

@EngHabu
Copy link
Copy Markdown
Contributor

@EngHabu EngHabu commented Aug 22, 2025

  • Add flyteidl and minimal publishing workflows

Tracking issue

Why are the changes needed?

What changes were proposed in this pull request?

How was this patch tested?

Labels

Please add one or more of the following labels to categorize your PR:

  • added: For new features.
  • changed: For changes in existing functionality.
  • deprecated: For soon-to-be-removed features.
  • removed: For features being removed.
  • fixed: For any bug fixed.
  • security: In case of vulnerabilities

This is important to improve the readability of release notes.

Setup process

Screenshots

Check all the applicable boxes

  • I updated the documentation accordingly.
  • All new and existing tests passed.
  • All commits are signed-off.

Related PRs

Docs link

* Add flyteidl and minimal publishing workflows

Signed-off-by: Haytham Abuelfutuh <haytham@afutuh.com>

* pretend version for buf python

Signed-off-by: Haytham Abuelfutuh <haytham@afutuh.com>

---------

Signed-off-by: Haytham Abuelfutuh <haytham@afutuh.com>
@flyte-bot
Copy link
Copy Markdown
Collaborator

Bito Automatic Review Skipped - Branch Excluded

Bito didn't auto-review because the source or target branch is excluded from automatic reviews.
No action is needed if you didn't intend for the agent to review it. Otherwise, to manually trigger a review, type /review in a comment and save.
You can change the branch exclusion settings here, or contact your Bito workspace admin at eduardo@union.ai.

* python version fix and use OIDC to authenticate crates

Signed-off-by: Haytham Abuelfutuh <haytham@afutuh.com>

* Fix python versioning

Signed-off-by: Haytham Abuelfutuh <haytham@afutuh.com>

* Set version before setting up env

Signed-off-by: Haytham Abuelfutuh <haytham@afutuh.com>

* echo version

Signed-off-by: Haytham Abuelfutuh <haytham@afutuh.com>

* Write to GITHUB_ENV

Signed-off-by: Haytham Abuelfutuh <haytham@afutuh.com>

* Use env:

Signed-off-by: Haytham Abuelfutuh <haytham@afutuh.com>

* buf generate

Signed-off-by: Haytham Abuelfutuh <haytham@afutuh.com>

---------

Signed-off-by: Haytham Abuelfutuh <haytham@afutuh.com>
@flyte-bot
Copy link
Copy Markdown
Collaborator

Bito Automatic Review Skipped - Branch Excluded

Bito didn't auto-review because the source or target branch is excluded from automatic reviews.
No action is needed if you didn't intend for the agent to review it. Otherwise, to manually trigger a review, type /review in a comment and save.
You can change the branch exclusion settings here, or contact your Bito workspace admin at eduardo@union.ai.

Signed-off-by: Haytham Abuelfutuh <haytham@afutuh.com>
@flyte-bot
Copy link
Copy Markdown
Collaborator

Bito Automatic Review Skipped - Branch Excluded

Bito didn't auto-review because the source or target branch is excluded from automatic reviews.
No action is needed if you didn't intend for the agent to review it. Otherwise, to manually trigger a review, type /review in a comment and save.
You can change the branch exclusion settings here, or contact your Bito workspace admin at eduardo@union.ai.

* Mark go package as /v2 to be able to import both at the same time

Signed-off-by: Haytham Abuelfutuh <haytham@afutuh.com>

* Upgrade to go 1.24

Signed-off-by: Haytham Abuelfutuh <haytham@afutuh.com>

* Rename python, rust and npm to flyteidl2

Signed-off-by: Haytham Abuelfutuh <haytham@afutuh.com>

---------

Signed-off-by: Haytham Abuelfutuh <haytham@afutuh.com>
@flyte-bot
Copy link
Copy Markdown
Collaborator

Bito Automatic Review Skipped - Branch Excluded

Bito didn't auto-review because the source or target branch is excluded from automatic reviews.
No action is needed if you didn't intend for the agent to review it. Otherwise, to manually trigger a review, type /review in a comment and save.
You can change the branch exclusion settings here, or contact your Bito workspace admin at eduardo@union.ai.

…6610)

* trying idl2

Signed-off-by: Haytham Abuelfutuh <haytham@afutuh.com>

* wip

Signed-off-by: Haytham Abuelfutuh <haytham@afutuh.com>

* wip

Signed-off-by: Haytham Abuelfutuh <haytham@afutuh.com>

* Update trace def

Signed-off-by: Haytham Abuelfutuh <haytham@afutuh.com>

* refactor: rename idl2 to flyteidl2

Signed-off-by: machichima <nary12321@gmail.com>

* generate -> buf

Signed-off-by: Haytham Abuelfutuh <haytham@afutuh.com>

* refactor: rename idl2 to flyteidl2

Signed-off-by: machichima <nary12321@gmail.com>

* build: re-gen

Signed-off-by: machichima <nary12321@gmail.com>

* fix: exclude python venv

Signed-off-by: machichima <nary12321@gmail.com>

* Add translator service

Signed-off-by: Haytham Abuelfutuh <haytham@afutuh.com>

* Regen

Signed-off-by: Haytham Abuelfutuh <haytham@afutuh.com>

* buf gen

Signed-off-by: Haytham Abuelfutuh <haytham@afutuh.com>

* refactor

Signed-off-by: Haytham Abuelfutuh <haytham@afutuh.com>

* wip

Signed-off-by: Haytham Abuelfutuh <haytham@afutuh.com>

* remove node execution service proto

Signed-off-by: Haytham Abuelfutuh <haytham@afutuh.com>

* Breakup task service into a separate package

Signed-off-by: Haytham Abuelfutuh <haytham@afutuh.com>

* Help strings for make file targets

Signed-off-by: Haytham Abuelfutuh <haytham@afutuh.com>

* Update TaskLog with read bool

Signed-off-by: Haytham Abuelfutuh <haytham@afutuh.com>

* Trying the updated protovalidate plugin

Signed-off-by: Haytham Abuelfutuh <haytham@afutuh.com>

* protovalidate with semver

Signed-off-by: Haytham Abuelfutuh <haytham@afutuh.com>

* try claude :shrug

Signed-off-by: Haytham Abuelfutuh <haytham@afutuh.com>

* add buf

Signed-off-by: Haytham Abuelfutuh <haytham@afutuh.com>

* update

Signed-off-by: Haytham Abuelfutuh <haytham@afutuh.com>

* update deps

Signed-off-by: Haytham Abuelfutuh <haytham@afutuh.com>

* build: remove generate validate for python

Signed-off-by: machichima <nary12321@gmail.com>

* build: use managed mode on buf.gen.yaml

Signed-off-by: machichima <nary12321@gmail.com>

* Update validate go plugin

Signed-off-by: Haytham Abuelfutuh <haytham@afutuh.com>

* Update imagebuild

Signed-off-by: Haytham Abuelfutuh <haytham@afutuh.com>

* Fix oneof buf lint

Signed-off-by: Haytham Abuelfutuh <haytham@afutuh.com>

* wip

Signed-off-by: Haytham Abuelfutuh <haytham@afutuh.com>

* Add deps to generated python code

Signed-off-by: Haytham Abuelfutuh <haytham@afutuh.com>

* Be explicit about the index

Signed-off-by: Haytham Abuelfutuh <haytham@afutuh.com>

* updated idl

Signed-off-by: Haytham Abuelfutuh <haytham@afutuh.com>

* Catch up to main

Signed-off-by: Haytham Abuelfutuh <haytham@afutuh.com>

* Generate mocks for workflow package

Signed-off-by: Haytham Abuelfutuh <haytham@afutuh.com>

* generate mocks for other packages

Signed-off-by: Haytham Abuelfutuh <haytham@afutuh.com>

* buf improvements

Signed-off-by: Haytham Abuelfutuh <haytham@afutuh.com>

* updates

Signed-off-by: Haytham Abuelfutuh <haytham@afutuh.com>

* feat: generate mock for common

Signed-off-by: machichima <nary12321@gmail.com>

* use make gen in GH Actions

Signed-off-by: Haytham Abuelfutuh <haytham@afutuh.com>

* Do not setup python env in generation CI

Signed-off-by: Haytham Abuelfutuh <haytham@afutuh.com>

* setup uv

Signed-off-by: Haytham Abuelfutuh <haytham@afutuh.com>

* set working directory

Signed-off-by: Haytham Abuelfutuh <haytham@afutuh.com>

* Try to reference files by path

Signed-off-by: Haytham Abuelfutuh <haytham@afutuh.com>

* wip

Signed-off-by: Haytham Abuelfutuh <haytham@afutuh.com>

* Explicitly move .venv to root

Signed-off-by: Haytham Abuelfutuh <haytham@afutuh.com>

* try exclude in buf.yaml

Signed-off-by: Haytham Abuelfutuh <haytham@afutuh.com>

* set UV_PROJECT_ENVIRONMENT

Signed-off-by: Haytham Abuelfutuh <haytham@afutuh.com>

* wip

Signed-off-by: Haytham Abuelfutuh <haytham@afutuh.com>

* download tooling

Signed-off-by: Haytham Abuelfutuh <haytham@afutuh.com>

* add which mockery

Signed-off-by: Haytham Abuelfutuh <haytham@afutuh.com>

* wip

Signed-off-by: Haytham Abuelfutuh <haytham@afutuh.com>

---------

Signed-off-by: Haytham Abuelfutuh <haytham@afutuh.com>
Signed-off-by: machichima <nary12321@gmail.com>
Co-authored-by: machichima <nary12321@gmail.com>
@flyte-bot
Copy link
Copy Markdown
Collaborator

Bito Automatic Review Skipped - Branch Excluded

Bito didn't auto-review because the source or target branch is excluded from automatic reviews.
No action is needed if you didn't intend for the agent to review it. Otherwise, to manually trigger a review, type /review in a comment and save.
You can change the branch exclusion settings here, or contact your Bito workspace admin at eduardo@union.ai.

* Add Trigger APIs

Signed-off-by: Haytham Abuelfutuh <haytham@afutuh.com>

* update triggers

Signed-off-by: Haytham Abuelfutuh <haytham@afutuh.com>

* wip

Signed-off-by: Haytham Abuelfutuh <haytham@afutuh.com>

* refactor

Signed-off-by: Haytham Abuelfutuh <haytham@afutuh.com>

---------

Signed-off-by: Haytham Abuelfutuh <haytham@afutuh.com>
@bito-code-review
Copy link
Copy Markdown

Bito Automatic Review Skipped - Branch Excluded

Bito didn't auto-review because the source or target branch is excluded from automatic reviews.
No action is needed if you didn't intend for the agent to review it. Otherwise, to manually trigger a review, type /review in a comment and save.
You can change the branch exclusion settings here, or contact your Bito workspace admin at haytham@union.ai.

* Fix cargo build and add CI check

Signed-off-by: Haytham Abuelfutuh <haytham@afutuh.com>

* lint lib.rs

Signed-off-by: Haytham Abuelfutuh <haytham@afutuh.com>

* Update Readme and contributing guides

Signed-off-by: Haytham Abuelfutuh <haytham@afutuh.com>

* signed?

Signed-off-by: Haytham Abuelfutuh <haytham@afutuh.com>

* fix build target

Signed-off-by: Haytham Abuelfutuh <haytham@afutuh.com>

* fix build-crate target

Signed-off-by: Haytham Abuelfutuh <haytham@afutuh.com>

* fix included ts files in npm

Signed-off-by: Haytham Abuelfutuh <haytham@afutuh.com>

---------

Signed-off-by: Haytham Abuelfutuh <haytham@afutuh.com>
@bito-code-review
Copy link
Copy Markdown

Bito Automatic Review Skipped - Branch Excluded

Bito didn't auto-review because the source or target branch is excluded from automatic reviews.
No action is needed if you didn't intend for the agent to review it. Otherwise, to manually trigger a review, type /review in a comment and save.
You can change the branch exclusion settings here, or contact your Bito workspace admin at haytham@union.ai.

Signed-off-by: Yee Hing Tong <wild-endeavor@users.noreply.github.com>
@bito-code-review
Copy link
Copy Markdown

Bito Automatic Review Skipped - Branch Excluded

Bito didn't auto-review because the source or target branch is excluded from automatic reviews.
No action is needed if you didn't intend for the agent to review it. Otherwise, to manually trigger a review, type /review in a comment and save.
You can change the branch exclusion settings here, or contact your Bito workspace admin at haytham@union.ai.

* Add device class to `GpuAccelerator`

Signed-off-by: Jeev B <jeevb@users.noreply.github.com>

* make gen

Signed-off-by: Jeev B <jeevb@users.noreply.github.com>

* Add HABANA_GAUDI as supported device class

Signed-off-by: Jeev B <jeevb@users.noreply.github.com>

---------

Signed-off-by: Jeev B <jeevb@users.noreply.github.com>
@bito-code-review
Copy link
Copy Markdown

Bito Automatic Review Skipped - Branch Excluded

Bito didn't auto-review because the source or target branch is excluded from automatic reviews.
No action is needed if you didn't intend for the agent to review it. Otherwise, to manually trigger a review, type /review in a comment and save.
You can change the branch exclusion settings here, or contact your Bito workspace admin at haytham@union.ai.

Signed-off-by: Haytham Abuelfutuh <haytham@afutuh.com>
Signed-off-by: Haytham Abuelfutuh <haytham@afutuh.com>
Signed-off-by: Haytham Abuelfutuh <haytham@afutuh.com>
Signed-off-by: Haytham Abuelfutuh <haytham@afutuh.com>
Signed-off-by: Haytham Abuelfutuh <haytham@afutuh.com>
@bito-code-review
Copy link
Copy Markdown

Bito Automatic Review Skipped - Branch Excluded

Bito didn't auto-review because the source or target branch is excluded from automatic reviews.
No action is needed if you didn't intend for the agent to review it. Otherwise, to manually trigger a review, type /review in a comment and save.
You can change the branch exclusion settings here, or contact your Bito workspace admin at haytham@union.ai.

Signed-off-by: Iaroslav Ciupin <iaroslav@union.ai>
@bito-code-review
Copy link
Copy Markdown

Bito Automatic Review Skipped - Branch Excluded

Bito didn't auto-review because the source or target branch is excluded from automatic reviews.
No action is needed if you didn't intend for the agent to review it. Otherwise, to manually trigger a review, type /review in a comment and save.
You can change the branch exclusion settings here, or contact your Bito workspace admin at haytham@union.ai.

Signed-off-by: Iaroslav Ciupin <iaroslav@union.ai>
@bito-code-review
Copy link
Copy Markdown

Bito Automatic Review Skipped - Branch Excluded

Bito didn't auto-review because the source or target branch is excluded from automatic reviews.
No action is needed if you didn't intend for the agent to review it. Otherwise, to manually trigger a review, type /review in a comment and save.
You can change the branch exclusion settings here, or contact your Bito workspace admin at haytham@union.ai.

* Clean up triggers api

Signed-off-by: Iaroslav Ciupin <iaroslav@union.ai>

* make gen

Signed-off-by: Iaroslav Ciupin <iaroslav@union.ai>

---------

Signed-off-by: Iaroslav Ciupin <iaroslav@union.ai>
@bito-code-review
Copy link
Copy Markdown

Bito Automatic Review Skipped - Branch Excluded

Bito didn't auto-review because the source or target branch is excluded from automatic reviews.
No action is needed if you didn't intend for the agent to review it. Otherwise, to manually trigger a review, type /review in a comment and save.
You can change the branch exclusion settings here, or contact your Bito workspace admin at haytham@union.ai.

@pingsutw pingsutw changed the title Flyte 2 WIP Flyte 2 Apr 15, 2026
@github-actions github-actions bot mentioned this pull request Apr 15, 2026
3 tasks
popojk and others added 2 commits April 15, 2026 10:21
Signed-off-by: Alex Wu <c.alexwu@gmail.com>
Signed-off-by: Kevin Su <pingsutw@apache.org>
Co-authored-by: Kevin Su <pingsutw@apache.org>
)

Signed-off-by: Kevin Su <pingsutw@apache.org>
Signed-off-by: Kevin Su <pingsutw@apache.org>
@github-actions github-actions bot mentioned this pull request Apr 16, 2026
3 tasks
Signed-off-by: Alex Wu <c.alexwu@gmail.com>
… numbering (#7220)

Signed-off-by: Kevin Su <pingsutw@apache.org>
* proto stubs

Signed-off-by: Katrina Rogan <katroganGH@gmail.com>

* add demo cluster impl

Signed-off-by: Katrina Rogan <katroganGH@gmail.com>

* impl

Signed-off-by: Katrina Rogan <katroganGH@gmail.com>

* mocks

Signed-off-by: Katrina Rogan <katroganGH@gmail.com>

* Cleaner

Signed-off-by: Katrina Rogan <katroganGH@gmail.com>

* tests

Signed-off-by: Katrina Rogan <katroganGH@gmail.com>

* dockerfile

Signed-off-by: Katrina Rogan <katroganGH@gmail.com>

* move log_streamer to dataproxy

Signed-off-by: Katrina Rogan <katroganGH@gmail.com>

* regen

Signed-off-by: Katrina Rogan <katroganGH@gmail.com>

* regen

Signed-off-by: Katrina Rogan <katroganGH@gmail.com>

* revert, restore, redo

Signed-off-by: Katrina Rogan <katroganGH@gmail.com>

* revert

Signed-off-by: Katrina Rogan <katroganGH@gmail.com>

---------

Signed-off-by: Katrina Rogan <katroganGH@gmail.com>
@github-actions github-actions bot mentioned this pull request Apr 16, 2026
3 tasks
Signed-off-by: Kevin Su <pingsutw@apache.org>
…K8s cascade (#7225)

Signed-off-by: M. Adil Fayyaz <62440954+AdilFayyaz@users.noreply.github.com>
Signed-off-by: Kevin Su <pingsutw@apache.org>
Co-authored-by: Kevin Su <pingsutw@apache.org>
SVilgelm and others added 4 commits April 17, 2026 10:08
…logKeys (#7229)

* [V2] Expand contextutils with org, service_name, cluster_id keys and logKeys

The shared interceptor chain in Union Cloud sets context values using v1
contextutils.Key types, but v2 services (actions, leaseworker, leasor,
events-proxy, executorv2) use the v2 logger which reads v2 contextutils.Key.
Both are `type Key string` but are different Go types, so context.Value()
lookups fail silently and all context-propagated log fields are missing from
v2 service logs.

Add OrganizationKey, ServiceNameKey, and ClusterNameKey constants to v2
contextutils, and expand logKeys with ProjectKey, DomainKey, and the three
new keys so that GetLogFields() includes them in structured log output.

This is one half of the fix — the other half is a context bridge interceptor
in the cloud repo that copies v1 values onto v2 keys at request time.

Assisted-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Signed-off-by: Sergey Vilgelm <sergey@union.ai>

* make gen

Signed-off-by: Sergey Vilgelm <sergey@union.ai>

---------

Signed-off-by: Sergey Vilgelm <sergey@union.ai>
* add: dep

Signed-off-by: M. Adil Fayyaz <62440954+AdilFayyaz@users.noreply.github.com>

* restructure

Signed-off-by: M. Adil Fayyaz <62440954+AdilFayyaz@users.noreply.github.com>

* wip

Signed-off-by: M. Adil Fayyaz <62440954+AdilFayyaz@users.noreply.github.com>

* restruct

Signed-off-by: M. Adil Fayyaz <62440954+AdilFayyaz@users.noreply.github.com>

* fix

Signed-off-by: M. Adil Fayyaz <62440954+AdilFayyaz@users.noreply.github.com>

* move config

Signed-off-by: M. Adil Fayyaz <62440954+AdilFayyaz@users.noreply.github.com>

* remove db.go

Signed-off-by: M. Adil Fayyaz <62440954+AdilFayyaz@users.noreply.github.com>

* address comments

Signed-off-by: M. Adil Fayyaz <62440954+AdilFayyaz@users.noreply.github.com>

* comments

Signed-off-by: M. Adil Fayyaz <62440954+AdilFayyaz@users.noreply.github.com>

---------

Signed-off-by: M. Adil Fayyaz <62440954+AdilFayyaz@users.noreply.github.com>
…inistic (#7231)

* fix(runs): make TestWatchActionUpdates_OnlyStreamsTargetAction deterministic

Start watcher before creating actions so the subscriber is registered
when the creation NOTIFY arrives. Drain the creation notification before
the real assertions to eliminate the race that caused spurious
"unexpected update for action target" failures.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Signed-off-by: Sergey Vilgelm <sergey@union.ai>

* fix(runs): replace sleep with deterministic subscriber check

Replace time.Sleep with require.Eventually polling actionSubscribers
to ensure the watcher is registered before creating actions. This
removes all timing dependencies from the test.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Signed-off-by: Sergey Vilgelm <sergey@union.ai>

---------

Signed-off-by: Sergey Vilgelm <sergey@union.ai>
Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* fix(Makefile): preserve exit code in timed macro

The timed macro previously lost the exit code of the wrapped command due to
chaining with semicolons — make always saw the exit code from the final echo
statement (0). Now the macro captures the command's exit code in a variable,
prints the timing, then exits with the original code, ensuring CI targets
like buf-lint properly fail when violations are detected.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Signed-off-by: Sergey Vilgelm <sergey@union.ai>

* fix(proto): resolve all buf lint violations

- Remove unused import of google/api/annotations.proto from cluster/service.proto
- Remove unused import of flyteidl2/task/common.proto from queue_service.proto
- Fix package mismatch in settings protos: change package from flyteidl2.org to
  flyteidl2.settings and update all field option references accordingly
- Remove unused buf dependency buf.build/grpc-ecosystem/grpc-gateway
- Regenerate all generated files (Go, Python, Rust, TypeScript)

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Signed-off-by: Sergey Vilgelm <sergey@union.ai>

---------

Signed-off-by: Sergey Vilgelm <sergey@union.ai>
Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
@github-actions github-actions bot mentioned this pull request Apr 17, 2026
1 task
EngHabu and others added 2 commits April 17, 2026 11:03
Signed-off-by: Haytham Abuelfutuh <haytham@afutuh.com>
* add: dep

Signed-off-by: M. Adil Fayyaz <62440954+AdilFayyaz@users.noreply.github.com>

* restructure

Signed-off-by: M. Adil Fayyaz <62440954+AdilFayyaz@users.noreply.github.com>

* wip

Signed-off-by: M. Adil Fayyaz <62440954+AdilFayyaz@users.noreply.github.com>

* restruct

Signed-off-by: M. Adil Fayyaz <62440954+AdilFayyaz@users.noreply.github.com>

* fix

Signed-off-by: M. Adil Fayyaz <62440954+AdilFayyaz@users.noreply.github.com>

* move config

Signed-off-by: M. Adil Fayyaz <62440954+AdilFayyaz@users.noreply.github.com>

* remove db.go

Signed-off-by: M. Adil Fayyaz <62440954+AdilFayyaz@users.noreply.github.com>

* address comments

Signed-off-by: M. Adil Fayyaz <62440954+AdilFayyaz@users.noreply.github.com>

* impl: internal service

Signed-off-by: M. Adil Fayyaz <62440954+AdilFayyaz@users.noreply.github.com>

* add tests

Signed-off-by: M. Adil Fayyaz <62440954+AdilFayyaz@users.noreply.github.com>

* address comments

Signed-off-by: M. Adil Fayyaz <62440954+AdilFayyaz@users.noreply.github.com>

---------

Signed-off-by: M. Adil Fayyaz <62440954+AdilFayyaz@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.