Add GH Action workflow to update downstream repos by joverlee521 · Pull Request #73 · nextstrain/shared

joverlee521 · 2026-04-13T20:30:05Z

Description of proposed changes

Searches the Nextstrain GitHub org to find repos that have the .gitrepo file with the nextstrain/shared or nextstrain/ingest remote to create a matrix of repos to potentially update. Installs and uses git subrepo to pull in the latest changes with the --force flag to avoid merge conflicts to due rebasing in the downstream repo. If there are changes pulled down, then git subrepo will create a single commit. If there is a single commit, then push up the changes to a branch and create or update the PR in the downstream repo. Nothing happens if there were no changes and workflow exits with error if it encounters more than one commit.

Related issue(s)

Prompted by nextstrain/public#39 (comment)

Checklist

Checks pass
~~If adding a script, add an entry for it in the README.~~
Test run creates/updates PRs in vendored repos
latest test run

jameshadfield · 2026-04-13T20:52:55Z

+          git config user.email "${{ vars.GIT_USER_EMAIL_NEXTSTRAIN_BOT }}"
+
+          git switch -c "$branch"
+          git subrepo pull "$VENDORED_PATH" --force


(it's been (thankfully) a while since I interacted with subrepo) if there were local changes, in what situation(s) would we want to override them?

In practice, I have not seen any of the vendored repos use long-term local changes. I've only seen temporary test of local changes that are ported to the central nextstrain/shared repo. I've mostly had to use the --force flag to work around failures due to rebasing in the vendored repo, as described in the README:

shared/README.md

Lines 38 to 44 in c29898f

> **Warning**

> Beware of rebasing/dropping the parent commit of a `git subrepo` update

`git subrepo` relies on metadata in the `shared/vendored/.gitrepo` file,

which includes the hash for the parent commit in the pathogen repos.

If this hash no longer exists in the commit history, there will be errors when

running future `git subrepo pull` commands.

--force is only necessary if someone had done a manual rebase of the git subrepo-generated commit, right? That shouldn't be an issue going forwards with these automated PRs.

I'd vote to drop --force for visibility into any local changes, instead of silently discarding them.

victorlin · 2026-04-14T14:39:32Z

+          matrix=$(gh api -X GET search/code \
+            -f q='org:nextstrain filename:.gitrepo "remote = https://github.com/nextstrain/shared"' \


This should also search for references to the old repo name nextstrain/ingest. See code search

Ah, good catch, totally forgot about the old name.

Added in ada3160.

This actually flagged an issue that I didn't consider before: avian-flu has both shared/vendored and ingest/vendored, which causes one of the pushes to fail because of mismatched ref. avian-flu should be cleaned up to only have a single copy of the vendored repo, but this workflow should probably also flag when there are multiple vendored paths for the same repo.

avian-flu/ingest/vendored will need to be removed as part of nextstrain/avian-flu#67.

Added ab5399f to deduplicate by repo and only update nextstrain/shared if both are in the repo. I decided not to add another step for flagging multiple paths since that feels out of the purview of this repo...

This actually flagged an issue that I didn't consider before: avian-flu has both shared/vendored and ingest/vendored, which causes one of the pushes to fail because of mismatched ref.

The alternative is to push to different branches and open separate PRs for each. But that requires additional changes and probably not worth it if avian-flu is the only exception which is slated for removal anyways.

Searches the Nextstrain GitHub org to find repos that have the `.gitrepo` file with the nextstrain/shared remote to create a matrix of repos to potentially update. Installs and uses `git subrepo` to pull in the latest changes with the `--force` flag to avoid merge conflicts to due rebasing in the downstream repo. If there are changes pulled down, then `git subrepo` will create a single commit. If there is a single commit, then push up the changes to a branch and create or update the PR in the downstream repo. Nothing happens if there were no changes and workflow exits with error if it encounters more than one commit.

I was unable to get the search/code API to work with the 'OR' syntax so just added a separate query for nextstrain/ingest and concatenated the two arrays. Deduplicated the final array to guard against potential overlap.

joverlee521 · 2026-04-15T18:07:23Z

nextstrain/rubella#53 shows example of automated update causing workflow error flagged by CI. I created a separate branch/PR to fix the error in nextstrain/rubella#54 so that future updates from this GHA does not overwrite the manual fixes.

We should only be keeping a single copy of the vendored repo in each downstream repo, so deduplicate the matrix by repo. In cases where there are multiple copies, we are prioritizing the `nextstrain/shared` remote since that is the newer version. This is prompted by the error in the workflow when avian-flu had two paths to update. <#73 (comment)>

victorlin · 2026-04-15T22:20:19Z

+          shared_matrix=$(gh api -X GET search/code \
+            -f q='org:nextstrain filename:.gitrepo "remote = https://github.com/nextstrain/shared"' \


GitHub API calls are limited to 30 results by default. --paginate + --slurp should work well to get everything. Something like this (copied from LLM output, untested):

shared_matrix=$(gh api --paginate --slurp -X GET search/code \ -f q='org:nextstrain filename:.gitrepo "remote = https://github.com/nextstrain/shared"' \ | jq -c ' [.[].items[] | { "repo": .repository.full_name, "path": (.path | split("/")[0:-1] | join("/")) }] ')

Same goes for the other API call.

victorlin · 2026-04-15T22:28:33Z

+          if [[ "$changes" == "1" ]]; then
+            git push --force origin HEAD


How about --force-with-lease? This would allow us to make edits directly to the PR safely without risk of being overwritten by bot force-push - an alternative to the flow of creating a new PR entirely e.g. nextstrain/rubella#54

👍 I exclusively use --force-with-lease now for this reason

victorlin · 2026-04-15T22:29:25Z

+          repository: ${{ matrix.repo }}
+          token: ${{ secrets.GH_TOKEN_NEXTSTRAIN_BOT_REPO }}
+        # Checkout git-subrepo _after_ the downstream repo to ensure that we
+        # keep it in a path within the downstream repo that does not interefere


Suggested change

# keep it in a path within the downstream repo that does not interefere

# keep it in a path within the downstream repo that does not interfere

victorlin · 2026-04-15T22:29:50Z

+          GH_TOKEN: ${{ secrets.GH_TOKEN_NEXTSTRAIN_BOT_REPO }}
+          title: '[bot] Update ${{ env.VENDORED_PATH }}'
+          body: |
+            This PR was automaticaly created by http://github.com/nextstrain/shared/actions/runs/${{ github.run_id }}


Suggested change

This PR was automaticaly created by http://github.com/nextstrain/shared/actions/runs/${{ github.run_id }}

This PR was automatically created by https://github.com/nextstrain/shared/actions/runs/${{ github.run_id }}

jameshadfield reviewed Apr 13, 2026

View reviewed changes

victorlin reviewed Apr 14, 2026

View reviewed changes

joverlee521 changed the title ~~Add GH Action workflow to update vendored repos~~ Add GH Action workflow to update downstream repos Apr 14, 2026

joverlee521 force-pushed the update-vendored branch 3 times, most recently from ca62f4a to e3df43a Compare April 14, 2026 22:12

.github/update-downstream-repos: Include old nextstrain/ingest

ada3160

I was unable to get the search/code API to work with the 'OR' syntax so just added a separate query for nextstrain/ingest and concatenated the two arrays. Deduplicated the final array to guard against potential overlap.

joverlee521 force-pushed the update-vendored branch from e3df43a to ada3160 Compare April 14, 2026 22:15

victorlin reviewed Apr 15, 2026

View reviewed changes

	> Warning
	> Beware of rebasing/dropping the parent commit of a `git subrepo` update

	`git subrepo` relies on metadata in the `shared/vendored/.gitrepo` file,
	which includes the hash for the parent commit in the pathogen repos.
	If this hash no longer exists in the commit history, there will be errors when
	running future `git subrepo pull` commands.

		matrix=$(gh api -X GET search/code \
		-f q='org:nextstrain filename:.gitrepo "remote = https://github.com/nextstrain/shared"' \

		shared_matrix=$(gh api -X GET search/code \
		-f q='org:nextstrain filename:.gitrepo "remote = https://github.com/nextstrain/shared"' \

		if [[ "$changes" == "1" ]]; then
		git push --force origin HEAD

	# keep it in a path within the downstream repo that does not interefere
	# keep it in a path within the downstream repo that does not interfere

	This PR was automaticaly created by http://github.com/nextstrain/shared/actions/runs/${{ github.run_id }}
	This PR was automatically created by https://github.com/nextstrain/shared/actions/runs/${{ github.run_id }}

Conversation

joverlee521 commented Apr 13, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description of proposed changes

Related issue(s)

Checklist

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

joverlee521 commented Apr 15, 2026

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

joverlee521 commented Apr 13, 2026 •

edited

Loading