Skip to content
Open
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
126 changes: 126 additions & 0 deletions .github/workflows/update-downstream-repos.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,126 @@
name: Update downstream repos

on:
push:
branches:
- main
workflow_dispatch:

concurrency:
group: ${{ github.workflow }}
cancel-in-progress: true

jobs:
build-downstream-matrix:
runs-on: ubuntu-latest
outputs:
downstream-matrix: ${{ steps.downstream-matrix.outputs.downstream-matrix }}
steps:
- id: downstream-matrix
env:
GH_TOKEN: ${{ secrets.GH_TOKEN_NEXTSTRAIN_BOT_REPO }}
# Create an array of potential repos to update that will be used as the
# matrix to the next job.
# [
# { "repo": "nextstrain/zika", "path": "shared/vendored"},
# { "repo": "nextstrain/mpox", "path": "shared/vendored"},
# ...
# ]
run: |
shared_matrix=$(gh api -X GET search/code \
-f q='org:nextstrain filename:.gitrepo "remote = https://github.com/nextstrain/shared"' \
Comment on lines +30 to +31
Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

GitHub API calls are limited to 30 results by default. --paginate + --slurp should work well to get everything. Something like this (copied from LLM output, untested):

shared_matrix=$(gh api --paginate --slurp -X GET search/code \
  -f q='org:nextstrain filename:.gitrepo "remote = https://github.com/nextstrain/shared"' \
  | jq -c '
      [.[].items[] | {
        "repo": .repository.full_name,
        "path": (.path | split("/")[0:-1] | join("/"))
      }]
    ')

Same goes for the other API call.

| jq -c '
.items
| map({
"repo": "\(.repository.full_name)",
"path": "\(.path | split("/")[0:-1] | join("/"))"
})
')

# I was unable to get the 'OR' syntax to work with the search/code API,
# so making a separate query for the old nextstrain/ingest repo name.
# -Jover, 14 Apr 2026.
ingest_matrix=$(gh api -X GET search/code \
-f q='org:nextstrain filename:.gitrepo "remote = https://github.com/nextstrain/ingest"' \
| jq -c '
.items
| map({
"repo": "\(.repository.full_name)",
"path": "\(.path | split("/")[0:-1] | join("/"))"
})
')

# Deduplicate by repo since each repo should only have a single copy
# of the vendored repo. In cases where a repo has both,
# we are prioritizing the nextstrain/shared remote since that is
# the newer repo.
# -Jover, 15 Apr 2026.
matrix=$(jq -n \
--argjson matrix1 "$shared_matrix" \
--argjson matrix2 "$ingest_matrix" \
-c '$matrix1 + $matrix2 | unique_by(.repo)')

echo "downstream-matrix=$matrix" | tee -a "$GITHUB_OUTPUT"
update-downstream:
name: update-downstream (${{ matrix.repo }}, ${{ matrix.path }})
needs: [build-downstream-matrix]
strategy:
fail-fast: false
matrix:
include: ${{ fromJson(needs.build-downstream-matrix.outputs.downstream-matrix) }}
env:
GIT_SUBREPO_DIR: .git/git-subrepo
VENDORED_PATH: ${{ matrix.path }}
branch: nextstrain-bot/update-vendored
runs-on: ubuntu-latest
steps:
- name: Checkout ${{ matrix.repo }}
uses: actions/checkout@v6
with:
repository: ${{ matrix.repo }}
token: ${{ secrets.GH_TOKEN_NEXTSTRAIN_BOT_REPO }}
# Checkout git-subrepo _after_ the downstream repo to ensure that we
# keep it in a path within the downstream repo that does not interefere
Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
# keep it in a path within the downstream repo that does not interefere
# keep it in a path within the downstream repo that does not interfere

# with the subrepo changes
- name: Checkout git-subrepo
uses: actions/checkout@v6
with:
repository: "ingydotnet/git-subrepo"
path: ${{ env.GIT_SUBREPO_DIR }}
- name: Add git-subrepo to PATH
run: echo "$GIT_SUBREPO_DIR/lib" >> "$GITHUB_PATH"
- name: Update vendored path
run: |
git config user.name "${{ vars.GIT_USER_NAME_NEXTSTRAIN_BOT }}"
git config user.email "${{ vars.GIT_USER_EMAIL_NEXTSTRAIN_BOT }}"

git switch -c "$branch"
git subrepo pull "$VENDORED_PATH" --force
- name: Create pull request
env:
GH_TOKEN: ${{ secrets.GH_TOKEN_NEXTSTRAIN_BOT_REPO }}
title: '[bot] Update ${{ env.VENDORED_PATH }}'
body: |
This PR was automaticaly created by http://github.com/nextstrain/shared/actions/runs/${{ github.run_id }}
Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
This PR was automaticaly created by http://github.com/nextstrain/shared/actions/runs/${{ github.run_id }}
This PR was automatically created by https://github.com/nextstrain/shared/actions/runs/${{ github.run_id }}

to update the vendored subrepo in ${{ env.VENDORED_PATH }}.

Subrepo updates were made with the `--force` flag so it overwrites any local changes in the subrepo.
run: |
default_branch=$(git remote show origin | sed -n '/HEAD branch/s/.*: //p')
changes=$(git rev-list --count "$default_branch".."$branch")
if [[ "$changes" == "1" ]]; then
git push --force origin HEAD
Comment on lines +111 to +112
Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

How about --force-with-lease? This would allow us to make edits directly to the PR safely without risk of being overwritten by bot force-push - an alternative to the flow of creating a new PR entirely e.g. nextstrain/rubella#54

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

👍 I exclusively use --force-with-lease now for this reason

pr_url=$(gh pr list --head "$branch" --json url | jq -r '.[0].url')

if [[ "$pr_url" == "null" ]]; then
pr_url="$(gh pr create --head "$branch" --title "$title" --body "$body")"
echo "Pull request created: $pr_url" >> "$GITHUB_STEP_SUMMARY"
else
echo "Pull request updated: $pr_url" >> "$GITHUB_STEP_SUMMARY"
fi
elif [[ "$changes" == "0" ]]; then
echo "No pull request created or updated because no changes were made" >> "$GITHUB_STEP_SUMMARY"
else
echo "ERROR: Encountered an unexpected number of changes: $changes"
exit 1
fi
Loading