Skip to content

Fix bug with ALiBi in Bloom Architecture Adapter#1213

Merged
jlarson4 merged 1 commit intodev-3.x-canaryfrom
weight-processing/bloom-bug
Mar 25, 2026
Merged

Fix bug with ALiBi in Bloom Architecture Adapter#1213
jlarson4 merged 1 commit intodev-3.x-canaryfrom
weight-processing/bloom-bug

Conversation

@jlarson4
Copy link
Collaborator

Description

The Bloom Architecture Adapter did not properly split joint QKV, and did not properly handle Bloom's ALiBi attention. This resolves those issues.

Type of change

Please delete options that are not relevant.

  • Bug fix (non-breaking change which fixes an issue)

Checklist:

  • I have commented my code, particularly in hard-to-understand areas
  • I have made corresponding changes to the documentation
  • My changes generate no new warnings
  • I have added tests that prove my fix is effective or that my feature works
  • New and existing unit tests pass locally with my changes
  • I have not rewritten tests relating to key interfaces which would affect backward compatibility

@jlarson4 jlarson4 merged commit 535d2e3 into dev-3.x-canary Mar 25, 2026
35 of 36 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant