Is your feature request related to a problem? Please describe.
Users who operate within their own AWS environments would like a straightforward way to contribute compute resources to MAAP Data Processing System from an external AWS account. While there may be existing mechanisms to connect external compute, the process is not obvious or well-documented, making it difficult for teams with dedicated cloud infrastructure, organizational billing requirements, or compliance constraints to leverage their own AWS accounts for running worker nodes. Running worker nodes in a separate AWS account allows teams to track and attribute compute costs independently from MAAP's own infrastructure, providing clearer visibility into project-specific spending and enabling more accurate cost reporting and budgeting.
Describe the solution you'd like
Allow users to provision and run DPS worker nodes from an external AWS account (separate from the MAAP account). These worker nodes should be configurable to listen for and process jobs from a specific project queue. This would involve:
- Worker nodes in the external account polling the queue and executing jobs as they would if running within the MAAP account
- Clear documentation on how to configure and run worker nodes in the external account and how to update any MAAP configuration needed to enable this
Describe alternatives you've considered
N/A
Additional context
N/A
Is your feature request related to a problem? Please describe.
Users who operate within their own AWS environments would like a straightforward way to contribute compute resources to MAAP Data Processing System from an external AWS account. While there may be existing mechanisms to connect external compute, the process is not obvious or well-documented, making it difficult for teams with dedicated cloud infrastructure, organizational billing requirements, or compliance constraints to leverage their own AWS accounts for running worker nodes. Running worker nodes in a separate AWS account allows teams to track and attribute compute costs independently from MAAP's own infrastructure, providing clearer visibility into project-specific spending and enabling more accurate cost reporting and budgeting.
Describe the solution you'd like
Allow users to provision and run DPS worker nodes from an external AWS account (separate from the MAAP account). These worker nodes should be configurable to listen for and process jobs from a specific project queue. This would involve:
Describe alternatives you've considered
N/A
Additional context
N/A