feat:fs s3 integration and tilesDeletionStartegy(MAPCO-9806) by almog8k · Pull Request #24 · MapColonies/cleaner

almog8k · 2026-05-06T08:04:43Z

Question	Answer
New feature	✔

This pull request introduces a complete implementation of a provider-agnostic tiles deletion strategy, supporting both S3 and filesystem backends. It adds new storage provider interfaces and implementations, integrates them into the DI container, and exposes configuration for provider-specific settings. The tiles deletion strategy now efficiently deletes tiles in batches, handles partial failures, and reports progress.

Tiles Deletion Strategy Implementation:

Implemented the TilesDeletionStrategy to support deleting tiles from both S3 and filesystem backends, including batching, concurrency, error handling, and progress reporting. The strategy uses provider abstraction and is fully configurable.

Storage Provider Abstraction:

Added the IStorageProvider interface and concrete implementations: S3StorageProvider (using @aws-sdk/client-s3) and FsStorageProvider (using Node.js file system APIs). These providers handle batch deletions and error handling specific to their storage type. [1] [2] [3] [4]
Registered the storage providers in the DI container and exposed them via the STORAGE_PROVIDERS symbol. [1] [2] [3] [4]

Configuration and Environment:

Added configuration and environment variable support for S3 and tiles deletion strategy parameters (such as bucket, base path, batch size, and concurrency) in config/default.json, config/custom-environment-variables.json, and Helm charts. [1] [2] [3] [4]

Dependency and Schema Updates:

Added dependencies on @aws-sdk/client-s3 and updated to use the tiles deletion schema and types from @map-colonies/raster-shared. [1] [2]

- Added "@aws-sdk/client-s3" dependency to package.json for S3 interactions. - Introduced STORAGE_PROVIDERS constant in constants.ts for future storage provider implementations.

Co-authored-by: Copilot <copilot@github.com>

…ogress tracking

…ories

…pe imports Co-authored-by: Copilot <copilot@github.com>

… strategy Co-authored-by: Copilot <copilot@github.com>

github-actions · 2026-05-06T08:06:42Z

🎫 Related Jira Issue: MAPCO-9806

Co-authored-by: Copilot <copilot@github.com>

- upgraded @map-colonies/mc-priority-queue from ^9.0.2 to ^9.1.0 - upgraded @map-colonies/raster-shared from ^8.1.0-alpha.2 to ^8.1.0-alpha.3

CL-SHLOMIKONCHA · 2026-05-07T10:41:40Z

    "delay": "exponential",
    "shouldResetTimeout": true
+  },
+  "s3": {


check if Storage Class is needed

We don't need to specify the Storage Class when deleting an object because S3 only needs the object's exact "address" to find and destroy it.

CL-SHLOMIKONCHA · 2026-05-07T11:00:06Z

+  private async cleanupEmptyDirs(relativePaths: string[], storageTarget: string): Promise<void> {
+    const dirsByLevel = new Map<number, Set<string>>();
+
+    for (const relativePath of relativePaths) {


try to add more comment to explain what going on here

Added comments.

CL-SHLOMIKONCHA · 2026-05-07T11:17:10Z

+  public constructor(
+    @inject(SERVICES.LOGGER) private readonly logger: Logger,
+    @inject(SERVICES.CONFIG) config: ConfigType,
+    @inject(SERVICES.STORAGE_PROVIDERS) private readonly storageProviders: Map<string, IStorageProvider>,


check if changed string to SourceType

Used(SourceType)

CL-SHLOMIKONCHA · 2026-05-07T11:20:31Z

+    if (provider === undefined) {
+      throw new UnrecoverableError(`Unknown storage provider: ${params.sourceProvider}`);
+    }
+    const storageTarget = params.sourceProvider === 'S3' ? this.s3Bucket : this.fsBasePath;


use SourceType

Used(SourceType)

CL-SHLOMIKONCHA · 2026-05-07T11:22:03Z

+  }
+
+  private countTiles(params: TilesDeletionParams): number {
+    return params.ranges.reduce((sum, r) => sum + (r.maxX - r.minX + 1) * (r.maxY - r.minY + 1), 0);


add comment to explain this code line

Added comment explaining the countTiles func

CL-SHLOMIKONCHA · 2026-05-07T11:32:43Z

+  }
+
+  private async flushBatches(provider: IStorageProvider, storageTarget: string, pendingBatches: string[][], failedPaths: string[]): Promise<number> {
+    const flushedCount = pendingBatches.reduce((sum, b) => sum + b.length, 0);


add comment to explain

CL-SHLOMIKONCHA · 2026-05-07T11:40:33Z

+
+    if (failedPaths.length > 0) {
+      const sample = failedPaths.slice(0, this.failureSampleSize);
+      this.logger.warn({


change to error log

CL-SHLOMIKONCHA · 2026-05-07T11:44:05Z

+        pendingBatches.push(batch);
+        batch = [];
+        if (pendingBatches.length === this.concurrency) {
+          processedTiles += await this.flushBatches(provider, storageTarget, pendingBatches, failedPaths);


add info log indicate for x/total deleted

CL-SHLOMIKONCHA · 2026-05-07T11:45:41Z

+        useFactory: instancePerContainerCachingFactory((container) => {
+          const config = container.resolve<ConfigType>(SERVICES.CONFIG);
+          const logger = container.resolve<Logger>(SERVICES.LOGGER);
+          return new Map<string, IStorageProvider>([


CL-SHLOMIKONCHA · 2026-05-07T11:46:22Z

    "@godaddy/terminus": "^4.12.1",
    "@map-colonies/config": "^4.0.1",
    "@map-colonies/jobnik-sdk": "^0.1.0",
    "@map-colonies/js-logger": "^3.0.2",


upgrade to 5.0.0

…orkerBoilerplateV2

…viders Co-authored-by: Copilot <copilot@github.com>

…n testing

almog8k and others added 8 commits April 29, 2026 17:49

feat: add AWS SDK S3 client and new storage provider constant

7db52cf

- Added "@aws-sdk/client-s3" dependency to package.json for S3 interactions. - Introduced STORAGE_PROVIDERS constant in constants.ts for future storage provider implementations.

feat: integrate S3 storage provider with configuration updates

c539c90

feat: update tilesDeletion strategy configuration for S3 integration

96412d0

feat: tiles deletion strategy with S3 integration and validation schemas

2a55831

Co-authored-by: Copilot <copilot@github.com>

feat: register task context and update tiles deletion strategy for pr…

0c236f3

…ogress tracking

feat: enhance file deletion process by adding cleanup of empty direct…

de75c99

…ories

feat: improve logging level during S3 deletion and clean up unused ty…

72d3889

…pe imports Co-authored-by: Copilot <copilot@github.com>

feat: integrate @map-colonies/raster-shared and update tiles deletion…

e0a8630

… strategy Co-authored-by: Copilot <copilot@github.com>

mapcolonies-devops assigned almog8k May 6, 2026

almog8k changed the title ~~feat:fs s3 integration and tilesDeletionStartegy~~ feat:fs s3 integration and tilesDeletionStartegy(MAPCO-9806) May 6, 2026

almog8k and others added 5 commits May 6, 2026 11:10

feat: remove unused tiles deletion validation exports

18623b6

feat: update storage provider keys to use SourceType constants

6c4eac0

Co-authored-by: Copilot <copilot@github.com>

fix: correct file extension formatting in generateRangePaths method

8785aa4

fix: replace console.debug with logger.debug in validateSchema function

633bc77

chore: update dependencies for mc-priority-queue and raster-shared

fa25125

- upgraded @map-colonies/mc-priority-queue from ^9.0.2 to ^9.1.0 - upgraded @map-colonies/raster-shared from ^8.1.0-alpha.2 to ^8.1.0-alpha.3

CL-SHLOMIKONCHA requested changes May 7, 2026

View reviewed changes

almog8k and others added 7 commits May 10, 2026 11:23

fix: pr issues

b22f4f8

feat: update js-logger to version 5.0.0 and switch schemas to commonW…

e262f90

…orkerBoilerplateV2

feat: add opentelemetry options to logger configuration in JSON files

ee226de

feat: implement targetExists method for filesystem and S3 storage pro…

ad74141

…viders Co-authored-by: Copilot <copilot@github.com>

helm: deployment adjustments

0f76f65

feat: update simulate-deletion script and Dockerfile for tile deletio…

6714172

…n testing

feat: update @map-colonies/schemas dependency to version 1.20.0

56d18f2

Conversation

almog8k commented May 6, 2026

Uh oh!

github-actions Bot commented May 6, 2026 • edited by atlassian Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

github-actions Bot commented May 6, 2026 •

edited by atlassian Bot

Loading