feat: tiles deletion creation (MAPCO-5735) by razbroc · Pull Request #95 · MapColonies/overseer

razbroc · 2026-05-07T11:35:54Z

Question	Answer
Bug fix	✖
New feature	✔
Breaking change	✖
Deprecations	✖
Documentation	✖
Tests added	✖
Chore	✖

Related issues: #MAPCO-5735

This pull request introduces several enhancements and refactors to support a new report storage provider, improve S3 integration, and add new ingestion task types. The most significant changes include configuration and infrastructure updates for report storage, the addition of a new tile deletion ingestion task, and improvements to error handling and external service integration.

Storage Provider and S3 Integration:

Added support for a reportStorageProvider option in configuration files (config/default.json, config/custom-environment-variables.json) and Helm charts (helm/templates/configmap.yaml, helm/values.yaml) to allow the selection of storage backends for reports, similar to existing GPKG and tile storage providers. [1] [2] [3] [4]
Updated Helm configuration logic to ensure S3-related environment variables are set if either the GPKG or report storage providers use S3.
Refactored the StorageProvider type in src/common/constants.ts to use values from @map-colonies/raster-shared, ensuring consistency across the codebase. [1] [2] [3]

Ingestion Task Enhancements:

Introduced a new tilesDeletion ingestion task type, including configuration options (type, tileBatchSize, taskBatchSize) and corresponding TypeScript interfaces. [1] [2] [3]
Updated the ingestion job handler to build and push tile deletion tasks using the new TileDeletionTaskManager, ensuring deletion tasks are processed before merge tasks. [1] [2] [3]

Dependency and Infrastructure Updates:

Bumped the @map-colonies/raster-shared package to version 8.1.0-alpha.3, enabling new features and types.
Added the unzip utility to the Docker image to support expanded data handling requirements.
Ensured S3 configuration is injected for both ingestion and export service instances in the dependency container setup.

Error Handling and External Service Integration:

Added a new IntersectionError class for improved error reporting when handling intersection operations with the polygon parts manager.
Extended the PolygonPartsMangerClient to support intersection requests, including error handling and logging for intersection failures. [1] [2]

…gurations Co-authored-by: Copilot <copilot@github.com>

Co-authored-by: Copilot <copilot@github.com>

…ion in task building Co-authored-by: Copilot <copilot@github.com>

…e for downloading reports Co-authored-by: Copilot <copilot@github.com>

… refactor related interfaces and task parameters Co-authored-by: Copilot <copilot@github.com>

…efining deletion task creation Co-authored-by: Copilot <copilot@github.com>

Co-authored-by: Copilot <copilot@github.com>

…and deletion task manager Co-authored-by: Copilot <copilot@github.com>

…URL handling - Removed S3Service dependency from TileDeletionTaskManager and updated report handling to use a URL instead of a path. - Modified readConflictFeatures function to download ZIP reports from a URL and extract shapefiles. - Enhanced logging messages for clarity in TileMergeTaskManager. - Cleaned up FSService constructor and type annotations for better readability. Co-authored-by: Copilot <copilot@github.com>

…les-deletion-skeleton

CL-SHLOMIKONCHA · 2026-05-11T13:16:55Z

  TELEMETRY_METRICS_BUCKETS: {{ $metrics.buckets | toJson | quote }}
  {{ end }}
-  {{- if eq (upper $storage.gpkgProvider) "S3" }}
+  {{- if (eq (upper $storage.gpkgProvider) "S3") }}


CL-SHLOMIKONCHA · 2026-05-11T13:18:29Z

+      type: ""
+    finalize:
+      type: ""
+    merge:


please add "tilesDeletion" task types values

CL-SHLOMIKONCHA · 2026-05-11T13:23:15Z

+} as const;
+/* eslint-enable @typescript-eslint/naming-convention */
+
+export type StorageProvider = (typeof StorageProvider)[keyof typeof StorageProvider];


export type StorageProvider = Omit<typeof SourceType, 'GPKG'>

CL-SHLOMIKONCHA · 2026-05-11T13:35:07Z


+  public async getIntersection(polygonPartsEntityName: string, payload: IntersectionFeatureCollection): Promise<IntersectedFeatureCollection> {
+    try {
+      this.logger.info({ msg: 'getIntersection', polygonPartsEntityName });


please change to debug log level and print the union geometry (fc)

CL-SHLOMIKONCHA · 2026-05-11T13:43:31Z

+        const { polygonPartsEntityName } = this.validateAndGenerateLayerNameFormats(job);
+        const layerRelativePath = taskBuildParams.taskMetadata.layerRelativePath;
+
+        const deletionTasks = this.tileDeletionTaskManager.buildTasks(


for the buildTasks - define a dedicated type like we did on merge task type and reuse it

CL-SHLOMIKONCHA · 2026-05-11T15:40:56Z

+    const span = createChildSpan(`${TileDeletionTaskManager.name}.createTasksForPart.zoom.${zoom}`, parentSpan);
+
+    try {
+      const part = turfFeature(geometry);


CL-SHLOMIKONCHA · 2026-05-11T15:47:34Z

        const fileStream = createReadStream(filePath);

-        fileStream.on('data', (data) => {
+        fileStream.on('data', (data: string | Uint8Array<ArrayBuffer>) => {


check again

CL-SHLOMIKONCHA · 2026-05-11T15:48:24Z

    "@map-colonies/mc-priority-queue": "^9.1.0",
    "@map-colonies/mc-utils": "^5.1.0",
-    "@map-colonies/raster-shared": "^7.10.2",
+    "@map-colonies/raster-shared": "^8.1.0-alpha.3",


should be replaced by the alpha version whenever realeased

CL-SHLOMIKONCHA · 2026-05-11T15:48:56Z

CL-SHLOMIKONCHA · 2026-05-11T15:49:09Z

check if needed - if not remove

razbroc and others added 10 commits April 26, 2026 18:07

feat: basic development

a1b5529

feat: implement tiles deletion functionality and update related confi…

ac9efa7

…gurations Co-authored-by: Copilot <copilot@github.com>

feat: enhance conflict feature extraction by unzipping shapefile reports

9f85918

Co-authored-by: Copilot <copilot@github.com>

feat: enhance tile deletion task manager to include ingestion resolut…

b82c9df

…ion in task building Co-authored-by: Copilot <copilot@github.com>

feat: add report storage provider configuration and enhance S3 servic…

4de04de

…e for downloading reports Co-authored-by: Copilot <copilot@github.com>

feat: update @map-colonies/raster-shared to version 8.1.0-alpha.3 and…

3b687a5

… refactor related interfaces and task parameters Co-authored-by: Copilot <copilot@github.com>

feat: enhance conflict feature processing by uniting geometries and r…

124411f

…efining deletion task creation Co-authored-by: Copilot <copilot@github.com>

style: lint

e9b4080

feat: refactor conflict feature reading into a separate utility function

79d08da

Co-authored-by: Copilot <copilot@github.com>

feat: add TileDeletionTaskManager mock to update job handler test setup

09e08fa

mapcolonies-devops assigned razbroc May 7, 2026

razbroc and others added 6 commits May 7, 2026 14:37

style: lint

71192e3

chore: update tests configuration

5e044f4

Co-authored-by: Copilot <copilot@github.com>

refactor: remove reportStorageProvider references from configuration …

0dbb490

…and deletion task manager Co-authored-by: Copilot <copilot@github.com>

Merge branch 'master' of github.com:MapColonies/overseer into feat/ti…

8406e05

…les-deletion-skeleton

style: lint

d88cc20

CL-SHLOMIKONCHA changed the title ~~feat: tiles deletion creation MAPCO-5735~~ feat: tiles deletion creation (MAPCO-5735) May 11, 2026

CL-SHLOMIKONCHA requested changes May 11, 2026

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat: tiles deletion creation (MAPCO-5735)#95

feat: tiles deletion creation (MAPCO-5735)#95
razbroc wants to merge 16 commits into
masterfrom
feat/tiles-deletion-skeleton

razbroc commented May 7, 2026

Uh oh!

CL-SHLOMIKONCHA May 11, 2026

Uh oh!

CL-SHLOMIKONCHA May 11, 2026

Uh oh!

CL-SHLOMIKONCHA May 11, 2026

Uh oh!

CL-SHLOMIKONCHA May 11, 2026

Uh oh!

CL-SHLOMIKONCHA May 11, 2026

Uh oh!

CL-SHLOMIKONCHA May 11, 2026

Uh oh!

CL-SHLOMIKONCHA May 11, 2026

Uh oh!

CL-SHLOMIKONCHA May 11, 2026

Uh oh!

CL-SHLOMIKONCHA May 11, 2026

Uh oh!

CL-SHLOMIKONCHA May 11, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Conversation

razbroc commented May 7, 2026

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants