GNARF: chunk datavalue import, multi text support on dataElements, override rule, fix single row export

[gqcorneby]

📌 References

• Issue: Closes #869c4fuc1 https://app.clickup.com/t/869c4fuc1

📝 Implementation

• Issue 1 - large data payload
  • the bulk load template created for this tasks includes 11,355 dataValues if all fields are populated. This results to the system task eventually returning an empty response.
  • The root cause was that when large payloads were imported asynchronously, the ImportSummary (including errors and counts) was never being persisted to the job's store. So polling taskSummaries returned empty so don't get success or error information (https://community.dhis2.org/t/heavy-server-load-in-dhis2-instance-results-in-no-response-to-users/51763/4)
• Fix: Reduce payload by chunking dataValues
  • Chunk size 5000 still resulted to an empty response for the 1st one, 3000 worked sometimes but still resulted in empty response sometimes, 1000 has been consistently successful.

Issue 2 - multi text

• Add multi-text support to dataElements
• Fix: add similar delimiter handling as TEI attributes. Detect mutli-text and add handling if multiTextDataElementDelimiter is set

Issue 3 - template formulas

• The template has dataElements mapped to cells with formulas. When exported, the actual values overwrite the formulas
• Fix: add override dataProcessing rule to "reroute" the export to a different cell. Original cell mapping for import keeps the sum formula

Issue 4 - single row export

• fillRows iterates dataEntries (1 per orgUnit/period/COC). With 1 entry, only 1 row gets filled, remaining rows in the range are skipped.
• Why it only affects GNARF: it's the only template with rowEnd defined on "row" type datasources. All others either use "cell" datasources or have no rowEnd. - Fix: differentiate handling between templates that expect one dataEntry per row (fillRowsByDataEntry) vs templates with one data entry with fields spread out ( fillRowsByKeyLookup).
• Assumption is that templates with set fixedOrgUnit and fixedPeriod are single entry templates
• refactored lines from fillRows to resolveDataElementValues and resolveDataElementValue to be reused

Issue 5 - Defined-name formula resolution on export

• Template mapping cells store DE/COC IDs as Excel defined-name formulas (e.g. =_). When the template file has cached label text, export read the label instead of the ID, causing zero data to populate
• generated templates are not affected because they wouldn't have cached values.
• Fix: Extract readCellResolvingDefinedNames from ExcelReader to a shared helper that checks if a cell's formula is a defined name and returns the ID, otherwise return value.

This is the template used to test. Update the period or org unit in "Introduction" tab to test

NTD_GNARF_Template_filled.xlsm

🔥 Notes for the reviewer

🎨 Screenshots

• Chunk size 5000. This is what it would look like if there are chunks that have empty results

📑 Others

[bundlemon]

BundleMon

No change in files bundle size

Groups updated (1)

Status	Path	Size	Limits
✅	Build Folder ./**/*	1.72MB (+2.39KB +0.14%)	+20%

Final result: ✅

View report in BundleMon website ➡️

---

Current branch size history | Target branch size history

[Ramon-Jimenez]

@gqcorneby I gave it a try using American Samoa 2023 and I got the 'existing data' pop-up showing that there were 3000 DEs with data already. Just wondering if, in fact, there are 3000 DEs with data, or if there is some additional fix to be done in the 'existing data' part that is making it look like there is only 3000 DEs (because of the chunk size) when there are, in fact, more DEs with data

Also, I got this warnings:

Thanks!

[gqcorneby]

I gave it a try using American Samoa 2023 and I got the 'existing data' pop-up showing that there were 3000 DEs with data already. Just wondering if, in fact, there are 3000 DEs with data, or if there is some additional fix to be done in the 'existing data' part that is making it look like there is only 3000 DEs (because of the chunk size) when there are, in fact, more DEs with data

Hi @Ramon-Jimenez! The chunking should be isolated during saving so it shouldn't affect this part of the code. It seems like the 3000 displayed here is just a coincidence. I tried importing the the filled file again and it showed 11,355

In terms of the warning, these dataElements are indeed integer types but they have an optionSet. All of the Country Progress 3.1 dataElements are integers but some have warnings and some are saved. I'll investigate this further.

https://dev.eyeseetea.com/who-dev-41/api/dataElements?filter=name:ilike:gnarf&filter=name:ilike:3.1&fields=id,name,formName,valueType,optionSet[id,name,options[name]]

[gqcorneby]

Hi @Ramon-Jimenez! I updated the PR description to include all the issues and fixes to make the template work for import and export. I was able to test import, export, edit, then import.

[Ramon-Jimenez]

Thanks @gqcorneby , working great