Change in deployment fails due to corrupt JSON in container createOptions due to a fault JSON-merge

[ortogonal]

Expected Behavior

When changing from deployment A to B the edgeAgent try and merge (on JSON-level) the old deployment with the new. The expected behavior is that this merge shall succeed so edgeAgent can configure the modules correctly.

Current Behavior

The JSON-merge fails in special cases when going from a long createOptions to a shorter one.

Steps to Reproduce

Provide a detailed set of steps to reproduce the bug.

1. Create a deployment A with a module with a really long createOptions. It must be long enough so it's divided into createOptions and createOptions01.
2. Install it on a device.
3. Create a deployment B with the same module as in A but make the createOptions shorter so it only get a createOptions and not a createOptions01.
4. Change the deployment from A to B on the device.
5. edgeAgent will halt the update and complain about broken JSON data.

Context (Environment)

Output of iotedge check

Click here

Configuration checks (aziot-identity-service)
---------------------------------------------
√ keyd configuration is well-formed - OK
√ certd configuration is well-formed - OK
√ tpmd configuration is well-formed - OK
√ identityd configuration is well-formed - OK
√ daemon configurations up-to-date with config.toml - OK
√ identityd config toml file specifies a valid hostname - OK
‼ aziot-identity-service package is up-to-date - Warning
    Installed aziot-identity-service package has version 1.4.3 but 1.4.9 is the latest stable version available.
    Please see https://aka.ms/aziot-update-runtime for update instructions.
√ host time is close to reference time - OK
√ preloaded certificates are valid - OK
√ keyd is running - OK
√ certd is running - OK
√ identityd is running - OK
√ read all preloaded certificates from the Certificates Service - OK
√ read all preloaded key pairs from the Keys Service - OK
√ check all EST server URLs utilize HTTPS - OK
√ ensure all preloaded certificates match preloaded private keys with the same ID - OK

Connectivity checks (aziot-identity-service)
--------------------------------------------
√ host can connect to and perform TLS handshake with iothub AMQP port - OK
√ host can connect to and perform TLS handshake with iothub HTTPS / WebSockets port - OK
√ host can connect to and perform TLS handshake with iothub MQTT port - OK

Configuration checks
--------------------
√ aziot-edged configuration is well-formed - OK
√ configuration up-to-date with config.toml - OK
√ container engine is installed and functional - OK
√ configuration has correct URIs for daemon mgmt endpoint - OK
‼ aziot-edge package is up-to-date - Warning
    Installed IoT Edge daemon has version 1.4.9 but 1.4.41 is the latest stable version available.
    Please see https://aka.ms/iotedge-update-runtime for update instructions.
√ container time is close to host time - OK
√ DNS server - OK
√ production readiness: logs policy - OK
√ production readiness: Edge Agent's storage directory is persisted on the host filesystem - OK
√ production readiness: Edge Hub's storage directory is persisted on the host filesystem - OK
√ Agent image is valid and can be pulled from upstream - OK
√ proxy settings are consistent in aziot-edged, aziot-identityd, moby daemon and config.toml - OK

Connectivity checks
-------------------
√ container on the default network can connect to upstream AMQP port - OK
√ container on the default network can connect to upstream HTTPS / WebSockets port - OK
√ container on the IoT Edge module network can connect to upstream AMQP port - OK
√ container on the IoT Edge module network can connect to upstream HTTPS / WebSockets port - OK
33 check(s) succeeded.
2 check(s) raised warnings. Re-run with --verbose for more details.
2 check(s) were skipped due to errors from other checks. Re-run with --verbose for more details.

Device Information

• Host OS [e.g. Ubuntu 22.04, Windows Server IoT 2019]: Custom Yocto dist
• Architecture [e.g. amd64, arm32, arm64]: arm64
• Container OS [e.g. Linux containers, Windows containers]: Linux containers

Runtime Versions

• aziot-edged [run iotedge version]: 1.4.9
• Edge Agent [image tag (e.g. 1.0.0)]: 1.5.31
• Edge Hub [image tag (e.g. 1.0.0)]: 1.5.31
• Docker/Moby [run docker version]: 20.10.25-ce

Logs

aziot-edged logs

<Paste here between the triple backticks>

edge-agent logs

<4> 2025-12-04 15:23:40.269 +00:00 [WRN] - Reconcile failed because of invalid configuration format
Microsoft.Azure.Devices.Edge.Agent.Core.ConfigSources.ConfigFormatException: Agent configuration format is invalid.
 ---> Newtonsoft.Json.JsonReaderException: Error parsing comment. Expected: *, got d. Path '', line 1, position 416.
   at Newtonsoft.Json.JsonTextReader.ParseComment(Boolean setToken)
   at Newtonsoft.Json.JsonTextReader.Read()
   at Newtonsoft.Json.Serialization.JsonSerializerInternalReader.Deserialize(JsonReader reader, Type objectType, Boolean checkAdditionalContent)
   at Newtonsoft.Json.JsonSerializer.DeserializeInternal(JsonReader reader, Type objectType)
   at Newtonsoft.Json.JsonConvert.DeserializeObject(String value, Type type, JsonSerializerSettings settings)
   at Microsoft.Azure.Devices.Edge.Agent.Docker.DockerConfig.GetCreateOptions(String createOptions) in /mnt/vss/_work/1/s/edge-agent/src/Microsoft.Azure.Devices.Edge.Agent.Docker/DockerConfig.cs:line 82
   at Microsoft.Azure.Devices.Edge.Agent.Docker.DockerConfig..ctor(String image, String createOptions, Option`1 digest) in /mnt/vss/_work/1/s/edge-agent/src/Microsoft.Azure.Devices.Edge.Agent.Docker/DockerConfig.cs:line 28
   at Microsoft.Azure.Devices.Edge.Agent.Docker.DockerConfig.DockerConfigJsonConverter.ReadJson(JsonReader reader, Type objectType, Object existingValue, JsonSerializer serializer) in /mnt/vss/_work/1/s/edge-agent/src/Microsoft.Azure.Devices.Edge.Agent.Docker/DockerConfig.cs:line 213
   at Newtonsoft.Json.Serialization.JsonSerializerInternalReader.ResolvePropertyAndCreatorValues(JsonObjectContract contract, JsonProperty containerProperty, JsonReader reader, Type objectType)
   at Newtonsoft.Json.Serialization.JsonSerializerInternalReader.CreateObjectUsingCreatorWithParameters(JsonReader reader, JsonObjectContract contract, JsonProperty containerProperty, ObjectConstructor`1 creator, String id)
   at Newtonsoft.Json.Serialization.JsonSerializerInternalReader.CreateObject(JsonReader reader, Type objectType, JsonContract contract, JsonProperty member, JsonContainerContract containerContract, JsonProperty containerMember, Object existingValue)
   at Newtonsoft.Json.Serialization.JsonSerializerInternalReader.CreateValueInternal(JsonReader reader, Type objectType, JsonContract contract, JsonProperty member, JsonContainerContract containerContract, JsonProperty containerMember, Object existingValue)
   at Newtonsoft.Json.Serialization.JsonSerializerInternalReader.Deserialize(JsonReader reader, Type objectType, Boolean checkAdditionalContent)
   at Newtonsoft.Json.JsonSerializer.DeserializeInternal(JsonReader reader, Type objectType)
   at Newtonsoft.Json.JsonConvert.DeserializeObject(String value, Type type, JsonSerializerSettings settings)
   at Newtonsoft.Json.JsonConvert.DeserializeObject(String value, Type type)
   at Microsoft.Azure.Devices.Edge.Agent.Core.Serde.TypeSpecificSerDe`1.TypeSpecificJsonConverter.ReadJson(JsonReader reader, Type objectType, Object existingValue, JsonSerializer serializer) in /mnt/vss/_work/1/s/edge-agent/src/Microsoft.Azure.Devices.Edge.Agent.Core/serde/TypeSpecificSerDe.cs:line 104
   at Newtonsoft.Json.Serialization.JsonSerializerInternalReader.DeserializeConvertable(JsonConverter converter, JsonReader reader, Type objectType, Object existingValue)
   at Newtonsoft.Json.Serialization.JsonSerializerInternalReader.PopulateDictionary(IDictionary dictionary, JsonReader reader, JsonDictionaryContract contract, JsonProperty containerProperty, String id)
   at Newtonsoft.Json.Serialization.JsonSerializerInternalReader.CreateObject(JsonReader reader, Type objectType, JsonContract contract, JsonProperty member, JsonContainerContract containerContract, JsonProperty containerMember, Object existingValue)
   at Newtonsoft.Json.Serialization.JsonSerializerInternalReader.CreateValueInternal(JsonReader reader, Type objectType, JsonContract contract, JsonProperty member, JsonContainerContract containerContract, JsonProperty containerMember, Object existingValue)
   at Newtonsoft.Json.Serialization.JsonSerializerInternalReader.ResolvePropertyAndCreatorValues(JsonObjectContract contract, JsonProperty containerProperty, JsonReader reader, Type objectType)
   at Newtonsoft.Json.Serialization.JsonSerializerInternalReader.CreateObjectUsingCreatorWithParameters(JsonReader reader, JsonObjectContract contract, JsonProperty containerProperty, ObjectConstructor`1 creator, String id)
   at Newtonsoft.Json.Serialization.JsonSerializerInternalReader.CreateObject(JsonReader reader, Type objectType, JsonContract contract, JsonProperty member, JsonContainerContract containerContract, JsonProperty containerMember, Object existingValue)
   at Newtonsoft.Json.Serialization.JsonSerializerInternalReader.CreateValueInternal(JsonReader reader, Type objectType, JsonContract contract, JsonProperty member, JsonContainerContract containerContract, JsonProperty containerMember, Object existingValue)
   at Newtonsoft.Json.Serialization.JsonSerializerInternalReader.Deserialize(JsonReader reader, Type objectType, Boolean checkAdditionalContent)
   at Newtonsoft.Json.JsonSerializer.DeserializeInternal(JsonReader reader, Type objectType)
   at Newtonsoft.Json.JsonConvert.DeserializeObject(String value, Type type, JsonSerializerSettings settings)
   at Newtonsoft.Json.JsonConvert.DeserializeObject[T](String value, JsonSerializerSettings settings)
   at Microsoft.Azure.Devices.Edge.Agent.Core.Serde.TypeSpecificSerDe`1.Deserialize[TU](String json) in /mnt/vss/_work/1/s/edge-agent/src/Microsoft.Azure.Devices.Edge.Agent.Core/serde/TypeSpecificSerDe.cs:line 63
   at Microsoft.Azure.Devices.Edge.Agent.Core.Serde.TypeSpecificSerDe`1.Deserialize(String json) in /mnt/vss/_work/1/s/edge-agent/src/Microsoft.Azure.Devices.Edge.Agent.Core/serde/TypeSpecificSerDe.cs:line 56
   at Microsoft.Azure.Devices.Edge.Agent.IoTHub.EdgeAgentConnection.UpdateDeploymentConfig(TwinCollection desiredProperties) in /mnt/vss/_work/1/s/edge-agent/src/Microsoft.Azure.Devices.Edge.Agent.IoTHub/EdgeAgentConnection.cs:line 435
   --- End of inner exception stack trace ---
   at Microsoft.Azure.Devices.Edge.Agent.IoTHub.EdgeAgentConnection.UpdateDeploymentConfig(TwinCollection desiredProperties) in /mnt/vss/_work/1/s/edge-agent/src/Microsoft.Azure.Devices.Edge.Agent.IoTHub/EdgeAgentConnection.cs:line 447
   at Microsoft.Azure.Devices.Edge.Agent.IoTHub.EdgeAgentConnection.ApplyPatchAsync(TwinCollection desiredProperties, TwinCollection patch) in /mnt/vss/_work/1/s/edge-agent/src/Microsoft.Azure.Devices.Edge.Agent.IoTHub/EdgeAgentConnection.cs:line 411
   at Microsoft.Azure.Devices.Edge.Agent.Core.Agent.ReconcileAsync(CancellationToken token) in /mnt/vss/_work/1/s/edge-agent/src/Microsoft.Azure.Devices.Edge.Agent.Core/Agent.cs:line 132
   at Microsoft.Azure.Devices.Edge.Agent.Core.Agent.ReconcileAsync(CancellationToken token) in /mnt/vss/_work/1/s/edge-agent/src/Microsoft.Azure.Devices.Edge.Agent.Core/Agent.cs:line 132
   at Microsoft.Azure.Devices.Edge.Agent.Core.Agent.ReconcileAsync(CancellationToken token) in /mnt/vss/_work/1/s/edge-agent/src/Microsoft.Azure.Devices.Edge.Agent.Core/Agent.cs:line 132
   at Microsoft.Azure.Devices.Edge.Agent.Core.Agent.ReconcileAsync(CancellationToken token) in /mnt/vss/_work/1/s/edge-agent/src/Microsoft.Azure.Devices.Edge.Agent.Core/Agent.cs:line 132
   at Microsoft.Azure.Devices.Edge.Agent.Core.Agent.ReconcileAsync(CancellationToken token) in /mnt/vss/_work/1/s/edge-agent/src/Microsoft.Azure.Devices.Edge.Agent.Core/Agent.cs:line 132
   at Microsoft.Azure.Devices.Edge.Agent.Core.Agent.ReconcileAsync(CancellationToken token) in /mnt/vss/_work/1/s/edge-agent/src/Microsoft.Azure.Devices.Edge.Agent.Core/Agent.cs:line 132
   at Microsoft.Azure.Devices.Edge.Agent.Core.Agent.ReconcileAsync(CancellationToken token) in /mnt/vss/_work/1/s/edge-agent/src/Microsoft.Azure.Devices.Edge.Agent.Core/Agent.cs:line 132

edge-hub logs

<Paste here between the triple backticks>

Additional Information

More detailed example.
Deployment A has a module that looks like this. Note that there is a createOptions and a createOptions01.

"mosquitto": {
    "restartPolicy": "always",
    "settings": {
        "createOptions": "{\"HostConfig\":{\"LogConfig\":{\"Type\":\"local\",\"Config\":{\"max-size\":\"6m\",\"max-file\":\"2\"}},\"Privileged\":true,\"ExtraHosts\":[\"host.docker.internal:host-gateway\"],\"ExposedPorts\":{\"8884/tcp\":{}},\"PortBindings\":{\"1883/tcp\":[{\"HostIp\":\"127.0.0.1\",\"HostPort\":1883}],\"8884/tcp\":[{\"HostPort\":\"8884\"}],\"9001/tcp\":[{\"HostPort\":\"9001\"}],\"1888/tcp\":[{\"HostPort\":\"1888\"}]},\"Binds\":[\"/mnt/storage/mosquitto/data:/var/tmp/data\",\"/mnt/storage/mosquitto/log:/var/tmp/log\",\"/mnt/secure/mosquitto/certificates:/var/tmp/certificates:ro\",\"",
        "image": "foo.azurecr.io/aarch64/mosquitto:v1.0.0",
        "createOptions01": "/dev:/dev\"]},\"Cmd\":[\"sh\",\"-c\",\"run-mosquitto.sh\"]}"
    },
    "startupOrder": 100,
    "status": "running",
    "type": "docker"
},

Deployment B has a module that looks like this. Not only a createOptions.

"mosquitto": {
    "restartPolicy": "always",
    "settings": {
        "createOptions": "{\"HostConfig\":{\"LogConfig\":{\"Type\":\"local\",\"Config\":{\"max-size\":\"6m\",\"max-file\":\"2\"}},\"ExposedPorts\":{\"8884/tcp\":{}},\"PortBindings\":{\"1883/tcp\":[{\"HostIp\":\"127.0.0.1\",\"HostPort\":1883}],\"8884/tcp\":[{\"HostPort\":\"8884\"}]},\"Binds\":[\"/mnt/storage/mosquitto/data:/var/tmp/data\",\"/mnt/storage/mosquitto/log:/var/tmp/log\",\"/mnt/secure/mosquitto/certificates:/var/tmp/certificates:ro\"]},\"Cmd\":[\"sh\",\"-c\",\"run-mosquitto.sh\"]}",
        "image": "foo.azurecr.io/aarch64/mosquitto:v1.0.0"
    },
    "startupOrder": 100,
    "status": "running",
    "type": "docker"
},

In EdgeAgentConnections.cs (ApplyPathAsync) a JSON merge between A and B i done.

string mergedJson = JsonEx.Merge(desiredProperties, patch, true);

Sine the class JsonEx doesn't know that createOptions and createOptions01are special (chunked) it will merge it into this:

"mosquitto": {
    "restartPolicy": "always",
    "settings": {
        "createOptions": "{\"HostConfig\":{\"LogConfig\":{\"Type\":\"local\",\"Config\":{\"max-size\":\"6m\",\"max-file\":\"2\"}},\"ExposedPorts\":{\"8884/tcp\":{}},\"PortBindings\":{\"1883/tcp\":[{\"HostIp\":\"127.0.0.1\",\"HostPort\":1883}],\"8884/tcp\":[{\"HostPort\":\"8884\"}]},\"Binds\":[\"/mnt/storage/mosquitto/data:/var/tmp/data\",\"/mnt/storage/mosquitto/log:/var/tmp/log\",\"/mnt/secure/mosquitto/certificates:/var/tmp/certificates:ro\"]},\"Cmd\":[\"sh\",\"-c\",\"run-mosquitto.sh\"]}",
        "image": "foo.azurecr.io/aarch64/mosquitto:v1.0.0",
        "createOptions01": "/dev:/dev\"]},\"Cmd\":[\"sh\",\"-c\",\"run-mosquitto.sh\"]}"
    },
    "startupOrder": 100,
    "status": "running",
    "type": "docker"
},

The result is that createOptions01 is still left in the JSON. When later DockerConfig.cs takes that JSON object to create a docker instance it will see createOptions and createOptions01 and use them as a chunked string. The result in invalid JSON which leads to a halt in the deployment change.

[ortogonal]

I've made a fix for this (#7490).
I did the fix just to proof that my assumptions was correct and also to be able to add test-cases that show the bug more explicitly.