Hello!
I've run into an odd little bug via ActivitySim in a use case where I'm trying to use Sharrow (via activitysim) to write input skims as zarr that I can then modify in place and re-use for later runs. Running in a container I'd get logs saying
writing zarr skims to .../skims.zarr
and then die immediately afterward. After digging in some more, the failure only cropped up once Sharrow started writing the 3d arrays in the skims. Here's a simple script that reproduces the error on a small omx file:
import os
import tempfile
import numpy as np
import openmatrix as omx
import sharrow as sh
from tables.exceptions import ClosedNodeError
TIME_PERIODS = ["EA", "AM", "MD", "PM", "EV"]
VAR_NAME = "DRV_COM_WLK_BOARDS"
work = tempfile.mkdtemp(prefix="tiny_omx_repro_")
omx_path = os.path.join(work, "tiny.omx")
zarr_path = os.path.join(work, "lazy-closed-node.zarr")
f = omx.open_file(omx_path, "w")
n = 32
base = np.arange(n * n, dtype=np.float32).reshape(n, n)
f["DIST"] = base
for i, tp in enumerate(TIME_PERIODS):
f[f"{VAR_NAME}__{tp}"] = base + i
f.create_mapping("zone_number", np.arange(1, n + 1, dtype=np.int32))
f.close()
ds = sh.dataset.from_omx_3d(
omx_path,
indexes="one-based",
time_periods=TIME_PERIODS,
)
ds[[VAR_NAME]].to_zarr(zarr_path, mode="w")which gives
Traceback (most recent call last):
File "/Users/zaneedell/git/sharrow/repro.py", line 30, in <module>
ds[[VAR_NAME]].to_zarr(zarr_path, mode="w")
File "sharrow/.venv/lib/python3.12/site-packages/xarray/core/dataset.py", line 2371, in to_zarr
return to_zarr( # type: ignore[call-overload,misc]
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "sharrow/.venv/lib/python3.12/site-packages/xarray/backends/writers.py", line 798, in to_zarr
writes = writer.sync(
^^^^^^^^^^^^
File "sharrow/.venv/lib/python3.12/site-packages/xarray/backends/common.py", line 416, in sync
delayed_store = chunkmanager.store(
^^^^^^^^^^^^^^^^^^^
File "sharrow/.venv/lib/python3.12/site-packages/xarray/namedarray/daskmanager.py", line 247, in store
return store(
^^^^^^
File "sharrow/.venv/lib/python3.12/site-packages/dask/array/core.py", line 1221, in store
dask.compute(arrays, **kwargs)
File "sharrow/.venv/lib/python3.12/site-packages/dask/base.py", line 681, in compute
results = schedule(expr, keys, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "sharrow/.venv/lib/python3.12/site-packages/tables/array.py", line 660, in __getitem__
self._g_check_open()
File "sharrow/.venv/lib/python3.12/site-packages/tables/node.py", line 346, in _g_check_open
raise ClosedNodeError("the node object is closed")
tables.exceptions.ClosedNodeError: the node object is closed
Is this a known issue and/or a non-intended use for from_omx_3d and/or to_zarr? The issue seems to be associated with the lazy Dask view, and I'm happy to poke around and try out a fix if that would be helpful.
Thank you!
Hello!
I've run into an odd little bug via ActivitySim in a use case where I'm trying to use Sharrow (via activitysim) to write input skims as zarr that I can then modify in place and re-use for later runs. Running in a container I'd get logs saying
and then die immediately afterward. After digging in some more, the failure only cropped up once Sharrow started writing the 3d arrays in the skims. Here's a simple script that reproduces the error on a small omx file:
which gives
Is this a known issue and/or a non-intended use for
from_omx_3dand/orto_zarr? The issue seems to be associated with the lazy Dask view, and I'm happy to poke around and try out a fix if that would be helpful.Thank you!