Bind mounts using Windows and Docker Desktop #1
Description
I tried setting up a demo and experimentation environment for ClickStack starting with the documentation [0] using the single image distribution and loading the data as described in the sample data section [1]. I eventually moved to the docker compose setup in an attempt to figure out what was going on [2].
My setup is Windows with Docker for windows installed. After doing the setup and running the import it seems that the OTel collector will accept all the data and return a 200 on all the requests. The data would never appear in HyperDX though, leading me to think there was an issue with the OTel collector.
Turns out this was a red herring and the database couldn't write to the bound volumes. There were a few message like this one;
ch-server-1 | 2026.01.27 05:39:22.744026 [ 724 ] {} <Error> void DB::SystemLog<DB::AsynchronousMetricLogElement>::flushImpl(const std::vector<LogElement> &, uint64_t) [LogElement = DB::AsynchronousMetricLogElement]: Failed to flush system log system.asynchronous_metric_log with 2912 entries up to offset 2912: std::exception. Code: 1001, type: std::__1::filesystem::filesystem_error, e.what() = filesystem error: in rename: Permission denied ["/var/lib/clickhouse/store/d01/d01be757-5763-4d27-bb4c-2dcb98d86347/tmp_insert_202601_1_1_0/"] ["/var/lib/clickhouse/store/d01/d01be757-5763-4d27-bb4c-2dcb98d86347/202601_1_1_0/"], Stack trace (when copying this message, always include the lines below):
ch-server-1 |
ch-server-1 | 0. std::system_error::system_error(std::error_code, String const&) @ 0x000000001b6225f7
ch-server-1 | 1. std::filesystem::filesystem_error::filesystem_error[abi:ne190107](String const&, std::filesystem::path const&, std::filesystem::path const&, std::error_code) @ 0x000000001b5da520
ch-server-1 | 2. void std::filesystem::__throw_filesystem_error[abi:ne190107]<String&, std::filesystem::path const&, std::filesystem::path const&, std::error_code const&>(String&, std::filesystem::path const&, std::filesystem::path const&, std::error_code const&) @ 0x000000001b5da234
ch-server-1 | 3. std::filesystem::detail::ErrorHandler<void>::report(std::error_code const&) const @ 0x000000001b5d881a
ch-server-1 | 4. std::filesystem::__rename(std::filesystem::path const&, std::filesystem::path const&, std::error_code*) @ 0x000000001b5df75d
ch-server-1 | 5. DB::DiskLocal::moveDirectory(String const&, String const&) @ 0x0000000013338f1b
ch-server-1 | 6. DB::DataPartStorageOnDiskBase::rename(String, String, std::shared_ptr<Poco::Logger>, bool, bool) @ 0x0000000014dabc82
ch-server-1 | 7. DB::IMergeTreeDataPart::renameTo(String const&, bool) @ 0x0000000014de7d1b
ch-server-1 | 8. DB::MergeTreeData::preparePartForCommit(std::shared_ptr<DB::IMergeTreeDataPart>&, DB::MergeTreeData::Transaction&, bool, bool) @ 0x0000000014ecf8a2
ch-server-1 | 9. DB::MergeTreeData::renameTempPartAndReplaceImpl(std::shared_ptr<DB::IMergeTreeDataPart>&, DB::MergeTreeData::Transaction&, DB::DataPartsLock&, std::vector<std::shared_ptr<DB::IMergeTreeDataPart const>, std::allocator<std::shared_ptr<DB::IMergeTreeDataPart const>>>*, bool) @ 0x0000000014f08997
ch-server-1 | 10. DB::MergeTreeData::renameTempPartAndAdd(std::shared_ptr<DB::IMergeTreeDataPart>&, DB::MergeTreeData::Transaction&, DB::DataPartsLock&, bool) @ 0x0000000014f09a28
ch-server-1 | 11. DB::MergeTreeSink::finishDelayedChunk() @ 0x00000000152d7e57
ch-server-1 | 12. DB::runStep(std::function<void ()>, std::shared_ptr<DB::ThreadGroup>&) @ 0x0000000015927503
ch-server-1 | 13. DB::ExceptionKeepingTransform::work() @ 0x0000000015926f5f
ch-server-1 | 14. DB::ExecutionThreadContext::executeTask() @ 0x000000001568ad62
ch-server-1 | 15. DB::PipelineExecutor::executeStepImpl(unsigned long, std::atomic<bool>*) @ 0x000000001567e865
ch-server-1 | 16. DB::PipelineExecutor::executeStep(std::atomic<bool>*) @ 0x000000001567dc32
ch-server-1 | 17. DB::SystemLog<DB::AsynchronousMetricLogElement>::savingThreadFunction() @ 0x0000000013f87f8c
ch-server-1 | 18. void std::__function::__policy_invoker<void ()>::__call_impl[abi:ne190107]<std::__function::__default_alloc_func<ThreadFromGlobalPoolImpl<true, true>::ThreadFromGlobalPoolImpl<DB::SystemLogBase<DB::AsynchronousMetricLogElement>::startup()::'lambda'()>(DB::SystemLogBase<DB::AsynchronousMetricLogElement>::startup()::'lambda'()&&)::'lambda'(), void ()>>(std::__function::__policy_storage const*) @ 0x000000000fdc9b23
ch-server-1 | 19. ThreadPoolImpl<std::thread>::ThreadFromThreadPool::worker() @ 0x000000000fd6c752
ch-server-1 | 20. void* std::__thread_proxy[abi:ne190107]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void (ThreadPoolImpl<std::thread>::ThreadFromThreadPool::*)(), ThreadPoolImpl<std::thread>::ThreadFromThreadPool*>>(void*) @ 0x000000000fd73c1a
ch-server-1 | 21. ? @ 0x0000000000094ac3
ch-server-1 | 22. ? @ 0x00000000001268c0
And turning the bound directories to named volumes in the docker-compose.yml file fixed this issue.
services:
ch-server:
volumes:
- ./docker/clickhouse/local/config.xml:/etc/clickhouse-server/config.xml
- ./docker/clickhouse/local/users.xml:/etc/clickhouse-server/users.xml
# When using WSL creating a volume mapped to the file system causes the database to be unable to write to it
# use some named volumes instead, these can still be persisted for later inspection and all that.
#- .volumes/ch_data:/var/lib/clickhouse
#- .volumes/ch_logs:/var/log/clickhouse-server
- ch_data:/var/lib/clickhouse
- ch_logs:/var/log/clickhouse-server
volumes:
ch_data:
ch_logs:
It took me a while to figure this out as there was quite a bit of noise in the database log and being unfamiliar with the tool I don't know what to look for. I'm assuming this would be the same issue in the single image setup, although I haven't gone through the trouble of validating that fact. This might be worth mentioning in the docker compose deployment documentation and the getting started.
Output from `docker info` for completeness
$ docker info
Client:
Version: 27.2.0
Context: desktop-linux
Debug Mode: false
Plugins:
buildx: Docker Buildx (Docker Inc.)
Version: v0.16.2-desktop.1
Path: C:\Program Files\Docker\cli-plugins\docker-buildx.exe
compose: Docker Compose (Docker Inc.)
Version: v2.29.2-desktop.2
Path: C:\Program Files\Docker\cli-plugins\docker-compose.exe
debug: Get a shell into any image or container (Docker Inc.)
Version: 0.0.34
Path: C:\Program Files\Docker\cli-plugins\docker-debug.exe
desktop: Docker Desktop commands (Alpha) (Docker Inc.)
Version: v0.0.15
Path: C:\Program Files\Docker\cli-plugins\docker-desktop.exe
dev: Docker Dev Environments (Docker Inc.)
Version: v0.1.2
Path: C:\Program Files\Docker\cli-plugins\docker-dev.exe
extension: Manages Docker extensions (Docker Inc.)
Version: v0.2.25
Path: C:\Program Files\Docker\cli-plugins\docker-extension.exe
feedback: Provide feedback, right in your terminal! (Docker Inc.)
Version: v1.0.5
Path: C:\Program Files\Docker\cli-plugins\docker-feedback.exe
init: Creates Docker-related starter files for your project (Docker Inc.)
Version: v1.3.0
Path: C:\Program Files\Docker\cli-plugins\docker-init.exe
sbom: View the packaged-based Software Bill Of Materials (SBOM) for an image (Anchore Inc.)
Version: 0.6.0
Path: C:\Program Files\Docker\cli-plugins\docker-sbom.exe
scout: Docker Scout (Docker Inc.)
Version: v1.13.0
Path: C:\Program Files\Docker\cli-plugins\docker-scout.exe
Server:
Containers: 8
Running: 4
Paused: 0
Stopped: 4
Images: 26
Server Version: 27.2.0
Storage Driver: overlay2
Backing Filesystem: extfs
Supports d_type: true
Using metacopy: false
Native Overlay Diff: true
userxattr: false
Logging Driver: json-file
Cgroup Driver: cgroupfs
Cgroup Version: 2
Plugins:
Volume: local
Network: bridge host ipvlan macvlan null overlay
Log: awslogs fluentd gcplogs gelf journald json-file local splunk syslog
Swarm: inactive
Runtimes: io.containerd.runc.v2 nvidia runc
Default Runtime: runc
Init Binary: docker-init
containerd version: 8fc6bcff51318944179630522a095cc9dbf9f353
runc version: v1.1.13-0-g58aa920
init version: de40ad0
Security Options:
seccomp
Profile: unconfined
cgroupns
Kernel Version: 6.6.87.2-microsoft-standard-WSL2
Operating System: Docker Desktop
OSType: linux
Architecture: x86_64
CPUs: 22
Total Memory: 15.35GiB
Name: docker-desktop
ID: 5f4801ad-2c0d-4367-ae2e-abcc2ace2de9
Docker Root Dir: /var/lib/docker
Debug Mode: false
HTTP Proxy: http.docker.internal:3128
HTTPS Proxy: http.docker.internal:3128
No Proxy: hubproxy.docker.internal
Labels:
com.docker.desktop.address=npipe://\\.\pipe\docker_cli
Experimental: false
Insecure Registries:
hubproxy.docker.internal:5555
127.0.0.0/8
Live Restore Enabled: false
WARNING: daemon is not using the default seccomp profile