When using batch_solr_request and download=True, very large requests (e.g., all of information in the 'statistical-result' core) can take very long to download and write.
The time it takes to write to disk could be improved by adding an option to compress the file. Perhaps adding a new parameter compression=True and compressing with gzip.
Another option would be to add parquet as one of the formats and write lighter files.
When using
batch_solr_requestanddownload=True, very large requests (e.g., all of information in the 'statistical-result' core) can take very long to download and write.The time it takes to write to disk could be improved by adding an option to compress the file. Perhaps adding a new parameter
compression=Trueand compressing with gzip.Another option would be to add
parquetas one of the formats and write lighter files.