feat(rust): add streaming deserialization support#3369
Open
Zakir032002 wants to merge 19 commits intoapache:mainfrom
Open
feat(rust): add streaming deserialization support#3369Zakir032002 wants to merge 19 commits intoapache:mainfrom
Zakir032002 wants to merge 19 commits intoapache:mainfrom
Conversation
Author
|
Hey @chaokunyang, Have a look at the implementation and let me know your reviews |
Collaborator
|
Hi @Zakir032002 , could you address the conflict? |
Collaborator
|
Please take #3307 as reference to finish the remaining works. And create a Deseralize help methods in tests, then use that instead of Then run benchmarks/rust to compare with asf/main to enure this pr doesn't introduce any performance regression. |
Introduces ForyStreamBuf and Reader::from_stream for incremental stream-backed deserialization from any Read source. Preserves existing in-memory fast path with zero overhead. Closes apache#3300
Zakir032002
force-pushed
the
feat/rust-streaming-deserialization
branch
from
18de059 to
78a59c7
Compare
Collaborator
|
Hi @Zakir032002 , Are you still working on this PR? |
Author
|
Yes, I have been busy for the last 3-4 days. Now that I am free, I will
complete all my PRs as soon as possible.
…On Thu, Mar 5, 2026 at 6:59 PM Shawn Yang ***@***.***> wrote:
*chaokunyang* left a comment (apache/fory#3369)
<#3369 (comment)>
Hi @Zakir032002 <https://github.com/Zakir032002> , Are you still working
on this PR?
—
Reply to this email directly, view it on GitHub
<#3369 (comment)>, or
unsubscribe
<https://github.com/notifications/unsubscribe-auth/BVRYNH2HPC6HWQXS7MZTSE34PF6KZAVCNFSM6AAAAACVVKAMV6VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHM2DAMBVGA3DKNZTGI>
.
You are receiving this because you were mentioned.Message ID:
***@***.***>
|
Zakir032002
force-pushed
the
feat/rust-streaming-deserialization
branch
from
dc1c898 to
010ae68
Compare
- fix reader attach/detach lifecycle in deserialize_from_stream - isolate stream and in-memory paths in deserialize_from - add struct round-trip test for reference parity with C++/Go - fix clippy lints in stream_test.rs
- implement ForyStreamBuf stream buffer - integrate Reader streaming support - add OneByteStream tests for streaming validation - add deserialize helper parity tests - fix varuint64 streaming decode to match C++ implementation - add sequential stream decoding tests - validate truncated stream error behavior
- add overflow guard in ensure_readable - propagate set_reader_index failure in fill_to - remove dead shrink_buffer call in deserialize_from_stream - clarify varuint64 stream decoding comment
- Add stream_reader_index() and stream_remaining() public accessors to Reader - Fix unused Result warnings in test_fory.rs and test_buffer.rs - Apply rustfmt formatting to test_stream.rs
…alize_check helper Updated 13 test files to use the centralized deserialize_check helper which verifies deserialization through both in-memory and stream-backed paths (including byte-at-a-time OneByte stress testing), asserting identical results. Files updated: - test_simple_struct, test_complex_struct, test_ext, test_list, test_map - test_tuple (29 calls), test_tuple_struct (19 calls), test_enum (6 calls) - test_collection (2 calls), test_complex_refs (5 calls) - test_rc_arc (21 calls), test_box (10 calls) - mod.rs: removed test_tuple/test_collection as submodules (standalone) - test_helpers.rs: OneByte wrapper + deserialize_check helper Exceptions (kept as fory.deserialize): - Schema evolution tests (different serialize/deserialize types) - BinaryHeap tests (no PartialEq) - test_any tests (dyn Any types) - test_stream.rs (has its own stream-specific helpers) All tests pass, clippy clean, fmt clean.
Author
|
Hey @chaokunyang, have added helper , and tell me if you need any changes, do you want me to change every call site of fory.deserialize(_) with the helper? |
Collaborator
|
@Zakir032002 Are you still working on this PR? If not, I will close this PR. Other pull requests rely on this pr got merged to continue their work. For example, https://github.com/apache/fory/pull/3483/changes |
chaokunyang
reviewed
Mar 18, 2026
| /// # Buffer size limit | ||
| /// The internal buffer is capped at `u32::MAX` bytes (~4 GiB). | ||
| /// Requesting more than this returns [`Error::buffer_out_of_bound`]. | ||
| pub struct ForyStreamBuf { |
Collaborator
There was a problem hiding this comment.
Fory C++ don't use ForyStreamBuf, please align with Fory C++ instead
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Introduces streaming deserialization support for the Rust implementation of Fory, mirroring the C++
ForyInputStreamBufmodel.Changes
stream.rs— NewForyStreamBufstruct andReader::from_streamfor incremental stream-backed deserialization from anyReadsourcebuffer.rs— Extended to support stream-backed buffer operationsfory.rs— Wired up stream reader supportlib.rs— Exported new stream modulestream_test.rs— Integration tests for stream deserializationDesign
std::io::Readsource (files, sockets, byte slices)ForyInputStreamBuffrom PR feat(c++/python): support stream deserialization for c++ and python #3307Closes #3300