Check out apache arrow in rust for transferring large amounts data between rust and python. With it you can get zero copy transfers between the languages. Afaik there is some performance penalty involved when converting python types to native rust types.
It used to be hard to do such transfers between rust and python using arrow but from v3 or v4, arrow implementation in rust started supporting the c data interface. Creating the array in rust and sending it to python might involve leaking a boxed object though
As for performance, YMMV but I think it could be faster since storing a list of strings as a numpy array would involve padding. Arrow arrays can get away without padding
Comments
Check out apache arrow in rust for transferring large amounts data between rust and python. With it you can get zero copy transfers between the languages. Afaik there is some performance penalty involved when converting python types to native rust types.
It used to be hard to do such transfers between rust and python using arrow but from v3 or v4, arrow implementation in rust started supporting the c data interface. Creating the array in rust and sending it to python might involve leaking a boxed object though
Check this out if you want to see how https://github.com/jhoekx/python-rust-arrow-interop-example/...
i was using rust numpy for this, i wonder how much faster this is and if it supports passing arrays of strings
Yup. It supports arrays of strings.
https://docs.rs/arrow/6.2.0/arrow/array/type.LargeStringArra...
As for performance, YMMV but I think it could be faster since storing a list of strings as a numpy array would involve padding. Arrow arrays can get away without padding