> You can change the size of the allocation on the fly, and just keep writing bytes as you go.
How do you do this technically? mremap(), I assume?
You can do something very similar on windows, by VirtualAlloc()ing at the end of the previous allocation, and if that fails, be committing, VirtualFreeing and remapping at a different address.
Yes, I agree mremap() is much cleaner. I am mostly a Linux guy myself. What I'm missing in Linux is "mremap2()", which would map an existing block of memory onto a file (whereas mmap maps an existing file onto a block of memory ). redis achieves similar functionality (and more) by fork()ing and write()ing instead, but it would be nicer if it could be done with one call and no fork()ing. a "copy this file from here to here" could then be done by mmap()ing the source into memory, mremap2()ing the memory into the destination file, and forgetting about it.
added:
This would also allow btrfs/zfs to have O(1) file copy, even between independent media if possible (i.e. source file already exists on target media, even if in a different directory) - without any consideration from the cp implementation.
Yes, with ftruncate(), see the link above. The problem is Windows doesn't have an ftruncate() that works while the file is open and being written to. (Note: my memory is a little fuzzy here, it might be that there was no implementation that took a file descriptor instead of a filename, either way I couldn't pull it off).
As for your 'zero-copy dump memory into a file after-the-fact' idea, I'm sure it's done somewhere. My brain is too frizzled right now to scrounge up how though :).
When I was still following the kernel mailing lists, there were discussions about how to achieve this for sockets, and none about files, so that would surprise me.
However, if there is a way to achieve that without forking, I would be very happy to hear.
Comments
> You can change the size of the allocation on the fly, and just keep writing bytes as you go.
How do you do this technically? mremap(), I assume?
You can do something very similar on windows, by VirtualAlloc()ing at the end of the previous allocation, and if that fails, be committing, VirtualFreeing and remapping at a different address.
Yes, I agree mremap() is much cleaner. I am mostly a Linux guy myself. What I'm missing in Linux is "mremap2()", which would map an existing block of memory onto a file (whereas mmap maps an existing file onto a block of memory ). redis achieves similar functionality (and more) by fork()ing and write()ing instead, but it would be nicer if it could be done with one call and no fork()ing. a "copy this file from here to here" could then be done by mmap()ing the source into memory, mremap2()ing the memory into the destination file, and forgetting about it.
added: This would also allow btrfs/zfs to have O(1) file copy, even between independent media if possible (i.e. source file already exists on target media, even if in a different directory) - without any consideration from the cp implementation.
> mremap(), I assume
Yes, with ftruncate(), see the link above. The problem is Windows doesn't have an ftruncate() that works while the file is open and being written to. (Note: my memory is a little fuzzy here, it might be that there was no implementation that took a file descriptor instead of a filename, either way I couldn't pull it off). As for your 'zero-copy dump memory into a file after-the-fact' idea, I'm sure it's done somewhere. My brain is too frizzled right now to scrounge up how though :).
When I was still following the kernel mailing lists, there were discussions about how to achieve this for sockets, and none about files, so that would surprise me.
However, if there is a way to achieve that without forking, I would be very happy to hear.