Sparing that, why not use fseek() and do a binary search to get to where you wanted to go? Even better, save the prior offset somewhere and jump there immediately on T+1?
That, I should have done. I thought to save the number of lines previously read so that I could skip them; I don't know why it didn't occur to me to just save the whole damn offset.
I used this trick the other day -- compute byte range partitions on a large file and use fseek when processing each. Is there a standard unix program that can output an arbitrary byte range from a seekable input? At first blush dd(1) looked like the ticket, but block-oriented operation means extra invocations to deal with a byte range that's not necessarily block-aligned.
Comments
Why didn't you split it?
Sparing that, why not use fseek() and do a binary search to get to where you wanted to go? Even better, save the prior offset somewhere and jump there immediately on T+1?
That, I should have done. I thought to save the number of lines previously read so that I could skip them; I don't know why it didn't occur to me to just save the whole damn offset.
I used this trick the other day -- compute byte range partitions on a large file and use fseek when processing each. Is there a standard unix program that can output an arbitrary byte range from a seekable input? At first blush dd(1) looked like the ticket, but block-oriented operation means extra invocations to deal with a byte range that's not necessarily block-aligned.
tail -c offset [input] | head -c length
will do the job.
You can set dd's "bs" parameter to 1, then seek, skip & count are all in terms of bytes.
I couldn't, I didn't have control over the file.