Comment on Show HN: Parsing CSV files with GPUComments−maxhou11yIn the real world, the slow part of "parsing" a CSV file is IO: reading the file content from disk to memory, and from memory to CPU cache.You would avoid reading the file content more than once if you had to parse it.The first line counts the number of lines in a buffer (assuming that file is read into memory and copied to gpu buffer d_readbuff).but this is what is done here, first search to find all \n, then multi-core GPU stuff for each line content.−azurezyq11yThings have been changed in a world of SSDs and machines w/ much memory. Actually, parsing a CSV in a single thread will never reach several hundred megabytes per second.
Comments
In the real world, the slow part of "parsing" a CSV file is IO: reading the file content from disk to memory, and from memory to CPU cache.
You would avoid reading the file content more than once if you had to parse it.
but this is what is done here, first search to find all \n, then multi-core GPU stuff for each line content.
Things have been changed in a world of SSDs and machines w/ much memory. Actually, parsing a CSV in a single thread will never reach several hundred megabytes per second.