This was not a mistake I made, but it was one that I fixed. A product I worked on had a CSV import feature with an option to run the import in parallel to speed things up. For a while this worked fine. Then one day imports using the parallelization feature started only importing a portion of the CSV and generally leaving a mess that had to be manually cleaned up. No one seemed to know what was wrong.
One day I was assigned to look at it. I discovered that the import feature worked by spawning multiple subprocesses to process the import in parallel (PHP app). There was a calculation used to determine how many CPU cores were available on the host and then divide that number in half to determine how many processes to spawn with some math that was added later to reduce the number of processes spawned to avoid overburdening the machine. In another there was the same code to determine the number of processes to spawn, but no math to divide as in the other location causing a mismatch. This value was used to determine how to split the file for processing. Unfortunately, the new math for determining how many processes to spawn was not applied in both places.
I found the issue and fixed it by adding `/ 2`. Three characters that before being added had probably costs the company thousands of hours of having relatively high payed employees manually correct the data from the broken import process. In addition this was supposed to be a new product replacing a supposedly unmaintainable legacy product and the broken import process really hamstrung the migration process.
The whole product was a mess, but that really illustrated how much of a mess it was to me. A lot of it was caused by a couple of individuals who incredibly productive at writing sloppy and repetitive code that frequently made problems worse rather than better.
Comments
This was not a mistake I made, but it was one that I fixed. A product I worked on had a CSV import feature with an option to run the import in parallel to speed things up. For a while this worked fine. Then one day imports using the parallelization feature started only importing a portion of the CSV and generally leaving a mess that had to be manually cleaned up. No one seemed to know what was wrong.
One day I was assigned to look at it. I discovered that the import feature worked by spawning multiple subprocesses to process the import in parallel (PHP app). There was a calculation used to determine how many CPU cores were available on the host and then divide that number in half to determine how many processes to spawn with some math that was added later to reduce the number of processes spawned to avoid overburdening the machine. In another there was the same code to determine the number of processes to spawn, but no math to divide as in the other location causing a mismatch. This value was used to determine how to split the file for processing. Unfortunately, the new math for determining how many processes to spawn was not applied in both places.
I found the issue and fixed it by adding `/ 2`. Three characters that before being added had probably costs the company thousands of hours of having relatively high payed employees manually correct the data from the broken import process. In addition this was supposed to be a new product replacing a supposedly unmaintainable legacy product and the broken import process really hamstrung the migration process.
The whole product was a mess, but that really illustrated how much of a mess it was to me. A lot of it was caused by a couple of individuals who incredibly productive at writing sloppy and repetitive code that frequently made problems worse rather than better.