Photo by Tania Melnyczuk on Unsplash
What's the easiest way to divide your data into equal-size pieces ?
You can split 150K data into 10K packets in just 40 ticks. This value was obtained on .NET4.8 (C# 7.3) on a personal computer. By the way there are 10,000 ticks in a millisecond.
You can use Parallel.For() or Parallel.Foreach() if you have million data. Parallel is defined in .Net 4.0 and above frameworks. ForEach loop runs on multiple threads and the processing takes place in a parallel manner.
List<String> list = new List<String>();
for (int i = 0; i < 150001; i++)
{
list.Add(i.ToString());
}
int chunkSize = 9999;
var numOfChunks = list.Count / chunkSize; // initial number of chunks
if (list.Count % chunkSize > 0) { numOfChunks++; } // add one chunk for remainder if there is a remainder
for (var i = 0; i <= numOfChunks; i++)
{
var chunk = list.Skip(i * chunkSize).Take(chunkSize);
// Do something with chunk, like writing to file
}