So this post is a bit different as it doesn’t relate to mobile or web development particularly, but is more of a conceptualization and analysis. Recently, I’ve been tasked to move and summarize large sets of data; we’re talking tens of millions of records, resulting in hundreds of gigabytes of space needed. At first I took a rudimentary approach which involved in writing a console app in C# using ADO.NET. I won’t even going into using an ORM (Object Relational Mapper) like Entity Framework, lets just say: they’re not efficient in this scenario.