Loading Two Million Entities as Fast as You Can



“Just when it seemed I’d finished squeezing out performance improvements for the framework – the 0.6 release gives us even more! The performance gap has therefore widened even more comparing CodexMicroORM to Entity Framework and Hibernate.”

The title sums up a use case I recently threw at CodexMicroORM. You might wonder, “that’s not typical of my workload” – but it’s an important requirement for ZableDB (aka ZDB). As I started to lay out in my prior article, ZableDB is a data storage provider that sits on top of CodexMicroORM. It acts as an object oriented database, removing the need to formally model a separate database layer: your native object model is persisted (including the relationships between objects), using familiar ORM-type semantics.

If we’re talking about rows in a database table, “two million” isn’t that big of a number, so it’s fair to insist on handling this much data using ZableDB. “Handling” in this case requires us to a) generate the data in memory so it can be saved, and b) move the data from disk back to memory on start-up. The latter requirement comes from the fact ZableDB is an in-memory database, and we make every effort to bring everything into memory early in your app life-cycle. It was during the implementation of these two requirements that it became obvious we needed to “tweak” parts of the CodexMicroORM foundation layer. The remainder of this paper discusses more detail on the how’s and why’s behind the changes made.


A Problem and a Solution


To fill a ZableDB instance with lots of data to perform early testing, we need to populate an object model. I chose to use the set of “Widget-related” classes I previously discussed, creating 10 WidgetType’s, 500,000 Widget objects, 700,000 WidgetReview’s, 200,000 Customer’s, 200,000 Receipt’s, and 400,000 GroupItem’s. This was all done using the same pattern we’ve seen demonstrated before, where we instantiate objects in a “new” state and call CEF.DBSave() to commit pending changes:

// Create 700,000 widget reviews (70,000 users x 10 types)

Parallel.For(1, 700001, (i) =>


    using (CEF.UseServiceScope(ss))


        CEF.NewObject(new WidgetReview() { RatingFor = alltype[Convert.ToInt32(Math.Floor((i - 1) / 70000.0))], Rating = Convert.ToInt32(Math.Round((Math.Pow((i % 10), 1.3) / Math.Pow(9, 1.3) * 10), 0) / 10.0), Username = $"User{((i - 1) % 70000)}" });







In this case, I wasn’t thrilled with overall populate-and-save performance. After drilling into the hot paths that were causing bottlenecks, it became evident that the ConcurrentIndexedList class that I covered in “Indexing In-Memory Collections for Blazing Fast Access” had some limitations contributing to the problem. Specifically, it performed poorly in write-heavy situations like you’d find when aggressively creating / adding objects in service scopes – exactly what I was doing above. The main issue was in the locking being used, despite the fact I’d gone with reader/writer locking which should have had limited overhead in read-heavy workloads. As I was finding, reads were still super-important, but writes were being penalized too much.

My answer was to find a more balanced read-write trade-off. I switched out basic dictionaries for a new class: SlimConcurrentDictionary. This class sounds like the .NET Framework Class Library’s (FCL) ConcurrentDictionary – and it shares some principles – but raw performance testing shows it performs more than twice as good as ConcurrentDictionary for writes, and only slightly worse for reads. The trade-off becomes obvious for my heavy write workloads with the data generation process being halved. Another change was to use a custom class called LightweightLongList that’s used internally by ConcurrentIndexedList, serving as a thread-safe list that also supports better-than-average write performance.

Part of ZableDB is the concept of your in-memory data store. This can be accessed directly using LINQ such as:

var ws = (from a in ZDB.All<Widget>().FindByEquality(nameof(Widget.SerialNumber), "SN" + i.ToString("000000")) select a).FirstOrDefault();


In this example, ZDB.All<Widget>() is returning all 500,000 Widget’s I previously instantiated and saved. The read performance of this query is pretty astounding compared to traditional RDBMS implementations: it’s on the order of about 20 microseconds per request on my dev machine. (In fact, there’s some up-front cost to index these collections, but accessing them in indexed form after that becomes O(1) in the case of the equality checking seen here.)

The collection returned by ZDB.All() posed another problem for me: as an IndexedSet<T> (inheriting from EntitySet<T>, part of CodexMicroORM), locks were being taken out on writes to the collection (i.e. “Add”), killing concurrency during the “load phase” of ZDB. To address this, I introduced the ConcurrentObservableCollection<T> class to CodexMicroORM.

ConcurrentObservableCollection resembles FCL’s ObservableCollection class, but it only shares similar interfaces – it uses no FCL base classes and is therefore an effective re-write. It offers performance characteristics like SlimConcurrentDictionary: writes are significantly faster than the previous implementation and reads pay a small penalty. In this case, the performance differences become obvious by doing some simple benchmarking with a 15,000,000-iteration loop:

ConcurrentObservableCollection<TestItem> l2 = new ConcurrentObservableCollection<TestItem>();


Parallel.For(1, 1500001, (i) =>


    l2.Add(new TestItem() { ID = i });



Comparing this against FCL’s ConcurrentBag<T> class (something similar in terms of being thread-safe and “list-like”), it’s on reads where it blows away the competition: reads are 95% faster - but writes are in fact 50% slower. Sounds bad for my specific workload? Writes are still 90% faster than reads for ConcurrentBag<T>, and the previous implementation wasn’t using ConcurrentBag but instead locking over the ObservableCollection base, arguably much worse for both reads and writes. (I mention ConcurrentBag only to emphasize that alternatives out of the FCL were considered and dismissed based on empirical testing.) The benefits of this change became clear in the ZDB “load on start” process: that time dropped by 50% with the use of ConcurrentObservableCollection. This was an easy change since ConcurrentObservableCollection exposes a signature that closely resembles the original ObservableCollection base.

The proof that the change for writes wasn’t a net-negative versus the previous versions is supported by running our existing benchmarks that have mixed read-write workloads. Without re-doing every one of them, a sample shows improvements of 12%, 14% and 13% over previously reported timings, as presented here and in other content. I’m calling version 0.6 a productive release as such!


Other Changes


Staying on the topic of performance, I added a method in Performance.cs called FastCreateNoParm(). This method lets you instantiate a new object (with its default constructor), much like Activator.CreateInstance() does – but much faster as proven by raw timings. (Needless to say, almost all  Activator.CreateInstance() were replaced in the code base.)

Another enhancement was the addition of a Globals setting called ConnectionScopePerThread. The default is “true” which matches the behavior in prior releases. If you set this to “false”, it means connection scopes are bound to your service scope instead – when a service scope goes “out of scope”, it will close any associated connection scope. This might be clearer for some and has importance for synchronization: when we’re running things like asynchronous saves, when our connection scope ends, we can be sure all work that’s part of any saving done on that connection has been completed.

I also added some additional settings related to “system parameters” such as default dictionary sizes, estimated scope sizes, etc. The defaults for these are generally “good enough” for most situations – but when you get 2,000,000 objects in a service scope (like I had in my example!), adjusting these settings can help improve performance further (e.g. by avoiding large, expensive dictionary resizes).


What’s Next?


My intent is to with the next release, provide some ZableDB bits that can be pulled from NuGet. ZableDB will be a data provider extension that sits on top of CodexMicroORM, as illustrated here:

(In blue are components that are included in the existing CodexMicroORM package on NuGet.)

I encourage you to sign-up on xskrape.com to get e-mail notifications when new articles are published – I’ll be providing updates like this one, showing my thought process along the way. I also encourage you to visit our proposed ad on StackOverflow: by voting for it, after a certain threshold we can participate in getting some free promotion as an open-source project – a big “thank you” in advance!


- codexguy

Did you like this article? Please rate it!

Back to Article List