Using BenchMarkDotNet for Performance BenchMarking

This article is for my second post for c# advent calendar. Many might be using the BenchmarkDotNet nuget package for measuring the performance of your .Net Application Code. This nuget package is used in many projects to run their performance benchmarks.

I recently came to know about BenchMarkDotNet nuget while I was working with Robert in one of his github project(memstate). That was my first pull request to github where I implemented a surrogate converter for Json Serialization.  As part of implementing Surrogate converter, we had two different options to implement. We knew that one method was not good with respect to performance while second approach is good from performance perspective. So I asked Robert about his opinion, on which option to select to proceed with the implementation. Robert suggested me to take any option as they will be performing Benchmark later. This was my introduction to BenchMarkDotNet. So I wanted to explore the nuget package further.

There is good documentation present on how to use the nuget package along with some guidance on best practice.

For my sample test, I created a class library with below code.We need to add BenchMarkDotNet nuget package to both class library and console application.

namespace BenchMarkTest
{
[Config(typeof(Config))]
[MemoryDiagnoser]
public class TestClass1
{
private class Config : ManualConfig
{
public Config()
{
Add(new Job(EnvMode.LegacyJitX64, EnvMode.Clr, BenchmarkDotNet.Jobs.RunMode.Dry)
{
Env = { Runtime = Runtime.Clr },
Run = { LaunchCount = 3, WarmupCount = 5, TargetCount = 10 },
Accuracy = { MaxRelativeError = 0.01 }
});
Add(new Job(EnvMode.LegacyJitX86, EnvMode.Clr, BenchmarkDotNet.Jobs.RunMode.Dry)
{
Env = { Runtime = Runtime.Clr },
Run = { LaunchCount = 3, WarmupCount = 5, TargetCount = 10 },
Accuracy = { MaxRelativeError = 0.01 }
});

}
}
[Benchmark]
public void TestMethod()
{
Console.WriteLine("This is a benchMark test");
}
}
}

We need to use the attribute BenchMark against the method which we want to know the performance. I have also included the details of BenchMark jobs as part of the Configuration.

In my sample am trying to benchmark code against CLR which is my case is .Net Framework 4.6 and trying to measure performance against x86 and x64 configuration. We can also specify the LaunchCount, WarmupCount and TargetCount to be used as part of our benchmark.

Using the attribute of Memory Diagnoser we can measure the memory allocation against the method.

There are different run modes or run strategy which you can select like cold start, monitoring or throughput. Additional details on choosing an appropriate strategy can be found in the benchmark documentation.

To run the benchmark, we need to add a console application. Below is the code for my console application, where am trying to perform benchmark against my TestClass.

using System;
using System.Collections.Generic;
using System.Linq;
using System.Text;
using System.Threading.Tasks;
using BenchmarkDotNet.Running;
using BenchMarkTest;
namespace ConsoleApplication1
{
class Program
{
static void Main(string[] args)
{
TestClass1 mytest =new TestClass1();
BenchmarkRunner.Run<TestClass1>();
Console.ReadLine();
}
}
}

BenchmarkRunner.Run is used to initiate the benchmark against the class which we want to measure performance.

It is recommended to run the benchmark in release mode. They have made sure to provide a warning when the benchmark is run under Debug mode.

So I went ahead and made sure the build mode is release and tried run a benchmark from Visual Studio by executing my console application.

Under the bin folder of console application, under Release folder a benchmark artifact folder will be created, which has the results of the benchmark summary.

Results are exported in csv format, html format and summary is generated.

So in these results I can see that it is using an Attached Debugger. This is not recommended approach. The best way as per documentation is to launch the console application from command prompt from the Release folder.

Below are the results when launched from command prompt.

In these results, us means microseconds.

Am still trying to explore completely the nuget package. Some of the attributes mentioned in the benchmark documentation like MinColumn and MaxColumn were not working with the new package, may be I might have missed something, which I will be exploring further.

Thanks to Matthew Groves for allowing me to sign up for two slots as part of c# advent calendar. Follow the further posts as part of advent calendar here.

Click here to find the documentation on BenchMarkDotNet

 

Advertisement