Lately, I have encountered several questions regarding the performance of Entity Framework (EF). I have been conducting feasibility study on EF since version 4.0 and followed through several versions but every time I see the performance results, I was not convinced to use it. But I can see performance improvements in every new version. Wanting to satisfy my curiosity and also to update my impressions towards EF, I have thought to conduct another round of performance test on it.
I realized that not only have I built an arsenal of samples but I have also created an avenue to use those samples for performance testing (all thanks to my Associate who reminded me by always playing with using the samples in demos and test runs). For the showdown, I will pit the ASPNET-WCF-EF sample against the ASPNET-WCF-DAAB sample, since both uses exactly the same code for all the layers but only differs in data access technology (all thanks to the swap-in-swap-out capability of Layered Architecture).
The Test Machine
- Windows Server 2012 R2 x64
- Intel Core i7-4800MQ CPU 2.7GHz (Quad Core HT)
- 16 GB RAM
- 500 GB Solid-State Hybrid Drive
- Visual Studio 2013 Ultimate Update 1
- Microsoft SQL Server 2012 x64
Preparation
To make use of the samples for the performance tests, few changes had to be made to both sets of code.
1. Disabling IsOverlap Check in Business Component
The samples contain an IsOverlap business logic check to prevent duplicate records from being inserted into the Leaves table. This will cause the unit tests to fail if high volume of test data were induced into it. To allow the application to continue to run even with overlaps, I commented out the line where the exception is thrown. You can locate the line of code in the Apply method of the LeaveComponent.cs.
// Check for overlapping leaves.
if (leaveDAC.IsOverlap(leave))
{
//throw new ApplicationException("Date range is overlapping with another leave.");
}
This will allow the query to continue to run but the exception will be suppressed.
2. Disable Enable Edit and Continue
Enable Edit and Continue needs to be disabled in the LeaveSample.Hosts.Web project to allow it to be running (See here for details) for the test. Alternatively, you can publish it to IIS if you want. For this test, I'm using the default IIS Express.
With this done, we can start the LeaveSample.Hosts.Web project in debug mode to launch the IIS Express instance and then close the browser that was launch. The Host will be running in the background.
Take note that this needs to be done for both samples and at any one time, only one of the samples can be opened and run in Visual Studio. This is to avoid any conflicts.
A Glimpse of the Code
In case you do not want to download the samples and wondering how the code looks like, basically, here are some code snippets to give you an idea. I will recommend you download the samples and play with it yourself.
The EF code looks like...
public Leave Create(Leave leave)
{
using (var db = new DbContext(CONNECTION_NAME))
{
db.Set<Leave>().Add(leave);
db.SaveChanges();
return leave;
}
}
and the DAAB code looks like...
public Leave Create(Leave leave)
{
const string SQL_STATEMENT =
"INSERT INTO dbo.Leaves ([CorrelationID], [Category], [Employee], [StartDate], [EndDate], [Description], [Duration], [Status], [IsCompleted], [Remarks], [DateSubmitted]) " +
"VALUES(@CorrelationID, @Category, @Employee, @StartDate, @EndDate, @Description, @Duration, @Status, @IsCompleted, @Remarks, @DateSubmitted); SELECT SCOPE_IDENTITY();";
// Connect to database.
Database db = DatabaseFactory.CreateDatabase(CONNECTION_NAME);
using (DbCommand cmd = db.GetSqlStringCommand(SQL_STATEMENT))
{
// Set parameter values.
db.AddInParameter(cmd, "@CorrelationID", DbType.Guid, leave.CorrelationID);
db.AddInParameter(cmd, "@Category", DbType.Byte, leave.Category);
db.AddInParameter(cmd, "@Employee", DbType.AnsiString, leave.Employee);
db.AddInParameter(cmd, "@StartDate", DbType.DateTime, leave.StartDate);
db.AddInParameter(cmd, "@EndDate", DbType.DateTime, leave.EndDate);
db.AddInParameter(cmd, "@Description", DbType.AnsiString, leave.Description);
db.AddInParameter(cmd, "@Duration", DbType.Byte, leave.Duration);
db.AddInParameter(cmd, "@Status", DbType.Byte, leave.Status);
db.AddInParameter(cmd, "@IsCompleted", DbType.Boolean, leave.IsCompleted);
db.AddInParameter(cmd, "@Remarks", DbType.AnsiString, leave.Remarks);
db.AddInParameter(cmd, "@DateSubmitted", DbType.DateTime, leave.DateSubmitted);
// Get the primary key value.
leave.LeaveID = Convert.ToInt64(db.ExecuteScalar(cmd));
}
return leave;
}
I know you are already screaming - "Holy Cow!!!"
Unit Test: Single Run
I noticed that the Unit Test in Visual Studio now provides execution times. This is very handy and I would like to take advantage of it. I chose the ApplyThenApproveTest unit test method since it simulates a complete Apply and Approval of a leave transaction which should give a good mix of INSERTs and UPDATEs to multiple tables and SELECT operations.
Running a single unit test on each sample produces the following results:
Method using Entity Framework 6.0.2 took 171 ms to complete 1 transaction.
Method using Enterprise Library 6.0 DAAB took 139 ms to complete 1 transaction.
Unit Test: Looping in 1000
That's pretty good for both since they completed in milliseconds. Let's raise the stake to loop 1000 times and see the results:
Method using Entity Framework 6.0.2 took 44 secs to complete 1000 transactions.
Method using Enterprise Library 6.0 DAAB took 30 secs to complete 1000 transactions.
Instrumented Performance and Diagnostics Profiler
Let's dig slightly deeper to see the time breakdown. Visual Studio Ultimate comes with an awesome Performance and Diagnostics Profiler. I used Instrumented profiling on both the samples and here are the results from the web-server's perspective. The result is from web-server to database.
Apply method using Entity Framework 6.0.2 took an Average Elapse Time of 1,124.97
Extra observation: Querying took an Average Elapse Time of 2,937.66
Apply method using Enterprise Library 6.0 DAAB took an Average Elapse Time of 432.23
Extra observation: Querying took an Average Elapse Time of 861.90
And here are the detail breakdown of how much time it took for instrumenting only the app-server side. Results are from app-server to database:
Create method in data layer for Entity Framework 6.0.2 took an Avg. Elapse Time of 517.30. The EF methods DbContext.SaveChanges() took Avg. Elapse Time of 477.78 and DbSet.Add() took 38.74.
Create method in data layer for Enterprise Library 6.0 DAAB took an Avg. Elapse Time of 9.27. The DAAB methods Database.ExecuteScalar() took Avg. Elapse Time of 4.96 and DatabaseFactory.CreateDatabase() took 0.02.
From the Call Tree we can observe that the functions using DAAB is considerably faster than EF.
Load Test
Finally, let's see how the two perform under load. I will use Visual Studio Load Test to load the single run ApplyThenApproveTest unit test method. I will use a constant load of 100 users and run the test for 1 minute.
Entity Framework 6.0.2 completed with 1692 Total Test runs with an Avg. Test Time of 3.41 sec but it gave 154 errors. The error thrown was 'An error occurred while reading from the store provider's data reader'.
Enterprise Library 6.0 DAAB completed with 8236 Total Test runs with an Avg. Test Time of 0.70 sec and gave 0 errors.
I was surprised (and concerned) by this result as it appears that EF may not be able to perform under high stressed load.
What is Going On Behind The Scene?
Out of curiosity, I fired-up SQL Profiler to see what is being sent to the SQL Server.
INSERT statement generated by Entity Framework 6.0.2
INSERT statement going through Enterprise Library 6.0 DAAB
SELECT statement with Paging generated by Entity Framework 6.0.2
Custom Paging SELECT statement going through Enterprise Library 6.0 DAAB
Summary and Wrap Up
From the performance test results it clearly shows that Entity Framework is still not as efficient as wrapper libraries such as DAAB which uses native ADO.NET. However, EF does make code a lot more easier to read and a lot less to write. The load test findings do raise some concerns for high performance applications that requires high concurrency.
Because of these findings, I would have to skip EF implementation again for this round :( and wait for more improvements to it. The choice of not being able to use EF for me is bind by the constraints I have in my environment that requires the applications to process millions of transactions per hour. This is the volume of processing in Telco environment.
Whether to use EF or not is entirely up to your environment and choice (as long as you know the limitations of the chosen technology). If your system does not require such crazy high loads, I don't foresee you will have any problems in using EF.
I hope this post has given you good enough insights into Entity Framework. If you have ideas for me to fix the EF exception, please feel free to post it in the comments.
No comments:
Post a Comment