We had used the Entity Framework in our projects earlier, but it was only recently that we tried its Inheritance support (or the lack of it, as we will soon see).

Here are the different Inheritance Types supported by the Entity Framework:

Table Per Hierarchy (TPH) – With this approach, we create a single “Super Wide Table” for a hierarchy, containing the fields required for all classes in an inheritance hierarchy. If we had a Lawyer, Physician and an Artist, all inheriting from a Person class, we would create a table containing all fields from each of these types.

Huh! I am left wondering why this approach is even supported in the first place. Not only does this give a horrible database schema, you will also have to make compromises like making non-nullable fields nullable.

Table Per Type (TPT) – TPT gives you a good database schema with normalization, plus inheritance. In fact, with TPT the schema is very similar to an approach using Composition instead of Inheritance. This was the approach we adopted for the application early on, since there seemed to be nothing wrong with it.

But when we finished modeling the domain and hit the code, we got a bad surprise. The TPT implementation in Entity Framework is so slow that it can only be called broken. A simple join on an inherited table gave me a 3200 line query, 170KB in size, and it took 15 seconds for Entity Framework to compile the expression to SQL. And this was just for _generating_ the SQL query!

There are some discussions on this, but nobody seemed to have an answer. There is no way this code can go anywhere near a production server.

One option was to avoid Inheritance and use Composition. Using Composition leads to code that makes less semantic sense; if you had to access a lawyer’s FirstName you had to call – Lawyer.User.FirstName. And base class methods that should be available in the Lawyer class now belong to the the User inside Lawyer. It gets worse when Lawyer inherits from another class, say LegalWorker which in turn inherits from User. We decided not to go this route.

Option 2: Translating Inheritance to Composition at runtime

We know that the Entity Framework handles composition queries without (too many) performance issues. If we can transform our queries involving inheritance into ones having only composition, Entity Framework can do the rest without measurable loss of performance.

To do that, while we design our Entity classes using the EDMX Designer we will avoid using inheritance. If ideally a Lawyer is a User, we use composition instead of inheritance and say Lawyer has a User.

Sounds ugly? Yes it is. But don’t worry, we will not be using these classes in code. They are being used only to make Entity Framework do the heavy lifting. Alright. Let us assume that we have created our entire domain model using the EDMX designer.

The next task is to define the same set of classes, but using inheritance. These are the classes we will be using while writing code.

Fig: Relation between classes with inheritance, and those with composition.

Auto-generating Models from EDMX

Manually modifying our Models every time we make a modification to the database or the EDMX is cumbersome and error prone. We wrote a code-generator for this, which analyzes EDMX and outputs C# classes with Inheritance.

For the figure represented above,  the code generator creates

public class User
{
public string FirstName { get; set; }
……..
}

public class Lawyer : User
{
public ICollection<Customer> Customers { get; set; }
……..
}

This code generator is available under AgileFx.Tools.

Now we can write queries like:
from lawyer in Lawyers
where lawyer.FirstName == “Hemchand”
select lawyer;
//although FirstName did not belong to Lawyer in the Entity Framework Designer generated model.

But that was the easy part. If we pass this expression (compiler converts this query to an expression tree) to the Entity Framework, it will not know what to do with it. This is because the classes used in the expression are not classes that the Entity Framework can translate to an SQL query. The Lawyer class referenced in the expression is just a mirror (albeit with inheritance) of the EF.Lawyer class, which is the one the Entity Framework is aware of.

We need to write a Query Translator, to convert an expression referencing our new classes into ones that reference Entity Framework Designer-Generated classes (ie, from App.Models.Lawyer to App.Models.EF.Lawyer).

Writing the Query Translator

How it works

Let us consider a simple Where query on the Entity set lawyers.
lawyers.Where(lawyer => lawyer.FirstName == “Hemchand”)

1. The compiler translates lawyer => lawyer.FirstName == “Hemchand” into an expression, say EXP.

2. lawyers.Where is an extension method defined in System.Linq.Queryable
so the actual call is – Queryable.Where(lawyers, EXP)

3. “Queryable.Where” calls lawyers.Provider.CreateQuery(expression-representing-the-Where-call, EXP, …)

4. Since we implemented our own provider to replace Entity Framework’s Query Provider, Provider is an instance of our CustomQueryProvider. Once we receive EXP in CustomQueryProvider.CreateQuery() we parse EXP using a Query ExpressionVisitor and rewrite the query after converting references of lawyer.FirstName to eflawyer.User.FirstName.

References

If you are writing your own IQueryable/IQueryProvider implementations, this 15-post series by Matt Warren will be very helpful. Actually, it is not just helpful; it is the TextBook for writing your own Query Providers. Matt has also made the code available on CodePlex (IQToolkit), which we have re-used in our implementation.

If you want to learn more about LINQ and Querying itself, you should see this series by Wes Dyer. Reading this blog is highly recommended, even if you are not writing an IQueryable Provider. These posts are profoundly insightful.

Shortcomings of the Translation

When we ask Entity Framework to fetch a User, it fetches an instance of the actual type of the User; like a Lawyer. (Trying to get this working is what possibly caused the performance issue in Entity Framework; just guessing here).

That is,
User u = <some_query>;
Now u.GetType() will get you Lawyer, Artist etc.

With our translation layer, you get an instance of User when we issue the same query. Based on the a field like UserType we have to fetch the actual Lawyer or the Artist.

There are two reasons why this is not so bad; one is that you probably want just the User (base class) since that was what you queried for. The second reason is that there is no way we can fetch the actual instance without querying all the tables in the inheritance hierarchy, and the associated performance problems that come with it.

Some useful side effects

By implementing Query Translation, our framework became conceptually ORM Framework independent. We can now translate queries to any framework. Presently we support only Entity Framework, but the next goal is to support Linq to Sql.

Someone fix the Entity Framework!

Finally, some complaining. Considering that the Entity Framework shipped with .Net Framework SP1, one would expect it to be sufficiently tested. But it seems not. If inheritance is a supported feature, how is it that even the simplest queries involving inheritance result in 15 second query generation times?

On the other hand, Linq to Sql is awesome. It is lightweight, and generated simpler SQL. Which is why we are happy that our code base is not ORM dependent anymore. There is some talk of Linq to Sql being abandoned, which I hope is not true. Many of us need just a lightweight ORM, and Linq to Sql is just that.

Thanks.

AgileFx is not yet ready for public release or even a BETA, but you could take a look at it on codeplex -  http://agilefx.codeplex.com/

Credits:
Most of the code for the Query Translator was written by our in-house C# grandmaster Hemchand T . If something blows up, catch him. ;)

– Jeswin P

Going with Mono, in 2006

In early 2006, we were working on a fairly large Social Networking project which included building a few long running processes. We decided to build these in the popular, but not yet widely deployed platform called Mono. Two factors tilted the decision in favor of Mono, the most important being that we were building the application on Linux, and the second being that we had decided to use a mainstream object-oriented, statically-typed language for building these components. That leaves us with Java and C#; while Java was left in the quagmire of community-driven, slow design process, C# was getting increasingly productive leaving Java behind in terms of expressiveness. By 2006, Mono had an excellent C# compiler, an efficient runtime, and good coverage of the .Net Base Class Libraries.

Though the project eventually failed due to non-technical reasons, Mono itself served us well for 3 years. Over that period, Mono improved by leaps and bounds. Compatibility is great with Net 2.0 libraries being almost entirely supported, compilers are exceptional, MonoDevelop 2.0 has become a capable IDE, and performance is comparable to Microsoft .Net itself. In specific benchmarks, the Mono runtime even bests the official  .Net runtime. Often, the mono compiler supports upcoming C# features ahead of the official Microsoft release.

We could say that the “Mono Project” has done well over the years. I actually admire the Mono Team; they have faced unnecessary criticism within the Open Source community, often by people who knew little about the project. That did not stop them from building a great product.

However, my confidence in Mono has been waning for some time.

In the Shadows of .Net

Back in 2006, we put our trust in Mono because we refused, or perhaps disliked, to vilify a project solely because it emulated something created at Microsoft. While Open Source backers generally dislike Microsoft technology, with Mono they had another argument that being a clone it could be affected by a number of patents that Microsoft holds related to the .Net framework. This point often comes up in debates about the “safety” of the Mono project, the defense of Mono being that large parts of the .Net specification are an open, published ECMA standard. I sided with the Mono supporters then, downplaying the risk of patents from Microsoft. But then in November, Microsoft and Novell announced their Patent Agreement, which guarantees patent protection exclusively for users of Novell Linux. The Mono project is largely supported by Novell, and such an agreement is disastrous for a community project like Mono. At this point, the fence-sitters in the Open Source community largely crossed over to the anti-Mono camp. Perhaps, they were justified in doing so. I could no longer defend Mono, and my belief in the framework getting wider acceptance has diminished significantly since then.

One might wonder why acceptance or ideological reasons should affect the platform we choose to build a product on. It just needs to work well, right? It is not that easy. Today my company works with customers on Microsoft and non-Microsoft platforms. We have skills on Microsoft and Open Source technologies, and while consulting or providing services to our customers we choose the right tool for the job irrespective of our ideological leaning. But when building our own products, we  are convinced about the advantages of Open Source. And as a company, we see Open Source as a principle and ideology – not just for its commercial benefit. Building a new product requires a lot of sacrifice, stamina, coffee and redbull. And you will only build it on a technology you believe in.

There are other minor issues that make Mono a stranger in Open Source land. Mono uses very Windows specific conventions, like “.exe” for executables and “.dll” for libraries. For non-windows users some of these conventions look odd. Again, this is not Mono’s fault; rather a consequence of emulating a platform designed for a very different Operating System.

It is entirely possible that Mono can suddenly gain acceptance if Microsoft decides to relinquish patent claims regarding the .Net framework. If it happens, .Net and Mono could well become an powerful challenger to the dominance of Java. This is very unlikely, Microsoft’s current strategy seems to be relying strongly on patents and IP to ward off the looming threat from Linux.

For now, we decided to look beyond Mono and C#.

What about Python?

In the last two years, I have mostly used Python with Django for web development. It is easy to learn and to apply. Documentation is plentiful, and there are libraries for getting most things done. While Python is  easy and practical, there was nothing quite remarkable about it. I felt Python’s strengths were not a consequence of well thought-out language design; instead it was the flexibility inherent in the dynamic nature of the language.

The language itself seems to be going backwards. I do not deny that I have an affection for functional-style programming, but its relevance cannot be overlooked anymore. In a variety of situations, Functional Programming (FP) is more concise, and it is also more suited for the dawning Parallel Programming era. For this reason alone, languages like C# are incorporating functional concepts in future versions. But not Guido;  Python’s benevolent dictator has a well known dislike for functional style programming. Guido deserves credit for creating a very popular programming language, but you wonder if he is paying attention to the evolution of contemporary programming languages.

Some people say that Python’s legacy will be like that of Visual Basic; if it ever gets that popular. I doubt whether it will be as popular as “VeeBee”, but I don’t disagree. There are more capable languages out there. If we ever accomplish interoperability across dynamic languages, I predict that Python would have a bleak future. Why aren’t the Parrots speaking, yet?

The JVM

Java is the most common language for building applications today; due to its maturity, multi-platform support, proven reliability and the huge-huge ecosystem built around it. But Java itself is a really sucky language. It is boring to write code in Java, and even simple tasks tend to be very lengthy. Steve Yegge’s ‘Kingdom of Nouns‘ captures this verbosity in a humorous style.

The JVM is however, quite an interesting platform.  While originally designed for the Java language, it is now the target of many new languages. Some of these languages have been around for some time now and have gained some momentum, like Scala, JRuby, Groovy and Clojure. Groovy has not caught on as much as the others; so it fails my “excellent community” criteria. JRuby’s biggest problem is Ruby itself; rockstars are unlikely to use the JVM.

But Scala is impressive; has great interop with Java, has a growing community around it,  and recently acquired a really high-profile user – Twitter. If you have been programming in Java or C#, Scala is a wonderful alternative. For Java developers, Scala offers a leap in productivity and the ability to call and get called by Java code. If you are on the Microsoft stack and hate the JVM because you hate Java, Scala offers the expressiveness that was recently introduced into C#.

But I knew Scala wasn’t what I was looking for. I actually knew what I wanted; I wanted a language with few syntactic rules and constructs. I wanted to move to a language I do not have to discard any time soon.  I wanted something like the ‘Hundred Year Language‘.

LISP will win, eventually

LISP is a bit like our company’s motto: “Simplicity is the ultimate sophistication.” – originally by Leonardo DaVinci. The language itself is so simple that you could learn the semantics in less time than it takes to read this article. This simplicity is achieved by abstracting ideas in programming languages to the extreme; yielding a simple form which is infinitely extensible. This extensibility comes from the ability to treat a program as data, and manipulate the program itself, using a feature called Macros.

While the syntax of LISP is very simple and elegant, writing a typical business or web application in LISP is (or was) much harder; perhaps an order of magnitude harder. This is especially true for newbies; unless they are really determined it is difficult to find documentation and useful libraries. This happened to me as well. I had read many books on LISP, but I did not have the confidence to execute a real project in LISP. It has been done before by so many others, but there is a big challenge in transitioning from a theoretical understanding of LISP and getting it to work in practice.

Although I was unable to use LISP in any production code, what I learnt from LISP profoundly impacted the way I saw programming. And since then, I have been keeping a close watch on upcoming languages bearing any resemblance to LISP. One such language was Clojure.

Clojure, the acceptable LISP

Clojure is a dialect of LISP and a dynamic programming language, running on the JVM.   It has all the benefits of LISP, including the powerful macro system. It also has useful features which help with parallel programming; like Software Transactional Memory for dealing with shared state without worries. Clojure makes some of LISP’s biggest problems go away. Lack of Libraries and other supporting infrastructure like Application Servers. Java has libraries for everything, from connecting to SAP to talking to Twitter. And you could deploy on robust servers like Tomcat.

Everything ever built for Java can be used with Clojure.

Clojure is very young, and like any new programming language you would encounter minor inconsistencies and bugs. But it is stable enough to get work done. There is an active and rapidly growing community around the language now, contributing useful modules and supporting new users. Clojure is like a breath of fresh air.

We are excited about building our new product on Clojure. Such unconventional choices are only possible when you are building your own product; but hey, that’s one good reason to do one. Enthusiasm is  vital ingredient in building products; they sprout from a variety of reasons. Like usefulness, financial rewards, challenges, technology, ..and programming languages like Clojure.

Jeswin P Kumar
(jeswinpk@my-company-name-is-AGILEHEAD.com)