Technobabble: The official AgileHead blog

Productivity Techniques for .Net Application Development

Sun, 24 Oct 2010 13:53:00 -0400

In a software company, competitiveness has a linear co-relation to productivity. The huge variance in productivity at different companies let better companies run circles around slower ones. They innovate more, they are more profitable, and they get more customers.

It is interesting to look at the situation in India; one of the top destinations for IT projects. I have been working here for about a decade now. Since the early days, programmer salaries have seen at least a 4-fold increase. Competent programmers in India are no longer an order of magnitude less expensive than their western counterparts. Indian IT Services companies can no longer afford to throw large teams at problems, ignoring team productivity statistics. At the same time, a modest improvement in productivity can significantly contribute to the bottom line.

But indeed, productivity is a truly global challenge in software engineering. In the context of the .Net development platform, this article looks at using recent technological innovation to dramatically improve productivity in IT projects.

The Big Leap

Enterprises and IT Services companies have predominantly used Java or Microsoft platforms for building Line of Business applications. Within these two platforms, programming idioms have been more or less the same since C# was largely based on Java. While there were improvements in tooling, language capabilities and expressiveness had not changed much. Tooling can only improve productivity to an extent, before hitting the limitations of the programming language.

But the main .Net languages C# and VB.Net made huge leaps over the last couple of years.. So much so that it eclipses everything done till that point. Microsoft borrowed many concepts from Functional Programming and the academia, which suddenly opened up new ways to express logic. The techniques described in this article are based on these language innovations.

Three Pillars of productivity

1. Declarative Programming

I will borrow Wikipedia’s definition of Declarative Programming:
Declarative Programming is a programming paradigm that expresses the logic of a computation without describing its control flow. Many languages applying this style attempt to minimize or eliminate side effects by describing what the program should accomplish, rather than describing how to go about accomplishing it.

Here is an example:

Example 1: Imperative (typical)var collection = new List<int> { 1, 2, 3, 4 };
var oddNumbers = new List<int>();
foreach (var num in collection)
{
    if (num % 2 != 0)
        oddNumbers.Add(num);
}

Example 2: Declarativevar collection = new List<int> { 1, 2, 3, 4 };
var oddNumbers = collection
    .Where(num => num % 2 != 0);

In Example 1 (left), the task of finding odd numbers is accomplished by looping through the collection and adding matching entries to another list. But Example 2 simply defines what an odd number is. That is, it should not be divisible by two!

The difference is very fundamental. When applied to a business function, an imperative approach specifies how to execute the business function while a declarative approach merely defines the business function.

Consider some real business logic, written in imperative style:

//Get Customers who deposited money in 2010;
public IEnumerable<Customer> GetCustomers_With_RecentDeposits()
{
    var customers = new List<Customer>();
    var conn = new SqlConnection(CONN_STR);
    var cmd = new SqlCommand(“SELECT * FROM Customers C JOIN Deposits D ON “ +
        “D.CustomerId = C.Id WHERE D.Year = 2010”, conn);
    //Lengthy code to retrieve the dataset and create customers omitted.
    return customers;
}
//Get Indians who deposited money in 2010;
public IEnumerable<Customer> GetIndianCustomers_With_RecentDeposits()
{
    var customers = GetCustomers_With_RecentDeposits();
    var indians = new List<Customer>();
    foreach (var cust in customers)
    {
        if (cust.Country == “India”)
            indians.Add(cust);
    }
    return indians;
}

Here is the same example, written in a declarative manner using LINQ

//Get Customers who deposited money in 2010;
public IEnumerable<Customer> GetCustomers_With_RecentDeposits()
{
    var customers = from customer in Db.Customers()
                    where customer.Deposits.Any(d => d.Year == 2010)
                    select customer;
    return customers.ToList();
}
//Get Indians who deposited money in 2010;
public IEnumerable<Customer> GetIndianCustomers_With_RecentDeposits()
{
    return GetCustomers_With_RecentDeposits()
        .Where(customer => customer.Country == “India”);
}

With the declarative approach, we simply specified what we wanted and left it to the runtime and compiler to decide how to go about doing it.

2. Type-safe, Verifiable Code

One of the biggest benefits of using a LINQ query as in the example above is that they can be verified for correctness at compile time. The LINQ query is a data structure built using the types referenced in the query and can be checked for type-safety. As a result, if a field name or data type was incorrectly specified in the query the compiler would immediately complain and report the mismatch. In contrast, if the SQL string in the imperative approach referred to an incorrect column name or an invalid relationship, the error is only detected when the application is run.

Of course, Type-Safety is not just about queries. All code should be type safe; strings should only be used to in Resource files or in messages displayed to the user.

Here are more examples:

//Avoid the dictionary, use a strongly typed view
ViewData[“Username”] = “jeswin”;

//Strings used for column names!.
customer.FirstName = (string)reader[“FirstName”];
customer.LastName = (string)reader[“LastName”];
customer.Age = (int)reader[“Age”];

Being able to identify bugs at compile time makes a huge difference in productivity. With such code, the programmer is able to change code without worrying about its impact on other parts of the application.

In addition to this, making code analyzable by the compiler through strong typing allows for easier refactoring. For example, renaming a Field in an Entity (used in LINQ queries) is as easy as right-clicking and selecting ‘Rename’. The IDE will automatically change all LINQ queries to use the new Field name.

3. Composable Business Logic

One of the biggest advantages of LINQ is that it makes queries ‘composable’. In other words, simpler queries and business logic can be combined to form more complex queries.

This is better explained with an example. Assume we are writing and banking application and there is a function called GetHighNetWorthCustomers() which lists people with a large account balance. Say we want to find high net worth customers who also have a PREMIUM plan.

If we are not using LINQ for data access, we would probably have two Stored Procedures or SQL Statements which fetch this data from the database. An oversimplified version of this might look like:
High Net-Worth: SELECT * from Customers WHERE Spending > 100,000
High Net-Worth and PREMIUM: SELECT * from Customers WHERE AccountType = ‘PREMIUM’ AND Spending > 100,000

As we see, SQL doesn’t yield to easy re-use. They sit in isolated form, each confined to a separate function. Not re-usable, Not ‘composable’.

By using LINQ for Data Access, such logic can be made composable:

public IQueryable<Customer> GetHighNetWorthCustomers() {
    return Db.Customers().Where(customer => customer.Balance > 1000000);  
}
//High Net-Worth customers with a PREMIUM Plan
public IQueryable<Customer> GetRichPremiumCustomers() {
    return GetHighNetWorthCustomers().Where(customer => customer.AccountType == “PREMIUM”);
}

It was straight forward to re-use the existing query which fetched High Net-Worth customers. In a large application, such composability will dramatically improve productivity. Also, in line with the techniques discussed earlier it is expressed declaratively, and is type-safe and verifiable. Not to mention, a lot more concise.

Finally, here is how to get Rich Indians:

var richIndians = GetRichPremiumCustomers().Where(customer => customer.Country == “India”);

Composability techniques with Extension Methods

There are people who prefer to use extension methods to compose functions. This provides for slightly more concise code. Whether to use this or not is a matter of taste.

public static class CustomerModule
{
    public static IQueryable<Customer> HighNetWorth(this IQueryable<Customer> customers) {
        return customers.Where(customer => customer.Balance > 1000000);
    }
    public static IQueryable<Customer> Premium(this IQueryable<Customer> customers) {
        return customers.Where(customer => customer.AccountType == “PREMIUM”);
    }
    public static IQueryable<Customer> Indians(this IQueryable<Customer> customers) {
        return customers.Where(customer => customer.Country == “India”);
    }
}

This allows us to combine these functions in a very elegant manner:

var richIndians = Db.Customers().HighNetWorth().Premium().Indians();

Refactor-ability

A few months or a few sprints into a project, teams are generally hesitant to fix less optimal code written earlier in the project for fear of introducing bugs and having to re-test the entire application. This should not happen; inefficiencies and bugs multiply exponentially as more code is written on top of the problematic parts.

The reluctance on the part of developers to fix problems can only be avoided if the cost of making changes are manageable. For example, half way through a project splitting the “CustomerName” column in Customers table into “FirstName” and “LastName” is many hours of work since the entire application might have to be retested. With the new approaches we discussed, this is likely to be 30 minutes of work. It is important to keep your codebase Refactoring-friendly and Agile.

Last words

My company provides architectural and technical consulting services to various IT Services companies in India. Our first-hand experience with a variety of projects in these companies tells us that a significant number of problems faced by teams can be eliminated by using the methods discussed in this article. In our opinion, recent changes to .Net languages represent the most significant mainstream language innovation since the advent of object oriented programming many years ago. The question is how quickly companies will adapt and adopt them.

Especially for IT Services companies in India, the future is moving away from offering lower cost talent to high quality engineering expertise.

Jeswin Kumar
AgileHead

AgileHead offers architectural consulting services and training to IT companies in India. You can email us at services@agilehead.com.

AgileHead launches new Training Courses (.Net Platform)

Wed, 06 Oct 2010 09:28:00 -0400

We happy to announce our corporate training program for IT Services companies based in India. We will have four tracks available at launch, targeting Technical Leads and Architects working on the .Net Platform.

T1: Rapid Development with Asp.Net MVC & Entity Framework

Level: Intermediate-Advanced
Duration: 16 Hours

This two day session targets Intermediate-Advanced level programmers, and focuses on rapidly developing Web Applications using Asp.Net MVC.

Day 1: (8 hours)

Introduction to Asp.Net MVC
N-Tier Applications
Project Structure and Code Layout
Designing the Business Logic Layer
- Entity Modeling
- Data Access Techniques
View and Controllers
Introduction to ADO.NET Entity Framework (optional: LINQ to SQL)

Day 2: (8 hours)

Writing Semantic Code
- Optimal use of LINQ
- Fluent Interfaces
Using jQuery
Security Best Practices
Web Services and REST-ful services
Unit Testing with Asp.Net MVC
Improving Code Quality
- Writing Verifiable Code
- Automation of Error Checking
Deployment Scenarios
Scalability and Performance
Introduction to Windows Azure Cloud
Examples

T2: Advanced Entity Framework and LINQ to SQL

Level: Intermediate-Advanced
Duration: 16 Hours

This two day session targets Intermediate-Advanced level programmers, and focuses on mastering ADO.Net Entity Framework with real world applications. This course can be customized for specific scenarios and projects.

Day 1: (8 hours)

Introduction to LINQ
Advantages of LINQ over conventional techniques
LINQ to SQL
ADO.NET Entity Framework v4
N-Tier Applications
Entity Modeling with the Entity Designer
Model First Approach
Using POCO and Templating
Entity Framework Internals
Debugging and Optimizing Queries

Day 2: (8 hours)

Using Stored Procedures
Performance and Scalability Considerations
Building a Web App with Asp.Net MVC
Building WCF Services
Improving Code Quality
- Writing Verifiable Code
- Automation of Error Checking
Writing Unit Tests
Migrating applications to use Entity Framework
Examples

T3: Windows Azure Cloud Platform

Level: Intermediate-Advanced
Duration: 16 Hours

This two day session introduces developers to the Windows Azure cloud platform. It takes them through an overview of fundamental concepts and then into developing and deploying applications on Microsoft Windows Azure.

Day 1: (8 hours)

Introduction to Cloud Platforms
Virtualization Concepts
Platform as a Service (PaaS)
Windows Azure Fundamentals
Comparison with Amazon AWS, Google App Engine
Developing .Net Applications for Windows Azure
Testing Cloud Applications
Deploying a Simple Asp.Net MVC Web Application
Entity Modeling with Entity Framework or LINQ to SQL
SQL Azure

Day 2: (8 hours)

Developing a full-fledged Business Application
Windows Azure Storage Services
- Blob Service
- Queue Service
- Table Service
Enhanced Scalability with Non-Relational Databases
Deploying and Managing Windows Azure Applications
Application Security
Scalability and Performance Considerations
Windows Azure Billing and Pricing Models
Examples

T4: Improving productivity with better engineering (for .Net Technical Architects)

Level: Advanced
Duration: 8 Hours

This one day session targets Architects and discusses various techniques for improving productivity and code quality within their teams.

Day 1:

Typical Problems faced by Teams
Agile Processes and Agile Code
DRY (Don’t Repeat Yourself) Principles
Convention over Configuration
Declarative and Functional Programming
Object Oriented Design Principles
Composable Business Logic
- Fluent Interfaces
Writing Statically Verifiable Code
Avoiding over-engineering
- Knowing Computational Complexity and I/O Costs
Semantic HTML and CSS
Advanced Javascript
Designing for Continuous Refactoring
Automated Testing
- via Setup Scripts
- Unit Test Frameworks
Continuous Integration
Conventions for Modern Database Design
Examples

From C# on Mono, to Clojure on the JVM

Tue, 26 May 2009 06:52:00 -0400

Last week, we had to take a decision that reminded us of a similar situation we faced three years back. On which platform and language will we develop back-end services?

Going with Mono, in 2006

In early 2006, we were working on a fairly large Social Networking project which included building a few long running processes. We decided to build these in the popular, but not yet widely deployed platform called Mono. Two factors tilted the decision in favor of Mono, the most important being that we were building the application on Linux, and the second being that we had decided to use a mainstream object-oriented, statically-typed language for building these components. That leaves us with Java and C#; while Java was left in the quagmire of community-driven, slow design process, C# was getting increasingly productive leaving Java behind in terms of expressiveness. By 2006, Mono had an excellent C# compiler, an efficient runtime, and good coverage of the .Net Base Class Libraries.

Though the project eventually failed due to non-technical reasons, Mono itself served us well for 3 years. Over that period, Mono improved by leaps and bounds. Compatibility is great with Net 2.0 libraries being almost entirely supported, compilers are exceptional, MonoDevelop 2.0 has become a capable IDE, and performance is comparable to Microsoft .Net itself. In specific benchmarks, the Mono runtime even bests the official .Net runtime. Often, the mono compiler supports upcoming C# features ahead of the official Microsoft release.

We could say that the “Mono Project” has done well over the years. I actually admire the Mono Team; they have faced unnecessary criticism within the Open Source community, often by people who knew little about the project. That did not stop them from building a great product.

However, my confidence in Mono has been waning for some time.

In the Shadows of .Net

Back in 2006, we put our trust in Mono because we refused, or perhaps disliked, to vilify a project solely because it emulated something created at Microsoft. While Open Source backers generally dislike Microsoft technology, with Mono they had another argument that being a clone it could be affected by a number of patents that Microsoft holds related to the .Net framework. This point often comes up in debates about the “safety” of the Mono project, the defense of Mono being that large parts of the .Net specification are an open, published ECMA standard. I sided with the Mono supporters then, downplaying the risk of patents from Microsoft. But then in November, Microsoft and Novell announced their Patent Agreement, which guarantees patent protection exclusively for users of Novell Linux. The Mono project is largely supported by Novell, and such an agreement is disastrous for a community project like Mono. At this point, the fence-sitters in the Open Source community largely crossed over to the anti-Mono camp. Perhaps, they were justified in doing so. I could no longer defend Mono, and my belief in the framework getting wider acceptance has diminished significantly since then.

One might wonder why acceptance or ideological reasons should affect the platform we choose to build a product on. It just needs to work well, right? It is not that easy. Today my company works with customers on Microsoft and non-Microsoft platforms. We have skills on Microsoft and Open Source technologies, and while consulting or providing services to our customers we choose the right tool for the job irrespective of our ideological leaning. But when building our own products, we are convinced about the advantages of Open Source. And as a company, we see Open Source as a principle and ideology – not just for its commercial benefit. Building a new product requires a lot of sacrifice, stamina, coffee and redbull. And you will only build it on a technology you believe in.

There are other minor issues that make Mono a stranger in Open Source land. Mono uses very Windows specific conventions, like “.exe” for executables and “.dll” for libraries. For non-windows users some of these conventions look odd. Again, this is not Mono’s fault; rather a consequence of emulating a platform designed for a very different Operating System.

It is entirely possible that Mono can suddenly gain acceptance if Microsoft decides to relinquish patent claims regarding the .Net framework. If it happens, .Net and Mono could well become an powerful challenger to the dominance of Java. This is very unlikely, Microsoft’s current strategy seems to be relying strongly on patents and IP to ward off the looming threat from Linux.

For now, we decided to look beyond Mono and C#.

What about Python?

In the last two years, I have mostly used Python with Django for web development. It is easy to learn and to apply. Documentation is plentiful, and there are libraries for getting most things done. While Python is easy and practical, there was nothing quite remarkable about it. I felt Python’s strengths were not a consequence of well thought-out language design; instead it was the flexibility inherent in the dynamic nature of the language.

The language itself seems to be going backwards. I do not deny that I have an affection for functional-style programming, but its relevance cannot be overlooked anymore. In a variety of situations, Functional Programming (FP) is more concise, and it is also more suited for the dawning Parallel Programming era. For this reason alone, languages like C# are incorporating functional concepts in future versions. But not Guido; Python’s benevolent dictator has a well known dislike for functional style programming. Guido deserves credit for creating a very popular programming language, but you wonder if he is paying attention to the evolution of contemporary programming languages.

Some people say that Python’s legacy will be like that of Visual Basic; if it ever gets that popular. I doubt whether it will be as popular as “VeeBee”, but I don’t disagree. There are more capable languages out there. If we ever accomplish interoperability across dynamic languages, I predict that Python would have a bleak future. Why aren’t the Parrots speaking, yet?

The JVM

Java is the most common language for building applications today; due to its maturity, multi-platform support, proven reliability and the huge-huge ecosystem built around it. But Java itself is a really sucky language. It is boring to write code in Java, and even simple tasks tend to be very lengthy. Steve Yegge’s ‘Kingdom of Nouns‘ captures this verbosity in a humorous style.

The JVM is however, quite an interesting platform. While originally designed for the Java language, it is now the target of many new languages. Some of these languages have been around for some time now and have gained some momentum, like Scala, JRuby, Groovy and Clojure. Groovy has not caught on as much as the others; so it fails my “excellent community” criteria. JRuby’s biggest problem is Ruby itself; rockstars are unlikely to use the JVM.

But Scala is impressive; has great interop with Java, has a growing community around it, and recently acquired a really high-profile user – Twitter. If you have been programming in Java or C#, Scala is a wonderful alternative. For Java developers, Scala offers a leap in productivity and the ability to call and get called by Java code. If you are on the Microsoft stack and hate the JVM because you hate Java, Scala offers the expressiveness that was recently introduced into C#.

But I knew Scala wasn’t what I was looking for. I actually knew what I wanted; I wanted a language with few syntactic rules and constructs. I wanted to move to a language I do not have to discard any time soon. I wanted something like the ‘Hundred Year Language‘.

LISP will win, eventually

LISP is a bit like our company’s motto: “Simplicity is the ultimate sophistication.” – originally by Leonardo DaVinci. The language itself is so simple that you could learn the semantics in less time than it takes to read this article. This simplicity is achieved by abstracting ideas in programming languages to the extreme; yielding a simple form which is infinitely extensible. This extensibility comes from the ability to treat a program as data, and manipulate the program itself, using a feature called Macros.

While the syntax of LISP is very simple and elegant, writing a typical business or web application in LISP is (or was) much harder; perhaps an order of magnitude harder. This is especially true for newbies; unless they are really determined it is difficult to find documentation and useful libraries. This happened to me as well. I had read many books on LISP, but I did not have the confidence to execute a real project in LISP. It has been done before by so many others, but there is a big challenge in transitioning from a theoretical understanding of LISP and getting it to work in practice.

Although I was unable to use LISP in any production code, what I learnt from LISP profoundly impacted the way I saw programming. And since then, I have been keeping a close watch on upcoming languages bearing any resemblance to LISP. One such language was Clojure.

Clojure, the acceptable LISP

Clojure is a dialect of LISP and a dynamic programming language, running on the JVM. It has all the benefits of LISP, including the powerful macro system. It also has useful features which help with parallel programming; like Software Transactional Memory for dealing with shared state without worries. Clojure makes some of LISP’s biggest problems go away. Lack of Libraries and other supporting infrastructure like Application Servers. Java has libraries for everything, from connecting to SAP to talking to Twitter. And you could deploy on robust servers like Tomcat.

Everything ever built for Java can be used with Clojure.

Clojure is very young, and like any new programming language you would encounter minor inconsistencies and bugs. But it is stable enough to get work done. There is an active and rapidly growing community around the language now, contributing useful modules and supporting new users. Clojure is like a breath of fresh air.

We are excited about building our new product on Clojure. Such unconventional choices are only possible when you are building your own product; but hey, that’s one good reason to do one. Enthusiasm is vital ingredient in building products; they sprout from a variety of reasons. Like usefulness, financial rewards, challenges, technology, ..and programming languages like Clojure.

Jeswin P Kumar
(jeswinpk@agilehead.com)

These comments were imported into Tumblr from the original post.

Joseph Gutierrez said at 10:23 AM on June 13, 2009 :
What IDE do you plan on using?
jeswinpk said at 11:31 AM on June 13, 2009 :
enclojure looks more friendly to me, but perhaps that’s only because I am not so familiar with emacs.
KL said at 1:04 PM on June 13, 2009 :
enclosure looks very cool.

For those people who really like Eclipse, there is the clojure feature clojure-dev.

Also, you might consider looking at the “D” language for applications that will need to connect to native C/C++ libraries, e.g. desktop apps. Where Clojure feels like “Lisp done right”, D is “C++ done right”.
Tony said at 1:26 PM on June 13, 2009 :
Hi Jeswin, Clojure and Scala are really nice languages for sure but fall way behind Groovy in terms of industry take up. In a recent survey in Germany, 30% of Java shops are now using Groovy. If you add up use of Scala, JRuby, Jython and Clojure they are in use in only about 10% of Java shops combined. I haven’t done the numbers but I think it is about the same here in Australia. Language choice is a very subjective thing, so I encourage people to choose what they prefer - just providing you with some feedback on your perceptions of the Groovy community. Cheers, Paul.
Paul King said at 4:30 PM on June 13, 2009 :
Hi Jeswin, Clojure and Scala are really nice languages for sure but fall way behind Groovy in terms of industry take up. In a recent survey in Germany, 30% of Java shops are now using Groovy. If you add up use of Scala, JRuby, Jython and Clojure they are in use in only about 10% of Java shops combined. I haven?t done the numbers but I think it is about the same here in Australia. Language choice is a very subjective thing, so I encourage people to choose what they prefer - just providing you with some feedback on your perceptions of the Groovy community. Cheers, Paul.
jjames said at 6:38 PM on June 13, 2009 :
Lots of smart hackers are already using Clojure.
Steve Kos said at 7:40 PM on June 13, 2009 :
I’m puzzled.

A few years ago, you took a risky bet and decided to base your company on an obscure and potentially dangerous technology.

That project failed for exactly the reasons you suspected when you made that decision.

And now you want to make the same mistake by basing your company on yet another bleeding edge technology?

If seems to me you are more interested in experimenting with fun software stuff than building a sound business.

As for the hundred year language, I?m betting that C# or Java is probably in a better position to achieve that goal than any of the other languages you listed in this post.

Good luck, though, I admire your bravado :-)
jeswinpk said at 8:17 PM on June 13, 2009 :
@Steve

Just like most other failures, ours had little to do with our choice of technology. I was suggesting that the choice of technology is a small company (or a small group of programmers), serves to drum up interest and enthusiasm.

As long as the interface remains the same and the app is fast, I don’t think users will care. And as I was saying, those Mono services never needed restarts ? so Mono really served us well as far as the technology is concerned.
Jacques Chester said at 11:10 PM on June 13, 2009 :
Nitpick:

It’s generally referred to as “Lisp”, not “LISP”.
mark said at 7:09 AM on June 14, 2009 :
Cobol will be the 100 year language first, it’s already 50….

:D
Thomas said at 10:35 PM on June 14, 2009 :
Strictly speaking, both Fortran and Lisp are older then Cobol….
Amr said at 5:18 AM on June 15, 2009 :
you can still stick with Mono though. Boo and Nemrle deserves your attention !
Thomas said at 11:13 PM on June 22, 2009 :
Lisp will not win!
(Too (many) pa(r) ent (hese) s)

There has to be a reason why Lisp didn’t pick up since when it was introduced…. readability.

Parens indicate in literature (a side thought) or something that can be skipped.

It is just not natural to express 2+2 as (+(2,2)) or whatever, in code.

IMHO, it will forever remain a cult language. Something like Sanskrit, Latin etc - good to know and helpful to understand the finer things in (coding (can’t resist)) life!
Chris said at 1:55 AM on June 23, 2009 :
And what about F# ?
Nick B said at 8:59 PM on June 29, 2009 :
> Also, you might consider looking at the “D” language for applications that will need to connect to native C/C++ libraries, e.g. desktop apps. Where Clojure feels like “Lisp done right”, D is “C++ done right”.

To be precise - check out “D programming language” & “TANGO” (the framework).