Apress Introducing Dot Net 4 With Visual Studio_9 potx

CHAPTER 16 ■ LINQ: LANGUAGE INTEGRATED QUERY 561 using System; using System.Linq; public class GroupExample { static void Main() { int[] numbers = { 0, 1, 2, 3, 4, 5, 6, 7, 8, 9 }; // Partition numbers into odd and // even numbers. var query = from x in numbers group x by x % 2 into partition where partition.Key == 0 select new { Key = partition.Key, Count = partition.Count(), Group = partition }; foreach( var item in query ) { Console.WriteLine( "mod2 == {0}", item.Key ); Console.WriteLine( "Count == {0}", item.Count ); foreach( var number in item.Group ) { Console.Write( "{0}, ", number ); } Console.WriteLine( "\n" ); } } } In this query, the continuation (the part of the query after the into clause) filters the series of groups where Key is 0 by using a where clause. This filters out the group of even numbers. I then project that group out into an anonymous type, producing a count of items in the group to go along with the Key property and the items in the group. Thus the output to the console includes only one group. But what if I wanted to add a count to each group in the partition? As I said before, the into clause is a generator. So I can produce the desired result by changing the query to this: var query = from x in numbers group x by x % 2 into partition select new { Key = partition.Key, Count = partition.Count(), Group = partition }; Notice that I removed the where clause, thus removing any filtering. When executed with this version of the query, the example produces the following desired output: mod2 == 0 CHAPTER 16 ■ LINQ: LANGUAGE INTEGRATED QUERY 562 Count == 5 0, 2, 4, 6, 8, mod2 == 1 Count == 5 1, 3, 5, 7, 9, In both of the previous query expressions, note that the result is not an IEnumerable<IGrouping<T>> as it commonly is when the group clause is the final projector. Rather, the end result is an IEnumerable<T> where T is replaced with our anonymous type. The Virtues of Being Lazy When you build a LINQ query expression and assign it to a query variable, very little code is executed in that statement. The data becomes available only when you iterate over that query variable, which executes the query once for each result in the result set. So, for example, if the result set consists of 100 items and you only iterate over the first 10, you don’t pay the price for computing the remaining 90 items in the result set unless you apply some sort of operator such as Average, which requires you to iterate over the entire collection. ■ Note You can use the Take extension method, which produces a deferred execution enumerator, to access a specified number of elements at the head of the given stream. Similarly useful methods are TakeWhile, Skip, and SkipWhile. The benefits of this deferred execution approach are many. First of all, the operations described in the query expression could be quite expensive. Because those operations are provided by the user, and the designers of LINQ have no way of predicting the complexity of those operations, it’s best to harvest each item only when necessary. Also, the data could be in a database halfway around the world. You definitely want lazy evaluation on your side in that case. And finally, the range variable could actually iterate over an infinite sequence. I’ll show an example of that in the next section. C# Iterators Foster Laziness Internally, the query variable is implemented using C# iterators by using the yield keyword. I explained in Chapter 9 that code containing yield statements actually compiles into an iterator object. Therefore, when you assign the LINQ expression to the query variable, just about the only code that is executed is the constructor for the iterator object. The iterator might depend on other nested objects, and they are CHAPTER 16 ■ LINQ: LANGUAGE INTEGRATED QUERY 563 initialized as well. You get the results of the LINQ expression once you start iterating over the query variable using a foreach statement, or by using the IEnumerator interface. As an example, let’s have a look at a query slightly modified from the code in the earlier section “LINQ Query Expressions.” For convenience, here is the relevant code: var query = from employee in employees where employee.Salary > 100000 select new { LastName = employee.LastName, FirstName = employee.FirstName }; Console.WriteLine( "Highly paid employees:" ); foreach( var item in query ) { Console.WriteLine( "{0}, {1}", item.LastName, item.FirstName ); Notice that the only difference is that I removed the orderby clause from the original LINQ expression; I’ll explain why in the next section. In this case, the query is translated into a series of chained extension method calls on the employees variable. Each of those methods returns an object that implements IEnumerable<T>. In reality, those objects are iterators created from a yield statement. Let’s consider what happens when you start to iterate over the query variable in the foreach block. To obtain the next result, first the from clause grabs the next item from the employees collection and makes the range variable employee reference it. Then, under the covers, the where clause passes the next item referenced by the range variable to the Where extension method. If it gets trapped by the filter, execution backtracks to the from clause to obtain the next item in the collection. It keeps executing that loop until either employees is completely empty or an element of employees passes the where clause predicate. Then the select clause projects the item into the format we want by creating an anonymous type and returning it. Once it returns the item from the select clause, the enumerator’s work is done until the query variable cursor is advanced by the next iteration. ■ Note LINQ query expressions can be reused. For example, suppose you have started iterating over the results of a query expression. Now, imagine that the range variable has iterated over just a few of the items in the input collection, and the variable referencing the collection is changed to reference a different collection. You can continue to iterate over the same query and it will pick up the changes in the new input collection without requiring you to redefine the query. How is that possible? Hint: think about closures and variable capture and what happens if the captured variable is modified outside the context of the closure. Subverting Laziness In the previous section, I removed the orderby clause from the query expression, and you might have been wondering why. That’s because there are certain query operations that foil lazy evaluation. After all, how can orderby do its work unless it has a look at all the results from the previous clauses? Of course it can’t, and therefore orderby forces the clauses prior to it to iterate to completion. CHAPTER 16 ■ LINQ: LANGUAGE INTEGRATED QUERY 564 ■ Note orderby is not the only clause that subverts lazy evaluation, or deferred execution, of query expressions. group . . . by and join do as well. Additionally, any time you make an extension method call on the query variable that produces a singleton value (as opposed to an IEnumerable<T> result), such as Count, you force the entire query to iterate to completion. The original query expression used in the earlier section “LINQ Query Expressions” looked like the following: var query = from employee in employees where employee.Salary > 100000 orderby employee.LastName, employee.FirstName select new { LastName = employee.LastName, FirstName = employee.FirstName }; Console.WriteLine( "Highly paid employees:" ); foreach( var item in query ) { Console.WriteLine( "{0}, {1}", item.LastName, item.FirstName ); } I have bolded the orderby clause to make it stand out. When you ask for the next item in the result set, the from clause sends the next item in employees to the where clause filter. If it passes, that is sent on to the orderby clause. However, now the orderby clause needs to see the rest of the input that passes the filter, so it forces execution back up to the from clause to get the next item that passes the filter. It continues in this loop until there are no more items left in the employees collection. Then, after ordering the items based on the criteria, it passes the first item in the ordered set to the select projector. When foreach asks for the next item in the result set, evaluation starts with the orderby clause because it has cached all the results from every clause prior. It takes the next item in its internal cache and passes it on to the select projector. This continues until the consumer of the query variable iterates over all the results, thus draining the cache formed by orderby. Now, earlier I mentioned the case where the range variable in the expression iterates over an infinite loop. Consider the following example: using System; using System.Linq; using System.Collections.Generic; public class InfiniteList { static IEnumerable<int> AllIntegers() { int count = 0; while( true ) { yield return count++; } } static void Main() { CHAPTER 16 ■ LINQ: LANGUAGE INTEGRATED QUERY 565 var query = from number in AllIntegers() select number * 2 + 1; foreach( var item in query.Take(10) ) { Console.WriteLine( item ); } } } Notice in the bolded query expression, it makes a call to AllIntegers, which is simply an iterator that iterates over all integers starting from zero. The select clause projects those integers into all the odd numbers. I then use Take and a foreach loop to display the first ten odd numbers. Notice that if I did not use Take, the program would run forever unless you compile it with the /checked+ compiler option to catch overflows. ■ Note Methods that create iterators over infinite sets like the AllIntegers method in the previous example are sometimes called streams. The Queryable and Enumerable classes also contain useful methods that generate finite collections. Those methods are Empty, which returns an empty set of elements; Range, which returns a sequence of numbers; and Repeat, which generates a repeated stream of constant objects given the object to return and the number of times to return it. I wish Repeat would iterate forever if a negative count is passed to it. Consider what would happen if I modified the query expression ever so slightly as shown here: var query = from number in AllIntegers() orderby number descending select number * 2 + 1; If you attempt to iterate even once over the query variable to get the first result, then you had better be ready to terminate the application. That’s because the orderby clause forces the clauses before it to iterate to completion. In this case, that will never happen. Even if your range variable does not iterate over an infinite set, the clauses prior to the orderby clause could be very expensive to execute. So the moral of the story is this: be careful of the performance penalty associated with using orderby, group . . . by, and join in your query expressions. Executing Queries Immediately Sometimes you need to execute the entire query immediately. Maybe you want to cache the results of your query locally in memory or maybe you need to minimize the lock length to a SQL database. You can do this in a couple of ways. You could immediately follow your query with a foreach loop that iterates over the query variable, stuffing each result into a List<T>. But that’s so imperative! Wouldn’t you rather be functional? Instead, you could call the ToList extension method on the query variable, which does the same thing in one simple method call. As with the orderby example in the previous section, be careful when calling ToList on a query that returns an infinite result set. There is also a ToArray extension method for converting the results into an array. I show an interesting usage of ToArray in the later section titled “Replacing foreach Statements.” CHAPTER 16 ■ LINQ: LANGUAGE INTEGRATED QUERY 566 Along with ToList, there are other extension methods that force immediate execution of the entire query. They include such methods as Count, Sum, Max, Min, Average, Last, Reverse and any other method that must execute the entire query in order to produce its result. Expression Trees Revisited In Chapter 15, I described how lambda expressions can be converted into expression trees. I also made a brief mention of how this is very useful for LINQ to SQL. When you use LINQ to SQL, the bodies of the LINQ clauses that boil down to lambda expressions are represented by expression trees. These expression trees are then used to convert the entire expression into a SQL statement for use against the server. When you perform LINQ to Objects, as I have done throughout this chapter, the lambda expressions are converted to delegates in the form of IL code instead. Clearly that’s not acceptable for LINQ to SQL. Can you imagine how difficult it would be to convert IL into SQL? As you know by now, LINQ clauses boil down to extension method calls implemented in either System.Linq.Enumerable or System.Linq.Queryable. But which set of extension methods are used and when? If you look at the documentation for the methods in Enumerable, you can see that the predicates are converted to delegates because the methods all accept a type based on the Func<> generic delegate type. However, the extension methods in Queryable, which have the same names as those in Enumerable, all convert the lambda expressions into an expression tree because they take a parameter of type Expression<T>. Clearly, LINQ to SQL uses the extension methods in Queryable. ■ Note Incidentally, when you use the extension methods in Enumerable, you can pass either lambda expressions or anonymous functions to them because they accept a delegate in their parameter lists. However, the extension methods in Queryable can accept only lambda expressions because anonymous functions cannot be converted into expression trees. Techniques from Functional Programming In the following sections, I want to explore some more of the functional programming concepts that are prevalent throughout the features added in C# 3.0. As you’ll soon see, some problems are solved with clever use of delegates created from lambda expressions to add the proverbial extra level of indirection. I’ll also show how you can replace many uses of the imperative programming style constructs such as for loops and foreach loops using a more functional style. Custom Standard Query Operators and Lazy Evaluation In this section, I will revisit an example introduced in Chapter 14, in which I showed how to implement a Lisp-style forward-linked list along with some extension methods to perform on that list. The primary interface for the list is shown here: public interface IList<T> { T Head { get; } CHAPTER 16 ■ LINQ: LANGUAGE INTEGRATED QUERY 567 IList<T> Tail { get; } } A possible implementation of a collection based on this type was shown in Chapter 14; I repeat it here for convenience: public class MyList<T> : IList<T> { public static IList<T> CreateList( IEnumerable<T> items ) { IEnumerator<T> iter = items.GetEnumerator(); return CreateList( iter ); } public static IList<T> CreateList( IEnumerator<T> iter ) { if( !iter.MoveNext() ) { return new MyList<T>( default(T), null ); } return new MyList<T>( iter.Current, CreateList(iter) ); } public MyList( T head, IList<T> tail ) { this.head = head; this.tail = tail; } public T Head { get { return head; } } public IList<T> Tail { get { return tail; } } private T head; private IList<T> tail; } Now, let’s say that you want to implement the Where and Select standard query operators. Based on this implementation of MyList, those operators could be implemented as shown here: public static class MyListExtensions { public static IEnumerable<T> GeneralIterator<T>( this IList<T> theList, Func<IList<T>, bool> finalState, Func<IList<T>, IList<T>> incrementer ) { while( !finalState(theList) ) { yield return theList.Head; CHAPTER 16 ■ LINQ: LANGUAGE INTEGRATED QUERY 568 theList = incrementer( theList ); } } public static IList<T> Where<T>( this IList<T> theList, Func<T, bool> predicate ) { Func<IList<T>, IList<T>> whereFunc = null; whereFunc = list => { IList<T> result = new MyList<T>(default(T), null); if( list.Tail != null ) { if( predicate(list.Head) ) { result = new MyList<T>( list.Head, whereFunc(list.Tail) ); } else { result = whereFunc( list.Tail ); } } return result; }; return whereFunc( theList ); } public static IList<R> Select<T,R>( this IList<T> theList, Func<T,R> selector ) { Func<IList<T>, IList<R>> selectorFunc = null; selectorFunc = list => { IList<R> result = new MyList<R>(default(R), null); if( list.Tail != null ) { result = new MyList<R>( selector(list.Head), selectorFunc(list.Tail) ); } return result; }; return selectorFunc( theList ); } } Each of the two methods, Where and Select, uses an embedded lambda expression that is converted to a delegate in order to get the work done. ■ Note Chapter 14 demonstrated a similar technique, but because lambda expressions had not been introduced yet, it used anonymous methods instead. Of course, lambda expressions clean up the syntax quite a bit. CHAPTER 16 ■ LINQ: LANGUAGE INTEGRATED QUERY 569 In both methods, the embedded lambda expression is used to perform a simple recursive computation to compute the desired results. The final result of the recursion produces the product you want from each of the methods. I encourage you to follow through the execution of this code in a debugger to get a good feel for the execution flow. The GeneralIterator method in the previous example is used to create an iterator that implements IEnumerable on the MyList object instances. It is virtually the same as that shown in the example in Chapter 14. Finally, you can put all of this together and execute the following code to see it in action: public class SqoExample { static void Main() { var listInts = new List<int> { 5, 2, 9, 4, 3, 1 }; var linkList = MyList<int>.CreateList( listInts ); // Now go. var linkList2 = linkList.Where( x => x > 3 ).Select( x => x * 2 ); var iterator2 = linkList2.GeneralIterator( list => list.Tail == null, list => list.Tail ); foreach( var item in iterator2 ) { Console.Write( "{0}, ", item ); } Console.WriteLine(); } } Of course, you will have to import the appropriate namespaces in order for the code to compile. Those namespaces are System, System.Linq, and System.Collections.Generic. If you execute this code, you will see the following results: 10, 18, 8, There are some very important points and problems to address in this example, though. Notice that my query was not written using a LINQ query expression even though I do make use of the standard query operators Where and Select. This is because the from clause requires that the given collection must implement IEnumerable. Because the IList interface does not implement IEnumerable, it is impossible to use foreach or a from clause. You could use the GeneralIterator extension method to get an IEnumerable interface on the IList and then use that in the from clause of a LINQ query expression. In that case, there would be no need to implement custom Where and Select methods because you could just use the ones already implemented in the Enumerable class. However, your results of the query would be in the form of an IEnumerable and not an IList, so you would then have to reconvert the results of the query back to an IList. Although these conversions are all possible, for the sake of example, let’s assume that the requirement is that the standard query operators must accept the custom IList type and return the custom IList type. Under such a requirement, it is impossible to use LINQ query expressions, and you must invoke the standard query operators directly. CHAPTER 16 ■ LINQ: LANGUAGE INTEGRATED QUERY 570 ■ Note You can see the power of the LINQ layered design and implementation. Even when your custom collection type does not implement IEnumerable, you can still perform operations using custom designed standard query operators, even though you cannot use LINQ query expressions. There is one major problem with the implementation of MyList and the extension methods in the MyListExtensions class as shown so far. They are grossly inefficient! One of the functional programming techniques employed throughout the LINQ implementation is that of lazy evaluation. In the section titled “The Virtues of Being Lazy,” I showed that when you create a LINQ query expression, very little code is executed at that point, and operations are performed only as needed while you iterate the results of the query. The implementations of Where and Select for IList, as shown so far, don’t follow this methodology. For example, when you call Where, the entire input list is processed before any results are returned to the caller. That’s bad because what if the input IList were an infinite list? The call to Where would never return. ■ Note When developing implementations of the standard query operators or any other method in which lazy evaluation is desirable, I like to use an infinite list for input as the litmus test of whether my lazy evaluation code is working as expected. Of course, as shown in the section “Subverting Laziness,” there are certain operations that just cannot be coded using lazy evaluation. Let’s turn to reimplementing the custom standard query operators in the previous example using lazy evaluation. Let’s start by considering the Where operation. How could you reimplement it to use lazy evaluation? It accepts an IList and returns a new IList, so how is it possible that Where could return only one item at a time? The solution actually lies in the implementation of the MyList class. Let’s consider the typical IEnumerator implementation for a moment. It has an internal cursor that points to the item that the IEnumerable.Current property returns, and it has a MoveNext method to go to the next item. The IEnumerable.MoveNext method is the key to retrieving each value only when needed. When you call MoveNext, you are invoking the operation to produce the next result, but only when needed, thus using lazy evaluation. I’ve mentioned Andrew Koenig’s “Fundamental Theorem of Software Engineering,” in which all problems can be solved by introducing an extra level of indirection. 4 Although it’s not really a theorem, it is true and very useful. In the C language, that form of indirection is typically in the form of a pointer. In C++ and other object-oriented languages, that extra level of indirection is typically in the form of a class (sometimes called a wrapper class). In functional programming, that extra level of indirection is typically a function in the form of a delegate. 4 I first encountered Koenig’s so called fundamental theorem of software engineering in his excellent book co- authored with Barbara Moo titled Ruminations on C++ (Boston: Addison-Wesley Professional, 1996). [...]... functionality is at par with VB .NET with respect to working with dynamically typed objects To better illustrate what I am talking about, let’s consider a short example Suppose that you want to create a new Excel document with some text in the first cell Additionally, force yourself to use only the late bound IDispatch interfaces for the sake of the example If you are familiar with coding against Office... between the NET runtime and the COM object, translates reflection operations into IDispatch operations This allows you to reflect over a COM object that implements the IDispatch automation interface If you used VB .NET rather than C# 3.0, the experience would have been much more pleasant because VB .NET shields you from all the reflection work Now that C# 4. 0 offers dynamic type support in concert with the... Additionally, you might consider LINQ for Visual C# 2005 by Fabio Claudio Ferracchiati or Pro LINQ: Language Integrated Query in C# 2008 by Joseph C Rattz, Jr., both published by Apress In the next chapter, I will introduce one of the coolest new features added in the C# 4. 0 language It is the new dynamic type and it brings interoperability in C# to a level of parity with Visual Basic, among other things 576... potential confusion When programming in C#, you are usually programming against static NET types that might have been coded in C#, C++/CLI, and so on But what about when you have to interoperate with types created 1 The DLR is at the heart of NET- based dynamic languages such as IronPython and IronRuby It provides an environment within which it is easy to implement dynamic languages as well as add dynamic capabilities... through it within a debugger to get a better feel for the execution flow Thus, we have achieved lazy evaluation Notice that each lambda expression in each method forms a closure that uses the passed-in information to form the recursive code that generates the next element in the list Test the lazy evaluation by introducing an infinite linked list of values Before you can prove the lazy evaluation with an... comes to diagnosing problems at run time because you are presented with the same errors that you’re familiar with To illustrate this point, consider the following code that will not compile: class C { public void Foo() {} } static class EntryPoint { static void Main() { C obj = new C(); obj.Bar(); } } As you would expect, you end up with a compiler error The output looks like the following: error CS1061:... IDynamicMetaObjectProvider (which I will explain later in the section “Objects with Custom Dynamic Behavior”) • A plain old NET statically typed object For plain old NET objects, the call site uses reflection to bind to the proper member If the object is a COM object, it reflects over the RCW that acts as a NET proxy object to the COM object The RCW translates the reflection operations into the matching IDispatch... Doing work Ticks: 1858 Doing work Ticks: 1 845 Doing work Ticks: 1981 Doing work Ticks: 1853 Doing work Ticks: 18 34 Doing work Ticks: 1995 Doing work Ticks: 1887 I first call DoWork once through a static receiver to make sure the method is JIT compiled before I gather the numbers That way, the first tick count should not reflect any JIT compiler time Boxing with Dynamic Boxing is one of those areas... EntryPoint { static void Main() { dynamic d = 42 ; ++d; object o = 42 ; o = (int)o + 1; Console.WriteLine( "d = " + d + "\no = " + o ); } } In this example, you have a dynamic instance that contains an integer Behind the scenes, it is an object that boxes the integer value In the dynamic case, you can simply invoke the increment operator to modify the value within the dynamic object’s box Right after that,... resolution (a topic covered in the next section) ■ Note At one point during the development of C# 4. 0, this type of implicit conversion from dynamic expressions to reference types was called assignment conversion Therefore, if you read blogs and articles on the Internet written during the development of C# 4. 0, you might see references to that term Dynamic expression conversion comes into play in cases . VB .NET rather than C# 3.0, the experience would have been much more pleasant because VB .NET shields you from all the reflection work. Now that C# 4. 0 offers dynamic type support in concert with. offers dynamic type support in concert with the DLR, its functionality is at par with VB .NET with respect to working with dynamically typed objects. To better illustrate what I am talking about,. of the coolest new features added in the C# 4. 0 language. It is the new dynamic type and it brings interoperability in C# to a level of parity with Visual Basic, among other things. C H A