CHAPTER 3 ■ WORKING WITH CLOUD QUEUE AND BLOB STORAGE 68 For service-oriented applications built on a front-end architecture with distributed infrastructure, the immediate advantages to be taken from Azure Queue (in conjunction with Azure Blob storage) are related to scalability: • Back-end traffic measurement using length of queue: If an application is built on Azure Queue and needs to determine scalability based on back-end traffic, the application can do so by measuring the queue length. Since the queue length directly reflects the delay time of the back-end workload, a growing queue length indicates that the back-end servers cannot process the work fast enough, so increasing the back-end instances for the application would help the server to process the workload in the queue more quickly. In contrast, if the measurement of queue length is close to zero, it is an indication that the capacity of the server process may be more than is needed, and you can consider decreasing the back- end server instances to save system resources. Therefore applications can monitor the queue length to make resource usage more efficient and scale-up more smooth. • System module decoupling: Applications built based on the message publish/subscribe architecture are loosely coupled, which gives applications extreme flexibility for extension and scaling up. If all modules and all components in an application use the message queue to communicate with each other either in the front end or in the back end, an application can easily replace any component independently, adjust the workflow logic, and upgrade the features without either interrupting irrelevant components (the components from presentation UI layers, business logical layers, or data storage access layers) or recompiling the code. • Efficient resource management: Message-queue-based applications can manage system resource allocation more efficiently. System resources can be grouped and assigned into distinct queues according to their critical levels. For instance, components that consume large system resources, such as audio or video processing, can have their own dedicated queues for communication in order to reduce the impact on other components or processes in the system. • Buffering messages when traffic soars: Queue-based system architecture allows message buffering and delayed processing without data loss if the traffic flow suddenly soars. Azure Queue has been designed to support guaranteed message delivery, which frees applications from handling data persistence due to traffic burst or other message-delivery difficulties. Traditionally, an application needs to persist the inbound message if it cannot process the message due to the volume of the message. To build data persistence at runtime is very costly. Using the message-buffering feature provided by the Azure Queue tremendously reduces the cost of application development for persisting inbound messages and for resubmitting messages if there are errors during transmission. Usually, to use Azure Queue as a message bus an application needs to use Azure Blob storage as well in order to reduce memory usage and improve performance. The exercises provided in this chapter show examples of how to achieve that goal. CHAPTER 3 ■ WORKING WITH CLOUD QUEUE AND BLOB STORAGE 69 Azure Blob Storage Blob storage can be understood as special table storage in the cloud. Basically cloud Blob storage extends table storage and targets large amounts of data. The difference between the two types of cloud storage is: • Table storage uses PrimaryKey and RowKey to manage the tables, whereas Blob storage uses a storage container and a blob ID (GUID) to manage the storage. • Table storage stores all portable system-defined data types, such as characters, strings, texts, integers, float numbers, and XML in storage tables. Blob storage stores data in binary format as data chunks. Figure 3-1 shows the concepts of cloud Blob storage. Blob storage access is based on an account. An Azure account can create multiple Blob containers. A Blob container can be understood as a placeholder for a group of Blob storages. Blob storage can have metadata, which can be understood as a collection of header attributes. Blob storage can be partially updated and committed using block objects. Each Blob can own a set of metadata in NameValueCollection string format. The Microsoft specification for blob storage can be found in Appendix A. Azure Blob Container XML Images Musics Movies User Account Blob Storage Metadata Figure 3-1. Cloud blob storage structure concepts The data object model of Azure Blob storage is shown in Figure 3-2. (This data map is generated using SQL Server Management Studio against the DevelopmentStorageDb on a local machine. The DevelopmentStorageDb database will be created the first time development storage is accessed and the local development storage is initialized by the Azure framework.) This database map actually reflects the concept diagram of Figure 3-1 and the relationship among the participants of all Blob storage tables. For example, a Blob container has an ID as its primary key, which is a foreign key of the Blob table. As I have mentioned, the local development environment and runtime environment fabric simulate the remote cloud environment. In the Azure cloud environment a physical table may have multiple instances running from distinct remote virtual machines. CHAPTER 3 ■ WORKING WITH CLOUD QUEUE AND BLOB STORAGE 70 Figure 3-2. Object models of blob storage Creating a Cloud Queue This exercise shows the basic steps to create a cloud queue, and how to put messages in and retrieve messages from queues. A queue can be created from either a worker role or a web role. In order to explore the collaboration mechanism between a web role and worker role in a cloud application, this exercise creates a queue from a worker role. ■ Note The code for this example is in the Exercise 3-1 bundle from the code download. Add a Worker Role in the Cloud Service Solution Add a worker role and associate that role to the service, as Figure 3-3 shows. The purpose of this approach is to demonstrate that the queue message can be manipulated from different processes. The responsibilities of this worker role are defined as: • Retrieve account information from the configuration. • Create a named queue storage container from cloud storage. • Create a named queue within the queue storage. CHAPTER 3 ■ WORKING WITH CLOUD QUEUE AND BLOB STORAGE 71 These responsibilities will be implemented in one function and called by the start handler of the worker role when the service application starts from the Azure fabric. Figure 3-3. Insert a worker role for queue initialization Create a Queue Storage Container and Create a Named Queue from the Start Handler of the Worker Role The data object model for Azure Queue storage is shown in Figure 3-4. The AcctQueueContainerMap table is used to map a queue name to the unique queue ID. This ID is used as a foreign key in the QueueContainer table and the Message table. These tables will be generated using the same approach we used in Chapter 1 to create the data entity C# classes; that is, we'll use Visual Studio to generate them. Figure 3-4. The data object model of Azure Queue All queue names must be alphanumerical characters and are case-sensitive as well. Characters only in lowercase are accepted for a queue name. A QueueStorage instance needs to be instantiated before a named queue can be created. The constructor of QueueStorage accepts an account information object as a parameter. As is the case when CHAPTER 3 ■ WORKING WITH CLOUD QUEUE AND BLOB STORAGE 72 creating table storage, an account with a base-64 security key and storage end points is assigned at the time you register a storage type project on the Azure portal page. In the development environment the account information can either be hard-coded in the configuration file or entered programmatically. Listing 3-1 shows the configuration settings from the configuration file for queue storage. Listing 3-1. Account and HTTP Port End Point Configuration for Queue Storage in the Development Environment <appSettings> <add key = "AccountName" value="devstoreaccount1"/> <add key = "AccountSharedKey" value="<ACCOUNT KEY>"/> <add key="QueueStorageEndpoint" value="http://127.0.0.1:10001" /> </appSettings> Create the Queue Using Account Information from the Configuration File Listing 3-2 is the implementation of the worker role project. The lines in bold show an example of how to create a named queue. Listing 3-2. Implementation of WorkerRole Shows How to Create Queue Using Configuration Files using System; using System.Threading; using Microsoft.ServiceHosting.ServiceRuntime; using Microsoft.Samples.ServiceHosting.StorageClient; using System.IO; using System.Configuration; using System.Net; using System.Xml; namespace CloudTableStorageService WorkerRole { public class WorkerRole : RoleEntryPoint { public const string XML PAYLOAD QUEUE NAME = "createxmlmessagequeue"; public const string XML CONTAINER NAME = "xmlpayload"; private Stream CreateXmlStreamBlob(byte [] byteData) { return new MemoryStream(byteData); } public override void Start() { QueueStorage queueStorage = QueueStorage.Create(StorageAccountInfo .GetDefaultQueueStorageAccountFromConfiguration()); MessageQueue queue = queueStorage.GetQueue(XML_PAYLOAD_QUEUE_NAME);