Windows Store App to monitor your Azure bill

Windows Store App to monitor your Azure bill

Introducing WIndows Azure Media Services App for SharePoint

http://blog.appliedis.com/2012/12/14/introducing-media-center-app-for-sharepoint-2013/

SharePoint App Dev Platform: The Journey So Far & the Road Ahead

Our SharePoint journey so far and a look ahead

http://blog.appliedis.com/2013/01/02/sharepoint-app-dev-platform-the-journey-so-far-the-road-ahead/

Netizen is now available in the Windows Store!

http://blog.appliedis.com/2012/12/28/introducing-netizen-for-windows-8/

 

Netizen – Windows 8 app that makes it easy to follow congressional votes

( this post is a response to ACT4Apps questionnaire posted by ACT and AT&T )

As a software developer who is also very interested in public policy, I am always looking to build apps that make it easier for folks to follow the public policy discourse. This is why I decided to build Netizen.   Netizen  is a Windows 8 app (soon to be  available in the Windows Store for free) that brings the voting record of your congressional representative to your finger tips. Simply select the Member of Congress you want to follow and "flick through" their voting record.

But I don’t want users of Netizen to merely follow how their  representative are voting in congress, instead, I want them to ensure that their voices are being heard.  For each bill brought to the floor of the house, Netizen automatically provisions a Facebook page dedicated to your member of congress. This page acts as a virtual ballot for a bill, as well as, a community hub where fellow constituents can gather to express their support. If enough of your friends and neighbors like or unlike the a vote, your congressional representative is sure to pay attention.

Pasted below are some screenshots from the application:

Screenshot #1 – Select state or find your congressional district based on your current location

Screenshot #2 – Select Representative

Screenshot #3 – “Flick through” the most recent roll call votes

Screenshot #4 –  “Virtual ballot” – Dynamically generated Facebook page,

Screenshot #5 –  Search for a representative

I am working on making this app available on other devices as well. Unfortunately the fragmented app landscape makes it harder for hobbyists and self-funded projects such as Netizen to publish across multiple app stores. If any of you are interested in helping develop versions of this app that run on other mobile platforms, please contact me via my twitter handle @vlele.

I hope that by the time CES 2022 rolls around, we can build an electronic, friction free  voting system that allows anyone with a tablet or a smart phone to securely cast a vote on the important issues of the day. Such a capability, combined with a vote  counting system that leverages the seemingly infinite computational resources available to us today, in order to make the voting results available in a snap. Perhaps that is the only hope to make law-making work again.

Finally, in order for sustain continued innovation in the app development, especially for hobbyists and startups, we need to strike a balance between regulation and room for innovation. Case in point, some of the recent changes proposed to the privacy laws for apps can lead to unintended consequences for the overall app economy. 

IaaS and PaaS Together: A Video Walkthrough of Hybrid Scenarios

Short video that walks through IaaS & PaaS hydrid scenarios:

• Using Windows Azure Virtual Network to provision a VPN to connect our on-premised infrastructure with a Windows Azure datacenter.
• Set up front-end and back-end subnets.
• Provision a set of Azure IaaS Virtual Machines and Azure Web Roles.
• Install System Center Monitoring Pack for Windows Azure Applications on Azure-based machines.
• Install System Center Operations on-premises in order to manage Azure-based resources.

http://blog.appliedis.com/2012/11/19/iaas-and-paas-together-a-video-walkthrough-of-hybrid-scenarios/

20 Things That May Be ‘Clouding’ Your Choice About the Cloud, But Shouldn’t

This paper covers 20 of the most common and not-so-common objections in moving software applications to the cloud. It highlights some of the most recent and up-to-date advancements in cloud technology that address these concerns head-on.

AIS Top 20 Reasons Clouding Cloud Adoption White Paper (PDF)

Application Server and Integration Server: Growing Closer Together

 

This blog post is about the evolution of Microsoft’s application server – a collection of products including IIS and .NET framework libraries such as WCF and WF. Over the last few releases Microsoft has added 1) runtime container and activation service, via WAS; 2) multi-protocol support that goes beyond Http, via IIS 7.0; and 3) manageability extensions and caching, via AppFabric. Some of these enhancements overlap with the capabilities offered by Microsoft’s flagship integration server, BizTalk. BizTalk is well known for its robust hosting and integration capabilities. This overlap has stirred conversation in blogs and news stories over the last several months.

My goal in this post is to step back and look into some of the reasons causing application and integration servers to grow closer; we will do a side-by-side comparison of a scenario that is right in the middle of the overlap which I alluded to earlier. Additionally, as pure conjecture on my part, we will take a look at an approach to making them live together closely.

Application and Integration Server

Application servers are well known for the capabilities that enable hosting of application logic. As you would expect, they offer capabilities such as robust hosting – auto start, load balancing, fault tolerance etc. Integration servers are well known for enabling communication with external systems and, as you would expect, offer support for various message exchange patterns, guaranteed delivery, and family of adapters to connect to desperate systems and so on.

Note -as stated earlier, BizTalk is more than a typical integration server. The diagram below is intended to be more generic and depicts typical workloads of application and integration servers.

 

With emphasis on composite applications, application designers are increasingly looking for ways to combine the best of the capabilities offered by application and integration server. For example, they want to deploy a business service (exposed as a SOAP endpoint) and at the same time, expose it as a service endpoint that is consumable as an external system via MSMQ. Conversely, they want to expose integration endpoints that have low latency requirements and don’t necessarily need features such as guaranteed delivery and business activity monitoring.

Another factor that is driving application and integration servers closer is the move towards the cloud. Traditional on-premise based application logic is being refactored to take advantage of cloud based services such as storage, compute and access services. This is also placing a demand for integration capabilities to be added to application servers.

Not surprisingly, Microsoft’s recent announcements suggest that the line between AppFabric and BizTalk is being blurred. For instance, the BizTalk team has stated publicly that they are looking to build the next generation of BizTalk on top of AppFabric – http://www.microsoftpdc.com/2009/SVR15.

Current Dilemma

If you are architecting an enterprise application today, should you just move all your integration logic to AppFabric? There are a number of reasons why you would be tempted to do this, including 1) AppFabric is part of the OS; 2) most integration is now SOA/ WS based anyways, thus reducing the need for custom adapters; 3) operational setup and maintenance cost of running an AppFabric WebFarm is likely lower than a BizTalk farm; and 4) development of components is more approachable to a .NET developer who does not have prior knowledge of BizTalk. However, if you go down in this direction, be aware that there will be some features that will not be available. To better understand what is available and what is not available, let us use a scenario that is well supported by both – hosting a WCF endpoint. The following table compares the hosting a WCF endpoint in BizTalk and AppFabric.

 

WCF hosted Inside BizTalk	WCF hosted inside App Fabric
Binding flexibility – support for custom WCF configuration	Binding flexibility – support for custom WCF configuration
Load balanced, fault tolerant setup · Central administration covers all aspects. For example, a change to the WCF binding needs to be applied to a single location (port setting dialog).	Load balanced, fault tolerant setup · Central administration for tracking purposes only. A change to a WCF binding needs to be replicated to all nodes using tools such as msdeploy. http://download.microsoft.com/download/3/1/C/31CED722-2E5F-48D6-96B1-E73AAFD9873F/AppFabricWebFarm.docx
Common tracking database	Common tracking database
Tooling to review tracked instances · Tooling is somewhat dated –BAM portal.	Tooling to review tracked instances · More modern tooling.
Cross – cutting concerns – exception handling, BAM, pre and post processing (formatting etc.) · Advanced support in the form of suspended message queue, BAM portal.	Cross – cutting concerns – exception handling, BAM, pre and post processing (Formatting etc.) · Not as advanced as BTS yet.
Guaranteed delivery	No guaranteed delivery (except when provided by the transport such as msmq)
Better suited for async message exchange. Request /Response for low latency scenarios are not easily handled.	Better support for different message exchange patterns – Request/Response, Asynchronous.
Extensibility points for transforming incoming and outgoing messages · Pipeline components (, EDI, AS2, MIME, etc.) · Mapper · Flat File Parser	Extensibility points for transforming incoming and outgoing messages as defined by WCF · Custom Channels
Business logic is baked into a set of code and configuration components spread across the system · Custom pipeline component · Receive and send port definition · Custom mapper · Orchestration This setup allows flexibility in handling changes. For example, a new message decoder can be added for incoming messages, by simply adding a custom decoder step in the receive pipeline.	Business logic is encapsulated into the body of method (of course, factored appropriately into a class libraries etc.). This setup is makes it easy to single step through the code. But this also means that every aspect of the logic manifests itself as code. In essence, you need to roll out a new version of binary to make a change.
Large message handling · Every message in BizTalk by default is routed via the message box. This makes it a challenge to handle large messages. However, there are well known workarounds for this. One of them is described in detail by Paolo http://blogs.msdn.com/b/paolos/archive/2010/05/25/large-message-transfer-with-wcf-adapters-part-1.aspx.	Large message handling
Type safety · WCF adapters use un-typed contracts for receiving messages. This allows it to receive any type of message.	Type safety · Contracts can be typed and un-typed
No built-in caching	Caching
Secure Storage Service(SSS) to secure store credentials	No equivalent capability

What can we expect next?

The latest release of BizTalk 2010 is already bridging the gap between the two products. For instance, AppFabric connect feature of BizTalk 2010 offers ways to leverage BizTalk capabilities inside AppFabric hosted programs. Specifically:

1) Ability to leverage the BizTalk mapping capability inside a WF program using the Mapper activity.

2) Ability to generate WF activities that invoke operations on WCF LOB adapters provided by BizTalk.

For more information see – http://blogs.msdn.com/b/biztalk_server_team_blog/archive/2010/06/09/biztalk-appfabric-an-introduction.aspx

I expect this trend to continue, in fact, I envision that a future version of App Fabric would look something like the diagram below (again, this is pure conjecture on my part). As you can see, the BizTalk provided capabilities will be moved into AppFabric (shown in red). This will allow WCF endpoints to take advantage of guaranteed delivery, as well as, the configuration flexibility offered by BizTalk ports (pipeline components, mapper etc.).

 

 

 

Call to Action

It is clear that AppFabric and BizTalk are moving closer together. In future, we can expect to have access to capabilities offered by both, rather than choosing between them.

Messaging, with its defining characteristics including adapters, reliable communication and mediation, is generally accepted as the preferred way to connect disparate systems (look no further than Gregor Hohpe’s seminal book on Enterprise Integration Patterns for more on messaging). As we have seen in this post, BizTalk has built-in support for messaging. This is why it is recommended that BizTalk be considered for most integration scenarios

AppFabric should be considered for hosting most of the application logic. Microsoft is making strategic investment in AppFabric to not only to bring advanced management and robust hosting capabilities but also as a way to seamlessly enable application migration between on-premise and the cloud based deployments (commonly referred to as Server / Service symmetry).

Public, Private, Hybrid or On-Premise – which cloud model is right for you?

you are about to embark on a distributed application development project and are wondering how you can leverage the Windows Azure Platform. Should  you invest in private cloud, move your application to the public cloud, choose a hydrid approach or keep the application on-premise?   In this blog post, I will try to briefly describe each model and provide some commentary on the impact each of the aforementioned model can have on development and operations.

Before we begin, let us sketch a fairly common  multi-tier application architecture. We will use it through the course of this post.  Along the name of each layer, I have appended, in parenthesis, a few implementation choices. For instance the data layer may be implemented using SQL, Analysis Service, and Master Data Services (MDS).

 

 

Private Cloud

The term private cloud has been used to describe two very different models. The first is the ability to run a variant of Windows Azure in a customer’s datacenter. While at least one Microsoft executive has hinted at this possibility, it is highly unlikely that we will see this anytime soon.

The second model refers to the techniques like virtualization, automated management, and utility-billing models, within their own data centers. Microsoft has been increasingly talking about toolkits such Dynamic Datacenter Toolkit  that will allow IT managers to implement the aforementioned cloud computing concepts, in their data centers. This is different from running a cloud OS on-premise. 

In addition to the two private cloud models described above, there is also the notion of Virtual Private Cloud wherein a cloud provider will segregate a bunch of machines in its data center and dedicate them for a given customer. While vendors such as Amazon have talked about this option quite a bit, Microsoft has not announced anything significant in this area.

Impact on Development and Operations

1) Development and operational impact will depend on the degree to which an organization commits to building a private cloud. If the private cloud is primarily designed to offer Infrastructure as a Service (IaaS), the impact on development is minimal. In other words, the development effort is not very different from a traditional on-premise development.

2) Setting up a private cloud requires significant investment and careful planning and is therefore only recommended for very large enterprises with special security needs.

 

Public Cloud

Public cloud refers to the approach wherein the entire application is hosted on the Windows Azure platform. For our candidate architecture it would mean that all layers of architecture are in the cloud. While this is the most cost effective model, there are a number of limitations associated with it. The ability to install custom software is limited unless one can use xcopy to deploy third-party applications. All of the Azure compute instances  run the same base image (64 bit Windows Server, .NET 3.5 SP1). Microsoft has announced that future versions of the Azure platform will allow customers to create their own base images. Another limitation is the SQL Server functionality that is available as part of SQL Azure. For instance, MDS is not available with SQL Azure today. Additionally, SQL Azure databases are limited to a  maximum size of 50 GB.

 

Impact on Development and Operations

1) The design should adhere to a patterns including multi-tenancy, statelessness, and ability to dynamically handle the changes to the configuration (adding web front end nodes with increase in load).

2) Limits imposed by the Windows Azure Platform must be taken into account. For example, the data layer will need to utilize some sort of sharding  scheme to get around SQL Azure max size limits.

3) Public cloud option offers other significant advantages such as  automated service management, fault detection and notification that can potentially reduce the operational cost.

4) Upfront capital expenditure is replaced with ongoing operational expenditure based on the resource used.

Hybrid Cloud

Hybrid cloud refers to a style of cloud computing that combines functionality available in the cloud with the resources based on on-premise. Such an arrangement could be motivated by special security requirement (such as the Payment Card Industry security standards)for data.   For our proposed architecture, one scenario would be to have the UI and business services layer hosted in the cloud, while the data layer resides on-premise. Technologies such as Azure AppFabric ServiceBus and Project Sydney can be used to bridge the connection between cloud and on-premise datacenter.

 

 

Impact on Development and Operations

1)The design should adhere to a patterns including multi-tenancy, statelessness, and the ability to  handle changes to the configuration dynamically(i.e., ability to add additional web front ends to handle higher load).

2) Application design will need to compensate for the latency as a result of the data layer being remote. For instance, a caching layer may need to be introduced.

3) The operations team will need to plan, review, and setup adequate trust between the Azure and the on-premise data center.

4) A scalable / robust connected infrastructure between Windows Azure and the on-premise data center needs to be available.

5) IT will need to evaluate the impact on security, compliance, and availability.

Cloud Ready Design

Perhaps you are not ready for any of the aforementioned cloud options. In this instance, you can consider designing your application in a manner that will allow you to leverage the cloud in the future.  The “server-services" symmetry depicted in the diagram below exemplifies this approach. It illustrates how Microsoft plans to align the building blocks across its server OS (Windows Server 2008 and beyond) with the “Azure OS”. A good example of the overlap between server and services is AppFabric Caching -  According to Gopal Kavivaya[1], the underlying architecture of velocity is based on data fabric technology that powers SQL Azure.

 

Impact on Development and Operations

1) Utilize existing management tools such as System Center that are being refactored to work with the on-premise, as well as, Azure based applications.

2) Understand and leverage the App Fabric as new services become available.

3) Be cognizant of the differences between the SQL Azure and SQL Server running on-premise and avoid large monolithic database instances, as well as, features such as CLR and Service Broker that may not be immediately available in SQL Azure.

4) Avoid relying on special security privileges, file system or registry access etc. that will not be available when executing on Windows Azure.

[1] http://channel9.msdn.com/pdc2008/BB03/

Monte Carlo Simulation on Azure

 

Live Demo  http://ais.cloudapp.net  **

** To limit  the cost of hosting this demo application, the availability is limited to  regular business hours -  9:00 am to 5:00 pm EST. An on-premise  utility, based on Windows Azure Service Management cmdlets, is used automate the creation and removal  of this application.

 [ Readers who are not already familiar with Windows Azure concepts may find it useful to review this first ]

This project was motivated by an article by Christian Stitch:   In his article, Christian Stitch describes an approach for financial option valuation implemented with Monte Carlo simulation using Excel Services. Of course, with Windows Azure, we have now  easy access to  highly elastic computational capability. This prompted me to take  Christian’s idea and refactor the code to run on the Windows Azure Platform.

Monte Carlo Simulations

You can read more about Monte Carlo Simulation on the Wikipedia page here.  But here is an explanation from Christian’s article that I found succinct and useful: 

“Monte Carlo simulations are extremely useful in those cases where no closed form solutions to a problem exist. In some of those cases approximations to the solution exist; however, often the approximations do not have sufficient accuracy over the entire range. The power of Monte Carlo simulations becomes obvious when the underlying distributions do not have an analytical solution. Monte Carlo simulations are extensively used in the physical and life sciences, engineering, financial modeling and many other areas.”

It is also important to note that there is no single Monte Carlo method of algorithm.  For this project I follow these steps:

Ask the user to define a domain of possible inputs (Mean, StdDev, Distribution, MinVal and MaxVal).

Generate inputs using  Box-Muller transform.

Perform a deterministic computation using the inputs for number of iteration requested.

Aggregate the results of the individual computations into the final result.

Why use the Windows Azure Platform?

Monte Carlo simulation results require a very large number of iterations to get the desired accuracy. As a result, access to elastic, cheap computational resources is a key requirement. This is where the Windows Azure Platform comes in. It is possible to dynamically add compute instances as needed (as you would imagine, you only pay for what you use). Furthermore, it is also possible to select from a set of small (2 cores), medium (4 cores) and large (6 cores) compute instances.

As part of my testing, I ran a simulation involving a billion iterations. In an unscientific test, running this simulation using 2 small compute instances took more than four hours.  I was able to run the same simulation in minutes ( < 20 minutes ) by provisioning four large compute instances. 

In addition, to the elastic computation resources, Windows Azure also offers a variety of persistence options including Queues, Blobs and Tables that can be used to store any amount of data  for any length of time.

Last but not the least, as a .NET/ C# developer, I was able to port the existing C# code with multi-threading, ASP.NET and other constructs, easily to Windows Azure. 

Architecture

Let us take a brief look at the architecture of this application. I will follow  Philip Krutchen’s “4+1” view model to describe the architecture of this application. Philip Krutchen’s approach uses different viewpoints such as logical, development, process and physical view to describe the system.

Functional View

There is  a single use case for this application. Users can submit a Monte Carlo simulation request by providing domain inputs such as Mean, StdDev, Distribution, MinVal and MaxVals. Users also have to specify the number of iterations. Once the request has been successfully submitted, a unique task identifier is assigned. Users can then monitor the completion status of their tasks by clicking on the Monitor tab. Upon completion of the task, users can  analyze the calculation results using  two charts that depict the density curve based on the results of the calculation.

Logical View

Overall, the system follows a  simple, yet powerful, asynchronous pattern  wherein the  Web role(s) place a request for calculations to an Azure Queue. A set of Worker roles then retrieve these requests, perform the necessary calculations and store the results in the Azure table storage. 

Worker and Web roles are stateless and completely decoupled. In fact, Web and Worker roles are packaged as two distinct Azure services. As a result, it is possible to scale this application up and down as needed. For instance,  if a large number of calculation requests were to come in at the same time, it is  possible to add  Web and Worker roles in real time. At the same time, it is  also possible to completely tear down the Worker roles during periods of inactivity (thereby incurring no Windows Azure hosting charge)

Development View

As indicated earlier, the code is broken up into two Azure services.

UI Service

As the name suggests, this service is responsible for the UI.  It is based on  a MVC Web Role. This service  accepts the calculation task details  from the user, chops it up into smaller subsets (referred to as jobs) and  writes them to the queue. There is one message for each subset (interestingly, the worker roles, in turn, further subdivides this subset based on the number of VM cores ).  

The  Submit and Monitor tabs (described in the functional view) are built using  straightforward MVC code. The “Analyze” tab has some rich charts built using Silverlight Control Toolkit. The two charts depict the density curve based on the results of calculation (cumulative  normal and inverse cumulative respectively).

The Silverlight application uses a WCF service hosted within the web role to  retrieve the calculation results from the Azure Table storage. The WCF service acts as an intermediary since the Silverlight application cannot make a cross-domain call to the Azure Table storage directly ( it is possible to access the Azure Blob storage directly by placing a ClientAccessPolicy.xml in the root container).

Calc Service

Again, as the name suggests, this service is responsible for performing the calculations. Each worker role periodically checks for any new requests. Once it retrieves a new request, it distributes the calculation across a number of threads ( number of threads equals the number of available cores within a VM). Once the calculation is complete, each worker marks the appropriate job as complete and stores the results of the calculations  in a Azure table.

Azure Queue has semantics to ensure that every message in the queue will have the chance to be processed to completion at least once.  So if the worker role instance crashes after it dequeues a message and before it completes the calculation, the request will re-appear in the queue after the VisibilityTimeout. This allows another worker role to come along and process the request to completion.

 Physical View

As stated earlier, each calculation task is chopped up into a bunch of jobs. Details about the  calculation are stored in a single  Azure table. The combination of TaskId and JobId   serve as the partition and row key. The following snapshot ( created using Cerebrata’s excellent Cloud Storage Studio tool) depicts the remaining elements of the schema.

The results of the calculation are stored in a separate Azure table . The following snapshot depicts the schema for that table where results are stored

 

 Summary

Windows Azure Platform is a good fit for applications such as the Monte Carlo method that require elastic computational and storage resources.  Azure Queue provides a simple scheme for implementing the asynchronous pattern. Finally, moving existing C# code – calculation algorithm, multithreading and other constructs to Azure is straightforward.