http://blog.appliedis.com/2012/12/28/introducing-netizen-for-windows-8/

 

( this post is a response to ACT4Apps questionnaire posted by ACT and AT&T )

As a software developer who is also very interested in public policy, I am always looking to build apps that make it easier for folks to follow the public policy discourse. This is why I decided to build Netizen.   Netizen  is a Windows 8 app (soon to be  available in the Windows Store for free) that brings the voting record of your congressional representative to your finger tips. Simply select the Member of Congress you want to follow and "flick through" their voting record.

But I don’t want users of Netizen to merely follow how their  representative are voting in congress, instead, I want them to ensure that their voices are being heard.  For each bill brought to the floor of the house, Netizen automatically provisions a Facebook page dedicated to your member of congress. This page acts as a virtual ballot for a bill, as well as, a community hub where fellow constituents can gather to express their support. If enough of your friends and neighbors like or unlike the a vote, your congressional representative is sure to pay attention.

Pasted below are some screenshots from the application:

Screenshot #1 – Select state or find your congressional district based on your current location

clip_image002

Screenshot #2 – Select Representative

clip_image004

Screenshot #3 – “Flick through” the most recent roll call votes

clip_image006

Screenshot #4 –  “Virtual ballot” – Dynamically generated Facebook page,

clip_image008

Screenshot #5 –  Search for a representative

clip_image010

I am working on making this app available on other devices as well. Unfortunately the fragmented app landscape makes it harder for hobbyists and self-funded projects such as Netizen to publish across multiple app stores. If any of you are interested in helping develop versions of this app that run on other mobile platforms, please contact me via my twitter handle @vlele.

I hope that by the time CES 2022 rolls around, we can build an electronic, friction free  voting system that allows anyone with a tablet or a smart phone to securely cast a vote on the important issues of the day. Such a capability, combined with a vote  counting system that leverages the seemingly infinite computational resources available to us today, in order to make the voting results available in a snap. Perhaps that is the only hope to make law-making work again.

Finally, in order for sustain continued innovation in the app development, especially for hobbyists and startups, we need to strike a balance between regulation and room for innovation. Case in point, some of the recent changes proposed to the privacy laws for apps can lead to unintended consequences for the overall app economy. 

Short video that walks through IaaS & PaaS hydrid scenarios:

  • Using Windows Azure Virtual Network to provision a VPN to connect our on-premised infrastructure with a Windows Azure datacenter.
  • Set up front-end and back-end subnets.
  • Provision a set of Azure IaaS Virtual Machines and Azure Web Roles.
  • Install System Center Monitoring Pack for Windows Azure Applications on Azure-based machines.
  • Install System Center Operations on-premises in order to manage Azure-based resources.

http://blog.appliedis.com/2012/11/19/iaas-and-paas-together-a-video-walkthrough-of-hybrid-scenarios/

This paper covers 20 of the most common and not-so-common objections in moving software applications to the cloud. It highlights some of the most recent and up-to-date advancements in cloud technology that address these concerns head-on.

AIS Top 20 Reasons Clouding Cloud Adoption White Paper (PDF)

 

This blog post is about the evolution of Microsoft’s application server – a collection of products including IIS and .NET framework libraries such as WCF and WF. Over the last few releases Microsoft has added 1) runtime container and activation service, via WAS; 2) multi-protocol support that goes beyond Http, via IIS 7.0; and 3) manageability extensions and caching, via AppFabric. Some of these enhancements overlap with the capabilities offered by Microsoft’s flagship integration server, BizTalk. BizTalk is well known for its robust hosting and integration capabilities. This overlap has stirred conversation in blogs and news stories over the last several months.

My goal in this post is to step back and look into some of the reasons causing application and integration servers to grow closer; we will do a side-by-side comparison of a scenario that is right in the middle of the overlap which I alluded to earlier. Additionally, as pure conjecture on my part, we will take a look at an approach to making them live together closely.

Application and Integration Server

Application servers are well known for the capabilities that enable hosting of application logic. As you would expect, they offer capabilities such as robust hosting – auto start, load balancing, fault tolerance etc. Integration servers are well known for enabling communication with external systems and, as you would expect, offer support for various message exchange patterns, guaranteed delivery, and family of adapters to connect to desperate systems and so on.

Note -as stated earlier, BizTalk is more than a typical integration server. The diagram below is intended to be more generic and depicts typical workloads of application and integration servers.

image

 

With emphasis on composite applications, application designers are increasingly looking for ways to combine the best of the capabilities offered by application and integration server. For example, they want to deploy a business service (exposed as a SOAP endpoint) and at the same time, expose it as a service endpoint that is consumable as an external system via MSMQ. Conversely, they want to expose integration endpoints that have low latency requirements and don’t necessarily need features such as guaranteed delivery and business activity monitoring.

Another factor that is driving application and integration servers closer is the move towards the cloud. Traditional on-premise based application logic is being refactored to take advantage of cloud based services such as storage, compute and access services. This is also placing a demand for integration capabilities to be added to application servers.

Not surprisingly, Microsoft’s recent announcements suggest that the line between AppFabric and BizTalk is being blurred. For instance, the BizTalk team has stated publicly that they are looking to build the next generation of BizTalk on top of AppFabric – http://www.microsoftpdc.com/2009/SVR15.

Current Dilemma

If you are architecting an enterprise application today, should you just move all your integration logic to AppFabric? There are a number of reasons why you would be tempted to do this, including 1) AppFabric is part of the OS; 2) most integration is now SOA/ WS based anyways, thus reducing the need for custom adapters; 3) operational setup and maintenance cost of running an AppFabric WebFarm is likely lower than a BizTalk farm; and 4) development of components is more approachable to a .NET developer who does not have prior knowledge of BizTalk. However, if you go down in this direction, be aware that there will be some features that will not be available. To better understand what is available and what is not available, let us use a scenario that is well supported by both – hosting a WCF endpoint. The following table compares the hosting a WCF endpoint in BizTalk and AppFabric.

 

WCF hosted Inside BizTalk

WCF hosted inside App Fabric

Binding flexibility – support for custom WCF

configuration

Binding flexibility – support for custom WCF configuration

Load balanced, fault tolerant setup

· Central administration covers all aspects. For example, a change to the WCF binding needs to be applied to a single location (port setting dialog).

Load balanced, fault tolerant setup

· Central administration for tracking purposes only. A change to a WCF binding needs to be replicated to all nodes using tools such as msdeploy.

http://download.microsoft.com/download/3/1/C/31CED722-2E5F-48D6-96B1-E73AAFD9873F/AppFabricWebFarm.docx

Common tracking database

Common tracking database

Tooling to review tracked instances

· Tooling is somewhat dated –BAM portal.

Tooling to review tracked instances

· More modern tooling.

Cross – cutting concerns – exception handling, BAM, pre and post processing (formatting etc.)

· Advanced support in the form of suspended message queue, BAM portal.

Cross – cutting concerns – exception handling, BAM, pre and post processing

(Formatting etc.)

· Not as advanced as BTS yet.

Guaranteed delivery

No guaranteed delivery (except when provided by the transport such as msmq)

Better suited for async message exchange. Request /Response for low latency scenarios are not easily handled.

Better support for different message exchange patterns – Request/Response, Asynchronous.

Extensibility points for transforming incoming and outgoing messages

· Pipeline components (, EDI, AS2, MIME, etc.)

· Mapper

· Flat File Parser

Extensibility points for transforming incoming and outgoing messages as defined by WCF

· Custom Channels

Business logic is baked into a set of code and configuration components spread across the system

· Custom pipeline component

· Receive and send port definition

· Custom mapper

· Orchestration

This setup allows flexibility in handling changes. For example, a new message decoder can be added for incoming messages, by simply adding a custom decoder step in the receive pipeline.

Business logic is encapsulated into the body of method (of course, factored appropriately into a class libraries etc.).

This setup is makes it easy to single step through the code. But this also means that every aspect of the logic manifests itself as code. In essence, you need to roll out a new version of binary to make a change.

Large message handling

· Every message in BizTalk by default is routed via the message box. This makes it a challenge to handle large messages. However, there are well known workarounds for this. One of them is described in detail by Paolo

http://blogs.msdn.com/b/paolos/archive/2010/05/25/large-message-transfer-with-wcf-adapters-part-1.aspx.

Large message handling

Type safety

· WCF adapters use untyped contracts for receiving messages. This allows it to receive any type of message.

Type safety

· Contracts can be typed and un-typed

No built-in caching

Caching

Secure Storage Service(SSS) to secure store credentials

No equivalent capability

What can we expect next?

The latest release of BizTalk 2010 is already bridging the gap between the two products. For instance, AppFabric connect feature of BizTalk 2010 offers ways to leverage BizTalk capabilities inside AppFabric hosted programs. Specifically:

1) Ability to leverage the BizTalk mapping capability inside a WF program using the Mapper activity.

2) Ability to generate WF activities that invoke operations on WCF LOB adapters provided by BizTalk.

For more information see – http://blogs.msdn.com/b/biztalk_server_team_blog/archive/2010/06/09/biztalk-appfabric-an-introduction.aspx

I expect this trend to continue, in fact, I envision that a future version of App Fabric would look something like the diagram below (again, this is pure conjecture on my part). As you can see, the BizTalk provided capabilities will be moved into AppFabric (shown in red). This will allow WCF endpoints to take advantage of guaranteed delivery, as well as, the configuration flexibility offered by BizTalk ports (pipeline components, mapper etc.).

 

image

 

 

Call to Action

It is clear that AppFabric and BizTalk are moving closer together. In future, we can expect to have access to capabilities offered by both, rather than choosing between them.

Messaging, with its defining characteristics including adapters, reliable communication and mediation, is generally accepted as the preferred way to connect disparate systems (look no further than Gregor Hohpe’s seminal book on Enterprise Integration Patterns for more on messaging). As we have seen in this post, BizTalk has built-in support for messaging. This is why it is recommended that BizTalk be considered for most integration scenarios

AppFabric should be considered for hosting most of the application logic. Microsoft is making strategic investment in AppFabric to not only to bring advanced management and robust hosting capabilities but also as a way to seamlessly enable application migration between on-premise and the cloud based deployments (commonly referred to as Server / Service symmetry).

you are about to embark on a distributed application development project and are wondering how you can leverage the Windows Azure Platform. Should  you invest in private cloud, move your application to the public cloud, choose a hydrid approach or keep the application on-premise?   In this blog post, I will try to briefly describe each model and provide some commentary on the impact each of the aforementioned model can have on development and operations.

Before we begin, let us sketch a fairly common  multi-tier application architecture. We will use it through the course of this post.  Along the name of each layer, I have appended, in parenthesis, a few implementation choices. For instance the data layer may be implemented using SQL, Analysis Service, and Master Data Services (MDS).

 

image 

Private Cloud

The term private cloud has been used to describe two very different models. The first is the ability to run a variant of Windows Azure in a customer’s datacenter. While at least one Microsoft executive has hinted at this possibility, it is highly unlikely that we will see this anytime soon.

The second model refers to the techniques like virtualization, automated management, and utility-billing models, within their own data centers. Microsoft has been increasingly talking about toolkits such Dynamic Datacenter Toolkit  that will allow IT managers to implement the aforementioned cloud computing concepts, in their data centers. This is different from running a cloud OS on-premise. 

In addition to the two private cloud models described above, there is also the notion of Virtual Private Cloud wherein a cloud provider will segregate a bunch of machines in its data center and dedicate them for a given customer. While vendors such as Amazon have talked about this option quite a bit, Microsoft has not announced anything significant in this area.

Impact on Development and Operations

1) Development and operational impact will depend on the degree to which an organization commits to building a private cloud. If the private cloud is primarily designed to offer Infrastructure as a Service (IaaS), the impact on development is minimal. In other words, the development effort is not very different from a traditional on-premise development.

2) Setting up a private cloud requires significant investment and careful planning and is therefore only recommended for very large enterprises with special security needs.

 

Public Cloud

Public cloud refers to the approach wherein the entire application is hosted on the Windows Azure platform. For our candidate architecture it would mean that all layers of architecture are in the cloud. While this is the most cost effective model, there are a number of limitations associated with it. The ability to install custom software is limited unless one can use xcopy to deploy third-party applications. All of the Azure compute instances  run the same base image (64 bit Windows Server, .NET 3.5 SP1). Microsoft has announced that future versions of the Azure platform will allow customers to create their own base images. Another limitation is the SQL Server functionality that is available as part of SQL Azure. For instance, MDS is not available with SQL Azure today. Additionally, SQL Azure databases are limited to a  maximum size of 50 GB.

 

image

Impact on Development and Operations

1) The design should adhere to a patterns including multi-tenancy, statelessness, and ability to dynamically handle the changes to the configuration (adding web front end nodes with increase in load).

2) Limits imposed by the Windows Azure Platform must be taken into account. For example, the data layer will need to utilize some sort of sharding  scheme to get around SQL Azure max size limits.

3) Public cloud option offers other significant advantages such as  automated service management, fault detection and notification that can potentially reduce the operational cost.

4) Upfront capital expenditure is replaced with ongoing operational expenditure based on the resource used.

Hybrid Cloud

Hybrid cloud refers to a style of cloud computing that combines functionality available in the cloud with the resources based on on-premise. Such an arrangement could be motivated by special security requirement (such as the Payment Card Industry security standards)for data.   For our proposed architecture, one scenario would be to have the UI and business services layer hosted in the cloud, while the data layer resides on-premise. Technologies such as Azure AppFabric ServiceBus and Project Sydney can be used to bridge the connection between cloud and on-premise datacenter.

image

 

 

Impact on Development and Operations

1)The design should adhere to a patterns including multi-tenancy, statelessness, and the ability to  handle changes to the configuration dynamically(i.e., ability to add additional web front ends to handle higher load).

2) Application design will need to compensate for the latency as a result of the data layer being remote. For instance, a caching layer may need to be introduced.

3) The operations team will need to plan, review, and setup adequate trust between the Azure and the on-premise data center.

4) A scalable / robust connected infrastructure between Windows Azure and the on-premise data center needs to be available.

5) IT will need to evaluate the impact on security, compliance, and availability.

Cloud Ready Design

Perhaps you are not ready for any of the aforementioned cloud options. In this instance, you can consider designing your application in a manner that will allow you to leverage the cloud in the future.  The “server-services" symmetry depicted in the diagram below exemplifies this approach. It illustrates how Microsoft plans to align the building blocks across its server OS (Windows Server 2008 and beyond) with the “Azure OS”. A good example of the overlap between server and services is AppFabric Caching -  According to Gopal Kavivaya[1], the underlying architecture of velocity is based on data fabric technology that powers SQL Azure.

clip_image014

 

Impact on Development and Operations

1) Utilize existing management tools such as System Center that are being refactored to work with the on-premise, as well as, Azure based applications.

2) Understand and leverage the App Fabric as new services become available.

3) Be cognizant of the differences between the SQL Azure and SQL Server running on-premise and avoid large monolithic database instances, as well as, features such as CLR and Service Broker that may not be immediately available in SQL Azure.

4) Avoid relying on special security privileges, file system or registry access etc. that will not be available when executing on Windows Azure.

[1] http://channel9.msdn.com/pdc2008/BB03/

 

Live Demo  http://ais.cloudapp.net  **

** To limit  the cost of hosting this demo application, the availability is limited to  regular business hours -  9:00 am to 5:00 pm EST. An on-premise  utility, based on Windows Azure Service Management cmdlets, is used automate the creation and removal  of this application.

 [ Readers who are not already familiar with Windows Azure concepts may find it useful to review this first ]

This project was motivated by an article by Christian Stitch:   In his article, Christian Stitch describes an approach for financial option valuation implemented with Monte Carlo simulation using Excel Services. Of course, with Windows Azure, we have now  easy access to  highly elastic computational capability. This prompted me to take  Christian’s idea and refactor the code to run on the Windows Azure Platform.

Monte Carlo Simulations

You can read more about Monte Carlo Simulation on the Wikipedia page here.  But here is an explanation from Christian’s article that I found succinct and useful: 

“Monte Carlo simulations are extremely useful in those cases where no closed form solutions to a problem exist. In some of those cases approximations to the solution exist; however, often the approximations do not have sufficient accuracy over the entire range. The power of Monte Carlo simulations becomes obvious when the underlying distributions do not have an analytical solution. Monte Carlo simulations are extensively used in the physical and life sciences, engineering, financial modeling and many other areas.”

It is also important to note that there is no single Monte Carlo method of algorithm.  For this project I follow these steps:

  • Ask the user to define a domain of possible inputs (Mean, StdDev, Distribution, MinVal and MaxVal).
  • Generate inputs using  Box-Muller transform.
  • Perform a deterministic computation using the inputs for number of iteration requested.
  • Aggregate the results of the individual computations into the final result.

    Why use the Windows Azure Platform?

    Monte Carlo simulation results require a very large number of iterations to get the desired accuracy. As a result, access to elastic, cheap computational resources is a key requirement. This is where the Windows Azure Platform comes in. It is possible to dynamically add compute instances as needed (as you would imagine, you only pay for what you use). Furthermore, it is also possible to select from a set of small (2 cores), medium (4 cores) and large (6 cores) compute instances.

    As part of my testing, I ran a simulation involving a billion iterations. In an unscientific test, running this simulation using 2 small compute instances took more than four hours.  I was able to run the same simulation in minutes ( < 20 minutes ) by provisioning four large compute instances. 

    In addition, to the elastic computation resources, Windows Azure also offers a variety of persistence options including Queues, Blobs and Tables that can be used to store any amount of data  for any length of time.

    Last but not the least, as a .NET/ C# developer, I was able to port the existing C# code with multi-threading, ASP.NET and other constructs, easily to Windows Azure. 

    Architecture

    Let us take a brief look at the architecture of this application. I will follow  Philip Krutchen’s “4+1” view model to describe the architecture of this application. Philip Krutchen’s approach uses different viewpoints such as logical, development, process and physical view to describe the system.

    Functional View

    There is  a single use case for this application. Users can submit a Monte Carlo simulation request by providing domain inputs such as Mean, StdDev, Distribution, MinVal and MaxVals. Users also have to specify the number of iterations. Once the request has been successfully submitted, a unique task identifier is assigned. Users can then monitor the completion status of their tasks by clicking on the Monitor tab. Upon completion of the task, users can  analyze the calculation results using  two charts that depict the density curve based on the results of the calculation.

    clip_image002clip_image002[4]

  • Logical View

    Overall, the system follows a  simple, yet powerful, asynchronous pattern  wherein the  Web role(s) place a request for calculations to an Azure Queue. A set of Worker roles then retrieve these requests, perform the necessary calculations and store the results in the Azure table storage.  image

    Worker and Web roles are stateless and completely decoupled. In fact, Web and Worker roles are packaged as two distinct Azure services. As a result, it is possible to scale this application up and down as needed. For instance,  if a large number of calculation requests were to come in at the same time, it is  possible to add  Web and Worker roles in real time. At the same time, it is  also possible to completely tear down the Worker roles during periods of inactivity (thereby incurring no Windows Azure hosting charge)

    Development View

    As indicated earlier, the code is broken up into two Azure services.

    UI Service

    As the name suggests, this service is responsible for the UI.  It is based on  a MVC Web Role. This service  accepts the calculation task details  from the user, chops it up into smaller subsets (referred to as jobs) and  writes them to the queue. There is one message for each subset (interestingly, the worker roles, in turn, further subdivides this subset based on the number of VM cores ).  

    The  Submit and Monitor tabs (described in the functional view) are built using  straightforward MVC code. The “Analyze” tab has some rich charts built using Silverlight Control Toolkit. The two charts depict the density curve based on the results of calculation (cumulative  normal and inverse cumulative respectively).

    The Silverlight application uses a WCF service hosted within the web role to  retrieve the calculation results from the Azure Table storage. The WCF service acts as an intermediary since the Silverlight application cannot make a cross-domain call to the Azure Table storage directly ( it is possible to access the Azure Blob storage directly by placing a ClientAccessPolicy.xml in the root container).

    Calc Service

    Again, as the name suggests, this service is responsible for performing the calculations. Each worker role periodically checks for any new requests. Once it retrieves a new request, it distributes the calculation across a number of threads ( number of threads equals the number of available cores within a VM). Once the calculation is complete, each worker marks the appropriate job as complete and stores the results of the calculations  in a Azure table.

    Azure Queue has semantics to ensure that every message in the queue will have the chance to be processed to completion at least once.  So if the worker role instance crashes after it dequeues a message and before it completes the calculation, the request will re-appear in the queue after the VisibilityTimeout. This allows another worker role to come along and process the request to completion.

  •  Physical View

    As stated earlier, each calculation task is chopped up into a bunch of jobs. Details about the  calculation are stored in a single  Azure table. The combination of TaskId and JobId   serve as the partition and row key. The following snapshot ( created using Cerebrata’s excellent Cloud Storage Studio tool) depicts the remaining elements of the schema. clip_image002[10]

    The results of the calculation are stored in a separate Azure table . The following snapshot depicts the schema for that table where results are stored

    clip_image002[12]

     

     Summary

    Windows Azure Platform is a good fit for applications such as the Monte Carlo method that require elastic computational and storage resources.  Azure Queue provides a simple scheme for implementing the asynchronous pattern. Finally, moving existing C# code – calculation algorithm, multithreading and other constructs to Azure is straightforward.

  • Higher Order Software Development

    I cannot claim it to be an act of volition, but, over the last three years I have found myself involved with building software designed to allow the business users to create the programs they need. I now have a somewhat presumptuous name for this approach – Higher Order Software Development. This approach is similar to the composite application site wherein applications are assembled using existing software assets. The difference, however, is that composite applications such as mashups are about assembling assets that have been built independently. Higher order software development is about software building blocks, designed from the ground-up, to allow business users to “develop”. In the last few years, many approaches including BPM (business process management) have espoused empowering the end-users. Even though BPM systems have been successful in improving the process agility, they have fallen short of making software development end-user ready. This is mainly because the BPM tools are designed to address a broad set of application scenarios, and in most instances, BPM offerings represent a collection of tools (workflow designer, rules engines, modeling, and so on). This has inadvertently raised the level of complexity, thus making it harder for end-users to participate. On the other hand, higher order software development has a much narrower focus — it is about solving a specific business problem within a given business domain. This approach has been made possible because of the rise in the level of abstractions available within the development platform itself. As we will discuss shortly, platforms such as SharePoint have served as a catalyst for higher order software development. 

    Why is this important?  First, business users know the requirements the best. Unfortunately, most business users are not very good at communicating them – hence the famous phrase “I will not know what I want until I see it working”. Furthermore, a portion of the requirements are invariably lost in translation, as they are conveyed to development teams. Second, rather than learn new interfaces, business users want to continue to work with the tools they are already familiar with. What could be better than enabling the business users to create programs using familiar tools like Microsoft Office? Third, IT departments, already backed up in supporting existing operational systems, seldom have the resources to take on new application development projects.  Thus, empowering the business users to build their own software may be the only real way to scale and enable the business to adapt/react quickly to external market forces.  Since business users are directly involved in building, the cost and scope of development can be more efficiently managed. Business users are in a much better position to respond to questions such as – “Do we really need this feature? Is the customization really needed, or will the out-of-the box functionality do?”  Last, but not least, unlocking the true potential of IT and effectively competing in the next generation of business cycles will require a greater ability to adjust and change software systems cost effectively. By evolving how programs are put together, businesses can implement a process of continuous improvement over time.

    For the remaining portion of this post, I will describe four concrete examples of applications to which we have applied this paradigm.

    Example #1: Custom Calculation Engine in the Professional Services Industry

    A common requirement in this domain is to implement calculations/reports that adhere to the Financial Accounting Standards Board (FASB) standards. These types of reports are often large and complex. The calculations in the reports are specific to a geographical region, so a multi-national company needs to implement different versions of these calculations.  Furthermore, over time these calculations have to be adjusted to comply with changing laws. Traditionally these calculations have been implemented using custom code, and, as a result, suffer from the challenges outlined above, including: the high cost of development and maintenance, requirements being lost in translation, the lack of traceability, and the lack of a robust mechanism for making a quick change to a calculation in response to  a change in a standard. In a nutshell, the analysts needed a flexible calculation engine tool that makes it easy to develop and maintain the calculations, and at the same time, provides the necessary robustness and scalability.

    Here is a quick overview of the Excel Services based solution we developed. For an introduction to Excel Services please refer to the MSDN Magazine article – http://msdn.microsoft.com/en-us/magazine/cc163374.aspx. We used XSDs to capture all the input and output data elements needed for implementing a given calculation.  Using a custom Excel pre-compiler, we translated the XSD into named ranges. For example, the generated template workbook has three sheets – one each for input, output, and calculation. Analysts could then use the generated template workbooks to develop the algorithms.  As long as they worked within the contract – as defined in input, output and calculation sheets – they could use any Excel functions. Once the workbook was developed and tested, they placed the workbook into an Excel Services trusted folder for execution. To support high scalability, we used a cluster of Excel Services nodes. The following figure shows the basic architecture:

    clip_image001[4]

    clip_image003[4]

    Who built what

    Development Team

    Built a framework based Excel Services

    Business Users

    Developed the calculation logic using Excel

    As you can see from above example, we built the software building blocks using Excel Services, which in turn enabled the analysts to implement the calculations as needed. 

    Example #2: Incident Management System for Law Enforcement Agencies

    Law enforcement agencies are looking for ways to improve the processes surrounding incident reporting and management. Managing an incident entails gathering a variety of information to aid in the investigation, including documentation about the case (date, location, type of incident, summaries from eye witnesses, photographs, and evidence), as well as information about contacts and sources who can provide more information. In addition to providing an intuitive repository for this information, the system should enable an agency-wide collaborative work amongst the law enforcement personnel assigned to work on the cases. This requires the ability to assign and track tasks, initiate process steps such as review and approval, provide access to a group calendar and messaging, record the status of the case (open, active, closed, and archived), and generate centralized reports (such as the status of all cases or assignments). Information about cases also needs to be searchable and shareable with extra-agency organizations as necessary.

    In addition to the high-level functional requirements, incident management systems need to be highly secure and be able to protect confidential information. This typically implies a number of requirements to control access to information based on the role, organization, and level of data sensitivity. There are also typically requirements for these types of systems to support auditing and policy compliance monitoring, tracking of email correspondence, support of e-discovery requests, and enforcement of agency governance rules.

    The above requirements highlight the need to store large and variable amount of information about the case in one “container.”  The different document types that make up the case can have unique metadata and lifecycles associated with them. Law enforcement personnel working on the case need the flexibility to add notes to cases, include ad hoc documents, and store different versions of a document. They also need to be able to initiate a number of workflows such as approval and disposition for individual documents or a group of documents.  These workflows can be pre-defined or dynamic in nature. 

    We decided to build this system on the SharePoint platform. We provisioned the appropriate container (site collection, site, document library) for a case. The provisioned container is based on a pre-defined blueprint we built. The blueprint includes document types, folder structure, web parts and workflows. Users are then able to customize the provisioned container as needed, including changing the folder structure, adding new ad hoc workflows, and installing additional web parts for reports and data visualization. A nice side-benefit of using a SharePoint based container is the built-in support for archival and restoration of cases.

    clip_image005[4]

    Snapshot of SharePoint based Incident Reporting System

    Who built what

    Development Team

    Built SharePoint artifacts including site definitions, web parts, VS.NET based workflows

    Business Users

    Provisioned the site, customized the site, content types, adhoc workflows, custom lists and views

    As you can see from above example, we built the software building blocks using the SharePoint platform that in turn enabled the law enforcement officials to effectively manage the incidents.

    Example #3: Data Analysis Tool for the Telecommunication Industry

    Network optimization engineers often need to convert vast amount of raw network performance data into useful data visualizations and data products that can assist them in identifying network performance bottlenecks. The goal is to allow non-programmers (in this instance, network engineers) to visually define complex multi-step algorithms  that specify the steps and control dependencies for analysis of raw data.  

    The key requirement is that of flexibility and ease of use needed by network engineer to define custom data processing workflow steps. A solution that is completely built by IT would be too expensive and at the same time not be able to accord the flexibility needed in this instance.

    We decided to build the solution using Windows Workflow Foundation (WF). The WF Designer tools provides the network engineers with useful and flexible way to author the data processing algorithms.  To make it easier for network engineers to author the workflows, we developed a set of custom domain specific activities. A lot of attention was devoted to making the workflow authoring experience as simple as possible.

    Once the workflows are defined, they are executed asynchronously and results are made available for further analysis. The following diagram depicts the user interface for authoring the workflows.

    clip_image007[4]

    Snapshot of Workflow Foundation based designer used by network engineers

    Who built what

    Development Team

    Built a framework to host WF programs, developed domain specific WF activities, customized the WF designer to make it business-user ready

    Business Users

    Developed custom workflows

    As you can see from above example, by building a set of building blocks in the form of custom workflow activities,  we were able allow network engineers to develop the programs that allowed them to analyze vast amounts of raw data.

    Example #4: Management of Policy Data in the Insurance Domain

    A common requirement in the property and casualty (P&C) Insurance industry is to store and retrieve “snapshots” of a customer policy or contract, such as an automobile policy or a homeowners policy, as it existed at any given point in time in the past.  Clearly, there are a number of ways to implement this functionality. However, a key requirement is to be able to retrieve the policy snapshot very quickly – typically a sub-second response time is expected. Given the millions of rows worth of historical data that is common for such systems, it would be hard to retrieve a specific version of the policy data dynamically. An alternative approach is to pre-generate templates for the snapshots and tradeoff disk space in favor of response time. In other words, accept the additional cost of storing the entire policy snapshot for every change to the policy.   In addition to the response time requirement, there are two other  distinct requirements related to the historical data.  The first requirement is to generate reports for verification of compliance with various state laws. As you can imagine, compliance based reports tend to vary quite a bit, based on the laws of each state (within the US). The second requirement is to allow end-users to mine the historical data for interesting patterns, for instance: Why is there a higher rate of customer attrition in a given county over others?  What types of changes are more common in a given state?

    The key aspect of this scenario is that while there is a need to support high performing historical queries, there is also a competing need for flexibility in reporting and data mining. In short, the system needs to allow self-service business intelligence (BI) for the end-users.

    Let us now take a look at the solution we decided to build:  We used SQL Server Analysis Services (SSAS) based OLAP (online analytic processing) cube as an application building block.   To store the policy data snapshots we populated a OLAP cube in real time. The OLAP model we developed is depicted in the diagram below.  The dimensions (dimensions are reference information about the data) are obvious ones including customer, geography, and time. The interesting aspect of this model are the fact tables (facts are generally the numeric aspects of the transaction data). Since we are dealing with events (change in policy address, for example) as opposed to a classic business transaction that involves numbers, we ended up creating a “factless” that captures a log of all events on a policy. Additionally, to be able to retrieve the policy as of a certain date quickly, we maintain another “factless” table that captures the snapshot a given policy following any change to it.

    :

    clip_image009[4]

    Once the OLAP cube is in place, analysts are able to perform queries using a tool such as Excel, as shown in the diagram below.

    clip_image011[4]

    Who built what

    Development Team

    Organized the data in OLAP structure, developed a scheme to populate the OLAP structure in real time

    Business Users

    Developed the queries to respond to historization related requests, built adhoc reports to meet compliance and regulatory needs

    Readers who are familiar with BI (business intelligence) systems are probably wondering how the solution described above is different from a traditional BI solution. Furthermore, why do I see this as an example of higher order solution development? While this is indeed a BI-based solution, it is a different from a traditional BI Data Warehouse / Data Mart in several important ways. First, the dimensional model, with the factless and snapshot tables, is different from a traditional BI data warehouse. In this instance, the data model has been designed in a manner to promote “assembly” by business users. Secondly, the OLAP cube is very much part of the application architecture in the sense that  queries from the UI (user interface) layer are serviced by the OLAP cube in near real-time. This also unlike a traditional BI system which is usually a distinct system designed to serve as  a decision support system. Finally, by combining a traditional relational database with the OLAP cube, we are able to offer the notion of a composeable data services layer.

    As you can see from above example, the OLAP based building blocks built by the development team enabled business users to perform dynamic queries to respond to historization requests from the client and build adhoc reports to meet compliance and regulatory needs.

    In summary, with all of the above examples, we built the software tools that allowed the business-users to create the program they need.  This approach has a number of benefits including reduced cost of development and increased agility to adjust and change software systems.

    During the last year, I have spent quite a bit of time on Cloud Computing in general and specifically on the Windows Azure Platform. So I thought, I would share with you, my humble 2010 predictions related to Cloud Computing.

    • Cloud Computing Services Revenue – Leading System Integrators could see up to 10% of their revenue coming from cloud based services, by the end of 2010. System Integrators with an App/Dev focus should invest in the building competency around cloud based application development. For instance:  invest in developing expertise on cloud application development platforms such as Azure, evaluate how their existing product/offerings can leverage the cloud and  understand the new business models enabled by the cloud.
    • Small and Medium Businesses (SMBs)– Another interesting opportunity for the SIs is to bring the benefits of the cloud to the SMB space. Unlike the common perception that  cloud computing is mainly for large, elastic workloads, I think there is a real opportunity for SMBs to leverage the benefits of the cloud. For the  first time, cloud platforms enable SMBs to take advantage of enterprise functions such as workflow, BPM, CRM etc. at a price point that is approachable to them. This is an opportunity not only for the SMBs to make their businesses more agile, but also for  SIs to build competency on cloud computing (since the work with SMBs will allow them to start small and gradually build their expertise)
    •  No Letup in the Pace of Innovation by Cloud Providers -  In the past year, there were a staggering  number of changes to the Windows Azure Platform alone (other platforms were no different). Starting with the full relational based storage, support for Java/Python, ability to choose from different type of compute instances, market place for data and more. We can expect this trend to continue in 2010, which is why, it is key for SIs to invest in constantly keeping up with these fast paced innovations. 

    Happy New Year!

      Here are my personal favorite SharePoint 2010 new features and enhancements:

       

    • Access Services & Visio Services
    • Building on the successful Excel Services pattern, SPS 2010 allows users to publish Access and Visio applications to SharePoint which other users can access through their browsers. As can be expected, Access and Visio Services will be somewhat limited in functionality(when compared to their respective desktop versions), but on the flip side, the customers would benefit from the server-side capabilities such as scalability, one centralized version of the document (rendered in a browser friendly format), access to multiple users and security control.   

      Access Services will allow customers to publish their Access 2010 applications to SharePoint. For example, consider a departmental application such as a travel planner built using Access. Using Access Services it will be possible to bring such an application to Sharepoint, thereby reducing the total cost of ownership through one common user experience, development and deployment strategy.

      Similarly, Visio Services will allow customers to render Visio diagrams within the browser. An application of Visio Services would be a visualization of a SharePoint workflow that is rendered inside a browser as a Visio flowchart. Note that no client-side Visio software would be needed in this scenario.

     

    • Business Connectivity Services (BCS)
    • In MOSS 2007, Business Data Connector (BDC) was the tool of choice for integrating external data sources  into Sharepoint. BDC was available only to customers who purchased the enterprise version of MOSS 2007. Furthermore, BDC was limited to only reading data from the external data sources. Busincess Connectivity Services (BCS), which replaces the BDC functionality in SPS 2010, goes much further. Not only is BCS going to be available with the free, base version of SPS 2010 (renamed to Windows SharePoint Foundation from Windows SharePoint Services), it now supports updating, deletion and insertion of data into an external data source.  Another key advantage of BCS is that it will enable SharePoint Lists to be created directly based on a database table.

     

    • Developer Productivity Enhancements
    • First and foremost and unlike previous versions, it will now be possible to conduct SharePoint development on a client OS such as Windows 7. In earlier versions SharePoint development had to be undertaken on a server OS. This inevitably created overhead as the development had to be undertaken in a virtualized environment. Secondly, developer enhancements go beyond the support for client operating systems. Developers can now utilize familiar .NET patterns and practices such as LINQ (Language Integrated Query) to query data from a SharePoint List, XSLT for customizing a SharePoint list and Silverlight for creating richer visualizations. There are also a number of enhancements to the SharePoint programming model that offer greater flexibility in extending the out-of-the-box SharePoint behavior.  For example, SharePoint developers commonly extend the SharePoint behavior using small segments of code commonly referred to as event handlers. A common issue with event handlers in MOSS 2007 is that the event handler is not finished executing before the control is returned back to the user. This can lead to a confusing behavior for the users. SPS 2010 seeks to alleviate this by introducing “After-Synchronous” events which are guaranteed to complete execution before the control is returned to the user.

      Finally, Visual Studio 2010 contains extended capabilities for developers to create rich applications on the SPS 2010 platform. Many improvements have also been made in Visual Studio 2010 to increase developer productivity as well as take advantage of SPS 2010 functionality. These improvements help make it easier for .NET developers to create and deploy SharePoint solutions.

     

    • SharePoint Designer Enhancements
    • SharePoint Designer is a tool targeted towards business users and designers who have used it to quickly and easily perform actions such as customizing lists, changing layouts and creating simple workflows. The one area of improvement often requested is the ability to package and reuse the changes made using SharePoint Designer.  Fortunately, SharePoint Designer 2010 not only addresses this request, it also comes with major improvements with regard to designing workflows, editing SharePoint pages and setting up BCS ( discussed above). SharePoint administrators also have greater control over how SharePoint Designer 2010 is used within their environment. For instance, administrators can block the ability to perform certain actions such as modifying certain SharePoint pages and restricting access to certain areas with the Sharepoint setup.

      One other key feature provided by SharePoint Designer 2010 is the improved interaction between the business users and the IT department. Since SharePoint Designer 2010 uses the same packing and deployment format (commonly referred to as WSP solution package) as the rest of the SharePoint platform, it is now possible to take the work done by a business user within the SharePoint Designer 2010 and import it within Visual Studio 2010. This will allow the IT department to build upon and extend the work done by the business users. 

     

    • Business Intelligence Enhancements
    • Business Intelligence is another area with significant improvements. To begin, there are a number of scalability improvements to the desktop version of Excel 2010, including the support for the numbers of rows within a workbook that goes far beyond the 64K limit in the current version. The new version of Excel effectively utilizes enhancements in the OS such as the 64-bit support for access to large memory pools and enhancement in hardware such as the multi-core computers.

      Microsoft has made a strategic decision to leverage Excel as the primary BI analyst tool. As part of the self-service BI initiative, Excel users can import large amounts of data from various data sources and quickly generate pivot tables. Once the data is imported into Excel 2010, using a feature called “Slicers”, users will be able to easily filter the data. In other words, Excel users can extract, transform and load (ETL) data from multiple sources directly into Excel without requiring IT to build a formal data import setup. The pivot table generated from imported data can later be shared with other users by publishing it to SharePoint 2010. Under the covers, the publishing process creates an SQL Server Analysis Services instance that can be monitored by IT, using tools such as usage dashboard and resource quota.

      Excel Services, the server-side Excel functionality provided by SharePoint Server, has seen a number of improvements as well. First off, it directly benefits from the scalability improvements to the underlying calculation engine mentioned earlier. Secondly, it enables additional programmability enhancements. For example it, using a JavScript based program, it will be possible

     

    • In-Place Records Management
    • Records management relates to the process of marking a document as a record or “laminating” them for legal and compliance reasons. A records repository is a library of such records. SPS 2010 dramatically improves upon the capabilities for managing records by extending where and how the records are managed. Unlike MOSS 2007, SPS 2010 will support multiple records repositories within a single SharePoint installation.. This means that users can route records to multiple record repositories. Records managers can also define routing rules to aid in the classification of records. Records repositories now also support the use of hierarchical structures (commonly referred to as File Plan) for storing records in a manner that matches the customer’s organizational hierarchy.

      SharePoint 2010 also provides the flexibility for defining and applying retention rules. For instance, it is now possible to apply a recurring retention rule that specifies several stages of  record retention. This will be very helpful when a records manager wants to specify a retention rule such as “Maintain this document for three years after project completion then transfer it to the archive for the next five years”.

      In addition to using a records repository, SPS 2010 has the ability to declare records in-place, i.e., allow any document inside a document library to be marked as a record without the need to explicitly move it to a records repository. The new in-place record management capability exemplifies Microsoft’s mantra of “compliance everywhere”

     

    • Enhancements related to Large List Handling
    • Microsoft has placed a lot of emphasis on performance of lists with large numbers of items in them. This has resulted in the addition of the new SPS 2010 features known as : Query Throttling, Batch Query Support, Remote Blob Storage, and External Lists.

      Query Throttling allows IT administrators to limit -or “throttle” – queries that are executed against large lists from consuming too many resources. For example, SharePoint will now block queries that will return more than 5000 rows by default – and this is of course, a configurable setting.  Rather than iterating over each item, SharePoint also includes the capability to process multiple items in a list simultaneously, as a single batch operation.  If there is content that is more suitable for storage outside SharePoint (for instance large media files), the Remote Blob Storage feature will provide a mechanism to store it on file shares (SANS, NAS, RAID arrays, File Servers).

      Finally, External Lists is another mechanism by which large lists can be incorporated in SharePoint. As stated earlier, database-backed lists that are a part of BCS allow content to be stored in separate data stores – such as SQL Server – outside of the Content Database. The benefit of course is that this feature allows for providing supporting large lists without burdening the SharePoint 2010 content database.

     

    • Workflow Improvements
    • Workflow is a very useful feature of SharePoint 2007. In addition to the out-of-the box workflows that were included with SharePoint 2007, it is possible to use the SharePoint designer to build declarative (no-code) workflows. For more advanced scenarios, developers created custom workflows in Visual Studio. Two of the main challenges in building workflows with SharePoint 2007 relate to the limitations in SharePoint Designer 2007 and the workflow host that is part of SharePoint 2007. While SharePoint Designer allows business users to develop simple workflows, it requires a direct connection to the site for which the workflow is being developed, thus limiting the ability to reuse the workflow in different locations. Furthermore, there is no easy way to leverage code components developed by IT. The workflow host within SharePoint 2007 comes pre-configured with a set of services such as persistence, eventing and tracing. There is no way to add custom services to the runtime host. For instance, there is no direct way to allow SharePoint-based workflows to interact with events other than the list item events.

      SharePoint 2010 alleviates these challenges by allowing the workflows created using SharePoint Designer to be saved as templates for reuse. SharePoint Designer 2010 has also been enhanced significantly to allow business users to leverage tools such as Visio for workflow modeling and visualization. Another major improvement is the ability to modify out-of-the-box SharePoint workflows (e.g. approval workflow, three-state workflow).

      The workflow host within SharePoint 2010 now provides the extensibility option to inject custom services. It is also now possible to kickoff workflows without explicitly associating it to a list item.

      Finally, and from an overall scalability perspective, it is also going to be possible to designate application server nodes as workflow participants. In other words, unlike SharePoint 2007 where each application server node participated in executing the workflows, there will be a way to throttle the workflow execution to a limited set of machines. This allows isolation of services within the farm so system administrators can better manage resources and troubleshoot issues.

     

    • Social Networking Capabilities
    • One can debate the impact and applicability of social networking tools such as Facebook in the workplace, but there is no doubt that information workers today are demanding similar capabilities from the tools they use inside the workplace. This is why the SPS 2010 enhancements in this arena are so noteworthy. In SPS 2010, just about every element such as sites, documents, videos, blog posts are taggable. There is out-of-the-box support for navigating the tag clouds and lists. Users will be able to rate artifacts and recommend it to others. Co-workers can keep up with the latest by tracking the activity data available as a new feed. Individual users will be able to setup profiles, write to a common or personal board, indicate their presence/location and be able to take advantage of micro-blogging capabilities.

     

    • Deployment Advancements
    • Unlike the previous versions, SPS 2010 uses a solution package-based single, consistent scheme for packaging and deploying any customizations. The solution package is a compressed file containing all of the necessary components such as features, web parts, list definitions, assemblies, customized ASPX pages and workflows. Additionally, the previous versions required an extensive amount of manual modification of XML files. In SharePoint 2010, these modifications can be done through point and click configuration with SharePoint Designer 2010.

      SharePoint 2010 also introduces a notion of sandbox solution packages that are designed to improve security and manageability. Sandbox solution packages restrict the scope to which a customization can be applied (For example, restricting the scope for a list customization to site collection). Sandbox solutions are also restricted to a subset of SharePoint programmability options. Finally, administrators can enforce quote limits on resources consumed by sandbox solutions to prevent abuse.

      For a more detail review of the SPS 2010, I encourage you to download the white paper we released recently at http://tinyurl.com/sps2010 ( note – this link takes you directly to the PDF ~1 MB)