SharePoint and Azure Integration Opportunities

 

There is a lot of talk about the use of SharePoint, and other Office products, in the upcoming cloud offering Office365. Office 365 is SaaS (Software as a Service) style of cloud computing wherein one can subscribe and use the SharePoint functionality without worrying about the underlying hardware and software infrastructure. The entire underlying infrastructure is owned, built and administered by Microsoft. In addition, the continued management of the infrastructure – including application of patches, upgrades, backups and recovery operations, etc. – is also Microsoft’s responsibility. Subscribers pay one fixed monthly amount for Office 365’s services. Microsoft is able to offer a reasonable price point for a monthly subscription because Office365 is based on a shared, multi-tenant infrastructure. Of course, while the software and hardware is shared, data belonging to each subscriber is appropriately sequestered

While a multitenant offering built on a shared infrastructure can be a cost effective option, there are scenarios where it may not completely meet the needs. Here are some examples:

• Regulatory Compliance: A specific regulatory or compliance requirement may prohibit hosting documents in a shared or public cloud environment. (For example, Canadian privacy laws may restrict data that can be stored in a cloud data center located outside Canada). SharePoint Customization: Since the underlying infrastructure is shared across all subscribers, there is limited flexibility in installing custom modules. Customizations are generated limited to the constraints imposed by SharePoint sandbox solutions (Sandbox solutions run in a separate process and can only access a limited subset of the SharePoint OM).

One possible alternative to the above limitations is to obtain  a dedicated SharePoint instance in the cloud. This is possible in two ways today: i) For customers with a user base of 5000 of more users can take advantage of the dedicated level of service wherein the hardware is dedicated for the customer ii) For U.S. government organizations, Microsoft offers BPOS Federal, in which all data is stored in a secure facility dedicated to the customer. It is interesting to note that BPOS-Federal just achieved the FISMA (Federal Information Security Management Act) certification and accreditation.

• Other Platform Releases: There may be a need to run a previous version of SharePoint, such as Sharepoint 2007, and Office365 is based on SharePoint 2010. Conversely, there may be a need to tap into the functionality offered by a newer release of the .Net framework beyond the version that SharePoint is based on today (for instance, Windows Workflow 4.0).

The key points above represent dilemmas that customers face today. As shown in the diagram below, while SharePoint as a Service via Office365.com provides benefits such as lower costs and elasticity of resources, the flexibility to customize is limited, and there are potential compliance concerns. On the other hand, while setting up SharePoint on-premise addresses these issues , it also comes with the added responsibility of configuring and managing the underlying infrastructure.

Enter the Windows Azure Platform. Windows Azure platform is a PaaS (Platform as a Service) offering from Microsoft. It allows for building and running custom applications in the cloud. The key benefit of Windows Azure is that it offers elasticity of resources such as storage and computation. Customers can leverage these capabilities to “enhance” their on-premise, as well as SaaS-based, SharePoint installations. The diagram below depicts how both the on-premise and online SharePoint environments can leverage the Windrows Azure Platform: 

i) On-premise installations can scale out easily by relying on the virtually infinite storage and computing resources available on the Windows Azure platform. The key benefit, of course, is that massive scalability is available without undertaking a huge capital investment upfront. Instead, customers are able to dynamically scale up and down while only paying for what they use.

ii) Online installations can take advantage of the Windows Azure platform to run customizations. For instance, a custom-built workflow may be hosted within a Windows Azure role. Since the workflow is hosted on Windows Azure, there is minimal administrative overhead for ensuring that the workflow is up and running, OS patches are applied etc.

To better understand this concept let us briefly look at a few scenarios:

Scenario #1: Connecting to Cloud-based databases

This scenario is about leveraging SQL Azure-based databases within an on-premise SharePoint instance. SQL Azure is a database-as-a-service offering that is part of Windows Azure. It makes it easy to provision any number of databases that are easy to administer and come bundled with a highly availability setup.

As depicted in the diagram below, Business Connectivity Services (BCS) can be used to connect data contained within a SQL Azure database to an external list in SharePoint. You can choose to connect to the SQL Azure instance directly or via a Web or Worker role.

For detailed steps on connecting to SQL Azure using BCS please refer to this toolkit.

Scenario #2: Taking advantage of the data marketplace

This scenario is about leveraging the Windows Azure DataMarket to author SharePoint-based dashboards and mashups. The Windows Azure DataMarket provides a marketplace for data where content publishers can make their datasets available to consumers. This includes datasets such as Health and Wellness data, location-based services, demographics, etc. All of the data available on DataMarket is curated and highly available. It is also consumable via a consistent set of REST-based ODATA APIs. This makes it possible to consume the DataMarket data via PowerPivot add-in in Excel. Once an Excel workbook containing PowerPivot data has been authored, it can then be published to SharePoint. This allows SharePoint users to view and collaborate around the Power Pivot application.   For a brief demo of PowerPivot with a Windows Azure DataMarket datasource, please refer to this brief video I recorded a while back.

Scenario #3: Scalable routing of notifications

This scenario is about leveraging the Windows Azure-based infrastructure to route SharePoint notifications. The Windows Azure AppFabric Service Bus provides an infrastructure for large-scale event distribution. As shown in the diagram below, SharePoint notification such as the item added events (for instance adding a new announcement to a list) can be routed over the Windows Azure App Fabric Service Bus to a large number of listeners.

For detailed discussion on building custom SharePoint Service Application that can act as conduit to the Windows Azure AppFabric Service Bus, please refer to my earlier blog post.

Scenario #4: Dynamic Scaling

This scenario is about leveraging the elastic computational resources available on the Windows Azure platform. Windows Azure VM role allows customers to bring their own pre-created OS image to the cloud. Recently, as a proof of concept, we hosted an Excel Services instance via the VM Role. As shown in the diagram below, the benefit of such an approach is that number of VM role instances can be dynamically changed based on the load.  This means that the customer doesn’t need to have a large setup on-premise to handle expected and un-expected peaks. Instead, they can “burst out” to the cloud

For more information on setting up Excel Services in VM role please refer to a blog post by Harin Sandhoo from AIS.

Before I close, I would like to refer the readers to Steve Fox’s blog. Steve Fox has been a thought leader on this topic of integrating Windows Azure with SharePoint.

SharePoint as a services platform

Introduction

Over the last two years, SharePoint has increasingly been established as an enterprise platform  in organizations worldwide. Now, with the advent of SharePoint 2010, organizations have the ability to develop, host and manage custom services (as in SOA services) within SharePoint.

This new capability in SharePoint 2010 is  part of Service Application Framework (SAF) .  SAF  allows out of the box services ( such as search and managed metadata), as well as, custom developed services to be hosted on the SharePoint infrastructure. In this post, we will take a deeper look at SAF, explore the benefits it offers and discuss examples of services that can take advantage SAF.

Note – While SAF is new in SharePoint 2010, the Excel Services capability in MOSS is similar in concept. Excel Services introduced the notion of moving the processing out to a different application pool ( possibly to different machine altogether )

Services  Platform

SOA done right can bring a number of benefits to an organization including software reuse, better alignment with business processes and lower cost and  agile development cycles. However, achieving these benefits requires building on top of a platform that provides a number of capabilities including:

1) Governance and Registry – Capability to catalog and publish access to the available services.

2) Load Balancing and High Availability – Capability to make the services fault tolerant and resilient to varying loads.

3) Administration -  Capability to centrally manage services including configuration, security, leverage backup and restore functionality.

4) Security – Capability to secure services and set the security context.

5) Tooling  – Capability to author business logic by orchestrating different services through mashups and workflows.

For more information on  SOA terms and concepts please refer  to OASIS Reference Model for Service Oriented  Architecture at  http://docs.oasisopen.org/soa-rm/v1.0/

 

Services Application Framework (SAF)

As stated earlier, the latest version of SharePoint introduces the notion of Service Application Framework (SAF). SAF allows creation of custom services that can be hosted within SharePoint. These services can then be consumed by applications within  SharePoint. In fact, SAF forms the basis for all the core SharePoint services including Search, Managed Metadata and Excel Services.

Let us take a detailed look at SAF Object Model.

The diagram below depicts the overall SAF object model. SPFarm and SPServer classes have been available in the previous versions as well – SPFarm represents a SharePoint server farm while SPServer represents a server within the farm. SAF adds a few new classes to the mix. SPService represents a service (such as the Search service ) that is registered within a SharePoint farm. SPServiceInstance represents an instance of the service installed on an application server ( think of a service that is installed on more than one application server within the SharePoint farm). SPServiceApplication and SPServiceApplicationProxy represent logical endpoints (logical because they abstract the notion of multiple services instances from the consumers of the service). The fact that service is broken into distinct Application and ApplicationProxy classes, is a key concept that allows  services to take advantage of a flexible deployment , auto load-balancing and fault-tolerance.

Now that we have seen the key classes in the SAF object model we are ready to look into the various pieces required for creating a custom service.  The diagram below depicts the different pieces involved. 

Let us start from the bottom of the stack and look at the SQL Server tier.  Services rely on SharePoint configuration database for storing any settings. Additionally, it is possible to provision  a custom database ( separate from the content DB) as a persistence store.  Next is the application server tier. This is where the all the components that make up the service are installed. For instance, executables, Windows services and class libraries etc.  A WCF contract is the easiest way to expose the service endpoint. While SAF does not preclude the use of non-WCF stacks, using WCF offers a number of advantages such as the built-in service context, load balancer etc. Finally, let us look at the Web Front End. As stated earlier, a custom service is typically consumed by SharePoint Web Front End (WFE) such as a WebPart. WFE is where you would typically install the ServiceApplication proxy. The proxy abstracts the fact that a service can be installed on multiple application servers. Service consumers invoke the service via the proxy. The proxy, in turn,  figures out which application server to go to, by leveraging the built-in load balancer.

More concretely, implementing a custom service involves following high level steps:

1) Define the WCF endpoint (address, contract and binding) where the custom service will be hosted within the SharePoint farm.

2) Implement  the business logic and package them in the form of class libraries, executables, windows services.

3) Implement the SAF classes

Let us assume that we are building a service called Custom Service. Let us further assume that this service utilizes the WCF as its  communication stack and is hosted in IIS (note again that we are not required to use WCF/IIS)

Building Custom Service will require that we implement the following SAF classes:

CustomService – This class inherits from the SPIisWebService and is used to register the service with the farm. As part of inheriting from SPIisWebService we are required to provide installation steps, links to ASPX pages that are surfaced within the SharePoint central administration.

CustomServiceInstance- This class inherits from the SPIisWebServiceInstance and is used to register the service with an application server.

CustomServiceApplication- This class inherits from the SPIisWebServiceApplication and is used to register a logical endpoint. This class also encapsulated all the configuration settings associated with a service. For instance,   as part of inheriting from SPIisWebServiceApplication , we are required to implement a properties such as VirtualPath that tells SharePoint the URI of the service endpoint.

CustomServiceApplicationProxy- This class inherits from the SPIisWebServiceApplicationProxy and represents a connection to CustomServiceApplication.

4) Determine if a custom persistence store is required or not. If a custom persistence store is needed, we can have it  created as part of  the service application provisioning step and include it in backup and restore operation.

For more information on steps involved in building a custom service, including sample code, please refer DTaylor’s excellent SAF blog at http://blogs.msdn.com/b/dtaylor/

SharePoint as a Services  Platform

Now, armed with an understanding of  SAF, we are ready to look at how SAF enables SharePoint to be leveraged as a services platform.

 

1) Administration – Once the service has been registered with the farm, it can be managed via the SharePoint Central Administration. The following diagram depicts the central administration screen to add a new service  and manage an existing service.

2) Governance and Registry – SharePoint provides a topology service that allows SharePoint administrators to discover published services available within a farm. Typically, it is used by administrators to connect to services from remote farms. The following diagram depicts the UI for publishing an existing service via the topology service.

 

3) Load Balancing and High Availability – SAF allows multiple service deployment topologies to meet high load and availability requirements. The following diagrams depicts two different topologies.

 

This setup relies on a dedicated services farm (Farm 2)topology. Farm1 and Farm 3 subscribe to service offered by Farm 2. 

 

 

This setup depicts a single farm configuration. For load balancing purposes, the custom service is installed on three out of the four application servers in the farm. The service proxy is installed on both the WFEs.

 

 

4) Security – SharePoint supports claims-based authentication using Windows Identity Foundation (WIF). Custom services can take advantage of this capability to delegate user permissions to services. In other words, custom services can make authorization decisions based on delegated identity of the user involved( even if the communication is over multiple hops – across farms, for instance) 

5) Tooling – Tools such as SharePoint Designer,  Visio and third party tools such as Nyntex make it easy to create mashups by utilizing  OOB and custom services.

Custom Service Scenarios

Let us look at a couple of scenarios where developing custom services would be beneficial.

Integrate external applications into SharePoint

In this scenario, a customer has an existing proprietary .NET based task tracking application that they would like to  integrate into their SharePoint farm. In doing so, they are seeking to leverage the collaboration, maintenance and provisioning capabilities offered by SharePoint.

Using SAF they could package the backend code as a ServiceApplication that can be installed on one or more application servers. In addition, they would need to develop an application proxy that will be installed on WFEs.

The benefit of packaging the task tracking system as a custom SharePoint service is that the users can seamlessly use it in conjunction with other SharePoint capabilities such as authentication. In addition, the task tracking system is now managed within the unified governance model for the SharePoint farm and can thereby leverage its fault tolerance, backup and restore capabilities.

Externalize compute intensive processing

In this scenario, a customer requires elastic computing capability to handle high volumes of calculation requests. Rather than perform these calculations within the SharePoint infrastructure ( by adding application servers), it may make sense to take advantage of a cloud platform such as Windows Azure.

Using SAF, they could develop just the ServiceApplicationProxy that  simply forwards the incoming requests to Windows Azure. Note that ServiceApplication is not needed since the backend computation is happening within Windows Azure.

The benefit of abstracting the invocation of Windows Azure hosted calculations, as custom service is that customers can now seamlessly access it, not unlike how they would access the out of the box services such as search. So, for instance, consumers  can continue to use claims based scheme to invoke the service. Another benefit of this approach is that the configuration information, such as Windows Azure project details, can be maintained in the SharePoint configuration database.  

Summary

In this post, we have seen how SharePoint Service Framework (SAF) enables custom services to be developed, hosted and managed within SharePoint 2010. By taking advantage of this capability, organizations can develop new functionality and expose  existing software assets, as services that are managed within the SharePoint infrastructure. It is also expected that  ISVs will take advantage of this capability to expose their offerings as services within SharePoint.