Here are my personal favorite SharePoint 2010 new features and enhancements:

 

• Access Services & Visio Services

Building on the successful Excel Services pattern, SPS 2010 allows users to publish Access and Visio applications to SharePoint which other users can access through their browsers. As can be expected, Access and Visio Services will be somewhat limited in functionality(when compared to their respective desktop versions), but on the flip side, the customers would benefit from the server-side capabilities such as scalability, one centralized version of the document (rendered in a browser friendly format), access to multiple users and security control.   

Access Services will allow customers to publish their Access 2010 applications to SharePoint. For example, consider a departmental application such as a travel planner built using Access. Using Access Services it will be possible to bring such an application to Sharepoint, thereby reducing the total cost of ownership through one common user experience, development and deployment strategy.

Similarly, Visio Services will allow customers to render Visio diagrams within the browser. An application of Visio Services would be a visualization of a SharePoint workflow that is rendered inside a browser as a Visio flowchart. Note that no client-side Visio software would be needed in this scenario.

• Business Connectivity Services (BCS)

In MOSS 2007, Business Data Connector (BDC) was the tool of choice for integrating external data sources  into Sharepoint. BDC was available only to customers who purchased the enterprise version of MOSS 2007. Furthermore, BDC was limited to only reading data from the external data sources. Busincess Connectivity Services (BCS), which replaces the BDC functionality in SPS 2010, goes much further. Not only is BCS going to be available with the free, base version of SPS 2010 (renamed to Windows SharePoint Foundation from Windows SharePoint Services), it now supports updating, deletion and insertion of data into an external data source.  Another key advantage of BCS is that it will enable SharePoint Lists to be created directly based on a database table.

• Developer Productivity Enhancements

First and foremost and unlike previous versions, it will now be possible to conduct SharePoint development on a client OS such as Windows 7. In earlier versions SharePoint development had to be undertaken on a server OS. This inevitably created overhead as the development had to be undertaken in a virtualized environment. Secondly, developer enhancements go beyond the support for client operating systems. Developers can now utilize familiar .NET patterns and practices such as LINQ (Language Integrated Query) to query data from a SharePoint List, XSLT for customizing a SharePoint list and Silverlight for creating richer visualizations. There are also a number of enhancements to the SharePoint programming model that offer greater flexibility in extending the out-of-the-box SharePoint behavior.  For example, SharePoint developers commonly extend the SharePoint behavior using small segments of code commonly referred to as event handlers. A common issue with event handlers in MOSS 2007 is that the event handler is not finished executing before the control is returned back to the user. This can lead to a confusing behavior for the users. SPS 2010 seeks to alleviate this by introducing “After-Synchronous” events which are guaranteed to complete execution before the control is returned to the user.

Finally, Visual Studio 2010 contains extended capabilities for developers to create rich applications on the SPS 2010 platform. Many improvements have also been made in Visual Studio 2010 to increase developer productivity as well as take advantage of SPS 2010 functionality. These improvements help make it easier for .NET developers to create and deploy SharePoint solutions.

• SharePoint Designer Enhancements

SharePoint Designer is a tool targeted towards business users and designers who have used it to quickly and easily perform actions such as customizing lists, changing layouts and creating simple workflows. The one area of improvement often requested is the ability to package and reuse the changes made using SharePoint Designer.  Fortunately, SharePoint Designer 2010 not only addresses this request, it also comes with major improvements with regard to designing workflows, editing SharePoint pages and setting up BCS ( discussed above). SharePoint administrators also have greater control over how SharePoint Designer 2010 is used within their environment. For instance, administrators can block the ability to perform certain actions such as modifying certain SharePoint pages and restricting access to certain areas with the Sharepoint setup.

One other key feature provided by SharePoint Designer 2010 is the improved interaction between the business users and the IT department. Since SharePoint Designer 2010 uses the same packing and deployment format (commonly referred to as WSP solution package) as the rest of the SharePoint platform, it is now possible to take the work done by a business user within the SharePoint Designer 2010 and import it within Visual Studio 2010. This will allow the IT department to build upon and extend the work done by the business users. 

• Business Intelligence Enhancements

Business Intelligence is another area with significant improvements. To begin, there are a number of scalability improvements to the desktop version of Excel 2010, including the support for the numbers of rows within a workbook that goes far beyond the 64K limit in the current version. The new version of Excel effectively utilizes enhancements in the OS such as the 64-bit support for access to large memory pools and enhancement in hardware such as the multi-core computers.

Microsoft has made a strategic decision to leverage Excel as the primary BI analyst tool. As part of the self-service BI initiative, Excel users can import large amounts of data from various data sources and quickly generate pivot tables. Once the data is imported into Excel 2010, using a feature called “Slicers”, users will be able to easily filter the data. In other words, Excel users can extract, transform and load (ETL) data from multiple sources directly into Excel without requiring IT to build a formal data import setup. The pivot table generated from imported data can later be shared with other users by publishing it to SharePoint 2010. Under the covers, the publishing process creates an SQL Server Analysis Services instance that can be monitored by IT, using tools such as usage dashboard and resource quota.

Excel Services, the server-side Excel functionality provided by SharePoint Server, has seen a number of improvements as well. First off, it directly benefits from the scalability improvements to the underlying calculation engine mentioned earlier. Secondly, it enables additional programmability enhancements. For example it, using a JavScript based program, it will be possible

• In-Place Records Management

Records management relates to the process of marking a document as a record or “laminating” them for legal and compliance reasons. A records repository is a library of such records. SPS 2010 dramatically improves upon the capabilities for managing records by extending where and how the records are managed. Unlike MOSS 2007, SPS 2010 will support multiple records repositories within a single SharePoint installation.. This means that users can route records to multiple record repositories. Records managers can also define routing rules to aid in the classification of records. Records repositories now also support the use of hierarchical structures (commonly referred to as File Plan) for storing records in a manner that matches the customer’s organizational hierarchy.

SharePoint 2010 also provides the flexibility for defining and applying retention rules. For instance, it is now possible to apply a recurring retention rule that specifies several stages of  record retention. This will be very helpful when a records manager wants to specify a retention rule such as “Maintain this document for three years after project completion then transfer it to the archive for the next five years”.

In addition to using a records repository, SPS 2010 has the ability to declare records in-place, i.e., allow any document inside a document library to be marked as a record without the need to explicitly move it to a records repository. The new in-place record management capability exemplifies Microsoft’s mantra of “compliance everywhere”

• Enhancements related to Large List Handling

Microsoft has placed a lot of emphasis on performance of lists with large numbers of items in them. This has resulted in the addition of the new SPS 2010 features known as : Query Throttling, Batch Query Support, Remote Blob Storage, and External Lists.

Query Throttling allows IT administrators to limit -or “throttle” – queries that are executed against large lists from consuming too many resources. For example, SharePoint will now block queries that will return more than 5000 rows by default – and this is of course, a configurable setting.  Rather than iterating over each item, SharePoint also includes the capability to process multiple items in a list simultaneously, as a single batch operation.  If there is content that is more suitable for storage outside SharePoint (for instance large media files), the Remote Blob Storage feature will provide a mechanism to store it on file shares (SANS, NAS, RAID arrays, File Servers).

Finally, External Lists is another mechanism by which large lists can be incorporated in SharePoint. As stated earlier, database-backed lists that are a part of BCS allow content to be stored in separate data stores – such as SQL Server – outside of the Content Database. The benefit of course is that this feature allows for providing supporting large lists without burdening the SharePoint 2010 content database.

• Workflow Improvements

Workflow is a very useful feature of SharePoint 2007. In addition to the out-of-the box workflows that were included with SharePoint 2007, it is possible to use the SharePoint designer to build declarative (no-code) workflows. For more advanced scenarios, developers created custom workflows in Visual Studio. Two of the main challenges in building workflows with SharePoint 2007 relate to the limitations in SharePoint Designer 2007 and the workflow host that is part of SharePoint 2007. While SharePoint Designer allows business users to develop simple workflows, it requires a direct connection to the site for which the workflow is being developed, thus limiting the ability to reuse the workflow in different locations. Furthermore, there is no easy way to leverage code components developed by IT. The workflow host within SharePoint 2007 comes pre-configured with a set of services such as persistence, eventing and tracing. There is no way to add custom services to the runtime host. For instance, there is no direct way to allow SharePoint-based workflows to interact with events other than the list item events.

SharePoint 2010 alleviates these challenges by allowing the workflows created using SharePoint Designer to be saved as templates for reuse. SharePoint Designer 2010 has also been enhanced significantly to allow business users to leverage tools such as Visio for workflow modeling and visualization. Another major improvement is the ability to modify out-of-the-box SharePoint workflows (e.g. approval workflow, three-state workflow).

The workflow host within SharePoint 2010 now provides the extensibility option to inject custom services. It is also now possible to kickoff workflows without explicitly associating it to a list item.

Finally, and from an overall scalability perspective, it is also going to be possible to designate application server nodes as workflow participants. In other words, unlike SharePoint 2007 where each application server node participated in executing the workflows, there will be a way to throttle the workflow execution to a limited set of machines. This allows isolation of services within the farm so system administrators can better manage resources and troubleshoot issues.

• Social Networking Capabilities

One can debate the impact and applicability of social networking tools such as Facebook in the workplace, but there is no doubt that information workers today are demanding similar capabilities from the tools they use inside the workplace. This is why the SPS 2010 enhancements in this arena are so noteworthy. In SPS 2010, just about every element such as sites, documents, videos, blog posts are taggable. There is out-of-the-box support for navigating the tag clouds and lists. Users will be able to rate artifacts and recommend it to others. Co-workers can keep up with the latest by tracking the activity data available as a new feed. Individual users will be able to setup profiles, write to a common or personal board, indicate their presence/location and be able to take advantage of micro-blogging capabilities.

• Deployment Advancements

Unlike the previous versions, SPS 2010 uses a solution package-based single, consistent scheme for packaging and deploying any customizations. The solution package is a compressed file containing all of the necessary components such as features, web parts, list definitions, assemblies, customized ASPX pages and workflows. Additionally, the previous versions required an extensive amount of manual modification of XML files. In SharePoint 2010, these modifications can be done through point and click configuration with SharePoint Designer 2010.

SharePoint 2010 also introduces a notion of sandbox solution packages that are designed to improve security and manageability. Sandbox solution packages restrict the scope to which a customization can be applied (For example, restricting the scope for a list customization to site collection). Sandbox solutions are also restricted to a subset of SharePoint programmability options. Finally, administrators can enforce quote limits on resources consumed by sandbox solutions to prevent abuse.

For a more detail review of the SPS 2010, I encourage you to download the white paper we released recently at http://tinyurl.com/sps2010 ( note – this link takes you directly to the PDF ~1 MB)

That is my forecast, anyway. Over the last year, I have had the opportunity to talk to a number of customers, user group and conference attendees about Cloud Computing in general and specifically about the Windows Azure Platform. During these conversations, I have come across a number of concerns/ questions about Azure pricing, performance, security and so on. Clearly Windows Azure has a long way to go before it becomes a mature cloud platform. However, some broad-brush statements questioning Azure’s pricing, performance and applicability that I have come across lately, deserve some clarifications.  In this, post I have attempted to capture some of these concerns and provide my humble thoughts on why we need a take a holistic approach when evaluating the Windows Azure Platform.

Azure – A Platform as a Service offering

Let me begin by stating that not all cloud offerings are alike.  In their paper, “Toward a Unified Ontology of Cloud Computing” [0] Lamia Youssef et al. provide a detailed model for understanding the different classes of cloud providers. The following diagram depicts their proposed classification of cloud service providers including Platform as a Service (PaaS), Infrastructure as a Service(IaaS) and Software as a Service (SaaS).

Screen clipping from:  http://www.cs.ucsb.edu/~lyouseff/CCOntology/CloudOntologyPres.pdf

Microsoft Azure is a “Platform as a Service” offering, which is very different from what Amazon is offering via EC2 (commonly classified as “Infrastructure as a Service”) or even Google’s App Engine (GAE), commonly classified as a specialized “Platform as a Service” offering [1]. As a result of this difference in approach, the lifecycle of a Windows Azure application lifecycle is different as well. As depicted in the figure below – A developer develops the code in the developer fabric running locally and then simply publishes it to the cloud, along with a description of the desired infrastructure model. Based on the uploaded code and the infrastructure model, Azure orchestrates the provisioning of the computing, storage and networking resources in a manner that is fault tolerant and allows for rolling upgrades. It is clear that Microsoft has had to innovate at 100 mph in order to make this concept a reality. I encourage you to read a recently released Microsoft research paper on Helios that describes an OS abstraction across a number of heterogeneous CPUs. Make no mistake; Azure is the only cloud offering that is trying to create a “Cloud OS” abstraction around a fabric of Windows machines. This is not to suggest that EC2 or GAE offerings are not useful. Far from it – If one is looking for an on-demand Virtual Machine provider, EC2 is a great choice. After all, Amazon is the king of retail and as its executives like to say with a grin “we will gladly match the lowest price offered by our competitors”. On the other hand, if you have a Java or Python based web application that can fit GAE pre-defined application structure and framework  – GAE would be a great choice. GAE is already successful with over 45,000 applications by the last count. In the end, different requirements will require the use of different types of cloud offerings.

    1) Windows Azure manages applications not just servers

    2) Tell it what you want, and it will automate the details

   3) Model-driven automation

   4) Platform insures service isolation

Now that we have an understanding of the different flavors of cloud computing, let us circle back to where I started and address some of the broad-brush statements, I alluded to earlier. 

Concern #1: “The Car Analogy  -  Azure Pricing model is fundamentally broken”

“Azure is like a car that has its engine running constantly, even when it is not being driven “.

The author of the above analogy is trying to contrast Azure pricing with GAE where you only pay for the time you use the service.

“Azure is like a car that has its hood sealed shut. One cannot look what is under the hood, leave alone reconfigure it“.

The author of above analogy is trying to highlight the fact that one cannot directly access the virtualized instance that is running the Azure based application.

I think each of the above analogy only conveys a half-truth about Azure. I think a better analogy for Azure is this:

“Azure is a like a car you rent from ZipCar  (a hourly car rental company)”

The price of the ZipCar rental includes gas and insurance cost. Of course, you pay for the hour whether you use the car or not (ZipCar has to spend money to honor a reservation you have made). But at any time, one can simply return the car and incur no additional charges. Would someone care to look under the hood of a ZipCar – I certainly would not. This is against the very reason I want to get a car from ZipCar in the first place – once I pay for the hourly rental I don’t have to worry about insurance or fuel. If something goes wrong, I can simply wave my ZipCar card and get into the next available car. And if for any reasons, my co-passengers want to go on a little excursion – they can, on a short notice, get additional cars at the same fixed price and give them up when they are done with them.

So how can GAE offer a model that requires the customers to pay only for the time their application gets used, when EC2 and Azure cannot? The answer lies in the figure below – as you want more control the economy of scale goes down.  Azure and EC2 give you more control (in that order) and as a result, cannot take advantage of the economy of scale at the same level as the GAE. GAE, Azure and EC2 are each offering a multitenant platform but the degree to which each the tenant can configure their individual setup is different. I will not be surprised if an ISV were to come out with an Azure based cloud offering that is constrained in some way (for example, it allow customers to submit their HttpHandlers ) but in turn, offers a usage based pricing model.

l

There are two other pricing related concerns that I have come across.

“I can host 2 blogs and a wiki for the price of a single Azure Web role“.

Hosting a blog or wiki is not likely the target application for Azure Web Role. If you want to host a blog or wiki, Business Productivity Online Suite (BPOS) [5] offering will be a more cost effective.

“My hoster allows me to create multiple virtual directories (vdir) inside a single VM. With Azure, I need multiple Web Roles to achieve the same setup“.

First off, it is not appropriate to compare a vdir offered by a VM where you have completed access to the machine (and in turn assume all the responsibility for administering it), with the vdir offered by a web role – web role offers fault tolerance, monitoring, load balancing and other SLA guarantees. While these differences are important, in the end, it is really about raising the level of abstraction. Readers will recall that when IIS 6 came out with the default out-of-process model, there were some concerns that provisioning a distinct process for each web application would be expensive (as opposed to an in-process or pooled process model supported by IIS 5.0). But the isolation and fault tolerance offered by an out of process model quickly sidelined any concerns about the cost of provisioning additional processes.

Concern #2: “Azure does not scale dynamically”

This concern is based on the fact that one has to specify the exact number of Azure role instances that one wishes to provision. In the current CTP, there is no way to specify a range for the role instance. In other words, letting Azure determine when it is appropriate to ramp up or ramp down the number of running instances, automatically based on the load on the application.

Frankly, this is not as big a deal as it sounds. One can always write a small piece of code that spawns new instances (or conversely, shuts down unused ones) based on the health alerts. Also note that the underlying fabric does indeed support providing a range (refer to the PDC 08 session [2]  “Under the Hood: Inside the Windows Azure Hosting” – fast forward to minute 67:14). I think that it is a matter of time before Microsoft exposes this capability to the end users.

Concern #3: “Azure makes sense only for the large enterprises”

“The massive scalability, fault tolerance and high availability are well suited for large enterprises. Most of the applications we build as a small/medium business don’t need these capabilities”.

Let us consider a small/medium sized web retailer. Based on the available, Azure pricing it would typically cost ~$360/month to run (refer to the cost breakup provided below) a small /medium web site – all without any upfront investment in staging / production / disaster recovery hardware or software costs. 

To compare the above cost with a hosted/on-premise solution, I checked with one of our customers on how much they were paying to host their servers in a data center in Herndon, VA. Here is the information I was provided:

Furthermore, even small retailers are subject to compliance with the security and auditing standards such as Payment Card Industry – Data Security Standard (PCI-DSS) and Statement on Auditing Standards (SAS 70). By hosting their application in the Microsoft’s cloud infrastructure, even small retailers can take advantage of security certifications that Microsoft has earned including SAS 70 and ISO/IEC 27001-2005 certifications [4].

Let us consider another example – I am working with a customer who has a very limited budget for hosting a small HR application in Canada, because of the local laws, this application must be hosted within geographical boundaries of Canada – unfortunately they have a very small office in Canada with limited or no data center capabilities.  At first glance, the option to purchase a small Windows box and placing it in their office in Canada seemed most cost effective. But once they added up the cost to manage the setup, it was clear that total cost of ownership would be not be very cost effective. This customer is now looking at hosting their application in Azure – hoping to take advantage of Geo-Location guarantees it offers. The key point to take away from this example is this – by taking advantage of the global data centers, even small to medium businesses can achieve a world-wide reach.

One final point about pricing, it is ultimately the free market model that will determine where the PaaS pricing lands. As is evident from the major mid-course correction regarding the SQL Data Services offering, Microsoft is going to have to listen to and adjust, based on customer and market demands. I also expect that Microsoft would offer special promotions initially(for MSDN users, small businesses and startups – similar to the BizSpark program) to build the momentum.

So, do the cost estimates for Azure put it out of reach of most small/medium businesses, I will let you be the judge. 

Concern #4: “I am out if there is no Remote Desktop Access”

This is probably the most often requested feature. It is easy to see how remote desktop access would be helpful; it allow access to the desktop, install software components, custom configuration – registry tweaks etc. While remote desktop may very well be supported in the future, keep in mind that the tradeoff will be that customers would be required to assume the responsibility for additional administrative tasks including patch management and access control.

Once again the underlying fabric may already support this capability. During his PDC 08 Manuvir Das talked about an “escape hatch” or raw mode as something that is available under the covers.

Screen clipping taken from http://channel9.msdn.com/pdc2008/ES16/ Fast forward to the 21.44 minute mark in the presentation.

Concern #5: “You cannot seamlessly move your Azure application back to the datacenter”

The vendor lock-in argument against Azure is that once you build an application for Azure, it will be hard to move it back to an on-premise and hosted data center.  Frankly, vendor lock-in is a concern with any cloud provider. As I heard David Chappell say recently – “there is no lock-in like the cloud. Cloud provider can turn the spigot of the service off, rather abruptly” ( I have paraphrased his quote a bit as I cannot recall his exact words).

Clearly, trust in cloud computing can only build over time. And hopefully over time, through good experiences, concerns over vendor lock-in will diminish as well. 

If we take a closer look at what it takes to build an Azure hosted application, the lock-in concerns do not seem as grave.

For instance, it is encouraging to see that Microsoft along with IBM, Rackspace and others have agreed to join the Simple API [3] project by Zend Technologies that provides a common File Storage, Document Storage and Simple Queues API  against a cross-section of vendor offerings including Rackspace Cloud Files, Windows Azure Storage, Amazon S3 and Nirvanix.

The Windows Azure team has also doubled-down on the more open RESTful web API, thus making it easier for non .NET applications to consume it. The Azure team has also recently unveiled the native mode execution capability that lifts the restriction on running applications that are based on managed .NET code only. This means that it is now possible to host application built using Python/ CGI on Azure.

As I have stated earlier, Azure development is squarely based on .NET building blocks WCF/WF/ASPX and so on. So fundamentally you are writing .NET code – it is important to understand that the only new API introduced with Azure – called the Service Runtime API – fits in one screen (shown below). So there is not going to be a large body of Azure specific syntax that needs to be ported, should you decide to move the application back to your premises.

While the code you write for an Azure application is all .NET based, there are a number of guidelines [8] that one will need to adhere to, for a successful implementation on Azure.

· Develop for the sandbox

· Prefer scale out over scale up

· Separate out the state from UI ; store the state in a distributed horizontal storage

· Be loosely coupled  

· Be prepared to handle varying loads

· Deal with failures; build retry logic that is idempotent

· Rolling upgrades – upgrade your application without any downtime

· Rely on unified logging ; build an alert mechanism

Fortunately most of the above bullets are the guidelines that the industry as a whole has been chasing over the last decade. So to the extent we can inculcate these in our design, our applications will better off – whether we choose to host them in the cloud or not.  I would even go to the extent of saying that cloud computing done right is a great hope for all of the industry.

Concern #6: “Azure is slow compared to EC2 or GAE”

How would a performance of a Web application hosted inside Azure compare with a similar application developed using EC2 or GAE? First off, this is an apples-to-oranges comparison. With EC2 you can ask for a specific hardware configuration (for example, you have the option to select an EC2 instance that has 8 virtual cores with 2.5 EC2 compute units each [7]). Azure offers full relational capability that EC2 and GAE don’t. Azure offers a batch processing capability that GAE does not. I could go on and on but you get the idea. These are three different approaches to cloud computing and offer distinct performance optimization strategies.

Closing

In closing, let me say this – I would be pollyannaish to suggest that Azure is a perfect cloud offering. Only time will tell how successful Azure is.  After building a handful of Azure applications ( mainly POCs), I, like many others, have run into some challenges with the current CTP – Here are a few examples of the kinds of problems I am talking about  – Provisioning ( I am not talking about new applications, even upgrading applications where the model is unchanged ) is  very slow. The slowness in provisioning further exacerbates the fact that the provisioning is very coarse grained – even if you had to change a single master page, code behind or CSS file, one has to go through the slow process of upgrading my application. Another example would be the pricing of .NET Service bus – based on the information available so far, it is very hard to develop a pricing model for a complex app – how do you differentiate between TCP connection vs. streaming connection, what does a transaction mean exactly? The diagnostics support is rudimentary – all you have are the logs (no access to the event logs, server logs etc.). Furthermore, it takes several minutes to copy them to the blob storage. Finally, there is only support for two types of roles – Web and Worker. What if I wanted to add on specific software (for example Excel Services)? Or, what if I wanted to run .NET 2.0 or even a .NET 4.0 based applications? (Note that a number of enhancements to the logging functionality have been announced very recently [6] including ability to look at the performance counters and auto copying of logs on a periodic basis)

The Azure team has heard the above feedback many times and has their work cut out for them. They are working hard to alleviate some of aforementioned challenges by the time Azure is launched commercially in November and in subsequent releases. And for the rest of us, who want to build cloud application using the .NET building blocks we know and the VS.NET based tools we love, work is cut out for us as well – brainstorm about the applications that can leverage the Azure platform, build prototypes, provide feedback on the pricing models, develop tools & utilities and last but not the least, demand the best product possible.

References

[0] Towards a Unified Ontology of Cloud Computing – http://www.cs.ucsb.edu/~lyouseff/CCOntology/CloudOntology.pdf

[1] Above the Clouds: A Berkeley View of Cloud Computing http://www.eecs.berkeley.edu/Pubs/TechRpts/2009/EECS-2009-28.pdf

[2] Dynamic Scaling – http://channel9.msdn.com/pdc2008/ES19/

[3] Simple API – http://www.simplecloud.org/

[4] Securing Microsoft’s Cloud Infrastructure – http://www.globalfoundationservices.com/security/documents/SecuringtheMSCloudMay09.pdf

[5] Business Productivity Online Suite – http://www.microsoft.com/resources/Technet/en-us/MSOnline/bpos/html/99d9ede5-ce15-476c-9a3f-d42a481d287e.htm

[6] Azure team recently announced the following enhancements to Windows Azure Logging – http://blogs.msdn.com/windowsazure/archive/2009/10/03/upcoming-changes-to-windows-azure-logging.aspx

[7] Amazon EC2 Instance Types – http://aws.amazon.com/ec2/instance-types/

[8] Windows Azure: Cloud Service Development Best Practices - http://channel9.msdn.com/pdc2008/ES03/

Recently, I have been working on a number of SharePoint database issues related to large number of workflow instances.  Here are a few things I encountered:

• System.Runtime.InteropServices.COMException (0×80004005) is a catch all for all database and other interop errors  – You can get this error for any number of reasons – for example out of transaction log space, deadlocks etc. I have seen a number of posts on the web attributing this error to one condition or another.

• Turn on WF tracing (see my previous post for more information). Trace logs provide a great source of information on how a given workflow instance has progressed. One can find the workflow instance that encountered an error. Once you have the workflow instance, you can query the workflow and workflow association tables to determine the related SPWeb, SPTask etc.

• Because SharePoint uses GUID as a the primary key it can result in a fragmentation (Kimblerly Tripp has a nice post about this) that one needs to be aware of.

• We found that despite the weekly timer job ( note that there is a difference between SP1 and SP2)statistics on certain tables can be off, resulting in less than optimal plans, which in turn  can cause deadlocks in some instances.

• It is important to review the content database maintenance plan and tweak it as needed – Another useful post from Kimberly Tripp – http://www.sqlskills.com/BLOGS/KIMBERLY/post/Database-Maintenance-Best-Practices-Part-I-e28093-clarifying-ambiguous-recommendations-for-Sharepoint.aspx

• Check if max degrees of parallelism is having an impact? Try re-running your tests after MAXDOP to 1.

• Look for lock escalation warnings – Try altering the locking threshold using the trace flags (1211, 1224).

• Review your code for potentially expensive WSS OM calls such as BreakRoleInheritance. Perform static analysis using tool such as SPDisposeCheck to make sure resources are being correctly disposed.

• Because of their episodic nature, workflows are difficult to automate the testing of workflows – There is a nice MSDN  article on unit testing workflows.

Load testing workflows is even harder. We developed a tool that allowed us to load test our workflows. This tool is itself a workflow program that drives a configured number of workflows – you basically supply the workflow types, users, how the tasks need to be updated (accepted, rejected etc ) and in what order. Having a load test tool allowed us to consistently recreate some of the issues we were encountering in production.

• Active management of lists is very important. Archive old items from lists regularly as appropriate.

URL http://ais.cloudapp.net                                                        

Source Code  http://cid-818700175481d002.skydrive.live.com/browse.aspx/Blog

I (along with Harin Sandhoo from AIS) recently worked on converting a subset of DinnerNow.net application to Azure. This post captures some of highlights of the porting effort.

DinnerNow.net is a reference application developed by Microsoft to showcase .NET 3.5 functionality. This application  can be broken up into three subsystems:

1. Web site where customers can order food from a variety of restaurants in their local delivery area.

2. Smart client application that allows a restaurant manager to view the incoming orders and update their status.

3. Mobile application that allows a delivery person to be notified when orders are ready for delivery.

It made sense to start with porting the web site to Azure. In future, it may be interesting to look into building a Silverlight equivalent for the restaurant manager piece and perhaps, a live mesh based mobile delivery application.

Current Architecture 

DinnerNow.net web site is implemented as ASP.NET application hosted inside IIS 7. Data is currently stored in SQL 2005, a LINQ to SQL based data access layer is used to persist the data. Business service functionality is using WCF and workflow service (also hosted inside the IIS 7)

[ Readers who are not already familiar with Azure concepts such as roles, may find it useful to review this first ]

Proposed Architecture  

The Azure version of DinnerNow.net utilizes the web role to host ASP.NET code as well as the WCF services. For now (because of the limits on the # of projects under the CTP) the UI code and WCF endpoints are hosted within the same web role. In the future, it will make sense to move the WCF endpoints to a separate web role. An Azure Table based membership provider ( part of the Azure SDK sample) is used for authentication.

A worker role is used for some background processing tasks such as storing the order into the database. The communication between the web and worker role takes place via the Azure Queue. The worker role is also responsible for communicating with the workflow program (ProcessOrder.xoml) that places the submitted order to a queue for further processing by the restaurants. As of July 1st, Azure Workflow Service has been taken down.  As a result, we have taken out the workflow service related code. The key motivations for including Azure Workflow Service, in the first place, included 1) a robust host for the workflow program 2) connectivity to applications inside the enterprise (such as the restaurant manager application)

The biggest challenge in porting the application was related to the database. Even though the relational features for SDS have been announced, they are not yet available. This forced us to rely on Windows Azure Table as the persistence store. The key consideration in moving the data from relational DB to Azure Table is the partitioning strategy. Consider the following the dbml diagram depicting Order, OrderDetail and OrderPayment, LINQ to SQL classes. Azure Table supports a flexible schema  that allows entities of different types to be stored within a single table. Since there was a need to retrieve Order and OrderDetails together,  we clustered the two together inside a single table for efficient retrieval. We partitioned the data according to order id and the RowKey was used to differentiate between the entities (i.e. “OrderItem_”+ DateTime , “Order_” + DateTime). Since the sorting is lexicographic, we use fixed length format based on Ticks.

Azure Table supports ADO.NET Data Services and REST. Fortunately, the data access code is not significantly different.  So for example, the existing LINQ to SQL query inside the GetOrdersForRestaurant method that looks like this:

var ordersByRestaurant = (from o in db.Orders.Distinct()
                           where (from od in db.OrderDetails where od.RestaurantId == restaurantId select od.OrderId).Contains(o.OrderId)
                           select new DinnerNow.Business.Data.Order()

is changed to the following query when working with Azure Table:

var qResult = (from oItems in _context.CreateQuery<OrderItemEntity>(OrderTableName)
                   where oItems.RestaurantId == restaurantId
                   select oItems);

For additional details please refer to the file OrderProcessing.cs in the sample code that accompanies this blog post.

Code View

The following diagram depicts the code view for the project. DinnerNow.CloudService is the Azure Service project that comprises of the web and worker role. The web role is mapped to the DinnerNow.WebUX project. Similarly the worker role is mapped to the DinnerNow.WorkerRole.

Miscellanea

• It is recommended that configuration data be stored inside the CSCFG file (as opposed to web.config). This is because CSCFG file is stored outside the application package you upload to the Azure Portal – The application package is really a diff disk that gets applied to the base HyperV image. By storing the configuration data in the CSCFG file means that you can make changes without the need to upload a new  application page. To read config setting from the CSCFG file use GetConfigurationSetting method of the RoleManage class. Like so:

var sslPort = Microsoft.ServiceHosting.ServiceRuntime.RoleManager.GetConfigurationSetting(“sslport”);

• Our code is based SDK March CTP.
• ASP.NET code is running in  full trust (now available with the March CTP).
• Testing – We mostly relied on the mixed mode testing – wherein the code was executed in the developer fabric but the data was in the Azure Table.
• We used Cerebrata’s nifty cloud storage tool for  all our testing – https://onlinedemo.cerebrata.com/Cerebrata.CloudStorage/default.aspx

This blog post is about tips and tricks for monitoring the health of SharePoint workflows. I will like to suggest  this excellent MSDN article for additional information.

Logging information about the progress of workflows

The WF tracking service logs the events as the workflow progresses along.  For example, consider a simple workflow (see below) that creates a task and then loops until the task is 100% complete.

By turning the tracking service on (see [1]) we can capture entries like the following, the trace output below corresponds to the workflow in Figure 1 above:

As you can see from the snippets above, we have information about step by step execution of the workflow.

Workflow Failure Conditions

Some of the common reasons why a workflow can fail include:

a.      The WF program instance errors out because of an exception. For example, if a null object reference is encountered inside the CreateTask handler, in the workflow in Figure 1, above. This exception will cause the WF instance to move to the error state (reflected in the status column of the document library)

The tracking service will record this exception and typically log detailed information as shown below:

One more point about exception conditions – there are times when we need to thrown an exception ourselves. For instance, if the user does not have appropriate permission, or there is missing data. In those cases, we could set a custom error message (See [2])

b.      The Application pool is recycled.  The state from a previous persist point (if there was one – typically a delay activity or OnTaskChanged activity) is persisted in the database, but since there is no retry mechanism, there is no way to re-start the persisted workflow instance. For example, if the sample WF program instance was executing inside the While activity (whileActivity1 in Figure 1) when the app pool crashed, there is no automatic way to have the workflow restarted.

One potential solution would be to model the workflow as a state machine and include retry logic. But this would add complexity to the workflow.

c.       If the correlation token is being set dynamically, there is a chance that the value gets incorrectly set in some cases. As a result, the waiting WF program instance will never receive the event. A correlation token is an identifier WF uses to tie activities to a common task – for example if CreateTask, OnTaskChanged and CompleteTask relate to a single task; they should have the same correlation token.

d.      SharePoint workflow activities (like Create Task) delay database commits until a persist point is reached. This means that a CreateTask activity will not result in a “real” task being added to the list until a persist point is reached. So any direct SharePoint OM calls that attempt to reference the created task will fail until the point a persist point is reached. Please refer to [3] to dump out the

 [1] Workflow Diagnostics

Add the following section to the web.config:

Additionally, we can use stsadm to capture the trace messages from Workflow Infrstructure as shown below:

[2] Adding custom status message

1.       Add a custom status in workflow.xml

2.       Add the following code for the invoking method

 [3] Code to extract the persisted workflow state

Since the workflow runtime is finicky about changes such as adding private variables, it is useful to dump the persisted state.

I read Andrew’s column on support for the relational capability within SDS. While it is great to have “change the conn string to move to cloud” capability, I think it is useful to understand how the relational model is enabled by SDS.  Additionally, in my opinion, it will be also be important to understand how the entity based storage (offered via Azure Storage) can be leveraged when designing applications for the cloud.

The architecture of the data fabric, that powers SDS, is a node based, scale-out architecture. Let us go over some of the core concepts of the data fabric :

• Storage Unit – Basic unit of storage that supports the CRUD operations. Example – database row.
• Consistency Unit – Set of storage units that can be queried and updated in a consistent manner. Example – A set of rows with the same partition keys or even an entire database instance.
• Failover Unit – Group of consistency units that are guaranteed to be available. SDS replicates the failover units to a replica set to ensure that data is always available.

Here are some things to consider when using the SDS relational model (this is based on my limited understanding, of course. I urge you to view Gopal’s excellent Under the Hood talk for additional details)

• Consistency is guaranteed inside a single node or consistency unit. This means that if your entire on-premises database instance can fit into one consistency unit, you are OK.  But on the other hand, if you are going to exceed it, you will need to think about partitioning your relational model – Until the time Microsoft adds auto-partitioning functionality on top of the relational model. Note that there is no support for transactions that span across consistency units.
• Data fabric uses dynamic partitioning to improve performance  i.e. it can move a consistency unit around to spread the load evenly across the cluster of nodes. A single fail-over unit is designed to host one more storage units vs. dedicating the entire fail-over unit to a single consistency unit. This is because a) it easy to re-balance the load by moving the “smaller” consistency units around b) easier/faster to recreate a failed node. When you move the your entire relational DB instance to the cloud as a single consistency unit, chances are that data fabric will need to dedicate the entire fail-over unit to it (to improve performance). This can limit some of the benefits of dynamic data partitioning.
• Bear in mind that the data fabric is based on scale-out using commodity hardware (typically 1.5 to 1.7 GHz X64 and 1.7 GB of memory – this is based on Chuck Lenzmeier’s talk at the PDC and obviously subject to change).
• Consider the additional latency cost due to the fact that writes are propagated to a quorum of a replicas.

As stated earlier, it might be worthwhile to consider the entity based storage offered by the Azure Storage, for the following reasons:

• If you need flexible schema or are building analysis focused applications – an entity based storage may be more suitable.
• Entity based storage forces you to think about partitioning from ground up. This allows for almost linear scaling as additional nodes are added to the mix. 
• Single index (Partition Key + Row Key )may seem limiting but there are number of ways of get around this including dynamically adjusting the partition key, storing multiple types of entities inside a single table so we can partition the data  using partition key  + entityType.

Recently I had to work on an interop scenario where WCF client needed to call a WSS4J service. After a bit of experimentation, I came up with the following configuration.

Here are some of the key settings to note:

Authentication Mode  MutualCertificate

Message Protection Order  SignBeforeEncrypt

SOAP Version  1.1

Algorithm Basic128Rsa15

Message Security Version  WSSecurity10WSTrustFebruary2005WSSecureConversationFebruary2005WSSecurityPolicy11BasicSecurityProfile10

<?xml version=“1.0“ encoding=“utf-8“ ?>

<configuration>

    <system.serviceModel>

            <client>

                  <endpoint address=“http://gp64156.exampleCorp.net:8080/eservices/aService“

                behaviorConfiguration=“ClientCertBehavior“ binding=“customBinding“

                bindingConfiguration=“JavaInterop“ contract=“exampleCorp.Proxy.aProfileManager“

                name=“UserProfileManager“>

                        <identity>

                              <dns value=“userprofilews“ />

                        </identity>

                  </endpoint>

            </client>

            <bindings>

                  <customBinding>

                        <binding name=“JavaInterop“>

                              <security defaultAlgorithmSuite=“Basic128Rsa15“ allowSerializedSigningTokenOnReply=“true“

                        authenticationMode=“MutualCertificate“ requireDerivedKeys=“false“

                        securityHeaderLayout=“Lax“ includeTimestamp=“false“ messageProtectionOrder=“SignBeforeEncrypt“

                        messageSecurityVersion=“WSSecurity10WSTrustFebruary2005WSSecureConversationFebruary2005WSSecurityPolicy11BasicSecurityProfile10“>

                                    <localClientSettings detectReplays=“false“ />

                              </security>

                              <textMessageEncoding messageVersion=“Soap11“ />

                              <httpTransport />

                        </binding>

                  </customBinding>

            </bindings>

            <behaviors>

            <endpointBehaviors>

                <behavior name=“ClientCertBehavior“>

                    <clientCredentials>

                        <clientCertificate findValue=“CN=client.com“ />

                        <serviceCertificate>

                            <defaultCertificate findValue=“userprofilews“ storeLocation=“LocalMachine“

                                storeName=“TrustedPeople“ x509FindType=“FindByIssuerName“ />

                        </serviceCertificate>

                    </clientCredentials>

                </behavior>

            </endpointBehaviors>

        </behaviors>

    </system.serviceModel>

</configuration>

Generating reports based on the data contained within SharePoint lists is a common requirement for many projects, and yet, there is no easy or direct way to realize this functionality. Fortunately, extensibility options provided by MOSS and SQL Server Reporting Services (SSRS) can help. This post briefly discusses the various approaches for generating reports on MOSS based data.

SharePoint Storage Basics

Sharepoint provides an abstraction around the underlying data storage (aka Content DB) wherein users don’t interface directly with the content database. Instead, the users work with lists and SharePoint in turn, stores the information appropriately inside the content DB.

To be able to support various lists (lists with varying content types), SharePoint utilizes the universal pattern – where a single database table is used to capture information stored in different lists. A universal table (the userdata view inside the content DB) contains generic columns such as int1, int2, varchar1, varchar2, etc. that are mapped to the fields in a list. The mapping between fields and universal table columns is maintained inside the lists view as shown below.

Here is an example of a field (Title) that is based on an existing site column

<FieldRef Name=”Title” ColName=”nvarchar1″/>

Here is an example of a field (Address) that is defined as part of a list

<Field Type=”Text” DisplayName=”Address” Required=”FALSE” MaxLength=”255″ ID=”{76958f23-d514-4d2a-bb16-373429eb6569}” SourceID=”{45581111-aa7e-4567-9a03-684a7ef051e8}” StaticName=”Address” Name=”Address” ColName=”nvarchar3″ RowOrdinal=”0″/>

The universal table implementation in SharePoint goes a step further, it allows fields to span across multiple rows, if needed e.g. if a custom list has more integer columns than the 16 that are available in a single row.

As you can imagine, the universal pattern is really convenient from a manageability perspective. An alternative design would be to provision a new table for each list or content type. Unfortunately the downside of the universal pattern is pretty obvious as well. For example, SQL Server will lock multiple rows of a list (or folders within a list) even when a single list item is being updated. In an extreme case, the entire table may be locked leading to single-threaded access.

The other obvious disadvantage is the limited ability to tune the performance, for example optimizing the index. With WSS 3.0, we now have the ability to mark a single field within a list as an “indexed” field. Indexed field can lead to significant gains in performance as long as they are used in the where clause of the query.

While it is useful to look under the covers to understand the structure content db, Microsoft does not want users to have any dependencies to the underlying tables (for good reason – the underlying storage format is subject to change without notice!). Instead, they provide a number of different object model (OM) based access schemes to safely access this data. These include

1) Search OM – Allows executing queries against the search engine

2) Query OM – Allows executing CAML queries using classes such as SPQuery amd SPSiteData QueryThis comes in two flavors – class library and web service based OM.

The White Paper in reference [1] below discusses all the OM options and how they perform under load. I recommend reading this paper in its entirety. In a nutshell, SPQuery using indexed column is the best option in terms of performance. This paper also talks about using PortalSiteMapProvider based scheme that uses caching to improve performance. I am ignoring this option for the purposes of this discussion as it does not fit the reporting needs very well.

When to use Content DB as custom application data storage

It should be clear from the previous section that the universal table is not designed to be:

High performing - Because of the limited indexing and locking options it is not designed for a high throughput OLTP style access.

Tunable – Schemes such as partitioning to split very large lists are not available. There is also no ability to tune the default SQL (use custom SPs etc) used by the system.

Aggregation - No ability to aggregate, group information from multiple lists close to the source of data (CAML based aggregation across lists takes place at the application tier which is much more expensive)

So if your application truly requires the above traits, the best option is to store data outside the content DB and use one of the several schemes (webparts, _layout, ASPX based content pages to surface your application within SharePoint.

On the other hand, if you are building an application with modest sizing requirements you can take advantage of the abstraction provided by SharePoint and thus avoid the need for any custom data access code. As an example, we built an invoicing system entirely based on the content DB. Invoices were “split” across site collections to allow the solution to scale-out (as opposed to a scale-up approach)

Generating Reports based on SharePoint Data

Whether you store application data inside the content db or not, there is almost always a requirement to generate reports based on the metadata associated with the document stored in SharePoint. Before we look into various reporting options, let us briefly look at some of the key characteristics of a reporting technology such as SQL Server Reporting Services (SSRS) SSRS can be broken up into three parts (as shown in the figure below)

1) Designer -used to create a report layout

2) Processor – binds report data to the layout

3) Connectivity – to a data source

It is easy to see that the ability to scale reports reporting generation is dependent on how fast the report data source can return the requested data. Relational databases (RDBs) with their innate ability to process set-based operations are obviously very good at this. RDBs are also very good at grouping, sorting and filtering; other common reporting requirements.

SQL Server Reporting Services (SSRS), like many other reporting tools, supports connectivity to various data sources out of the box, including SQL Server, DB2, and Oracle. Again, similar to other reporting tools, SSRS allows custom data sources e.g. Active Directory, Disconnected DataSets etc., to act as the source for report data. The only requirement is that the custom data source support a forward-only stream based access to data.

The other key aspect of reporting is the ease of use in creating a report layout. Designers such as the one included with SSRS make it easy to support report layouts such as cross-tab reports, nested reports etc.

Finally, reporting tools exhibit many advanced capabilities such a caching, slicing (ability to support multiple reports via a single “super” dataset), scheduled report generation, delivery (PDF, Excel, Word etc.) and dynamic reports.

Reporting Options

Let us now look at some different options for generating reports on the SharePoint content metadata

Built-in and custom WebParts

If you are trying to generate a simple report (aggregated task list) based on a single list , it might be sufficient to use one of built-in webparts (for example Content Query Web part – see Reference [2] for detailed instructions).

The key advantage of this approach is that security trimming takes place automatically. Another advantage is that the report is always generated on data that is current. The downside is that manual work is required to create the layout (no report designer type tool is available). Any advanced functionality such as master-detail or nesting requires additional custom coding. Finally, unless a caching scheme is used, each view of the report causes the query to be computed again imposing additional load on the system.

Entity Framework

The ADO.NET Entity Framework is part of VS .NET 2008 SP1. The ADO.NET Entity Framework provides a level of abstraction over a data source. The ADO.NET Entity Framework is based on the Entity Data Model (EDM). Basically the EDM allows users to model the database like the way they designed the database in the first place, probably with an Entity-Relational (ER) diagram. This level of abstraction is important because as more and more application information is being maintained in the database, the relational schema more frequently gets modified for performance and normalization. Those changes can affect the data access code in the application. For example, a product table may become a join across a few tables, but as a programmer I only really only care about what the application defines as a Product entity.

So why is EDM important? It is expected that future version of SSRS works natively with the EDM. Using the EDM we could model a SharePoint list and then use SSRS to generate reports.

The key advantage is that this scheme is that it allows us native, yet safe, access to the SharePoint tables. This is because EDM acts as a firewall between the underlying SharePoint content DB structure and Entity-SQL based queries. If the structure of content DB changes at some point, all we need to do is change the EDM. The Entity-SQL queries are not impacted.

Native access to data stored in content DB means zero latency as well as full fidelity access to item level security.

The downside of this approach is that EDM model needs to be manually defined. The other down side is that tuning options are still not available.

Data Extraction Service and Operational Data Store

Another reporting option is to develop a Data Extraction Service (DES) shared service similar to BDC, except this DES would export the information stored inside SharePoint out to an operation data store (ODS). DES is a generic scheme that works across various list types without the need to develop list specific custom code. It is envisioned that users will upload a BDC style XML configuration. DES will then use the configuration file to export data to an external data source on a scheduled basis. To make the transfer as efficient as possible, only differential updates will be carried over.

Security (like the BDC) will be applied at the entity level and not at the SharePoint item.

This scheme offers the most scalability and flexibility. Once the data is available inside ODS, any SSRS technique can be applied. All the tuning and optimization options provided by SQL Server are available as well. Further, the SharePoint performance is not impacted by reporting (although there will be cost associated with running the DES service for extracting the data from SharePoint into the ODS)

The downside of this approach is the latency of data and effort required to generate the DES metadata.

If this scheme seems useful, we could invest in a tool that generates the DES metadata (similar to the BDC metadata generation tools available in the market today)

Reporting Services Data Extension

SSRS supports a modular architecture designed for extensibility. One such extensibility option is the data processing extension which allows SSRS to connect to data sources and retrieve data.

We have two options here:

1) Develop a custom data extension ourselves, by wrapping the SharePoint Web Service based API. Alternatively, we could have a custom web service return a dataset. SSRS 2005 has the ability to consume a dataset directly. The latter is a bit easier because we don’t have to implement all the data extension ourselves, although the report design experience is not rich as provided by data processing extension.

2) Use a commercial data extension such as the one provided by Enesys. Enesys data extension makes it possible to use SharePoint (MOSS 2007 & WSS V3) lists data for building reports with Microsoft SQL Server 2005 Reporting Services – a custom XML query (with portions of CAML) can be directly embedded into the RDL.

The biggest upside of this approach is that it allows the richness of the SSRS designer to be leveraged.

On the downside, Enesys RS uses SharePoint Web Service OM to extract information from the content DB and is thus subjected to the inherent performance limitations of CAML.

References

[1] White Paper: Working with large lists in Office SharePoint® Server 2007, Steve Peschka, Microsoft: http://technet.microsoft.com/en-us/library/cc262813.aspx

[2] Data Processing Extensions Overview, SQL Server 2008 Books Online, Microsoft,
http://msdn2.microsoft.com/en-us/library/ms152816.aspx?s=11

[3] How to: Customize the Content Query Web Part by using Custom Properties, Microsoft,
http://msdn2.microsoft.com/en-us/library/aa981241.aspx

[4] Enesys RS Data Extension
http://www.enesyssoftware.com/Products/EnesysRSDataExtension/Features/tabid/73/language/en-US/Default.aspx

I recently recorded a DNR show on MOSS. During the session, a question about the benefit of using MOSS platform elements such as "List" came up. How can developers leverage these to build applications more productively. Let me try to answer this question with an example. Imagine that we are required to build a web page that displays a list of webcast recordings. Users with administration permissions are allowed to upload new webcast recordings, and update (and delete) existing items in the list. Other users can only view existing items in the list. Further, we also have a UI requirement to customize the list rendering such that in addition to the name of the webcast and an icon (that allows users to initiate streaming), the list also allows users to download the associated presentation slide deck as well as view the description of the webcast.

One approach for implementing the above requirement would be to define a webcast content type that has fields that correspond to the columns described above. We can then associate the webcast content type with a SharePoint list. Using CAML (Collaborative Application Markup Language) we can customize the UI of the list to meet our requirements.

Note that we could have just created a list using site columns instead of defining a content type. The benefit of using a content type is that it is a reusable type that can be associated to other list instances. We can also create new content types that derive from webcast content type. SharePoint uses ASP.NET forms to allow users to insert and update list items. All data associated with the list is automatically stored in the content database. The Figure below depicts a custom webcast list.

While it may not seem difficult to add a list control on an ASP.NET page and hook-up some ADO.NET code to persist the data in the database, it should be noted that we have not written a single line of code thus far. We have relied on SharePoint list handling and content database to implement the list.

But imagine if we are required to extend the above functionality. For example, it is required that each item of the list be secured individually. Users also want to subscribe to any changes to the list (new recordings) made to the list either via RSS or email. From a QA standpoint, a content management process needs to be enforced when a new webcast recording is uploaded requiring versioning, check-in/check-out, and approval workflow. Content management requirement invariably necessitates the ability to maintain an audit trail of changes as well as the ability to undelete an item that was inadvertently deleted. Last but not the least, a search function on the site should include the information about the webcasts.

Now with the need for additional features, the custom ASP.NET solution is not easy. Fortunately, all of the above functions are provided by SharePoint List by default. We can even extend the SharePoint List behavior using event handlers.

All of the information stored inside a list is accessible, not only via the SharePoint UI, also via the Object Model (Class Library as well as Web Service based OM). This means that processes outside the host process can access the list information – a key to building transparent applications that are reusable.

As is well known by now, WSS 3.0 (and consequently MOSS) supports the ability to plug-in a custom role provider. So instead of being limited to the default AD based role provider, a custom store such as SQL database, can be used as the authorization store. As you can imagine, this gives you a lot flexibility in terms of storing and managing users.

Let us examine custom role provider in detail. But before we proceed, let us get a few core definitions out of the way. WSS defines the following core set of security related objects:

o SPUser and SPGroup – SPUser represents a user, while SPGroup as the name suggests, represents a collection of users. WSS allows you to create custom groups.

o SPBasePermission, SPRoleDefinition and SPRoleAssignment- SPBasePermission is right to perform a certain action within WSS. For example the right to insert list items. SPRoleDefinition is a collection of rights. For example, Contributor, Designer are built-in roles. Finally, SPRoleAssignment defines the assignment of roles for SPUser or SPGroup

Irrespective of whether you use a custom authorization provider or not, WSS creates instances of above mentioned set of objects to manage security within WSS. For example, SPRoleDefinition object that we discussed above, relies on SPUser and SPGroup for role-to-permission mapping.

So how does custom provider help? Using a custom provider you can create roles and include them as part of SPGroup (much like you would include instances of SPUser). This is an important benefit as you grant access based on custom authorization rules, stored outside of WSS.

There is another important benefit enabled by custom role providers – Ability to manage authorization rules across site collections (SPSiteCollection). A SPSiteCollection is a security boundary within WSS. This means that SPUsers and SPGroups defined within one SPSiteCollection are not available to other SPSiteCollection instances.

A custom role, however, can span multiples site collections since the authorization provider is defined at the Web Application level (SPWebApplication). This way you can manage authorization across sites from one central administrative interface. On the contrary, if you only had to rely on SPGroups for authorization, you would have to create a separate SPGroup for each SPSiteCollection and develop a scheme to keep them synchronized.

I should add that AD groups can be used to achieve the same behavior as described above (I.e. ability to span SPSiteCollections). However, the way AD is administered in most organizations, WSS Site owners do not have the permissions to create and modify AD groups.