Appendix: How Do We Upgrade?

The biggest difference between a SharePoint deployment and a standard web or database one is the fact that a SharePoint site will evolve as the business it is modeling changes and, hopefully, grows. Everyone loves to use the word "organic" here, but it's a great biological metaphor: SharePoint is designed to have its structure be flexible and malleable.

And not just in the tried and true CMS-y ways. SharePoint empowers its business users to tweak the environment without the help of IT. Fields become required. Content types have babies, taxonomies become folksonomies, and search, like a piano, is kept well-tuned. Views are created, web parts are reconfigured, and new master pages are applied.

Compare this to a data-driven web application backed by SQL. You deploy the site, people use it, and over time a change log of enhancements (and of course bugs) is maintained. Next you get back on the horse and make the updates to the tables and stored procedure and foreign keys. When they're done and tested, a maintenance window is scheduled, a SQL Schema Comparison in Visual Studio is performed, and database changes are published. Suddenly, we're in version next.

We don't have to deal with the inevitability of people not on the development team being involved in these updates. Could you imagine someone popping into the database and changing a date column to an nvarchar because a user wanted to specify AM and PM differently that the planned convention? That kind of thing happens in SharePoint all the time! It's greatest strengths as a CMS cause the biggest challenge for architects working on the next release.

This appendix is about how to allow these strengths to be capitalized on by the business while still maintaining order on the development side of the house. There are a couple different ways to do this. Although the technology makes this feasible, we'll see that it's planning and communication that could make it actually kinda easy.

The Handoff

Monday morning the site goes live. Tuesday morning we look at the analytics. Wednesday morning we monitor the servers. Thursday morning we review the feedback Emails. Friday morning we send the interns out to buy champagne. That next Monday, however, we're back at it; there are nice-to-haves that didn't make into version one, and have-to-haves that aren't working quite right.

I'm always fascinated by the first few weeks of the second phase of a project. Production, which was just another server farm a few days ago, is now sacred ground. It makes me anxious: we suddenly have a real threat of losing data. We can't be wild guns IISRESET-ing every weird little one-off SharePoint burp. And, the most stressful reality of them all: the next production deployment has to be perfect on the first try.

What makes this so hard is the fact that we no longer control the structure of the site. Permissions and SharePoint group membership can get out of hand real quick without a governance plan that has teeth. Team site hierarchies can spread throughout a content database like a virus. Pages can become wastelands of content editor web parts.

I'm being a bit dramatic here, but these are situations I've seen after what I call "The Handoff." The Handoff can be a week-long training session, an hour-long knowledge transfer meeting, or a sentence-long Email stating that no further code-based structure updates will be deployed; the site owners now control things directly via the settings screens.

Whatever it is, it is a formal declaration of content ownership independence. It's interesting to watch the progression of events that lead up to this: simple little bugs start to trickle in, asking for columns to be hidden or made visible, required or optional, or removed or added to a content type. Or names of pages in the data creator will change. And there's always time for one more CSS issue that only shows up in Firefox on Macs.

These always coalesce with the end of a project, and it's easy for an attitude of "Oh, just do it yourself through the UI!" to begin to pervade. But after code complete, UAT, and performance testing are complete, (assuming we've been deploying to production all the while) why pay consultants for a few hours of development, deployment, and regression testing when someone on the business side can knock it out in thirty seconds? Well, I feel like I've laid out many reasons why, but these points lose their persuasive power when the budget starts to run dry or during the last few days of the project.

So the first step in planning an upgrade is doing The Handoff. Even if we're still fixing bugs elsewhere in the project, the last few deployments, if possible, will be executed via new PowerShell wrapper scripts that don't activate Structure. All the beautiful All Code extension methods are quickly ushered into retirement. As I said above, the site owners are now responsible for owning their site.

After The Handoff, we need to plan how to build out the next version of the site's information architecture while keeping in mind that we can't be guaranteed what production will look like at any given time. Short of putting a lock on the site collection or commenting the ribbon out of the master page to handcuff the content managers, we need to work with them, and IT, to ensure that our local, development, and test environments remain a valid representation of production.

The best way to implement this second step of the plan is to set up a schedule of SharePoint backup / restores from production down to staging/test/whatever, then down to development servers, and finally down to local developer environments. The biggest challenge with this is the size of the database, as larger site collections, (roughly greater than ten gigs) in my experience, tend to resist living in a new home. Also, if you're running SQL Express locally, you're out of luck when the content grows beyond roughly three-and-a-half gigs.

To combat this, you can supplement your Deployment Driven Design with a little Backup / Restore Driven Design (although that's not nearly as catchy of a title). This just means planning your information architecture to make backups not a nightmare. For example, images can be off-loaded from your site, (specially, stored in separate content databases serving different web applications) and different site collections can be separated out.

Once this schedule takes hold, we can get a fairly reliable environment to code against. Ideally, most small-scale changes (modifications of views, requiredness of columns, etc.) won't break our logic. But even if you're getting weekly refreshes of production data, there's still a chance a content manger can make a breaking change.

So when I'm getting close to a release date, the third part of the plan is to politely ask the business to put a freeze on content changes a week or two before we schedule the deployment with IT. If that's not feasible, we should be able to at least ask them for a change log so that we can manually (ugh) keep our development environment in sync. But as long as nothing breaks SPMetal, we don't need to burn too many calories on this.

To summarize, here's how I go about scheduling The Handoff between version one and version two of my SharePoint site:

1. Agree upon a moment toward the end of the version one project where no more structural changes will be made to the code base. From this point on, content managers are doing these updates directly in production via the UI (or SharePoint Designer).
2. Modify/retire the scripts that activate the Structure feature, and deliver the final code base.
3. The final production deployment is completed and the site goes live.
4. Establish a scheduled recurring restore of production data into development environments.
5. Develop version two, deploying each build up the chain, but not into production.
6. Schedule a content freeze and deployment date.
7. Deploy version two to production.

This procedure allows us to keep a fairly sane development experience while allowing content managers to enjoy all the power of SharePoint as a CMS. Just keep up the chatter as things change on the site. The Handoff is only the first half of the upgrade problem; after scheduling everything, we have to start writing some code!

Upgrading Solutions

The code we'll be looking at to do our upgrades is a departure from the "blow away the site collection and recreate it from scratch" paradigm we saw in All Code. Simply put: since we have a live production site, we can't blow it away. Therefore, we'll be revisiting the "create if not already there" style of development I mentioned previously.

The implementation will be using the SharePoint feature / solution upgrade procedure, which is more or less out of the box. Despite the fact that it will require some XML to sling, it's a really clean way to literally version our Structure code. Some dark paths I've followed before embracing this one is creating new features called "Structure v2," "Structure v3," and so on, using reflection to detect the current assembly version of DDD.Web.dll, and even building completely new WSPs.

But that was simply too much custom effort to avoid a dozen or so lines of markup, and too great a departure from the clean, supported method of feature and solution upgrades. Following is a quick inventory of the additions we'll be making to DDD.Structure (this will be the same for any SharePoint project we have; Structure updates are the most tedious and version-able, we we'll only be considering those).

• Increasing the version number of the feature
• Defining the upgrade in the feature's Structure.Template.xml file
• Handling the FeatureUpgrading event in Structure.EventReceiver.cs
• Creating some new deployment PowerShell scripts to perform the upgrade

The XML

The template markup file for a feature is actually a really good place for the declarative logic needed to upgrade a SharePoint feature. This XML lets you establish a sort of "name" for a block of upgrade logic to execute in the feature receiver, pass in a dictionary of string parameters, and determine for which versions of the feature the event should fire.

These version numbers follow the same assembly versioning convention .NET does: [major].[minor].[build].[revision]. Generally, each production deployment increments the major version number; we will be upgrading DDD.Structure to version 2.0.0.0. The rest is up to you; I use the minor version number for "hot fixes" – critical bugs that must skip the standard development cycle and get into production immediately.

Build and revision versions are "internal" and for developers only during a dev cycle. They map to CI builds and minor bug fixes. So if we have version 3.0.0.0 live and I get assigned a critical bug, I'll fix it and deploy 3.0.0.1 to my local environment, check it in, and have my build server deploy 3.0.1.0 to the development environment. Then, after being tested, 3.1.0.0 will be deployed to the client.

Let's take a look at the updates we'll be making to our feature's XML (new lines are bolded):

First, on Line #2, we specify the incremented version number. Next, we hook up our feature receiver on Line #3 (you can steal the attribute values from the feature designer). Things then get interesting on Line's #4-8. Reading the XML literally, we're saying that for versions of this feature that are between 2.0.0.0 (inclusive) and 3.0.0.0 (exclusive), fire the FeatureUpgrading event for the receiver, pass the string "VersionTwo" for the "upgradeActionName" parameter, and add a key of "Enabled" to the "parameters" dictionary with a value of "true." Note that features without the "Version" attribute specified are stamped with "0.0.0.0."

The Code

Now let's take a look at the FeatureUpgrading event of our feature receiver. The above XML acts as a config file, letting us know what to expect the values of the method's arguments to be. The thing I like the most about this markup is the ability to infer the "history" of your Structure code. As your site grows, you can keep adding VersionRange elements each having multiple CustomUpgradeActions child nodes to tightly model our deployment code.

In Line #5, I grab the current version (what we've incremented it to in the XML template) in case I need it for some advanced functionality. For example, if Version.Revision is greater than zero, we can turn on some debugging logic. Next, Line #'s 6-13 handle my enabling logic. Basically, I use the parameters for each upgrade action to give me a quick Boolean value I can use to turn on or off the code. This can help you skip the Structure update logic in case you want to more quickly test other modules in the upgrade sequence.

Next, if we're indeed enabled (Line #15) then do a switch on the upgradeActionName. DDD doesn't like switching on non-constant strings, but since we explicitly control these values in the XML file, we can copy-and-paste them into the case statements relatively safely. Like I said at the beginning of this section, it's a small price to pay for such a clean upgrade procedure.

Finally, Line #22 (if we're under the "VersionTwo" code path) calls the method that executes the actual upgrade logic. I tend to push the implementations of each version into separate functions for readability of the code and navigability of the file; the receiver itself is just a broker to organize the different manners in which we determine which code to run.

UpgradeToVersionTwo, for the purposes of this book, is going to add the content type field to the default view of the Rollup Article page library. Yawn, I know, but like I said these are the sorts of requests we can expect to see after people have been testing an almost-ready-to-go-live site. All we need to pass it is our site collection; it handles the rest.

The code, like we discussed, follows the "create if not already there" paradigm because upgrade logic might run multiple times within a version range. Since we want to operate on our article sub site, we have to build a relative URL to it via the GetUrlFromWebName extension method, which simply removes spaces, encodes, and trims leading slashes. Like lists, we can't get a design-time guid to our sub sites, so this method makes their name (stored in Constants) a strong enough unique identifier.

We next use another All Code extension method, GetPagesList, to get the pages library for the publishing sub site (Line #10). Finally, we get the default view on Line #12, and then Line #13 calls the AddField extension to SPView, which only adds the field reference if it's not already present. Normally, I'd have more diligence around making sure the field exists, but since you can't delete the content type column, (which is certainly a good thing, and a fortunate selection for this example) we're safe.

The PowerShell

Now that we've implemented our event upgrade logic and wired it to our feature receiver XML, the last step is to script the deployment. The first thing to understand is that we can't upgrade a feature without upgrading its owning WSP. Technically, a solution upgrade does a retract and re-add, but that won't fire the feature upgrade, since the feature is itself uninstalled and reinstalled.

So that's why Update-SPSolution exists in Microsoft.SharePoint.PowerShell: it indeed cycles the WSP, but maintains guids so that its features know to be ungraded, instead of reconstituted. So we basically publish DDD.Web to DDD.Common\Deployment, and use Update-SPSolution to replace the currently-installed WSP.

Make sure you don't accidentally do a right-click-deploy when you mean to right-click-publish in Visual Studio. This will do an explicit solution deployment and invalidate our upgrade scenario. Why? Because the Structure feature, being site collection-scoped, will not be activated after a re-deployment, and deactivated features can't be upgraded.

Manually activating will then bomb out, because the site collection and its infrastructure already exist. If you make this mistake, you'll have to fire off a DoEverythinger.ps1 to reset things, increment the version number of the feature, and republish again. But once you've got your new WSP out there, let's look at the first (of three) PowerShell scripts we'll be using to give it its facelift: SolutionUpgrader.

This is a slight derivation of SolutionDeployer.ps1, and simply wraps the underlying Update-SPSolution commandlet with some parameterized love; there's really nothing new here. However, Line #'s 19-20 call our old friend Execadmsvcjobs.ps1. Since there is technically a redeployment of the WSP, it will find and patiently wait for the "solution-deployment-ddd.web.wsp-0" job to finish.

This is crucial, as it will allow our next script, FeatureUpgrader.ps1, to operate against the newly-deployed bits (including any packaged DLLs (like DDD.Common) instead of the previously-installed version). Now that our solution has been upgraded, we can check out what it takes to push our feature updates through.

As SolutionUpgrader followed SolutionDeployer, so does FeatureUpgrader follow FeatureEnsureer. All but the last ten Lines are essentially equivalent, except that Line #'s 18, 29, and 35 have to do a bit more work to get the SPFeature object, rather than the SPFeatureDefintiion that Get-SPFeature actually returns.

Line #23 is necessary since the SPFarm class doesn't have a Features collection that we can query. This is because all features scoped to the farm level are auto-activated by default; we don't need a collection to add SPFeatures to (as this is how they are programmatically activated). Next, after Line #39 ensures us that the feature in question has been found and is activated, we can proceed with the upgrade.

This happens on Line #43. The Boolean I pass to this method specifies weather or not to "force" the upgrade. This option represents the fact that by default, features won't upgrade if they don't have to. (For example, if the currently-deployed version number is higher than the about-to-be-deployed one.) Since our code is tightly-defended against the structure changes it wants to make having already been accomplished, there's no reason to have to redo a deployment because you botched the feature's version number. But of course, try not to do that.

Finally, we have one more script that's a wrapper around these two. This guy is called VersionTwoUpgrader.ps1. I don't mind getting a little liberal with my scripts here, essentially creating a new one for each major or minor version of the feature we're upgrading. Since we're used to production deployments comprising of only a single PowerShell command with a single URL parameter, I'd rather not rock that particular boat.

I also feel that many specialized wrapper scripts for specific deployment scenarios is much cleaner than trying to conceive of a super-master-wrapper FrankenPowerStienShell that's doing way too much work. Do you want your production script to have half a dozen parameters? Now that it's way too scary of a monster! Let's take a look at our upgrade wrapper:

The theme here is clearly that upgrade scripts closely mirror their big brother deployment scripts. All this guy does is distill the parameters to pass to its two constituent ps1's: SolutionUpgrader and FeatureUpgrader. If this or any other upgrade scenario has multiple solutions and features to deal with, just list them all out like we do in SolutionDeployer.ps1.

And there it is! After upgrading our Structure to 2.0.0.0, we can see our default view now shows us the content type for each page. Lovely. We needed some XML to get there, but it was a small price to pay. The only thing I want to leave you with here is an urgency not to take shortcuts when it comes to upgrading features that leave their fingerprints all over your site. Manually tweaking your Structure code after the first production environment can shove you off a cliff so high that not even the parachute provided by TFS' "View History" functionality will slow your plummet.