Importing SharePoint Designer Workflows Into Visual Studio (Part 1)

SharePoint is a universe, and like our own, new distant galaxies are discovered and elucidated. Similar to an astronomer, as a SharePoint architect, I occasionally find myself making assumptions. It's sort of like dark matter: we only know it's there because it has to be there; it makes all the math work out. Coming from a development background, I know an awful lot about web parts, feature receivers, client objet models, timer jobs, and so on: all of the code-based galaxies of SharePoint.

However, there are distant "SPStars" out there that I've only gazed at through a telescope; they are too far away from my .NET home planet to see clearly in the least, let along directly observe in the most. These are things like taxonomy, FAST search, BCS, workflow, and so on: galaxies of SharePoint about which I only know what I've had to know to successfully implement them on projects. This varies from code, which I know pervasively and can speak to, estimate against, and work with assumptionlessly.

But for these more ancillary technologies, I can only advise from first hand (and occasionally leverage trusted second hand) experience. The rest is assumptive. BCS is great example. I know that it can be used to index external data for SharePoint search integration. But I've only done it once. If I had to estimate how long it would take to accomplish this a second time, I would have to assume that it would be no longer than the first go-around. But until I've worked with BCS for ten years like I have with .NET, and it is second nature (or even first nature, as it is with me and code) there would be a lot of assumptions.

I want to close out my series about SharePoint Designer / Developer hybrid team development with a piece about assumptions. And SharePoint. And workflow. And how the former crushed the connection between the two latter.

My role on this project as architect was basically to make sure that all requirements deemed "code-ish" were implemented by my developers. In addition, I had to facilitate all things "SharePoint Designer-ish" being integrated into our Visual Studio-based WSP deployment. This was a balls-to-wall deployment, with code-provisioned site structure, taxonomy-guid-matching-across-environment features, and more PowerShell than I ever thought I'd know.

It was also technically challenging, making sure that page layouts, content types, site columns, and other artifacts that could have been provisioned manually in the environment were not blown away by our WSPs. It also required a lot of syncing of HTML, CSS, images, etc. across Email, SharePoint Designer, and TFS. It was a large team with a large range of SharePoint knowledge and toolsets, and it was my job to keep them all chugging along together.

So when the workflow phase of the project plan commenced, I thought that this would be the most elegant example of SharePoint Designer / Developer coordination. I thought that this would demonstrate the true value of my automated deployments.

I thought that it would work.

But my assumptions were horribly wrong. And time consuming. And expensive. And frankly, quite depressing at times. So following is a multi-parter about the problem in the first place, the solution in the second place, and the conclusion in the third place.

<Spoiler Alert>

We did basically get it to work, technically, but the one feature we needed and couldn't get working was the show-stopper (at the time of this writing, our workflow designer is connected to one of the client's production SharePoint web front end servers, manually configuring a workflow in SharePoint Designer) that made us, after over two months of technical, political, business, and ideological discussions and debates, pull the plug from importing SharePoint Designer 2010 (SPD heretofore) workflows (WF) into Visual Studio 2010 (of course VS).

</Spoiler Alert>

The Assumption

Being in VS all day, I always feel a little miffed and unadventurous scrolling through all the project and item templates I've never touched. Like despite how much WCF I've done over the years, I've never created a "WCF Workflow Service Application" project or an "AJAX-Enabled WCF Service" item. And under the "Data" section there's a "Service-based Database." That sounds cool, but I don't even know what it is. Anyway, spending a lot of time in the SharePoint -> 2010 templates, I've always seen the two "import" project templates, one for WF and one for general WSPs.

On the day this whole mess started, over three months ago now, I had just completed my final deployment proof of concept, (POC) and was feeling great about the things I could accomplish and the time I could save with my toolset. Sitting next to my WF designer, I realized that amidst all this SPD integration, WF was the only thing that we planned to deploy separately from the rest of the application. One-offs never sit well with me, so remembering seeing those VS templates and feeling super confident about my deployment machine, two things happened simultaneously:

• I assumed that the "Import Reusable Workflow" VS project template would create a VS project capable of importing a reusable workflow.
• I promised my team and my client that I would be able to use this tool to assimilate WF into VS and our deployment proper.

It was a fine day and I felt really good about my plan. This was, unfortunately, the best I'd ever feel about it. As I set about proving this out, my WF designer referred me to a few articles she'd found (one even written by someone she knew) about SPD VS WF integration. She was as excited as I was about this deployment: it meant she would only have to do the work once (verses having to reproduce her WFs manually in each target environment) and that she could learn a thing or two about how her WFs fit into the larger application by way of deployment.

The Problem

Full of excitement for a one hundred percent automated SharePoint deployment mechanism, I had her package up her WF as a WSP via SPD, and shoot me the file. I created a new "Import Reusable Workflow" project, went through the wizard, sucked in the WSP, and watched as VS created a SharePoint 2010 project and provisioned and configured all the necessary structure to represent an SPD WF. "This is awesome," I remember thinking. Right-click on the project, select "Package," and update my PowerShell script to include this with the rest of the solutions. "This will take like ten minutes!"

So I right clicked, packaged, but no packaging occurred; it didn't even compile. Ten minutes become an hour, most of which I spent digging around the GAC and ISAPI for DLLs. Yet I remained undeterred and finally got it to compile and package. Moment of truth: I turned the crank of my deployment, and the WSP was successfully added and deployed. Things were looking really good.

Then on the site, I was able to activate the web-scoped feature that was provisioned, and the WF was visible on lists. But that's as far as I got when the troubles began. I couldn't associate or instantiate the WF. (Read about these here and here.) Then I learned that I was too quick to glaze over the fact that it was a web-scoped feature; the WFs were not available on any sub sites. Then there's the fact that updates didn't really work as desired: upon the second deployment, it was reported that one environment, out of the box site columns disappeared, and in another a content type refused to redeploy.

I think these assets didn't uninstall during a retraction because they were in use, and the VS WSP deployment failed upon redeployment because an object with the same Id already existed. This is something I call "deployment limbo" when uninstallation fails, causing reinstallation to fail, leaving you with a broken component that you can't remove or update. The fix is to blow away the site collection, turn off VS feature auto activation, fix your logic, and redeploy manually via PowerShell.

So the ten minutes that became one hour quickly bloated into two days, and I still couldn't get InfoPath to deploy (read the above links for the back story on that issue). I consulted the aforementioned articles m WF designer gave me, but they all had the same story: import, click next a few times, and you've got it. I'm not sure what's special about our WFs in general or the SPD export process and the VS import process specifically, but somewhere along the way, I ran into many difficulties that apparently no one's seen before.

So why not just bail at this point? Because after getting my client excited about having the WF logic in VS, they quickly stipulated that they didn't want to support SPD at all, since they didn't have the in-house expertise for that tool. So our approach was to have them built in SPD, exported to VS, and then deployment and updated from there. Being in TFS, all code was in one "collaborative" place, and all deployments were automated and repeatable. If the WFs were outside of TFS, then they would have had to maintain the website using two different tools, one of which (SPD) required the manual touching of production.

Another thing to note is that even if everything worked via VS, I don't think that, in hind sight, it would have been a feasible tool to update the WFs. The XOML file designer, once imported into VS, is MONSTEROUS. A simple SPD WF generates many hundreds of activities, including several depths of nested container elements. If I didn't have 8GB RAM on my machine, I'm not sure if it would have been able to handle VS' WF designer while running SharePoint, SQL, and everything else you'd find on a development machine. Even if SharePoint WF is already in your wheel house, it's still probably easier to tweak these things in SPD manually than shuffle through this monstrosity of a designer.

Plus, of course, I hate reneging on above-and-beyond, gold-plated features. If I mention it, I'll build it. So I decided that since I was already two days overinvested in this approach, I might as well exhaust my ideas before bailing. So I went through the import process again, redeployed, and strangely enough, saw things working! What happened? Well, I recalled that the wizard seamed a bit different, and the project structure seemed a bit different too. Turns out, I used the "Import SharePoint Solution Package" by accident!

Although this approach was still flawed (provisioning web-scoped features, occasionally running into an error that the file paths were too long, updates not taking, and a few other small things) it provided the repeatable TFS development experience that our client was assured. However, what brought this approach to its knees wasn't the deployment itself, but rather the apparent bugginess of the WFs.

When created manually in the WF designer's SharePoint environment, (which was a VM) everything worked. However, when it was put into VS and deployed to our development environment, we'd experience sporadic issues. And what's super weird is the WFs would fail with different errors on different servers. These include Emails not being sent, tasks not being created, InfoPath forms not being deployed, WFs themselves not being visible on the association page for a list, and others.

Whereas any developer who uttered the dreaded phrase "but it worked on my machine" would be chastised for not writing robust enough code, the same symptom here could be caused by the deployment, by SharePoint, by Windows Workflow Foundation, or any of the other technologies playing a role here. So my claims of "bug in workflow" were countered with "bug in deployment."

But instead of allowing the team mojo to be muddled with politics and finger pointing, we decided on another approach: call Microsoft. We were looking for three things from them: first, that the process of importing SPD WFs into VS was the recommend approach; second, that creating WFs in SPD, exporting to WSPs, and deploying those was a supported approach; and third how to fix the integration issue we were having.

To make a long story anticlimactic, we did get the validation we were looking for with regard to the first two issues. I was told that SPD was a home run for "simple" WFs, although "simple" is indeed a subjective adjective. Is a WF that performs twenty simple tasks itself therefore simple? In an case, when things get complicated, it's time to turn to VS. And not just for the added goodies of WSP deployments and TFS integration; it's simple a more robust tool. Even though SPD 2010 is way more stable than 2007, (in my experience) I still find myself having to close out and reopen my site at least twice an hour. The main reason those "import" VS project templates even exist is to promote VS as the tool for complex WFs, even if they were created in SPD.

So thanks Microsoft, at least our heads (and certainly our hearts) were in the right place. But now what about the third question: getting this thing working? After several more phone calls and Emails about the proper process for this integration, and spending a lot of time testing, tweaking, and expanding upon it with my own improvements, I'm able to present Part 2 of this post: the solution. Like I said: it's not perfect. And like I haven't said: it's approximately 80 steps long. But regardless, in Part 2, I'll discuss in more detail the technical issues we encountered, and the step-by-step that almost addresses them.

The Conclusion

We don't need to review the procedure to discuss the outcome. Basically, my attitude is that if it's not one hundred percent correct, then it's wrong. Even if you're left with a happy client, it's still not right. Unfortunately, I have to voice a vote of no confidence in SPD WFs. If you need workflow, either build it in VS from scratch, buy it from Nintex or K2, or design it as a timer job, feature receiver, or something else. The VS import tools simply can't handle complex WFs, regardless of what "complex" means.

So if you're building "simple" WFs, then perhaps VS isn't even part of your project, or your world; in any case, considering all of the best practices it requires you to ignore, building your simple WFs in SPD manually and rolling updates on each environment separately might very well be the deployment mechanism that makes the most sense. And as far as assumptions go, you have to trust your tools in the first place, and learn from the experiences in which they fail you in the second place.