The SUESS Series

Introduction

Stage 1 - Upload

Stage 2 - Encode

Stage 3 - SmoothStreaming

Our in-depth media adventure begins, well, at the beginning of the SUESS "lifecycle" with Uploading. This stage contains two sub-components: the Uploader Silverlight control, and the WCF service on the Web Server that facilitates the file upload process. The only prerequisite required here is to have your data tier (Component 3) in place (which can simply be a folder on the Web Server).

Of course, there is nothing new about a file upload control; it's been in HTML longer than I have. Over the years, I've seen a lot of modifications to the familiar read only textbox and "Browse..." button to get the effects people are starting to expect in this whole Web 2.0 craze: progress bars, client-side file type filtering, large file support, etc.

To achieve these nice features, a lot of magic needs to happen client side. If you've read anything I've written before, you'll know that I don't consider AJAX to be magic; it's voodoo at best compared to the wizardry of Silverlight. The argument remains the Silverlight needs to be installed on the client's machine while AJAX comes down for free from the server along with the rest of the page. However, when it comes to programmatic benefits such as a first-class development experience (IntelliSense, compilation, UI designer, etc.), robustness of code, and extent of possibilities of what you can build, you cannot argue against Silverlight.

<RANT>

Now don't get me wrong: AJAX is a really cute technology that has a lot of niche uses. The difference, as far as I'm concerned, falls in the mindset of the developer. Take animating progress bars for example. In AJAX/HTML, you'd need to arrange a few nested divs, set all their widths and background colors, and then when the upload mechanism reports progress, update the width of the inner-most div. To me, that's not really a true progress bar; it's HTML kludginess.

Now, in Silverlight, you see, you literally just animate a ProgressBar.

THAT'S the point.

</RANT>

The example Uploader control I'll show you shortly lacks all the bells and whistles with which it could be adorned to truly make the life of a content manager much better. These features include drag and drop support, multi-file uploads, a slick UI, etc. However, the big ones, like progress bars and cancellation, will demonstrate how uploading large files (such as media) can be a pleasant experience for a user.

First of all, what does it look like? Well, not much, unfortunately. A pretty admin UI was not a requirement of the project through which I came up with SUESS, since this was all back end functionality. But here's what we have:

The title and description fields are watermarked textboxes with some gradient borders to implement the comps of the project. I really don't want to bore you with the details of the code that updates the visual state of the UI; it's hard to make hiding buttons and displaying error messages sexy. You'll be able to download SUESS in its entirety and comb through all the details yourself. Instead, let's take a quick trip through it, and then dive into the cool stuff.

The only input we actually need from the user is the file itself. Depending on your data tier, you could accept more metadata. (SUESS was actually born into SharePoint, so the sky was the limit for metadata.) But for now, we'll just have two optional textboxes for the name of the media and a description. If the name is blank, we'll use the file name. (As you'll see later, with the way Encoder outputs its SmoothStreaming formatted files, we don't have to worry about filename collisions; it's only a consideration during uploading.)

Next we have the "Encoding Quality" drop down. Once again, this is optional, and included only as an example to show another aspect of what you can do with SUESS in terms of explicitly controlling a myriad of Encoder options. (You'll see what I'm talking about in the Encoding post about SUESS.) These are hard-coded (barf, I know) values mimicking the Video Complexity (no documentation available) enumeration in the Encoder SDK. This is the best we can do to this extent, since we can't actually reference the DLL in our Silverlight project and reflect each enum member.

The purpose of including this in the example Uploader is to allow the user the ability to "throttle" the encoding process, which, as previously mentioned, could be very processor (and, of course, time) intensive. Unfortunately, I don't have a lot of metrics regarding just how much quality is sacrificed in terms of time taken to encode. However, I can say that media encoded with the highest quality settings indeed takes much longer than with the lowest.

Finally, we have the media itself. Silverlight gives us the OpenFileDialog control that is more or less identical to what you'd find in Windows Forms. The one immediate advantage it gives us over what is available in the HTML version is the ability to filter the file types (by extension) that the user is allowed to select in the dialog. This not only gives us a more elegant user experience (since we can "label" our filter with something like "Media Files,") but also makes the development much easier, since we don't have to go through the "hand slapping" input validation experience.

<RANT>

Back in the HTML/AJAX version of a file upload control, if you need to enforce a certain type (or types) of file(s), the best you can do is inspect the file the user selected, check its extension, and display an error message, forcing the user to have to go through the entire exercise again. Instead, with Silverlight, we can simply hide invalid file extensions in the folders they navigate within the dialog via the Filter property. This probably doesn't seem like a big deal, but I think it's huge: small UX improvements like these are what Web 2.0 is all about; with Silverlight, you are really using an application, not just a webpage.

</RANT>

Let's talk about a few of the more interesting code samples. The first one simply sets the OpenFileDialog to only allow the files that Encoder supports to be uploaded. It also has a hard coded value (which can easily be made to be configurable) to disallow files over 1GB (purely for sanity purposes).

Line #7 above sets the Filter property of the OpenFileDialog to the Encoder 3 supported file types. This is what ensures that Encoder will be able to handle whatever the user throws at it. Line #15 shows that "UI" code has been omitted. You'll see this in a lot the code samples throughout SUESS. Like I said, I don't want to bore you with the details of dealing with VisualStates in the application. Here are some more action shots:

Next let's look at the logic that implements the recursive "chunky" upload. If we send the entire file up to the server, we can't show progress bars, and really don't take advantage of client side functionality at all. Instead, the SUESS Uploader sends 1MB of the file up to the server at a time. After each call, the UI is updated with the current progress, and then the next "chunk" is uploaded.

Since all Silverlight service communication is done asynchronously, we need to daisy chain the "Completed" callback for each of these calls so that they are executed serially within it. Normally, if service calls can be done in parallel (such as downloading images or assembling the content of unselected tabs), you can kick off them all off at the same time; when they're done, they're done, and the corresponding part of the UI lights up.

Otherwise, if you need to call service A and then pass its result on to service B (or a subsequent call back to A, like the Uploader), then there will be some chaining going on. Here's a quick diagram that demonstrates these two paradigms.

In the top half, where we show "parallel" calls, all of the service references are explicit. We invoke a service, and when the associated completed event handler is fired, we do something back on the UI. What if we need to call the same service multiple (and in an unknowable amount of) times in a specific order? This is the case with the Uploader. Since we obviously can't know the size of the file, then we further don't know how many chunks we'll be dealing with. We need to make these calls serially.

The answer is recursion, as shown in the bottom half of the diagram. (Of course, this is only logical recursion, since the same physical method isn't actually calling itself.) I experimented with some crazy iterative algorithms, but they got real messy real quick, and were all ultimately besmirched by the asynchronousity of Silverlight. Instead, I created a method that uploads a chunk (array of bytes) of a file, and when it's done, increments a counter and progress bars. Finally, the same service is called again with the next chunk.

This algorithm, other than being sweet at uploading files, makes two additional features of the Uploader trivial: real time progress bars and upload cancellation. We'll discuss those next. First, however, let's look at this code. This piece is the StartUpload method that the above code block calls to kick off the recursion.

The Boolean passed as the second parameter to UploadFile in Line #7 simply tells the algorithm this is the first chunk. UploadFile actually doesn't care; it just passes this along to the service so that it knows whether to create a new file on the data tier or open an existing one. This can probably be inferred on the server rather than made explicit, but I didn't want to have to burn an extra trip to the disk for each chunk if I didn't have to.

Here's the algorithm:

The first thing to point out here is the utility method in Line #5. This method is a nice Silverlight helper that alleviates the need to have to deal with the "ServiceReferences.ClientConfig" files that store the Service URLs and other WCF settings. Using this method allows you to promote your control from development through production without having to worry about maintaining it.

Here are the two internal methods that build the WCF binding and get the URL of the service dynamically:

And the piece to get the service endpoint. (Note that since this example was lifted from a SharePoint implementation of SUESS, I had to be a bit creative with the URL.)

The rest of the main algorithm is actually pretty straight forward. The recursiveness happens in lines 39 through 42. We pause a bit in Line #39 so that Silverlight doesn't try to open the file from a new thread before the previous one properly closed it, increment the index of where we "are" in the file, and then recursively call the service.

How do we break out of the recursion? Three things can happen. First of all, if something goes terribly wrong on the server or some other exception is thrown, it will be caught on Line #20, handled, and then we'll hard return out of the method. (Again, since this isn't physical recursion, we don't have any stack trace "depth" to worry about.) Otherwise, we use the index. If it's greater than or equal to the length of the file, we know we've uploaded all the bytes: break out here, and start encoding.

The final way is through cancelation, which is one of the big features the Uploader. I know this isn't anything amazing, but it's another example of something that's pretty easy in Silverlight and probably pretty tough in HTML. As someone who's been in and around SharePoint for years, I've seen a lot of large file uploads quietly timeout after watching the page spin for ten minutes. That sort of behavior is not good enough; we need a big red self-destruct button to make sure we can cleanly stop an upload.

So as all this asynchronous uploading is happening on background threads, how do we cancel it from the click event of a cancel button on the UI thread? It turns out that it just works. Silverlight will automatically fire the completed event for a service call on the correct thread, alleviating the need for any dispatching. Since we don't have to worry about any cross threading, we can jump right in with the logic.

The Uploader cancel button click does two things. First, it simply sets the aforementioned index to -1, which basically throws a wrench in the recursive gears. Since all of our calls are on the same thread, we can check this index, see that we've been cancelled, and, well, stop making calls. This is all the housekeeping we need on the client. But what about the server?

The second cancellation task is to make one more call that tells the server that this upload has been cancelled so it can clean up the file. This is one of the reasons for the intermediate "upload" folder on the server: we never had to worry about IIS serving fragments of files. Other than cancellation, dropped connections will also leave broken files on the server. If the upload connections die, then we obviously won't be able to make another call to tell the server to clean up this file. Here is where the cleanup job finishes up for us.

This is a good transition to start looking at server code. We'll begin with the CancelFile method:

The interesting thing going on here is in Line #8 where I elevate to run as the SharePoint app pool account. This is important for two reasons (neither of which, of course, are what the RunWithElevatedPrivileges delegate is designed to do). First of all, we don't have to assign "Everyone" permissions on our folders. Second, it gets us around a potential IIS double hop issue, in case the GetTempUploadPath method (which is a wrapper around a config file call) returns a UNC path. (Kerberos is the right way to deal with IIS double hops, but that seems to be more configuration than most people - including me - are willing to deal with).

Next we have the method that accepts a chunk from the client and builds a file on the destination server:

There are a couple of things to note here. First of all, you'll notice that I use an Action delegate to pass the blocks of code that do the file writing (WriteChunk will be displayed shortly) to a method in Line #48 called ForceRetryFunction. This method takes in a Func (that is passed via an anonymous method in the invocation) which is basically processed in a while loop until it doesn't throw the type of exception (that is generically passed in as well). There are few other operations in the system that spawned SUESS, so I refactored it into ForceRetryMethod.

But WHY? Simply because certain operations need to be kicked in the ass to work. In this example, with I/O happening on several threads really fast, .NET can step on itself and open the file before it's properly closed (just like on the client). I'll put the code here for fun because it's a pretty cool algorithm, but the details are outside the scope of SUESS.

After all the trying, retrying, locking, and checking, the actual work that the server performs can be boiled down the simplest method in this section: WriteChunk. Here's the little guy:

Very straight forward, and unfortunately, a rather anticlimactic way to end our discussion of the SUESS Uploader. Once the file is up on the server, the Uploader's only remaining task is to kick off the Encoder stage. This is where all the really cool stuff happens, so stay tuned for the next post!

Have fun!