Defining the Vision

Mary Kirtland

Microsoft Corporation

January 10, 2001

In last week's column, I introduced the Web Services Guidance team and our mission: building, deploying, and operating sample Web Services to illustrate the kinds of issues you need to consider when you do the same. I also introduced our first sample project, the Favorites Service.

Thanks to everyone for your comments and feedback. We are keeping track of the issues you raise, so keep 'em coming!

Someone asked why it would take three months for us to release the sample and why we hadn't started on it before we announced the idea. In fact, we have been working on Favorites pretty much full time for the past two months. A lot of the functionality is, well, functioning… but there's still a bit of work to do before everything is packaged up nice and neat into a sample you can install on your own machines or try out over the Internet. It also takes some time to write, tech review, edit, and process all the articles that we want to ship with our sample sources. So that's why we're aiming for three-month project cycles. We're keeping our fingers crossed that we'll be able to get the first release of Favorites out sometime in February. (As I write this, the team is going through a scheduling exercise to get a better estimate of the release date.)

Some of the comments also assume that we're using the .NET Framework to develop this sample. That's not necessarily the case. We'll use whatever technologies we think are best for the job. And best doesn't always mean technically superior, easier to use, or most in the news. Whatever we choose, we'll tell you why and we'll tell you what issues we ran in to when we tried to use it.

This week, we'll start to look at the issues our team has encountered while building the Favorites Service. There's quite a backlog, but we'll start at the beginning: figuring out the goals and objectives for the project.

Getting Started

In the Microsoft Solution Framework process model, the first phase in any project is the envisioning phase. During the envisioning phase, the project team and customers create a high-level view of the project's goals and constraints. The primary deliverable from this phase is a vision document, which contains an analysis of the business problem, a description of the goals for the product, an outline of the solution concept, profiles of the product's users, design goals, and a definition of the project's scope. The envisioning phase culminates in the vision/scope approved milestone.

Our team began from an unusual starting point: We knew we wanted to write a Web Service, but we didn't have any particular problem domain in mind, nor did we have existing applications we had to use. So the first thing we needed to do was come up with a reasonably realistic business scenario that could justify building a scalable, reliable, etc., etc., etc. Web Service.

We began by locking a bunch of people in a room and brainstorming potential businesses or industries, services, and technical issues that would make good topics for guidance. Along the way, we came up with some reasons why you might want to build Web Services:

You are a source of time-sensitive or parameterized data that end users or other businesses want to use. If the data doesn't change often and clients almost always want all of the data, you might as well just post an XML document on your Web site. But if clients want to execute queries against your data, then a Web Service makes a lot of sense. If you want to push data to clients, the subscription and notification operations are also potential Web Services.
You implement algorithms that other businesses can't or don't want to implement themselves. In this case, each algorithm is a potential Web Service: Clients pass in the data, you respond with the answer.

You aggregate several other services to provide a higher-level service. In this case, clients use your service because it provides the combination of information they want, saving them the work of combining the existing services themselves.

You want to integrate your businesses processes with partners. Each step of the business process that goes across an enterprise boundary is a potential Web Service or Web Service operation.

You have a heterogeneous and/or geographically distributed enterprise architecture, in which using private networks or tightly coupled protocols for application integration is not practical. The API of each application you want to integrate is a potential Web Service.

You provide infrastructure services ("plumbing") for other Web application and Web Service developers, for example an identity service or a billing service. Instead of building custom API libraries for every client platform, you can expose your API as a Web Service.

Of course, for any of these reasons, you also need to consider whether there are a sufficient number of potential clients for your service to justify the costs of implementing and operating it.

We also quickly realized there were some other considerations when building sample services for educating a general developer audience. First, the business scenarios should not require extensive knowledge of a given industry. Second, we want you to be able to install and run the samples on your own machines. Third, many interesting scenarios require one or more data stores or feeds. There are lots of issues when it comes to shipping sample source code that shows how to access data sources we don't own. And we don't own any data sources…at least not that we're at liberty to give away with a sample.

This led us away from scenarios such as online banking, control home digital video recorder, check flight status, or daily comics server to something more along the lines of an infrastructure service. We started thinking about the kinds of Web Services that MSDN could provide (real services, not samples). One idea that people really liked was a way to keep track of MSDN articles they wanted to find again, regardless of which machine they used to access MSDN. That led us to the idea of server-side favorites.

The Vision, Rev 1

Once we had a rough idea of the service we wanted to build, we built up a business scenario around it:

We are looking for ways to expand our client base for our Web development practice and to generate a regular revenue stream. We believe that we can achieve both objectives by providing high-quality Web Services that Web sites will want to use. Because Web Services are a new concept, we feel that potential customers will need to be convinced they can bet part of their business on our services. Thus we need a teaser Web Service that:
Offers obvious value to potential customers' Web sites.

Does not provide a mission-critical service.

Shows the quality of our development and operations practices.

Can be implemented and deployed at reasonable cost to us.

Here's the first cut at our vision for the project:

The server-side Favorites Service will enable Web surfers to store their favorite links in a safe, secure, central location, so their favorites are accessible anywhere, any time. We will offer the service free of charge to all users; however because we want to protect each user's privacy, users will need to be authenticated to access their favorites. By using the service through a client Web site, users agree that we may use their favorite lists after all personally identifying information is stripped from the data.

In phase two of this project, we will license additional services to Web sites. For example:

Statistics on which pages from their sites are being bookmarked.

Recommended links based on an affinity analysis of all user favorites.

From a technical perspective, this scenario looked like it would address many of the questions we'd been hearing from customers. The business logic didn't look too hard to implement —or too hard to explain to our readers. But once the vision statement was written down for everyone to look at, some big red flags went up:

We'd need a global user identification scheme, a scheme common enough that Web sites would be able to pass along the user identifier to our service.

How would we authenticate the end user when requests come from a Web site rather than directly from the end user? Sounds like we'd need delegation, or we'd need to trust the Web sites to only make requests on a user's behalf when the end user is actually using their site.

Should any Web site be allowed to retrieve an end user's favorites that were added from another Web site? If so, would we want to let any Web site use our service, or should we restrict access to the service to licensed sites? Letting any site have access to any and all data stored in the Favorites Service without any sort of input from the end user sounds like a huge privacy problem.

Similarly, what if our Web Service provided update methods to maintain user favorite information. Should one Web site be allowed to modify an end user's favorites added from another Web site?

Wouldn't end users yell and scream if they found out that we were doing things like affinity analysis on their data collected from multiple Web sites? How would they even know that these sites were using our service? Would we need to do something like require an end user to register with our service and set their privacy preferences before any site could access their data? How would the end user know to register with us?

Perhaps some additional research was in order.

The Vision, Rev 2

After reviewing everything I could find about user privacy, I spent a couple hours with a member of Microsoft's Corporate Privacy Group discussing the scenarios potentially enabled by our service and the privacy implications. I took page after page of notes, and there were still a few open issues. But the scenarios where one site could access or modify data added from another site sounded awfully dicey from a privacy perspective. That's not to say they shouldn't be allowed, just that it's harder to write an accurate yet understandable privacy policy to cover these scenarios, and end users might not be comfortable with the scenarios.

We also did some preliminary research into the authentication issue. There really isn't a good way today to globally identify and authenticate users over the Internet when there's an app in the middle. Microsoft Passport works great when a user is interacting with a Web site through a browser. But there's no secure way for the Web site to pass the user's credentials on to another Web site. What about the single login feature of Passport, where users don't need to re-enter their user names and passwords when they move to another Passport-enabled site? That uses client-side redirects. And our Web Service never talks directly to the end user's machine.

Based on this preliminary research, we decided to implement Favorites in phases. This would give us more time to sort out the privacy implications, and maybe the Identity Web Service that was mentioned several times at last year's PDC would be available by the time we needed a global identity scheme.

Our revised vision looks like this:

We will implement and deploy a Favorites Web Service during CY2001 that will enable end users to bookmark selected pages from Web sites for later access. These favorites will be stored by the Favorites Service, so they are accessible from any machine or device the end user is using.

The Favorites Service will be used to advertise to potential customers and thus must be a highly available, scalable, secure service that showcases the firm's commitment to quality and personal privacy.

Purists may argue that this is far too long to be a vision statement. The important thing, however, is that everyone understands it, whatever you want to call it.

We defined the phases like this:

In phase one, we will implement and deploy a Favorites Web Service that can be licensed to Web site developers for use behind the scenes to manage their customers' favorites. (Effectively each licensee has a private data store of user favorites.) We will also provide a sample that shows how to integrate the Favorites Service into a Web site. The Service and sample will be available for licensing by March 1, 2001. Our goal is to license the service by March 1, 2001, to five to 10 sites that represent approximately 10,000 new end-users per month. (We believe this is a manageable rate of growth, given our current staff.)

In phase two, we will implement browser extensions for Internet Explorer and Netscape Navigator that will give end users a safe, secure way to manage their favorites to any Web site, in the same way they manage client-side favorites today. We will also implement and deploy a Web site end users can use to manage their favorites, read our privacy statement, set user profile options regarding the use of their personal information, and download browser extensions. The browser extensions and Web site will be delivered by May 1, 2001. Our goal is to reach 1,000 new end-users per month.

In phase three, we will extend the Favorites Web Service so that end users can see both favorites they have saved directly (via the browser add-in or Favorites Web site) and favorites that have been stored through licensees of the Favorites Service. Licensees will have the option of displaying a special logo that lets end users know the Consulting Favorites Service is being used (and hence these favorites will also be accessible through the browser add-in or Favorites Web site). Licensees may also continue to use the Favorites Service behind the scenes, in which case favorites stored through that site will not be available through the browser add-in or Favorites Web site. Phase three will be delivered by June 1, 2001.

Phase three delivers a foundation for future Web Services. For example, if a user visits a Web page, the site could display a list of Web pages that other people who liked the page also liked. The Web site would retrieve the list of pages from a Web Service. We have not yet decided whether to implement this type of profiling service.

With that in place, we could flesh out the rest of the Vision document.

The next step was to define some user profiles. It's important to think carefully about whom all your users are when you define the vision for a Web Service project. The user categories we identified during the envisioning phase are:

End Users—anyone who uses a Web browser to visit Web sites.

Licensees—any business with a Web site that uses the Favorites Service.

Licensee Developers—developers building a licensee's Web site.

Licensee Operators—operators of a licensee's Web site.

Operators—the people responsible for operating the Favorites service.

You can read the complete user profiles in the Favorites vision document that accompanies this column. It turns out we missed a couple categories: managers and managers of licensees. These cropped up when we started thinking about reporting requirements. We probably should have split out testers as a separate category as well. This list should provide a good starting point for most Web Service projects.

We also spent a bit of time thinking about business goals and design goals. One of the main questions is: Why are you building the Web Service? Are you trying to make money? Trying to streamline business processes? Or something else altogether? Whatever you decide, it's likely to have an impact on your requirements. For example, the primary goals for the Favorites Service are:

Showcase our commitment to quality products.

Avoid bad press due to unreliable service, security problems, or personal privacy issues.

This combination of goals added a significant number of requirements on our service, especially in the areas of licensing and the ability requirements (scalability, availability, and so on.). But making money is a complete non-goal for this project. If it were a goal, many of our licensing requirements would have come out a bit differently.

From a design perspective, you should think about your overall philosophy about user privacy, security, and other non-functional requirements. You should also consider whether clients around the world will use your Web Service, and what this implies about globalization and localization. For example, a couple of our design goals are:

Deliver a worldwide solution. The service and all supporting applications must be globalized.

Deliver a reliable, secure service. The service must be highly available, must not lose data, and must only permit access by authorized users.

You can find the rest of our business goals and design goals in the Favorites Vision document.

Conclusion

It's always easier to know when you're done with a project if the project team and your customers have agreed to the overall goals, objectives, and scope up front. Even if you think you're just adding a Web Service front end to an existing Web site, it's worth taking the time to write up a vision document. Why are you adding the front end? Whom do you expect to use it? How much additional load is that going to put on your site that your ops people aren't expecting?

Writing up the vision document for Favorites was a valuable exercise for us. Without it, we would have missed at least half the requirements that ended up in our functional spec. For example, we probably wouldn't have recognized the privacy and security issues until much later in the game. The Vision document does other things for us as well. It gives us a yardstick for evaluating feature requests and prioritizing requirements. The document reminds us what we want to do in future phases—we can account for that in the sample design so we don't need to completely rewrite things down the road.

Next week, we'll look at the privacy issues mentioned here in more detail. I'll let you know what I learned from our Corporate Privacy Group, talk about the hard problems we've deferred, and discuss the privacy issues that remain in phase one and how these have an impact on our design and implementation.

See you then!

Mary Kirtland is the architect for the MSDN Web Services Guidance team, where she does everything but coding, testing, or operations, including trying to keep the specs up to date, writing down everything the team has learned in articles for MSDN, asking annoying questions during reviews, and ordering lunch.