A Process for Context-Based Technology Evaluation

In order to determine a fit between systems and technology, it is necessary to evaluate technologies within the contexts that they will be used. This report describes a process called context-based evaluation that determines the fitness of a technology within a specific context. It includes hands-on experimentation with the technology for a greater understanding of its implications, as well as early competence development of the people conducting the experiments. An integral part of the process is the development of model problems; these are prototypes, situated in a specific context, with the goal of satisfying evaluation criteria. The focus of this report is on evaluation of software technologies, such as Web services, database systems, or architectural frameworks and development tools. The report also includes a case study of the use of this process for the evaluation of Web service technology.

List of Tables Table 1: Sample  In order to determine a fit between systems and technology, it is necessary to evaluate technologies within the context that they will be used.All technologies work well within a specific context and under certain conditions.For example, Web services work well for asynchronous communication over the Internet.In a business environment these conditions are very common.However, this may not be the case in a military tactical command and control environment where high performance and availability requirements prevail.
The process outlined in Figure 1  The focus of this report is on evaluation of software technologies, such as Web services, database systems, or architectural frameworks and development tools.We constrain the scope of the evaluation to technical issues and not the business, legal, and other concerns that are important in their own right but out of scope for this evaluation process.We believe these additional issues become important in the product evaluation and selection stage, but not in the experimentation and exploratory stage for which this process is intended.Here, nontechnical concerns are treated as constraints.
The next sections provide detail on each of the activities included in the context-based technology evaluation process.The evaluation team will then use the answers to these questions to describe the envisioned state of the organization and its operations if the new technology were implemented.
An organization may have many reasons for undertaking a technology evaluation.It is important that the evaluation team be aware of these reasons, so as to ask and answer the right questions throughout the evaluation effort.Depending on the particular situation, the team will need to address a number of concerns.The following examples illustrate this.
"* A bank currently uses a hierarchical database that has been heavily modified over the years.It now wants to evaluate relational database technology to partly replace the legacy database.One major concern is performance.
"* A manufacturing company wants to upgrade its communication mechanism with external providers from Electronic Data Interchange (EDI) to Web services.The evaluation is mainly driven by concerns about interoperability with internal and external systems.
"* An organization is exploring the introduction of e-mail encryption technology to secure electronic communication.The evaluation is mainly focused on usability, as the majority of their users are non-technical.
" An organization wants to explore future options and possibilities using open-source project management tools.The concern is whether these open-source tools will provide functionality similar to that of the current commercial tool.
It is also important to know what the scope of the evaluation effort is, the budget, and the expected time frame for delivering results.In many evaluations the scope will consist of just one technology to evaluate, but there are also situations where the evaluation concerns a choice between several competing technologies.
A list of expectations for the technology will provide an overall picture of the intended benefits the new technology may bring to the organization.Expectations include both requirements the technology must or should fulfill, and other benefits, such as improved quality of service, that are not necessarily required but can make adopting the technology worthwhile.Requirements may be further subdivided into requirements involving functionality and those that involve quality attributes, such as performance, usability, ease of development, or maintainability.These expectations are revisited in the last step of the process to determine whether they have been met to a sufficient degree to go ahead with adopting the technology.
A system's overall context may sometimes prove inflexible, thus placing constraints on the technology.Examples include legal requirements or organizational policies.A legal requirement will always place a constraint on technology use, whereas it might be possible to change a policy if so justified by the potential benefits of a new technology.If it is not possible for a technology to meet a constraint, then the technology cannot be used in the organization.
The evaluators will put expectations and constraints together to describe the envisioned state of the organization and its operations should the technology in question be implemented.
In summary, this task will provide answers to some basic questions, including SWhat is the environment in which the technology will be used?* In what ways is the system context flexible?

Step 2: Plan The Evaluation
The next step is to plan the evaluation.The details of every evaluation plan will differ based on the scope of the evaluation and the potential impact of the new technology on the business.Evaluating only one technology will require less effort than comparing multiple technologies, and the higher the impact the more detailed the evaluation will need to be.However, the following major activities will probably be part of most evaluation plans: "* Form an evaluation team.
"* Estimate effort and schedule.

Form Evaluation Team
Most importantly, the evaluation team must consist of members who have the technical and domain expertise needed to evaluate the technology in the system context.Domain experts will contribute information about the context in which the technology will need to function.Technical experts will delve into the details of the technology and to create and implement the model problems.Model problems are described in detail in Section 1.3.
It is also important to compose a team with enough authority and credibility to ensure the results are accepted in the organization.The higher the impact of the technology on the organization's business or mission, the more authority and credibility must be manifest in the evaluation team.Talking with supervisors to get commitment from the members of the evaluation team is crucial at this point to assure their dedication to the team.

Identify Stakeholders
The evaluation team must consider the points of view of all stakeholders in the evaluation effort.Different stakeholders will have different goals, expectations, and concerns regarding a new technology, and must be identified early enough to gather their input in the planning Sphase.Stakeholder input will influence the evaluation goals and scope and have expectations on the technology and the evaluation itself.Examples of stakeholders are end users of the technology, system developers and maintainers, system architects, managers, administrative staff, change agents, and sponsors.

Select an Approach
Based on the technology usage context and evaluation goals, a set of high-level tasks for the evaluation is defined.Any work that must be conducted before the definition of the model problems in the next step, such as surveys, purchases, initial investigations, and conversations with stakeholders, is included.Any reports or briefings that form part of the outcome of the evaluation are also included as tasks.
Even though most technology evaluations will be performed on one technology, in a situation where the team evaluates competing technologies and the goal of the evaluation is to choose one technology for potential implementation, it is also necessary to decide if the team should follow a best-fit or first-fit strategy.In a first-fit approach the evaluationcontinues until one technology has been identified that meets all expectations and constraints.In contrast, in a best-fit approach the team evaluates all technologies in scope and chooses the technology that meets all constraints and delivers most benefit to the organization.These benefits are measured by the degree to which the technology can fulfill the expectations of the stakeholders.In this case the team must also define how this degree will be measured and the relative weight to be assigned to each expectation.The selected strategy will affect the set of tasks and the length of the evaluation.

Estimate Effort and Schedule
Based on the list of tasks defined in the previous activity and the composition and skill set of the evaluation team, a preliminary plan is put together for the evaluation, indicating the effort required for each task as well as an overall schedule.

Step 3: Develop Model Problems
Model problems are prototypes, situated in a specific context, with the goal of satisfying evaluation criteria [Wallnau 01].In the case of technology evaluation, the purpose of using model problems is to conduct situated hands-on experimentation with the technologies, given the context of how the technologies will be used.Scenarios are constructed from the given context to provide a context-based technology fitness evaluation.
The steps for developing model problems are as follows: 1. Develop hypotheses.

Develop Hypotheses
The first task is to develop hypotheses.These are derived from the expectations placed on the technology.Hypotheses are claims about the technologies that are to be sustained or refuted.
Hypotheses provide the focus and structure that is necessitated by the expense involved in hands-on experimentation.
For Web services technology, 2 for example, valid hypotheses that correspond to a typical Web services context might be "* It is fairly easy for a developer to connect components developed for the same platform using Web services.
"*  (2) on the receiver side, nothing more than to extract the contents from the SOAP message and parse the appropriate XML document.
There will be no problems regarding data types if Web The two applications will exchange a complex data services are used to connect components developed for type, a date, and a floating point data type (known to different platforms (i.e., J2EE and .NET).
cause problems) with no additional effort other than that indicated in Criterion 3 for the previous hypothesis.

Design Model Solution
The model solution should be very efficient-it only needs to answer the question at hand.The model solution should also be realistic-it is framed within a specific scenario derived from the context.Therefore, a model solution is the simplest set of applications and/or components that are able to answer the questions posed by the hypotheses and associated criteria, within a given context.Designing a model solution will often require preparation work such as acquiring and installing components, performing interviews and surveys, or becoming quickly acquainted with the technology in order to propose a model solution.
Let us say, for example, that the context is a typical human resources arena, the organization's development environment is Java using the Eclipse IDE [Eclipse 05a], and there is knowledge of existing .NET applications that will expose functionality as Web services.The model solutions to evaluate the above hypotheses might be those described with the scenarios below: Hypothesis 1: It is fairly easy for a developer to connect components developed for the same platform using Web services.
Scenario: An organization has a Human Resources (HR) system that maintains personnel records.Training information is kept in a separate system with which the HR system must interact, using Web services to update and retrieve training information.
Model Solution: A simple J2EE HR application that -performs create-retrieve-update-delete (CRUD) operations on personnel basic data (social security number, name, address, salary) is

Figure 2: Model Solution for Hypothesis 1
Hypothesis 2: There will be no problems regarding data types if Web services are used to connect components developed for different platforms (i.e., J2EE and .NET).
Scenario: The HR system, a J2EE application, interacts with the payroll system, a .NET application, using Web services to report new employees and salary changes.The payroll system interacts with the HR system, also using Web services to indicate that the salary change has been processed.
Model Solution: A payroll application is generated using Visual Studio .NET [Microsoft 05b].The payroll application has a basic user interface that displays employee information and payroll-related events for that employee.A Web service with two operations is added to the payroll application: one to receive new employee records (a complex data type) and one to receive salary changes (social security number, a floating point type representing a percentage, and a date by which they should take effect).The HR application has an option for invoking the Web service operations.A Web service with an operation that receives salary change notifications (social security number, date processed, and a floating point representing the amount of the new salary) is added to the HR application.A simple user interface to record salary changes and view notifications is also added to the HR application.The payroll application has an option for processing the salary change and invoking the notification Web service.Figure 3  Given that there is a better idea of details of model problems at this point, it is a good idea to revise the overall plan for the evaluation that was put together in the previous step.

Implement and Evaluate Model Solution Against Criteria
In this activity, the model solution is implemented, run, and observed, until there is enough information to sustain or refute the set of hypotheses, Criteria sometimes require refining because they are too vague, making it difficult to decide whether the hypothesis is sustained or refuted.For example, Criterion 1 for Hypothesis 1 can be refined to indicate that not only should there be tools to generate WSDL documents, but also tools to generate code from WSDL documents.
A simple but effective way to capture experience and knowledge is to keep notes about product installation, problems encountered, interim results, deployment environment, and any information that might help future projects, should the technology be selected for use in the organization.The simplest mechanism to capture this information is through an informal blog that can be shared among all people participating in the model problem process.

Step 4: Analyze Model Problem Results Against Technology Usage Context
The last step in the process is to make a final determination with respect to the fitness of the technology for the context in which it is intended to be used.Model problems provide increased negotiation power and informed decision making because of the direct hands-on experimentation with the technologies.
Based on the results of the evaluation there should be enough information to determine if * the technology is a good fit with requirements * the technology is not a good fit with requirements * the technology has some mismatches that could potentially be solved by modifying the context or modifying the technology itself CMU/SEI-2005-TN-025 If the model solution provides information that sustains the set of hypotheses, and assuming sufficient confidence in the results, the technology can be declared a good fit with the requirements.Given that the model solution is a simplification of the real solution, it should be determined whether the results can scale so that they are valid in the larger context as well.
If the model solution provides information that refutes the set of hypotheses, and there is no room for negotiation, or very strict constraints that are not met, it can be stated that the technology is not a good fit with the requirements.In this case, it may be possible to find a different technology that will satisfy the requirements.The process should then be repeated, reusing the previously produced information and results as much as possible.
Most often, the model solution will provide information that sustains some of the hypotheses and refutes others.Sometimes it is also possible to modify the technology.Some examples follow: "* In the case of open source technologies, an option would be to modify the technology, making sure that the modifications are included in the main release line of the technology.This of course requires access to source code; the cost-benefit of this option must also be evaluated.The risk of making modifications that are not fed back into the open source project is an inability to take advantages of new releases, unless the same modifications are made to every new release.
"* Investigate complementary technologies that in combination would satisfy the hypotheses.
* Negotiate the possibility of performing modifications with the vendor, open source community, or standards organization.In the case of the first two, it is important to make sure that modifications are included in the main release line of the technology.This is usually easier with small vendors or small projects.In the case of standards organizations, there is usually a formal process to put forth modifications that can involve a lengthy approval time.

Technology
The following case study is an example of how the context-based technology evaluation process could be used in the evaluation of Web services in an organization that provides business intelligence and data mining services to its registered users.The case study does not include the actual technology evaluation.

Identify Technology Usage Context and Evaluation Goals Technology Usage Context
An organization is migrating from a federated suite of legacy applications and data stores, used for providing business intelligence data mining services, to a solution based on Web services.The plan is for most of the "as-is" systems to migrate via their adaptation to Web services.However, some of the older legacy systems will have to be replaced in order to fit with Web services, or eliminated, if no longer needed.
The current systems support approximately 2,000 simultaneous users.Typical queries are serviced with response times of a few (2-10) seconds, and some complex queries take as long as two minutes.These response times are acceptable by current users.Query results are provided in a variety of pre-defined and user-specified formats, such as database tables, spreadsheets, or advanced data visualization including high-resolution images.
Access to data is determined by business area.For example, users from customer relations management should not have access to long-term strategic company forecast data.
There is a requirement that the new Web-services-based system support single "sign-on" for users, rather than the current jumble of separate logins/passwords for different systems.
The organization-while new to Web services-has been in the software development business long enough to realize the significant gap that often exists between technologies touted as mature versus those that are mature in reality.The organization's main concerns are the feasibility of adapting its current systems, maintaining (or improving) current response time, allowing image transfer, and providing single sign-on. CMU/SEI-2005-TN-025

Goals of the Evaluation
The goals will be to determine how Web services use will affect the "* percentage of systems that can be adapted "* ability to maintain or improve current response time "* feasibility of transmitting images as part of messages "* possibility of having single sign-on

Scope of the Evaluation
The evaluation will be limited to the feasibility of using Web services as a modernization option for the set of legacy systems, limiting the evaluation to the four goals listed above.
Other aspects of Web services are out of scope for the present evaluation.
Expectations for the Technology "* The use of Web services will ease the eventual replacement of legacy applications due to a common interface and will provide a single point of entry and user interface for all applications.
"* Single sign-on will reduce the amount of work necessary for current users when they need data from more than one application.
Constraints "* Since this is a feasibility study, there is no budget for the purchase of new software.All software should be freeware, open source, or already licensed for the organization.
"* Java-based products are preferred given the current IT mandate to move towards Java as the development and production environment.Eclipse is the current Integrated Development Environment (IDE) tool.

Plan The Evaluation Evaluation Team
The evaluation team will be composed of "* a project lead "* a member from the core information technology (IT) group "* two members from the maintenance group "* a member from the development group Each member of the evaluation team is expected to contribute 50% of her or his time for two months.Direct supervisors will be requested to sign a commitment form to assure continued participation on the team.

Stakeholders
Stakeholders and their responsibilities are shown in Table 2.All stakeholders will be informed of the goals and dates of the evaluation, as well as possible requests for interviews and/or availability of personnel for receiving input.
Current system developers X Current system maintainers -X IT personnel X X

Approach
A system inventory will be performed before the evaluation starts, in collaboration with IT, to determine the number of systems in use and some basic characteristics regarding their potential for adaptation to Web services.
An Internet search will be conducted to select free/open-source software to satisfy the model solution.Some of the software elements below will be necessary for more than one platform, depending on the results of the systems inventory and the selected deployment configuration.
* HTTP Server * SOAP engine * Tool to convert from a WSDL document to code, to handle Web service requests in selected programming languages * SOAP libraries Model problems will be developed to address the goals of the evaluation. CMU/SEI-2005-TN-025

Model Problem Hypotheses and Criteria
Hypotheses and criteria for the model problems are shown in Table 3. Response time will not be degraded due to the use of Response times using Web services will be within the Web services.
same order of magnitude from current response times for representative applications.
Images can be transmitted as part of SOAP messages.An image can be requested using Web services and viewed on a client application.
It is possible to have single sign-on using Web A user is able to login once and obtain information services, from three different Web services residing on three different machines.

Model Solutions
Hypothesis 1: 75 % or more of existing applications can be adapted to Web services.
Preparation Work: A system inventory was conducted, gathering the following information for all applications: business area, function, name, technology (data storage, platform, programming language), and development type (commercial or in-house).The distribution of platforms/programming languages is shown in Figure 4.
Scenario: A user from one of the business areas needs data from nine different sources.A client application, using Web services, is able to obtain the data from the data sources that are located on different applications on different machines.
Model Solution: An application that is representative of each of the groups in Figure 4 is targeted for adaptation to Web services.The simplest operation available from the application should be selected for this model solution.If there is no such simple operation, a simple application is developed by one of the application maintainers that simply returns a set of data containing representative data types (integer, floating point, date, string).A Java client application is developed to connect to each of the Web services and display the results.These results are manually compared to the results from the current requests to the applications.Scenario: A user from one of the business areas requests data that is returned as an image. CMU/SEI-2005-TN-025 the application with full rights to all data and functions or impersonate individual users.In the latter case the implementation of the Web service is extended with a new component that authenticates Web service users with the legacy applications.The new authentication component will contain copies of the password databases of individual legacy applications.

Model Solution Architecture
A combined view of the component and connector view for the model solution is presented in Figure 5.The component and connector view shows a Java Client Application in the center.This application invokes the Web service operations from the different representative applications and displays results of the operations on the screen.It is instrumented to show response times from each of the legacy applications along with the results.The Web service that connects to the Java application contains two operations: one that exchanges basic data types as plain text and one that returns an image as part of the message.

WS interface Interface
Web Service

Figure 5: Component and Connector View for the Model Solution
Figure 6 shows potential deployment options for the model solution.The selection of the deployment option for each application will depend on the availability of tools and software for the legacy platform, as well as performance.
Option 1 is to deploy in the way indicated by the C, Visual Basic, and COBOL applications, as well as the commercial products with or without a Web service interface.
In this case, a machine is set up as an HTTP server.The HTTP server contains the SOAP Engine that redirects operations to the proper applications.Each application has a Web CMU/SEI-2005-TN-025 service adapter resident on the same machine with the application.This adapter is needed to make functions of the legacy application available to the Web service running in the SOAP Engine. 3The adapter converts the call from the Web service into calls to the application and then receives and sends results back to the Web service.The advantage of this approach is the unification of all interfaces to legacy applications.The disadvantage is that performance might degrade given that the HTTP server acts as a "router" for all Web service calls.Using one central component also reduces reliability because it introduces a potential single point of failure.For the single sign-on scenario the additional access control component would be deployed as a servlet on the central HTTP server node.
A variant of this option is to have several HTTP servers, depending on application usage patterns.Figure 6 shows an additional HTTP server for the mainframe application.This would indicate that the mainframe application is accessed frequently enough via Web services to warrant its own HTTP server.An extreme variant would have one HTTP server per application.The advantage of this variant is the load sharing between HTTP servers.The disadvantage is the need for additional nodes to run the HTTP servers.In this variant, access control components run as servIets in each HTTP server.
Option 2 is to deploy in the way indicated by the Lotus Notes application.In this case, the application resides on the same machine as the Web services adapter and the HTTP server.The selection of this option depends on the availability of both Web service libraries and tools and HTTP servers for the legacy platform.The advantage of this approach is that the load on the HTTP server is spread across its multiple instances.The disadvantage is the additional load on the legacy application host machine.With this option, an instance of the single sign-on access control component would run on all nodes that run an HTTP server.
Option 3 is represented by the commercial product with Web service interface and the Java application.In this case there is no need for a Web services adapter because the product is already enabled to be used in a Web service context or the application is enabled to receive remote calls.The advantage is that adapters would be unnecessary.However, the caveat is that applications or products must already be enabled to be integrated in this way, and this is probably only feasible in newer products or applications.
3 A nice depiction of how this works can be found at http://www.soapuser.com/basics4.html.Technology is moving at a faster pace than people can keep up with.Because of this, certain technologies or "buzzwords" become popular and are quickly embraced by practitioners and managers alike.Examples are client/server in the early 90s, e-anything in the late 90s, and now service-oriented architectures and Web services.The problem with embracing technologies based on their popularity is that they are often inserted into organizations--or mandated-without any formal evaluation as to their suitability to the organizations' environments.
Context-based technology evaluation allows for better informed decisions about the fitness of a technology within a particular context.The use of model problems within the evaluation process provides an organized and efficient way to "play" with technologies before they are inserted into organizations' systems.
As demonstrated by the case study, context-based technology evaluation requires extensive planning, effort, and research.But too frequently the alternative is embracing a technology that must later be abandoned because of mismatches between its capabilities and the expectations for its capabilities.It's almost certain that the cost of such an unfortunate effort will exceed and justify the cost of a context-based evaluation.
is a context-based technology evaluation process for determining the fitness of a technology within a specific context.An integral part of the process is the development of model problems, a technique initially codified for building systems from commercial components.This technique prescribes the use of focused hands-on experimentation as a way of reducing the risk associated with component integration [Wallnau 01, Lewis 05a].The process is also largely based on the PECAK process for the evaluation of commercial off-the-shelf (COTS) software products [Comella-Dorda 04].

,0
What are the goals of the evaluation?* What is the scope of the evaluation?* What decision authority will the evaluation team have?What would be a successful evaluation?* What constraints must the evaluation team adhere to?

Figure 3 :
Figure 3: Model Solution for Hypothesis 2 Figure 6: Deployment View for the Model Solution

Table 3 :
Hypotheses and Criteria for Case Study Model Problems .
................ 1 Context-Based Technology Evaluation Identify Technology Usage Context and Evaluation GoalsThe goal of this step is to determine and document the organization's reasons for conducting the evaluation, what is expected of the evaluation, what the expectations are for the technology, and what constraints, if any, must the evaluators adhere to.
Develop Model Problems) Analyze Model Problem Results Against Technology Usage Context (Mismatches Can Potentially Be Fixed] [Technology Is a Good Fit] [Technology Is Not a Good Fit] Modify Tech nology or Context-Figure 1: Context-Based Technology Evaluation Method 1.1 Step 1: There will be no problems regarding data types if Web services are used to connect Criteria are used to determine if a model solution sustains or refutes a hypothesis.Criteria are derived from requirements and are stated as a clearly measurable statement of capability.Each hypothesis can have one or more criteria, depending on the breadth covered by the hypothesis.Table1illustrates valid criteria for the above hypotheses. 2A service is a coarse-grained, discoverable, and self-contained software entity that interacts with applications and other services through a loosely coupled, often asynchronous, message-based communication model.A Service-Oriented Architecture (SOA) is a collection of services with well-defined interfaces and a shared communications model.Web services are the most common form of SOA, in which service interfaces are described using Web Services Description Language (WSDL), payload is transmitted using Simple Object Access Protocol (SOAP) over Hypertext Transfer Protocol (HTTP), and optionally Universal Description, Discovery and Integration (UDDI) is used as the directory service [Lewis 05b].6 CMU/SEI-2005-TN-025

Table 1 :
Sample Hypotheses and Associated Criteria using the Java Eclipse IDE.Tomcat is the Servlet engine, the J2EE application server is JBoss, and data is stored in an Oracle database [Apache 05a, JBoss 05, Oracle 05].A second J2EE application that keeps track of basic personnel training information (course name, date taken, location) is created using the same IDE and deployed on the same platform.A Web service with two operations is created in the training application using Eclipse plugins from the Web Standard Tools (WST) Platform subproject [Eclipse 05b].One operation submits new training information to the training application and the other operation returns training information for a given social security number.The HR application has an option for invoking the Web service operations.Figure2illustrates this model solution.
CMU/SEI-2005-TN-025 created In this case of mismatch, it is useful to determine if either the context or the technology itself can be modified.Some examples of context modification are "* Reset technology expectations.

Table 2 :
Stakeholders and Responsibilities for Case Study Technology Evaluation

Table 3 :
Hypotheses and Criteria for Case Study Model Problems Tools are acquired to help in the task, if available.Time is tracked for the adaptation and 14 CMU/SEI-2005-TN-025 client development tasks.Interviews with maintainers should help determine whether the effort can in fact be generalized to the rest of the applications in each group.The architecture for the solution is presented in Section 2.3.3.Hypothesis 2: Response time will not be degraded due to the use of Web services.Using the same representative applications as for Hypothesis 1, response time is measured for the current requests to those applications.The client application developed for Hypothesis 1 is instrumented to show response times for each of the requests.To make the results comparable, all times are collected during similar network traffic periods.Time permitting, the different deployment options presented in Section 2.3.3 should be tested to see how performance is affected.