Performance Analysis of WS-Security Mechanisms in SOAP-Based Web Services

Identity management (IdM) solutions in web services environments are often compared on the levels of performance and security they provide. Selecting the appropriate IdM solution for a given system or application often requires making tradeoffs between security and performance, while also considering the system’s contextual and environmental requirements and constraints. This paper presents the results of a series of experiments targeted at analyzing the performance impact of adding WS-Security, a common security standard used in IdM frameworks, to SOAP-based web services. The goal of this work is to establish a baseline of performance data that can be used to explore performance/security tradeoffs in environments with complex attributes, such as resource or bandwidth limitations.


Introduction
Security and performance are two primary quality attributes that shape the architecture of any software-based system.These two quality attributes are often in tension [5].This tension is amplified in systems that use XML-based web services.While using XML as the communication medium for web services promotes quality attributes such as interoperability, flexibility, modifiability, and shorter development time [11], its use to support security significantly affects performance-primarily due to its verbosity [4].The standard means of adding security is to increase the content of each XML message, which adds to the burden of an already heavyweight communication medium and magnifies the tension between security and performance.
This paper describes a series of experiments that focus on understanding the performance impact of different security-level mechanisms based on WS-Security. 1 The collected data represents a starting point for understanding tradeoffs between security and performance and forms a basis for making engineering and architectural decisions.Section 2 presents related work.Section 3 presents an overview of the security mechanisms that are being examined and the experimental methodology.Section 4 presents the experimental results and provides some analysis.Section 5 presents overall conclusions based on the experiment results.Finally, Section 6 presents possible future work to leverage the baseline established in this paper.
1 WS-Security has been defined as "a standard of SOAP extensions that can be used when building secure Web services to implement message content integrity and confidentiality" [6].

Related Work
There has been considerable effort to benchmark various aspects of web service performance.
Work by Head and associates [15] primarily focuses on comparing the performance of popular SOAP 1.1 toolkit implementations circa 2005 and thus is out of date with the most recent SOAP engines.Further research led by Head [16] expands that work to compare the performance of a wide variety of XML parsers and updates to SOAP 1.2 but is still orthogonal to our interests in security.Additional protocol comparison with no security considerations is found in "Performance of Web Services" by Jeckle and Melzer [21].Finally, the work reported in "Comparing Web Service Performance" [22] demonstrates a performance comparison of two popular application servers running web service benchmark tests.
Early discussions of the cost of implementing security included the blog posts in "Fat protocols slow Web services" [23] and "Fat Protocols" [24].The work in "Characterizing Secure Dynamic Web Applications Scalability" [17] takes the first step in examining the cost of security in web services, but restricts its scope to channel-based (SSL) security."Comparison of performance of Web services, WS-Security, RMI, and RMI-SSL" [19] takes the next step by comparing two different methods of distributed communication (RMI and web services) and comparing the impact of implementing comparable security measures via WS-Security and RMI-SSL, but does not consider different levels of security.
However, there are fewer examples of research that specifically examine the performance impact of different levels of security in SOAP-based web service environments.Sosnoski's work [18] approaches our topic by discussing different approaches to securing web services, but does not offer any quantitative comparison.The work by Liu, Pallickara, and Fox [20] is notable in this context as it focuses on the comparison of performance of different encryption algorithms and does a limited comparison of different security mechanisms.It differs from the work described in this paper due to the far more extensive set of security approaches and metrics that are considered in this paper.
The work that is most closely related to the work presented in this paper is by Chen, Zic, Tang, and Levy [3]; this work compares different security mechanisms in detail and addresses payload size.The contribution of the work described in this paper is the comparison of a larger range of payload types and sizes, analyses of different payload complexities, the secure conversation security mechanism, and mechanism startup costs, as well as the examination of the cost of security from the perspective of message size and resource usage.

Experimental Approach
Adding security to web services has a cost that depends on the type (or types) of security mechanisms used.This cost can range from performance degradation, to increased message size, to additional consumption of resources, or some combination thereof.It is commonly accepted that more complex security mechanisms incur greater costs due to increased overhead.However, this may or may not be always true because the cost incurred by security is context-and situationdependent.In order to make informed decisions about performance/security tradeoffs, it is necessary to measure and compare these costs.
The experiments described in this paper seek to establish a baseline of results that can be used to compare different security mechanisms in a web services context.With such a baseline of measurements, further experiments can be designed and executed to compare the tradeoffs in more complex scenarios, such as contexts in which network connectivity is intermittent or where available computing resources are constrained or limited.

Experiment Goals
The goal of the experimentation was to find the answers to three high-level questions that address the cost/benefit tradeoffs of security in the context of WS-Security and SOAP-based web services.
1. What is the cost of security with respect to roundtrip (request-response pair) time?
2. What is the cost of security with respect to message size?
3. What is the cost of security with respect to resource usage?

Experiment Subjects
This section describes the different end-to-end, message-level security mechanisms that were selected to be benchmarked by the experiments.Each security mechanism under test was evaluated from the perspective of the three experiment questions.In addition, combinations of the mechanisms (e.g., integrity and confidentiality) were benchmarked to compare their performance with that of the individual mechanisms.Combinations of mechanisms will generally provide a more secure environment overall than individual mechanisms alone and are commonly used in commercial and industrial security frameworks such as multifactor/multilayer authentication [12].

Security Mechanism #1: No Message-Level Security (No Security)
The first mechanism under test used a plain SOAP message exchange with no security overhead.This established a performance baseline.

Security Mechanism #2: Message Authentication (Password Only)
The second mechanism involved username/hashed password authentication implemented at the message level.To accomplish this, authentication tokens within the SOAP message were enabled using WS-Security.

Security Mechanism #3: Message Integrity (Sign Only)
The third mechanism involved ensuring message integrity through the use of digital signatures and hashing.To accomplish this, WS-Security in the message header was enabled by adding a digital signature to the SOAP document, using XML Signature, for both the request and the response.
Security Mechanism #4: Message Confidentiality (Encrypt Only) The fourth mechanism involved encryption to ensure message confidentiality.To accomplish this, WS-Security was used in the message header to encrypt the SOAP document, using XML Encryption, for both the request and the response.

Security Mechanism #5: Integrity and Confidentiality (Sign Then Encrypt)
The fifth mechanism involved a combination of two different approaches, digital signatures and encryption.To accomplish this, WS-Security was used in the message header first to apply a digital signature and then to encrypt the SOAP document, for both the request and the response.

Security Mechanism #6: Integrity and Confidentiality (Encrypt Then Sign)
The sixth mechanism reverses the order of operations in Security Mechanism #5.To accomplish this, WS-Security was used in the message header first to encrypt and then to apply a digital signature to the SOAP/XML document, for both the request and the response.

Security Mechanism #7: Secure Conversation
The seventh mechanism provides a conversation-level (as opposed to a message-level) approach to security.A conversation, or sequence of messages exchanged between two participants, can have security data abstracted to a predetermined token that is established at the beginning of the conversation.The token is attached to the header of each following message and specifies the mechanisms to be used for each message in the conversation.This approach is in contrast to mechanisms 2 through 6, in which all necessary security mechanism information is included in the header of each message.
The implementation of the Secure Conversation mechanism for this experiment is slightly different from the WS-SecureConversation standard.This implementation uses the sample implementation provided by Apache Rampart that leverages a SymmetricBinding construct for bootstrapping-with a timestamp, a signature, and encryption as the security mechanisms [10, 1].

Experiment Architecture
The test suite consisted of several experiments that focused on the performance aspects of interest: roundtrip time, message size, and resource usage.
The experiments were run on a pair of servers running Apache Tomcat with Apache Axis2 as the SOAP engine and Apache Rampart as the security module [14,2,1].The experiments leveraged sample implementations of the different security mechanisms provided by Apache Rampart.These examples consisted of a simple echo pattern in which the client passes a "Hello World" string to the service and the service echoes the string back [1].Most of the experiments modify the basic service in minor ways to test different aspects of the communication.These modifications are explained in detail in the following sections.To gather performance measurements, the examples were modified using the Perf4J and Apache Log4J framework and inserting instances of the Perf4J StopWatch class [7,8].
All of the test cases are based on request-response message exchanges using HTTP as the communications channel.Because an actual implementation is likely to use HTTPS/SSL, this aspect is considered a constant and therefore was removed from these test cases.Figure 1 provides a highlevel view of the basic architecture of the testing framework.For further technical details on the experiments, please consult Appendices A-E.

Experiment Configurations
The testing efforts looked at several ways to benchmark the impact of security on roundtrip response time, message size, and resource usage.To analyze this impact, each experiment exercised one of two web services using one of two test configurations.

Web Services
"Echo" Web Service (Simple Payload Structure): This service takes an input string and returns the same string as the response.
"Employee Details" Web Service (Complex Payload Structure): This service takes no input and returns a one-element array of employee objects.An employee object contains an employee's details, such as name, e-mail address, phone number(s), and social security number.

Configurations
"Static Payload" Configuration: In this configuration, a client application executes the target web service 10,000 consecutive times with the default input.(The Employee Details service has no default input.)Both services were exercised in this configuration.

"Dynamic Payload" Configuration:
In this configuration, a client application executes the target web service 10,000 consecutive times with an input payload that is increased in each execution.This configuration is only used with the Echo service, which has a default initial payload of a simple "Hello World" string. 2 The payload is increased with each execution by appending an ad- The experiments with the Echo service used a "Hello World " string, a 12-character string with a space character at the end.There was no particular reason to use this string instead of the 11-character "Hello World" string.
It was an arbitrary choice.
ditional "Hello World" string to the payload used in the previous execution.This results in a range of payload sizes from 12 bytes up to 100+ kilobytes over the 10,000 executions.

Experiment Measures
Consistent with the experiment goals, each experiment analyzed three different benchmarking measures: 1. roundtrip response time 2. message size

resource usage
While we can measure roundtrip response time and message size directly, our approach to measuring resource usage had to be less obvious.One approach we could have taken to analyze the resource usage of the different security mechanisms would be to record CPU usage and memory usage over the course of the execution of a test.However, CPU and memory usage are extremely dependent upon the resources available on a given platform.We chose instead to simply record those measures and observe the impact upon them (if any) of different Java Virtual Machine (JVM) environments.
Our results (which will be discussed in detail later) showed that in resource-constrained environments, memory management tasks such as garbage collection can have significant performance impact upon the execution of even simple web services.Therefore, our approach was to identify the inflection point at which the memory management tasks stopped impacting the performance of the experiments.While exact by no measure, this approach can give a general idea of the resources necessary to effectively leverage the security mechanisms under test.

Results and Analysis
This section describes experiments we performed to test the performance implications of the security mechanisms discussed in Section 3.2 and summarizes our results.The purpose of this analysis is to understand the impact of using different security mechanisms on system performance.As discussed in Section 3, the experiments also establish a baseline of metrics for future work on tradeoffs in complex environments.

The Cost of Security: Roundtrip Response Time
Our first focus was on the performance of each security mechanism for a roundtrip response time.
The initial experiments confirmed our hypothesis that adding security to web services considerably increases roundtrip response time.Roundtrip response time of SOAP-based web services with different security mechanisms can be as much as 20 times higher than that of an unsecured web service, as shown in Table 1.This table shows the average, median, and standard deviation roundtrip time in milliseconds of each mechanism in a test using the basic Echo service (simple payload structure) in the static payload configuration.Building on these results, we looked at the performance of each security mechanism by running an experiment with the Echo service using the dynamic payload configuration.Figure 2 shows the performance measures of each mechanism in this experiment.Finally, we re-ran the experiment with the Employee Details service (complex payload structure) in the static payload configuration to examine how the different security mechanisms handle the difference in payload type and size.These results are shown in Table 2.The data collected from these experiments can be analyzed in a number of ways.The only overall conclusion is that there is no clear "winner"; however, the analysis presented in the following sections should help in making design decisions.

Simple vs. Complex Payload Structure
From the results in Table 1, and Table 2, and Figure 2, it is clear that message payload structure (whether a simple string or a complex object converted to an XML-based structure) can significantly impact the performance of each security mechanism.Even though it would be logical to assume that the more the complex a mechanism the larger the impact on roundtrip response time, results from the three experiments do not support this assumption.We analyzed Sign Only versus Encrypt Only, the two Sign/Encrypt mechanisms, and Secure Conversation.
It is useful to compare the performance of Sign Only to Encrypt Only for the Echo service case to see where they diverge, because Figure 2 suggests that they are roughly equal for some small message payload sizes.Figure 3 shows this comparison.Analysis of this data suggests that for messages up to approximately 7kb in payload size with simple payload structure, Encrypt Only performs as well as Sign Only.However, past the 7kb payload size point Sign Only provides superior performance.The data in those tables also suggests that both Sign/Encrypt mechanisms have virtually identical performance in the simple payload structure case, but Encrypt then Sign performs better in the complex payload structure case.This mechanism has a lower average roundtrip time and greater reliability (shown as a smaller standard deviation).While no data collected provides conclusive explanation for this difference, one possibility is that Encrypt then Sign causes less work because it does not require encryption/decryption of the digital signature prior to verification.
Finally, in the simple payload case Secure Conversation shows the best performance of all of the mechanisms that use encryption (see Table 1).However, when used with complex structure payloads, it shows the worst performance of all mechanisms (see Table 2).In addition, for complex payload structures, Secure Conversation appears to have somewhat unreliable performance, exhibited by the high standard deviation numbers noted in Table 2.The reason for the low reliability is unknown; one possibility is that the overhead for maintaining the conversation token may be responsible.

•
In all cases, the best performing mechanism (aside from no security at all) was the Password Only mechanism.Because this mechanism provides relatively weak security, its use is usually not recommended.However, in situations where the need for message-level security is low (e.g., secured networks, SSL-based communications) Password Only may be a viable security mechanism that provides maximum performance.
• There are several tradeoffs between Sign Only and Encrypt Only approaches to message-level security.In contexts where either will provide sufficient security and performance, the choice may be made based on the complexity of the message payload structure.For simple payload structures, a Sign Only approach provides the best performance, while for complex payload structures the Encrypt Only security mechanism performs the best.Also, it should be noted that for messages with simple payload structure and payload size under 7kb or so, both mechanisms will result in approximately the same performance profile.

•
In the case of messages with simple payload structure, both Sign/Encrypt mechanisms have the same performance.For messages with complex payload structures, Encrypt then Sign outperforms the Sign then Encrypt approach.Also, if a token-based session mechanism such as Secure Conversation is unnecessary or undesired, the Encrypt then Sign mechanism is probably the better approach.
• When used with messages with simple payload structure, Secure Conversation is the top performer of all the encryption-enabled mechanisms we tested.However, Secure Conversation does not perform well with messages with complex payload structure and thus seems to be appropriate only when performance is not the driving quality attribute.

Startup Times
Startup time is the amount of time that a roundtrip communication requires on the first execution of an experiment.This time can be relevant when lost connections need to be reestablished.Our test results confirmed the intuitive expectation that more complex security mechanisms require longer startup times (see Figure 4).One interesting phenomenon is that there are two classes of startup times, initial communications and all subsequent communications.Initial communications represent the first roundtrip communication that a service participates in after deployment or redeployment.Results show that the startup time associated with initial communications tends to be roughly twice as long on average than the startup time for all subsequent communications with that service.The data in Figure 4 for Sign Only startup time matches to some extent data for its non-startup roundtrip time relative to performance for each mechanism in Table 1, Table 2, and Figure 2. The differences observed between startup and non-startup times are that

•
Startup time for Sign Only shows significantly better performance than any mechanism with encryption, with Echo or Employee Details service (see Figure 4).
• Non-startup time for Sign Only, Encrypt Only, and Secure Conversation is similar with Echo service (see Table 1).
• Non-startup time for Sign Only has worse performance than Encrypt Only, though better than Secure Conversation by a significant margin, with Employee Details service (see Table 2).
Encrypt Only startup time is similar to both Sign/Encrypt options, a result that does not correlate to its non-startup performance.This result is unexpected and cannot be explained by our data or analysis.Perhaps the time required to initially access the shared keys is significant.Secure Conversation is especially poor for initial communications: a 59% increase over the Encrypt then Sign (versus a 22% increase over Encrypt then Sign in subsequent communications).

Startup time performance of Secure Conversation correlates best with the non-startup roundtrip time performance results presented in
One other notable result is the increase in roundtrip time when the Employee Details web service is used instead of the Echo service.One reason for the increase might be payload size. 3The base payload of the Echo web service is 12 bytes, while that of Employee Details is approximately 112 bytes [26,27].Considering that the call to the Employee Details service has no arguments while the Echo service has the same input as output, the roundtrip payload size with the Employee Details service is approximately 450% larger than that with the Echo (112 versus 24 bytes).
To examine how greater payload size increases roundtrip time, we repeated the experiments using the Echo service with static payloads of 60 bytes (5 concatenated "Hello World" strings) and of 120 bytes (10 concatenated "Hello World" strings).In the 60-byte test, total message payload over the roundtrip sequence was 120 bytes, close to the Employee Details payload, testing the impact of sending the same total payload.The 120-byte test evaluated the impact of the same payload size being transferred on the initial communication.These tests allow us to examine if roundtrip or initial payload size was the reason for the roundtrip time increase observed when using the Employee Details web service.
As shown in Figure 5, the effect of payload size on roundtrip time appears to be negligible.Some small increases in time resulted as payload size increased with the Echo web service, but some small decreases were seen as well.For each security mechanism, in addition, there is a relatively constant increase in roundtrip time when the Employee Details service case was used, compared to the Echo service.As such, the data seems to suggest that payload complexity drives performance difference.The final observations that can be made based on the startup data have to do with total cost of communication.While Table 2 shows that Encrypt Only outperforms Sign Only in non-startup communications with complex payloads (157 versus 174ms), Figure 4 shows that startup times for Encrypt Only exceed those of Sign Only. Figure 6 plots the total time of the two mechanisms, showing that for roughly the first 46 messages Sign Only performs better than Encrypt Only.Also, for simple payloads, the total cost of both Sign/Encrypt mechanisms can be lower than that of the Secure Conversation mechanism despite the lower non-startup costs of Secure Conversation for some finite set of messages.Figure 7 shows this inflection point to be roughly after the 12 th message.

Conclusions
• Initial communications (the first that a service has after deployment or redeployment) will have roughly twice (or more) the subsequent communications cost.
• Any security mechanism that uses encryption will experience significantly longer startup times than any non-encryption mechanism.
• Each security mechanism has a significant impact on roundtrip response time of the initial communications a client application has with a given service.The more complex the security mechanism, the higher the cost incurred.(Recommendation: If possible, newly deployed web services should be immediately executed by some application, such as a test or dummy application, to eliminate the post-deployment initial communications startup cost.) • Communications with services that handle complex structure payloads (such as XML data types) will have a higher startup cost than communications that deal with simple structure payloads.
• For short message sequences (< 50 message exchanges), the Sign Only mechanism performs better than the Encrypt Only mechanism, due to lower startup time.For extremely short message sequences (< 10 message exchanges) either Sign/Encrypt mechanism performs better than the Secure Conversation mechanism, due to lower startup time.

Rate of Cost Increase as Payload Size Increases
The data represented by Table 3 suggests that increasing payload size causes some security mechanisms to experience higher roundtrip time.If true, this finding provides insight on which mechanisms are better suited for larger or smaller payload sizes.Table 3 presents an analysis of the rates for each mechanism at minimum payload size and 117kb payload size.Due to the relatively high standard deviations documented in Table 1 and Table 2, it is difficult to calculate a rate of increase for each mechanism.We took the minimum value found during the dynamic payload Echo service experiment (results documented in Figure 2) and compared it to the average of the last 10 results from that data set (where payload is roughly 120,000 bytes or ~117k).We used the average time to mitigate spikes due to memory management issues.This method is inexact for determining the rate of increase, but it provides a general approximation that confirms what was observed in Figure 2. Of the mechanisms with encryption, Sign Only performs best, Encrypt Only worst, with the Sign/Encrypt mechanisms and Secure Conversation rating between the two.
This analysis confirms the hypothesis that the most complex algorithms have the highest rate of increase in roundtrip time cost as payload increases.The most interesting result is Secure Conver-sation, which performs well compared to the encryption-based mechanisms and confirms the early analyses from examining Figure 2.

•
For message payloads with simple structure, the Sign Only mechanism is the best non-trivial security mechanism for passing data of increasing sizes.
• For message payloads with simple structure, Secure Conversation can be a good choice for enabling security while ensuring good performance as payload size increases, compared to other mechanisms.

Convergence of Security Mechanisms
One property that may be important to examine is the speed at which a given security mechanism converges to its normal values.As shown in Figure 8, each mechanism needs a certain number of communication exchanges before it stabilizes to some predictable level of performance.This stabilization behavior can be defined by calculating the mathematical convergence of the sequence.A sequence of real numbers is said to converge if there is some number X such that past some point in the sequence, all following numbers in the sequence are within some range of X [25].For a given element Z of the sequence, if it is the Y th or later number in the sequence, |X -Z| < some value ε.
Unfortunately, inherent noise in the data prevented the calculation of a useful value of Y for any value of ε smaller than 30ms, a significantly large range for data sets that show standard deviations no larger than 4ms (as shown in Table 1).We removed noise from the equation by looking not at ranges, but at how far into the data set we have to go for each mechanism to produce a value that is less than or equal to the resulting average.While not nearly as precise a measure of convergence, this approach eliminates the effect of noise and provides a general sense of the speed of convergence.Table 4 shows the results of this analysis on the experiment that used the Echo service in static payload configuration.To contrast this, the same numbers were calculated for the Employee Details service.The results are presented in Table 5.On the one hand, the numbers between Table 4 and Table 5 are similar, in that No Security takes a short time to reach its ultimate average while other mechanisms take much longer.This comparison confirms the hypothesis that adding security mechanisms is costly to roundtrip time from a convergence standpoint.On the other hand, the stabilization occurs much faster in the complex payload structure case overall (with the Employee Details service), which is not easily explained and is outside of the scope of this work.
The outlier data point to note is the performance of Secure Conversation in the complex payload structure case.It outperforms even the No Security option.While extremely unexpected, this result can be explained by looking at the variance in this mechanism's performance.Figure 9 is a graph of its performance compared to the other mechanisms for the complex payload structure configuration.The significant variance in the performance of Secure Conversation affects the average roundtrip response, allowing the convergence test to be met sooner than expected.The only appreciable difference in convergence rate between the security mechanisms is with the No Security option.There may be some variance between the different mechanisms.However the startup costs associated with each mechanism (Section 4.1.2)overwhelm the effects of any observable differences in convergence rate.

•
There is strong evidence of significant noise or some other effect that prevents Secure Conversation from converging at a predictable rate relative to the other mechanisms.
• Overall, it is difficult to take the results on rate of convergence as a strong recommendation about a security mechanism, due to an inability to measure the values precisely; therefore this generalized data should be used sparingly.If used at all, convergence rate should be considered only in simple payload structure cases.In complex payload structure contexts, this data suggests that it has even less influence.

The Cost of Security: Message Size
The second focus of the experimentation was message size.Our hypothesis was that adding security to messages will increase their size.Further, if adding security does increase message size, we wanted to determine how much of an increase to predict for each mechanism.
We extracted message sizes associated with each security mechanism from roundtrip response times.We assessed the impact of security as message payload size increases using the Echo service in a dynamic payload configuration (with payloads increasing to 100kb).As Figure 10 shows, security mechanisms that do not use encryption, such as Sign Only, add a constant amount of size (overhead) to the message, but encryption-enabled mechanisms add message overhead at a linear rate with payload size increase.In addition to identifying a trend across increasing payload sizes, it is instructive to compare the base overhead sizes of each mechanism.This information is presented in  In Table 6, the raw payload size data in is the message, and the request size is the message placed in a SOAP envelope.Putting the message in the SOAP envelope has a significant size cost that increases as security mechanisms are added.Further, the increased request size correlates with the response time for a mechanism, showing that cost increases with complexity.One additional result worth noting from this experiment is that Encrypt Only outperforms Sign Only, where the message with request size is 4kb or less.

•
As with roundtrip response time, the Password Only security mechanism adds the least message overhead.Also, as with roundtrip response time, this mechanism is not recommended except in environments that have low message-level security requirements.
• Where it is important to limit message overhead while using a non-trivial security mechanism, Encrypt Only offers the best performance up to payload sizes of roughly 4k.Past this size, Sign Only should be used to avoid the steadily increasing costs of using encryption.

The Cost of Security: System Resources
The third focus of the experiments was the cost of security in terms of system resources.Our hypothesis was that security mechanisms would require substantial system resources to perform effectively.The goal was to examine the resource requirements of the security mechanisms in order to provide guidance on how to select computing resources for a given security framework.
One observation made early during the experimentation is that security carries a significant resource cost.Even using simple services, we found performance degradation due to memory management issues for all but the most basic message payloads.We addressed the problems for the most part by tuning the JVM with a more effective garbage collection algorithm and with increased memory allocation [9].
The results show that Secure Conversation is more sensitive to the amount of available resources than the other mechanisms.Figure 11 compares the impact of increasing the allocated memory to the test application for this mechanism.Realizing the maximum potential roundtrip time performance of Secure Conversation may depend upon the resources available to the infrastructure.

4
This data was not available for the Secure Conversation mechanism.Future work will gather this data.

Conclusions
• Security mechanisms such as Sign Only and Encrypt Only (e.g., XML Signature and XML Encryption) require significant resources to perform at peak levels.Services and applications intended to implement these mechanisms as security services will likely require more than the default memory allocated to them by a standard Sun Hotspot JVM [13].In order to meet these resource needs, developers should identify expected message payload sizes and structure complexities early in the design process and tune the services to perform optimally on the available hardware.
• Secure Conversation and similar mechanisms will likely require more resources than nonsession-based mechanisms in order to perform at peak levels.Implementations of these mechanisms should be tuned aggressively to ensure that the necessary resources are available to ensure the expected performance.

Summary
There is no simple answer to the critical question of whether the cost of web service security is acceptable or too high.The exact cost of security on the runtime performance of a SOAP-based web service will depend on many factors.A good understanding of the quality expectations in a specific environment, as well as of the cost of security relative to the costs of service execution, network latency, and overhead can be determining factors.Variables such as JVM heap memory, garbage collection algorithm, and network bandwidth contribute to roundtrip response time.Our work shows that there is a sufficient range of alternative security mechanisms to perform an effective tradeoff analysis, once the driving use cases in a specific context have been analyzed.
Selecting the appropriate mechanisms for web service security involves weighing the security benefits of individual mechanisms against their costs.Whether measured in terms of roundtrip response time, message size, or resource usage, security mechanisms have a considerable impact on the performance of SOAP-based web services.Our experiments reveal that adding security to web services considerably increases roundtrip response time and message size and that more complicated security mechanisms require greater system resources.The experiments also establish a baseline from which to explore different security framework contexts and establish general guidelines for selecting appropriate mechanisms.

Future Work
There are many opportunities for further investigation.These include high-level questions raised by the initial experimental results, as well as the effect of additional configuration parameters on the overall performance of secure web services.In this section, we briefly describe the following possible future work in those areas.

Encryption Overhead vs. Message Overhead
Overhead associated with an encryption increases with payload size.In contrast, message overhead stays constant when a message has been authenticated or digitally signed.Further investigation could reveal why this difference occurs.It would also be useful to find whether there is an optimal point at which the overhead associated with encrypting a large message outweighs the message overhead associated with splitting the message into smaller chunks and sending each chunk individually.

Potential Alternative Experiment Configuration Elements
Possible configuration elements that could be analyzed in further experiments include algorithm variation (for encryption and digital signing), simulation of constrained environments, alternative security mechanisms, and alternative security framework infrastructures.

Encryption and Signing Algorithms
The existing experiment setup can be leveraged as a baseline and expanded to include measurement of the performance impact of combinations of the following encryption and signing algorithms: •

Message Size
Security testing can be expanded to include measurement of the performance impact of larger files (1MB+) in order to understand the impact of moving large files such as media (pictures, video, etc.) or programs.

Resource Limitations
Security testing can be expanded to include measurement of the performance impact of the various security implementation alternatives in a resource-constrained environment.

Network Bandwidth
One possible experiment is the simulation of an unreliable network environment to measure the performance impact of various security implementation alternatives under conditions such as lowbandwidth connection, unexpected connection loss, and unexpected packet loss.Possible software that could be used to support these tests includes NistNet [26] and WanEm [27].

CPU Speed / Available Memory
Another possible experiment is the simulation of a reduced-hardware environment from the standpoint of CPU and memory.

JVM Settings
Tuning the JVM environment includes increasing heap size and modifying the garbage collection algorithm to reduce the impact of the garbage collector as message size increases.Refinements like these would be helpful to in determining minimum recommended settings for different mechanism and payload combinations.Additional experimentation can also be performed to see if any other JVM settings impact performance.

Partial Encryption
This alternative includes encrypting part of the message, such as the XML content, but not all of it.

Secure Session Token
This alternative implements the WS-SecureConveration standard and adds a Security Context Token (SCT) to the message.

Double Encryption
This alternative performs encryption twice on a message to provide additional security.

REST-Style Web Services
This alternative examines how REST may be used to improve performance in certain cases [22].

Token by Reference
For constrained devices, token-by-reference (as in SAML) may offer an improvement over passing tokens by value.

Alternative Security Framework Infrastructures
The experiments in this report use the Axis2 SOAP engine with the Rampart module for providing WS-* security implementations [2,1].Alternative SOAP engines such as the IBM WebSphere SOAP engine can be tested to see if there is possible impact by the engine on different security implementations [28].

Glossary of Technical Terms
Axis2 Apache Axis2, a SOAP engine (i.e., a software module that manages the communication of web service messages in SOAP format)

Rampart
Apache Rampart, a software module that works on a SOAP engine (specifically Apache Axis2) to automatically manage WS-Security standards on the services running on the SOAP engine.

WS-SecureConversation
An OASIS web service standard that creates a session token for a set of communications between web services and associates a user-defined set of security standards to that token.The token can be transferred in each message instead of the security standard specifics, reducing security overhead.

WS-Security
An OASIS web service standard for managing security of web service communications

Figure 1 :Figure 2 :Figure 3 : 2 DataFigure 4 :Figure 5 :Figure 6 :Figure 7 :Figure 8 :Figure 9 :Figure 11 :
Figure 1: Basic Implementation Architecture of the Experiments Figure 2: Roundtrip Time per Mechanism, Echo Service/Dynamic Payload Figure 3: Sign Only versus Encrypt Only for Message Payloads up to 20kb, from Figure 2 DataFigure 4: Startup Times per Mechanism/Service, Initial versus Subsequent Communications Figure 5: Comparing Startup Times for Different Payload Sizes for Each Mechanism Figure 6: Total Cost of Sign Only versus Encrypt Only, First 100 Messages Figure 7: Total Cost of Both Sign/Encrypt Mechanisms versus Secure Conversation for Simple Structure Payloads Figure 8: Startup Behavior of each Security Mechanism with Echo Service in Static Payload Configuration Figure 9: Performance of Each Mechanism, Employee Details Service in Static Payload Configuration Figure 10: Message Overhead per Mechanism as Payload Size Increases Figure 11: Secure ConversationPerformance, Default JVM Memory vs. Manually Set 25 MB-50MB Range

Figure 1 :
Figure 1: Basic Implementation Architecture of the Experiments

Figure 2 :
Figure 2: Roundtrip Time per Mechanism, Echo Service/Dynamic Payload

Figure 3 :
Figure 3: Sign Only versus Encrypt Only for Message Payloads up to 20kb, from Figure 2 Data

Figure 4 :
Figure 4: Startup Times per Mechanism/Service, Initial versus Subsequent Communications

Figure 5 :
Figure 5: Comparing Startup Times for Different Payload Sizes for Each Mechanism3

Figure 6 :
Figure 6: Total Cost of Sign Only versus Encrypt Only, First 100 Messages

Figure
Figure 7: Total Cost of Both Sign/Encrypt Mechanisms versus Secure Conversation for Simple Structure Payloads

Figure 8 :
Figure 8: Startup Behavior of each Security Mechanism with Echo Service in Static Payload Configuration

Figure 9 :
Figure 9: Performance of Each Mechanism, Employee Details Service in Static Payload Configuration Conclusions •

Figure 10 :
Figure 10: Message Overhead per Mechanism as Payload Size Increases

Table 1 :
Statistical Summary per Mechanism, Echo Service/Static Payload (ms)

Table 2 :
Statistical Summary per Mechanism, Employee Details/Static Payload (ms)

Table 3 :
Approximate Rates of Increase of Each Security Mechanism, from Figure 2 Data

Table 4 :
Convergence Points for Each Mechanism for Echo Service in Static Payload Configuration

Table 5 :
Convergence Points for Each Mechanism for Employee Details Service in Static Payload Configuration

Table 6 :
Message Size for Echo Service with Static Payload Configuration (Bytes)

Table 8 :
Development Environments

Table 10 :
Mapping of Test Cases to Rampart Samples Used in the Experiments

Table 1 :
Statistical Summary per Mechanism, Echo Service/Static Payload (ms)

Table 2 :
Statistical Summary per Mechanism, Employee Details/Static Payload (ms) Values in Table1and Table2indicate that Sign Only performs better than Encrypt Only for messages with simple payload structure, but Encrypt Only outperforms Sign Only for messages with complex payload structure.Sign Only roundtrip time increases more slowly as payload size increases for the simple payload structure case (for more detail, see Section 4.1.3).

Table 2
, where it was the worst performer.Startup time for

Table 3 :
Approximate Rates of Increase of Each Security Mechanism, from Figure 2 Data

Table 4 :
Convergence Points for Each Mechanism for Echo Service in Static Payload Configuration

Table 5 :
Convergence Points for Each Mechanism for Employee Details Service in Static Payload Configuration

Table 6 :
Message Size for Echo Service with Static Payload Configuration (Bytes)