US 20050229043 A1
Systems and methods for evaluating the testing of a computer program wherein a test automator generates code to test the computer program with respect to predetermined testing criteria. A test results analyst reviews test results generated by applying the code to test the computer program. The environments of a test automator and a test analyst are separated, thereby relieving a test results analyst from being required to have knowledge of the code that was used to test the computer program.
1. A system for evaluating tests of a computer program, comprising:
a test automation environment that provides for generation of testing code so that the computer program can be tested with respect to predetermined testing criteria;
a test results analysis environment that provides for review of test results generated by executing the testing code with respect to the computer program;
wherein the test results analysis environment is separate and insulated from the test automation environment;
wherein the generated test results are automatically transferred to the test results analysis environment.
2. The system of
3. The system of
4. The system of
5. The system of
6. The system of
7. The system of
wherein the transferring is automatically transferred either based upon request of a test results analyst or without being requested by a test results analyst.
8. The system of
9. The system of
10. The system of
11. The system of
an automation independent tool for use by the test results analyst;
wherein the automation independent tool insulates the test results analyst from details of the testing coding generated by the test automator to test the computer program.
12. The system of
13. The system of
14. The system of
15. The system of
16. The system of
17. The system of
18. The system of
wherein the test results analyst is not required to know code details associated the computer program in order to perform the analysis of the generated test results.
19. The system of
wherein the test results are provided such that the test results analyst is not required to have knowledge of the test automation code that was used to test the computer program.
20. A data signal that is transmitted using a network, wherein the data signal includes the test results of
21. The method of
22. A method for evaluating tests of a computer program, comprising the steps of:
generating within a test automation environment test automation code for automatically testing a computer program;
transferring automatically to a test results analysis environment the test results generated by executing the test automation code upon the computer program;
receiving the test results analysis within a test results analysis environment without requiring users of the test results analysis environment to know where the test results were stored within the test automation environment and without requiring the users themselves from having to enter into the test automation environment.
23. The method of
24. An end-to-end system for evaluating tests of a computer program, comprising:
means for generating within a test design environment testing criteria for testing the computer program;
means for generating within a test automation environment test automation code for automatically testing a computer program based upon the testing criteria;
means for transferring automatically to a test results analysis environment test results generated by executing the test automation code with respect to the computer program;
means for receiving the test results within the test results analysis environment without requiring a user of the test results analysis environment to know where the test results were stored within the test automation environment and without requiring the user from personally having to enter into the test automation environment;
wherein a test designer and a test results analyst operate in separate and insulated software environments from the test automation environment.
The present invention relates generally to software testing and more particularly to computer systems for the testing of software.
Software testing environments exist for assisting a test automator to generate code to automatically test a target software application with respect to predetermined testing criteria. However, these environments tend to be confusing and otherwise inefficient to operate within for individuals who design test cases and/or for individuals who evaluate test results.
In accordance with the teachings provided herein, software testing systems and methods are disclosed herein for addressing these and other issues. As an example of a system and method, a test automator generates code to test the computer program with respect to predetermined testing criteria, and a test results analyst reviews test results generated by applying the code to test the computer program. The environments of a test automator and a test analyst are separated, thereby relieving a test results analyst from being required to have knowledge of the code or environment that was used to test the computer program.
The separated environments keep the test designer 34 and the test results analyst 44 insulated from the test coding automation implementation details 46. This allows non-programmers to develop and analyze sophisticated, robust automated test cases without programming knowledge as well as use automation independent tools to access the full power of automation code 40 without knowledge of its implementation details. The system also insulates the test automator 38 from application specific details, thereby allowing the test automator 38 to focus on successfully automating the application 32 under test without requiring any application specific knowledge.
A test automator is a testing programming skilled person and is insulated from an application's test details. Within a separated test automator environment 70, the test automator takes the high-level action list 68 and generates code 72 to drive the application 64 under test. For each action specified, the test automator's code 72 can be run independently. A test of the software generates one or more test result files 80 for a test analyst to validate.
Within a separated test analyst environment 82, a test analyst reviews test results 80 for the application 64 under test using a test results automation independent tool which compares the results of each automation run with bench files which have already been validated as correct. The automation tool 64 visually displays this comparison for analysis and validation. In addition to bench results, the automation tool 84 can allow a test analyst to include in the comparison previous runs to see how the test results are evolving. This can be helpful to a test analyst to provide her analysis 86 of whether the software application is getting closer to the bench results.
The test analyst's interface and environment 82 are insulated from the test automation details. As an illustration, the test analyst's interface automatically receives or copies the automation test result files 80 from the test automation environment 70. In this way, the system 60 allows test results to be provided in a separate, file comparison GUI environment, thereby removing the need for a test analyst to perform test analysis within the test automation framework as well as removing the confusion that test analysts may have because they are not familiar with the technical details of the automation environment.
The environment separation between the roles of a test analyst and test automator results in a robust, reusable, easily maintainable, and cost effective testing system 60 in that, organizations comprised mainly of non-programming individuals, whose skill sets are targeted to the expertise required by the application under test, can take advantage of software testing automation. It should be understood that terms such as test designer, test automator, and test results analyst refer to the role or function of a test designer or test automator or test results analyst. As an illustration, one or more individuals can be used to fulfill the role of a test analyst for a particular test run. Moreover, while the separated environments allow for the different roles to operate in an insulated fashion, the same person could fulfill two or more of the roles when utilizing the separated environments.
The test designer determines what testing needs to be performed on the report creation application by deciding to start with a test, called Bar Graph Sales Data Test. This test will validate that the Report Creator application can successfully enable a user to log on, create, view and save a report that contains a bar graph. The test designer decides what test actions need to be taken inside the Report Creator application to accomplish this bar graph test. The test designer inputs their list of actions into a test design automation independent tool which generates a file 102 containing test actions for a test automator to automate. As an example, a test actions file 102 can specify that the logon functionality is to be exercised as well as selecting different data and graphs in order to generate a report. Such a test actions file 102 may resemble the following:
As shown at 104, a test automator, employing any number of tools and technologies, designs and implements software code to automate the test steps that are necessary to make the test actions specified in the Bar Graph Sales Data Test action file above repeatable. Based upon the top-level actions in the test design file, the test automator generates code to perform more detailed steps. For example, the test automator could generate test automation code for the Logon action in order to perform such lower-level steps as:
The test results 106 are provided to a test analyst using a format that the test analyst environment can handle. Graphic-based test results can be provided as JPEG or other format, such as in a PDF format. Text-based results can be provided in a word processing document or rendered in a text document (e.g., “.txt” file). Web-based pages can be provided as HTML files for viewing by a test analysis automation independent tool having web browsing capabilities.
A test analyst reviews at 108 test results produced by the automated Bar Graph Sales Data Test run by interacting with a test analysis automation independent tool. The tool can compare the test results of the latest automated Bar Graph Sales Data Test run with the results previously deemed to be valid, benchmarked results by the same or different test analyst. Because this comparison is surfaced to the test analyst via an automation independent tool, they are insulated from the heart of the automation details implemented by the test automator. The test analyst is not required to have any understanding of how the test results were produced. However, they are typically very knowledgeable in evaluating the results in the context of the resultant file generated by the test designer.
A test design environment 202 may include a test case design tool 210 such as the one 300 shown in
GUI region 306 provides a list of action words and test cases stored in a test repository. This helps ensure that relevant and acceptable action words are used in formulating a test design. The test cases provided in region 306 are previously created testing scenarios. The stored test cases can be retrieved by a test designer so that the test designer can use or modify a stored test case to fit the testing situation at hand.
The actions designated by the test designer through interaction with region 306 are added to region 302. Region 302 provides for manipulation of the actions specified in the region, such as allowing a user to reorder or delete actions in region 302.
With reference back to
Examples of the test automation environment 204 are shown in
Examples of code portions that a test automator might enter into region 406 of
The code of
The automation application map is a lower-level version of the automation engine code. As an illustration, line 460 on
The code contained in the automation application map is executed in order to test the application. As shown in
With reference back to
As an illustration, region 532 may contain benchmark pie chart graphical information 550 as shown in
Another test analyst tool that may be included in a test analyst environment is shown in
As an illustration, region 604 may contain benchmark tabular textual information 610 as shown in
It should be understood that a test results analysis environment may contain other tools, such as an HTML results differencing tool shown in
While examples have been used to disclose the invention, including the best mode, and also to enable any person skilled in the art to make and use the invention, the patentable scope of the invention is defined by claims, and may include other examples that occur to those skilled in the art. For example, the steps and the order of the steps in the process flows described herein may be altered, modified and/or augmented and still achieve the desired outcome.
As another example of the wide variation of the disclosed systems and methods, the systems and methods may be implemented on various types of computer architectures, such as for example on a networked system, or in a client-server configuration, or in an application service provider configuration. In multiple computer systems, data signals may be conveyed via networks (e.g., local area network, wide area network, internet, etc.), fiber optic medium, carrier waves, wireless networks, etc. for communication among multiple computers or computing devices.
The systems' and methods' data disclosed herein, such as generated test results, may be stored as one or more data structures in computer memory and/or storage depending upon the application at hand. Data structures describe formats for use in organizing and storing data in memory or other computer-readable media for use by a computer program. It should be further understood that many different types of software applications can be tested, including those applications that do not have user interfaces. Furthermore, the testing of a software application may examine a wide variety of characteristics, such as user interface characteristics, performance characteristics, etc.
The systems and methods may be provided on many different types of computer-readable media including computer storage mechanisms (e.g., CD-ROM, diskette, RAM, flash memory, computer's hard drive, etc.) that contain instructions for use in execution by a processor to perform the methods' operations and implement the systems described herein.
The computer components, software modules, functions and data structures described herein may be connected directly or indirectly to each other in order to allow the flow of data needed for their operations. It is also noted that a module or processor includes but is not limited to a unit of code that performs a software operation, and can be implemented for example as a subroutine unit of code, or as a software function unit of code, or as an object (as in an object-oriented paradigm), or as an applet, or in a computer script language, or as another type of computer code. The software components and/or functionality may be located on a single computer or distributed across multiple computers depending upon the situation at hand.
It should be understood that as used in the description herein and throughout the claims that follow, the meaning of “a,” “an,” and “the” includes plural reference unless the context clearly dictates otherwise. Also, as used in the description herein and throughout the claims that follow, the meaning of “in” includes “in” and “on” unless the context clearly dictates otherwise. Finally, as used in the description herein and throughout the claims that follow, the meanings of “and” and “or” include both the conjunctive and disjunctive and may be used interchangeably unless the context clearly dictates otherwise; the phrase “exclusive or” may be used to indicate situation where only the disjunctive meaning may apply.