(1) the complete title of one (or more) paper(s) published in the open literature describing the work that the author claims describes a human-competitive result Evolutionary Testing of Stateful Systems: a Holistic Approach (2) the name, complete physical mailing address, e-mail address, and phone number of EACH author of EACH paper, Matteo Miraz Dipartimento di Elettronica e Informazione, Politecnico di Milano piazza Leonardo da Vinci, 32 20133 - Milano (Italy) email: miraz@elet.polimi.it phone: +39 02 2399 3770 (3) the name of the corresponding author (i.e., the author to whom notices will be sent concerning the competition), Matteo Miraz - miraz@elet.polimi.it (4) the abstract of the paper(s), Testing should be one of the key activities of every software development process. However it requires up to half of the software development effort when it is properly done. One of the main problems is the generation of "smart" tests to probe the system, which is both difficult and time-consuming. The research community has been proposing several ways to automate the generation of these tests; among them, the search-based techniques recently achieved significant results. This doctoral dissertation presents TestFul, our evolutionary testing approach for stateful systems; it is tailored to work on object-oriented systems. It uses a holistic approach to make the state of object evolve, to enable all the features the class provides, and to generate the shortest test with the utmost coverage for the class under test. We employ several complementary coverage criteria to drive the evolutionary search. We aim to generate tests with high fault detection effectiveness. To this end, we consider the system from complementary perspectives and we combine white-box analysis techniques with black-box ones. The evolutionary search is completed with a local one, and we establish a synergic cooperation between them. The evolutionary search concentrates on evolving the state of objects, while the local search detects the functionality not yet exercised, and directly targets them. All the proposal were subject to an extensive empirical validation. We devised a benchmark composed of independent benchmarks for tests, public libraries, and third party studies. As comparison, we consider both search-based, symbolic, and traditional (i.e., manually generated by human being) approaches. The achieved results were encouraging: TestFul efficiently generate tests for complex classes and outperforms the other approaches. The proposals presented in this dissertation open new interesting research directions. On one side, one can continue refining the search strategy, by considering more advanced search techniques and by leveraging more advanced coverage criteria. On the other side, one can adapt the approach to work either at a coarse-grained level -and focus on the integration testing- or on other kind of stateful systems (e.g., components or services). (5) a list containing one or more of the eight letters (A, B, C, D, E, F, G, or H) that correspond to the criteria (see above) that the author claims that the work satisfies, (B) The result is equal to or better than a result that was accepted as a new scientific result at the time when it was published in a peer-reviewed scientific journal. (C) The result is equal to or better than a result that was placed into a database or archive of results maintained by an internationally recognized panel of scientific experts. (H) The result holds its own or wins a regulated competition involving human contestants (in the form of either live human players or human-written computer programs). (6) a statement stating why the result satisfies the criteria that the contestant claims (see examples of statements of human-competitiveness as a guide to aid in constructing this part of the submission), In [1] the authors leverage a group of students to compare two well known techniques to (manually) generate tests. The authors perform a careful empirical study, which includes an extensive training on the base principles of testing and an evaluated session where the students are asked to generate tests. The quality of these tests is measured by means of "mutation analysis", and it is published in [1]. We applied our evolutionary technique on the same benchmark used in [1], and we measured the quality of tests by means of the same technique. The results show how the tests generated by TestFul are equal or better than the one generated by the students. Moreover, we executed a similar experiment in our own Software Engineering class, and we experienced a similar results. Finally, we run TestFul on some classes of the Sun's Java Collection Framework, and we found a bug which is reported in the sun's official bug database. Interestingly, other approaches to automatically generate tests were not able to find such error, albeit "container classes are the de facto benchmark for testing software with internal state" [Arcuri]. Referring to the eight criteria for establishing that an automatically created result is competitive with a human-produced result, our approach to create unit tests for java classes by means of evolutionary algorithms satisfies the following three of the eight criteria: (B) The result is equal to or better than a result that was accepted as a new scientific result at the time when it was published in a peer-reviewed scientific journal. (C) The result is equal to or better than a result that was placed into a database or archive of results maintained by an internationally recognized panel of scientific experts. (H) The result holds its own or wins a regulated competition involving human contestants (in the form of either live human players or human-written computer programs). [1] Samar Mouchawrab, Lionel C. Briand, Yvan Labiche, Massimiliano Di Penta: Assessing, Comparing, and Combining State Machine-Based Testing and Structural Testing: A Series of Experiments. IEEE Trans. Software Eng. (TSE) 37(2):161-187 (2011) (7) a full citation of the paper (that is, author names; publication date; name of journal, conference, technical report, thesis, book, or book chapter; name of editors, if applicable, of the journal or edited book; publisher name; publisher city; page numbers, if applicable); Matteo Miraz; 2010; Evolutionary Testing of Stateful Systems: a Holistic Approach; Ph.D. thesis; Politecnico di Milano (8) a statement either that “any prize money, if any, is to be divided equally among the co-authors” OR a specific percentage breakdown as to how the prize money, if any, is to be divided among the co-authors; Matteo Miraz will receive the entire prize money (if any) (9) a statement stating why the judges should consider the entry as “best” in comparison to other entries that may also be “human-competitive.” Software testing should be a fundamental activity of any software development process: even if a systematic testing activity does not guarantee error-freeness, a good and sound campaign would increase the quality of the product. Tassey estimated in $20 billion the amount of money that could be saved every year if better testing were performed. Beizer in 1990 estimated that an adequate testing campaign might require up to half of the software development effort. To alleviate this burden, the research community have proposed several ways to (semi-)automatically generate test data. The approaches presented so far "face challenges in generating test inputs to achieve high structural coverage when they are applied on complex programs in practice" [2]. Considering these circumstances, the results achieved by TestFul are even more important. Instead of relying on traditional techniques of Software Engineers (e.g., SAT solvers), it maps the generation of test-input data to a search problem, which is addressed by means of an evolutionary algorithm. [2] Xusheng Xiao, Tao Xie, Nikolai Tillmann, and Jonathan de Halleux, Precise Identification of Problems for Structural Test Generation, ICSE 2011