1. the complete title of one (or more) paper(s) published in the open literature describing the work that the author claims describes a human-competitive result; Tong, 2. the name, complete physical mailing address, e-mail address, and phone number of EACH author of EACH paper(s); Authors: a) Fei Liu, fliu36-c@my.cityu.edu.hk, +852 60440852 b) Xialiang Tong, tongxialiang@huawei.com, +86 13428727227 c) Mingxuan Yuan, yuan.mingxuan@huawei.com, +86 13066860406 d) Xi Lin, xi.lin@my.cityu.edu.hk, +852 68410459 e) Fu Luo, luof2023@mail.sustech.edu.cn, +86 18378775363 f) Zhenkun Wang, wangzk3@sustech.edu.cn, +86 15332445564 g) Zhichao Lu, luzhichaocn@gmail.com, +86 13925130907 h) Qingfu Zhang, qingfu.zhang@cityu.edu.hk, +852 97895524 Address for authors a), d), g), and h): Department of Computer Science City University of Hong Kong 83 Tat Chee Avenue Kowloon Hong Kong Address for authors b): HUAWEI Industrial Base Bantian Longgang District Shenzhen, China Address for authors c): Huawei Hong Kong Research Center Science Park W Ave Science Park Hong Kong Address for authors e) and f): The School of System Design and Intelligent Manufacturing Southern University of Science and Technology No. 1088, Xueyuan Avenue Nanshan District Shenzhen, China 3. the name of the corresponding author (i.e., the author to whom notices will be sent concerning the competition); Qingfu Zhang qingfu.zhang@cityu.edu.hk 4. the abstract of the paper(s); Heuristics are widely used for dealing with complex search and optimization problems. However, manual design of heuristics can be often very labour extensive and requires rich working experience and knowledge. This paper proposes Evolution of Heuristic (EoH), a novel evolutionary paradigm that leverages both Large Language Models (LLMs) and Evolutionary Computation (EC) methods for Automatic Heuristic Design (AHD). EoH represents the ideas of heuristics in natural language, termed thoughts. They are then translated into executable codes by LLMs. The evolution of both thoughts and codes in an evolutionary search framework makes it very effective and efficient for generating high-performance heuristics. Experiments on three widely studied combinatorial optimization benchmark problems demonstrate that EoH outperforms commonly used handcrafted heuristics and other recent AHD methods including FunSearch. Particularly, the heuristic produced by EoH with a low computational budget (in terms of the number of queries to LLMs) significantly outperforms widely-used human hand-crafted baseline algorithms for the online bin packing problem. 5. a list containing one or more of the eight letters (A, B, C, D, E, F, G, or H) that correspond to the criteria (see above) that the author claims that the work satisfies; (D) The result is publishable in its own right as a new scientific result independent of the fact that the result was mechanically created. (E) The result is equal to or better than the most recent human-created solution to a long-standing problem for which there has been a succession of increasingly better human-created solutions. (F) The result is equal to or better than a result that was considered an achievement in its field at the time it was first discovered. (G) The result solves a problem of indisputable difficulty in its field. 6. a statement stating why the result satisfies the criteria that the contestant claims (see examples of statements of human-competitiveness as a guide to aid in constructing this part of the submission); (D) The result is publishable in its own right as a new scientific result independent of the fact that the result was mechanically created. EoH combines evolutionary algorithms and Large Language Models (LLMs) and proposes a new approach towards automatic algorithm design. In contrast to classic evolutionary algorithms, EoH does "evolution of thoughts". The LLMs are used to create a linguistic description of the idea (thought) and its detailed code implementation. With LLMs, crossover and mutation operators can generate new and novel algorithms. We believe that the "evolution of thoughts" should be an important research direction. (E) The result is equal to or better than the most recent human-created solution to a long-standing problem for which there has been a succession of increasingly better human-created solutions. The resultant heuristics outperform human hand-crafted heuristics as well as deep-learning-based heuristics on three well-known optimization problems. For example, on flow shop scheduling problem, it outperforms human-designed heuristic methods including CDS[1], GUPTA[2], NEH[3], NEHFF[4], and PFSPNet[5] and PFSPNet_NEH[5]. [1] Campbell, H. G., Dudek, R. A., and Smith, M. L. A heuristic algorithm for the n job, m machine sequencing problem. Management science, 16(10):B–630, 1970. [2] Gupta, J. N. A functional heuristic algorithm for the flow-shop scheduling problem. Journal of the Operational Research Society, 22:39–47, 1971. [3] Nawaz, M., Enscore Jr, E. E., and Ham, I. A heuristic algorithm for the m-machine, n-job flow-shop sequencing problem. Omega, 11(1):91–95, 1983. [4] Fernandez-Viagas, V. and Framinan, J. M. On insertion tie-breaking rules in heuristics for the permutation flowshop scheduling problem. Computers & Operations Research, 45:60–67, 2014. [5] Pan, Z., Wang, L., et al. Deep reinforcement learning based optimization algorithm for permutation flow-shop scheduling. IEEE Transactions on Emerging Topics in Computational Intelligence, 2021. (F) The result is equal to or better than a result that was considered an achievement in its field at the time it was first discovered. EoH generates a heuristic on online bin packing problem and outperforms the very recent results published on: [1] Romera-Paredes, B., Barekatain, M., et al. Mathematical discoveries from program search with large language models. Nature, 625 (7995):468–475, 2024. Our work is developed independently of [1]. (G) The result solves a problem of indisputable difficulty in its field. The objective of EoH is to do automatic algorithm design, which is a very difficult and important research issue. Our work proposes to evolve both "thoughts" and "codes", and design several efficient genetic operators for evolution of thoughts and codes. As the PC chair of ICML comments: “EoH [our method] presents a strong starting point for subsequent algorithm design". 7. a full citation of the paper (that is, author names; title, publication date; name of journal, conference, or book in which article appeared; name of editors, if applicable, of the journal or edited book; publisher name; publisher city; page numbers, if applicable); Fei Liu, Xialiang Tong, Mingxuan Yuan, Xi Lin, Fu Luo, Zhenkun Wang, Zhichao Lu, and Qingfu Zhang. "Evolution of Heuristics: Towards Efficient Automatic Algorithm Design Using Large Language Model". To appear in ICML, 2024. arXiv preprint arXiv:2401.02051 8. a statement either that "any prize money, if any, is to be divided equally among the co-authors" OR a specific percentage breakdown as to how the prize money, if any, is to be divided among the co-authors; Prize money, if any, will be divided equally among those co-authors who wish to receive an equal share. 9. a statement stating why the authors expect that their entry would be the "best"; We expect that our entry would be the "best" due to the groundbreaking paradigm introduced in the paper, Evolution of Heuristic (EoH). EoH combines Large Language Models (LLMs) and Evolutionary Computation (EC) to Automate Algorithm Design (AAD), shifting from the traditional evolution of solutions to a more intelligent evolution of thoughts, leading to a new era in automatic algorithm design. As recommended by the Program Chairs of ICML comments: “EoH [our method] presents a strong starting point for subsequent algorithm design using LLMs and is of great importance to the community and hence deserves wider dissemination". We posit that the evolution of thought facilitated by EC and LLM will become a standard approach in the future, not just for algorithm design but also for various decision-making tasks, such as scientific discoveries and engineering challenges. This work represents the initial stride toward this future direction. By integrating thought and code evolution in an evolutionary framework, EoH delivers superior performance in AAD while reducing computational costs and minimizing human effort. As evidenced by comprehensive evaluations on optimization benchmarks, EoH outperforms existing AAD methods as well as human-designed heuristics. better heuristics can be automatically designed without human interaction and model training. Furthermore, the focus on reproducibility and accessibility, with the availability of the source code, adds to the credibility and potential impact to the community. 10. An indication of the general type of genetic or evolutionary computation used, such as GA (genetic algorithms), GP (genetic programming), ES (evolution strategies), EP (evolutionary programming), LCS (learning classifier systems), GI (genetic improvement), GE (grammatical evolution), GEP (gene expression programming), DE (differential evolution), etc. Genetic Algorithms 11. The date of publication of each paper. If the date of publication is not on or before the deadline for submission, but instead, the paper has been unconditionally accepted for publication and is “in press” by the deadline for this competition, the entry must include a copy of the documentation establishing that the paper meets the "in press" requirement. The publication Evolution of Heuristics: Towards Efficient Automatic Algorithm Design Using Large Language Model has been unconditionally accepted at 41st International Conference on Machine Learning (ICML) 2024, which will be held in Vienna, Austria, July 21st - 27th. You can verify the acceptance at: https://urldefense.com/v3/__https://icml.cc/virtual/2024/papers.html?filter=titles__;!!KjDnqvtInNPT!nYNVjGOQ9grkb_GXMa9w9S36qigvzX2Iy-NmZSZ1wL2jI5eZ9s_9NEDVyiI0Loa7uuDqQWMY-V_mGf9jPhIWddlLP6oC_H4F0tA$ Following is a copy of the acceptance notification: Dear Fei Liu, We are happy to notify you that your submission Evolution of Heuristics: Towards Efficient Automatic Algorithm Design Using Large Language Model (https://urldefense.com/v3/__https://openreview.net/forum?id=BwAkaxqiLB__;!!KjDnqvtInNPT!nYNVjGOQ9grkb_GXMa9w9S36qigvzX2Iy-NmZSZ1wL2jI5eZ9s_9NEDVyiI0Loa7uuDqQWMY-V_mGf9jPhIWddlLP6oCxj-TgB0$ ), is accepted at ICML 2024. To access your reviews, please log in to your author console at https://urldefense.com/v3/__https://openreview.net/group?id=ICML.cc*2024*Conference*Authors__;Ly8v!!KjDnqvtInNPT!nYNVjGOQ9grkb_GXMa9w9S36qigvzX2Iy-NmZSZ1wL2jI5eZ9s_9NEDVyiI0Loa7uuDqQWMY-V_mGf9jPhIWddlLP6oCm0QV2Bo$ . This year, ICML received 9,473 submissions (not including desk rejected papers), an increase of 44% from last year. Among these, we have accepted 2,609 submissions for presentation at the conference, an acceptance rate of 27.5%. These numbers include 286 position paper submissions, out of which 75 were accepted. Submissions were reviewed by at least three reviewers, an area chair, and a senior area chair, in order to ensure each submission was assessed properly; in the rare cases where a paper ultimately received only two reviews, ACs provided a detailed reading of the paper as well. Note that while some of the meta-reviews mention recommending the paper for a talk versus a poster, no final decisions have yet been made on the assignment of papers to presentations as a poster alone or as an oral presentation. These will be made in the coming weeks, subject to availability of space and timing at the venue. All accepted papers will receive an email in the next few weeks designating the paper as a poster or additional oral presentation. ICML will take place in Vienna, Austria on July 21-27 and will be an in-person conference. To accommodate this format, here is some important information: Every paper will be presented as a poster at one of the poster sessions during the main conference. Every paper will be given an opportunity to record and make available a short video presentation of the paper. This is encouraged but not required. Papers eventually designated for oral presentation will receive further instructions on the presentation format. Please update your paper as appropriate based on the review process. The camera ready deadline is May 29 (11:59pm AoE). Additional instructions and camera-ready requirements will be provided soon. Congratulations, and thank you for submitting your work to ICML. We are looking forward to seeing you at the conference! Sincerely, ICML 2024 Program Chairs Please note that responding to this email will direct your reply to program-chairs@icml.cc.