Recent performance breakthroughs in artificial intelligence (AI) and machine learning (ML), including advances in deep learning (DL) and the availability of powerful, easy-to-use ML toolboxes, have led to unprecedented interest in AI/ML among nuclear engineers. However, the full potential of AI/ML techniques has yet to be fully realised. One barrier is that existing ML methods often do not meet the needs of nuclear engineering (NE) applications. Application-agnostic algorithms, or those designed for more traditional ML applications such as computer vision and natural language processing, cannot typically be directly applied to scientific data in NE and require non-trivial, task-specific modifications. Furthermore, there is no common AI/ML benchmark or activity to validate and compare different developed ML methods and algorithms. Finally, there are significant gaps in the predictive capability assessment and improvement of ML models. Specifically, to enable more trustworthy applications in high-consequence systems like nuclear reactors that are subject to nuclear safety regulations, the ML practitioners have to consider a few critical unresolved issues, including the verification, validation and uncertainty quantification (VVUQ) of AI/ML, data scarcity, scaling-induced uncertainty, and lack of physics in black-box ML models. All this requires the development of an AI/ML benchmark that will provide guidelines and exercises in various computational domains to help participants to develop and evaluate the performance of their ML methods.
The Task Force (TF) will design benchmark specifications to evaluate the performance of AI/ML in multi-physics modeling and simulation (M&S) of reactor systems, under the guidance of the Expert Group on Reactor Systems Multi-Physics (EGMUP), will provide guidelines to the Working Party on Scientific issues and Uncertainty Analysis of Reactor Systems (WPRS), and the nuclear community on the scientific development needs (data, methods, and benchmark exercises) for trustworthy AI/ML applications to nuclear scientific computing problems. This benchmark will focus on establishing and improving ML predictive capability/credibility, through consistent, comprehensive, and rigorous VVUQ of AI/ML which matches the quality standards for VVUQ of traditional nuclear M&A. Innovative methodologies for unresolved issues such as data scarcity, scaling-induced uncertainty, and lack of physics in black-box ML models will be developed through the benchmark exercises. Finally, the participants will also contribute to enhancing the trustworthiness of AI/ML for nuclear applications in this benchmark, in order to enable wide acceptance of AI/ML by the nuclear regulators, stakeholders, and policy and decision makers. Among the broad concepts incorporated in AI/ML trustworthiness, this benchmark will focus on accuracy, robustness (reproducibility, applicability) and transparency (explainability, interpretability).
This TF will design and supervise the execution of benchmark exercises on both single physics (reactor physics, thermal-hydraulics, fuel performance and structural mechanics) and multi-physics coupled simulation problems. The exercises will target specific challenges of each computational domain. Certified experimental data, as well as verified and validated high-fidelity computational data from other WPRS expert groups will be leveraged as training dataset for benchmark exercises. Both steady-state and time-dependent problems will be included. A broad spectrum of AI/ML sub-domains will be explored, including supervised ML, unsupervised ML, semi-supervised ML, deep generative learning, and probabilistic ML. In each exercise, participants are asked to select AI/ML algorithms to perform tasks specified in the exercises (prediction, generalisation, uncertainty quantification, etc.). To fulfil the above, this TF will provide the following:
Under the guidance of the WPRS, and with contributions of the Expert Group on Physics of Reactor Systems (EGPRS), the Expert Group on Reactor Core Thermal-Hydraulics and Mechanics (EGTHM), and the Expert Group on Reactor Fuel Performance (EGRFP), this TF will promote the use of AI/ML to make impact on solutions of critical issues in nuclear engineering and accelerate research on AI/ML for advanced reactor design and safety/licensing analysis. The TF will develop and supervise benchmark exercises and will establish comprehensive, systematic, and quantifiable VVUQ guidelines and metrics to establish AI/ML predictive capability by building trustworthy ML models with considerations of AI/ML accuracy, robustness (reproducibility, applicability) and transparency (explainability, interpretability). The guidelines will be developed and demonstrated on several benchmark exercises that consider the following:
The AI/ML benchmarks will be organised as exercises based on the specific AI/ML task and each exercise will contain test cases for various computational domains of interest that will span neutronics, thermal-hydraulics, fuel performance and multi-physics:
Challenging problems using real-world measurement data from nuclear industry will also be considered. Data for these problems will be made available at OECD/NEA GitLab repositories and made accessible to the benchmark participants. Participants are encouraged to propose state-of-the-art AI/ML-based solutions to these problems, thus significantly promoting the use of innovative AI/ML to make impact on solutions of critical issues in nuclear industry and accelerate the AI/ML research and development for advanced reactor design and safety/licensing analysis.
The development of the benchmark specifications will be conducted in two phases. The first phase of the benchmark will focus on Exercise I and II while the second phase on Exercise III and IV.
Participants will meet once per month during the first year, and bimonthly in the later phase.
The TF is supervised by the EGMUP, and the co-ordinators report on the TF progress during the EGMUP annual meetings. Short courses on AI/ML training, as well as workshops on the latest advancement of AI/ML for nuclear scientific computing will be organised as part of the WPRS workshops and education activities.
Co-ordinators: Xu WU, Gregory DELIPEI (North Carolina State University, USA)
NEA Secretariat: Oliver BUSS
Participation is open to all NEA member states. Please contact the NEA Secretariat to join the Task Force.