D4.2 AI-driven Search Engine API for dark, deep and clear web

This report presents an AI-driven Search Engine API for dark, deep and clear web, a cutting-edge data gathering platform within the IDUNN project. Utilizing Infrastructure as Code (IaC), the platform is designed to advance data scraping, crawling, and searching technologies in order to integrate actionable cyber threat intelligence and Indicators of Compromise (IoC) into cyber defense systems. The platform incorporates three services: 1) THOR-CTI for extracting IoCs using AI and focused crawling, 2) THOR-Hunt as a search interface for threat hunting, and 3) THOR-Analytics for delivering machine learning models to enable data-driven decisions. This automated system is capable of collecting and analyzing unknown and reported malicious network artifacts from diverse sources. The resulting data services can be accessed through an API, enhancing machine-to-machine communication and facilitating human examination via two developed Graphical User Interface (GUI) interfaces. The report aims to provide a comprehensive characterization of the architectural and software aspects of these components while detailing their deployment and functionalities in cyber threat intelligence tasks.