Semantic-Driven Focused Crawling Using LASER and FAISS: A Novel Approach for Threat Detection and Improved Information Retrieval

Focused crawling is a technique used to retrieve topic-specific information from the internet by directing the crawling search towards a particular subset of webpages. This approach can significantly improve the discovery and extraction of relevant cybersecurity content, thereby enhancing the precision, recall, and overall relevancy of the collected data. The current state-of-the-art focused crawlers depend on basic keyword matching and phrase matching, or they utilize complex logic, all while demonstrating only average efficiency in identifying relevant webpages. In this study, the implementation of a semantic-driven focused crawler is carried out, expressly constructed for discovering pages that contain Cyber Threat Intelligence (CTI). The proposed method introduces a holistic and creative approach that seamlessly integrates keyword-based and link-based strategies with the power of LASER embeddings, enabling a more nuanced understanding of web content and boosting the efficiency and effectiveness of focused crawling for web page classification. To measure the accuracy and efficiency of the focused crawler, a transformer-based large language model (LLM) classifier has been trained, functioning as a web page classifier to identify the relevance of the pages that the crawler has processed. The effectiveness of the suggested approach is assessed based on two main metrics: harvest rate and irrelevance ratio. The proposed methodology demonstrates superior performance compared to existing methodologies, achieving an average harvest rate of 0.87 and an irrelevance ratio of 0.13.