Introduction
As part of the IDUNN project,another research was carried out on TSTEM, a cognitive platform designed for the efficient collection of cyber threat intelligence (CTI) from open sources. The exponential growth of internet usage has led to an increase in malicious activities, making it essential for organizations to have robust CTI collection mechanisms. TSTEM addresses this need by enhancing the resilience of both Information Technology (IT) and Operational Technology (OT) environments against large-scale cyber-attacks.
About TSTEM
TSTEM (Threat Streaming and Extraction Machine) is a cognitive platform developed to autonomously search, extract, and index Indicators of Compromise (IOCs) from various online sources in real-time. Leveraging a containerized microservice architecture, TSTEM utilizes advanced technologies including Tweepy, Scrapy, Terraform, ELK (Elasticsearch, Logstash, and Kibana), Kafka, and MLOps.
Key Features
- Containerized Microservice Architecture: Ensures scalability, flexibility, and ease of deployment. This architecture allows different components of the platform to operate independently and efficiently.
- Infrastructure as Code (IaC): The provisioning, monitoring, and management of TSTEM are achieved through IaC, which ensures a streamlined and automated infrastructure setup, reducing human error and enhancing reliability.
- Custom Focus Crawlers: Tailored to identify and gather relevant information from diverse online sources, ensuring comprehensive data collection.
- Multi-Level Classification and Extraction: Collected web content undergoes a first-level classification to identify potential IOCs. Relevant content is then passed to a second-level extraction process for further analysis, ensuring precise identification of relevant information with low false positives.
- State-of-the-Art NLP Models: Utilizes advanced Natural Language Processing (NLP) models like BERT and Longformer for classification and entity extraction, enhancing the accuracy and efficiency of the IOC extraction process.
Experimental Results
The experimental results of TSTEM are impressive, demonstrating high accuracy rates exceeding 98% in classification and extraction tasks. The platform achieves this performance within a time frame of less than a minute, highlighting its efficiency in processing large volumes of data in real-time.
Conclusion
TSTEM represents a significant advancement in the field of cyber threat intelligence as part of the IDUNN project. By leveraging cutting-edge technologies and methodologies, TSTEM provides an efficient and effective solution for collecting, processing, and sharing CTI from open sources. This platform not only addresses the technical challenges associated with CTI extraction but also enhances the overall resilience of IT and OT environments against cyber threats.
Authors
- Prasasthy Balasubramanian
- Sadaf Nazari
- Danial Khosh Kholgh
- Alireza Mahmoodi
- Justin Seby
- Panos Kostakos
Publication Details
- Published in: Computers & Security, July 3, 2024
- Keywords: Cyber Security, CTI, NER, Transformer, BERT, Longformer, Classification, Streaming
- Full Citation: Balasubramanian, P., Nazari, S., Khosh Kholgh, D., Mahmoodi, A., Seby, J., & Kostakos, P. (2024). TSTEM: A Cognitive Platform for Collecting Cyber Threat Intelligence in the Wild. Computers & Security, Vol. 142, p. 103885. Elsevier BV. DOI: 10.1016/j.cose.2024.103885
For more detailed information, you can access the full publication here. Stay tuned for more updates and advancements in cybersecurity from the IDUNN project and the Center for Ubiquitous Computing at the University of Oulu.