OSINT and Threat Intelligence Solutions
Open-source tools are critical for both internal and external threat intelligence programs.
Valued at only $13.5 billion in 2023, the market for risk and threat intelligence solutions is expected to increase by nearly 221%, to $43.3 billion, by 2033.[1] Increasingly, both governments and businesses need insight into the threats facing their personnel, populations, infrastructures, operations, data, and IT systems. Hence the tremendous increase in spending on risk intelligence solutions: automated systems that search and analyze publicly and commercially available information (PAI/CAI). The threat intelligence gleaned from these volumes of data is often called open-source intelligence (OSINT).
What are PAI and CAI?
Risk intelligence solutions glean insight from searching, collating, and analyzing PAI (data accessible for free) and CAI (data available for a price).
Some sources of PAI are websites (including those hosted on the deep and dark web), social media platforms, message board interactions, online comments, mainstream news media sites, government data, and court records. In a world where people create more than 2.5 quintillion bytes of data daily,[2] there is a massive amount of PAI available for analysis.
CAI includes data produced by market research firms, financial and investment analyses, consumer data, academic articles, geospatial information, intellectual property data, industry newsletters, firmographics, and more.
The threat landscape
Governments and private enterprises understand that both internal and external threats are on the rise.
External threats to a country’s security include terrorism, cyberattacks, attacks against infrastructure, military incursions, foreign interference with elections, drug trafficking, human trafficking, trafficking in counterfeit goods, and disinformation campaigns. Many of these (terrorism, disinformation campaigns, cyberattacks, infrastructure attacks) double as internal threats. Additional internal threats include leaks of classified information, corruption, general crime and violence, and natural disasters.
The private sector is not immune to these and similar dangers. External threats can include cyberattacks, economic uncertainty, regulatory noncompliance, physical threats to employees or the workplace, frivolous lawsuits, supply chain disruptions, and event and venue attacks. Internal business threats are most often posed by employees and former employees, partners, and contractors. Network vulnerabilities and the loss of corporate data — through theft, accident, or inadvertent disclosure — are significant concerns. Additional internal threats include sabotage of IT systems, operations failures, employee violence, and theft of devices.
Developing threat management programs
There’s no silver bullet for spotting, mitigating, and preventing danger. Rather, threat is best managed as part of an organization-wide program.
To manage external threats, organizations should develop a program of threat assessment, prioritization, and prevention/mitigation, considering the likely business impact of each type of threat. These programs should include:
- Geopolitical risk assessment — Identifying the risks associated with wars, terrorism, and tensions among nations
- Cybersecurity measures — Protecting systems, networks, data, and applications from attack
- Supply chain vendor vetting — Minimizing the risk of supply chain disruptions
- Emergency response and business continuity planning — Ensuring the organization can continue operating in times of disaster or unrest
- Regulatory compliance — Avoiding the fines, reputational damage, and other repercussions of failing to comply with regulatory mandates
- Regular security audits and updates — Continually reassessing the organization’s security posture
- Adoption of a risk intelligence platform — Uncovering threats early by scanning PAI and CAI
To protect against internal threats, organizations should develop holistic insider risk management programs. These programs identify insider threats across the organization; assess the impact of these threats on operations and missions; mitigate those threats; and consistently scan the digital landscape for emerging dangers. Holistic risk management typically consists of:
- Policy — Developing cohesive policies for identifying, assessing, and mitigating threats
- Leadership — Empowering strong leadership to engender employee buy-in, and to secure the financing needed for program implementation
- Robust security protocols and access control — Preventing unauthorized access to critical systems and data
- Employee education — Educating employees on the ways they and their colleagues may — intentionally or unintentionally — put their organizations and themselves at risk
- Technology — Deploying OSINT tools to automate and streamline examination of employees’ online behavior and assess threats
Threat intelligence capabilities are critical to both internal and external threat management programs.
Learn the basics
eBook
The Honest Guide to Insider Threat Detection
The value that AI-powered PAI tools can bring to the Department of War for the prevention and mitigation of data leaks and spills.
Data Sheet
Secure Access for Intelligence Operations
How Babel Street Secure Access managed attribution solution protects intelligence agents during online investigations.
Use Case
Using Babel Street to Detect and Mitigate Insider Threat
How Babel Street can help detect insider threat
White Paper
Deploying OSINT Solutions to Understand China’s Political-Military Operations
How OSINT tools can uncover China's actions and intentions
Using a risk intelligence platform
A risk intelligence platform helps private- and public sector entities obtain critical insights around strategic threats, identity risk, and vendor risk. Using targeted keywords and search terms, a risk intelligence platform can rapidly and persistently scan global data sources. The best of these platforms can reach hard-to-access sites on the deep and dark web and within adversarial regions. Searches are conducted in real time, so threat mitigation efforts can begin more quickly.
How does threat intelligence capabilities differ from user activity monitoring?
To improve security, many organizations have deployed user activity monitoring (UAM) software. This software tracks user behavior on employee-owned devices and networks. It can spot instances of unusual network access, which may indicate cyber hacking and other illegal behavior. It can also spot if Hal in R&D is using the email system on his office computer to sell the company’s biotech breakthroughs to a competitor.
You know what UAM can’t do? Detect if Hal is using his home computer to access a dark web marketplace and selling information there.
The tracking capabilities of UAM and related technologies are insufficient for the digital age. They provide information only on the use of enterprise-issued or authorized devices. They cannot track user activity on employee-owned devices. Supplementing the information obtained from UAM systems with AI-powered risk intelligence platforms that actively scan PAI and CAI can close this security gap. Risk intelligence platforms enable organizations to examine personnel’s online behavior regardless of the device employed.
Using OSINT to your advantage
Webinar
The Year of the Dragon? Using OSINT for a Decision Advantage on China
How Babel Street can aid understanding of China's actions.
Data Sheet
Babel Street Insights Helps Stop Drug Trafficking
How Babel Street Insights facilitates the use of publicly available information to fight drug trafficking.
eBook
Improving DoW Data Strategy: 6 Ways Babel Street Can Help
The specific strategic needs of the Department of War and how Babel Street aligns with those needs.
eBook
The Honest Guide to OSINT Citation and Reference Standards
Ebook explaining OSINT citation standards along with examples
How can these capabilities help protect against threats?
Here are just a few use cases from a handful of sectors and geographies.
- Immigration officials can use a threat intelligence capabilities to pre-screen travelers for visas. They can examine social media posts and other content to determine whether an applicant is in any way related to a criminal appearing on a watch list.
- Border security officers can use threat intelligence capabilities to detect and track illegal cross-border activity; monitor the movements of individuals and groups of interest; and begin response planning.
- National security agencies can monitor the social media activity of suspected terrorist organizations.
- Since mass shooters tend to announce their plans online[3], law enforcement can use a threat intelligence application to monitor social media for potential mass shooters in their area.
- Airport security can learn more about the security of their facilities. If an application detects someone tweeting, “Just saw a woman abandon a bag @Liverpool John Lennon Airport, Gate 8,” it can trigger an alert to airport authorities.
- The United States Department of War can search PAI and CAI worldwide to detect words and phrases associated with leaks of classified or sensitive information.
- Law enforcement can scour the deep and dark web to identify drug traffickers, potential human traffickers, and human trafficking victims. They can also analyze PAI and CAI for insights into trafficking patterns, criminal recruitment methods, and recruitment advertising.
- Business executives can use social media monitoring capabilities to investigate the online behavior of current and former employees, contractors, and others suspected of malicious behavior.
- Public health and safety officials can deploy threat intelligence capabilities to determine the scope of natural disasters, and coordinate responses appropriately. People often post about these events, even before calling emergency services. Information gleaned from monitoring these social media posts can inform governments and emergency services about what is happening where.
Using risk intelligence at different stages of the threat lifecycle
Finding, understanding, and acting upon risk intelligence is a multi-stage process, ranging from collection to action and review. A risk intelligence platform plays a significant role in many of these steps.
Collection
The first step in obtaining threat intelligence is data collection. Analysts must gather relevant data — both structured and unstructured — from a huge array of disparate PAI/CAI sources. In-country access to sites worldwide is important. When an analyst accesses a Chinese site, he must see the information provided to Chinese nationals, rather than just the information China presents to the wider world. The best platforms provide these collection capabilities. They then translate data into the user’s language of choice and transform that information into relevant insight.
Processing
For improved threat intelligence, the data must be properly organized and structured. Today’s platforms can categorize information, remove irrelevant data, and otherwise prepare it for analysis.
Analysis
Cutting-edge threat intelligence platforms help in analysis. They not only find information, but they also identify themes and sentiment in data. They map relationships — spotting social, business, and political connections. They can help analysts graph or otherwise visualize data to better understand these connections.
Integration
Technologically advanced platforms combine third party information with an organization’s internal data (including information from databases, network logs, and incident reports) to provide a more complete view of potential threats.
Prioritization
A risk intelligence platform can prioritize threats by severity and likelihood of occurrence.
Dissemination
Sharing intelligence with relevant stakeholders and partners
Action
Implementing measures to combat threats
Iteration
Continuously evaluating the effectiveness of risk intelligence processes to improve future intelligence gathering
What to look for in a threat intelligence platform
There are many risk and threat intelligence platforms on the market. What should you look for in a solution?
To meet the challenge of detecting today’s threats, your platform must be capable of finding, analyzing, and coalescing vast amounts of data. Look for an automated solution that can access all layers of the internet, including the deep and dark web. Choose a platform that can enhance PAI with a large and diverse library of enriched data, originating from a broad array of free and commercially available sources. And don’t forget internal data. A risk intelligence solution should be able to find data wherever it lives in your organization. To accomplish this in a cost-effective manner, you should consider an API-based solution, one that works on top of existing systems to facilitate sharing from one application or data silo to another — avoiding the need to replace or re-tool older systems.
You should also look for:
Why Babel Street?
Babel Street delivers mission-grade risk intelligence to protect nations, empower organizations, and create a safer world. The Babel Street Risk Intelligence Platform sets a new standard for threat detection. It offers all the risk intelligence tools and capabilities discussed in this article. It rapidly and persistently searches PAI and CAI published in more than 200 languages. This data originates from more than a billion top-level domains; the deep-and dark web; and other commercially and publicly available sources.
Babel Street’s AI-powered data analytics capabilities, along with our extensive data library, can enrich data already appearing in government or enterprise databases. We provide graph-powered clarity: Our knowledge graphs help analysts visualize individuals, corporations, owners, subsidiaries, associated entities, and the relationships among them.
Everyone from terrorists to disgruntled employees threaten governments, private enterprises, and other organizations. The Babel Street Risk Intelligence Platform can help you spot and mitigate these threats.
You might also like
Webinar
Securing Your Inner Circle: Mastering Insider Risk Management with Public Data Insights
Data Sheet
Using AI-powered Technology to Improve Human Trafficking Investigations
How Babel Street Insights facilitates the use of publicly available information to fight human trafficking.
Data Sheet
AI-powered Social Media Monitoring for Curbing Gang Violence
How Babel Street Insights facilitates social media monitoring to help track and investigate gang-related crime.
Webinar
Intelligence & Insider Threats: Exploring & Mitigating Risk
Endnotes
1. Global Market Insights, “Threat Intelligence Market Size,” accessed January 2024, https://www.gminsights.com/industry-analysis/threat-intelligence-market
2. Skelly, William, “Turning Quintillion Bytes of Data Into Opportunities,” Datanami, February 2023, https://www.datanami.com/2023/02/16/turning-quintillion-bytes-of-data-into-opportunities/#:~:text=Approximately%202.5%20quintillion%20bytes%20of,and%20opportunity%20of%20organized%20data.
3. Peterson, J., Densley, J., Spaulding, J., & Higgins, S., “How Mass Public Shooters Use Social Media: Exploring Themes and Future Direction,” Social Media + Society, accessed October 2023, https://doi.org/10.1177/20563051231155101
Disclaimer
All names, companies, and incidents portrayed in this document are fictitious. No identification with actual persons (living or deceased), places, companies, and products are intended or should be inferred.
Frequently asked questions
The market was valued at $13.5 billion in 2023 and is expected to grow by nearly 221%, to $43.3 billion, by 2033.