Skip to content

Latest commit

 

History

History
175 lines (124 loc) · 8.82 KB

README.md

File metadata and controls

175 lines (124 loc) · 8.82 KB
Raw

Logo

CRAWNET

CRAWNET is a graph-based domain discovery tool that helps you gather information about domains and potential relationships with other actors. With the power of graph databases, we can find and display quickly information and relationship with other actors, to identify threats before they happen and much more. This tool is perfect for digital forensic, red teaming, penetration testing and cybersecurity researchers.
Explore the docs »

Report Bug · Request Feature · Report Vulnerability

Table of Contents

Installation

To install the project and its dependencies, follow these steps:

Make sure you have docker installed!

  1. Clone the repository to your local machine:
git clone https://github.com/cryxnet/crawnet.git
  1. Go to the directory
cd crawnet
  1. Rename (to .env) and edit the configuration file
mv sample.env .env
&& sed -i 's/NEO4J_PASSWORD=CHANGEME/NEO4J_PASSWORD=your_password_here/' .env \
&& sed -i 's/FLASK_APP_URL=http:\/\/CHANGEME:5000/FLASK_APP_URL=http:\/\/your_machine_ip_or_localhost:5000/' .env \
&& sed -i 's/FLASK_DEBUG=1/FLASK_DEBUG=0/' .env
  1. Start the docker stack
docker compose -f docker-compose.prod.yaml up

Roadmap

v1

  • The overseen service (the api) can collect data from various open source intelligence services.
  • The overseen service (the api) can create entity nodes and relationships in neo4j.
  • The user can easily interact with the api
  • The user can submit a new domain to discover via an formular
  • The user has an interactive way to see the nodes and their relationships on the dashboard
  • First Version Release (09.05.2023)

v2 (Development begins Q3 2023)

  • Collectors Handling (Easy to add new intelligence and entity nodes)
  • Relationship detection & connection engine (Database engine that automatically connects nodes if they have anything in common together)
  • Threat Intelligence Data
  • Better UI/UX

Intelligence Data

The Intelligence Service is using the following sources:

Subdomains

To gather subdomains from a domain we use technique like google dorks and certifcate fingerprinting.

Architecture

architecture

  • For the dashboard we use Next.js with the MUI, vis-data and vis-network libraries (we also use other libraries). vis-data and vis-network is needed to display the nodes and their relationships.
  • For the API we use the Flask framework for python.
  • For the database we use the graph database Neo4j

Testing

  • We have a pipeline that is automatically testing the intergration between the 3 services and source code for any vulnerabilities
Test Case Expected Outcome Outcome
Reach Website Response status code is 200 and "Success" is printed to console Integration Test
Reach API Response status code is 200 or 404 and "Success" is printed to console Integration Test
Reach Neo4j Neo4j connection is successful and "Success" is printed to console Integration Test
API-Website Communication Response status code is 200 and "Success" is printed to console Integration Test
API-Neo4j Communication Response JSON is {'nodes': [], 'relationships': []} and "Success" is printed to console Integration Test
Vulnerable Python Code Nothing Found Actions Status
Vulnerable TS Code Nothing Found Actions Status

Local Testing

python tests/intergration.py

Pipelines

Security Policy

  • Please take a moment to review and familiarize yourself with our security policy.
  • We encourage you to report any issues, bugs, or vulnerabilities that you encounter while using our service.
  • Your help in identifying and reporting any security issues or vulnerabilities is greatly appreciated. Please refer to our security policy for guidance.

Hosting in the Cloud

A short guide if you wan't to deploy this tool for your team in the cloud. This are the minimum requirements.

Cloud Provider Instance Type vCPUs RAM Storage Cost (ca. per Hour)
AWS t4g.medium 2 4GB 20GB $0.0336
Azure A2 v2 2 4GB 20GB $0.136

Disclaimer

YOUR USAGE OF THIS PROJECT CONSTITUTES YOUR AGREEMENT TO THE FOLLOWING TERMS:

THE MISUSE OF THE DATA PROVIDED BY THIS PROJECT MAY LEAD TO CRIMINAL CHARGES AGAINST THE PERSONS CONCERNED.

I DO NOT TAKE ANY RESPONSIBILITY FOR THE CASE. USE THIS PROJECT ONLY FOR RESEARCH PURPOSES, EDUCATIONAL PURPOSES & ETHICAL ONLY.

CRAWNET is a project related to Computer Security and for Educational Purposes and not a project that promotes illegal activities.

Don't use this Project for any illegal activities.

If something happens, we do not take any liability.

CRAWNET should be considered as a project for educational purposes.

Author

Created by cryxnet

If you find this project helpful, please give it a ⭐️ on GitHub to show your support. I would also appreciate it if you shared it with others who might find it useful!