Skip to content

Commit

Permalink
Improved README file.
Browse files Browse the repository at this point in the history
  • Loading branch information
@dreibh
dreibh committed Dec 19, 2024
1 parent 226e5af commit 73c2ce8
Showing 1 changed file with 134 additions and 64 deletions.
198 changes: 134 additions & 64 deletions README.md
View file
Original file line number Diff line number Diff line change
@@ -1,15 +1,16 @@
High-Performance Connectivity Tracer (HiPerConTracer) is a Ping/Traceroute measurement framework. It performs regular Ping and Traceroute runs among sites and can import the results into an SQL or NoSQL database. HiPerConTracer currently offers runs over ICMP and UDP.
The HiPerConTracer framework furthermore provides additional tools for helping to process the collected data:

* HiPerConTracer Trigger Tool for triggering HiPerConTracer measurements in the reverse direction.
* HiPerConTracer Sync Tool for copying data to a remote collector (via [RSync](https://rsync.samba.org/)/[SSH](https://www.openssh.com/)).
* HiPerConTracer Viewer Tool for displaying the contents of data files.
* HiPerConTracer UDP Echo Server as UDP Echo ([RFC 862](https://datatracker.ietf.org/doc/html/rfc862)) protocol endpoint.
* HiPerConTracer Importer Tool for storing measurement data from data files into SQL or NoSQL databases. Currently, database backends for [MariaDB](https://mariadb.com/)/[MySQL](https://www.mysql.com/), [PostgreSQL](https://www.postgresql.org/) and [MongoDB](https://www.mongodb.com/) are provided.
* HiPerConTracer Query Tool for querying data from a database and storing it into a data file.
* HiPerConTracer Results Tool for merging and converting data files, e.g. to create a Comma-Separated Value (CSV) file.
* HiPerConTracer Database Shell as simple command-line front-end for the underlying database backends.
* HiPerConTracer Database Tools with some helper programs to e.g. join HiPerConTracer database configurations into an existing DBeaver (SQL database GUI) configuration.
The HiPerConTracer framework furthermore provides additional tools for helping to obtain, process, collect, store, and retrieve measurement data:

* [HiPerConTracer](#run-a-hipercontracer-measurement) is the measurement tool itself.
* [HiPerConTracer Viewer Tool](#the-hipercontracer-viewer-tool) for displaying the contents of data files.
* [HiPerConTracer Results Tool](#the-hipercontracer-results-tool) for merging and converting data files, e.g. to create a Comma-Separated Value (CSV) file.
* [HiPerConTracer Sync Tool]() for copying data to a remote collector (via [RSync](https://rsync.samba.org/)/[SSH](https://www.openssh.com/)).
* [HiPerConTracer Trigger Tool]() for triggering HiPerConTracer measurements in the reverse direction.
* [HiPerConTracer Importer Tool](#the-hipercontracer-importer-tool) for storing measurement data from data files into SQL or NoSQL databases. Currently, database backends for [MariaDB](https://mariadb.com/)/[MySQL](https://www.mysql.com/), [PostgreSQL](https://www.postgresql.org/) and [MongoDB](https://www.mongodb.com/) are provided.
* [HiPerConTracer Query Tool](#the-hipercontracer-query-tool) for querying data from a database and storing it into a data file.
* [HiPerConTracer Database Shell]() as simple command-line front-end for the underlying database backends.
* [HiPerConTracer Database Tools]() with some helper programs to e.g. join HiPerConTracer database configurations into an existing DBeaver (SQL database GUI) configuration.
* [HiPerConTracer UDP Echo Server]() as UDP Echo ([RFC 862](https://datatracker.ietf.org/doc/html/rfc862)) protocol endpoint.

# Build the HiPerConTracer Framework from Sources

Expand All @@ -28,113 +29,182 @@ Notes:

# Run a HiPerConTracer Measurement

HiPerConTracer is the measurement tool itself.

## Example 1
A simply Ping run, without data storage, to any address of [www.heise.de](https://www.heise.de) via ICMP (default):
A simply Ping run, without data storage, from arbitrary local addresses, to all IPv4 and IPv6 addresses of [www.heise.de](https://www.heise.de) via ICMP (default):
```
sudo hipercontracer --destination www.heise.de --ping --verbose
```

## Example 2
Run HiPerConTracer measurement #1000, from arbitrary local IPv4 address to destination 193.99.144.80 with Traceroute and Ping. Store data into sub-directory "data" in the current directory; run as current user $USER:
Run HiPerConTracer measurement #1000000, from arbitrary local IPv4 address to destination 193.99.144.80 with Traceroute and Ping. Store data into sub-directory "data" in the current directory; run as current user $USER:
```
sudo ./hipercontracer --user $USER -#1000 --source 0.0.0.0 --destination 193.99.144.80 --traceroute --ping --resultsdirectory data --verbose
sudo hipercontracer --user $USER -#1000000 --source 0.0.0.0 --destination 193.99.144.80 --traceroute --ping --resultsdirectory data --verbose
```
Notes:
- HiPerConTracer expects the sub-directory "data" (provided with "--resultsdirectory") to write the results files to. The directory must be writable for the user $USER!
- See the manpage "hipercontracer" for various options to set Ping and Traceroute intervals, results file lengths, etc.!
- See the manpage "hipercontracer" for various options to set Ping and Traceroute intervals, results file lengths, results file compression, etc.!

## Example 3
Run HiPerConTracer measurement #1001, from arbitrary local IPv4 (0.0.0.0) and IPv6 (::) addresses to destinations 193.99.144.80 and 2a02:2e0:3fe:1001:302:: with Traceroute and Ping via ICMP (default). Store data into sub-directory "data" in the current directory; run as current user $USER:
Run HiPerConTracer measurement #1000001, from arbitrary local IPv4 (0.0.0.0) and IPv6 (::) addresses to destinations 193.99.144.80 and 2a02:2e0:3fe:1001:302:: with Traceroute and Ping via ICMP (default). Store data into sub-directory "data" in the current directory; run as current user $USER:
```
sudo ./hipercontracer --user $USER -#1001 --source 0.0.0.0 --source :: --destination 193.99.144.80 --destination 2a02:2e0:3fe:1001:302:: --traceroute --ping --resultsdirectory data --verbose
sudo hipercontracer --user $USER -#1000001 --source 0.0.0.0 --source :: --destination 193.99.144.80 --destination 2a02:2e0:3fe:1001:302:: --traceroute --ping --resultsdirectory data --verbose
```

## Data Examples

Some simple data file examples (from [src/results-examples](https://github.com/dreibh/hipercontracer/tree/master/src/results-examples)):

- [HiPerConTracer ICMP Ping over IPv4](https://github.com/dreibh/hipercontracer/blob/master/src/results-examples/Ping-ICMP-%231000001-0.0.0.0-20241219T090830.364329-000000001.hpct)
- [HiPerConTracer ICMP Ping over IPv6](https://github.com/dreibh/hipercontracer/blob/master/src/results-examples/Ping-ICMP-%231000001-%3A%3A-20241219T090830.364464-000000001.hpct)
- [HiPerConTracer ICMP Traceroute over IPv4](https://github.com/dreibh/hipercontracer/blob/master/src/results-examples/Traceroute-ICMP-%231000001-0.0.0.0-20241219T090830.364422-000000001.hpct)
- [HiPerConTracer ICMP Traceroute over IPv6](https://github.com/dreibh/hipercontracer/blob/master/src/results-examples/Traceroute-ICMP-%231000001-%3A%3A-20241219T090830.364515-000000001.hpct)

Notes:

- See the the [manpage of "hipercontracer"](https://github.com/dreibh/hipercontracer/blob/master/src/hipercontracer.1) for a detailed description of the results file formats: ```man hipercontracer```
- The HiPerConTracer Viewer Tool can be used to display results files, including uncompressed ones.
- The HiPerConTracer Results Tool can be used to merge and/or convert the results files.

## Further Details
See the the manpage of "hipercontracer" for all options, including a description of the results file formats!


# The HiPerConTracer Viewer Tool

The HiPerConTracer Viewer Tool displays the contents of data files.

## Example
```
hpct-viewer src/results-examples/Traceroute-UDP-#88888888-fdb6:6d27:be73:4::50-20231018T102656.821891-000000001.results.xz
```

## Further Details
See the the manpage "hipercontracer" for all options, including a description of the results file formats!
See the the [manpage of "hpct-viewer"](https://github.com/dreibh/hipercontracer/blob/master/src/hpct-viewer.1) for a detailed description of the available options: ```man hpct-viewer```


# Convert Results into a CSV File
# The HiPerConTracer Results Tool

- find ./data -maxdepth 1 -name "Ping*.hpct.*" | ./hpct-results --input-file-names-from-stdin --separator=, -o ping.csv.gz
- find ./data -maxdepth 1 -name "Traceroute*.hpct.*" | ./hpct-results --input-file-names-from-stdin --separator=, -o traceroute.csv.gz
The HiPerConTracer Results Tool provides merging and converting data files, e.g. to create a Comma-Separated Value (CSV) file.

## Example 1
Merge all files matching the pattern "Ping\*.hpct.\*" into CSV file, with "," as separator:
```
find data -maxdepth 1 -name "Ping*.hpct.*" | hpct-results --input-file-names-from-stdin --separator=, -o ping.csv.gz
```
Hint: You can use extension .gz for GZip, .bz for BZip2, .xz for XZ, or none for uncompressed output into the output CSV file!

See the the manpage "hpct-results" for all options!
## Example 2
Merge all files matching the pattern "Traceroute\*.hpct.\*" into CSV file, with ";" as separator:
```
find data -maxdepth 1 -name "Traceroute*.hpct.*" | hpct-results --input-file-names-from-stdin --separator=; -o traceroute.csv.gz
```

Note: For simple measurements, you can use HiPerConTracer and the Results Tool directly, without using a database!
## Processing Results from a CSV File

See [src/results-examples](https://github.com/dreibh/hipercontracer/blob/master/src/results-examples) for some examples.

# Setting Up a Database
### GNU R

See src/SQL and src/NoSQL for schema, user and permission setups. Use the database of your choice (MariaDB/MySQL, PostgreSQL, MongoDB).
See [src/results-examples/r-install-dependencies](https://github.com/dreibh/hipercontracer/blob/master/src/results-examples/r-install-dependencies) to get the necessary library packages installed!

Hint: All HiPerConTracer tools support TLS setup. It is **strongly** recommended to properly setup TLS for secure access to a database!
- [r-ping-example](https://github.com/dreibh/hipercontracer/blob/master/src/results-examples/r-ping-example) ping.csv
- [r-traceroute-example](https://github.com/dreibh/hipercontracer/blob/master/src/results-examples/r-traceroute-example) traceroute.csv.gz

### LibreOffice (or any similar spreadsheet program)

# Using the Importer to Store Results in a Database
Import CSV file into spreadsheet.

## Write a Configuration File for the Importer
Hints:

See [src/hipercontracer-database.conf](src/hipercontracer-database.conf) for an example. Make sure that the database access details are correct, so that the Importer tool can connect to the right database and has the required permissions! See src/SQL and src/NoSQL for schema, user and permission setups.
- Use English (US) language, to avoid strange number conversions.
- Choose column separator (" ", ",", etc.), if not automatically detected.

Note: Make sure the "data" directory, as well as the directory for "good" imports and the directory for "bad" (i.e. failed) imports are existing and accessible by the user running the importer!
## Further Details
See the the [manpage of "hpct-results"](https://github.com/dreibh/hipercontracer/blob/master/src/hpct-results.1) for a detailed description of the available options: ```man hpct-results```

## Run the Importer

Examples:
- ./hpct-importer ~/testdb-users-mariadb-importer.conf -verbose
- ./hpct-importer ~/testdb-users-mariadb-importer.conf -verbose --quit-when-idle
# Setting Up a Database

Note: If running without "--quit-when-idle" (recommended), the importer keeps running and imports new files as soon as they appear in the results directory. The importer uses INotify!
See [src/SQL](https://github.com/dreibh/hipercontracer/tree/master/src/SQL) and [src/NoSQL](https://github.com/dreibh/hipercontracer/tree/master/src/NoSQL) for schema, user and permission setups. Use the database of your choice (MariaDB/MySQL, PostgreSQL, MongoDB).

See the the manpage "hpct-importer" for all options!
Hint: All HiPerConTracer tools support TLS setup. It is **strongly** recommended to properly setup TLS for secure access to a database! See [src/TestDB](https://github.com/dreibh/hipercontracer/tree/master/src/TestDB) as example; this is the CI test, which includes a full TLS setup with all supported database backends.


# Using the Query Tool to Retrieve Results from a Database
# The HiPerConTracer Importer Tool

## Write a Configuration File for the Query Tool
The HiPerConTracer Importer Tool provides the continuous storage of collected measurement data from data files into SQL or NoSQL databases. Currently, database backends for [MariaDB](https://mariadb.com/)/[MySQL](https://www.mysql.com/), [PostgreSQL](https://www.postgresql.org/) and [MongoDB](https://www.mongodb.com/) are provided.

See [src/hipercontracer-database.conf](src/hipercontracer-database.conf) for an example. Make sure that the database access details are correct, so that the Query tool can connect to the right database and has the required permissions! See src/SQL and src/NoSQL for schema, user and permission setups.
## Write a Configuration File for the Importer

## Run the Query Tool
See [src/hipercontracer-database.conf](src/hipercontracer-database.conf) for an example. Make sure that the database access details are correct, so that the Query tool can connect to the right database and has the required permissions! See [src/SQL](https://github.com/dreibh/hipercontracer/tree/master/src/SQL) and [src/NoSQL](https://github.com/dreibh/hipercontracer/tree/master/src/NoSQL) for schema, user and permission setups.

Examples:
- ./hpct-query ~/testdb-users-mariadb-researcher.conf ping -o ping.hpct.gz
- ./hpct-query ~/testdb-users-mariadb-researcher.conf ping -o ping.hpct.gz --from-measurement-id 1000 --to-measurement-id 1000
- ./hpct-query ~/testdb-users-mariadb-researcher.conf traceroute -o traceroute.hpct.gz --loglevel 0 --from-measurement-id 1000 --to-measurement-id 1000
- ./hpct-query ~/testdb-users-mariadb-researcher.conf traceroute -o traceroute.hpct.gz --verbose --from-time "2023-09-22 00:00:00"
- ./hpct-query ~/testdb-users-mariadb-researcher.conf traceroute -o traceroute.hpct.gz --verbose --from-time "2023-09-22 00:00:00" --to-time "2023-09-23 00:00:00"
Note: Make sure the "data" directory, as well as the directory for "good" imports and the directory for "bad" (i.e. failed) imports are existing and accessible by the user running the importer!

The output is in the same format as the originally written HiPerConTracer results. See the the manpage "hipercontracer" for all options, including a description of the results file formats!
## Run the Importer Tool

Note: Make sure to specify a Measurement ID range, or a time range. Otherwise, the Query tool will export **everything**!
### Example 1
Continuously run, waiting for new files to import:
```
hpct-importer ~/testdb-users-mariadb-importer.conf -verbose
```

Hint: You can use extension .gz for GZip, .bz for BZip2, .xz for XZ, or none for uncompressed output into the output CSV file!
### Example 2
Just run one import round, quit when there are no more files to import:
```
hpct-importer ~/testdb-users-mariadb-importer.conf -verbose --quit-when-idle
```

See the the manpage "hpct-query" for all options!
Note: If running without "--quit-when-idle" (recommended), the importer keeps running and imports new files as soon as they appear in the results directory. The importer uses INotify!

To convert the results to CSV format, use for example:
- ./hpct-results ping.hpct.gz --separator=, --output ping.csv.gz
- ./hpct-results traceroute.hpct.gz --separator=, --output traceroute.csv.gz
## Further Details
See the the [manpage of "hpct-importer"](https://github.com/dreibh/hipercontracer/blob/master/src/hpct-importer.1) for a detailed description of the available options: ```man hpct-importer```


# Examples to Process Results from CSV
# The HiPerConTracer Query Tool

See [src/examples](src/examples) for some examples.
## Write a Configuration File for the Query Tool

## GNU R
See [src/hipercontracer-database.conf](src/hipercontracer-database.conf) for an example. Make sure that the database access details are correct, so that the Query tool can connect to the right database and has the required permissions! See [src/SQL](https://github.com/dreibh/hipercontracer/tree/master/src/SQL) and [src/NoSQL](https://github.com/dreibh/hipercontracer/tree/master/src/NoSQL) for schema, user and permission setups.

See [src/examples/install-packages.R](src/results-examples/install-packages.R) to get the necessary library packages installed!
## Run the Query Tool

- ./r-ping-example ../ping.csv
- ./r-traceroute-example ../traceroute.csv.gz
### Example 1
Export all Ping data to ping.hpct.gz (GZip-compressed data file):
```
hpct-query ~/testdb-users-mariadb-researcher.conf ping -o ping.hpct.gz
```

## LibreOffice (or any similar spreadsheet program)
Notes:

Import CSV file into spreadsheet.
- Make sure to specify a Measurement ID range, or a time range. Otherwise, the Query tool will export **everything**!
- The output is in the same format as the originally written HiPerConTracer results. See the the manpage "hipercontracer" for all options, including a description of the results file formats!
- You can use extension .gz for GZip, .bz for BZip2, .xz for XZ, or none for uncompressed output!

Hints:
- Use English (US) language, to avoid strange number conversions.
- Choose column separator (" ", ",", etc.), if not automatically detected.
### Example 2
Export all Ping data of Measurement ID #1000 to ping.hpct.gz (GZip-compressed data file):
```
hpct-query ~/testdb-users-mariadb-researcher.conf ping -o ping.hpct.gz --from-measurement-id 1000 --to-measurement-id 1000
```

### Example 3
Export all Traceroute data of Measurement ID #1000 to traceroute.hpct.bz2 (BZip2-compressed data file), with verbose logging:
```
hpct-query ~/testdb-users-mariadb-researcher.conf traceroute -o traceroute.hpct.bz2 --loglevel 0 --from-measurement-id 1000 --to-measurement-id 1000
```

### Example 4
Export all Traceroute data from 2023-09-22 00:00:00 to traceroute.hpct.xz (XZ-compressed data file), with verbose logging:
```
hpct-query ~/testdb-users-mariadb-researcher.conf traceroute -o traceroute.hpct.xz --verbose --from-time "2023-09-22 00:00:00"
```

### Example 5
Export all Traceroute data between 2023-09-22 00:00:00 and 2023-09-23 00:00:00 to traceroute.hpct.xz (XZ-compressed data file):
```
hpct-query ~/testdb-users-mariadb-researcher.conf traceroute -o traceroute.hpct.xz --verbose --from-time "2023-09-22 00:00:00" --to-time "2023-09-23 00:00:00"
```

## Further Details
See the the [manpage of "hpct-query"](https://github.com/dreibh/hipercontracer/blob/master/src/hpct-query.1) for a detailed description of the available options: ```man hpct-query```

0 comments on commit 73c2ce8

Please sign in to comment.