Skip to content

Commit

Permalink
[1421][cygnus][doc] Fix titles according to Github rendering
Browse files Browse the repository at this point in the history
  • Loading branch information
Francisco Romero Bueno authored and Francisco Romero Bueno committed Mar 30, 2017
1 parent 13c693d commit 61c1f0b
Show file tree
Hide file tree
Showing 103 changed files with 704 additions and 705 deletions.
12 changes: 6 additions & 6 deletions README.md
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
#Cygnus
# Cygnus
[![License badge](https://img.shields.io/badge/license-AGPL-blue.svg)](https://opensource.org/licenses/AGPL-3.0)
[![Documentation badge](https://readthedocs.org/projects/fiware-cygnus/badge/?version=latest)](http://fiware-cygnus.readthedocs.org/en/latest/?badge=latest)
[![Support badge]( https://img.shields.io/badge/support-sof-yellowgreen.svg)](http://stackoverflow.com/questions/tagged/fiware-cygnus)
Expand All @@ -8,7 +8,7 @@
[![Docker badge](https://img.shields.io/docker/pulls/fiware/cygnus-ngsi.svg)](https://hub.docker.com/r/fiware/cygnus-ngsi/)
[![Docker badge](https://img.shields.io/docker/pulls/fiware/cygnus-twitter.svg)](https://hub.docker.com/r/fiware/cygnus-twitter/)

##Welcome
## Welcome
This project is part of [FIWARE](http://fiware.org), being part of the [Cosmos](http://catalogue.fiware.org/enablers/bigdata-analysis-cosmos) Ecosystem.

Cygnus is a connector in charge of persisting certain sources of data in certain configured third-party storages, creating a historical view of such data.
Expand All @@ -34,12 +34,12 @@ Current stable release is able to persist the following sources of data in the f

**IMPORTANT NOTE**: for the time being, cygnus-ngsi and cygus-twitter agents cannot be installed in the same base path, because of an incompatibility with the required version of the `httpclient` library. Of course, if you are going to use just one of the agents, there is no problem at all.

##Cyngus place in FIWARE architecture
## Cyngus place in FIWARE architecture
Cygnus (more specifically, cygnus-ngsi agent) plays the role of a connector between Orion Context Broker (which is a NGSI source of data) and many FIWARE storages such as CKAN, Cosmos Big Data (Hadoop) and STH Comet. Of course, as previously said, you may add MySQL, Kafka, Carto, etc as other non FIWARE storages to the FIWARE architecture.

![FIWARE architecture](doc/images/fiware_architecture.png)

##Further documentation
## Further documentation
The per agent **Quick Start Guide** found at readthedocs.org provides a good documentation summary ([cygnus-ngsi](http://fiware-cygnus.readthedocs.io/en/latest/cygnus-ngsi/quick_start_guide/index.html), [cygnus-twitter](http://fiware-cygnus.readthedocs.io/en/latest/cygnus-twitter/quick_start_guide/index.html)).

Nevertheless, both the **Installation and Administration Guide** and the **User and Programmer Guide** for each agent also found at [readthedocs.org](http://fiware-cygnus.readthedocs.io/en/latest/) cover more advanced topics.
Expand All @@ -53,8 +53,8 @@ Other interesting links are:
* [cygnus-ngsi](https://edu.fiware.org/mod/resource/view.php?id=1037) **introductory course** in FIWARE Academy.
* The [Contributing Guidelines](doc/contributing/contributing_guidelines.md) if your aim is to extend Cygnus.

##Licensing
## Licensing
Cygnus is licensed under Affero General Public License (GPL) version 3. You can find a [copy of this license in the repository](./LICENSE).

##Reporting issues and contact information
## Reporting issues and contact information
Any doubt you may have, please refer to the [Cygnus Core Team](./reporting_issues_and_contact.md).
24 changes: 12 additions & 12 deletions cygnus-common/README.md
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
#<a name="top"></a>cygnus-common
# <a name="top"></a>cygnus-common
Content:

* [Welcome to cygnus-common](#section1)
Expand All @@ -13,21 +13,21 @@ Content:
* [Features summary](#section4)
* [Reporting issues and contact information](#section5)

#<a name="section1"></a>Welcome to cygnus-common
# <a name="section1"></a>Welcome to cygnus-common
cygnus-common is the base for any Cygnus agent (e.g. cygnus-ngsi). Cygnus agents are based on [Apache Flume](http://flume.apache.org/) agents, which are basically composed of a source in charge of receiving the data, a channel where the source puts the data once it has been transformed into a Flume event, and a sink, which takes Flume events from the channel in order to persist the data within its body into a third-party storage.

cygnus-common provides a set of extensions for Apache Flume, for instance, defining how a Http source handler must look like or adding channels suitable for reading Cygnus-like counters. But not only Flume extensions, but interesting functionality for any agent in terms of a common Management Interface, common backend classes for HDFS, MySQL, MongoDB, PostgreSQL and many others, unified logging classes and error handling, etc.

[Top](#top)

##<a name="section2"></a>Basic operation
###<a name="section2.1"></a>Hardware requirements
## <a name="section2"></a>Basic operation
### <a name="section2.1"></a>Hardware requirements
* RAM: 1 GB, specially if abusing of the batching mechanism.
* HDD: A few GB may be enough unless the channel types are configured as `FileChannel` type.

[Top](#top)

###<a name="section2.2"></a>Installation (CentOS/RedHat)
### <a name="section2.2"></a>Installation (CentOS/RedHat)
Simply configure the FIWARE repository if not yet configured:

$ cat > /etc/yum.repos.d/fiware.repo <<EOL
Expand All @@ -46,12 +46,12 @@ The above will install cygnus-common in `/usr/cygnus/`.

[Top](#top)

###<a name="section2.3"></a>Configuration
### <a name="section2.3"></a>Configuration
Configuring cygnus-common is just configuring Apache Flume since no agent-related functionality is added (that's something agents as cygnus-ngsi do). Please, check [this](https://flume.apache.org/FlumeUserGuide.html#setup) official guidelines.

[Top](#top)

###<a name="section2.4"></a>Running
### <a name="section2.4"></a>Running
cygnus-common can be run as a service by simply typing:

$ service cygnus-common start
Expand All @@ -60,7 +60,7 @@ Logs are written in `/var/log/cygnus/cygnus.log`, and the PID of the process wil

[Top](#top)

###<a name="section2.5"></a>Unit testing
### <a name="section2.5"></a>Unit testing
Running the tests require [Apache Maven](https://maven.apache.org/) installed and cygnus-common sources downloaded.

$ git clone https://github.com/telefonicaid/fiware-cygnus.git
Expand All @@ -69,7 +69,7 @@ Running the tests require [Apache Maven](https://maven.apache.org/) installed an

[Top](#top)

###<a name="section2.6"></a>Management API overview
### <a name="section2.6"></a>Management API overview
Run the following `curl` in order to get the version (assuming cygnus-common runs on `localhost`):

```
Expand All @@ -84,13 +84,13 @@ Many other operations, like getting/putting/updating/deleting the grouping rules

[Top](#top)

##<a name="section3"></a>Further reading
## <a name="section3"></a>Further reading

Further information can be found in the documentation at [fiware-cygnus.readthedocs.io](https://fiware-cygnus.readthedocs.io)

[Top](#top)

##<a name="section4"></a>Features summary
## <a name="section4"></a>Features summary
<table>
<tr><td rowspan="6">Management Interface</td><td>GET /version</td><td>0.5.0</td></tr>
<tr><td>GET /stats</td><td>0.13.0</td></tr>
Expand All @@ -112,7 +112,7 @@ Further information can be found in the documentation at [fiware-cygnus.readthed

[Top](#top)

##<a name="section5"></a>Reporting issues and contact information
## <a name="section5"></a>Reporting issues and contact information
Any doubt you may have, please refer to the [Cygnus Core Team](../reporting_issues_and_contact.md).

[Top](#top)
26 changes: 13 additions & 13 deletions cygnus-ngsi/README.md
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
#<a name="top"></a>Cygnus NGSI
# <a name="top"></a>Cygnus NGSI
Content:

* [Welcome to Cygnus NGSI](#section1)
Expand All @@ -14,7 +14,7 @@ Content:
* [Features summary](#section4)
* [Reporting issues and contact information](#section5)

##<a name="section1"></a>Welcome to Cygnus NGSI
## <a name="section1"></a>Welcome to Cygnus NGSI
Cygnus NGSI is a connector in charge of persisting [Orion](https://github.com/telefonicaid/fiware-orion) context data in certain configured third-party storages, creating a historical view of such data. In other words, Orion only stores the last value regarding an entity's attribute, and if an older value is required then you will have to persist it in other storage, value by value, using Cygnus NGSI.

Cygnus NGSI uses the subscription/notification feature of Orion. A subscription is made in Orion on behalf of Cygnus NGSI, detailing which entities we want to be notified when an update occurs on any of those entities attributes.
Expand All @@ -37,14 +37,14 @@ You may consider to visit [Cygnus NGSI Quick Start Guide](../doc/cygnus-ngsi/qui

[Top](#top)

##<a name="section2"></a>Basic operation
###<a name="section2.1"></a>Hardware requirements
## <a name="section2"></a>Basic operation
### <a name="section2.1"></a>Hardware requirements
* RAM: 1 GB, specially if abusing of the batching mechanism.
* HDD: A few GB may be enough unless the channel types are configured as `FileChannel` type.

[Top](#top)

###<a name="section2.2"></a>Installation (CentOS/RedHat)
### <a name="section2.2"></a>Installation (CentOS/RedHat)
Simply configure the FIWARE repository if not yet configured:

$ cat > /etc/yum.repos.d/fiware.repo <<EOL
Expand All @@ -65,7 +65,7 @@ Please observe, as part of the installation process, cygnus-common is installed

[Top](#top)

###<a name="section2.3"></a>Configuration
### <a name="section2.3"></a>Configuration
Cygnus NGSI is a tool with a high degree of configuration required for properly running it. The reason is the configuration describes the Flume-based agent chosen to be run.

So, the starting point is choosing the internal architecture of the Cygnus NGSI agent. Let's assume the simplest one:
Expand Down Expand Up @@ -121,7 +121,7 @@ POLLING_INTERVAL=30

[Top](#top)

###<a name="section2.4"></a>Running
### <a name="section2.4"></a>Running
Cygnus NGSI can be run as a service by simply typing:

$ (sudo) service cygnus start
Expand All @@ -130,7 +130,7 @@ Logs are written in `/var/log/cygnus/cygnus.log`, and the PID of the process wil

[Top](#top)

###<a name="section2.5"></a>Unit testing
### <a name="section2.5"></a>Unit testing
Running the tests require [Apache Maven](https://maven.apache.org/) installed and Cygnus NGSI sources downloaded.

$ git clone https://github.com/telefonicaid/fiware-cygnus.git
Expand All @@ -139,7 +139,7 @@ Running the tests require [Apache Maven](https://maven.apache.org/) installed an

[Top](#top)

###<a name="section2.6"></a>e2e testing
### <a name="section2.6"></a>e2e testing
Cygnus NGSI works by receiving NGSI-like notifications, which are finally persisted. In order to test this, you can run any of the notification scripts located in the [resources folder](./resources/ngsi-examples) of this repo, which emulate certain notification types.

```
Expand Down Expand Up @@ -167,7 +167,7 @@ Or you can connect a real NGSI source such as [Orion Context Broker](https://git

[Top](#top)

###<a name="section2.7"></a>Management API overview
### <a name="section2.7"></a>Management API overview
Run the following `curl` in order to get the version (assuming Cygnus NGSI runs on `localhost`):

```
Expand Down Expand Up @@ -226,7 +226,7 @@ Many other operations, like getting/putting/updating/deleting the grouping rules

[Top](#top)

##<a name="section3"></a>Advanced topics and further reading
## <a name="section3"></a>Advanced topics and further reading
Detailed information regarding cygus-ngsi can be found in the [Installation and Administration Guide](../doc/cygnus-ngsi/installation_and_administration_guide/introduction.md), the [User and Programmer Guide](../doc/cygnus-ngsi/user_and_programmer_guide/introduction.md) and the [Flume extensions catalogue](../doc/cygnus-ngsi/flume_extensions_catalogue/introduction.md). The following is just a list of shortcuts regarding the most popular topics:

* [Installation with docker](../doc/cygnus-ngsi/installation_and_administration_guide/install_with_docker). An alternative to RPM installation, docker is one of the main options when installing FIWARE components.
Expand All @@ -241,7 +241,7 @@ Detailed information regarding cygus-ngsi can be found in the [Installation and

[Top](#top)

##<a name="section4"></a>Features summary
## <a name="section4"></a>Features summary
<table>
<tr><th>Component</th><th>Feature</th><th>From version</th></tr>
<tr><td rowspan="11">NGSIHDFSSink</td><td>First implementation</td><td>0.1.0</td></tr>
Expand Down Expand Up @@ -303,7 +303,7 @@ Detailed information regarding cygus-ngsi can be found in the [Installation and

[Top](#top)

##<a name="section5"></a>Reporting issues and contact information
## <a name="section5"></a>Reporting issues and contact information
Any doubt you may have, please refer to the [Cygnus Core Team](../doc/cygnus-ngsi/user_and_programmer_guide/issues_and_contact.md).

[Top](#top)
4 changes: 2 additions & 2 deletions cygnus-ngsi/conf/README.md
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
#`cygnus-ngsi` configuration notes
# `cygnus-ngsi` configuration notes
* The `agent_ngsi.conf.template` file is meant for substituting the `agent.conf.template` file installed by `cygnus-common`:

```
Expand All @@ -9,4 +9,4 @@ $ cp agent_ngsi.conf.template [APACHE_FLUME_HOME]/conf/

```
$ cp grouping_rules.conf.template [APACHE_FLUME_HOME]/conf/
``
``
4 changes: 2 additions & 2 deletions cygnus-ngsi/test/e2e/test_environment.md
Original file line number Diff line number Diff line change
@@ -1,10 +1,10 @@
##Introduction
## Introduction

This is a private environment for e2e testing addressed to fiware-cygnus developers. All the information about it is described in a document in Google Docs where we will update all the changes. The document has restricted access (contact us for more information):

* [Information about environment](https://docs.google.com/document/d/1XmoIn6TViEfss2PuBfVZ7SUB-57_S6EAsI4DB79yagA/edit?usp=sharing)

#Reporting issues and contact information
# Reporting issues and contact information
There are several channels suited for reporting issues and asking for doubts in general. Each one depends on the nature of the question:

* Use [stackoverflow.com](http://stackoverflow.com) for specific questions about this software. Typically, these will be related to installation problems, errors and bugs. Development questions when forking the code are welcome as well. Use the `fiware-cygnus` tag.
Expand Down
18 changes: 9 additions & 9 deletions cygnus-twitter/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -10,7 +10,7 @@
* [Licensing](#section6)
* [Reporting issues and contact information](#section7)

##<a name="section1"></a>Welcome to Cygnus-twitter
## <a name="section1"></a>Welcome to Cygnus-twitter
This project is part of [FIWARE](http://fiware.org), being part of the [Cosmos](http://catalogue.fiware.org/enablers/bigdata-analysis-cosmos) Ecosystem.

Cygnus-twitter is a connector in charge of persisting tweets (https://dev.twitter.com/overview/api/tweets) in certain configured third-party storages, creating a historical view of such data.
Expand All @@ -23,14 +23,14 @@ Current stable release is able to persist Twitter data in:

[Top](#top)

##<a name="section2"></a>Basic operation
###<a name="section2.1"></a>Hardware requirements
## <a name="section2"></a>Basic operation
### <a name="section2.1"></a>Hardware requirements
* RAM: 1 GB, specially if abusing of the batching mechanism.
* HDD: A few GB may be enough unless the channel types are configured as `FileChannel` type.

[Top](#top)

###<a name="section2.3"></a>Configuration
### <a name="section2.3"></a>Configuration
Cygnus-twitter is a tool with a high degree of configuration required for properly running it.

So, the starting point is choosing the internal architecture of the Cygnus agent. Let's assume the simplest one:
Expand Down Expand Up @@ -183,7 +183,7 @@ Check the [User and Programmer Guide](../../doc/cygnus-twitter/user_and_programm

[Top](#top)

###<a name="section2.5"></a>Unit testing
### <a name="section2.5"></a>Unit testing
Running the tests require [Apache Maven](https://maven.apache.org/) installed and Cygnus sources downloaded.

$ git clone https://github.com/telefonicaid/fiware-cygnus.git
Expand All @@ -192,7 +192,7 @@ Running the tests require [Apache Maven](https://maven.apache.org/) installed an

[Top](#top)

###<a name="section2.7"></a>Management API overview
### <a name="section2.7"></a>Management API overview
Run the following `curl` in order to get the version (assuming Cygnus runs on `localhost`):

```
Expand Down Expand Up @@ -251,7 +251,7 @@ Many other operations, like getting/putting/updating/deleting the grouping rules

[Top](#top)

##<a name="section5"></a>Features summary
## <a name="section5"></a>Features summary
<table>
<tr><th>Component</th><th>Feature</th><th>From version</th></tr>
<tr><td rowspan="7">TwitterHDFSSink</td><td>First implementation</td><td>1.1.0</td></tr>
Expand All @@ -267,12 +267,12 @@ Many other operations, like getting/putting/updating/deleting the grouping rules

[Top](#top)

##<a name="section6"></a>Licensing
## <a name="section6"></a>Licensing
Cygnus is licensed under Affero General Public License (GPL) version 3. You can find a [copy of this license in the repository](../../LICENSE).

[Top](#top)

##<a name="section7"></a>Reporting issues and contact information
## <a name="section7"></a>Reporting issues and contact information
There are several channels suited for reporting issues and asking for doubts in general. Each one depends on the nature of the question:

* Use [stackoverflow.com](http://stackoverflow.com) for specific questions about this software. Typically, these will be related to installation problems, errors and bugs. Development questions when forking the code are welcome as well. Use the `fiware-cygnus` tag.
Expand Down
10 changes: 5 additions & 5 deletions doc/architecture.md
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
#<a name="top"></a>Cygnus architecture
# <a name="top"></a>Cygnus architecture
Cygnus runs Flume agents. Thus, Cygnus agents architecture is Flume agents one. Let's see how this architecture ranges from the most basic configuration to the most complex one.

##Flume architecture
## Flume architecture
As stated in [flume.apache.org](http://flume.apache.org/FlumeDeveloperGuide.html):

>An Event is a unit of data that flows through a Flume agent. The Event flows from Source to Channel to Sink, and is represented by an implementation of the Event interface. An Event carries a payload (byte array) that is accompanied by an optional set of headers (string attributes). A Flume agent is a process (JVM) that hosts the components that allow Events to flow from an external source to a external destination.
Expand All @@ -12,7 +12,7 @@ As stated in [flume.apache.org](http://flume.apache.org/FlumeDeveloperGuide.html
[Top](#top)

##Basic Cygnus agent architecture
## Basic Cygnus agent architecture
The simplest way of using Cygnus is to adopt basic constructs/flows of <i>source - channel - sink</i> as described in the Apache Flume documentation. There can be as many basic constructs/flows as persistence elements, i.e. one for HDFS, another one for MySQL, etc.

For each one of this flows, a [`HttpSource`](http://flume.apache.org/FlumeUserGuide.html#http-source) has to be used. The way this native sources process the Orion notifications is by means of a specific REST handler: `NGSIRESTHandler`. Nevetheless, this basic approach requires each source receives its own event notifications. This is not a problem if the architect clearly defines which flows must end in a HDFS storage, or in a Carto storage, if talking about a NGSI agent. But, what happens if the same event must be stored at HDFS and Carto at the same time? In this case, the constructs are modified in order all of them have the same Http source; then, the notified event is replicated for each channel connected to the source.
Expand All @@ -38,7 +38,7 @@ Finally, the sinks are custom ones, one per each persistence element covered by

[Top](#top)

##Advanced Cygnus architectures
## Advanced Cygnus architectures
All the advanced archictures arise when trying to improve the performance of Cygnus. As seen above, basic Cygnus configuration is about a source writting Flume events into a single channel where a single sink consumes those events. This can be clearly moved to a multiple sink configuration running in parallel; there are several possibilities:

### Multiple sinks, single channel
Expand All @@ -63,7 +63,7 @@ Due to the available <i>Channel Selectors</i> do not fit our needs, a custom sel

[Top](#top)

##High availability Cygnus architecture
## High availability Cygnus architecture
High Availability (or HA) is achieved by replicating a whole Cygnus agent, independently of the internal architecture (basic or advance), in an active-passive standard schema. I.e. when the active Cygnus agent fails, a load balancer redirects all the incoming Orion notifications to the passive one. Both Cygnus agents are able to persist the notified context data using the same set of sinks with identical configuration.

![](./images/ha_architecture.jpg)
Expand Down
Loading

0 comments on commit 61c1f0b

Please sign in to comment.