-
Notifications
You must be signed in to change notification settings - Fork 0
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
- Loading branch information
Showing
8 changed files
with
165 additions
and
0 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,49 @@ | ||
--- | ||
title: ESP(Event Streaming Platform)에 대한 이해 | ||
date: 2022-11-08 | ||
categories: [Bigdata, Kafka] | ||
tags: [Bigdata, Kafka] | ||
--- | ||
|
||
## ESP의 개념 | ||
|
||
Event Streaming Platform | ||
|
||
- Event Streaming | ||
- 이벤트 소스와 이벤트 대상간의 지속적인 이벤트 전송 | ||
- 하나의 이벤트 소스에 하나의 이벤트 대상이 지정된다면 ETL로 할 수 있다. | ||
- 다양한 이벤트가 있고, 이벤트 소스와 이벤트 대상이 다를 때 ESP를 통해 이벤트 관리 | ||
|
||
### ETL | ||
|
||
![image](/assets/img/_posts/bigdata/kafka/etl.png) | ||
|
||
### Without ESP | ||
|
||
![image](/assets/img/_posts/bigdata/kafka/without_esp.png) | ||
|
||
### With ESP | ||
|
||
![image](/assets/img/_posts/bigdata/kafka/with_esp.png) | ||
|
||
## ESP의 구성 | ||
|
||
### 1. Broker | ||
|
||
이벤트를 수신, 소비하는 브로커 | ||
|
||
#### Ingestor : 이벤트 수신 | ||
|
||
#### Processor : 이벤트 처리 | ||
|
||
직렬화, 역직렬화 / 암호화, 비암호화 / 압축 | ||
|
||
#### Consumption : 이벤트 배포 | ||
|
||
### 2. Storage | ||
|
||
수신된 이벤트를 저장하는 스토리지 | ||
|
||
### 3. Analytic and Query Engine | ||
|
||
이벤트를 분석, 쿼리하는 엔진 |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,28 @@ | ||
--- | ||
title: Kafka에 대한 이해 | ||
date: 2022-11-08 | ||
categories: [Bigdata, Kafka] | ||
tags: [Bigdata, Kafka] | ||
--- | ||
|
||
## Kafka | ||
|
||
### Architecture | ||
|
||
분산된 클라이언트, 서버 형태 | ||
|
||
kafka 브로커 | ||
|
||
- 여러개의 관련 서버가 있는 클러스터 | ||
- 이벤트 수신, 저장 배포 하는 전용 서버 | ||
- 브로커는 ZooKeeper라는 분산 시스템에서 관리됨 | ||
- 브로커가 효율적이고 협력적으로 업무 수행 | ||
- TCP기반의 통신을 통해 클라이언트와 서버 간에 데이터를 교환 | ||
|
||
### Example | ||
|
||
```plaintext | ||
[tweets in JSON] -> [Twitter Producer] -> [twitter_topic] | ||
[twitter_topic] -> [Twitter Comsumer] -> [tweets in JSON] | ||
[tweets in JSON] -> [DB Writer] -> [RDBMS] | ||
``` |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,38 @@ | ||
--- | ||
title: chmod를 확실하게 이해하자 | ||
date: 2022-11-08 | ||
categories: [Think, Linux] | ||
tags: [Linux] | ||
--- | ||
|
||
## 권한 | ||
|
||
권한의 종류 | ||
- r : read | ||
- w : write | ||
- x : execute | ||
|
||
## 유저 | ||
|
||
유저의 종류 | ||
- u : user (본인) | ||
- g : group (그룹) | ||
- o : others (다른유저) | ||
- a : all (전체) | ||
|
||
## 간단한 예시 | ||
|
||
```plaintext | ||
drwxr-xr-x⠀staff⠀64⠀Apr 24 19:03:26⠀ ⠀my_dir/ | ||
-rw-r--r--⠀staff⠀0 ⠀Apr 24 19:03:32⠀ ⠀my_file_01 | ||
-rw-r--r--⠀staff⠀0 ⠀Apr 24 19:03:43⠀ ⠀my_file_02 | ||
-rw-r--r--⠀staff⠀0 ⠀Apr 24 19:03:44⠀ ⠀my_file_03 | ||
0123456789 | ||
권한의 해석 | ||
0 : is_directory (d / -) | ||
1 : User : read (r / -) | ||
2 : User : write (w / -) | ||
3 : User : execute (e / -) | ||
``` |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,42 @@ | ||
--- | ||
title: 리눅스를 살펴보자 | ||
date: 2022-11-08 | ||
categories: [Think, Linux] | ||
tags: [Linux] | ||
--- | ||
|
||
## Five Layers of Linux | ||
|
||
1. UI(User Interface) | ||
1. Application | ||
1. Operating System | ||
1. Kernel | ||
1. Hard Ware | ||
|
||
--- | ||
|
||
## Shell / Terminal | ||
|
||
1. User | ||
1. Terminal | ||
1. Shell / OS / Kernel | ||
1. Hard Ware | ||
|
||
--- | ||
|
||
## Debian / Redhat | ||
|
||
### Debian | ||
|
||
`apt based` | ||
|
||
### Redhat | ||
|
||
`yum based` | ||
|
||
--- | ||
|
||
## 권한 | ||
|
||
- drwx : \[directory]\[read]\[write]\[execute] | ||
- -rwx : \[file]\[read]\[write]\[execute] |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,8 @@ | ||
--- | ||
title: 국비지원교육 및 기업연계프로젝트를 마치며 | ||
date: 2023-06-01 | ||
categories: [writes] | ||
tags: [_All, _Writes] | ||
--- | ||
|
||
## |
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.