5th semester

kvetinac97 · Jan 30, 2022 · 5040709 · 5040709
1 parent 707c1e1
commit 5040709
Show file tree

Hide file tree

Showing 436 changed files with 141,769 additions and 0 deletions.
diff --git a/BI-BIG/README.adoc b/BI-BIG/README.adoc
@@ -0,0 +1,14 @@
+= DB Technologies for Big Data
+
+Got 100 % from use-cases (tasks on labs). +
+Got 79 % from semestral test, resulting in **grade A**.
+
+== Use-cases
+
+Tasks from labs, with the following themes:
+
+- UC1: Docker
+- UC2: MongoDB
+- UC3: Open data
+- UC4: Apache Spark
+- UC5: ElasticSearch, Kibana
diff --git a/BI-BIG/grades.pdf b/BI-BIG/grades.pdf
diff --git a/BI-BIG/uc1/assignment.adoc b/BI-BIG/uc1/assignment.adoc
@@ -0,0 +1,3 @@
+= UC1
+
+We had to write a Docker compose in order to start a functioning web server with Node.js and PostgreSQL.
diff --git a/BI-BIG/uc1/backend/.dockerignore b/BI-BIG/uc1/backend/.dockerignore
@@ -0,0 +1,3 @@
+node_modules
+npm-debug.log
+
diff --git a/BI-BIG/uc1/backend/Dockerfile b/BI-BIG/uc1/backend/Dockerfile
@@ -0,0 +1,10 @@
+FROM node:14
+
+WORKDIR /usr/src/app
+
+COPY package.json ./
+RUN npm install
+
+COPY . .
+
+CMD [ "node", "server.js" ]
diff --git a/BI-BIG/uc1/backend/package.json b/BI-BIG/uc1/backend/package.json
@@ -0,0 +1,10 @@
+{
+  "name": "simp-todo-list",
+  "version": "0.0.42",
+  "dependencies": {
+    "cors": "*",
+    "express": "*",
+    "pg": "*"
+  }
+}
+
diff --git a/BI-BIG/uc1/backend/server.js b/BI-BIG/uc1/backend/server.js
@@ -0,0 +1,61 @@
+const { Pool } = require('pg');
+
+const env = process.env;
+const config = {
+    host: env.DB_HOST,
+    port: env.DB_PORT || '5432',
+    user: env.DB_USER,
+    password: env.DB_PASSWORD,
+    database: env.DB_NAME,
+}; // odkud se asi tak berou tyto údaje? ;)
+
+const pool = new Pool(config);
+
+async function query(query, params) {
+    const {rows, fields} = await pool.query(query, params);
+    return rows;
+}
+
+const express = require('express');
+var app = express();
+
+app.use(express.json());
+app.use(express.urlencoded({ extended: false }));
+
+const cors = require('cors');
+app.use(cors({ origin: '*' }));
+
+app.get('/tasks', async function(req, res, next) {
+    const tasks = await query(
+        'SELECT * FROM task ORDER BY createdOn ASC', // todo: vyber všechny úkoly seřazené od nejstarsiho
+        []
+    );
+    res.json({"tasks": tasks ? tasks : []});
+});
+
+app.put('/tasks', async function(req, res, next) {
+    const tasks = await query(
+        'INSERT INTO task (name, createdOn) VALUES ($1, NOW()) RETURNING taskId, name, createdOn, completedOn;',
+        [ req.body.name ]
+    );
+    res.json(tasks ? tasks[0] : {});
+});
+
+app.patch('/task/:taskId', async function(req, res) {
+    await query(
+        'UPDATE task SET completedOn = NOW() WHERE taskId = $1;', // todo: nastavit completedOn na aktuální čas pro daný úkol
+        [ req.params.taskId ]
+    );
+    res.json({});
+});
+
+app.delete('/task/:taskId', async function(req, res) {
+    await query(
+        'DELETE FROM task WHERE taskId = $1;', // todo: smazat daný úkol
+        [ req.params.taskId ]
+    );
+    res.json({});
+});
+
+app.listen(3001);
+
diff --git a/BI-BIG/uc1/database/.gitkeep b/BI-BIG/uc1/database/.gitkeep
@@ -0,0 +1,2 @@
+You have to remove this file before starting Postgres,
+otherwise, it will become crazy.
diff --git a/BI-BIG/uc1/db/init.sql b/BI-BIG/uc1/db/init.sql
@@ -0,0 +1,7 @@
+-- todo: skip when table already exists
+CREATE TABLE IF NOT EXISTS task (
+    taskId      SERIAL PRIMARY KEY,
+    name        VARCHAR(100) NOT NULL,
+    createdOn   TIMESTAMP NOT NULL,
+    completedOn TIMESTAMP
+);
diff --git a/BI-BIG/uc1/docker-compose.yml b/BI-BIG/uc1/docker-compose.yml
@@ -0,0 +1,36 @@
+version: '3'
+
+services:
+  frontend:
+    image: nginx:alpine
+    volumes:
+     - ./frontend:/usr/share/nginx/html
+    ports:
+     - 3000:80
+  backend:
+    build: ./backend
+    environment:
+     DB_HOST: db
+     DB_PORT: 5432
+     DB_USER: bibig
+     DB_PASSWORD: bibig
+     DB_NAME: bibig
+    ports:
+     - 3001:3001
+  db:
+    image: postgres
+    restart: always
+    volumes:
+     - ./database:/var/lib/postgresql/data
+     - ./db:/docker-entrypoint-initdb.d/
+    environment:
+      POSTGRES_USER: bibig
+      POSTGRES_PASSWORD: bibig
+      POSTGRES_DB: bibig
+    ports:
+     - 5432:5432
+  adminer:
+    image: adminer
+    restart: always
+    ports:
+     - 8080:8080
diff --git a/BI-BIG/uc1/frontend/index.html b/BI-BIG/uc1/frontend/index.html
@@ -0,0 +1,95 @@
+<!DOCTYPE html>
+<html>
+<head>
+    <meta charset="UTF-8"/>
+    <title>Úkolníček</title>
+    <style type="text/css">
+        body { max-width: 500px; margin: auto; font-family: sans-serif; }
+        table { margin-top: 50px; border-collapse: collapse; }
+        tr:nth-child(even) { background-color: #f8f8f8; }
+        td { padding: 5px 0; }
+        .name { width: 100%; }
+        .name a { color: black; text-decoration: none; }
+        .name a:hover { color: black; text-decoration: line-through; }
+        .done .name a, a:hover { color: gray; text-decoration: line-through; }
+        .delete a { display: block; background-color: gray; color: white; text-decoration: none; border-radius: 3px; padding: 5px; }
+        .delete a:hover { background-color: darkred; color: white; text-decoration: none; }
+        .done .delete a { background-color: lightgray; }
+        #template { display: none; }
+    </style>
+</head>
+<body>
+<h1>Úkolníček</h1>
+<form onsubmit="return createTask(this)">
+<input type="text" class="text" placeholder="Zadej úkol" autofocus="autofocus"/><input type="submit" value="Vytvoř úkol">
+</form>
+<table>
+    <tbody id="tasks">
+        <tr id="template" data-id="0"><td class="name"><a href="#" onclick="finishTask(this)">Název</a></td><td class="delete"><a href="#" onclick="removeTask(this)">Smazat</a></td></tr>
+    </tbody>
+</table>
+<script type="text/javascript">
+const apiUrl = "http://localhost:3001";
+// on page load, fetch task data
+function createTask(e) {
+    let i = e.querySelector('.text');
+    fetch(apiUrl + '/tasks', {
+        method: 'PUT',
+        body: JSON.stringify({
+            name: i.value
+        }),
+        headers: new Headers({
+            'Content-Type': 'application/json'
+        })
+    }).then(res => res.json())
+        .then(res => appendTask(res))
+        .catch(err => console.error(err));
+    i.value = "";
+    return false;
+}
+
+function appendTask(data) {
+    let c = document.querySelector('#template').cloneNode(true);
+    c.id = "";
+    c.dataset.id = "" + data.taskid;
+    if(data.completedon) c.classList.add('done');
+    c.querySelector('.name a').textContent = data.name;
+    let t = document.querySelector('#tasks');
+    t.insertBefore(c,t.firstChild);
+}
+
+function finishTask(e) {
+    let c = e.parentNode.parentNode;
+    if(c.classList.contains('done')) return false;
+    fetch(apiUrl + '/task/' + c.dataset.id, {
+        method: 'PATCH',
+        headers: {
+            'Content-type': 'application/json; charset=UTF-8',
+        }
+    }).catch(err => console.error(err));
+    c.classList.add('done');
+    return false;
+}
+
+function removeTask(e) {
+    let c = e.parentNode.parentNode;
+    fetch(apiUrl + '/task/' + c.dataset.id, {
+        method: 'DELETE',
+        headers: {
+            'Content-type': 'application/json; charset=UTF-8',
+        }
+    }).catch(err => console.error(err));
+    c.parentNode.removeChild(c);
+    return false;
+}
+
+document.addEventListener("DOMContentLoaded", function() {
+    fetch(apiUrl + '/tasks')
+        .then(res => res.json())
+        .then(res => res.tasks.map(appendTask))
+        .catch(err => console.error(err));
+}, false);
+</script>
+</body>
+</html>
+
diff --git a/BI-BIG/uc2/assignment.adoc b/BI-BIG/uc2/assignment.adoc
@@ -0,0 +1,3 @@
+= UC2
+
+We got a MongoDB database, and we had to write queries based on given assignment.
diff --git a/BI-BIG/uc2/solution.txt b/BI-BIG/uc2/solution.txt
@@ -0,0 +1,67 @@
+// 1
+db.cars.find({"znacka": null}).count()
+
+// 2
+db.cars.find({"evidence": "Praha"}, {"jmeno": 1, "prijmeni": 1, "_id": 0}).sort({"prijmeni": 1, "jmeno": 1})
+
+// 3
+db.cars.deleteMany({"evidence": "Karlovy Vary", "$or": [{"jmeno": "Roman", "prijmeni": "Komarov"}, {"vyrobce": "Mercedes", "barva": "cerna"}]})
+
+// 4
+db.cars.insert({
+	"stari": 0,
+	"barva": "cerna",
+	"znacka": "BIGDATA",
+	"jmeno": "Michal",
+	"prijmeni": "Valenta",
+	"evidence": "Praha",
+	"vyrobce": "Audi"
+})
+
+// 5
+db.cars.update({"znacka": "BIGDATA"}, {
+	"$push": {"kontroly": {
+		"stanice": "Beroun",
+		"uspech": true,
+		"datum": new Date("2021-10-21"),
+	}},
+	"$set": {"jmeno": "Josef", "prijmeni": "Gattemayer"}
+})
+
+// 6
+db.cars.aggregate([
+	{ "$group": {
+		"_id": "$evidence",
+		"prumerneStari": {"$avg": "$stari"}
+	}}
+])
+
+// 7
+db.cars.aggregate([
+	{ "$unwind": { "path": "$kontroly", preserveNullAndEmptyArrays: true } },
+	{ "$match": { "kontroly.uspech": true, "kontroly.datum": {"$gte": new Date("2018-10-21")} } },
+	{ "$group": {"_id": "$znacka"} },
+	{ "$sort": { "_id": 1 } }
+])
+
+// 8
+db.cars.aggregate([
+	{ "$group": {
+		"_id": "$znacka",
+		"pocet": {"$sum": 1}
+	}},
+	{ "$match": { "pocet": {"$gt": 1} } }
+]).forEach(function(doc) { db.getCollection("cars").remove({"znacka": doc._id}); })
+
+db.cars.ensureIndex({"znacka": 1}, {"unique": true})
+
+
+
+
+
+
+
+
+
+
+
diff --git a/BI-BIG/uc3/assignment.adoc b/BI-BIG/uc3/assignment.adoc
@@ -0,0 +1,16 @@
+= UC3
+
+Pro splnění UC3 je nutné předložit/ukázat a vysvětlit vyučující v rámci konání cvičení:
+
+* minimálně 3 různé datasety, které tvoří jeden tématický celek (budou dále využity pro UC4 a UC5)
+* alespoň jeden vstupní dataset bude mít nejméně 5.000 záznamů
+* vybrané datové sady musí být navzájem propojitelné skrz nějaký společný identifikátor či jiný sloupec
+* uvést zdroje odkud jsou jednotlivé datové sety čerpány
+* uvést jeden tématický celek, jinými slovy porozumět vybraným datům a uvést jakou oblast řeší
+* předložené datasety je nutné vybrat tak, aby bylo možné vytvářet smysluplné vizualizace
+* uvést, co lze z dat získat
+* uvést, jak rozumíte jednotlivým datovým sadám, například jakého datového typu je každý sloupec, co je obsahem každého sloupce a případná integritní omezení na sloupcích
+* uvést, jakého datového formátu jsou datové sady
+* uvést, zda a jak budete data připravovat pro UC4+UC5 (zmenšení datové sady, vyčištění chybných položek, vyčištění prázdných položek, sjednocení na různé úrovni granularity dat, apod.)
+* hotový UC3 odevzdejte vyučující dle níže uvedeného způsobu odevzdání
+* níže uvedené ukázkové řešení a data nelze použít, dále nevyužívejte prosím databázi IMDB
diff --git a/BI-BIG/uc3/solution.adoc b/BI-BIG/uc3/solution.adoc
@@ -0,0 +1,10 @@
+= UC3 solution
+
+I chose 4 datasets, all containing data from the Czech Republic throughout year 2020:
+
+ 1. Death reasons
+ 2. Hospitalizations
+ 3. List of economic subjects
+ 4. Helper table mapping districts to cities
+
+My idea was to watch how number of open restaurants is influencing hospitalizations and deaths, to watch death reasons and number of deaths in various cities and months.
diff --git a/BI-BIG/uc4/assignment.adoc b/BI-BIG/uc4/assignment.adoc
@@ -0,0 +1,4 @@
+= UC4
+
+The task was to create set of queries on our datasets from UC3. +
+Unfortunately, I did not manage to find my solution.
Original file line number	Diff line number	Diff line change
		@@ -0,0 +1,3 @@
		= UC1

		We had to write a Docker compose in order to start a functioning web server with Node.js and PostgreSQL.
Original file line number	Diff line number	Diff line change
		@@ -0,0 +1,2 @@
		You have to remove this file before starting Postgres,
		otherwise, it will become crazy.
Original file line number	Diff line number	Diff line change
		@@ -0,0 +1,3 @@
		= UC2

		We got a MongoDB database, and we had to write queries based on given assignment.