-
Notifications
You must be signed in to change notification settings - Fork 1
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
- Loading branch information
Showing
6 changed files
with
355 additions
and
0 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
|
@@ -134,6 +134,8 @@ if [ $1 -eq 1 ];then | |
echo "1" | ||
elif [ $1 -eq 2 ];then | ||
echo "2" | ||
else | ||
echo "3" | ||
fi | ||
``` | ||
|
||
|
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,224 @@ | ||
# 搭建Hadoop开发环境 | ||
|
||
## 虚拟机配置 | ||
|
||
### 修改IP | ||
|
||
`/etc/sysconfig/network-scripts/ifcfg-eth0` | ||
|
||
```shell | ||
ONBOOT="yes" | ||
BOOTPROTO="static" | ||
# 修改为自己的ip,网关和dns | ||
IPADDR="10.211.55.11" | ||
GATEWAY="10.211.55.1" | ||
DNS1="10.211.55.1" | ||
``` | ||
|
||
### 修改主机名 | ||
|
||
`/etc/sysconfig/network` | ||
|
||
```shell | ||
NETWORKING=yes | ||
HOSTNAME=box11 | ||
``` | ||
|
||
### 修改 host | ||
|
||
`/etc/hosts` | ||
|
||
```shell | ||
10.211.55.11 box11 | ||
10.211.55.12 box12 | ||
10.211.55.13 box13 | ||
10.211.55.14 box14 | ||
10.211.55.15 box15 | ||
10.211.55.16 box16 | ||
``` | ||
|
||
### 创建用户 | ||
|
||
```shell | ||
# 创建用户 | ||
useradd box | ||
## 修改用户密码 | ||
passwd box | ||
``` | ||
|
||
### 添加用户为root用户 | ||
|
||
`/etc/sudoers` | ||
|
||
```shell | ||
# 在 root ALL=(ALL) ALL 下面添加 | ||
box ALL=(ALL) ALL | ||
``` | ||
|
||
### 配置用户组 | ||
|
||
通过用户组为用户增加操作权限, 一下所有操作的文件`root`组必须有权限操作 | ||
|
||
```shell | ||
# 将box用户添加到root组中 | ||
gpasswd -a box root | ||
``` | ||
|
||
### 暂时关闭防火墙 | ||
|
||
```shell | ||
systemctl stop firewalld | ||
systemctl disable firewalld | ||
``` | ||
|
||
## 配置Java | ||
|
||
解压文件 | ||
|
||
`tar -zxvf jdk-xxx.tar.gz` | ||
|
||
配置环境变量, 编辑`/etc/profile` | ||
|
||
```shell | ||
export JAVA_HOME=jdk解压目录的绝对路径 | ||
export PATH=$PATH:$JAVA_HOME/bin | ||
``` | ||
|
||
## 配置hadoop | ||
|
||
解压文件 | ||
|
||
`tar -zxvf hadoop-xxx.tar.gz` | ||
|
||
配置环境变量, 编辑`/etc/profile/` | ||
|
||
```shell | ||
export HADOOP_HOME=hadoop解压目录的绝对路径 | ||
export PATH=$PATH:$HADOOP_HOME/bin:$HADOOP_HOME/sbin | ||
``` | ||
|
||
### 单机模式 | ||
|
||
Hadoop在无需进行任何配置的情况下,默认为单机模式 | ||
|
||
### 伪分布式模式 | ||
|
||
#### 配置文件 | ||
|
||
`etc/hadoop/hadoop-env.sh` | ||
|
||
```shell | ||
# 将 JAVA_HOME=${JAVA_HOME} 替换为本机绝对路径 | ||
JAVA_HOME=java home的绝对路径 | ||
``` | ||
|
||
`etc/hadoop/core-site.xml` | ||
|
||
```xml | ||
<configuration> | ||
<property> | ||
<name>fs.defaultFS</name> | ||
<value>hdfs://localhost:9000</value> | ||
</property> | ||
</configuration> | ||
``` | ||
|
||
`etc/hadoop/hdfs-site.xml` | ||
|
||
```xml | ||
<configuration> | ||
<property> | ||
<name>dfs.replication</name> | ||
<value>1</value> | ||
</property> | ||
</configuration> | ||
``` | ||
|
||
#### 配置ssh免密登陆 | ||
|
||
1.检查ssh是否为免密登陆状态 | ||
|
||
```shell | ||
$ ssh localhost | ||
``` | ||
|
||
2.如果上一步需要输入密码,使用如下命令配置 | ||
|
||
```shell | ||
$ ssh-keygen -t rsa -P '' -f ~/.ssh/id_rsa | ||
$ cat ~/.ssh/id_rsa.pub >> ~/.ssh/authorized_keys | ||
$ chmod 0600 ~/.ssh/authorized_keys | ||
``` | ||
|
||
#### 执行 | ||
|
||
1.格式化namenode, 仅能执行一次 | ||
|
||
```shell | ||
$ hdfs namenode -format | ||
``` | ||
|
||
2.启动namenode守护进程 | ||
|
||
```shell | ||
$ start-hdfs.sh | ||
``` | ||
|
||
日志默认打印在`${HADOOP_HOME}/logs`下面 | ||
|
||
3.通过浏览器访问`http://localhost:50070`即可展示namenode的信息 | ||
|
||
4.关闭hdfs | ||
|
||
```shell | ||
stop-hdfs.sh | ||
``` | ||
|
||
#### 配置yarn | ||
|
||
`etc/hadoop/yarn-env.sh` | ||
|
||
```shell | ||
# 将 JAVA_HOME=${JAVA_HOME} 替换为本机绝对路径 | ||
JAVA_HOME=java home的绝对路径 | ||
``` | ||
|
||
`etc/hadoop/mapred-site.xml` | ||
|
||
```xml | ||
<configuration> | ||
<property> | ||
<name>mapreduce.framework.name</name> | ||
<value>yarn</value> | ||
</property> | ||
</configuration> | ||
``` | ||
|
||
`etc/hadoop/yarn-site.xml` | ||
|
||
```xml | ||
<configuration> | ||
<property> | ||
<name>yarn.nodemanager.aux-services</name> | ||
<value>mapreduce_shuffle</value> | ||
</property> | ||
|
||
<property> | ||
<name>yarn.resourcemanager.hostname</name> | ||
<value>localhost</value> | ||
</property> | ||
</configuration> | ||
``` | ||
|
||
启动 | ||
|
||
```shell | ||
start-yarn.sh | ||
``` | ||
|
||
停止 | ||
|
||
```shell | ||
stop-yarn.sh | ||
``` | ||
|
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,54 @@ | ||
# 如何从git中忽略指定的文件 | ||
|
||
现在有两种方式, `.gitignore`或者`.git/info/exclude`,这两种方式都表示指定的文件不不会被git进行管理. | ||
|
||
但现在有一个需求, 我想忽略某些已经被git管理的文件, 例如, 我有一些配置文件, 我需要git帮我管理着他们, 但是这些配置文件每个开发人员需要配置的项目不一样, 这时我想修改这个文件,但是又不想让git提醒我提交它. | ||
|
||
## skip-worktree | ||
|
||
`git update-index --skip-worktree path/to/file` | ||
|
||
`skip-worktree`的作用时, 指定的文件一定会被修改, 但是git要认为它没有改变.这样我们就可以在本地修改被指定的文件, 而不同担心`git add .`会将修改的文件添加到版本控制中. | ||
|
||
检查文件是否被忽略: | ||
|
||
```shell | ||
git ls-files -v | grep ^s | ||
``` | ||
|
||
* `git ls-files`显示所有被git管理的文件 | ||
* `-v`选择被忽略的文件 | ||
* `s`是`--skip-worktree`的缩写 | ||
|
||
将忽略的文件重新添加到版本控制需要使用如下命令: | ||
|
||
```shell | ||
git update-index --no-skip-worktree path/to/file | ||
``` | ||
|
||
|
||
|
||
## assume-unchanged | ||
|
||
```shell | ||
git update-index --assume-unchanged path/to/file | ||
``` | ||
|
||
`assume-unchanged`的作用是, 指定的文件在本地肯定不会被修改, git可以忽略对指定文件的检测. | ||
|
||
当我们认为某些文件我们一定不会尽心修改, 又想加快git的运行速度时,可以使用该命令. | ||
|
||
检查文件是否被忽略: | ||
|
||
```shell | ||
git ls-files -v | grep ^h | ||
``` | ||
|
||
* `h`是`assume-unchanged`的缩写 | ||
|
||
恢复使用如下命令: | ||
|
||
```shell | ||
git update-index --no-assume-unchanged path/to/file | ||
``` | ||
|
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,46 @@ | ||
# git 控制台中文乱码 | ||
|
||
默认情况下, 通过git status查询修改的文件,如果文件名中包含中文, 则会显示乱码, 如下: | ||
|
||
```shell | ||
git status | ||
位于分支 master | ||
您的分支领先 'origin/master' 共 2 个提交。 | ||
(使用 "git push" 来发布您的本地提交) | ||
|
||
尚未暂存以备提交的变更: | ||
(使用 "git add <文件>..." 更新要提交的内容) | ||
(使用 "git restore <文件>..." 丢弃工作区的改动) | ||
修改: linux/shell.md | ||
|
||
未跟踪的文件: | ||
(使用 "git add <文件>..." 以包含要提交的内容) | ||
"\345\244\247\346\225\260\346\215\256/hadoop/\345\207\206\345\244\207\350\231\232\346\213\237\346\234\272.md" | ||
|
||
修改尚未加入提交(使用 "git add" 和/或 "git commit -a") | ||
``` | ||
|
||
出现的原始是: git会对0x80以上的字符进行quote. | ||
|
||
解决的方案为: `git config --global core.quotePath false` | ||
|
||
再次查看, 中文就显示正常了. | ||
|
||
```shell | ||
git status | ||
位于分支 master | ||
您的分支领先 'origin/master' 共 2 个提交。 | ||
(使用 "git push" 来发布您的本地提交) | ||
|
||
尚未暂存以备提交的变更: | ||
(使用 "git add <文件>..." 更新要提交的内容) | ||
(使用 "git restore <文件>..." 丢弃工作区的改动) | ||
修改: linux/shell.md | ||
|
||
未跟踪的文件: | ||
(使用 "git add <文件>..." 以包含要提交的内容) | ||
大数据/hadoop/准备虚拟机.md | ||
|
||
修改尚未加入提交(使用 "git add" 和/或 "git commit -a") | ||
``` | ||
|
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,26 @@ | ||
在各操作系统下,文本文件所使用的换行符是不一样的。UNIX/Linux 使用的是 0x0A(LF),早期的 Mac OS 使用的是0x0D(CR),后来的 OS X 在更换内核后与 UNIX 保持一致了。但 DOS/Windows 一直使用 0x0D0A(CRLF)作为换行符。Git提供了一个“换行符自动转换”功能。这个功能默认处于“自动模式”,当你在签出文件时,它试图将 UNIX 换行符(LF)替换为 Windows 的换行符(CRLF);当你在提交文件时,它又试图将 CRLF 替换为 LF。Git 的“换行符自动转换”功能听起来似乎很智能、很贴心,因为它试图一方面保持仓库内文件的一致性(UNIX 风格),一方面又保证本地文件的兼容性(Windows 风格)。但遗憾的是,这个功能是有 bug 的,而且在短期内都不太可能会修正。 | ||
|
||
```shell | ||
AutoCRLF | ||
#提交时转换为LF,检出时转换为CRLF | ||
git config --global core.autocrlf true | ||
|
||
#提交时转换为LF,检出时不转换 (一般仅设置该项) | ||
git config --global core.autocrlf input | ||
|
||
#提交检出均不转换 | ||
git config --global core.autocrlf false | ||
SafeCRLF | ||
#拒绝提交包含混合换行符的文件 | ||
git config --global core.safecrlf true | ||
|
||
#允许提交包含混合换行符的文件 | ||
git config --global core.safecrlf false | ||
|
||
#提交包含混合换行符的文件时给出警告 | ||
git config --global core.safecrlf warn | ||
``` | ||
|
||
|
||
|
||
引用自: [git 换行符LF与CRLF转换问题 ](https://www.cnblogs.com/sdgf/p/6237847.html) |