運維筆記_搭建配置Moosefs檔案伺服器

 搭建硬體環境:Citrix_Xencenter


     1.找一臺效能相對來說不是很差的機器,安裝幾臺Linux伺服器就可以完成,是用vmware也一樣。


 搭建系統環境:Centos 7


備註:Mfs_Data2server 借用了之前一臺zabbix伺服器擔任了此角色,不在Citrix伺服器之內;

 


Moosefs特點:

     moosefs 是一個具備冗餘容錯功能的分散式網路檔案系統,它將資料分別存放多個物理伺服器或單獨磁碟及分割槽上,確保
一份資料多個備份副本。對於訪問的客戶端或者使用者來說,整個分散式網路檔案系統叢集看起來像一個資源一樣。關鍵是可以免費使用不用還錢挺好的!

 1.高冗餘,高可用;

 2.安裝簡單方便;

 3.管理起來也方便;

開始安裝:

1.Mfs_Masterserver:10.1.66.13

   因為我的系統是Centos 7 全新安裝的,我們需要安裝一下編譯環境,為了方便做實驗我也關閉了selinux和fireall服務

[[email protected]_master ~]# vim /etc/selinux/config

# This file controls the state of SELinux on the system.
# SELINUX= can take one of these three values:
#     enforcing – SELinux security policy is enforced.
#     permissive – SELinux prints warnings instead of enforcing.
#     disabled – No SELinux policy is loaded.
#SELINUX=enforcing
SELINUX=disabled  #關閉掉服務
# SELINUXTYPE= can take one of these two values:
#     targeted – Targeted processes are protected,
#     minimum – Modification of targeted policy. Only selected processes are protected. 
#     mls – Multi Level Security protection.
#SELINUXTYPE=targeted 

[[email protected]_master ~]# systemctl disable firewalld.service #禁止firewall開機啟動

建立mfs使用者和組

useradd mfs -s /sbin/nologin

以下是一些Mfs需要的環境安裝包,安裝後就可以一路暢通!

下載安裝zlib包

wget http://zlib.net/zlib-1.2.8.tar.gz

tar -zxvf  zlib-1.2.8.tar.gz

cd zlib-1.2.8

./configure 

make && make install

下載安裝fuse包

wget http://ncu.dl.sourceforge.net/project/fuse/fuse-2.X/2.8.1/fuse-2.8.1.tar.gz

tar -zxvf  fuse-2.8.1.tar.gz

cd fuse-2.8.1

./configure 

make && make install

下載安裝MFS包:

wget http://pro.hit.gemius.pl/hitredir/id=.WCbG2t.7Ln5k1s3Q9xPg8cPfX.wVMc5kyXfrKcJTDH.c7/url=moosefs.org/tl_files/mfscode/mfs-1.6.27-1.tar.gz

tar -zxvf mfs-1.6.27-1.tar.gz

cd mfs-1.6.27

./configure –prefix=/usr/local/mfs   –with-default-user=mfs –with-default-group=mfs

make && make install

目錄結構:

[[email protected]_master ~]# ll /usr/local/mfs

   drw-xr-x 2 root root 4096 Jan 31 04:09 etc   #mfs配置檔案目錄
    drwxr-xr-x 2 root root 4096 Jan 31 04:09 sbin  #mfs命令路徑
    drwxr-xr-x 4 root root 4096 Jan 31 04:09 share #mfs幫助檔案目錄
    drwxr-xr-x 3 root root 4096 Jan 31 04:09 var   #mfs資料及日誌目錄

修改配置檔案

cp /usr/local/mfs/etc/mfs/mfsmaster.cfg.dist    /usr/local/mfs/etc/mfs/mfsmaster.cfg  #主配置檔案
cp /usr/local/mfs/etc/mfs/mfsexports.cfg.dist   /usr/local/mfs/etc/mfs/mfsexports.cfg  #許可權配置檔案

修改主配置檔案:

[[email protected]_master ~]# vim /usr/local/mfs/etc/mfs/mfsmaster.cfg

# WORKING_USER = mfs 執行master server 的使用者
 
# WORKING_GROUP = mfs 執行master server 的組
 
# SYSLOG_IDENT = mfsmaster master server 在syslog中的標識,說明是由master serve 產生的
 
# LOCK_MEMORY = 0 是否執行mlockall()以避免mfsmaster 程序溢位(預設為0)
 
# NICE_LEVEL = -19 執行的優先順序(如果可以預設是-19; 注意: 程序必須是用root啟動)
 
# EXPORTS_FILENAME = /usr/local/mfs/etc/mfsexports.cfg 被掛接目錄及其許可權控制檔案的存放位置
 
# DATA_PATH = /usr/local/mfs/var/mfs 資料存放路徑,此目錄下大致有三類檔案,changelog,sessions和stats;
 
# BACK_LOGS = 50 metadata 的改變log 檔案數目(預設是50);
 
# REPLICATIONS_DELAY_INIT = 300 延遲複製的時間(預設是300s)
;
# REPLICATIONS_DELAY_DISCONNECT = 3600 chunkserver 斷開的複製延遲(預設是3600);
 
# MATOML_LISTEN_HOST = * metalogger 監聽的IP 地址(預設是*,代表任何IP);
 
# MATOML_LISTEN_PORT = 9419 metalogger 監聽的埠地址(預設是9419)

# MATOCS_LISTEN_HOST = * 用於chunkserver 連線的IP 地址(預設是*,代表任何IP);
 
# MATOCS_LISTEN_PORT = 9420 用於chunkserver 連線的埠地址(預設是9420);
 
# MATOCU_LISTEN_HOST = * 用於客戶端掛接連線的IP 地址(預設是*,代表任何IP);
 
# MATOCU_LISTEN_PORT = 9421 用於客戶端掛接連線的埠地址(預設是9421);
 
# CHUNKS_LOOP_TIME = 300 chunks 的迴環頻率(預設是:300 秒);
 
# CHUNKS_DEL_LIMIT = 100
# CHUNKS_WRITE_REP_LIMIT = 1 在一個迴圈裡複製到一個chunkserver 的最大chunk數目(預設是1)
 
# CHUNKS_READ_REP_LIMIT = 5 在一個迴圈裡從一個chunkserver 複製的最大chunk數目(預設是5)
 
# REJECT_OLD_CLIENTS = 0 彈出低於1.6.0 的客戶端掛接(0 或1,預設是0)

修改許可權配置檔案

[email protected]_master ~]# vim /usr/local/mfs/etc/mfs/mfsexports.cfg
# Allow everything but “meta”.
echo 10.1.66.0/24 / rw,alldirs,mapall=mfs:mfs,password=888 >mfsexports.cfg #新增訪問網段

修改目錄許可權:

chown -R mfs:mfs /usr/local/mfs/*

啟動與關閉mfs服務:

/usr/local/mfs/sbin/mfsmaster start #開啟
/usr/local/mfs/sbin/mfsmaster -s    #關閉

加入到開機自動啟動:

echo “#startup master” >> /etc/rc.local 
echo “/usr/local/mfs/sbin/mfsmaster start ” >>/etc/rc.local 

檢視MFS監聽埠是否成功開啟:

[email protected]_master ~]# netstat -antlpe |grep mfs
tcp        0      0 0.0.0.0:9419            0.0.0.0:*               LISTEN      1001       23210      2642/mfsmaster      
tcp        0      0 0.0.0.0:9420            0.0.0.0:*               LISTEN      1001       23211      2642/mfsmaster      
tcp        0      0 0.0.0.0:9421            0.0.0.0:*               LISTEN      1001       23212      2642/mfsmaster

開啟mfsmaster圖形監控服務:

[[email protected]_master ~]# /usr/local/mfs/sbin/mfscgiserv restart
sending SIGTERM to lock owner (pid:17831)
lockfile created and locked
starting simple cgi server (host: any , port: 9425 , rootpath: /usr/local/mfs/share/mfscgi)

使用瀏覽器訪問:10.1.66.13:9425


2安裝MFS_Backupserver IP:10.1.66.14 

安裝backup和master步驟完全一致,需要注意的是配置檔案的改動

[[email protected]_backup~]# vim /etc/selinux/config

# This file controls the state of SELinux on the system.
# SELINUX= can take one of these three values:
#     enforcing – SELinux security policy is enforced.
#     permissive – SELinux prints warnings instead of enforcing.
#     disabled – No SELinux policy is loaded.
#SELINUX=enforcing
SELINUX=disabled  #關閉掉服務
# SELINUXTYPE= can take one of these two values:
#     targeted – Targeted processes are protected,
#     minimum – Modification of targeted policy. Only selected processes are protected. 
#     mls – Multi Level Security protection.
#SELINUXTYPE=targeted 

[[email protected]_backup ~]# systemctl disable firewalld.service #禁止firewall開機啟動

建立mfs使用者和組

useradd mfs -s /sbin/nologin

以下是一些Mfs需要的環境安裝包

下載安裝zlib包

wget http://zlib.net/zlib-1.2.8.tar.gz

tar -zxvf  zlib-1.2.8.tar.gz

cd zlib-1.2.8

./configure 

make && make install

下載安裝fuse包

wget http://ncu.dl.sourceforge.net/project/fuse/fuse-2.X/2.8.1/fuse-2.8.1.tar.gz

tar -zxvf  fuse-2.8.1.tar.gz

cd fuse-2.8.1

./configure 

make && make install

下載安裝MFS包:

wget http://pro.hit.gemius.pl/hitredir/id=.WCbG2t.7Ln5k1s3Q9xPg8cPfX.wVMc5kyXfrKcJTDH.c7/url=moosefs.org/tl_files/mfscode/mfs-1.6.27-1.tar.gz

tar -zxvf mfs-1.6.27-1.tar.gz

cd mfs-1.6.27

./configure –prefix=/usr/local/mfs   –with-default-user=mfs –with-default-group=mfs

make && make install

修改配置檔案:

cp /usr/local/mfs/etc/mfs/mfsmetalogger.cfg.dist  /usr/local/mfs/etc/mfs/mfsmetalogger.cfg

[[email protected]_backup ~]# vim /usr/local/mfs/etc/mfs/mfsmetalogger.cfg

# WORKING_USER = mfs
# WORKING_GROUP = mfs
# SYSLOG_IDENT = mfsmetalogger
# LOCK_MEMORY = 0
# NICE_LEVEL = -19

# DATA_PATH = /usr/local/mfs/var/mfs

# BACK_LOGS = 50
# BACK_META_KEEP_PREVIOUS = 3
# META_DOWNLOAD_FREQ = 24

# MASTER_RECONNECTION_DELAY = 5

MASTER_HOST = 10.1.66.13 #master主機IP地址或域名
# MASTER_PORT = 9419

# MASTER_TIMEOUT = 60

# deprecated, to be removed in MooseFS 1.7
# LOCK_FILE = /var/run/mfs/mfsmetalogger.lock

啟動服務:

/usr/local/mfs/sbin/mfsmetalogger start

加入開啟啟動:

 echo “#start mfsmetalogger ” >>/etc/rc.local
 echo “/usr/local/mfs/sbin/mfsmetalogger start”  >>/etc/rc.local

檢視監聽埠是否正常啟動:

[email protected]_backup ~]# netstat -antlpe |grep mfs
tcp        0      0 10.1.66.14:57895        10.1.66.13:9419         ESTABLISHED 1001       111713     32302/mfsmetalogger

安裝Mfs_Dataserver IP:10.1.66.16和105

Mfs_Dataserver (chunk server) 安裝步驟和master完全一致,只是配置檔案不同

[[email protected]_data~]# vim /etc/selinux/config

# This file controls the state of SELinux on the system.
# SELINUX= can take one of these three values:
#     enforcing – SELinux security policy is enforced.
#     permissive – SELinux prints warnings instead of enforcing.
#     disabled – No SELinux policy is loaded.
#SELINUX=enforcing
SELINUX=disabled  #關閉掉服務
# SELINUXTYPE= can take one of these two values:
#     targeted – Targeted processes are protected,
#     minimum – Modification of targeted policy. Only selected processes are protected. 
#     mls – Multi Level Security protection.
#SELINUXTYPE=targeted 

[[email protected]_data ~]# systemctl disable firewalld.service #禁止firewall開機啟動

建立mfs使用者和組

useradd mfs -s /sbin/nologin

以下是一些Mfs需要的環境安裝包

下載安裝zlib包

wget http://zlib.net/zlib-1.2.8.tar.gz

tar -zxvf  zlib-1.2.8.tar.gz

cd zlib-1.2.8

./configure 

make && make install

下載安裝fuse包

wget http://ncu.dl.sourceforge.net/project/fuse/fuse-2.X/2.8.1/fuse-2.8.1.tar.gz

tar -zxvf  fuse-2.8.1.tar.gz

cd fuse-2.8.1

./configure 

make && make install

下載安裝MFS包:

wget http://pro.hit.gemius.pl/hitredir/id=.WCbG2t.7Ln5k1s3Q9xPg8cPfX.wVMc5kyXfrKcJTDH.c7/url=moosefs.org/tl_files/mfscode/mfs-1.6.27-1.tar.gz

tar -zxvf mfs-1.6.27-1.tar.gz

cd mfs-1.6.27

./configure –prefix=/usr/local/mfs   –with-default-user=mfs –with-default-group=mfs

make && make install

修改配置檔案:

cp /usr/local/mfs/etc/mfs/mfschunkserver.cfg.dist /usr/local/mfs/etc/mfs/mfschunkserver.cfg
cp /usr/local/mfs/etc/mfs/mfshdd.cfg.dist /usr/local/mfs/etc/mfs/mfshdd.cfg

[[email protected]_data ~]vim /usr/local/mfs/etc/mfs/mfschunkserver.cfg

# WORKING_USER = mfs
# WORKING_GROUP = mfs
# SYSLOG_IDENT = mfschunkserver
# LOCK_MEMORY = 0
# NICE_LEVEL = -19

# DATA_PATH = /usr/local/mfs/var/mfs

# MASTER_RECONNECTION_DELAY = 5

# BIND_HOST = *
MASTER_HOST = 10.1.66.13
# MASTER_PORT = 9420

# MASTER_TIMEOUT = 60

# CSSERV_LISTEN_HOST = *
# CSSERV_LISTEN_PORT = 9422

# HDD_CONF_FILENAME = /usr/local/mfs/etc/mfs/mfshdd.cfg
# HDD_TEST_FREQ = 10

# deprecated, to be removed in MooseFS 1.7
# LOCK_FILE = /var/run/mfs/mfschunkserver.lock
# BACK_LOGS = 50
# CSSERV_TIMEOUT = 5

 [[email protected]_data ~]vim /usr/local/mfs/etc/mfs/mfshdd.cfg

# mount points of HDD drives
#
#/mnt/hd1
#/mnt/hd2
#etc.
/mfsdata

新增掛載點目錄:

[[email protected]_data ~]#mkdir /mnt/mfs

[[email protected]_data ~]#mkdir /mfsdata

[[email protected]_data ~]#chown -R mfs.mfs /mfsdata

 [[email protected]_data ~]# mount /dev/sda5      /mnt/mfsdata  #/dev/sda5 /是我的跟目錄,實際環境中建議單獨分割槽方便管理;

[[email protected]_data ~]#vim /etc/fstab 

/dev/sda5               /mfsdata               ext3     defaults      0 0  #新增一條

echo “/mfsdata ” >mfshdd.cfg  #新增共享掛載點

啟動服務:

[[email protected]_data ~]#/usr/local/mfs/sbin/mfschunkserver start

檢查埠是否啟動:

[[email protected]_data ~]# netstat -tunlp |grep 9422
tcp        0      0 0.0.0.0:9422            0.0.0.0:*               LISTEN

安裝MFs_client ip:10.1.66.15

[[email protected]_data~]# vim /etc/selinux/config

# This file controls the state of SELinux on the system.
# SELINUX= can take one of these three values:
#     enforcing – SELinux security policy is enforced.
#     permissive – SELinux prints warnings instead of enforcing.
#     disabled – No SELinux policy is loaded.
#SELINUX=enforcing
SELINUX=disabled  #關閉掉服務
# SELINUXTYPE= can take one of these two values:
#     targeted – Targeted processes are protected,
#     minimum – Modification of targeted policy. Only selected processes are protected. 
#     mls – Multi Level Security protection.
#SELINUXTYPE=targeted 

[[email protected]_data ~]# systemctl disable firewalld.service #禁止firewall開機啟動

建立mfs使用者和組

useradd mfs -s /sbin/nologin

以下是一些Mfs需要的環境安裝包

下載安裝zlib包

wget http://zlib.net/zlib-1.2.8.tar.gz

tar -zxvf  zlib-1.2.8.tar.gz

cd zlib-1.2.8

./configure 

make && make install

下載安裝fuse包

wget http://ncu.dl.sourceforge.net/project/fuse/fuse-2.X/2.8.1/fuse-2.8.1.tar.gz

tar -zxvf  fuse-2.8.1.tar.gz

cd fuse-2.8.1

./configure 

make && make install

下載安裝MFS包:

wget http://pro.hit.gemius.pl/hitredir/id=.WCbG2t.7Ln5k1s3Q9xPg8cPfX.wVMc5kyXfrKcJTDH.c7/url=moosefs.org/tl_files/mfscode/mfs-1.6.27-1.tar.gz

tar -zxvf mfs-1.6.27-1.tar.gz

cd mfs-1.6.27

./configure –prefix=/usr/local/mfs   –with-default-user=mfs –with-default-group=mfs –enable-mfsmount #注意不同地方

make && make install

建立掛載步驟:

mkdir /mnt/mfs -p
chown -R mfs.mfs /mnt/mfs 許可權賦予
載入fuse 模組到核心:modprobe fuse
/usr/local/mfs/bin/mfsmount /mnt/mfs -H 10.1.66.13 -p

輸入密碼:888

檢查是否掛載成功:

[[email protected] ~]# df -h
檔案系統                 容量  已用  可用 已用% 掛載點
/dev/mapper/centos-root   18G  4.0G   14G   23% /
devtmpfs                 486M     0  486M    0% /dev
tmpfs                    495M   80K  495M    1% /dev/shm
tmpfs                    495M  6.9M  488M    2% /run
tmpfs                    495M     0  495M    0% /sys/fs/cgroup
/dev/xvda1               497M  118M  380M   24% /boot
10.1.66.13:9421          118G     0  118G    0% /mnt/mfs

暫時先到這裡,留個記號有時間在來完成後續操作;

~