在 Ubuntu/Debian 安裝 NFS Ganesha 取代 NFS-kernel service

因為詢問 ChatGPT 是否有提升 NFS 服務效能的輔助軟體, 結果給了這答案:
使用 NFS 緩存 : 某些軟體和硬體方案提供 NFS 緩存，可減少對伺服器的存取，提升效能。這些緩存可以是軟體層面的，例如 nfs-ganesha 或硬體層面的存儲緩存。
也看到預設 Ubuntu/Debian 可透過 apt 直接安裝, 門檻很低就直接安裝測試

安裝與設定方式

安裝程序

sudo apt update
sudo apt install nfs-ganesha nfs-ganesha-vfs

設定 nfs 分享目錄 Exp. /nfsshare 可讀寫, 只允許 10.20.0.0/22 這網段的用戶存取

vi /etc/ganesha/ganesha.conf

EXPORT
{
    Export_ID = 1;
    Path = /mnt/datastore/zfs-backup/nfsshare;
    Pseudo = /nfsshare;
    Access_Type = RW;
    Squash = No_Root_Squash;
    FSAL {
        Name = VFS;
    }
    SecType = sys;
    CLIENT {
        Clients = 10.20.0.0/22;
        Access_Type = RW;
    }
}

設定啟動 nfs-ganesha 服務

sudo systemctl restart nfs-ganesha
sudo systemctl enable nfs-ganesha

查詢服務狀態
```
sudo systemctl status nfs-ganesha
```

如果服務狀態不是完全正常

確認是否原本的 nfs-kernel service 還沒關閉, 關閉方式如下:

sudo systemctl stop nfs-kernel-server
sudo systemctl disable nfs-kernel-server

查詢 log 了解問題發生的原因: Exp.

tail -100 /var/log/ganesha/ganesha.log

看範例紀錄

# tail -100 /var/log/ganesha/ganesha.log
03/01/2024 05:24:53 : epoch 6594efa5 : test-jonathan-45 : ganesha.nfsd-4407[main] config_errs_to_log :CONFIG :CRIT :Config File (/etc/ganesha/ganesha.conf:6): Unexpected character (▒)
03/01/2024 05:24:53 : epoch 6594efa5 : test-jonathan-45 : ganesha.nfsd-4407[main] config_errs_to_log :CONFIG :CRIT :Config File (/etc/ganesha/ganesha.conf:6): Unexpected character (▒)
03/01/2024 05:24:53 : epoch 6594efa5 : test-jonathan-45 : ganesha.nfsd-4407[main] config_errs_to_log :CONFIG :CRIT :Config File (/etc/ganesha/ganesha.conf:6): Unexpected character (▒)
03/01/2024 05:24:53 : epoch 6594efa5 : test-jonathan-45 : ganesha.nfsd-4407[main] config_errs_to_log :CONFIG :CRIT :Config File (/etc/ganesha/ganesha.conf:6): Unexpected character (▒)
:
03/01/2024 05:24:53 : epoch 6594efa5 : test-jonathan-45 : ganesha.nfsd-4407[main] config_errs_to_log :CONFIG :CRIT :Config File (/etc/ganesha/ganesha.conf:10): Unexpected character (▒)
03/01/2024 05:24:53 : epoch 6594efa5 : test-jonathan-45 : ganesha.nfsd-4407[main] config_errs_to_log :CONFIG :CRIT :Config File (/etc/ganesha/ganesha.conf:3): Syntax error in block
03/01/2024 05:24:53 : epoch 6594efa5 : test-jonathan-45 : ganesha.nfsd-4407[main] config_errs_to_log :CONFIG :CRIT :Config File (/etc/ganesha/ganesha.conf:0): Configuration syntax errors found
03/01/2024 05:24:53 : epoch 6594efa5 : test-jonathan-45 : ganesha.nfsd-4407[main] main :NFS STARTUP :FATAL :Fatal errors.  Server exiting...
03/01/2024 05:26:18 : epoch 6594effa : test-jonathan-45 : ganesha.nfsd-4431[main] main :MAIN :EVENT :ganesha.nfsd Starting: Ganesha Version 3.0.3
03/01/2024 05:26:18 : epoch 6594effa : test-jonathan-45 : ganesha.nfsd-4432[main] load_rados_config :CONFIG :CRIT :Unknown urls backend
03/01/2024 05:26:18 : epoch 6594effa : test-jonathan-45 : ganesha.nfsd-4432[main] nfs_set_param_from_conf :NFS STARTUP :EVENT :Configuration file successfully parsed
03/01/2024 05:26:18 : epoch 6594effa : test-jonathan-45 : ganesha.nfsd-4432[main] init_server_pkgs :NFS STARTUP :EVENT :Initializing ID Mapper.
03/01/2024 05:26:18 : epoch 6594effa : test-jonathan-45 : ganesha.nfsd-4432[main] init_server_pkgs :NFS STARTUP :EVENT :ID Mapper successfully initialized.
03/01/2024 05:26:18 : epoch 6594effa : test-jonathan-45 : ganesha.nfsd-4432[main] nfs_start_grace :STATE :EVENT :NFS Server Now IN GRACE, duration 90
03/01/2024 05:26:18 : epoch 6594effa : test-jonathan-45 : ganesha.nfsd-4432[main] load_fsal :NFS STARTUP :FATAL :Could not dlopen module: /usr/lib/x86_64-linux-gnu/ganesha/libfsalvfs.so Error: /usr/lib/x86_64-linux-gnu/ganesha/libfsalvfs.so: cannot open shared object file: No such file or directory. You might want to install the nfs-ganesha-VFS package
03/01/2024 05:27:38 : epoch 6594f04a : test-jonathan-45 : ganesha.nfsd-4589[main] main :MAIN :EVENT :ganesha.nfsd Starting: Ganesha Version 3.0.3
03/01/2024 05:27:38 : epoch 6594f04a : test-jonathan-45 : ganesha.nfsd-4591[main] load_rados_config :CONFIG :CRIT :Unknown urls backend
03/01/2024 05:27:38 : epoch 6594f04a : test-jonathan-45 : ganesha.nfsd-4591[main] nfs_set_param_from_conf :NFS STARTUP :EVENT :Configuration file successfully parsed
:
03/01/2024 05:27:39 : epoch 6594f04a : test-jonathan-45 : ganesha.nfsd-4591[main] nfs_Start_threads :THREAD :EVENT :admin thread was started successfully
03/01/2024 05:27:39 : epoch 6594f04a : test-jonathan-45 : ganesha.nfsd-4591[_9p_disp] _9p_dispatcher_thread :9P DISP :EVENT :9P dispatcher started
03/01/2024 05:27:39 : epoch 6594f04a : test-jonathan-45 : ganesha.nfsd-4591[main] nfs_Start_threads :THREAD :EVENT :reaper thread was started successfully
03/01/2024 05:27:39 : epoch 6594f04a : test-jonathan-45 : ganesha.nfsd-4591[main] nfs_Start_threads :THREAD :EVENT :General fridge was started successfully
03/01/2024 05:27:39 : epoch 6594f04a : test-jonathan-45 : ganesha.nfsd-4591[main] nfs_start :NFS STARTUP :EVENT :-------------------------------------------------
03/01/2024 05:27:39 : epoch 6594f04a : test-jonathan-45 : ganesha.nfsd-4591[main] nfs_start :NFS STARTUP :EVENT :             NFS SERVER INITIALIZED
03/01/2024 05:27:39 : epoch 6594f04a : test-jonathan-45 : ganesha.nfsd-4591[main] nfs_start :NFS STARTUP :EVENT :-------------------------------------------------
03/01/2024 05:29:09 : epoch 6594f04a : test-jonathan-45 : ganesha.nfsd-4591[reaper] nfs_lift_grace_locked :STATE :EVENT :NFS Server Now NOT IN GRACE

進階調教

調整自訂記憶體當 Cache

Mem_Max 預設可能是 64 MB 調高至 2048 MB → Mem_Max = 2147483648
Mem_Cache_Size 預設可能是 32 MB 調高至 512 MB → Mem_Cache_Size = 536870912
Dir_MaxCacheSize 預設可能是 512 KB 調高至 4096 KB → Dir_MaxCacheSize = 4194304

將設定加入 ganesha.conf 內

vi /etc/ganesha/ganesha.conf

NFS_Core_Param
{
    MNT_Port = 20048;          # MNT 協議端口
    NLM_Port = 32803;          # NLM 協議端口
    Bind_Addr = 0.0.0.0;       # 監聽所有網卡
    Protocols = 3,4;           # 支持 NFSv3 和 NFSv4
}

_9P {
    Nb_Worker = 16;  # 工作線程現在在這裡
}

MDCACHE {
    Cache_Size = 536870912;  # 512MB
    Dir_Chunk = 4194304;     # 4MB
}

TIRPC {
    Max_Connections = 1024;    # 替代 RPC_Max_Connections
    Idle_Timeout_S = 300;      # 替代 RPC_Idle_Timeout_S
    Default_Buffer_Size = 1048576;  # 替代 RPC_Buffer_Size
}

Cache_Parameters {
    Entries_HWMark = 100000;       # 設定緩存條目上限
    NParts = 32;                     # 增加分區數量以減少鎖競爭
    Cache_Size = 1000;               # 緩存大小
    Use_Getattr_Directory_Invalidation = true;
}

# 關閉 Kerberos 以避免那些錯誤
NFS_KRB5 {
    Active_krb5 = false;
    KeytabPath = "/dev/null";  # 避免嘗試讀取實際的 keytab 文件
    PrincipalName = "";        # 清空主體名稱
    CCacheDir = "";            # 清空憑證緩存目錄
}

LOG {
    Default_Log_Level = EVENT;
    RPC_Debug_Flags = 0;        # 減少 RPC 調試信息
}

EXPORT
:

重啟 nfs-ganesha 服務讓設定生效
```
systemctl restart nfs-ganesha
```

使用 fio 來驗證調教後效能提升大概 18%

調教前看完整測試資料

Run status group 0 (all jobs):
   READ: bw=1637KiB/s (1676kB/s), 1637KiB/s-1637KiB/s (1676kB/s-1676kB/s), io=192MiB (202MB), run=120272-120272msec
  WRITE: bw=1637KiB/s (1677kB/s), 1637KiB/s-1637KiB/s (1677kB/s-1677kB/s), io=192MiB (202MB), run=120360-120360msec

調教後看完整測試資料

Run status group 0 (all jobs):
   READ: bw=1935KiB/s (1981kB/s), 1935KiB/s-1935KiB/s (1981kB/s-1981kB/s), io=228MiB (239MB), run=120434-120434msec
  WRITE: bw=1953KiB/s (2000kB/s), 1953KiB/s-1953KiB/s (2000kB/s-2000kB/s), io=230MiB (241MB), run=120500-120500msec

常見問題

關閉 Ganasha 的作法

systemctl stop nfs-ganesha
systemctl disable nfs-ganesha

nfs