Hi,
Firstly, thanks for this application and for giving us the opportunity to use it.
To try it out, I deployed ProxyFS on a CentOS 7.4 VM using the Vagrantfile
in the saio
subfolder. By the way, the referenced vagrant box in this file seems to be down, I used this box (config.vm.box = "CentosBox/Centos-7-v7.4-Minimal-CLI"
with the virtualbox
provider to continue.
After the vagrant_provision.sh
finished running, I compiled the ProxyFS project using make
: everything went well. I then used the script start_and_mount_pfs
to mount the NFS and SMB share.
I can create folders/files in both shares without issues, and then view everything with the swift
CLI:
[vagrant@localhost ~]$ ll /mnt/smb_proxyfs_mount/
total 0
drwxr-xr-x. 2 vagrant vagrant 0 Jun 17 16:25 test
drwxr-xr-x. 2 vagrant vagrant 0 Jun 14 15:48 test_container
drwxr-xr-x. 2 vagrant vagrant 0 Jun 14 15:56 test_container2
[vagrant@localhost ~]$ ll /mnt/smb_proxyfs_mount/test_container
total 0
-rwxr-xr-x. 1 vagrant vagrant 8 Jun 14 15:48 test_file.txt
[vagrant@localhost ~]$ cat /mnt/smb_proxyfs_mount/test_container/test_file.txt
abcdefg
[vagrant@localhost ~]$ swift -A http://127.0.0.1:8080/auth/v1.0 -U test:tester -K testing list
test
test_container
test_container2
[vagrant@localhost ~]$ swift -A http://127.0.0.1:8080/auth/v1.0 -U test:tester -K testing list test_container
test_file.txt
[vagrant@localhost ~]$ curl -i http://127.0.0.1:8080/v1/AUTH_test/test_container/test_file.txt -X GET -H "X-Auth-Token: AUTH_tka85032f655f249cca7d43b5c71184858"
HTTP/1.1 200 OK
Content-Length: 8
Accept-Ranges: bytes
Last-Modified: Fri, 14 Jun 2019 13:48:25 GMT
Etag: "pfsv2/AUTH_test/00000311/00000001-32"
X-Timestamp: 1560520104.65309
Content-Type: text/plain
X-Trans-Id: txb4100d18d9de43d094d35-005d08d72a
X-Openstack-Request-Id: txb4100d18d9de43d094d35-005d08d72a
Date: Tue, 18 Jun 2019 12:20:58 GMT
abcdefg
I've been looking for a way to use ProxyFS in my existing OpenStack Swift/Keystone installation:
- Swift version
rocky
(installed with this link)
- Keystone v3 version
rocky
(installed using this link)
So far, I have been able to deploy a CentOS 7.4 VM using the Vagrantfile
in the saio
subfolder. I removed everything regarding the installation of Swift (including the creation of the user swift
) since I already have one installed.
I then fiddled with the ProxyFS configuration on this VM to point to my existing Swift Proxy server. I installed the pfs
and meta
middlewares on the machine hosting my Swift Proxy server, added them to the pipeline.
I also launched another instance of the Proxy server listening on port 8090 with the /etc/swift/proxy-server/proxy-noauth.cond.d/20_settings.conf
file:
/usr/bin/python2 /usr/bin/swift-proxy-server /etc/swift/proxy-server/proxy-noauth.cond.d
Finally I used the script start_and_mount_pfs
, after removing the lines about starting Swift, to launch ProxyFS and mount the NFS and SMB network shares.
The NFS share seems to work well (I can create folders and write files), but I'm getting an error trying to mount the SMB one. Relevant info: since I haven't created a swift
user, I replaced it with the vagrant
user that was already existing in the VM in the smb.conf
file, and used smbpasswd -a vagrant
.
Command line error:
[vagrant@localhost ~]$ sudo mount -t cifs -o user=vagrant,uid=1000,gid=1000,vers=3.0,iocharset=utf8,actimeo=0 //127.0.0.1/proxyfs /mnt/smb_proxyfs_mount/
Password for vagrant@//127.0.0.1/proxyfs: *******
mount error(5): Input/output error
Refer to the mount.cifs(8) manual page (e.g. man mount.cifs)
What I find in /var/log/samba/log.smbd
after adding log level = 3 passdb:5 auth:5
in smb.conf
:
[2019/06/18 13:41:48.820712, 3] ../lib/util/access.c:361(allow_access)
Allowed connection from 127.0.0.1 (127.0.0.1)
[2019/06/18 13:41:48.821084, 3] ../source3/smbd/oplock.c:1322(init_oplocks)
init_oplocks: initializing messages.
[2019/06/18 13:41:48.821353, 3] ../source3/smbd/process.c:1958(process_smb)
Transaction 0 of length 106 (0 toread)
[2019/06/18 13:41:48.821806, 3] ../source3/smbd/smb2_negprot.c:290(smbd_smb2_request_process_negprot)
Selected protocol SMB3_00
[2019/06/18 13:41:48.821849, 5] ../source3/auth/auth.c:491(make_auth_context_subsystem)
Making default auth method list for server role = 'standalone server', encrypt passwords = yes
[2019/06/18 13:41:48.821873, 5] ../source3/auth/auth.c:48(smb_register_auth)
Attempting to register auth backend trustdomain
[2019/06/18 13:41:48.821926, 5] ../source3/auth/auth.c:60(smb_register_auth)
Successfully added auth method 'trustdomain'
[2019/06/18 13:41:48.821945, 5] ../source3/auth/auth.c:48(smb_register_auth)
Attempting to register auth backend ntdomain
[2019/06/18 13:41:48.821956, 5] ../source3/auth/auth.c:60(smb_register_auth)
Successfully added auth method 'ntdomain'
[2019/06/18 13:41:48.821970, 5] ../source3/auth/auth.c:48(smb_register_auth)
Attempting to register auth backend guest
[2019/06/18 13:41:48.821983, 5] ../source3/auth/auth.c:60(smb_register_auth)
Successfully added auth method 'guest'
[2019/06/18 13:41:48.821994, 5] ../source3/auth/auth.c:48(smb_register_auth)
Attempting to register auth backend sam
[2019/06/18 13:41:48.822004, 5] ../source3/auth/auth.c:60(smb_register_auth)
Successfully added auth method 'sam'
[2019/06/18 13:41:48.822015, 5] ../source3/auth/auth.c:48(smb_register_auth)
Attempting to register auth backend sam_ignoredomain
[2019/06/18 13:41:48.822026, 5] ../source3/auth/auth.c:60(smb_register_auth)
Successfully added auth method 'sam_ignoredomain'
[2019/06/18 13:41:48.822060, 5] ../source3/auth/auth.c:48(smb_register_auth)
Attempting to register auth backend winbind
[2019/06/18 13:41:48.822076, 5] ../source3/auth/auth.c:60(smb_register_auth)
Successfully added auth method 'winbind'
[2019/06/18 13:41:48.822086, 5] ../source3/auth/auth.c:378(load_auth_module)
load_auth_module: Attempting to find an auth method to match guest
[2019/06/18 13:41:48.822099, 5] ../source3/auth/auth.c:403(load_auth_module)
load_auth_module: auth method guest has a valid init
[2019/06/18 13:41:48.822110, 5] ../source3/auth/auth.c:378(load_auth_module)
load_auth_module: Attempting to find an auth method to match sam
[2019/06/18 13:41:48.822122, 5] ../source3/auth/auth.c:403(load_auth_module)
load_auth_module: auth method sam has a valid init
[2019/06/18 13:41:48.823791, 3] ../auth/gensec/gensec_start.c:918(gensec_register)
GENSEC backend 'gssapi_spnego' registered
[2019/06/18 13:41:48.823830, 3] ../auth/gensec/gensec_start.c:918(gensec_register)
GENSEC backend 'gssapi_krb5' registered
[2019/06/18 13:41:48.823904, 3] ../auth/gensec/gensec_start.c:918(gensec_register)
GENSEC backend 'gssapi_krb5_sasl' registered
[2019/06/18 13:41:48.823935, 3] ../auth/gensec/gensec_start.c:918(gensec_register)
GENSEC backend 'spnego' registered
[2019/06/18 13:41:48.823949, 3] ../auth/gensec/gensec_start.c:918(gensec_register)
GENSEC backend 'schannel' registered
[2019/06/18 13:41:48.823964, 3] ../auth/gensec/gensec_start.c:918(gensec_register)
GENSEC backend 'naclrpc_as_system' registered
[2019/06/18 13:41:48.823976, 3] ../auth/gensec/gensec_start.c:918(gensec_register)
GENSEC backend 'sasl-EXTERNAL' registered
[2019/06/18 13:41:48.823988, 3] ../auth/gensec/gensec_start.c:918(gensec_register)
GENSEC backend 'ntlmssp' registered
[2019/06/18 13:41:48.824000, 3] ../auth/gensec/gensec_start.c:918(gensec_register)
GENSEC backend 'ntlmssp_resume_ccache' registered
[2019/06/18 13:41:48.824014, 3] ../auth/gensec/gensec_start.c:918(gensec_register)
GENSEC backend 'http_basic' registered
[2019/06/18 13:41:48.824030, 3] ../auth/gensec/gensec_start.c:918(gensec_register)
GENSEC backend 'http_ntlm' registered
[2019/06/18 13:41:48.824789, 5] ../source3/auth/auth.c:491(make_auth_context_subsystem)
Making default auth method list for server role = 'standalone server', encrypt passwords = yes
[2019/06/18 13:41:48.824822, 5] ../source3/auth/auth.c:378(load_auth_module)
load_auth_module: Attempting to find an auth method to match guest
[2019/06/18 13:41:48.824836, 5] ../source3/auth/auth.c:403(load_auth_module)
load_auth_module: auth method guest has a valid init
[2019/06/18 13:41:48.824847, 5] ../source3/auth/auth.c:378(load_auth_module)
load_auth_module: Attempting to find an auth method to match sam
[2019/06/18 13:41:48.824859, 5] ../source3/auth/auth.c:403(load_auth_module)
load_auth_module: auth method sam has a valid init
[2019/06/18 13:41:48.825052, 3] ../auth/ntlmssp/ntlmssp_util.c:69(debug_ntlmssp_flags)
Got NTLMSSP neg_flags=0xa0080205
[2019/06/18 13:41:48.825484, 3] ../auth/ntlmssp/ntlmssp_server.c:452(ntlmssp_server_preauth)
Got user=[vagrant] domain=[LOCALHOST] workstation=[] len1=0 len2=132
[2019/06/18 13:41:48.825565, 3] ../source3/param/loadparm.c:3823(lp_load_ex)
lp_load_ex: refreshing parameters
[2019/06/18 13:41:48.825665, 3] ../source3/param/loadparm.c:542(init_globals)
Initialising global parameters
[2019/06/18 13:41:48.825810, 3] ../source3/param/loadparm.c:2752(lp_do_section)
Processing section "[global]"
[2019/06/18 13:41:48.825983, 2] ../source3/param/loadparm.c:2769(lp_do_section)
Processing section "[proxyfs]"
[2019/06/18 13:41:48.826162, 3] ../source3/param/loadparm.c:1592(lp_add_ipc)
adding IPC service
[2019/06/18 13:41:48.826198, 5] ../source3/auth/auth_util.c:123(make_user_info_map)
Mapping user [LOCALHOST]\[vagrant] from workstation []
[2019/06/18 13:41:48.826220, 5] ../source3/auth/user_info.c:62(make_user_info)
attempting to make a user_info for vagrant (vagrant)
[2019/06/18 13:41:48.826236, 5] ../source3/auth/user_info.c:70(make_user_info)
making strings for vagrant's user_info struct
[2019/06/18 13:41:48.826244, 5] ../source3/auth/user_info.c:108(make_user_info)
making blobs for vagrant's user_info struct
[2019/06/18 13:41:48.826251, 3] ../source3/auth/auth.c:178(auth_check_ntlm_password)
check_ntlm_password: Checking password for unmapped user [LOCALHOST]\[vagrant]@[] with the new password interface
[2019/06/18 13:41:48.826259, 3] ../source3/auth/auth.c:181(auth_check_ntlm_password)
check_ntlm_password: mapped user is: [LOCALHOST]\[vagrant]@[]
[2019/06/18 13:41:48.826554, 3] ../source3/passdb/lookup_sid.c:1680(get_primary_group_sid)
Forcing Primary Group to 'Domain Users' for vagrant
[2019/06/18 13:41:48.826646, 4] ../source3/auth/check_samsec.c:183(sam_account_ok)
sam_account_ok: Checking SMB password for user vagrant
[2019/06/18 13:41:48.826661, 5] ../source3/auth/check_samsec.c:165(logon_hours_ok)
logon_hours_ok: user vagrant allowed to logon at this time (Tue Jun 18 11:41:48 2019
)
[2019/06/18 13:41:48.827099, 5] ../source3/auth/server_info_sam.c:122(make_server_info_sam)
make_server_info_sam: made server info for user vagrant -> vagrant
[2019/06/18 13:41:48.827130, 3] ../source3/auth/auth.c:249(auth_check_ntlm_password)
check_ntlm_password: sam authentication for user [vagrant] succeeded
[2019/06/18 13:41:48.827153, 5] ../source3/auth/auth.c:292(auth_check_ntlm_password)
check_ntlm_password: PAM Account for user [vagrant] succeeded
[2019/06/18 13:41:48.827160, 2] ../source3/auth/auth.c:305(auth_check_ntlm_password)
check_ntlm_password: authentication for user [vagrant] -> [vagrant] -> [vagrant] succeeded
[2019/06/18 13:41:48.827343, 3] ../source3/auth/token_util.c:548(finalize_local_nt_token)
Failed to fetch domain sid for WORKGROUP
[2019/06/18 13:41:48.827371, 3] ../source3/auth/token_util.c:580(finalize_local_nt_token)
Failed to fetch domain sid for WORKGROUP
[2019/06/18 13:41:48.827624, 5] ../source3/passdb/pdb_interface.c:1749(lookup_global_sam_rid)
lookup_global_sam_rid: looking up RID 513.
[2019/06/18 13:41:48.827655, 5] ../source3/passdb/pdb_tdb.c:658(tdbsam_getsampwrid)
pdb_getsampwrid (TDB): error looking up RID 513 by key RID_00000201.
[2019/06/18 13:41:48.827672, 5] ../source3/passdb/pdb_interface.c:1825(lookup_global_sam_rid)
Can't find a unix id for an unmapped group
[2019/06/18 13:41:48.827679, 5] ../source3/passdb/pdb_interface.c:1535(pdb_default_sid_to_id)
SID S-1-5-21-2240567756-3470875878-3910347872-513 belongs to our domain, but there is no corresponding object in the database.
[2019/06/18 13:41:48.827699, 5] ../source3/passdb/pdb_interface.c:1749(lookup_global_sam_rid)
lookup_global_sam_rid: looking up RID 513.
[2019/06/18 13:41:48.827711, 5] ../source3/passdb/pdb_tdb.c:658(tdbsam_getsampwrid)
pdb_getsampwrid (TDB): error looking up RID 513 by key RID_00000201.
[2019/06/18 13:41:48.827723, 5] ../source3/passdb/pdb_interface.c:1825(lookup_global_sam_rid)
Can't find a unix id for an unmapped group
[2019/06/18 13:41:48.827729, 5] ../source3/passdb/pdb_interface.c:1535(pdb_default_sid_to_id)
SID S-1-5-21-2240567756-3470875878-3910347872-513 belongs to our domain, but there is no corresponding object in the database.
[2019/06/18 13:41:48.827829, 3] ../source3/smbd/password.c:144(register_homes_share)
Adding homes service for user 'vagrant' using home directory: '/home/vagrant'
[2019/06/18 13:41:48.828148, 3] ../lib/util/access.c:361(allow_access)
Allowed connection from 127.0.0.1 (127.0.0.1)
[2019/06/18 13:41:48.828191, 3] ../libcli/security/dom_sid.c:210(dom_sid_parse_endp)
string_to_sid: SID vagrant is not in a valid format
[2019/06/18 13:41:48.828274, 3] ../source3/passdb/lookup_sid.c:1680(get_primary_group_sid)
Forcing Primary Group to 'Domain Users' for vagrant
[2019/06/18 13:41:48.828374, 3] ../source3/smbd/service.c:576(make_connection_snum)
Connect path is '/mnt/CommonVolume' for service [proxyfs]
[2019/06/18 13:41:48.828407, 3] ../libcli/security/dom_sid.c:210(dom_sid_parse_endp)
string_to_sid: SID vagrant is not in a valid format
[2019/06/18 13:41:48.828483, 3] ../source3/passdb/lookup_sid.c:1680(get_primary_group_sid)
Forcing Primary Group to 'Domain Users' for vagrant
[2019/06/18 13:41:48.828562, 3] ../source3/smbd/vfs.c:113(vfs_init_default)
Initialising default vfs hooks
[2019/06/18 13:41:48.828589, 3] ../source3/smbd/vfs.c:139(vfs_init_custom)
Initialising custom vfs hooks from [/[Default VFS]/]
[2019/06/18 13:41:48.828598, 3] ../source3/smbd/vfs.c:139(vfs_init_custom)
Initialising custom vfs hooks from [proxyfs]
[2019/06/18 13:41:48.831109, 2] ../lib/util/modules.c:196(do_smb_load_module)
Module 'proxyfs' loaded
[2019/06/18 13:41:48.834266, 1] vfs_proxyfs.c:230(vfs_proxyfs_connect)
proxyfs_mount_failed: Volume : CommonVolume Connection_path /mnt/CommonVolume Service proxyfs user vagrant errno 19
[2019/06/18 13:41:48.834293, 1] ../source3/smbd/service.c:636(make_connection_snum)
make_connection_snum: SMB_VFS_CONNECT for service 'proxyfs' at '/mnt/CommonVolume' failed: No such device
[2019/06/18 13:41:48.834344, 3] ../source3/smbd/smb2_server.c:3097(smbd_smb2_request_error_ex)
smbd_smb2_request_error_ex: smbd_smb2_request_error_ex: idx[1] status[NT_STATUS_UNSUCCESSFUL] || at ../source3/smbd/smb2_tcon.c:135
[2019/06/18 13:41:48.960403, 3] ../source3/smbd/server_exit.c:246(exit_server_common)
Server exit (NT_STATUS_END_OF_FILE)
[2019/06/18 13:41:48.966933, 3] ../source3/lib/util_procid.c:54(pid_to_procid)
pid_to_procid: messaging_dgm_get_unique failed: No such file or directory
It looks like the samba authentication went well, but the relevant error to me are the following lines:
[2019/06/18 13:41:48.831109, 2] ../lib/util/modules.c:196(do_smb_load_module)
Module 'proxyfs' loaded
[2019/06/18 13:41:48.834266, 1] vfs_proxyfs.c:230(vfs_proxyfs_connect)
proxyfs_mount_failed: Volume : CommonVolume Connection_path /mnt/CommonVolume Service proxyfs user vagrant errno 19
[2019/06/18 13:41:48.834293, 1] ../source3/smbd/service.c:636(make_connection_snum)
make_connection_snum: SMB_VFS_CONNECT for service 'proxyfs' at '/mnt/CommonVolume' failed: No such device
I tried troubleshooting this, but no luck so far. Would anyone be able to help on this?
Here's my df -H
output if needed:
Filesystem Size Used Avail Use% Mounted on
/dev/mapper/cl-root 19G 3.1G 16G 17% /
devtmpfs 3.1G 0 3.1G 0% /dev
tmpfs 3.1G 0 3.1G 0% /dev/shm
tmpfs 3.1G 9.0M 3.1G 1% /run
tmpfs 3.1G 0 3.1G 0% /sys/fs/cgroup
/dev/sda1 1.1G 240M 824M 23% /boot
tmpfs 609M 0 609M 0% /run/user/1000
CommonMountPoint 110T 0 110T 0% /CommonMountPoint
127.0.0.1:/CommonMountPoint 110T 0 110T 0% /mnt/nfs_proxyfs_mount
I also tried to get containers and objects I created via the NFS share with the Object Storage API, but I got the following error on my Swift Proxy server:
[root@controller adminuser]# swift -A http://controller:8080/auth/v1.0 -U test:tester -K testing stat --debug
DEBUG:urllib3.connectionpool:Starting new HTTP connection (1): controller:8080
DEBUG:urllib3.connectionpool:http://controller:8080 "GET /auth/v1.0 HTTP/1.1" 200 0
DEBUG:swiftclient:REQ: curl -i http://controller:8080/auth/v1.0 -X GET
DEBUG:swiftclient:RESP STATUS: 200 OK
DEBUG:swiftclient:RESP HEADERS: {u'Content-Length': u'0', u'X-Trans-Id': u'tx6493625ff99f4486a7f5b-005d08d170', u'X-Auth-Token-Expires': u'76663', u'X-Auth-Token': u'AUTH_tk24c8619d99964285a356cbf294531184', u'X-Storage-Token': u'AUTH_tk24c8619d99964285a356cbf294531184', u'Date': u'Tue, 18 Jun 2019 11:56:32 GMT', u'X-Storage-Url': u'http://controller:8080/v1/AUTH_test', u'Content-Type': u'text/html; charset=UTF-8', u'X-Openstack-Request-Id': u'tx6493625ff99f4486a7f5b-005d08d170'}
DEBUG:urllib3.connectionpool:Starting new HTTP connection (1): controller:8080
DEBUG:urllib3.connectionpool:http://controller:8080 "HEAD /v1/AUTH_test HTTP/1.1" 500 0
INFO:swiftclient:REQ: curl -i http://controller:8080/v1/AUTH_test -I -H "X-Auth-Token: AUTH_tk24c8619d99964285a356cbf294531184"
INFO:swiftclient:RESP STATUS: 500 Internal Error
INFO:swiftclient:RESP HEADERS: {u'Date': u'Tue, 18 Jun 2019 11:56:32 GMT', u'Content-Length': u'17', u'Content-Type': u'text/plain', u'X-Openstack-Request-Id': u'tx3bdf2145377d4050a7044-005d08d170', u'X-Trans-Id': u'tx3bdf2145377d4050a7044-005d08d170'}
Relevant lines in /var/log/messages
regarding the error:
Jun 18 13:57:57 controller proxy-server: STDERR: (23786) accepted ('192.168.71.37', 52024)
Jun 18 13:57:57 controller proxy-server: - - 18/Jun/2019/11/57/57 HEAD /auth/v1.0 HTTP/1.0 400 - Swift - - - - tx1c9994434391428a82261-005d08d1c5 - 0.0002 RL - 1560859077.899780035 1560859077.899970055 -
Jun 18 13:57:57 controller proxy-server: 192.168.71.37 192.168.71.37 18/Jun/2019/11/57/57 GET /auth/v1.0 HTTP/1.0 200 - python-swiftclient-3.6.0 - - - - tx1c9994434391428a82261-005d08d1c5 - 0.0021 - - 1560859077.899091005 1560859077.901160955 -
Jun 18 13:57:57 controller proxy-server: STDERR: 192.168.71.37 - - [18/Jun/2019 11:57:57] "GET /auth/v1.0 HTTP/1.1" 200 417 0.002583 (txn: tx1c9994434391428a82261-005d08d1c5)
Jun 18 13:57:57 controller proxy-server: STDERR: (23786) accepted ('192.168.71.37', 52026)
Jun 18 13:57:57 controller proxy-server: 192.168.71.37 192.168.71.37 18/Jun/2019/11/57/57 HEAD /v1/AUTH_test%3Fformat%3Djson HTTP/1.0 500 - python-swiftclient-3.6.0 AUTH_tk24c8619d9... - - - txc715be53ba9e476483a71-005d08d1c5 - 0.0013 - - 1560859077.906188011 1560859077.907531023 -
Jun 18 13:57:57 controller proxy-server: Erreur : une erreur s'est produite: Hôte inaccessible (txn: txc715be53ba9e476483a71-005d08d1c5)
Jun 18 13:57:57 controller proxy-server: STDERR: 192.168.71.37 - - [18/Jun/2019 11:57:57] "HEAD /v1/AUTH_test HTTP/1.1" 500 222 0.001975 (txn: txc715be53ba9e476483a71-005d08d1c5)
On another subject, does ProxyFS support Keystone authentication, instead of the tempauth
used in the main pipeline?
More broadly, has anyone tried to connect ProxyFS to an existing OpenStack Swift/Keystone installation?
Regards