Multitenancy support?

[cloudfoundry]

Multitenancy support?

[cf-gitbot]

We have created an issue in Pivotal Tracker to manage this:

https://www.pivotaltracker.com/story/show/146688249

The labels on this github issue will be updated when the story is started.

[paulcwarren]

The way our cephfs-volume-release works is actually a function of the way cephfs itself works. We had to implement that way in order to partition each service's data from each other.

For nfs-volume-release, that can mount any number of exports, the situation is a little different. Typically, apps that require an nfs volume will connect to an app-specific export and usually also use an app-specific user and achieve multi-tenancy that way.

It would definitely be possible to modify the nfs broker to work the same way as the ceph-broker though. Is there something about how you are setup that would mean this broker behavior would be advantagious for you?

We are also happy to chat on a conference call if you would like?

[nota-ja]

Thank you for the quick response, @paulcwarren

Now I think I've understood that the basic approach of nfs-volume-service to enable multitenancy is Providing different NFS exports for each developer .

Is there something about how you are setup that would mean this broker behavior would be advantagious for you?

I think it's better to answer this question after we collect voices of users and organize them. So please wait a bit.

We are also happy to chat on a conference call if you would like?

Because I live in Japan, time difference would be a major hurdle for a conference call.
I will attend the CF Summit in the next week, so it's good to have f2f if you and we can.

[julian-hj]

@nota-ja : Thanks for your inputs, and we'd be very happy to speak with you in person. Please do come find us at the Dell EMC booth, and/or at the Diego Persistence office hours.

The big difference between our nfs release and the ceph release as I see it is that ceph comes pre-packaged with a storage server, and the broker is designed to allocate space on that storage server.

NFS is intended to be used with pre-existing already allocated storage. That means that our nfs release can work with just about any external NFS shares that support NFSv3, but it also means that the broker is not capable of allocating storage when a service instance is created. We do package an nfs test server with the nfs-volume-release, but that is really only intended for testing, and comes with a few hard coded shares exposed. We didn't intend that it would get used for production, and in fact today it puts data on ephemeral storage, so it is really not suitable for production.

We envisioned that additional service brokers would be built specific to different kinds of storage, and that those service brokers would call into the management APIs for those servers to provision new shares, but as yet, those aren't published to pivotal network or similar marketplaces. We've been working with the ECS team (Dell EMC Elastic Cloud Storage) on just such a broker for ECS, which ought to be in production later this year.

Anyway, we look forward to meeting you! Thanks, -Julian

[nota-ja]

Thank you for the kind response, @julian-hj
It will be great to have a time for discussion with you and the persi team. I think I'll go to the Dell EMC booth hopefully on the first day.

The big difference between our nfs release and the ceph release as I see it is that ceph comes pre-packaged with a storage server, and the broker is designed to allocate space on that storage server.
..

Yes.
When our team discussed yesterday about this topic, we reached a similar conclusion.

The current nfs-volume-release seems designed to use one or more existing NFS shares from applications deployed on the CF. There might have been exsisting files in those NFS shares and the apps want to read and write such the "old" files, want to share them with "old" applications.

On the other hand, volume services (generally, I think) such like cephfs-volume-release seems designed to share disk space between app instances / apps on a CF. So it is natural that creating service instance is creating an empty directory.

It will be great if nfs-volume-release supports both usage, but it may be difficult in one single volume service supports the two different design concepts that should have different characteristics.

---

Meanwhile, I'll return to the question I reserved to answer before.

Is there something about how you are setup that would mean this broker behavior would be advantagious for you?

With the current implementation of nfs-volume-release to use it especially in shared CF environment (disclosure: NTT Communications, one of our group companies, provides a shared service offering of Cloud Foundry), our major concerns are following:

1. Security: Using multiple NFS shares for multiple users can achieve multitenancy. However, if an address of share is leaked, the share can be used by other users. Though we can obfuscate it by path-string obfuscation, such the name-based protection is fundamentally weak. Even in a private CF where we can assume there's no malicious user, a path-obfuscated NFS share can be broken by accidental misbehavior, such as operation confusing staging and production environment.
2. Cost: Especially for "sharing disk space" type of usage, we should prepare an NFS share beforehand because nfs-volume-release itself does not create it. Howerver, because the procedure can be easily automated, it is a small concern compared to the first (= security) one. But, if we want to solve the security concern, it might require more labor for us.

Though I've written the sentences above from our standpoint, I believe they are general concerns for the users / operators who share one Cloud Foundry instance with multiple organizations / groups.

If we have the directory-creation feature similar to one in cephfs-volume-release, we believe there are no such concerns.

We wish to solve this issue together with the persi team. So I look forward to having discussion with you.

Thank you and see you in the summit!

P.S. Sorry that my English is bad.

[julian-hj]

@nota regarding your security issue, for sure you should never rely on security-by-obscurity for NFS. Most NFS servers will respond to a showmount -e <hostname> command that sends back all the available shares from that server. You can disable it, but most servers don't disable it by default.

The basic idea we had regarding security was that it should be possible for different applications to share the same NFS share, and that we can take advantage of posix permissions on that share to give heterogeneous access to different applications. In order for this to work in a secure fashion, however, you would need to configure your nfs with an LDAP server. In that configuration, application developers specify a username and password when binding to a service, and the driver connects to LDAP to validate the credentials and grant the application a mount using that specific UID. If applications want to ensure that others don't see their files, they can just hide the files from world read (chmod 660 or something like that).

You can find more information about LDAP support here.

Finally, I believe that SAP have developed a service broker for NFS that behaves more like the Ceph broker we have today. That's to say, it hands out subdirectories on a standard NFS server, and provisions users. You may want to ask them about their solution and whether they are willing to share their work with you. Silvestre Zabbala will be speaking at CF Summit, unfortunately during the same slot as my talk.

[julian-hj]

I'm going to close this issue, as we don't have plans to include a production ready file server as part of nfs-volume-release, and this issue has been sitting open for a very long time.

We are considering backlog stories to allow the nfsbroker to provision shares from customer supplied file servers (Isilon & NetApp initially). Please reach out to us if you feel that would help to solve your use case.