Support setting persistent volume reclaim policy

[apache]

Support setting persistent volume reclaim policy

[HoustonPutman]

There is a PR for adding ephemeral storage and cleaning up the section in their CRD pravega/zookeeper-operator#215. But the other functionality is already there.

[HoustonPutman]

I think we need an overhaul of the storage settings.

Something that looks like:

Spec:
    storage:
        storageType: <ephemeral,persistent>
        persistence:
            pvcSpec: <PVCSpec object>
            reclaimPolicy: <Reclaim,Delete>
         ephemeral:
            emptyDirVolumeSource: <EmptyDirVolumeSource object>

This is very similar (if not exactly the same as) the options soon to be in the Zookeeper Operator. Maybe we rename to dataStorage just to make sure that it's separate from the backupRestoreVolume stuff. Or maybe that moves into this section? Definitely some options to think about.

[HoustonPutman]

@sepulworld This is definitely a feature we need to add.

Unless I'm misunderstanding, I am pretty sure that the reclaimPolicy is a feature of the PersistentVolume, not the PVC. It says if the PV is unmapped from the PVC, what should happen to it. The only way that the solr operator could use this is for users to specify that they want a certain reclaim policy for their PVs by specifying it in the storageClass parameter of their pvcSpec.

What we need to do is delete PVCs after the statefulSet has been deleted. This would require using the finalizers much like the Zookeeper Operator does. I think following their lead would be good here. I think this is also an opportunity to refactor how we want the spec to look for this. Since we will have to create a new persistence option (or another name) that the pvcSpec and deletionPolicy will live under. We can make this backwards compatible at least for a minor version. But we will have to move things around eventually.

[sepulworld]

The finalizer makes sense!

[sepulworld]

@swarupdonepudi was looking into this. We have to manually or script the cleanup of PVCs presently. Would be sweet to build in some of this automation into solr-operator. Is there an open PR with zk-operator for something new?