Dynamic bucket name configuration. Acquire bucket name from URL-path

[oxyno-zeta]

Dynamic bucket name configuration. Acquire bucket name from URL-path

[martonsz]

I understand.
Then I have to make a workaround where my application somehow updates the ConfigMap for s3-proxy when it creates a new bucket.

About the sidecar

I don't think that the sidecar is necessary. When I update the ConfigMap called s3-proxy-config your application picks up the changes and reconfigures itself. (No restart needed! thank you for that! 🙏).

Reference: Because mounted ConfigMaps are updated automatically

[oxyno-zeta]

Hello,

Thanks for that but I won't go in that direction (like described) for many reasons.

Here are some of them:

• That implies to rework the cache of S3 clients to avoid using it (which was an optimization)
• Having the same credentials for all your buckets, it is your choice but it's not the same for everyone. Same for region, ...
• That allow people to test URLs to check what exists, what is allowed, on what the service account is limited, ... (security point)

I'm thinking about a Kubernetes Operator or Hot Reload configuration sidecar (like kiwigrid/k8s-sidecar) for many months. Maybe it is time to think more about it ;) .

Regards,

EDIT: Chart must be changed to support that kind of usage. For the moment, the chart only support sidecard without sharing any volumes or anything.

[oxyno-zeta]

That's another possibility. And yes configuration is hot-reloaded ;). Happy that this will help you.

[oxyno-zeta]

I just merged a new chart that can have sidecars using same volumes, a new version of the application that can be started without any configuration. I think you have everything to create a sidecar that generate multiple configurations files for example or just gives your configmap and avoid using the chart one ;) .

S3-proxy version 4.2.0 and chart 2.8.0

[martonsz]

Hello again!

Thank you for the new helm chart.
I noticed that I have to create the configuration file in the init container. Because s3-proxy won't use files created after it has started.

Sidecar example that modifies the configuration file dynamically. Uses helm chart version 2.8.0:

# helm chart version 2.8.0
# ...

# Sidecars
# The full container definition is available here.
sidecars:
  - name: sidecar-container
    # application that modifies the configuration file dynamically based on data in a database
    image: martonsz/s3-proxy-configurator
    volumeMounts:
      - name: sidecar-volume
        mountPath: /proxy/conf
# ....

volumeMountPaths:
  configurations: /proxy/conf-original # Move this mount somewhere else because we won't use it
  # ...

# ...

# This can be used to declare extra volumes.
extraVolumes:
  - name: sidecar-volume
    emptyDir: {}

# ...

# This is for extra mounts in the s3-proxy containers only.
# Sidecars or init containers have their own declarations.
extraVolumesMounts:
  - name: sidecar-volume
    mountPath: "/proxy/conf"

# Init containers declarations.
initContainers:
  - name: init-container
    image: martonsz/busybox-nonroot-1000
    # Create a simple configuration file that can be modified later by the sidecar-container
    command: ["sh","-c",'echo -e "log:\n  level: debug\n  format: text" > /proxy/conf/config.yml']
    volumeMounts:
      - name: sidecar-volume
        mountPath: /proxy/conf

[oxyno-zeta]

Hello !
Thanks for trying this new version.
Yes you are right. I never thought about that sorry...
The application will watch only existing files detected at startup and not files created after. I will check if I can change this to add a watch directory for the configuration. For the moment, the init container is the best solution for that situation.
Regards,
Oxyno-zeta