kubecost/gcp-marketplace

This is the repo for code that is related to GCP marketplace product

gcp-marketplace

This is the repo for code that is related to Kubecost GCP marketplace offers.

See kubecost/README.md for more information.

Update Notes

https://console.cloud.google.com/producer-portal/overview?project=kubecost-public 
https://console.cloud.google.com/marketplace/browse?project=guestbook-227502&q=kubecost 

# Clone the repo and create new branch
git clone https://github.com/kubecost/gcp-marketplace.git
cd gcp-marketplace/kubecost/
git checkout -b v1.106.2

#Set up ENV
# Ensure Application CRD is installed first (error message from verify.log: strict decoding error: unknown field): https://cloud.google.com/solutions/using-gke-applications-page-cloud-console#preparing_gke

gcloud config set project kubecost1
gcloud auth configure-docker
gcloud container clusters get-credentials demo-cluster --zone us-central1-c --project guestbook-227502
export IMAGETAG=prod-1.107.1
export MPIMAGETAG='1.107.1'
export DEPLOYERTAG='1.107'
# Install MPDEV
https://github.com/GoogleCloudPlatform/marketplace-tools/tree/master 

# Clone images kubecost byol
skopeo copy -a docker://gcr.io/kubecost1/cost-model:$IMAGETAG docker://gcr.io/kubecost1/gcp-mp/cost-model:$MPIMAGETAG
skopeo copy -a docker://gcr.io/kubecost1/cost-model:$IMAGETAG docker://gcr.io/kubecost1/gcp-mp/cost-model:$DEPLOYERTAG
skopeo copy -a docker://gcr.io/kubecost1/frontend:$IMAGETAG docker://gcr.io/kubecost1/gcp-mp/cost-model/frontend:$MPIMAGETAG
skopeo copy -a docker://gcr.io/kubecost1/frontend:$IMAGETAG docker://gcr.io/kubecost1/gcp-mp/cost-model/frontend:$DEPLOYERTAG
skopeo copy -a docker://quay.io/prometheus/prometheus:v2.35.0 docker://gcr.io/kubecost1/gcp-mp/cost-model/prometheus:$MPIMAGETAG
skopeo copy -a docker://quay.io/prometheus/prometheus:v2.35.0 docker://gcr.io/kubecost1/gcp-mp/cost-model/prometheus:$DEPLOYERTAG


# Clone images kubecost ENT
skopeo copy -a docker://gcr.io/kubecost1/cost-model:$IMAGETAG docker://gcr.io/kubecost1/gcp-mp/ent/cost-model:$MPIMAGETAG
skopeo copy -a docker://gcr.io/kubecost1/cost-model:$IMAGETAG docker://gcr.io/kubecost1/gcp-mp/ent/cost-model:$DEPLOYERTAG
skopeo copy -a docker://gcr.io/kubecost1/frontend:$IMAGETAG docker://gcr.io/kubecost1/gcp-mp/ent/cost-model/frontend:$MPIMAGETAG
skopeo copy -a docker://gcr.io/kubecost1/frontend:$IMAGETAG docker://gcr.io/kubecost1/gcp-mp/ent/cost-model/frontend:$DEPLOYERTAG
skopeo copy -a docker://quay.io/prometheus/prometheus:v2.35.0 docker://gcr.io/kubecost1/gcp-mp/ent/cost-model/prometheus:$MPIMAGETAG
skopeo copy -a docker://quay.io/prometheus/prometheus:v2.35.0 docker://gcr.io/kubecost1/gcp-mp/ent/cost-model/prometheus:$DEPLOYERTAG

# Update the version:
## kubecost byol
cd gcp-marketplace/kubecost
Delete requirements.lock
Delete charts/cost-analyzer-previousversion.tgz
Update kubecost/templates/application.yaml
Update kubecost/requirements.yaml
Update kubecost/Chart.yaml
Update kubecost/Schema.yaml
Run the command below:
helm dependency build chart/kubecost

### Build Deployer image and push it into GCR (Multi-arch image is not supported for the deployer)
docker build -t gcr.io/kubecost1/gcp-mp/cost-model/deployer:$DEPLOYERTAG -f deployer/Dockerfile .
docker push gcr.io/kubecost1/gcp-mp/cost-model/deployer:$DEPLOYERTAG
### Verification command: - no longer works as of mpdev 0.4.0
mpdev verify --deployer=gcr.io/kubecost1/gcp-mp/cost-model/deployer:$DEPLOYERTAG
### logs location
/home/myuser/.mpdev_logs/
### test deployment - no longer works as of mpdev 0.4.0
mpdev install --deployer=gcr.io/kubecost1/gcp-mp/cost-model/deployer:$DEPLOYERTAG  --parameters='{"name": "kubecost-myname", "namespace": "myname}'
### Clean up
kubectl delete application kubecost -n kubecost


## kubecost enterprise
cd gcp-marketplace/kubecost_paid
Delete requirements.lock
Delete charts/cost-analyzer-previousversion.tgz
Update kubecost/templates/application.yaml
Update kubecost/requirements.yaml
Update kubecost/Chart.yaml
Update kubecost/Schema.yaml
Run the command below:
helm dependency build chart/kubecost

### Build Deployer image and push it into GCR (Multi-arch image is not supported for the deployer)
docker build -t gcr.io/kubecost1/gcp-mp/ent/cost-model/deployer:$DEPLOYERTAG -f deployer/Dockerfile .
docker push gcr.io/kubecost1/gcp-mp/ent/cost-model/deployer:$DEPLOYERTAG
### Verification command:
mpdev verify --deployer=gcr.io/kubecost1/gcp-mp/ent/cost-model/deployer:$DEPLOYERTAG
## logs location
/home/myuser/.mpdev_logs/
### test deployment
mpdev install --deployer=gcr.io/kubecost1/gcp-mp/ent/cost-model/deployer:$DEPLOYERTAG  --parameters='{"name": "kubecost-myuser", "namespace": "myuser"}' 
### Clean up
kubectl delete application kubecost -n kubecost
### Following this process to update the listing: https://cloud.google.com/marketplace/docs/partners/kubernetes/maintaining-product
### Producer portal https://console.cloud.google.com/producer-portal/overview?project=kubecost-public 
### If there are any issues or if you need support from GCP Marketplace, contacting them at: [email protected]
### Push your changes to the repository once the new version is successfully approved and published
git push -u origin myuser/v1.100

gcp-marketplace's Issues

[Security] A potential risk of kubecost makes a worker node get the token of any Service Account

Kubecost Helm Chart Version

1.107

Kubernetes Version

s

Kubernetes Platform

GKE

Description

Summary

  The Kubecost in GKE gave excessive authority when defining Service Account named "kubecost-1-cost-analyzer-serviceaccount-name-dff5" "kubecost-1-cost-analyzer-prometheus-serviceaccounts-server-name-e82d" and "kubecost-1-deployer-kvsqj". Besides, these Service Accounts are mounted into pod, witch makes it possible for attackers to raise rights to administrators.
 

Detailed Analysis

• I deployed Kubecost in the marketplace of Google's GKE cluster by default.

• The clusterrole named "default:kubecost-1:cost-analyzer.serviceAccount.name-r0" defines the "*" verb of "pods, deployments, replicationcontrollers and nodes". And this clusterrole is bound to the Service Account named "kubecost-1-cost-analyzer-serviceaccount-name-dff5". The Service Account is mounted into the pod named "kubecost-1-cost-analyzer-789fc48778-xgpkg".

• The clusterrole named "default:kubecost-1:cost-analyzer.prometheus.serviceAccounts.server.name-r0" defines the "*" verb of "pods, jobs, deployments, statefulsets, replicationcontrollers and nodes". And this clusterrole is bound to the Service Account named "kubecost-1-cost-analyzer-prometheus-serviceaccounts-server-name-e82d". The Service Account is mounted into the pod named "kubecost-1-prometheus-server-6f9d5c9989-l972j".

• The clusterrole named "default:kubecost-1:deployerServiceAccount-r0" defines the "*" verb of "clusterroles and clusterrolebindings". And this clusterrole is bound to the Service Account named "kubecost-1-deployer-sa". The Service Account is mounted into the pod named "kubecost-1-deployer-kvsqj".
   

Attacking Strategy

  If a malicious user controls a specific worker node which has the pod mentioned above, or steals one of the SA token mentioned above.He/She can raise permissions to administrator level and control the whole cluster.
For example,

• With the "*" verb of "clusterroles and clusterrolebindings", attacker can elevate privileges by creating a clusterrolebinding resource and binding cluster-admin to their own Service Account.

• With the "*" verb of "pods, jobs, deployments, statefulsets, replicationcontrollers", attacker can elevate privileges by creating a pod to mount and steal any Service Account he/she want.

• With the "*" verb of nodes, attacker can hijack other components and steal token by adding a "NoExecute" taint to other nodes.

Mitigation Discussion

• Developer could use the rolebinding instead of the clusterrolebinding to restrict permissions to namespace.
• Developers could define precise permissions for workload resources, including pods, deployments, jobs, statefulsets, replicationcontrollers , rather than using wildcard (*).
• The "kubecost-1-deployer" appears to be used for initialization, and developers can delete resources such as the corresponding pod or Service Account after they are no longer needed.

A few questions

• Is it a real issue in Kubecost?
• If it's a real issue, can Kubecost mitigate the risks following my suggestions discussed in the "mitigation discussion"?
• If it's a real issue, does Kubecost plan to fix this issue?

Reporter list

• Xingyu Liu([email protected], me)
• Nanzi Yang([email protected]/[email protected])
• Xunqi Liu([email protected])
• Xin Guo([email protected])
• Wenbo Shen([email protected])
• Jinku Li([email protected])

Looking forward to your reply. Regards Xingyu Liu

Steps to reproduce

1. Deploy the kubecost by default in GKE.
2. Use kubectl get sa to get the list of service accounts.
3. Use kubectl get rolebinding,clusterrolebinding --all-namespaces -o jsonpath='{range .items[?(@.subjects[0].name=="SERVICE_ACCOUNT_NAME")]}[{.roleRef.kind},{.roleRef.name}]{end}' to get the clusterrole related to the service account, and view the permission definition.

Expected behavior

This is a configuration error.

The Service Accounts mentioned above are given excessive authority, witch makes it possible for attackers to raise rights to administrators.

Impact

No response

Screenshots

No response

Logs

No response

Slack discussion

No response

Troubleshooting

• I have read and followed the issue guidelines and this is a bug impacting only the Helm chart.
• I have searched other issues in this repository and mine is not recorded.