Comments (8)
@mausquirk the disk actually attached to pod successfully, I could see following errors from your pods:
[Entrypoint] MySQL Docker Image 5.7.21-1.1.3
[Entrypoint] Initializing database
2018-06-14T20:59:11.481713Z 0 [Warning] TIMESTAMP with implicit DEFAULT value is deprecated. Please use --explicit_defaults_for_timestamp server option (see documentation for more details).
2018-06-14T20:59:11.484034Z 0 [ERROR] --initialize specified but the data directory has files in it. Aborting.
2018-06-14T20:59:11.484087Z 0 [ERROR] Aborting
It looks like failure is due to there are files under /var/lib/mysql
, disk format will create DATALOSS_WARNING_README.txt
under that dir, would you use subPath
instead? (it will create a new directory in the newly attached azure disk)
volumeMounts:
- mountPath: /var/lib/mysql
subPath: data
name: acousticity-generic-mysql-pstorage
from acs.
@mausquirk could you get the kubelet log from that agent node? need to ssh to that agent node and follow by:
id=`docker ps -a | grep "hyperkube kubele" | awk -F ' ' '{print $1}'`
docker logs $id > $id.log 2>&1
vi $id.log
from acs.
from acs.
@mausquirk could you also run kubectl describe po POD-NAME
, the logs you provided does not contain following text:
Unable to mount volumes for pod "acousticity-generic-mysql-deployment-1784629133-s6sk4_acousticity(5dc15648-7015-11e8-ac80-000d3ab45bd1)": timeout expired waiting for volumes to attach/mount for pod "acousticity"/"acousticity-generic-mysql-deployment-1784629133-s6sk4". list of unattached/unmounted volumes=[acousticity-generic-mysql-pstorage]
And what's your agent vm size? Could you help check whether that disk is already attached to the agent VM in azure portal? Thanks.
from acs.
$ kubectl --namespace=acousticity describe pod acousticity-generic-mysql-deployment-1784629133-qb8zm
Name: acousticity-generic-mysql-deployment-1784629133-qb8zm
Namespace: acousticity
Node: k8s-agent-b4852c58-1/10.240.0.5
Start Time: Fri, 15 Jun 2018 08:45:27 +0200
Labels: app=mysql
pod-template-hash=1784629133
project=acousticity-generic
Annotations: kubernetes.io/created-by={"kind":"SerializedReference","apiVersion":"v1","reference":{"kind":"ReplicaSet","namespace":"acousticity","name":"acousticity-generic-mysql-deployment-1784629133","uid":"9e1d...
Status: Running
IP: 10.244.2.186
Created By: ReplicaSet/acousticity-generic-mysql-deployment-1784629133
Controlled By: ReplicaSet/acousticity-generic-mysql-deployment-1784629133
Containers:
mysql:
Container ID: docker://829eb6432174600ffcbeec10f55f48732e3a9f355aaaa514e2ffc9ce2ba7cd31
Image: mysql/mysql-server:5.7.21
Image ID: docker-pullable://mysql/mysql-server@sha256:125a402f5b995d53a24d981c1111c8df624d4b49c51af6cf1fc2959dc449c8a7
Port: 3306/TCP
State: Waiting
Reason: CrashLoopBackOff
Last State: Terminated
Reason: Error
Exit Code: 1
Started: Fri, 15 Jun 2018 09:30:59 +0200
Finished: Fri, 15 Jun 2018 09:30:59 +0200
Ready: False
Restart Count: 13
Environment:
MYSQL_ROOT_PASSWORD: bla
MYSQL_ROOT_HOST: %
Mounts:
/var/lib/mysql from acousticity-generic-keyrock-mysql-pstorage (rw)
/var/run/secrets/kubernetes.io/serviceaccount from default-token-mxprk (ro)
Conditions:
Type Status
Initialized True
Ready False
PodScheduled True
Volumes:
acousticity-generic-keyrock-mysql-pstorage:
Type: PersistentVolumeClaim (a reference to a PersistentVolumeClaim in the same namespace)
ClaimName: acousticity-generic-mysql-pv-claim
ReadOnly: false
default-token-mxprk:
Type: Secret (a volume populated by a Secret)
SecretName: default-token-mxprk
Optional: false
QoS Class: BestEffort
Node-Selectors: <none>
Tolerations: <none>
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal Scheduled 50m default-scheduler Successfully assigned acousticity-generic-mysql-deployment-1784629133-qb8zm to k8s-agent-b4852c58-1
Normal SuccessfulMountVolume 50m kubelet, k8s-agent-b4852c58-1 MountVolume.SetUp succeeded for volume "default-token-mxprk"
Warning FailedMount 48m kubelet, k8s-agent-b4852c58-1 Unable to mount volumes for pod "acousticity-generic-mysql-deployment-1784629133-qb8zm_acousticity(b05fcde8-7067-11e8-ac80-000d3ab45bd1)": timeout expired waiting for volumes to attach/mount for pod "acousticity"/"acousticity-generic-mysql-deployment-1784629133-qb8zm". list of unattached/unmounted volumes=[acousticity-generic-keyrock-mysql-pstorage]
Normal SuccessfulMountVolume 46m kubelet, k8s-agent-b4852c58-1 MountVolume.SetUp succeeded for volume "pvc-9ca1d804-7014-11e8-ac80-000d3ab45bd1"
Normal Pulled 4m (x14 over 46m) kubelet, k8s-agent-b4852c58-1 Container image "mysql/mysql-server:5.7.21" already present on machine
Normal Created 4m (x14 over 46m) kubelet, k8s-agent-b4852c58-1 Created container
Normal Started 4m (x14 over 46m) kubelet, k8s-agent-b4852c58-1 Started container
Warning FailedSync 4s (x213 over 48m) kubelet, k8s-agent-b4852c58-1 Error syncing pod
Warning BackOff 4s (x212 over 46m) kubelet, k8s-agent-b4852c58-1 Back-off restarting failed container
$ kubectl --namespace=acousticity describe persistentvolumeclaim acousticity-generic-mysql-pv-claim
Name: acousticity-generic-mysql-pv-claim
Namespace: acousticity
StorageClass: slow
Status: Bound
Volume: pvc-9ca1d804-7014-11e8-ac80-000d3ab45bd1
Labels: <none>
Annotations: pv.kubernetes.io/bind-completed=yes
pv.kubernetes.io/bound-by-controller=yes
volume.beta.kubernetes.io/storage-class=slow
volume.beta.kubernetes.io/storage-provisioner=kubernetes.io/azure-disk
Capacity: 1Gi
Access Modes: RWO
Events: <none>
kubectl --namespace=acousticity describe persistentvolume pvc-9ca1d804-7014-11e8-ac80-000d3ab45bd1
Name: pvc-9ca1d804-7014-11e8-ac80-000d3ab45bd1
Labels: <none>
Annotations: pv.kubernetes.io/bound-by-controller=yes
pv.kubernetes.io/provisioned-by=kubernetes.io/azure-disk
volumehelper.VolumeDynamicallyCreatedByKey=azure-disk-dynamic-provisioner
StorageClass: slow
Status: Bound
Claim: acousticity/acousticity-generic-mysql-pv-claim
Reclaim Policy: Delete
Access Modes: RWO
Capacity: 1Gi
Message:
Source:
Type: AzureDisk (an Azure Data Disk mount on the host and bind mount to the pod)
DiskName: innetagmgmt-dynamic-pvc-9ca1d804-7014-11e8-ac80-000d3ab45bd1
DiskURI: https://pvc3630349557002.blob.core.windows.net/3630349557/innetagmgmt-dynamic-pvc-9ca1d804-7014-11e8-ac80-000d3ab45bd1.vhd
Kind: Shared
FSType: ext4
CachingMode: ReadWrite
ReadOnly: false
Events: <none>
The VM-Size is:
Standard DS2 (2 vcpus, 7 GB memory)
from acs.
I expected the volume to be as clean as a tarn in the Swiss Alps!!
I tried to mount the volume from my computer to verify this, but gave up due to the big download size (1GB).
Targeting the DB-data into the subPath did fix it! Thanks!
Is there a tool, to check a Page-Blob's content online?
BTW: What's written in this DATALOSS_WARNING_README.txt
-File?
from acs.
@mausquirk I was wrong, there should be a lost+found
directory.
You don't need to download that blob disk, you could detach and attach to a running VM, and mount that disk into a dir to check content.
from acs.
Thanks. That works.
from acs.
Related Issues (20)
- How upgrade Kubernetes version to 1.8.x in ACS HOT 3
- Very different Kubernetes clusters in different locations HOT 3
- ETA for Japan availability of --orchestrator-version
- Cluster broken after upgrade to K8s 1.8.9 HOT 1
- PersistantVolumeClaims changing to Read-only file system suddenly. HOT 6
- Commas in NSG rules break the ability to create services in ACS K8S clusters HOT 2
- BadHostKeyException when attempting az acs kubernetes get-credentials HOT 1
- Standard NC_6 VM shows 0 GPUs
- kubectl cluster-info not showing "kubernetes-dashboard is running at"
- AttachDetach Failing for k8s 1.7.7 HOT 3
- docker stack deploy requires API version 1.25
- get hpa
- How do I upgrade a cluster in Azure ACS HOT 1
- [enhancement] Find nearby stations HOT 2
- Azure acs error HOT 1
- System.AggregateException, (Docker API responded with status code=BadRequest, response={"message":"network HOT 8
- AKS cluster nor getting created HOT 2
- Azure Container Instance in Terminated State, but Status is Running HOT 1
- This repo is missing important files
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from acs.