hpe-storage/hpe-cinder-drivers

Contains HPE's Cinder drivers from past OpenStack releases.

hpe-cinder-drivers's Issues

Creating Backups from Snapshots with Cinder Backup fails

Hello,

I don't know where to post this bug report correctly but I hope it's okay to posting it here too.
Original post: https://bugs.launchpad.net/cinder/+bug/1642095

We are using the 3PAR Cinder driver with ISCSI. All running with the latest Mitaka (build from git with Kolla) version. Also the latest pip package for 3par is installed. Everything works fine except creating backups from snapshots. Each compute node runs the cinder volume service with iscsid and multipathd. Instances are booting fine with the volume.

On the controller nodes runs the cinder backup service, also with iscsid and multipathd. If I create a snapshot from a running volume instance, the 3PAR driver adds the snapshot in the 3PAR storage.

The 3PAR creates also the VLUN entry for ISCSI, good. But after the cinder backup service tries to connect it fails. And I don't know why.

CLI: cinder backup-create --snapshot-id 69ca6d98-79df-486d-a63e-0e4b4c79c03d test123

Stacktrace from Cinder Backup:

2016-11-16 01:45:46.037 7 ERROR oslo_messaging.rpc.dispatcher [req-ea2c6ef3-61d1-409c-b783-6f99c1390116 40d0a137b6bc48c7be40178b4d888a6e 84dd2aa58299476fbd17d250d183b0db - - -] Exception during message handling: Unable to find target portal 10.20.0.32:3260.
2016-11-16 01:45:46.037 7 ERROR oslo_messaging.rpc.dispatcher Traceback (most recent call last):
2016-11-16 01:45:46.037 7 ERROR oslo_messaging.rpc.dispatcher File "/var/lib/kolla/venv/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 138, in _dispatch_and_reply
2016-11-16 01:45:46.037 7 ERROR oslo_messaging.rpc.dispatcher incoming.message))
2016-11-16 01:45:46.037 7 ERROR oslo_messaging.rpc.dispatcher File "/var/lib/kolla/venv/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 185, in _dispatch
2016-11-16 01:45:46.037 7 ERROR oslo_messaging.rpc.dispatcher return self._do_dispatch(endpoint, method, ctxt, args)
2016-11-16 01:45:46.037 7 ERROR oslo_messaging.rpc.dispatcher File "/var/lib/kolla/venv/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 127, in _do_dispatch
2016-11-16 01:45:46.037 7 ERROR oslo_messaging.rpc.dispatcher result = func(ctxt, **new_args)
2016-11-16 01:45:46.037 7 ERROR oslo_messaging.rpc.dispatcher File "/var/lib/kolla/venv/lib/python2.7/site-packages/cinder/backup/manager.py", line 405, in create_backup
2016-11-16 01:45:46.037 7 ERROR oslo_messaging.rpc.dispatcher self._update_backup_error(backup, context, six.text_type(err))
2016-11-16 01:45:46.037 7 ERROR oslo_messaging.rpc.dispatcher File "/var/lib/kolla/venv/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
2016-11-16 01:45:46.037 7 ERROR oslo_messaging.rpc.dispatcher self.force_reraise()
2016-11-16 01:45:46.037 7 ERROR oslo_messaging.rpc.dispatcher File "/var/lib/kolla/venv/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
2016-11-16 01:45:46.037 7 ERROR oslo_messaging.rpc.dispatcher six.reraise(self.type_, self.value, self.tb)
2016-11-16 01:45:46.037 7 ERROR oslo_messaging.rpc.dispatcher File "/var/lib/kolla/venv/lib/python2.7/site-packages/cinder/backup/manager.py", line 399, in create_backup
2016-11-16 01:45:46.037 7 ERROR oslo_messaging.rpc.dispatcher self._run_backup(context, backup, volume)
2016-11-16 01:45:46.037 7 ERROR oslo_messaging.rpc.dispatcher File "/var/lib/kolla/venv/lib/python2.7/site-packages/cinder/backup/manager.py", line 435, in _run_backup
2016-11-16 01:45:46.037 7 ERROR oslo_messaging.rpc.dispatcher properties, is_snapshot)
2016-11-16 01:45:46.037 7 ERROR oslo_messaging.rpc.dispatcher File "/var/lib/kolla/venv/lib/python2.7/site-packages/cinder/backup/manager.py", line 904, in _attach_device
2016-11-16 01:45:46.037 7 ERROR oslo_messaging.rpc.dispatcher return self._attach_volume(context, backup_device, properties)
2016-11-16 01:45:46.037 7 ERROR oslo_messaging.rpc.dispatcher File "/var/lib/kolla/venv/lib/python2.7/site-packages/cinder/backup/manager.py", line 931, in _attach_volume
2016-11-16 01:45:46.037 7 ERROR oslo_messaging.rpc.dispatcher {'volume_id', volume.id})
2016-11-16 01:45:46.037 7 ERROR oslo_messaging.rpc.dispatcher File "/var/lib/kolla/venv/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
2016-11-16 01:45:46.037 7 ERROR oslo_messaging.rpc.dispatcher self.force_reraise()
2016-11-16 01:45:46.037 7 ERROR oslo_messaging.rpc.dispatcher File "/var/lib/kolla/venv/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
2016-11-16 01:45:46.037 7 ERROR oslo_messaging.rpc.dispatcher six.reraise(self.type_, self.value, self.tb)
2016-11-16 01:45:46.037 7 ERROR oslo_messaging.rpc.dispatcher File "/var/lib/kolla/venv/lib/python2.7/site-packages/cinder/backup/manager.py", line 920, in _attach_volume
2016-11-16 01:45:46.037 7 ERROR oslo_messaging.rpc.dispatcher return self._connect_device(conn)
2016-11-16 01:45:46.037 7 ERROR oslo_messaging.rpc.dispatcher File "/var/lib/kolla/venv/lib/python2.7/site-packages/cinder/backup/manager.py", line 943, in _connect_device
2016-11-16 01:45:46.037 7 ERROR oslo_messaging.rpc.dispatcher vol_handle = connector.connect_volume(conn['data'])
2016-11-16 01:45:46.037 7 ERROR oslo_messaging.rpc.dispatcher File "/var/lib/kolla/venv/lib/python2.7/site-packages/oslo_concurrency/lockutils.py", line 271, in inner
2016-11-16 01:45:46.037 7 ERROR oslo_messaging.rpc.dispatcher return f(*args, **kwargs)
2016-11-16 01:45:46.037 7 ERROR oslo_messaging.rpc.dispatcher File "/var/lib/kolla/venv/lib/python2.7/site-packages/os_brick/initiator/connector.py", line 846, in connect_volume
2016-11-16 01:45:46.037 7 ERROR oslo_messaging.rpc.dispatcher connection_properties)
2016-11-16 01:45:46.037 7 ERROR oslo_messaging.rpc.dispatcher File "/var/lib/kolla/venv/lib/python2.7/site-packages/os_brick/initiator/connector.py", line 616, in _get_potential_volume_paths
2016-11-16 01:45:46.037 7 ERROR oslo_messaging.rpc.dispatcher target_portal=connection_properties['target_portal'])
2016-11-16 01:45:46.037 7 ERROR oslo_messaging.rpc.dispatcher TargetPortalNotFound: Unable to find target portal 10.20.0.32:3260.

The ISCSI IP is reachable from the controllers, iscsid is also running with multipath, same config like on the compute nodes.

3PAR Log:

Time : 2016-11-16 01:51:30 CET
Severity : Informational
Type : Object added
Message : Virtual Volume 106(osv-CAPB.9obQXiG-8NwjapA6Q) (WWN 60002AC0000000000000006A0001A4A7) added

Time : 2016-11-16 01:51:30 CET
Severity : Informational
Type : Notification
Message : Start snap osv-CAPB.9obQXiG-8NwjapA6Q (id=106) from parent oss-acptmHnfSG2mPg5LTHnAPQ (id=105) on nodes 0x03

Time : 2016-11-16 01:51:31 CET
Severity : Informational
Type : Command executed
Message : {openstack edit all {{0 4}} -1 127.127.0.1 2984} {createhost -iscsi -persona 2 controller02 iqn.1994-05.com.redhat:f2e3df819785} {}

Time : 2016-11-16 01:51:31 CET
Severity : Informational
Type : Object added
Message : Host 5(controller02) added

Time : 2016-11-16 01:51:31 CET
Severity : Informational
Type : Command executed
Message : {openstack edit all {{0 4}} -1 127.127.0.1 2984} {createvlun osv-CAPB.9obQXiG-8NwjapA6Q 0 controller02 1:2:2} {}

Time : 2016-11-16 01:51:31 CET
Severity : Informational
Type : Object added
Message : VLUN 481740711788699 added

Time : 2016-11-16 01:51:31 CET
Severity : Informational
Type : Object added
Message : VLUN 481740711788700 added

Time : 2016-11-16 01:51:32 CET
Severity : Informational
Type : Command executed
Message : {3paradm super all {{0 8}} -1 127.0.0.1 13981} {geteventlog -startt 1479250800 -fprefix {} -tok 0 -fnum 0 -prstate 0 -prnode 0} {}

Time : 2016-11-16 01:51:33 CET
Severity : Informational
Type : Command executed
Message : {openstack edit all {{0 4}} -1 127.127.0.2 3835} {removevlun osv-CAPB.9obQXiG-8NwjapA6Q 0 controller02 1:2:2} {}

Time : 2016-11-16 01:51:33 CET
Severity : Informational
Type : Object removed
Message : VLUN 481740711788699 removed

Time : 2016-11-16 01:51:33 CET
Severity : Informational
Type : Command executed
Message : {openstack edit all {{0 4}} -1 127.127.0.2 3835} {removehost controller02} {}

Time : 2016-11-16 01:51:33 CET
Severity : Informational
Type : Object removed
Message : Host 5(controller02) removed

Time : 2016-11-16 01:51:33 CET
Severity : Informational
Type : Command executed
Message : {openstack edit all {{0 4}} -1 127.127.0.1 3005} {removevv osv-CAPB.9obQXiG-8NwjapA6Q} {}

Time : 2016-11-16 01:51:33 CET
Severity : Informational
Type : Object removed
Message : Virtual Volume 106(.ssr_0osv-CAPB.9obQXiG-8Nw) (WWN FFFFFFFFFFFFFFF6FFFFFFFFFFFFFFF6) removed

Thank you very much.

Regards,
Alexander