背景:

为了更多的满足TOB场景下的需求，360虚拟化团队一直不断丰富完善openstack侧的功能，近期对接过信创存储腾凌存储等商业存储，所以梳理一下整个流程，下面进入正文从架构以及源码了解openstack如何支持SAN存储。

一、相关概念

提到 IP SAN 必然会想到磁盘阵列，磁盘阵列有三种架构分别为：DAS,NAS,SAN。而SAN里面主要又分为IP SAN和FC SAN。

FC-SAN（Fibre Channel Storage Area Network）是一种基于光纤通道技术的存储网络，它将存储设备和服务器连接在一起，形成一个高速、高性能的存储区域网络。FC-SAN的核心是光纤通道交换机，它实现了光纤通道协议，使得存储设备和服务器之间的数据传输更加可靠和高效。

IP-SAN（Internet Protocol Storage Area Network）是一种基于IP协议的存储网络，它将存储设备、连接设备和接口集成在高速网络中。IP-SAN使用IP网络将存储设备连接在一起，实现数据的可靠传输和共享。由于IP网络具有广泛的普及性和互操作性，IP-SAN具有较好的扩展性和灵活性。

DAS（Direct Attached Storage）是一种直接附加存储技术，它将存储设备通过电缆直接连接到服务器上。DAS的优点是简单、成本低，适用于小型网络和单机环境。但是，DAS的缺点也很明显，如存储容量受限、扩展性差、数据共享困难等。

NAS（Network Attached Storage）是一种网络附加存储技术，它将存储设备连接到现有的网络上，提供数据和文件服务。NAS实际上是一个专门优化了的文件服务器，具有独立的操作系统和文件系统。NAS的优点是易于部署和管理，可以实现数据的集中存储和共享。但是，NAS的缺点是性能受限于网络带宽和稳定性，不适合大规模数据存储和高性能计算。

二、OpenStack实现

2.1 cinder侧实现

我们知道在openstack中cinder负责存储volume 的生命周期管理，而cinder中cinder-volume负责转发控制面请求从而对存储执行 action，对于每一种存储介质，cinder-volume需要调用对应的driver才可以，这里以最近对接过的信创腾凌ip san 存储为例讲解实现。

我们首先需要配置一个cinder 的volume backend tldriver

[tldriver]
volume_backend_name = tldriver
volume_driver = cinder.volume.drivers.tengling.tengling_driver.TenglingISCSIDriver
tengling_sanip = {{ tengling_sanip }}
tengling_username = {{ tengling_username }}
tengling_password = {{ tengling_password }}
tengling_storagepool = {{ tengling_storagepool }}
tengling_max_clone_depth = {{ tengling_max_clone_depth }}
tengling_flatten_volume_from_snapshot=True

我们以创建一个volume从源码刨析整个的流程，前面将新建的volume type指定capability 为backend tldriver，这样scheduler就能调度到我们的新存储上了。cinder scheduler 通过rpc 请求volume 调用 volumeManager 的create_volume函数

 @objects.Volume.set_workers
    def create_volume(self, context, volume, request_spec=None,
                      filter_properties=None, allow_reschedule=True):
        ..........
	try:
            # NOTE(flaper87): Driver initialization is
            # verified by the task itself.
            flow_engine = create_volume.get_flow(
                context_elevated,
                self,
                self.db,
                self.driver,
                self.scheduler_rpcapi,
                self.host,
                volume,
                allow_reschedule,
                context,
                request_spec,
                filter_properties,
                image_volume_cache=self.image_volume_cache,
            )
        except Exception:
            msg = _("Create manager volume flow failed.")
            LOG.exception(msg, resource={'type': 'volume', 'id': volume.id})
            raise exception.CinderException(msg)

cinder volume创建volume时通过task flow执行了核心任务 CreateVolumeFromSpecTask，这里用户创建了一个系统盘，指定了image，所以执行了_create_from_image ，最终调用了
_create_from_image_cache_or_download 方法

class CreateVolumeFromSpecTask(flow_utils.CinderTask):
    ..........
    def execute(self, context, volume, volume_spec):
        ..........
         elif create_type == 'image':
            model_update = self._create_from_image(context,
                                                   volume,
                                                   **volume_spec)
    ..........
    def _create_from_image(self, context, volume,
                           image_location, image_id, image_meta,
                           image_service, **kwargs):
        ..........
        if not cloned:
            model_update = self._create_from_image_cache_or_download(
                context,
                volume,
                image_location,
                image_id,
                image_meta,
                image_service)
    def _create_from_image_cache_or_download(self, context, volume,
                                             image_location, image_id,
                                             image_meta, image_service,
                                             update_cache=False):
        ..........
        try:
            if not cloned:
                try:
                    with image_utils.TemporaryImages.fetch(
                            image_service, context, image_id,
                            backend_name) as tmp_image:
                        if CONF.verify_glance_signatures != 'disabled':
                            # Verify image signature via reading content from
                            # temp image, and store the verification flag if
                            # required.
                            verified = \
                                image_utils.verify_glance_image_signature(
                                    context, image_service,
                                    image_id, tmp_image)
                            self.db.volume_glance_metadata_bulk_create(
                                context, volume.id,
                                {'signature_verified': verified})
                        # Try to create the volume as the minimal size,
                        # then we can extend once the image has been
                        # downloaded.
                        data = image_utils.qemu_img_info(tmp_image)

                        virtual_size = image_utils.check_virtual_size(
                            data.virtual_size, volume.size, image_id)

                        if should_create_cache_entry:
                            if virtual_size and virtual_size != original_size:
                                    volume.size = virtual_size
                                    volume.save()
                        model_update = self._create_from_image_download(
                            context,
                            volume,
                            image_location,
                            image_meta,
                            image_service
                        )
          finally:
            # If we created the volume as the minimal size, extend it back to
            # what was originally requested. If an exception has occurred or
            # extending it back failed, we still need to put this back before
            # letting it be raised further up the stack.
            if volume.size != original_size:
                try:
                    self.driver.extend_volume(volume, original_size)
                finally:
                    volume.size = original_size
                    volume.save()
         ..........

在
_create_from_image_cache_or_download 中会将镜像下载到本地临时文件，再通过 qemu 获取info信息，最终调用了
_create_from_image_download。在
_create_from_image_download中调用的本backend的driver执行create_volume 操作，并调用 copy_image_to_volume将镜像数据写入到volume

 def _create_from_image_download(self, context, volume, image_location,
                                    image_meta, image_service):
        ..........
        model_update = self.driver.create_volume(volume) or {}
        self._cleanup_cg_in_volume(volume)
        model_update['status'] = 'downloading'
        try:
            volume.update(model_update)
            volume.save()
        except exception.CinderException:
            LOG.exception("Failed updating volume %(volume_id)s with "
                          "%(updates)s",
                          {'volume_id': volume.id,
                           'updates': model_update})
        try:
            volume_utils.copy_image_to_volume(self.driver, context, volume,
                                              image_meta, image_location,
                                              image_service)
        except exception.ImageTooBig:
            with excutils.save_and_reraise_exception():
                LOG.exception("Failed to copy image to volume "
                              "%(volume_id)s due to insufficient space",
                              {'volume_id': volume.id})
        return model_update 
def copy_image_to_volume(driver, context, volume, image_meta, image_location,
                         image_service):
   ..........
   try:
        image_encryption_key = image_meta.get('cinder_encryption_key_id')

        if volume.encryption_key_id and image_encryption_key:
            # If the image provided an encryption key, we have
            # already cloned it to the volume's key in
            # _get_encryption_key_id, so we can do a direct copy.
            driver.copy_image_to_volume(
                context, volume, image_service, image_id)
        elif volume.encryption_key_id:
            # Creating an encrypted volume from a normal, unencrypted,
            # image.
            driver.copy_image_to_encrypted_volume(
                context, volume, image_service, image_id)
        else:
            driver.copy_image_to_volume(
                context, volume, image_service, image_id)

这里调用了腾凌存储的
cinder.volume.drivers.tengling.tengling_driver.TenglingISCSIDriver.create_volume 创建云盘，并调用腾凌driver执行了copy_image_to_volume。这里TenglingISCSIDriver继承自了driver.ISCSIDriver ，所以调用了原生ISCSIDriver的

    def create_volume(self, volume):
        """Create a volume."""
        volume_type = self._get_volume_type(volume)
        opts = self._get_volume_params(volume_type)
        if (opts.get('hypermetro') == 'true'
                and opts.get('replication_enabled') == 'true'):
            err_msg = _("Hypermetro and Replication can not be "
                        "used in the same volume_type.")
            LOG.error(err_msg)
            raise exception.VolumeBackendAPIException(data=err_msg)

        lun_params, lun_info, model_update = (
            self._create_base_type_volume(opts, volume, volume_type))

        model_update = self._add_extend_type_to_volume(opts, lun_params,
                                                       lun_info, model_update)
        return model_update

原生ISCSIDriver 会通过os_brick模块远程iscsi挂载磁盘到本地

class ISCSIDriver(VolumeDriver):
    .........
    def copy_image_to_volume(self, context, volume, image_service, image_id):
        """Fetch image from image_service and write to unencrypted volume.

        This does not attach an encryptor layer when connecting to the volume.
        """
        self._copy_image_data_to_volume(
            context, volume, image_service, image_id, encrypted=False)
        .........
    def _copy_image_data_to_volume(self, context, volume, image_service,
                                   image_id, encrypted=False):
        """Fetch the image from image_service and write it to the volume."""
        LOG.debug('copy_image_to_volume %s.', volume['name'])

        use_multipath = self.configuration.use_multipath_for_image_xfer
        enforce_multipath = self.configuration.enforce_multipath_for_image_xfer
        properties = utils.brick_get_connector_properties(use_multipath,
                                                          enforce_multipath)
        attach_info, volume = self._attach_volume(context, volume, properties) # 这里会挂载远程disk
        try:
            if encrypted:
                encryption = self.db.volume_encryption_metadata_get(context,
                                                                    volume.id)
                utils.brick_attach_volume_encryptor(context,
                                                    attach_info,
                                                    encryption)
            try:
                image_utils.fetch_to_raw(
                    context,
                    image_service,
                    image_id,
                    attach_info['device']['path'],
                    self.configuration.volume_dd_blocksize,
                    size=volume['size']) #这里写入镜像数据 
            except exception.ImageTooBig:
                with excutils.save_and_reraise_exception():
                    LOG.exception("Copying image %(image_id)s "
                                  "to volume failed due to "
                                  "insufficient available space.",
                                  {'image_id': image_id})

            finally:
                if encrypted:
                    utils.brick_detach_volume_encryptor(attach_info,
                                                        encryption)
        finally:
            self._detach_volume(context, attach_info, volume, properties,
                                force=True)

fetch_to_volume_format 中会fetch镜像文件到本地临时目录，最后通过执行qemu-img convert 命令将镜像临时文件数据写入到本地iscsi远程磁盘中，至此虚机的系统盘数据写入完成。

def fetch_to_volume_format(context, image_service,
                           image_id, dest, volume_format, blocksize,
                           volume_subformat=None, user_id=None,
                           project_id=None, size=None, run_as_root=True):
    .........
    convert_image(tmp, dest, volume_format,
                      out_subformat=volume_subformat,
                      src_format=disk_format,
                      run_as_root=run_as_root)

def _convert_image(prefix, source, dest, out_format,
                   out_subformat=None, src_format=None,
                   run_as_root=True, cipher_spec=None, passphrase_file=None):
    cmd = _get_qemu_convert_cmd(source, dest, 
                                out_format=out_format,
                                src_format=src_format,
                                out_subformat=out_subformat,
                                cache_mode=cache_mode,
                                prefix=prefix,
                                cipher_spec=cipher_spec,
                                passphrase_file=passphrase_file)  #拼接 qemu-img convert 命令

最后cinder侧更新数据库，至此cinder侧工作完成了，概括性的流程如下:

2.2 nova侧实现

nova侧在给虚机挂载磁盘时，nova-compute收到请求后attach volume的请求后调用nova.compute.manager.ComputeManager.attach_volume

    def _attach_volume(self, context, instance, bdm):
        context = context.elevated()
        LOG.info('Attaching volume %(volume_id)s to %(mountpoint)s',
                 {'volume_id': bdm.volume_id,
                  'mountpoint': bdm['mount_device']},
                 instance=instance)
        compute_utils.notify_about_volume_attach_detach(
            context, instance, self.host,
            action=fields.NotificationAction.VOLUME_ATTACH,
            phase=fields.NotificationPhase.START,
            volume_id=bdm.volume_id)
        try:
            bdm.attach(context, instance, self.volume_api, self.driver,
                       do_driver_attach=True)
            ................

nova.virt.block_device.
DriverVolumeBlockDevice.attach中会 调用 _legacy_volume_attach 进行挂载磁盘

@update_db
    def attach(self, context, instance, volume_api, virt_driver,
               do_driver_attach=False, **kwargs):
        .................. 
        # Check to see if we need to lock based on the shared_targets value.
        # Default to False if the volume does not expose that value to maintain
        # legacy behavior.
        if volume.get('shared_targets', False):
            # Lock the attach call using the provided service_uuid.
            @utils.synchronized(volume['service_uuid'])
            def _do_locked_attach(*args, **_kwargs):
                self._do_attach(*args, **_kwargs)

            _do_locked_attach(context, instance, volume, volume_api,
                              virt_driver, do_driver_attach)
        else:
            # We don't need to (or don't know if we need to) lock.
            self._do_attach(context, instance, volume, volume_api,
                            virt_driver, do_driver_attach)

    def _do_attach(self, context, instance, volume, volume_api, virt_driver,
                   do_driver_attach):
        """Private method that actually does the attach.

        This is separate from the attach() method so the caller can optionally
        lock this call.
        """
        context = context.elevated()
        connector = virt_driver.get_volume_connector(instance)
        if not self['attachment_id']:
            self._legacy_volume_attach(context, volume, connector, instance,
                                       volume_api, virt_driver,
                                       do_driver_attach)
        else:
            self._volume_attach(context, volume, connector, instance,
                                volume_api, virt_driver,
                                self['attachment_id'],
                                do_driver_attach)

legacyvolume_attach 会 调用cinder 的 initialize_connection 获取volume挂载的connection info，这里因为是iscsi 云盘，cinder会返回iSCSI的 connection info。

    def _legacy_volume_attach(self, context, volume, connector, instance,
                              volume_api, virt_driver,
                              do_driver_attach=False):
        volume_id = volume['id']

        connection_info = volume_api.initialize_connection(context,
                                                           volume_id,
                                                           connector)
       .................. 

        # If do_driver_attach is False, we will attach a volume to an instance
        # at boot time. So actual attach is done by instance creation code.
        if do_driver_attach:
            encryption = encryptors.get_encryption_metadata(
                context, volume_api, volume_id, connection_info)

            try:
                virt_driver.attach_volume(
                        context, connection_info, instance,
                        self['mount_device'], disk_bus=self['disk_bus'],
                        device_type=self['device_type'], encryption=encryption)
            except Exception:
                with excutils.save_and_reraise_exception():
                    LOG.exception("Driver failed to attach volume "
                                  "%(volume_id)s at %(mountpoint)s",
                                  {'volume_id': volume_id,
                                   'mountpoint': self['mount_device']},
                                  instance=instance)
                    volume_api.terminate_connection(context, volume_id,
                                                    connector)
        .................. 
        if volume['attach_status'] == "detached":
            # NOTE(mriedem): save our current state so connection_info is in
            # the database before the volume status goes to 'in-use' because
            # after that we can detach and connection_info is required for
            # detach.
            self.save()
            try:
                volume_api.attach(context, volume_id, instance.uuid,
                                  self['mount_device'], mode=mode)

virt_driver.attach_volume 会调用libvirt 挂载磁盘 ，self._connect_volume 会依据connection info判断为iSCSI driver 会Calling os-brick to attach iSCSI Volume ，最终libvirt 在线添加了disk到虚机里面，至此虚机侧挂载完成！

 def attach_volume(self, context, connection_info, instance, mountpoint,
                      disk_bus=None, device_type=None, encryption=None):
        guest = self._host.get_guest(instance)

        disk_dev = mountpoint.rpartition("/")[2]
        bdm = {
            'device_name': disk_dev,
            'disk_bus': disk_bus,
            'device_type': device_type}

        .................. 

        self._connect_volume(context, connection_info, instance,
                             encryption=encryption)  

        .................. 
 	try:
            state = guest.get_power_state(self._host)
            live = state in (power_state.RUNNING, power_state.PAUSED)

            guest.attach_device(conf, persistent=True, live=live)

总结:

IP SAN 由于基于传统IP网络，所以优势是成本低、易维护且比较灵活，但同时也限制了他只适合应用于对于性能要求并不是太高的场景，性能上比较不如FS SAN以及nvmf等，因此IP SAN 适合于中小型应用场景，所以需要针对不同的使用场景下进行选择不同的存储协议。

本文借以兼容腾凌存储梳理了openstack侧虚机挂载使用商业 IP SAN存储的整个流程，对于理解包括支持其他类型的SAN/NVMF等存储有一定的参考意义。