mmsegmentation 框架技巧

启动 Test 阶段 Metric

1. 在 test_pipeline 中添加 dict(type=‘LoadAnnotations’, reduce_zero_label=False) ，如果使用 TTA的话，也需要在 tta 模块添加。

   dict(type='LoadAnnotations', reduce_zero_label=False)
   

2. 在 test_evaluator 中设置 format_only=False。

   test_evaluator = dict(type='IoUMetric', iou_metrics=['mFscore', 'mIoU', 'mDice'], format_only=False)
   

3. 为 test_dataloader 中的 seg_map_path 设置正确路径。

   test_dataloader = dict(
       dataset=dict(
           type=dataset_type,
           data_root=data_root,
           data_prefix=dict(
               img_path='test/' + data_name,  img_path2='test/' + data_name2, seg_map_path='test/labels_20240316'),
   		    pipeline=test_pipeline))
   

   这里要注意 mm 框架中存在 层级覆盖，一定要确保数据修改后不会被下一层错误覆盖。

输入: 3 通道改多通道

这里以三通道改四通道为例

1. 设置数据加载类: 修改 mmseg\datasets\transforms[loading.py](http://loading.py/) ，修改 LoadMultipleRSImageFromFile 类中 transform 函数（如果已安装好 Gdal 库的可以跳过本步，这里修改是因为原方法使用了 Gdal 库来进行数据加载，但是 Gdal 库比较难安装，所以换成 Cv2 来读入）

     def transform(self, results: Dict) -> Dict:
           """Functions to load image.
   
           Args:
               results (dict): Result dict from :obj:``mmcv.BaseDataset``.
   
           Returns:
               dict: The dict contains loaded image and meta information.
           """
   
           filename = results['img_path']
           filename2 = results['img_path2']
   
           # ds = gdal.Open(filename)
           # ds2 = gdal.Open(filename2)
   
           # if ds is None:
           #     raise Exception(f'Unable to open file: {filename}')
           # if ds2 is None:
           #     raise Exception(f'Unable to open file: {filename2}')
   
           # img = np.einsum('ijk->jki', ds.ReadAsArray())
           # img2 = np.einsum('ijk->jki', ds2.ReadAsArray())
   
           img_bytes = fileio.get(filename)
           img =  mmcv.imfrombytes(img_bytes, flag='color', backend='cv2')
   
           img2_bytes = fileio.get(filename2)
           img2 =  mmcv.imfrombytes(img2_bytes, flag='color', backend='cv2')
   
           if self.to_float32:
               img = img.astype(np.float32)
               img2 = img2.astype(np.float32)
               img = np.concatenate((img, img2[:, :, :1]), axis=2)
               # img = np.concatenate((img, img2), axis=2)
           # if img.shape != img2.shape:
           #     raise Exception(f'Image shapes do not match:'
           #                     f' {img.shape} vs {img2.shape}')
   
           results['img'] = img
           results['img_shape'] = img.shape[:2]
           results['ori_shape'] = img.shape[:2]
           return results
   

   💡 当然也可以考虑不修改 LoadMultipleRSImageFromFile 类，而是基于 LoadMultipleRSImageFromFile 类重写一个类，Such as LoadMultipleRSImageFromFileByCv2，然后需要在 mmseg\datasets\init.py 这里注册一下新类，具体可参考 LoadMultipleRSImageFromFile 类的注册方法。

2. 在数据集配置文件中调整数据加载类: 修改 configs\base\datasets[igarss2024track2.py](http://igarss2024track2.py/) ，把配置文件里的 LoadImageFromFile 替换成第一步修改后的类，比如 LoadMultipleRSImageFromFile 。

   train_pipeline = [
       dict(type='LoadMultipleRSImageFromFile'),
       # dict(type='LoadImageFromFile'),
       dict(type='LoadAnnotations', reduce_zero_label=False),
       dict(
           type='RandomResize',
           scale=(2048, 512),
           ratio_range=(0.5, 2.0),
           keep_ratio=True),
       dict(type='RandomCrop', crop_size=crop_size, cat_max_ratio=0.75),
       dict(type='RandomFlip', prob=0.5),
       dict(type='PhotoMetricDistortion'),
       dict(type='PackSegInputs')
   ]
   val_pipeline = [
       dict(type='LoadMultipleRSImageFromFile'),
       # dict(type='LoadImageFromFile'),
       dict(type='Resize', scale=(2048, 512), keep_ratio=True),
       # add loading annotation after ``Resize`` because ground truth
       # does not need to do resize data transform
       dict(type='LoadAnnotations', reduce_zero_label=False),
       dict(type='PackSegInputs')
   ]
   test_pipeline = [
       dict(type='LoadMultipleRSImageFromFile'),
       # dict(type='LoadImageFromFile'),
       dict(type='Resize', scale=(2048, 512), keep_ratio=True),
       # add loading annotation after ``Resize`` because ground truth
       # does not need to do resize data transform
       # dict(type='LoadAnnotations', reduce_zero_label=False),
       dict(type='PackSegInputs')
   ]
   img_ratios = [0.5, 0.75, 1.0, 1.25, 1.5, 1.75, 2.0]
   # img_ratios = [4.0, 4.5, 5.0, 5.5, 6.0]
   # img_ratios = [0.75]
   tta_pipeline = [
       dict(type='LoadMultipleRSImageFromFile', backend_args=None),
       dict(
           type='TestTimeAug',
           transforms=[
               [dict(type='Resize', scale=(2048, 512), keep_ratio=True)],
               [
                   dict(type='Resize', scale_factor=r, keep_ratio=True)
                   for r in img_ratios
               ],
               [
                   dict(type='RandomFlip', prob=0., direction='horizontal'),
                   dict(type='RandomFlip', prob=1., direction='horizontal')
               ], [dict(type='PackSegInputs')]
           ])
   ]
   

3. 修改 mode-backone 输入通道: 修改 configs\knet\knet-s3_resnest101-d8_fcn_8xb2-adamw-80k_ade20k-512x512.py , 将 in_channels 值改成我们需要输入的通道数 (这里 in_channels = 4 )。

   model = dict(
       type='EncoderDecoder',
       data_preprocessor=data_preprocessor,
       pretrained='open-mmlab://resnest101', # diff
       backbone=dict(
           type='ResNeSt',
           in_channels = 4, # 添加这一行，具体数字表示通道数
           stem_channels=128, # 网络通道大小
           radix=2, # diff
           reduction_factor=4, # diff
           avg_down_stride=True, # diff
           depth=101,
           num_stages=4,
           out_indices=(0, 1, 2, 3),
           dilations=(1, 1, 2, 4),
           strides=(1, 2, 1, 1),
           norm_cfg=norm_cfg,
           norm_eval=False,
           style='pytorch',
           contract_dilation=True),
   

ResNeSt 特征层融合

模型特征层融合: 我使用的是 Knet 模型，其中 Backbone 采用的是 ResNeSt，所以需要对ResNeSt 提取的特征进行融合（3 通道和单通道融合），在经过对 ResNet 和 ResNeSt 网络架构了解后，发现 ResNeSt 主要工作是使用 Split-Attention模块替换了 ResNet 中四个 Stage 的 Conv，mmsegmentation 框架中的 ResNeSt 是继承 ResNet，所以进而需要做的是把 ResNet的特征部分进行融合。

1. 修改 mmseg\models\backbones[resnet.py](http://resnet.py/) , 修改 ResNet_feature_fusion 类中 forward 函数

       def forward(self, x):
           """Forward function."""
           if self.deep_stem:
               x1 = self.stem_first_channels(x[:, 0:3, :, :])
               x2 = self.stem_second_channels(x[:, 3:4, :, :])
           else:
               x = self.conv1(x)
               x = self.norm1(x)
               x = self.relu(x)
   
           outs1 = []
           outs2 = []
           x1 = self.maxpool(x1)
           x2 = self.maxpool(x2)
           for i, layer_name in enumerate(self.res_layers):
               res_layer = getattr(self, layer_name)
               x1 = res_layer(x1)
               x2 = res_layer(x2)
               if i in self.out_indices:
                   outs1.append(x1)
                   outs2.append(x2)
   
           outs = []
           for out1, out2 in zip(outs1, outs2):
               outs.append(torch.add(out1, out2)) 
   

2. 定义 self.stem_first_channels 和 self.stem_second_channels: 由于是 3 通道和单通道融合，所有特征提取器的 in_channels 是不一样的，所以在这里需要定义两个特征提取器 self.stem, 只是 in_channels 不一样，其它都是一样的。

   def _make_stem_layer(self, in_first_channels, in_second_channels, stem_channels):
           """Make stem layer for ResNet."""
           if self.deep_stem:
               self.stem_first_channels = nn.Sequential(
                   build_conv_layer(
                       self.conv_cfg,
                       in_first_channels,
                       stem_channels // 2,
                       kernel_size=3,
                       stride=2,
                       padding=1,
                       bias=False),
                   build_norm_layer(self.norm_cfg, stem_channels // 2)[1],
                   nn.ReLU(inplace=True),
                   build_conv_layer(
                       self.conv_cfg,
                       stem_channels // 2,
                       stem_channels // 2,
                       kernel_size=3,
                       stride=1,
                       padding=1,
                       bias=False),
                   build_norm_layer(self.norm_cfg, stem_channels // 2)[1],
                   nn.ReLU(inplace=True),
                   build_conv_layer(
                       self.conv_cfg,
                       stem_channels // 2,
                       stem_channels,
                       kernel_size=3,
                       stride=1,
                       padding=1,
                       bias=False),
                   build_norm_layer(self.norm_cfg, stem_channels)[1],
                   nn.ReLU(inplace=True))
               self.stem_second_channels = nn.Sequential(
                   build_conv_layer(
                       self.conv_cfg,
                       in_second_channels,
                       stem_channels // 2,
                       kernel_size=3,
                       stride=2,
                       padding=1,
                       bias=False),
                   build_norm_layer(self.norm_cfg, stem_channels // 2)[1],
                   nn.ReLU(inplace=True),
                   build_conv_layer(
                       self.conv_cfg,
                       stem_channels // 2,
                       stem_channels // 2,
                       kernel_size=3,
                       stride=1,
                       padding=1,
                       bias=False),
                   build_norm_layer(self.norm_cfg, stem_channels // 2)[1],
                   nn.ReLU(inplace=True),
                   build_conv_layer(
                       self.conv_cfg,
                       stem_channels // 2,
                       stem_channels,
                       kernel_size=3,
                       stride=1,
                       padding=1,
                       bias=False),
                   build_norm_layer(self.norm_cfg, stem_channels)[1],
                   nn.ReLU(inplace=True))
           else:
               self.conv1 = build_conv_layer(
                   self.conv_cfg,
                   in_channels,
                   stem_channels,
                   kernel_size=7,
                   stride=2,
                   padding=3,
                   bias=False)
               self.norm1_name, norm1 = build_norm_layer(
                   self.norm_cfg, stem_channels, postfix=1)
               self.add_module(self.norm1_name, norm1)
               self.relu = nn.ReLU(inplace=True)
           self.maxpool = nn.MaxPool2d(kernel_size=3, stride=2, padding=1)
   

3. 在 __init__函数中初始化 in_first_channels=3, in_second_channels=1 并赋值给 self._make_stem_layer(in_first_channels, in_second_channels, stem_channels)

    def __init__(self,
                    depth,
                    in_channels=3,
                    in_first_channels=3, 
                    in_second_channels=1,
                    stem_channels=64,
                    base_channels=64,
                    num_stages=4,
                    strides=(1, 2, 2, 2),
                    dilations=(1, 1, 1, 1),
                    out_indices=(0, 1, 2, 3),
                    style='pytorch',
                    deep_stem=False,
                    avg_down=False,
                    frozen_stages=-1,
                    conv_cfg=None,
                    norm_cfg=dict(type='BN', requires_grad=True),
                    norm_eval=False,
                    dcn=None,
                    stage_with_dcn=(False, False, False, False),
                    plugins=None,
                    multi_grid=None,
                    contract_dilation=False,
                    with_cp=False,
                    zero_init_residual=True,
                    pretrained=None,
                    init_cfg=None):
           super().__init__(init_cfg)
   
           self._make_stem_layer(in_first_channels, in_second_channels, stem_channels)
   
   

   💡 当然也可以考虑不修改 mmseg\models\backbones[resnet.py](http://resnet.py/)，而是基于 resnet.py重写一个resnet_feature_fusion.py, 这种方法就需要在 mmseg\models\backbones\init.py 这里注册一下新类，具体可参考 resnet.py 中类的注册方法，然后在 configs_4x3090_192.168.29.165\knet\knet-s3_resnest101-d8_fcn_8xb2-adamw-80k_ade20k-512x512.py 修改 backbone-type 为 注册的新类。

   ---

   上面方法中只是将特征进行简单的相加，下面是扩展部分对特征进行特殊处理进行融合。

   考虑到单纯的将特征相加的效果不太好，所以我参考了论文Self-Supervised Model Adaptation for Multimodal Semantic Segmentation 中的 SSMA 方法。

   it merges together features coming from parallel layers, using Squeeze and Excitation
   blocks, then rescaled the input using channel (sigmoid-based) attention

   class MultimodalAdapter(nn.Module):
       """Self-Supervised Multi-Modal Adaptation block, from https://arxiv.org/abs/1808.03833
       Also called SSMA, it merges together features coming from parallel layers, using Squeeze and Excitation
       blocks, then rescaled the input using channel (sigmoid-based) attention.
       """
       def __init__(self,
                    sar_channels: int,
                    cdem_channels: int,
                    act_layer: Type[nn.Module],
                    norm_layer: Type[nn.Module],
                    bottleneck_factor: int = 4):
           super().__init__()
           # compute input channels and bottleneck channels
           total_chs = sar_channels + cdem_channels
           bottleneck_chs = total_chs // bottleneck_factor
           self.bottleneck = nn.Sequential(nn.Conv2d(total_chs, bottleneck_chs, kernel_size=3, padding=1, bias=False),
                                           norm_layer(bottleneck_chs), act_layer(),
                                           nn.Conv2d(bottleneck_chs, total_chs, kernel_size=3, padding=1, bias=False),
                                           norm_layer(total_chs), nn.Sigmoid())
           # the output is given by the RGB network, which is supposed to be bigger
           # also, this allows for easier integration with decoders
           self.out_bn = norm_layer(total_chs)
           self.out_conv = nn.Conv2d(total_chs, sar_channels, kernel_size=3, padding=1, bias=False)
   
       def forward(self, sar: torch.Tensor, dem: torch.Tensor) -> torch.Tensor:
           x1 = torch.cat((sar, dem), dim=1)
           x = self.bottleneck(x1)
           recalibrated = x1 * x
           x = self.out_bn(recalibrated)
           return self.out_conv(x)
   

设置在线 Cutmix 数据增强

1. 修改 mmengine/dataset/base_dataset.py 文件，修改 BaseDataset 类中 _getitem_ 方法，具体如下:

     def __getitem__(self, idx: int) -> dict:
         """Get the idx-th image and data information of dataset after
         ``self.pipeline``, and ``full_init`` will be called if the dataset has
         not been fully initialized.
   
         During training phase, if ``self.pipeline`` get ``None``,
         ``self._rand_another`` will be called until a valid image is fetched or
          the maximum limit of refetech is reached.
   
         Args:
             idx (int): The index of self.data_list.
   
         Returns:
             dict: The idx-th image and data information of dataset after
             ``self.pipeline``.
         """
         # Performing full initialization by calling `__getitem__` will consume
         # extra memory. If a dataset is not fully initialized by setting
         # `lazy_init=True` and then fed into the dataloader. Different workers
         # will simultaneously read and parse the annotation. It will cost more
         # time and memory, although this may work. Therefore, it is recommended
         # to manually call `full_init` before dataset fed into dataloader to
         # ensure all workers use shared RAM from master process.
         if not self._fully_initialized:
             print_log(
                 'Please call `full_init()` method manually to accelerate '
                 'the speed.',
                 logger='current',
                 level=logging.WARNING)
             self.full_init()
   
         if self.test_mode:
             data = self.prepare_data(idx)
             if data is None:
                 raise Exception('Test time pipline should not get `None` '
                                 'data_sample')
             return data
         ###########
         ################
         def rand_bbox(size, lam):
             if len(size) == 4:
                 W = size[2]
                 H = size[3]
             elif len(size) == 3:
                 W = size[1]
                 H = size[2]
             else:
                 raise Exception
   
             cut_rat = np.sqrt(1. - lam)
             cut_w = int(W * cut_rat)
             cut_h = int(H * cut_rat)
   
             # uniform
             cx = np.random.randint(W)
             cy = np.random.randint(H)
   
             bbx1 = np.clip(cx - cut_w // 2, 0, W)
             bby1 = np.clip(cy - cut_h // 2, 0, H)
             bbx2 = np.clip(cx + cut_w // 2, 0, W)
             bby2 = np.clip(cy + cut_h // 2, 0, H)
   
             return bbx1, bby1, bbx2, bby2
         for _ in range(self.max_refetch + 1):
             data = self.prepare_data(idx)
             ####################
             idx_2 = self._rand_another()
             data_2 = self.prepare_data(idx_2)
   
             img = data['inputs']
             img_2 = data_2['inputs']
             if img.shape[2] < img_2.shape[2]:
                 shape = img.shape
             else:
                 shape = img_2.shape
   
             mask = data['data_samples']._gt_sem_seg.data
             mask_2 = data_2['data_samples']._gt_sem_seg.data
             lam = np.random.beta(1,1)
             bbx1, bby1, bbx2, bby2 = rand_bbox(shape, lam)
             img[:, bbx1:bbx2, bby1:bby2] = img_2[:, bbx1:bbx2, bby1:bby2]
             mask[:, bbx1:bbx2, bby1:bby2] = mask_2[:, bbx1:bbx2, bby1:bby2]
             data['inputs'] = img
             data['data_samples']._gt_sem_seg.data = mask
             data['data_samples'].gt_sem_seg.data = mask
   
             # Broken images or random augmentations may cause the returned data
             # to be None
             if data is None:
                 idx = self._rand_another()
                 continue
             return data
         # for _ in range(self.max_refetch + 1):
         #     data = self.prepare_data(idx)
         #     # Broken images or random augmentations may cause the returned data
         #     # to be None
         #     if data is None:
         #         idx = self._rand_another()
         #         continue
         #     return data
   
         raise Exception(f'Cannot find valid image after {self.max_refetch}! '
                         'Please check your image path and pipeline')
   

2. 在 configs/base/datasets/igarss2024track2_muti_channel_640x640_red-nir-swir1-cdem_train+val+test.py 文件中 test_dataloader-dataset 位置添加 test_mode = True。

   test_dataloader = dict(
       batch_size=1,
       num_workers=4,
       persistent_workers=True,
       sampler=dict(type='DefaultSampler', shuffle=False),
       dataset=dict(
           type=dataset_type,
           data_root=data_root,
           test_mode = True, 
           data_prefix=dict(
               img_path='test/' + data_name,  img_path2='test/' + data_name2, seg_map_path='test/labels_20240316'),
               # img_path='val/' + 'images_normalization-[1000, 1600, 3000]_1_2_3', seg_map_path='val/labels_' + data_name[7:]),
               # img_path='channel_r_g_b_uint8/val/' + data_name, seg_map_path='val/labels'),
           pipeline=test_pipeline))