# Faster R-CNN

> [Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks](https://arxiv.org/abs/1506.01497)

<!-- [ALGORITHM] -->

## Abstract

State-of-the-art object detection networks depend on region proposal algorithms to hypothesize object locations. Advances like SPPnet and Fast R-CNN have reduced the running time of these detection networks, exposing region proposal computation as a bottleneck. In this work, we introduce a Region Proposal Network (RPN) that shares full-image convolutional features with the detection network, thus enabling nearly cost-free region proposals. An RPN is a fully convolutional network that simultaneously predicts object bounds and objectness scores at each position. The RPN is trained end-to-end to generate high-quality region proposals, which are used by Fast R-CNN for detection. We further merge RPN and Fast R-CNN into a single network by sharing their convolutional features---using the recently popular terminology of neural networks with 'attention' mechanisms, the RPN component tells the unified network where to look. For the very deep VGG-16 model, our detection system has a frame rate of 5fps (including all steps) on a GPU, while achieving state-of-the-art object detection accuracy on PASCAL VOC 2007, 2012, and MS COCO datasets with only 300 proposals per image. In ILSVRC and COCO 2015 competitions, Faster R-CNN and RPN are the foundations of the 1st-place winning entries in several tracks.

<div align=center>
<img src="https://user-images.githubusercontent.com/40661020/143881188-ab87720f-5059-4b4e-a928-b540fb8fb84d.png" height="300"/>
</div>

## Results and Models

|    Backbone     |  Style  | Lr schd | Mem (GB) | Inf time (fps) | box AP |                      Config                       |                                                                                                                                                    Download                                                                                                                                                     |
| :-------------: | :-----: | :-----: | :------: | :------------: | :----: | :-----------------------------------------------: | :-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------: |
|     R-50-C4     |  caffe  |   1x    |    -     |       -        |  35.6  |  [config](./faster-rcnn_r50-caffe_c4-1x_coco.py)  |                [model](https://download.openxlab.org.cn/models/mmdetection/FasterR-CNN/weight/faster-rcnn_r50-caffe-c4_1x_coco) \| [log](https://download.openmmlab.com/mmdetection/v2.0/faster_rcnn/faster_rcnn_r50_caffe_c4_1x_coco/faster_rcnn_r50_caffe_c4_1x_coco_20220316_150152.log.json)                |
|    R-50-DC5     |  caffe  |   1x    |    -     |       -        |  37.2  | [config](./faster-rcnn_r50-caffe-dc5_1x_coco.py)  |              [model](https://download.openxlab.org.cn/models/mmdetection/FasterR-CNN/weight/faster-rcnn_r50-caffe-dc5_1x_coco) \| [log](https://download.openmmlab.com/mmdetection/v2.0/faster_rcnn/faster_rcnn_r50_caffe_dc5_1x_coco/faster_rcnn_r50_caffe_dc5_1x_coco_20201030_151909.log.json)               |
|    R-50-FPN     |  caffe  |   1x    |   3.8    |                |  37.8  | [config](./faster-rcnn_r50-caffe_fpn_1x_coco.py)  |              [model](https://download.openxlab.org.cn/models/mmdetection/FasterR-CNN/weight/faster-rcnn_r50-caffe_fpn_1x_coco) \| [log](https://download.openmmlab.com/mmdetection/v2.0/faster_rcnn/faster_rcnn_r50_caffe_fpn_1x_coco/faster_rcnn_r50_caffe_fpn_1x_coco_20200504_180032.log.json)               |
|    R-50-FPN     | pytorch |   1x    |   4.0    |      21.4      |  37.4  |    [config](./faster-rcnn_r50_fpn_1x_coco.py)     |                       [model](https://download.openxlab.org.cn/models/mmdetection/FasterR-CNN/weight/faster-rcnn_r50_fpn_1x_coco) \| [log](https://download.openmmlab.com/mmdetection/v2.0/faster_rcnn/faster_rcnn_r50_fpn_1x_coco/faster_rcnn_r50_fpn_1x_coco_20200130_204655.log.json)                        |
| R-50-FPN (FP16) | pytorch |   1x    |   3.4    |      28.8      |  37.5  |  [config](./faster-rcnn_r50_fpn_amp-1x_coco.py)   | [model](https://download.openmmlab.com/mmdetection/v2.0/fp16/faster_rcnn_r50_fpn_fp16_1x_coco/faster_rcnn_r50_fpn_fp16_1x_coco_20200204-d4dc1471.pth) \| [log](https://download.openmmlab.com/mmdetection/v2.0/fp16/faster_rcnn_r50_fpn_fp16_1x_coco/faster_rcnn_r50_fpn_fp16_1x_coco_20200204_143530.log.json) |
|    R-50-FPN     | pytorch |   2x    |    -     |       -        |  38.4  |    [config](./faster-rcnn_r50_fpn_2x_coco.py)     |                       [model](https://download.openxlab.org.cn/models/mmdetection/FasterR-CNN/weight/faster-rcnn_r50_fpn_2x_coco) \| [log](https://download.openmmlab.com/mmdetection/v2.0/faster_rcnn/faster_rcnn_r50_fpn_2x_coco/faster_rcnn_r50_fpn_2x_coco_20200504_210434.log.json)                        |
|    R-101-FPN    |  caffe  |   1x    |   5.7    |                |  39.8  | [config](./faster-rcnn_r101-caffe_fpn_1x_coco.py) |             [model](https://download.openxlab.org.cn/models/mmdetection/FasterR-CNN/weight/faster-rcnn_r101-caffe_fpn_1x_coco) \| [log](https://download.openmmlab.com/mmdetection/v2.0/faster_rcnn/faster_rcnn_r101_caffe_fpn_1x_coco/faster_rcnn_r101_caffe_fpn_1x_coco_20200504_180057.log.json)             |
|    R-101-FPN    | pytorch |   1x    |   6.0    |      15.6      |  39.4  |    [config](./faster-rcnn_r101_fpn_1x_coco.py)    |                      [model](https://download.openxlab.org.cn/models/mmdetection/FasterR-CNN/weight/faster-rcnn_r101_fpn_1x_coco) \| [log](https://download.openmmlab.com/mmdetection/v2.0/faster_rcnn/faster_rcnn_r101_fpn_1x_coco/faster_rcnn_r101_fpn_1x_coco_20200130_204655.log.json)                      |
|    R-101-FPN    | pytorch |   2x    |    -     |       -        |  39.8  |    [config](./faster-rcnn_r101_fpn_2x_coco.py)    |                      [model](https://download.openxlab.org.cn/models/mmdetection/FasterR-CNN/weight/faster-rcnn_r101_fpn_2x_coco) \| [log](https://download.openmmlab.com/mmdetection/v2.0/faster_rcnn/faster_rcnn_r101_fpn_2x_coco/faster_rcnn_r101_fpn_2x_coco_20200504_210455.log.json)                      |
| X-101-32x4d-FPN | pytorch |   1x    |   7.2    |      13.8      |  41.2  | [config](./faster-rcnn_x101-32x4d_fpn_1x_coco.py) |             [model](https://download.openxlab.org.cn/models/mmdetection/FasterR-CNN/weight/faster-rcnn_x101-32x4d_fpn_1x_coco) \| [log](https://download.openmmlab.com/mmdetection/v2.0/faster_rcnn/faster_rcnn_x101_32x4d_fpn_1x_coco/faster_rcnn_x101_32x4d_fpn_1x_coco_20200203_000520.log.json)             |
| X-101-32x4d-FPN | pytorch |   2x    |    -     |       -        |  41.2  | [config](./faster-rcnn_x101-32x4d_fpn_2x_coco.py) |             [model](https://download.openxlab.org.cn/models/mmdetection/FasterR-CNN/weight/faster-rcnn_x101-32x4d_fpn_2x_coco) \| [log](https://download.openmmlab.com/mmdetection/v2.0/faster_rcnn/faster_rcnn_x101_32x4d_fpn_2x_coco/faster_rcnn_x101_32x4d_fpn_2x_coco_20200506_041400.log.json)             |
| X-101-64x4d-FPN | pytorch |   1x    |   10.3   |      9.4       |  42.1  | [config](./faster-rcnn_x101-64x4d_fpn_1x_coco.py) |             [model](https://download.openxlab.org.cn/models/mmdetection/FasterR-CNN/weight/faster-rcnn_x101-64x4d_fpn_1x_coco) \| [log](https://download.openmmlab.com/mmdetection/v2.0/faster_rcnn/faster_rcnn_x101_64x4d_fpn_1x_coco/faster_rcnn_x101_64x4d_fpn_1x_coco_20200204_134340.log.json)             |
| X-101-64x4d-FPN | pytorch |   2x    |    -     |       -        |  41.6  | [config](./faster-rcnn_x101-64x4d_fpn_2x_coco.py) |             [model](https://download.openxlab.org.cn/models/mmdetection/FasterR-CNN/weight/faster-rcnn_x101-64x4d_fpn_2x_coco) \| [log](https://download.openmmlab.com/mmdetection/v2.0/faster_rcnn/faster_rcnn_x101_64x4d_fpn_2x_coco/faster_rcnn_x101_64x4d_fpn_2x_coco_20200512_161033.log.json)             |

## Different regression loss

We trained with R-50-FPN pytorch style backbone for 1x schedule.

| Backbone |   Loss type    | Mem (GB) | Inf time (fps) | box AP |                         Config                         |                                                                                                                                                         Download                                                                                                                                                         |
| :------: | :------------: | :------: | :------------: | :----: | :----------------------------------------------------: | :----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------: |
| R-50-FPN |     L1Loss     |   4.0    |      21.4      |  37.4  |       [config](./faster-rcnn_r50_fpn_1x_coco.py)       |                            [model](https://download.openxlab.org.cn/models/mmdetection/FasterR-CNN/weight/faster-rcnn_r50_fpn_1x_coco) \| [log](https://download.openmmlab.com/mmdetection/v2.0/faster_rcnn/faster_rcnn_r50_fpn_1x_coco/faster_rcnn_r50_fpn_1x_coco_20200130_204655.log.json)                            |
| R-50-FPN |    IoULoss     |          |                |  37.9  |     [config](./faster-rcnn_r50_fpn_iou_1x_coco.py)     |                      [model](https://download.openxlab.org.cn/models/mmdetection/FasterR-CNN/weight/faster-rcnn_r50_fpn_iou_1x_coco) \| [log](https://download.openmmlab.com/mmdetection/v2.0/faster_rcnn/faster_rcnn_r50_fpn_iou_1x_coco/faster_rcnn_r50_fpn_iou_1x_coco_20200506_095954.log.json)                      |
| R-50-FPN |    GIoULoss    |          |                |  37.6  |    [config](./faster-rcnn_r50_fpn_giou_1x_coco.py)     |                       [model](https://download.openxlab.org.cn/models/mmdetection/FasterR-CNN/weight/faster-rcnn_r50_fpn_giou_1x_coco) \| [log](https://download.openmmlab.com/mmdetection/v2.0/faster_rcnn/faster_rcnn_r50_fpn_1x_coco/faster_rcnn_r50_fpn_giou_1x_coco_20200505_161120.log.json)                       |
| R-50-FPN | BoundedIoULoss |          |                |  37.4  | [config](./faster-rcnn_r50_fpn_bounded-iou_1x_coco.py) | [model](https://download.openmmlab.com/mmdetection/v2.0/faster_rcnn/faster_rcnn_r50_fpn_1x_coco/faster_rcnn_r50_fpn_bounded_iou_1x_coco-98ad993b.pth) \| [log](https://download.openmmlab.com/mmdetection/v2.0/faster_rcnn/faster_rcnn_r50_fpn_1x_coco/faster_rcnn_r50_fpn_bounded_iou_1x_coco_20200505_160738.log.json) |

## Pre-trained Models

We also train some models with longer schedules and multi-scale training. The users could finetune them for downstream tasks.

|                           Backbone                            |  Style  | Lr schd | Mem (GB) | Inf time (fps) | box AP |                        Config                        |                                                                                                                                                                                 Download                                                                                                                                                                                 |
| :-----------------------------------------------------------: | :-----: | :-----: | :------: | :------------: | :----: | :--------------------------------------------------: | :----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------: |
|      [R-50-C4](./faster-rcnn_r50-caffe-c4_ms-1x_coco.py)      |  caffe  |   1x    |    -     |                |  35.9  |  [config](./faster-rcnn_r50-caffe-c4_ms-1x_coco.py)  |   [model](https://download.openmmlab.com/mmdetection/v2.0/faster_rcnn/faster_rcnn_r50_caffe_c4_mstrain_1x_coco/faster_rcnn_r50_caffe_c4_mstrain_1x_coco_20220316_150527-db276fed.pth) \| [log](https://download.openmmlab.com/mmdetection/v2.0/faster_rcnn/faster_rcnn_r50_caffe_c4_mstrain_1x_coco/faster_rcnn_r50_caffe_c4_mstrain_1x_coco_20220316_150527.log.json)   |
|     [R-50-DC5](./faster-rcnn_r50-caffe-dc5_ms-1x_coco.py)     |  caffe  |   1x    |    -     |                |  37.4  | [config](./faster-rcnn_r50-caffe-dc5_ms-1x_coco.py)  | [model](https://download.openmmlab.com/mmdetection/v2.0/faster_rcnn/faster_rcnn_r50_caffe_dc5_mstrain_1x_coco/faster_rcnn_r50_caffe_dc5_mstrain_1x_coco_20201028_233851-b33d21b9.pth) \| [log](https://download.openmmlab.com/mmdetection/v2.0/faster_rcnn/faster_rcnn_r50_caffe_dc5_mstrain_1x_coco/faster_rcnn_r50_caffe_dc5_mstrain_1x_coco_20201028_233851.log.json) |
|     [R-50-DC5](./faster-rcnn_r50-caffe-dc5_ms-3x_coco.py)     |  caffe  |   3x    |    -     |                |  38.7  | [config](./faster-rcnn_r50-caffe-dc5_ms-3x_coco.py)  | [model](https://download.openmmlab.com/mmdetection/v2.0/faster_rcnn/faster_rcnn_r50_caffe_dc5_mstrain_3x_coco/faster_rcnn_r50_caffe_dc5_mstrain_3x_coco_20201028_002107-34a53b2c.pth) \| [log](https://download.openmmlab.com/mmdetection/v2.0/faster_rcnn/faster_rcnn_r50_caffe_dc5_mstrain_3x_coco/faster_rcnn_r50_caffe_dc5_mstrain_3x_coco_20201028_002107.log.json) |
|     [R-50-FPN](./faster-rcnn_r50-caffe_fpn_ms-2x_coco.py)     |  caffe  |   2x    |   3.7    |                |  39.7  | [config](./faster-rcnn_r50-caffe_fpn_ms-2x_coco.py)  |                                 [model](https://download.openxlab.org.cn/models/mmdetection/FasterR-CNN/weight/faster-rcnn_r50-caffe_fpn_ms-2x_coco) \| [log](https://download.openmmlab.com/mmdetection/v2.0/faster_rcnn/faster_rcnn_r50_caffe_fpn_mstrain_2x_coco/faster_rcnn_r50_caffe_fpn_mstrain_2x_coco_20200504_231813.log.json)                                  |
|     [R-50-FPN](./faster-rcnn_r50-caffe_fpn_ms-3x_coco.py)     |  caffe  |   3x    |   3.7    |                |  39.9  | [config](./faster-rcnn_r50-caffe_fpn_ms-3x_coco.py)  |                                 [model](https://download.openxlab.org.cn/models/mmdetection/FasterR-CNN/weight/faster-rcnn_r50-caffe_fpn_ms-3x_coco) \| [log](https://download.openmmlab.com/mmdetection/v2.0/faster_rcnn/faster_rcnn_r50_caffe_fpn_mstrain_3x_coco/faster_rcnn_r50_caffe_fpn_mstrain_3x_coco_20210526_095054.log.json)                                  |
|        [R-50-FPN](./faster-rcnn_r50_fpn_ms-3x_coco.py)        | pytorch |   3x    |   3.9    |                |  40.3  |    [config](./faster-rcnn_r50_fpn_ms-3x_coco.py)     |             [model](https://download.openmmlab.com/mmdetection/v2.0/faster_rcnn/faster_rcnn_r50_fpn_mstrain_3x_coco/faster_rcnn_r50_fpn_mstrain_3x_coco_20210524_110822-e10bd31c.pth) \| [log](https://download.openmmlab.com/mmdetection/v2.0/faster_rcnn/faster_rcnn_r50_fpn_mstrain_3x_coco/faster_rcnn_r50_fpn_mstrain_3x_coco_20210524_110822.log.json)             |
|    [R-101-FPN](./faster-rcnn_r101-caffe_fpn_ms-3x_coco.py)    |  caffe  |   3x    |   5.6    |                |  42.0  | [config](./faster-rcnn_r101-caffe_fpn_ms-3x_coco.py) |                                [model](https://download.openxlab.org.cn/models/mmdetection/FasterR-CNN/weight/faster-rcnn_r101-caffe_fpn_ms-3x_coco) \| [log](https://download.openmmlab.com/mmdetection/v2.0/faster_rcnn/faster_rcnn_r101_caffe_fpn_mstrain_3x_coco/faster_rcnn_r101_caffe_fpn_mstrain_3x_coco_20210526_095742.log.json)                                |
|       [R-101-FPN](./faster-rcnn_r101_fpn_ms-3x_coco.py)       | pytorch |   3x    |   5.8    |                |  41.8  |    [config](./faster-rcnn_r101_fpn_ms-3x_coco.py)    |                                         [model](https://download.openxlab.org.cn/models/mmdetection/FasterR-CNN/weight/faster-rcnn_r101_fpn_ms-3x_coco) \| [log](https://download.openmmlab.com/mmdetection/v2.0/faster_rcnn/faster_rcnn_r101_fpn_mstrain_3x_coco/faster_rcnn_r101_fpn_mstrain_3x_coco_20210524_110822.log.json)                                         |
| [X-101-32x4d-FPN](./faster-rcnn_x101-32x4d_fpn_ms-3x_coco.py) | pytorch |   3x    |   7.0    |                |  42.5  | [config](./faster-rcnn_x101-32x4d_fpn_ms-3x_coco.py) |                                [model](https://download.openxlab.org.cn/models/mmdetection/FasterR-CNN/weight/faster-rcnn_x101-32x4d_fpn_ms-3x_coco) \| [log](https://download.openmmlab.com/mmdetection/v2.0/faster_rcnn/faster_rcnn_x101_32x4d_fpn_mstrain_3x_coco/faster_rcnn_x101_32x4d_fpn_mstrain_3x_coco_20210524_124151.log.json)                                |
| [X-101-32x8d-FPN](./faster-rcnn_x101-32x8d_fpn_ms-3x_coco.py) | pytorch |   3x    |   10.1   |                |  42.4  | [config](./faster-rcnn_x101-32x8d_fpn_ms-3x_coco.py) |                                [model](https://download.openxlab.org.cn/models/mmdetection/FasterR-CNN/weight/faster-rcnn_x101-32x8d_fpn_ms-3x_coco) \| [log](https://download.openmmlab.com/mmdetection/v2.0/faster_rcnn/faster_rcnn_x101_32x8d_fpn_mstrain_3x_coco/faster_rcnn_x101_32x8d_fpn_mstrain_3x_coco_20210604_182954.log.json)                                |
| [X-101-64x4d-FPN](./faster-rcnn_x101-64x4d_fpn_ms-3x_coco.py) | pytorch |   3x    |   10.0   |                |  43.1  | [config](./faster-rcnn_x101-64x4d_fpn_ms-3x_coco.py) |                                [model](https://download.openxlab.org.cn/models/mmdetection/FasterR-CNN/weight/faster-rcnn_x101-64x4d_fpn_ms-3x_coco) \| [log](https://download.openmmlab.com/mmdetection/v2.0/faster_rcnn/faster_rcnn_x101_64x4d_fpn_mstrain_3x_coco/faster_rcnn_x101_64x4d_fpn_mstrain_3x_coco_20210524_124528.log.json)                                |

We further finetune some pre-trained models on the COCO subsets, which only contain only a few of the 80 categories.

| Backbone                                                                 | Style | Class name         | Pre-traind model                                               | Mem (GB) | box AP | Config                                                                 | Download                                                                                                                                                                                                                                                                                                                                                                                     |
| ------------------------------------------------------------------------ | ----- | ------------------ | -------------------------------------------------------------- | -------- | ------ | ---------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| [R-50-FPN](./faster-rcnn_r50-caffe_fpn_ms-1x_coco-person.py)             | caffe | person             | [R-50-FPN-Caffe-3x](./faster-rcnn_r50-caffe_fpn_ms-3x_coco.py) | 3.7      | 55.8   | [config](./faster-rcnn_r50-caffe_fpn_ms-1x_coco-person.py)             | [model](https://download.openmmlab.com/mmdetection/v2.0/faster_rcnn/faster_rcnn_r50_fpn_1x_coco-person/faster_rcnn_r50_fpn_1x_coco-person_20201216_175929-d022e227.pth) \| [log](https://download.openmmlab.com/mmdetection/v2.0/faster_rcnn/faster_rcnn_r50_fpn_1x_coco-person/faster_rcnn_r50_fpn_1x_coco-person_20201216_175929.log.json)                                                 |
| [R-50-FPN](./faster-rcnn_r50-caffe_fpn_ms-1x_coco-person-bicycle-car.py) | caffe | person-bicycle-car | [R-50-FPN-Caffe-3x](./faster-rcnn_r50-caffe_fpn_ms-3x_coco.py) | 3.7      | 44.1   | [config](./faster-rcnn_r50-caffe_fpn_ms-1x_coco-person-bicycle-car.py) | [model](https://download.openmmlab.com/mmdetection/v2.0/faster_rcnn/faster_rcnn_r50_fpn_1x_coco-person-bicycle-car/faster_rcnn_r50_fpn_1x_coco-person-bicycle-car_20201216_173117-6eda6d92.pth) \| [log](https://download.openmmlab.com/mmdetection/v2.0/faster_rcnn/faster_rcnn_r50_fpn_1x_coco-person-bicycle-car/faster_rcnn_r50_fpn_1x_coco-person-bicycle-car_20201216_173117.log.json) |

## Torchvision New Receipe (TNR)

Torchvision released its high-precision ResNet models. The training details can be found on the [Pytorch website](https://pytorch.org/blog/how-to-train-state-of-the-art-models-using-torchvision-latest-primitives/). Here, we have done grid searches on learning rate and weight decay and found the optimal hyper-parameter on the detection task.

|                       Backbone                       |  Style  | Lr schd | Mem (GB) | Inf time (fps) | box AP |                       Config                       |                                                                                                                                                                               Download                                                                                                                                                                               |
| :--------------------------------------------------: | :-----: | :-----: | :------: | :------------: | :----: | :------------------------------------------------: | :------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------: |
| [R-50-TNR](./faster-rcnn_r50-tnr-pre_fpn_1x_coco.py) | pytorch |   1x    |    -     |                |  40.2  | [config](./faster-rcnn_r50-tnr-pre_fpn_1x_coco.py) | [model](https://download.openmmlab.com/mmdetection/v2.0/faster_rcnn/faster_rcnn_r50_fpn_tnr-pretrain_1x_coco/faster_rcnn_r50_fpn_tnr-pretrain_1x_coco_20220320_085147-efedfda4.pth) \| [log](https://download.openmmlab.com/mmdetection/v2.0/faster_rcnn/faster_rcnn_r50_fpn_tnr-pretrain_1x_coco/faster_rcnn_r50_fpn_tnr-pretrain_1x_coco_20220320_085147.log.json) |

## Citation

```latex
@article{Ren_2017,
   title={Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks},
   journal={IEEE Transactions on Pattern Analysis and Machine Intelligence},
   publisher={Institute of Electrical and Electronics Engineers (IEEE)},
   author={Ren, Shaoqing and He, Kaiming and Girshick, Ross and Sun, Jian},
   year={2017},
   month={Jun},
}
```
