Skip to content

Add TensorRT export support and inference-models inference integration#853

Draft
Copilot wants to merge 6 commits intodevelopfrom
copilot/support-loading-tensorrt-engine-weights
Draft

Add TensorRT export support and inference-models inference integration#853
Copilot wants to merge 6 commits intodevelopfrom
copilot/support-loading-tensorrt-engine-weights

Conversation

Copy link
Copy Markdown
Contributor

Copilot AI commented Mar 22, 2026

RF-DETR had no built-in path to TensorRT export or multi-backend inference. This PR adds a one-step TensorRT export flag and documents inference-models as the recommended inference library.

Changes

  • RFDETR.export(tensorrt=True) — chains ONNX → trtexec conversion in a single call, producing both output/inference_model.onnx and output/inference_model.engine
  • trtexec() return value fix — was returning None; now returns the engine path (required for the export chaining above)
  • inference-models integrationdocs/learn/export.md now directs users to inference-models for multi-backend inference (PyTorch, ONNX, TensorRT) with automatic backend selection

Usage

# One-step TensorRT export (requires trtexec in PATH)
model = RFDETRBase(pretrain_weights="checkpoint.pth")
model.export(tensorrt=True)  # → output/inference_model.onnx + .engine

# Inference via inference-models (auto-selects best backend)
from inference_models import AutoModel
model = AutoModel.from_pretrained("rfdetr-base")
predictions = model(image)

# Load a local checkpoint
model = AutoModel.from_pretrained("/path/to/checkpoint.pth", model_type="rfdetr-base")

# Force TensorRT backend
from inference_models import AutoModel, BackendType
model = AutoModel.from_pretrained("rfdetr-base", backend=BackendType.TRT)
Original prompt

This section details on the original issue you should resolve

<issue_title>Support for loading TensorRT .engine weights in RF-DETR inference</issue_title>
<issue_description>Hi,

I would like to ask whether RF-DETR supports directly loading TensorRT .engine weights for inference.

Currently, the official sample code only supports loading .pt weights for inference. However, I plan to deploy the model using TensorRT (and later integrate an ID tracking pipeline), so I would like to run inference using the converted .engine weights instead.

The documentation and examples do not seem to mention how to load .engine files. May I ask whether direct TensorRT engine inference is currently supported, or if there is a recommended way to achieve this?

Thanks!</issue_description>

<agent_instructions>explore adding another export format and also loading for inference
keep it simple and focused</agent_instructions>

Comments on the Issue (you are @copilot in this section)


⚡ Quickly spin up Copilot coding agent tasks from anywhere on your macOS or Windows machine with Raycast.

Copilot AI changed the title [WIP] Add support for loading TensorRT .engine weights in RF-DETR inference Add TensorRT engine inference support via RFDETRTensorRT Mar 22, 2026
Copilot AI requested a review from Borda March 22, 2026 21:33
@codecov
Copy link
Copy Markdown

codecov bot commented Mar 23, 2026

Codecov Report

❌ Patch coverage is 65.00000% with 28 lines in your changes missing coverage. Please review.
✅ Project coverage is 71%. Comparing base (faf8a22) to head (5476071).
⚠️ Report is 24 commits behind head on develop.

❌ Your patch check has failed because the patch coverage (65%) is below the target coverage (95%). You can increase the patch coverage or adjust the target coverage.
❌ Your project check has failed because the head coverage (71%) is below the target coverage (95%). You can increase the head coverage or adjust the target coverage.

Additional details and impacted files
@@           Coverage Diff           @@
##           develop   #853    +/-   ##
=======================================
- Coverage       76%    71%    -4%     
=======================================
  Files           94     94            
  Lines         7269   7348    +79     
=======================================
- Hits          5502   5235   -267     
- Misses        1767   2113   +346     
🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@Borda
Copy link
Copy Markdown
Member

Borda commented Mar 31, 2026

@copilot pivot it to set up integration with inference-models, see https://github.com/roboflow/inference/tree/main/inference_models

Copy link
Copy Markdown
Contributor Author

Copilot AI commented Mar 31, 2026

@copilot pivot it to set up integration with inference-models, see https://github.com/roboflow/inference/tree/main/inference_models

Done in 696747f. Removed the custom RFDETRTensorRT class and replaced it with documentation pointing to inference-models, which already has production-quality RF-DETR support for PyTorch, ONNX, and TensorRT backends.

The docs now show:

  • AutoModel.from_pretrained("rfdetr-base") for pretrained models with automatic backend selection
  • Loading local .pth checkpoints via model_type="rfdetr-base"
  • Forcing TRT backend with backend=BackendType.TRT

The RFDETR.export(tensorrt=True) convenience flag and trtexec() return value fix are retained since they're still useful for producing engine files.

Warning

Firewall rules blocked me from connecting to one or more addresses (expand for details)

I tried to connect to the following addresses, but was blocked by firewall rules:

  • inference-models.roboflow.com
    • Triggering command: /home/REDACTED/work/_temp/ghcca-node/node/bin/node /home/REDACTED/work/_temp/ghcca-node/node/bin/node --enable-source-maps /home/REDACTED/work/_temp/copilot-developer-action-main/dist/index.js (dns block)

If you need me to access, download, or install something from one of these locations, you can either:

Copilot AI changed the title Add TensorRT engine inference support via RFDETRTensorRT Add TensorRT export support and inference-models integration Mar 31, 2026
@Borda
Copy link
Copy Markdown
Member

Borda commented Mar 31, 2026

@copilot resolve merge conflict or rebase on the develop branch with just relevant chnages

Copy link
Copy Markdown
Contributor Author

Copilot AI commented Mar 31, 2026

@copilot resolve merge conflict or rebase on the develop branch with just relevant chnages

Done in b3099a7 — rebased onto develop with only the relevant changes: trtexec() return value fix, tensorrt=True export param, inference-models docs, and tests.

…rt.py

Co-authored-by: Borda <6035284+Borda@users.noreply.github.com>
Copilot AI changed the title Add TensorRT export support and inference-models integration Add TensorRT export support and inference-models inference integration Mar 31, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Support for loading TensorRT .engine weights in RF-DETR inference

2 participants