该模型于 2017-12-20 发布在 HF 论文中，并于 2024-03-19 贡献给 Hugging Face Transformers。

SuperPoint

SuperPoint 是对用于特征点检测和描述的全卷积网络进行自监督训练的结果。该模型能够检测在单应性变换下具有重复性的特征点，并为每个点提供描述符。虽然其独立用途有限，但它可以用作单应性估计和图像匹配等其他任务的特征提取器。

您可以在 Magic Leap Community 组织下找到所有原始的 SuperPoint 检查点。

该模型由 stevenbucaille 贡献。

点击右侧侧边栏中的 SuperPoint 模型，查看有关如何将 SuperPoint 应用于不同计算机视觉任务的更多示例。

下面的示例演示了如何使用 AutoModel 类检测图像中的特征点。

import requests
import torch
from PIL import Image

from transformers import AutoImageProcessor, SuperPointForKeypointDetection


url = "http://images.cocodataset.org/val2017/000000039769.jpg"
image = Image.open(requests.get(url, stream=True).raw)

processor = AutoImageProcessor.from_pretrained("magic-leap-community/superpoint")
model = SuperPointForKeypointDetection.from_pretrained("magic-leap-community/superpoint", device_map="auto")

inputs = processor(image, return_tensors="pt").to(model.device)
with torch.no_grad():
    outputs = model(**inputs)

# Post-process to get keypoints, scores, and descriptors
image_size = (image.height, image.width)
processed_outputs = processor.post_process_keypoint_detection(outputs, [image_size])

注意事项

• SuperPoint 为每张图像输出动态数量的关键点，这使其适用于需要变长特征表示的任务。

  已复制

  from transformers import AutoImageProcessor, SuperPointForKeypointDetection
  import torch
  from PIL import Image
  import requests
  processor = AutoImageProcessor.from_pretrained("magic-leap-community/superpoint")
  model = SuperPointForKeypointDetection.from_pretrained("magic-leap-community/superpoint", device_map="auto")
  url_image_1 = "http://images.cocodataset.org/val2017/000000039769.jpg"
  image_1 = Image.open(requests.get(url_image_1, stream=True).raw)
  url_image_2 = "http://images.cocodataset.org/test-stuff2017/000000000568.jpg"
  image_2 = Image.open(requests.get(url_image_2, stream=True).raw)
  images = [image_1, image_2]
  inputs = processor(images, return_tensors="pt").to(model.device)
  # Example of handling dynamic keypoint output
  outputs = model(**inputs)
  keypoints = outputs.keypoints  # Shape varies per image
  scores = outputs.scores        # Confidence scores for each keypoint
  descriptors = outputs.descriptors  # 256-dimensional descriptors
  mask = outputs.mask # Value of 1 corresponds to a keypoint detection

• 该模型在单次前向传播中同时提供关键点坐标及其相应的描述符（256 维向量）。

• 对于多张图像的批量处理，您需要使用 mask 属性来检索每张图像的相应信息。您可以使用 SuperPointImageProcessor 中的 post_process_keypoint_detection 来检索每张图像的信息。

  已复制

  # Batch processing example
  images = [image1, image2, image3]
  inputs = processor(images, return_tensors="pt").to(model.device)
  outputs = model(**inputs)
  image_sizes = [(img.height, img.width) for img in images]
  processed_outputs = processor.post_process_keypoint_detection(outputs, image_sizes)

• 然后，您可以将关键点打印在所选的图像上以使结果可视化

  已复制

  import matplotlib.pyplot as plt
  plt.axis("off")
  plt.imshow(image_1)
  plt.scatter(
      outputs[0]["keypoints"][:, 0],
      outputs[0]["keypoints"][:, 1],
      c=outputs[0]["scores"] * 100,
      s=outputs[0]["scores"] * 50,
      alpha=0.8
  )
  plt.savefig(f"output_image.png")

资源

• 有关推理和可视化示例，请参阅此 notebook。

SuperPointConfig

>>> from transformers import SuperPointConfig, SuperPointForKeypointDetection

>>> # Initializing a SuperPoint superpoint style configuration
>>> configuration = SuperPointConfig()
>>> # Initializing a model from the superpoint style configuration
>>> model = SuperPointForKeypointDetection(configuration)
>>> # Accessing the model configuration
>>> configuration = model.config

SuperPointImageProcessor

SuperPointImageProcessorPil

SuperPointForKeypointDetection

>>> from transformers import AutoImageProcessor, SuperPointForKeypointDetection
>>> import torch
>>> from PIL import Image
>>> import httpx
>>> from io import BytesIO

>>> url = "http://images.cocodataset.org/val2017/000000039769.jpg"
>>> with httpx.stream("GET", url) as response:
...     image = Image.open(BytesIO(response.read()))

>>> processor = AutoImageProcessor.from_pretrained("magic-leap-community/superpoint")
>>> model = SuperPointForKeypointDetection.from_pretrained("magic-leap-community/superpoint")

>>> inputs = processor(image, return_tensors="pt")
>>> outputs = model(**inputs)