🟠 Large Model Prompt Segmentation

Function Description

The operator integrates the Segment Anything Model, i.e., SAM model. SAM can segment any object in images based on user-provided real-time prompt points and prompt boxes without requiring additional training.

  • Prompt points: Provide one or multiple points on the target object to guide the model for segmentation.

  • Prompt boxes: Provide an approximate bounding box to let the model segment the main object within the box.

Usage Scenarios

Suitable for segmenting objects that have never been trained.

Input Output

Input

Image: Single color image to perform segmentation.

Prompt point list: A list containing N (X,Y) pixel coordinates, used to specify one or multiple points on objects to be segmented.

Prompt box list: One or multiple 2D rectangular boxes, used to prompt the model to segment objects within the boxes.

Output

Detection result: Detection instance list. Each element in the list represents a segmented object, containing its user-specified class, model-given confidence score and contour polygon.

Parameter Description

  • The operation of this operator depends on the Python library segment-anything. If your environment hasn’t installed it yet, please use the command pip install segment-anything to install.

  • Either "prompt point list" or "prompt box list" at the input end must be provided, otherwise the operator cannot execute. Inputs can be obtained through image processing or point cloud processing conversions such as point cloud bounding box to 2D bounding box, point cloud point to pixel point nodes.

Weight File

Parameter Description

Load officially released SAM model weight files (.pth format). The model size determines its accuracy, speed and resource consumption. Model download address (internal network):

Parameter Adjustment

Base model has the fastest speed and smallest resource occupation, suitable for scenarios with high real-time requirements; Large/Huge models have higher accuracy but slower speed and require more GPU memory. Please choose the appropriate model file according to actual application needs.

Enable GPU

Parameter Description

Controls whether the operator uses CPU or GPU for computation.

Parameter Adjustment

  • Enabled (recommended): SAM model has extremely large computation, especially for Large and Huge models, it’s almost necessary to run on GPU to obtain acceptable performance.

  • Disabled: Uses CPU for computation, speed will be very slow.

Confidence Threshold

Parameter Description

SAM will give each generated segmentation result an IOU prediction score (confidence), filtering out segmentation results below this score through this threshold.

Parameter Adjustment

  • Increase threshold: Get more reliable segmentation results, but may filter out some correct segmentations;

  • Decrease threshold: More lenient filtering, but may introduce more incorrect segmentations.

Parameter Range

[0,1], default value: 0.5

Output Multiple Results

Parameter Description

For a potentially ambiguous prompt, the model can generate multiple logically reasonable segmentation results.

Parameter Adjustment

  • Disabled (default): For a single prompt, the operator only outputs the one segmentation result that the model considers most reliable.

  • Enabled: For a single prompt, the operator will output multiple possible segmentation results (for example, an object, part of the object, larger whole containing the object).

Class Name

Parameter Description

Used to manually specify a class name for the final output segmentation results.

Parameter Adjustment

SAM itself doesn’t recognize any specific objects, the "class" of its segmentation results is specified by this parameter.