**TL;DR**

Cell segmentation in spatial transcriptomics is the step that turns raw images and barcoded spot grids into cell-level units you can trust. In this guide, we explain what cell segmentation is, compare StarDist, Cellpose, CellBin and SCS, and share practical tips for choosing a segmentation method for 10x Visium, Stereo-seq and high-resolution spatial epigenomics projects.

Figure 1. From images and barcoded spots to cell-level spatial readouts: choosing between StarDist, Cellpose, CellBin and SCS for cell segmentation in spatial transcriptomics.

Why Cell Segmentation Is the First Gate for Reliable Spatial Transcriptomics

If two groups use the same 10x Visium kit on similar tissues, why do their spatial clusters and cell–cell interaction maps sometimes look completely different? In many cases, the divergence starts long before clustering or domain detection—at the point where the pipeline decides where one cell ends and the next begins.

Cell segmentation sits between image registration and downstream spatial analysis. It influences:

• Cell type calling

  When two neighbouring cells are merged into one mask, their expression profiles blend into an artificial "hybrid" cell that does not exist in the tissue.

• Cell–cell interaction analysis

  Adjacency graphs and ligand–receptor maps depend on which cells count as neighbours. Shift the boundaries and the interaction landscape changes.

• Spatial heterogeneity and niche detection

  Overly fragmented or merged masks can blur gradients across layers, tumour margins or germinal centres and change inferred microenvironments.

Benchmarking work on imaging-based spatial assays and spatial transcriptomics consistently shows that segmentation choices can change cluster composition, neighbourhood statistics and even the apparent presence or absence of certain cell states. For teams running spatial transcriptomics services, and for project leads reading reports, segmentation is therefore the first gate: if the masks are wrong, every downstream interpretation sits on shaky ground.

What We Mean by Cell Segmentation in Spatial Transcriptomics

Cell segmentation in spatial transcriptomics is the process of turning tissue images and spatial barcodes into cell-level masks, so that each transcript is assigned to one cell.

Recent work such as the BOMS algorithm illustrates this idea by clustering spatial spots and gene labels into coherent cell instances that match image-derived boundaries.

Figure 2. Workflow of the BOMS algorithm for converting spatial transcriptomics spots and gene labels into cell instances, illustrating neighbourhood gene expression vectors, mean-shift clustering and final cell outlines (Kamboj O. et al. (2025) PLOS One).

In most spatial transcriptomics projects, that means:

• Starting from H&E, DAPI or multiplex IF images aligned to the capture area.
• Detecting nuclei as anchors for individual cells.
• Estimating cell boundaries by expanding around nuclei and separating touching cells.
• Intersecting barcoded spots or molecule coordinates with cell masks.
• Building a cell-by-gene expression matrix for downstream analysis.

If you have already worked through "Spatial Transcriptomics Data Analysis: A Practical Introduction" on the Spatial Omics Lab resource page, you can think of segmentation as the bridge between image alignment and the clustering, domain detection and cell type mapping steps described there. It is also closely connected to the computational themes covered in "Computational Strategies and Machine Learning for Spatial Genomics Data", which discusses how spatial structure and sparsity affect later models.

StarDist, Cellpose, CellBin and SCS: Core Methods at a Glance

Over the last few years, cell segmentation for spatial omics has shifted from classical watershed and thresholding to deep learning and hybrid image–transcript approaches. Among many tools, four families show up repeatedly in spatial pipelines:

• StarDist – U-Net-based model that represents objects as star-convex polygons.
• Cellpose – a generalist model that predicts flow fields guiding pixels into cell centres.
• CellBin – a Stereo-seq–oriented pipeline that uses chip design for precise registration.
• SCS – a Transformer-based method that integrates images and spatial transcriptomics spots.

For quick orientation, it helps to summarise them in a simple comparison table that readers and analysis tools can both parse.

This is not a ranking. Each method lives inside a broader spatial genomics or spatial epigenomics workflow, and the right choice depends on platform, tissue architecture and project goals. If you need a refresher on platforms themselves, the "Spatial Transcriptomics vs DBiT-seq vs Slide-seq Comparison Guide" on Spatial Omics Lab is a useful companion read.

How to Choose a Cell Segmentation Strategy for Your Platform, Tissue and Resolution

There is no universal "best" segmentation algorithm. Instead, there is a best-fit for your platform, stains, tissue type and resolution. A practical way to make that decision is to start with the technology and work outward.

10x Visium and related array-based spatial transcriptomics

For standard-resolution 10x Visium (55 μm spots) with H&E or DAPI images, StarDist is a natural first choice:

• It models nuclei and cells as star-convex shapes, which suits many epithelial and immune tissues.
• Its off-the-shelf models, including those integrated into Space Ranger, are trained on diverse tissue types.

From a practical standpoint:

• If nuclei are clear, moderately crowded and roughly round → StarDist is an efficient default.
• If your sample contains highly elongated or branched cells (neurons, fibroblasts, smooth muscle) → consider Cellpose or a hybrid approach where StarDist detects nuclei and Cellpose refines cell bodies.

For cell segmentation methods for 10x Visium, a small pilot test goes a long way: run both StarDist and Cellpose on 5–10 tiles, overlay masks, and see which model produces more realistic cell sizes and boundaries in your tissue of interest.

Stereo-seq and 1 μm grids

Stereo-seq introduces a 1 μm or near-cellular grid, where image-molecule registration and binning are critical. CellBin takes advantage of chip layout and track lines to:

• Align images and molecular coordinates at subcellular resolution.
• Segment tissue regions, nuclei and cells while preserving the underlying grid.
• Assign molecules to cells in a way that respects both imaging and chip design.

If you work with Stereo-seq spatial transcriptomics, CellBin is a strong default for primary analysis. For teams that want to experiment, it can sit alongside newer models such as SCS, with both run on the same dataset to compare segmentation quality and downstream readouts.

DBiT-seq, spatial ATAC-seq and ultra-dense data

High-resolution platforms such as DBiT-seq, as well as near-cellular spatial ATAC-seq datasets, produce dense grids of barcodes. In these settings:

• Neighbouring barcodes often belong to the same cell.
• Expression or chromatin accessibility patterns carry information about cell boundaries.

SCS addresses this by using a Transformer model that:

• Looks at spots and their nearest neighbours.
• Learns a direction from each point to its cell centre.
• Estimates the probability that each spot belongs to a cell.

This design makes SCS especially appealing for high-resolution spatial epigenomics projects, where you want segmentation that respects both images and local expression patterns.

When Cellpose is the better choice

Cellpose is particularly attractive when:

• Your lab uses multiple imaging modalities (live-cell fluorescence, fixed IF, H&E).
• You need a flexible model across several projects and tissues.
• You want to fine-tune a base model for your specific staining protocol.

A practical recipe is:

1. Run the default Cellpose model on a handful of representative tiles.
2. Overlay segmentation masks on the raw images and visually score performance.
3. If you see systematic errors in certain regions, fine-tune Cellpose with a small, manually annotated dataset from your own project.

This kind of "human-in-the-loop" refinement is often faster than writing a custom algorithm from scratch.

Figure 3. Effect of incorporating Cellpose-derived flow fields as auxiliary information in BOMS segmentation for Allen smFISH and osmFISH datasets, demonstrating how increasing the influence of Cellpose flows improves alignment between BOMS and image-based cell boundaries (Kamboj O. et al. (2025) PLOS One).

Practical Workflow: From H&E / DAPI Images to Cell-Level Expression Matrices

Once you have chosen a segmentation strategy, you still need a clean, repeatable workflow that converts images and spatial barcodes into cell-level data. A robust spatial transcriptomics data analysis pipeline often follows these steps:

1. Image QC and preprocessing

• Check for focus issues, stitching artefacts and uneven staining.
• Apply basic background correction and illumination normalisation so models see consistent contrast.

2. Tiling and model selection

• Split whole-slide images into overlapping tiles to control memory usage.
• Decide whether to use StarDist, Cellpose, CellBin or SCS "as is" or to fine-tune on project-specific annotations.

3. Segmentation inference

• Run the chosen model across all tiles, ideally on GPUs to keep runtimes reasonable.
• Export label images or vector polygons for each segmented cell.

4. Spot or molecule assignment

• For array-based spatial transcriptomics (e.g., 10x Visium): assign each spot barcode to the cell mask that covers its centre.
• For molecule-based platforms and high-resolution spatial ATAC-seq: assign each (x, y) coordinate to the containing cell polygon.

5. Cell-by-gene matrix construction and filtering

• Sum counts by cell and gene to create a cell-by-gene matrix.
• Filter out cells that are too small, too large or abnormally low in counts.

6. Integration with downstream analysis

• Feed the matrix into your existing pipeline for cell type mapping, spatial clustering or multi-omics integration.

Two pieces of hands-on advice from real projects:

• Always inspect 50–100 random overlays (image + mask + spot positions) before trusting a full run. Many subtle errors are obvious to the eye but not to summary metrics.
• Keep coordinate systems consistent. Mixing pixels, micrometres and array indices without clear conversions is a common source of hidden misalignment.

CD Genomics typically embeds these steps in broader spatial genomics services, so that project teams receive both segmented cell-by-gene matrices and documentation of how those masks were generated.

Quality Control, Metrics and Common Failure Modes in Cell Segmentation

Segmentation is an approximation. The key question is not "Is it perfect?" but "Are remaining errors small enough that they do not change the biological story?"

Figure 4. Examples of BOMS cell segmentation results on several imaging-based spatial transcriptomics datasets, showing coloured molecules, overlaid cell boundaries and typical success and failure modes such as missed detections and merged cells (Kamboj O. et al. (2025) PLOS One).

Practical QC metrics you can track

Easy-to-compute checks that fit into most pipelines include:

• Cell size distribution

  Abnormally tiny or extremely large masks can signal noise, doublets or merged clusters.

• Nuclei-to-cell ratio

  A very high ratio may suggest oversegmentation into fragments; a very low ratio can reflect merged cells or missing objects.

• Fraction of unassigned counts

  If many barcodes fall outside any cell mask, you may be losing a large share of signal at tissue edges or in low-contrast regions.

• Per-cell expression outliers

  Masks with exceptionally high total counts or unusual marker patterns may be artefacts, especially if they cluster near staining artefacts or tears.

In published evaluations, researchers sometimes also report object-level metrics such as F1-scores, Dice coefficients and boundary precision against expert annotations. You may not have labelled ground truth, but you can still compare these metrics across methods or parameter sets within your own project.

Figure 5. Quantitative comparison of BOMS with Baysor, pciSeq and original "silver-standard" segmentations across multiple spatial transcriptomics datasets, including runtime, mutual information with silver-standard masks and concordance of differentially expressed genes (Kamboj O. et al. (2025) PLOS One).

Common failure modes—and how to fix them

Across different tissues and platforms, several failure patterns recur:

• Merged cells in dense tissue regions
  • Where it shows up: lymphoid organs, tumour nests, germinal centres.
  • Symptoms: very large masks spanning multiple nuclei.
  • Fixes: lower object probability thresholds; use models like StarDist that bias towards splitting; add post-processing that splits large objects along intensity valleys.
• Fragmented large cells (oversegmentation)
  • Where it shows up: hepatocytes, certain neurons, stromal cells with broad cytoplasm.
  • Symptoms: many tiny masks surrounding one nucleus or bright region.
  • Fixes: adjust minimum object size filters; consider Cellpose for more flexible shapes; refine training data to include large, intact cells.
• Masks leaking into background or artefacts
  • Where it shows up: uneven backgrounds, debris, folded tissue.
  • Symptoms: cells apparently growing over blank slide or stray bright spots.
  • Fixes: improve background subtraction; include "pure background" examples in training; use conservative thresholds in mask expansion.
• Misassigned barcodes near tissue edges
  • Where it shows up: tissue boundaries, tears, holes.
  • Symptoms: many spots unassigned or assigned to implausible cells.
  • Fixes: inspect edge areas; expand masks slightly while tracking which counts come from edge pixels; in some cases, drop a thin edge band from downstream distance-sensitive analyses.

In our experience, it is useful to treat segmentation like any other assay: document parameters, review QC plots and avoid mixing very different segmentation settings across samples in the same comparison. That principle also appears in spatial QC-oriented resources such as "Spatial Transcriptomics Data Analysis: A Practical Introduction" and "Spatial ATAC-seq Tools and Datasets", which emphasise consistency across samples.

From Segmentation Masks to Biology: Cell Typing, Spatial Interactions and Epigenomic Context

Segmentation is not the final deliverable. Its value comes from the biological questions it enables you to answer more confidently.

Cell typing and spatial clustering

Once each cell has a well-defined expression profile, you can:

• Cluster cells and annotate types using marker genes.
• Map known structures such as cortical layers, tumour–stroma boundaries or germinal centres.
• Compare spatial patterns with those described in "Methods and Application of Spatial Transcriptomics" and "Applications of Spatial Genomics in Oncology and Developmental Biology."

Poor segmentation tends to show up here as blurred layer boundaries, unexpected double-positive populations or clusters that do not match known histology.

Cell–cell interactions and microenvironment analysis

Many teams invest in spatial transcriptomics to understand local microenvironments:

• Which cell types sit next to each other?
• Where are specific ligand–receptor pairs enriched?
• How do immune cells, tumour cells and stromal cells organise in space?

These analyses use adjacency graphs built on segmented cells. If two real cells are merged into one mask, you may create artificial "hub" cells with many partners. If one real cell is broken into fragments, you may dilute genuine interactions.

Whenever you see surprising cell–cell communication results, it is worth checking whether they survive modest changes in segmentation parameters or algorithms.

Integration with spatial epigenomics

For spatial epigenomics services, segmentation has a parallel role:

• In spatial ATAC-seq, it defines which barcodes belong to the same cell, producing cell-level chromatin accessibility profiles that researchers can combine with external high-resolution transcriptome data or spatial transcriptomics in downstream studies.
• In spatial CUT&Tag, it helps localise histone mark signals within nuclei and across cell boundaries, especially when you integrate signal tracks with gene expression.

Resource articles such as "Spatial ATAC-seq Experimental Workflow and Principles" and "Spatial ATAC-seq & scRNA-seq Integration Strategy for Spatial Epigenomics" describe many of these ideas. Robust segmentation simply ensures that chromatin and histone mark signals are assigned to the correct cells before you apply those integration strategies.

How CD Genomics Supports Cell Segmentation in Spatial Transcriptomics and Epigenomics Projects

Building and maintaining a segmentation-aware spatial pipeline requires image processing, deep learning and domain knowledge about each platform. Many research teams therefore prefer to collaborate with experienced partners rather than managing every step internally.

Figure 6. Support from CD Genomics: segmentation-aware spatial pipelines that turn raw spatial data into QC-documented cell-level matrices across spatial transcriptomics, epigenomics and integrated spatial genomics projects.

At CD Genomics' Spatial Omics Lab, segmentation sits inside broader:

• Spatial transcriptomics services – including 10x Genomics Visium projects on frozen and FFPE tissues.
• Spatial epigenomics services – such as spatial ATAC-seq and spatial CUT&Tag, where cell-level maps of chromatin and histone marks are essential.
• Spatial genomics services – integrating segmentation, cell type mapping, domain detection and multi-omics analysis.

In typical projects, support includes:

• Reviewing platform, tissue type and staining to recommend StarDist, Cellpose, CellBin, SCS or hybrid approaches.
• Configuring segmentation workflows with version-controlled code, so results are reproducible and transparent.
• Generating QC summaries that report cell size distributions, unassigned fractions and example overlays.
• Linking segmentation results to downstream analyses in line with the workflows described in Spatial Omics Lab resource articles.

All services are provided strictly for research use only and are not intended for any clinical diagnosis, treatment or individual patient decisions.

If you are planning a new spatial transcriptomics or spatial epigenomics study:

• Use this guide as a checklist when you review or design your cell segmentation strategy.
• Explore related resources on data analysis, machine learning and spatial ATAC-seq workflows on the Spatial Omics Lab website.
• Consider discussing your project with the CD Genomics team to design a segmentation and analysis plan that matches your platform, tissue and research questions.

Frequently Asked Questions About Cell Segmentation in Spatial Transcriptomics

1. Do I always need cell segmentation for spatial transcriptomics?

Not always. For some exploratory questions, spot-level or bin-based analysis on Visium or high-resolution arrays can already show useful patterns. However, if you need cell type-specific expression, cell–cell interaction maps or integration with high-resolution RNA-seq datasets from dissociated cells, cell segmentation is usually required for robust results.

2. How do I decide between StarDist and Cellpose for 10x Visium H&E images?

As a starting point, use StarDist when nuclei are roughly round and moderately crowded, because its star-convex shapes match that regime well. If your tissue contains very irregular, elongated or branched cells, or you expect to reuse the same model across several imaging conditions, Cellpose may be more flexible. In practice, testing both models on a small set of tiles and visually comparing overlays is the fastest way to decide.

3. Can I reuse the same segmentation model across tissue types and projects?

Pre-trained models are often surprisingly robust, but they are not universal. Changes in staining intensity, microscope settings or tissue architecture can reduce performance. If you see systematic errors after overlay inspection—such as consistent merging in a particular region—it may be worth fine-tuning the model using a small, manually annotated training set from your new project.

4. How do segmentation errors affect my cell–cell interaction analysis?

Segmentation errors affect who counts as neighbours and which ligand and receptor counts are assigned to each cell. Merged masks can create artificial "high-interaction" cells with combined expression, while fragmented masks can split real interactions across several small objects. When interaction results drive key conclusions, it is wise to check whether they are stable under modest changes in segmentation settings.

References

1. Kamboj, Simarpreet Singh, et al. "From Spots to Cells: Cell Segmentation in Spatial Transcriptomics with BOMS." PLOS ONE, vol. 20, no. 6, 2025, e0311458. PLOS, doi:10.1371/journal.pone.0311458.
2. Schmidt, Uwe, et al. "Cell Detection with Star-Convex Polygons." Medical Image Computing and Computer-Assisted Intervention – MICCAI 2018. Lecture Notes in Computer Science, vol. 11071, Springer, 2018, pp. 265–273. arXiv:1806.03535.
3. Stringer, Carsen, et al. "Cellpose: A Generalist Algorithm for Cellular Segmentation." Nature Methods, vol. 18, no. 1, 2021, pp. 100–106. doi:10.1038/s41592-020-01018-x.
4. Chen, Hao, Dongshunyi Li, and Ziv Bar-Joseph. "SCS: Cell Segmentation for High-Resolution Spatial Transcriptomics." Nature Methods, vol. 20, no. 8, 2023, pp. 1237–1243. doi:10.1038/s41592-023-01939-3.
5. Li, M., et al. "CellBin: A Highly Accurate Single-Cell Gene Expression Matrix Construction Method for Stereo-seq." bioRxiv, 2023. doi:10.1101/2023.02.28.530414.
6. Moses, Lambda, and Lior Pachter. "Museum of Spatial Transcriptomics." Nature Methods, vol. 19, no. 5, 2022, pp. 534–546. doi:10.1038/s41592-022-01409-2.
7. Chen, Ao, et al. "Spatiotemporal Transcriptomic Atlas of Mouse Organogenesis Using Stereo-seq." Cell, vol. 185, no. 10, 2022, pp. 1777–1792.e21. doi:10.1016/j.cell.2022.03.047.
8. Deng, Yang, et al. "Spatial Profiling of Chromatin Accessibility in Mouse and Human Tissues." Nature, vol. 609, no. 7929, 2022, pp. 796–806. doi:10.1038/s41586-022-05094-1.
9. Deng, Yang, et al. "Spatially Resolved Chromatin Modification Profiling at the Tissue Level." Science, vol. 375, no. 6580, 2022, pp. 681–686. doi:10.1126/science.abg7216.
10. Llorens-Bobadilla, Enric, et al. "Solid-Phase Capture and Profiling of Open Chromatin by Spatial ATAC." Nature Biotechnology, vol. 41, 2023, pp. 1085–1088. doi:10.1038/s41587-022-01603-9.
11. Li, Hu, et al. "Spatially Resolved Genome-Wide Joint Profiling of Transcription and the Epigenome in Tissues." Nature Protocols, 2025. doi:10.1038/s41596-025-01145-9.
12. Teng, Haotian, Ye Yuan, and Ziv Bar-Joseph. "Clustering Spatial Transcriptomics Data." Bioinformatics, vol. 38, no. 4, 2022, pp. 997–1004. doi:10.1093/bioinformatics/btab711.