Deep-Learning Holography Detects Elusive Circulating Tumor Cells in Prostate Cancer

By Dr. Chinta SidharthanReviewed by Susha Cheriyedath, M.Sc.Jul 27 2026

A pilot platform combines cell morphology with molecular profiling to capture rare circulating cells that reliance on a single epithelial marker may leave undetected.

Paper: Circulating tumor cell detection in cancer patients using in-flow deep learning holography

Research on circulating tumor cells (CTCs) has significantly advanced the study of cancer metastasis and treatment response. Studies are now exploring whether combining optical imaging with artificial intelligence (AI) can improve the detection of these rare cells.

In a recent study published in npj Biosensing, researchers at the University of Minnesota and Astrin Biosciences described a new system integrating light-based imaging with AI-driven analysis to identify CTCs more reliably.

CTCs as biomarkers

Cancer cells that break away from a tumor and travel through the bloodstream may reach distant organs, where some can seed metastases, and may be detectable before scans or symptoms reveal the disease. A simple blood test that can spot these CTCs has long been the focus of oncological research, since drawing blood is far easier than invasive biopsies of a tumor.

The challenge has been that the concentration of these cells in blood is very low, and CTCs look different from one patient to the next. Moreover, physical sorting methods that separate cells by size or density encounter a similar problem, since tumor cells can closely resemble ordinary white blood cells and be missed or mistaken.

About the study

In the present study, the research team built a platform that processed whole blood using three integrated technologies: microfluidic enrichment followed by dual-modality imaging. Blood was first diluted and passed through a specialized microfluidic chip that used the physical forces of fluid flow to remove more than 99.999% of red blood cells and approximately 99.6% of white blood cells while retaining around 95% of spiked cancer cells.

Samples begin as whole blood prior to microfluidic enrichment, during which red blood cells are primarily depleted while retaining CTCs and most WBCs. Subsequently, holograms of each cell and IF signal from the field-of-view as a whole are captured during passage through a second microfluidic chip. Detections in the IF signal for emitting cells are visible as broad peaks whose time-scale is inversely proportional to the frame rate (i.e., equivalent to the time taken for a cell to passage the field-of-view). Holograms pass through a neural network to classify each cell and detect CTCs, while the IF data can be used for additional filtering to support cell enumeration.

The remaining sample was then passed through a second chip, where a pulsed 405 nm laser and digital camera captured holographic images of cells as they flowed through the channel. Unlike conventional microscopy, this technique allows both the shape and the optical thickness of a cell to be recorded without the need for chemical staining. This allowed the holographic component to examine cells based on their physical structure, although the overall classification system also used fluorescent molecular labels.

Two fluorescent channels were integrated with the imaging setup to detect prostate-specific membrane antigen (PSMA) and epithelial cell adhesion molecule (EpCAM), two proteins associated with prostate cancer and epithelial cells, respectively.

A deep learning model, built on a modified high-resolution neural network, converted each hologram into a probability map that flagged likely cell locations. The curated development dataset included 5.9 million images from healthy blood samples and 2.0 million images from five cancer cell lines. A separate set of 1.1 million images from 25 cancer cell lines was used to estimate recovery under controlled conditions, testing whether the model generalized beyond the cell lines used during development.

Positive and negative samples are generated by imaging cell lines and healthy blood (post-microfluidic enrichment) in separate samples. All images derived from healthy blood are labeled negatively (i.e., lacking any Gaussian keypoint label). Cell lines are spiked into the buffer, and all nucleated cells are labeled as positive using a pseudo-label neural network trained on human-generated labels. The pseudo-labeler is never applied to images with blood cells. Although the pseudo-labeling model is trained on human-generated labels, the CTC detection model can be trained automatically, enabling the use of large datasets. During each training epoch, the effects of false negatives from the pseudo-labeler (i.e., unlabeled cell line cells) are mitigated using pixel-wise asymmetric loss, wherein CTC model predictions lacking a corresponding pseudo-labeler label (depicted as yellow Gaussian blobs) are penalized at 1/10^th the rate of false positives and false negatives. Each training epoch also uses hard sample mining such that images from low-performing runs from healthy controls are preferentially sampled in subsequent epochs.

The researchers then used 18 healthy donor blood samples spiked with LNCaP prostate cancer cells after microfluidic enrichment to test the downstream imaging and detection stages and compared detection rates with those from 18 unspiked control samples. Before spiking, the LNCaP cells were labeled with a cell-tracker dye, creating favorable fluorescence conditions with uniform marker expression and minimal nonspecific labeling. They then applied the platform to blood drawn from 13 men with metastatic castration-resistant prostate cancer and eight healthy male donors, and classified a cell as a CTC candidate only when it met both the imaging model's confidence threshold and showed a positive PSMA signal.

Key findings

The study found that this combined imaging and labeling approach classified more CTC candidates in cancer patients than in healthy donors, while also indicating that many of these cells could potentially be missed by EpCAM-based methods. Patients with prostate cancer showed a median of 12.5 CTC candidates per milliliter of blood, compared to 1.5 cells per milliliter in healthy donors, demonstrating higher median counts in this small pilot cohort.

The study also revealed an interesting finding regarding the EpCAM protein that many existing tests depend on. Only 37% of the CTC candidates that exceeded the holography threshold and were PSMA-positive also carried EpCAM, suggesting that EpCAM-dependent platforms could miss some of these candidates. However, the researchers did not directly compare the platform with a conventional CTC assay. This finding is consistent with possible epithelial-to-mesenchymal phenotypic changes, during which EpCAM can be downregulated, but does not establish that these cells were undergoing this process or were more aggressive.

In the separate laboratory validation experiment, at the prespecified model threshold of 0.5, the system correctly identified about 60% of spiked tumor cells while producing five false-positive events across approximately 55 milliliters of unspiked blood from the 18 laboratory control samples, equivalent to fewer than 0.1 false positives per milliliter. Even at the more permissive threshold of 0.1, the false-positive rate remained below one cell per milliliter. A model-based calculation estimated a cell-level positive predictive value of approximately 98%, but this estimate assumed a CTC abundance of 10 cells per milliliter and was not measured directly in the clinical cohort.

However, the authors acknowledged that their patient cohort included only 13 men with late-stage prostate cancer, and that the patient and healthy-control samples came from separate studies, meaning the analysis was not strictly blinded. The platform was not compared directly with an established CTC assay, and broader testing across cancer stages and tumor types, along with disease-control cohorts that include inflammatory or infectious conditions, remains necessary before the tool can move toward clinical use.

Conclusions

In summary, the study provided pilot evidence that combining label-free imaging with targeted molecular markers can identify PSMA-positive CTC candidates, including many that lacked EpCAM and might therefore be missed by assays that rely on EpCAM. These findings pointed toward the platform's potential use in future blood tests to track cancer progression and treatment response without invasive biopsies, although these clinical applications were not evaluated in the study and further validation across diverse patient groups and tumor types remains an important next step.