Image-based drug testing in patient biopsies has recently been shown to identify potent treatments for patients suffering from relapsed or refractory hematological cancers. Here we investigate the use of weakly-supervised deep learning on cell morphologies (DML) to complement immunofluorescence (IF) in the classification of cancer and healthy cells in such drug testing. Across 390 biopsies from 289 patients with diverse blood cancers, DML-based drug responses show improved reproducibility and clustering of drugs with the same mode of action. DML does so by adapting to batch effects and by autonomously recognizing disease-associated cell morphologies. In a post-hoc analysis of 66 patients, DML-recommended treatments led to improved progression free survival compared to IF-based recommendations and physician’s choice-based treatments. Treatments recommended by both IF and DML enriched for patients achieving exceptional clinical responses. Thus, DML-enhanced ex vivo drug screening is a promising new tool in the identification of effective personalized treatments.