A Visual Query-Driven Search Engine for Brain Tissue Image Analysis

We present a versatile multiscale visual search engine for visual query-driven analysis of whole-slide multiplex IHC scans of brain tissue, without the confines, limitations, and programming needs of conventional script-based image analysis. Our unsupervised machine learning-based method adaptively learns the cytoarchitectural characteristics of provided training images, without any human effort or intervention. Then, visual queries can be submitted by indicating individual cell(s), and/or multicellular tissue patch(es) of interest, upon which the search engine retrieves a rank-ordered spatially mapped list of similar other cells or tissue patches based on similar cell morphologies, protein expression, cytoarchitecture, myeloarchitecture, vasculature, etc. Retrievals from multiple queries can be co-analyzed intuitively to generate complex inferences that would otherwise require sophisticated programming. We envision a broad range of uses, e.g., identifying cell populations, discovering cellular/cytoarchitectural similarities and differences across brain regions, delineating brain regions, fitting/refining/building atlases, delineating cortical cell layers, and proofreading automated image analysis results.