Touchstone

Touchstone

[NeurIPS 2024] Touchstone - Benchmarking AI on 5,172 o.o.d. CT volumes and 9 anatomical structures

Stars: 98

Visit
screenshot

README:

Touchstone Benchmark

Touchstone Benchmark

Participate - Touchstone 1.0 Participate - Touchstone 2.0

touchstone leaderboard touchstone dataset touchstone model
visitors GitHub Repo stars Follow on Twitter
Subscribe us: https://groups.google.com/u/2/g/bodymaps

We present Touchstone, a large-scale medical segmentation benchmark based on annotated 5,195 CT volumes from 76 hospitals for training, and 6,933 CT volumes from 8 additional hospitals for testing. We invite AI inventors to train their models on AbdomenAtlas, and we independently evaluate their algorithms. We have already collaborated with 14 influential research teams, and we remain accepting new submissions.

Paper

Touchstone Benchmark: Are We on the Right Way for Evaluating AI Algorithms for Medical Segmentation?
Pedro R. A. S. Bassi1, Wenxuan Li1, Yucheng Tang2, Fabian Isensee3, ..., Alan Yuille1, Zongwei Zhou1
1Johns Hopkins University, 2NVIDIA, 3DKFZ
NeurIPS 2024
YouTube

Touchstone 1.0 Leaderboard

rank model organization average DSC paper github
🏆 MedNeXt DKFZ 89.2 arXiv GitHub stars
🏆 STU-Net-B Shanghai AI Lab 89.0 arXiv GitHub stars
🏆 MedFormer Rutgers 89.0 arXiv GitHub stars
🏆 nnU-Net ResEncL DKFZ 88.8 arXiv GitHub stars
🏆 UniSeg NPU 88.8 arXiv GitHub stars
🏆 Diff-UNet HKUST 88.5 arXiv GitHub stars
🏆 LHU-Net UR 88.0 arXiv GitHub stars
🏆 NexToU HIT 87.8 arXiv GitHub stars
9 SegVol BAAI 87.1 arXiv GitHub stars
10 U-Net & CLIP CityU 87.1 arXiv GitHub stars
11 Swin UNETR & CLIP CityU 86.7 arXiv GitHub stars
12 Swin UNETR NVIDIA 80.1 arXiv GitHub stars
13 UNesT NVIDIA 79.1 arXiv GitHub stars
14 SAM-Adapter Duke 73.4 arXiv GitHub stars
15 UNETR NVIDIA 64.4 arXiv GitHub stars
Aorta - NexToU 🏆
rank model organization DSC paper github
🏆 NexToU HIT 86.4 arXiv GitHub stars
2 MedNeXt DKFZ 83.1 arXiv GitHub stars
3 UniSeg NPU 82.3 arXiv GitHub stars
4 STU-Net-B Shanghai AI Lab 82.1 arXiv GitHub stars
5 nnU-Net ResEncL DKFZ 81.4 arXiv GitHub stars
6 Diff-UNet HKUST 81.2 arXiv GitHub stars
7 Swin UNETR NVIDIA 81.1 arXiv GitHub stars
8 SegVol BAAI 80.2 arXiv GitHub stars
9 UNesT NVIDIA 78.6 arXiv GitHub stars
10 Swin UNETR & CLIP CityU 78.1 arXiv GitHub stars
11 U-Net & CLIP CityU 77.1 arXiv GitHub stars
12 SAM-Adapter Duke 62.8 arXiv GitHub stars
13 UNETR NVIDIA 52.1 arXiv GitHub stars
Gallbladder - STU-Net-B & MedNeXt 🏆
rank model organization DSC paper github
🏆 STU-Net-B Shanghai AI Lab 85.5 arXiv GitHub stars
🏆 MedNeXt DKFZ 85.3 arXiv GitHub stars
3 nnU-Net ResEncL DKFZ 84.9 arXiv GitHub stars
4 UniSeg NPU 84.7 arXiv GitHub stars
5 Diff-UNet HKUST 83.8 arXiv GitHub stars
6 NexToU HIT 82.3 arXiv GitHub stars
7 U-Net & CLIP CityU 82.1 arXiv GitHub stars
8 Swin UNETR & CLIP CityU 80.2 arXiv GitHub stars
9 SegVol BAAI 79.3 arXiv GitHub stars
10 Swin UNETR NVIDIA 69.2 arXiv GitHub stars
11 UNesT NVIDIA 62.1 arXiv GitHub stars
12 SAM-Adapter Duke 49.4 arXiv GitHub stars
13 UNETR NVIDIA 43.8 arXiv GitHub stars
KidneyL - Diff-UNet 🏆
rank model organization DSC paper github
🏆 Diff-UNet HKUST 91.9 arXiv GitHub stars
2 nnU-Net ResEncL DKFZ 91.9 arXiv GitHub stars
3 STU-Net-B Shanghai AI Lab 91.9 arXiv GitHub stars
4 MedNeXt DKFZ 91.8 arXiv GitHub stars
5 SegVol BAAI 91.8 arXiv GitHub stars
6 UniSeg NPU 91.5 arXiv GitHub stars
7 U-Net & CLIP CityU 91.1 arXiv GitHub stars
8 Swin UNETR & CLIP CityU 91.0 arXiv GitHub stars
9 NexToU HIT 89.6 arXiv GitHub stars
10 SAM-Adapter Duke 87.3 arXiv GitHub stars
11 Swin UNETR NVIDIA 85.5 arXiv GitHub stars
12 UNesT NVIDIA 85.4 arXiv GitHub stars
13 UNETR NVIDIA 63.7 arXiv GitHub stars
KidneyR - Diff-UNet 🏆
rank model organization DSC paper github
🏆 Diff-UNet HKUST 92.8 arXiv GitHub stars
2 MedNeXt DKFZ 92.6 arXiv GitHub stars
3 nnU-Net ResEncL DKFZ 92.6 arXiv GitHub stars
4 STU-Net-B Shanghai AI Lab 92.5 arXiv GitHub stars
5 SegVol BAAI 92.5 arXiv GitHub stars
6 UniSeg NPU 92.2 arXiv GitHub stars
7 U-Net & CLIP CityU 91.9 arXiv GitHub stars
8 Swin UNETR & CLIP CityU 91.7 arXiv GitHub stars
9 SAM-Adapter Duke 90.4 arXiv GitHub stars
10 NexToU HIT 90.1 arXiv GitHub stars
11 UNesT NVIDIA 83.6 arXiv GitHub stars
12 Swin UNETR NVIDIA 81.7 arXiv GitHub stars
13 UNETR NVIDIA 69.6 arXiv GitHub stars
Liver - MedNeXt 🏆
rank model organization DSC paper github
🏆 MedNeXt DKFZ 96.3 arXiv GitHub stars
2 nnU-Net ResEncL DKFZ 96.3 arXiv GitHub stars
3 Diff-UNet HKUST 96.2 arXiv GitHub stars
4 STU-Net-B Shanghai AI Lab 96.2 arXiv GitHub stars
5 UniSeg NPU 96.1 arXiv GitHub stars
6 U-Net & CLIP CityU 96.0 arXiv GitHub stars
7 SegVol BAAI 96.0 arXiv GitHub stars
8 Swin UNETR & CLIP CityU 95.8 arXiv GitHub stars
9 NexToU HIT 95.7 arXiv GitHub stars
10 SAM-Adapter Duke 94.1 arXiv GitHub stars
11 UNesT NVIDIA 93.6 arXiv GitHub stars
12 Swin UNETR NVIDIA 93.5 arXiv GitHub stars
13 UNETR NVIDIA 90.5 arXiv GitHub stars
Pancreas - MedNeXt 🏆
rank model organization DSC paper github
🏆 MedNeXt DKFZ 83.3 arXiv GitHub stars
2 STU-Net-B Shanghai AI Lab 83.2 arXiv GitHub stars
3 nnU-Net ResEncL DKFZ 82.9 arXiv GitHub stars
4 UniSeg NPU 82.7 arXiv GitHub stars
5 Diff-UNet HKUST 81.9 arXiv GitHub stars
6 U-Net & CLIP CityU 80.8 arXiv GitHub stars
7 Swin UNETR & CLIP CityU 80.2 arXiv GitHub stars
8 NexToU HIT 80.2 arXiv GitHub stars
9 SegVol BAAI 79.1 arXiv GitHub stars
10 Swin UNETR NVIDIA 68.5 arXiv GitHub stars
11 UNesT NVIDIA 68.3 arXiv GitHub stars
12 UNETR NVIDIA 55.1 arXiv GitHub stars
13 SAM-Adapter Duke 50.2 arXiv GitHub stars
Postcava - STU-Net-B & MedNeXt 🏆
rank model organization DSC paper github
🏆 STU-Net-B Shanghai AI Lab 81.3 arXiv GitHub stars
🏆 MedNeXt DKFZ 81.3 arXiv GitHub stars
3 UniSeg NPU 81.2 arXiv GitHub stars
4 Diff-UNet HKUST 80.8 arXiv GitHub stars
5 nnU-Net ResEncL DKFZ 80.5 arXiv GitHub stars
6 U-Net & CLIP CityU 78.5 arXiv GitHub stars
7 NexToU HIT 78.1 arXiv GitHub stars
8 SegVol BAAI 77.8 arXiv GitHub stars
9 Swin UNETR & CLIP CityU 76.8 arXiv GitHub stars
10 Swin UNETR NVIDIA 69.9 arXiv GitHub stars
11 UNesT NVIDIA 66.2 arXiv GitHub stars
12 UNETR NVIDIA 53.9 arXiv GitHub stars
13 SAM-Adapter Duke 48.0 arXiv GitHub stars
Spleen - nnU-Net ResEncL 🏆
rank model organization DSC paper github
🏆 nnU-Net ResEncL DKFZ 95.2 arXiv GitHub stars
2 MedNeXt DKFZ 95.2 arXiv GitHub stars
3 STU-Net-B Shanghai AI Lab 95.1 arXiv GitHub stars
4 Diff-UNet HKUST 95.0 arXiv GitHub stars
5 UniSeg NPU 94.9 arXiv GitHub stars
6 SegVol BAAI 94.5 arXiv GitHub stars
7 NexToU HIT 94.7 arXiv GitHub stars
8 U-Net & CLIP CityU 94.3 arXiv GitHub stars
9 Swin UNETR & CLIP CityU 94.1 arXiv GitHub stars
10 SAM-Adapter Duke 90.5 arXiv GitHub stars
11 Swin UNETR NVIDIA 87.9 arXiv GitHub stars
12 UNesT NVIDIA 86.7 arXiv GitHub stars
13 UNETR NVIDIA 76.5 arXiv GitHub stars
Stomach - STU-Net-B & MedNeXt & nnU-Net ResEncL 🏆
rank model organization DSC paper github
🏆 STU-Net-B Shanghai AI Lab 93.5 arXiv GitHub stars
🏆 MedNeXt DKFZ 93.5 arXiv GitHub stars
🏆 nnU-Net ResEncL DKFZ 93.4 arXiv GitHub stars
4 UniSeg NPU 93.3 arXiv GitHub stars
5 Diff-UNet HKUST 93.1 arXiv GitHub stars
6 NexToU HIT 92.7 arXiv GitHub stars
7 SegVol BAAI 92.5 arXiv GitHub stars
8 U-Net & CLIP CityU 92.4 arXiv GitHub stars
9 Swin UNETR & CLIP CityU 92.2 arXiv GitHub stars
10 SAM-Adapter Duke 88.0 arXiv GitHub stars
11 UNesT NVIDIA 87.6 arXiv GitHub stars
12 Swin UNETR NVIDIA 84.1 arXiv GitHub stars
13 UNETR NVIDIA 74.2 arXiv GitHub stars

Touchstone 1.0 Dataset

Training set

Test set

metadata

Figure 1. Metadata distribution in the test set.

Touchstone 1.0 Model

[!NOTE] We are releasing the trained AI models evaluated in Touchstone right here. Stay tuned!

rank model average DSC parameter infer. speed download
🏆 MedNeXt 89.2 61.8M ★☆☆☆☆
🏆 STU-Net-B 89.0 58.3M ★★☆☆☆ checkpoint
🏆 MedFormer 89.0 38.5M ★★★☆☆
🏆 nnU-Net ResEncL 88.8 102.0M ★★★★☆ checkpoint
🏆 UniSeg 88.8 31.0M ☆☆☆☆☆
🏆 Diff-UNet 88.5 434.0M ★★★☆☆
🏆 LHU-Net 88.0 8.6M ★★★★★
🏆 NexToU 87.8 81.9M ★★★★☆ checkpoint
9 SegVol 87.1 181.0M ★★★★☆ checkpoint
10 U-Net & CLIP 87.1 19.1M ★★★☆☆
11 Swin UNETR & CLIP 86.7 62.2M ★★★☆☆
12 Swin UNETR 80.1 72.8M ★★★★★
13 UNesT 79.1 87.2M ★★★★★
14 SAM-Adapter 73.4 11.6M ★★★★☆
15 UNETR 64.4 101.8M ★★★★★

Citation

Please cite the following papers if you find our study helpful.

@article{bassi2024touchstone,
  title={Touchstone Benchmark: Are We on the Right Way for Evaluating AI Algorithms for Medical Segmentation?},
  author={Bassi, Pedro RAS and Li, Wenxuan and Tang, Yucheng and Isensee, Fabian and Wang, Zifu and Chen, Jieneng and Chou, Yu-Cheng and Kirchhoff, Yannick and Rokuss, Maximilian and Huang, Ziyan and Ye, Jin and He, Junjun and Wald, Tassilo and Ulrich, Constantin and Baumgartner, Michael and Roy, Saikat and Maier-Hein, Klaus H. and Jaeger, Paul and Ye, Yiwen and Xie, Yutong and Zhang, Jianpeng and Chen, Ziyang and Xia, Yong and Xing, Zhaohu and Zhu, Lei and Sadegheih, Yousef and Bozorgpour, Afshin and Kumari, Pratibha and Azad, Reza and Merhof, Dorit and Shi, Pengcheng and Ma, Ting and Du, Yuxin and Bai, Fan and Huang, Tiejun and Zhao, Bo and Wang, Haonan and Li, Xiaomeng and Gu, Hanxue and Dong, Haoyu and Yang, Jichen and Mazurowski, Maciej A. and Gupta, Saumya and Wu, Linshan and Zhuang, Jiaxin and Chen, Hao and Roth, Holger and Xu, Daguang and Blaschko, Matthew B. and Decherchi, Sergio and Cavalli, Andrea and Yuille, Alan L. and Zhou, Zongwei},
  journal={Conference on Neural Information Processing Systems},
  year={2024},
  utl={https://github.com/MrGiovanni/Touchstone}
}

@article{li2024abdomenatlas,
  title={AbdomenAtlas: A large-scale, detailed-annotated, \& multi-center dataset for efficient transfer learning and open algorithmic benchmarking},
  author={Li, Wenxuan and Qu, Chongyu and Chen, Xiaoxi and Bassi, Pedro RAS and Shi, Yijia and Lai, Yuxiang and Yu, Qian and Xue, Huimin and Chen, Yixiong and Lin, Xiaorui and others},
  journal={Medical Image Analysis},
  pages={103285},
  year={2024},
  publisher={Elsevier}
}

@inproceedings{li2024well,
  title={How Well Do Supervised Models Transfer to 3D Image Segmentation?},
  author={Li, Wenxuan and Yuille, Alan and Zhou, Zongwei},
  booktitle={The Twelfth International Conference on Learning Representations},
  year={2024}
}

@article{qu2023abdomenatlas,
  title={Abdomenatlas-8k: Annotating 8,000 CT volumes for multi-organ segmentation in three weeks},
  author={Qu, Chongyu and Zhang, Tiezheng and Qiao, Hualin and Tang, Yucheng and Yuille, Alan L and Zhou, Zongwei and others},
  journal={Advances in Neural Information Processing Systems},
  volume={36},
  year={2023}
}

Acknowledgement

This work was supported by the Lustgarten Foundation for Pancreatic Cancer Research and the McGovern Foundation. Paper content is covered by patents pending.

Touchstone Benchmark

For Tasks:

Click tags to check more tools for each tasks

For Jobs:

Alternative AI tools for Touchstone

Similar Open Source Tools

Touchstone Screenshot
Touchstone

github
: 98
Awesome-LLM-Resources-List Screenshot
Awesome-LLM-Resources-List

Awesome LLM Resources is a curated collection of resources for Large Language Models (LLMs) covering various aspects such as serverless hosting, accessing off-the-shelf models via API, local inference, LLM serving frameworks, open-source LLM web chat UIs, renting GPUs for fine-tuning, fine-tuning with no-code UI, fine-tuning frameworks, OS agentic/AI workflow, AI agents, co-pilots, voice API, open-source TTS models, OS RAG frameworks, research papers on chain-of-thought prompting, CoT implementations, CoT fine-tuned models & datasets, and more.

github
: 95
EmoLLM Screenshot
EmoLLM

EmoLLM is a series of large-scale psychological health counseling models that can support **understanding-supporting-helping users** in the psychological health counseling chain, which is fine-tuned from `LLM` instructions. Welcome everyone to star~⭐⭐. The currently open source `LLM` fine-tuning configurations are as follows:

github
: 1.2k
cgft-llm Screenshot
cgft-llm

The cgft-llm repository is a collection of video tutorials and documentation for implementing large models. It provides guidance on topics such as fine-tuning llama3 with llama-factory, lightweight deployment and quantization using llama.cpp, speech generation with ChatTTS, introduction to Ollama for large model deployment, deployment tools for vllm and paged attention, and implementing RAG with llama-index. Users can find detailed code documentation and video tutorials for each project in the repository.

github
: 924
DI-engine Screenshot
DI-engine

github
: 2.9k
widgets Screenshot
widgets

Widgets is a desktop component front-end open source component. The project is still being continuously improved. The desktop component client can be downloaded and run in two ways: 1. https://www.microsoft.com/store/productId/9NPR50GQ7T53 2. https://widgetjs.cn After cloning the code, you need to download the dependency in the project directory: `shell pnpm install` and run: `shell pnpm serve`

github
: 228
JiwuChat Screenshot
JiwuChat

JiwuChat is a lightweight multi-platform chat application built on Tauri2 and Nuxt3, with various real-time messaging features, AI group chat bots (such as 'iFlytek Spark', 'KimiAI' etc.), WebRTC audio-video calling, screen sharing, and AI shopping functions. It supports seamless cross-device communication, covering text, images, files, and voice messages, also supporting group chats and customizable settings. It provides light/dark mode for efficient social networking.

github
: 227
llm-book Screenshot
llm-book

The 'llm-book' repository is dedicated to the introduction of large-scale language models, focusing on natural language processing tasks. The code is designed to run on Google Colaboratory and utilizes datasets and models available on the Hugging Face Hub. Note that as of July 28, 2023, there are issues with the MARC-ja dataset links, but an alternative notebook using the WRIME Japanese sentiment analysis dataset has been added. The repository covers various chapters on topics such as Transformers, fine-tuning language models, entity recognition, summarization, document embedding, question answering, and more.

github
: 291
MedicalGPT Screenshot
MedicalGPT

MedicalGPT is a training medical GPT model with ChatGPT training pipeline, implement of Pretraining, Supervised Finetuning, RLHF(Reward Modeling and Reinforcement Learning) and DPO(Direct Preference Optimization).

github
: 3.6k
ruoyi-vue-pro Screenshot
ruoyi-vue-pro

The ruoyi-vue-pro repository is an open-source project that provides a comprehensive development platform with various functionalities such as system features, infrastructure, member center, data reports, workflow, payment system, mall system, ERP system, CRM system, and AI big model. It is built using Java backend with Spring Boot framework and Vue frontend with different versions like Vue3 with element-plus, Vue3 with vben(ant-design-vue), and Vue2 with element-ui. The project aims to offer a fast development platform for developers and enterprises, supporting features like dynamic menu loading, button-level access control, SaaS multi-tenancy, code generator, real-time communication, integration with third-party services like WeChat, Alipay, and cloud services, and more.

github
: 28.9k
Awesome-LLM-Tabular Screenshot
Awesome-LLM-Tabular

This repository is a curated list of research papers that explore the integration of Large Language Model (LLM) technology with tabular data. It aims to provide a comprehensive resource for researchers and practitioners interested in this emerging field. The repository includes papers on a wide range of topics, including table-to-text generation, table question answering, and tabular data classification. It also includes a section on related datasets and resources.

github
: 335
yudao-boot-mini Screenshot
yudao-boot-mini

yudao-boot-mini is an open-source project focused on developing a rapid development platform for developers in China. It includes features like system functions, infrastructure, member center, data reports, workflow, mall system, WeChat official account, CRM, ERP, etc. The project is based on Spring Boot with Java backend and Vue for frontend. It offers various functionalities such as user management, role management, menu management, department management, workflow management, payment system, code generation, API documentation, database documentation, file service, WebSocket integration, message queue, Java monitoring, and more. The project is licensed under the MIT License, allowing both individuals and enterprises to use it freely without restrictions.

github
: 54
yudao-cloud Screenshot
yudao-cloud

Yudao-cloud is an open-source project designed to provide a fast development platform for developers in China. It includes various system functions, infrastructure, member center, data reports, workflow, mall system, WeChat public account, CRM, ERP, etc. The project is based on Java backend with Spring Boot and Spring Cloud Alibaba microservices architecture. It supports multiple databases, message queues, authentication systems, dynamic menu loading, SaaS multi-tenant system, code generator, real-time communication, integration with third-party services like WeChat, Alipay, and more. The project is well-documented and follows the Alibaba Java development guidelines, ensuring clean code and architecture.

github
: 16.5k
PaddleNLP Screenshot
PaddleNLP

PaddleNLP is an easy-to-use and high-performance NLP library. It aggregates high-quality pre-trained models in the industry and provides out-of-the-box development experience, covering a model library for multiple NLP scenarios with industry practice examples to meet developers' flexible customization needs.

github
: 12.4k
MEGREZ Screenshot
MEGREZ

MEGREZ is a modern and elegant open-source high-performance computing platform that efficiently manages GPU resources. It allows for easy container instance creation, supports multiple nodes/multiple GPUs, modern UI environment isolation, customizable performance configurations, and user data isolation. The platform also comes with pre-installed deep learning environments, supports multiple users, features a VSCode web version, resource performance monitoring dashboard, and Jupyter Notebook support.

github
: 77
awesome-ai-painting Screenshot
awesome-ai-painting

This repository, named 'awesome-ai-painting', is a comprehensive collection of resources related to AI painting. It is curated by a user named 秋风, who is an AI painting enthusiast with a background in the AIGC industry. The repository aims to help more people learn AI painting and also documents the user's goal of creating 100 AI products, with current progress at 4/100. The repository includes information on various AI painting products, tutorials, tools, and models, providing a valuable resource for individuals interested in AI painting and related technologies.

github
: 11.0k

For similar tasks

No tools available

For similar jobs

No tools available