Touchstone

Touchstone

[NeurIPS 2024] Touchstone - Benchmarking AI on 5,172 o.o.d. CT volumes and 9 anatomical structures

Stars: 98

Visit
screenshot

README:

Touchstone Benchmark

Touchstone Benchmark

Participate - Touchstone 1.0 Participate - Touchstone 2.0

touchstone leaderboard touchstone dataset touchstone model
visitors GitHub Repo stars Follow on Twitter
Subscribe us: https://groups.google.com/u/2/g/bodymaps

We present Touchstone, a large-scale medical segmentation benchmark based on annotated 5,195 CT volumes from 76 hospitals for training, and 6,933 CT volumes from 8 additional hospitals for testing. We invite AI inventors to train their models on AbdomenAtlas, and we independently evaluate their algorithms. We have already collaborated with 14 influential research teams, and we remain accepting new submissions.

Paper

Touchstone Benchmark: Are We on the Right Way for Evaluating AI Algorithms for Medical Segmentation?
Pedro R. A. S. Bassi1, Wenxuan Li1, Yucheng Tang2, Fabian Isensee3, ..., Alan Yuille1, Zongwei Zhou1
1Johns Hopkins University, 2NVIDIA, 3DKFZ
NeurIPS 2024
YouTube

Touchstone 1.0 Leaderboard

rank model organization average DSC paper github
🏆 MedNeXt DKFZ 89.2 arXiv GitHub stars
🏆 STU-Net-B Shanghai AI Lab 89.0 arXiv GitHub stars
🏆 MedFormer Rutgers 89.0 arXiv GitHub stars
🏆 nnU-Net ResEncL DKFZ 88.8 arXiv GitHub stars
🏆 UniSeg NPU 88.8 arXiv GitHub stars
🏆 Diff-UNet HKUST 88.5 arXiv GitHub stars
🏆 LHU-Net UR 88.0 arXiv GitHub stars
🏆 NexToU HIT 87.8 arXiv GitHub stars
9 SegVol BAAI 87.1 arXiv GitHub stars
10 U-Net & CLIP CityU 87.1 arXiv GitHub stars
11 Swin UNETR & CLIP CityU 86.7 arXiv GitHub stars
12 Swin UNETR NVIDIA 80.1 arXiv GitHub stars
13 UNesT NVIDIA 79.1 arXiv GitHub stars
14 SAM-Adapter Duke 73.4 arXiv GitHub stars
15 UNETR NVIDIA 64.4 arXiv GitHub stars
Aorta - NexToU 🏆
rank model organization DSC paper github
🏆 NexToU HIT 86.4 arXiv GitHub stars
2 MedNeXt DKFZ 83.1 arXiv GitHub stars
3 UniSeg NPU 82.3 arXiv GitHub stars
4 STU-Net-B Shanghai AI Lab 82.1 arXiv GitHub stars
5 nnU-Net ResEncL DKFZ 81.4 arXiv GitHub stars
6 Diff-UNet HKUST 81.2 arXiv GitHub stars
7 Swin UNETR NVIDIA 81.1 arXiv GitHub stars
8 SegVol BAAI 80.2 arXiv GitHub stars
9 UNesT NVIDIA 78.6 arXiv GitHub stars
10 Swin UNETR & CLIP CityU 78.1 arXiv GitHub stars
11 U-Net & CLIP CityU 77.1 arXiv GitHub stars
12 SAM-Adapter Duke 62.8 arXiv GitHub stars
13 UNETR NVIDIA 52.1 arXiv GitHub stars
Gallbladder - STU-Net-B & MedNeXt 🏆
rank model organization DSC paper github
🏆 STU-Net-B Shanghai AI Lab 85.5 arXiv GitHub stars
🏆 MedNeXt DKFZ 85.3 arXiv GitHub stars
3 nnU-Net ResEncL DKFZ 84.9 arXiv GitHub stars
4 UniSeg NPU 84.7 arXiv GitHub stars
5 Diff-UNet HKUST 83.8 arXiv GitHub stars
6 NexToU HIT 82.3 arXiv GitHub stars
7 U-Net & CLIP CityU 82.1 arXiv GitHub stars
8 Swin UNETR & CLIP CityU 80.2 arXiv GitHub stars
9 SegVol BAAI 79.3 arXiv GitHub stars
10 Swin UNETR NVIDIA 69.2 arXiv GitHub stars
11 UNesT NVIDIA 62.1 arXiv GitHub stars
12 SAM-Adapter Duke 49.4 arXiv GitHub stars
13 UNETR NVIDIA 43.8 arXiv GitHub stars
KidneyL - Diff-UNet 🏆
rank model organization DSC paper github
🏆 Diff-UNet HKUST 91.9 arXiv GitHub stars
2 nnU-Net ResEncL DKFZ 91.9 arXiv GitHub stars
3 STU-Net-B Shanghai AI Lab 91.9 arXiv GitHub stars
4 MedNeXt DKFZ 91.8 arXiv GitHub stars
5 SegVol BAAI 91.8 arXiv GitHub stars
6 UniSeg NPU 91.5 arXiv GitHub stars
7 U-Net & CLIP CityU 91.1 arXiv GitHub stars
8 Swin UNETR & CLIP CityU 91.0 arXiv GitHub stars
9 NexToU HIT 89.6 arXiv GitHub stars
10 SAM-Adapter Duke 87.3 arXiv GitHub stars
11 Swin UNETR NVIDIA 85.5 arXiv GitHub stars
12 UNesT NVIDIA 85.4 arXiv GitHub stars
13 UNETR NVIDIA 63.7 arXiv GitHub stars
KidneyR - Diff-UNet 🏆
rank model organization DSC paper github
🏆 Diff-UNet HKUST 92.8 arXiv GitHub stars
2 MedNeXt DKFZ 92.6 arXiv GitHub stars
3 nnU-Net ResEncL DKFZ 92.6 arXiv GitHub stars
4 STU-Net-B Shanghai AI Lab 92.5 arXiv GitHub stars
5 SegVol BAAI 92.5 arXiv GitHub stars
6 UniSeg NPU 92.2 arXiv GitHub stars
7 U-Net & CLIP CityU 91.9 arXiv GitHub stars
8 Swin UNETR & CLIP CityU 91.7 arXiv GitHub stars
9 SAM-Adapter Duke 90.4 arXiv GitHub stars
10 NexToU HIT 90.1 arXiv GitHub stars
11 UNesT NVIDIA 83.6 arXiv GitHub stars
12 Swin UNETR NVIDIA 81.7 arXiv GitHub stars
13 UNETR NVIDIA 69.6 arXiv GitHub stars
Liver - MedNeXt 🏆
rank model organization DSC paper github
🏆 MedNeXt DKFZ 96.3 arXiv GitHub stars
2 nnU-Net ResEncL DKFZ 96.3 arXiv GitHub stars
3 Diff-UNet HKUST 96.2 arXiv GitHub stars
4 STU-Net-B Shanghai AI Lab 96.2 arXiv GitHub stars
5 UniSeg NPU 96.1 arXiv GitHub stars
6 U-Net & CLIP CityU 96.0 arXiv GitHub stars
7 SegVol BAAI 96.0 arXiv GitHub stars
8 Swin UNETR & CLIP CityU 95.8 arXiv GitHub stars
9 NexToU HIT 95.7 arXiv GitHub stars
10 SAM-Adapter Duke 94.1 arXiv GitHub stars
11 UNesT NVIDIA 93.6 arXiv GitHub stars
12 Swin UNETR NVIDIA 93.5 arXiv GitHub stars
13 UNETR NVIDIA 90.5 arXiv GitHub stars
Pancreas - MedNeXt 🏆
rank model organization DSC paper github
🏆 MedNeXt DKFZ 83.3 arXiv GitHub stars
2 STU-Net-B Shanghai AI Lab 83.2 arXiv GitHub stars
3 nnU-Net ResEncL DKFZ 82.9 arXiv GitHub stars
4 UniSeg NPU 82.7 arXiv GitHub stars
5 Diff-UNet HKUST 81.9 arXiv GitHub stars
6 U-Net & CLIP CityU 80.8 arXiv GitHub stars
7 Swin UNETR & CLIP CityU 80.2 arXiv GitHub stars
8 NexToU HIT 80.2 arXiv GitHub stars
9 SegVol BAAI 79.1 arXiv GitHub stars
10 Swin UNETR NVIDIA 68.5 arXiv GitHub stars
11 UNesT NVIDIA 68.3 arXiv GitHub stars
12 UNETR NVIDIA 55.1 arXiv GitHub stars
13 SAM-Adapter Duke 50.2 arXiv GitHub stars
Postcava - STU-Net-B & MedNeXt 🏆
rank model organization DSC paper github
🏆 STU-Net-B Shanghai AI Lab 81.3 arXiv GitHub stars
🏆 MedNeXt DKFZ 81.3 arXiv GitHub stars
3 UniSeg NPU 81.2 arXiv GitHub stars
4 Diff-UNet HKUST 80.8 arXiv GitHub stars
5 nnU-Net ResEncL DKFZ 80.5 arXiv GitHub stars
6 U-Net & CLIP CityU 78.5 arXiv GitHub stars
7 NexToU HIT 78.1 arXiv GitHub stars
8 SegVol BAAI 77.8 arXiv GitHub stars
9 Swin UNETR & CLIP CityU 76.8 arXiv GitHub stars
10 Swin UNETR NVIDIA 69.9 arXiv GitHub stars
11 UNesT NVIDIA 66.2 arXiv GitHub stars
12 UNETR NVIDIA 53.9 arXiv GitHub stars
13 SAM-Adapter Duke 48.0 arXiv GitHub stars
Spleen - nnU-Net ResEncL 🏆
rank model organization DSC paper github
🏆 nnU-Net ResEncL DKFZ 95.2 arXiv GitHub stars
2 MedNeXt DKFZ 95.2 arXiv GitHub stars
3 STU-Net-B Shanghai AI Lab 95.1 arXiv GitHub stars
4 Diff-UNet HKUST 95.0 arXiv GitHub stars
5 UniSeg NPU 94.9 arXiv GitHub stars
6 SegVol BAAI 94.5 arXiv GitHub stars
7 NexToU HIT 94.7 arXiv GitHub stars
8 U-Net & CLIP CityU 94.3 arXiv GitHub stars
9 Swin UNETR & CLIP CityU 94.1 arXiv GitHub stars
10 SAM-Adapter Duke 90.5 arXiv GitHub stars
11 Swin UNETR NVIDIA 87.9 arXiv GitHub stars
12 UNesT NVIDIA 86.7 arXiv GitHub stars
13 UNETR NVIDIA 76.5 arXiv GitHub stars
Stomach - STU-Net-B & MedNeXt & nnU-Net ResEncL 🏆
rank model organization DSC paper github
🏆 STU-Net-B Shanghai AI Lab 93.5 arXiv GitHub stars
🏆 MedNeXt DKFZ 93.5 arXiv GitHub stars
🏆 nnU-Net ResEncL DKFZ 93.4 arXiv GitHub stars
4 UniSeg NPU 93.3 arXiv GitHub stars
5 Diff-UNet HKUST 93.1 arXiv GitHub stars
6 NexToU HIT 92.7 arXiv GitHub stars
7 SegVol BAAI 92.5 arXiv GitHub stars
8 U-Net & CLIP CityU 92.4 arXiv GitHub stars
9 Swin UNETR & CLIP CityU 92.2 arXiv GitHub stars
10 SAM-Adapter Duke 88.0 arXiv GitHub stars
11 UNesT NVIDIA 87.6 arXiv GitHub stars
12 Swin UNETR NVIDIA 84.1 arXiv GitHub stars
13 UNETR NVIDIA 74.2 arXiv GitHub stars

Touchstone 1.0 Dataset

Training set

Test set

metadata

Figure 1. Metadata distribution in the test set.

Touchstone 1.0 Model

[!NOTE] We are releasing the trained AI models evaluated in Touchstone right here. Stay tuned!

rank model average DSC parameter infer. speed download
🏆 MedNeXt 89.2 61.8M ★☆☆☆☆
🏆 STU-Net-B 89.0 58.3M ★★☆☆☆ checkpoint
🏆 MedFormer 89.0 38.5M ★★★☆☆
🏆 nnU-Net ResEncL 88.8 102.0M ★★★★☆ checkpoint
🏆 UniSeg 88.8 31.0M ☆☆☆☆☆
🏆 Diff-UNet 88.5 434.0M ★★★☆☆
🏆 LHU-Net 88.0 8.6M ★★★★★
🏆 NexToU 87.8 81.9M ★★★★☆ checkpoint
9 SegVol 87.1 181.0M ★★★★☆ checkpoint
10 U-Net & CLIP 87.1 19.1M ★★★☆☆
11 Swin UNETR & CLIP 86.7 62.2M ★★★☆☆
12 Swin UNETR 80.1 72.8M ★★★★★
13 UNesT 79.1 87.2M ★★★★★
14 SAM-Adapter 73.4 11.6M ★★★★☆
15 UNETR 64.4 101.8M ★★★★★

Citation

Please cite the following papers if you find our study helpful.

@article{bassi2024touchstone,
  title={Touchstone Benchmark: Are We on the Right Way for Evaluating AI Algorithms for Medical Segmentation?},
  author={Bassi, Pedro RAS and Li, Wenxuan and Tang, Yucheng and Isensee, Fabian and Wang, Zifu and Chen, Jieneng and Chou, Yu-Cheng and Kirchhoff, Yannick and Rokuss, Maximilian and Huang, Ziyan and Ye, Jin and He, Junjun and Wald, Tassilo and Ulrich, Constantin and Baumgartner, Michael and Roy, Saikat and Maier-Hein, Klaus H. and Jaeger, Paul and Ye, Yiwen and Xie, Yutong and Zhang, Jianpeng and Chen, Ziyang and Xia, Yong and Xing, Zhaohu and Zhu, Lei and Sadegheih, Yousef and Bozorgpour, Afshin and Kumari, Pratibha and Azad, Reza and Merhof, Dorit and Shi, Pengcheng and Ma, Ting and Du, Yuxin and Bai, Fan and Huang, Tiejun and Zhao, Bo and Wang, Haonan and Li, Xiaomeng and Gu, Hanxue and Dong, Haoyu and Yang, Jichen and Mazurowski, Maciej A. and Gupta, Saumya and Wu, Linshan and Zhuang, Jiaxin and Chen, Hao and Roth, Holger and Xu, Daguang and Blaschko, Matthew B. and Decherchi, Sergio and Cavalli, Andrea and Yuille, Alan L. and Zhou, Zongwei},
  journal={Conference on Neural Information Processing Systems},
  year={2024},
  utl={https://github.com/MrGiovanni/Touchstone}
}

@article{li2024abdomenatlas,
  title={AbdomenAtlas: A large-scale, detailed-annotated, \& multi-center dataset for efficient transfer learning and open algorithmic benchmarking},
  author={Li, Wenxuan and Qu, Chongyu and Chen, Xiaoxi and Bassi, Pedro RAS and Shi, Yijia and Lai, Yuxiang and Yu, Qian and Xue, Huimin and Chen, Yixiong and Lin, Xiaorui and others},
  journal={Medical Image Analysis},
  pages={103285},
  year={2024},
  publisher={Elsevier}
}

@inproceedings{li2024well,
  title={How Well Do Supervised Models Transfer to 3D Image Segmentation?},
  author={Li, Wenxuan and Yuille, Alan and Zhou, Zongwei},
  booktitle={The Twelfth International Conference on Learning Representations},
  year={2024}
}

@article{qu2023abdomenatlas,
  title={Abdomenatlas-8k: Annotating 8,000 CT volumes for multi-organ segmentation in three weeks},
  author={Qu, Chongyu and Zhang, Tiezheng and Qiao, Hualin and Tang, Yucheng and Yuille, Alan L and Zhou, Zongwei and others},
  journal={Advances in Neural Information Processing Systems},
  volume={36},
  year={2023}
}

Acknowledgement

This work was supported by the Lustgarten Foundation for Pancreatic Cancer Research and the McGovern Foundation. Paper content is covered by patents pending.

Touchstone Benchmark

For Tasks:

Click tags to check more tools for each tasks

For Jobs:

Alternative AI tools for Touchstone

Similar Open Source Tools

Touchstone Screenshot
Touchstone

github
: 98
cgft-llm Screenshot
cgft-llm

The cgft-llm repository is a collection of video tutorials and documentation for implementing large models. It provides guidance on topics such as fine-tuning llama3 with llama-factory, lightweight deployment and quantization using llama.cpp, speech generation with ChatTTS, introduction to Ollama for large model deployment, deployment tools for vllm and paged attention, and implementing RAG with llama-index. Users can find detailed code documentation and video tutorials for each project in the repository.

github
: 2.0k
EmoLLM Screenshot
EmoLLM

EmoLLM is a series of large-scale psychological health counseling models that can support **understanding-supporting-helping users** in the psychological health counseling chain, which is fine-tuned from `LLM` instructions. Welcome everyone to star~⭐⭐. The currently open source `LLM` fine-tuning configurations are as follows:

github
: 1.3k
JiwuChat Screenshot
JiwuChat

JiwuChat is a lightweight multi-platform chat application built on Tauri2 and Nuxt3, with various real-time messaging features, AI group chat bots (such as 'iFlytek Spark', 'KimiAI' etc.), WebRTC audio-video calling, screen sharing, and AI shopping functions. It supports seamless cross-device communication, covering text, images, files, and voice messages, also supporting group chats and customizable settings. It provides light/dark mode for efficient social networking.

github
: 627
Awesome-LLM-Resources-List Screenshot
Awesome-LLM-Resources-List

Awesome LLM Resources is a curated collection of resources for Large Language Models (LLMs) covering various aspects such as serverless hosting, accessing off-the-shelf models via API, local inference, LLM serving frameworks, open-source LLM web chat UIs, renting GPUs for fine-tuning, fine-tuning with no-code UI, fine-tuning frameworks, OS agentic/AI workflow, AI agents, co-pilots, voice API, open-source TTS models, OS RAG frameworks, research papers on chain-of-thought prompting, CoT implementations, CoT fine-tuned models & datasets, and more.

github
: 126
MedicalGPT Screenshot
MedicalGPT

MedicalGPT is a training medical GPT model with ChatGPT training pipeline, implement of Pretraining, Supervised Finetuning, RLHF(Reward Modeling and Reinforcement Learning) and DPO(Direct Preference Optimization).

github
: 3.6k
DeepAudit Screenshot
DeepAudit

DeepAudit is an AI audit team accessible to everyone, making vulnerability discovery within reach. It is a next-generation code security audit platform based on Multi-Agent collaborative architecture. It simulates the thinking mode of security experts, achieving deep code understanding, vulnerability discovery, and automated sandbox PoC verification through multiple intelligent agents (Orchestrator, Recon, Analysis, Verification). DeepAudit aims to address the three major pain points of traditional SAST tools: high false positive rate, blind spots in business logic, and lack of verification means. Users only need to import the project, and DeepAudit automatically starts working: identifying the technology stack, analyzing potential risks, generating scripts, sandbox verification, and generating reports, ultimately outputting a professional audit report. The core concept is to let AI attack like a hacker and defend like an expert.

github
: 4.6k
Firefly Screenshot
Firefly

Firefly is an open-source large model training project that supports pre-training, fine-tuning, and DPO of mainstream large models. It includes models like Llama3, Gemma, Qwen1.5, MiniCPM, Llama, InternLM, Baichuan, ChatGLM, Yi, Deepseek, Qwen, Orion, Ziya, Xverse, Mistral, Mixtral-8x7B, Zephyr, Vicuna, Bloom, etc. The project supports full-parameter training, LoRA, QLoRA efficient training, and various tasks such as pre-training, SFT, and DPO. Suitable for users with limited training resources, QLoRA is recommended for fine-tuning instructions. The project has achieved good results on the Open LLM Leaderboard with QLoRA training process validation. The latest version has significant updates and adaptations for different chat model templates.

github
: 4.8k
TigerBot Screenshot
TigerBot

TigerBot is a cutting-edge foundation for your very own LLM, providing a world-class large model for innovative Chinese-style contributions. It offers various upgrades and features, such as search mode enhancements, support for large context lengths, and the ability to play text-based games. TigerBot is suitable for prompt-based game engine development, interactive game design, and real-time feedback for playable games.

github
: 2.2k
ruoyi-vue-pro Screenshot
ruoyi-vue-pro

The ruoyi-vue-pro repository is an open-source project that provides a comprehensive development platform with various functionalities such as system features, infrastructure, member center, data reports, workflow, payment system, mall system, ERP system, CRM system, and AI big model. It is built using Java backend with Spring Boot framework and Vue frontend with different versions like Vue3 with element-plus, Vue3 with vben(ant-design-vue), and Vue2 with element-ui. The project aims to offer a fast development platform for developers and enterprises, supporting features like dynamic menu loading, button-level access control, SaaS multi-tenancy, code generator, real-time communication, integration with third-party services like WeChat, Alipay, and cloud services, and more.

github
: 28.9k
yudao-boot-mini Screenshot
yudao-boot-mini

yudao-boot-mini is an open-source project focused on developing a rapid development platform for developers in China. It includes features like system functions, infrastructure, member center, data reports, workflow, mall system, WeChat official account, CRM, ERP, etc. The project is based on Spring Boot with Java backend and Vue for frontend. It offers various functionalities such as user management, role management, menu management, department management, workflow management, payment system, code generation, API documentation, database documentation, file service, WebSocket integration, message queue, Java monitoring, and more. The project is licensed under the MIT License, allowing both individuals and enterprises to use it freely without restrictions.

github
: 54
yudao-cloud Screenshot
yudao-cloud

Yudao-cloud is an open-source project designed to provide a fast development platform for developers in China. It includes various system functions, infrastructure, member center, data reports, workflow, mall system, WeChat public account, CRM, ERP, etc. The project is based on Java backend with Spring Boot and Spring Cloud Alibaba microservices architecture. It supports multiple databases, message queues, authentication systems, dynamic menu loading, SaaS multi-tenant system, code generator, real-time communication, integration with third-party services like WeChat, Alipay, and more. The project is well-documented and follows the Alibaba Java development guidelines, ensuring clean code and architecture.

github
: 16.5k
DI-engine Screenshot
DI-engine

github
: 2.9k
AstrBot Screenshot
AstrBot

AstrBot is a powerful and versatile tool that leverages the capabilities of large language models (LLMs) like GPT-3, GPT-3.5, and GPT-4 to enhance communication and automate tasks. It seamlessly integrates with popular messaging platforms such as QQ, QQ Channel, and Telegram, enabling users to harness the power of AI within their daily conversations and workflows.

github
: 6.6k
erp-pro Screenshot
erp-pro

ERP-Pro is an intelligent manufacturing integrated platform developed using Springboot (microservices) + UNI-APP + Ant Design Vue's zero-code development mode. It includes over 50 electronic processes, CRM, PM, ERP, MES, ADM, OA, EHR, AI, projects, malls, finance, multi-shift attendance, payroll, recruitment, cloud after-sales, forums, surveys, report design, workflow, Saas, and more. It aims to create the first set of zero-code, most comprehensive, and most cost-effective intelligent manufacturing industry supply chain integrated management software on the web.

github
: 599
awesome-ai-painting Screenshot
awesome-ai-painting

This repository, named 'awesome-ai-painting', is a comprehensive collection of resources related to AI painting. It is curated by a user named 秋风, who is an AI painting enthusiast with a background in the AIGC industry. The repository aims to help more people learn AI painting and also documents the user's goal of creating 100 AI products, with current progress at 4/100. The repository includes information on various AI painting products, tutorials, tools, and models, providing a valuable resource for individuals interested in AI painting and related technologies.

github
: 11.0k

For similar tasks

No tools available

For similar jobs

No tools available