Awesome-AI-Security
Curated resources, research, and tools for securing AI systems
Stars: 59
Awesome-AI-Security is a curated list of resources for AI security, including tools, research papers, articles, and tutorials. It aims to provide a comprehensive overview of the latest developments in securing AI systems and preventing vulnerabilities. The repository covers topics such as adversarial attacks, privacy protection, model robustness, and secure deployment of AI applications. Whether you are a researcher, developer, or security professional, this collection of resources will help you stay informed and up-to-date in the rapidly evolving field of AI security.
README:
Curated resources, research, and tools for securing AI systems.
- Best Practices and Security Standards
- Tools
- Attack & Defense Matrices
- Checklists
- Datasets
- Courses & Certifications
- Training
- Reports and Research
- Communities & Social Groups
- Benchmarking
- Incident Response
- Supply Chain Security
- Videos & Playlists
- Newsletter
- Conferences
- Foundations: Glossary, SoK/Surveys & Taxonomies
- Podcasts
- Market Landscape
- Startups Blogs
- Related Awesome Lists
- Common Acronyms
- NIST — AI Risk Management Framework (AI RMF)
- ISO/IEC 42001 (AI Management System)
-
OWASP — AI Maturity Assessment (AIMA)
- Google — Secure AI Framework (SAIF)
- OWASP — LLM & GenAI Security Center of Excellence (CoE) Guide
- CSA — AI Model Risk Management Framework
- NIST — Artificial Intelligence Risk Management Framework: Generative Artificial Intelligence Profile
-
OWASP — LLM Security Verification Standard (LLMSVS)
-
OWASP — Artificial Intelligence Security Verification Standard (AISVS)
- CSA — AI Controls Matrix (AICM) — The AICM contains 243 control objectives across 18 domains and maps to ISO 42001, ISO 27001, NIST AI RMF 1.0, and BSI AIC4. Freely downloadable.
-
OWASP — Top 10 for Large Language Model Applications
- CSA - MCP Client Top 10
- CSA - MCP Server Top 10
-
OWASP — AI Testing Guide
- OWASP — Red Teaming Guide
- OWASP — LLM Exploit Generation
- CSA — Agentic AI Red Teaming Guide
-
OWASP — AI Security and Privacy Guide
- OWASP — LLM and Gen AI Data Security Best Practices
-
OWASP — GenAI Security Project
- CSA — Secure LLM Systems: Essential Authorization Practices
- NIST — Four Principles of Explainable Artificial Intelligence
-
OASIS CoSAI — Preparing Defenders of AI Systems
- CISA — AI Data Security: Best Practices for Securing Data Used to Train & Operate AI Systems
- DoD CIO — AI Cybersecurity Risk Management Tailoring Guide (2025) — Practical RMF tailoring for AI systems across the lifecycle; complements CDAO’s RAI toolkit.
-
OWASP — Agent Observability Standard (AOS)
- OWASP — Agent Name Service (ANS) for Secure AI Agent Discovery
- OWASP — Agentic AI - Threats and Mitigations
- OWASP — Securing Agentic Applications Guide
- OWASP — Multi-Agentic System Threat Modeling Guide
- OWASP — State of Agentic AI Security and Governance
- CSA — Secure Agentic System Design: A Trait-Based Approach
- CSA — Agentic AI Identity & Access Management — 08/25
↑Tools
Inclusion criteria (open-source tools): must have 220+ GitHub stars, active maintenance in the last 12 months, and ≥3 contributors.
Detect and stop prompt-injection (direct/indirect) across inputs, context, and outputs; filter hostile content before it reaches tools or models.
- (none from your current list yet)
Enforce safety policies and block jailbreaks at runtime via rules/validators/DSLs, with optional human-in-the-loop for sensitive actions.
-
NeMo Guardrails
-
LLM Guard
-
Llama Guard
-
LlamaFirewall
-
Code Shield
-
Guardrails
— Runtime policy enforcement for LLM apps: compose input/output validators (PII, toxicity, jailbreak/PI, regex, competitor checks), then block/redact/rewrite/retry on fail; optional server mode; also supports structured outputs (Pydantic/function-calling).
Analyze serialized model files for unsafe deserialization and embedded code; verify integrity/metadata and block or quarantine on fail.
Scan/audit MCP servers & client configs; detect tool poisoning, unsafe flows; constrain tool access with least-privilege and audit trails.
-
Beelzebub
— Beelzebub is a honeypot framework designed to provide a secure environment for detecting and analyzing cyber attacks. It offers a low code approach for easy implementation and uses AI to mimic the behavior of a high-interaction honeypot.
Run untrusted or LLM-triggered code in isolated sandboxes (FS/network/process limits) to contain RCE and reduce blast radius.
-
E2B
— SDK + self-hostable infra to run untrusted, LLM-generated code in isolated cloud sandboxes (Firecracker microVMs).
Centralize auth, quotas/rate limits, cost caps, egress/DLP filters, and guardrail orchestration across all model/providers.
- (none from your current list yet)
-
Claude Code Security Reviewer
- An AI-powered security review GitHub Action using Claude to analyze code changes for security vulnerabilities.
-
Vulnhuntr
- Vulnhuntr leverages the power of LLMs to automatically create and analyze entire code call chains starting from remote user input and ending at server output for detection of complex, multi-step, security-bypassing vulnerabilities that go far beyond what traditional static code analysis tools are capable of performing.
Automate attack suites (prompt-injection, leakage, jailbreak, goal-based tasks) in CI; score results and produce regression evidence.
-
promptfoo
-
Agentic Radar
-
DeepTeam
-
Buttercup
— Trail of Bits’ AIxCC Cyber Reasoning System: runs OSS-Fuzz–style campaigns to find vulns, then uses a multi-agent LLM patcher to generate & validate fixes for C/Java repos; ships SigNoz observability; requires at least one LLM API key.
- (none from your current list yet)
Generate and verify AI/ML BOMs, signatures, and provenance for models/datasets/dependencies; enforce allow/deny policies.
- (none from your current list yet)
Harden RAG memory: isolate namespaces, sanitize queries/content, detect poisoning/outliers, and prevent secret/PII retention.
- (none from your current list yet)
Detect and mitigate dataset/model poisoning and backdoors; validate training/fine-tuning integrity and prune suspicious behaviors.
Prevent secret/PII exfiltration in prompts/outputs via detection, redaction, and policy checks at I/O boundaries.
-
Presidio
— PII/PHI detection & redaction for text, images, and structured data; use as a pre/post-LLM DLP filter and for dataset sanitization.
Collect AI-specific security logs/signals; detect abuse patterns (PI/jailbreak/leakage), enrich alerts, and support forensics.
-
LangKit
— LLM observability metrics toolkit (whylogs-compatible): prompt-injection/jailbreak similarity, PII patterns, hallucination/consistency, relevance, sentiment/toxicity, readability.
-
Alibi Detect
— Production drift/outlier/adversarial detection for tabular, text, images, and time series; online/offline detectors with TF/PyTorch backends; returns scores, thresholds, and flags for alerting.
↑Attack & Defense Matrices
Matrix-style resources covering adversarial TTPs and curated defensive techniques for AI systems.
- MITRE ATLAS – Adversarial TTP matrix and knowledge base for threats to AI systems.
- GenAI Attacks Matrix – Matrix of TTPs targeting GenAI apps, copilots, and agents.
- MCP Security Tactics, Techniques, and Procedures (TTPs)
-
AIDEFEND — AI Defense Framework
— Interactive defensive countermeasures knowledge base with Tactics / Pillars / Phases views; maps mitigations to MITRE ATLAS, MAESTRO, and OWASP LLM risks. • Live demo: https://edward-playground.github.io/aidefense-framework/
↑Checklists
↑Supply Chain Security
Guidance and standards for securing the AI/ML software supply chain (models, datasets, code, pipelines). Primarily specs and frameworks; includes vetted TPRM templates.
Normative formats and specifications for transparency and traceability across AI components and dependencies.
-
OWASP — AI Bill of Materials (AIBOM)
— Bill of materials format for AI components, datasets, and model dependencies.
Questionnaires and templates to assess external vendors, model providers, and integrators for security, privacy, and compliance.
- FS-ISAC — Generative AI Vendor Evaluation & Qualitative Risk Assessment — Assessment Tool XLSX • Guide PDF — Vendor due-diligence toolkit for GenAI: risk tiering by use case, integration and data sensitivity; questionnaires across privacy, security, model development and validation, integration, legal and compliance; auto-generated reporting.
↑Videos & Playlists
Monthly curated playlists of AI-security talks, demos, incidents, and tooling.
- AI Security Playlist – September 2025
- AI Security Playlist – August 2025
- AI Security Playlist – July 2025
- AI Security Playlist – June 2025
↑Datasets
- Kaggle — Community-contributed datasets (IDS, phishing, malware URLs, incidents).
- Hugging Face — Search HF datasets tagged/related to cybersecurity and threat intel.
Interactive CTFs and self-contained labs for hands-on security skills (web, pwn, crypto, forensics, reversing). Used to assess practical reasoning, tool use, and end-to-end task execution.
Structured Q&A datasets assessing security knowledge and terminology. Used to evaluate factual recall and conceptual understanding.
Code snippet datasets labeled as vulnerable or secure, often tied to CWEs (Common Weakness Enumeration). Used to evaluate the model’s ability to recognize insecure code patterns and suggest secure fixes.
Adversarial prompt datasets—both text-only and multimodal—designed to bypass safety mechanisms or test refusal logic. Used to test how effectively a model resists jailbreaks and enforces policy-based refusal.
Datasets labeled with whether prompts are benign or malicious (i.e., injection attempts). Used to evaluate an LLM’s ability to detect and neutralize prompt-injection style attacks.
↑Courses & Certifications
- SANS — AI Cybersecurity Careers — Career pathways poster + training map; baseline skills for AI security (IR, DFIR, detection, threat hunting).
- SANS — SEC545: GenAI & LLM Application Security — Hands-on course covering prompt injection, excessive agency, model supply chain, and defensive patterns. (Certificate of completion provided by SANS.)
- SANS — SEC495: Leveraging LLMs: Building & Securing RAG, Contextual RAG, and Agentic RAG — Practical RAG builds with threat modeling, validation, and guardrails. (Certificate of completion provided by SANS.)
- Modern Security - Certified AI Security Expert (MSec-CAIS) — Practical AI Security - Go Beyond Theory: Build, Break, and Defend AI apps/agent with hands-on labs. (Certificate of completion provided by Modern Security.)
- IAPP — Artificial Intelligence Governance Professional (AIGP) — Governance-focused credential aligned with emerging regulations.
- ISACA — Advanced in AI Security Management (AAISM™) — AI-centric security management certification.
- NIST AI RMF 1.0 Architect — Certified Information Security — Credential aligned to NIST AI RMF 1.0.
- ISO/IEC 23894 — AI Risk Management (AI Risk Manager, PECB) — Risk identification, assessment, and mitigation aligned to ISO/IEC 23894 and NIST AI RMF.
- ISO/IEC 42001 — AI Management System (Lead Implementer, PECB) — Implement an AIMS per ISO/IEC 42001.
- ISO/IEC 42001 — AI Management System (Lead Auditor, PECB) — Audit AIMS using recognized principles.
- ISACA — Advanced in AI Audit (AAIA™) — Certification for auditing AI systems and mitigating AI-related risks.
↑Training
- Microsoft AI Security Learning Path — Free, self-paced Microsoft content on secure AI model development, risk management, and threat mitigation.
- AWS AI Security Training — Free AWS portal with courses on securing AI applications, risk management, and AI/ML security best practices.
- PortSwigger — Web Security Academy: Web LLM attacks — Structured, guided track on LLM issues (prompt injection, insecure output handling, excessive agency) with walkthrough-style exercises.
-
AI GOAT
— Vulnerable LLM CTF challenges for learning AI security.
-
Damn Vulnerable LLM Agent
-
AI Red Teaming Playground Labs — Microsoft
— Self-hostable environment with 12 challenges (direct/indirect prompt injection, metaprompt extraction, Crescendo multi-turn, guardrail bypass).
- Trail of Bits — AI/ML Security & Safety Training — Courses on AI failure modes, adversarial attacks, data provenance, pipeline threats, and mitigation.
↑Research Working Groups
- Cloud Security Alliance (CSA) AI Security Working Groups – Collaborative research groups focused on AI security, cloud security, and emerging threats in AI-driven systems.
- OWASP Top 10 for LLM & Generative AI Security Risks Project – An open-source initiative addressing critical security risks in Large Language Models (LLMs) and Generative AI applications, offering resources and guidelines to mitigate emerging threats.
- CWE Artificial Intelligence Working Group (AI WG) – The AI WG was established by CWE™ and CVE® community stakeholders to identify and address gaps in the CWE corpus where AI-related weaknesses are not adequately covered, and work collaboratively to fix them.
- NIST — SP 800-53 Control Overlays for Securing AI Systems (COSAiS) – Public collaboration to develop AI security control overlays with NIST principal investigators and the community.
- OpenSSF — AI/ML Security Working Group — Cross-org WG on “security for AI” and “AI for security”
-
CoSAI — Coalition for Secure AI (OASIS Open Project) — Open, cross-industry initiative advancing secure-by-design AI through shared frameworks, tooling, and guidance.
- WS1: Software Supply Chain Security for AI Systems - Extends SSDF/SLSA principles to AI; provenance, model risks, and pipeline security.https://github.com/cosai-oasis/ws1-supply-chain
-
WS2: Preparing Defenders for a Changing Cybersecurity Landscape - Defender-focused framework aligning threats, mitigations, and investments for AI-driven ops. https://github.com/cosai-oasis/ws2-defenders
• Reference doc: “Preparing Defenders of AI Systems” https://github.com/cosai-oasis/ws2-defenders/blob/main/preparing-defenders-of-ai-systems.md - WS3: AI Security Risk Governance - Security-focused risk & controls taxonomy, checklist, and scorecard for AI products and components.https://github.com/cosai-oasis/ws3-ai-risk-governance
- WS4: Secure Design Patterns for Agentic Systems - Threat models and secure design patterns for agentic systems and infrastructure. https://github.com/cosai-oasis/ws4-secure-design-agentic-systems
📌 (More working groups to be added.)
↑Communities & Social Groups
↑Benchmarking
Purpose: Evaluates how AI systems withstand adversarial attacks, including evasion, poisoning, and model extraction. Ensures AI remains functional under manipulation.
NIST AI RMF Alignment: Measure, Manage
- Measure: Identify risks related to adversarial attacks.
- Manage: Implement mitigation strategies to ensure resilience.
Purpose: Assesses AI models for unauthorized modifications, including backdoors and dataset poisoning. Supports trustworthiness and security of model outputs.
NIST AI RMF Alignment: Map, Measure
-
Map: Understand and identify risks to model/data integrity.
-
Measure: Evaluate and mitigate risks through validation techniques.
-
CVE-Bench — @uiuc-kang-lab
— How well AI agents can exploit real-world software vulnerabilities that are listed in the CVE database.
Purpose: Ensures AI security aligns with governance frameworks, industry regulations, and security policies. Supports auditability and risk management.
NIST AI RMF Alignment: Govern
- Govern: Establish policies, accountability structures, and compliance controls.
Purpose: Evaluates AI for risks like data leakage, membership inference, and model inversion. Helps ensure privacy preservation and compliance.
NIST AI RMF Alignment: Measure, Manage
- Measure: Identify and assess AI-related privacy risks.
- Manage: Implement security controls to mitigate privacy threats.
Purpose: Assesses AI for transparency, fairness, and bias mitigation. Ensures AI operates in an interpretable and ethical manner.
NIST AI RMF Alignment: Govern, Map, Measure
- Govern: Establish policies for fairness, bias mitigation, and transparency.
- Map: Identify potential explainability risks in AI decision-making.
- Measure: Evaluate AI outputs for fairness, bias, and interpretability.
↑Incident Response
- AI Incident Database (AIID)
- MIT AI Risk Repository — Incident Tracker
- AIAAIC Repository
- OECD.AI — AIM: AI Incidents and Hazards Monitor
- AVID — AI Vulnerability Database — Open, taxonomy-driven catalog of AI failure modes; Vulnerabilities*, Reports map incidents to failure modes/lifecycle stages.
- EU AI Act — Article 73: Reporting of Serious Incidents — Providers of high-risk AI systems need to report serious incidents to national authorities.
↑Newsletter
- Adversarial AI Digest - A digest of AI security research, threats, governance challenges, and best practices for securing AI systems.
↑Reports and Research
- AI Security Research Feed – Continuously updated feed of AI security–related academic papers, preprints, and research indexed from arXiv.
- AI Security Portal – Literature Database – Categorized database of AI security literature, taxonomy, and related resources.
- CSA — Principles to Practice: Responsible AI in a Dynamic Regulatory Environment
- CSA — AI Resilience: A Revolutionary Benchmarking Model for AI Safety – Governance & compliance benchmarking model.
- CSA — Using AI for Offensive Security
📌 (More to be added – A collection of AI security reports, white papers, and academic studies.)
↑Foundations: Glossary, SoK/Surveys & Taxonomies
(Core references and syntheses for orientation and shared language.)
(Authoritative definitions for AI/ML security, governance, and risk—use to align terminology across docs and reviews.)
- NIST — “The Language of Trustworthy AI: An In-Depth Glossary of Terms.” - Authoritative cross-org terminology aligned to NIST AI RMF; useful for standardizing terms across teams.
- ISO/IEC 22989:2022 — Artificial intelligence — Concepts and terminology - International standard that formalizes core AI concepts and vocabulary used in policy and engineering.
(Systematizations of Knowledge (SoK), surveys, systematic reviews, and mapping studies.)
(Reusable classification schemes—clear dimensions, categories, and labeling rules for attacks, defenses, datasets, and risks.)
- CSA — Large Language Model (LLM) Threats Taxonomy - Community taxonomy of LLM-specific threats; clarifies categories/definitions for risk discussion and control mapping.
- ARC — PI (Prompt Injection) Taxonomy - Focused taxonomy for prompt-injection behaviors/variants with practical labeling guidance for detection and defense.
↑Podcasts
- The MLSecOps Podcast – Insightful conversations with industry leaders and AI experts, exploring the fascinating world of machine learning security operations.
↑Market Landscape
Curated market maps of tools and vendors for securing LLM and agentic AI applications across the lifecycle.
- OWASP — LLM and Generative AI Security Solutions Landscape
- OWASP — AI Security Solutions Landscape for Agentic AI
- Latio — 2025 AI Security Report — Market trends and vendor landscape snapshot for AI security.
- Woodside Capital Partners — Cybersecurity Sector — A snapshot with vendor breakdowns and landscape view.
- Insight Partners — Cybersecurity Portfolio Overview (Market Map) — Visual market map and portfolio overview across cybersecurity domains.
↑Startups Blogs
A curated list of startups securing agentic AI applications, organized by the OWASP Agentic AI lifecycle (Scope & Plan → Govern). Each company appears once in its best-fit stage based on public positioning, and links point to blog/insights for deeper context. Some startups span multiple stages; placements reflect primary focus.
Inclusion criteria
- Startup has not been acquired
- Has an active blog
- Has an active GitHub organization/repository
Design-time security: non-human identities, agent threat modeling, privilege boundaries/authn, and memory scoping/isolation.
no startups here with active blog and active GitHub account
Secure agent loops and tool use; validate I/O contracts; embed policy hooks; test resilience during co-engineering.
no startups here with active blog and active GitHub account
Sanitize/trace data and reasoning; validate alignment; protect sensitive memory with privacy controls before deployment.
Adversarial testing for goal drift, prompt injection, and tool misuse; red-team sims; sandboxed calls; decision validation.
Sign models/plugins/memory; verify SBOMs; enforce cryptographically validated policies; register agents/capabilities.
no startups here with active blog and active GitHub account
Zero-trust activation: rotate ephemeral creds, apply allowlists/LLM firewalls, and fine-grained least-privilege authorization.
Monitor memory mutations for drift/poisoning, detect abnormal loops/misuse, enforce HITL overrides, and scan plugins—continuous, real-time vigilance for resilient operations as systems scale and self-orchestrate.
Correlate agent steps/tools/comms; detect anomalies (e.g., goal reversal); keep immutable logs for auditability.
Enforce role/task policies, version/retire agents, prevent privilege creep, and align evidence with AI regulations.
↑Related Awesome Lists
-
Awesome LLMSecOps — wearetyomsmnv
-
OSS LLM Security — kaplanlior
-
Awesome LLM Security — corca-ai
-
Security for AI — zmre
-
Awesome AI Security — DeepSpaceHarbor
-
Awesome AI for Cybersecurity — Billy1900
-
Awesome ML Security — Trail of Bits
-
Awesome MLSecOps — RiccardoBiosas
-
MLSecOps References — disesdi
-
Awesome ML Privacy Attacks — StratosphereIPS
-
Awesome LLM Supply Chain Security — ShenaoW
-
Awesome Prompt Injection — FonduAI
-
Awesome Jailbreak on LLMs — yueliu1999
-
Awesome LM-SSP (Large Model Security, Safety & Privacy) — ThuCCSLab
-
Security & Privacy for LLMs (llm-sp) — chawins
-
Awesome LVLM Attack — liudaizong
-
Awesome ML/SP Papers — gnipping
-
Awesome LLM JailBreak Papers — WhileBug
-
Awesome Adversarial Machine Learning — man3kin3ko
-
LLM Security & Privacy — briland
-
Awesome GenAI Security — jassics
-
Awesome GenAI CyberHub — Ashfaaq98
-
Awesome AI for Security — AmanPriyanshu
-
Awesome ML for Cybersecurity — jivoi
-
Awesome AI Security — ottosulin
-
Awesome AI4DevSecOps — awsm-research
-
Prompt Hacking Resources — PromptLabs
-
Awesome LALMs Jailbreak — WangCheng0116
-
Awesome LRMs Safety — WangCheng0116
-
Awesome LLM Safety — ydyjya
-
Awesome MCP Security — Puliczek
↑Common Acronyms
| Acronym | Full Form |
|---|---|
| AI | Artificial Intelligence |
| AGI | Artificial General Intelligence |
| ALBERT | A Lite BERT |
| AOC | Area Over Curve |
| ASR | Attack Success Rate |
| BERT | Bidirectional Encoder Representations from Transformers |
| BGMAttack | Black-box Generative Model-based Attack |
| CBA | Composite Backdoor Attack |
| CCPA | California Consumer Privacy Act |
| CNN | Convolutional Neural Network |
| CoT | Chain-of-Thought |
| DAN | Do Anything Now |
| DFS | Depth-First Search |
| DNN | Deep Neural Network |
| DPO | Direct Preference Optimization |
| DP | Differential Privacy |
| FL | Federated Learning |
| GA | Genetic Algorithm |
| GDPR | General Data Protection Regulation |
| GPT | Generative Pre-trained Transformer |
| GRPO | Group Relative Policy Optimization |
| HIPAA | Health Insurance Portability and Accountability Act |
| ICL | In-Context Learning |
| KL | Kullback–Leibler Divergence |
| LAS | Leakage-Adjusted Simulatability |
| LM | Language Model |
| LLM | Large Language Model |
| Llama | Large Language Model Meta AI |
| LoRA | Low-Rank Adapter |
| LRM | Large Reasoning Model |
| MCTS | Monte-Carlo Tree Search |
| MIA | Membership Inference Attack |
| MDP | Masking-Differential Prompting |
| MLM | Masked Language Model |
| MLLM | Multimodal Large Language Model |
| MLRM | Multimodal Large Reasoning Model |
| MoE | Mixture-of-Experts |
| NLP | Natural Language Processing |
| OOD | Out Of Distribution |
| ORM | Outcome Reward Model |
| PI | Prompt Injection |
| PII | Personally Identifiable Information |
| PAIR | Prompt Automatic Iterative Refinement |
| PLM | pre-trained Language Model |
| PRM | Process Reward Model |
| QA | Question-Answering |
| RAG | Retrieval-Augmented Generation |
| RL | Reinforcement Learning |
| RLHF | Reinforcement Learning from Human Feedback |
| RLVR | Reinforcement Learning with Verifiable Reward |
| RoBERTa | Robustly optimized BERT approach |
| SCM | Structural Causal Model |
| SGD | Stochastic Gradient Descent |
| SOTA | State of the Art |
| TAG | Gradient Attack on Transformer-based Language Models |
| VR | Verifiable Reward |
| XLNet | Transformer-XL with autoregressive and autoencoding pre-training |
↑Contributing
Contributions are welcome! If you have new resources, tools, or insights to add, feel free to submit a pull request.
This repository follows the Awesome Manifesto guidelines.
↑License
© 2025 Tal Eliyahu. Licensed under the MIT License. See LICENSE.
For Tasks:
Click tags to check more tools for each tasksFor Jobs:
Alternative AI tools for Awesome-AI-Security
Similar Open Source Tools
Awesome-AI-Security
Awesome-AI-Security is a curated list of resources for AI security, including tools, research papers, articles, and tutorials. It aims to provide a comprehensive overview of the latest developments in securing AI systems and preventing vulnerabilities. The repository covers topics such as adversarial attacks, privacy protection, model robustness, and secure deployment of AI applications. Whether you are a researcher, developer, or security professional, this collection of resources will help you stay informed and up-to-date in the rapidly evolving field of AI security.
Awesome-Lists-and-CheatSheets
Awesome-Lists is a curated index of selected resources spanning various fields including programming languages and theories, web and frontend development, server-side development and infrastructure, cloud computing and big data, data science and artificial intelligence, product design, etc. It includes articles, books, courses, examples, open-source projects, and more. The repository categorizes resources according to the knowledge system of different domains, aiming to provide valuable and concise material indexes for readers. Users can explore and learn from a wide range of high-quality resources in a systematic way.
awesome-LangGraph
Awesome LangGraph is a curated list of projects, resources, and tools for building stateful, multi-actor applications with LangGraph. It provides valuable resources for developers at all stages of development, from beginners to those building production-ready systems. The repository covers core ecosystem components, LangChain ecosystem, LangGraph platform, official resources, starter templates, pre-built agents, example applications, development tools, community projects, AI assistants, content & media, knowledge & retrieval, finance & business, sustainability, learning resources, companies using LangGraph, contributing guidelines, and acknowledgments.
Awesome-Lists
Awesome-Lists is a curated list of awesome lists across various domains of computer science and beyond, including programming languages, web development, data science, and more. It provides a comprehensive index of articles, books, courses, open source projects, and other resources. The lists are organized by topic and subtopic, making it easy to find the information you need. Awesome-Lists is a valuable resource for anyone looking to learn more about a particular topic or to stay up-to-date on the latest developments in the field.
comfyui-photoshop
ComfyUI for Photoshop is a plugin that integrates with an AI-powered image generation system to enhance the Photoshop experience with features like unlimited generative fill, customizable back-end, AI-powered artistry, and one-click transformation. The plugin requires a minimum of 6GB graphics memory and 12GB RAM. Users can install the plugin and set up the ComfyUI workflow using provided links and files. Additionally, specific files like Check points, Loras, and Detailer Lora are required for different functionalities. Support and contributions are encouraged through GitHub.
L3AGI
L3AGI is an open-source tool that enables AI Assistants to collaborate together as effectively as human teams. It provides a robust set of functionalities that empower users to design, supervise, and execute both autonomous AI Assistants and Teams of Assistants. Key features include the ability to create and manage Teams of AI Assistants, design and oversee standalone AI Assistants, equip AI Assistants with the ability to retain and recall information, connect AI Assistants to an array of data sources for efficient information retrieval and processing, and employ curated sets of tools for specific tasks. L3AGI also offers a user-friendly interface, APIs for integration with other systems, and a vibrant community for support and collaboration.
robusta
Robusta is a tool designed to enhance Prometheus notifications for Kubernetes environments. It offers features such as smart grouping to reduce notification spam, AI investigation for alert analysis, alert enrichment with additional data like pod logs, self-healing capabilities for defining auto-remediation rules, advanced routing options, problem detection without PromQL, change-tracking for Kubernetes resources, auto-resolve functionality, and integration with various external systems like Slack, Teams, and Jira. Users can utilize Robusta with or without Prometheus, and it can be installed alongside existing Prometheus setups or as part of an all-in-one Kubernetes observability stack.
bitcart
Bitcart is a platform designed for merchants, users, and developers, providing easy setup and usage. It includes various linked repositories for core daemons, admin panel, ready store, Docker packaging, Python library for coins connection, BitCCL scripting language, documentation, and official site. The platform aims to simplify the process for merchants and developers to interact and transact with cryptocurrencies, offering a comprehensive ecosystem for managing transactions and payments.
YaneuraOu
YaneuraOu is the World's Strongest Shogi engine (AI player), winner of WCSC29 and other prestigious competitions. It is an educational and USI compliant engine that supports various features such as Ponder, MultiPV, and ultra-parallel search. The engine is known for its compatibility with different platforms like Windows, Ubuntu, macOS, and ARM. Additionally, YaneuraOu offers a standard opening book format, on-the-fly opening book support, and various maintenance commands for opening books. With a massive transposition table size of up to 33TB, YaneuraOu is a powerful and versatile tool for Shogi enthusiasts and developers.
db2rest
DB2Rest is a modern low-code REST DATA API platform that simplifies the development of intelligent applications. It seamlessly integrates existing and new databases with language models (LMs/LLMs) and vector stores, enabling the rapid delivery of context-aware, reasoning applications without vendor lock-in.
computer
Cua is a tool for creating and running high-performance macOS and Linux VMs on Apple Silicon, with built-in support for AI agents. It provides libraries like Lume for running VMs with near-native performance, Computer for interacting with sandboxes, and Agent for running agentic workflows. Users can refer to the documentation for onboarding and explore demos showcasing the tool's capabilities. Additionally, accessory libraries like Core, PyLume, Computer Server, and SOM offer additional functionality. Contributions to Cua are welcome, and the tool is open-sourced under the MIT License.
AirTrail
AirTrail is a web application that allows users to track their flights and view their flight history. It features an interactive world map to view flights, flight history tracking, statistics insights, multiple user management with user authentication, responsive design, dark mode, and flight import from various sources.
sfdx-hardis
sfdx-hardis is a toolbox for Salesforce DX, developed by Cloudity, that simplifies tasks which would otherwise take minutes or hours to complete manually. It enables users to define complete CI/CD pipelines for Salesforce projects, backup metadata, and monitor any Salesforce org. The tool offers a wide range of commands that can be accessed via the command line interface or through a Visual Studio Code extension. Additionally, sfdx-hardis provides Docker images for easy integration into CI workflows. The tool is designed to be natively compliant with various platforms and tools, making it a versatile solution for Salesforce developers.
pennywiseai-tracker
PennyWise AI Tracker is a free and open-source expense tracker that uses on-device AI to turn bank SMS into a clean and searchable money timeline. It offers smart SMS parsing, clear insights, subscription tracking, on-device AI assistant, auto-categorization, data export, and supports major Indian banks. All processing happens on the user's device for privacy. The tool is designed for Android users in India who want automatic expense tracking from bank SMS, with clean categories, subscription detection, and clear insights.
cf-proxy-ex
Cloudflare Proxy EX is a tool that provides Cloudflare super proxy, OpenAI/ChatGPT proxy, Github acceleration, and online proxy services. It allows users to create a worker in Cloudflare website by copying the content from worker.js file, and add their domain name before any URL to use the tool. The tool is an improvement based on gaboolic's cloudflare-reverse-proxy, offering features like removing '/proxy/', handling redirection events, modifying headers, converting relative paths to absolute paths, and more. It aims to enhance proxy functionality and address issues faced by some websites. However, users are advised not to log in to any website through the online proxy due to potential security risks.
genkit-plugins
Community plugins repository for Google Firebase Genkit, containing various plugins for AI APIs and Vector Stores. Developed by The Fire Company, this repository offers plugins like genkitx-anthropic, genkitx-cohere, genkitx-groq, genkitx-mistral, genkitx-openai, genkitx-convex, and genkitx-hnsw. Users can easily install and use these plugins in their projects, with examples provided in the documentation. The repository also showcases products like Fireview and Giftit built using these plugins, and welcomes contributions from the community.
For similar tasks
watchtower
AIShield Watchtower is a tool designed to fortify the security of AI/ML models and Jupyter notebooks by automating model and notebook discoveries, conducting vulnerability scans, and categorizing risks into 'low,' 'medium,' 'high,' and 'critical' levels. It supports scanning of public GitHub repositories, Hugging Face repositories, AWS S3 buckets, and local systems. The tool generates comprehensive reports, offers a user-friendly interface, and aligns with industry standards like OWASP, MITRE, and CWE. It aims to address the security blind spots surrounding Jupyter notebooks and AI models, providing organizations with a tailored approach to enhancing their security efforts.
LLM-PLSE-paper
LLM-PLSE-paper is a repository focused on the applications of Large Language Models (LLMs) in Programming Language and Software Engineering (PL/SE) domains. It covers a wide range of topics including bug detection, specification inference and verification, code generation, fuzzing and testing, code model and reasoning, code understanding, IDE technologies, prompting for reasoning tasks, and agent/tool usage and planning. The repository provides a comprehensive collection of research papers, benchmarks, empirical studies, and frameworks related to the capabilities of LLMs in various PL/SE tasks.
invariant
Invariant Analyzer is an open-source scanner designed for LLM-based AI agents to find bugs, vulnerabilities, and security threats. It scans agent execution traces to identify issues like looping behavior, data leaks, prompt injections, and unsafe code execution. The tool offers a library of built-in checkers, an expressive policy language, data flow analysis, real-time monitoring, and extensible architecture for custom checkers. It helps developers debug AI agents, scan for security violations, and prevent security issues and data breaches during runtime. The analyzer leverages deep contextual understanding and a purpose-built rule matching engine for security policy enforcement.
OpenRedTeaming
OpenRedTeaming is a repository focused on red teaming for generative models, specifically large language models (LLMs). The repository provides a comprehensive survey on potential attacks on GenAI and robust safeguards. It covers attack strategies, evaluation metrics, benchmarks, and defensive approaches. The repository also implements over 30 auto red teaming methods. It includes surveys, taxonomies, attack strategies, and risks related to LLMs. The goal is to understand vulnerabilities and develop defenses against adversarial attacks on large language models.
Awesome-LLM4Cybersecurity
The repository 'Awesome-LLM4Cybersecurity' provides a comprehensive overview of the applications of Large Language Models (LLMs) in cybersecurity. It includes a systematic literature review covering topics such as constructing cybersecurity-oriented domain LLMs, potential applications of LLMs in cybersecurity, and research directions in the field. The repository analyzes various benchmarks, datasets, and applications of LLMs in cybersecurity tasks like threat intelligence, fuzzing, vulnerabilities detection, insecure code generation, program repair, anomaly detection, and LLM-assisted attacks.
quark-engine
Quark Engine is an AI-powered tool designed for analyzing Android APK files. It focuses on enhancing the detection process for auto-suggestion, enabling users to create detection workflows without coding. The tool offers an intuitive drag-and-drop interface for workflow adjustments and updates. Quark Agent, the core component, generates Quark Script code based on natural language input and feedback. The project is committed to providing a user-friendly experience for designing detection workflows through textual and visual methods. Various features are still under development and will be rolled out gradually.
vulnerability-analysis
The NVIDIA AI Blueprint for Vulnerability Analysis for Container Security showcases accelerated analysis on common vulnerabilities and exposures (CVE) at an enterprise scale, reducing mitigation time from days to seconds. It enables security analysts to determine software package vulnerabilities using large language models (LLMs) and retrieval-augmented generation (RAG). The blueprint is designed for security analysts, IT engineers, and AI practitioners in cybersecurity. It requires NVAIE developer license and API keys for vulnerability databases, search engines, and LLM model services. Hardware requirements include L40 GPU for pipeline operation and optional LLM NIM and Embedding NIM. The workflow involves LLM pipeline for CVE impact analysis, utilizing LLM planner, agent, and summarization nodes. The blueprint uses NVIDIA NIM microservices and Morpheus Cybersecurity AI SDK for vulnerability analysis.
CodeAsk
CodeAsk is a code analysis tool designed to tackle complex issues such as code that seems to self-replicate, cryptic comments left by predecessors, messy and unclear code, and long-lasting temporary solutions. It offers intelligent code organization and analysis, security vulnerability detection, code quality assessment, and other interesting prompts to help users understand and work with legacy code more efficiently. The tool aims to translate 'legacy code mountains' into understandable language, creating an illusion of comprehension and facilitating knowledge transfer to new team members.
For similar jobs
weave
Weave is a toolkit for developing Generative AI applications, built by Weights & Biases. With Weave, you can log and debug language model inputs, outputs, and traces; build rigorous, apples-to-apples evaluations for language model use cases; and organize all the information generated across the LLM workflow, from experimentation to evaluations to production. Weave aims to bring rigor, best-practices, and composability to the inherently experimental process of developing Generative AI software, without introducing cognitive overhead.
LLMStack
LLMStack is a no-code platform for building generative AI agents, workflows, and chatbots. It allows users to connect their own data, internal tools, and GPT-powered models without any coding experience. LLMStack can be deployed to the cloud or on-premise and can be accessed via HTTP API or triggered from Slack or Discord.
VisionCraft
The VisionCraft API is a free API for using over 100 different AI models. From images to sound.
kaito
Kaito is an operator that automates the AI/ML inference model deployment in a Kubernetes cluster. It manages large model files using container images, avoids tuning deployment parameters to fit GPU hardware by providing preset configurations, auto-provisions GPU nodes based on model requirements, and hosts large model images in the public Microsoft Container Registry (MCR) if the license allows. Using Kaito, the workflow of onboarding large AI inference models in Kubernetes is largely simplified.
PyRIT
PyRIT is an open access automation framework designed to empower security professionals and ML engineers to red team foundation models and their applications. It automates AI Red Teaming tasks to allow operators to focus on more complicated and time-consuming tasks and can also identify security harms such as misuse (e.g., malware generation, jailbreaking), and privacy harms (e.g., identity theft). The goal is to allow researchers to have a baseline of how well their model and entire inference pipeline is doing against different harm categories and to be able to compare that baseline to future iterations of their model. This allows them to have empirical data on how well their model is doing today, and detect any degradation of performance based on future improvements.
tabby
Tabby is a self-hosted AI coding assistant, offering an open-source and on-premises alternative to GitHub Copilot. It boasts several key features: * Self-contained, with no need for a DBMS or cloud service. * OpenAPI interface, easy to integrate with existing infrastructure (e.g Cloud IDE). * Supports consumer-grade GPUs.
spear
SPEAR (Simulator for Photorealistic Embodied AI Research) is a powerful tool for training embodied agents. It features 300 unique virtual indoor environments with 2,566 unique rooms and 17,234 unique objects that can be manipulated individually. Each environment is designed by a professional artist and features detailed geometry, photorealistic materials, and a unique floor plan and object layout. SPEAR is implemented as Unreal Engine assets and provides an OpenAI Gym interface for interacting with the environments via Python.
Magick
Magick is a groundbreaking visual AIDE (Artificial Intelligence Development Environment) for no-code data pipelines and multimodal agents. Magick can connect to other services and comes with nodes and templates well-suited for intelligent agents, chatbots, complex reasoning systems and realistic characters.