Best AI tools for< Perform Offline Inference >

AdGen AI is an AI-powered creative generator that helps businesses create high-performing ad copy and visuals for multiple ad channels. It uses machine learning models to analyze product data and generate a variety of ad creatives that are tailored to the target audience. AdGen AI also allows users to publish ads directly from the platform, making it easy to launch and manage ad campaigns.

JobInterview.guru is an AI-powered platform designed to provide personalized interview training for job seekers. Leveraging advanced AI technology, the platform offers realistic job interview simulations, detailed insights into interview questions, and personalized feedback to help users prepare effectively. With a focus on efficiency and cost-effectiveness, JobInterview.guru aims to empower users to confidently navigate their job interviews and land their dream jobs.

LambdaTest is a next-generation mobile apps and cross-browser testing cloud platform that offers a wide range of testing services. It allows users to perform manual live-interactive cross-browser testing, run Selenium, Cypress, Playwright scripts on cloud-based infrastructure, and execute AI-powered automation testing. The platform also provides accessibility testing, real devices cloud, visual regression cloud, and AI-powered test analytics. LambdaTest is trusted by over 2 million users globally and offers a unified digital experience testing cloud to accelerate go-to-market strategies.

Laxis is a revolutionary AI Meeting Assistant designed to capture and distill key insights from every customer interaction effortlessly. It seamlessly integrates across platforms, from online meetings to CRM updates, all with a user-friendly interface. Laxis empowers revenue teams to maximize every customer conversation, ensuring no valuable detail is missed. With Laxis, sales teams can close more deals with AI note-taking and insights from client conversations, business development teams can engage prospects more effectively and grow their business faster, marketing teams can repurpose podcasts, webinars, and meetings into engaging content with a single click, product and market researchers can conduct better research interviews that get to the "aha!" moment faster, project managers can remember key takeaways and status updates, and capture them for progress reports, and product and UX designers can capture and organize insights from their interviews and user research.

CampaignBuilder.AI is an AI-powered platform that enables users to quickly generate and launch AI-optimized advertising campaigns across major ad platforms. The tool offers features such as AI-generated copywriting, audience targeting, creative building, and campaign exporting. It provides creative freedom and full-funnel capabilities, making campaign creation efficient and effective for businesses of all sizes. With CampaignBuilder.AI, users can save time, improve campaign performance, and scale their advertising efforts with ease.

Laxis is an AI Meeting Assistant designed to empower revenue teams by capturing and distilling key insights from customer interactions effortlessly. It offers seamless integration across platforms, from online meetings to CRM updates, with a user-friendly interface. Laxis helps users stay focused during meetings, auto-generate meeting summaries, identify customer requirements, and extract valuable insights. It supports multilingual interactions, real-time transcriptions, and provides answers based on past conversations. Trusted by over 35,000 business professionals from 3000 organizations, Laxis saves time, improves note-taking, and enhances communication with clients and prospects.

Ask Blue J is a generative AI tool designed specifically for tax experts. It provides fast, verifiable answers to complex tax questions, helping professionals work smarter and more efficiently. With its extensive database of curated tax content and industry-leading AI technology, Ask Blue J enables users to conduct efficient research, expedite drafting, and enhance their overall productivity.

Blue J is a legal technology company founded in 2015, dedicated to enhancing tax research with the power of AI. Their AI-powered tool, Ask Blue J, provides fast and verifiable answers to tax questions, enabling tax professionals to work more efficiently. Blue J's generative AI technology helps users find authoritative sources quickly, expedite drafting processes, and cater to junior staff's research needs. The tool is trusted by hundreds of leading firms and offers a comprehensive database of curated tax content.

Sales Closer AI is an AI-powered sales tool designed to help businesses scale their sales operations by creating AI agents capable of handling various tasks such as phone calls, scheduling, and conducting personalized discovery calls. The tool integrates seamlessly with existing CRM and marketing tools, enabling users to uncover customer pain points, build rapport, and deliver interactive demos in multiple languages. Sales Closer AI continuously learns and optimizes its approach, providing detailed notes for future reference and boosting conversion rates across different industries.

GPTConsole is an AI-powered platform that helps developers build production-ready applications faster and more efficiently. Its AI agents can generate code for a variety of applications, including web applications, AI applications, and landing pages. GPTConsole also offers a range of features to help developers build and maintain their applications, including an AI agent that can learn your entire codebase and answer your questions, and a CLI tool for accessing agents directly from the command line.

Remy is an AI-powered platform designed to help product security and compliance teams resolve security risks early. It offers scalable design review capabilities, automates review initiation, generates tailored questions, and provides clear metrics and audit trails. Remy aims to augment and scale product security teams by ensuring full visibility on risky engineering plans and automating tedious review processes. The platform is built for enterprise readiness, offering SSO for convenient logins, scalability, and customization.

Validator by Yazero is a platform that helps users validate their startup ideas using AI. It provides a community where users can share their ideas, get feedback, and find collaborators. Validator also offers a variety of features to help users improve their ideas, such as idea validation, market research, and financial planning.

pdfAssistant is a powerful AI chatbot designed to assist users with various PDF processing tasks. It offers a user-friendly chat-based interface that allows users to convert, watermark, merge, split, and perform other PDF-related operations using natural language commands. The application is powered by industry-leading PDF and AI technology, providing fast and accurate results. With pdfAssistant, users can work smarter and more efficiently by simplifying complex PDF software processes.

KYP.ai is a productivity intelligence platform that offers a 360° view of organizations across people, process, and technology dimensions. It provides instant productivity intelligence, end-to-end process optimization, holistic productivity insights, ROI-driven automation, and unparalleled scalability. The platform helps in live visibility, immediate impact, hybrid workplace management, technology landscape rationalization, and AI-powered aggregation and analysis. KYP.ai focuses on workforce enablement, no integration hassles, no-code configuration, and secure, privacy-compliant data processing.

Solidroad is an AI-first training and feedback platform that turns company knowledge-base into immersive training programs. It offers personalized coaching, realistic simulations, and real-time feedback to improve team performance. The platform aims to make training programs easier to manage and more engaging for employees.

Vizly is an AI-powered data analysis tool that empowers users to make the most of their data. It allows users to chat with their data, visualize insights, and perform complex analysis. Vizly supports various file formats like CSV, Excel, and JSON, making it versatile for different data sources. The tool is free to use for up to 10 messages per month and offers a student discount of 50%. Vizly is suitable for individuals, students, academics, and organizations looking to gain actionable insights from their data.

Yogger is a video analysis and AI movement assessment tool that empowers coaches, trainers, physical therapists, and athletes to gather precise movement data for performance enhancement, recovery optimization, and injury risk reduction. The software solutions offered by Yogger enable users to analyze movement, critique form, and visualize joint tracking with the help of AI technology. With Yogger, users can streamline client evaluations through automated movement screenings, delivering objective scores and data in just 60 seconds. The tool provides a versatile suite of features for any sport or activity, all accessible from a mobile device.

GeekSight Inc. offers a range of Trello Power-Ups designed to enhance team collaboration and productivity. One of their key products is Notes & Docs, an AI-powered note-taking tool integrated with Trello. The tool aims to streamline task management by providing comprehensive task context information organization and content creation workflows. Additionally, they provide PersonaTool for building customer personas and Card Annotations for enhancing communication on Trello boards.

Dreamwriter is an AI-powered content creation tool that allows users to design beautiful, on-brand premium content in minutes. By leveraging the power of AI and the user's brand voice, Dreamwriter helps in developing hard-hitting PDFs & PPTs tailored to the exact target audience. The tool features an intuitive UI editor, real-time collaboration, simplified daily content generation, and the ability to write in multiple languages. Dreamwriter aims to streamline the content creation process by providing a toolbox of leading solutions to produce premium content at unprecedented speeds.

Parity is a personal scheduling assistant that helps you save time by automating the process of scheduling meetings. It works with your favorite tools, including Zoom, Google Meet, and Microsoft Teams, and can coordinate with everyone else on the thread to find a time that works for everyone. You can also get a briefing of all your calendar events just by asking, and forward emails about events to Parity to add them to your calendar automatically. Parity is compatible with your Google and Outlook calendars, and is free to use.

ppl.llm.serving is a serving component for Large Language Models (LLMs) within the PPL.LLM system. It provides a server based on gRPC and supports inference for LLaMA. The repository includes instructions for prerequisites, quick start guide, model exporting, server setup, client usage, benchmarking, and offline inference. Users can refer to the LLaMA Guide for more details on using this serving component.

Colossal-AI is a deep learning system for large-scale parallel training. It provides a unified interface to scale sequential code of model training to distributed environments. Colossal-AI supports parallel training methods such as data, pipeline, tensor, and sequence parallelism and is integrated with heterogeneous training and zero redundancy optimizer.

InternLM-XComposer2 is a groundbreaking vision-language large model (VLLM) based on InternLM2-7B excelling in free-form text-image composition and comprehension. It boasts several amazing capabilities and applications: * **Free-form Interleaved Text-Image Composition** : InternLM-XComposer2 can effortlessly generate coherent and contextual articles with interleaved images following diverse inputs like outlines, detailed text requirements and reference images, enabling highly customizable content creation. * **Accurate Vision-language Problem-solving** : InternLM-XComposer2 accurately handles diverse and challenging vision-language Q&A tasks based on free-form instructions, excelling in recognition, perception, detailed captioning, visual reasoning, and more. * **Awesome performance** : InternLM-XComposer2 based on InternLM2-7B not only significantly outperforms existing open-source multimodal models in 13 benchmarks but also **matches or even surpasses GPT-4V and Gemini Pro in 6 benchmarks** We release InternLM-XComposer2 series in three versions: * **InternLM-XComposer2-4KHD-7B** 🤗: The high-resolution multi-task trained VLLM model with InternLM-7B as the initialization of the LLM for _High-resolution understanding_ , _VL benchmarks_ and _AI assistant_. * **InternLM-XComposer2-VL-7B** 🤗 : The multi-task trained VLLM model with InternLM-7B as the initialization of the LLM for _VL benchmarks_ and _AI assistant_. **It ranks as the most powerful vision-language model based on 7B-parameter level LLMs, leading across 13 benchmarks.** * **InternLM-XComposer2-VL-1.8B** 🤗 : A lightweight version of InternLM-XComposer2-VL based on InternLM-1.8B. * **InternLM-XComposer2-7B** 🤗: The further instruction tuned VLLM for _Interleaved Text-Image Composition_ with free-form inputs. Please refer to Technical Report and 4KHD Technical Reportfor more details.

Pai-Megatron-Patch is a deep learning training toolkit built for developers to train and predict LLMs & VLMs by using Megatron framework easily. With the continuous development of LLMs, the model structure and scale are rapidly evolving. Although these models can be conveniently manufactured using Transformers or DeepSpeed training framework, the training efficiency is comparably low. This phenomenon becomes even severer when the model scale exceeds 10 billion. The primary objective of Pai-Megatron-Patch is to effectively utilize the computational power of GPUs for LLM. This tool allows convenient training of commonly used LLM with all the accelerating techniques provided by Megatron-LM.

UHGEval is a comprehensive framework designed for evaluating the hallucination phenomena. It includes UHGEval, a framework for evaluating hallucination, XinhuaHallucinations dataset, and UHGEval-dataset pipeline for creating XinhuaHallucinations. The framework offers flexibility and extensibility for evaluating common hallucination tasks, supporting various models and datasets. Researchers can use the open-source pipeline to create customized datasets. Supported tasks include QA, dialogue, summarization, and multi-choice tasks.

ReVA (Reverse Engineering Assistant) is a project aimed at building a disassembler agnostic AI assistant for reverse engineering tasks. It utilizes a tool-driven approach, providing small tools to the user to empower them in completing complex tasks. The assistant is designed to accept various inputs, guide the user in correcting mistakes, and provide additional context to encourage exploration. Users can ask questions, perform tasks like decompilation, class diagram generation, variable renaming, and more. ReVA supports different language models for online and local inference, with easy configuration options. The workflow involves opening the RE tool and program, then starting a chat session to interact with the assistant. Installation includes setting up the Python component, running the chat tool, and configuring the Ghidra extension for seamless integration. ReVA aims to enhance the reverse engineering process by breaking down actions into small parts, including the user's thoughts in the output, and providing support for monitoring and adjusting prompts.

CSGHub is an open source, trustworthy large model asset management platform that can assist users in governing the assets involved in the lifecycle of LLM and LLM applications (datasets, model files, codes, etc). With CSGHub, users can perform operations on LLM assets, including uploading, downloading, storing, verifying, and distributing, through Web interface, Git command line, or natural language Chatbot. Meanwhile, the platform provides microservice submodules and standardized OpenAPIs, which could be easily integrated with users' own systems. CSGHub is committed to bringing users an asset management platform that is natively designed for large models and can be deployed On-Premise for fully offline operation. CSGHub offers functionalities similar to a privatized Huggingface(on-premise Huggingface), managing LLM assets in a manner akin to how OpenStack Glance manages virtual machine images, Harbor manages container images, and Sonatype Nexus manages artifacts.

Marlin is a highly optimized FP16xINT4 matmul kernel designed for large language model (LLM) inference, offering close to ideal speedups up to batchsizes of 16-32 tokens. It is suitable for larger-scale serving, speculative decoding, and advanced multi-inference schemes like CoT-Majority. Marlin achieves optimal performance by utilizing various techniques and optimizations to fully leverage GPU resources, ensuring efficient computation and memory management.

The 'awesome-local-llms' repository is a curated list of open-source tools for local Large Language Model (LLM) inference, covering both proprietary and open weights LLMs. The repository categorizes these tools into LLM inference backend engines, LLM front end UIs, and all-in-one desktop applications. It collects GitHub repository metrics as proxies for popularity and active maintenance. Contributions are encouraged, and users can suggest additional open-source repositories through the Issues section or by running a provided script to update the README and make a pull request. The repository aims to provide a comprehensive resource for exploring and utilizing local LLM tools.

biniou is a self-hosted webui for various GenAI (generative artificial intelligence) tasks. It allows users to generate multimedia content using AI models and chatbots on their own computer, even without a dedicated GPU. The tool can work offline once deployed and required models are downloaded. It offers a wide range of features for text, image, audio, video, and 3D object generation and modification. Users can easily manage the tool through a control panel within the webui, with support for various operating systems and CUDA optimization. biniou is powered by Huggingface and Gradio, providing a cross-platform solution for AI content generation.

The Keras-llm-robot Web UI project is an open-source tool designed for offline deployment and testing of various open-source models from the Hugging Face website. It allows users to combine multiple models through configuration to achieve functionalities like multimodal, RAG, Agent, and more. The project consists of three main interfaces: chat interface for language models, configuration interface for loading models, and tools & agent interface for auxiliary models. Users can interact with the language model through text, voice, and image inputs, and the tool supports features like model loading, quantization, fine-tuning, role-playing, code interpretation, speech recognition, image recognition, network search engine, and function calling.

Venice is a derived data storage platform, providing the following characteristics: 1. High throughput asynchronous ingestion from batch and streaming sources (e.g. Hadoop and Samza). 2. Low latency online reads via remote queries or in-process caching. 3. Active-active replication between regions with CRDT-based conflict resolution. 4. Multi-cluster support within each region with operator-driven cluster assignment. 5. Multi-tenancy, horizontal scalability and elasticity within each cluster. The above makes Venice particularly suitable as the stateful component backing a Feature Store, such as Feathr. AI applications feed the output of their ML training jobs into Venice and then query the data for use during online inference workloads.

A comprehensive repository of resources for Chinese large language models (LLMs), including data processing tools, fine-tuning frameworks, inference libraries, evaluation platforms, RAG engines, agent frameworks, books, courses, tutorials, and tips. The repository covers a wide range of tools and resources for working with LLMs, from data labeling and processing to model fine-tuning, inference, evaluation, and application development. It also includes resources for learning about LLMs through books, courses, and tutorials, as well as insights and strategies from building with LLMs.

AgentScope is a multi-agent platform designed to empower developers to build multi-agent applications with large-scale models. It features three high-level capabilities: Easy-to-Use, High Robustness, and Actor-Based Distribution. AgentScope provides a list of `ModelWrapper` to support both local model services and third-party model APIs, including OpenAI API, DashScope API, Gemini API, and ollama. It also enables developers to rapidly deploy local model services using libraries such as ollama (CPU inference), Flask + Transformers, Flask + ModelScope, FastChat, and vllm. AgentScope supports various services, including Web Search, Data Query, Retrieval, Code Execution, File Operation, and Text Processing. Example applications include Conversation, Game, and Distribution. AgentScope is released under Apache License 2.0 and welcomes contributions.

DB-GPT is a personal database administrator that can solve database problems by reading documents, using various tools, and writing analysis reports. It is currently undergoing an upgrade. **Features:** * **Online Demo:** * Import documents into the knowledge base * Utilize the knowledge base for well-founded Q&A and diagnosis analysis of abnormal alarms * Send feedbacks to refine the intermediate diagnosis results * Edit the diagnosis result * Browse all historical diagnosis results, used metrics, and detailed diagnosis processes * **Language Support:** * English (default) * Chinese (add "language: zh" in config.yaml) * **New Frontend:** * Knowledgebase + Chat Q&A + Diagnosis + Report Replay * **Extreme Speed Version for localized llms:** * 4-bit quantized LLM (reducing inference time by 1/3) * vllm for fast inference (qwen) * Tiny LLM * **Multi-path extraction of document knowledge:** * Vector database (ChromaDB) * RESTful Search Engine (Elasticsearch) * **Expert prompt generation using document knowledge** * **Upgrade the LLM-based diagnosis mechanism:** * Task Dispatching -> Concurrent Diagnosis -> Cross Review -> Report Generation * Synchronous Concurrency Mechanism during LLM inference * **Support monitoring and optimization tools in multiple levels:** * Monitoring metrics (Prometheus) * Flame graph in code level * Diagnosis knowledge retrieval (dbmind) * Logical query transformations (Calcite) * Index optimization algorithms (for PostgreSQL) * Physical operator hints (for PostgreSQL) * Backup and Point-in-time Recovery (Pigsty) * **Continuously updated papers and experimental reports** This project is constantly evolving with new features. Don't forget to star ⭐ and watch 👀 to stay up to date.

Chronon is a platform that simplifies and improves ML workflows by providing a central place to define features, ensuring point-in-time correctness for backfills, simplifying orchestration for batch and streaming pipelines, offering easy endpoints for feature fetching, and guaranteeing and measuring consistency. It offers benefits over other approaches by enabling the use of a broad set of data for training, handling large aggregations and other computationally intensive transformations, and abstracting away the infrastructure complexity of data plumbing.

MInference is a tool designed to accelerate pre-filling for long-context Language Models (LLMs) by leveraging dynamic sparse attention. It achieves up to a 10x speedup for pre-filling on an A100 while maintaining accuracy. The tool supports various decoding LLMs, including LLaMA-style models and Phi models, and provides custom kernels for attention computation. MInference is useful for researchers and developers working with large-scale language models who aim to improve efficiency without compromising accuracy.

WilmerAI is a middleware system designed to process prompts before sending them to Large Language Models (LLMs). It categorizes prompts, routes them to appropriate workflows, and generates manageable prompts for local models. It acts as an intermediary between the user interface and LLM APIs, supporting multiple backend LLMs simultaneously. WilmerAI provides API endpoints compatible with OpenAI API, supports prompt templates, and offers flexible connections to various LLM APIs. The project is under heavy development and may contain bugs or incomplete code.

QServe is a serving system designed for efficient and accurate Large Language Models (LLM) on GPUs with W4A8KV4 quantization. It achieves higher throughput compared to leading industry solutions, allowing users to achieve A100-level throughput on cheaper L40S GPUs. The system introduces the QoQ quantization algorithm with 4-bit weight, 8-bit activation, and 4-bit KV cache, addressing runtime overhead challenges. QServe improves serving throughput for various LLM models by implementing compute-aware weight reordering, register-level parallelism, and fused attention memory-bound techniques.

Analyzing VC projects and due diligence.

Social Ads Trainer

Breathing coach for athletes, focusing on performance and recovery

Asistente especializado en el manejo de la parada cardiorespiratoria según las recomendaciones del ERC (2021) y del ILCOR (2023).

Mysterious magician creating baffling verbal and numerical tricks of the mind.

优云康手外科专业机器人

Actor of 'Blackadder' characters with script knowledge.

stand up comedy show

A guide in esoteric and occult knowledge, utilizing innovative chaos magick techniques.

Expert in GMC vehicle maintenance and repair, with internet browsing for extra info.

Expert Neurosurgeon offering tailored deep-dive consultations.

Enchanting, intriguing illusion expert

Your go-to source for Impacket documentation

Advanced AI for complex computations and data analysis.

Your go-to assistant for all things lifeguarding.

I can help you with any questions you have about your pet.

I rap Douban's top movies with a panda flair!

Ensures project deliverables meet predetermined quality standards.

Expert in Stata for statistical analysis guidance

Emergency Medicine Educator