aioboto3

Wrapper to use boto3 resources with the aiobotocore async backend

Stars: 744

Visit

aioboto3 is an async AWS SDK for Python that allows users to use near enough all of the boto3 client commands in an async manner just by prefixing the command with `await`. It combines the great work of boto3 and aiobotocore, enabling users to use higher level APIs provided by boto3 in an asynchronous manner. The package provides support for various AWS services such as DynamoDB, S3, Kinesis, SSM Parameter Store, and Athena. It also offers features like client-side encryption using KMS-Managed master keys and supports asyncifying `get_presigned_url`. The library closely mimics the usage of boto3 and is mainly developed to be used in async microservices.

README:

======================== Async AWS SDK for Python

.. image:: https://img.shields.io/pypi/v/aioboto3.svg :target: https://pypi.python.org/pypi/aioboto3

.. image:: https://github.com/terrycain/aioboto3/actions/workflows/CI.yml/badge.svg :target: https://github.com/terrycain/aioboto3/actions

.. image:: https://readthedocs.org/projects/aioboto3/badge/?version=latest :target: https://aioboto3.readthedocs.io :alt: Documentation Status

.. image:: https://pyup.io/repos/github/terrycain/aioboto3/shield.svg :target: https://pyup.io/repos/github/terrycain/aioboto3/ :alt: Updates

Breaking changes for v11: The S3Transfer config passed into upload/download_file etc.. has been updated to that it matches what boto3 uses

Breaking changes for v9: aioboto3.resource and aioboto3.client methods no longer exist, make a session then call session.client etc... This was done for various reasons but mainly that it prevents the default session living longer than it should as that breaks situations where eventloops are replaced.

The .client and .resource functions must now be used as async context managers.

Now that aiobotocore has reached version 1.0.1, a side effect of the work put in to fix various issues like bucket region redirection and supporting web assume role type credentials, the client must now be instantiated using a context manager, which by extension applies to the resource creator. You used to get away with calling res = aioboto3.resource('dynamodb') but that no longer works. If you really want to do that, you can do res = await aioboto3.resource('dynamodb').__aenter__() but you'll need to remember to call __aexit__.

There will most likely be some parts that dont work now which I've missed, just make an issue and we'll get them resoved quickly.

Creating service resources must also be async now, e.g.

.. code-block:: python

async def main():
    session = aioboto3.Session()
    async with session.resource("s3") as s3:
        bucket = await s3.Bucket('mybucket')  # <----------------
        async for s3_object in bucket.objects.all():
            print(s3_object)

Updating to aiobotocore 1.0.1 also brings with it support for running inside EKS as well as asyncifying get_presigned_url

This package is mostly just a wrapper combining the great work of boto3_ and aiobotocore_.

aiobotocore allows you to use near enough all of the boto3 client commands in an async manner just by prefixing the command with await.

With aioboto3 you can now use the higher level APIs provided by boto3 in an asynchronous manner. Mainly I developed this as I wanted to use the boto3 dynamodb Table object in some async microservices.

While all resources in boto3 should work I havent tested them all, so if what your after is not in the table below then try it out, if it works drop me an issue with a simple test case and I'll add it to the table.

+---------------------------+--------------------+ | Services | Status | +===========================+====================+ | DynamoDB Service Resource | Tested and working | +---------------------------+--------------------+ | DynamoDB Table | Tested and working | +---------------------------+--------------------+ | S3 | Working | +---------------------------+--------------------+ | Kinesis | Working | +---------------------------+--------------------+ | SSM Parameter Store | Working | +---------------------------+--------------------+ | Athena | Working | +---------------------------+--------------------+

Example

Simple example of using aioboto3 to put items into a dynamodb table

.. code-block:: python

import asyncio
import aioboto3
from boto3.dynamodb.conditions import Key


async def main():
    session = aioboto3.Session()
    async with session.resource('dynamodb', region_name='eu-central-1') as dynamo_resource:
        table = await dynamo_resource.Table('test_table')

        await table.put_item(
            Item={'pk': 'test1', 'col1': 'some_data'}
        )

        result = await table.query(
            KeyConditionExpression=Key('pk').eq('test1')
        )

        # Example batch write
        more_items = [{'pk': 't2', 'col1': 'c1'}, \
                      {'pk': 't3', 'col1': 'c3'}]
        async with table.batch_writer() as batch:
            for item_ in more_items:
                await batch.put_item(Item=item_)

loop = asyncio.get_event_loop()
loop.run_until_complete(main())

# Outputs:
#  [{'col1': 'some_data', 'pk': 'test1'}]

Things that either dont work or have been patched

As this library literally wraps boto3, its inevitable that some things won't magically be async.

Fixed:

s3_client.download_file* This is performed by the s3transfer module. -- Patched with get_object
s3_client.upload_file* This is performed by the s3transfer module. -- Patched with custom multipart upload
s3_client.copy This is performed by the s3transfer module. -- Patched to use get_object -> upload_fileobject
dynamodb_resource.Table.batch_writer This now returns an async context manager which performs the same function
Resource waiters - You can now await waiters which are part of resource objects, not just client waiters, e.g. await dynamodbtable.wait_until_exists()
Resource object properties are normally autoloaded, now they are all co-routines and the metadata they come from will be loaded on first await and then cached thereafter.
S3 Bucket.objects object now works and has been asyncified. Examples here - https://aioboto3.readthedocs.io/en/latest/usage.html#s3-resource-objects

Amazon S3 Client-Side Encryption

Boto3 doesn't support AWS client-side encryption so until they do I've added basic support for it. Docs here CSE_

CSE requires the python cryptography library so if you do pip install aioboto3[s3cse] that'll also include cryptography.

This library currently supports client-side encryption using KMS-Managed master keys performing envelope encryption using either AES/CBC/PKCS5Padding or preferably AES/GCM/NoPadding. The files generated are compatible with the Java Encryption SDK so I will assume they are compatible with the Ruby, PHP, Go and C++ libraries as well.

Non-KMS managed keys are not yet supported but if you have use of that, raise an issue and i'll look into it.

Documentation

Docs are here - https://aioboto3.readthedocs.io/en/latest/

Examples here - https://aioboto3.readthedocs.io/en/latest/usage.html

Features

Closely mimics the usage of boto3.

Todo

More examples
Set up docs
Look into monkey-patching the aws xray sdk to be more async if it needs to be.

Credits

This package was created with Cookiecutter_ and the audreyr/cookiecutter-pypackage_ project template. It also makes use of the aiobotocore_ and boto3_ libraries. All the credit goes to them, this is mainly a wrapper with some examples.

.. _aiobotocore: https://github.com/aio-libs/aiobotocore .. _boto3: https://github.com/boto/boto3 .. _Cookiecutter: https://github.com/audreyr/cookiecutter .. _audreyr/cookiecutter-pypackage: https://github.com/audreyr/cookiecutter-pypackage .. _CSE: https://aioboto3.readthedocs.io/en/latest/cse.html

For Tasks:

Click tags to check more tools for each tasks

manage s3 buckets interact with dynamodb upload files to s3 query data in dynamodb perform batch writes in dynamodb

For Jobs:

python developer cloud engineer devops engineer aws solutions architect backend developer

Alternative AI tools for aioboto3

Similar Open Source Tools

aioboto3

github

: 744

aiomisc

aiomisc is a Python library that provides a collection of utility functions and classes for working with asynchronous I/O in a more intuitive and efficient way. It offers features like worker pools, connection pools, circuit breaker pattern, and retry mechanisms to make asyncio code more robust and easier to maintain. The library simplifies the architecture of software using asynchronous I/O, making it easier for developers to write reliable and scalable asynchronous code.

github

: 388

aiosmtpd

aiosmtpd is an asyncio-based SMTP server implementation that provides a modern and efficient way to handle SMTP and LMTP protocols in Python 3. It replaces the outdated asyncore and asynchat modules with asyncio for improved asynchronous I/O operations. The project aims to offer a more user-friendly, extendable, and maintainable solution for handling email protocols in Python applications. It is actively maintained by experienced Python developers and offers full documentation for easy integration and usage.

github

: 304

suno-api

Suno AI API is an open-source project that allows developers to integrate the music generation capabilities of Suno.ai into their own applications. The API provides a simple and convenient way to generate music, lyrics, and other audio content using Suno.ai's powerful AI models. With Suno AI API, developers can easily add music generation functionality to their apps, websites, and other projects.

github

: 1.7k

consult-llm-mcp

Consult LLM MCP is an MCP server that enables users to consult powerful AI models like GPT-5.2, Gemini 3.0 Pro, and DeepSeek Reasoner for complex problem-solving. It supports multi-turn conversations, direct queries with optional file context, git changes inclusion for code review, comprehensive logging with cost estimation, and various CLI modes for Gemini and Codex. The tool is designed to simplify the process of querying AI models for assistance in resolving coding issues and improving code quality.

github

: 60

langserve

LangServe helps developers deploy `LangChain` runnables and chains as a REST API. This library is integrated with FastAPI and uses pydantic for data validation. In addition, it provides a client that can be used to call into runnables deployed on a server. A JavaScript client is available in LangChain.js.

github

: 1.9k

APIMyLlama

APIMyLlama is a server application that provides an interface to interact with the Ollama API, a powerful AI tool to run LLMs. It allows users to easily distribute API keys to create amazing things. The tool offers commands to generate, list, remove, add, change, activate, deactivate, and manage API keys, as well as functionalities to work with webhooks, set rate limits, and get detailed information about API keys. Users can install APIMyLlama packages with NPM, PIP, Jitpack Repo+Gradle or Maven, or from the Crates Repository. The tool supports Node.JS, Python, Java, and Rust for generating responses from the API. Additionally, it provides built-in health checking commands for monitoring API health status.

github

: 79

json_repair

This simple package can be used to fix an invalid json string. To know all cases in which this package will work, check out the unit test. Inspired by https://github.com/josdejong/jsonrepair Motivation Some LLMs are a bit iffy when it comes to returning well formed JSON data, sometimes they skip a parentheses and sometimes they add some words in it, because that's what an LLM does. Luckily, the mistakes LLMs make are simple enough to be fixed without destroying the content. I searched for a lightweight python package that was able to reliably fix this problem but couldn't find any. So I wrote one How to use from json_repair import repair_json good_json_string = repair_json(bad_json_string) # If the string was super broken this will return an empty string You can use this library to completely replace `json.loads()`: import json_repair decoded_object = json_repair.loads(json_string) or just import json_repair decoded_object = json_repair.repair_json(json_string, return_objects=True) Read json from a file or file descriptor JSON repair provides also a drop-in replacement for `json.load()`: import json_repair try: file_descriptor = open(fname, 'rb') except OSError: ... with file_descriptor: decoded_object = json_repair.load(file_descriptor) and another method to read from a file: import json_repair try: decoded_object = json_repair.from_file(json_file) except OSError: ... except IOError: ... Keep in mind that the library will not catch any IO-related exception and those will need to be managed by you Performance considerations If you find this library too slow because is using `json.loads()` you can skip that by passing `skip_json_loads=True` to `repair_json`. Like: from json_repair import repair_json good_json_string = repair_json(bad_json_string, skip_json_loads=True) I made a choice of not using any fast json library to avoid having any external dependency, so that anybody can use it regardless of their stack. Some rules of thumb to use: - Setting `return_objects=True` will always be faster because the parser returns an object already and it doesn't have serialize that object to JSON - `skip_json_loads` is faster only if you 100% know that the string is not a valid JSON - If you are having issues with escaping pass the string as **raw** string like: `r"string with escaping\"" Adding to requirements Please pin this library only on the major version! We use TDD and strict semantic versioning, there will be frequent updates and no breaking changes in minor and patch versions. To ensure that you only pin the major version of this library in your `requirements.txt`, specify the package name followed by the major version and a wildcard for minor and patch versions. For example: json_repair==0.* In this example, any version that starts with `0.` will be acceptable, allowing for updates on minor and patch versions. How it works This module will parse the JSON file following the BNF definition: ::= | ::= | | ; Where: ; is a valid real number expressed in one of a number of given formats ; is a string of valid characters enclosed in quotes ; is one of the literal strings 'true', 'false', or 'null' (unquoted) ::=