
dataengineering-roadmap
Un repositorio más con conceptos básicos, desafíos técnicos y recursos sobre ingeniería de datos en español 🧙✨
Stars: 574

A repository providing basic concepts, technical challenges, and resources on data engineering in Spanish. It is a curated list of free, Spanish-language materials found on the internet to facilitate the study of data engineering enthusiasts. The repository covers programming fundamentals, programming languages like Python, version control with Git, database fundamentals, SQL, design concepts, Big Data, analytics, cloud computing, data processing, and job search tips in the IT field.
README:
Un repositorio más con conceptos básicos, desafíos técnicos y recursos sobre ingeniería de datos en español 🧙✨
¿Te gustaría aportar al repositorio? Visitá la guía de contribución
Nota: la siguiente ruta de aprendizaje está diseñada a criterio personal con la idea de facilitar el estudio de aquellos interesados en la ingeniería de datos con material libre, gratuito y en español que encontré en internet. No es una guía definitiva ni un curso, es una lista de recursos que puede ser mejorada con el tiempo con contribuciones de la comunidad.
📚 Libros de ingeniería de datos en inglés
📖 Patrones de Diseño para DE en inglés
Comenzamos con la comprensión de los conceptos fundamentales de programación y lógica. Esta sección puede ser desarrollada simultáneamente con el aprendizaje del lenguaje de programación que elijan.
- Curso: Programación Básica de Platzi
- Videos: Introducción a los Algoritmos y la Programación de TodoCode
- Videos: Ejercicios de Pseudocódigo de TodoCode
- Videos: Linea de Comandos de Datademia
- Videos: Bash scripting de Fazt
- Lectura: Introducción a la línea de comandos de Linux y el shell de Microsoft Learn
Recomiendo iniciar con Python debido a su curva de aprendizaje amigable y su prevalencia en la industria actual. No obstante, es importante destacar que el procesamiento de datos también puede realizarse con R, Java, Scala, Julia, entre otros.
- Videos: Python desde 0 de PildorasInformáticas
- Curso: Computación científica con Python de FreeCodeCamp
- Curso: Álgebra universitaria con Python de FreeCodeCamp
- Curso: Harvard CS50’s Introducción a la programación con Python subtítulado de FreeCodeCamp
- Curso: Python intermedio subtitulado de FreeCodeCamp
- Curso: Pandas de Kaggle
- Videos: Expresiones Regulares de Ada Lovecode
- Video: Principios de la Programación Orientada a Objetos de BettaTech
- Videos: Programación Orientada a Objetos explicada con Minecraft de Absolute
- Curso: Julia para gente con prisa de Miguel Raz
El aprendizaje sobre el control de versiones no solo es valioso al trabajar en equipos, sino que también nos proporciona la capacidad de rastrear, comprender y gestionar los cambios realizados en nuestro proyecto y así mantener un desarrollo eficiente y colaborativo.
- Video: ¿Qué es el control de versiones y porque es tan importante para programar? de Datademia
- Curso: Git y Github de MoureDev
- Videos: Git y Github de TodoCode
- Lectura: Usa Git correctamente de Attlasian
- Juego: Learn Git Branching
- Notebooks: Google Collab, Jupyter o Deepnote
- IDE: VSCode o Spyder
En esta instancia toca aprender sobre las bases de datos. La elección del gestor de bases de datos a utilizar queda a tu criterio, aunque personalmente recomiendo PostgreSQL para datos estructurados y MongoDB para datos no estructurados. Sin embargo, existen muchas otras opciones: MySQL, SQLite y demás.
- Videos: Introducción a las bases de datos de TodoCode
- Lectura: Diferencias entre DDL, DML y DCL de TodoPostgreSQL
- Video: Procedimientos almacenados #1 de Héctor de León
- Video: Procedimientos almacenados #2 de Héctor de León
- Video: MongoDB de Fazt
- Videos: MongoDB de MitoCode
También aprenderás SQL, un lenguaje de consulta para gestionar y manipular las bases de datos relacionales.
Ahora seguimos con conceptos más avanzados que nos servirán para diseñar bases de datos, datalake, datawarehouses, esquemas, etcétera.
- Video: ¿Cuándo utilizar SQL y cuando NoSQL? de Héctor de León
- Video: ¿Cómo se modelan las bases de datos NoSQL? de HolaMundo
- Lectura: Bases de datos orientadas a grafos de Oracle
- Video: Bases de Datos de Grafos, Fundamentos y Práctica de Datahack
Lo siguiente es entender algunos conceptos de Big Data. Además, resulta interesante adquirir conocimientos básicos sobre inteligencia artificial, inteligencia de negocios y análisis de datos sin la necesidad de profundizar demasiado.
- Video: Big Data para dummies de Datahack
- Lectura: Big Data: ¿Qué es y cómo ayuda a mi negocio? de Salesforce
- Certificación: Diseña y programa soluciones IoT con el uso de Big Data de Universidad del Rosario
- Certificación: Big Data de University of California San Diego
- Video: Big data y privacidad de Databits
- Videos: Gobierno de Datos de Smart Data
- Video: Cómo Iniciar con Gobierno de Datos sin Romper el Presupuesto de Software Gurú
- Certificación: Fundamentos profesionales del análisis de datos, de Microsoft y LinkedIn
- Certificación: Certificado profesional de Google Data Analytics
- Certificación: Certificado profesional de Analista de datos de IBM
- Curso: Análisis de datos con Python de FreeCodeCamp
- Video: Storytelling: ¿Cómo convertir tu contenido en una historia? de Coderhouse
- Curso: Machine Learning con Python de FreeCodeCamp
- Canal: AprendeIA con Ligdi Gonzalez
- Videos: Aprende Inteligencia Artificial de Dot CSV
- Video: Cómo usar ChatGPT en ingeniería de datos de Datalytics
- Curso: Inteligencia Artificial subtitulado de Universidad de Columbia
- Videos: Google Business Intelligence Certificate subtitulado de Google Career
- Videos: ¡Business Intelligence para Todos! de PEALCALA
En esta sección está el corazón de la ingeniería de datos, veremos que son los data pipelines, qué es un ETL, orquestadores, y más. Además, dejo una lista de conceptos clave qué voy a ir actualizando con sus recursos respectivos a futuro, si te interesa aprenderlos en detalle, podés buscar en los libros subidos en el repositorio.
- Video: Ingeniería de datos: viaje al corazón de los proyectos de datos de RockingData
- Video: ¿Cómo convertirte en un verdadero Ingeniero de Datos? de Databits
- Videos: Preprocesamiento de Datos en Python de Rocio Chavez
- Videos: Preprocesamiento de Datos en R de Rocio Chavez
- Video: Pruebas A/B: Datos, no opiniones de SantanDev
- Cargas incrementales
- Colas de mensajería
- Expresiones Cron
- Modelo relacional
- Modelo dimensional
- Facts y dimensiones
- Datalake, Datamart, Datawarehouse y Dataqube
- Diseño por columnas y basada por filas
- Esquemas star y snowflake
- Esquemas on read y on write
- Videos: Airflow de Data Engineering LATAM
- Video: Automatizando ideas con Apache Airflow - Yesi Díaz de Software Gurú
- Videos: Pentaho Spoon de LEARNING-BI
- Videos: Luigi subtitulado de Seattle Data Guy
- Lectura: Azure Data Factory de Microsoft
- Procesamiento de datos por lotes o batch
- Procesamiento en tiempo real o streaming
- Arquitecturas lambda y kappa
- Lectura: Diferencias clave entre el OLAP y el OLTP de AWS
- Video: Construye ETL en batch y streaming con Spark de Databits
- Lectura: Comparación de contenedores y máquinas virtuales de Atlassian
- Videos: Docker de Pelado Nerd
- Videos: Kubernetes de Pelado Nerd
- Lectura: ¿Qué es un sistema distribuido? de Atlassian
- Videos: Spark de Data Engineering LATAM
- Video: Infraestructura como código para ingeniería de datos de Spark México
- Videos: Apache Spark de NullSafe Architect
- Videos: Apache Kafka de NullSafe Architect
- Video: Great Expectations: Validar Data Pipelines como un Profesional por CodingEric en la PyConAr 2020
- Video: ETL Testing y su Automatización con Python por Patricio Miner en la #QSConf 2023
Es útil tener conocimientos de cloud computing. Llegado a este punto, te recomendaría considerar la preparación de certificaciones oficiales. Aunque estos exámenes suelen tener un costo, puedes encontrar recursos de preparación gratuitos y oficiales de los proveedores más conocidos en la industria.
- Video: Fundamentos de Cloud Computing de Datahack
- Lectura: Descubre las ventajas y desventajas de la nube de Platzi
- Lectura: Arquitectura para Big Data en Cloud de Platzi
- Ingeniería de datos de Google Cloud
- Ingeniería de datos de Microsoft Azure
- Ingeniería de datos de AWS (pronto)
Finalmente te dejo algunas lecturas y videos que ofrecen consejos y experiencias relacionadas con la búsqueda laboral en el ámbito de sistemas. Más adelante, se agregarán desafíos técnicos y otros recursos vinculados al tema.
- Video: ¿Cómo obtener tu primer empleo en ingeniería de datos? de Spark México
- Videos: Consejos Laborales para el mundo IT de TodoCode
- Videos: Esenciales para comenzar en el mundo de los sistemas de Maxi Programa
- Hilo: Consejos para completar el perfil de LinkedIn de @natayadev
- Hilo: Consejos para conseguir un trabajo remoto en IT de @natayadev
- Hilo: Cómo crear un CV ordenado y legible de @iamdoomling
- Hilo: Te dejo estos tips para sobrevivir entrevistas con recursos humanos de @iamdoomling
- Video: Programar en empresas, startups o freelance ¿Qué es mejor? de @iamdoomling
- Video: Terminé el bootcamp de programación ¿Y ahora qué? de @iamdoomling
- Video: Trabajar como contractor desde Argentina de @iamdoomling
- Podcast: DevRock de Jonatan Ariste
- (2023) Repositorio: Desafíos de código de la comunidad de MoureDev
- (2024) Repositorio: Roadmap retos de programación de la comunidad de MoureDev
En proceso 😊
Si te resultó útil este repositorio, regalame una estrella ⭐
For Tasks:
Click tags to check more tools for each tasksFor Jobs:
Alternative AI tools for dataengineering-roadmap
Similar Open Source Tools

dataengineering-roadmap
A repository providing basic concepts, technical challenges, and resources on data engineering in Spanish. It is a curated list of free, Spanish-language materials found on the internet to facilitate the study of data engineering enthusiasts. The repository covers programming fundamentals, programming languages like Python, version control with Git, database fundamentals, SQL, design concepts, Big Data, analytics, cloud computing, data processing, and job search tips in the IT field.

Ai-Hoshino
Ai Hoshino - MD is a WhatsApp bot tool with features like voice and text interaction, group configuration, anti-delete, anti-link, personalized welcome messages, chatbot functionality, sticker creation, sub-bot integration, RPG game, YouTube music and video downloads, and more. The tool is actively maintained by Starlights Team and offers a range of functionalities for WhatsApp users.

LLMs-Zero-to-Hero
LLMs-Zero-to-Hero is a repository dedicated to training large language models (LLMs) from scratch, covering topics such as dense models, MOE models, pre-training, supervised fine-tuning, direct preference optimization, reinforcement learning from human feedback, and deploying large models. The repository provides detailed learning notes for different chapters, code implementations, and resources for training and deploying LLMs. It aims to guide users from being beginners to proficient in building and deploying large language models.

fairseq
Fairseq is a sequence modeling toolkit that enables researchers and developers to train custom models for translation, summarization, language modeling, and other text generation tasks. It provides reference implementations of various sequence modeling papers covering CNN, LSTM networks, Transformer networks, LightConv, DynamicConv models, Non-autoregressive Transformers, Finetuning, and more. The toolkit supports multi-GPU training, fast generation on CPU and GPU, mixed precision training, extensibility, flexible configuration based on Hydra, and full parameter and optimizer state sharding. Pre-trained models are available for translation and language modeling with a torch.hub interface. Fairseq also offers pre-trained models and examples for tasks like XLS-R, cross-lingual retrieval, wav2vec 2.0, unsupervised quality estimation, and more.

Plug-play-modules
Plug-play-modules is a comprehensive collection of plug-and-play modules for AI, deep learning, and computer vision applications. It includes various convolution variants, latest attention mechanisms, feature fusion modules, up-sampling/down-sampling modules, suitable for tasks like image classification, object detection, instance segmentation, semantic segmentation, single object tracking (SOT), multi-object tracking (MOT), infrared object tracking (RGBT), image de-raining, de-fogging, de-blurring, super-resolution, and more. The modules are designed to enhance model performance and feature extraction capabilities across various tasks.

open-source-ops
This repository contains various tools, scripts, instructions, and guides that can be useful when creating open-source projects. All materials are available under the BSD-3 license.

wenxin-starter
WenXin-Starter is a spring-boot-starter for Baidu's "Wenxin Qianfan WENXINWORKSHOP" large model, which can help you quickly access Baidu's AI capabilities. It fully integrates the official API documentation of Wenxin Qianfan. Supports text-to-image generation, built-in dialogue memory, and supports streaming return of dialogue. Supports QPS control of a single model and supports queuing mechanism. Plugins will be added soon.

awesome-object-detection-datasets
This repository is a curated list of awesome public object detection and recognition datasets. It includes a wide range of datasets related to object detection and recognition tasks, such as general detection and recognition datasets, autonomous driving datasets, adverse weather datasets, person detection datasets, anti-UAV datasets, optical aerial imagery datasets, low-light image datasets, infrared image datasets, SAR image datasets, multispectral image datasets, 3D object detection datasets, vehicle-to-everything field datasets, super-resolution field datasets, and face detection and recognition datasets. The repository also provides information on tools for data annotation, data augmentation, and data management related to object detection tasks.

awesome-llm-unlearning
This repository tracks the latest research on machine unlearning in large language models (LLMs). It offers a comprehensive list of papers, datasets, and resources relevant to the topic.

LLaMA-Factory
LLaMA Factory is a unified framework for fine-tuning 100+ large language models (LLMs) with various methods, including pre-training, supervised fine-tuning, reward modeling, PPO, DPO and ORPO. It features integrated algorithms like GaLore, BAdam, DoRA, LongLoRA, LLaMA Pro, LoRA+, LoftQ and Agent tuning, as well as practical tricks like FlashAttention-2, Unsloth, RoPE scaling, NEFTune and rsLoRA. LLaMA Factory provides experiment monitors like LlamaBoard, TensorBoard, Wandb, MLflow, etc., and supports faster inference with OpenAI-style API, Gradio UI and CLI with vLLM worker. Compared to ChatGLM's P-Tuning, LLaMA Factory's LoRA tuning offers up to 3.7 times faster training speed with a better Rouge score on the advertising text generation task. By leveraging 4-bit quantization technique, LLaMA Factory's QLoRA further improves the efficiency regarding the GPU memory.

fastapi-admin
智元 Fast API is a one-stop API management system that unifies various LLM APIs in terms of format, standards, and management to achieve the ultimate in functionality, performance, and user experience. It includes features such as model management with intelligent and regex matching, backup model functionality, key management, proxy management, company management, user management, and chat management for both admin and user ends. The project supports cluster deployment, multi-site deployment, and cross-region deployment. It also provides a public API site for registration with a contact to the author for a 10 million quota. The tool offers a comprehensive dashboard, model management, application management, key management, and chat management functionalities for users.

jvm-openai
jvm-openai is a minimalistic unofficial OpenAI API client for the JVM, written in Java. It serves as a Java client for OpenAI API with a focus on simplicity and minimal dependencies. The tool provides support for various OpenAI APIs and endpoints, including Audio, Chat, Embeddings, Fine-tuning, Batch, Files, Uploads, Images, Models, Moderations, Assistants, Threads, Messages, Runs, Run Steps, Vector Stores, Vector Store Files, Vector Store File Batches, Invites, Users, Projects, Project Users, Project Service Accounts, Project API Keys, and Audit Logs. Users can easily integrate this tool into their Java projects to interact with OpenAI services efficiently.

langchat
LangChat is an enterprise AIGC project solution in the Java ecosystem. It integrates AIGC large model functionality on top of the RBAC permission system to help enterprises quickly customize AI knowledge bases and enterprise AI robots. It supports integration with various large models such as OpenAI, Gemini, Ollama, Azure, Zhifu, Alibaba Tongyi, Baidu Qianfan, etc. The project is developed solely by TyCoding and is continuously evolving. It features multi-modality, dynamic configuration, knowledge base support, advanced RAG capabilities, function call customization, multi-channel deployment, workflows visualization, AIGC client application, and more.

PyTorch-Tutorial-2nd
The second edition of "PyTorch Practical Tutorial" was completed after 5 years, 4 years, and 2 years. On the basis of the essence of the first edition, rich and detailed deep learning application cases and reasoning deployment frameworks have been added, so that this book can more systematically cover the knowledge involved in deep learning engineers. As the development of artificial intelligence technology continues to emerge, the second edition of "PyTorch Practical Tutorial" is not the end, but the beginning, opening up new technologies, new fields, and new chapters. I hope to continue learning and making progress in artificial intelligence technology with you in the future.
For similar tasks

Azure-Analytics-and-AI-Engagement
The Azure-Analytics-and-AI-Engagement repository provides packaged Industry Scenario DREAM Demos with ARM templates (Containing a demo web application, Power BI reports, Synapse resources, AML Notebooks etc.) that can be deployed in a customer’s subscription using the CAPE tool within a matter of few hours. Partners can also deploy DREAM Demos in their own subscriptions using DPoC.

sorrentum
Sorrentum is an open-source project that aims to combine open-source development, startups, and brilliant students to build machine learning, AI, and Web3 / DeFi protocols geared towards finance and economics. The project provides opportunities for internships, research assistantships, and development grants, as well as the chance to work on cutting-edge problems, learn about startups, write academic papers, and get internships and full-time positions at companies working on Sorrentum applications.

tidb
TiDB is an open-source distributed SQL database that supports Hybrid Transactional and Analytical Processing (HTAP) workloads. It is MySQL compatible and features horizontal scalability, strong consistency, and high availability.

zep-python
Zep is an open-source platform for building and deploying large language model (LLM) applications. It provides a suite of tools and services that make it easy to integrate LLMs into your applications, including chat history memory, embedding, vector search, and data enrichment. Zep is designed to be scalable, reliable, and easy to use, making it a great choice for developers who want to build LLM-powered applications quickly and easily.

telemetry-airflow
This repository codifies the Airflow cluster that is deployed at workflow.telemetry.mozilla.org (behind SSO) and commonly referred to as "WTMO" or simply "Airflow". Some links relevant to users and developers of WTMO: * The `dags` directory in this repository contains some custom DAG definitions * Many of the DAGs registered with WTMO don't live in this repository, but are instead generated from ETL task definitions in bigquery-etl * The Data SRE team maintains a WTMO Developer Guide (behind SSO)

mojo
Mojo is a new programming language that bridges the gap between research and production by combining Python syntax and ecosystem with systems programming and metaprogramming features. Mojo is still young, but it is designed to become a superset of Python over time.

pandas-ai
PandasAI is a Python library that makes it easy to ask questions to your data in natural language. It helps you to explore, clean, and analyze your data using generative AI.

databend
Databend is an open-source cloud data warehouse that serves as a cost-effective alternative to Snowflake. With its focus on fast query execution and data ingestion, it's designed for complex analysis of the world's largest datasets.
For similar jobs

AirGo
AirGo is a front and rear end separation, multi user, multi protocol proxy service management system, simple and easy to use. It supports vless, vmess, shadowsocks, and hysteria2.

mosec
Mosec is a high-performance and flexible model serving framework for building ML model-enabled backend and microservices. It bridges the gap between any machine learning models you just trained and the efficient online service API. * **Highly performant** : web layer and task coordination built with Rust 🦀, which offers blazing speed in addition to efficient CPU utilization powered by async I/O * **Ease of use** : user interface purely in Python 🐍, by which users can serve their models in an ML framework-agnostic manner using the same code as they do for offline testing * **Dynamic batching** : aggregate requests from different users for batched inference and distribute results back * **Pipelined stages** : spawn multiple processes for pipelined stages to handle CPU/GPU/IO mixed workloads * **Cloud friendly** : designed to run in the cloud, with the model warmup, graceful shutdown, and Prometheus monitoring metrics, easily managed by Kubernetes or any container orchestration systems * **Do one thing well** : focus on the online serving part, users can pay attention to the model optimization and business logic

llm-code-interpreter
The 'llm-code-interpreter' repository is a deprecated plugin that provides a code interpreter on steroids for ChatGPT by E2B. It gives ChatGPT access to a sandboxed cloud environment with capabilities like running any code, accessing Linux OS, installing programs, using filesystem, running processes, and accessing the internet. The plugin exposes commands to run shell commands, read files, and write files, enabling various possibilities such as running different languages, installing programs, starting servers, deploying websites, and more. It is powered by the E2B API and is designed for agents to freely experiment within a sandboxed environment.

pezzo
Pezzo is a fully cloud-native and open-source LLMOps platform that allows users to observe and monitor AI operations, troubleshoot issues, save costs and latency, collaborate, manage prompts, and deliver AI changes instantly. It supports various clients for prompt management, observability, and caching. Users can run the full Pezzo stack locally using Docker Compose, with prerequisites including Node.js 18+, Docker, and a GraphQL Language Feature Support VSCode Extension. Contributions are welcome, and the source code is available under the Apache 2.0 License.

learn-generative-ai
Learn Cloud Applied Generative AI Engineering (GenEng) is a course focusing on the application of generative AI technologies in various industries. The course covers topics such as the economic impact of generative AI, the role of developers in adopting and integrating generative AI technologies, and the future trends in generative AI. Students will learn about tools like OpenAI API, LangChain, and Pinecone, and how to build and deploy Large Language Models (LLMs) for different applications. The course also explores the convergence of generative AI with Web 3.0 and its potential implications for decentralized intelligence.

gcloud-aio
This repository contains shared codebase for two projects: gcloud-aio and gcloud-rest. gcloud-aio is built for Python 3's asyncio, while gcloud-rest is a threadsafe requests-based implementation. It provides clients for Google Cloud services like Auth, BigQuery, Datastore, KMS, PubSub, Storage, and Task Queue. Users can install the library using pip and refer to the documentation for usage details. Developers can contribute to the project by following the contribution guide.

fluid
Fluid is an open source Kubernetes-native Distributed Dataset Orchestrator and Accelerator for data-intensive applications, such as big data and AI applications. It implements dataset abstraction, scalable cache runtime, automated data operations, elasticity and scheduling, and is runtime platform agnostic. Key concepts include Dataset and Runtime. Prerequisites include Kubernetes version > 1.16, Golang 1.18+, and Helm 3. The tool offers features like accelerating remote file accessing, machine learning, accelerating PVC, preloading dataset, and on-the-fly dataset cache scaling. Contributions are welcomed, and the project is under the Apache 2.0 license with a vendor-neutral approach.

aiges
AIGES is a core component of the Athena Serving Framework, designed as a universal encapsulation tool for AI developers to deploy AI algorithm models and engines quickly. By integrating AIGES, you can deploy AI algorithm models and engines rapidly and host them on the Athena Serving Framework, utilizing supporting auxiliary systems for networking, distribution strategies, data processing, etc. The Athena Serving Framework aims to accelerate the cloud service of AI algorithm models and engines, providing multiple guarantees for cloud service stability through cloud-native architecture. You can efficiently and securely deploy, upgrade, scale, operate, and monitor models and engines without focusing on underlying infrastructure and service-related development, governance, and operations.