#python #analytics #dagster #data_orchestrator #data_pipelines #data_science #etl #scheduler #workflow #workflow_automation

https://github.com/dagster-io/dagster

#python #ai #collaboration #data_pipelines #data_science #data_scientists #data_version_control #dataset #dataset_management #datasets #deep_learning #developer_tools #distributed #hacktoberfest #machine_learning #ml #mlops #pytorch #tensorflow #training

https://github.com/activeloopai/Hub

#python #cloud #data_pipelines #data_science #deployment #docker #ide #jupyter #jupyterlab #machine_learning #notebooks #orchest #pipelines #self_hosted

https://github.com/orchest/orchest

#html #data_pipelines #deep_learning #document_ai #document_image_analysis #document_image_processing #document_parser #document_parsing #docx #donut #information_retrieval #langchain #machine_learning #ml #natural_language_processing #nlp #ocr #pdf #pdf_to_json #pdf_to_text #preprocessing

https://github.com/Unstructured-IO/unstructured

#java #airflow #azkaban #cloud_native #data_pipelines #job_scheduler #orchestration #powerful_data_pipelines #task_scheduler #workflow #workflow_orchestration #workflow_schedule

Apache DolphinScheduler is a powerful tool for managing data workflows. It makes it easy to create and manage complex tasks with a user-friendly interface and low-code options. You can deploy it in several ways, including standalone, cluster, Docker, and Kubernetes, making it flexible for different environments. It's highly reliable, scalable, and performs much faster than other platforms, supporting millions of tasks daily. The tool also offers features like versioning, state control of workflows, multi-tenancy support, and permission control. This helps you manage your data pipelines efficiently and reliably, saving time and effort.

https://github.com/apache/dolphinscheduler

#python #analytics #dagster #data_engineering #data_integration #data_orchestrator #data_pipelines #data_science #etl #metadata #mlops #orchestration #python #scheduler #workflow #workflow_automation

Dagster is a tool that helps you manage and automate your data workflows. You can define your data assets, like tables or machine learning models, using Python functions. Dagster then runs these functions at the right time and keeps your data up-to-date. It offers features like integrated lineage and observability, making it easier to track and manage your data. This tool is useful for every stage of data development, from local testing to production, and it integrates well with other popular data tools. Using Dagster, you can build reusable components, spot data quality issues early, and scale your data pipelines efficiently. This makes your work more productive and helps maintain control over complex data systems.

https://github.com/dagster-io/dagster

#python #agent #agents #ai_search #chatbot #chatgpt #data_pipelines #deep_learning #document_parser #document_understanding #genai #graph #graphrag #llm #nlp #pdf_to_text #preprocessing #rag #retrieval_augmented_generation #table_structure_recognition #text2sql

RAGFlow is an open-source tool that helps businesses answer questions accurately using large language models and deep document understanding. It extracts information from various complex data formats, such as Word documents, Excel files, and web pages, and provides grounded citations to support its answers. You can try a demo online or set it up on your own server using Docker. The setup is relatively straightforward, requiring a few steps like cloning the repository, building the Docker image, and configuring the system settings. RAGFlow offers key features like template-based chunking, reduced hallucinations, and compatibility with multiple data sources, making it a powerful tool for truthful question-answering capabilities. This benefits users by providing reliable and explainable answers, streamlining their workflow, and supporting integration with their business systems.

https://github.com/infiniflow/ragflow

#python #airflow #apache #apache_airflow #automation #dag #data_engineering #data_integration #data_orchestrator #data_pipelines #data_science #elt #etl #machine_learning #mlops #orchestration #python #scheduler #workflow #workflow_engine #workflow_orchestration

Apache Airflow is a tool that helps you manage and automate workflows. You can write your workflows as code, making them easier to maintain, version, test, and collaborate on. Airflow lets you schedule tasks and monitor their progress through a user-friendly interface. It supports dynamic pipeline generation, is highly extensible, and scalable, allowing you to define your own operators and executors.

Using Airflow benefits you by making your workflows more organized, efficient, and reliable. It simplifies the process of managing complex tasks and provides clear visualizations of your workflow's performance, helping you identify and troubleshoot issues quickly. This makes it easier to manage data processing and other automated tasks effectively.

https://github.com/apache/airflow