#go #argo #argo_cd #cd #ci_cd #cicd #continuous_delivery #continuous_deployment #devops #docker #gitops #helm #ksonnet #kubernetes #pipeline

https://github.com/argoproj/argo-cd

#java #airbyte #connectors #data #data_analysis #data_ingestion #data_integration #data_science #data_transfers #elt #etl #incremental_updates #integration #open_source #pipeline #pipelines #replications

https://github.com/airbytehq/airbyte

#python #cleandata #data_engineering #data_profilers #data_profiling #data_quality #data_science #data_unit_tests #datacleaner #datacleaning #dataquality #dataunittest #eda #exploratory_analysis #exploratory_data_analysis #exploratorydataanalysis #mlops #pipeline #pipeline_debt #pipeline_testing #pipeline_tests

https://github.com/great-expectations/great_expectations

#python #data_parallelism #deep_learning #distributed_training #hpc #large_scale #model_parallelism #pipeline_parallelism

https://github.com/hpcaitech/ColossalAI

#java #data #data_engineering #data_orchestration #data_orchestrator #data_pipeline #dataflow #elt #etl #kestra #orchestration #pipeline #scheduler #workflow #workflow_automation #workflow_engine

https://github.com/kestra-io/kestra

#python #computer_vision #convolutional_networks #embedding_vectors #embeddings #feature_extraction #feature_vector #image_processing #image_retrieval #machine_learning #milvus #pipeline #towhee #transformer #unstructured_data #video_processing #vision_transformer #vit

https://github.com/towhee-io/towhee

#rust #data #pipeline #sql

https://github.com/PRQL/prql

#jupyter_notebook #ai #aihub #argo #automl #gpt #inference #kubeflow #kubernetes #llmops #mlops #notebook #pipeline #pytorch #spark #vgpu #workflow

https://github.com/tencentmusic/cube-studio

#python #billion_parameters #compression #data_parallelism #deep_learning #gpu #inference #machine_learning #mixture_of_experts #model_parallelism #pipeline_parallelism #pytorch #trillion_parameters #zero

DeepSpeed is a powerful tool for training and using large artificial intelligence models quickly and efficiently. It allows you to train models with billions or even trillions of parameters, which is much faster and cheaper than other methods. With DeepSpeed, you can achieve significant speedups, reduce costs, and improve the performance of your models. For example, it can train ChatGPT-like models 15 times faster than current state-of-the-art systems. This makes it easier to work with large language models without needing massive resources, making AI more accessible and efficient for everyone.

https://github.com/microsoft/DeepSpeed

#cplusplus #android #audio_processing #c_plus_plus #calculator #computer_vision #deep_learning #framework #graph_based #graph_framework #inference #machine_learning #mediapipe #mobile_development #perception #pipeline_framework #stream_processing #video_processing

MediaPipe is a tool that helps you add smart machine learning features to your apps and devices. It works on mobile, web, desktop, and other devices. You can use pre-made solutions for tasks like vision, text, and audio processing, or customize the models to fit your needs. MediaPipe also offers tools like Model Maker and Studio to help you create and test your solutions easily. This makes it easier to delight your customers with innovative features without needing deep machine learning expertise.

https://github.com/google-ai-edge/mediapipe

#typescript #cd #ci #git #gitlab #gitlab_ci #local #pipeline #push #uncomitted #untracked

You can run GitLab CI pipelines locally using `gitlab-ci-local`, which saves you time and effort by not having to push changes to test your `.gitlab-ci.yml` files. This tool allows you to execute pipelines as a shell executor or docker executor, eliminating the need for development-specific scripts. It also offers convenience features like CLI options, environment files, bash aliases, and tab completion. You can list pipeline jobs before running them and customize variables and artifacts easily. This makes your development process more efficient and streamlined.

https://github.com/firecow/gitlab-ci-local

#python #artificial_intelligence #dag #data_science #data_visualization #dataflow #developer_tools #machine_learning #notebooks #pipeline #python #reactive #web_app

Marimo is a powerful tool for Python users that makes working with notebooks much easier and more efficient. Here’s what it offers When you run a cell or interact with UI elements, marimo automatically updates dependent cells, keeping your code and outputs consistent.
- **Interactive** Marimo ensures no hidden state and deterministic execution, making your work reliable.
- **Executable** Notebooks are stored as `.py` files, making version control easy.
- **Modern Editor**: It includes features like GitHub Copilot, AI assistants, and more quality-of-life tools.

Using marimo helps you avoid errors, keeps your code organized, and makes sharing and deploying your work simpler.

https://github.com/marimo-team/marimo

#rust #events #forwarder #logs #metrics #observability #parser #pipeline #router #rust #stream_processing #vector

Vector is a powerful tool for managing your observability data, such as logs and metrics. It allows you to collect, transform, and route your data to any vendor you choose, giving you full control. Vector is reliable, fast (up to 10x faster than alternatives), and secure. It helps reduce costs, improve data quality, and consolidate agents, making your observability processes more efficient and reliable. With a strong community support and extensive documentation, Vector is used by many big companies and is downloaded over 100,000 times daily. This makes it a valuable tool for anyone looking to manage their data effectively.

https://github.com/vectordotdev/vector

#python #automation #data #data_engineering #data_ops #data_science #infrastructure #ml_ops #observability #orchestration #pipeline #prefect #python #workflow #workflow_engine

Prefect is a tool that helps you automate and manage data workflows in Python. It makes it easy to turn your scripts into reliable and flexible workflows that can handle unexpected changes. With Prefect, you can schedule tasks, retry failed operations, and monitor your workflows. You can install it using `pip install -U prefect` and start creating workflows with just a few lines of code. This helps data teams work more efficiently, reduce errors, and save time. You can also use Prefect Cloud for more advanced features and support.

https://github.com/PrefectHQ/prefect

#python #cloud_native #cncf #deep_learning #docker #fastapi #framework #generative_ai #grpc #jaeger #kubernetes #llmops #machine_learning #microservice #mlops #multimodal #neural_search #opentelemetry #orchestration #pipeline #prometheus

Jina-serve is a tool that helps you build and deploy AI services easily. It supports major machine learning frameworks and allows you to scale your services from local development to production quickly. You can use it to create AI services that communicate via gRPC, HTTP, and WebSockets. It has features like built-in Docker integration, one-click cloud deployment, and support for Kubernetes and Docker Compose, making it easy to manage and scale your AI applications. This makes it simpler for you to focus on the core logic of your AI projects without worrying about the technical details of deployment and scaling.

https://github.com/jina-ai/serve