#go #argo #argo_cd #cd #ci_cd #cicd #continuous_delivery #continuous_deployment #devops #docker #gitops #helm #ksonnet #kubernetes #pipeline

https://github.com/argoproj/argo-cd

#java #airbyte #connectors #data #data_analysis #data_ingestion #data_integration #data_science #data_transfers #elt #etl #incremental_updates #integration #open_source #pipeline #pipelines #replications

https://github.com/airbytehq/airbyte

#python #cleandata #data_engineering #data_profilers #data_profiling #data_quality #data_science #data_unit_tests #datacleaner #datacleaning #dataquality #dataunittest #eda #exploratory_analysis #exploratory_data_analysis #exploratorydataanalysis #mlops #pipeline #pipeline_debt #pipeline_testing #pipeline_tests

https://github.com/great-expectations/great_expectations

#python #data_parallelism #deep_learning #distributed_training #hpc #large_scale #model_parallelism #pipeline_parallelism

https://github.com/hpcaitech/ColossalAI

#java #data #data_engineering #data_orchestration #data_orchestrator #data_pipeline #dataflow #elt #etl #kestra #orchestration #pipeline #scheduler #workflow #workflow_automation #workflow_engine

https://github.com/kestra-io/kestra

#python #computer_vision #convolutional_networks #embedding_vectors #embeddings #feature_extraction #feature_vector #image_processing #image_retrieval #machine_learning #milvus #pipeline #towhee #transformer #unstructured_data #video_processing #vision_transformer #vit

https://github.com/towhee-io/towhee

#rust #data #pipeline #sql

https://github.com/PRQL/prql

#jupyter_notebook #ai #aihub #argo #automl #gpt #inference #kubeflow #kubernetes #llmops #mlops #notebook #pipeline #pytorch #spark #vgpu #workflow

https://github.com/tencentmusic/cube-studio

#python #billion_parameters #compression #data_parallelism #deep_learning #gpu #inference #machine_learning #mixture_of_experts #model_parallelism #pipeline_parallelism #pytorch #trillion_parameters #zero

DeepSpeed is a powerful tool for training and using large artificial intelligence models quickly and efficiently. It allows you to train models with billions or even trillions of parameters, which is much faster and cheaper than other methods. With DeepSpeed, you can achieve significant speedups, reduce costs, and improve the performance of your models. For example, it can train ChatGPT-like models 15 times faster than current state-of-the-art systems. This makes it easier to work with large language models without needing massive resources, making AI more accessible and efficient for everyone.

https://github.com/microsoft/DeepSpeed

#cplusplus #android #audio_processing #c_plus_plus #calculator #computer_vision #deep_learning #framework #graph_based #graph_framework #inference #machine_learning #mediapipe #mobile_development #perception #pipeline_framework #stream_processing #video_processing

MediaPipe is a tool that helps you add smart machine learning features to your apps and devices. It works on mobile, web, desktop, and other devices. You can use pre-made solutions for tasks like vision, text, and audio processing, or customize the models to fit your needs. MediaPipe also offers tools like Model Maker and Studio to help you create and test your solutions easily. This makes it easier to delight your customers with innovative features without needing deep machine learning expertise.

https://github.com/google-ai-edge/mediapipe

#typescript #cd #ci #git #gitlab #gitlab_ci #local #pipeline #push #uncomitted #untracked

You can run GitLab CI pipelines locally using `gitlab-ci-local`, which saves you time and effort by not having to push changes to test your `.gitlab-ci.yml` files. This tool allows you to execute pipelines as a shell executor or docker executor, eliminating the need for development-specific scripts. It also offers convenience features like CLI options, environment files, bash aliases, and tab completion. You can list pipeline jobs before running them and customize variables and artifacts easily. This makes your development process more efficient and streamlined.

https://github.com/firecow/gitlab-ci-local

#python #artificial_intelligence #dag #data_science #data_visualization #dataflow #developer_tools #machine_learning #notebooks #pipeline #python #reactive #web_app

Marimo is a powerful tool for Python users that makes working with notebooks much easier and more efficient. Here’s what it offers When you run a cell or interact with UI elements, marimo automatically updates dependent cells, keeping your code and outputs consistent.
- **Interactive** Marimo ensures no hidden state and deterministic execution, making your work reliable.
- **Executable** Notebooks are stored as `.py` files, making version control easy.
- **Modern Editor**: It includes features like GitHub Copilot, AI assistants, and more quality-of-life tools.

Using marimo helps you avoid errors, keeps your code organized, and makes sharing and deploying your work simpler.

https://github.com/marimo-team/marimo

#rust #events #forwarder #logs #metrics #observability #parser #pipeline #router #rust #stream_processing #vector

Vector is a powerful tool for managing your observability data, such as logs and metrics. It allows you to collect, transform, and route your data to any vendor you choose, giving you full control. Vector is reliable, fast (up to 10x faster than alternatives), and secure. It helps reduce costs, improve data quality, and consolidate agents, making your observability processes more efficient and reliable. With a strong community support and extensive documentation, Vector is used by many big companies and is downloaded over 100,000 times daily. This makes it a valuable tool for anyone looking to manage their data effectively.

https://github.com/vectordotdev/vector