MetaSpore

中文介绍

MetaSpore: One-stop machine learning development platform

MetaSpore is a one-stop end-to-end machine learning development platform that provides a full-cycle framework and development interface for from data preprocessing, model training, offline experiments, online predictions to online experiment traffic bucketization and ab-testing.

MetaSpore is developed and opensourced by DMetaSoul team. You could also join our slack user discussion space.

Core Features

MetaSpore has the following features:

1. One-stop end-to-end development, from offline model training to online prediction and bucketing experiments, with a unified development experience across the entire process;
2. Deep learning training framework, compatible with PyTorch ecology, supports distributed large-scale sparse feature learning;
3. The training framework is connected with PySpark to seamlessly read the training data from the data lake and data warehouse;
4. High-performance online prediction service, supports fast inference for neural network, decision tree, Spark ML, SKLearn and other models; supports heterogeneous hardware inference acceleration;
5. In the offline unified feature extraction framework, the online feature reading logic is automatically generated, and the feature extraction logic is unified cross offline and online;
6. Online algorithm application framework, providing model prediction, experiment bucketing and traffic splitting, dynamic hot loading of parameters and rich debug functions;
7. Rich industry algorithm examples and end-to-end solutions.

Documentation and examples

• Offline Training Getting Started Tutorial

• Online Algorithm Application (Java implementation)

  • Online Model Serving
  • Online Feature Extraction
  • Online Experiment Pipeline

• A MovieLens end-to-end recommender system demo, including

  • Offline models
  • Online algorithm application (Java implementation)

Installation package download

Training package

We provide precompiled offline training wheel package on pypi, install it via pip:

pip install metaspore

The minimum Python version required is 3.8.

After installation, also install pytorch and pyspark (they are not included as depenencies of metaspore wheel so you could choose pyspark and pytorch versions as needed):

pip install pyspark
pip install torch==1.11.0+cpu -f https://download.pytorch.org/whl/cpu/torch_stable.html

Serving package

We provide prebuilt docker images for MetaSpore Serving Service:

CPU only image

docker pull dmetasoul/metaspore-serving-release:cpu-v1.0.1

GPU image

docker pull dmetasoul/metaspore-serving-release:gpu-v1.0.1

See Run Serving Service in Docker for details.

Compile the code

• Offline training framework compilation

Community guidelines

Community guidelines

Feedback

For questions about usage, you can post questions in GitHub Discussion, or through GitHub Issue.

Mail

Email us at [email protected].

Slack

Join our user discussion slack channel: MetaSpore User Discussion

Open source projects

MetaSpore is a completely open source project released under the Apache License 2.0. Participation, feedback, and code contributions are welcome.