The  feature stores evaluated were chosen because they were (1) reproducible - you can create an account and re-run the code to reproduce the results, (2) they have a ready-to-use feature store (not a virtual feature store). We gladly receive contributions for new feature stores, DeWitt Clause permitting. See how to contribute, below, for how to add a new feature store.

Read Pandas DataFrame Benchmarks 🔗

Feature Store Online Read Latency 🔗

------

Feature Freshness Benchmark 🔗

‍

These benchmarks were conducted in collaboration with KTH University, Hopsworks, and the Karolinska Institute.

The benchmark results presented here should follow these database benchmarking principles:

• Reproducibility - you should be able to easily setup the feature store and re-run the source code provided,
• Fairness - there should be no cherry-picking of results, hidden configuration parameters, unrealistic workload tuning,
• Realistic Workloads - the workloads benchmarked should be typical for the use of feature stores in the wild.

Although new benchmarks for AI systems have recently appeared (such as TPCx-AI), these cover a very wide array of use cases, including video and images. In contrast, feature stores are designed primarily to manage structured data that comes from databases, data warehouses, and files.

In this context, the feature store community developed a first set of benchmarks for common usage patterns of feature stores. So far, two benchmarks have been published:

• Offline API Benchmark:
  Measures the throughput of a feature store for the creation of training data as Pandas DataFrames or files.
• Online API Benchmark:
  Measures the latency of online feature serving to AI-enabled applications.

‍

Find in-Depth Benchmarks at the following links:

Contribute

Feel free to create a PR to add a new feature store or benchmark. Be sure to include all the hardware setup and software version numbers, that should be as close as possible to existing benchmarks to ensure apple-to-apple comparisons. For virtual feature stores, include the automated complete setup of the feature store plus online and offline stores.