Real-Time Fraud Detection Pipeline

Streaming inference under severe class imbalance

Status: Academic prototype with a simulated streaming benchmark
Role: Named project co-author; individual implementation ownership is not documented in the public repository.
Domain: Streaming Data / Machine Learning
Source: Repository (opens in a new tab)

Figure 1Shows the Kafka to Spark to inference path together with the benchmark's own measured values: 3,285 events per second, a 26.53 ms median batch, 99.2% AUC retained on a reduced feature set, 284K+ transactions, and 577:1 class imbalance. Every figure comes from a simulated local streaming benchmark, not production traffic.

§1Brief

The problem this had to solve.

Study whether an aggressively reduced fraud feature set can preserve ranking quality while lowering inference cost under severe class imbalance.

The project processes a dataset of more than 284,000 transactions with a 577:1 class imbalance through Kafka, Spark Structured Streaming, feature processing, and Spark MLlib inference.

Its benchmarks compare multiple model and feature-selection combinations. A three-feature configuration retained 99.2% of the full-feature AUC-ROC while reducing inference latency by approximately 30%, alongside measured throughput of 3,285 transactions per second and 26.53 ms median batch latency.

Academic collaboration with Aditya Ravi and Atharva Indulkar; the public repository is owned by Aditya Ravi and does not imply sole ownership by Aayush.

Constraints

The experiment uses the public ULB/Kaggle European cardholder dataset: 284,807 historical transactions and 492 fraud cases.
Spark runs in local[*] mode and Kafka traffic is simulated by replaying held-out transactions in 100-record micro-batches.
SMOTE is applied only to the training split; the held-out test set contains 42,722 transactions and 74 fraud cases.
Reported throughput and latency describe one local experimental setup, not production payment infrastructure.

§2Architecture

How the system is put together.

Transaction stream

284K+ transaction project dataset

source

Kafka

Carries the incoming event stream

process

Spark Structured Streaming

Processes transactions in streaming batches

process

Feature pipeline

Selects 3 features from the 30-feature baseline

process

Model inference

Scores transactions using Spark MLlib

decision

Fraud score

Emits a scored or flagged transaction state

output

§3Decisions

What was chosen, and what it cost.

Optimize a compact feature subset

Choice: Use a Quantum-Inspired Evolutionary Algorithm with a Random Forest proxy and a parsimony penalty to select three of 30 features.
Rationale: The experiment directly tests the quality-versus-inference-cost tradeoff rather than assuming every available feature is required.
Cost: The three-feature Random Forest produces far more false positives than the 30-feature baseline: 466 versus 27 on the held-out test set.

Compare search and model combinations

Choice: Benchmark all-features, PCA, mutual information, RFE, and QIEA across Logistic Regression, Random Forest, and Gradient Boosted Trees.
Rationale: Fifteen combinations expose whether an apparent gain belongs to feature selection, classifier choice, or both.

Keep streaming results separate from model latency

Choice: Report per-transaction classifier latency alongside end-to-end simulated micro-batch throughput and latency.
Rationale: These measurements describe different boundaries and should not be conflated.

§4Validation

How it was checked.

A stratified 70/15/15 split and training-only SMOTE were used; the repository reports all 15 feature-selection/classifier results on the held-out test split.
QIEA + Random Forest achieved AUC-ROC 0.956 versus 0.964 for all-features + Random Forest, retaining 99.2% of that AUC-ROC value with three features.
The simulated streaming replay processed 42,700 held-out transactions in about 13 seconds: 3,285 transactions/s, 26.53 ms median batch latency, and 49.10 ms p95.

Streaming latency chart from the collaborative real-time fraud-detection benchmark. — **Figure 2 · Repository artifact**Latency output from the repository's simulated local streaming replay; it is experimental evidence, not a production payment-system benchmark. Inspect the source (opens in a new tab)

§5Evidence

What was measured.

Each figure states the conditions it was measured under. None of them describe production traffic.

3,285/s

transactions processed

Verified streaming throughput from the project benchmark.

26.53 ms

median batch latency

Verified median latency from the streaming benchmark.

99.2%

full-feature AUC-ROC retained

Three selected features compared with the 30-feature baseline.

~30%

lower inference latency

Measured after reducing the feature set from 30 to 3.

§6Limits

Where this stops being true.

This is a notebook-scale local experiment, not a deployed fraud-detection service or a live Kafka/Spark cluster benchmark.
AUC-ROC retention masks a material precision tradeoff: QIEA + Random Forest precision is 11.2% versus 69.7% for all-features + Random Forest at the evaluated thresholds.
The dataset contains anonymised historical European card transactions from a 48-hour period, so external validity and concept drift are untested.
The repository names three authors but does not document each person's individual contribution.

§7Technology

What it is built with.

Data & evaluation

Spark MLlib
NumPy

Infrastructure & providers

Apache Kafka
PySpark
Spark Structured Streaming

Explore the repository

Next documentCivicLens