Image Classification with ML.NET in .NET 8

By Khaled Md Tuhidul Hossain · 2024-06-28 · Technology

Machine learning in .NET has come a long way. With ML.NET 3.0 and .NET 8, you can build production-grade image classification models without leaving the .NET ecosystem or relying on Python. ML.NET's image classification API uses transfer learning under the hood, taking a pretrained deep learning model and fine-tuning it on your dataset. This means you can achieve strong results even with relatively small datasets. We will build a full image classification pipeline from data loading to prediction.

What Is ML.NET?

ML.NET is Microsoft's open-source, cross-platform machine learning framework for .NET. It supports a wide range of ML tasks including classification, regression, recommendation, anomaly detection, and image classification. Version 3.0, released alongside .NET 8, brought performance improvements, better TensorFlow integration, and expanded API surface for deep learning scenarios.

Project Setup

Create a new .NET 8 console application and add the required packages:

dotnet new console -n ImageClassifier
cd ImageClassifier
dotnet add package Microsoft.ML --version 3.0.1
dotnet add package Microsoft.ML.Vision --version 3.0.1
dotnet add package SciSharp.TensorFlow.Redist --version 2.16.0

The Microsoft.ML.Vision package provides the image classification trainer, and SciSharp.TensorFlow.Redist provides the TensorFlow runtime that powers the transfer learning.

Preparing the Data

For this example, we will classify images of flowers. Organize your images in a folder structure where each subfolder is a class label:

images/
  daisy/
    img001.jpg
    img002.jpg
  rose/
    img001.jpg
    img002.jpg
  sunflower/
    img001.jpg
    img002.jpg
  tulip/
    img001.jpg
    img002.jpg

Define the data model classes:

using Microsoft.ML.Data;

public class ImageData
{
    [LoadColumn(0)]
    public string ImagePath { get; set; } = string.Empty;

    [LoadColumn(1)]
    public string Label { get; set; } = string.Empty;
}

public class ImagePrediction : ImageData
{
    public float[] Score { get; set; } = Array.Empty<float>();

    public string PredictedLabel { get; set; } = string.Empty;
}

Loading and Splitting the Data

We need to scan the image directory, create the dataset, and split it into training and validation sets:

using Microsoft.ML;

var mlContext = new MLContext(seed: 42);

// Load image paths and labels from directory structure
var imageFiles = Directory.GetFiles("images", "*.jpg", SearchOption.AllDirectories)
    .Select(path => new ImageData
    {
        ImagePath = Path.GetFullPath(path),
        Label = Directory.GetParent(path)!.Name,
    })
    .ToList();

Console.WriteLine(quot;Total images: {imageFiles.Count}");

// Group by label to verify distribution
var labelCounts = imageFiles.GroupBy(x => x.Label)
    .Select(g => quot;{g.Key}: {g.Count()}")
    .ToList();
Console.WriteLine(quot;Classes: {string.Join(", ", labelCounts)}");

// Load into IDataView
var data = mlContext.Data.LoadFromEnumerable(imageFiles);

// Split: 80% train, 20% validation
var splitData = mlContext.Data.TrainTestSplit(data, testFraction: 0.2, seed: 42);
var trainData = splitData.TrainSet;
var testData = splitData.TestSet;

Building the Classification Pipeline

The ML.NET image classification pipeline handles image loading, resizing, pixel extraction, and model training in a single fluent API:

var pipeline = mlContext.Transforms.Conversion
    .MapValueToKey("LabelKey", nameof(ImageData.Label))
    .Append(mlContext.Transforms.LoadRawImageBytes(
        outputColumnName: "Image",
        imageFolder: null,
        inputColumnName: nameof(ImageData.ImagePath)))
    .Append(mlContext.MulticlassClassification.Trainers
        .ImageClassification(new Microsoft.ML.Vision.ImageClassificationTrainer.Options
        {
            FeatureColumnName = "Image",
            LabelColumnName = "LabelKey",
            Arch = Microsoft.ML.Vision.ImageClassificationTrainer.Architecture.ResnetV2101,
            Epoch = 100,
            BatchSize = 16,
            LearningRate = 0.01f,
            MetricsCallback = (metrics) =>
            {
                Console.WriteLine(quot;Phase: {metrics.Train?.Phase}, " +
                    quot;Accuracy: {metrics.Train?.Accuracy:P2}, " +
                    quot;Loss: {metrics.Train?.CrossEntropy:F4}");
            },
            ValidationSet = testData,
        }))
    .Append(mlContext.Transforms.Conversion
        .MapKeyToValue("PredictedLabel", "PredictedLabel"));

The Arch parameter specifies the pretrained model architecture. ML.NET supports several options:

ResnetV2101 - High accuracy, slower training
ResnetV2050 - Good balance of accuracy and speed
MobilenetV2 - Fast inference, suitable for mobile and edge deployment
InceptionV3 - Strong accuracy for general image classification

Training the Model

Training is straightforward. The transfer learning process freezes most of the pretrained layers and only trains the final classification layers on your data:

Console.WriteLine("Training started...");
var stopwatch = System.Diagnostics.Stopwatch.StartNew();

var model = pipeline.Fit(trainData);

stopwatch.Stop();
Console.WriteLine(quot;Training completed in {stopwatch.Elapsed.TotalMinutes:F1} minutes");

Depending on your hardware and dataset size, training can take anywhere from a few minutes to an hour. GPU acceleration is available through the TensorFlow runtime if you have a compatible NVIDIA GPU.

Evaluating the Model

Evaluate the trained model against the test set to understand its performance:

var predictions = model.Transform(testData);
var metrics = mlContext.MulticlassClassification.Evaluate(
    predictions,
    labelColumnName: "LabelKey");

Console.WriteLine(quot;\nEvaluation Metrics:");
Console.WriteLine(quot;  Macro Accuracy:    {metrics.MacroAccuracy:P2}");
Console.WriteLine(quot;  Micro Accuracy:    {metrics.MicroAccuracy:P2}");
Console.WriteLine(quot;  Log Loss:          {metrics.LogLoss:F4}");
Console.WriteLine(quot;  Log Loss Reduction:{metrics.LogLossReduction:F4}");

Console.WriteLine(quot;\nPer-class metrics:");
Console.WriteLine(metrics.ConfusionMatrix.GetFormattedConfusionTable());

With a dataset of a few hundred images per class and ResnetV2101, you can typically expect 85-95% accuracy. The confusion matrix shows which classes the model confuses most, guiding further data collection or augmentation efforts.

Making Predictions

Create a prediction engine and classify new images:

var predictionEngine = mlContext.Model
    .CreatePredictionEngine<ImageData, ImagePrediction>(model);

var testImage = new ImageData
{
    ImagePath = Path.GetFullPath("test_images/unknown_flower.jpg"),
};

var prediction = predictionEngine.Predict(testImage);

Console.WriteLine(quot;\nPrediction: {prediction.PredictedLabel}");
Console.WriteLine("Confidence scores:");
var labels = new[] { "daisy", "rose", "sunflower", "tulip" };
for (int i = 0; i < prediction.Score.Length; i++)
{
    Console.WriteLine(quot;  {labels[i]}: {prediction.Score[i]:P2}");
}

Saving and Loading the Model

Save the trained model so you can use it later without retraining:

// Save
var modelPath = "flower_classifier.zip";
mlContext.Model.Save(model, trainData.Schema, modelPath);
Console.WriteLine(quot;Model saved to {modelPath}");

// Load
var loadedModel = mlContext.Model.Load(modelPath, out var schema);
var loadedPredictionEngine = mlContext.Model
    .CreatePredictionEngine<ImageData, ImagePrediction>(loadedModel);

The saved model file contains the entire pipeline including preprocessing steps, so you do not need to replicate the pipeline when loading.

Integrating with ASP.NET Core

For serving predictions via an API, wrap the prediction engine in a minimal API endpoint:

// In Program.cs of an ASP.NET Core project
var mlContext = new MLContext();
var model = mlContext.Model.Load("flower_classifier.zip", out _);
var predictionEngine = mlContext.Model
    .CreatePredictionEngine<ImageData, ImagePrediction>(model);

app.MapPost("/classify", async (HttpRequest request) =>
{
    var form = await request.ReadFormAsync();
    var file = form.Files.FirstOrDefault();
    if (file is null) return Results.BadRequest("No image provided");

    var tempPath = Path.GetTempFileName();
    using (var stream = File.OpenWrite(tempPath))
    {
        await file.CopyToAsync(stream);
    }

    var prediction = predictionEngine.Predict(new ImageData
    {
        ImagePath = tempPath
    });

    File.Delete(tempPath);

    return Results.Ok(new
    {
        label = prediction.PredictedLabel,
        scores = prediction.Score,
    });
});

Note that PredictionEngine is not thread-safe. In production, use PredictionEnginePool from the Microsoft.Extensions.ML package for concurrent request handling.

Key Considerations

Working with ML.NET for image classification has taught me a few practical lessons:

Transfer learning is the right default. Training a deep learning model from scratch requires millions of images. Transfer learning gets you strong results with hundreds per class.
Architecture choice matters. MobilenetV2 is 10x faster at inference than ResnetV2101 with only a modest accuracy trade-off. Choose based on your deployment constraints.
Data quality beats quantity. A small dataset of well-labeled, representative images outperforms a large dataset with noisy labels.
Monitor the confusion matrix, not just accuracy. Overall accuracy can hide systematic failures on specific classes.

ML.NET brings machine learning to .NET developers without requiring a context switch to Python. For teams already invested in the .NET ecosystem, it is a practical and capable option for production image classification.