Crypto Currency Software Monitor

My Friends,

I want to share some of my work on Crypto, and Software tracking changes in the Market. I am using AI as Prediction Engines, Data Engines, to download data and so on.

Disclaimer:

I am not a Financial Expert, what I share with you is just an experiment, so please take from it what you will! Crypto is risky, you can very easily loose your money! Any of my comments must not be accepted as advice, it is not, and no responsibility will be accepted. You must accept the risks and your own activities are your own responsibility.

Datasets

There are very few Datasets available for Crypto! Its a real shame, because with the right Dataset, one could do a lot more to make accurate predictions! I have put together a Dataset: Here

• Size: 270Mb
• Resolutions: 5, 15, 30, 60, 120, 240, D, 3D, W
• Cutoff Date: Thursday 6th March 2025
• Download: Here

// Get the Data and read in as a Json String:
string jsonString = File.ReadAllText("Path to your data");

// Deserialize the latest price data:
List<LatestPrice> dataset = JsonSerializer.Deserialize<List<LatestPrice>>(jsonString);

    /// <summary>
    /// Represents a cryptocurrency price update with status and candlestick data.
    /// </summary>
    public class LatestPrice
    {
        /// <summary>
        /// The status of the price update (e.g., "ok" indicates successful data).
        /// </summary>
        [JsonPropertyName("s")]
        public string Status { get; set; }

        /// <summary>
        /// The current or closing price of the cryptocurrency.
        /// </summary>
        [JsonPropertyName("price")]
        public double Price { get; set; }

        /// <summary>
        /// The opening price of the cryptocurrency for the period.
        /// </summary>
        [JsonPropertyName("open")]
        public double OpenPrice { get; set; }

        /// <summary>
        /// The lowest price of the cryptocurrency during the period.
        /// </summary>
        [JsonPropertyName("low")]
        public double LowPrice { get; set; }

        /// <summary>
        /// The highest price of the cryptocurrency during the period.
        /// </summary>
        [JsonPropertyName("high")]
        public double HighPrice { get; set; }

        /// <summary>
        /// The highest price of the cryptocurrency during the period.
        /// </summary>
        public int Timestamp { get; internal set; }
    }

Typically, one data record would look like:

  {
    "s": "ok",
    "price": 145567.4031468,
    "open": 145123.87091991,
    "low": 140956.87897272,
    "high": 148669.89984942,
    "Timestamp": 1741254545
  }

I think most would agree, the OHLCV Data, Open, High, Low, Close and Volume Data, is the best data to look at. However, this data does not include the most important data, the Tick Data or the 10 Second Data!

The One Tick Data

The Tick Data or some refer to the 10 Second Data is the data we see in the Charts, every update, where the line on the chart moves with every Update, each Candle displayed is made up of X number of Ticks, which are Market updates for the timeframe given:

Above, the the Chart, we are on the One Hour time frame, which is the 60 minutes shown. If one watches the Chart, we have a Candle for every One Hour shown. However, the Red Line, shown at: 4.068133, is on a different Time Frame, typically, it is 10 Seconds, and moves up or Down according to market value, every 10 Seconds. So, you ask: "Why is this important?"

This data is important, if one wants to make predictions on Market movement, one can evaluate this 10 second data to see big shifts in the market very early on in the game!

For Example:

Each Candle shown is for One Hour, and this is how a Candle works:

Where:

• Bullish Candle = Increase in Market Value.
• Bearish Candle = Decrease in Market Value.

This means, each Candle might be made up of:

[
  {
    "s": "ok",
    "price": 145567.4031468,
    "open": 145123.87091991,
    "low": 140956.87897272,
    "high": 148669.89984942,
    "Timestamp": 1741254545
  },
  {
    "s": "ok",
    "price": 145520.46327524,
    "open": 145123.87091991,
    "low": 140956.87897272,
    "high": 148669.89984942,
    "Timestamp": 1741254614
  },
  {
    "s": "ok",
    "price": 145564.42804449,
    "open": 145123.87091991,
    "low": 140956.87897272,
    "high": 148669.89984942,
    "Timestamp": 1741254654
  },
  {
    "s": "ok",
    "price": 145617.9267137,
    "open": 145123.87091991,
    "low": 140956.87897272,
    "high": 148669.89984942,
    "Timestamp": 1741254690
  },
  {
    "s": "ok",
    "price": 145578.68046191,
    "open": 145123.87091991,
    "low": 140956.87897272,
    "high": 148669.89984942,
    "Timestamp": 1741254727
  },
  {
    "s": "ok",
    "price": 145637.58190353,
    "open": 145123.87091991,
    "low": 140956.87897272,
    "high": 148669.89984942,
    "Timestamp": 1741254761
  },
  {
    "s": "ok",
    "price": 145618.64815215,
    "open": 145123.87091991,
    "low": 140956.87897272,
    "high": 148669.89984942,
    "Timestamp": 1741254797
  },
  {
    "s": "ok",
    "price": 145580.54016992,
    "open": 145123.87091991,
    "low": 140956.87897272,
    "high": 148669.89984942,
    "Timestamp": 1741254830
  },
  {
    "s": "ok",
    "price": 145531.77093057,
    "open": 145123.87091991,
    "low": 140956.87897272,
    "high": 148669.89984942,
    "Timestamp": 1741254865
  },
  {
    "s": "ok",
    "price": 145509.7590418,
    "open": 145123.87091991,
    "low": 140956.87897272,
    "high": 148669.89984942,
    "Timestamp": 1741254900
  },
  {
    "s": "ok",
    "price": 145317.62793503,
    "open": 145123.87091991,
    "low": 140956.87897272,
    "high": 148669.89984942,
    "Timestamp": 1741254937
  },
  {
    "s": "ok",
    "price": 145433.38322856,
    "open": 145123.87091991,
    "low": 140956.87897272,
    "high": 148669.89984942,
    "Timestamp": 1741254975
  },
  {
    "s": "ok",
    "price": 145439.97079788,
    "open": 145123.87091991,
    "low": 140956.87897272,
    "high": 148669.89984942,
    "Timestamp": 1741255008
  },
  {
    "s": "ok",
    "price": 145439.44186895,
    "open": 145123.87091991,
    "low": 140956.87897272,
    "high": 148669.89984942,
    "Timestamp": 1741255052
  },
  {
    "s": "ok",
    "price": 145460.67916664,
    "open": 145123.87091991,
    "low": 140956.87897272,
    "high": 148669.89984942,
    "Timestamp": 1741255089
  },
  {
    "s": "ok",
    "price": 145436.23623911,
    "open": 145123.87091991,
    "low": 140956.87897272,
    "high": 148669.89984942,
    "Timestamp": 1741255121
  },
  {
    "s": "ok",
    "price": 145420.62482179,
    "open": 145123.87091991,
    "low": 140956.87897272,
    "high": 148669.89984942,
    "Timestamp": 1741255155
  },
  {
    "s": "ok",
    "price": 145458.88401393,
    "open": 145123.87091991,
    "low": 140956.87897272,
    "high": 148669.89984942,
    "Timestamp": 1741255186
  }
]

Of course, at Ten Seconds, in Sixty Minutes, we would have: 

1. Convert 60 minutes to seconds:
   • 60 minutes × 60 seconds/minute = 3,600 seconds
2. Divide total seconds by 10-second intervals:
   • 3,600 seconds ÷ 10 seconds/interval = 360 intervals

So, there are 360 ten-second intervals in 60 minutes, thus making up one Candle on the One Hour interval.

The Input Schema Class: AIBreakoutInputSchema.cs

namespace AdvancedCryptoTrader.AI
{



    #region Using Statements:



    using Microsoft.ML.Data;



    #endregion



    /// <summary>
    /// Represents the input schema for breakout prediction, including both historical and short-term features.
    /// </summary>
    public class AIBreakoutInputSchema
    {
        /// <summary>
        /// The label for the breakout prediction (optional for training, "Bullish", "Bearish", or "NoBreakout").
        /// </summary>
        [ColumnName("Label")]
        public string Label { get; set; }

        /// <summary>
        /// Historical timestamp in Unix seconds for the candlestick.
        /// </summary>
        [ColumnName("Timestamp")]
        public long Timestamp { get; set; }

        /// <summary>
        /// Closing price of the historical candlestick.
        /// </summary>
        [ColumnName("ClosePrice")]
        public float ClosePrice { get; set; }

        /// <summary>
        /// Opening price of the historical candlestick.
        /// </summary>
        [ColumnName("OpenPrice")]
        public float OpenPrice { get; set; }

        /// <summary>
        /// Highest price during the historical candlestick period.
        /// </summary>
        [ColumnName("HighPrice")]
        public float HighPrice { get; set; }

        /// <summary>
        /// Lowest price during the historical candlestick period.
        /// </summary>
        [ColumnName("LowPrice")]
        public float LowPrice { get; set; }

        /// <summary>
        /// Trading volume during the historical candlestick period.
        /// </summary>
        [ColumnName("Volume")]
        public float Volume { get; set; }

        /// <summary>
        /// Price change percentage for the historical candlestick ((Close - Open) / Open * 100).
        /// </summary>
        [ColumnName("PriceChangePct")]
        public float PriceChangePct { get; set; }

        /// <summary>
        /// Price range for the historical candlestick (High - Low).
        /// </summary>
        [ColumnName("PriceRange")]
        public float PriceRange { get; set; }

        /// <summary>
        /// Simple Moving Average over a lookback period (e.g., 20 periods).
        /// </summary>
        [ColumnName("Sma")]
        public float Sma { get; set; }

        /// <summary>
        /// Relative Strength Index over a lookback period (e.g., 14 periods).
        /// </summary>
        [ColumnName("Rsi")]
        public float Rsi { get; set; }

        /// <summary>
        /// Average True Range over a lookback period (e.g., 14 periods) for historical volatility.
        /// </summary>
        [ColumnName("Atr")]
        public float Atr { get; set; }

        /// <summary>
        /// Short-term (10-second) price difference magnitude (absolute value of Close price change over 10 seconds).
        /// </summary>
        [ColumnName("ShortTermPriceDiffMagnitude")]
        public float ShortTermPriceDiffMagnitude { get; set; }

        /// <summary>
        /// Short-term (10-second) rolling average of price difference magnitude over a window (e.g., 50 updates, 500 seconds).
        /// </summary>
        [ColumnName("ShortTermPriceDiffRollingAvg")]
        public float ShortTermPriceDiffRollingAvg { get; set; }

        /// <summary>
        /// Short-term (10-second) volatility (standard deviation of price differences over a window).
        /// </summary>
        [ColumnName("ShortTermVolatility")]
        public float ShortTermVolatility { get; set; }

        /// <summary>
        /// Slope of short-term price movement over a window (e.g., last 20 updates, 200 seconds).
        /// </summary>
        [ColumnName("ShortTermPriceSlope")]
        public float ShortTermPriceSlope { get; set; }
    }
}

Here is the C# ML.NET Class: AIBreakoutPredictor.cs

namespace AdvancedCryptoTrader.AI
{



    #region Using Statements:



    using Microsoft.ML;
    using Microsoft.ML.Data;
    using Microsoft.ML.Transforms;

    using System;
    using System.Linq;
    using System.Collections.Generic;



    #endregion


    /// <summary>
    /// AIBreakoutPredictor class that uses a Transformer model to predict market breakout trends (bullish, bearish, or no breakout)
    /// based on historical and short-term (10-second) price data.
    /// </summary>
    public partial class AIBreakoutPredictor
    {



        #region Fields:



        /// <summary>
        /// The MLContext for the Machine Learning Pipeline.
        /// </summary>
        private readonly MLContext _mlContext;



        /// <summary>
        /// The Transformer Model.
        /// </summary>
        private ITransformer _model;



        /// <summary>
        /// Historical Loopback Period.
        /// </summary>
        private const int HistoricalLookbackPeriod = 50; // Number of historical candlesticks (e.g., 1H) for sequence



        /// <summary>
        /// Shortterm Loopback Period.
        /// </summary>
        private const int ShortTermLookbackPeriod = 50;  // Number of 10-second updates for short-term features



        /// <summary>
        /// Breakout threshold.
        /// </summary>
        private const float BreakoutThreshold = 0.05f;   // 5% price change threshold for defining breakouts



        #endregion



        #region Properties:



        #endregion



        /// <summary>
        /// Initializes a new instance of the AIBreakoutPredictor class.
        /// </summary>
        /// <param name="ailogger">Optional logging action for debugging and monitoring. Defaults to Console.WriteLine if null.</param>
        public AIBreakoutPredictor(Action<string> ailogger = null)
        {

            _mlContext = new MLContext(seed: 0);
            LoadModelFromDisk(ailogger ?? (s => Console.WriteLine(s))); // Load existing model or start fresh
        }



        /// <summary>
        /// Trains the Transformer model with historical and short-term price data to predict breakouts.
        /// </summary>
        /// <param name="prices">List of Price objects containing historical candlestick data (e.g., 1H timeframe).</param>
        /// <param name="shortTermPrices">List of Price objects containing short-term (10-second) price updates.</param>
        /// <param name="ailogger">Logging action for training progress and errors.</param>
        /// <summary>
        /// Trains the Transformer model with historical and short-term price data to predict breakouts.
        /// </summary>
        /// <param name="prices">List of Price objects containing historical candlestick data (e.g., 1H timeframe).</param>
        /// <param name="shortTermPrices">List of Price objects containing short-term (10-second) price updates.</param>
        /// <param name="ailogger">Logging action for training progress and errors.</param>
        public void Train(List<Price> prices, List<Price> shortTermPrices, Action<string> ailogger)
        {

            if (prices == null || prices.Count == 0 || shortTermPrices == null || shortTermPrices.Count == 0)
            {
                ailogger("Train: Error - No price data provided for training.");
                throw new ArgumentException("Price data cannot be null or empty.");
            }

            ailogger($"Train: Processing {prices.Count} historical candlesticks and {shortTermPrices.Count} short-term updates.");

            // Calculate features from historical and short-term data
            var historicalFeatures = CalculateHistoricalFeatures(prices).ToList();
            var shortTermFeatures = CalculateShortTermFeatures(shortTermPrices).ToList();

            // Align historical and short-term data by timestamp
            var trainingData = AlignData(historicalFeatures, shortTermFeatures, ailogger).ToList();

            if (trainingData.Count == 0)
            {
                ailogger("Train: Error - No aligned training data generated.");
                throw new InvalidOperationException("No training data available after alignment.");
            }

            ailogger($"Train: Generated {trainingData.Count} training records. Sample - Timestamp={trainingData.First().Timestamp}, ClosePrice={trainingData.First().ClosePrice:F2}, ShortTermPriceDiffMagnitude={trainingData.First().ShortTermPriceDiffMagnitude:F2}");

            // Create ML.NET data view
            var dataView = _mlContext.Data.LoadFromEnumerable(trainingData);
            ailogger("Train: DataView created for Transformer training.");

            // Define the pipeline for non-sequence multiclass classification (no Sequence transform for older ML.NET)
            var pipeline = _mlContext.Transforms
                .Conversion.MapValueToKey("Label", "Label") // Convert string labels to keys for multiclass
                .Append(_mlContext.Transforms.Concatenate("Features", // Concatenate all numeric features
                    "ClosePrice", "OpenPrice", "HighPrice", "LowPrice", "Volume", "PriceChangePct", "PriceRange",
                    "Sma", "Rsi", "Atr", "ShortTermPriceDiffMagnitude", "ShortTermPriceDiffRollingAvg",
                    "ShortTermVolatility", "ShortTermPriceSlope"))
                .Append(_mlContext.Transforms.NormalizeMinMax("Features")) // Normalize features for better training
                .Append(_mlContext.MulticlassClassification.Trainers
                    .LbfgsMaximumEntropy(
                        labelColumnName: "Label",
                        featureColumnName: "Features",
                        l1Regularization: 1f,
                        l2Regularization: 1f,
                        optimizationTolerance: 1E-07f,
                        historySize: 20,
                        enforceNonNegativity: false)) // Use L-BFGS for multiclass without advancedSettings
                .Append(_mlContext.Transforms.Conversion.MapKeyToValue("PredictedLabel", "PredictedLabel")); // Convert keys back to labels

            ailogger("Train: Pipeline defined for L-BFGS multiclass model.");
            _model = pipeline.Fit(dataView);
            ailogger("Train: Model training completed successfully.");

            SaveModelToDisk(ailogger); // Save the trained model
        }



        /// <summary>
        /// Predicts the next breakout trend (Bullish, Bearish, or NoBreakout) for a sequence of prices.
        /// </summary>
        /// <param name="prices">List of historical Price objects (e.g., 1H candlesticks).</param>
        /// <param name="shortTermPrices">List of short-term (10-second) Price updates.</param>
        /// <param name="ailogger">Logging action for prediction details.</param>
        /// <returns>Predicted breakout trend with probability scores and confidence.</returns>
        public AIBreakoutOutputSchema Predict(List<Price> prices, List<Price> shortTermPrices, Action<string> ailogger)
        {

            if (_model == null)
                throw new InvalidOperationException("Model must be trained before predicting.");

            if (prices == null || prices.Count == 0 || shortTermPrices == null || shortTermPrices.Count == 0)
            {
                ailogger("Predict: Error - No price data provided for prediction.");
                throw new ArgumentException("Price data cannot be null or empty.");
            }

            ailogger($"Predict: Processing {prices.Count} historical candlesticks and {shortTermPrices.Count} short-term updates.");

            // Calculate features from historical and short-term data
            var historicalFeatures = CalculateHistoricalFeatures(prices).ToList();
            var shortTermFeatures = CalculateShortTermFeatures(shortTermPrices).ToList();

            // Align data for prediction
            var predictionInput = AlignData(historicalFeatures, shortTermFeatures, ailogger).Last();

            // Create prediction engine for single-row prediction
            var predEngine = _mlContext.Model.CreatePredictionEngine<AIBreakoutInputSchema, AIBreakoutOutputSchema>(_model);
            var prediction = predEngine.Predict(predictionInput);

            // Calculate confidence as the highest probability
            prediction.Confidence = prediction.Probabilities.Max();

            ailogger($"Predict: Output - PredictedLabel={prediction.PredictedLabel}, Probabilities=[{string.Join(", ", prediction.Probabilities.Select(p => p.ToString("F2")))}], Confidence={prediction.Confidence:F2}");
            return prediction;
        }



        /// <summary>
        /// Calculates technical and derived features from historical price data (e.g., 1H candlesticks).
        /// </summary>
        /// <param name="prices">List of Price objects for historical data.</param>
        /// <returns>Enumerable of InputBreakoutSchema with calculated features.</returns>
        private IEnumerable<AIBreakoutInputSchema> CalculateHistoricalFeatures(List<Price> prices)
        {

            var priceList = prices.ToList();
            var features = new List<AIBreakoutInputSchema>();

            for (int i = 0; i < priceList.Count; i++)
            {
                var price = priceList[i];
                var feature = new AIBreakoutInputSchema
                {
                    Timestamp = price.Timestamp,
                    ClosePrice = (float)price.ClosePrice,
                    OpenPrice = (float)price.OpenPrice,
                    HighPrice = (float)price.HighPrice,
                    LowPrice = (float)price.LowPrice,
                    Volume = (float)price.Volume,
                    PriceChangePct = (float)((price.ClosePrice - price.OpenPrice) / price.OpenPrice * 100),
                    PriceRange = (float)(price.HighPrice - price.LowPrice)
                };

                // Calculate SMA (Simple Moving Average) over HistoricalLookbackPeriod
                if (i >= HistoricalLookbackPeriod)
                {
                    var periodPrices = priceList.Skip(i - HistoricalLookbackPeriod).Take(HistoricalLookbackPeriod);
                    feature.Sma = (float)periodPrices.Average(p => p.ClosePrice);
                }

                // Calculate RSI (Relative Strength Index) over HistoricalLookbackPeriod
                if (i >= HistoricalLookbackPeriod)
                {
                    var periodPrices = priceList.Skip(i - HistoricalLookbackPeriod).Take(HistoricalLookbackPeriod);
                    feature.Rsi = (float)CalculateRSI(periodPrices.Select(p => p.ClosePrice));
                }

                // Calculate ATR (Average True Range) over HistoricalLookbackPeriod
                if (i >= HistoricalLookbackPeriod)
                {
                    var periodPrices = priceList.Skip(i - HistoricalLookbackPeriod).Take(HistoricalLookbackPeriod);
                    feature.Atr = (float)CalculateATR(periodPrices);
                }

                features.Add(feature);
            }

            return features;
        }



        /// <summary>
        /// Calculates short-term features from 10-second price updates.
        /// </summary>
        /// <param name="prices">List of Price objects for 10-second updates.</param>
        /// <returns>Enumerable of short-term feature aggregates for alignment with historical data.</returns>
        private IEnumerable<AIBreakoutInputSchema> CalculateShortTermFeatures(List<Price> prices)
        {

            var priceList = prices.ToList();
            var features = new List<AIBreakoutInputSchema>();

            for (int i = 0; i < priceList.Count; i++)
            {
                var price = priceList[i];
                var feature = new AIBreakoutInputSchema
                {
                    Timestamp = price.Timestamp,
                    ClosePrice = (float)price.ClosePrice
                };

                // Calculate 10-second price difference (magnitude)
                if (i > 0)
                {
                    var prevPrice = priceList[i - 1];
                    float priceDiff = (float)(price.ClosePrice - prevPrice.ClosePrice);
                    feature.ShortTermPriceDiffMagnitude = Math.Abs(priceDiff);
                }

                // Calculate rolling average of price difference magnitude over ShortTermLookbackPeriod
                if (i >= ShortTermLookbackPeriod)
                {
                    var periodDiffs = priceList.Skip(i - ShortTermLookbackPeriod).Take(ShortTermLookbackPeriod)
                        .Zip(priceList.Skip(i - ShortTermLookbackPeriod + 1).Take(ShortTermLookbackPeriod),
                            (prev, curr) => Math.Abs((float)(curr.ClosePrice - prev.ClosePrice)));
                    feature.ShortTermPriceDiffRollingAvg = periodDiffs.Average();
                }

                // Calculate short-term volatility (standard deviation of price differences)
                if (i >= ShortTermLookbackPeriod)
                {
                    var periodDiffs = priceList.Skip(i - ShortTermLookbackPeriod).Take(ShortTermLookbackPeriod)
                        .Zip(priceList.Skip(i - ShortTermLookbackPeriod + 1).Take(ShortTermLookbackPeriod),
                            (prev, curr) => (float)(curr.ClosePrice - prev.ClosePrice));
                    feature.ShortTermVolatility = (float)periodDiffs.StandardDeviation();
                }

                // Calculate slope of price movement over ShortTermLookbackPeriod
                if (i >= ShortTermLookbackPeriod)
                {
                    var periodTimestamps = priceList.Skip(i - ShortTermLookbackPeriod).Take(ShortTermLookbackPeriod)
                        .Select(p => (double)p.Timestamp);
                    var periodPrices = priceList.Skip(i - ShortTermLookbackPeriod).Take(ShortTermLookbackPeriod)
                        .Select(p => (float)p.ClosePrice);
                    feature.ShortTermPriceSlope = (float)CalculateSlope(periodTimestamps, periodPrices);
                }

                features.Add(feature);
            }

            return features;
        }



        /// <summary>
        /// Aligns historical and short-term features by timestamp for training or prediction.
        /// </summary>
        /// <param name="historicalFeatures">Historical features from 1H candlesticks.</param>
        /// <param name="shortTermFeatures">Short-term features from 10-second updates.</param>
        /// <param name="ailogger">Logging action for alignment details.</param>
        /// <returns>Enumerable of aligned InputBreakoutSchema records.</returns>
        private IEnumerable<AIBreakoutInputSchema> AlignData(IEnumerable<AIBreakoutInputSchema> historicalFeatures, IEnumerable<AIBreakoutInputSchema> shortTermFeatures, Action<string> ailogger)
        {

            var historicalList = historicalFeatures.ToList();
            var shortTermList = shortTermFeatures.ToList();
            var alignedData = new List<AIBreakoutInputSchema>();

            // Assume short-term data is continuous and recent; align with the most recent historical data
            for (int i = 0; i < historicalList.Count; i++)
            {
                var historical = historicalList[i];
                var shortTermWindow = shortTermList
                    .Where(st => st.Timestamp >= historical.Timestamp && st.Timestamp < (i + 1 < historicalList.Count ? historicalList[i + 1].Timestamp : long.MaxValue))
                    .ToList();

                if (shortTermWindow.Any())
                {
                    var aligned = new AIBreakoutInputSchema
                    {
                        Timestamp = historical.Timestamp,
                        ClosePrice = historical.ClosePrice,
                        OpenPrice = historical.OpenPrice,
                        HighPrice = historical.HighPrice,
                        LowPrice = historical.LowPrice,
                        Volume = historical.Volume,
                        PriceChangePct = historical.PriceChangePct,
                        PriceRange = historical.PriceRange,
                        Sma = historical.Sma,
                        Rsi = historical.Rsi,
                        Atr = historical.Atr,
                        ShortTermPriceDiffMagnitude = shortTermWindow.Last().ShortTermPriceDiffMagnitude,
                        ShortTermPriceDiffRollingAvg = shortTermWindow.Last().ShortTermPriceDiffRollingAvg,
                        ShortTermVolatility = shortTermWindow.Last().ShortTermVolatility,
                        ShortTermPriceSlope = shortTermWindow.Last().ShortTermPriceSlope
                    };

                    // Set label based on historical breakout (look ahead)
                    if (i + 1 < historicalList.Count)
                    {
                        float nextPriceChange = (historicalList[i + 1].ClosePrice - historical.ClosePrice) / historical.ClosePrice;
                        aligned.Label = Math.Abs(nextPriceChange) >= BreakoutThreshold
                            ? (nextPriceChange > 0 ? "Bullish" : "Bearish")
                            : "NoBreakout";
                    }

                    alignedData.Add(aligned);
                }
            }

            ailogger($"AlignData: Aligned {alignedData.Count} records from {historicalList.Count} historical and {shortTermList.Count} short-term features.");
            return alignedData;
        }



        /// <summary>
        /// Calculates the Relative Strength Index (RSI) for a sequence of closing prices.
        /// </summary>
        /// <param name="prices">Sequence of closing prices.</param>
        /// <returns>RSI value (0-100).</returns>
        private double CalculateRSI(IEnumerable<double> prices)
        {

            var priceChanges = prices.Zip(prices.Skip(1), (prev, curr) => curr - prev).ToList();
            var gains = priceChanges.Where(x => x > 0).DefaultIfEmpty(0).Average();
            var losses = Math.Abs(priceChanges.Where(x => x < 0).DefaultIfEmpty(0).Average());

            if (losses == 0) return 100;
            var rs = gains / losses;
            return 100 - (100 / (1 + rs));
        }



        /// <summary>
        /// Calculates the Average True Range (ATR) for a sequence of prices.
        /// </summary>
        /// <param name="prices">Sequence of Price objects.</param>
        /// <returns>ATR value.</returns>
        private double CalculateATR(IEnumerable<Price> prices)
        {

            var trueRanges = prices.Zip(prices.Skip(1), (prev, curr) =>
            {
                double highLow = curr.HighPrice - curr.LowPrice;
                double highClose = Math.Abs(curr.HighPrice - prev.ClosePrice);
                double lowClose = Math.Abs(curr.LowPrice - prev.ClosePrice);
                return Math.Max(highLow, Math.Max(highClose, lowClose));
            }).ToList();

            return trueRanges.DefaultIfEmpty(0).Average();
        }



        /// <summary>
        /// Calculates the slope of a linear regression for timestamps and prices.
        /// </summary>
        /// <param name="timestamps">Sequence of timestamps (as doubles).</param>
        /// <param name="prices">Sequence of prices (as floats).</param>
        /// <returns>Slope of the linear regression line.</returns>
        private double CalculateSlope(IEnumerable<double> timestamps, IEnumerable<float> prices)
        {

            var ts = timestamps.ToList();
            var ps = prices.ToList();
            if (ts.Count != ps.Count || ts.Count < 2) return 0;

            double sumX = ts.Sum();
            double sumY = ps.Sum();
            double sumXY = ts.Zip(ps, (x, y) => x * y).Sum();
            double sumXX = ts.Sum(x => x * x);
            double n = ts.Count;

            double slope = (n * sumXY - sumX * sumY) / (n * sumXX - sumX * sumX);
            return slope;
        }



        /// <summary>
        /// Loads a pre-trained model from disk if it exists, otherwise initializes a new model.
        /// </summary>
        /// <param name="ailogger">Logging action for model loading status.</param>
        private void LoadModelFromDisk(Action<string> ailogger)
        {

            string modelPath = "Models/BreakoutPredictorModel.zip";

            if (System.IO.File.Exists(modelPath))
            {
                try
                {
                    _model = _mlContext.Model.Load(modelPath, out var schema);
                    ailogger($"Loaded BreakoutPredictor model from {modelPath}");
                }
                catch (Exception ex)
                {
                    ailogger($"Model load failed: {ex.Message}. Starting fresh!");
                    _model = null; // Reset to force retraining
                }
            }
            else
            {
                ailogger("No model found at Models/BreakoutPredictorModel.zip—starting with a new one!");
                _model = null; // Fresh start
            }
        }



        /// <summary>
        /// Saves the trained model to disk atomically to prevent corruption.
        /// </summary>
        /// <param name="ailogger">Logging action for model saving status.</param>
        private void SaveModelToDisk(Action<string> ailogger)
        {

            string modelPath = "Models/BreakoutPredictorModel.zip";
            string tempModelPath = modelPath + ".tmp";

            try
            {
                if (_model != null)
                {
                    _mlContext.Model.Save(_model, null, tempModelPath);
                    System.IO.File.Move(tempModelPath, modelPath, overwrite: true);
                    ailogger($"BreakoutPredictor model saved to {modelPath}");
                }
                else
                {
                    ailogger("No model to save—train it first!");
                }
                System.Threading.Thread.Sleep(1000); // Brief pause to ensure file operations settle
            }
            catch (Exception ex)
            {
                ailogger($"Save failed: {ex.Message}. Keeping the model in memory!");
                if (System.IO.File.Exists(tempModelPath))
                    System.IO.File.Delete(tempModelPath); // Clean up temp file on failure
            }
        }
    }



    /// <summary>
    /// BreakoutPredictor class that uses a Transformer model to predict market breakout trends (bullish, bearish, or no breakout)
    /// based on historical and short-term (10-second) price data.
    /// </summary>
    public partial class AIBreakoutPredictor
    {



        /// <summary>
        /// Tests the BreakoutPredictor by training it on historical and short-term price data, then predicting
        /// breakout trends for the current market, evaluating performance metrics like accuracy, win rate, and confidence.
        /// </summary>
        /// <param name="historicalPrices">List of historical Price objects (e.g., 1-hour candlesticks) for training and testing.</param>
        /// <param name="shortTermPrices">List of short-term Price objects (e.g., 10-second updates) for real-time analysis.</param>
        /// <param name="ailogger">Logging action for test progress, results, and errors. Defaults to Console.WriteLine if null.</param>
        /// <param name="windowSize">Number of historical candlesticks to use for each prediction (default: 50, matching HistoricalLookbackPeriod).</param>
        /// <param name="testEpisodes">Number of prediction episodes to simulate (default: 100).</param>
        /// <param name="trainingRatio">Ratio of data to use for training (0-1, default: 0.7 or 70%).</param>
        /// <returns>Tuple containing overall accuracy, win rate, and average confidence for predictions.</returns>
        public (float Accuracy, float WinRate, float AverageConfidence) TestBreakoutPrediction(List<Price> historicalPrices,
                                                                                               List<Price> shortTermPrices,
                                                                                               Action<string> ailogger = null,
                                                                                               int windowSize = 50,
                                                                                               int testEpisodes = 100,
                                                                                               float trainingRatio = 0.7f)
        {

            // Default logger to Console.WriteLine if none provided
            ailogger = ailogger ?? (s => Console.WriteLine(s));

            // Validate input data
            if (historicalPrices == null || historicalPrices.Count < windowSize + 1)
            {
                ailogger($"TestBreakoutPrediction: Error - Insufficient historical price data: {historicalPrices?.Count ?? 0} prices, need at least {windowSize + 1}");
                return (0f, 0f, 0f);
            }

            if (shortTermPrices == null || shortTermPrices.Count == 0)
            {
                ailogger("TestBreakoutPrediction: Error - No short-term price data provided.");
                return (0f, 0f, 0f);
            }

            // Calculate the number of data points for training and testing
            int trainingCount = (int)(historicalPrices.Count * trainingRatio);
            if (trainingCount < windowSize || trainingCount >= historicalPrices.Count)
            {
                ailogger($"TestBreakoutPrediction: Error - Invalid training split: need between {windowSize} and {historicalPrices.Count - 1} prices for training");
                return (0f, 0f, 0f);
            }

            // Split data into training and testing sets
            var trainingHistoricalPrices = historicalPrices.Take(trainingCount).ToList();
            var testingHistoricalPrices = historicalPrices.Skip(trainingCount).ToList();
            int maxEpisodes = Math.Min(testEpisodes, testingHistoricalPrices.Count - windowSize);

            if (maxEpisodes <= 0)
            {
                ailogger("TestBreakoutPrediction: Error - Not enough testing data for specified episodes and window size.");
                return (0f, 0f, 0f);
            }

            ailogger($"TestBreakoutPrediction: Starting test with {maxEpisodes} episodes, {trainingRatio * 100:F1}% training data, window size={windowSize}");

            // Train the model with training data
            try
            {
                Train(trainingHistoricalPrices, shortTermPrices.Take(trainingCount * 6).ToList(), ailogger); // Assuming 10s updates, 360 per hour, 6 per 1H candle
                ailogger("TestBreakoutPrediction: Model trained successfully.");
            }
            catch (Exception ex)
            {
                ailogger($"TestBreakoutPrediction: Training failed - {ex.Message}");
                return (0f, 0f, 0f);
            }

            // Initialize metrics
            int correctPredictions = 0;
            int totalPredictions = 0;
            int bullishWins = 0, bearishWins = 0, totalBullish = 0, totalBearish = 0;
            float totalConfidence = 0f;
            const float confidenceThreshold = 0.65f; // Minimum confidence for a prediction to be considered valid

            // Simulate predictions for testing episodes
            for (int i = 0; i < maxEpisodes; i++)
            {
                // Get a window of historical data for prediction
                var predictionWindow = testingHistoricalPrices.Skip(i).Take(windowSize).ToList();
                var currentShortTermWindow = shortTermPrices
                    .Skip(i * (windowSize * 6)) // Assuming 6 short-term updates per 1H candle (10s * 6 = 60s)
                    .Take(windowSize * 6) // Match the number of short-term updates to historical window
                    .ToList();

                // Predict the next breakout trend
                AIBreakoutOutputSchema prediction;
                try
                {
                    prediction = Predict(predictionWindow, currentShortTermWindow, ailogger);
                }
                catch (Exception ex)
                {
                    ailogger($"TestBreakoutPrediction: Prediction failed for episode {i + 1} - {ex.Message}");
                    continue;
                }

                // Determine the actual outcome (look ahead one step in historical data)
                string actualLabel = "NoBreakout";
                if (i + windowSize < testingHistoricalPrices.Count)
                {
                    float nextPriceChange = (float)((testingHistoricalPrices[i + windowSize].ClosePrice - testingHistoricalPrices[i + windowSize - 1].ClosePrice) /
                                           (testingHistoricalPrices[i + windowSize - 1].ClosePrice));
                    actualLabel = Math.Abs(nextPriceChange) >= BreakoutThreshold
                        ? (nextPriceChange > 0 ? "Bullish" : "Bearish")
                        : "NoBreakout";
                }

                // Calculate confidence (highest probability among classes)
                float confidence = prediction.Probabilities.Max();
                totalConfidence += confidence;

                // Log prediction details
                ailogger($"TestBreakoutPrediction Episode {i + 1}: Predicted={prediction.PredictedLabel}, Actual={actualLabel}, " +
                         $"Confidence={confidence:F2}, Probabilities=[{string.Join(", ", prediction.Probabilities.Select(p => p.ToString("F2")))}]");

                // Evaluate if prediction is valid (above confidence threshold) and correct
                if (confidence >= confidenceThreshold)
                {
                    totalPredictions++;
                    if (prediction.PredictedLabel == actualLabel)
                    {
                        correctPredictions++;
                        if (prediction.PredictedLabel == "Bullish") bullishWins++;
                        if (prediction.PredictedLabel == "Bearish") bearishWins++;
                    }

                    // Track totals for win rate calculation
                    if (prediction.PredictedLabel == "Bullish") totalBullish++;
                    if (prediction.PredictedLabel == "Bearish") totalBearish++;
                }
            }

            // Calculate performance metrics
            float accuracy = totalPredictions > 0 ? (float)correctPredictions / totalPredictions * 100 : 0f;
            float winRate = (totalBullish + totalBearish) > 0
                ? ((float)(bullishWins + bearishWins) / (totalBullish + totalBearish)) * 100
                : 0f;
            float averageConfidence = totalPredictions > 0 ? totalConfidence / totalPredictions : 0f;

            // Log final results
            ailogger($"TestBreakoutPrediction Complete: Accuracy={accuracy:F2}%, Win Rate={winRate:F2}%, " +
                     $"Average Confidence={averageConfidence:F2}, Total Predictions={totalPredictions}, Correct={correctPredictions}");
            ailogger($"Breakdown: Bullish Predictions={totalBullish}, Bullish Wins={bullishWins}, " +
                     $"Bearish Predictions={totalBearish}, Bearish Wins={bearishWins}");

            return (accuracy, winRate, averageConfidence);
        }
    }
}

The Output Schema: AIBreakoutOutputSchema.cs

namespace AdvancedCryptoTrader.AI
{



    #region Using Statements:



    using Microsoft.ML.Data;



    #endregion



    /// <summary>
    /// Represents the output schema for breakout prediction, including the predicted label and confidence scores.
    /// </summary>
    public class AIBreakoutOutputSchema
    {
        /// <summary>
        /// The predicted breakout label ("Bullish", "Bearish", or "NoBreakout").
        /// </summary>
        [ColumnName("PredictedLabel")]
        public string PredictedLabel { get; set; }

        /// <summary>
        /// Probability scores for each possible breakout class (Bullish, Bearish, NoBreakout).
        /// </summary>
        [ColumnName("Score")]
        public float[] Probabilities { get; set; }

        /// <summary>
        /// Confidence score for the predicted label (highest probability among classes).
        /// </summary>
        public float Confidence { get; internal set; }
    }
}

More coming soon...

Best Wishes,

   Chris