Status Updates
Last updated
Last updated
12/9/2024 Incident
On 12/08/2024 @ approx. 5:00pm CST; trading services for a few partner firms went offline - limiting the ability for traders to enter, exit and manage existing simulated / paper trades. This lasted until approximately 5:25pm CST when all services were fully restored.
On 12/09/2024 @ approx. 1:21pm CST; a security group within our Amazon AWS cluster was inadvertently changed, resulting in a connection disruption to our primary VPC group responsible for managing trading services. This lasted until 1:25pm CST (4 minutes) when the change was reverted and system access restored. Intra-day candle building services were affected as a result and will be 're-built' after market close on 12/9. 7/31/2024 Incident On 7/31/2024 @ approx. 6:15pm CST; our market data ingestion services suffered a critical fault that caused the incoming CME 'Best Ask' market data to stop updating and propagating the order book on all incoming updates (millisecond intervals).
This led to a cascade effect that caused the Best Ask to migrate below the Best Bid. In addition, fault tolerant checks that would normally prohibit trading activity in the event of a 'crossed market', failed to trigger & simulated paper trading was allowed to continue under these conditions. At no time was Live market brokerage accounts or trading effected. This allowed traders the ability to open simulated positions at prices well below the Best Ask, and immediately sell those positions well above at the Best Bid, providing unrealistic entries. Conversely, traders who had open positions prior to the event were able to exit those positions at grossly inflated profit levels. This fault state was quickly identified by our team and immediately remedied. Total time of incident was less than 15 minutes. Prop firm / Evaluation Services customers: please await updates from your firm(s) on planned resolutions to trading accounts scheduled for 8/1/2024. Live Brokerage / Retail trading customers: No action needed.
The image above shows normal market spread (difference between Best Ask and Best Bid) up until approx. 16:13PST, at which time the Best Ask went below the Best Bid, causing a crossed market.
