Real-Time Game Health Analytics with WebSockets, Python 3 and Redis PubSub

Many factors can impact the success of a Mobile Game; for example the throughput of the CDNs serving game assets, the crash rate or the frame rate per second. Collecting this information at scale poses 2 challenges. On the server side, a lot of storage and processing power are needed to answer arbitrary and always changing questions. On the client side the instrumentation should be lightweight to minimize the observer effect. By using an in-memory PubSub system on the server side, we take the disk storage out of the equation, and get full flexibility by publishing structured JSON documents to specific per event type channels equivalent to different tables in a SQL world. The client-server communication is made through encrypted and persistent WebSocket connections. We wrote our own WebSocket C++ implementation and open-sourced it on GitHub at https://github.com/machinezone/IXWebSocket. Multiple subscribers programs are consuming the events, transforming and ingesting them in charting tools such as Grafana, or Error Logging tools such as Sentry. Our system is able to handle 100 billions events per month.