开颜

Kaiyan is an analytics platform for 7TV emotes on Twitch. It provides various indicators of emote popularity and allows identifying the least used emotes so that streamers can quickly decide which ones to remove when it's time for a cleanup.

Architecture

Kaiyan is built as a collection of small components.

Subscription manager

The subscription manager registers new channels and their 7TV emotes. It communicates to the indexer to describe emotes which are added or removed in real time. It also handles channels that revoke their subscription.

This is the component which handles OAuth2 authentication with Twitch.

Message ingest

Message ingest consumes chat messages, parsing metadata and content. It emplaces an event for each message onto Kafka for the indexers to process.

Indexer

The indexer records emote usage in ClickHouse, on open-source column store database for online analytical processing.

API

The API computes full metrics on demand from what the indexer has recorded.

Load tester

The load tester is an independent component that pushes a high volume of simulated messages into message ingest. This produces internal metrics to determine how message ingest, indexers, Kafka, and ClickHouse can scale relative to each other.