As we shared in our bot trading analysis report, the Klaytn mainnet has recently seen a sudden and sustained increase in trading. This caused great inconvenience to Klaytn users by congesting the network, tricking them into thinking the network was down or not working properly, when in fact the network was producing blocks normally but the high transaction load was slowing down processing times. In this article, we would like to share the progress of our technological efforts to overcome this problem.
The Spam Throttler was created in response to bot transactions towards the end of 2021. These bot transactions were found to be generated by a few entities, which almost took over the entire network for a few hours. While they used various sender addresses, the number of recipient addresses was limited. Over 99% of transactions were intentionally manipulated to be reversed. The Spam Throttler limits the rate at which recurring canceled transactions are propagated, giving more opportunities for normal transactions to be included in a block.
Instead of “blacklisting”, in which transactions of accounts and addresses are blocked from using the network with this mode of centralized processing, we opted for a method by which a PN can analyze the most recent blocks via an algorithm. We wanted network resources to be decentralized and evenly distributed to a large number of users, instead of being concentrated on a few creating another layer of centralization. The criteria for spam transactions has also been set to only cover substantial amounts of canceled transactions.
Such a solution, which consists in limiting the performance of the system, carries the risk of being exploited by malicious actors. To prevent such a malicious user from using the Spam Throttler to cripple the network, we have also implemented the following measures which will limit any unintended side effects of the Spam Throttler.
- The Spam Throttler only limits the speed of propagation during periods of transaction overload on the Klaytn network
- A certain level of TPS is guaranteed for Throttle Lists (Current: 100 TPS)
- Non-faulty services can be whitelisted in case of emergency
The Spam Throttler was enforced on the mainnet in early January 2022, which immediately led to a dramatic drop in canceled transactions. It was also able to regulate excessive reverse transactions, decongesting the network by reducing these spammy reverse transactions. However, while this feature was successful in dealing with canceled transaction spam, ultimately it was not enough on its own to completely eliminate bot-generated spam transactions. Since the introduction of the Spam Throttler feature, bots have found a way around canceled transactions by adjusting their logic to gracefully return instead of outright return. In short, the Spam Throttler do reduce the number of canceled transactions, but did not reduce the total number of burst transactions.
Apart from the Spam Throttler, the Klaytn team is also testing various technological solutions to alleviate the problem of transaction overload. To do this, we have categorized the inconveniences of users into three groups in order to implement technological solutions adapted to each one.
- Transaction deployment error or failures when using endpoint nodes
This problem occurs when transactions suddenly increase beyond what an EN can accept. ENs are the primary transaction submission and gossip processing nodes within the Klaytn network. If you’ve ever seen the “Txpool is full” error message on Kaikas, a crowded transaction pool on the EN was the cause. In order to mitigate this issue, the Klaytn team is working on a solution to limit the number of transactions received externally when each node stores transactions. In early January 2022, many more experimental features were applied to some ENs, including the Kaikas EN. The number of users encountering “Txpool is full” on Kaikas EN has decreased significantly. Through various experiments, this feature will evolve so that an appropriate number of external transactions can be set manually per EN by the trader.
- Network monopoly of certain users/services
This problem is what the Spam Throttler is supposed to help alleviate, and is also the most technologically complex to solve. In the long term, we are working on ways to increase network throughput in terms of TPS and scaling solutions. We also find ways to advocate and enforce fair use of network resources. Other solutions currently being explored include first-come, first-served block propagation and inclusion, and fair transaction management of multiple users per node. Policy-level adjustments such as transaction fee adjustments are also considered.
- Inability to prioritize transactions during network overload
Klaytn’s policy of maintaining a fixed gas price, and in turn deterministic gas fees, provides many benefits to developers. However, a major drawback is that transactions cannot be prioritized during network congestion. The Klaytn team is aware of this issue and is considering an adjustment to the current pricing model. Possible options include a more flexible fee model, a differentiated fee model, or a fee consumption model. Since this is a change that can have a significant impact on the ecosystem, we will make sure to base any decision backed by sufficient preparations, research, real-world data and communication.
The Klaytn team also intends to work with our ecosystem to address these issues. We are aware that these issues have several additional effects such as freezing of transactions on exchanges and bot transactions during NFT/DeFi project launches, which could cause great inconvenience to Klaytn users. Since these are issues that cannot be considered only in the context of the single Klaytn protocol layer, but affect the ecosystem as a whole, we will work together with the support and assistance of our ecosystem and members of the community. The development of the Klaytn ecosystem is truly open-source and as such we welcome all contributions to the Klaytn ecosystem and communities via GitHub, Discord or our developer forum.