篇名 |
A Memory-Aware Spark Cache Replacement Strategy
|
---|---|
並列篇名 | A Memory-Aware Spark Cache Replacement Strategy |
作者 | Jingyu Zhang、Ruihan Zhang、Osama Alfarraj、Amr Tolba、Gwang-Jun Kim |
英文摘要 | Spark is currently the most widely used distributed computing framework, and its key data abstraction concept, Resilient Distributed Dataset (RDD), brings significant performance improvements in big data computing. In application scenarios, Spark jobs often need to replace RDDs due to insufficient memory. Spark uses the Least Recently Used (LRU) algorithm by default as the cache replacement strategy. This algorithm only considers the most recent use time of RDDs as the replacement basis. This characteristic may cause the RDDs that need to be reused to be evicted when performing cache replacement, resulting in a decrease in Spark performance. In response to the above problems, this paper proposes a memory-aware Spark cache replacement strategy, which comprehensively considers the cluster memory usage, RDD size, RDD dependencies, usage times and other information when performing cache replacement and selects the RDDs to be evicted. Furthermore, this paper designs extensive corresponding experiments to test and analyze the performance of the memory-aware Spark cache replacement strategy. The experimental data show that the proposed strategy can improve the performance by up to 13% compared with the LRU algorithm in different scenarios.
|
起訖頁 | 1185-1190 |
關鍵詞 | Big data、Spark、Cache replacement、Memory resource utilization |
刊名 | 網際網路技術學刊 |
期數 | 202211 (23:6期) |
出版單位 | 台灣學術網路管理委員會 |
DOI |
|
QR Code | |
該期刊 上一篇
| Promoting Students’ Math Learning Performance and Engagement: A Help-seeking Mech+C3:C18anism-based Mobile Gaming Approach |
該期刊 下一篇
| SV-PBFT: An Efficient and Stable Blockchain PBFT Improved Consensus Algorithm for Vehicle-to-Vehicle Energy Transactions |