Storm vs spark streaming

Jiedao jdon 2021-05-04 18:54:07
storm vs spark streaming

Storm Storms and Spark Streaming Spark flow is an open source framework for distributed flow processing . Here we compare them and point out their important differences .

Processing model , Delay

Although both frameworks provide scalability and fault tolerance , The fundamental difference between them is their processing model . and Storm It deals with one event at a time , and Spark Streaming It deals with the flow of events in a window of time . therefore ,Storm Processing an event can be delayed by seconds , and Spark Streaming There's a delay of a few seconds .

Fault tolerance 、 Data assurance

The trade-off in fault-tolerant data assurance is ,Spark Streaming Provides better support for fault-tolerant state computation . stay Storm in , Each individual record must be tracked as it passes through the system , therefore Storm It can guarantee that each record will be processed at least once , But duplicate records are allowed when recovering from errors . This means that the variable state may be incorrectly updated twice .

On the other hand ,Spark Streaming Just trace at the batch level , Therefore, it can effectively ensure that every mini-batch Will be dealt with completely once , Even if a node fails .( actually ,Storm Of Trident library The library also provides a complete process . however , It depends on the transaction update state , It's slow , Usually it has to be implemented by the user .)

In short , If you need a second delay ,Storm It's a good choice , And no data loss . If you need stateful computation , And make sure that every event is handled only once ,Spark Streaming Better .Spark Streaming Programming logic may also be easier , Because it's like a batch program (Hadoop), Especially when you use batches ( Small as it is ) when .

Realization , Programming api

Storm The first time was by Clojure Realization , and Spark Streaming It's using Scala. If you want to look at the code or make your own customization, you need to pay attention to , In order to find out how each system works .Storm By BackType and Twitter Development ; Spark Streaming It was developed at the University of California, Berkeley .

Storm There is one Java API, Other languages are also supported , and Spark Streaming In order to Scala Programming , Yes, of course Java

Spark Streaming A good feature is that it runs on Spark On . So you can write the same code for your batch , There is no need to write separate code to deal with real-time stream data and historical data .

Product support

Storm It's been released for a few years , stay Twitter from 2011 It has been running since , It's also used by other companies , and Spark Streaming It's a new project , It is from 2013 In Sharethrough There's a project running .

Hadoop Support

Storm It's a Hortonworks Hadoop Streaming solutions on data platforms , and Spark Streaming Yes MapR There are also versions of Cloudera Our enterprise data platform ,Databricks Also provide Spark Support .

Cluster management integration

Although both systems run on their own clusters ,Storm It can also run in Mesos, and Spark Streaming Can run in YARN and Mesos On .

本文为[Jiedao jdon]所创,转载请带上原文链接,感谢

  1. Prototype与JQuery对比
  2. Three ways of data transmission between threads in Java and source code display
  3. Jdon causes 99% of CPU and Tomcat dies -- banq replies
  4. docker 原理之 user namespace(下)
  5. Simulating AOP injection with domain events
  6. Spring 3.1 finally adds cache support
  7. Comparison between prototype and jquery
  8. User namespace of docker principle (2)
  9. The way to learn java IO stream and XML
  10. Why does a seemingly correct code cause the Dubbo thread pool to be full
  11. 0 基础 Java 自学之路(2021年最新版)
  12. 0 basic Java self study road (latest version in 2021)
  13. c#—基础拾遗(1) 面向对象
  14. C - basic information (1) object oriented
  15. 技术分享|SQL和 NoSQL数据库之间的差异:MySQL(VS)MongoDB
  16. Technology sharing differences between SQL and NoSQL databases: MySQL (VS) mongodb
  17. PHP教程/面向对象-3~构造函数和析构函数
  18. Spring Cloud的Feign客户端入门
  19. 优化Spring Boot应用的Docker打包速度
  20. PHP tutorial / object oriented - 3 ~ constructor and destructor
  21. Introduction to feign client of spring cloud
  22. Optimizing docker packaging speed of spring boot application
  23. 尚硅谷宋红康Java基础教程2019版
  24. 尚硅谷宋红康Java基础教程2019版
  25. Song Hongkang Java foundation course 2019
  26. Song Hongkang Java foundation course 2019
  27. Redis 6 的多线程
  28. Multithreading of redis 6
  29. SpringCloud-微服务架构编码构建
  30. SpringCloud-微服务架构编码构建
  31. Linux作业控制
  32. Coding construction of springcloud microservice architecture
  33. Java中几个常用并发队列比较 | Baeldung
  34. 为什么Java后端在创业企业中并不流行? -reddit
  35. Coding construction of springcloud microservice architecture
  36. Linux job control
  37. Comparison of several common concurrent queues in Java
  38. Why is java backend not popular in start-ups- reddit
  39. docker 资源限制之 cgroup
  40. 大数据环境: hadoop和jdk部署
  41. CGroup of docker resource limitation
  42. Big data environment: Hadoop and JDK deployment
  43. Spring与Hibernate与JPA的整合(详细配置和源码)
  44. Integration of spring, hibernate and JPA (detailed configuration and source code)
  45. 《精通JPA与Hibernate:Java对象持久化技术详解》的源代码下载
  46. 《精通JPA与Hibernate:Java对象持久化技术详解》的源代码下载
  47. "Proficient in JPA and Hibernate: Java Object Persistence technology" source code download
  48. "Proficient in JPA and Hibernate: Java Object Persistence technology" source code download
  49. Redis、Kafka或RabbitMQ:选择哪个作为微服务消息代理? - otonomo
  50. Java Stream和Collection比较:何时以及如何从Java API返回Stream而不是集合Collection? - TomaszKiełbowicz
  51. 如何在SpringBoot中使用Hibernate @NaturalId?
  52. RPC框架设计----NIO编程通道(Channel)
  53. The use of jquery
  54. Redis, Kafka or rabbitmq: which one to choose as the micro service message broker- otonomo
  55. Comparison of Java stream and collection: when and how to return stream instead of collection from Java API- TomaszKie ł bowicz
  56. How to use hibernate @ naturalid in springboot?
  57. RPC framework design -- NiO programming channel
  58. RPC框架设计----NIO编程Selector (选择器)
  59. RPC framework design -- NiO programming selector
  60. Linux centos重启命令