骑士卡:基于Kafka搭建消息中心,上亿消息推送轻松完成

InfoQ 2021-02-23 13:41:01
kafka 搭建 消息 中心 骑士


{"type":"doc","content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"全球购骑士卡是国内领先的会员制特权电商平台,汇聚国内外“吃喝玩乐买”超 300 项会员专属优惠特权。全球购骑士卡基于移动互联生活方式,打通线上、线下消费场景,汇集时下热门、高频的商品及服务优惠。会员可享全国超万家大型商超购物8折起、全国加油7折起、热门电商平台专属4折起、大牌美食餐饮 5 折起等,满足用户吃、喝、玩、乐、买各场景的消费需求。截至2020年,全球购骑士卡已累计服务用户超 5000 万名。2020年4月,全球购骑士卡完成 A 轮数千万美元融资;同年 5 月,全球购骑士卡完成数千万美元 A+轮融资。"}]},{"type":"heading","attrs":{"align":null,"level":2},"content":[{"type":"text","text":"新的需求"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"全球购骑士特权业务的飞速发展,当前每天平均发送的短信量达到了约 200 万+,需要 PUSH 的推送量达到了约 1 亿+,通过微信推送量达到了5000 万+。因此,如何构造建设一个"},{"type":"text","marks":[{"type":"strong"}],"text":"高性能、高稳定性、可扩展的消息中心"},{"type":"text","text":"迫在眉睫。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"消息中心技术选型主要参考以下因素:"}]},{"type":"bulletedlist","content":[{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong"}],"text":"削峰填谷能力"},{"type":"text","text":":消息中心需要处理各条业务线的通知和营销任务的信息,而这些信息根据转化的需要,很大可能会集中化地在短期内进行推送,所以需要系统有削峰填谷的能力。"}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong"}],"text":"接口通用能力"},{"type":"text","text":":消息中心的接入方不希望被绑定在某个接口上,不需要对该接口进行维护可以供多个业务方进行发送处理。"}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong"}],"text":"灵活类型划分"},{"type":"text","text":":消息中心需要支持灵活的业务分类配置, 因为我们消息中心这里的业务配置非常多,大类就有短信、PUSH、微信推送,短信里又分通知、验证码和营销类别,而 PUSH 又区分 APNS、渠道服务商等第三方通道,以及 Android 厂商通道。"}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong"}],"text":"稳定处理能力"},{"type":"text","text":":所依赖的技术产品运行稳定,因为处于消息中心的通道位置,不能忍受产品本身的稳定性波动带来的业务损失。"}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong"}],"text":"集群扩展能力"},{"type":"text","text":":所依赖的技术产品没有扩容瓶颈,对于我们的业务继续发展有扩展的足够空间,可以快速进行业务扩容诉求。"}]}]}]},{"type":"heading","attrs":{"align":null,"level":2},"content":[{"type":"text","text":"新的解法"}]},{"type":"image","attrs":{"src":"https:\/\/static001.geekbang.org\/infoq\/9d\/9d37a261a2358c64f57c3a6c6f8f60f1.png","alt":"图片","title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"使用消息中间件来做消息中心的通道是显现而见的目标选项,综合对比多种消息的产品,由于骑士卡并没有需要顺序消息、事务消息等高阶功能,而是重点关注以下这些功能点:"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"bulletedlist","content":[{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong"}],"text":"队列的扩展能力"},{"type":"text","text":":在这方面,RabbitMQ 的单 Queue 的处理能力不容易扩展;而 RocketMQ 的 Topic 是有 ConsumerQueue 的参数来进行配置扩容的,在 Broker 的配置文件里指定,但是对 Broker 层面生效的;而 Kafka 的 Partition 可以每个 Topic 拥有不同的取值。这样在分类灵活性方面,Kafka 是最优的选择,RocketMQ 次之。"}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong"}],"text":"通用的接入方式"},{"type":"text","text":":本质上 RabbitMQ、RocketMQ、Kafka 都是私有协议的方式接入,比较云上商业版本的接入方式,对于 Kafka 支持最纯粹友好,可以使用官方的接入方式进行接入。"}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong"}],"text":"消息的吞吐能力"},{"type":"text","text":":在各类消息的对比测试中, 因为 Kafka 本身的处理机制原因,都是由客户端进行拉消息,整个 Broker 的处理方式比别的消息中间件要简洁,而 Kafka 的读写能力\/吞吐量都是最大的。"}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong"}],"text":"集群稳定性能力"},{"type":"text","text":":云上的消息产品都很友好地保持业务的连续性来进行升配操作,并且对于商业版本的 Kafka 做了 Broker 上的优化,存储上的优化,运维上的优化后,不需要担心自建集群出现的不稳定问题,完全满足骑士卡的需求。"}]}]}]},{"type":"heading","attrs":{"align":null,"level":2},"content":[{"type":"text","text":"业务价值"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"使用 Kafka 构建消息中心,对骑士卡来说最重要的是"},{"type":"text","marks":[{"type":"strong"}],"text":"保障了业务的稳健"},{"type":"text","text":"。利用 Kafka 的吞吐能力,自定义的 partition 设定(扩展),通过弹性扩展消费者实例的方式,自消息中心上线以来,一直运行平稳,没有出现过影响业务的故障。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"同时,"},{"type":"text","marks":[{"type":"strong"}],"text":"系统运维起来十分简单"},{"type":"text","text":"。利用云上的Kafka能力,避免了测试期自建集群莫名其妙的 Broker 故障,不需要投入额外的资源来保障消息中间件正常工作。并且可以通过白屏化的升级操作来匹配骑士卡的业务发展,也可以按需要来快速调整实例数。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"值得一提的是,使用云产品 Kafka,无论在生产环境还是本地开发测试环境,都可以直接使用云产品,最大限度减少通用产品依赖,让团队专注于业务的开拓实现,"},{"type":"text","marks":[{"type":"strong"}],"text":"极大的提升了团队工作效率。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"blockquote","content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"“在全球购骑士卡消息中心的搭建过程中,我们使用阿里云的Kafka完成了消息中心高吞吐量,稳定以及可扩展的目标。目前,消息中心作为业务运营推广的基石,发挥着重要作用,对于新业务的接入,通过消息队列的配置修改即可完成,对现有业务可以做到无侵入,尽可能的减少了故障发生的可能。”"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":"br"}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"——骑士卡CTO"}]}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"本文转载自:阿里巴巴中间件(ID:Aliware_2018)"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"原文链接:"},{"type":"link","attrs":{"href":"https:\/\/mp.weixin.qq.com\/s\/yU6Fpy_4DYTOiPf7gg4pBg","title":"xxx","type":null},"content":[{"type":"text","text":"骑士卡:基于Kafka搭建消息中心,上亿消息推送轻松完成"}]}]}]}
版权声明
本文为[InfoQ]所创,转载请带上原文链接,感谢
https://www.infoq.cn/article/ScsH9cK6WLhWc3qDx7O0?utm_source=rss&utm_medium=article

  1. Java collection processing / null value processing / exception processing, use experience sharing!
  2. mysql Innodb_flush_log_at_trx_commit 和 sync_binlog
  3. mysql Innodb_ flush_ log_ at_ trx_ Commit and sync_ binlog
  4. 不能回滚的Redis事务还能用吗
  5. 不能回滚的Redis事务还能用吗
  6. Can redis transactions that cannot be rolled back be used
  7. 23种java设计模式
  8. Java、JavaScript、C、C++、PHP、Python都是用来开发什么?
  9. Docker overlay 清理空间
  10. 「Linux」- 安装 Opera 浏览器 @20210223
  11. java的byte和C#的byte的不同之处
  12. Can redis transactions that cannot be rolled back be used
  13. 23 Java design patterns
  14. What are Java, JavaScript, C, C + +, PHP and python used to develop?
  15. Docker overlay cleaning space
  16. "Linux" - install opera browser @ 20210223
  17. Differences between Java byte and C byte
  18. SAP UI5 JavaScript文件的lazy load - 懒加载
  19. Java 在Excel中添加筛选器并执行筛选
  20. LiteOS:盘点那些重要的数据结构
  21. Lazy load lazy load of SAP ui5 JavaScript files
  22. Add filter and execute filter in excel by Java
  23. Liteos: inventory those important data structures
  24. HDFS依然是存储的王者
  25. [MySQL]事务的MVCC原理与幻读
  26. 93.7%的程序员!竟然都不知道Redis为什么默认16个数据库?
  27. Java 集合处理/ 空值处理/ 异常处理,使用心得分享!
  28. Spring Authorization Server 全新授权服务器整合使用
  29. Spring Security 实战干货:OAuth2登录获取Token的核心逻辑
  30. Java中各种锁的原理解析
  31. java的byte和C#的byte的不同之处
  32. Java 在Excel中添加筛选器并执行筛选
  33. HDFS is still the king of storage
  34. Mvcc principle and unreal reading of [MySQL] transaction
  35. 93.7% of programmers! Why does redis default to 16 databases?
  36. Java collection processing / null value processing / exception processing, use experience sharing!
  37. Integrated use of new authorization server of spring authorization server
  38. Spring security real combat dry goods: the core logic of oauth2 login to obtain token
  39. Principle analysis of various locks in Java
  40. Differences between Java byte and C byte
  41. Add filter and execute filter in excel by Java
  42. Dialogue in spring
  43. 解决Docker MySQL无法被宿主机访问的问题
  44. Oracle OCP 19c 认证1Z0-083考试题库(第1题)
  45. Solve the problem that docker MySQL cannot be accessed by the host
  46. Oracle OCP 19C certification 1z0-083 examination question bank (question 1)
  47. 在 2021 年你需要掌握的 7 种关于 JavaScript 的数组方法
  48. Seven array methods for JavaScript you need to master in 2021
  49. 在 2021 年你需要掌握的 7 种关于 JavaScript 的数组方法
  50. Struts2 + Json _ 配置,异常解决及深入了解Struts2返回JSON数据的原理及具体应用范例
  51. Seven array methods for JavaScript you need to master in 2021
  52. Struts2 + Json _ Configuration, exception resolution and in-depth understanding of Struts2 return JSON data principle and specific application examples
  53. (三)MySQL锁机制 + 事务
  54. (3) MySQL lock mechanism + transaction
  55. 在 2021 年你需要掌握的 7 种关于 JavaScript 的数组方法
  56. Seven array methods for JavaScript you need to master in 2021
  57. 基于Kafka和Elasticsearch构建实时站内搜索功能的实践
  58. Practice of building real time search function in the website based on Kafka and elasticsearch
  59. Golang 实现 Redis(9): 使用GeoHash 搜索附近的人
  60. RxHttp - 轻量级、可扩展、易使用、完美兼容MVVM、MVC架构的网络封装类库