Using docker buildx to realize multi platform compilation

Home 2020-11-09 22:42:47
using docker buildx realize multi


Use docker buildx Implement multi platform compilation - Case report

In the previous article Use docker buildx Implement multi platform compilation - Environment chapter Describes how to deploy docker buildx Environmental Science .

The author will share several meaningful cases in use

0x00 Say first conclusion

  1. docker buildx It runs in a container environment , therefore scheduler and builder Native configuration (ex, /etc/hosts, /etc/docker/daemon.json ) Most of the configurations and scenarios for It's actually not available .
  2. Use ssh://user@host You can easily perform remote builds , Especially in building the most basic image , When you need an overseas mirror .
  3. docker buildx It's still experimental , Constantly updated . Especially in Dockerfile There are many new features to use , You can refer to dockerfile experimental
  4. It's very convenient to transfer and merge images , combination github action Can be fully hosted .

0x01 Distal builder

In the previous article , We go through qemu Multi architecture simulation of the environment . But actually in use , Execution efficiency is particularly low in a simulation architecture , If you have already performed some project compilation , Then you should feel it .

As the author previously maintained opencv Dual architecture project , adopt github action Cross platform compilation , For three or four hours . Under this framework, only a short period of 10 A few minutes .

opencv-multi-arch-github-action.jpg

therefore , The need to perform compilation or other work under the same architecture is particularly urgent .

  1. Fortunately docker buildx Can support It's special for the special plane - Perform the corresponding tasks according to the machine architecture .
  2. in addition , docker It also supports based on ssh://user@host Remote call to protocol . Through password free certificates , Very easy to manage .

execution environment

Suppose there are three machines , And scheduler Can pass the certificate password free login two builder.

purpose The host address CPU framework
scheduler 10.100.100.10 -
arm64 builder 192.168.100.101 arm64 / aarch64
amd64 builder 192.168.100.102 amd64 / x86_64

establish remotebuilder

stay scheduler(10.100.100.10) establish remotebuilder , The order is as follows

ARM64=ssh://root@192.168.100.101
AMD64=ssh://root@192.168.100.102
## Be careful : Name it here remotebuilder
DOCKER_HOST=${AMD64} docker buildx create --name remotebuilder --node hk-amd64 --platform=amd64
### --append Indicates append , Instead of recreating 
DOCKER_HOST=${ARM64} docker buildx create --append --name remotebuilder --node hk-arm64 --platform=arm64
## Use remotebuilder
docker buildx use remotebuilder
## see remotebuilder state 
docker buildx ls --builder remotebuilder

This is the place , remote builder And created successfully .

When we do it locally docker buildx build ... During the mission , The task will be distributed to the corresponding cpu On the host of the architecture .

Perform a mirror build

  1. establish Dockerfile

here Dockerfile It's simple , It's just pulling centos:8 Mirror image .

mkdir -p centos/ && cd centos/
echo "FROM centos:8" > Dockerfile
tree
# .
# └── Dockerfile
# 0 directories, 1 file
  1. Use docker buildx Command pull image .

Be careful : We don't use it here --tag Specify the target image name .

docker buildx build --platform=linux/amd64,linux/arm64 .

remote-builder.jpg

  1. Re execution docker buildx command .

Be careful : This time we used --tag and --push, After the build is complete , Will be pushed to the target docker registry

Be careful 2: If you want to execute --push success , Only need scheduler Execute on the host docker login that will do , builder There is no login requirement on .

docker buildx build --platform=linux/amd64,linux/arm64 --tag tangx/centos:8 --push .

remote-builder.jpg

Pay attention to the position of the red box , The direct result of this time is , More exporting and merging

When task execution changes to , Built layer The cache will exist with remote builder On , meanwhile scheduler It will save multiple architectures manifest Information .

Output results tangx/centos:8 Can be in dockerhub See above .

0x02 Configuration optimization

Use in China docker buildx There's one big problem , It's the Internet .

because driver It's running in the container , Refer to official documentation buildx - github.com, As a result, many of the native configurations do not work .

Such as docker.io In this way, the official image of foreign countries , The pulling speed is not ideal .

Use mirrors to accelerate optimization

  1. New configuration file buildkitd.toml
# buildkitd.toml
[registry."docker.io"]
mirrors = ["wlzfs4t4.mirror.aliyuncs.com"]

And create local builder

### create builder with mirror
docker buildx create --use --name localbuilder --platform=linux/amd64,linux/arm64 --config=buildkitd.toml
  1. newly build Dockerfile
# Dockerfile
FROM centos:8

And build a mirror image

### build a image
docker buildx build --platform=linux/amd64,linux/arm64 .

builder-with-mirror.png

The results are shown in the figure above , It takes about 14s about

Using default parameters , No mirror optimization

  1. Create... That doesn't use mirror optimization mirror, And build
### without mirror
docker buildx create --use --name localbuilder-no-mirror --platform=linux/amd64,linux/arm64 
### build a image
docker buildx build --platform=linux/amd64,linux/arm64 .

builder-without-mirror.png

You can see from the picture , In the absence of mirror Under the circumstances , There's a network problem .

This is because builder Run in container class , It's not using the host /etc/docker/daemon.json To configure ( Suppose the host is already configured mirror), It is equivalent to domestic Direct connection docker.io Pull out the mirror image .

0x03 Dockerfile Case study

If you read Dockerfile ARG Usage and scope analysis , So the following two Dockerfile It's easy .

sync

The only effect : Rename the image , The synchronization of non mirror image is realized by parameter .

ARG IMAGE
FROM ${IMAGE}
image=tangx/alpine
tag=3.12
docker buildx build --platform=linux/amd64,linux/arm64 \
--tag ${image}:${tag} \
--file Dockerfile
--build-arg IMAGE=alpine:3.12 \
.

combine

Through here Multistage In the building Alias And ${TARGETARCH} The way , Will the two separate tag The images merge into one . for example minio/minio Mirror image .

ARG TARGETARCH
FROM example.com/alpine:3.12-arm64 as arm64
FROM example.com/alpine:3.12-amd64 as amd64
FROM ${TARGETARCH}
docker buildx build --platform=linux/amd64,linux/arm64 \
--tag example.in/alpine:3.12 \
--file Dockerfile
--build-arg IMAGE=alpine:3.12 \
.

This article starts with the author's blog : https://tangx.in/2020/11/07/docker-buildx-examples/


This article is originally contained in the author's personal blog https://tangx.in

If you find this helpful , You can pay attention to the author's official account of WeChat

mp-weixin


Please have a cup of coffee

weixin-pay alipay

版权声明
本文为[Home]所创,转载请带上原文链接,感谢

  1. 【计算机网络 12(1),尚学堂马士兵Java视频教程
  2. 【程序猿历程,史上最全的Java面试题集锦在这里
  3. 【程序猿历程(1),Javaweb视频教程百度云
  4. Notes on MySQL 45 lectures (1-7)
  5. [computer network 12 (1), Shang Xuetang Ma soldier java video tutorial
  6. The most complete collection of Java interview questions in history is here
  7. [process of program ape (1), JavaWeb video tutorial, baidu cloud
  8. Notes on MySQL 45 lectures (1-7)
  9. 精进 Spring Boot 03:Spring Boot 的配置文件和配置管理,以及用三种方式读取配置文件
  10. Refined spring boot 03: spring boot configuration files and configuration management, and reading configuration files in three ways
  11. 精进 Spring Boot 03:Spring Boot 的配置文件和配置管理,以及用三种方式读取配置文件
  12. Refined spring boot 03: spring boot configuration files and configuration management, and reading configuration files in three ways
  13. 【递归,Java传智播客笔记
  14. [recursion, Java intelligence podcast notes
  15. [adhere to painting for 386 days] the beginning of spring of 24 solar terms
  16. K8S系列第八篇(Service、EndPoints以及高可用kubeadm部署)
  17. K8s Series Part 8 (service, endpoints and high availability kubeadm deployment)
  18. 【重识 HTML (3),350道Java面试真题分享
  19. 【重识 HTML (2),Java并发编程必会的多线程你竟然还不会
  20. 【重识 HTML (1),二本Java小菜鸟4面字节跳动被秒成渣渣
  21. [re recognize HTML (3) and share 350 real Java interview questions
  22. [re recognize HTML (2). Multithreading is a must for Java Concurrent Programming. How dare you not
  23. [re recognize HTML (1), two Java rookies' 4-sided bytes beat and become slag in seconds
  24. 造轮子系列之RPC 1:如何从零开始开发RPC框架
  25. RPC 1: how to develop RPC framework from scratch
  26. 造轮子系列之RPC 1:如何从零开始开发RPC框架
  27. RPC 1: how to develop RPC framework from scratch
  28. 一次性捋清楚吧,对乱糟糟的,Spring事务扩展机制
  29. 一文彻底弄懂如何选择抽象类还是接口,连续四年百度Java岗必问面试题
  30. Redis常用命令
  31. 一双拖鞋引发的血案,狂神说Java系列笔记
  32. 一、mysql基础安装
  33. 一位程序员的独白:尽管我一生坎坷,Java框架面试基础
  34. Clear it all at once. For the messy, spring transaction extension mechanism
  35. A thorough understanding of how to choose abstract classes or interfaces, baidu Java post must ask interview questions for four consecutive years
  36. Redis common commands
  37. A pair of slippers triggered the murder, crazy God said java series notes
  38. 1、 MySQL basic installation
  39. Monologue of a programmer: despite my ups and downs in my life, Java framework is the foundation of interview
  40. 【大厂面试】三面三问Spring循环依赖,请一定要把这篇看完(建议收藏)
  41. 一线互联网企业中,springboot入门项目
  42. 一篇文带你入门SSM框架Spring开发,帮你快速拿Offer
  43. 【面试资料】Java全集、微服务、大数据、数据结构与算法、机器学习知识最全总结,283页pdf
  44. 【leetcode刷题】24.数组中重复的数字——Java版
  45. 【leetcode刷题】23.对称二叉树——Java版
  46. 【leetcode刷题】22.二叉树的中序遍历——Java版
  47. 【leetcode刷题】21.三数之和——Java版
  48. 【leetcode刷题】20.最长回文子串——Java版
  49. 【leetcode刷题】19.回文链表——Java版
  50. 【leetcode刷题】18.反转链表——Java版
  51. 【leetcode刷题】17.相交链表——Java&python版
  52. 【leetcode刷题】16.环形链表——Java版
  53. 【leetcode刷题】15.汉明距离——Java版
  54. 【leetcode刷题】14.找到所有数组中消失的数字——Java版
  55. 【leetcode刷题】13.比特位计数——Java版
  56. oracle控制用户权限命令
  57. 三年Java开发,继阿里,鲁班二期Java架构师
  58. Oracle必须要启动的服务
  59. 万字长文!深入剖析HashMap,Java基础笔试题大全带答案
  60. 一问Kafka就心慌?我却凭着这份,图灵学院vip课程百度云