This article is based on 〖2016 Global operation and Maintenance Conference • ShenZhen Railway Station 〗 The content shared by the guests of the live speech is organized , The editor is Wu Zhaojun @ tencent .
Welcome to your attention “ Efficient operation and maintenance ( WeChat ID：greatops)” official account , In order to preemptively read a variety of original articles full of dry goods .
About the lecturer
Graduated from Beijing University , Get a bachelor's degree and a master's degree ; Currently responsible for the research and development of products related to Alibaba cloud container technology .
Before joining Ali , I was in IBM China Development Center work 14 year , As senior technical specialist , be responsible for IBM Enterprise platform cloud product line PureApplication System R & D of ; Also responsible for and involved in a series of IBM stay Web 2.0,SOA Middleware development and innovation , We have also provided SOA Technical consultation and project implementation .
Hello everyone , The theme of my speech is 《 The cloud is based on Docker Micro service and continuous delivery practice of 》, I will mainly introduce ：
Docker And microservices
Cloud production environment deployment
application Docker Transformation
Continuous delivery process practice
Changes in delivery have changed the global economy
A lot of people start with the container story , I'm no exception . One hundred years ago , Something called a container , Changed the global law , Now in the cross-border trade 90% All the goods are transported in containers .
A humble single , A change in delivery , You can change the whole economy .
Docker In this era, it's a revolution that changes the whole software delivery , Now it is used in almost all operation and maintenance or architecture Docker To deliver , Why? ？
Docker in Alibaba
Alibaba Docker The use of is everywhere ,2011 year , Taobao began to adopt container technology , With some of the internal structure of Ali , It greatly improves the efficiency of development and operation and maintenance , Make the whole development more efficient . stay “ double 11” Such a big promotion , Played a huge role .
Ali Baichuan is an e-commerce company for mobile platform PaaS, It USES Docker Can support different programming environments , So that users can quickly run applications online .
Based on Ali's practice , We have been implementing container services on public cloud computing platforms since last year , You don't have to worry about Docker The underlying technology , Network storage 、 Don't care about resource management , You just care about yourself Docker The application deployment is OK .
meanwhile ,Docker It will play a huge role in many fields .
For example, a while ago, Google's AlphaGo, In fact, one of the most important frameworks behind computing is the deep learning framework , Building a deep learning framework is a very complex learning , You need to configure the software of the corresponding deep computing framework .
But the services provided to customers in Ali , All you have to do is use Docker Images can be quickly grouped together HPC To build a machine , You can get the computing power you need for deep learning on demand , Maybe one day you can develop one for PK.
Actually Docker The use of Alibaba is ubiquitous , You can imagine , A system as big as Ali , Including virtualization 、 database 、 big data 、 The Internet , Such a huge system , Hundreds of components , We're dealing with it now Docker Handle , Speed up its development, operation and output .
We can build the whole alicloud in a few hours . So ,Docker Has completely changed some of our scenarios for software delivery and operation and maintenance .
Why? Docker So important ？
Accelerate application delivery , Shorten the dream distance
We're talking about Internet companies , What is the core competitiveness of Internet companies ？
It's efficiency , It depends on whether you can launch the product as quickly as possible , Lowest cost , The fastest iteration .
And in the What was our software development like ten years ago ？
Our application is a holistic application , It's developed by a large team , This team is divided into development team and operation and maintenance team , The development team has a front-end team 、 Back end team 、 Database team , These teams may argue with each other for business needs , In the end, the whole software development iteration is extremely slow .
In traditional enterprises, the cycle of iteration is from half a year to one year , This speed is far from meeting the needs of the business side , At the same time, it is even more difficult for our operation and maintenance personnel , Until the last moment , Software personnel explain that it will be online soon , Give a bunch of installation scripts to the O & M staff , Let the operation and maintenance personnel to install and deploy , To ensure its high availability .
Can you imagine ？ How could such a thing happen ！
So , Everyone is carrying a Great challenge :
The first is slow change , In the Internet age, who becomes slow , Whoever dies quickly ;
second , In order to ensure such a large-scale overall application , It's very hard to scale , Usually in order to adapt to large traffic , We can only add new CPU And so on .
Everybody knows , If you want to guarantee the growth of linear computing performance , It may cost more . in addition , The continuous operation ability of the whole system is also bad , The bigger, the more unstable , One of the components is broken , There's an avalanche effect , The whole system goes down .
It's a very painful process , Ali went through this process , Today's Internet companies, including Alibaba, have evolved into the following structure .
The bottom layer is a computing architecture based on cloud services or virtualization , Every business such as e-commerce , There can be user management 、 Commodity management , Our shopping cart 、 shoppers 、 advertisement .
These are different modules , Each module is made up of a team , Each image can be deployed independently , Components communicate with each other through standardized protocols .
Only in this way can speed be guaranteed , In order to ensure that we can survive in the competition .
Docker And virtualization technology
Docker The emergence of the emergence of further push this matter ,Docker Is a lightweight operating system virtualization solution , Deliver more quickly .
Docker It has good portability , This is more important , We can develop 、 test 、 Use a unified method in production 、 Unified media to deliver software .
Imagine , If it's in a hybrid cloud scenario , for instance “ double 11 Great promotion ”, A lot of computing power now has been moved to the public cloud , Because it's distributed on demand , You can quickly extend the application to the entire data center .
combination Docker Container and virtualization technology
A lot of people say “Docker Changed the life of virtualization ”, We think Docker And virtualization are complementary in many ways .
Docker Technology still has its own limitations , such as Docker It's not good at isolating the system , Like virtualization Docker Can't do , At present, one of the most mature solutions is to combine virtualization technology with container technology .
In the public cloud, we are not recommended to adopt the multi tenant approach , Because there are a lot of security holes .
Cloud Native Computing
With the development of microservice and container technology , Last year, Google took the lead in setting up a Cloud Native Computing The foundation .
It defines some basic elements and frameworks for future native applications , Delivered as a container in a microservice architecture , Support DevOps, This platform is dynamically managed by itself , It's not manual .
Why do you do this ？
We should look at it from the perspective of sustainable development , The single application will touch the ceiling sooner or later , Its complexity 、 Scalability must hit the wall .
therefore , We use microservices , Microservices are not free lunches , It brings complexity as well as benefits , I used to run an app , Now we have to operate dozens of services .
I have been told by my colleagues that there are 20 Services , After decomposing into microservices, there are nearly 400 , His management is a lot more .
How to let services and services monitor their health ？
Once a service fails , We need to isolate it 、 Fuse 、 Downgrade , Then how do we update the version of this microservice , Previous products are not guaranteed to be interrupted ？
These are huge impacts .
If you let people do these things manually , Definitely not , Make sure you use the platform 、 In an automated way . therefore , That's why people are emphasizing the need for a platform to support .
Walking in the clouds ： Start Docker The journey
In fact, using servers on the cloud is not complicated , People are very used to their own development 、 test 、 Data center use Docker technology , however Why don't you all on the cloud ？
People think virtual machines are needed 、 It's a lot of trouble with all kinds of things , But it's not like this .
Docker stay 2014 Launched at the end of the year Docker Machine, utilize Docker Machine You can quickly create a Docker The cloud environment of .
The thing to do is very simple , download Docker Machine Then look for the driver of the cloud provider , For example, Alibaba cloud now 、 Amazon 、Aure Some of these Driver, adopt Driver, From the command line, I can deploy my containerized application on it , It's simple .
Used in a production environment Docker
But if it's really on the cloud 、 Choose... In the growing environment Docker, The challenge you face is much more than that .
One Docker Certainly not enough , It must be a cluster , How to manage this cluster , How to bridge the network , What about storage , How to schedule resources , How to arrange , It's a very complicated thing .
For this matter , Many Internet companies offer so-called products , Let me introduce some of them to you .
Docker Cloud,Docker In the last year 11 I bought tutum.co In this year 2 Launched in April Docker Cloud, It basically provides Docker A choreographed of its own API.
In the amazon 2014 year 11 Launched in April EC2 Container Service, It was originally based on a private of its own API Provides container described Services , But in the last year, it gradually began to support a wider range of Docker, To describe a composite containerized application .
Google's Compose template, It's also in 2014 year 7 Open source in , It integrates a lot of ideas from the past , A lot of history of the corresponding scheduling . It was very popular after it was pushed out , But it provides a set of its own unique abstraction of containerized applications .
Container cluster management - Docker Swarm
To better illustrate some of our content , I'm going to introduce Ali container service today . In order to solve the problem that users are developing 、 Seamless migration of test environment , We are completely compatible with Docker Native choreography .
Docker What does the native choreography include ？
First of all Docker Swarm, It's a very delicate design , Can I put a group of Docker engine Become a virtual Docker engine, I'm all for this virtual Docker engine Give orders , From its control node to a real node to execute .
Its architecture is very simple , Only one is needed on each node Docker engine And then install another Agent Just fine ,Agent By reporting, you can automatically register a machine , Through this function, there is a node above that can find the node information inside , The cluster can be built automatically .
It's a very delicate abstraction , Because it's almost 99% To support all Docker Native API.
It brings two benefits ：
It can be owned by three parties and Docker Connected tools are integrated , Don't make any changes ;
Another advantage is that it provides a pluggable Architecture , Like its scheduler 、 Both storage and networking can be easily expanded .
And it has a big shortcoming ：
Swarm and Docker equally , Its basic abstract unit is the container , Not from the perspective of service .
Container arrangement - Docker Compose
Docker Compose From the Docker An acquisition of ,Docker Compose It describes how to combine a set of containers with resources related to the container .
For example, we take Wordpress For example , then Mysql, A simple layout template can describe it , adopt Wordpress Mirror together , Connect to through a link Mysql, adopt volume To create .
In this way, it is very elegant to describe how a set of containers work together in association , And you can start the whole application station with one button . If we scale it a little bit , It's very simple .
Simple to use , To facilitate the development . It's a great development tool , stay Docker There are more than 70% People using Docker Compose Do image development .
Expanded on the network 、 Storage support . Not only can you describe and container , You can also describe some associations between a container and its underlying resources .
For development and deployment , Automatic operation and maintenance is not supported . for instance , How to monitor the operation and maintenance , Is it possible to perform elastic contraction , It didn't do it , Because it's a development tool itself .
Alibaba container service
What is our ideal container development platform ？
One of the capabilities provided by our alicloud container service , First of all, the bottom layer is the public cloud computing platform or the enterprise's proprietary cloud . Above that is the container layer , except Docker outside ,Docker Warehouse , There's storage and networking .
Native Docker It's not enough , It provides a mechanism , We also recommend this mechanism to store blocks in the cloud 、 Object storage 、 Network storage can be easily integrated in .
Above the container layer is the cluster management and scheduling layer , We've done a lot of optimization and improvement , For example, we talked about an application just now , We need to make sure it's moving , We have so many things to do , We can't guarantee it won't go down , What can we guarantee ？
We can schedule resources , Even if one data center loses power , A regional data center is down , It can also guarantee resource scheduling .
in addition , From the perspective of container arrangement , How can we easily expose a service , It's easy to collect and monitor the container log ？ We've done a lot of expansion , Still can do very good container application control .
Above that is our service layer , Alibaba's own microservice architecture has been built for a long time , It's very popular in the open source world , We internalize these experiences into our support capabilities .
One of the big keys is how to do service discovery 、 Routing of services , We've expanded a lot , adopt BNS Find out , Through load balancing to achieve dynamic load balancing between service nodes , Through these things , Let your micro service do well .
Above the service layer is the access layer , Give Way Web Applications can be very easy to access to your own applications .
This is the core of our entire container platform , But you know , A closed platform is not enough , Because containers don't solve all the problems , The container must be connected with the existing enterprise applications or cloud services .
We have done a good integration capability , It's very easy to integrate with cloud services , We can integrate with third-party tools , Integrate the container technology into your own development process , At the same time, we provide cloud control .
Control ability , In addition to its own access to our cloud monitoring log , In fact, all the control frameworks in our whole system can be extended at will , Because we think that a platform that is not open enough is basically playing rogue , Containers are not your islands of information , Be sure to keep up with the existing IT Control comes together .
We also provide many examples of how to use open source frameworks , Quickly build a cloud monitoring capability that you need , We have some examples in the following documents , You can go to see .
What applications can run in containers
So much has been said , In fact, people will care about ：
How does my app run in the container ？
Which of my applications can be containerized ？
People ask these questions .
This is a very famous classification method that I extracted , It can help you choose what kind of application is suitable for running in the container 、 What kind of application is not suitable for .
It's based on two dimensions ：
One dimension is the application of longevity or short life ;
The other is to see whether it is stateful or stateless .
The best use of containerized applications is the short-lived, stateless applications on the left , Because such an application is the easiest to deploy .
For example, a Web application , We can get rid of it easily and quickly , We don't care about it , You can deploy one very quickly Web application .
The other dimension is the short-lived , Like high performance computing 、 The batch .
Rendering a video must be a lot of state information , But this information can be done through Web Object storage to save , This kind of computing intensive task is also very suitable for doing on it , Because we can quickly identify a large group of clusters , Running such a task through a container like this .
There are also long-lived but stateless , For example, our development test environment has been used or our monitoring , It's going to be there all the time , But its state dependence is very small , We can also consider this kind of application .
Only the lower left is the most challenging dimension in general , It's a stateful service , Generally, stateful services need some adjustments , Including storage adjustment 、 Adjust the network , You still need to DBA Doing complicated work , It's not completely automated .
For some of these applications , Our suggestion is that container technology can be used in test and development , But it's not recommended to use such technology in production .
Docker It's a practical application : Ghost Blog
Next, I'll take a very popular example to explain how to container an application .
Ghost Blog is one of my favorite blog applications , It's simple , Very light , The mirror is also very easy to use , In a very simple way Ghost Mirror it .
But there are still many problems ：
It's not scalable ;
It's not highly available .
All its data is stored locally Database Inside , If the virtual machine node is down , Migrate to another node , Data status is lost .
How can we solve this problem ？
In fact, you can refer to THE TWELVE-FACTOR standard , This specification is supported by most manufacturers , It's a very important programming rule now .
It has a few Core principles ：
The application should be decoupled from the running environment
The application should be decoupled from the external calling service
The application should be decoupled from the configuration
Through this decoupling, we can have some applications become stateless , Be able to deploy and operate on the cloud quickly .
Ghost Blog High availability cluster 1
We just need to add one more MySQL, Let it support MySQL It's OK to drive , Start a ghost+MySQ, adopt MySQL Connect , They are shared , Our container service has been optimized a lot .
Ghost Blog High availability cluster 2
We don't recommend using stateful services like databases in a production environment , What should we do ？
We see a lot of articles that say Docker It's very hard to use , Said it couldn't run the database ,Docker It doesn't run a database .
Why don't I connect directly to my database instance ？
Of course , There are different approaches , But I think the best thing to do is to make a minimal change , The application layer can mirror a database without any perception , Put one Docker The running use of becomes a Web, We've added an ability to expand .
We expand to quote Web Service for , We can deploy in the production environment , And your application layer doesn't change .
Ghost Blog High availability cluster 3
In the process, there is still a problem to be solved , Attachments uploaded by users , Like pictures , Still stored in local storage , It's not going to work . In order to do this , We have another thing .
We can go through Docker Of Volueme Plugin To solve , It provides a very flexible mechanism to support different storage types , Block storage is now supported 、 Object storage 、 Network file system .
And the more interesting thing is , All of our network drivers and Volume In fact, all the drivers run in the container , Because only in this way , We can carry out unified operation and maintenance and unified management of the whole system .
however Docker There are still flaws in this area ,Docker It's impossible to distinguish some of these network drivers , Will cause a restart Docker Engine When , It's possible to kill your Volume Driver, Your data is broken without saving it ;
Or say , It's the same with your network , It may kill your network driver without waiting for your application to kill , Your application and network will be completely disrupted , That's not going to work .
We actually made changes in the community , We can grade different daemons , You can start some containers , It can have a higher system level , It is loaded first at boot time , When it is stopped, it will be stopped finally , In the community 1.11 There will be similar work in the version , The community will solve this kind of work .
By treating both network drivers and volume drivers as containers , It can also bring us greater benefits .
Our whole system is very scalable , For example, we are talking about cooperation with a third-party network storage company , It now takes a container to deliver the storage driver , We can run the storage driver on the server without modifying a single line of code , This will make our system more scalable .
Containerized continuous integration and delivery
Docker One of the important benefits of is portability , Through portability, you can develop 、 Throughout the software life cycle of testing and production , Deliver our software products in the same way .
for instance , That's what our developers are like now , You can start the local development environment with one click , Submit the code when it's done , When it is submitted, it is the corresponding self code and the original .
With this , dependent Docker infrastructure , Such as the container 、 Mirror warehouse , You can compile the relevant code into Docker Mirror image , And throughout the test 、 This image has been used in production , All the steps can be repeated , And ensure consistency .
In the process , Everything can support Docker Managed , And it can be updated and managed quickly .
What's the benefit of doing this ？
The developer thought on the first day “ How my code goes online ”, It's a huge cultural and temporal change .
We used to talk about it DevOps, It's no use just for operation and maintenance personnel , You have to let developers think about it on the first day of development “ How software is delivered 、 How to be highly available in the cloud 、 Telescopic ”, This is a culture and an idea that has to be changed .
If that doesn't change , You can't change... With any technology .
We used to be the same in Ali , Developers are amazing , The operation and maintenance personnel are hard pressed , Developers develop , Operation and maintenance personnel stay up late online , Roll back when something goes wrong , Very inefficient .
But we now require developers to deliver one for each function Docker Mirror image , Put your preconditions 、 Postcondition 、 Check the script 、 The health check is delivered at the beginning , Without delivery , We operators stopped accepting code like this .
Through this , We can evolve quickly .
Simplified continuous delivery process
Source control , We can have a mirror service , It can subscribe to notifications from the source repository , Our container service can also subscribe to notification of mirror changes .
When your code changes , For example, modify a web page , Turn two columns into three columns , It tells the mirror service to pull the corresponding code build , After packaging it into a mirror image , Automatically notify the corresponding container service to update the ready-made application .
In a few minutes , Change online , We can also integrate with other services .
Complete continuous delivery process
How can Alibaba social platform monitor the change of source code warehouse , After code launch changes , It pulls the code and mirrors it , After passing the unit test, mirror it , Notify the continuous delivery server to proceed with the next step , It's an assembly line .
Take... In the assembly line Docker Mirror image and Docker File in test environment 、 Pretest environment 、 Deployment on production environment .
We have the same one Docker Mirror image , The same one Docker Templates , It can be used in different environments , So I can make sure that from the development 、 test 、 Consistency of everything online .
We must adhere to some of these ideas , because DevOps A lot of things we all know , What is lacking is persistence .
Immutable Architecture （immutable infrastructure）
Docker Why do you get DevOps Cheers from people in the field ？
In fact, the way we want it to be operated .
understand OOpenStac We all know the famous fable stock market , Your app is like a pet that needs your care all day , It's like a cow in a herd , It can be killed at any time without sadness , You can always have another cow to add .
Make this system self-sustaining , Very robust , The whole system will not be terminated due to the failure of any node .
Take advantage of immutability to maintain the infrastructure ： Once instantiated , Never change ; It will only be replaced correctly by another instance .
Avoid inconsistencies between environments . It's a big part of our daily life , exceed 30% The online errors are caused by the inconsistency between the development environment and the test environment and the online environment .
As like as two peas, I can guarantee that all code is exactly the same as the door , Always run the way you expect , Test and launch 、 Production is the same thing .
Simplify deployment complexity . It's very difficult to patch and upgrade in place , Especially a lot of system software , There are many pairs. How many .
Low cost rollback . It's a lot more complicated to write a rollback code , And it's hard to be right , Because we never test rollback .
In fact, it's not new , There used to be a virtual machine , It's also possible to have an automated tool at home , however Docker Make it faster , Make it easier .
It's like saying , Every Docker The mirror image is actually immutable , It's a process , You can always replace it with a new image .
Docker It starts very fast , It used to take a few minutes to roll back a virtual machine , But if you use Docker The rollback time is in seconds , Users can't feel the interruption .
Docker： Immutable architecture dream come true
To achieve this, we also need to pay attention to , Because I know a lot of people still put Docker Containers are used as lightweight virtual machines , There's nothing wrong with this , It's just that everyone's scene is different .
however , I ask you to think twice when you do this , When you think of the container as a lightweight virtual machine , You have to think about it again , You may lose Docker A lot of good features , The most important thing is immutability .
To achieve this , We need to do something right ：
Never manually modify the contents of the container , Your container should always be built out of code .
Try not to use latest As a mirror label , You have to roll back in production , You need to know what issues to roll back 、 Which version , One of the easiest ways to do this is to use Git Commit As a mirror image tag Part of the , Easy to track , Make sure your online product knows exactly what version is running .
Don't store any variable data in a mirror image , To pass the Volume abstract , You can combine the application changes with the lifecycle of your container , The benefits it brings far outweigh laziness 、 The benefits of saving time .
Cross host container network Cloud practice
On the Internet, you can see a lot about Docker A lot of discussion on the Internet , Most of the discussion is based on your own data center , Because it's very simple in your own data center , You can even control the switch .
But on the public cloud, it's hard , We have to choose on the cloud Docker If the containers are interconnected , It must be adapted according to the maximum optimization of cloud manufacturer's configuration .
Generally, there are two solutions on the cloud to realize the interconnection between container networks across virtual machines ：
adopt Overlay Methods , As long as the three layers are connected , adopt Overlay Realize virtual network .
Overlay This is a very general way , It can be used in different network environments , Even across different cloud providers . But it also has weaknesses , Its own performance is limited .
We're at Amazon 、 Alibaba cloud 、IBM It's been measured on the cloud , adopt Overlay Container interconnection performance and container communication through native virtual machine , Only bandwidth 70%, Also increases 20%-30% Delay of , For people who are sensitive to network performance, we don't recommend this .
Through the network characteristics of cloud provider network itself , such as VPC,VPC It's very different from today's Internet , stay VPC We can control some IT Distribute 、 Routing rules .
We get a lot of benefits in this way , Because in one VSwitch Inside, the whole two were once connected , We can even avoid Web technology , In the corresponding node with the corresponding routing table can achieve interoperability between containers .
But in one VSwitch If you want to do it, you have to bear a consequence , When a data center loses the chain, the application will be suspended .
In general, we recommend this method in production , Through one VRouter Deploy your application in different VSwitch On , And then in VRouter Configure routing rules , This is a very general way , Whether it's on Amazon or alicloud , We all use this way .
Its benefits , The bandwidth of communication between containers , On Alibaba cloud and Amazon , There is no difference between bandwidth and native speed , But the delay will be a little bit more , May be in 10% about .
therefore , If you pursue performance, you may consider , Of course, it's also limited by a VRouter How many routing tables are there , Finally, limit the maximum size of your cluster nodes .
Everybody knows , There is no best solution in the world , Be sure to choose the right plan according to your own situation .
My speech is over today , This is my blog address （ The cloud community Docker Team blog ）, There are often a lot of and Docker Related examples , And these examples don't necessarily need to run on alicloud , Our big goal is any use of Docker Mirror image 、Docker Template applications can run in the cloud . Thank you for your time .