Preface

We discussed it in the group some time ago , If you want to upload a file to the server regularly, how can you do it . I remember doing it before
A project : For Qualcomm's iot Module writing FOTA function : The implementation module can download the upgrade image package remotely , Realize version upgrade function . And a super powerful tool used at that time cURL. be prompted by a sudden impulse , Decided to write a special article , To friends in need .

The article is divided into two parts

  1. First introduced cURL Usage of ;
  2. To achieve a regular upload log file of a simple program .
     Insert picture description here

Concept

cURL Is a common command line tool , Used to request Web The server . Its name is client (client) Of URL Tool means .

cURL The original author of is Daniel Stenberg ( At present, it is cURL The core developer of ), It's also IETF HTTPbis Senior members of the working group .Daniel stay 1998 Created in 2003 curl project , He wrote the original curl edition , And created libcurl library . up to now , Code warehouse includes 24000 Time commit More than half of them are Daniel I submitted , He is still the core developer of the project .Daniel Indicate that you have curl Think of yourself as a child .

As a powerful tool , Supported protocols include (DICT, FILE, FTP, FTPS, GOPHER, HTTP, HTTPS, IMAP, IMAPS, LDAP, LDAPS, POP3, POP3S, RTMP, RTSP, SCP, SFTP, SMTP, SMTPS, TELNET and TFTP), And support POST、cookies、 authentication 、 Download some files from the specified offset , Has a user agent string 、 The speed limit 、 file size 、 Progress bar 、cookie Support 、 User authentication 、 Breakpoint continuation and other characteristics .

One 、 Command installation

sudo apt-get install curl

Two 、cURL Command syntax :

curl [options] [URL...]

3、 ... and 、URL Format

URL The format definition of should refer to RFC 1808 .

 Address :http://www.w3.org/Addressing/rfc1808.txt
《Relative Uniform Resource Locators 》

URL It's made up of three parts : The resource type 、 Host domain name where resources are stored 、 Resource file name .
It can also be said that 4 Part of it is made up of : agreement 、 host 、 port 、 route
URL The general syntax of is :

protocol :// hostname[:port] / path / [;parameters][?query]#fragment

( Square bracket [] Of is optional ).

protocol( agreement )

Specify the transport protocol used , The following table lists protocol Valid scheme name for property . The most common is HTTP agreement , So is it WWW The most widely used protocol in .

  • file Resources are files on the local computer . Format file:///, Note that there should be three slashes at the back .
  • ftp adopt FTP Access resources . Format FTP://
  • gopher adopt Gopher Protocol access to the resource .
  • http adopt HTTP Access the resource . Format HTTP://
  • https Through safe HTTPS Access the resource . Format HTTPS://
  • mailto The resource is the email address , adopt SMTP visit . Format mailto:
  • MMS adopt Support MMS( Streaming media ) Play the resource by protocol .( For Software :Windows Media Player) Format MMS://
  • ed2k adopt Support ed2k( Dedicated download link ) Agreed P2P Software access to the resource .( For Software : Emule ) Format ed2k://
  • Flashget adopt Support Flashget:( Dedicated download link ) Agreed P2P Software access to the resource .( For Software : express ) Format Flashget://
  • thunder adopt Support thunder( Dedicated download link ) Agreed P2P Software access to the resource .( For Software : thunderbolt ) Format thunder://
  • news adopt NNTP Access the resource .

hostname( Host name )

It refers to the domain name system of the server where the resources are stored (DNS) Host name or IP Address . Sometimes , Before the host name, you can also include the user name and password required to connect to the server ( Format :username:[email protected]).

port( Port number )

Integers , Optional , Use the default port of the scheme when omitted , All kinds of transmission protocols have default port numbers , Such as http The default port is 80. If you omit , Then use the default port number . Sometimes for safety or other reasons , Ports can be redefined on the server , Non standard port number is adopted , here ,URL We can't omit the item of port number .

path( route )

By zero or more “/” String separated by symbols , Generally used to represent a directory or file address on the host .

parameters( Parameters )

This is an option for specifying special parameters .

query( Inquire about )

Optional , For dynamic web pages ( If you use CGI、ISAPI、PHP/JSP/ASP/ASP.NET And so on ) Pass parameters , There can be multiple parameters , use “&” Symbols separate , The name and value of each parameter are “=” Symbols separate .

fragment( Pieces of information )

character string , Used to specify fragments in network resources . For example, a web page has multiple noun explanations , You can use fragment Go straight to a noun to explain .

Four 、curl Command parameter details :

because linux curl Very powerful , So there are a lot of command parameters , The following table only filters out some commonly used parameters , For more parameters, please run “man curl” Command view .

Parameters function
-a/–append When uploading files , Attach to target file
-A/–user-agent Set up the user agent to send to the server
-anyauth have access to “ whatever ” Authentication methods
-b/–cookie <name=string/file> cookie String or file read location
–basic Use HTTP Basic verification
-B/–use-ascii Use ASCII / Text transfer
-c/–cookie-jar After the operation, put cookie Write to this file
-C/–continue-at The breakpoint continues
-d/–data HTTP POST Way to transmit data
–data-ascii With ascii The way post data
–data-binary In a binary way post data
–negotiate Use HTTP Authentication
–digest Use digital authentication
–disable-eprt No use EPRT or LPRT
–disable-epsv No use EPSV
-D/–dump-header hold header Information is written to the file
–egd-file For random data (SSL) Set up EGD socket route
–tcp-nodelay Use TCP_NODELAY Options
-e/–referer Source URL
-E/–cert <cert[:passwd]> Client certificate file and password (SSL)
–cert-type Certificate file type (DER/PEM/ENG) (SSL)
–key Private key filename (SSL)
–key-type Private key file type (DER/PEM/ENG) (SSL)
–pass Private key password (SSL)
–engine The encryption engine uses (SSL). “–engine list” for list
–cacert CA certificate (SSL)
–capath CA Catalog (made using c_rehash) to verify peer against (SSL)
–ciphers SSL password
–compressed Asking for a return is a compressed situation (using deflate or gzip)
–connect-timeout Set the maximum request time
–create-dirs Create a directory hierarchy of local directories
–crlf Upload is to LF Into a CRLF
-f/–fail Do not show... When connection fails http error
–ftp-create-dirs If the remote directory does not exist , Create a remote directory
–ftp-method [multicwd/nocwd/singlecwd] control CWD Use
–ftp-pasv Use PASV/EPSV Instead of ports
–ftp-skip-pasv-ip Use PASV When , Ignore this IP Address
–ftp-ssl Try to use SSL/TLS To carry out ftp The data transfer
–ftp-ssl-reqd Asked to use SSL/TLS To carry out ftp The data transfer
-F/–form <name=content> simulation http Form submission data
–form-string <name=string> simulation http Form submission data
-g/–globoff Disable URL sequence and range usage {} and []
-G/–get With get To send data
-H/–header Custom header information is passed to the server
–ignore-content-length Neglected HTTP Length of header information
-i/–include Output includes protocol Header information
-I/–head Display request header information only
-j/–junk-session-cookies Read file and ignore session cookie
–interface Use the specified network interface / Address
–krb4 Use... With the specified security level krb4
-k/–insecure Allow certificates not to be used to SSL Site
-K/–config Specified profile read
-l/–list-only List ftp The name of the file in the directory
–limit-rate Set the transmission speed
–local-port Force local port number
-m/–max-time Set the maximum transfer time
–max-redirs Set the maximum number of directories to read
–max-filesize Set the maximum number of files to download
-M/–manual Display full manual
-n/–netrc from netrc Read the user name and password in the file
–netrc-optional Use .netrc perhaps URL To cover -n
–ntlm Use HTTP NTLM Authentication
-N/–no-buffer Disable buffered output
-o/–output Write the output to this file
-O/–remote-name Write the output to this file , Keep the filename of the remote file
-p/–proxytunnel Use HTTP agent
–proxy-anyauth Choose any proxy authentication method
–proxy-basic Use basic authentication on agents
–proxy-digest Use digital authentication on agents
–proxy-ntlm Use... On agents ntlm Authentication
-P/–ftp-port
Use port address , Instead of using PASV
-q As the first parameter , close .curlrc
-Q/–quote Before file transfer , Send command to server
-r/–range Search from HTTP/1.1 or FTP Server byte range
–range-file Read (SSL) A random file of
-R/–remote-time When generating files locally , Keep remote files for
–retry When there is a problem with the transmission , The number of retries
–retry-delay When there is a problem with the transmission , Set retry interval
–retry-max-time When there is a problem with the transmission , Set the maximum retry time
-s/–silent silent mode . Don't export anything
-S/–show-error Display error
–socks4 <host[:port]> use socks4 Agent gives host and port
–socks5 <host[:port]> use socks5 Agent gives host and port
–stderr
-t/–telnet-option <OPT=val> Telnet Option is set
–trace On the specified file debug
–trace-ascii Like -- Tracking but not hex Output
–trace-time track / Detailed output , Add time stamp
-T/–upload-file Upload files
–url Spet URL to work with
-u/–user <user[:password]> Set the user and password of the server
-U/–proxy-user <user[:password]> Set the proxy user name and password
-w/–write-out [format] After what output is complete
-x/–proxy <host[:port]> Use... On a given port HTTP agent
-X/–request What order to give
-y/–speed-time The time it takes to give up the speed limit , The default is 30
-Y/–speed-limit Stop transmission speed limit , Speed time

5、 ... and 、Linux curl Command exit code :

Here is linux curl Command error codes and their corresponding error messages , When the command execution error, you can check the error reason through the error code , Convenient for development and debugging .

refund Out code Error description
1 Unsupported protocol. This build of curl has no support for this protocol.
2 Failed to initialize.
3 URL malformed. The syntax was not correct.
5 Couldn’t resolve proxy. The given proxy host could not be resolved.
6 Couldn’t resolve host. The given remote host was not resolved.
7 Failed to connect to host.
8 FTP weird server reply. The server sent data curl couldn’t parse.
9 FTP access denied. The server denied login or denied access to the particular resource or directory you wanted to reach. Most often you tried to change to a directory that doesn’t exist on the server.
11 FTP weird PASS reply. Curl couldn’t parse the reply sent to the PASS request.
13 FTP weird PASV reply, Curl couldn’t parse the reply sent to the PASV request.
14 FTP weird 227 format. Curl couldn’t parse the 227-line the server sent.
15 FTP can’t get host. Couldn’t resolve the host IP we got in the 227-line.
17 FTP couldn’t set binary. Couldn’t change transfer method to binary.
18 Partial file. Only a part of the file was transferred.
19 FTP couldn’t download/access the given file, the RETR (or similar) command failed.
21 FTP quote error. A quote command returned error from the server.
22 HTTP page not retrieved. The requested url was not found or returned another error with the HTTP error code being 400 or above. This return code only appears if -f/–fail is used.
23 Write error. Curl couldn’t write data to a local filesystem or similar.
25 FTP couldn’t STOR file. The server denied the STOR operation, used for FTP uploading.
26 Read error. Various reading problems.
27 Out of memory. A memory allocation request failed.
28 Operation timeout. The specified time-out period was reached according to the conditions.
30 FTP PORT failed. The PORT command failed. Not all FTP servers support the PORT command, try doing a transfer using PASV instead!
31 FTP couldn’t use REST. The REST command failed. This command is used for resumed FTP transfers.
33 HTTP range error. The range “command” didn’t work.
34 HTTP post error. Internal post-request generation error.
35 SSL connect error. The SSL handshaking failed.
36 FTP bad download resume. Couldn’t continue an earlier aborted download.
37 FILE couldn’t read file. Failed to open the file. Permissions?
38 LDAP cannot bind. LDAP bind operation failed.
39 LDAP search failed.
41 Function not found. A required LDAP function was not found.
42 Aborted by callback. An application told curl to abort the operation.
43 Internal error. A function was called with a bad parameter.
45 Interface error. A specified outgoing interface could not be used.
47 Too many redirects. When following redirects, curl hit the maximum amount.
48 Unknown TELNET option specified.
49 Malformed telnet option.
51 The peer’s SSL certificate or SSH MD5 fingerprint was not ok.
52 The server didn’t reply anything, which here is considered an error.
53 SSL crypto engine not found.
54 Cannot set SSL crypto engine as default.
55 Failed sending network data.
56 Failure in receiving network data.
58 Problem with the local certificate.
59 Couldn’t use specified SSL cipher.
60 Peer certificate cannot be authenticated with known CA certificates.
61 Unrecognized transfer encoding.
62 Invalid LDAP URL.
63 Maximum file size exceeded.
64 Requested FTP SSL level failed.
65 Sending the data requires a rewind that failed.
66 Failed to initialize SSL Engine.
67 The user name, password, or similar was not accepted and curl failed to log in.
68 File not found on TFTP server.
69 Permission problem on TFTP server.
70 Out of disk space on TFTP server.
71 Illegal TFTP operation.
72 Unknown TFTP transfer ID.
73 File already exists (TFTP).
74 No such user (TFTP).
75 Character conversion failed.
76 Character conversion functions required.
77 Problem with reading the SSL CA cert (path? access rights?).
78 The resource referenced in the URL does not exist.
79 An unspecified error occurred during the SSH session.
80 Failed to shut down the SSL connection.
82 Could not load CRL file, missing or wrong format (added in 7.19.0).
83 Issuer check failed (added in 7.19.0).
XX More error codes will appear here in future releases. The existing ones are meant to never change.

6、 ... and 、 Usage demonstration :

To save space , Some operations are no longer pasted with execution results .

1、 View source code of webpage

Directly in curl Add the web address after the order , You can see the source code of the web page . We use the website www.sina.com For example ( Choose the website , Mainly because of its short web code ):

[email protected]:/home/peng# curl www.sohu.com<html><head><title>307 Temporary Redirect</title></head><body bgcolor="white"><center><h1>307 Temporary Redirect</h1></center><hr><center>nginx</center></body></html>

Execution results show 307 Temporary Redirect, The URL needs to be redirected .

If you want to save this page , have access to -o Parameters , This is equivalent to using wget The command .

curl -o [ file name ] www.sohu.com

2、 Automatic jump

Some URLs jump automatically . Use -L Parameters ,curl Will jump to the new website .

curl -L www.sohu.com

Type the command above , The result automatically jumps to www.sohu.com.cn.

3、 Display header information

-i Parameters can show http response The header information , Along with the web code .

[email protected]:/home/peng/driver/test#  curl -i www.sohu.comHTTP/1.1 307 Temporary Redirect
Content-Type: text/html
Content-Length: 180
Connection: keep-alive
Server: nginx
Date: Tue, 25 Aug 2020 10:10:54 GMT
Location: https://www.sohu.com/
FSS-Cache: from 9790436.18244590.10468709
FSS-Proxy: Powered by 2384755.3433341.3062915<html><head><title>307 Temporary Redirect</title></head><body bgcolor="white"><center><h1>307 Temporary Redirect</h1></center><hr><center>nginx</center></body></html>

-I The parameter shows only http response The header information .

4、 Show communication process

-v Parameters can be displayed once http The whole process of communication , Includes port connections and http request Header information .

[email protected]:/home/peng/driver/test#  curl -v www.sohu.com* About to connect() to www.sohu.com port 80 (#0)*   Trying 240e:83:201:3700::5... connected> GET / HTTP/1.1> User-Agent: curl/7.22.0 (i686-pc-linux-gnu) libcurl/7.22.0 OpenSSL/1.0.1 zlib/1.2.3.4 libidn/1.23 librtmp/2.3> Host: www.sohu.com> Accept: */*> < HTTP/1.1 307 Temporary Redirect< Content-Type: text/html< Content-Length: 180< Connection: keep-alive< Server: nginx< Date: Tue, 25 Aug 2020 10:11:49 GMT< Location: https://www.sohu.com/< FSS-Cache: from 9855973.18375663.10534247< FSS-Proxy: Powered by 2450292.3564414.3128453< <html><head><title>307 Temporary Redirect</title></head><body bgcolor="white"><center><h1>307 Temporary Redirect</h1></center><hr><center>nginx</center></body></html>* Connection #0 to host www.sohu.com left intact* Closing connection #0

If you think the above information is not enough , Then the following command can view the more detailed communication process .

curl --trace output.txt www.sohu.com

perhaps

curl --trace-ascii output.txt www.sohu.com

After operation , Please open the output.txt The file to view .

5、 Send form information

Sending form information includes GET and POST The two methods .GET The method is relatively simple , Just attach the data to the website .

curl example.com/form.cgi?data=xxx

POST The method must separate the data from the web address ,curl You use –data Parameters .

curl -X POST --data "data=xxx" example.com/form.cgi

If your data is not form coded , Can also let curl Code for you , Parameter is --data-urlencode.

curl -X POST--data-urlencode "date=April 1" example.com/form.cgi

6、HTTP Verb

curl default HTTP The verb is GET, Use -X Parameters can support other verbs .

curl -X POST www.example.comcurl -X DELETE www.example.com

7、 Upload files

Let's say the form uploaded by the file is as follows :

<form method="POST" enctype='multipart/form-data' action="upload.cgi"> <input type=file name=upload> <input type=submit name=press value="OK"></form>

You can use it. curl Upload files like this :

curl --form [email protected] --form press=OK [URL]

8、Referer Field

Sometimes you need to be in http request In header information , Provide a referer Field , Where did you jump from .

curl --referer http://www.example.com http://www.example.com

9、User Agent Field

This field is used to represent the device information of the client . The server sometimes uses this field , For different devices , Returns a web page in a different format , For example, mobile version and desktop version .

iPhone4 Of User Agent yes

Mozilla/5.0 (iPhone; U; CPU iPhone OS 4_0 like Mac OS X; en-us) AppleWebKit/532.9 (KHTML, like Gecko) Version/4.0.5 Mobile/8A293 Safari/6531.22.7

curl You can simulate :

curl --user-agent "[User Agent]" [URL]

10、cookie

Use --cookie Parameters , It can make curl send out cookie.

curl --cookie "name=xxx" www.example.com

As for the concrete cookie Value , It can be downloaded from http response Head message Set-Cookie Field to get .

-c cookie-file You can save the... Returned by the server cookie To the file ,-b cookie-file You can use this file as cookie Information , Make subsequent requests .

curl -c cookies http://example.comcurl -b cookies http://example.com

11、 Add header information

Sometimes it needs to be in http request In , Add a header of your own .--header Parameters can do this .

$ curl --header "Content-Type:application/json" http://example.com

12、 authentication

Use curl Options -u Can finish HTTP perhaps FTP Certification of , You can specify a password , You can also enter the password in subsequent operations without specifying a password :

curl -u user:pwd http://man.linuxde.netcurl -u user http://man.linuxde.net

13、FTP

1)、 List ftp List of directories on the server

curl ftp://www.xxx.com/ --user name:passwdcurl ftp://www.xxx.com/ –u name:passwd # Simple writing curl ftp://name:[email protected] # Simple writing 2

for example : stay IP Address 192.168.43.117 To build FTP The server , And set the user name to user, The password for 123456
Now we want to display all the file information in the root directory of the server , The order is as follows :

curl -u user:123456 ftp://192.168.43.117

The results are as follows :
 Insert picture description here
Simple writing :

curl ftp://user:[email protected]

The results are as follows :
 Insert picture description here

2)、 List only directories , Don't show progress bar

curl ftp://www.xxx.com –u name:passwd -s

3)、 Download a file :

Format

curl ftp://www.xxx.com/size.zip –u name:passwd -o size.zip

Examples are as follows :
Download the file from the root of the server test.c, Save to local , The local file name is also test.c.
【 Be careful 】 without -o Options , The program will direct the data stream to stdout, That is to directly display the file content to the terminal .

curl ftp://user:[email protected]/test.c -o test.c

The results are as follows :
 Insert picture description here
Simple mode

curl -u user:123456 ftp://192.168.43.117/list.h -o list.h

The results are as follows :
 Insert picture description here

4)、 Upload a file :

curl –u name:passwd -T size.mp3 ftp://www.xxx.com/mp3/

Examples are as follows :

curl -u user:123456 ftp://192.168.43.117/ -T list.h

 Insert picture description here
You can see that the file was not uploaded successfully , The return error code is 25, Refer to Chapter 5

25 FTP couldn't STOR file. The server denied the STOR operation, used for FTP uploading.

You know , It's because the server doesn't have permission to store , So set the server's write Permissions can be .

 Insert picture description here

5)、 Delete files from the server ( Use curl Pass on ftp Agreed DELE command ):

curl –u name:passwd ftp://www.xxx.com/ -X 'DELE mp3/size.mp3'

6)、 in addition curl Recursive download is not supported , But you can download files in array mode , For example, we need to download 1-10.gif Serially named files :**

curl –u name:passwd ftp://www.xxx.com/img/[1-10].gif –O #O Capital letters 

7)、 To download multiple files in succession :

curl –u name:passwd ftp://www.xxx.com/img/[one,two,three].jpg –O #O Capital letters 

6、 ... and 、 Realize the regular upload of log files

function

  1. When the program is running, the last modification time of the current log file should be recorded ;
  2. Every 10 Check it in seconds log Whether the document has been modified , Sleep if not modified 10 Second ;
  3. If log The document has been modified , Copy the current log file into a backup file , Backup file name plus current time ;
  4. adopt curl Send to ftp The server ;
  5. Delete backup file , Repeat step 2.

The flow chart of the program is as follows :

 Insert picture description here

Function function introduction

init()

First record the current log File time , And record the global variables last_mtime in .

check_file_change()

Read file last modified time , And on and on last_mtime Compare , If it's the same, go back to 0, If it's different, go back 1.

file_name_add_time()

Copy the current log file into a backup file , Backup file name plus current time .

stat()

Get the attribute information of the corresponding file , Store in struct stat In structural variables .

system()

Execute the command corresponding to the string in the parameter

The code is as follows :

/*   Copyright (C)    official account : yikoulinux  */#include <sys/stat.h>#include <stdio.h>#include <stdlib.h>#include <unistd.h>#include <time.h>
 typedef struct stat ST;unsigned long last_mtime;/* The user name and password are temporarily dead , It should actually be saved in the configuration file */char name[32]="user";char pass[32] ="123456";char ip[32]     ="192.168.43.117";char filename[32]="t.log";char dstfile[256]  ={0};int init(void){// Prepare the structure
ST status;// call stat function int res = stat(filename,&status);if(-1 == res){perror("error:open file fail\n");return 0;}
last_mtime = status.st_mtime;printf("init time:%s \n",ctime(&last_mtime));return 1;}int  check_file_change(void){// Prepare the structure
ST status;// call stat function int res = stat(filename,&status);if(-1 == res){perror("error:open file fail\n");return 0;}// printf("old:%s new:%s",ctime(&last_mtime),ctime(&status.st_mtime));if(last_mtime == status.st_mtime){printf("file not change\n");return 0;}else{printf("file updated\n");
last_mtime = status.st_mtime;return 1;}}void file_name_add_time(void){
ST status;
time_t t; 
struct tm *tblock;char cmd[1024]={0};
t = time(NULL);
tblock = localtime(&t);sprintf(dstfile,"t-%d-%d-%d-%d-%d-%d.log",
tblock->tm_year+1900,
tblock->tm_mon,
tblock->tm_mday,
tblock->tm_hour,
tblock->tm_min,
tblock->tm_sec);sprintf(cmd,"cp %s %s",filename,dstfile);// printf("cdm=%s\n",cmd);system(cmd);}int main(void){char cmd[1024]={0};init();while(1){if(check_file_change() == 1){file_name_add_time();sprintf(cmd,"curl -u %s:%s ftp://%s/ -T %s",name,pass,ip,dstfile);// printf("cdm=%s\n",cmd);system(cmd);unlink(dstfile);}sleep(10);}}

Run a screenshot :

First step :
 Insert picture description here
because log The document has not been modified , So the program won't upload .

The second step :
Manually enter the string yikoulinux To log file t.log in .
 Insert picture description here The third step :
Because the file has changed , So print “file updated”, And you can see that curl Of uploaded files log Information .
 Insert picture description here Here are FTP The root directory of the server , You can see , Uploaded log files :t-2020-7-26-1-19-45.log.
 Insert picture description here

【 Add 】

  1. Configuration information , Write dead directly in the code , Usually it should be read from the configuration file , For the convenience of readers , This code does not add this function ;
  2. FTP Server setup , This article does not explain , There are many related documents , You can search for , A mouthful of you is File zilla;
     Insert picture description here
  3. Usually this kind of program needs a long time to run , Need to be set as a daemon , This article does not add the corresponding function , Readers can search on their own . If you have a strong demand, you can write a detailed introduction .
  4. In the code time Management function of , Please search for related articles by yourself .
  5. curl Also provides the related function library curl.lib, If you want to achieve more flexible functions, you can use the corresponding api.
  6. The reason for copying files into backup files first , The main consideration is that other modules may modify the log file at any time , Play a protective role .