常用libcurl异步使用方法

作者: mfdalf | 来源:发表于2020-07-07 22:16 被阅读0次

正文

1 背景知识:

1.1 基本网络通信cs模式,select 框架,网上例子很多.下面只介绍epoll的难点.其他内容请自行搜索.

1.2 epoll 用法

1.2.1 基础知识:

epoll in/out 与socket io 缓冲区关系？
socket 可读可写是指io 缓冲区的情况,这层由内核控制.socket io 对应的epoll in/out 是应用层.
epoll 通常采用ET模型.ET触发方式是指当fd到状态发生变化时通知,比如:read buffer从无到有,write buffer从满到不满才会通知.
所以用while 不停的读,read完缓冲区.下次来数据的时候,缓冲区又是由无到有,又会触发epoll in.
while 不停的写直到返回缓冲区满返回eagain,然后os立刻会发送,“发送缓冲区的”数据.
“发送缓冲区的”由满变成不满,再次触发epoll out.进入epoll out分支,开始下一轮while写.
如果没写满,但是也写完了,不会再次进入epoll out分支.
epoll out的使用时机?

在自己端准备write之前,通过epoll ctrl设置成epoll out.
epoll in 是被动监听接收,epoll out是主动设置.

1.2.2 epoll 模型：网上例子很多.

2 libcurl 基础知识

libcurl作用:它只能做client. 用于文件上传下载和发送http get,post命令.libcurl不是一个简单的api,是一组api实现的模块,有自己的使用steps. 默认curl是get url 网页,与callback write function连动，写入文件。
两种方案:
curl_multi_socket_action()：通常和select/poll/epoll/libev 连用.
curl_multi_perform() + curl_multi_wait().

这个是最简单异步，先发送--等待--接收，这种用法很少用了
https://blog.csdn.net/lijinqi1987/article/details/53925835 and https://blog.csdn.net/Rong_Toa/article/details/105712677.
本文讨论curl_multi_socket_action 方案.

3 libcurl两种模式

**3.1 easy 模式(这种模式比较简单) **

调用curl_easy_setopt函数设置传输的一些基本参数,CULROPT_URL必填.设置完成后,调用curl_easy_perform函数发送数据.
CURLcode curl_easy_setopt(CURL *handle, CURLOPT_WRITEDATA, void *pointer);
curl_easy_setopt(conn->easy, CURLOPT_URL, conn->url);
curl_easy_setopt(conn->easy, CURLOPT_WRITEFUNCTION, write_cb);
curl_easy_setopt(conn->easy, CURLOPT_WRITEDATA, conn);
第一句 curl建立与url 连接，第二句将respond 发送到write_cb,第三句是将write_cb处理内容写入conn指向的文件

3.2 multi 模式

1 ) 常用函数介绍(注意参数)
a ) CURLMOPT_SOCKETFUNCTION:设置socket的回调函数.是所有socket 变化都会调用callback(不仅仅是socket connect，read/write也调用).

int socket_callback(CURL *easy,      /* easy handle */
                    curl_socket_t s, /* socket */
                    int what,        /* describes the socket */
                    void *userp,     /* private callback pointer */
                    void *socketp);  /* private socket pointer */
CURLMOPT_SOCKETDATA: it is params userp

对于接收的话，CURLMOPT_SOCKETDATA没用。socketp是返回值。
CURLMOPT_TIMERDATA and CURLOPT_WRITEDATA 都是对应 curl function的入参.

b) CURLMOPT_TIMERFUNCTION :set callback to receive timeout values

int timer_callback(CURLM *multi,    /* multi handle */
                   long timeout_ms, /* timeout in number of ms */
                   void *userp);    /* private callback pointer */

You can also use the curl_multi_timeout function to poll the value at any given time,
but for an event-based system using the callback is far better than relying on polling the timeout value.
系统超时操作触发callback.
c ) CURLOPT_WRITEFUNCTION:set callback for writing received data

size_t write_callback(char *ptr, size_t size, size_t nmemb, void *userdata);

CURLOPT_WRITEDATA:custom pointer passed to the write callback
设置CURLOPT_WRITEFUNCTION的参数4,把参数1 ptr指向的数据拷到参数4 userdata
d ) curl_multi_setopt

CURLMcode curl_multi_setopt(CURLM *handle, CURLMOPT_TIMERFUNCTION, timer_callback);

CURLMOPT_TIMERDATA:The userp pointer is set with CURLMOPT_TIMERDATA. 入参
curl_multi_setopt is used to tell a libcurl multi handle how to behave.
CURLMOPT_SOCKETFUNCTION,CURLMOPT_SOCKETDATA,CURLMOPT_TIMERFUNCTION,CURLMOPT_TIMERDATA
e) curl_multi_assign

curl_multi_assign(CURLM *multi_handle, curl_socket_t sockfd,   void *sockptr);

set association sockfd with sockptr ;curl系统中将sockfd和sockpt结构体绑定

curl_multi_info_read
Use curl_multi_info_read(3) to figure out which easy handle that completed. 当前只发现这一个功能
注意这个函数不是read data，而是read info ,具体说就是completed flag，是结束检测函数.
重点:

CURLMcode curl_multi_socket_action(CURLM * multi_handle,
                                   curl_socket_t sockfd,
                                   int ev_bitmask,
                                   int *running_handles);

执行curl命令.curl_multi_socket_action=curl_multi_perform：reads/writes available data given an action.通知libcurl读写数据
执行curl_multi_add_handle中的东东. 注意参数3ev_bitmask是action.
比如: 设置了CURLMOPT_SOCKETFUNCTION就从server download file.设置CURLMOPT_UPLOAD就是上传file.
CURLOPT_WRITEFUNCTION 就调用CURLOPT_WRITEDATA将download data写入file.

4 curl实例:

curl同时和socket，epoll，file 同时打交道.
curl精华：curl_multi_socket_action可以自动get file from server(CURLOPT_URL) and 自动write file to local(CURLOPT_WRITEFUNCTION)
4.1 curl+ epoll
CURLOPT_URL=connect,CURLMOPT_SOCKETFUNCTION=epoll_ctrl, curl_multi_socket_action=get remote file and write file to local ,epoll=select.

image

实现步骤和时序:

创建curl multi handle
当timeout的时候,初始化socket
socket变化触发socekt 回调函数, 设置epoll_ctrl 并且将socket和curl绑定.
epoll_wait调用epoll_out 分支,curl_multi_socket_action执行发送.
epoll_in 调用epoll_in 分支,curl_multi_socket_action执行接收.
curl_multi_info_read 判断是否读完. 读完,删除curl multi handle.
代码:

epoll_ctl(curl_epoll_fd, EPOLL_CTL_ADD, curl_timer_fd, &ev);//put timer into epoll
curl_easy_setopt(curl, CURLOPT_URL, request_uri);
curl_easy_setopt(curl, CURLOPT_WRITEFUNCTION, curl_read_http_body);
curl_easy_setopt(curl, CURLOPT_WRITEDATA, relay_session);
CURLMcode rc = curl_multi_add_handle(curl_multi, curl);
curl_multi_setopt(curl_multi, CURLMOPT_TIMERFUNCTION, curl_timer_setup);//call back for timer
curl_multi_setopt(curl_multi, CURLMOPT_SOCKETFUNCTION, curl_sock_setup);//socket fd 有变化,会调用curl_sock_setup 
curl_multi_thread  = g_thread_try_new("curl multi thread", &curl_multi_thread_run, NULL, NULL);

static void* curl_multi_thread_run(void* user_data) {
    int i, nfds;
    struct epoll_event *ev = (struct epoll_event *)malloc(sizeof(struct epoll_event) * MAX_EPOLL_FDS_NUM);
    gint64 start_time, end_time;
    while (!janus_is_stopping()) {
        nfds = epoll_wait(curl_epoll_fd, ev, MAX_EPOLL_FDS_NUM, 3000);//wait 3 seconds
        if (nfds > 0) {
            for (i = 0; i < nfds; i++) {
                curl_event(ev[i].data.fd, ev[i].events);//curl_event//curl_event
            }
        }
    }
}
static void curl_event(int fd, int revents) {
    CURLMcode rc;
    int still_running = -1;
    int action = ((revents & EPOLLIN) ? CURL_CSELECT_IN : 0) |
                 ((revents & EPOLLOUT) ? CURL_CSELECT_OUT : 0);
    //call CURLOPT_WRITEFUNCTION callback;read/write curl data
    rc = curl_multi_socket_action(curl_multi, fd, action, &still_running);
    if (rc == CURLM_OK) {
        check_multi_info();//check result
    }
}
static void check_multi_info(void) {
    CURLMsg *msg = NULL;
    int msgs_left;
    CURL *curl = NULL;
    CURLcode res;
    janus_session* session = NULL;
    json_error_t error;
    json_t *root = NULL;
    //从curl msg queue中读取
    while ((msg = curl_multi_info_read(curl_multi, &msgs_left))) {
        curl = msg->easy_handle;
        res = msg->data.result;
        curl_easy_getinfo(curl, CURLINFO_PRIVATE, &session);//it is from CURLOPT_PRIVATE
        curl_multi_remove_handle(curl_multi, curl);
        curl_easy_cleanup(curl);
    }
}

https://curl.haxx.se/libcurl/c/ephiperfifo.html
[https://www.jianshu.com/p/80274bc54aff]
(https://www.jianshu.com/p/80274bc54aff)
https://blog.csdn.net/Rong_Toa/article/details/105712677
https://blog.csdn.net/lijinqi1987/article/details/53996129
https://www.cnblogs.com/zl-graduate/articles/6724446.html

网友评论

本文标题：常用libcurl异步使用方法

本文链接：https://www.haomeiwen.com/subject/uyktcktx.html

延伸阅读

深度阅读

您也可以注册成为美文阅读网的作者，发表您的原创作品、分享您的心情！