Commit fe426157 authored by Caleb Raitto's avatar Caleb Raitto Committed by Commit Bot

Add libcurl headers and examples to //components/cronet/third_party.

This doesn't include the libcurl implementation -- just headers and
samples to build an alternative implementation of the libcurl API.

The full libcurl repo with its implementation and repo history would be
nice to have, but not strictly necessary.

Since this code isn't intended to be shipped with Chrome, but rather as
headers for an alternative API for the components/cronet library, it
makes sense to avoid having Chromium developers sync the full library.

Bug: 973603
Change-Id: I3879241ab7b2aa0177cd09cc2e8ef73739fd5feb
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/src/+/1664790Reviewed-by: default avatarJochen Eisinger <jochen@chromium.org>
Reviewed-by: default avatarPaul Jensen <pauljensen@chromium.org>
Reviewed-by: default avatarChris Palmer <palmer@chromium.org>
Commit-Queue: Caleb Raitto <caraitto@chromium.org>
Cr-Commit-Position: refs/heads/master@{#672184}
parent 3c79ae68
COPYRIGHT AND PERMISSION NOTICE
Copyright (c) 1996 - 2019, Daniel Stenberg, <daniel@haxx.se>, and many
contributors, see the THANKS file.
All rights reserved.
Permission to use, copy, modify, and distribute this software for any purpose
with or without fee is hereby granted, provided that the above copyright
notice and this permission notice appear in all copies.
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT OF THIRD PARTY RIGHTS. IN
NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM,
DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR
OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE
OR OTHER DEALINGS IN THE SOFTWARE.
Except as contained in this notice, the name of a copyright holder shall not
be used in advertising or otherwise to promote the sale, use or other dealings
in this Software without prior written authorization of the copyright holder.
COPYRIGHT AND PERMISSION NOTICE
Copyright (c) 1996 - 2019, Daniel Stenberg, <daniel@haxx.se>, and many
contributors, see the THANKS file.
All rights reserved.
Permission to use, copy, modify, and distribute this software for any purpose
with or without fee is hereby granted, provided that the above copyright
notice and this permission notice appear in all copies.
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT OF THIRD PARTY RIGHTS. IN
NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM,
DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR
OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE
OR OTHER DEALINGS IN THE SOFTWARE.
Except as contained in this notice, the name of a copyright holder shall not
be used in advertising or otherwise to promote the sale, use or other dealings
in this Software without prior written authorization of the copyright holder.
pauljensen@chromium.org
caraitto@chromium.org
# Component: Internals>Network>Library
Name: libcurl (headers and examples only)
Short Name: libcurl
URL: https://github.com/curl/curl
Version: 7.65.1
Revision: 69248b58f649e35b09a126c12781353e3471f5c6
License: MIT/X11-like
License File: NOT_SHIPPED
Security Critical: no
Description:
Headers and examples from the libcurl repository. Useful for creating
alternative implementations of the libcurl API.
Local Modifications:
Only includes COPYING, include/, and docs/examples/ -- all other files and
directories from the original repository are not included -- DOES NOT contain
the implementation of the libcurl API.
10-at-a-time
anyauthput
certinfo
chkspeed
cookie_interface
debug
externalsocket
fileupload
fopen
ftp-wildcard
ftpget
ftpgetinfo
ftpgetresp
ftpsget
ftpupload
ftpuploadfrommem
ftpuploadresume
getinfo
getinmemory
getredirect
http-post
http2-download
http2-pushinmemory
http2-serverpush
http2-upload
httpcustomheader
httpput
https
imap
imap-append
imap-copy
imap-create
imap-delete
imap-examine
imap-fetch
imap-list
imap-lsub
imap-multi
imap-noop
imap-search
imap-ssl
imap-store
imap-tls
multi-app
multi-debugcallback
multi-double
multi-formadd
multi-post
multi-single
parseurl
persistent
pop3-dele
pop3-list
pop3-multi
pop3-noop
pop3-retr
pop3-ssl
pop3-stat
pop3-tls
pop3-top
pop3-uidl
pop3s
pop3slist
post-callback
postinmemory
postit2
postit2-formadd
progressfunc
resolve
rtsp
sendrecv
sepheaders
sftpget
sftpuploadresume
shared-connection-cache
simple
simplepost
simplesmtp
simplessl
smtp-expn
smtp-mail
smtp-mime
smtp-multi
smtp-ssl
smtp-tls
smtp-vrfy
sslbackend
urlapi
url2file
usercertinmem
xmlstream
/***************************************************************************
* _ _ ____ _
* Project ___| | | | _ \| |
* / __| | | | |_) | |
* | (__| |_| | _ <| |___
* \___|\___/|_| \_\_____|
*
* Copyright (C) 1998 - 2019, Daniel Stenberg, <daniel@haxx.se>, et al.
*
* This software is licensed as described in the file COPYING, which
* you should have received as part of this distribution. The terms
* are also available at https://curl.haxx.se/docs/copyright.html.
*
* You may opt to use, copy, modify, merge, publish, distribute and/or sell
* copies of the Software, and permit persons to whom the Software is
* furnished to do so, under the terms of the COPYING file.
*
* This software is distributed on an "AS IS" basis, WITHOUT WARRANTY OF ANY
* KIND, either express or implied.
*
***************************************************************************/
/* <DESC>
* Download many files in parallel, in the same thread.
* </DESC>
*/
#include <errno.h>
#include <stdlib.h>
#include <string.h>
#ifndef WIN32
# include <unistd.h>
#endif
#include <curl/curl.h>
static const char *urls[] = {
"https://www.microsoft.com",
"https://opensource.org",
"https://www.google.com",
"https://www.yahoo.com",
"https://www.ibm.com",
"https://www.mysql.com",
"https://www.oracle.com",
"https://www.ripe.net",
"https://www.iana.org",
"https://www.amazon.com",
"https://www.netcraft.com",
"https://www.heise.de",
"https://www.chip.de",
"https://www.ca.com",
"https://www.cnet.com",
"https://www.mozilla.org",
"https://www.cnn.com",
"https://www.wikipedia.org",
"https://www.dell.com",
"https://www.hp.com",
"https://www.cert.org",
"https://www.mit.edu",
"https://www.nist.gov",
"https://www.ebay.com",
"https://www.playstation.com",
"https://www.uefa.com",
"https://www.ieee.org",
"https://www.apple.com",
"https://www.symantec.com",
"https://www.zdnet.com",
"https://www.fujitsu.com/global/",
"https://www.supermicro.com",
"https://www.hotmail.com",
"https://www.ietf.org",
"https://www.bbc.co.uk",
"https://news.google.com",
"https://www.foxnews.com",
"https://www.msn.com",
"https://www.wired.com",
"https://www.sky.com",
"https://www.usatoday.com",
"https://www.cbs.com",
"https://www.nbc.com/",
"https://slashdot.org",
"https://www.informationweek.com",
"https://apache.org",
"https://www.un.org",
};
#define MAX_PARALLEL 10 /* number of simultaneous transfers */
#define NUM_URLS sizeof(urls)/sizeof(char *)
static size_t write_cb(char *data, size_t n, size_t l, void *userp)
{
/* take care of the data here, ignored in this example */
(void)data;
(void)userp;
return n*l;
}
static void add_transfer(CURLM *cm, int i)
{
CURL *eh = curl_easy_init();
curl_easy_setopt(eh, CURLOPT_WRITEFUNCTION, write_cb);
curl_easy_setopt(eh, CURLOPT_URL, urls[i]);
curl_easy_setopt(eh, CURLOPT_PRIVATE, urls[i]);
curl_multi_add_handle(cm, eh);
}
int main(void)
{
CURLM *cm;
CURLMsg *msg;
unsigned int transfers = 0;
int msgs_left = -1;
int still_alive = 1;
curl_global_init(CURL_GLOBAL_ALL);
cm = curl_multi_init();
/* Limit the amount of simultaneous connections curl should allow: */
curl_multi_setopt(cm, CURLMOPT_MAXCONNECTS, (long)MAX_PARALLEL);
for(transfers = 0; transfers < MAX_PARALLEL; transfers++)
add_transfer(cm, transfers);
do {
curl_multi_perform(cm, &still_alive);
while((msg = curl_multi_info_read(cm, &msgs_left))) {
if(msg->msg == CURLMSG_DONE) {
char *url;
CURL *e = msg->easy_handle;
curl_easy_getinfo(msg->easy_handle, CURLINFO_PRIVATE, &url);
fprintf(stderr, "R: %d - %s <%s>\n",
msg->data.result, curl_easy_strerror(msg->data.result), url);
curl_multi_remove_handle(cm, e);
curl_easy_cleanup(e);
}
else {
fprintf(stderr, "E: CURLMsg (%d)\n", msg->msg);
}
if(transfers < NUM_URLS)
add_transfer(cm, transfers++);
}
if(still_alive)
curl_multi_wait(cm, NULL, 0, 1000, NULL);
} while(still_alive || (transfers < NUM_URLS));
curl_multi_cleanup(cm);
curl_global_cleanup();
return EXIT_SUCCESS;
}
#***************************************************************************
# _ _ ____ _
# Project ___| | | | _ \| |
# / __| | | | |_) | |
# | (__| |_| | _ <| |___
# \___|\___/|_| \_\_____|
#
# Copyright (C) 1998 - 2019, Daniel Stenberg, <daniel@haxx.se>, et al.
#
# This software is licensed as described in the file COPYING, which
# you should have received as part of this distribution. The terms
# are also available at https://curl.haxx.se/docs/copyright.html.
#
# You may opt to use, copy, modify, merge, publish, distribute and/or sell
# copies of the Software, and permit persons to whom the Software is
# furnished to do so, under the terms of the COPYING file.
#
# This software is distributed on an "AS IS" basis, WITHOUT WARRANTY OF ANY
# KIND, either express or implied.
#
###########################################################################
AUTOMAKE_OPTIONS = foreign nostdinc
EXTRA_DIST = README Makefile.example Makefile.inc Makefile.m32 \
Makefile.netware makefile.dj $(COMPLICATED_EXAMPLES)
# Specify our include paths here, and do it relative to $(top_srcdir) and
# $(top_builddir), to ensure that these paths which belong to the library
# being currently built and tested are searched before the library which
# might possibly already be installed in the system.
#
# $(top_srcdir)/include is for libcurl's external include files
AM_CPPFLAGS = -I$(top_srcdir)/include
LIBDIR = $(top_builddir)/lib
# Avoid libcurl obsolete stuff
AM_CPPFLAGS += -DCURL_NO_OLDIES
if USE_CPPFLAG_CURL_STATICLIB
AM_CPPFLAGS += -DCURL_STATICLIB
endif
# Prevent LIBS from being used for all link targets
LIBS = $(BLANK_AT_MAKETIME)
# Dependencies
if USE_EXPLICIT_LIB_DEPS
LDADD = $(LIBDIR)/libcurl.la @LIBCURL_LIBS@
else
LDADD = $(LIBDIR)/libcurl.la
endif
# This might hold -Werror
CFLAGS += @CURL_CFLAG_EXTRAS@
# Makefile.inc provides the check_PROGRAMS and COMPLICATED_EXAMPLES defines
include Makefile.inc
all: $(check_PROGRAMS)
CHECKSRC = $(CS_$(V))
CS_0 = @echo " RUN " $@;
CS_1 =
CS_ = $(CS_0)
checksrc:
$(CHECKSRC)(@PERL@ $(top_srcdir)/lib/checksrc.pl -ASNPRINTF $(srcdir)/*.c)
#***************************************************************************
# _ _ ____ _
# Project ___| | | | _ \| |
# / __| | | | |_) | |
# | (__| |_| | _ <| |___
# \___|\___/|_| \_\_____|
#
# Copyright (C) 1998 - 2011, Daniel Stenberg, <daniel@haxx.se>, et al.
#
# This software is licensed as described in the file COPYING, which
# you should have received as part of this distribution. The terms
# are also available at https://curl.haxx.se/docs/copyright.html.
#
# You may opt to use, copy, modify, merge, publish, distribute and/or sell
# copies of the Software, and permit persons to whom the Software is
# furnished to do so, under the terms of the COPYING file.
#
# This software is distributed on an "AS IS" basis, WITHOUT WARRANTY OF ANY
# KIND, either express or implied.
#
###########################################################################
# What to call the final executable
TARGET = example
# Which object files that the executable consists of
OBJS= ftpget.o
# What compiler to use
CC = gcc
# Compiler flags, -g for debug, -c to make an object file
CFLAGS = -c -g
# This should point to a directory that holds libcurl, if it isn't
# in the system's standard lib dir
# We also set a -L to include the directory where we have the openssl
# libraries
LDFLAGS = -L/home/dast/lib -L/usr/local/ssl/lib
# We need -lcurl for the curl stuff
# We need -lsocket and -lnsl when on Solaris
# We need -lssl and -lcrypto when using libcurl with SSL support
# We need -lpthread for the pthread example
LIBS = -lcurl -lsocket -lnsl -lssl -lcrypto
# Link the target with all objects and libraries
$(TARGET) : $(OBJS)
$(CC) -o $(TARGET) $(OBJS) $(LDFLAGS) $(LIBS)
# Compile the source files into object files
ftpget.o : ftpget.c
$(CC) $(CFLAGS) $<
#***************************************************************************
# _ _ ____ _
# Project ___| | | | _ \| |
# / __| | | | |_) | |
# | (__| |_| | _ <| |___
# \___|\___/|_| \_\_____|
#
# Copyright (C) 1998 - 2018, Daniel Stenberg, <daniel@haxx.se>, et al.
#
# This software is licensed as described in the file COPYING, which
# you should have received as part of this distribution. The terms
# are also available at https://curl.haxx.se/docs/copyright.html.
#
# You may opt to use, copy, modify, merge, publish, distribute and/or sell
# copies of the Software, and permit persons to whom the Software is
# furnished to do so, under the terms of the COPYING file.
#
# This software is distributed on an "AS IS" basis, WITHOUT WARRANTY OF ANY
# KIND, either express or implied.
#
###########################################################################
# These are all libcurl example programs to be test compiled
check_PROGRAMS = 10-at-a-time anyauthput cookie_interface debug fileupload \
fopen ftpget ftpgetresp ftpupload getinfo getinmemory http-post httpput \
https multi-app multi-debugcallback multi-double multi-post multi-single \
persistent post-callback postit2 sepheaders simple simplepost simplessl \
sendrecv httpcustomheader certinfo chkspeed ftpgetinfo ftp-wildcard \
smtp-mail smtp-mime smtp-multi smtp-ssl smtp-tls smtp-vrfy smtp-expn \
rtsp externalsocket resolve progressfunc pop3-retr pop3-list pop3-uidl \
pop3-dele pop3-top pop3-stat pop3-noop pop3-ssl pop3-tls pop3-multi \
imap-list imap-lsub imap-fetch imap-store imap-append imap-examine \
imap-search imap-create imap-delete imap-copy imap-noop imap-ssl \
imap-tls imap-multi url2file sftpget ftpsget postinmemory http2-download \
http2-upload http2-serverpush getredirect ftpuploadfrommem \
ftpuploadresume sslbackend postit2-formadd multi-formadd \
shared-connection-cache sftpuploadresume http2-pushinmemory parseurl \
urlapi
# These examples require external dependencies that may not be commonly
# available on POSIX systems, so don't bother attempting to compile them here.
COMPLICATED_EXAMPLES = curlgtk.c curlx.c htmltitle.cpp cacertinmem.c \
ghiper.c hiperfifo.c htmltidy.c multithread.c opensslthreadlock.c \
sampleconv.c synctime.c threaded-ssl.c evhiperfifo.c \
smooth-gtk-thread.c version-check.pl href_extractor.c asiohiper.cpp \
multi-uv.c xmlstream.c usercertinmem.c sessioninfo.c \
threaded-shared-conn.c crawler.c ephiperfifo.c
_ _ ____ _
___| | | | _ \| |
/ __| | | | |_) | |
| (__| |_| | _ <| |___
\___|\___/|_| \_\_____|
This directory is for libcurl programming examples. They are meant to show
some simple steps on how you can build your own application to take full
advantage of libcurl.
If you end up with other small but still useful example sources, please mail
them for submission in future packages and on the web site.
BUILDING
The Makefile.example is an example makefile that could be used to build these
examples. Just edit the file according to your system and requirements first.
Most examples should build fine using a command line like this:
$ `curl-config --cc --cflags --libs` -o example example.c
Some compilers don't like having the arguments in this order but instead
want you do reorganize them like:
$ `curl-config --cc` -o example example.c `curl-config --cflags --libs`
*PLEASE* do not use the curl.haxx.se site as a test target for your libcurl
applications/experiments. Even if some of the examples use that site as a URL
at some places, it doesn't mean that the URLs work or that we expect you to
actually torture our web site with your tests! Thanks.
EXAMPLES
Each example source code file is designed to be and work stand-alone and
rather self-explanatory. The examples may at times lack the level of error
checks you need in a real world, but that is then only for the sake of
readability: to make the code smaller and easier to follow.
#!/usr/bin/env perl
# pass files as argument(s)
my $docroot="https://curl.haxx.se/libcurl/c";
for $f (@ARGV) {
open(NEW, ">$f.new");
open(F, "<$f");
while(<F>) {
my $l = $_;
if($l =~ /\/* $docroot/) {
# just ignore preciously added refs
}
elsif($l =~ /^( *).*curl_easy_setopt\([^,]*, *([^ ,]*) *,/) {
my ($prefix, $anc) = ($1, $2);
$anc =~ s/_//g;
print NEW "$prefix/* $docroot/curl_easy_setopt.html#$anc */\n";
print NEW $l;
}
elsif($l =~ /^( *).*(curl_([^\(]*))\(/) {
my ($prefix, $func) = ($1, $2);
print NEW "$prefix/* $docroot/$func.html */\n";
print NEW $l;
}
else {
print NEW $l;
}
}
close(F);
close(NEW);
system("mv $f $f.org");
system("mv $f.new $f");
}
/***************************************************************************
* _ _ ____ _
* Project ___| | | | _ \| |
* / __| | | | |_) | |
* | (__| |_| | _ <| |___
* \___|\___/|_| \_\_____|
*
* Copyright (C) 1998 - 2017, Daniel Stenberg, <daniel@haxx.se>, et al.
*
* This software is licensed as described in the file COPYING, which
* you should have received as part of this distribution. The terms
* are also available at https://curl.haxx.se/docs/copyright.html.
*
* You may opt to use, copy, modify, merge, publish, distribute and/or sell
* copies of the Software, and permit persons to whom the Software is
* furnished to do so, under the terms of the COPYING file.
*
* This software is distributed on an "AS IS" basis, WITHOUT WARRANTY OF ANY
* KIND, either express or implied.
*
***************************************************************************/
/* <DESC>
* HTTP PUT upload with authentiction using "any" method. libcurl picks the
* one the server supports/wants.
* </DESC>
*/
#include <stdio.h>
#include <fcntl.h>
#include <sys/types.h>
#include <sys/stat.h>
#include <curl/curl.h>
#ifdef WIN32
# include <io.h>
# define READ_3RD_ARG unsigned int
#else
# include <unistd.h>
# define READ_3RD_ARG size_t
#endif
#if LIBCURL_VERSION_NUM < 0x070c03
#error "upgrade your libcurl to no less than 7.12.3"
#endif
/*
* This example shows a HTTP PUT operation with authentiction using "any"
* type. It PUTs a file given as a command line argument to the URL also given
* on the command line.
*
* Since libcurl 7.12.3, using "any" auth and POST/PUT requires a set ioctl
* function.
*
* This example also uses its own read callback.
*/
/* ioctl callback function */
static curlioerr my_ioctl(CURL *handle, curliocmd cmd, void *userp)
{
int *fdp = (int *)userp;
int fd = *fdp;
(void)handle; /* not used in here */
switch(cmd) {
case CURLIOCMD_RESTARTREAD:
/* mr libcurl kindly asks as to rewind the read data stream to start */
if(-1 == lseek(fd, 0, SEEK_SET))
/* couldn't rewind */
return CURLIOE_FAILRESTART;
break;
default: /* ignore unknown commands */
return CURLIOE_UNKNOWNCMD;
}
return CURLIOE_OK; /* success! */
}
/* read callback function, fread() look alike */
static size_t read_callback(void *ptr, size_t size, size_t nmemb, void *stream)
{
ssize_t retcode;
curl_off_t nread;
int *fdp = (int *)stream;
int fd = *fdp;
retcode = read(fd, ptr, (READ_3RD_ARG)(size * nmemb));
nread = (curl_off_t)retcode;
fprintf(stderr, "*** We read %" CURL_FORMAT_CURL_OFF_T
" bytes from file\n", nread);
return retcode;
}
int main(int argc, char **argv)
{
CURL *curl;
CURLcode res;
int hd;
struct stat file_info;
char *file;
char *url;
if(argc < 3)
return 1;
file = argv[1];
url = argv[2];
/* get the file size of the local file */
hd = open(file, O_RDONLY);
fstat(hd, &file_info);
/* In windows, this will init the winsock stuff */
curl_global_init(CURL_GLOBAL_ALL);
/* get a curl handle */
curl = curl_easy_init();
if(curl) {
/* we want to use our own read function */
curl_easy_setopt(curl, CURLOPT_READFUNCTION, read_callback);
/* which file to upload */
curl_easy_setopt(curl, CURLOPT_READDATA, (void *)&hd);
/* set the ioctl function */
curl_easy_setopt(curl, CURLOPT_IOCTLFUNCTION, my_ioctl);
/* pass the file descriptor to the ioctl callback as well */
curl_easy_setopt(curl, CURLOPT_IOCTLDATA, (void *)&hd);
/* enable "uploading" (which means PUT when doing HTTP) */
curl_easy_setopt(curl, CURLOPT_UPLOAD, 1L);
/* specify target URL, and note that this URL should also include a file
name, not only a directory (as you can do with GTP uploads) */
curl_easy_setopt(curl, CURLOPT_URL, url);
/* and give the size of the upload, this supports large file sizes
on systems that have general support for it */
curl_easy_setopt(curl, CURLOPT_INFILESIZE_LARGE,
(curl_off_t)file_info.st_size);
/* tell libcurl we can use "any" auth, which lets the lib pick one, but it
also costs one extra round-trip and possibly sending of all the PUT
data twice!!! */
curl_easy_setopt(curl, CURLOPT_HTTPAUTH, (long)CURLAUTH_ANY);
/* set user name and password for the authentication */
curl_easy_setopt(curl, CURLOPT_USERPWD, "user:password");
/* Now run off and do what you've been told! */
res = curl_easy_perform(curl);
/* Check for errors */
if(res != CURLE_OK)
fprintf(stderr, "curl_easy_perform() failed: %s\n",
curl_easy_strerror(res));
/* always cleanup */
curl_easy_cleanup(curl);
}
close(hd); /* close the local file */
curl_global_cleanup();
return 0;
}
/***************************************************************************
* _ _ ____ _
* Project ___| | | | _ \| |
* / __| | | | |_) | |
* | (__| |_| | _ <| |___
* \___|\___/|_| \_\_____|
*
* Copyright (C) 1998 - 2019, Daniel Stenberg, <daniel@haxx.se>, et al.
*
* This software is licensed as described in the file COPYING, which
* you should have received as part of this distribution. The terms
* are also available at https://curl.haxx.se/docs/copyright.html.
*
* You may opt to use, copy, modify, merge, publish, distribute and/or sell
* copies of the Software, and permit persons to whom the Software is
* furnished to do so, under the terms of the COPYING file.
*
* This software is distributed on an "AS IS" basis, WITHOUT WARRANTY OF ANY
* KIND, either express or implied.
*
***************************************************************************/
/* <DESC>
* CA cert in memory with OpenSSL to get a HTTPS page.
* </DESC>
*/
#include <openssl/err.h>
#include <openssl/ssl.h>
#include <curl/curl.h>
#include <stdio.h>
static size_t writefunction(void *ptr, size_t size, size_t nmemb, void *stream)
{
fwrite(ptr, size, nmemb, (FILE *)stream);
return (nmemb*size);
}
static CURLcode sslctx_function(CURL *curl, void *sslctx, void *parm)
{
CURLcode rv = CURLE_ABORTED_BY_CALLBACK;
/** This example uses two (fake) certificates **/
static const char mypem[] =
"-----BEGIN CERTIFICATE-----\n"
"MIIH0zCCBbugAwIBAgIIXsO3pkN/pOAwDQYJKoZIhvcNAQEFBQAwQjESMBAGA1UE\n"
"AwwJQUNDVlJBSVoxMRAwDgYDVQQLDAdQS0lBQ0NWMQ0wCwYDVQQKDARBQ0NWMQsw\n"
"CQYDVQQGEwJFUzAeFw0xMTA1MDUwOTM3MzdaFw0zMDEyMzEwOTM3MzdaMEIxEjAQ\n"
"BgNVBAMMCUFDQ1ZSQUlaMTEQMA4GA1UECwwHUEtJQUNDVjENMAsGA1UECgwEQUND\n"
"VjELMAkGA1UEBhMCRVMwggIiMA0GCSqGSIb3DQEBAQUAA4ICDwAwggIKAoICAQCb\n"
"qau/YUqXry+XZpp0X9DZlv3P4uRm7x8fRzPCRKPfmt4ftVTdFXxpNRFvu8gMjmoY\n"
"HtiP2Ra8EEg2XPBjs5BaXCQ316PWywlxufEBcoSwfdtNgM3802/J+Nq2DoLSRYWo\n"
"G2ioPej0RGy9ocLLA76MPhMAhN9KSMDjIgro6TenGEyxCQ0jVn8ETdkXhBilyNpA\n"
"0KIV9VMJcRz/RROE5iZe+OCIHAr8Fraocwa48GOEAqDGWuzndN9wrqODJerWx5eH\n"
"k6fGioozl2A3ED6XPm4pFdahD9GILBKfb6qkxkLrQaLjlUPTAYVtjrs78yM2x/47\n"
"JyCpZET/LtZ1qmxNYEAZSUNUY9rizLpm5U9EelvZaoErQNV/+QEnWCzI7UiRfD+m\n"
"AM/EKXMRNt6GGT6d7hmKG9Ww7Y49nCrADdg9ZuM8Db3VlFzi4qc1GwQA9j9ajepD\n"
"vV+JHanBsMyZ4k0ACtrJJ1vnE5Bc5PUzolVt3OAJTS+xJlsndQAJxGJ3KQhfnlms\n"
"tn6tn1QwIgPBHnFk/vk4CpYY3QIUrCPLBhwepH2NDd4nQeit2hW3sCPdK6jT2iWH\n"
"7ehVRE2I9DZ+hJp4rPcOVkkO1jMl1oRQQmwgEh0q1b688nCBpHBgvgW1m54ERL5h\n"
"I6zppSSMEYCUWqKiuUnSwdzRp+0xESyeGabu4VXhwOrPDYTkF7eifKXeVSUG7szA\n"
"h1xA2syVP1XgNce4hL60Xc16gwFy7ofmXx2utYXGJt/mwZrpHgJHnyqobalbz+xF\n"
"d3+YJ5oyXSrjhO7FmGYvliAd3djDJ9ew+f7Zfc3Qn48LFFhRny+Lwzgt3uiP1o2H\n"
"pPVWQxaZLPSkVrQ0uGE3ycJYgBugl6H8WY3pEfbRD0tVNEYqi4Y7\n"
"-----END CERTIFICATE-----\n"
"-----BEGIN CERTIFICATE-----\n"
"MIIFtTCCA52gAwIBAgIIYY3HhjsBggUwDQYJKoZIhvcNAQEFBQAwRDEWMBQGA1UE\n"
"AwwNQUNFRElDT00gUm9vdDEMMAoGA1UECwwDUEtJMQ8wDQYDVQQKDAZFRElDT00x\n"
"CzAJBgNVBAYTAkVTMB4XDTA4MDQxODE2MjQyMloXDTI4MDQxMzE2MjQyMlowRDEW\n"
"MBQGA1UEAwwNQUNFRElDT00gUm9vdDEMMAoGA1UECwwDUEtJMQ8wDQYDVQQKDAZF\n"
"RElDT00xCzAJBgNVBAYTAkVTMIICIjANBgkqhkiG9w0BAQEFAAOCAg8AMIICCgKC\n"
"AgEA/5KV4WgGdrQsyFhIyv2AVClVYyT/kGWbEHV7w2rbYgIB8hiGtXxaOLHkWLn7\n"
"09gtn70yN78sFW2+tfQh0hOR2QetAQXW8713zl9CgQr5auODAKgrLlUTY4HKRxx7\n"
"XBZXehuDYAQ6PmXDzQHe3qTWDLqO3tkE7hdWIpuPY/1NFgu3e3eM+SW10W2ZEi5P\n"
"gvoFNTPhNahXwOf9jU8/kzJPeGYDdwdY6ZXIfj7QeQCM8htRM5u8lOk6e25SLTKe\n"
"I6RF+7YuE7CLGLHdztUdp0J/Vb77W7tH1PwkzQSulgUV1qzOMPPKC8W64iLgpq0i\n"
"5ALudBF/TP94HTXa5gI06xgSYXcGCRZj6hitoocf8seACQl1ThCojz2GuHURwCRi\n"
"ipZ7SkXp7FnFvmuD5uHorLUwHv4FB4D54SMNUI8FmP8sX+g7tq3PgbUhh8oIKiMn\n"
"MCArz+2UW6yyetLHKKGKC5tNSixthT8Jcjxn4tncB7rrZXtaAWPWkFtPF2Y9fwsZ\n"
"o5NjEFIqnxQWWOLcpfShFosOkYuByptZ+thrkQdlVV9SH686+5DdaaVbnG0OLLb6\n"
"zqylfDJKZ0DcMDQj3dcEI2bw/FWAp/tmGYI1Z2JwOV5vx+qQQEQIHriy1tvuWacN\n"
"GHk0vFQYXlPKNFHtRQrmjseCNj6nOGOpMCwXEGCSn1WHElkQwg9naRHMTh5+Spqt\n"
"r0CodaxWkHS4oJyleW/c6RrIaQXpuvoDs3zk4E7Czp3otkYNbn5XOmeUwssfnHdK\n"
"Z05phkOTOPu220+DkdRgfks+KzgHVZhepA==\n"
"-----END CERTIFICATE-----\n";
BIO *cbio = BIO_new_mem_buf(mypem, sizeof(mypem));
X509_STORE *cts = SSL_CTX_get_cert_store((SSL_CTX *)sslctx);
int i;
STACK_OF(X509_INFO) *inf;
(void)curl;
(void)parm;
if(!cts || !cbio) {
return rv;
}
inf = PEM_X509_INFO_read_bio(cbio, NULL, NULL, NULL);
if(!inf) {
BIO_free(cbio);
return rv;
}
for(i = 0; i < sk_X509_INFO_num(inf); i++) {
X509_INFO *itmp = sk_X509_INFO_value(inf, i);
if(itmp->x509) {
X509_STORE_add_cert(cts, itmp->x509);
}
if(itmp->crl) {
X509_STORE_add_crl(cts, itmp->crl);
}
}
sk_X509_INFO_pop_free(inf, X509_INFO_free);
BIO_free(cbio);
rv = CURLE_OK;
return rv;
}
int main(void)
{
CURL *ch;
CURLcode rv;
curl_global_init(CURL_GLOBAL_ALL);
ch = curl_easy_init();
curl_easy_setopt(ch, CURLOPT_VERBOSE, 0L);
curl_easy_setopt(ch, CURLOPT_HEADER, 0L);
curl_easy_setopt(ch, CURLOPT_NOPROGRESS, 1L);
curl_easy_setopt(ch, CURLOPT_NOSIGNAL, 1L);
curl_easy_setopt(ch, CURLOPT_WRITEFUNCTION, *writefunction);
curl_easy_setopt(ch, CURLOPT_WRITEDATA, stdout);
curl_easy_setopt(ch, CURLOPT_HEADERFUNCTION, *writefunction);
curl_easy_setopt(ch, CURLOPT_HEADERDATA, stderr);
curl_easy_setopt(ch, CURLOPT_SSLCERTTYPE, "PEM");
curl_easy_setopt(ch, CURLOPT_SSL_VERIFYPEER, 1L);
curl_easy_setopt(ch, CURLOPT_URL, "https://www.example.com/");
/* Turn off the default CA locations, otherwise libcurl will load CA
* certificates from the locations that were detected/specified at
* build-time
*/
curl_easy_setopt(ch, CURLOPT_CAINFO, NULL);
curl_easy_setopt(ch, CURLOPT_CAPATH, NULL);
/* first try: retrieve page without ca certificates -> should fail
* unless libcurl was built --with-ca-fallback enabled at build-time
*/
rv = curl_easy_perform(ch);
if(rv == CURLE_OK)
printf("*** transfer succeeded ***\n");
else
printf("*** transfer failed ***\n");
/* use a fresh connection (optional)
* this option seriously impacts performance of multiple transfers but
* it is necessary order to demonstrate this example. recall that the
* ssl ctx callback is only called _before_ an SSL connection is
* established, therefore it will not affect existing verified SSL
* connections already in the connection cache associated with this
* handle. normally you would set the ssl ctx function before making
* any transfers, and not use this option.
*/
curl_easy_setopt(ch, CURLOPT_FRESH_CONNECT, 1L);
/* second try: retrieve page using cacerts' certificate -> will succeed
* load the certificate by installing a function doing the necessary
* "modifications" to the SSL CONTEXT just before link init
*/
curl_easy_setopt(ch, CURLOPT_SSL_CTX_FUNCTION, *sslctx_function);
rv = curl_easy_perform(ch);
if(rv == CURLE_OK)
printf("*** transfer succeeded ***\n");
else
printf("*** transfer failed ***\n");
curl_easy_cleanup(ch);
curl_global_cleanup();
return rv;
}
/***************************************************************************
* _ _ ____ _
* Project ___| | | | _ \| |
* / __| | | | |_) | |
* | (__| |_| | _ <| |___
* \___|\___/|_| \_\_____|
*
* Copyright (C) 1998 - 2017, Daniel Stenberg, <daniel@haxx.se>, et al.
*
* This software is licensed as described in the file COPYING, which
* you should have received as part of this distribution. The terms
* are also available at https://curl.haxx.se/docs/copyright.html.
*
* You may opt to use, copy, modify, merge, publish, distribute and/or sell
* copies of the Software, and permit persons to whom the Software is
* furnished to do so, under the terms of the COPYING file.
*
* This software is distributed on an "AS IS" basis, WITHOUT WARRANTY OF ANY
* KIND, either express or implied.
*
***************************************************************************/
/* <DESC>
* Extract lots of TLS certificate info.
* </DESC>
*/
#include <stdio.h>
#include <curl/curl.h>
static size_t wrfu(void *ptr, size_t size, size_t nmemb, void *stream)
{
(void)stream;
(void)ptr;
return size * nmemb;
}
int main(void)
{
CURL *curl;
CURLcode res;
curl_global_init(CURL_GLOBAL_DEFAULT);
curl = curl_easy_init();
if(curl) {
curl_easy_setopt(curl, CURLOPT_URL, "https://www.example.com/");
curl_easy_setopt(curl, CURLOPT_WRITEFUNCTION, wrfu);
curl_easy_setopt(curl, CURLOPT_SSL_VERIFYPEER, 0L);
curl_easy_setopt(curl, CURLOPT_SSL_VERIFYHOST, 0L);
curl_easy_setopt(curl, CURLOPT_VERBOSE, 0L);
curl_easy_setopt(curl, CURLOPT_CERTINFO, 1L);
res = curl_easy_perform(curl);
if(!res) {
struct curl_certinfo *certinfo;
res = curl_easy_getinfo(curl, CURLINFO_CERTINFO, &certinfo);
if(!res && certinfo) {
int i;
printf("%d certs!\n", certinfo->num_of_certs);
for(i = 0; i < certinfo->num_of_certs; i++) {
struct curl_slist *slist;
for(slist = certinfo->certinfo[i]; slist; slist = slist->next)
printf("%s\n", slist->data);
}
}
}
curl_easy_cleanup(curl);
}
curl_global_cleanup();
return 0;
}
/***************************************************************************
* _ _ ____ _
* Project ___| | | | _ \| |
* / __| | | | |_) | |
* | (__| |_| | _ <| |___
* \___|\___/|_| \_\_____|
*
* Copyright (C) 1998 - 2017, Daniel Stenberg, <daniel@haxx.se>, et al.
*
* This software is licensed as described in the file COPYING, which
* you should have received as part of this distribution. The terms
* are also available at https://curl.haxx.se/docs/copyright.html.
*
* You may opt to use, copy, modify, merge, publish, distribute and/or sell
* copies of the Software, and permit persons to whom the Software is
* furnished to do so, under the terms of the COPYING file.
*
* This software is distributed on an "AS IS" basis, WITHOUT WARRANTY OF ANY
* KIND, either express or implied.
*
***************************************************************************/
/* <DESC>
* Show transfer timing info after download completes.
* </DESC>
*/
/* Example source code to show how the callback function can be used to
* download data into a chunk of memory instead of storing it in a file.
* After successful download we use curl_easy_getinfo() calls to get the
* amount of downloaded bytes, the time used for the whole download, and
* the average download speed.
* On Linux you can create the download test files with:
* dd if=/dev/urandom of=file_1M.bin bs=1M count=1
*
*/
#include <stdio.h>
#include <stdlib.h>
#include <string.h>
#include <time.h>
#include <curl/curl.h>
#define URL_BASE "http://speedtest.your.domain/"
#define URL_1M URL_BASE "file_1M.bin"
#define URL_2M URL_BASE "file_2M.bin"
#define URL_5M URL_BASE "file_5M.bin"
#define URL_10M URL_BASE "file_10M.bin"
#define URL_20M URL_BASE "file_20M.bin"
#define URL_50M URL_BASE "file_50M.bin"
#define URL_100M URL_BASE "file_100M.bin"
#define CHKSPEED_VERSION "1.0"
static size_t WriteCallback(void *ptr, size_t size, size_t nmemb, void *data)
{
/* we are not interested in the downloaded bytes itself,
so we only return the size we would have saved ... */
(void)ptr; /* unused */
(void)data; /* unused */
return (size_t)(size * nmemb);
}
int main(int argc, char *argv[])
{
CURL *curl_handle;
CURLcode res;
int prtall = 0, prtsep = 0, prttime = 0;
const char *url = URL_1M;
char *appname = argv[0];
if(argc > 1) {
/* parse input parameters */
for(argc--, argv++; *argv; argc--, argv++) {
if(strncasecmp(*argv, "-", 1) == 0) {
if(strncasecmp(*argv, "-H", 2) == 0) {
fprintf(stderr,
"\rUsage: %s [-m=1|2|5|10|20|50|100] [-t] [-x] [url]\n",
appname);
exit(1);
}
else if(strncasecmp(*argv, "-V", 2) == 0) {
fprintf(stderr, "\r%s %s - %s\n",
appname, CHKSPEED_VERSION, curl_version());
exit(1);
}
else if(strncasecmp(*argv, "-A", 2) == 0) {
prtall = 1;
}
else if(strncasecmp(*argv, "-X", 2) == 0) {
prtsep = 1;
}
else if(strncasecmp(*argv, "-T", 2) == 0) {
prttime = 1;
}
else if(strncasecmp(*argv, "-M=", 3) == 0) {
long m = strtol((*argv) + 3, NULL, 10);
switch(m) {
case 1:
url = URL_1M;
break;
case 2:
url = URL_2M;
break;
case 5:
url = URL_5M;
break;
case 10:
url = URL_10M;
break;
case 20:
url = URL_20M;
break;
case 50:
url = URL_50M;
break;
case 100:
url = URL_100M;
break;
default:
fprintf(stderr, "\r%s: invalid parameter %s\n",
appname, *argv + 3);
exit(1);
}
}
else {
fprintf(stderr, "\r%s: invalid or unknown option %s\n",
appname, *argv);
exit(1);
}
}
else {
url = *argv;
}
}
}
/* print separator line */
if(prtsep) {
printf("-------------------------------------------------\n");
}
/* print localtime */
if(prttime) {
time_t t = time(NULL);
printf("Localtime: %s", ctime(&t));
}
/* init libcurl */
curl_global_init(CURL_GLOBAL_ALL);
/* init the curl session */
curl_handle = curl_easy_init();
/* specify URL to get */
curl_easy_setopt(curl_handle, CURLOPT_URL, url);
/* send all data to this function */
curl_easy_setopt(curl_handle, CURLOPT_WRITEFUNCTION, WriteCallback);
/* some servers don't like requests that are made without a user-agent
field, so we provide one */
curl_easy_setopt(curl_handle, CURLOPT_USERAGENT,
"libcurl-speedchecker/" CHKSPEED_VERSION);
/* get it! */
res = curl_easy_perform(curl_handle);
if(CURLE_OK == res) {
curl_off_t val;
/* check for bytes downloaded */
res = curl_easy_getinfo(curl_handle, CURLINFO_SIZE_DOWNLOAD_T, &val);
if((CURLE_OK == res) && (val>0))
printf("Data downloaded: %" CURL_FORMAT_CURL_OFF_T " bytes.\n", val);
/* check for total download time */
res = curl_easy_getinfo(curl_handle, CURLINFO_TOTAL_TIME_T, &val);
if((CURLE_OK == res) && (val>0))
printf("Total download time: %" CURL_FORMAT_CURL_OFF_T ".%06ld sec.\n",
(val / 1000000), (long)(val % 1000000));
/* check for average download speed */
res = curl_easy_getinfo(curl_handle, CURLINFO_SPEED_DOWNLOAD_T, &val);
if((CURLE_OK == res) && (val>0))
printf("Average download speed: %" CURL_FORMAT_CURL_OFF_T
" kbyte/sec.\n", val / 1024);
if(prtall) {
/* check for name resolution time */
res = curl_easy_getinfo(curl_handle, CURLINFO_NAMELOOKUP_TIME_T, &val);
if((CURLE_OK == res) && (val>0))
printf("Name lookup time: %" CURL_FORMAT_CURL_OFF_T ".%06ld sec.\n",
(val / 1000000), (long)(val % 1000000));
/* check for connect time */
res = curl_easy_getinfo(curl_handle, CURLINFO_CONNECT_TIME_T, &val);
if((CURLE_OK == res) && (val>0))
printf("Connect time: %" CURL_FORMAT_CURL_OFF_T ".%06ld sec.\n",
(val / 1000000), (long)(val % 1000000));
}
}
else {
fprintf(stderr, "Error while fetching '%s' : %s\n",
url, curl_easy_strerror(res));
}
/* cleanup curl stuff */
curl_easy_cleanup(curl_handle);
/* we're done with libcurl, so clean it up */
curl_global_cleanup();
return 0;
}
/***************************************************************************
* _ _ ____ _
* Project ___| | | | _ \| |
* / __| | | | |_) | |
* | (__| |_| | _ <| |___
* \___|\___/|_| \_\_____|
*
* Copyright (C) 1998 - 2018, Daniel Stenberg, <daniel@haxx.se>, et al.
*
* This software is licensed as described in the file COPYING, which
* you should have received as part of this distribution. The terms
* are also available at https://curl.haxx.se/docs/copyright.html.
*
* You may opt to use, copy, modify, merge, publish, distribute and/or sell
* copies of the Software, and permit persons to whom the Software is
* furnished to do so, under the terms of the COPYING file.
*
* This software is distributed on an "AS IS" basis, WITHOUT WARRANTY OF ANY
* KIND, either express or implied.
*
***************************************************************************/
/* <DESC>
* Import and export cookies with COOKIELIST.
* </DESC>
*/
#include <stdio.h>
#include <string.h>
#include <stdlib.h>
#include <errno.h>
#include <time.h>
#include <curl/curl.h>
static void
print_cookies(CURL *curl)
{
CURLcode res;
struct curl_slist *cookies;
struct curl_slist *nc;
int i;
printf("Cookies, curl knows:\n");
res = curl_easy_getinfo(curl, CURLINFO_COOKIELIST, &cookies);
if(res != CURLE_OK) {
fprintf(stderr, "Curl curl_easy_getinfo failed: %s\n",
curl_easy_strerror(res));
exit(1);
}
nc = cookies;
i = 1;
while(nc) {
printf("[%d]: %s\n", i, nc->data);
nc = nc->next;
i++;
}
if(i == 1) {
printf("(none)\n");
}
curl_slist_free_all(cookies);
}
int
main(void)
{
CURL *curl;
CURLcode res;
curl_global_init(CURL_GLOBAL_ALL);
curl = curl_easy_init();
if(curl) {
char nline[256];
curl_easy_setopt(curl, CURLOPT_URL, "https://www.example.com/");
curl_easy_setopt(curl, CURLOPT_VERBOSE, 1L);
curl_easy_setopt(curl, CURLOPT_COOKIEFILE, ""); /* start cookie engine */
res = curl_easy_perform(curl);
if(res != CURLE_OK) {
fprintf(stderr, "Curl perform failed: %s\n", curl_easy_strerror(res));
return 1;
}
print_cookies(curl);
printf("Erasing curl's knowledge of cookies!\n");
curl_easy_setopt(curl, CURLOPT_COOKIELIST, "ALL");
print_cookies(curl);
printf("-----------------------------------------------\n"
"Setting a cookie \"PREF\" via cookie interface:\n");
#ifdef WIN32
#define snprintf _snprintf
#endif
/* Netscape format cookie */
snprintf(nline, sizeof(nline), "%s\t%s\t%s\t%s\t%lu\t%s\t%s",
".example.com", "TRUE", "/", "FALSE",
(unsigned long)time(NULL) + 31337UL,
"PREF", "hello example, i like you very much!");
res = curl_easy_setopt(curl, CURLOPT_COOKIELIST, nline);
if(res != CURLE_OK) {
fprintf(stderr, "Curl curl_easy_setopt failed: %s\n",
curl_easy_strerror(res));
return 1;
}
/* HTTP-header style cookie. If you use the Set-Cookie format and don't
specify a domain then the cookie is sent for any domain and will not be
modified, likely not what you intended. Starting in 7.43.0 any-domain
cookies will not be exported either. For more information refer to the
CURLOPT_COOKIELIST documentation.
*/
snprintf(nline, sizeof(nline),
"Set-Cookie: OLD_PREF=3d141414bf4209321; "
"expires=Sun, 17-Jan-2038 19:14:07 GMT; path=/; domain=.example.com");
res = curl_easy_setopt(curl, CURLOPT_COOKIELIST, nline);
if(res != CURLE_OK) {
fprintf(stderr, "Curl curl_easy_setopt failed: %s\n",
curl_easy_strerror(res));
return 1;
}
print_cookies(curl);
res = curl_easy_perform(curl);
if(res != CURLE_OK) {
fprintf(stderr, "Curl perform failed: %s\n", curl_easy_strerror(res));
return 1;
}
curl_easy_cleanup(curl);
}
else {
fprintf(stderr, "Curl init failed!\n");
return 1;
}
curl_global_cleanup();
return 0;
}
/***************************************************************************
* _ _ ____ _
* Project ___| | | | _ \| |
* / __| | | | |_) | |
* | (__| |_| | _ <| |___
* \___|\___/|_| \_\_____|
*
* Web crawler based on curl and libxml2.
* Copyright (C) 2018 Jeroen Ooms <jeroenooms@gmail.com>
* License: MIT
*
* To compile:
* gcc crawler.c $(pkg-config --cflags --libs libxml-2.0 libcurl)
*
*/
/* <DESC>
* Web crawler based on curl and libxml2 to stress-test curl with
* hundreds of concurrent connections to various servers.
* </DESC>
*/
/* Parameters */
int max_con = 200;
int max_total = 20000;
int max_requests = 500;
int max_link_per_page = 5;
int follow_relative_links = 0;
char *start_page = "https://www.reuters.com";
#include <libxml/HTMLparser.h>
#include <libxml/xpath.h>
#include <libxml/uri.h>
#include <curl/curl.h>
#include <stdlib.h>
#include <string.h>
#include <math.h>
#include <signal.h>
int pending_interrupt = 0;
void sighandler(int dummy)
{
pending_interrupt = 1;
}
/* resizable buffer */
typedef struct {
char *buf;
size_t size;
} memory;
size_t grow_buffer(void *contents, size_t sz, size_t nmemb, void *ctx)
{
size_t realsize = sz * nmemb;
memory *mem = (memory*) ctx;
char *ptr = realloc(mem->buf, mem->size + realsize);
if(!ptr) {
/* out of memory */
printf("not enough memory (realloc returned NULL)\n");
return 0;
}
mem->buf = ptr;
memcpy(&(mem->buf[mem->size]), contents, realsize);
mem->size += realsize;
return realsize;
}
CURL *make_handle(char *url)
{
CURL *handle = curl_easy_init();
/* Important: use HTTP2 over HTTPS */
curl_easy_setopt(handle, CURLOPT_HTTP_VERSION, CURL_HTTP_VERSION_2TLS);
curl_easy_setopt(handle, CURLOPT_URL, url);
/* buffer body */
memory *mem = malloc(sizeof(memory));
mem->size = 0;
mem->buf = malloc(1);
curl_easy_setopt(handle, CURLOPT_WRITEFUNCTION, grow_buffer);
curl_easy_setopt(handle, CURLOPT_WRITEDATA, mem);
curl_easy_setopt(handle, CURLOPT_PRIVATE, mem);
/* For completeness */
curl_easy_setopt(handle, CURLOPT_ACCEPT_ENCODING, "");
curl_easy_setopt(handle, CURLOPT_TIMEOUT, 5L);
curl_easy_setopt(handle, CURLOPT_FOLLOWLOCATION, 1L);
curl_easy_setopt(handle, CURLOPT_MAXREDIRS, 10L);
curl_easy_setopt(handle, CURLOPT_CONNECTTIMEOUT, 2L);
curl_easy_setopt(handle, CURLOPT_COOKIEFILE, "");
curl_easy_setopt(handle, CURLOPT_FILETIME, 1L);
curl_easy_setopt(handle, CURLOPT_USERAGENT, "mini crawler");
curl_easy_setopt(handle, CURLOPT_HTTPAUTH, CURLAUTH_ANY);
curl_easy_setopt(handle, CURLOPT_UNRESTRICTED_AUTH, 1L);
curl_easy_setopt(handle, CURLOPT_PROXYAUTH, CURLAUTH_ANY);
curl_easy_setopt(handle, CURLOPT_EXPECT_100_TIMEOUT_MS, 0L);
return handle;
}
/* HREF finder implemented in libxml2 but could be any HTML parser */
size_t follow_links(CURLM *multi_handle, memory *mem, char *url)
{
int opts = HTML_PARSE_NOBLANKS | HTML_PARSE_NOERROR | \
HTML_PARSE_NOWARNING | HTML_PARSE_NONET;
htmlDocPtr doc = htmlReadMemory(mem->buf, mem->size, url, NULL, opts);
if(!doc)
return 0;
xmlChar *xpath = (xmlChar*) "//a/@href";
xmlXPathContextPtr context = xmlXPathNewContext(doc);
xmlXPathObjectPtr result = xmlXPathEvalExpression(xpath, context);
xmlXPathFreeContext(context);
if(!result)
return 0;
xmlNodeSetPtr nodeset = result->nodesetval;
if(xmlXPathNodeSetIsEmpty(nodeset)) {
xmlXPathFreeObject(result);
return 0;
}
size_t count = 0;
for(int i = 0; i < nodeset->nodeNr; i++) {
double r = rand();
int x = r * nodeset->nodeNr / RAND_MAX;
const xmlNode *node = nodeset->nodeTab[x]->xmlChildrenNode;
xmlChar *href = xmlNodeListGetString(doc, node, 1);
if(follow_relative_links) {
xmlChar *orig = href;
href = xmlBuildURI(href, (xmlChar *) url);
xmlFree(orig);
}
char *link = (char *) href;
if(!link || strlen(link) < 20)
continue;
if(!strncmp(link, "http://", 7) || !strncmp(link, "https://", 8)) {
curl_multi_add_handle(multi_handle, make_handle(link));
if(count++ == max_link_per_page)
break;
}
xmlFree(link);
}
xmlXPathFreeObject(result);
return count;
}
int is_html(char *ctype)
{
return ctype != NULL && strlen(ctype) > 10 && strstr(ctype, "text/html");
}
int main(void)
{
signal(SIGINT, sighandler);
LIBXML_TEST_VERSION;
curl_global_init(CURL_GLOBAL_DEFAULT);
CURLM *multi_handle = curl_multi_init();
curl_multi_setopt(multi_handle, CURLMOPT_MAX_TOTAL_CONNECTIONS, max_con);
curl_multi_setopt(multi_handle, CURLMOPT_MAX_HOST_CONNECTIONS, 6L);
/* enables http/2 if available */
#ifdef CURLPIPE_MULTIPLEX
curl_multi_setopt(multi_handle, CURLMOPT_PIPELINING, CURLPIPE_MULTIPLEX);
#endif
/* sets html start page */
curl_multi_add_handle(multi_handle, make_handle(start_page));
int msgs_left;
int pending = 0;
int complete = 0;
int still_running = 1;
while(still_running && !pending_interrupt) {
int numfds;
curl_multi_wait(multi_handle, NULL, 0, 1000, &numfds);
curl_multi_perform(multi_handle, &still_running);
/* See how the transfers went */
CURLMsg *m = NULL;
while((m = curl_multi_info_read(multi_handle, &msgs_left))) {
if(m->msg == CURLMSG_DONE) {
CURL *handle = m->easy_handle;
char *url;
memory *mem;
curl_easy_getinfo(handle, CURLINFO_PRIVATE, &mem);
curl_easy_getinfo(handle, CURLINFO_EFFECTIVE_URL, &url);
if(m->data.result == CURLE_OK) {
long res_status;
curl_easy_getinfo(handle, CURLINFO_RESPONSE_CODE, &res_status);
if(res_status == 200) {
char *ctype;
curl_easy_getinfo(handle, CURLINFO_CONTENT_TYPE, &ctype);
printf("[%d] HTTP 200 (%s): %s\n", complete, ctype, url);
if(is_html(ctype) && mem->size > 100) {
if(pending < max_requests && (complete + pending) < max_total) {
pending += follow_links(multi_handle, mem, url);
still_running = 1;
}
}
}
else {
printf("[%d] HTTP %d: %s\n", complete, (int) res_status, url);
}
}
else {
printf("[%d] Connection failure: %s\n", complete, url);
}
curl_multi_remove_handle(multi_handle, handle);
curl_easy_cleanup(handle);
free(mem->buf);
free(mem);
complete++;
pending--;
}
}
}
curl_multi_cleanup(multi_handle);
curl_global_cleanup();
return 0;
}
/*****************************************************************************
* _ _ ____ _
* Project ___| | | | _ \| |
* / __| | | | |_) | |
* | (__| |_| | _ <| |___
* \___|\___/|_| \_\_____|
*
* Copyright (c) 2000 David Odin (aka DindinX) for MandrakeSoft
*/
/* <DESC>
* use the libcurl in a gtk-threaded application
* </DESC>
*/
#include <stdio.h>
#include <gtk/gtk.h>
#include <curl/curl.h>
GtkWidget *Bar;
size_t my_write_func(void *ptr, size_t size, size_t nmemb, FILE *stream)
{
return fwrite(ptr, size, nmemb, stream);
}
size_t my_read_func(void *ptr, size_t size, size_t nmemb, FILE *stream)
{
return fread(ptr, size, nmemb, stream);
}
int my_progress_func(GtkWidget *bar,
double t, /* dltotal */
double d, /* dlnow */
double ultotal,
double ulnow)
{
/* printf("%d / %d (%g %%)\n", d, t, d*100.0/t);*/
gdk_threads_enter();
gtk_progress_set_value(GTK_PROGRESS(bar), d*100.0/t);
gdk_threads_leave();
return 0;
}
void *my_thread(void *ptr)
{
CURL *curl;
curl = curl_easy_init();
if(curl) {
gchar *url = ptr;
const char *filename = "test.curl";
FILE *outfile = fopen(filename, "wb");
curl_easy_setopt(curl, CURLOPT_URL, url);
curl_easy_setopt(curl, CURLOPT_WRITEDATA, outfile);
curl_easy_setopt(curl, CURLOPT_WRITEFUNCTION, my_write_func);
curl_easy_setopt(curl, CURLOPT_READFUNCTION, my_read_func);
curl_easy_setopt(curl, CURLOPT_NOPROGRESS, 0L);
curl_easy_setopt(curl, CURLOPT_PROGRESSFUNCTION, my_progress_func);
curl_easy_setopt(curl, CURLOPT_PROGRESSDATA, Bar);
curl_easy_perform(curl);
fclose(outfile);
/* always cleanup */
curl_easy_cleanup(curl);
}
return NULL;
}
int main(int argc, char **argv)
{
GtkWidget *Window, *Frame, *Frame2;
GtkAdjustment *adj;
/* Must initialize libcurl before any threads are started */
curl_global_init(CURL_GLOBAL_ALL);
/* Init thread */
g_thread_init(NULL);
gtk_init(&argc, &argv);
Window = gtk_window_new(GTK_WINDOW_TOPLEVEL);
Frame = gtk_frame_new(NULL);
gtk_frame_set_shadow_type(GTK_FRAME(Frame), GTK_SHADOW_OUT);
gtk_container_add(GTK_CONTAINER(Window), Frame);
Frame2 = gtk_frame_new(NULL);
gtk_frame_set_shadow_type(GTK_FRAME(Frame2), GTK_SHADOW_IN);
gtk_container_add(GTK_CONTAINER(Frame), Frame2);
gtk_container_set_border_width(GTK_CONTAINER(Frame2), 5);
adj = (GtkAdjustment*)gtk_adjustment_new(0, 0, 100, 0, 0, 0);
Bar = gtk_progress_bar_new_with_adjustment(adj);
gtk_container_add(GTK_CONTAINER(Frame2), Bar);
gtk_widget_show_all(Window);
if(!g_thread_create(&my_thread, argv[1], FALSE, NULL) != 0)
g_warning("can't create the thread");
gdk_threads_enter();
gtk_main();
gdk_threads_leave();
return 0;
}
/***************************************************************************
* _ _ ____ _
* Project ___| | | | _ \| |
* / __| | | | |_) | |
* | (__| |_| | _ <| |___
* \___|\___/|_| \_\_____|
*
* Copyright (C) 1998 - 2018, Daniel Stenberg, <daniel@haxx.se>, et al.
*
* This software is licensed as described in the file COPYING, which
* you should have received as part of this distribution. The terms
* are also available at https://curl.haxx.se/docs/copyright.html.
*
* You may opt to use, copy, modify, merge, publish, distribute and/or sell
* copies of the Software, and permit persons to whom the Software is
* furnished to do so, under the terms of the COPYING file.
*
* This software is distributed on an "AS IS" basis, WITHOUT WARRANTY OF ANY
* KIND, either express or implied.
*
***************************************************************************/
/* <DESC>
* Show how CURLOPT_DEBUGFUNCTION can be used.
* </DESC>
*/
#include <stdio.h>
#include <curl/curl.h>
struct data {
char trace_ascii; /* 1 or 0 */
};
static
void dump(const char *text,
FILE *stream, unsigned char *ptr, size_t size,
char nohex)
{
size_t i;
size_t c;
unsigned int width = 0x10;
if(nohex)
/* without the hex output, we can fit more on screen */
width = 0x40;
fprintf(stream, "%s, %10.10lu bytes (0x%8.8lx)\n",
text, (unsigned long)size, (unsigned long)size);
for(i = 0; i<size; i += width) {
fprintf(stream, "%4.4lx: ", (unsigned long)i);
if(!nohex) {
/* hex not disabled, show it */
for(c = 0; c < width; c++)
if(i + c < size)
fprintf(stream, "%02x ", ptr[i + c]);
else
fputs(" ", stream);
}
for(c = 0; (c < width) && (i + c < size); c++) {
/* check for 0D0A; if found, skip past and start a new line of output */
if(nohex && (i + c + 1 < size) && ptr[i + c] == 0x0D &&
ptr[i + c + 1] == 0x0A) {
i += (c + 2 - width);
break;
}
fprintf(stream, "%c",
(ptr[i + c] >= 0x20) && (ptr[i + c]<0x80)?ptr[i + c]:'.');
/* check again for 0D0A, to avoid an extra \n if it's at width */
if(nohex && (i + c + 2 < size) && ptr[i + c + 1] == 0x0D &&
ptr[i + c + 2] == 0x0A) {
i += (c + 3 - width);
break;
}
}
fputc('\n', stream); /* newline */
}
fflush(stream);
}
static
int my_trace(CURL *handle, curl_infotype type,
char *data, size_t size,
void *userp)
{
struct data *config = (struct data *)userp;
const char *text;
(void)handle; /* prevent compiler warning */
switch(type) {
case CURLINFO_TEXT:
fprintf(stderr, "== Info: %s", data);
/* FALLTHROUGH */
default: /* in case a new one is introduced to shock us */
return 0;
case CURLINFO_HEADER_OUT:
text = "=> Send header";
break;
case CURLINFO_DATA_OUT:
text = "=> Send data";
break;
case CURLINFO_SSL_DATA_OUT:
text = "=> Send SSL data";
break;
case CURLINFO_HEADER_IN:
text = "<= Recv header";
break;
case CURLINFO_DATA_IN:
text = "<= Recv data";
break;
case CURLINFO_SSL_DATA_IN:
text = "<= Recv SSL data";
break;
}
dump(text, stderr, (unsigned char *)data, size, config->trace_ascii);
return 0;
}
int main(void)
{
CURL *curl;
CURLcode res;
struct data config;
config.trace_ascii = 1; /* enable ascii tracing */
curl = curl_easy_init();
if(curl) {
curl_easy_setopt(curl, CURLOPT_DEBUGFUNCTION, my_trace);
curl_easy_setopt(curl, CURLOPT_DEBUGDATA, &config);
/* the DEBUGFUNCTION has no effect until we enable VERBOSE */
curl_easy_setopt(curl, CURLOPT_VERBOSE, 1L);
/* example.com is redirected, so we tell libcurl to follow redirection */
curl_easy_setopt(curl, CURLOPT_FOLLOWLOCATION, 1L);
curl_easy_setopt(curl, CURLOPT_URL, "https://example.com/");
res = curl_easy_perform(curl);
/* Check for errors */
if(res != CURLE_OK)
fprintf(stderr, "curl_easy_perform() failed: %s\n",
curl_easy_strerror(res));
/* always cleanup */
curl_easy_cleanup(curl);
}
return 0;
}
/***************************************************************************
* _ _ ____ _
* Project ___| | | | _ \| |
* / __| | | | |_) | |
* | (__| |_| | _ <| |___
* \___|\___/|_| \_\_____|
*
* Copyright (C) 1998 - 2017, Daniel Stenberg, <daniel@haxx.se>, et al.
*
* This software is licensed as described in the file COPYING, which
* you should have received as part of this distribution. The terms
* are also available at https://curl.haxx.se/docs/copyright.html.
*
* You may opt to use, copy, modify, merge, publish, distribute and/or sell
* copies of the Software, and permit persons to whom the Software is
* furnished to do so, under the terms of the COPYING file.
*
* This software is distributed on an "AS IS" basis, WITHOUT WARRANTY OF ANY
* KIND, either express or implied.
*
***************************************************************************/
/* <DESC>
* An example demonstrating how an application can pass in a custom
* socket to libcurl to use. This example also handles the connect itself.
* </DESC>
*/
#include <stdio.h>
#include <string.h>
#include <stdlib.h>
#include <curl/curl.h>
#ifdef WIN32
#include <windows.h>
#include <winsock2.h>
#include <ws2tcpip.h>
#define close closesocket
#else
#include <sys/types.h> /* socket types */
#include <sys/socket.h> /* socket definitions */
#include <netinet/in.h>
#include <arpa/inet.h> /* inet (3) functions */
#include <unistd.h> /* misc. Unix functions */
#endif
#include <errno.h>
/* The IP address and port number to connect to */
#define IPADDR "127.0.0.1"
#define PORTNUM 80
#ifndef INADDR_NONE
#define INADDR_NONE 0xffffffff
#endif
static size_t write_data(void *ptr, size_t size, size_t nmemb, void *stream)
{
size_t written = fwrite(ptr, size, nmemb, (FILE *)stream);
return written;
}
static int closecb(void *clientp, curl_socket_t item)
{
(void)clientp;
printf("libcurl wants to close %d now\n", (int)item);
return 0;
}
static curl_socket_t opensocket(void *clientp,
curlsocktype purpose,
struct curl_sockaddr *address)
{
curl_socket_t sockfd;
(void)purpose;
(void)address;
sockfd = *(curl_socket_t *)clientp;
/* the actual externally set socket is passed in via the OPENSOCKETDATA
option */
return sockfd;
}
static int sockopt_callback(void *clientp, curl_socket_t curlfd,
curlsocktype purpose)
{
(void)clientp;
(void)curlfd;
(void)purpose;
/* This return code was added in libcurl 7.21.5 */
return CURL_SOCKOPT_ALREADY_CONNECTED;
}
int main(void)
{
CURL *curl;
CURLcode res;
struct sockaddr_in servaddr; /* socket address structure */
curl_socket_t sockfd;
#ifdef WIN32
WSADATA wsaData;
int initwsa = WSAStartup(MAKEWORD(2, 0), &wsaData);
if(initwsa != 0) {
printf("WSAStartup failed: %d\n", initwsa);
return 1;
}
#endif
curl = curl_easy_init();
if(curl) {
/*
* Note that libcurl will internally think that you connect to the host
* and port that you specify in the URL option.
*/
curl_easy_setopt(curl, CURLOPT_URL, "http://99.99.99.99:9999");
/* Create the socket "manually" */
sockfd = socket(AF_INET, SOCK_STREAM, 0);
if(sockfd == CURL_SOCKET_BAD) {
printf("Error creating listening socket.\n");
return 3;
}
memset(&servaddr, 0, sizeof(servaddr));
servaddr.sin_family = AF_INET;
servaddr.sin_port = htons(PORTNUM);
servaddr.sin_addr.s_addr = inet_addr(IPADDR);
if(INADDR_NONE == servaddr.sin_addr.s_addr) {
close(sockfd);
return 2;
}
if(connect(sockfd, (struct sockaddr *) &servaddr, sizeof(servaddr)) ==
-1) {
close(sockfd);
printf("client error: connect: %s\n", strerror(errno));
return 1;
}
/* no progress meter please */
curl_easy_setopt(curl, CURLOPT_NOPROGRESS, 1L);
/* send all data to this function */
curl_easy_setopt(curl, CURLOPT_WRITEFUNCTION, write_data);
/* call this function to get a socket */
curl_easy_setopt(curl, CURLOPT_OPENSOCKETFUNCTION, opensocket);
curl_easy_setopt(curl, CURLOPT_OPENSOCKETDATA, &sockfd);
/* call this function to close sockets */
curl_easy_setopt(curl, CURLOPT_CLOSESOCKETFUNCTION, closecb);
curl_easy_setopt(curl, CURLOPT_CLOSESOCKETDATA, &sockfd);
/* call this function to set options for the socket */
curl_easy_setopt(curl, CURLOPT_SOCKOPTFUNCTION, sockopt_callback);
curl_easy_setopt(curl, CURLOPT_VERBOSE, 1);
res = curl_easy_perform(curl);
curl_easy_cleanup(curl);
close(sockfd);
if(res) {
printf("libcurl error: %d\n", res);
return 4;
}
}
#ifdef WIN32
WSACleanup();
#endif
return 0;
}
/***************************************************************************
* _ _ ____ _
* Project ___| | | | _ \| |
* / __| | | | |_) | |
* | (__| |_| | _ <| |___
* \___|\___/|_| \_\_____|
*
* Copyright (C) 1998 - 2016, Daniel Stenberg, <daniel@haxx.se>, et al.
*
* This software is licensed as described in the file COPYING, which
* you should have received as part of this distribution. The terms
* are also available at https://curl.haxx.se/docs/copyright.html.
*
* You may opt to use, copy, modify, merge, publish, distribute and/or sell
* copies of the Software, and permit persons to whom the Software is
* furnished to do so, under the terms of the COPYING file.
*
* This software is distributed on an "AS IS" basis, WITHOUT WARRANTY OF ANY
* KIND, either express or implied.
*
***************************************************************************/
/* <DESC>
* Upload to a file:// URL
* </DESC>
*/
#include <stdio.h>
#include <curl/curl.h>
#include <sys/stat.h>
#include <fcntl.h>
int main(void)
{
CURL *curl;
CURLcode res;
struct stat file_info;
curl_off_t speed_upload, total_time;
FILE *fd;
fd = fopen("debugit", "rb"); /* open file to upload */
if(!fd)
return 1; /* can't continue */
/* to get the file size */
if(fstat(fileno(fd), &file_info) != 0)
return 1; /* can't continue */
curl = curl_easy_init();
if(curl) {
/* upload to this place */
curl_easy_setopt(curl, CURLOPT_URL,
"file:///home/dast/src/curl/debug/new");
/* tell it to "upload" to the URL */
curl_easy_setopt(curl, CURLOPT_UPLOAD, 1L);
/* set where to read from (on Windows you need to use READFUNCTION too) */
curl_easy_setopt(curl, CURLOPT_READDATA, fd);
/* and give the size of the upload (optional) */
curl_easy_setopt(curl, CURLOPT_INFILESIZE_LARGE,
(curl_off_t)file_info.st_size);
/* enable verbose for easier tracing */
curl_easy_setopt(curl, CURLOPT_VERBOSE, 1L);
res = curl_easy_perform(curl);
/* Check for errors */
if(res != CURLE_OK) {
fprintf(stderr, "curl_easy_perform() failed: %s\n",
curl_easy_strerror(res));
}
else {
/* now extract transfer info */
curl_easy_getinfo(curl, CURLINFO_SPEED_UPLOAD_T, &speed_upload);
curl_easy_getinfo(curl, CURLINFO_TOTAL_TIME_T, &total_time);
fprintf(stderr, "Speed: %" CURL_FORMAT_CURL_OFF_T " bytes/sec during %"
CURL_FORMAT_CURL_OFF_T ".%06ld seconds\n",
speed_upload,
(total_time / 1000000), (long)(total_time % 1000000));
}
/* always cleanup */
curl_easy_cleanup(curl);
}
fclose(fd);
return 0;
}
/***************************************************************************
* _ _ ____ _
* Project ___| | | | _ \| |
* / __| | | | |_) | |
* | (__| |_| | _ <| |___
* \___|\___/|_| \_\_____|
*
* Copyright (C) 1998 - 2018, Daniel Stenberg, <daniel@haxx.se>, et al.
*
* This software is licensed as described in the file COPYING, which
* you should have received as part of this distribution. The terms
* are also available at https://curl.haxx.se/docs/copyright.html.
*
* You may opt to use, copy, modify, merge, publish, distribute and/or sell
* copies of the Software, and permit persons to whom the Software is
* furnished to do so, under the terms of the COPYING file.
*
* This software is distributed on an "AS IS" basis, WITHOUT WARRANTY OF ANY
* KIND, either express or implied.
*
***************************************************************************/
/* <DESC>
* Use getinfo to get content-type after completed transfer.
* </DESC>
*/
#include <stdio.h>
#include <curl/curl.h>
int main(void)
{
CURL *curl;
CURLcode res;
curl = curl_easy_init();
if(curl) {
curl_easy_setopt(curl, CURLOPT_URL, "https://www.example.com/");
res = curl_easy_perform(curl);
if(CURLE_OK == res) {
char *ct;
/* ask for the content-type */
res = curl_easy_getinfo(curl, CURLINFO_CONTENT_TYPE, &ct);
if((CURLE_OK == res) && ct)
printf("We received Content-Type: %s\n", ct);
}
/* always cleanup */
curl_easy_cleanup(curl);
}
return 0;
}
This diff is collapsed.
This diff is collapsed.
This diff is collapsed.
This diff is collapsed.
Markdown is supported
0%
or
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment