Interesting article, but do many people use curl for binary downloads? I thought most sysadmins used it for viewing headers and/or parsing raw text. For binary downloads wget seems to be favoured. So with that in mind, I'd be curious to see how wget compared with the other solutions in those benchmarks.
DISCLAIMER: I'm not trying to imply that my way is "better" nor the only way HTTP GETs should be called. I'm only commenting on how I generally work and the trend I've anecdotally observed. Though I'd be interested to see if many others on here do prefer curl over wget for pulling binary files.
[13:17:18] i13:klausa:~% uname -a
Darwin i13.local 12.4.0 Darwin Kernel Version 12.4.0: Wed May 1 17:57:12 PDT 2013; root:xnu-2050.24.15~1/RELEASE_X86_64 x86_64
[13:17:19] i13:klausa:~% curl
curl: try 'curl --help' or 'curl --manual' for more information
[13:17:22] i13:klausa:~% wget
wget: Command not found.
A demonstration about how OS X ships curl but not wget. Essentially he's just running
which {curl,wget}
to demonstrate that curl is in $PATH where as the shell cannot find wget (since we're talking about base installs it's safe to assume that the aforementioned utilities should be in $PATH by default - if included at all).
I'd be interested to see if many others on here do prefer curl over wget for pulling binary files.
This is how I ended up consistently prefering curl:
Last time I looked closely, I had better control with curl, and curl had better support for stuff. For example HTTP 1.1 was supported by curl, but not wget.
This makes curl a far better choice for testing HTTP servers.
And next, since I now know curl pretty well, and have it installed all over the place, it is my go-to HTTP command line client. I don't know of any reason to use both, or any reason to prefer wget, although in some situations maybe "wget <url>" acts slightly more like you want than "curl <url>". (Which is of course easy to work around by using "curl -OL <url>", if that's what you want, but convenient defaults are still convenient)
> "Last time I looked closely, I had better control with curl, and curl had better support for stuff. For example HTTP 1.1 was supported by curl, but not wget.
"This makes curl a far better choice for testing HTTP servers."
Well yes. You're just reiterating what I said about curl being great for server testing. But this article is about download performance. When just downloading a largish binary file (eg a tarball), it's easier to just:
wget example.com/source.tar.gz
With that, you get a progress bar, it saves the file to disk and basically just does all the sane options you'd want for downloading by default (which I think is why most INSTALL / README docs tend to recommend wget in their instructions).
Again, I'm not saying one is better than the other, but the convenience of wget's defaults (which I think you also highlighted later in you post) does tend to make me favour it for downloading binary content (with curl being my go-to for any server testing or text resources).
> "And next, since I now know curl pretty well, and have it installed all over the place, it is my go-to HTTP command line client."
wget also comes as part of the basic install with almost all Linux and other UNIX-like OSs as well. In fact wget actually pre-dates curl (albeit they're both >= 15 years old, so there's not really much between the two relatively speaking) so I'd have expected even your oldest live systems would still have wget.
I can totally relate to the "know[ing] curl pretty well" point though. There is definitely an argument for reusing the tools that you're already familiar with.
We seem to be in full agreement. I only wanted to spell out my own reasoning for using curl and not wget, since you explicitly said you were interested in this information, and I suppose there are others that think like me in this regard as well :)
One popular UNIX-like OS that comes with curl and not wget is OS X. That might also drive adoption.
(I seem to remember having had to install both curl and wget on Ubuntu, but I am unable to verify this now)
> "We seem to be in full agreement. I only wanted to spell out my own reasoning for using curl and not wget, since you explicitly said you were interested in this information, and I suppose there are others that think like me in this regard as well :)"
I'm grateful for your input. Sorry if I came across as elitist - that wasn't my intention but in reflection, in does read a little that way.
> "One popular UNIX-like OS that comes with curl and not wget is OS X. That might also drive adoption."
Ahh interesting point. I guess that would have an impact.
Yeah, I've spent some time playing with libcurl as well. However I hadn't considered your point about wget being out of scope because this is about libraries rather than utilities. Good point :)
That said, including JVM startup time is a very odd way to benchmark HTTP libraries. Is it a common enough use case to make Java apps that just start up and download a single file?
I use curl across the board. I can't even begin to count the amount of times that I have seen script kiddies use some pre-made script to hack {Wordpress, Drupal, Joomla, whatever other PHP script} and try to use wget to download their new code. wget isn't installed on my servers, so it fails. curl is installed, but they never seem to reach for that.
Keeping wget off my servers has stopped countless attacks, yes the PHP scripts should be updated... but shared hosting isn't all that great as a sys-admin.
Curl is the default on most systems I've used. The only place wget seems to be a default is on Ubuntu (and probably some other flavors of Linux).
But most places where I need such a tool: My desktop (Mac), the load-balancers (OpenBSD), the database servers (FreeBSD), the hypervisors (SmartOS), the NFS servers (FreeBSD), basically all the infrastructurey stuff, curl is the default. I wouldn't swear to it, but I don't think wget is installed on most of those systems.
Comments
Interesting article, but do many people use curl for binary downloads? I thought most sysadmins used it for viewing headers and/or parsing raw text. For binary downloads wget seems to be favoured. So with that in mind, I'd be curious to see how wget compared with the other solutions in those benchmarks.
DISCLAIMER: I'm not trying to imply that my way is "better" nor the only way HTTP GETs should be called. I'm only commenting on how I generally work and the trend I've anecdotally observed. Though I'd be interested to see if many others on here do prefer curl over wget for pulling binary files.
and this is...?
A demonstration about how OS X ships curl but not wget. Essentially he's just running
to demonstrate that curl is in $PATH where as the shell cannot find wget (since we're talking about base installs it's safe to assume that the aforementioned utilities should be in $PATH by default - if included at all).Thanks, that makes sense now.
This is how I ended up consistently prefering curl:
Last time I looked closely, I had better control with curl, and curl had better support for stuff. For example HTTP 1.1 was supported by curl, but not wget.
This makes curl a far better choice for testing HTTP servers.
And next, since I now know curl pretty well, and have it installed all over the place, it is my go-to HTTP command line client. I don't know of any reason to use both, or any reason to prefer wget, although in some situations maybe "wget <url>" acts slightly more like you want than "curl <url>". (Which is of course easy to work around by using "curl -OL <url>", if that's what you want, but convenient defaults are still convenient)
> "Last time I looked closely, I had better control with curl, and curl had better support for stuff. For example HTTP 1.1 was supported by curl, but not wget.
Well yes. You're just reiterating what I said about curl being great for server testing. But this article is about download performance. When just downloading a largish binary file (eg a tarball), it's easier to just:
With that, you get a progress bar, it saves the file to disk and basically just does all the sane options you'd want for downloading by default (which I think is why most INSTALL / README docs tend to recommend wget in their instructions).Again, I'm not saying one is better than the other, but the convenience of wget's defaults (which I think you also highlighted later in you post) does tend to make me favour it for downloading binary content (with curl being my go-to for any server testing or text resources).
> "And next, since I now know curl pretty well, and have it installed all over the place, it is my go-to HTTP command line client."
wget also comes as part of the basic install with almost all Linux and other UNIX-like OSs as well. In fact wget actually pre-dates curl (albeit they're both >= 15 years old, so there's not really much between the two relatively speaking) so I'd have expected even your oldest live systems would still have wget.
I can totally relate to the "know[ing] curl pretty well" point though. There is definitely an argument for reusing the tools that you're already familiar with.
We seem to be in full agreement. I only wanted to spell out my own reasoning for using curl and not wget, since you explicitly said you were interested in this information, and I suppose there are others that think like me in this regard as well :)
One popular UNIX-like OS that comes with curl and not wget is OS X. That might also drive adoption.
(I seem to remember having had to install both curl and wget on Ubuntu, but I am unable to verify this now)
> "We seem to be in full agreement. I only wanted to spell out my own reasoning for using curl and not wget, since you explicitly said you were interested in this information, and I suppose there are others that think like me in this regard as well :)"
I'm grateful for your input. Sorry if I came across as elitist - that wasn't my intention but in reflection, in does read a little that way.
> "One popular UNIX-like OS that comes with curl and not wget is OS X. That might also drive adoption."
Ahh interesting point. I guess that would have an impact.
Well cURL is also an HTTP library which makes it a better comparison with other HTTP libraries than a file download utility.
Yeah, I've spent some time playing with libcurl as well. However I hadn't considered your point about wget being out of scope because this is about libraries rather than utilities. Good point :)
That said, including JVM startup time is a very odd way to benchmark HTTP libraries. Is it a common enough use case to make Java apps that just start up and download a single file?
Do people use curl from the command line directly? Maybe, maybe not. But this article is really about libcurl's performance.
And libcurl is used quite extensively to download all sorts of things, including binaries. Git, for example, makes heavy use of curl for http remotes.
I use curl across the board. I can't even begin to count the amount of times that I have seen script kiddies use some pre-made script to hack {Wordpress, Drupal, Joomla, whatever other PHP script} and try to use wget to download their new code. wget isn't installed on my servers, so it fails. curl is installed, but they never seem to reach for that.
Keeping wget off my servers has stopped countless attacks, yes the PHP scripts should be updated... but shared hosting isn't all that great as a sys-admin.
Curl is the default on most systems I've used. The only place wget seems to be a default is on Ubuntu (and probably some other flavors of Linux).
But most places where I need such a tool: My desktop (Mac), the load-balancers (OpenBSD), the database servers (FreeBSD), the hypervisors (SmartOS), the NFS servers (FreeBSD), basically all the infrastructurey stuff, curl is the default. I wouldn't swear to it, but I don't think wget is installed on most of those systems.
I typically use curl and prefer it, but one feature I find it lacking which wget has is recursive downloading.
https://www.gnu.org/software/wget/manual/html_node/Recursive...