Download file from url with curl incomplete file

The problem is with your source: the server. I tried running your scraper on scraperwiki and here is what it shows: Also, same problem occured when I personally tried loading the xml and it worked for me the third time. Community Bot 1 1 1 silver badge. Prasanth Prasanth 5, 2 2 gold badges 25 25 silver badges 58 58 bronze badges.

I was afraid that this is the case. Unfortunately I can't do anything about it. The only think that was bugging me is that when I run the script from my localhost it has by far more chances to get the file successfully and in first attempt, but when running the script from my host server it does worse by far.

Is there anything tha may be causing this difference in my host server? Sign up or log in Sign up using Google. Sign up using Facebook. Sign up using Email and Password. Post as a guest Name. Email Required, but never shown. The Overflow Blog. It will probably have excess parameters you can trim-away.

So this will result in a working response from a web browser but not a working response in a backend web request like curl. I just ran into a site that required all the following headers. Failure to specify them resulted in a timeout. Sign up to join this community. The best answers are voted up and rise to the top. Stack Overflow for Teams — Collaborate and share knowledge with a private group. Create a free Team What is Teams? Learn more. Why won't curl download this link when a browser will?

In this case, you have to add the -L or --location option to tell cURL to follow the redirects. The number of bytes can be abbreviated by appending a suffix: kilobytes k or K , megabytes m or M , and gigabytes g or G. Do not! In the case of our readme, the complete command would like this:.

So what if we want to use cURL to save a file to our server? For that, we must use the -O option:. You'll notice that cURL will display a download progress table rather than the file contents this time:. If you'd like the file to have a different file name perhaps readme is too bossy for you , specify it after -O :. Out of the box, without any command-line arguments, the curl command will fetch a file and display its contents to the standard output.

Fetching a file and display its contents is all well and good, but what if you want to actually save the file to your system? You can check on things with the cat command:. Execute the following command to download the remote robots. Now use the cat command to display the contents of do-bots. If you happened to try to fetch the robots. You can verify this by using the -I flag, which displays the request headers rather than the contents of the file:.