Rectangle 27 1

How to download and save a file from Internet using Java?


8388608
Integer.MAX_VALUE
Long.MAX_VALUE
URL website = new URL("http://www.website.com/information.asp");
ReadableByteChannel rbc = Channels.newChannel(website.openStream());
FileOutputStream fos = new FileOutputStream("information.html");
fos.getChannel().transferFrom(rbc, 0, Long.MAX_VALUE);

A single call isn't adequate. transferFrom() isnt' specified to complete the entire transfer in a single call. That's why it returns a count. You have to loop.

Give a try to Java NIO:

Using transferFrom() is potentially much more efficient than a simple loop that reads from the source channel and writes to this channel. Many operating systems can transfer bytes directly from the source channel into the filesystem cache without actually copying them.

hey what is that shifty bit?

Note
Rectangle 27 1

How to download and save a file from Internet using Java?


BufferedInputStream in = new BufferedInputStream(new URL("http://www.website.com/information.asp").openStream())
    byte data[] = new byte[1024];
    int count;
    while((count = in.read(data,0,1024)) != -1)
    {
        out.write(data, 0, count);
    }

Downloading a file requires you to read it, either way you will have to go through the file in some way. Instead of line by line, you can just read it by bytes from the stream:

Java complains about initializing in a while line, and also gives the error "The operator != is undefined for the argument type(s) int, null"

Note
Rectangle 27 1

How to download and save a file from Internet using Java?


import java.io.*;
import java.net.*;

public class filedown {
    public static void download(String address, String localFileName) {
        OutputStream out = null;
        URLConnection conn = null;
        InputStream in = null;

        try {
            URL url = new URL(address);
            out = new BufferedOutputStream(new FileOutputStream(localFileName));
            conn = url.openConnection();
            in = conn.getInputStream();
            byte[] buffer = new byte[1024];

            int numRead;
            long numWritten = 0;

            while ((numRead = in.read(buffer)) != -1) {
                out.write(buffer, 0, numRead);
                numWritten += numRead;
            }

            System.out.println(localFileName + "\t" + numWritten);
        } 
        catch (Exception exception) { 
            exception.printStackTrace();
        } 
        finally {
            try {
                if (in != null) {
                    in.close();
                }
                if (out != null) {
                    out.close();
                }
            } 
            catch (IOException ioe) {
            }
        }
    }

    public static void download(String address) {
        int lastSlashIndex = address.lastIndexOf('/');
        if (lastSlashIndex >= 0 &&
        lastSlashIndex < address.length() - 1) {
            download(address, address.substring(lastSlashIndex + 1));
        } 
        else {
            System.err.println("Could not figure out local file name for "+address);
        }
    }

    public static void main(String[] args) {
        for (int i = 0; i < args.length; i++) {
            download(args[i]);
        }
    }
}

If in.close throws an exception, out.close is not called.

Note
Rectangle 27 1

How to download and save a file from Internet using Java?


BufferedInputStream
public void saveUrl(final String filename, final String urlString)
        throws MalformedURLException, IOException {
    BufferedInputStream in = null;
    FileOutputStream fout = null;
    try {
        in = new BufferedInputStream(new URL(urlString).openStream());
        fout = new FileOutputStream(filename);

        final byte data[] = new byte[1024];
        int count;
        while ((count = in.read(data, 0, 1024)) != -1) {
            fout.write(data, 0, count);
        }
    } finally {
        if (in != null) {
            in.close();
        }
        if (fout != null) {
            fout.close();
        }
    }
}

@ComFreek That is simply untrue. Using a BufferedInputStream has precisely zero effect on socket timeouts. I had already refuted that as 'urban myth' in my comments to the 'background details' you cited. Three years earlier.

How do I show the percentage downloaded

If in.close throws an exception, fout.close is not called.

Note
Rectangle 27 1

How to download and save a file from Internet using Java?


public static void downloadFromUrl(String downloadUrl,
        String targetFilePath) throws MalformedURLException, IOException
{
    URL url = new URL(downloadUrl);

    Files.copy(url.openStream(), new File(targetFilePath).toPath(),
            StandardCopyOption.REPLACE_EXISTING);
}

When using Java 7+ simply use the following method to download a file from the Internet and save it to some directory:

Note
Rectangle 27 1

How to download and save a file from Internet using Java?


URL website = new URL("http://www.website.com/information.asp");
try (InputStream in = website.openStream()) {
    Files.copy(in, target, StandardCopyOption.REPLACE_EXISTING);
}

@Miere It is impossible for InputStream.read() to return zero unless you provided a zero length buffer or count, 'little pause' or otherwise. It will block until at least one byte has been transferred or end of stream or an error occurs. Your claim about the internals of Files.copy() is baseless.

Despite the fact this is an elegant solution, behind the scenes this approach could silently betray you. Files.copy( InputStream, Paths, FileOption) delegates the copy process to the Files.copy( InputStream, OutputStream ). This last method does not check for the end of stream (-1) but checks for no byte read (0). It means that, if your network had a little pause, it could read 0 bytes and end the copy process, even if the stream isn't finished to be downloaded by the OS.

Even though the InputStream of the URL never gets closed, this should be the accepted answer. (See comment of @EmmanuelTouzery). Files.copy(in, path) is by far the simplest way to this.

Great way! I'd still put the .openStream() in a try-with-resources block.

I have an unit test that reads a binary file with 2.6TiB. Using Files.copy it always fails on my HDD storage server (XFS) but it fails only a few times my SSH one. Looking at JDK 8 the code of File.copy I've identified that it checks for '> 0' to leave the 'while' loop. I just copied the exactly same code with the -1 and both unit tests never stopped again. Once InputStream can represent Network and local file descriptors, and both IO operations are subject to OS context switching, I cant see why my claim is baseless. One may claim it be working by luck, but it gave no headaches any more.

Note
Rectangle 27 1

How to download and save a file from Internet using Java?


FileUtils.copyURLToFile(URL, File)

...and when using that overloaded version, remember that the timeouts are specified in milliseconds, not seconds.

Nice! Just what I'm looking for! I knew Apache libraries would already cover this. BTW, it's recommended to use the overloaded version with timeout parameters!

Take note that copyURLToFile with timeout parameter is only available since version 2.0 of Commons IO library. See Java docs

what if basic authentication header has to be added to the request? is there a workaround?

Note
Rectangle 27 1

How to download and save a file from Internet using Java?


public static void downloadFileFromURL(String urlString, File destination) throws Throwable {

      URL website = new URL(urlString);
      try(
              ReadableByteChannel rbc = Channels.newChannel(website.openStream());
              FileOutputStream fos = new FileOutputStream(destination);  
              ){
          fos.getChannel().transferFrom(rbc, 0, Long.MAX_VALUE);
      }

  }

A single call isn't adequate. transferFrom() isnt' specified to complete the entire transfer in a single call. That's why it returns a count. You have to loop.

Note
Rectangle 27 1

How to download and save a file from Internet using Java?


Personally, I've found Apache's HttpClient to be more than capable of everything I've needed to do with regards to this. Here is a great tutorial on using HttpClient

Note
Rectangle 27 1

How to download and save a file from Internet using Java?


/**
 * Downloads from a (http/https) URL and saves to a file. 
 * Does not consider a connection error an Exception. Instead it returns:
 *  
 *    0=ok  
 *    1=connection interrupted, timeout (but something was read)
 *    2=not found (FileNotFoundException) (404) 
 *    3=server error (500...) 
 *    4=could not connect: connection timeout (no internet?) java.net.SocketTimeoutException
 *    5=could not connect: (server down?) java.net.ConnectException
 *    6=could not resolve host (bad host, or no internet - no dns)
 * 
 * @param file File to write. Parent directory will be created if necessary
 * @param url  http/https url to connect
 * @param secsConnectTimeout Seconds to wait for connection establishment
 * @param secsReadTimeout Read timeout in seconds - trasmission will abort if it freezes more than this 
 * @return See above
 * @throws IOException Only if URL is malformed or if could not create the file
 */
public static int saveUrl(final Path file, final URL url, 
  int secsConnectTimeout, int secsReadTimeout) throws IOException {
    Files.createDirectories(file.getParent()); // make sure parent dir exists , this can throw exception
    URLConnection conn = url.openConnection(); // can throw exception if bad url
    if( secsConnectTimeout > 0 ) conn.setConnectTimeout(secsConnectTimeout * 1000);
    if( secsReadTimeout > 0 ) conn.setReadTimeout(secsReadTimeout * 1000);
    int ret = 0;
    boolean somethingRead = false;
    try (InputStream is = conn.getInputStream()) {
        try (BufferedInputStream in = new BufferedInputStream(is); OutputStream fout = Files
                .newOutputStream(file)) {
            final byte data[] = new byte[8192];
            int count;
            while((count = in.read(data)) > 0) {
                somethingRead = true;
                fout.write(data, 0, count);
            }
        }
    } catch(java.io.IOException e) { 
        int httpcode = 999;
        try {
            httpcode = ((HttpURLConnection) conn).getResponseCode();
        } catch(Exception ee) {}
        if( somethingRead && e instanceof java.net.SocketTimeoutException ) ret = 1;
        else if( e instanceof FileNotFoundException && httpcode >= 400 && httpcode < 500 ) ret = 2; 
        else if( httpcode >= 400 && httpcode < 600 ) ret = 3; 
        else if( e instanceof java.net.SocketTimeoutException ) ret = 4; 
        else if( e instanceof java.net.ConnectException ) ret = 5; 
        else if( e instanceof java.net.UnknownHostException ) ret = 6;  
        else throw e;
    }
    return ret;
}

Here's a method that does not throw Exceptions for network errors (only for truly exceptional problems, as malformed url or problems writing to the file)

There are many elegant and efficient answers here. But the conciseness can make us lose some useful information. In particular, one often does not want to consider a connection error an Exception, and one might want to treat differently some kind of network-related errors - for example, to decide if we should retry the download.

Note