org.netpreserve.jwarc

Class WarcWriter

java.lang.Object

org.netpreserve.jwarc.WarcWriter

All Implemented Interfaces:

Closeable, AutoCloseable

public class WarcWriter
extends Object
implements Closeable

Writes records to a WARC file.

Constructor Summary

Constructors

Constructor and Description
WarcWriter(OutputStream stream)
WarcWriter(Path path) Opens a WARC file for writing.
WarcWriter(WritableByteChannel channel)
WarcWriter(WritableByteChannel channel, WarcCompression compression)

Method Summary

Modifier and Type	Method and Description
void	close() Closes the channel.
FetchResult	fetch(URI uri) Downloads a remote resource recording the request and response as WARC records.
FetchResult	fetch(URI uri, FetchOptions options) Downloads a remote resource recording the request and response as WARC records.
FetchResult	fetch(URI uri, HttpRequest httpRequest, FetchOptions options) Downloads a remote resource recording the request and response as WARC records.
FetchResult	fetch(URI uri, HttpRequest httpRequest, OutputStream copyTo) Downloads a remote resource recording the request and response as WARC records.
long	position() Returns the byte position the next record will be written to.
void	write(WarcRecord record)

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

WarcWriter

public WarcWriter(WritableByteChannel channel,
                  WarcCompression compression)
           throws IOException

Throws:

IOException

WarcWriter

public WarcWriter(WritableByteChannel channel)
           throws IOException

Throws:

IOException

WarcWriter

public WarcWriter(OutputStream stream)
           throws IOException

Throws:

IOException

WarcWriter

public WarcWriter(Path path)
           throws IOException

Opens a WARC file for writing. Compression is determined by the file extension.

Parameters:

path - the path to the file

Throws:

IOException - if an I/O error occurs

Method Detail

write

public void write(WarcRecord record)
           throws IOException

Throws:

IOException

fetch

public FetchResult fetch(URI uri)
                  throws IOException

Downloads a remote resource recording the request and response as WARC records.

Throws:

IOException

fetch

public FetchResult fetch(URI uri,
                         FetchOptions options)
                  throws IOException

Downloads a remote resource recording the request and response as WARC records.

Parameters:

uri - URL to download

options - fetch options to use

Throws:

IOException - if an IO error occurred

fetch

public FetchResult fetch(URI uri,
                         HttpRequest httpRequest,
                         OutputStream copyTo)
                  throws IOException

Downloads a remote resource recording the request and response as WARC records.

Parameters:

uri - URL to download

httpRequest - request to send

copyTo - if not null will receive a copy of the (raw) http response bytes

Throws:

IOException - if an IO error occurred

fetch

public FetchResult fetch(URI uri,
                         HttpRequest httpRequest,
                         FetchOptions options)
                  throws IOException

Downloads a remote resource recording the request and response as WARC records.

Parameters:

uri - URL to download

httpRequest - request to send

options - fetch options to use

Throws:

IOException - if an IO error occurred

position

public long position()

Returns the byte position the next record will be written to.

If the underlying channel is not seekable the returned value will be relative to the position the channel was in when the WarcWriter was created.

close

public void close()
           throws IOException

Closes the channel. If there are any active fetches they will be truncated and current progress written.

Specified by:

close in interface Closeable

Specified by:

close in interface AutoCloseable

Throws:

IOException