Compress files faster using Snappy

       0

We aggregate and tag open source projects. We have collections of more than one million projects. Check out the projects section.

Snappy is the fast compression/decompression library from Google. It does not target to reduce compression size but it does faster compression. Most of the open source products use Snappy.

<dependency>
    <groupId>org.xerial.snappy</groupId>
    <artifactId>snappy-java</artifactId>
    <version>1.1.7.2</version>
</dependency> 

public class SnappyTest {

  public static void testCompress(String inFilePath, String outFilePath) {

   try {
       long start = Calendar.getInstance().getTimeInMillis();
       Path inPath = FileSystems.getDefault().getPath(inFilePath);
       Path outPath = FileSystems.getDefault().getPath(outFilePath);

       byte[] compressed = Snappy.compress(Files.readAllBytes(inPath));
       Files.write(outPath, compressed, StandardOpenOption.CREATE);

       System.out.println("Snappy time " + (Calendar.getInstance().getTimeInMillis() - start));
  } 
  catch (IOException e) {
     e.printStackTrace();
  }
}

 public static void testCompressZip(String inFilePath, String outFilePath) {

 try {
     long start = Calendar.getInstance().getTimeInMillis();
     Path inPath = FileSystems.getDefault().getPath(inFilePath);
     Path outPath = FileSystems.getDefault().getPath(outFilePath);

     ZipOutputStream zos = new ZipOutputStream(new FileOutputStream(outFilePath));
     zos.putNextEntry(new ZipEntry(inPath.getFileName().toString()));

     byte data[] = Files.readAllBytes(inPath);
     zos.write(data, 0, data.length);
     zos.closeEntry();
     zos.close();
     System.out.println("Zip time " + (Calendar.getInstance().getTimeInMillis() - start));
 } 
 catch (IOException e) {
     e.printStackTrace();
 }
}
}

public static void main( String[] args )
{
   String inDir = "<input-directory>";
   String outDir = "<output-directory>";

   File inDirFile = new File(inDir);
   String inputFiles[] = inDirFile.list();
   for (String inputFile : inputFiles) {
        String inFile = inDir + File.separator + inputFile;
        String outSFile = outDir + File.separator + "output-s" + index + ".out";
        String outzFile = outDir + File.separator + "output-z" + index + ".zip";

        System.out.println("File: " + inputFile);
        SnappyTest.testCompress(inFile, outSFile);
        SnappyTest.testCompressZip(inFile, outzFile);
    }

    Snappy.cleanUp();
}

Intially i tested for couple of files and saw Snappy is taking more time and size. Then i dropped lot of files (close to 100) in the input directory and tried to run the above code, Snappy outperfomed JDK compression.

Snappy is good library, when we have lots of data to compress and we need faster compression with slight compromise on compression size.

Reference:
https://google.github.io/snappy/