Quick Tip: EXPORT TO PARQUET Compression with GZIP, Brotli, and ZSTD

EXPORT TO PARQUET exports a table, columns from a table, or query results to files in the Parquet format. These Parquet files use Snappy compression…

OpenText profile picture
OpenText

May 25, 20212 minute read

EXPORT TO PARQUET exports a table, columns from a table, or query results to files in the Parquet format. These Parquet files use Snappy compression by default.

Starting in Vertica 10.1.1, EXPORT TO PARQUET supports the GZIP, Brotli, and ZSTD compression types!

Let’s see how these compression types compare in disk usage:

verticademos=> SELECT COUNT(*) FROM big;
   COUNT
-----------
134217728
(1 row)

Snappy compression:

verticademos=> EXPORT TO PARQUET (directory = '/home/dbadmin/parq_snappy') AS SELECT * FROM big;
Rows Exported
---------------
     134217728
(1 row)

verticademos=> \! du --summarize -h /home/dbadmin/parq_snappy
2.6G    /home/dbadmin/parq_snappy

GZIP compression:

verticademos=> EXPORT TO PARQUET (directory = '/home/dbadmin/parq_gzip', compression='GZIP') AS SELECT * FROM big;
Rows Exported
---------------
     134217728
(1 row)

verticademos=> \! du --summarize -h /home/dbadmin/parq_gzip
1.9G    /home/dbadmin/parq_gzip

Brotli compression:

verticademos=> EXPORT TO PARQUET (directory = '/home/dbadmin/parq_Brotli', compression='Brotli') AS SELECT * FROM big;
Rows Exported
---------------
    134217728
(1 row)

verticademos=> \! du --summarize -h /home/dbadmin/parq_Brotli
1.7G    /home/dbadmin/parq_Brotli

ZSTD compression:

verticademos=> EXPORT TO PARQUET (directory = '/home/dbadmin/parq_ZSTD', compression='ZSTD') AS SELECT * FROM big;
Rows Exported
---------------
     134217728
(1 row)

verticademos=> \! du --summarize -h /home/dbadmin/parq_ZSTD
1.7G    /home/dbadmin/parq_ZSTD

Hint: Although we can see in the example that the Brotli and ZSTD compression methods offer similar savings on disk space, there are other factors to keep in mind about using these methods in practice; in reality, the ZSTD performs much, much better than Brotli.

Have fun!

Share this post

Share this post to x. Share to linkedin. Mail to
OpenText avatar image

OpenText

OpenText, The Information Company, enables organizations to gain insight through market-leading information management solutions, powered by OpenText Cloud Editions.

See all posts

More from the author

Manutan combines digital services with the human touch to delight customers

Manutan combines digital services with the human touch to delight customers

At Manutan, we equip businesses and communities with the products and services they require to succeed. Headquartered in France, our company has three divisions, serving…

4 minute read

Reaching new markets in Europe and beyond

Reaching new markets in Europe and beyond

How information management specialists at One Fox slashed time to market for innovative products with OpenText Cloud Platform Services At One Fox, we’ve driven some…

4 minute read

SoluSoft helps government agencies tackle fraud faster

SoluSoft helps government agencies tackle fraud faster

Fraud, in all its forms, is a pervasive problem, spanning industries and preying on vulnerabilities in federal and state government systems. Each year in the…

3 minute read

Stay in the loop!

Get our most popular content delivered monthly to your inbox.