Microsoft unzips Zipline, lets world+dog have a go with cloudy storage compression tech

[Mark-x]

 

 Microsoft used the Open Compute Project (OCP) Global Summit to announce the open-sourcing of the company's cloudy compression technology, Project Zipline.

Disappointingly not a wire strung between data centres, along which techies can whizz with armfuls of USB sticks – an aerial Sneakernet if you will – Project Zipline is all about a compression algorithm, aimed at addressing the exponential growth of data created across the world.

Those playing the Microsoft drinking game – namely taking a swig as soon as "Artificial Intelligence" is mentioned in the company's emissions – will be delighted to know that the company is waving the finger of blame at the technology as more and more datapoints are required to feed algorithms that drive everything from business processes through to taking a stab at predicting a meatbag's upcoming schedule.

Pointing to IDC forecasts that over 22 ZB of storage capacity must ship across all media types from 2018 to 2025 to keep up with storage demand, Microsoft has been working on methods to prevent its Azure data centres bursting at their seams while data-intensive workloads continue to grow.

Enter Project Zipline, a compression algorithm which, along with some optimised hardware, is geared up for the types of data Microsoft has spied in its cloud storage. We checked with Microsoft to find out if that included a look at customer bits and bytes, but have yet to receive a response.

Naturally, the company reckons that it has "been able to simultaneously achieve higher compression ratios, higher throughput, and lower latency than the other algorithms that are currently available" before casting a little shade on the commonly used Zlib-L4 64KB model. Microsoft claims the Project Zipline tech can achieve up to 2X high compression ratios in comparison.

Microsoft is open-sourcing the compression algorithms, hardware specs and Verilog code in the hope that the community and hardware vendors will pick it up and run with it. Contributions from the OCP ecosystem wouldn’t go amiss either – after all, everyone benefits, right? "Including Azure and our customers" according to the software giant. ®