“Every day, developers check in large volumes of code, and each line of code translates into additional bits in the apps that people ultimately download onto their phones,” he said. “Left unchecked, this added code would make the app bigger and bigger until eventually the time it takes to download would become unacceptable.”
In a lengthy post on the Facebook Engineering blog today, Facebook software engineer Saphan Batia talks about the unique challenge of managing application size. He explains that Facebook’s developers are constantly adding new bits of code to the company’s mobile apps, including the Facebook app, Messenger, WhatsApp and Instagram.
Facebook’s app developers already employ a bunch of data compression techniques, but the company said these methods cannot keep up with the rate at which it adds new features and updates.
According to Batia, Superpack’s strength lies in its ability to compress specific types of code such as machine code and bytecode as well as some kinds of structured data. He explained that the basic underlying approach to Superpack is based on an insight in Kolmogorov’s algorithmic measure of complexity, which defines the information content of a piece of data as the length of the shortest program that can generate that data. What that means is that it’s possible to compress data by representing it as a program that generates that data, as opposed to compressing the information itself.
So for the last two years, Facebook’s team has been working on the Superpack technique, which it claims has been able to achieve “significantly better” compression ratios than existing tools.
“When that data is code to begin with then it can be transformed into one with a smaller compressed representation,” he wrote. “A program that generates Fibonacci numbers coupled with a list of indices is a highly compressed representation of a file containing such numbers.”
He added that the idea of reducing Kolmogorov complexity is not a new one in the area of data compression. What’s new is that Superpack employs a combination of compiler methods with modern compression techniques to achieve that goal.
Batia said Facebook has employed Superpack to put a check on developer-induced app growth in the past couple of years and actually managed to reduce the size of its Android apps. They are now “substantially smaller in comparison to regular Android APK compression,” Batia wrote, with the average app size reduction amounting to about 20%.
The main Facebook app on Android, for example, has actually been reduced by more than 40% using Superpack, while Messenger has achieved a reduction of over 30%. For reasons that were not discussed, the technique appears to have been less successful with WhatsApp. Although Superpack was conceived as a way of reducing mobile app size, Facebook believes it has potential in many other areas where data compression is desirable. The company is currently working on applying the technique to a new on-demand executable file format that can save disk space by keeping shared libraries compressed and decoding them only at load time.
It’s also looking at using Superpack to enable compression of code to reduce the size of software updates. Finally, Facebook sees potential to use Superpack as a compressor for cold storage files that are rarely used. “Presently, Superpack is only available to our engineers but we aspire to bring the benefits of Superpack to everyone,” Batia said. “To this end we are exploring ways to improve the compatibility of our compression work with the Android ecosystem. We may someday consider open-sourcing Superpack.”
Images: Facebook Show your support for our mission by joining our Cube Club and Cube Event Community of experts. Join the community that includes Amazon Web Services and Amazon.com CEO Andy Jassy, Dell Technologies founder and CEO Michael Dell, Intel CEO Pat Gelsinger and many more luminaries and experts.
Join Our Community We are holding our third cloud startup showcase on Sept. 22. Click here to join the free and open Startup Showcase event.
“TheCUBE is part of re:Invent, you know, you guys really are a part of the event and we really appreciate your coming here and I know people appreciate the content you create as well” – Andy Jassy We really want to hear from you, and we’re looking forward to seeing you at the event and in theCUBE Club.