64

Many Android devices, including the Google Nexus line, are now receiving monthly security patches via OTA updates, accompanied by the Android Security Bulletins. However, these updates are often released in what is known as "staggered roll outs," where the update is available for more devices (of the same model) over time, instead of becoming available to all users instantaneously.

I understand that for feature updates, gradual roll outs allow errors or bugs to be fixed before all users receive the new software. However, for security patches, wouldn't a staggered release make it much easier for blackhat hackers to utilize the now-public vulnerabilities against users whose devices have not yet received the OTA, even though a patch for their device model is already available?

For example, I have a Google Nexus phone that is supposed to be among the first to receive all Android updates. The latest Android security update was publicly released on May 4, 2016, along with its source code; it is already May 16, 2016, and my Nexus device is still telling me that my "system is up to date," even when I click the "check for update" button. Of course, I can manually download the latest firmware images and flash it manually; but shouldn't security updates be made available to all devices of the same model as quickly as possible once it becomes public?

Edit: Thank you for the very thoughtful responses. While this question is intended to be widely applicable to different devices, my specific concern is the intentional "staggered roll out" to identical devices of the same model, once a patch has already been developed for that specific model. For example, the May 2016 security update has been released for the Google Nexus 6P at the beginning of the month, but as of May 16 not all Nexus 6P devices have received the OTA update. Google releases security patches for Nexus devices every month, and each month some devices receive them a few weeks later than other devices of the same model.

tonytan
  • 698
  • 5
  • 8

3 Answers3

57

Excellent question.

Yes, your understanding is correct, as well as your rationale behind it.

Staggering roll outs for new features often makes good sense.

Staggering roll outs for security patches rarely is a good idea. As you pointed out, this gives even more opportunity for the vulnerabilities to be exploited. Perhaps even more importantly, the patches can be quickly reverse engineered to develop exploits in a rapid fashion.

Microsoft often publicly releases their patches on the second Tuesday of the month (and also sometimes on the fourth Tuesday). This has commonly been referred to as "Patch Tuesday". There's a reason we call the next day "Exploit Wednesday".

It's unfortunate that a significant chunk of the Android ecosystem has not learned from this phenomenon.

Updates:
Several knowledgeable people have pointed out potential impact on internet infrastructure, including fears of overloading the entire internet. The volume of internet traffic is monumental; security patches, even extremely large ones, are tiny drops in the bucket. Microsoft releases large patches to hundreds of millions of users on the same day each month, and they have yet to "crash the internet". Netflix, YouTube, and Twitch stream videos to millions of people every day, and even with their combined traffic, they have yet to "crash the internet".

On the other hand, Android patches are predominately (but not exclusively) delivered to wireless users. There are solutions to any potential issues:

  1. Provide users with a choice of when to download the patches. This provides numerous benefits:
    • Does not disrupt the user's workflow
    • Creates traffic staggering due to human interaction and decision variability
    • Allows the user to wait until they are connected to a higher bandwidth system (perhaps at work, their university, at home WiFi)
    • Allows the user, at their own risk, to wait and see if others report problems with the patches
  2. When distributing patches to specific regions known to have limited infrastructure that could be impacted, stagger the patches over the minimal number of days to avoid overloading infrastructure.

In regards to specifically the Google Nexus 6P security updates not being released to all users promptly, that's simply a poor choice by Google that is not in the best interest of their customers. Compared to the massive volume of internet traffic, those patches are minuscule.

On top of that, that device is relatively rare in the Android ecosystem. This further supports the statement that releasing the patches to all customers at once would not harm any internet providers.

Even the entire Google Nexus product line comprises only a tiny part of the Android world. As a product line, however, there could be a little impact on infrastructure in select regions. As such, the following methodology, while combined with the recommendations outlined above, will minimize infrastructure impact while maximizing patch distribution:

  1. Release zero-day exploit patches immediately
  2. Release scheduled updates on different days each month, a different day for each product
  3. If a product has significant market share that could reasonably impact infrastructure in a region, stagger the roll out over the minimal number of days required to avoid infrastructure overload in that region only

Finally, according to your statements, it has been over two weeks since Google initially released those patches for the Google Nexus 6P. That's more than enough time to know if their patches are causing havoc. I have found no documentation from Google recognizing or apologizing for a bad bunch of patches, nor anecdotal evidence of any serious problems.

One could make the argument that staggering patches out over a few days could possibly be reasonable in order to detect flawed patches and to reduce traffic load. But leaving customers unpatched for weeks is unreasonable, unnecessary, and not an effective policy from an information security standpoint.

In conclusion, based on the statements above, and your statement that Google has not rolled out the security patches to your device, my conclusion is that Google, by not delivering security patches to all affected Google Nexus 6P customers, is making a poor decision and is doing a disservice to their customers.

RockPaperLz- Mask it or Casket
  • 3,114
  • 21
  • 50
  • 6
    When in doubt, manually force it with the biggest sledge hammer you can find. I've had auto-update staggered rollouts where there's a week difference between first and last application. On a zero day, that's inexcusable and tells you a lot about how much the offender values you. – Fiasco Labs May 16 '16 at 05:13
  • 1
    Part of the difficulty is load balancing. If everyone's phone tries to download the OTA at exactly midnight of the same day... the servers are going to melt and it will take even *longer* for the update to get out. – Kevin May 16 '16 at 06:37
  • 8
    @Kevin When we're talking about Google, Microsoft and Apple, I see this as an excuse. For companies this size, it would definitly not be a problem to deliver 100mio patches a day (at least from the network perspective) – Lukas May 16 '16 at 07:51
  • 8
    It would certainly be trivial for Google to make an update available to every device at the same time, especially if they'd already rolled it out to local servers around the world. This is not the issue. – James Hyde May 16 '16 at 08:42
  • 8
    There's a tension between potentially bricking millions of devices because of a bad patch and widening the vulnerability window. The former is much worse for PR. – Roger Lipscombe May 16 '16 at 10:56
  • 1
    @RogerLipscombe I believe that premise may not be accurate. A patch, once reviewed and tested, would likely not *brick* millions of devices. A poorly coded patch may not work, or could cause some compatibility issues, but if it's gone through even a little testing, it would be very unlikely that it would *brick* millions of devices. – RockPaperLz- Mask it or Casket May 16 '16 at 11:03
  • 10
    @RockPaperLizard - The real world NEVER matches the QA lab, no matter how good your QA lab. The fear that a new release is going to break stuff is VERY WELL founded in a history (from even the largest companies) of perfectly tested patches breaking everything in site. – Michael Kohne May 16 '16 at 11:26
  • 3
    @MichaelKohne Even Microsoft, who has a questionable record of releasing quality patches, rarely *bricks* devices, even though they release security patches simultaneously to **hundreds of millions of systems**. – RockPaperLz- Mask it or Casket May 16 '16 at 11:29
  • 3
    @JamesHyde android OS is heavily modified by manufacturers and carriers and sometimes are incompatible with one another. So an update cannot be rolled into all devices because it needs to be modified to work with the modded OS'es. – Mindwin May 16 '16 at 13:45
  • 1
    @Mindwin I was referring to the Nexus line of devices which receive monthly updates directly from Google. – James Hyde May 16 '16 at 14:32
  • @JamesHyde acknowledge, that was not clear in the previous comment. Well done. – Mindwin May 16 '16 at 15:41
  • 2
    There have been a lot of comments about capacity to deliver patches, but these comments focus on servers and the wired network. Remember that the "last mile" for many of these devices is a wireless network. And the carriers have their concerns, too. They don't want these updates to turn into a DOS on their system. – user19474 May 16 '16 at 19:08
  • 1
    I thought we were getting past the era of "I have no experience with the development/deployment side of the house, but my opinion must be totally correct.". Whether or not to stagger is dependent upon multiple factors, many listed in these comments. The strain on the Internet itself even matters for certain extremely large patches, What is right for a particular company depends upon that company's risk tolerances - all the risks, not just security. – atk May 17 '16 at 00:15
  • @atk Your assumptions are erroneous. If you disagree with this answer, you are encouraged to write your own. In your answer, we look forward to you explaining how a security patch could be so massive as to place a "strain on the Internet". We look forward to learning from your extensive experience and knowledge. – RockPaperLz- Mask it or Casket May 17 '16 at 01:04
  • @Lukas I see this actually as being good corporate citizens. Microsoft and Google can certainly handle the load that a synchronized patch generates. But not every infrastructure owner between MSFT/GOOG and you will be able to handle that load. It is the small ISPs and SOHO admins which will suffer when their systems overload... – Aron May 17 '16 at 09:08
  • 2
    If there really were a valid concern about bandwidth / server load (I doubt there is) it would be possible to devise a system where the patches were delivered encrypted, and the decryption key only supplied later (and simultaneously to all devices). I would thus strongly suspect there are other reasons for the staggering. – abligh May 18 '16 at 08:40
4

However, for security patches, wouldn't a staggered release make it much easier for blackhat hackers to utilize the now-public vulnerabilities against users whose devices have not yet received the OTA, even though a patch for their device model is already available?

Easier than what?, is the important question. Yes it will be easier for the hacker for a few days while the update is being pushed out. But it will be much harder for the hacker then if then if the update was not sent at all.

A staggered release keeps the network moving, the servers running, and the 1s and 0s flowing while everyone gets their update. If every Android device out there tried to update all at once, you would end up with a massive amount of traffic hitting a tiny resource.

The other option is only doing an update when people "scan" for it. That's horrible too.

So while the staggered approach is not the absolute best, it is the best available in regard to resources.

Keep in mind that in order to take advantage of this window you would need to find a device between the time the patch was released, and when it (the device) got it's update.

Also remember that Android is Open Source. It's far more profitable to write exploits that are still viable, than ones that you know will be in the next release (cause you can see the code).

So, in summery:

  • The fixes are not a secret before the security push starts anyway.
  • It's better than no updates at all
  • It's better than crashing the mobile network or update server.
  • The window of opportunity is such that very few people should be effected, that are not already an active target.
Community
  • 1
coteyr
  • 1,506
  • 8
  • 12
  • 2
    I find it highly unlikely that an android update is going to crash the download of a static file like an OS patch. – Paul Becotte May 16 '16 at 16:06
  • 1
    I'm not sure how to respond to that. Of course it can. They range in size from 10 Megs to 1.1 Gigs. Mobile internet still sucks throughout most of the world. A 100 Meg update that takes a long time to download will fill a decent server pretty fast if 4.6 Million people do it all at once (Sales guess from the Nexus 7) – coteyr May 16 '16 at 16:20
  • 3
    @coteyr I don't think those large updates are security updates, or at best they are mostly other updates bundled with some security changes - do you have any examples? (preferably with a changelog?) – user2813274 May 16 '16 at 17:24
  • 1
    Google has what is probably the world's largest collection of servers. A little thing like a worldwide upgrade of Android won't cause it the slightest problem. – Mark May 16 '16 at 21:27
0

I would not say that I am an expert in this so my point may not be valid.

I quickly readed on Wikipedia

On modern mobile devices such as smartphones, an over-the-air update may refer simply to a software update that is distributed over Wi-Fi or mobile broadband using a function built into the operating system

From this quote we may note that the update is distributed over WiFi or mobile broadband, which not always may be available at the same time, eg. If I live in place where there is no WiFi modules, or have low signal strength, then there is possibility that I will miss the moment in which the updates are sent, and then it will be downloaded after a while.

Updates carried out by built-in manager

Some built in updating program may have settings, which allow you to customize when you want to check and download for OTA updates. Your settings may be set to check updates when device is turned on, twice per day, daily, weekly, monthly or even manually. If some devices have different settings then they may receive the updates at different time.

Updates carried out by producer

In this case the updates are carried by producer of firmware, in this case google. As already mentioned you may not be able to receive information that new update is available, because for e.g. dead battery, no signal, etc.
The updates are tend to be made when the user wont use his phone, so it do not stop user from doing whatever he/she is doing at the moment, so the update may be done at night.

Hardware difference between two the same phone models

(NOTE: this point may not be valid for some devices, including yours, as some devices may be exactly the same in first and its final release)

Sometimes even the same model of device may be a little bit different. For example at beginning xbox 360 did not had HDMI output, but after a while it was added and this probably required some other software to be made as well. Even that both types of xbox are different the model name is still the same. This make some devices with the same model name not be actually be the same inside.

There is one major difference between eg. windows 10 64 bit, and android. The difference is that the 64 bit version of windows will run on CPU which support long mode (64 bit mode) and instruction set in it will be the same in AMD and Intel. The problem with android is that different devices may have different processors with different instruction set, and this causes that phones may even require totally different code. This is causing that the updates for different devices may come in different times.

For example of this could be if you would have few WiFi cards. Lets say that you installed driver for one of them. There is very likely possibility that other cards wont run with this driver, so you would need to install different driver, which will definitely have different code. Now if the producer of this WiFi cards will find bug in all of those drivers, it is only possible to fix it one by one, or have lots of people working at the same time on different versions. However even if multiple people are working on multiple files at the same time, one may finish fix earlier than the other. In this case the update will be released quicker than for the other one.

Note that the producer could also wait for every version of code to be fixed then make update on all devices at the same time, but updating bugs even on one device decreases amount of people who would be vulnerable for this bug.

Also note that the drivers in windows are installed by you, most likely using .exe installers. In android all of those drivers are built-in, and as I mentioned before if you need different version of driver for different phones then you will spend some time on fixing all those drivers. I guess that the way google works with the update is that they are updating phones as fast as possible after they got fix for it.

Also you mentioned that even if you click manually 'check for update' button, your device says that 'system is up to date'. This may be caused because of unavailability of updating service, which is really similar to updates carried out by producer.

vakus
  • 3,743
  • 3
  • 20
  • 32
  • 2
    The question is about staggered roll outs for the **same device**. There is 100% the same hardware in all this devices, still Google decides to not update them all at once. – Josef May 17 '16 at 09:09
  • 1
    @Josef I have updated my answer to fit the topic. Thank you for informing me that I misread the question, as I did not noticed it – vakus May 17 '16 at 15:31