9

My company (an inward-looking institution without customer or external interface) has moved all operations from a local data centre to AWS a few years back, mostly to a handful of large AWS accounts (network, management, billing, sandbox, dev, test, prod). Developers have been having a hard time innovating in this initial setup, because IAM policies and service whitelisting had to be quite restrictive due to the shared nature of those accounts.

So, the latest thing we're busy with now, is introducing more accounts, e.g. one triplet of dev/test/prod accounts per product or development team. With a few key accounts creating guardrails around those product accounts (network VPCs providing, security scanning, log/reporting, etc...). Separate account triplets for each dev team will mean the splash radius of what they do will more easily be contained via inter-account firewalls, security groups, etc... Once developers get their accounts, they also get an admin account to more freely explore, and if they do something suspicious (e.g. a machine opening up all ports to 0.0.0.0/0), we've got Turbot.com nuking the rogue resources/account in no time.

Seems easy enough, but for some reason our 3rd party account/billing provider who I will not mention in here, has been having challenges providing us new accounts on the fly (we're talking dozens of account triplets as we start up new projects/apps internally).

I am puzzled by a few things:

  • why would our internal cloud team not buy accounts straight from AWS ?
  • what can possibly be the issue in them not being to provision new accounts swiftly ?
  • what's complex about that service offering ?
  • How are they possibly architected ?
  • How do they provide security, IAM, and care for our root credentials ? I understand we're restricted from creating roles and policies with certain names that they reserve for themselves
  • What does it take to automate 3rd party AWS account provisioning the way they do ? I understand they're kind of owning each of our accounts, and billing gets aggregated to a specific account they've large control over.

I'm trying to understand the complexity of what they do and whether there are alternative routes. As a developer, I've been waiting weeks for my accounts internally and I'm searching for answers. To me it looks like red tape that was decided upon arbitrarily in the past.

BuZz
  • 193
  • 4

1 Answers1

14
  • why would our internal cloud team not buy accounts straight from AWS?

They probably get discounts. These large "billing consolidation providers" invoice their customers millions of dollars per month and for that they get decent discounts from AWS, Azure, etc, and pass on some of that to the customers. Typically you get around 5% discount.

The big downside is that they technically own your accounts and depending on the billing provider and the configuration you may be quite restricted in what you can and can't do. Typically you can't create sub-accounts as you have seen yourself, often you can't view billing data directly (e.g. AWS Cost Explorer) and will be forced to use the billing provider's portal for that, you may not be able to buy reserved instances or spending plans or even subscribe to support plans directly, etc.

I personally hate these billing arrangements - the little discount is not worth the limitations and the loss in productivity it brings. But the management often only sees the 5% discount on the bill and not the frustration and the extra hours their engineers spend fighting and working around the restrictions. It probably costs them way more than those 5% in the lost productivity, but hey, that's not as visible as a discount item on an invoice.

That'd be my explanation ;)

MLu
  • 23,798
  • 5
  • 54
  • 81
  • Thanks for this, I was quite oblivious to that fact. Our management certainly fits that bill! Thinking of selling them to do all this multi-account governance internally possibly... Do you have insight as to what those billing consolidation providers' AWS architecture might look like ? The key ideas ? – BuZz Oct 18 '20 at 22:58
  • 1
    @BuZz if you want to do your account management internally look at **AWS ControlTower** – MLu Oct 19 '20 at 10:48
  • that sounds juicy! Will be taking a look. – BuZz Oct 19 '20 at 10:58
  • 4
    @BuZz I can tell you that it's almost guaranteed that their internal infrastructure is the most rickety piece of junk you'd ever not want to see... that's one of the ways they're able to compete with other such providers and give discounts to customers, by not hiring decent devs to architect and maintain what was probably originally built as a proof-of-concept, and hiring the lowest of the low support staff (hence the delays and general pain in getting them to do anything quickly). – Ian Kemp Oct 19 '20 at 14:22
  • 2
    @BuZz If you don't think your management would be swayed by arguments about extra internal dev/admin costs due to dealing with the intermediary, the risks from the 3rd party company technically owning all your AWS infrastructure might have better traction. In any sort of dispute they could hold all your companies data/systems hostage to force you to accede to their demands. – Dan Is Fiddling By Firelight Oct 19 '20 at 15:28
  • Thanks both for your added wisdom! Noted and noted. – BuZz Oct 19 '20 at 20:16