63

Our security experts, database administrators, network team and infrastructure team are all saying it's OK to have the database server located in the DMZ along with the HTTP server and middle-ware server.

Their reason:

If the database server is compromised (because of an insecure middle tier), at least the database server is outside the internal system. If it is inside our network, the hacker can then use the database server to access other systems.

What they are saying is:

  1. Let's not put the middle-ware server behind a second firewall and the database server behind a third firewall.
  2. Let's use just one firewall (the HTTP server's) in case a hacker wants to get our database's sensitive data, at least that's all they can get.

The second statement was actually said... verbatim.

Please note that this database server will hold sensitive information, including bank details.

Now, are these experts making any sense to you? I'm a software developer, and I can't get their logic. It's like, "Put the jewelry box outside the house so that robbers won't bother getting in for the TV?"

Peter Mortensen
  • 877
  • 5
  • 10
bruce bana
  • 633
  • 1
  • 5
  • 7
  • 50
    my conclusion - these people are not experts in security – Rory Alsop Nov 27 '14 at 12:12
  • There is an entire Security Team that reviewed this. It's hard to imagine that not even one could see a problem--unless I'm really just misinterpreting their point. – bruce bana Nov 27 '14 at 13:05
  • In this situation, does DMZ stand for demilitarized zone? Or does it have a different meaning with regards to DBs and such? – Daft Nov 27 '14 at 14:52
  • "DMZ stands for De-Militarized Zone, an industry-standard term referring from the Korean War. A DMZ is a server that is isolated by firewalls from both the Internet and the intranet, thus forming a buffer between the two." from: https://docs.oracle.com/cd/B14099_19/core.1012/b13999/rectop.htm – bruce bana Nov 27 '14 at 15:03
  • 3
    But you don't have to have a single DMZ. You can have several rings of trust. – Sobrique Nov 27 '14 at 18:55
  • Have you considered that you may have to take action into your own hands at some point to draw attention from the final users and may be the general public, in order to make sure that this kind of "security" is never actually used? – njzk2 Nov 27 '14 at 21:07
  • 3
    If the web server got compromised, what good would a firewall do for you? You would have to allow the web server to connect to the database, and it would also have the credentials to do so. Any data the web server is allowed to access in the data base could be accessed by an attacker who compromised the web server. – kasperd Nov 27 '14 at 21:46
  • Sobrique, I don't understand what you mean by rings of trust. I'm not a security expert. please explain? njzk2, It's really tempting but that would make me lose my job. And they might hire someone with similar IQ as their's as replacement.. then that wouldn't have been a responsible action on my part. :-) kasperd, What I was suggesting was have both the middleware and the database servers outside the DMZ and both should have their own firewalls. – bruce bana Nov 27 '14 at 23:06
  • 3
    This question provides no context whatsoever to evaluate the situation properly. What is the new system about, what kind of data is in the DMZ, what are your threat model? There are many factors that needs to be considered to evaluate whether putting DB in the existing secured zone is sensible, or whether you need a new secured zone, or whether putting it outside in the DMZ is sensible. Lacking context, there are only uninformed answers below making generalized statements. – Lie Ryan Nov 28 '14 at 00:28
  • 1
    I have the same question as kasperd above. Once your web server is compromised, why does it matter how many firewalls your database server is behind? Clearly the web server must a) have access to the database server through the firewalls to perform queries (even if it has to do queries through the middleware) and b) those credentials are stored in the web server. Assuming your DB only holds data your webserver can access, no matter how many firewalls you place, once your webserver is compromised, you're out of luck, the attacker can slurp everything out of the database. – mpontes Nov 28 '14 at 13:27
  • 1
    I generally am very confused about why security experts, database administrators, network team and infrastructure team would say/encourage something like this. Is this truly a critical data on this database server? Is there sensitive customer data and is that data reachable outside of localhost? Meaning the web server is basically just making `localhost` calls? – Giacomo1968 Nov 30 '14 at 05:57
  • 1
    @kasperd This is one of several reasons I'm still a huge fan of stored procedures. You can grant the web server privileges to execute a stored procedure without granting it any access to the base tables. But I'm also completely aware that the "norm" these days in many cases is to just grant the web server full access to the database. Which of course I disagree with. ;-) – Craig Tullis Nov 30 '14 at 08:20
  • You tender your resignation and ask for all of your personal data to be scrubbed from their systems... You will be better for it. –  Dec 01 '14 at 02:43
  • Don't explain too much if they can't/won't understand them. Warn them, in written form. After every idiot thing forced on you, give them a warning. After you collect enough warning, you had to consider two options: 1. do you have a better job possibility 2. is the company yet in the position to work with you. Every company (every human collective) has its limit, until that it lets you to elevate. If you reached this point, you can't elevate higher more efficiently. In many cases they simply underestimate you. – peterh Dec 01 '14 at 09:51
  • 1
    Why do you even need to (or would want to) have a database server in an externally-accessible zone? A database server should only ever execute stored procedures from a limited number of well-known hosts (here, the webservers), and it should not be visible or accessible in any way from outside, nor should it be controllable by "trusted" hosts that are externally accessible (and possibly compromised) other than by running rigidly checked stored procs. Plus, I'd replace these experts the moment they say "at least they _only_ get the database", to be honest. – Damon Dec 01 '14 at 10:52

8 Answers8

38

It's like, "Put the jewelry box outside the house so that robbers won't bother getting in for the TV?"

Yes, it is exactly like that.

If you don't care about the value of the database, relatively speaking, then sure it makes sense to leave it outside - if the assumption is that the application is horridly insecure, but you need to put it up anyway for whatever reason, and don't want to secure it, then this is makes sense as a way to isolate this horrible insecure system from all the other internal systems.

On the other hand, there really is no excuse to expect to have SUCH an insecure application that allows complete database server takeover. Also there is no real reason to use "exposure" as an alternative to "isolation" - there are simple solutions to do that right too.

When it comes down to it, it sounds like this is one of 2 possible situations:

  1. These so-called "experts" are really clueless.
  2. There are other business requirements that are being traded off here.

I think your analogy works perfectly. If they then tell you the equivalent of "Actually, those jewelries are fake, oh and btw the TV is really a 60" 5K custom job with gold rims", then that might be a sensible tradeoff (still better to do it right, though).
Otherwise, it's likely that there is no way to explain it, since they are working from a lack of knowledge.

AviD
  • 72,138
  • 22
  • 136
  • 218
  • 1
    Problem is I'm part of the team that is developing the web application. If information is leaked, it's quite easy to point at the application as the culprit when everything is in the DMZ. We're talking about billions of dollars here... not just petty cash. From my point of view, it's #1. That's why it's frustrating and would really want to kindly educate them without being arrogant. Besides, if port and IP addresses are filtered then the only way to have access to the DB is through DB calls... can someone really use SQL scripts to run a terminal and gain access to other internal systems? – bruce bana Nov 27 '14 at 12:10
  • By the way, to give context, the business requires the web application two-factor authentication through password authentication and token key generator... that's how sensitive the data are. So I'm really scratching my head on this one. – bruce bana Nov 27 '14 at 13:02
  • 1
    Hmm... perhaps you should adjust your analogy "Leave the jewelry box outside the house so the robbers don't make a mess in the kitchen when they make a sandwich because they got hungry while doing all the robbing." – AviD Nov 27 '14 at 13:47
  • 3
    Though in that case the better solution would be to leave out a plate of cookies, but I think at that point the analogy starts breaking down.... – AviD Nov 27 '14 at 13:48
  • 2
    Nobody really knows how to be lonely until they are surrounded by idiots. *sigh* – bruce bana Nov 27 '14 at 15:18
  • I wonder whether the strong authentication is a factor in their thinking...to take the analogy one step further, perhaps they consider the strong authentication to be akin to a guarded perimeter with a mantrap access gate (so in theory only people we trust not to take the jewellery box can get in the garden). Doesn't stop someone jumping over the fence when the guards aren't looking though. – R15 Nov 27 '14 at 16:39
  • Yeah. The two-factor authentication makes the web application very secure. However, the hardware configuration compromises this security measure. Got a large steel gate and guards at the front, then just have a picket fence at the back. – bruce bana Nov 27 '14 at 22:27
31

SANS' "Making Your Network Safe for Databases" (http://www.sans.org/reading-room/whitepapers/application/making-network-safe-databases-24) reads a little dated in some sections, but provides a decent "for dummies" level of guidance in the direction you're after.

You could also exhaust yourself poking through the US NIST's resource centre (http://csrc.nist.gov/). I think ISO's ISO/IEC 27033-2:2012 would be on topic too, but don't have a copy at hand to be sure.

You're trying to separate/isolate the most sensitive servers (the database servers) from the most exposed (and therefore vulnerable).

You're proposing a "defense in depth" approach, that seeks to a) prevent attacks where possible, and b) delay their progress (and access to the important stuff) when not.

Ideally, everything is always hardened and patched, servers only listen for traffic on required ports, and only from allowed devices, all traffic "in flight" is inaccessible to unauthorized listeners (through encryption and/or isolation), and everything is monitored for intrusion and integrity.

If all that is in place with 100% certainty, then great, your "opposition" have addressed point a) above, as much as is possible. Great start, but what about point b)?

If a web server does get compromised, your proposed architecture is in a much better spot. Their potential attack footprint, and vector, is much larger than it needs to be.

The justification for separate database from web servers is no different than the justification they've accepted for separating web servers from LAN. More bluntly: if they're so convinced a compromised web server presents no danger to other systems in the same security zone, why do they think a DMZ is required at all?

It's awfully frustrating to be in your situation. At the very least, in your position I'd create a risk memo outlining your concerns and suggestions, and ensure they acknowledge it. CYA, anyway.

DerekM
  • 326
  • 2
  • 2
  • Thanks everyone for your answers. I wish I could accept all answers but this one was what I directly used to explain the pros and cons of using a 3 tier network security architecture. I was able to convince one of the young member of the security team by giving a summary of all your answers with this as the main point. He eventually explained to the other teams why it should be done. – bruce bana Nov 28 '14 at 10:52
  • 1
    For more information on this topic in the future (in case you need more ammo) the term is "N Tiered Architecture". (for google searches) – k1DBLITZ Nov 28 '14 at 23:44
  • I posted a comment on the question, but the reality is: What kind of database server is this? What is the application? What is the true risk of a complete data compromise? – Giacomo1968 Nov 30 '14 at 05:58
17

If the database holds card details, it can be very easily argued that you aren't fulfilling the PCI DSS requirement on appropriate protection.

It also fails the sanity checks on single points of failure, and protecting your core assets.

If the data is worth billions, why would you not spend a few thousand more to add layers of protection? Industry good practice is to have layered security zones.

You could point them at PCI, ISF's Good Practice Guide and many others.

Rory Alsop
  • 61,367
  • 12
  • 115
  • 320
  • 1
    No credit card details so I can't use PCI compliance as a motivating factor. Thanks for pointing out ISF. I'm not aware of this. I only rely on the good practices of my peers from previous projects I had and this current configuration is just making my alarm bells go crazy. – bruce bana Nov 27 '14 at 12:31
  • Do you know of any international regulations that requires non-exposure of bank accounts and payment details? – bruce bana Nov 27 '14 at 13:45
  • 1
    If you aren't holding Credit Card details, what are you holding? Account information? Patient visit information? blueprints for American Made UFOs? If it's that sensitive, maybe you should aim for that level of security or similar levels demanded by say Electronic Health Record systems used for Patient info. No CC info there, but you be screwed sideways if you lose THAT database.... – WernerCD Nov 27 '14 at 16:32
  • I don't think storing CC details in the db is a problem as long as they are encrypted and keys are away from the DB. – AKS Nov 27 '14 at 17:10
  • 3
    @AyeshK - PCI compliance requires payment information to be stored on a secured network. A DMZ would not pass a PCI scan even a little. – Logarr Nov 27 '14 at 20:01
  • @Logarr sorry I didn't meant a DMZ. In general, in don't think it's straight away insecure to store card data in a database, provided the DB itself is secured and data is encrypted. – AKS Nov 27 '14 at 20:14
11

If you are suggesting that the database server gets moved from being in the same security zone as the web server to being in the same security zone as some internal systems, then one could reasonably conclude that you are reducing security.

If status quo is that web server and database server are both in the DMZ, and no connections are permitted from DMZ to internal network, then moving the database server from DMZ to internal network would require allowing connections from DMZ to internal network. That would mean you are reducing the security of the internal network without getting significantly better protection for the database.

If you truly believe that the database is more sensitive than the internal network (which may very well be true), then you wouldn't want any compromise of the internal network to get direct access to the database.

So the point you need to make is, that you don't want to move database server from one security zone to another, instead what you want to do is to create an entirely new security zone. To really follow through on that you would actually have to buy an additional firewall and put that between the DMZ and the extra secured network.

Then it boils down to evaluating the cost of buying and maintaining this additional firewall against the additional security it provides.

kasperd
  • 5,402
  • 1
  • 19
  • 38
  • Thanks. That makes sense. Do you know web resources with figures for this setup? I don't think they would be concerned about the cost. However, the additional work they would do to maintain it would prevent them from moving. That's why their brain can only think of "internal network" and "DMZ". They would scatter if they think they have to work. – bruce bana Nov 27 '14 at 22:53
  • @brucebana I would just draw the figures on a whiteboard or on a piece of paper. After all the figures would have to take your current network as starting point. – kasperd Nov 28 '14 at 07:29
  • @kasperd I'm actually trying to soak up as much info as I can from this brilliant question that was made. And your answer seems to be easy enough to learn (for a 'starter'). Would you mind to draw it on paper or using DIA (http://dia-installer.de/)? That would be awesome of you if you made that drawing. – Ismael Miguel Nov 28 '14 at 22:11
5

AvID has already covered the main question, but coming at this from a slightly different angle most firewalls will support multiple interfaces and can provide control of traffic between the interfaces.

Configuring the multiple interfaces to host each of the aspects of the solution (frontend, middleware, backend) would reduce the risk of onward compromise of the database from the web server without necessitating multiple physical firewalls (if that is what the other teams are worried about...is it a cost issue?) and without introducing a route to the internal network.

Just wondering whether this would give you a practical 'lever' to support the argument for implementing some separation i.e. it brings something different to the discussion that they have to counter?

Failing that what about a what-if scenario based on an historic web server vulnerability, which would have resulted in onward compromise of any other server on the same network segment (including the database) and how a layered DMZ would prevent the onward compromise, presumably there would be a monetary cost so it would be 'easy' to illustrate the cost/effort benefit?

R15
  • 2,923
  • 1
  • 11
  • 21
  • Good that you also gave practical solutions. My feeling was that if they wanted practical solutions it would be trivial to find one, even for incompetents - I hope that's not the case :-) – AviD Nov 27 '14 at 12:05
  • @AviD Very true...just trying to give the OP as much ammunition as possible to 'beat' the incompetents into submission with! – R15 Nov 27 '14 at 12:09
  • I agree. Cost isn't really an issue. The hardware cost so small compared to the monetary transactions that is saved in the DBs. I sense that it's more of being lazy--don't want to be bothered with configuring and interacting with other teams to setup the secure infrastructure. I wish FUD would work on them but dirty tactics doesn't really suit my personality. I really prefer to have a concrete and very simple way of explaining the pros and cons of this configuration. – bruce bana Nov 27 '14 at 12:20
  • Do you know of a resource on network topology with the DB outside the DMZ and also outside the internal network? Giving them the link might work... especially if it has the words "best practice" or "industry standards". – bruce bana Nov 27 '14 at 15:17
  • 1
    @brucebana: Create an additional zone (separate LAN or VLAN) with a new IP subnet directly connected to the firewall over a new interface (either physical or statically configured VLAN). There put the database server. --- This is a common practice. Some companies have a very large number of separated zones (almost one zone per server). ------ Of course this brings higher performance requirements on the firewall as the traffic between the application and the database is not being checked in the current design. – pabouk - Ukraine stay strong Nov 27 '14 at 17:40
  • Yes. I've seen it in a lot of previous projects I had. However, the suggestion shouldn't come from us web application developers. They think we're idiots and they're experts (because of their tenure in the organization). The solution should come from an information security resource/website with lots of warnings about the dangers if they pursue what they were planning. Something so compelling that it would be hard to refute that what they are suggesting IS THE SECURITY COMPROMISE. These people only respond to accountability. They only want to cover their asses when things blow up. – bruce bana Nov 27 '14 at 22:37
  • @brucebana: Some examples of online materials after quick search: http://www.servicecatalog.dts.ca.gov/services/professional/security/docs/3117_Network_Architecture_Standard.pdf https://www.owasp.org/images/a/a5/OWASP_miami_Architect%E2%80%99s_View_of_Application_Security-2009_02.ppt http://zeltser.com/multi-firewall/ http://www.ibm.com/developerworks/cloud/library/cl-tsamextensions/ http://blogs.citrix.com/2013/09/25/cloudplatform-enables-every-workload/ – pabouk - Ukraine stay strong Nov 29 '14 at 00:02
  • @brucebana "FUD" isn't really a dirty tactic in a case like this, if you ask me. The simple fact is that active attacks against Internet-connected infrastructure are taking place constantly. *Constantly*. Not implementing the kind of security you are talking here is probably borderline negligent. Actually, speaking of that angle, would the current configuration without multiple network zones pass a real independent security audit? – Craig Tullis Dec 06 '14 at 22:36
4

Assuming you are attempting to persuade them to do it rather than (necessarily) convince them it is correct:

Explain that when their large customers and prospects come to do a security audit they will fail.

If the obstacle is the business then that will be the only sufficient, and only necessary, reason.

Ben
  • 3,697
  • 1
  • 18
  • 24
  • 1
    Yes, I am trying to persuade them. I don't care if they take the credit or if they think they came up with the idea--as long as the final topology is secure. Business isn't really the obstacle. Clients don't really have a say on how the organization does its business. The answers I'm getting here also verified my suspicion. I think they would respond to the words "best practice" or "industry standards" and other business-sounding catch phrase. – bruce bana Nov 27 '14 at 15:21
2

One good argument is that the bar really isn't that high to separate the web servers and database servers into separate DMZ's.

Use a real router/firewall, and put the web servers and database servers on separate VLANs, both of them outside the internal secure LAN, with firewall rules controlling access to the bare minimum required ports from the Internet to the web servers, from the web servers to the database servers, and no access at all originating from the web servers to the secure LAN.

The firewall would also prevent any direct access at all from the internet to the database servers, and tightly control any access from the database servers inward to the secure LAN (for authentication purposes or whatever).

That way, the attacker can't even get to the network containing the database servers directly.

If they get a beachhead on one of your web servers, they're still not on the same network with unrestricted access to attack your database server(s), and if you have any kind of log monitoring in place you should receive notifications about the breach of the web servers before the attackers have had much time to attack anything else.

Even if they then manage to breach your database servers after some period of time through the one open port that the web server gets to use to communicate with the database, they've wasted all that time accomplishing relatively little, during which time you have been aware of their attack, instead of spending all of that time getting into your secure LAN.

They can't even reach the LAN from the DMZ where the web servers live, so their only route into the LAN in any form at all is to hop onto the database servers, protected in the other DMZ. Chances are that your database servers are or will be tied into some kind of enterprisey authentication system (Active Directory or whatever). Do you want that capability in the same DMZ with your public web servers?

If I can be concerned enough about security issues to create guest and DMZ subnets at my home, to have a place to put "things" ("Internet of Things") without having them directly on my LAN, a concern worth billions can surely afford the mindshare and time to do the same thing with important web servers and database servers. I'm doing this at the office with a combination of a stack of several-thousand-dollar Procurve managed L2/L3 switches, a SonicWall UTM and a Ubiquiti EdgeMax router. At home, I have a $100 Ubiquiti EdgeRouter Lite, a $100 managed 8-port HP switch and one Ubiquiti Unifi AP that supports multiple VLANs, and my setup is totally capable of doing what we're talking about.

And I get the peace of mind of not worrying that my network-connected DVR, printer, BluRay player, thermostat and whatever else, running who-knows-what buggy out-of-date firmware with who-knows-what undiscovered exploits, might get hacked and reach out to my computer and personal files over the network.

It isn't super hard for security experts to configure this sort of thing, and it certainly doesn't have to involve separate physical hardware for each firewall. SDN (software defined networking) is all the rage these days, right?

Even the $100 EdgeRouter Lite can forward nearly 1 Gbps through the router, with support for multiple virtual interfaces and firewall rules between all of those interfaces.

One tiny box is really a whole basket full of firewalls.

So if you spend real money on a higher-end router, you'll get all those features and a few more with beefier routing performance.

Even something like the Ubiquiti EdgeRouter Pro 8 gives you 2 million packets per second for only $375, with 8 physical interfaces and VLAN subinterfaces on each of those if you need them. If you need higher performance than that, look to Brocade (Vyatta), Cisco, Juniper, etc. for bigger hardware. Or something like Dell/Sonicwall's SuperMassive series. Or run the Vyatta virtual router on a beefy multi-core Xeon server.

I'm not trying to peddle routers, just making the point that the bar isn't actually all that high to get the kind of security separation you should probably have and obviously want here.

Craig Tullis
  • 1,483
  • 10
  • 13
0

Tell your direct manager that you are not authorized to accept that risk on behalf of the company. Point out that the value of the data in the database is $xxx and that that level of risk acceptance must come from someone authorized to do so, and ask him to champion getting a member of executive management to sign-off on the risk acceptance.

To estimate the value of the data, determine whether it is data tied to PCI DSS (if so, look at the cost-per-customer that Target / Home Depot have incurred in lost sales, credit monitoring, fraud), or if it is PII (if so, look at the financial damage per-patient that the medical systems have incurred in lost confidence, identity theft monitoring, lost reputation), or if it is your company's competitive edge (trade secrets, not-yet-released financials, intellectual property) then the value of the data could be the current valuation of the firm.

If you are given a direct order to accept the risk on behalf of the company, you may be asked to begin acting in the capacity of a company officer, and as such, you have an obligation (possibly a legal obligation) to point out that you are a non-appointed person asked to perform as one, and should brief this up the management chain to the company's board of directors. Be prepared for significant blow-back and to expend significant political capital to make this known.

Good luck!

DTK
  • 1,190
  • 7
  • 8