Not inherently dangerous, but there is some risk. This is the way L2 switches work (if you don't want the odd misdirected packet use a point to point link ;-) Unless you have a locked-down layer 2 configuration this is unavoidable, essentially a trade off of security versus ease of configuration & management. Mitigating options include ports with pre-configured MAC addresses and 802.1x port-based NAC.
A point that seems to be misunderstood: some assume that the switch MAC table being empty corresponds with a quiescent state of the network and clients, e.g. the only packets that are flooded are "empty" SYN packets (or more likely ARP or DHCP).
This is not necessarily true, you may have an exposure if an attacker can flush the MAC table directly or indirectly (through SNMP, forced restart by administrative means or through a software bug, resource exhaustion, or good old fashioned power cycling). Such methods aren't great ways to capture packets since most involve (short) interruptions to connectivity, but it may be exploitable. (A layer 2 switch does not behave like a L3/L4 firewall, it only cares about forwarding frames, once it has recovered client connections will continue unaffected.)
A related issue is flooding in the opposite case: when the MAC table is full and the switch has no choice but to drop or flood. This is arguably more useful to an attacker, if he can sustain the CAM exhaustion he has a longer window to capture useful traffic.