I install a plugin that allows me to click on any part of a page and it gives me the color of the clicked object
For this to work, the plugin needs to register a click event handler and it needs to interact with the document object model of the current HTML page.
HTML was originally designed to share scientific documents. And while we build complex web applications today, it remains document orientated at its core.
The browsers cannot know what parts of a HTML page contain sensitive information. For example if you login to StackExchange, it will display your username in the top navigation bar. On your bank website there will be information on your money transfer and account balance. There is no structural difference between those informations and a text saying "Your answer".
Chrome does offer some protection against malicious plugins by preventing them from arbitrarily accessing your harddisk. But a plugin that needs to interact with webpages, can do anything to them. All other browsers have the same issues, they just don't go into detail.