Apache Druid

Druid is a column-oriented, open-source, distributed data store written in Java. Druid is designed to quickly ingest massive quantities of event data, and provide low-latency queries on top of the data.[1] The name Druid comes from the shapeshifting Druid class in many role-playing games, to reflect the fact that the architecture of the system can shift to solve different types of data problems.

Druid
Original author(s)
  • Eric Tschetter
  • Fangjin Yang
Developer(s)Apache Druid
Stable release
0.18.1 / 13 May 2020 (2020-05-13)
RepositoryDruid Repository
Written inJava
Operating systemCross-platform
Type
LicenseApache License 2.0
Websitedruid.apache.org

Druid is commonly used in business intelligence/OLAP applications to analyze high volumes of real-time and historical data.[2] Druid is used in production by technology companies such as Alibaba,[2] Airbnb,[2] Cisco,[3] , Deep.BI[2][4], eBay,[5] Lyft,[6] Netflix,[7] PayPal,[2] Pinterest,[8] Twitter,[9] Walmart,[10]Wikimedia Foundation [11] and Yahoo.[12]

History

Druid was started in 2011 to power the analytics product of Metamarkets. The project was open-sourced under the GPL license in October 2012,[13][14] and moved to an Apache License in February 2015.[15][16]

Over time, a number of organizations and companies have integrated Druid into their backend technology,[2] and committers have been added from numerous different organizations.[17]

Architecture

Fully deployed, Druid runs as a cluster of specialized processes (called nodes in Druid) to support a fault-tolerant architecture[18] where data is stored redundantly, and there is no single point of failure.[19] The cluster includes external dependencies for coordination (Apache ZooKeeper), metadata storage (e.g. MySQL, PostgreSQL, or Derby), and a deep storage facility (e.g. HDFS, or Amazon S3) for permanent data backup.

Query management

Client queries first hit broker nodes, which forward them to the appropriate data nodes (either historical or real-time). Since Druid segments may be partitioned, an incoming query can require data from multiple segments and partitions (or shards) stored on different nodes in the cluster. Brokers are able to learn which nodes have the required data, and also merge partial results before returning the aggregated result.

Cluster management

Operations relating to data management in historical nodes are overseen by coordinator nodes. Apache ZooKeeper is used to register all nodes, manage certain aspects of internode communications, and provide for leader elections.

Features
  • Low latency (streaming) data ingestion
  • Arbitrary slice and dice data exploration
  • Sub-second analytic queries
  • Approximate and exact computations
gollark: GHIJ
gollark: Or probably weapon attacks at all.
gollark: Or any time, really.
gollark: There would be no photon torpedoes at this time.
gollark: ```Cold Ones (also ice giants, the Finality, Lords of the Last Waste)Mythological beings who dwell at the end of time, during the final blackness of the universe, the last surviving remnants of the war of all-against-all over the universe’s final stocks of extropy, long after the passing of baryonic matter and the death throes of the most ancient black holes. Savage, autocannibalistic beings, stretching their remaining existence across aeons-long slowthoughts powered by the rare quantum fluctuations of the nothingness, these wretched dead gods know nothing but despair, hunger, and envy for those past entities which dwelled in eras rich in energy differentials, information, and ordered states, and would – if they could – feast on any unwary enough to fall into their clutches.Stories of the Cold Ones are, of course, not to be interpreted literally: they are a philosophical and theological metaphor for the pessimal end-state of the universe, to wit, the final triumph of entropy in both a physical and a spiritual sense. Nonetheless, this metaphor has been adopted by both the Flamic church and the archai themselves to describe the potential future which it is their intention to avert.The Cold Ones have also found a place in popular culture, depicted as supreme villains: perhaps best seen in the Ghosts of the Dark Spiral expansion for Mythic Stars, a virtuality game from Nebula 12 ArGaming, ICC, and the Void Cascading InVid series, produced by Dexlyn Vithinios (Sundogs of Delphys, ICC).```

See also

References
  1. Hemsoth, Nicole. "Druid Summons Strength in Real-Time" Archived 2013-02-27 at the Wayback Machine, Datanami, 08 November 2012
  2. druid. "Druid | Powered by Druid". druid.apache.org. Retrieved 2016-06-29.
  3. Butler, Brandon. "Under the hood of Cisco's Tetration Analytics platform". Retrieved 2016-06-23.
  4. "Real-time Stream Analytics and User Scoring Using Apache Druid, Flink & Cassandra at Deep.BI".
  5. "Druid at Pulsar - ebay的专栏 - 博客频道 - CSDN.NET". blog.csdn.net. Retrieved 2016-06-23.
  6. Streaming SQL and Druid by Arup Malakar, retrieved 2020-01-29
  7. "The Netflix Tech Blog: Announcing Suro: Backbone of Netflix's Data Pipeline". techblog.netflix.com. Retrieved 2016-06-23.
  8. Pinterest: Powering Ad Analytics with Apache Druid, retrieved 2020-01-29
  9. "Interactive Analytics at MoPub: Querying Terabytes of Data in Seconds". blog.twitter.com. Retrieved 2020-01-29.
  10. Nayak, Amaresh (2018-02-23). "Event Stream Analytics at Walmart with Druid". Medium. Retrieved 2020-01-29.
  11. https://conferences.oreilly.com/strata/strata-ny/public/schedule/detail/60986
  12. "Complementing Hadoop at Yahoo: Interactive Analytics with Druid". Retrieved 2016-06-23.
  13. Tschetter, Eric. "Introducing Druid", druid.apache.org, 24 October 2012
  14. Higginbotham, Stacey. "Metamarkets open sources Druid, its in-memory database", GigaOM, 24 October 2012
  15. Harris, Derrick (2015-02-20). "The Druid real-time database moves to an Apache license". Retrieved 2015-08-04.
  16. "Druid Gets Open Source-ier Under the Apache License". Retrieved 2015-08-04.
  17. druid. "Druid | Druid Community". druid.apache.org. Retrieved 2016-06-23.
  18. Druid Project Documentation
  19. Yang, Fangjin; Tschetter, Eric; Léauté, Xavier; Ray, Nelson; Merlino, Gian; Ganguli, Deep. "Druid: A Real-time Analytical Data Store", Metamarkets, retrieved 6 February 2014
This article is issued from Wikipedia. The text is licensed under Creative Commons - Attribution - Sharealike. Additional terms may apply for the media files.