Clever Geek Handbook
📜 ⬆️ ⬇️

Clickhouse

ClickHouse is an open-source column database analytic database that allows real-time analytical queries to be performed on structured big data , developed by Yandex [3] [4] [5] .

Clickhouse
Clickhouse.png
Type ofrelational DBMS
DeveloperYandex
Written onC ++
operating systemLinux
First editionJune 15, 2016 ( 2016-06-15 )
Latest versionv19.3.3-stable [1] (February 13, 2019 ( 2019-02-13 ) )
LicenseApache (2.0) [2]
Websiteclickhouse.yandex

ClickHouse uses its own SQL dialect close to standard, but containing various extensions: arrays and nested data structures, higher-order functions, probabilistic structures, functions for working with URIs , the ability to work with external key-value repositories ("dictionaries"), specialized aggregate functions, functionality for sampling , approximate calculations, the ability to create stored views with aggregation, populating a table from the Apache Kafka message flow, etc.

However, there are also limitations: lack of transactions , lack of point-based UPDATE / DELETE (batch UPDATE / DELETE was introduced in June 2018), limited support for JOIN syntax, strict types with the need for explicit casting, for some operations, intermediate data should be placed in RAM , the lack of window functions, the lack of a full-fledged query optimizer, point reading, the presence of restrictions on the implementation of some functions related to the specifics of using ClickHouse in Yandex, etc.

The system is optimized for storing data on hard drives (taking advantage of linear reading, data compression). To ensure fault tolerance and scalability, ClickHouse can be deployed on a cluster ( Apache ZooKeeper is used to coordinate the replication process) [6] . To work with the database, there is a console client, web client, HTTP interface, ODBC and JDBC drivers [7] , as well as ready-made libraries for integration with many popular programming languages ​​and libraries [8] .

In many tests, ClickHouse shows very high performance, beating competitors such as Greenplum , Vertica [9] , Amazon Redshift [10] , Druid [11] , InfiniDB / MariaDB ColumnStore [12] , Apache Spark [13] [14 ] by this indicator. ] , Presto , Elasticsearch [15] .

Content

  • 1 History
  • 2 Distribution
  • 3 notes
  • 4 References

History

ClickHouse was developed to solve web analytics tasks for Yandex.Metrica , the third most popular web analytics system in the world [16] .

Initially, Yandex.Metrica used pre-aggregated data to build reports [17] .

This approach allowed us to reduce the size of the stored data, but had a number of limitations and disadvantages:

  • the need to fix a list of reports available to the user in advance (the inability to build an arbitrary report);
  • pre-aggregation by a large number of keys or by keys of high cardinality (such as URLs ) can lead to the opposite effect (increase in data volume);
  • maintaining logical integrity during storage of a large number of different aggregations is difficult.

An alternative approach is to store “raw” non-aggregated data, performing all the necessary calculations at the time of the user's request. For this, a DBMS was needed that would be able to process Yandex.Metrica non-aggregated data (petabytes of data) with very high efficiency and in real time, while at the same time having an acceptable cost. Since at that time there were no such solutions on the market, Yandex began to develop its own DBMS .

The first ClickHouse prototype appeared in 2009 [18] . By the end of 2014, Metrica 2.0, powered by ClickHouse, was launched, which allowed users to build custom reports.

In June 2016, the source code of the system was uploaded to open-source under the Apache 2.0 license [19] .

Distribution

In 2016, in addition to Yandex.Metrica , ClickHouse was used in a number of different projects inside Yandex , for example, in the open-source Yandex.Tank project for storing telemetry data [19] , Yandex.Market for monitoring the health of the service [20] , and in external projects, for example, to analyze metadata about events in the LHCb experiment in CERN [21] (about a billion events and 1000 parameters for each event).

Currently, many companies successfully use ClickHouse, including: Cloudflare , Bloomberg [22] , Spotify [23] , VKontakte [24] , Rambler [25] , Tinkoff Bank [26] , NIC Labs Chile [27] , Amadeus [ 28] , Avito.ru [29] , Criteo , ContentSquare [30] , Media2 [31] , ivi.ru [32] , Mail.ru ,Adtelligent , Carto, Lifestreet, Infinidat [33] [34] , SemRush [35] , etc.

Notes

  1. ↑ ClickHouse releases . yandex / ClickHouse . Github Date of treatment February 14, 2019.
  2. ↑ ClickHouse License . yandex / ClickHouse . Github Date of treatment August 17, 2018.
  3. ↑ Yandex ClickHouse. Nowhere faster. / Filippov Oleg // System Administrator. - 2017. - No. 1-2. - S. 56-58.
  4. ↑ ClickHouse in statistics collection systems / Alexander Kalendarev // System Administrator. - 2017. - No. 3. - S. 56-59.
  5. ↑ ClickHouse - National Library named after N.E. Bauman (Russian) . ru.bmstu.wiki. Date accessed August 20, 2018.
  6. ↑ Afanasyev G.I., Belonogov I.B., Bulatova I.G., Tonoyan S.A. Organization of clusters for data processing based on YANDEX CLICKHOUSE DBMS and distributed coordination service for distributed applications APACHE ZOOKEEPER // Science Alley. - 2018.- T. 3 , No. 1 . - S. 850-860 . - ISSN 2587-6244 .
  7. ↑ Access to ClickHouse using JDBC (Russian) . Date accessed August 19, 2018.
  8. ↑ Interfaces . ClickHouse Documentation . clickhouse.yandex. Date of treatment August 17, 2018.
  9. ↑ Performance comparison of analytical DBMS . clickhouse.yandex. Date of treatment August 17, 2018.
  10. ↑ ClickHouse vs Amazon RedShift Benchmark . www.altinity.com. Date of treatment August 17, 2018.
  11. ↑ SREcon18 Americas - Monitoring DNS with Open-Source Solutions on YouTube starting at 8:50
  12. ↑ InfiniDB vs ClickHouse (China) . www.verynull.com (August 22, 2016). Date of treatment August 17, 2018.
  13. ↑ Column Store Database Benchmarks: MariaDB ColumnStore vs. ClickHouse vs. Apache Spark www.percona.com (March 15, 2017). Date of treatment August 17, 2018.
  14. ↑ A Look at ClickHouse: A New Open Source Columnar Database - DZone Database , dzone.com . Date accessed August 20, 2018.
  15. ↑ Mark Litwintschik. Summary of the 1.1 Billion Taxi Rides Benchmarks . tech.marksblogg.com. Date of treatment August 17, 2018.
  16. ↑ Usage Statistics and Market Share of Traffic Analysis Tools for Websites, April 2019 (neopr.) . w3techs.com. Date of appeal April 18, 2019.
  17. ↑ Evolution of data structures in Yandex.Metrica (rus.) , Yandex Blog , habr.com (December 17, 2015). Date of treatment August 17, 2018.
  18. ↑ ClickHouse: High-Performance Distributed DBMS for Analytics | Percona Live Amsterdam - Open Source Database Conference 2016 (Neopr.) . www.percona.com. Date of treatment October 20, 2016.
  19. ↑ 1 2 Yandex opens ClickHouse . Date of treatment October 20, 2016.
  20. ↑ Market Health: how we turn logs into graphics, Dmitry Andreev (Yandex) - Yandex Events (unopened) . events.yandex.ru. Date of treatment October 20, 2016.
  21. ↑ Yandex - Yandex Launches Search Tool for LHC Events at CERN , Yandex . Date of treatment October 20, 2016.
  22. ↑ Alex Bocharov . HTTP Analytics for 6M requests per second using ClickHouse , The Cloudflare Blog (March 6, 2018). Date of treatment August 17, 2018.
  23. ↑ Gleb Kanterov. ClickHouse for Experimentation . ClickHouse Community Meetup in Berlin (Neopr.) . ClickHouse Blog . clickhouse.yandex (July 3, 2018) . Date accessed August 20, 2018.
  24. ↑ A bit of backstage VK (Russian) , habr.com (June 22, 2018). Date accessed August 20, 2018.
  25. ↑ Demyan Kudryavtsev . Development of ClickHouse API for Rambler / top-100 (rus.) , Rambler Group blog, habr.com (May 17, 2018). Date accessed August 19, 2018.
  26. ↑ M. Belousov, D. Nemchin, G. Bezrukikh, D. Pavlov . Comparison of analytical in-memory databases (Russian) , IT's Tinkoff.ru Blog , Habr (November 11, 2016). Date of treatment August 17, 2018.
  27. ↑ Felipe Espinoza and Javier Bustos. Monitoring DNS with Open-Source Solutions | USENIX SREcon18 Americas . www.usenix.org (March 29, 2018). Date of treatment August 17, 2018.
  28. ↑ Amadeus Technologies Launches Investment and Insights Tool Based on Machine Learning and Strategy Algorithms , Kodiak Data (March 27, 2018).
  29. ↑ Vladimir Kolobaev . Storage of metrics: how we switched from Graphite + Whisper to Graphite + ClickHouse (Russian) , Avito 's blog, habr.com. Date accessed August 19, 2018.
  30. ↑ ClickHouse Meetup in Paris . Altinity Date of treatment October 8, 2018.
  31. ↑ Igor Strykhar . How to launch ClickHouse on your own and win the jackpot (Russian) , Media2 Blog , habr.com (November 7, 2016). Date accessed August 18, 2018.
  32. ↑ Konyaev Andrey . As we in ivi rewrote etl: Flink + Kafka + ClickHouse (Russian) , ivi Online Cinema Blog , habr.com (January 24, 2018). Date accessed August 19, 2018.
  33. ↑ Alexander Zaytsev . Who and Why is Using ClickHouse (Eng.) , Altinity (10 August 2017). Date of treatment August 17, 2018.
  34. ↑ ClickHouse Meetup in Berlin . yandex.imtqy.com. Date of treatment August 17, 2018.
  35. ↑ “Sometimes you have to look into the Spark code”: Alexander Morozov (SEMrush) about using Scala, Spark and ClickHouse (Russian) , Blog of the JUG.ru Group company , habr.ru (October 30, 2017). Date accessed August 19, 2018.

Links

  • Official ClickHouse Page
  • ClickHouse repository
Source - https://ru.wikipedia.org/w/index.php?title=ClickHouse&oldid=99291778


More articles:

  • Barman, Tom
  • Jaggy, Michelle
  • Inokovka (Station)
  • Brodsky Synagogue (Odessa)
  • Andreeva, Elena Leonidovna
  • Yurginsky
  • Rybinka (Volgograd Oblast)
  • Lekerica, Jose Felix de
  • Stein, Boris Samuilovich
  • Harmonia (game)

All articles

Clever Geek | 2019