Clickhouse单机版快速入门

作者: 捞月亮的阿汤哥 | 来源:发表于2021-04-21 15:44 被阅读0次

Clickhouse单机版快速入门
clickhouse
搭建本地debs源安装clickhouse
【clickhouse专栏】单机版的安装与验证
篇一|ClickHouse快速入门
VersionedCollapsingMergeTree 与 C
clickhouse入门
Clickhouse入门
ClickHouse 入门
Clickhouse 入门

本文以操作系统为macos为例
参考链接 https://clickhouse.tech/docs/zh/getting-started/tutorial/

macos下载二进制包

curl -O 'https://builds.clickhouse.tech/master/macos/clickhouse' && chmod a+x ./clickhouse

创建config.xml和users.xml
文件的内容如下

https://github.com/ClickHouse/ClickHouse/blob/master/programs/server/config.xml

https://github.com/ClickHouse/ClickHouse/blob/master/programs/server/users.xml

启动服务端和客户端

修改文件句柄数

如果文件句柄数设置的比较小，client或者server端在运行中可能会报too many open files xxx的错。在运行server和client的终端都运行一下

ulimit -n 10240

ulimit -a # 查看文件句柄数是否修改成功

启动服务端

在一个终端运行

sudo ./clickhouse server --config-file=config.xml

启动客户端

在另一个终端运行

./clickhouse client

命令行clickhouse-client的快速提示

# 交互模式
clickhouse-client
clickhouse-client --host=... --port=... --user=... --password=...

# 多行查询
clickhouse-client -m
clickhouse-client --multiline

# 批处理
clickhouse-client --query='SELECT 1'
echo 'SELECT 1' | clickhouse-client
clickhouse-client <<< 'SELECT 1'

# 从指定文件中插入数据
clickhouse-client --query='INSERT INTO table VALUES' < data.txt
clickhouse-client --query='INSERT INTO table FORMAT TabSeparated' < data.tsv

使用DataGrip连接clickhouse

下载并且提取数据

下载数据集

下载完后解压，解压完后两个文件大概是10G的大小

https://datasets.clickhouse.tech/visits/tsv/visits_v1.tsv.xz
https://datasets.clickhouse.tech/hits/tsv/hits_v1.tsv.xz

创建数据库

CREATE DATABASE IF NOT EXISTS tutorial;

创建表引擎

与创建数据库相比，创建表的语法要复杂得多（请参阅参考资料. 一般CREATE TABLE声明必须指定三个关键的事情:

要创建的表的名称。
表结构，例如：列名和对应的数据类型。
表引擎及其设置，这决定了对此表的查询操作是如何在物理层面执行的所有细节。

创建hits_v1表

CREATE TABLE tutorial.hits_v1
(
    `WatchID` UInt64,
    `JavaEnable` UInt8,
    `Title` String,
    `GoodEvent` Int16,
    `EventTime` DateTime,
    `EventDate` Date,
    `CounterID` UInt32,
    `ClientIP` UInt32,
    `ClientIP6` FixedString(16),
    `RegionID` UInt32,
    `UserID` UInt64,
    `CounterClass` Int8,
    `OS` UInt8,
    `UserAgent` UInt8,
    `URL` String,
    `Referer` String,
    `URLDomain` String,
    `RefererDomain` String,
    `Refresh` UInt8,
    `IsRobot` UInt8,
    `RefererCategories` Array(UInt16),
    `URLCategories` Array(UInt16),
    `URLRegions` Array(UInt32),
    `RefererRegions` Array(UInt32),
    `ResolutionWidth` UInt16,
    `ResolutionHeight` UInt16,
    `ResolutionDepth` UInt8,
    `FlashMajor` UInt8,
    `FlashMinor` UInt8,
    `FlashMinor2` String,
    `NetMajor` UInt8,
    `NetMinor` UInt8,
    `UserAgentMajor` UInt16,
    `UserAgentMinor` FixedString(2),
    `CookieEnable` UInt8,
    `JavascriptEnable` UInt8,
    `IsMobile` UInt8,
    `MobilePhone` UInt8,
    `MobilePhoneModel` String,
    `Params` String,
    `IPNetworkID` UInt32,
    `TraficSourceID` Int8,
    `SearchEngineID` UInt16,
    `SearchPhrase` String,
    `AdvEngineID` UInt8,
    `IsArtifical` UInt8,
    `WindowClientWidth` UInt16,
    `WindowClientHeight` UInt16,
    `ClientTimeZone` Int16,
    `ClientEventTime` DateTime,
    `SilverlightVersion1` UInt8,
    `SilverlightVersion2` UInt8,
    `SilverlightVersion3` UInt32,
    `SilverlightVersion4` UInt16,
    `PageCharset` String,
    `CodeVersion` UInt32,
    `IsLink` UInt8,
    `IsDownload` UInt8,
    `IsNotBounce` UInt8,
    `FUniqID` UInt64,
    `HID` UInt32,
    `IsOldCounter` UInt8,
    `IsEvent` UInt8,
    `IsParameter` UInt8,
    `DontCountHits` UInt8,
    `WithHash` UInt8,
    `HitColor` FixedString(1),
    `UTCEventTime` DateTime,
    `Age` UInt8,
    `Sex` UInt8,
    `Income` UInt8,
    `Interests` UInt16,
    `Robotness` UInt8,
    `GeneralInterests` Array(UInt16),
    `RemoteIP` UInt32,
    `RemoteIP6` FixedString(16),
    `WindowName` Int32,
    `OpenerName` Int32,
    `HistoryLength` Int16,
    `BrowserLanguage` FixedString(2),
    `BrowserCountry` FixedString(2),
    `SocialNetwork` String,
    `SocialAction` String,
    `HTTPError` UInt16,
    `SendTiming` Int32,
    `DNSTiming` Int32,
    `ConnectTiming` Int32,
    `ResponseStartTiming` Int32,
    `ResponseEndTiming` Int32,
    `FetchTiming` Int32,
    `RedirectTiming` Int32,
    `DOMInteractiveTiming` Int32,
    `DOMContentLoadedTiming` Int32,
    `DOMCompleteTiming` Int32,
    `LoadEventStartTiming` Int32,
    `LoadEventEndTiming` Int32,
    `NSToDOMContentLoadedTiming` Int32,
    `FirstPaintTiming` Int32,
    `RedirectCount` Int8,
    `SocialSourceNetworkID` UInt8,
    `SocialSourcePage` String,
    `ParamPrice` Int64,
    `ParamOrderID` String,
    `ParamCurrency` FixedString(3),
    `ParamCurrencyID` UInt16,
    `GoalsReached` Array(UInt32),
    `OpenstatServiceName` String,
    `OpenstatCampaignID` String,
    `OpenstatAdID` String,
    `OpenstatSourceID` String,
    `UTMSource` String,
    `UTMMedium` String,
    `UTMCampaign` String,
    `UTMContent` String,
    `UTMTerm` String,
    `FromTag` String,
    `HasGCLID` UInt8,
    `RefererHash` UInt64,
    `URLHash` UInt64,
    `CLID` UInt32,
    `YCLID` UInt64,
    `ShareService` String,
    `ShareURL` String,
    `ShareTitle` String,
    `ParsedParams` Nested(
        Key1 String,
        Key2 String,
        Key3 String,
        Key4 String,
        Key5 String,
        ValueDouble Float64),
    `IslandID` FixedString(16),
    `RequestNum` UInt32,
    `RequestTry` UInt8
)
ENGINE = MergeTree()
PARTITION BY toYYYYMM(EventDate)
ORDER BY (CounterID, EventDate, intHash32(UserID))
SAMPLE BY intHash32(UserID)

创建visits_v1表

CREATE TABLE tutorial.visits_v1
(
    `CounterID` UInt32,
    `StartDate` Date,
    `Sign` Int8,
    `IsNew` UInt8,
    `VisitID` UInt64,
    `UserID` UInt64,
    `StartTime` DateTime,
    `Duration` UInt32,
    `UTCStartTime` DateTime,
    `PageViews` Int32,
    `Hits` Int32,
    `IsBounce` UInt8,
    `Referer` String,
    `StartURL` String,
    `RefererDomain` String,
    `StartURLDomain` String,
    `EndURL` String,
    `LinkURL` String,
    `IsDownload` UInt8,
    `TraficSourceID` Int8,
    `SearchEngineID` UInt16,
    `SearchPhrase` String,
    `AdvEngineID` UInt8,
    `PlaceID` Int32,
    `RefererCategories` Array(UInt16),
    `URLCategories` Array(UInt16),
    `URLRegions` Array(UInt32),
    `RefererRegions` Array(UInt32),
    `IsYandex` UInt8,
    `GoalReachesDepth` Int32,
    `GoalReachesURL` Int32,
    `GoalReachesAny` Int32,
    `SocialSourceNetworkID` UInt8,
    `SocialSourcePage` String,
    `MobilePhoneModel` String,
    `ClientEventTime` DateTime,
    `RegionID` UInt32,
    `ClientIP` UInt32,
    `ClientIP6` FixedString(16),
    `RemoteIP` UInt32,
    `RemoteIP6` FixedString(16),
    `IPNetworkID` UInt32,
    `SilverlightVersion3` UInt32,
    `CodeVersion` UInt32,
    `ResolutionWidth` UInt16,
    `ResolutionHeight` UInt16,
    `UserAgentMajor` UInt16,
    `UserAgentMinor` UInt16,
    `WindowClientWidth` UInt16,
    `WindowClientHeight` UInt16,
    `SilverlightVersion2` UInt8,
    `SilverlightVersion4` UInt16,
    `FlashVersion3` UInt16,
    `FlashVersion4` UInt16,
    `ClientTimeZone` Int16,
    `OS` UInt8,
    `UserAgent` UInt8,
    `ResolutionDepth` UInt8,
    `FlashMajor` UInt8,
    `FlashMinor` UInt8,
    `NetMajor` UInt8,
    `NetMinor` UInt8,
    `MobilePhone` UInt8,
    `SilverlightVersion1` UInt8,
    `Age` UInt8,
    `Sex` UInt8,
    `Income` UInt8,
    `JavaEnable` UInt8,
    `CookieEnable` UInt8,
    `JavascriptEnable` UInt8,
    `IsMobile` UInt8,
    `BrowserLanguage` UInt16,
    `BrowserCountry` UInt16,
    `Interests` UInt16,
    `Robotness` UInt8,
    `GeneralInterests` Array(UInt16),
    `Params` Array(String),
    `Goals` Nested(
        ID UInt32,
        Serial UInt32,
        EventTime DateTime,
        Price Int64,
        OrderID String,
        CurrencyID UInt32),
    `WatchIDs` Array(UInt64),
    `ParamSumPrice` Int64,
    `ParamCurrency` FixedString(3),
    `ParamCurrencyID` UInt16,
    `ClickLogID` UInt64,
    `ClickEventID` Int32,
    `ClickGoodEvent` Int32,
    `ClickEventTime` DateTime,
    `ClickPriorityID` Int32,
    `ClickPhraseID` Int32,
    `ClickPageID` Int32,
    `ClickPlaceID` Int32,
    `ClickTypeID` Int32,
    `ClickResourceID` Int32,
    `ClickCost` UInt32,
    `ClickClientIP` UInt32,
    `ClickDomainID` UInt32,
    `ClickURL` String,
    `ClickAttempt` UInt8,
    `ClickOrderID` UInt32,
    `ClickBannerID` UInt32,
    `ClickMarketCategoryID` UInt32,
    `ClickMarketPP` UInt32,
    `ClickMarketCategoryName` String,
    `ClickMarketPPName` String,
    `ClickAWAPSCampaignName` String,
    `ClickPageName` String,
    `ClickTargetType` UInt16,
    `ClickTargetPhraseID` UInt64,
    `ClickContextType` UInt8,
    `ClickSelectType` Int8,
    `ClickOptions` String,
    `ClickGroupBannerID` Int32,
    `OpenstatServiceName` String,
    `OpenstatCampaignID` String,
    `OpenstatAdID` String,
    `OpenstatSourceID` String,
    `UTMSource` String,
    `UTMMedium` String,
    `UTMCampaign` String,
    `UTMContent` String,
    `UTMTerm` String,
    `FromTag` String,
    `HasGCLID` UInt8,
    `FirstVisit` DateTime,
    `PredLastVisit` Date,
    `LastVisit` Date,
    `TotalVisits` UInt32,
    `TraficSource` Nested(
        ID Int8,
        SearchEngineID UInt16,
        AdvEngineID UInt8,
        PlaceID UInt16,
        SocialSourceNetworkID UInt8,
        Domain String,
        SearchPhrase String,
        SocialSourcePage String),
    `Attendance` FixedString(16),
    `CLID` UInt32,
    `YCLID` UInt64,
    `NormalizedRefererHash` UInt64,
    `SearchPhraseHash` UInt64,
    `RefererDomainHash` UInt64,
    `NormalizedStartURLHash` UInt64,
    `StartURLDomainHash` UInt64,
    `NormalizedEndURLHash` UInt64,
    `TopLevelDomain` UInt64,
    `URLScheme` UInt64,
    `OpenstatServiceNameHash` UInt64,
    `OpenstatCampaignIDHash` UInt64,
    `OpenstatAdIDHash` UInt64,
    `OpenstatSourceIDHash` UInt64,
    `UTMSourceHash` UInt64,
    `UTMMediumHash` UInt64,
    `UTMCampaignHash` UInt64,
    `UTMContentHash` UInt64,
    `UTMTermHash` UInt64,
    `FromHash` UInt64,
    `WebVisorEnabled` UInt8,
    `WebVisorActivity` UInt32,
    `ParsedParams` Nested(
        Key1 String,
        Key2 String,
        Key3 String,
        Key4 String,
        Key5 String,
        ValueDouble Float64),
    `Market` Nested(
        Type UInt8,
        GoalID UInt32,
        OrderID String,
        OrderPrice Int64,
        PP UInt32,
        DirectPlaceID UInt32,
        DirectOrderID UInt32,
        DirectBannerID UInt32,
        GoodID String,
        GoodName String,
        GoodQuantity Int32,
        GoodPrice Int64),
    `IslandID` FixedString(16)
)
ENGINE = CollapsingMergeTree(Sign)
PARTITION BY toYYYYMM(StartDate)
ORDER BY (CounterID, StartDate, intHash32(UserID), VisitID)
SAMPLE BY intHash32(UserID)

hits_v1使用 MergeTree引擎，而visits_v1使用 Collapsing引擎

导入数据

从文件中导入数据

./clickhouse client --query "INSERT INTO tutorial.hits_v1 FORMAT TSV" --max_insert_block_size=100000 < hits_v1.tsv
./clickhouse client --query "INSERT INTO tutorial.visits_v1 FORMAT TSV" --max_insert_block_size=100000 < visits_v1.tsv

查看默认配置

SELECT name, value, changed, description
FROM system.settings
WHERE name LIKE '%max_insert_b%'

OPTIMIZE导入后的表

optimize table tutorial.hits_v1 final
optimize table tutorial.visits_v1 final

检查数据是否插入成功

./clickhouse client --query "SELECT COUNT(*) FROM tutorial.hits_v1"
./clickhouse client --query "SELECT COUNT(*) FROM tutorial.visits_v1"

查询数据

可以在DataGrip中使用

截屏2021-04-21 下午3.42.45.png

select StartURL      as URL,
       avg(Duration) as AvgDuration
from tutorial.visits_v1
where StartDate between '2014-03-23' and '2014-03-30'
group by URL
ORDER BY AvgDuration DESC
limit 10

SELECT sum(Sign)                           AS visits,
       sumIf(Sign, has(Goals.ID, 1105530)) AS goal_visits,
       (100. * goal_visits) / visits       AS goal_percent
FROM tutorial.visits_v1
WHERE (CounterID = 912887)
  AND (toYYYYMM(StartDate) = 201403)
  AND (domain(StartURL) = 'yandex.ru')

Clickhouse单机版快速入门
本文以操作系统为macos为例参考链接 https://clickhouse.tech/docs/zh/getti...
clickhouse
clickhouse相关内容单机版安装端口 9009 端口：用于在ClickHouse服务器之间交换数据的端口...
搭建本地debs源安装clickhouse
最近在学习clickhouse，在单机版上玩了一段时间之后，决定搭建clickhouse cluster。我买了4...
【clickhouse专栏】单机版的安装与验证
《clickhouse专栏》第三节内容，先安装一个单机版的clickhouse，是后续学习多副本或者分布式集群安装...
篇一|ClickHouse快速入门
ClickHouse简介 ClickHouse是一个用于联机分析(OLAP)的列式数据库管理系统(DBMS)。Cl...
VersionedCollapsingMergeTree 与 C
ClickHouse入门：表引擎-VersionedCollapsingMergeTree_高矮-CSDN博客[h...
clickhouse入门
clickhouse使用 clickhouse介绍启动方式 clickhouse-server --daemon...
Clickhouse入门
我们先获取一些开源数据样本集，我们将使用美国1987到2015年的民用航班数据，很难称这个样本为大数据（只包含1亿...
ClickHouse 入门
1 概述什么是 ClickHouse？ ClickHouse 是俄罗斯的Yandex于2016年开源的列式存储数...
Clickhouse 入门
clickhouse 简介ck是一个列式存储的数据库，其针对的场景是OLAP。OLAP的特点是：数据不经常写，即...

Clickhouse单机版快速入门

macos下载二进制包

启动服务端和客户端

修改文件句柄数

启动服务端

启动客户端

下载并且提取数据

下载数据集

创建数据库

创建表引擎

导入数据

从文件中导入数据

查看默认配置

OPTIMIZE导入后的表

检查数据是否插入成功

查询数据

相关文章

Clickhouse单机版快速入门

clickhouse

搭建本地debs源安装clickhouse

【clickhouse专栏】单机版的安装与验证

篇一|ClickHouse快速入门

VersionedCollapsingMergeTree 与 C

clickhouse入门

Clickhouse入门

ClickHouse 入门

Clickhouse 入门

网友评论

延伸阅读

深度阅读

栏目导航

热点阅读