《MongoDB极简教程》第一章 NoSQL简史 & M

作者: 光剑书架上的书 | 来源:发表于2017-03-19 02:59 被阅读389次

MongoDB 是一款开源的文档数据库，并且是业内领先的 NoSQL 数据库，用 C++ 编写而成。

NoSQL

(NoSQL = Not Only SQL )，意即"不仅仅是SQL"。

在现代的计算系统上每天网络上都会产生庞大的数据量。
这些数据有很大一部分是由关系数据库管理系统（RDMBSs）来处理。 1970年 E.F.Codd's提出的关系模型的论文 "A relational model of data for large shared data banks"，这使得数据建模和应用程序编程更加简单。

通过应用实践证明，关系模型是非常适合于客户服务器编程，远远超出预期的利益，今天它是结构化数据存储在网络和商务应用的主导技术。

NoSQL 是一项全新的数据库革命性运动，早期就有人提出，发展至2009年趋势越发高涨。NoSQL的拥护者们提倡运用非关系型的数据存储，相对于铺天盖地的关系型数据库运用，这一概念无疑是一种全新的思维的注入。

RDBMS vs NoSQL
RDBMS

高度组织化结构化数据
结构化查询语言（SQL） (SQL)
数据和关系都存储在单独的表中。
数据操纵语言，数据定义语言
严格的一致性
基础事务
NoSQL
代表着不仅仅是SQL
没有声明性查询语言
没有预定义的模式
-键 - 值对存储，列存储，文档存储，图形数据库
最终一致性，而非ACID属性
非结构化和不可预知的数据
CAP定理
高性能，高可用性和可伸缩性

NoSQL 简史

NoSQL一词最早出现于1998年，是Carlo Strozzi开发的一个轻量、开源、不提供SQL功能的关系数据库。

2009年，Last.fm的Johan Oskarsson发起了一次关于分布式开源数据库的讨论[2]，来自Rackspace的Eric Evans再次提出了NoSQL的概念，这时的NoSQL主要指非关系型、分布式、不提供ACID的数据库设计模式。

2009年在亚特兰大举行的"no:sql(east)"讨论会是一个里程碑，其口号是"select fun, profit from real_world where relational=false;"。因此，对NoSQL最普遍的解释是"非关联型的"，强调Key-Value Stores和文档数据库的优点，而不是单纯的反对RDBMS。

CAP定理（CAP theorem）

在计算机科学中, CAP定理（CAP theorem）, 又被称作布鲁尔定理（Brewer's theorem）, 它指出对于一个分布式计算系统来说，不可能同时满足以下三点:

一致性(Consistency) (所有节点在同一时间具有相同的数据)
可用性(Availability) (保证每个请求不管成功或者失败都有响应)
分隔容忍(Partition tolerance) (系统中任意信息的丢失或失败不会影响系统的继续运作)

CAP理论的核心是：一个分布式系统不可能同时很好的满足一致性，可用性和分区容错性这三个需求，最多只能同时较好的满足两个。
因此，根据 CAP 原理将 NoSQL 数据库分成了满足 CA 原则、满足 CP 原则和满足 AP 原则三大类：

CA - 单点集群，满足一致性，可用性的系统，通常在可扩展性上不太强大。
CP - 满足一致性，分区容忍性的系统，通常性能不是特别高。
AP - 满足可用性，分区容忍性的系统，通常可能对一致性要求低一些。

cap-theoram-image

NoSQL的优点/缺点

优点:

高可扩展性
分布式计算
低成本
架构的灵活性，半结构化数据
没有复杂的关系

缺点:

没有标准化
有限的查询功能（到目前为止）
最终一致是不直观的程序

BASE

BASE：Basically Available, Soft-state, Eventually Consistent。由 Eric Brewer 定义。

CAP理论的核心是：一个分布式系统不可能同时很好的满足一致性，可用性和分区容错性这三个需求，最多只能同时较好的满足两个。
BASE是NoSQL数据库通常对可用性及一致性的弱要求原则:

Basically Availble --基本可用
Soft-state --软状态/柔性事务。 "Soft state" 可以理解为"无连接"的, 而 "Hard state" 是"面向连接"的
Eventual Consistency --最终一致性最终一致性，也是是 ACID 的最终目的。

MongoDB 特性&优势

MongoDB 特性	优势
事务支持 MongoDB	目前只支持单文档事务，需要复杂事务支持的场景暂时不适合
灵活的文档模型	JSON 格式存储最接近真实对象模型，对开发者友好，方便快速开发迭代
高可用复制集	满足数据高可靠、服务高可用的需求，运维简单，故障自动切换
可扩展分片集群	海量数据存储，服务能力水平扩展
高性能	mmapv1、wiredtiger、mongorocks（rocksdb）、in-memory 等多引擎支持满足各种场景需求
强大的索引支持	地理位置索引可用于构建各种 O2O 应用、文本索引解决搜索的需求、TTL索引解决历史数据自动过期的需求
Gridfs	解决文件存储的需求
aggregation & mapreduce	解决数据分析场景需求，用户可以自己写查询语句或脚本，将请求都分发到 MongoDB 上完成

文档参考

中文社区

MongoDB 极简实践入门

安装&环境配置

下载：https://www.mongodb.com/download-center#community

安装：
https://docs.mongodb.com/manual/administration/install-community/

解压，配置环境变量PATH即可。

MONGODB_HOME=/Users/jack/soft/mongodb-osx-x86_64-3.4.2
export PATH=$PATH:$MONGODB_HOME/bin

启动脚本

cat start_mongodb.sh 
#!/bin/bash
source ~/.bashrc

mongod --dbpath ~/dbdata

启动服务进程

./start_mongodb.sh 

2017-03-19T02:50:44.491+0800 I CONTROL  [initandlisten] MongoDB starting : pid=67062 port=27017 dbpath=/Users/jack/dbdata 64-bit host=ChenJackdeMacBook-Pro.local
2017-03-19T02:50:44.492+0800 I CONTROL  [initandlisten] db version v3.4.2
2017-03-19T02:50:44.492+0800 I CONTROL  [initandlisten] git version: 3f76e40c105fc223b3e5aac3e20dcd026b83b38b
2017-03-19T02:50:44.492+0800 I CONTROL  [initandlisten] allocator: system
2017-03-19T02:50:44.492+0800 I CONTROL  [initandlisten] modules: none
2017-03-19T02:50:44.492+0800 I CONTROL  [initandlisten] build environment:
2017-03-19T02:50:44.492+0800 I CONTROL  [initandlisten]     distarch: x86_64
2017-03-19T02:50:44.492+0800 I CONTROL  [initandlisten]     target_arch: x86_64
2017-03-19T02:50:44.492+0800 I CONTROL  [initandlisten] options: { storage: { dbPath: "/Users/jack/dbdata" } }
2017-03-19T02:50:44.494+0800 I STORAGE  [initandlisten] wiredtiger_open config: create,cache_size=3584M,session_max=20000,eviction=(threads_max=4),config_base=false,statistics=(fast),log=(enabled=true,archive=true,path=journal,compressor=snappy),file_manager=(close_idle_time=100000),checkpoint=(wait=60,log_size=2GB),statistics_log=(wait=0),
2017-03-19T02:50:45.352+0800 I CONTROL  [initandlisten] 
2017-03-19T02:50:45.353+0800 I CONTROL  [initandlisten] ** WARNING: Access control is not enabled for the database.
2017-03-19T02:50:45.353+0800 I CONTROL  [initandlisten] **          Read and write access to data and configuration is unrestricted.
2017-03-19T02:50:45.353+0800 I CONTROL  [initandlisten] 
2017-03-19T02:50:45.353+0800 I CONTROL  [initandlisten] 
2017-03-19T02:50:45.353+0800 I CONTROL  [initandlisten] ** WARNING: soft rlimits too low. Number of files is 256, should be at least 1000
2017-03-19T02:50:45.748+0800 I FTDC     [initandlisten] Initializing full-time diagnostic data capture with directory '/Users/jack/dbdata/diagnostic.data'
2017-03-19T02:50:46.073+0800 I INDEX    [initandlisten] build index on: admin.system.version properties: { v: 2, key: { version: 1 }, name: "incompatible_with_version_32", ns: "admin.system.version" }
2017-03-19T02:50:46.073+0800 I INDEX    [initandlisten]      building index using bulk method; build may temporarily use up to 500 megabytes of RAM
2017-03-19T02:50:46.123+0800 I INDEX    [initandlisten] build index done.  scanned 0 total records. 0 secs
2017-03-19T02:50:46.124+0800 I COMMAND  [initandlisten] setting featureCompatibilityVersion to 3.4
2017-03-19T02:50:46.124+0800 I NETWORK  [thread1] waiting for connections on port 27017



2017-03-19T02:51:24.533+0800 I NETWORK  [thread1] connection accepted from 127.0.0.1:49370 #1 (1 connection now open)
2017-03-19T02:51:24.534+0800 I NETWORK  [conn1] received client metadata from 127.0.0.1:49370 conn1: { application: { name: "MongoDB Shell" }, driver: { name: "MongoDB Internal Client", version: "3.4.2" }, os: { type: "Darwin", name: "Mac OS X", architecture: "x86_64", version: "16.4.0" } }

Terminal连接客户端

$ mongo
MongoDB shell version v3.4.2
connecting to: mongodb://127.0.0.1:27017
MongoDB server version: 3.4.2
Server has startup warnings: 
2017-03-19T02:50:45.352+0800 I CONTROL  [initandlisten] 
2017-03-19T02:50:45.353+0800 I CONTROL  [initandlisten] ** WARNING: Access control is not enabled for the database.
2017-03-19T02:50:45.353+0800 I CONTROL  [initandlisten] **          Read and write access to data and configuration is unrestricted.
2017-03-19T02:50:45.353+0800 I CONTROL  [initandlisten] 
2017-03-19T02:50:45.353+0800 I CONTROL  [initandlisten] 
2017-03-19T02:50:45.353+0800 I CONTROL  [initandlisten] ** WARNING: soft rlimits too low. Number of files is 256, should be at least 1000
> 
> 
> show dbs;
admin  0.000GB
local  0.000GB

强大的 MongoDB 管理工具

MongoDB Shell Enhancements

https://github.com/Jason-Chen-2017/mongo-hacker

Installation

npm install -g mongo-hacker

Enhancements

Basic UX

Sort document keys by default
Highlight querytime if verboseShell is enabled
- In green if querytime is at or below slowms
- In red if query time is above slowms
Default indent is 2 spaces instead of tab
- Customizable by setting indent key of config
Verbose shell is enabled by default -- to disable: setVerboseShell(false)
Disable notfication of "Type 'it' for more"
Custom prompt: hostname(process-version)[rs_status:set_name] db>
Always pretty print. You can still use default format by appending .ugly() to the end of db statement.
Show DBs has aligned columns, is sorted by database name and shows less significant digits (in master for Mongo 2.5/2.6)
Nicer sh.status() output (remove lastmod, take up less space, colorize chunk's shard)
Colorized query output for console/terminal windows supporting ANSI color codes.

Colorized Output

Additional shell commands

The MongoDB shell offers various "shell commands" (sometimes referred to as "shell helpers" as well) that make interactive use of the shell much more convenient than proper, Javascript-only scripted use of the shell.

To make interactive use of the MongoDB shell even more convenient, mongo-hacker adds the following shell commands:

count collections/count tables: count the number of collections in each of the mongo server's databases - by @pvdb
count documents/count docs: count the number of documents in all (non-system) collections in the database - by @pvdb
count indexes: list all collections and display the size of all indexes - by @cog-g

Some of these commands have hidden features that can be enabled in the mongo-hacker config, to make the command output even more useful:

by changing the count_deltas setting to true in config.js, the count documents command will also print out the change in the number of documents since the last count - by @pvdb

API Additions

Scripting

Get a list of database names: (by @pvdb)

db.getMongo().getDatabaseNames()

(note that this method is similar - functionality-wise and usage-wise - to the existing db.getCollectionNames() API method and allows for advanced, cross-database scripting in the MongoDB shell)

General

Filter for a collection of documents:

db.collection.filter(<criteria>)

One for finding a single document:

db.collection.find({ ... }).one() == db.collection.findOne({ ... })

Select for selecting fields to return (projection):

db.collection.find({ ... }).select({ name: 1 })

Reverse for descending sort by insertion order (default) or arbitrary field:

db.collection.find({ ... }).reverse()
db.collection.find({ ... }).reverse('createDate')

Last for finding last inserted document (default) or document last by given field:

db.collection.find({ ... }).last()
db.collection.find({ ... }).last('createDate')

Update, Replace, Upsert and Remove can be called on a DBQuery Object:

db.collection.find({ ... }).update({ ... })  // multi update
db.collection.find({ ... }).replace({ ... }) // single replacement
db.collection.find({ ... }).upsert({ ... })  // single upsert
db.collection.find({ ... }).remove()         // multi remove

Sort, limit, and skip through multi updates and removes:

db.collection.find({ ... }).limit(7).update({ ... })
db.collection.find({ ... }).sort({ ... }).skip(1).limit(3).update({ ... })
db.collection.find({ ... }).limit(3).remove()

Note: The performance of multi updates involving a skip or limit may be worse than those without those specifications due to there being absolutely no native support for this feature in MongoDB itself. It should be understood by the user of this software that use of this feature (by calling update on a cursor rather than a collection) is advanced and experimental. The option to do this sort of operation is purely additive to the MongoDB experience with MongoHacker and usage of it is in no way required. Furthermore, its inclusion in this enhancement does not effect the operation of updates invoked through collections and, in practice, is insanely useful.

Aggregation Framework

The aggregation framework is now fluent as well. You can use it as currently documented or via the chainable methods.

Calling aggregate without an array of operations or $operations will make it a match.

// matches every document
db.collection.aggregate()
db.collection.aggregate({})

// matches documents where the "a" is equal to 1
db.collection.aggregate({a: 1})

// matches documents where "a" is greater than 7
db.collection.aggregate({a: {$gt: 7}})

Additional methods can then be chained on top of the inital match in order to make more complicated aggregations.

// Match and project
db.collection.aggregate(<querydoc>).project(<projection>)
db.collection.aggregate({a: 1}).project({a: 1, _id: 0})

// Match, group and sort
db.collection.aggregate({<match>}).group({<group>}).sort({<sort>})
db.test.aggregate().group({_id: '$a', 'sum': {'$sum': 1}}).sort({sum: -1})

Data Generation

For easy and simple random data generation you can utilise these methods below. You can use any of these functions in a loop. For example:

// Inserts 20 documents with random data. 
for (i=1; i<21; i++) { 
    db.collection.insert(
            {
             word: randomWord(), 
             number: randomNumber(), 
             date: randomDate() 
            }
    ); 
}

randomWord

You can specify the length of each word, the number of words, and an optional seeded word in a sentence randomly. Use the optional seed parameter for testing text search.

randomWord(length=5, words=1, seed=undefined)

// Inserts a random sentence consisting of 5 letters per word, 5 words in total, 
// with a probability to insert the word 'needle' in the sentence
db.collection.insert( { words: randomWord(5, 5, 'needle') } )

// Inserts a random word consisting of 16 letters
db.collection.insert( { words: randomWord(16) } )

randomNumber

You can specify maximum number to be randomly generated (exclusive)

randomNumber(max=100)

// Inserts a random number in the range of 0 or 1. 
db.collection.insert( { number: randomNumber(2) } )

// Inserts a random number in the range of 0 or 999. 
db.collection.insert( { number: randomNumber(1000) } )

randomDate

You can specify start and end dates range to be randomly generated. (exclusive)

randomDate(start= <2 years ago> , end=Date() )

// Inserts a random date object in the range of 1st January 2016 to 1st February 2016
db.collection.insert( { date: randomDate(ISODate("2016-01-01T00:00:00"), ISODate("2016-02-01T00:00:00")) })

// If today is 19th May 2016 and you specify only the start of the day, 
// this will generate random date object between 00:00:00 to current time.  
db.collection.insert( { date: randomDate(ISODate("2016-05-19T00:00:00")) })

Helpers

General Shell Helpers

findCommand('search') list commands that match the search string

Aggregation Framework Helpers -- on collections

Group and Count: gcount(group_field, filter)
Group and Sum: gsum(group_field, sum_field, filter)
Group and Average: gavg(group_field, avg_field, filter)

Run function on some/all databases

runOnDbs(/db_names_regexp/, function(db) {
    // callback is ran for each database which name matches regular expression
    // db is that selected database
});

mongo-express

https://github.com/Jason-Chen-2017/mongo-express

Web-based MongoDB admin interface, written with Node.js and express

终端查看的效果：

jack@jacks-MacBook-Air:~$ mongo
MongoDB shell version: 3.2.4
connecting to: test
Mongo-Hacker 0.0.14
Server has startup warnings: 
2017-03-13T10:31:13.023+0800 I CONTROL  [initandlisten] ** WARNING: You are running this process as the root user, which is not recommended.
2017-03-13T10:31:13.023+0800 I CONTROL  [initandlisten] 
jacks-MacBook-Air(mongod-3.2.4) test> 
jacks-MacBook-Air(mongod-3.2.4) test> 
jacks-MacBook-Air(mongod-3.2.4) test> db
test
jacks-MacBook-Air(mongod-3.2.4) test> show dbs;
admin      → 0.000GB
local      → 0.000GB
restfiddle → 0.001GB
jacks-MacBook-Air(mongod-3.2.4) test> use restfiddle;
switched to db restfiddle
jacks-MacBook-Air(mongod-3.2.4) restfiddle> show collections;
EntityAuth        → 0.000MB / 0.004MB
activityLog       → 0.001MB / 0.035MB
baseNode          → 0.021MB / 0.043MB
config            → 0.000MB / 0.016MB
conversation      → 0.017MB / 0.035MB
httpRequestHeader → 0.005MB / 0.016MB
oAuth2            → 0.002MB / 0.016MB
project           → 0.002MB / 0.031MB
rfRequest         → 0.017MB / 0.035MB
rfResponse        → 0.036MB / 0.047MB
runnerLog         → 0.000MB / 0.016MB
tag               → 0.000MB / 0.016MB
user              → 0.000MB / 0.035MB
workspace         → 0.001MB / 0.031MB
jacks-MacBook-Air(mongod-3.2.4) restfiddle> l
2017-03-21T14:18:55.202+0800 E QUERY    [thread1] ReferenceError: l is not defined :
@(shell):1:1

jacks-MacBook-Air(mongod-3.2.4) restfiddle> db.runnerLog.find()
{
  "_id": ObjectId("575fad7dba637304fffbae85"),
  "_class": "com.restfiddle.entity.RunnerLog",
  "status": "ACTIVE",
  "version": NumberLong("0")
}
{
  "_id": ObjectId("575fad92ba637304fffbae86"),
  "_class": "com.restfiddle.entity.RunnerLog",
  "status": "ACTIVE",
  "version": NumberLong("0")
}
{
  "_id": ObjectId("575fad9cba637304fffbae87"),
  "_class": "com.restfiddle.entity.RunnerLog",
  "status": "ACTIVE",
  "version": NumberLong("0")
}
Fetched 3 record(s) in 7ms
jacks-MacBook-Air(mongod-3.2.4) restfiddle>

mongo-express的效果：

螢幕快照 2017-03-21 14.25.03.png

螢幕快照 2017-03-21 14.25.57.png