后台常用操作

How can I list my open network ports with netstat

netstat -ap tcp | grep -i "listen"

查看端口 lsof -I:8000

关闭端口 kill: usage: kill [-s sigspec | -n signum | -sigspec] pid | jobspec ... or kill -l [sigspec] kill -9 PID

查找文件

Ansible

增加sudo用户并设置无密码sudo

adduser <username>

set passwd

usermod -a -G sudo <username>

chmod u+w /etc/sudoers

set %sudo ALL=NOPASSWD: ALL in /etc/sudoers

su <username>

set ~/.ssh/authorized_keyscd

cat 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC//j84AdUKb+1lF9vASGJID+OStuDlbvNkCN3ld/Om525+BNl2qbk2NkY60zgwUcb5eETguxjj0c6xfFMQr/H3bsJSVwhfh8aUmR5jSp5dnCglaqIyLVleHnZETE4H5s089ogUnGV3JRAbqF2nuLtvXnl/G3O/YK38xZYCowLnsHfu5ek74SU0Z5Lv3x/q9MiiAcLLCC5oXf5E6ZtIHHqBUYkaMHdrgwaQVhnkB+dADRs7lxRkfQuCxUPjhUJBRznh+PVwxo4n+DkFgNuWnE1sXnnMOIBIfvhk/dzxNQwgEWY1V/I7H2RnOzHw0slU1KPW/XAKqO+82reDYrME/2/l deploy-meson' >> ~/.ssh/authorized_keys

ansible-playbook -i hosts meson_photon_job.yml

Running Playbooks

ansible-playbook myplaybook.yml -I

ansible-playbook myplaybook.yml --list-hosts

DOCKER

Docker for Mac国内镜像

Insecure registries:

registry.mirrors.aliyuncs.com

Registry mirrors:

[https://pfmzeohs.mirror.aliyuncs.com](javascript:void())(你自己的加速URL)

你的加速URL可以登录https://cr.console.aliyun.com/#/accelerator进行查询

建议加入其他国内镜像

https://docker.mirrors.ustc.edu.cn

https://hub-mirror.c.163.com

docker常用命令

docker build -t friendlyhello . # Create image using this directory's Dockerfile

docker run -p 4000:80 friendlyhello # Run "friendlyname" mapping port 4000 to 80

docker run -d -p 4000:80 friendlyhello # Same thing, but in detached mode

docker container ls # List all running containers

docker container ls -a # List all containers, even those not running

docker container stop <hash> # Gracefully stop the specified container

docker container kill <hash> # Force shutdown of the specified container

docker container rm <hash> # Remove specified container from this machine

docker container rm $(docker container ls -a -q) # Remove all containers

docker image ls -a # List all images on this machine

docker image rm <image id> # Remove specified image from this machine

docker image rm $(docker image ls -a -q) # Remove all images from this machine

docker login # Log in this CLI session using your Docker credentials

docker tag <image> username/repository:tag # Tag <image> for upload to registry

docker push username/repository:tag # Upload tagged image to registry

docker run username/repository:tag # Run image from a registry

查看所有状态的容器 docker container ls -a

新建并启动docker run

启动已终止容器 docker container start [OPTIONS] CONTAINER [CONTAINER...]

后台运行添加 -d 参数来实现 docker run -d

终止容器 docker container stop [OPTIONS] CONTAINER [CONTAINER...]

进入容器使用 docker attach 命令或 docker exec 命令，推荐大家使用 docker exec 命令 原因：从这个 stdin 中 exit，不会导致容器的停止 docker exec -it

删除容器 docker container rm [****OPTIONS****] CONTAINER [****CONTAINER...****]****[****flags****]

清理所有处于终止状态的容器docker container prune

Ubuntu docker mongodb install process

docker pull mongo

sudo adduser mongo password: 123456

sudo docker run --name mongodb --restart unless-stopped \

-v /home/mongo/data:/data/db -v /home/mongo/backups:/backups -d mongo --smallfiles

数据库

MongoDB

从镜像创建并启动服务器 docker run -p 27017:27017 -v $PWD/db:/data/db -d --name scrapy mongo

启动已暂停服务器 docker container start scrapy

进入客户端 mongo

show dbs use scrapy show collections use paper

db.col.find().pretty() db.col.find().count() db.col.drop()

使用limit()方法来读取指定数量的数据外，还可以使用skip()方法来跳过指定数量的数据

db.COLLECTION_NAME.find().limit(NUMBER).skip(NUMBER)

Mysql

进入客户端 mysql -uroot -p123456

show databases; use baryon; show tables;

select * from table_name limit 0,10 --通常0是可以省略的，直接写成 limit 10。0代表从第0条记录后面开始，也就是从第一条开始

select * from articles_article where created_at = (select created_at from articles_article order by created_at desc limit 1)

select * from papers_paper where created_at = (select created_at from papers_paper order by created_at desc limit 1);

select * from news_news where created_at in (select created_at from news_news order by created_at desc limit 2);

SELECT COUNT(*) FROM papers_paper;

DELETE FROM papers_paper;

忽略外键约束

SET foreign_key_checks = 0;

DELETE FROM articles_article WHERE id>100;

进入docker 客户端 docker run -p 3306:3306 --name mysql-baryon -e MYSQL_ROOT_PASSWORD=123456 -d mysql:latest

启动本地 mysql 服务器 mysql.server start

Connecting to MySQL through Docker

https://medium.com/coderscorner/connecting-to-mysql-through-docker-997aa2c090cc

Accessing MySQL through a Web Interface

1. docker pull phpmyadmin/phpmyadmin

2. docker run --name myadmin -d --link mysql-baryon:db -p 8080:80 phpmyadmin/phpmyadmin

1. http://localhost:8080/’

Redis

启动 docker run -p 0.0.0.0:6379:6379 -it --name="redis-server" -d redis

docker run:重新创建一个容器并运行命令，语法如下：

Docker run [OPTIONS] IMAGES [COMMAND][ARG]

通常用的OPTIONS为：

-t 为容器分配一个伪输入终端

-I 以交互模式运行容器

-name 为容器指定名称

-d 以后台模式运行

-p 端口映射

redis客户端

进入 docker exec [OPTIONS] CONTAINER COMMAND [ARG...]

docker exec -it redis-server redis-cli

127.0.0.1:6379> set day newDay

OK

127.0.0.1:6379> get day

"newDay"

127.0.0.1:6379> exit

常用命令：
1） 查看keys个数
keys * // 查看所有keys
keys prefix_* // 查看前缀为”prefix_”的所有keys

2） 清空数据库
flushdb // 清除当前数据库的所有keys
flushall // 清除所有数据库的所有keys

查看redis

docker exec -it 16db4f78420a redis-cli

KEYS *

2)”rq:job:ca82af29-5744-4d62-afe0-3cba45b2d31d"

2. "rq:queue:default"

3. "rq:queues"

TYPE rq:queues set SMEMBERS rq:queues

TYPE rq:queue:default list LRANGE rq:queue:default 0 12

TYPE rq:job:ca82af29-5744-4d62-afe0-3cba45b2d31d hash

HGETALL rq:job:ca82af29-5744-4d62-afe0-3cba45b2d31d

job的信息是以hash形式存储的，奇数行是key，偶数是value

python manage.py rqworker default low

rq:worker:myhost.10179是该worker的标识

Elasticsearch

docker run -d -p 9200:9200 --name="es" elasticsearch

Git

更新仓库： git pull

拉取分支：git branch paper

切换分支： git checkout paper

新建并切换分支： git checkout -b paper

解决每次提交出现.ideas文件夹下的更改

http://oomusou.io/phpstorm/phpstorm-git-rm-idea/

Scrapy

创建scrapy项目 scrapy startproject tutorial

运行项目 scrapy crawl quotes

新建爬虫 scrapy genspider example example.com

pycharm中调试爬虫

from scrapy import cmdline

cmdline.execute("scrapy crawl chainfor".split())

Scrapinghub

Portia

scrapyd 部署

取消任务 curl http://spider.coinbeef.com/cancel.json -d project=chainevent -d job=4deb19b06d3411e89011de22dba28bc9

删除项目 curl http://spider.coinbeef.com/delproject.json -d project=phonton

爬虫 调试 scrapy

from scrapy import cmdline

cmdline.execute('scrapy crawl eprint'.split())

-bash: scrapyd-client: command not found

pip install git+https://github.com/scrapy/scrapyd-client

which scrapyd-client

/Users/tarun.lalwani/.virtualenvs/sclient/bin/scrapyd-client

Django

创建admin账户

Meson—>lepton python manage.py createsuperuser

Baryon python manage.py createsuperuser --settings=baryon.settings.dev_wancheng

Django-rq

pytcharm中调试worker

python manage.py rqworker high default low

Django-extensions

./manage.py runjob papers crawl_papers

./manage.py runjob -l papers crawl_papers

创建虚拟环境

mkvirtualenv -p which python3 env_name

mkvirtualenv --no-setuptools --no-wheel -p which python3 rest 不安装 setuptools wheel 使用 Python

注意：符号不是引号’’,而是``

workon: command not found

source /usr/local/bin/virtualenvwrapper.sh

**Git **

git remote -v

git fetch origin master

git log -p master.. origin/master

git merge origin/master

git push <remote> <local branch name>:<remote branch to push into> example git push origin fix_paper_admin:master

How do I revert all local changes in Git managed project to previous state?

If you want to revert changes made to your working copy, do this:

git checkout .

If you want to revert changes made to the index (i.e., that you have added), do this. Warning this will reset all of your unpushed commits to master!:

git reset

If you want to revert a change that you have committed, do this:

git revert <commit 1> <commit 2>

If you want to remove untracked files (e.g., new files, generated files):

git clean -f

Or untracked directories (e.g., new or automatically generated directories):

git clean -fd

忽略 .idea 文件

fork

git remote -v

git remote add upstream https://github.com/ORIGINAL_OWNER/ORIGINAL_REPOSITORY.git

git remote add upstream https://github.com/Bit03/Baryon.git

git fetch upstream

git checkout master

git merge upstream/master

nothing to commit, working tree clean

git pull upstream master 获取上游项目 master 分支

fatal: refusing to merge unrelated histories —> git pull upstream master --allow-unrelated-histories