Week 1

Jan 9th Intro to Linux

1. What's an OS?

Operation System

2. OS services include: File system, Scheduler, I/O and communication

3. CLI (Command Line Interface) & GUI (Graphical Interface)

4. Remote server access using ssh

5. Linux commands

• date - Get the current date and time
• scp - copy a file from or onto a remote machine
  • Remote to Local scp linuxuser@216.230.228.88:~/iris.csv ~/Desktop/
  • Local to Remote scp ~/Desktop/iris.csv linuxuser@216.230.228.88:~/iris2.csv
• ls /etc: The /etc directory contains a lot of system files.
• pwd
• curl - download an html page from the web. The -O (that’s capital
  O, not zero) means the file should be saved locally with the same name
  as it has remotely (curl1.html, in this case). (wget is a very similar
  command; some systems have wget, some have curl, some have both.)
  curl -O "http://linuxcommand.org/man_pages/curl1.html"
less curl1.html
curl http://linuxcommand.org/man_pages/curl1.html > curhtml
less curhtml
• cd $HOME == cd ~ == cd
• man ls
• ls --help
• ls -a -t / ls -at: “-t” sorts the file listing by modification date.
• ls -l
  Structure of -l
• mkdir
$ mkdir -p ~/examples/multiple/levels/down
$ ls -R ~/examples # -R means list contents “recursively”
• cp [-r] source destination #Add the -r option if you are copying a folder.
  cp /etc/magic .
cp /etc/hosts nethosts
• mv source destination
mv magic examples #Move the magic folder into examples
• mv old_file_name new_file_name # When the destination is a new name (rather than an existing folder), mv acts as a renaming operation.
• rm magicData.txt
• rmdir: The rmdir command supports deleting empty directories
• rm -r files_folder
• less
  • Scroll up/down one line at a time with arrow keys.
  • Use the space bar to scroll through a screen at a time.
  • Use the / key to search for a term: /apple
  • Press h to see a quick list of other options.
  • Press q to exit less
• Global *
  • ls *txt - list the names of all files whose names end with txt.
  • cat *abc* - list the contents of all files whose names contain ‘abc’.
  • cat de* > de_files # Concatenate all the files that start with “de” into a file de-files (the ">"character redirect standard output to a file)
• “redirecting” the output of a command into a file. echo Some random text > random.txt
• The trick is the “>”, which says: instead of printing to the terminal, put
  the output into a file. This is called “I/O redirection.” ls -l /home > ls_out
• vi file, type i to go into insert mode, in command mode type ZZ to save the file and exit vi.

vi Command mode vi Insert mode

• grep word file1 [file2 ... ] - Search through a given file using a string
  grep user /etc/passwd
• wc file1 [file2 ... ] - Count lines and words
  wc /etc/*conf
wc -l README.md hi.txt ls_root # Get only lines
• sort file - Sort all lines in file Sort on the first field (permissions).
• sort -k9 ls_root - Sort on the name field
• sort -k5 -n ls_root - Sorting on a number field (add -n flag to use numerical value)

Command Line from LMD

1. Types of Shells

   • Bourne Shell (sh)
   • Bourne Again Shell (bash)

   which bash
bash --version

2. On most systems, the default Bash prompt is terminated by the $ character when you run it as a “non-super user” (i.e., as a user with normal system permissions) and the # character when you run it as a super user

3. Long-format Listing

   • ls -lh - -h flag prints the file sizes with units (bytes, kilobytes, megabytes)
   • Sorting the Listing
     • -S to sort files by size (in bytes)
     • -t to sort files by last-modified time
     • -u to sort files by time of last access
     • -U to sort files by time of creation
     • -r to reverse the sort order based on any of the previous options, or if none is provided, to reverse the sort order alphabetically

• Combining Options
  • ls -al to list hidden files using the long-format listing
  • ls -lrs to list files using the long-format listing in order of increasing size

4. Quoting provides techniques for removing the special meaning of certain characters or groups of characters.

   $ echo hello whoami # outputs: hello whoami
   $ echo 'hello whoami' # outputs: hello whoami
   $ echo "hello whoami" # outputs: hello whoami
   $ echo 'hello "whoami"' # outputs: hello "whoami"
   $ echo "hello 'whoami'" # outputs: hello 'whoami'
   $ echo 'hello $(whoami)' # outputs: hello     $(whoami)
   $ echo "hello $(whoami)" # outputs: hello josep
   $ echo "hello `whoami`" # outputs: hello josep
   

5. More If the contents do not fit, however, more
   will print the portion that does fit and allow you to scroll downwards through the remainder of the file using the down arrow key or the SPACEBAR
   . Press ESC or the Q key to exit.

6. Less allows you the convenience of being able to scroll upward, back through the file contents in case you missed something

7. Creating Empty Files:

   • touch new_file.txt
   • pico new_file.txt To open Pico’s help section, use Ctrl-G or the menu option labeled “Get Help.” To exit Pico, enter Ctrl-X

8. Deleting Files: While trying all examples, add always the “-i” flag. It will prompt you before deleting anything.

   • rm -f *.js to use the -f flag to try a force deletion
   • The rmdir command only works on empty directories.
   • To remove a non-empty directory, use the rm
     command with the -r flag to indicate recursive deletion
   • rm -rf my_directory to override the confirmation, you can use the -r flag coupled with the -f force deletion flag

9. Command Output

   • The > operator will add to a file or replace the contents of a file, while the >> operator will append to a file or add to the end of a file.
   • The tar command is a program used to create archives.
     tar -czf archivename.tar.gz file1 file2 file3 more The flag c means “create a new file”, the flag z means “compress, filter the archive through gzip”, and the flag f means “use an archive file (archivename.tar.gz)”.

10. Command substitution is used to execute a command as an argument (or option) of another command. Or to store the result of a command into a variable.

    $ echo My name is `whoami`
    $ echo My name is $(whoami)
    $ CURRENTDIR=`pwd`
    $ CURRENTDIR=$(pwd)
    

11. The curl command is a tool to transfer data from or to a server, it supports a wide variety of protocols. To write the output to a file, we could use the -o flag, followed by the local file name we want to create.
    curl login.moderndeveloper.com -o loginpage.html

12. Piping, | allows you to send the output of one command as the input of another.

    • ls | wc shows how to pipe the output of ls
      to wc

13. The grep command is a utility use to search for a pattern, and you can use it for something as simplest as finding text in an input. grep can be used on a file, or you can pipe the output of a command directly to grep

    grep "cats" favoriteanimals.md to use grep
    to find the word “cats” inside of a file
    echo 'foo\nbar' | grep 'foo' how pipe to grep

14. The find command is a powerful program that allows you to find files and folders based on patterns.

    find [PATTERN1] [PATTERN2] [PATTERN3...]
find navigation-* to find files that begin with the word “navigation”
    find . -name "project*" If you would like to recursively find all files starting with “project”, scanning from the current (.) directory

15. The ping command is useful to know if a host is up and responding, or if otherwise it is shut down or not responding. It sends ICMP packets over the network, and the packets come back, so the roundtrip can be measured (e.g. you see this in the command output: time=39.252 ms
    ). You can ping either a hostname or an IP:

16. A shell script, often referred to as a program, is a file containing a list of commands that execute sequentially, or in the order they are written.Programming languages that execute their commands sequentially are referred to as procedural programming languages

#!/usr/bin/env bash
echo "Welcome."
echo 'This is being echoed from a script, not the command line.'

17. Permission
    chmod +x myscript1.sh The chmod
    command allows you to change the permissions of files and folders. The +x
    flag indicates that you “give this file executable permission”(you can remember x, for executable).

18. As an example, say you want to easily create a Markdown file that contains a summary of all the text files inside your current directory.

$ echo $'SUMMARY:\n===' >> summary.md
$ cat *.txt >> summary.md
$ echo $'\n---\n' >> summary.md
$ echo "Word Count: `cat summary.md | wc -w`"   >> summary.md

Or, you could create a shell script, giving you the ability to repeatedly execute the above commands, with just a single command-line statement.

#!/usr/bin/env bash
echo $'SUMMARY:\n===' >> summary.md
cat *.txt >> summary.md
echo $'\n---\n' >> summary.md
echo "Word Count: `cat summary.md | wc -w`" >> summary.md

19. Positional parameters are variables representing the relative positions of the options that are passed to your shell script.

$ ./myscript somefile.txt fire water wind earth

somefile.txt is positional parameter $1
fire is positional parameter $2
water is positional parameter $3
wind is positional parameter $4
earth is positional parameter $5
and so on…

#!/usr/bin/env bash
echo My name is $1, I am $2, and I love $3.

$ ./myscript Marceline 25 pizza,
-> My name is Marceline, I am 25, and I love pizza.

A final note: The value at the positional parameter $0 is always the name of the script or command being executed.

20. Unix Filesystem Layout

    The filesystem always begins at the root directory /.

• /bin: The /bin, short for “binaries,” directory contains the majority of binary executable programs and utilities that live on the system. Some of these utilities, such as ls, echo, and pwd should now be familiar to you.

• /dev: Short for “devices,” the /dev
  directory contains system-generated links to peripheral devices and their ilk. In practice, you should not add or remove files in this directory, but you may have to access it from time to time when mounting(CD-ROM) new filesystems, accessing peripherals, or performing sophisticated administration tasks.

• /etc: Listed using the common abbreviation for “et cetera,” the /etc directory typically contains a wide-variety of important system configuration files. You will likely look through the content of this directory regularly while performing system administration tasks.

• /home: The /home directory contains the home folders for all users on the system. Because of the potentially massive size of this directory (due to users placing music, movies, images, or other multimedia in their home folders), many system administrators elect to place the /home
  directory on a separate disk partition.

• /lib: The /lib directory contains system-wide shared libraries, often needed by a variety of programs on the system.

• /usr: Short for “user,” the /usr
  directory contains less system-critical binaries, libraries, and other files, often placed in subdirectories such as /usr/bin and /usr/lib
  , that resemble the similarly-named directories living higher up in the hierarchy.

• /usr/local: The special “local” directory inside of /usr contains files that do not come with the operating system distribution. Typically, these files have been installed on the system after the operating system has been installed. For instance, if you create custom scripts or other custom executables that you want available system-wide, you should place them in the /usr/local/bindirectory, ensuring that /usr/local/bin exists in the $PATH

• /var: The /var directory, short for “variable,” typically contains files that change often, such as log files and temporary files.

21. File Permissions

    $ ls -l
     -rw-r--r--   1 username  staff   787B Dec  2 22:20 README.md
    -rw-r--r--   1 username  staff   1.1K Dec  2 22:20 LICENSE
    -rw-r--r--   1 username  staff   271B Dec  2 22:20 bower.json
    -rwxr-xr-x   1 username  staff   363B Dec  2 22:20 build.sh
    drwxr-xr-x   7 username  staff   238B Dec  2 22:20 src
    

The first ten characters of each line in the output above indicate the permissions for the corresponding file. The first character in this ten-character set describes the *file type,* such as whether the file is a directory (`d`), character special file (`c`), or regular file (`-`).

The last nine characters of the set are more important; they comprise three permission triads ascribing permissions for the file’s *owner,* *group,* and everyone else, respectively. The labels displayed immediately after the permission triads (“username” and “staff” in this example) indicate the *owner* and *group* for each file.

The system will display an r if reading is permitted; - if not.
The system will display a w if writing is permitted; - if not.
The system will display a x if file execution is permitted; - if not.

22. Changing File Permissions `$ chmod [ugoa...][+-=][rwx...]`

In the first operator group, the combination of letters `ugoa` represents the user(s) for whom you wish to change access:

- `u`: Represents the user who owns the file
- `g`: Represents users in the group to which the file belongs
- `o`: Represents users not in the file’s group
- `a`: Represents all other (non-owner) users.  

In the second group, the operators `+-=` instruct `chmod` how to change the permissions of the file:
  * `+`: Causes the permissions specified to be added
 * `-`: Causes the permissions specified to be removed
 * `=`: Forces the permissions specified to be the *only* permissions on the file

1. Add read, write, and execute permission for all non-owner users:
  `chmod a+rwx myfile`
2. Remove write permission for the group and non-owner users:
  `chmod w-ga myfile`
3. Add read and write permission to the group and others:
  `chmod go+rw myfile`
4. Give everyone read, write, and execute permission:
  `chmod ugoa=rwx myfile`

23. Changing Directory Permissions
  Permissions work slightly different for directories than they do for files. On directories, “read” permissions allow you to list files in the directory, “write” permissions allow you to add new files to the directory, and “execute” permissions allow you to access files in the directory.

24. Changing File Ownership

Remember: In order to change any file   permissions using chmod, you must also have permissions to do so. If you do not own the file in question or you are not the superuser, you cannot change its permissions. If you need access to change a file’s permissions, but do not own the file in question, you can take a few courses of action:

  1. Get superuser access.
  2. Ask the file’s owner to change the file’s permissions for you.
  3. Ask the file’s owner to give you ownership of the file using the `chown` command, short for “change ownership,” allows you to change both the owner and group for a file or collection of files.The syntax for the command uses the following format, where “USERNAME” is the name of the user you wish to make the file’s owner, and “GROUPNAME” is the name of the group to which you wish to give the file membership:

`chown USERNAME:GROUPNAME myfile`
`chown oscarwilde:wheel dorian_gray.txt` For example, to give the user “oscarwilde” and the group “wheel” ownership of the file named “dorian_gray.txt”.

The `chown` command accepts an optional -R
flag parameter, which instructs the command to recursively change ownership for all files in all subdirectories (and subdirectories of subdirectories) in a given directory. For instance, the following command instructs a change in ownership of all files inside a parent directory named “Pictures,” including those that may exist in subdirectories:
  `chown -R picasso:wheel Pictures/`

25. Configuring Bash

  The `.bash_profile` and `.bashrc` files are simply shell scripts that Bash executes on startup. 

  Bash looks for settings and initialization procedures in `.bash_profile` *only* when the system launches Bash as a *login shell*.If the system has been configured to use Bash as the login shell, Bash will look in .bash_profile
for settings needed for environment configuration *before* showing the initial command prompt. As such, you should put settings in `.bash_profile` *only* when you want them executed immediately after login and at no other time. Otherwise, you should put most settings in `.bashrc`

  A common practice–and one that is recommended–for proper Bash configuration involves putting login-only settings in `.bash_profile`; then, using Bash’s source
command to load additional settings from `.bashrc`.

Login environment settings go here ...

Next, load settings from .bashrc if it exists

if [ -f ~/.bashrc ]; then
source ~/.bashrc
fi

In this example, we use Bash scripting to test if the file `.bashrc` exists in the `$HOME` directory, and if it does, we use the source command to execute the commands listed in the file.

 Most web developers use `.bashrc`
as an opportunity to customize commonly used environment variables–such as `$PATH` or `$PS1`–as well as to set up command *aliases*, used as shorthand for commonly used commands. You should place all of the configuration and initialization settings that you want when Bash launches, as either a login shell or a non-login shell, inside `.bashrc`

26. Customizing Your Path
  Notice that each component of the $PATH
is separated by a colon (:). Generally speaking, you should *not* override the default $PATH
provided to you by your system. If you must alter the $PATH, append additional directories as needed. For instance, to add a directory called bin
located in your $HOME directory to your $PATH
, append ~/bin to the contents of the existing variable by placing the following in your .bashrc

`PATH=$PATH:~/bin`

You can also update the $PATH
directly on the command line, though the changes will be lost after you close Bash. You should use the `export` command when doing so, as this will allow $PATH changes to propagate to any sub-processes you launch from the command line:

`export PATH=$PATH:~/bin`

If you have specific executables you wish to give higher priority to in the `$PATH`, you must either:

  1. Place those executables in directories with the desired priority level.
  2. Alter the `$PATH` to suit your priority preferences.

  Editing and Saving Bash Configuration Files
  Whenever you make changes to Bash configuration files, for the changes to take effect, you will need to close and restart Bash.         

  Alternatively, you may use the `source` command to load changes from the command line, such as: `source .bashrc`

27. Setting Aliases

  Though you can use `alias` directly on the command line, a more common practice involves setting up aliases in `.bashrc`, allowing you to re-use the same aliases each time you start Bash.

  `alias SHORTHAND=“COMMAND”`
  
  `alias ll=“ls -l”` to create an alias called ll
that lists the contents of the present working directory in long format, place the following in `.bashrc`

 A common set of Git aliases in .bashrc
might look something like the following:

alias gst=“git status”
alias gd=“git diff”
alias gc=“git commit”
alias gca=“git commit -a”
alias gco=“git checkout”
alias glog=“git log --graph --abbrev-commit -- date=relative”

28. Customizing the Prompt

  Bash uses environment variables to store the settings and patterns it uses for various prompts. In total, Bash has four prompts, labeled PS1, PS2
, PS3, and PS4, each serving a unique purpose. You will likely only encounter PS1 and PS2
on any regular basis. The prompt named PS1
is the primary prompt displayed before any single-line command; PS2 is the secondary prompt displayed during multi-line commands.

The PS1 variable supports a long list of custom codes that you can use to display a variety of information as part of your prompt. An abbreviated list of codes follows:

![PS1 variable custom codes](https://img.haomeiwen.com/i3576786/c2f9a1f06f86bddf.png?imageMogr2/auto-orient/strip%7CimageView2/2/w/1240)

To change the format of any of these prompts, edit the contents of their corresponding environment variable in `.bashrc`. For example, you might edit PS1 such that it includes the current time along with the present working directory: 

 `PS1="[\$(date +%H:%M)][\w]$ "`

`[15:52][~/my_project]$`

29. Redirection 

  1. By using `>`, `>>`
  2. You can also redirect input *into* a command, allowing you to use a file as input, instead of the command line itself.
    When you first start Bash, it always opens three default files: *stdin*, which represents the keyboard input; *stdout*, which represents the screen; and *stderr*, a special output location for errors. When Bash opens a file, it represents that file with a *file descriptor*, a kind of internal pointer to the file that has been assigned by the operating system. Bash represents each of the three file descriptors with a corresponding number: 0 for stdin, 1 for stdout, and 2 for stderr.
    By default, whenever using `>`, Bash manipulates file descriptor 1, which originally points to stdout. 

    ```
    $ echo “I will never order asparagus.” > worstfood.txt

    <=> $ echo “I will never order asparagus.” 1> worstfood.txt
    ```

  However, if you want all the *error* output of a particular command to go to a file, you could use `2>` instead, redirecting stderr:
  
  `ping foo 2> ping_errors.txt` In this example, since the `ping foo` command will fail, Bash redirects the error message that would have gone into stderr into “ping_errors.txt.” If you instead issued the following:

  `ping foo 1> ping_errors.txt` You would not see any content in “ping_errors.txt” because stderr has not been redirected, and ping foo
will not generate output to stdout.

  With input redirection, we redirect the input of a command by manipulating the file descriptor pointing to stdin. To perform input redirection, use the `<` operator. `$ COMMAND < INPUT_FILE`

  We can redirect this file as input to the `sort` command, which will alphabetize the content and send the results to stdout:

  ```
  $ sort < fruit.txt
  apple
  banana
  cantaloupe
  orange
  pineapple
  ```
  
  The default file descriptor for `<` is 0, which points to stdin. Thus, the above example is the same as specifying 0 explicitly:

  `$ sort 0< fruit.txt`

  You can couple input redirection with output redirection, which helps to illustrate the powerful flexibility of this system. For instance, to save the sorted output of “fruit.txt” to another file named “sorted_fruit.txt,” use both `>` and `<`

  `$ sort < fruit.txt > sorted_fruit.txt`


30. Piping

   A common use of command piping involves directing the potentially lengthy output of one command into another command that supports searching or filtering.   

  `ps aux | grep java`

  In the above example, we use the ps
command, coupled with `grep`, to find all currently running Java programs. The `ps aux` command lists all currently running processes on the system; the `grep java ` command filters this potentially lengthy list to only those that contain the word “java”.

31. Variables in Bash

  `myvariable=somevalue`
  `echo $myvariable`

  **Note:** Whitespace–spaces, tabs, and new lines–has special significance in Bash. 
  
  ```
    # This is incorrect
    myvariable = somevalue
  ```

32. Special Variables and Environment Variables
  
- BASH_VERSION: A string containing the version of your Bash installation.
- PWD: Your present working directory.
- PS1: The formatting string for your primary prompt.
- HOME: The path to your home directory.
- PATH: A colon-delimited list of directories containing executable programs.
- HOSTNAME: The hostname of your computer.
- #: The number of parameters passed into your script.

33. To view the entire list of environment variables currently set in your environment, use the `printenv` command:

  Bash typically names environment variables using ALL CAPS, with whitespace separating words replaced with an underscore (_). As a best practice, you should follow suit, using lowercase letters for your own custom variables. You do not want to accidentally override the Bash-provided environment variables, including other special variables such as the numbered *positional parameters*.

34. Bash Exit Codes

  The standard convention for exit codes dictates that all codes be an integer from 0 to 255, with 0 indicating successful completion and any other number indicating an error.

  Bash provides a special environment variable `$?` that you can use to check the last-known exit code. For example, the `wc`
command will return an exit code of 1 when trying to open a non-existent file:

$ wc non_existent_file
wc: non_existent_file: open: No such file or directory
$ echo $?
1

35. Interactive Input with Read

    The `read` command will continue reading input from the command line until the user presses the [ENTER] key.

    For example, you might want to write a script that asks the user to input his or her full name, then prints a welcome message:

    ```
    #!/usr/bin/env bash
    echo -n "Enter your full name, then press [ENTER]:"
    read full_name
    echo "Welcome, $full_name!"
    ```

36. The following list is not exhaustive; see the manpage for read
 for a complete list
    * `-n nchars`: Forces the `read` command to read up to the number of characters specified instead of waiting for the [ENTER] key to proceed.
    *`-s`: Uses silent mode, which prevents read
 from echoing the user’s input to the command line.
    *`-t` timeout: Sets a timeout, in seconds, that will force `read` to fail if the user does not provide input within the set amount of time.

37. If Statements

    ```
    if true
    then echo “TRUE”
    fi
    ```

    Note that the keyword `true` is a Bash command, as is false. The true
command returns an exit code of 0; false
 returns an exit code of 1. You can try this out for yourself on the command line to see that this is indeed the case, using the `$?` environment variable to see the exit codes.

    The if keyword also supports else statements, allowing you to execute alternative commands when the conditional commands *do not* return 0:
    
    ```
    if true
    then echo “TRUE”
    else echo “FALSE”
    fi
    ```

38. Testing

    The test command, `[`, and its more powerful cousin, `[[`, allows you to test logical conditions, returning an exit code of 0 if the test passes and an exit code of 1 if the test fails. The format of the test command follows: `[ SOME_CONDITION ]`

$ [ 1 = 1 ]
$ echo $?
0
$ [ 1 = 2 ]
$ echo $?
1

Create some variables

num_apples=5
num_oranges=8

Compare the variable values

if [ $num_apples > $num_oranges ]
then echo “We have have more apples than oranges.”
else echo “We do not have enough oranges.”
fi

The `[[` command works similarly to `[` but adds support for pattern matching. With pattern matching support, `[[` allows you to test conditions that would be difficult or impossible to express otherwise. For instance, if you had a variable containing a filename, you might use pattern matching to test the file extension, using the wildcard to represent all characters up to, but not including, the desired extension:

if [[ $myfile = *.jpg ]]
then echo “Found a JPEG”
fi

**Note**: Keep in mind that `[[` only works in Bash. When writing shell scripts that should work across Bash and the standard Bourne shell (sh
), you cannot use `[[
for tests.

  


39. 



  

#####Jan 10th  Intro to Git

git checkout -- test.txt
git rm test.txt
git log