Linux

1 Linux Basics

Linux is a family of open-source Unix-like operating systems based on the Linux kernel. It is widely used for servers, desktops, and embedded systems. Linux provides a powerful command-line interface (CLI) that allows users to interact with the system and perform various tasks.
Linux commands are executed in a terminal or shell, and they can be combined to create powerful scripts and automate tasks. The Linux file system is organized in a hierarchical structure, with directories and files that can be manipulated using various commands.
In bioinformatics, Linux is commonly used for data analysis, bioinformatics pipelines, and high-performance computing (HPC) tasks. Many bioinformatics tools and software are designed to run on Linux systems, making it a preferred choice for researchers in the field.

1.1 File & directory

### Working with files
cp file1.txt file2.txt  # Copy file1.txt to file2.txt
mv file1.txt file2.txt  # Move or rename file1.txt to file2.txt
rm file1.txt            # Remove file1.txt
gzip file1.txt          # Compress file1.txt to file1.txt.gz
gunzip file1.txt.gz     # Decompress file1.txt.gz to file1.txt

### Working with directories
mkdir my_directory        # Create a new directory named my_directory
cd my_directory           # Change into the directory my_directory
ls                        # List files and directories in the current directory
ls -l                     # List files and directories with detailed information
ls -a                     # List all files, including hidden files
ls -lh                    # List files with human-readable sizes
rm -r my_directory        # Remove the directory my_directory and its contents

### View file contents
cat file.txt          # Display the entire contents of file.txt
head file.txt         # Display the first 10 lines of file.txt
tail file.txt         # Display the last 10 lines of file.txt
wc -l file.txt        # Count the number of lines in file.txt

less file.txt         # View file.txt with scrolling and searching capabilities

## Options for less:
# -n: Show line numbers
# +F: Follow the file (like tail -f)
# -S: Don't wrap long lines
# -U: Don't show control characters

grep "\*" file.txt | less -S  # Search for lines containing "*" in file.txt and view them with less, without line wrapping

1.2 File transfer

Transfering data efficiently between local and remote systems:
- wget: Download files directly from the internet to your local machine or server.
- scp: Securely copy files between local and remote systems over SSH.
- rsync: Synchronize files and directories efficiently, with options for selective copying and deletion.

## Basic command
rsync -av /source/folder/ /destination/folder/

## Sync source to destination
rsync -av --delete /source/folder/ /destination/folder/

## If you need two-way sync, run both directions
rsync -av --delete /folder1/ /folder2/
rsync -av --delete /folder2/ /folder1/

## sync multiple folders at once
rsync -avP /source/{folder1,folder2,folder3} /destination/
rsync -avP --include='v*/' --include='Scripts/' --exclude='*' /groups/fcoi/guorui/analysis/ /path/to/destination/

## Network sync
rsync -av --delete /local/folder/ user@remote:/remote/folder/

## Copy only new files (files that don't exist in the destination)
rsync -av --ignore-existing /source/folder/ /destination/folder/

## Copy only if source is newer
rsync -av --update /source/folder/ /destination/folder/

Note

--archive: Preserves permissions, timestamps, symbolic links, etc.
--verbose: Shows progress
--delete: Removes files in destination that don’t exist in source
--dry-run: Preview changes without actually syncing
--update: Only copy newer file
--ignore-existing: Skips files that already exist in destination (regardless of age)
--update: Only copies if source file is newer than destination
--no-perms: Don’t preserve Unix permissions
--no-times: Don’t preserve modification times
--inplace: Write directly without temp files

Sync remote server folders to local using rsync

## Remote server details
remote_user="username"
remote_host="hpcio2" # hpc2021-io1.hku.hk,hpc2021-io2.hku.hk

## Define folders to sync (absolute paths)
## Format: "local_absolute_path:remote_absolute_path"
folders_to_sync=(
    # "/mnt/m/Reference:/lustre1/g/path_my/Reference"
    # "/mnt/m/WES/DFSP/Annovar:/lustre1/g/path_my/pipeline/somatic_variants_calling/data/DFSP/Annovar"
    "/mnt/m/WES/SARC/BAM:/lustre1/g/path_my/pipeline/somatic_variants_calling/data/SARC/BAM"
    # Add more folders as needed in the same format
)

## Sync each folder
for folder_pair in "${folders_to_sync[@]}"; do
    # Split the pair into local and remote paths
    local_path="${folder_pair%%:*}"
    remote_path="${folder_pair#*:}"

    echo "Syncing ${remote_user}@${remote_host}:${remote_path} to ${local_path}"

    # Create local directory structure
    mkdir -p "${local_path}"

    # Sync the folder contents from remote to local (trailing slash on source copies contents)
    rsync -av --update --progress --inplace --no-perms --no-times \
        "${remote_user}@${remote_host}:${remote_path}/" \
        "${local_path}/"

    echo "-----------------------------------"
done

Sync local folders to remote server

## Remote server details
remote_user="username"
remote_host="hpcio1" # hpc2021-io1.hku.hk

## Define folders to sync (absolute paths)
## Format: "local_absolute_path:remote_absolute_path"
folders_to_sync=(
    "/mnt/f/Reference:/lustre1/g/path_my/Reference"
    # Add more folders as needed in the same format
)

## Sync each folder
for folder_pair in "${folders_to_sync[@]}"; do

    # Split the pair into local and remote paths
    local_path="${folder_pair%%:*}"
    remote_path="${folder_pair#*:}"

    echo "Syncing ${local_path} to ${remote_user}@${remote_host}:${remote_path}"

    # Create remote directory structure
    ssh ${remote_user}@${remote_host} "mkdir -p ${remote_path}"

    # Sync the folder contents (trailing slash on source copies contents)
    rsync -av --update --progress \
        "${local_path}/" \
        "${remote_user}@${remote_host}:${remote_path}/"

    echo "-----------------------------------"
done

1.3 File permission

Only the user can copy, execute, and change

chmod 700 /path/to/folder

700 means:

Owner: 7 (read = 4, write = 2, execute = 1; 4+2+1 = 7)
Group: 0 (no permissions)
Others: 0 (no permissions)

All can read and execute, only the owner can change

chmod 755 /path/to/folder

Owner: 7 (read = 4, write = 2, execute = 1; 4+2+1 = 7)
Group: 5 (read = 4, execute = 1; 4+1 = 5)
Others: 5 (read = 4, execute = 1; 4+1 = 5)

Give a user named someuser read, write, and execute permissions on the /path/to/folder directory

setfacl -m u:someuser:rwx /path/to/folder

Verify the Permissions

getfacl /path/to/folder

Remove all ACL entries for a user

setfacl -x u:someuser /path/to/folder

Change owner

sudo chown username:groupname /path/to/folder

1.4 File compression

## Backup a directory to a tar.gz file
tar -czvf backup.tar.gz /path/to/directory
## Restore a directory from a tar.gz file
tar -xzvf backup.tar.gz -C /path/to/restore

## Common tar flags:
# -c: create a new archive
# -x: extract files from an archive
# -z: filter the archive through gzip
# -v: verbose output (list files being processed)

Commone tar flags:
- -x = extract
- -f = file
- -v = verbose
- -z = gzip compression
- -j = bzip2 compression
- -J = xz compression
- -c = change to directory
- -C = specify directory
- -p = preserve permissions

## Extract .tar.gz file
tar -xf fuke.tar
tar -xzvf file.tar.gz
tar -xzf file.tgz

## Extract .tar.bz2 file
tar -xjf file.tar.bz2

## Extract .tar.xz file
tar -xJf file.tar.xz

## Extract to specific directory
tar -xzf file.tar.gz -C /path/to/directory

## List contents without extracting
tar -tzf file.tar.gz
tar -tf file.tar

## Extract zip files
unzip file.zip
unzip file.zip -d /path/to/directory

## List contents without extracting
unzip -l file.zip

## Extract .gz file
gunzip file.gz
gunzip -dk file.gz  # keep original file

## Extract .bz2 file
bunzip2 file.bz2

## Extract .xz file
unxz file.xz

## Extract .7z file
7z x file.7z

## Extract .rar file
unrar x file.rar

Common tar flags for compression
- -c = create archive
- -f = file name
- -z = gzip compression (.tar.gz)
- -j = bzip2 compression (.tar.bz2)
- -J = xz compression (.tar.xz)
- -v = verbose output

tar -cf archive.tar file1 file2 directory/

## gzip compressed
tar -czf archive.tar.gz file1 file2 directory/

## bzip2 compressed
tar -cjf archive.tar.bz2 file1 file2 directory/

## .tar.xz compressed
tar -cJf archive.tar.xz file1 file2 directory/

## Backup and Restore a Folder
tar -czvf mybackup.tar.gz myfolder
tar -xzvf mybackup.tar.gz

1.5 Regular expression

Regular expressions (regex) are patterns used to match character combinations in strings. In Linux, they are commonly used with commands like grep, sed, and awk for searching and manipulating text.

Symbol	Descriptions	Examples	Matched pattern
.	match any characters	A..T	ATGT, AGCT
^	match start of string	^A	ATG, ACC, AGTC
$	match end of string	T$	TCT, GAT, ACGT
*	match up zero or more times the preceding character	AT*	ATC, CATT, TGAC
\	escape a special (or meta) character	\^A	^A, ^ATG
()	group regular expressions	(ATC){3}	ATCATCATC

[]	match any one of a set characters	[AT]	ACG, CTG, ACTCG
[A-Z]	match upper case letter from A to Z	[A-Z]	>GL000226.1
[a-z]	math lower case letter form a to z	[a-z]	/home/user100
[0-9]	match digit from 0-9	[0-9]	>GL000226.1
[^]	match any character except those in the square bracket after carat ^	[^T]	GCCGAG, >GL000226.1

*	match up zero or more times the preceding character	AT*CG	ACG, ATCG, ATTCG
+	match up one or more times the preceding character	AT+CG	ATCG, ATTCG, ATTTCG
?	match up zero or one times the preceding character	AT?CG	ATCG,ACG
{N}	preceding character is matched exactly N times	AT{3}CG	ATTTCG
{N,}	preceding character is matched at least N times or more	AT{3,}CG	ATTTCG, ATTTTCG
{N,M}	preceding character is matched at least N times but not more than M times	AT{2,3}CG	ATTCG, ATTTCG

1.6 Text file manipulation

1.6.1 Filtering

grep Globally search a Regular Expression and Print: is a command-line utility for searching plain-text data sets for lines that match a specified pattern.

grep "pattern" file.txt          # Search for "pattern" in file.txt
grep -i "pattern" file.txt       # Case-insensitive search
grep -n "pattern" file.txt       # Show line numbers
grep -v "pattern" file.txt       # Invert the sense of matching, to select non-matching lines

cut is used to extract sections from each line of input: - Extract columns or fields from text files - Specify delimiter for fields - Useful for processing structured data like CSV or TSV files - Works line by line - Column extraction based on delimiter or character position

cut -f1 file.txt             # Extract first column from a tab-delimited file
cut -d',' -f1,3 file.csv     # Extract first and third columns from a comma-separated file
cut -c1-5 file.txt           # Extract characters from position 1 to 5

# Piping and combining commands
grep "pattern" file.txt | cut -f1,3 | sort | uniq

1.6.2 Sort

sort file.txt                 # Sort lines in file.txt alphabetically
sort -n file.txt              # Sort lines numerically
sort -r file.txt              # Sort lines in reverse order
sort -u file.txt              # Sort and remove duplicate lines
sort -k2 file.txt             # Sort by the second column

1.6.3 Duplicates

uniq file.txt                 # Remove duplicate lines (requires sorted input)
uniq -c file.txt              # Count occurrences of each line
sort file.txt | uniq -c | sort -nr  # Count and sort by frequency

1.6.4 Merge

paste file1.txt file2.txt > combined.txt  # Join files side by side

1.6.5 Find & replace

sed is a stream editor used to perform basic text transformations: * substituction, deletion, insertion, and more * works line by line, applying patterns and edits

## Remove all spaces from a string
echo "a b c" | sed 's/ //g'
#> "abc"
## Remove all occurences of "foo" with "bar"
echo "foo baz foo" | sed 's/foo/bar/g'
#> "bar baz bar"

## Delete lines containing a "pattern" pattern from file.txt
sed '/pattern/d' file.txt

## Print only lines matching a "pattern" pattern from file.txt
sed -n '/pattern/p' file.txt

## Print lines 2 to 5 of a file
sed -n '2,5p' file.txt

1.7 Data processing

awk used for pattern scanning and processing. * More powerful than cut for data extraction and reporting * Process text as fileds and records, support arithmetic and logic
awk is a powerful text-processing tool and programming language used for manipulating and analyzing data
Search files for lines (or other units of text) that contain certain patterns and perform specified actions on those lines
By default, awk prints every line of data from the specified file
Basic syntax: awk '<pattern> {<action>}' <file>
regular expression awk '/pattern/ {print $1}' file.txt will print the first column of lines that match the pattern
Build in variables:
- $0: entire row of input
- $1, $2, $3, …: n-th column of inut
- NR: the number of the current line (row) being processed.
- NF: the number of fields (columns) in the current line.
- SNF: the last field (column) in the current row.
- FS: Field separator (default is whitespace).
- OFS: Output field separator (default is whitespace).
- ORS: Output record separator (default is newline).
- BEGIN and END: Special blocks that allow you to specify actions to be performed before processing any lines and after processing all lines, respectively.
Operators
- > | >=: greater than or greater than or equal to
- < | <=: less than or less than or equal to
- ==: equal to
- !=: not equal to
- &&: And (reports “true” if both conditions are true)
- ||: Or (reports “true” if either condition, or both, are true)
- !: Not (negates the condition, reports “true” if the condition is false)

awk '/a/{print $1}' file.txt  # Print the first column of lines that contain the letter "a"

## Print the first column of a file
awk '{print $1}' file.txt

## Sum values in the first column
awk '{sum += $1} END {print sum}' file.txt

## Print lines where the third column is greater than 100
awk '$3 > 100' file.txt

## Print the lines that match a pattern "error"
awk '/error/' file.txt

## Print the number of lines in a file
awk 'END {print NR}' file.txt

Delimiters
- -F FS:
  - FS: input field separator (default is whitespace)
- -v OFS:
  - OFS: output field separator (default is whitespace)

awk -F ',' 'NR==1 || $3>10 {print $1, $2, $3}' file.csv  # Print the first three columns of a CSV file where the third column is greater than 10, including the header row

awk -F ',' -v OFS='\t' 'NR==1 || $3>10 {print $1, $2, $3}' file.csv

Syntax with BEGIN and END blocks
- awk 'BEGIN {<action1>} <pattern> {<action2>} END {<action3>}' <file>
- BEGIN {<actioin1>}: the BEGIN block is executed before any input lines are processed. It is often used to initialize variables, set field separators and setting up the environment.
- <pattern> {<action2>}: This is the main processing block for each input line. For each line in the input file, if the line matches the specified pattern, the associated action is executed. If no pattern is specified, the action is applied to all lines.
- END {<action3>}: The END block is executed after all input lines have been processed. It is typically used for final calculations, printing summaries, or performing cleanup tasks.
Arithmetic operations
- +: addition
- -: subtraction
- *: multiplication
- /: division
- ^: exponentiation
- %: remainder
- ++: increment
- --: decrement
- +=: add and assign
- -=: subtract and assign

1.8 Search anything

find is a powerful command to search for files and directories based on various criteria: * Search by name, type, size, modification time, permissions, etc. * Execute actions on found items (delete, move, etc.) * Supports complex expressions with logical operators

## Find all .txt files in current directory and subdirectories
find . -name "*.txt"

## Find files/folders by name
find /path/to/search -name "pattern"

## Find files larger than 100MB
find /path/to/search -type f -size +100M

## Find files in pattern and list details
find ./Clean -name "*Clean.fastq.gz" -type f -exec ls -lh {} \;

1.9 Redirection

Redirection is a way to direct the input and output of a command to a file or another command. It allows you to save the output of a command to a file, or to use the output of one command as the input for another command.

## Redirect output to a file
command > output.txt

## Append output to a file
command >> output.txt

Standard streams in Linux: * stdin (standard input): The default input stream for a command, 0 * stdout (standard output): The default output stream for a command, 1 * stderr (standard error): The default error output stream for a command, 2

2 Vim Text Editor

2.1 Modes

Vim text editor has three main modes:
- Esc, back to normal mode
- i, insert mode
- v, select mode

Command	What it does	Most common follow-up
i	Insert before cursor	typing text
a	Append after cursor	typing text
I	Insert at beginning of line	—
A	Append at end of line	—
o	Open new line below	typing
O	Open new line above	typing
Esc	Back to Normal mode	—
v	Visual mode	move + operator
V	Visual Line mode	—
Ctrl-v	Visual Block mode	—

2.2 Navigation

Command	Movement	Common combos
h j k l	← ↓ ↑ →	—
w / b	forward/backward by word start	ciw, daw
e / ge	forward/backward to word end	—
W / B	forward/backward by big word	(ignores punctuation)
0	Start of line	—
^	First non-blank character	—
$	End of line	—
gg	Top of file	—
G	Bottom of file	50G → line 50
Ctrl-u / Ctrl-d	Half page up/down	—
Ctrl-f / Ctrl-b	Full page down/up	—

2.3 Editing

Command	Meaning	examples
x	Delete character under cursor	—
dd	Delete current line	5dd = delete 5 lines
yy	Yank (copy) current line	p to paste
p / P	Paste after/before cursor	—
u	Undo	(most used command after Esc)
Ctrl-r	Redo	—
.	Repeat last change/command	(extremely powerful)
dw / d$	Delete to end of word / end of line	diw, daw
ciw / caw	Change (delete + insert) inside/around word	most common text object usage
“ci”” / ci( / ci{”	Change inside quotes/parentheses/braces	extremely common in code

2.4 Search & Replace

Command	Meaning
/pattern	Search forward
?pattern	Search backward
n / N	Next / previous match
* / #	Search forward/backward for word under cursor
:%s/old/new/g	Replace all occurrences (whole file)
:%s/old/new/gc	Replace with confirmation

2.5 Saving & Exiting

Command	Meaning
:w	Save (write)
:q	Quit (only if no changes)
:q!	Quit without saving
:wq or :x or ZZ	Save and quit
:w !sudo tee %	Save file needing root (very common workaround)

3 Nano Text Editor

Open a file with: nano filename.txt
Use the arrow keys to navigate.
Save the changes with ctrl + o,
Confirm the filename by pressing Enter.
Exit nano with ctrl + x.

3.1 Searching Text

To find words or phrases within a file:

Initiate search with ctrl + w.
Type the search term and press Enter.
Exit search mode with ctrl + c.

3.2 Customizing Nano

Enhance Nano’s functionality:

nano -miA file.txt

-m: Enable mouse support.
-i: Auto-indent new lines.
-A: Enable syntax highlighting.

4 Job Scheduling

4.1 OpenPBS

OpenPBS (Portable Batch System) is an open-source workload manager and job scheduler used in High-Performance Computing (HPC) environments to automate and optimize the execution of tasks across and clouds.

PBS job script

#!/bin/bash
#PBS -N job_name                  # Job name
#PBS -l nodes=1:ppn=4             # Number of nodes and processors per node
#PBS -l walltime=01:00:00         # Walltime (hh:mm:ss)
#PBS -l mem=8gb                   # Memory per node
#PBS -j oe                        # Combine stdout and stderr
#PBS -M user@example.com          # Email address for notifications
#PBS -m abe                       # Send email on (a) abort, (b) begin, (e) end
#PBS -q batch                     # Queue named
#PBS -V                           # Export all environment variables to the job

## example
cd $PBS_O_WORKDIR                 # Change to the directory where the job was submitted
module load python/3.8.5          # Load necessary modules
python my_script.py               # Command to run your program

PBS environment variables

$PBS_O_WORKDIR     # The directory from which the job was submitted
$PBS_O_HOME        # The home directory of the user who submitted the job
$PBS_JOBDIR        # The working directory for the job on the compute node
$TMPDIR            # The temporary directory for the job on the compute node
$PBS_JOBID         # The unique identifier assigned to the job
$PBS_NODEFILE      # The file containing the list of nodes allocated to the job
$PBS_NUM_NODES     # The number of nodes allocated to the job
$PBS_NUM_PPN       # The number of processors per node allocated to the job
$PBS_QUEUE         # The name of the queue to which the job was submitted
$PBS_JOBNAME       # The name of the job as specified in the job script

Submit job scripts

## Submit a job
qsub job_script.pbs

## Submit a job with environment variables.
qsub -v sample="$sample" -M "$EMAIL" pbs_1_cutadapt.pbs

Check job status

## List finished job details in summary
qstat -x job_id

## List finished job details in full details
qstat -H job_id
qstat -x -f job_id
qstat -x -f job_id | grepl "exec_host"

## List all jobs for user guorui
qstat -u guorui -a

## Find out all compute nodes allocated to a job
for job_id in $(find . -name "log.*.e*" -printf "%f\n" | sed 's/.*\.e\([0-9]*\)$/\1/' | sort -n | uniq); do
    echo -n "Job ID: $job_id: "; qstat -x -f $job_id | grep exec_host
done

## Delete a job
qdel job_id

## check all queues
qstat -Qf | less

for q in cgs small medium large cgs_queue; do
    echo "=== $q ==="
    qstat -Qf $q | grep -E "resources_max.(mem|ncpus|walltime)"
done

## Hold a job
qhold job_id

## Release a held job
qrls job_id

# See your groups
groups

# Check queue ACLs
qstat -Q -f | grep -E "^Queue:|acl_users|acl_groups"

Stat interactive job

## Start and interactive job with 16 processors, 64GB memory, 24 hours walltime in cgs_queue
qsub -I -q cgs_queue -l nodes=hpcf3-c01:ppn=16,walltime=12:00:00,mem=64gb
qsub -I -q cgs_queue -l nodes=hpcf3-c02:ppn=16,walltime=12:00:00,mem=64gb
qsub -I -q cgs_queue -l nodes=hpcf3-c03:ppn=16,walltime=12:00:00,mem=64gb

Check the permission of queue

qstat -Q cgs -f

# Queue: cgs
#     queue_type = Execution
#     total_jobs = 16
#     state_count = Transit:0 Queued:10 Held:0 Waiting:0 Running:6 Exiting:0 Begu
#         n:0
#     acl_user_enable = True
#     acl_users = bioinfo,emiklin,irisxu,itsupport,julia,junyanzhong,manzhai,nicklin,
#         rubyzhou,totochu
#     resources_max.mem = 1024gb
#     resources_max.ncpus = 64
#     resources_max.nodect = 1
#     resources_max.walltime = 1440:00:00
#     resources_default.mem = 1gb
#     resources_default.walltime = 24:00:00
#     default_chunk.Qlist = cgs
#     resources_assigned.mem = 3019898880kb
#     resources_assigned.mpiprocs = 168
#     resources_assigned.ncpus = 253
#     resources_assigned.nodect = 6
#     enabled = True
#     started = True

-Q: Show queue information (not job information)
-cgs: The specific queue name to query
-f: Full/detailed output showing all queue attributes

5 Terminal Multiplexer

5.1 Tmux

Tmux Configuration Consider adding these aliases to your shell configuration for quick tmux access.

# Terminal aliases for tmux
alias t="tmux"
alias ta="t a -t"
alias tls="t ls"
alias tn="t new -s"
alias tk="t kill-session -t"
alias tks="t kill-server"

🔧 Custom Prefix Key

Change the default prefix in .tmux.conf:

set-option -g prefix C-a
bind-key C-a send-prefix

Session Management

Action	Command/Shortcut
List sessions	`tmux list-sessions`
Attach to session	`tmux attach-session -t target-session`
Switch between sessions	`Ctrl + A + s`
Switch to latest session	`Ctrl + A + l`
Detach from session	`Ctrl + A + d`

Window Management

Action	Shortcut
List all windows	`Ctrl + A + w`
Rename current window	`Ctrl + A + ,`
Switch to next window	`Ctrl + A + →`
Switch to previous window	`Ctrl + A + ←`
Create new window	`Ctrl + A + c`
Kill current window	`Ctrl + A + q`

Pane Management

Action	Shortcut
Switch to pane above	`Shift + ↑`
Switch to pane below	`Shift + ↓`
Switch to pane left	`Shift + ←`
Switch to pane right	`Shift + →`
Kill current pane	`Ctrl + A + x`

Copy Mode

Action	Shortcut
Enter copy mode	Drag mouse to select text
Paste copied text	`Ctrl + A + ]`

5.2 Screen

💡 Screen Aliases

Add these aliases to your shell configuration for easier screen management.

# Terminal aliases for screen
alias s="screen"       # start a screen session
alias ss="s -S"        # start a named screen session
alias sr="s -r"        # reattach to a screen session
alias sls="s -ls"      # list current running screen sessions

Basic Commands

Action	Command
Start screen session	`screen`
Start named session	`screen -S session_name`
Reattach to session	`screen -r`
List running sessions	`screen -ls`

Window Management

Action	Shortcut
Create new window	`Ctrl + A + C`
Kill current window	`Ctrl + A + K`
List all windows	`Ctrl + A + W`
Go to window 0-9	`Ctrl + A + 0-9`
Go to next window	`Ctrl + A + N`
Toggle between windows	`Ctrl + A + Ctrl + A`
Rename current window	`Ctrl + A + A`

Region Management

Action	Shortcut
Split horizontally	`Ctrl + A + S`
Split vertically	`Ctrl + A + \\|`
Switch between regions	`Ctrl + A + Tab`
Close all regions but current	`Ctrl + A + Q`
Close current region	`Ctrl + A + X`

Session & Copy Mode

Action	Shortcut
Detach from session	`Ctrl + A + D`
Start copy mode	`Ctrl + A + [`
Paste copied text	`Ctrl + A + ]`
Show help	`Ctrl + A + ?`
Quit screen	`Ctrl + A + Ctrl + \`

Additional Resources

VSCode: Official Keyboard Shortcuts Reference
Vim: Vim Cheat Sheet
Tmux: Tmux Cheat Sheet
Terminal: Bash Reference Manual

6 Manage Environment

6.1 Conda

Installation


# Miniforge linux
wget "https://github.com/conda-forge/miniforge/releases/latest/download/Miniforge3-Linux-x86_64.sh"

# Miniforge Mac
wget "https://github.com/conda-forge/miniforge/releases/latest/download/Miniforge3-MacOSX-arm64.sh"

# Install it
bash Miniforge3-$(uname)-$(uname -m).sh

# Add conda to PATH if found
if [ -f "$HOME/miniconda3/etc/profile.d/conda.sh" ]; then
    . $HOME/miniconda3/etc/profile.d/conda.sh
fi

Configuration


# Init
conda init --all

# Use mamba for faster solving
conda update -n base conda
conda install -n base conda-libmamba-solver

# Config
conda config --set solver libmamba
conda config --set always_yes true
conda config --set auto_activate_base false

# Setup channels
conda config --add channels bioconda
conda config --add channels conda-forge
#conda config --set channel_priority strict

# Install software from a specific channel
conda install -c conda-forge numpy

Create Environment

# Create R environment with specific R version and packages
conda create -n renv r=4.5 r-languageserver r-tidyverse r-irkernel r-httpgd r-downlit r-xml2 r-markdown r-devtools radian

# Install bioconductor packages, all package names are lower case
conda install -n renv bioconductor-deseq2 bioconductor-edger

# Install repository packages
conda install -n renv r-qs r-fs r-tidyverse

# Remove a specific environment
conda remove --name renv --all

# Remove a software in a environment
conda remove --name renv package_name

# Export and save the conda env file with all software information
conda env export -n renv > renv.yml

# Export and save the conda env file with all software information
# without the prefix and bundle information
conda env export -n renv --no-builds > renv.yml
conda env export -n renv --no-builds --file renv.yml

# Create a new environment with a environment file
conda env create --file renv.yml

# Rename an environment
conda create --name new_env_name --clone old_env_name
conda activate new_env_name
conda remove --name old_env_name --all

6.2 Pixi

## Auto-activate pixi when entering project directory
eval "$(pixi completion --shell bash)"  # or zsh

## Create a global pixi environment with specific R version and packages
pixi global install -e renv \
    r-base=4.5 \
    r-languageserver \
    r-tidyverse \
    r-irkernel \
    r-httpgd \
    r-downlit \
    r-xml2 \
    r-markdown \
    r-devtools \
    radian

## Print out  global pixi environments
pixi global list

## Add Bioconda Permanently
pixi config append default-channels bioconda --global
pixi config set default-channels '["conda-forge", "bioconda"]' --global

7 Git Version Control

## keep your original Git repository but provide a clean version to others,
git archive --format=zip --output=my-project-clean.zip main

git init
git add .
git commit -m "first commit"
git remote add origin
git push -u origin main

git config --global user.name "" # set an user name that will be associated with each history marker
git config --global user.email "" # set an email address that will be associated with each history marker
git config --global color.ui auto # set automatic command line coloring for Git for easy reviewing
git init #


# 1. Delete the local branch
git branch -d branch-name     # Use -D to force delete

# 2. Delete the remote branch
git push origin --delete branch-name

# 3. Verify branches are gone
git branch -a                # List all branches

echo "" # create file ""
git status # check
git add # add a file as it looks now to your next commit (stage)
git commit # commit your staged content as a new commit snapshot
### if go to the viam editing mode, i to input, esc to quit, :wq to save and quit
git log # show all commits in the current branch’s history, check versions, q to quit
touch .gitignore
git branch "" # create branch ""
git branch # check branches
git checkout "" # switch to another branch and check it out into your working directory ""
git branch -D "" # delete branch ##
git checkout -b temp # create temporary branch ##
git merage # merge the specified branch’s history into the current one
git clone # retrieve an entire repository from a hosted location via URL
git remote -V # check remove repos information
git push # Transmit local branch commits to the remote repository branch
git fetch # fetch down all the branches from that Git remote
git diff # diff of what is changed but not staged
git pull # fetch and merge any commits from the tracking remote branch
git reset # unstage a file while retaining the changes in working directory
git rm -r --cached . # ignore already added files run

## Go back to a specifi commit
git reset --hard e4098a7

## Pull remote updates and ignore local changes
git fetch origin
git reset --hard origin/your-branch-name

## Change remote URL
git remote -v # check current remote configuration
git remote set-url origin <new-repository-url>
git remote remove origin
git remote add origin <new-repository-url>

## Reset to match remote
git fetch origin
git reset --hard origin/main

8 Windows WSL

System info


## Install the neofetch package
sudo apt install neofetch -y

## Show system info
neofetch --memory_unit gib

Change sudo password

In the windows terminal, run the following command to change the sudo password:


wsl -u root

## Input the new password
passwd <username>

WSL configuration
- WSL global config: C:\Users\<UserName>\.wslconfig
- WSL distro config: \\wsl.localhost\Ubuntu-24.04\etc\wsl.conf

--- title: Linux date: 2022-10-09 published-title: Created date-modified: last-modified title-block-banner: "#212529" subtitle: "Commonly used Linux commands" # toc: true # toc-location: left toc-title: "Contents" execute: eval: false format: html: code-tools: source: true toggle: true code-fold: false toc: true number-sections: true --- ## Linux Basics - Linux is a family of open-source Unix-like operating systems based on the Linux kernel. It is widely used for servers, desktops, and embedded systems. Linux provides a powerful command-line interface (CLI) that allows users to interact with the system and perform various tasks. - Linux commands are executed in a terminal or shell, and they can be combined to create powerful scripts and automate tasks. The Linux file system is organized in a hierarchical structure, with directories and files that can be manipulated using various commands. - In bioinformatics, Linux is commonly used for data analysis, bioinformatics pipelines, and high-performance computing (HPC) tasks. Many bioinformatics tools and software are designed to run on Linux systems, making it a preferred choice for researchers in the field. ### File & directory ``` bash ### Working with files cp file1.txt file2.txt # Copy file1.txt to file2.txt mv file1.txt file2.txt # Move or rename file1.txt to file2.txt rm file1.txt # Remove file1.txt gzip file1.txt # Compress file1.txt to file1.txt.gz gunzip file1.txt.gz # Decompress file1.txt.gz to file1.txt ``` ``` bash ### Working with directories mkdir my_directory # Create a new directory named my_directory cd my_directory # Change into the directory my_directory ls # List files and directories in the current directory ls -l # List files and directories with detailed information ls -a # List all files, including hidden files ls -lh # List files with human-readable sizes rm -r my_directory # Remove the directory my_directory and its contents ``` ``` bash ### View file contents cat file.txt # Display the entire contents of file.txt head file.txt # Display the first 10 lines of file.txt tail file.txt # Display the last 10 lines of file.txt wc -l file.txt # Count the number of lines in file.txt ``` ``` bash less file.txt # View file.txt with scrolling and searching capabilities ## Options for less: # -n: Show line numbers # +F: Follow the file (like tail -f) # -S: Don't wrap long lines # -U: Don't show control characters grep "\*" file.txt | less -S # Search for lines containing "*" in file.txt and view them with less, without line wrapping ``` ### File transfer - Transfering data efficiently between local and remote systems: - `wget`: Download files directly from the internet to your local machine or server. - `scp`: Securely copy files between local and remote systems over SSH. - `rsync`: Synchronize files and directories efficiently, with options for selective copying and deletion. ``` bash ## Basic command rsync -av /source/folder/ /destination/folder/ ## Sync source to destination rsync -av --delete /source/folder/ /destination/folder/ ## If you need two-way sync, run both directions rsync -av --delete /folder1/ /folder2/ rsync -av --delete /folder2/ /folder1/ ## sync multiple folders at once rsync -avP /source/{folder1,folder2,folder3} /destination/ rsync -avP --include='v*/' --include='Scripts/' --exclude='*' /groups/fcoi/guorui/analysis/ /path/to/destination/ ## Network sync rsync -av --delete /local/folder/ user@remote:/remote/folder/ ## Copy only new files (files that don't exist in the destination) rsync -av --ignore-existing /source/folder/ /destination/folder/ ## Copy only if source is newer rsync -av --update /source/folder/ /destination/folder/ ``` ::: callout-note - `--archive`: Preserves permissions, timestamps, symbolic links, etc. - `--verbose`: Shows progress - `--delete`: Removes files in destination that don't exist in source - `--dry-run`: Preview changes without actually syncing - `--update`: Only copy newer file - `--ignore-existing`: Skips files that already exist in destination (regardless of age) - `--update`: Only copies if source file is newer than destination - `--no-perms`: Don't preserve Unix permissions - `--no-times`: Don't preserve modification times - `--inplace`: Write directly without temp files ::: - Sync remote server folders to local using rsync ``` bash ## Remote server details remote_user="username" remote_host="hpcio2" # hpc2021-io1.hku.hk,hpc2021-io2.hku.hk ## Define folders to sync (absolute paths) ## Format: "local_absolute_path:remote_absolute_path" folders_to_sync=( # "/mnt/m/Reference:/lustre1/g/path_my/Reference" # "/mnt/m/WES/DFSP/Annovar:/lustre1/g/path_my/pipeline/somatic_variants_calling/data/DFSP/Annovar" "/mnt/m/WES/SARC/BAM:/lustre1/g/path_my/pipeline/somatic_variants_calling/data/SARC/BAM" # Add more folders as needed in the same format ) ## Sync each folder for folder_pair in "${folders_to_sync[@]}"; do # Split the pair into local and remote paths local_path="${folder_pair%%:*}" remote_path="${folder_pair#*:}" echo "Syncing ${remote_user}@${remote_host}:${remote_path} to ${local_path}" # Create local directory structure mkdir -p "${local_path}" # Sync the folder contents from remote to local (trailing slash on source copies contents) rsync -av --update --progress --inplace --no-perms --no-times \ "${remote_user}@${remote_host}:${remote_path}/" \ "${local_path}/" echo "-----------------------------------" done ``` - Sync local folders to remote server ``` bash ## Remote server details remote_user="username" remote_host="hpcio1" # hpc2021-io1.hku.hk ## Define folders to sync (absolute paths) ## Format: "local_absolute_path:remote_absolute_path" folders_to_sync=( "/mnt/f/Reference:/lustre1/g/path_my/Reference" # Add more folders as needed in the same format ) ## Sync each folder for folder_pair in "${folders_to_sync[@]}"; do # Split the pair into local and remote paths local_path="${folder_pair%%:*}" remote_path="${folder_pair#*:}" echo "Syncing ${local_path} to ${remote_user}@${remote_host}:${remote_path}" # Create remote directory structure ssh ${remote_user}@${remote_host} "mkdir -p ${remote_path}" # Sync the folder contents (trailing slash on source copies contents) rsync -av --update --progress \ "${local_path}/" \ "${remote_user}@${remote_host}:${remote_path}/" echo "-----------------------------------" done ``` ### File permission Only the user can copy, execute, and change ``` bash chmod 700 /path/to/folder ``` `700` means: - Owner: 7 (read = 4, write = 2, execute = 1; 4+2+1 = 7) - Group: 0 (no permissions) - Others: 0 (no permissions) All can read and execute, only the owner can change ``` bash chmod 755 /path/to/folder ``` - Owner: 7 (read = 4, write = 2, execute = 1; 4+2+1 = 7) - Group: 5 (read = 4, execute = 1; 4+1 = 5) - Others: 5 (read = 4, execute = 1; 4+1 = 5) Give a user named someuser read, write, and execute permissions on the /path/to/folder directory ``` bash setfacl -m u:someuser:rwx /path/to/folder ``` Verify the Permissions ``` bash getfacl /path/to/folder ``` Remove all ACL entries for a user ``` bash setfacl -x u:someuser /path/to/folder ``` Change owner ``` bash sudo chown username:groupname /path/to/folder ``` ### File compression ``` bash ## Backup a directory to a tar.gz file tar -czvf backup.tar.gz /path/to/directory ## Restore a directory from a tar.gz file tar -xzvf backup.tar.gz -C /path/to/restore ## Common tar flags: # -c: create a new archive # -x: extract files from an archive # -z: filter the archive through gzip # -v: verbose output (list files being processed) ``` - Commone tar flags: - `-x` = extract - `-f` = file - `-v` = verbose - `-z` = gzip compression - `-j` = bzip2 compression - `-J` = xz compression - `-c` = change to directory - `-C` = specify directory - `-p` = preserve permissions ``` bash ## Extract .tar.gz file tar -xf fuke.tar tar -xzvf file.tar.gz tar -xzf file.tgz ## Extract .tar.bz2 file tar -xjf file.tar.bz2 ## Extract .tar.xz file tar -xJf file.tar.xz ## Extract to specific directory tar -xzf file.tar.gz -C /path/to/directory ## List contents without extracting tar -tzf file.tar.gz tar -tf file.tar ``` ``` bash ## Extract zip files unzip file.zip unzip file.zip -d /path/to/directory ## List contents without extracting unzip -l file.zip ## Extract .gz file gunzip file.gz gunzip -dk file.gz # keep original file ## Extract .bz2 file bunzip2 file.bz2 ## Extract .xz file unxz file.xz ## Extract .7z file 7z x file.7z ## Extract .rar file unrar x file.rar ``` - Common tar flags for compression - `-c` = create archive - `-f` = file name - `-z` = gzip compression (.tar.gz) - `-j` = bzip2 compression (.tar.bz2) - `-J` = xz compression (.tar.xz) - `-v` = verbose output ``` bash tar -cf archive.tar file1 file2 directory/ ## gzip compressed tar -czf archive.tar.gz file1 file2 directory/ ## bzip2 compressed tar -cjf archive.tar.bz2 file1 file2 directory/ ## .tar.xz compressed tar -cJf archive.tar.xz file1 file2 directory/ ``` ``` bash ## Backup and Restore a Folder tar -czvf mybackup.tar.gz myfolder tar -xzvf mybackup.tar.gz ``` ### Regular expression Regular expressions (regex) are patterns used to match character combinations in strings. In Linux, they are commonly used with commands like `grep`, `sed`, and `awk` for searching and manipulating text. | Symbol | Descriptions | Examples | Matched pattern | |----------|-----------------------|----------------|------------------------| | . | match any characters | A..T | A[**TG**]{.underline}T, A[**GC**]{.underline}T | | \^ | match start of string | \^A | [**A**]{.underline}TG, [**A**]{.underline}CC, [**A**]{.underline}GTC | | \$ | match end of string | T\$ | TC[**T**]{.underline}, GA[**T**]{.underline}, ACG[**T**]{.underline} | | \* | match up zero or more times the preceding character | AT\* | [**AT**]{.underline}C, C[**ATT**]{.underline}, TG[**A**]{.underline}C | | \ | escape a special (or meta) character | \\\^A | [**\^A**]{.underline}, [**\^A**]{.underline}TG | | () | group regular expressions | (ATC){3} | [**ATCATCATC**]{.underline} | | | | | | | \[\] | match any one of a set characters | \[AT\] | [**A**]{.underline}CG, C[**T**]{.underline}G, [**A**]{.underline}C[**T**]{.underline}CG | | \[A-Z\] | match [**upper**]{.underline} case letter from A to Z | \[A-Z\] | \>GL000226.1 | | \[a-z\] | math [**lower**]{.underline} case letter form a to z | \[a-z\] | /[**home**]{.underline}/[**user**]{.underline}100 | | \[0-9\] | match digit from 0-9 | \[0-9\] | \>GL[**000226**]{.underline}.[**1**]{.underline} | | \[\^\] | match any character except those in the square bracket after carat \^ | \[\^T\] | [**GCCGAG**]{.underline}, [**\>GL000226.1**]{.underline} | | | | | | | \* | match up [**zero or more times**]{.underline} the preceding character | AT\*CG | ACG, A[**T**]{.underline}CG, A[**TT**]{.underline}CG | | \+ | match up [**one or more times**]{.underline} the preceding character | AT+CG | A[**T**]{.underline}CG, A[**TT**]{.underline}CG, A[**TTT**]{.underline}CG | | ? | match up [**zero or one times**]{.underline} the preceding character | AT?CG | A[**T**]{.underline}CG,ACG | | {N} | preceding character is matched [**exactly N times**]{.underline} | AT{3}CG | A[**TTT**]{.underline}CG | | {N,} | preceding character is matched [**at least N times or more**]{.underline} | AT{3,}CG | A[**TTT**]{.underline}CG, A[**TTTT**]{.underline}CG | | {N,M} | preceding character is matched [**at least N times but not more than M times**]{.underline} | AT{2,3}CG | A[**TT**]{.underline}CG, A[**TTT**]{.underline}CG | ### Text file manipulation #### Filtering `grep` Globally search a Regular Expression and Print: is a command-line utility for searching plain-text data sets for lines that match a specified pattern. ``` bash grep "pattern" file.txt # Search for "pattern" in file.txt grep -i "pattern" file.txt # Case-insensitive search grep -n "pattern" file.txt # Show line numbers grep -v "pattern" file.txt # Invert the sense of matching, to select non-matching lines ``` `cut` is used to extract sections from each line of input: - Extract columns or fields from text files - Specify delimiter for fields - Useful for processing structured data like CSV or TSV files - Works line by line - Column extraction based on delimiter or character position ``` bash cut -f1 file.txt # Extract first column from a tab-delimited file cut -d',' -f1,3 file.csv # Extract first and third columns from a comma-separated file cut -c1-5 file.txt # Extract characters from position 1 to 5 ``` ``` bash # Piping and combining commands grep "pattern" file.txt | cut -f1,3 | sort | uniq ``` #### Sort ```bash sort file.txt # Sort lines in file.txt alphabetically sort -n file.txt # Sort lines numerically sort -r file.txt # Sort lines in reverse order sort -u file.txt # Sort and remove duplicate lines sort -k2 file.txt # Sort by the second column ``` #### Duplicates ```bash uniq file.txt # Remove duplicate lines (requires sorted input) uniq -c file.txt # Count occurrences of each line sort file.txt | uniq -c | sort -nr # Count and sort by frequency ``` #### Merge ```bash paste file1.txt file2.txt > combined.txt # Join files side by side ``` #### Find & replace `sed` is a stream editor used to perform basic text transformations: \* substituction, deletion, insertion, and more \* works line by line, applying patterns and edits ``` bash ## Remove all spaces from a string echo "a b c" | sed 's/ //g' #> "abc" ## Remove all occurences of "foo" with "bar" echo "foo baz foo" | sed 's/foo/bar/g' #> "bar baz bar" ## Delete lines containing a "pattern" pattern from file.txt sed '/pattern/d' file.txt ## Print only lines matching a "pattern" pattern from file.txt sed -n '/pattern/p' file.txt ## Print lines 2 to 5 of a file sed -n '2,5p' file.txt ``` ### Data processing - `awk` used for pattern scanning and processing. \* More powerful than `cut` for data extraction and reporting \* Process text as fileds and records, support arithmetic and logic - `awk` is a powerful text-processing tool and programming language used for manipulating and analyzing data - Search files for lines (or other units of text) that contain certain patterns and perform specified actions on those lines - By default, awk prints every line of data from the specified file - Basic syntax: `awk '<pattern> {<action>}' <file>` - regular expression `awk '/pattern/ {print $1}' file.txt` will print the first column of lines that match the pattern - Build in variables: - `$0`: entire row of input - `$1`, `$2`, `$3`, ...: n-th column of inut - `NR`: the number of the current line (row) being processed. - `NF`: the number of fields (columns) in the current line. - `SNF`: the last field (column) in the current row. - `FS`: Field separator (default is whitespace). - `OFS`: Output field separator (default is whitespace). - `ORS`: Output record separator (default is newline). - `BEGIN` and `END`: Special blocks that allow you to specify actions to be performed before processing any lines and after processing all lines, respectively. - Operators - `> | >=`: greater than or greater than or equal to - `< | <=`: less than or less than or equal to - `==`: equal to - `!=`: not equal to - `&&`: And (reports "true" if both conditions are true) - `||`: Or (reports "true" if either condition, or both, are true) - `!`: Not (negates the condition, reports "true" if the condition is false) ```bash awk '/a/{print $1}' file.txt # Print the first column of lines that contain the letter "a" ## Print the first column of a file awk '{print $1}' file.txt ## Sum values in the first column awk '{sum += $1} END {print sum}' file.txt ## Print lines where the third column is greater than 100 awk '$3 > 100' file.txt ## Print the lines that match a pattern "error" awk '/error/' file.txt ## Print the number of lines in a file awk 'END {print NR}' file.txt ``` - Delimiters - `-F FS`: - `FS`: input field separator (default is whitespace) - `-v OFS`: - `OFS`: output field separator (default is whitespace) ```bash awk -F ',' 'NR==1 || $3>10 {print $1, $2, $3}' file.csv # Print the first three columns of a CSV file where the third column is greater than 10, including the header row awk -F ',' -v OFS='\t' 'NR==1 || $3>10 {print $1, $2, $3}' file.csv ``` - Syntax with `BEGIN` and `END` blocks - `awk 'BEGIN {<action1>} <pattern> {<action2>} END {<action3>}' <file>` - `BEGIN {<actioin1>}`: the `BEGIN` block is executed before any input lines are processed. It is often used to initialize variables, set field separators and setting up the environment. - `<pattern> {<action2>}`: This is the main processing block for each input line. For each line in the input file, if the line matches the specified pattern, the associated action is executed. If no pattern is specified, the action is applied to all lines. - `END {<action3>}`: The `END` block is executed after all input lines have been processed. It is typically used for final calculations, printing summaries, or performing cleanup tasks. - Arithmetic operations - `+`: addition - `-`: subtraction - `*`: multiplication - `/`: division - `^`: exponentiation - `%`: remainder - `++`: increment - `--`: decrement - `+=`: add and assign - `-=`: subtract and assign ### Search anything `find` is a powerful command to search for files and directories based on various criteria: \* Search by name, type, size, modification time, permissions, etc. \* Execute actions on found items (delete, move, etc.) \* Supports complex expressions with logical operators ``` bash ## Find all .txt files in current directory and subdirectories find . -name "*.txt" ## Find files/folders by name find /path/to/search -name "pattern" ## Find files larger than 100MB find /path/to/search -type f -size +100M ## Find files in pattern and list details find ./Clean -name "*Clean.fastq.gz" -type f -exec ls -lh {} \; ``` ### Redirection Redirection is a way to direct the input and output of a command to a file or another command. It allows you to save the output of a command to a file, or to use the output of one command as the input for another command. ``` bash ## Redirect output to a file command > output.txt ## Append output to a file command >> output.txt ``` Standard streams in Linux: \* `stdin` (standard input): The default input stream for a command, 0 \* `stdout` (standard output): The default output stream for a command, 1 \* `stderr` (standard error): The default error output stream for a command, 2 ## Vim Text Editor ### Modes - Vim text editor has three main modes: - `Esc`, back to normal mode - `i`, insert mode - `v`, select mode | Command | What it does | Most common follow-up | |---------|-----------------------------|-----------------------| | i | Insert before cursor | typing text | | a | Append after cursor | typing text | | I | Insert at beginning of line | — | | A | Append at end of line | — | | o | Open new line below | typing | | O | Open new line above | typing | | Esc | Back to Normal mode | — | | v | Visual mode | move + operator | | V | Visual Line mode | — | | Ctrl-v | Visual Block mode | — | ### Navigation | Command | Movement | Common combos | |-----------------|--------------------------------|-----------------------| | h j k l | ← ↓ ↑ → | — | | w / b | forward/backward by word start | ciw, daw | | e / ge | forward/backward to word end | — | | W / B | forward/backward by big word | (ignores punctuation) | | 0 | Start of line | — | | \^ | First non-blank character | — | | \$ | End of line | — | | gg | Top of file | — | | G | Bottom of file | 50G → line 50 | | Ctrl-u / Ctrl-d | Half page up/down | — | | Ctrl-f / Ctrl-b | Full page down/up | — | ### Editing | Command | Meaning | examples | |-----------------|-------------------------------|------------------------| | x | Delete character under cursor | — | | dd | Delete current line | 5dd = delete 5 lines | | yy | Yank (copy) current line | p to paste | | p / P | Paste after/before cursor | — | | u | Undo | (most used command after Esc) | | Ctrl-r | Redo | — | | . | Repeat last change/command | (extremely powerful) | | dw / d\$ | Delete to end of word / end of line | diw, daw | | ciw / caw | Change (delete + insert) inside/around word | most common text object usage | | "ci"" / ci( / ci{" | Change inside quotes/parentheses/braces | extremely common in code | ### Search & Replace | Command | Meaning | |----------------|-----------------------------------------------| | /pattern | Search forward | | ?pattern | Search backward | | n / N | Next / previous match | | \* / \# | Search forward/backward for word under cursor | | :%s/old/new/g | Replace all occurrences (whole file) | | :%s/old/new/gc | Replace with confirmation | ### Saving & Exiting | Command | Meaning | |-----------------|-------------------------------------------------| | :w | Save (write) | | :q | Quit (only if no changes) | | :q! | Quit without saving | | :wq or :x or ZZ | Save and quit | | :w !sudo tee % | Save file needing root (very common workaround) | ## Nano Text Editor 1. Open a file with: `nano filename.txt` 2. Use the arrow keys to navigate. 3. Save the changes with `ctrl + o`, 4. Confirm the filename by pressing `Enter`. 5. Exit nano with `ctrl + x`. ### Searching Text - To find words or phrases within a file: 1. Initiate search with `ctrl + w`. 2. Type the search term and press `Enter`. 3. Exit search mode with `ctrl + c`. ### Customizing Nano Enhance Nano's functionality: ``` bash nano -miA file.txt ``` - `-m`: Enable mouse support. - `-i`: Auto-indent new lines. - `-A`: Enable syntax highlighting. ## Job Scheduling ### OpenPBS OpenPBS (Portable Batch System) is an open-source workload manager and job scheduler used in High-Performance Computing (HPC) environments to automate and optimize the execution of tasks across and clouds. **PBS job script** ``` bash #!/bin/bash #PBS -N job_name # Job name #PBS -l nodes=1:ppn=4 # Number of nodes and processors per node #PBS -l walltime=01:00:00 # Walltime (hh:mm:ss) #PBS -l mem=8gb # Memory per node #PBS -j oe # Combine stdout and stderr #PBS -M user@example.com # Email address for notifications #PBS -m abe # Send email on (a) abort, (b) begin, (e) end #PBS -q batch # Queue named #PBS -V # Export all environment variables to the job ## example cd $PBS_O_WORKDIR # Change to the directory where the job was submitted module load python/3.8.5 # Load necessary modules python my_script.py # Command to run your program ``` **PBS environment variables** ``` bash $PBS_O_WORKDIR # The directory from which the job was submitted $PBS_O_HOME # The home directory of the user who submitted the job $PBS_JOBDIR # The working directory for the job on the compute node $TMPDIR # The temporary directory for the job on the compute node $PBS_JOBID # The unique identifier assigned to the job $PBS_NODEFILE # The file containing the list of nodes allocated to the job $PBS_NUM_NODES # The number of nodes allocated to the job $PBS_NUM_PPN # The number of processors per node allocated to the job $PBS_QUEUE # The name of the queue to which the job was submitted $PBS_JOBNAME # The name of the job as specified in the job script ``` **Submit job scripts** ``` bash ## Submit a job qsub job_script.pbs ## Submit a job with environment variables. qsub -v sample="$sample" -M "$EMAIL" pbs_1_cutadapt.pbs ``` **Check job status** ``` bash ## List finished job details in summary qstat -x job_id ## List finished job details in full details qstat -H job_id qstat -x -f job_id qstat -x -f job_id | grepl "exec_host" ## List all jobs for user guorui qstat -u guorui -a ## Find out all compute nodes allocated to a job for job_id in $(find . -name "log.*.e*" -printf "%f\n" | sed 's/.*\.e$[0-9]*$$/\1/' | sort -n | uniq); do echo -n "Job ID: $job_id: "; qstat -x -f $job_id | grep exec_host done ## Delete a job qdel job_id ## check all queues qstat -Qf | less for q in cgs small medium large cgs_queue; do echo "=== $q ===" qstat -Qf $q | grep -E "resources_max.(mem|ncpus|walltime)" done ## Hold a job qhold job_id ## Release a held job qrls job_id # See your groups groups # Check queue ACLs qstat -Q -f | grep -E "^Queue:|acl_users|acl_groups" ``` **Stat interactive job** ``` bash ## Start and interactive job with 16 processors, 64GB memory, 24 hours walltime in cgs_queue qsub -I -q cgs_queue -l nodes=hpcf3-c01:ppn=16,walltime=12:00:00,mem=64gb qsub -I -q cgs_queue -l nodes=hpcf3-c02:ppn=16,walltime=12:00:00,mem=64gb qsub -I -q cgs_queue -l nodes=hpcf3-c03:ppn=16,walltime=12:00:00,mem=64gb ``` **Check the permission of queue** ``` bash qstat -Q cgs -f # Queue: cgs # queue_type = Execution # total_jobs = 16 # state_count = Transit:0 Queued:10 Held:0 Waiting:0 Running:6 Exiting:0 Begu # n:0 # acl_user_enable = True # acl_users = bioinfo,emiklin,irisxu,itsupport,julia,junyanzhong,manzhai,nicklin, # rubyzhou,totochu # resources_max.mem = 1024gb # resources_max.ncpus = 64 # resources_max.nodect = 1 # resources_max.walltime = 1440:00:00 # resources_default.mem = 1gb # resources_default.walltime = 24:00:00 # default_chunk.Qlist = cgs # resources_assigned.mem = 3019898880kb # resources_assigned.mpiprocs = 168 # resources_assigned.ncpus = 253 # resources_assigned.nodect = 6 # enabled = True # started = True ``` - `-Q`: Show queue information (not job information) - `-cgs`: The specific queue name to query - `-f`: Full/detailed output showing all queue attributes ## Terminal Multiplexer ### Tmux - Tmux Configuration Consider adding these aliases to your shell configuration for quick tmux access. ::: code-block ``` bash # Terminal aliases for tmux alias t="tmux" alias ta="t a -t" alias tls="t ls" alias tn="t new -s" alias tk="t kill-session -t" alias tks="t kill-server" ``` ::: ::: callout-note ### 🔧 Custom Prefix Key Change the default prefix in `.tmux.conf`: ``` bash set-option -g prefix C-a bind-key C-a send-prefix ``` ::: - Session Management | **Action** | **Command/Shortcut** | |--------------------------|-----------------------------------------| | List sessions | `tmux list-sessions` | | Attach to session | `tmux attach-session -t target-session` | | Switch between sessions | `Ctrl + A + s` | | Switch to latest session | `Ctrl + A + l` | | Detach from session | `Ctrl + A + d` | - Window Management | **Action** | **Shortcut** | |---------------------------|----------------| | List all windows | `Ctrl + A + w` | | Rename current window | `Ctrl + A + ,` | | Switch to next window | `Ctrl + A + →` | | Switch to previous window | `Ctrl + A + ←` | | Create new window | `Ctrl + A + c` | | Kill current window | `Ctrl + A + q` | - Pane Management | **Action** | **Shortcut** | |----------------------|----------------| | Switch to pane above | `Shift + ↑` | | Switch to pane below | `Shift + ↓` | | Switch to pane left | `Shift + ←` | | Switch to pane right | `Shift + →` | | Kill current pane | `Ctrl + A + x` | - Copy Mode | **Action** | **Shortcut** | |-------------------|---------------------------| | Enter copy mode | Drag mouse to select text | | Paste copied text | `Ctrl + A + ]` | ------------------------------------------------------------------------ ### Screen ::: callout-tip ### 💡 Screen Aliases Add these aliases to your shell configuration for easier screen management. ::: ::: code-block ``` bash # Terminal aliases for screen alias s="screen" # start a screen session alias ss="s -S" # start a named screen session alias sr="s -r" # reattach to a screen session alias sls="s -ls" # list current running screen sessions ``` ::: - Basic Commands | **Action** | **Command** | |-----------------------|--------------------------| | Start screen session | `screen` | | Start named session | `screen -S session_name` | | Reattach to session | `screen -r` | | List running sessions | `screen -ls` | - Window Management | **Action** | **Shortcut** | |------------------------|-----------------------| | Create new window | `Ctrl + A + C` | | Kill current window | `Ctrl + A + K` | | List all windows | `Ctrl + A + W` | | Go to window 0-9 | `Ctrl + A + 0-9` | | Go to next window | `Ctrl + A + N` | | Toggle between windows | `Ctrl + A + Ctrl + A` | | Rename current window | `Ctrl + A + A` | - Region Management | **Action** | **Shortcut** | |-------------------------------|------------------| | Split horizontally | `Ctrl + A + S` | | Split vertically | `Ctrl + A + \|` | | Switch between regions | `Ctrl + A + Tab` | | Close all regions but current | `Ctrl + A + Q` | | Close current region | `Ctrl + A + X` | - Session & Copy Mode | **Action** | **Shortcut** | |---------------------|-----------------------| | Detach from session | `Ctrl + A + D` | | Start copy mode | `Ctrl + A + [` | | Paste copied text | `Ctrl + A + ]` | | Show help | `Ctrl + A + ?` | | Quit screen | `Ctrl + A + Ctrl + \` | ------------------------------------------------------------------------ ::: callout-note ### Additional Resources - **VSCode**: [Official Keyboard Shortcuts Reference](https://code.visualstudio.com/docs/getstarted/keybindings) - **Vim**: [Vim Cheat Sheet](https://vim.rtorr.com/) - **Tmux**: [Tmux Cheat Sheet](https://tmuxcheatsheet.com/) - **Terminal**: [Bash Reference Manual](https://www.gnu.org/software/bash/manual/) ::: ## Manage Environment ### Conda - Installation ``` bash # Miniforge linux wget "https://github.com/conda-forge/miniforge/releases/latest/download/Miniforge3-Linux-x86_64.sh" # Miniforge Mac wget "https://github.com/conda-forge/miniforge/releases/latest/download/Miniforge3-MacOSX-arm64.sh" # Install it bash Miniforge3-$(uname)-$(uname -m).sh # Add conda to PATH if found if [ -f "$HOME/miniconda3/etc/profile.d/conda.sh" ]; then . $HOME/miniconda3/etc/profile.d/conda.sh fi ``` - Configuration ``` bash # Init conda init --all # Use mamba for faster solving conda update -n base conda conda install -n base conda-libmamba-solver # Config conda config --set solver libmamba conda config --set always_yes true conda config --set auto_activate_base false # Setup channels conda config --add channels bioconda conda config --add channels conda-forge #conda config --set channel_priority strict # Install software from a specific channel conda install -c conda-forge numpy ``` - Create Environment ``` bash # Create R environment with specific R version and packages conda create -n renv r=4.5 r-languageserver r-tidyverse r-irkernel r-httpgd r-downlit r-xml2 r-markdown r-devtools radian # Install bioconductor packages, all package names are lower case conda install -n renv bioconductor-deseq2 bioconductor-edger # Install repository packages conda install -n renv r-qs r-fs r-tidyverse # Remove a specific environment conda remove --name renv --all # Remove a software in a environment conda remove --name renv package_name # Export and save the conda env file with all software information conda env export -n renv > renv.yml # Export and save the conda env file with all software information # without the prefix and bundle information conda env export -n renv --no-builds > renv.yml conda env export -n renv --no-builds --file renv.yml # Create a new environment with a environment file conda env create --file renv.yml # Rename an environment conda create --name new_env_name --clone old_env_name conda activate new_env_name conda remove --name old_env_name --all ``` ### Pixi ``` bash ## Auto-activate pixi when entering project directory eval "$(pixi completion --shell bash)" # or zsh ``` ``` bash ## Create a global pixi environment with specific R version and packages pixi global install -e renv \ r-base=4.5 \ r-languageserver \ r-tidyverse \ r-irkernel \ r-httpgd \ r-downlit \ r-xml2 \ r-markdown \ r-devtools \ radian ## Print out global pixi environments pixi global list ## Add Bioconda Permanently pixi config append default-channels bioconda --global pixi config set default-channels '["conda-forge", "bioconda"]' --global ``` ## Git Version Control ``` bash ## keep your original Git repository but provide a clean version to others, git archive --format=zip --output=my-project-clean.zip main git init git add . git commit -m "first commit" git remote add origin git push -u origin main git config --global user.name "" # set an user name that will be associated with each history marker git config --global user.email "" # set an email address that will be associated with each history marker git config --global color.ui auto # set automatic command line coloring for Git for easy reviewing git init # # 1. Delete the local branch git branch -d branch-name # Use -D to force delete # 2. Delete the remote branch git push origin --delete branch-name # 3. Verify branches are gone git branch -a # List all branches echo "" # create file "" git status # check git add # add a file as it looks now to your next commit (stage) git commit # commit your staged content as a new commit snapshot ### if go to the viam editing mode, i to input, esc to quit, :wq to save and quit git log # show all commits in the current branch’s history, check versions, q to quit touch .gitignore git branch "" # create branch "" git branch # check branches git checkout "" # switch to another branch and check it out into your working directory "" git branch -D "" # delete branch ## git checkout -b temp # create temporary branch ## git merage # merge the specified branch’s history into the current one git clone # retrieve an entire repository from a hosted location via URL git remote -V # check remove repos information git push # Transmit local branch commits to the remote repository branch git fetch # fetch down all the branches from that Git remote git diff # diff of what is changed but not staged git pull # fetch and merge any commits from the tracking remote branch git reset # unstage a file while retaining the changes in working directory git rm -r --cached . # ignore already added files run ## Go back to a specifi commit git reset --hard e4098a7 ## Pull remote updates and ignore local changes git fetch origin git reset --hard origin/your-branch-name ## Change remote URL git remote -v # check current remote configuration git remote set-url origin <new-repository-url> git remote remove origin git remote add origin <new-repository-url> ## Reset to match remote git fetch origin git reset --hard origin/main ``` ## Windows WSL - System info ``` bash ## Install the neofetch package sudo apt install neofetch -y ## Show system info neofetch --memory_unit gib ``` - Change sudo password In the windows terminal, run the following command to change the sudo password: ``` bash wsl -u root ## Input the new password passwd <username> ``` - WSL configuration - WSL global config: `C:\Users\<UserName>\.wslconfig` - WSL distro config: `\\wsl.localhost\Ubuntu-24.04\etc\wsl.conf`