Linux

Prompt

# mac
PROMPT='%F{cyan}%n%F{white} @%F{green} %m:%F{yellow} %~%f
%F{magenta}❯%f '

# Linux
PS1='\[\e[36m\]\u\[\e[37m\] @\[\e[32m\]\h:\[\e[33m\]\w\[\e[0m\]\n\[\e[35m\]❯\[\e[0m\] '

Vim

Modes

Vim text editor has three main modes:
- Esc, back to normal mode
- i, insert mode
- v, select mode

Command	What it does	Most common follow-up
i	Insert before cursor	typing text
a	Append after cursor	typing text
I	Insert at beginning of line	—
A	Append at end of line	—
o	Open new line below	typing
O	Open new line above	typing
Esc	Back to Normal mode	—
v	Visual mode	move + operator
V	Visual Line mode	—
Ctrl-v	Visual Block mode	—

Navigation

Command	Movement	Common combos
h j k l	← ↓ ↑ →	—
w / b	forward/backward by word start	ciw, daw
e / ge	forward/backward to word end	—
W / B	forward/backward by big word	(ignores punctuation)
0	Start of line	—
^	First non-blank character	—
$	End of line	—
gg	Top of file	—
G	Bottom of file	50G → line 50
Ctrl-u / Ctrl-d	Half page up/down	—
Ctrl-f / Ctrl-b	Full page down/up	—

Editing

Command	Meaning	examples
x	Delete character under cursor	—
dd	Delete current line	5dd = delete 5 lines
yy	Yank (copy) current line	p to paste
p / P	Paste after/before cursor	—
u	Undo	(most used command after Esc)
Ctrl-r	Redo	—
.	Repeat last change/command	(extremely powerful)
dw / d$	Delete to end of word / end of line	diw, daw
ciw / caw	Change (delete + insert) inside/around word	most common text object usage
“ci”” / ci( / ci{”	Change inside quotes/parentheses/braces	extremely common in code

Search & Replace

Command	Meaning
/pattern	Search forward
?pattern	Search backward
n / N	Next / previous match
* / #	Search forward/backward for word under cursor
:%s/old/new/g	Replace all occurrences (whole file)
:%s/old/new/gc	Replace with confirmation

Saving & Exiting

Command	Meaning
:w	Save (write)
:q	Quit (only if no changes)
:q!	Quit without saving
:wq or :x or ZZ	Save and quit
:w !sudo tee %	Save file needing root (very common workaround)

Nano

Open a file with: nano filename.txt
Use the arrow keys to navigate.
Save the changes with ctrl + o,
Confirm the filename by pressing Enter.
Exit nano with ctrl + x.

Searching Text

To find words or phrases within a file:

Initiate search with ctrl + w.
Type the search term and press Enter.
Exit search mode with ctrl + c.

Customizing Nano

Enhance Nano’s functionality:

nano -miA file.txt

-m: Enable mouse support.
-i: Auto-indent new lines.
-A: Enable syntax highlighting.

Jobs

OpenPBS

OpenPBS (Portable Batch System) is an open-source workload manager and job scheduler used in High-Performance Computing (HPC) environments to automate and optimize the execution of tasks across and clouds.

PBS job script

#!/bin/bash
#PBS -N job_name                  # Job name
#PBS -l nodes=1:ppn=4             # Number of nodes and processors per node
#PBS -l walltime=01:00:00         # Walltime (hh:mm:ss)
#PBS -l mem=8gb                   # Memory per node
#PBS -j oe                        # Combine stdout and stderr
#PBS -M user@example.com          # Email address for notifications
#PBS -m abe                       # Send email on (a) abort, (b) begin, (e) end
#PBS -q batch                     # Queue named
#PBS -V                           # Export all environment variables to the job

## example
cd $PBS_O_WORKDIR                 # Change to the directory where the job was submitted
module load python/3.8.5          # Load necessary modules
python my_script.py               # Command to run your program

PBS environment variables

$PBS_O_WORKDIR     # The directory from which the job was submitted
$PBS_O_HOME        # The home directory of the user who submitted the job
$PBS_JOBDIR        # The working directory for the job on the compute node
$TMPDIR            # The temporary directory for the job on the compute node
$PBS_JOBID         # The unique identifier assigned to the job
$PBS_NODEFILE      # The file containing the list of nodes allocated to the job
$PBS_NUM_NODES     # The number of nodes allocated to the job
$PBS_NUM_PPN       # The number of processors per node allocated to the job
$PBS_QUEUE         # The name of the queue to which the job was submitted
$PBS_JOBNAME       # The name of the job as specified in the job script

Submit job scripts

## Submit a job
qsub job_script.pbs

## Submit a job with environment variables.
qsub -v sample="$sample" -M "$EMAIL" pbs_1_cutadapt.pbs

Check job status

## List finished job details in summary
qstat -x job_id

## List finished job details in full details
qstat -H job_id
qstat -x -f job_id
qstat -x -f job_id | grepl "exec_host"

## List all jobs for user guorui
qstat -u guorui -a

## Find out all compute nodes allocated to a job
for job_id in $(find . -name "log.*.e*" -printf "%f\n" | sed 's/.*\.e\([0-9]*\)$/\1/' | sort -n | uniq); do
    echo -n "Job ID: $job_id: "; qstat -x -f $job_id | grep exec_host
done

## Delete a job
qdel job_id

## Delete all jobs for user guorui
qselect -u guorui | xargs qdel

## check all queues
qstat -Qf | less

for q in cgs small medium large cgs_queue; do
    echo "=== $q ==="
    qstat -Qf $q | grep -E "resources_max.(mem|ncpus|walltime)"
done

## Hold a job
qhold job_id

## Release a held job
qrls job_id

# See your groups
groups

# Check queue ACLs
qstat -Q -f | grep -E "^Queue:|acl_users|acl_groups"

Stat interactive job

## Start and interactive job with 16 processors, 64GB memory, 24 hours walltime in cgs_queue
qsub -I -q cgs_queue -l nodes=hpcf3-c01:ppn=16,walltime=12:00:00,mem=64gb
qsub -I -q cgs_queue -l nodes=hpcf3-c02:ppn=16,walltime=12:00:00,mem=64gb
qsub -I -q cgs_queue -l nodes=hpcf3-c03:ppn=16,walltime=12:00:00,mem=64gb

Check the permission of queue

qstat -Q cgs -f

# Queue: cgs
#     queue_type = Execution
#     total_jobs = 16
#     state_count = Transit:0 Queued:10 Held:0 Waiting:0 Running:6 Exiting:0 Begu
#         n:0
#     acl_user_enable = True
#     acl_users = bioinfo,emiklin,irisxu,itsupport,julia,junyanzhong,manzhai,nicklin,
#         rubyzhou,totochu
#     resources_max.mem = 1024gb
#     resources_max.ncpus = 64
#     resources_max.nodect = 1
#     resources_max.walltime = 1440:00:00
#     resources_default.mem = 1gb
#     resources_default.walltime = 24:00:00
#     default_chunk.Qlist = cgs
#     resources_assigned.mem = 3019898880kb
#     resources_assigned.mpiprocs = 168
#     resources_assigned.ncpus = 253
#     resources_assigned.nodect = 6
#     enabled = True
#     started = True

-Q: Show queue information (not job information)
-cgs: The specific queue name to query
-f: Full/detailed output showing all queue attributes

# Delete all held jobs for the current user
qstat -u $USER | awk '$10=="H" {print $1}' | xargs -r qdel

# Delete all running jobs for the current user
qstat -u $USER | awk '$10=="R" {print $1}' | xargs -r qdel

# Delete every job whose numeric ID is greater than a threshold. Preview first, then delete:
qselect -u $USER | awk -F. '$1+0 > 158899'
qselect -u $USER | awk -F. '$1+0 > 158899' | xargs -r qdel

SLURM

SLURM (Simple Linux Utility for Resource Management) is an open-source workload manager and job scheduler widely used in HPC clusters to allocate resources and run jobs across compute nodes.

SLURM job script

#!/bin/bash
#SBATCH --job-name=job_name          # Job name
#SBATCH --nodes=1                    # Number of nodes
#SBATCH --ntasks=1                   # Number of tasks
#SBATCH --cpus-per-task=4            # Processors per task
#SBATCH --time=01:00:00              # Walltime (hh:mm:ss)
#SBATCH --mem=8G                     # Memory per node
#SBATCH --output=log.%x.%j.out       # Stdout (%x = job name, %j = job ID)
#SBATCH --error=log.%x.%j.err        # Stderr
#SBATCH --mail-user=user@example.com # Email address for notifications
#SBATCH --mail-type=BEGIN,END,FAIL   # Send email on begin, end, fail
#SBATCH --partition=batch            # Partition (queue) name
#SBATCH --export=ALL                 # Export all environment variables to the job

## example
cd $SLURM_SUBMIT_DIR                 # Change to the directory where the job was submitted
module load python/3.8.5             # Load necessary modules
python my_script.py                  # Command to run your program

SLURM environment variables

$SLURM_SUBMIT_DIR       # The directory from which the job was submitted
$SLURM_JOB_ID           # The unique identifier assigned to the job
$SLURM_JOB_NAME         # The name of the job
$SLURM_JOB_NODELIST     # The list of nodes allocated to the job
$SLURM_JOB_NUM_NODES    # The number of nodes allocated to the job
$SLURM_NTASKS           # The number of tasks
$SLURM_CPUS_PER_TASK    # The number of processors per task
$SLURM_JOB_PARTITION    # The partition to which the job was submitted
$SLURM_ARRAY_TASK_ID    # The index of the task in a job array
$TMPDIR                 # The temporary directory for the job on the compute node

Submit job scripts

## Submit a job
sbatch job_script.sh

## Submit a job with environment variables
sbatch --export=ALL,sample="$sample" --mail-user="$EMAIL" slurm_1_cutadapt.sh

## Submit a job array (tasks 1-10, at most 4 running at once)
sbatch --array=1-10%4 job_script.sh

Check job status

## List all jobs for user guorui
squeue -u guorui

## Show details of a pending/running job
scontrol show job job_id

## List finished job details in summary
sacct -j job_id --format=JobID,JobName,Partition,State,Elapsed,MaxRSS,ExitCode

## List all jobs of user guorui since a date
sacct -u guorui -S 2026-01-01 --format=JobID,JobName,State,Elapsed,NodeList

## Delete a job
scancel job_id

## Delete all jobs for user guorui
scancel -u guorui

## Delete all pending / running jobs for the current user
scancel -u $USER --state=PENDING
scancel -u $USER --state=RUNNING

## Hold a job
scontrol hold job_id

## Release a held job
scontrol release job_id

## Check all partitions and nodes
sinfo
sinfo -N -l

## Show partition limits (memory, CPUs, walltime)
scontrol show partition

## See your account and associations (allowed partitions/QOS)
sacctmgr show assoc user=$USER format=User,Account,Partition,QOS

Start interactive job

## Start an interactive job with 16 processors, 64GB memory, 12 hours walltime
srun --partition=batch --nodes=1 --cpus-per-task=16 --mem=64G --time=12:00:00 --pty bash

## Request a specific node
srun --partition=batch --nodelist=hpcf3-c01 --cpus-per-task=16 --mem=64G --time=12:00:00 --pty bash

## Allocate resources first, then attach
salloc --partition=batch --cpus-per-task=16 --mem=64G --time=12:00:00

Check the permission of partition

scontrol show partition batch

# PartitionName=batch
#    AllowGroups=ALL AllowAccounts=ALL AllowQos=ALL
#    Default=YES MaxTime=60-00:00:00 State=UP
#    TotalCPUs=64 TotalNodes=1
#    DefMemPerNode=1024 MaxMemPerNode=1048576

scontrol show partition: Show partition (queue) attributes
batch: The specific partition name to query
AllowGroups/AllowAccounts: Who is permitted to submit to the partition

PBS to SLURM cheat sheet

Task	OpenPBS	SLURM
Submit job	`qsub script.pbs`	`sbatch script.sh`
Job status	`qstat -u $USER`	`squeue -u $USER`
Finished job info	`qstat -x -f job_id`	`sacct -j job_id`
Delete job	`qdel job_id`	`scancel job_id`
Hold / release	`qhold` / `qrls`	`scontrol hold` / `release`
Interactive job	`qsub -I`	`srun --pty bash` / `salloc`
Queue info	`qstat -Q`	`sinfo`
Working directory	`$PBS_O_WORKDIR`	`$SLURM_SUBMIT_DIR`
Job ID	`$PBS_JOBID`	`$SLURM_JOB_ID`
Node list	`$PBS_NODEFILE`	`$SLURM_JOB_NODELIST`

WSL

System info


## Install the neofetch package
sudo apt install neofetch -y

## Show system info
neofetch --memory_unit gib

Change sudo password

In the windows terminal, run the following command to change the sudo password:


wsl -u root

## Input the new password
passwd <username>

WSL configuration
- WSL global config: C:\Users\<UserName>\.wslconfig
- WSL distro config: \\wsl.localhost\Ubuntu-24.04\etc\wsl.conf

Tmux

Tmux Configuration Consider adding these aliases to your shell configuration for quick tmux access.

# Terminal aliases for tmux
alias t="tmux"
alias ta="t a -t"
alias tls="t ls"
alias tn="t new -s"
alias tk="t kill-session -t"
alias tks="t kill-server"

🔧 Custom Prefix Key

Change the default prefix in .tmux.conf:

set-option -g prefix C-a
bind-key C-a send-prefix

Session Management

Action	Command/Shortcut
List sessions	`tmux list-sessions`
Attach to session	`tmux attach-session -t target-session`
Switch between sessions	`Ctrl + A + s`
Switch to latest session	`Ctrl + A + l`
Detach from session	`Ctrl + A + d`

Window Management

Action	Shortcut
List all windows	`Ctrl + A + w`
Rename current window	`Ctrl + A + ,`
Switch to next window	`Ctrl + A + →`
Switch to previous window	`Ctrl + A + ←`
Create new window	`Ctrl + A + c`
Kill current window	`Ctrl + A + q`

Pane Management

Action	Shortcut
Switch to pane above	`Shift + ↑`
Switch to pane below	`Shift + ↓`
Switch to pane left	`Shift + ←`
Switch to pane right	`Shift + →`
Kill current pane	`Ctrl + A + x`

Copy Mode

Action	Shortcut
Enter copy mode	Drag mouse to select text
Paste copied text	`Ctrl + A + ]`

Screen

Config

## ~/.screenrc

# tells GNU Screen how to handle terminal initialization (ti) and termination (te) sequences for xterm and its variants (like xterm-color, gnome-terminal, etc.).
termcapinfo xterm* ti@:te@

# binds a specific keyboard key to send a command or string
bindkey -k kI stuff
bindkey -k kD stuff

💡 Screen Aliases

Add these aliases to your shell configuration for easier screen management.

# Terminal aliases for screen
alias s="screen"       # start a screen session
alias ss="s -S"        # start a named screen session
alias sr="s -r"        # reattach to a screen session
alias sls="s -ls"      # list current running screen sessions

Basic Commands

Action	Command
Start screen session	`screen`
Start named session	`screen -S session_name`
Reattach to session	`screen -r`
List running sessions	`screen -ls`

Window Management

Action	Shortcut
Create new window	`Ctrl + A + C`
Kill current window	`Ctrl + A + K`
List all windows	`Ctrl + A + W`
Go to window 0-9	`Ctrl + A + 0-9`
Go to next window	`Ctrl + A + N`
Toggle between windows	`Ctrl + A + Ctrl + A`
Rename current window	`Ctrl + A + A`

Region Management

Action	Shortcut
Split horizontally	`Ctrl + A + S`
Split vertically	`Ctrl + A + \\|`
Switch between regions	`Ctrl + A + Tab`
Close all regions but current	`Ctrl + A + Q`
Close current region	`Ctrl + A + X`

Session & Copy Mode

Action	Shortcut
Detach from session	`Ctrl + A + D`
Start copy mode	`Ctrl + A + [`
Paste copied text	`Ctrl + A + ]`
Show help	`Ctrl + A + ?`
Quit screen	`Ctrl + A + Ctrl + \`

Additional Resources

VSCode: Official Keyboard Shortcuts Reference
Vim: Vim Cheat Sheet
Tmux: Tmux Cheat Sheet
Terminal: Bash Reference Manual

Linux

Linux is a family of open-source Unix-like operating systems based on the Linux kernel. It is widely used for servers, desktops, and embedded systems. Linux provides a powerful command-line interface (CLI) that allows users to interact with the system and perform various tasks.
Linux commands are executed in a terminal or shell, and they can be combined to create powerful scripts and automate tasks. The Linux file system is organized in a hierarchical structure, with directories and files that can be manipulated using various commands.
In bioinformatics, Linux is commonly used for data analysis, bioinformatics pipelines, and high-performance computing (HPC) tasks. Many bioinformatics tools and software are designed to run on Linux systems, making it a preferred choice for researchers in the field.

File & director

### Working with files
cp file1.txt file2.txt  # Copy file1.txt to file2.txt
mv file1.txt file2.txt  # Move or rename file1.txt to file2.txt
rm file1.txt            # Remove file1.txt
gzip file1.txt          # Compress file1.txt to file1.txt.gz
gunzip file1.txt.gz     # Decompress file1.txt.gz to file1.txt

### Working with directories
mkdir my_directory        # Create a new directory named my_directory
cd my_directory           # Change into the directory my_directory
ls                        # List files and directories in the current directory
ls -l                     # List files and directories with detailed information
ls -a                     # List all files, including hidden files
ls -lh                    # List files with human-readable sizes
rm -r my_directory        # Remove the directory my_directory and its contents

### View file contents
cat file.txt          # Display the entire contents of file.txt
head file.txt         # Display the first 10 lines of file.txt
tail file.txt         # Display the last 10 lines of file.txt
wc -l file.txt        # Count the number of lines in file.txt

less file.txt         # View file.txt with scrolling and searching capabilities

## Options for less:
# -n: Show line numbers
# +F: Follow the file (like tail -f)
# -S: Don't wrap long lines
# -U: Don't show control characters

grep "\*" file.txt | less -S  # Search for lines containing "*" in file.txt and view them with less, without line wrapping

File transfer

Transfering data efficiently between local and remote systems:
- wget: Download files directly from the internet to your local machine or server.
- scp: Securely copy files between local and remote systems over SSH.
- rsync: Synchronize files and directories efficiently, with options for selective copying and deletion.

## Basic command
rsync -av /source/folder/ /destination/folder/

## Sync source to destination
rsync -av --delete /source/folder/ /destination/folder/

## If you need two-way sync, run both directions
rsync -av --delete /folder1/ /folder2/
rsync -av --delete /folder2/ /folder1/

## sync multiple folders at once
rsync -avP /source/{folder1,folder2,folder3} /destination/
rsync -avP --include='v*/' --include='Scripts/' --exclude='*' /groups/fcoi/guorui/analysis/ /path/to/destination/

## Network sync
rsync -av --delete /local/folder/ user@remote:/remote/folder/

## Copy only new files (files that don't exist in the destination)
rsync -av --ignore-existing /source/folder/ /destination/folder/

## Copy only if source is newer
rsync -av --update /source/folder/ /destination/folder/

Note

--archive: Preserves permissions, timestamps, symbolic links, etc.
--verbose: Shows progress
--delete: Removes files in destination that don’t exist in source
--dry-run: Preview changes without actually syncing
--update: Only copy newer file
--ignore-existing: Skips files that already exist in destination (regardless of age)
--update: Only copies if source file is newer than destination
--no-perms: Don’t preserve Unix permissions
--no-times: Don’t preserve modification times
--inplace: Write directly without temp files

Sync remote server folders to local using rsync

## Remote server details
remote_user="username"
remote_host="hpcio2" # hpc2021-io1.hku.hk,hpc2021-io2.hku.hk

## Define folders to sync (absolute paths)
## Format: "local_absolute_path:remote_absolute_path"
folders_to_sync=(
    # "/mnt/m/Reference:/lustre1/g/path_my/Reference"
    # "/mnt/m/WES/DFSP/Annovar:/lustre1/g/path_my/pipeline/somatic_variants_calling/data/DFSP/Annovar"
    "/mnt/m/WES/SARC/BAM:/lustre1/g/path_my/pipeline/somatic_variants_calling/data/SARC/BAM"
    # Add more folders as needed in the same format
)

## Sync each folder
for folder_pair in "${folders_to_sync[@]}"; do
    # Split the pair into local and remote paths
    local_path="${folder_pair%%:*}"
    remote_path="${folder_pair#*:}"

    echo "Syncing ${remote_user}@${remote_host}:${remote_path} to ${local_path}"

    # Create local directory structure
    mkdir -p "${local_path}"

    # Sync the folder contents from remote to local (trailing slash on source copies contents)
    rsync -av --update --progress --inplace --no-perms --no-times \
        "${remote_user}@${remote_host}:${remote_path}/" \
        "${local_path}/"

    echo "-----------------------------------"
done

Sync local folders to remote server

## Remote server details
remote_user="username"
remote_host="hpcio1" # hpc2021-io1.hku.hk

## Define folders to sync (absolute paths)
## Format: "local_absolute_path:remote_absolute_path"
folders_to_sync=(
    "/mnt/f/Reference:/lustre1/g/path_my/Reference"
    # Add more folders as needed in the same format
)

## Sync each folder
for folder_pair in "${folders_to_sync[@]}"; do

    # Split the pair into local and remote paths
    local_path="${folder_pair%%:*}"
    remote_path="${folder_pair#*:}"

    echo "Syncing ${local_path} to ${remote_user}@${remote_host}:${remote_path}"

    # Create remote directory structure
    ssh ${remote_user}@${remote_host} "mkdir -p ${remote_path}"

    # Sync the folder contents (trailing slash on source copies contents)
    rsync -av --update --progress \
        "${local_path}/" \
        "${remote_user}@${remote_host}:${remote_path}/"

    echo "-----------------------------------"
done

File permission

Only the user can copy, execute, and change

chmod 700 /path/to/folder

700 means:

Owner: 7 (read = 4, write = 2, execute = 1; 4+2+1 = 7)
Group: 0 (no permissions)
Others: 0 (no permissions)

All can read and execute, only the owner can change

chmod 755 /path/to/folder

Owner: 7 (read = 4, write = 2, execute = 1; 4+2+1 = 7)
Group: 5 (read = 4, execute = 1; 4+1 = 5)
Others: 5 (read = 4, execute = 1; 4+1 = 5)

Give a user named someuser read, write, and execute permissions on the /path/to/folder directory

setfacl -m u:someuser:rwx /path/to/folder

Verify the Permissions

getfacl /path/to/folder

Remove all ACL entries for a user

setfacl -x u:someuser /path/to/folder

Change owner

sudo chown username:groupname /path/to/folder

File compression

## Backup a directory to a tar.gz file
tar -czvf backup.tar.gz /path/to/directory
## Restore a directory from a tar.gz file
tar -xzvf backup.tar.gz -C /path/to/restore

## Common tar flags:
# -c: create a new archive
# -x: extract files from an archive
# -z: filter the archive through gzip
# -v: verbose output (list files being processed)

Commone tar flags:
- -x = extract
- -f = file
- -v = verbose
- -z = gzip compression
- -j = bzip2 compression
- -J = xz compression
- -c = change to directory
- -C = specify directory
- -p = preserve permissions

## Extract .tar.gz file
tar -xf fuke.tar
tar -xzvf file.tar.gz
tar -xzf file.tgz

## Extract .tar.bz2 file
tar -xjf file.tar.bz2

## Extract .tar.xz file
tar -xJf file.tar.xz

## Extract to specific directory
tar -xzf file.tar.gz -C /path/to/directory

## List contents without extracting
tar -tzf file.tar.gz
tar -tf file.tar

## Extract zip files
unzip file.zip
unzip file.zip -d /path/to/directory

## List contents without extracting
unzip -l file.zip

## Extract .gz file
gunzip file.gz
gunzip -dk file.gz  # keep original file

## Extract .bz2 file
bunzip2 file.bz2

## Extract .xz file
unxz file.xz

## Extract .7z file
7z x file.7z

## Extract .rar file
unrar x file.rar

Common tar flags for compression
- -c = create archive
- -f = file name
- -z = gzip compression (.tar.gz)
- -j = bzip2 compression (.tar.bz2)
- -J = xz compression (.tar.xz)
- -v = verbose output

tar -cf archive.tar file1 file2 directory/

## gzip compressed
tar -czf archive.tar.gz file1 file2 directory/

## bzip2 compressed
tar -cjf archive.tar.bz2 file1 file2 directory/

## .tar.xz compressed
tar -cJf archive.tar.xz file1 file2 directory/

## Backup and Restore a Folder
tar -czvf mybackup.tar.gz myfolder
tar -xzvf mybackup.tar.gz

Regular expression

Regular expressions (regex) are patterns used to match character combinations in strings. In Linux, they are commonly used with commands like grep, sed, and awk for searching and manipulating text.

Symbol	Descriptions	Examples	Matched pattern
.	match any characters	A..T	ATGT, AGCT
^	match start of string	^A	ATG, ACC, AGTC
$	match end of string	T$	TCT, GAT, ACGT
*	match up zero or more times the preceding character	AT*	ATC, CATT, TGAC
\	escape a special (or meta) character	\^A	^A, ^ATG
()	group regular expressions	(ATC){3}	ATCATCATC

[]	match any one of a set characters	[AT]	ACG, CTG, ACTCG
[A-Z]	match upper case letter from A to Z	[A-Z]	>GL000226.1
[a-z]	math lower case letter form a to z	[a-z]	/home/user100
[0-9]	match digit from 0-9	[0-9]	>GL000226.1
[^]	match any character except those in the square bracket after carat ^	[^T]	GCCGAG, >GL000226.1

*	match up zero or more times the preceding character	AT*CG	ACG, ATCG, ATTCG
+	match up one or more times the preceding character	AT+CG	ATCG, ATTCG, ATTTCG
?	match up zero or one times the preceding character	AT?CG	ATCG,ACG
{N}	preceding character is matched exactly N times	AT{3}CG	ATTTCG
{N,}	preceding character is matched at least N times or more	AT{3,}CG	ATTTCG, ATTTTCG
{N,M}	preceding character is matched at least N times but not more than M times	AT{2,3}CG	ATTCG, ATTTCG

Text file manipulation

Filtering

grep Globally search a Regular Expression and Print: is a command-line utility for searching plain-text data sets for lines that match a specified pattern.

grep "pattern" file.txt          # Search for "pattern" in file.txt
grep -i "pattern" file.txt       # Case-insensitive search
grep -n "pattern" file.txt       # Show line numbers
grep -v "pattern" file.txt       # Invert the sense of matching, to select non-matching lines

cut is used to extract sections from each line of input: - Extract columns or fields from text files - Specify delimiter for fields - Useful for processing structured data like CSV or TSV files - Works line by line - Column extraction based on delimiter or character position

cut -f1 file.txt             # Extract first column from a tab-delimited file
cut -d',' -f1,3 file.csv     # Extract first and third columns from a comma-separated file
cut -c1-5 file.txt           # Extract characters from position 1 to 5

# Piping and combining commands
grep "pattern" file.txt | cut -f1,3 | sort | uniq

Sort

sort file.txt                 # Sort lines in file.txt alphabetically
sort -n file.txt              # Sort lines numerically
sort -r file.txt              # Sort lines in reverse order
sort -u file.txt              # Sort and remove duplicate lines
sort -k2 file.txt             # Sort by the second column

Duplicates

uniq file.txt                 # Remove duplicate lines (requires sorted input)
uniq -c file.txt              # Count occurrences of each line
sort file.txt | uniq -c | sort -nr  # Count and sort by frequency

Merge

paste file1.txt file2.txt > combined.txt  # Join files side by side

Find & replace

sed is a stream editor used to perform basic text transformations: * substituction, deletion, insertion, and more * works line by line, applying patterns and edits

## Remove all spaces from a string
echo "a b c" | sed 's/ //g'
#> "abc"
## Remove all occurences of "foo" with "bar"
echo "foo baz foo" | sed 's/foo/bar/g'
#> "bar baz bar"

## Delete lines containing a "pattern" pattern from file.txt
sed '/pattern/d' file.txt

## Print only lines matching a "pattern" pattern from file.txt
sed -n '/pattern/p' file.txt

## Print lines 2 to 5 of a file
sed -n '2,5p' file.txt

Data processing

awk used for pattern scanning and processing. * More powerful than cut for data extraction and reporting * Process text as fileds and records, support arithmetic and logic
awk is a powerful text-processing tool and programming language used for manipulating and analyzing data
Search files for lines (or other units of text) that contain certain patterns and perform specified actions on those lines
By default, awk prints every line of data from the specified file
Basic syntax: awk '<pattern> {<action>}' <file>
regular expression awk '/pattern/ {print $1}' file.txt will print the first column of lines that match the pattern
Build in variables:
- $0: entire row of input
- $1, $2, $3, …: n-th column of inut
- NR: the number of the current line (row) being processed.
- NF: the number of fields (columns) in the current line.
- SNF: the last field (column) in the current row.
- FS: Field separator (default is whitespace).
- OFS: Output field separator (default is whitespace).
- ORS: Output record separator (default is newline).
- BEGIN and END: Special blocks that allow you to specify actions to be performed before processing any lines and after processing all lines, respectively.
Operators
- > | >=: greater than or greater than or equal to
- < | <=: less than or less than or equal to
- ==: equal to
- !=: not equal to
- &&: And (reports “true” if both conditions are true)
- ||: Or (reports “true” if either condition, or both, are true)
- !: Not (negates the condition, reports “true” if the condition is false)

awk '/a/{print $1}' file.txt  # Print the first column of lines that contain the letter "a"

## Print the first column of a file
awk '{print $1}' file.txt

## Sum values in the first column
awk '{sum += $1} END {print sum}' file.txt

## Print lines where the third column is greater than 100
awk '$3 > 100' file.txt

## Print the lines that match a pattern "error"
awk '/error/' file.txt

## Print the number of lines in a file
awk 'END {print NR}' file.txt

Delimiters
- -F FS:
  - FS: input field separator (default is whitespace)
- -v OFS:
  - OFS: output field separator (default is whitespace)

awk -F ',' 'NR==1 || $3>10 {print $1, $2, $3}' file.csv  # Print the first three columns of a CSV file where the third column is greater than 10, including the header row

awk -F ',' -v OFS='\t' 'NR==1 || $3>10 {print $1, $2, $3}' file.csv

Syntax with BEGIN and END blocks
- awk 'BEGIN {<action1>} <pattern> {<action2>} END {<action3>}' <file>
- BEGIN {<actioin1>}: the BEGIN block is executed before any input lines are processed. It is often used to initialize variables, set field separators and setting up the environment.
- <pattern> {<action2>}: This is the main processing block for each input line. For each line in the input file, if the line matches the specified pattern, the associated action is executed. If no pattern is specified, the action is applied to all lines.
- END {<action3>}: The END block is executed after all input lines have been processed. It is typically used for final calculations, printing summaries, or performing cleanup tasks.
Arithmetic operations
- +: addition
- -: subtraction
- *: multiplication
- /: division
- ^: exponentiation
- %: remainder
- ++: increment
- --: decrement
- +=: add and assign
- -=: subtract and assign

Search anything

find is a powerful command to search for files and directories based on various criteria: * Search by name, type, size, modification time, permissions, etc. * Execute actions on found items (delete, move, etc.) * Supports complex expressions with logical operators

## Find all .txt files in current directory and subdirectories
find . -name "*.txt"

## Find files/folders by name
find /path/to/search -name "pattern"

## Find files larger than 100MB
find /path/to/search -type f -size +100M

## Find files in pattern and list details
find ./Clean -name "*Clean.fastq.gz" -type f -exec ls -lh {} \;

Redirection

Redirection is a way to direct the input and output of a command to a file or another command. It allows you to save the output of a command to a file, or to use the output of one command as the input for another command.

## Redirect output to a file
command > output.txt

## Append output to a file
command >> output.txt

Standard streams in Linux: * stdin (standard input): The default input stream for a command, 0 * stdout (standard output): The default output stream for a command, 1 * stderr (standard error): The default error output stream for a command, 2

--- title: Linux date: 2022-10-09 published-title: Created date-modified: last-modified title-block-banner: "#212529" subtitle: "Commonly used Linux commands" # toc: true # toc-location: left toc-title: "Contents" execute: eval: false format: html: code-tools: source: true toggle: true code-fold: false toc: true number-sections: false --- ## Prompt ```bash # mac PROMPT='%F{cyan}%n%F{white} @%F{green} %m:%F{yellow} %~%f %F{magenta}❯%f ' # Linux PS1='\[\e[36m\]\u\[\e[37m\] @\[\e[32m\]\h:\[\e[33m\]\w\[\e[0m\]\n\[\e[35m\]❯\[\e[0m\] ' ``` ## Vim ### Modes - Vim text editor has three main modes: - `Esc`, back to normal mode - `i`, insert mode - `v`, select mode | Command | What it does | Most common follow-up | |---------|-----------------------------|-----------------------| | i | Insert before cursor | typing text | | a | Append after cursor | typing text | | I | Insert at beginning of line | — | | A | Append at end of line | — | | o | Open new line below | typing | | O | Open new line above | typing | | Esc | Back to Normal mode | — | | v | Visual mode | move + operator | | V | Visual Line mode | — | | Ctrl-v | Visual Block mode | — | ### Navigation | Command | Movement | Common combos | |-----------------|--------------------------------|-----------------------| | h j k l | ← ↓ ↑ → | — | | w / b | forward/backward by word start | ciw, daw | | e / ge | forward/backward to word end | — | | W / B | forward/backward by big word | (ignores punctuation) | | 0 | Start of line | — | | \^ | First non-blank character | — | | \$ | End of line | — | | gg | Top of file | — | | G | Bottom of file | 50G → line 50 | | Ctrl-u / Ctrl-d | Half page up/down | — | | Ctrl-f / Ctrl-b | Full page down/up | — | ### Editing | Command | Meaning | examples | |-----------------|-------------------------------|------------------------| | x | Delete character under cursor | — | | dd | Delete current line | 5dd = delete 5 lines | | yy | Yank (copy) current line | p to paste | | p / P | Paste after/before cursor | — | | u | Undo | (most used command after Esc) | | Ctrl-r | Redo | — | | . | Repeat last change/command | (extremely powerful) | | dw / d\$ | Delete to end of word / end of line | diw, daw | | ciw / caw | Change (delete + insert) inside/around word | most common text object usage | | "ci"" / ci( / ci{" | Change inside quotes/parentheses/braces | extremely common in code | ### Search & Replace | Command | Meaning | |----------------|-----------------------------------------------| | /pattern | Search forward | | ?pattern | Search backward | | n / N | Next / previous match | | \* / \# | Search forward/backward for word under cursor | | :%s/old/new/g | Replace all occurrences (whole file) | | :%s/old/new/gc | Replace with confirmation | ### Saving & Exiting | Command | Meaning | |-----------------|-------------------------------------------------| | :w | Save (write) | | :q | Quit (only if no changes) | | :q! | Quit without saving | | :wq or :x or ZZ | Save and quit | | :w !sudo tee % | Save file needing root (very common workaround) | ## Nano 1. Open a file with: `nano filename.txt` 2. Use the arrow keys to navigate. 3. Save the changes with `ctrl + o`, 4. Confirm the filename by pressing `Enter`. 5. Exit nano with `ctrl + x`. ### Searching Text - To find words or phrases within a file: 1. Initiate search with `ctrl + w`. 2. Type the search term and press `Enter`. 3. Exit search mode with `ctrl + c`. ### Customizing Nano Enhance Nano's functionality: ``` bash nano -miA file.txt ``` - `-m`: Enable mouse support. - `-i`: Auto-indent new lines. - `-A`: Enable syntax highlighting. ## Jobs ### OpenPBS OpenPBS (Portable Batch System) is an open-source workload manager and job scheduler used in High-Performance Computing (HPC) environments to automate and optimize the execution of tasks across and clouds. **PBS job script** ``` bash #!/bin/bash #PBS -N job_name # Job name #PBS -l nodes=1:ppn=4 # Number of nodes and processors per node #PBS -l walltime=01:00:00 # Walltime (hh:mm:ss) #PBS -l mem=8gb # Memory per node #PBS -j oe # Combine stdout and stderr #PBS -M user@example.com # Email address for notifications #PBS -m abe # Send email on (a) abort, (b) begin, (e) end #PBS -q batch # Queue named #PBS -V # Export all environment variables to the job ## example cd $PBS_O_WORKDIR # Change to the directory where the job was submitted module load python/3.8.5 # Load necessary modules python my_script.py # Command to run your program ``` **PBS environment variables** ``` bash $PBS_O_WORKDIR # The directory from which the job was submitted $PBS_O_HOME # The home directory of the user who submitted the job $PBS_JOBDIR # The working directory for the job on the compute node $TMPDIR # The temporary directory for the job on the compute node $PBS_JOBID # The unique identifier assigned to the job $PBS_NODEFILE # The file containing the list of nodes allocated to the job $PBS_NUM_NODES # The number of nodes allocated to the job $PBS_NUM_PPN # The number of processors per node allocated to the job $PBS_QUEUE # The name of the queue to which the job was submitted $PBS_JOBNAME # The name of the job as specified in the job script ``` **Submit job scripts** ``` bash ## Submit a job qsub job_script.pbs ## Submit a job with environment variables. qsub -v sample="$sample" -M "$EMAIL" pbs_1_cutadapt.pbs ``` **Check job status** ``` bash ## List finished job details in summary qstat -x job_id ## List finished job details in full details qstat -H job_id qstat -x -f job_id qstat -x -f job_id | grepl "exec_host" ## List all jobs for user guorui qstat -u guorui -a ## Find out all compute nodes allocated to a job for job_id in $(find . -name "log.*.e*" -printf "%f\n" | sed 's/.*\.e$[0-9]*$$/\1/' | sort -n | uniq); do echo -n "Job ID: $job_id: "; qstat -x -f $job_id | grep exec_host done ## Delete a job qdel job_id ## Delete all jobs for user guorui qselect -u guorui | xargs qdel ## check all queues qstat -Qf | less for q in cgs small medium large cgs_queue; do echo "=== $q ===" qstat -Qf $q | grep -E "resources_max.(mem|ncpus|walltime)" done ## Hold a job qhold job_id ## Release a held job qrls job_id # See your groups groups # Check queue ACLs qstat -Q -f | grep -E "^Queue:|acl_users|acl_groups" ``` **Stat interactive job** ``` bash ## Start and interactive job with 16 processors, 64GB memory, 24 hours walltime in cgs_queue qsub -I -q cgs_queue -l nodes=hpcf3-c01:ppn=16,walltime=12:00:00,mem=64gb qsub -I -q cgs_queue -l nodes=hpcf3-c02:ppn=16,walltime=12:00:00,mem=64gb qsub -I -q cgs_queue -l nodes=hpcf3-c03:ppn=16,walltime=12:00:00,mem=64gb ``` **Check the permission of queue** ``` bash qstat -Q cgs -f # Queue: cgs # queue_type = Execution # total_jobs = 16 # state_count = Transit:0 Queued:10 Held:0 Waiting:0 Running:6 Exiting:0 Begu # n:0 # acl_user_enable = True # acl_users = bioinfo,emiklin,irisxu,itsupport,julia,junyanzhong,manzhai,nicklin, # rubyzhou,totochu # resources_max.mem = 1024gb # resources_max.ncpus = 64 # resources_max.nodect = 1 # resources_max.walltime = 1440:00:00 # resources_default.mem = 1gb # resources_default.walltime = 24:00:00 # default_chunk.Qlist = cgs # resources_assigned.mem = 3019898880kb # resources_assigned.mpiprocs = 168 # resources_assigned.ncpus = 253 # resources_assigned.nodect = 6 # enabled = True # started = True ``` - `-Q`: Show queue information (not job information) - `-cgs`: The specific queue name to query - `-f`: Full/detailed output showing all queue attributes ```bash # Delete all held jobs for the current user qstat -u $USER | awk '$10=="H" {print $1}' | xargs -r qdel # Delete all running jobs for the current user qstat -u $USER | awk '$10=="R" {print $1}' | xargs -r qdel # Delete every job whose numeric ID is greater than a threshold. Preview first, then delete: qselect -u $USER | awk -F. '$1+0 > 158899' qselect -u $USER | awk -F. '$1+0 > 158899' | xargs -r qdel ``` ### SLURM SLURM (Simple Linux Utility for Resource Management) is an open-source workload manager and job scheduler widely used in HPC clusters to allocate resources and run jobs across compute nodes. **SLURM job script** ``` bash #!/bin/bash #SBATCH --job-name=job_name # Job name #SBATCH --nodes=1 # Number of nodes #SBATCH --ntasks=1 # Number of tasks #SBATCH --cpus-per-task=4 # Processors per task #SBATCH --time=01:00:00 # Walltime (hh:mm:ss) #SBATCH --mem=8G # Memory per node #SBATCH --output=log.%x.%j.out # Stdout (%x = job name, %j = job ID) #SBATCH --error=log.%x.%j.err # Stderr #SBATCH --mail-user=user@example.com # Email address for notifications #SBATCH --mail-type=BEGIN,END,FAIL # Send email on begin, end, fail #SBATCH --partition=batch # Partition (queue) name #SBATCH --export=ALL # Export all environment variables to the job ## example cd $SLURM_SUBMIT_DIR # Change to the directory where the job was submitted module load python/3.8.5 # Load necessary modules python my_script.py # Command to run your program ``` **SLURM environment variables** ``` bash $SLURM_SUBMIT_DIR # The directory from which the job was submitted $SLURM_JOB_ID # The unique identifier assigned to the job $SLURM_JOB_NAME # The name of the job $SLURM_JOB_NODELIST # The list of nodes allocated to the job $SLURM_JOB_NUM_NODES # The number of nodes allocated to the job $SLURM_NTASKS # The number of tasks $SLURM_CPUS_PER_TASK # The number of processors per task $SLURM_JOB_PARTITION # The partition to which the job was submitted $SLURM_ARRAY_TASK_ID # The index of the task in a job array $TMPDIR # The temporary directory for the job on the compute node ``` **Submit job scripts** ``` bash ## Submit a job sbatch job_script.sh ## Submit a job with environment variables sbatch --export=ALL,sample="$sample" --mail-user="$EMAIL" slurm_1_cutadapt.sh ## Submit a job array (tasks 1-10, at most 4 running at once) sbatch --array=1-10%4 job_script.sh ``` **Check job status** ``` bash ## List all jobs for user guorui squeue -u guorui ## Show details of a pending/running job scontrol show job job_id ## List finished job details in summary sacct -j job_id --format=JobID,JobName,Partition,State,Elapsed,MaxRSS,ExitCode ## List all jobs of user guorui since a date sacct -u guorui -S 2026-01-01 --format=JobID,JobName,State,Elapsed,NodeList ## Delete a job scancel job_id ## Delete all jobs for user guorui scancel -u guorui ## Delete all pending / running jobs for the current user scancel -u $USER --state=PENDING scancel -u $USER --state=RUNNING ## Hold a job scontrol hold job_id ## Release a held job scontrol release job_id ## Check all partitions and nodes sinfo sinfo -N -l ## Show partition limits (memory, CPUs, walltime) scontrol show partition ## See your account and associations (allowed partitions/QOS) sacctmgr show assoc user=$USER format=User,Account,Partition,QOS ``` **Start interactive job** ``` bash ## Start an interactive job with 16 processors, 64GB memory, 12 hours walltime srun --partition=batch --nodes=1 --cpus-per-task=16 --mem=64G --time=12:00:00 --pty bash ## Request a specific node srun --partition=batch --nodelist=hpcf3-c01 --cpus-per-task=16 --mem=64G --time=12:00:00 --pty bash ## Allocate resources first, then attach salloc --partition=batch --cpus-per-task=16 --mem=64G --time=12:00:00 ``` **Check the permission of partition** ``` bash scontrol show partition batch # PartitionName=batch # AllowGroups=ALL AllowAccounts=ALL AllowQos=ALL # Default=YES MaxTime=60-00:00:00 State=UP # TotalCPUs=64 TotalNodes=1 # DefMemPerNode=1024 MaxMemPerNode=1048576 ``` - `scontrol show partition`: Show partition (queue) attributes - `batch`: The specific partition name to query - `AllowGroups/AllowAccounts`: Who is permitted to submit to the partition **PBS to SLURM cheat sheet** | Task | OpenPBS | SLURM | |--------------------|--------------------------|--------------------------------| | Submit job | `qsub script.pbs` | `sbatch script.sh` | | Job status | `qstat -u $USER` | `squeue -u $USER` | | Finished job info | `qstat -x -f job_id` | `sacct -j job_id` | | Delete job | `qdel job_id` | `scancel job_id` | | Hold / release | `qhold` / `qrls` | `scontrol hold` / `release` | | Interactive job | `qsub -I` | `srun --pty bash` / `salloc` | | Queue info | `qstat -Q` | `sinfo` | | Working directory | `$PBS_O_WORKDIR` | `$SLURM_SUBMIT_DIR` | | Job ID | `$PBS_JOBID` | `$SLURM_JOB_ID` | | Node list | `$PBS_NODEFILE` | `$SLURM_JOB_NODELIST` | ## WSL - System info ``` bash ## Install the neofetch package sudo apt install neofetch -y ## Show system info neofetch --memory_unit gib ``` - Change sudo password In the windows terminal, run the following command to change the sudo password: ``` bash wsl -u root ## Input the new password passwd <username> ``` - WSL configuration - WSL global config: `C:\Users\<UserName>\.wslconfig` - WSL distro config: `\\wsl.localhost\Ubuntu-24.04\etc\wsl.conf` ## Tmux - Tmux Configuration Consider adding these aliases to your shell configuration for quick tmux access. ::: code-block ``` bash # Terminal aliases for tmux alias t="tmux" alias ta="t a -t" alias tls="t ls" alias tn="t new -s" alias tk="t kill-session -t" alias tks="t kill-server" ``` ::: ::: callout-note ### 🔧 Custom Prefix Key Change the default prefix in `.tmux.conf`: ``` bash set-option -g prefix C-a bind-key C-a send-prefix ``` ::: - Session Management | **Action** | **Command/Shortcut** | |--------------------------|-----------------------------------------| | List sessions | `tmux list-sessions` | | Attach to session | `tmux attach-session -t target-session` | | Switch between sessions | `Ctrl + A + s` | | Switch to latest session | `Ctrl + A + l` | | Detach from session | `Ctrl + A + d` | - Window Management | **Action** | **Shortcut** | |---------------------------|----------------| | List all windows | `Ctrl + A + w` | | Rename current window | `Ctrl + A + ,` | | Switch to next window | `Ctrl + A + →` | | Switch to previous window | `Ctrl + A + ←` | | Create new window | `Ctrl + A + c` | | Kill current window | `Ctrl + A + q` | - Pane Management | **Action** | **Shortcut** | |----------------------|----------------| | Switch to pane above | `Shift + ↑` | | Switch to pane below | `Shift + ↓` | | Switch to pane left | `Shift + ←` | | Switch to pane right | `Shift + →` | | Kill current pane | `Ctrl + A + x` | - Copy Mode | **Action** | **Shortcut** | |-------------------|---------------------------| | Enter copy mode | Drag mouse to select text | | Paste copied text | `Ctrl + A + ]` | ------------------------------------------------------------------------ ## Screen ### Config ```bash ## ~/.screenrc # tells GNU Screen how to handle terminal initialization (ti) and termination (te) sequences for xterm and its variants (like xterm-color, gnome-terminal, etc.). termcapinfo xterm* ti@:te@ # binds a specific keyboard key to send a command or string bindkey -k kI stuff bindkey -k kD stuff ``` ::: callout-tip ### 💡 Screen Aliases Add these aliases to your shell configuration for easier screen management. ::: ::: code-block ``` bash # Terminal aliases for screen alias s="screen" # start a screen session alias ss="s -S" # start a named screen session alias sr="s -r" # reattach to a screen session alias sls="s -ls" # list current running screen sessions ``` ::: - Basic Commands | **Action** | **Command** | |-----------------------|--------------------------| | Start screen session | `screen` | | Start named session | `screen -S session_name` | | Reattach to session | `screen -r` | | List running sessions | `screen -ls` | - Window Management | **Action** | **Shortcut** | |------------------------|-----------------------| | Create new window | `Ctrl + A + C` | | Kill current window | `Ctrl + A + K` | | List all windows | `Ctrl + A + W` | | Go to window 0-9 | `Ctrl + A + 0-9` | | Go to next window | `Ctrl + A + N` | | Toggle between windows | `Ctrl + A + Ctrl + A` | | Rename current window | `Ctrl + A + A` | - Region Management | **Action** | **Shortcut** | |-------------------------------|------------------| | Split horizontally | `Ctrl + A + S` | | Split vertically | `Ctrl + A + \|` | | Switch between regions | `Ctrl + A + Tab` | | Close all regions but current | `Ctrl + A + Q` | | Close current region | `Ctrl + A + X` | - Session & Copy Mode | **Action** | **Shortcut** | |---------------------|-----------------------| | Detach from session | `Ctrl + A + D` | | Start copy mode | `Ctrl + A + [` | | Paste copied text | `Ctrl + A + ]` | | Show help | `Ctrl + A + ?` | | Quit screen | `Ctrl + A + Ctrl + \` | ------------------------------------------------------------------------ ::: callout-note ### Additional Resources - **VSCode**: [Official Keyboard Shortcuts Reference](https://code.visualstudio.com/docs/getstarted/keybindings) - **Vim**: [Vim Cheat Sheet](https://vim.rtorr.com/) - **Tmux**: [Tmux Cheat Sheet](https://tmuxcheatsheet.com/) - **Terminal**: [Bash Reference Manual](https://www.gnu.org/software/bash/manual/) ::: ## Linux - Linux is a family of open-source Unix-like operating systems based on the Linux kernel. It is widely used for servers, desktops, and embedded systems. Linux provides a powerful command-line interface (CLI) that allows users to interact with the system and perform various tasks. - Linux commands are executed in a terminal or shell, and they can be combined to create powerful scripts and automate tasks. The Linux file system is organized in a hierarchical structure, with directories and files that can be manipulated using various commands. - In bioinformatics, Linux is commonly used for data analysis, bioinformatics pipelines, and high-performance computing (HPC) tasks. Many bioinformatics tools and software are designed to run on Linux systems, making it a preferred choice for researchers in the field. ### File & director ``` bash ### Working with files cp file1.txt file2.txt # Copy file1.txt to file2.txt mv file1.txt file2.txt # Move or rename file1.txt to file2.txt rm file1.txt # Remove file1.txt gzip file1.txt # Compress file1.txt to file1.txt.gz gunzip file1.txt.gz # Decompress file1.txt.gz to file1.txt ``` ``` bash ### Working with directories mkdir my_directory # Create a new directory named my_directory cd my_directory # Change into the directory my_directory ls # List files and directories in the current directory ls -l # List files and directories with detailed information ls -a # List all files, including hidden files ls -lh # List files with human-readable sizes rm -r my_directory # Remove the directory my_directory and its contents ``` ``` bash ### View file contents cat file.txt # Display the entire contents of file.txt head file.txt # Display the first 10 lines of file.txt tail file.txt # Display the last 10 lines of file.txt wc -l file.txt # Count the number of lines in file.txt ``` ``` bash less file.txt # View file.txt with scrolling and searching capabilities ## Options for less: # -n: Show line numbers # +F: Follow the file (like tail -f) # -S: Don't wrap long lines # -U: Don't show control characters grep "\*" file.txt | less -S # Search for lines containing "*" in file.txt and view them with less, without line wrapping ``` ### File transfer - Transfering data efficiently between local and remote systems: - `wget`: Download files directly from the internet to your local machine or server. - `scp`: Securely copy files between local and remote systems over SSH. - `rsync`: Synchronize files and directories efficiently, with options for selective copying and deletion. ``` bash ## Basic command rsync -av /source/folder/ /destination/folder/ ## Sync source to destination rsync -av --delete /source/folder/ /destination/folder/ ## If you need two-way sync, run both directions rsync -av --delete /folder1/ /folder2/ rsync -av --delete /folder2/ /folder1/ ## sync multiple folders at once rsync -avP /source/{folder1,folder2,folder3} /destination/ rsync -avP --include='v*/' --include='Scripts/' --exclude='*' /groups/fcoi/guorui/analysis/ /path/to/destination/ ## Network sync rsync -av --delete /local/folder/ user@remote:/remote/folder/ ## Copy only new files (files that don't exist in the destination) rsync -av --ignore-existing /source/folder/ /destination/folder/ ## Copy only if source is newer rsync -av --update /source/folder/ /destination/folder/ ``` ::: callout-note - `--archive`: Preserves permissions, timestamps, symbolic links, etc. - `--verbose`: Shows progress - `--delete`: Removes files in destination that don't exist in source - `--dry-run`: Preview changes without actually syncing - `--update`: Only copy newer file - `--ignore-existing`: Skips files that already exist in destination (regardless of age) - `--update`: Only copies if source file is newer than destination - `--no-perms`: Don't preserve Unix permissions - `--no-times`: Don't preserve modification times - `--inplace`: Write directly without temp files ::: - Sync remote server folders to local using rsync ``` bash ## Remote server details remote_user="username" remote_host="hpcio2" # hpc2021-io1.hku.hk,hpc2021-io2.hku.hk ## Define folders to sync (absolute paths) ## Format: "local_absolute_path:remote_absolute_path" folders_to_sync=( # "/mnt/m/Reference:/lustre1/g/path_my/Reference" # "/mnt/m/WES/DFSP/Annovar:/lustre1/g/path_my/pipeline/somatic_variants_calling/data/DFSP/Annovar" "/mnt/m/WES/SARC/BAM:/lustre1/g/path_my/pipeline/somatic_variants_calling/data/SARC/BAM" # Add more folders as needed in the same format ) ## Sync each folder for folder_pair in "${folders_to_sync[@]}"; do # Split the pair into local and remote paths local_path="${folder_pair%%:*}" remote_path="${folder_pair#*:}" echo "Syncing ${remote_user}@${remote_host}:${remote_path} to ${local_path}" # Create local directory structure mkdir -p "${local_path}" # Sync the folder contents from remote to local (trailing slash on source copies contents) rsync -av --update --progress --inplace --no-perms --no-times \ "${remote_user}@${remote_host}:${remote_path}/" \ "${local_path}/" echo "-----------------------------------" done ``` - Sync local folders to remote server ``` bash ## Remote server details remote_user="username" remote_host="hpcio1" # hpc2021-io1.hku.hk ## Define folders to sync (absolute paths) ## Format: "local_absolute_path:remote_absolute_path" folders_to_sync=( "/mnt/f/Reference:/lustre1/g/path_my/Reference" # Add more folders as needed in the same format ) ## Sync each folder for folder_pair in "${folders_to_sync[@]}"; do # Split the pair into local and remote paths local_path="${folder_pair%%:*}" remote_path="${folder_pair#*:}" echo "Syncing ${local_path} to ${remote_user}@${remote_host}:${remote_path}" # Create remote directory structure ssh ${remote_user}@${remote_host} "mkdir -p ${remote_path}" # Sync the folder contents (trailing slash on source copies contents) rsync -av --update --progress \ "${local_path}/" \ "${remote_user}@${remote_host}:${remote_path}/" echo "-----------------------------------" done ``` ### File permission Only the user can copy, execute, and change ``` bash chmod 700 /path/to/folder ``` `700` means: - Owner: 7 (read = 4, write = 2, execute = 1; 4+2+1 = 7) - Group: 0 (no permissions) - Others: 0 (no permissions) All can read and execute, only the owner can change ``` bash chmod 755 /path/to/folder ``` - Owner: 7 (read = 4, write = 2, execute = 1; 4+2+1 = 7) - Group: 5 (read = 4, execute = 1; 4+1 = 5) - Others: 5 (read = 4, execute = 1; 4+1 = 5) Give a user named someuser read, write, and execute permissions on the /path/to/folder directory ``` bash setfacl -m u:someuser:rwx /path/to/folder ``` Verify the Permissions ``` bash getfacl /path/to/folder ``` Remove all ACL entries for a user ``` bash setfacl -x u:someuser /path/to/folder ``` Change owner ``` bash sudo chown username:groupname /path/to/folder ``` ### File compression ``` bash ## Backup a directory to a tar.gz file tar -czvf backup.tar.gz /path/to/directory ## Restore a directory from a tar.gz file tar -xzvf backup.tar.gz -C /path/to/restore ## Common tar flags: # -c: create a new archive # -x: extract files from an archive # -z: filter the archive through gzip # -v: verbose output (list files being processed) ``` - Commone tar flags: - `-x` = extract - `-f` = file - `-v` = verbose - `-z` = gzip compression - `-j` = bzip2 compression - `-J` = xz compression - `-c` = change to directory - `-C` = specify directory - `-p` = preserve permissions ``` bash ## Extract .tar.gz file tar -xf fuke.tar tar -xzvf file.tar.gz tar -xzf file.tgz ## Extract .tar.bz2 file tar -xjf file.tar.bz2 ## Extract .tar.xz file tar -xJf file.tar.xz ## Extract to specific directory tar -xzf file.tar.gz -C /path/to/directory ## List contents without extracting tar -tzf file.tar.gz tar -tf file.tar ``` ``` bash ## Extract zip files unzip file.zip unzip file.zip -d /path/to/directory ## List contents without extracting unzip -l file.zip ## Extract .gz file gunzip file.gz gunzip -dk file.gz # keep original file ## Extract .bz2 file bunzip2 file.bz2 ## Extract .xz file unxz file.xz ## Extract .7z file 7z x file.7z ## Extract .rar file unrar x file.rar ``` - Common tar flags for compression - `-c` = create archive - `-f` = file name - `-z` = gzip compression (.tar.gz) - `-j` = bzip2 compression (.tar.bz2) - `-J` = xz compression (.tar.xz) - `-v` = verbose output ``` bash tar -cf archive.tar file1 file2 directory/ ## gzip compressed tar -czf archive.tar.gz file1 file2 directory/ ## bzip2 compressed tar -cjf archive.tar.bz2 file1 file2 directory/ ## .tar.xz compressed tar -cJf archive.tar.xz file1 file2 directory/ ``` ``` bash ## Backup and Restore a Folder tar -czvf mybackup.tar.gz myfolder tar -xzvf mybackup.tar.gz ``` ### Regular expression Regular expressions (regex) are patterns used to match character combinations in strings. In Linux, they are commonly used with commands like `grep`, `sed`, and `awk` for searching and manipulating text. | Symbol | Descriptions | Examples | Matched pattern | |----------|-----------------------|----------------|------------------------| | . | match any characters | A..T | A[**TG**]{.underline}T, A[**GC**]{.underline}T | | \^ | match start of string | \^A | [**A**]{.underline}TG, [**A**]{.underline}CC, [**A**]{.underline}GTC | | \$ | match end of string | T\$ | TC[**T**]{.underline}, GA[**T**]{.underline}, ACG[**T**]{.underline} | | \* | match up zero or more times the preceding character | AT\* | [**AT**]{.underline}C, C[**ATT**]{.underline}, TG[**A**]{.underline}C | | \ | escape a special (or meta) character | \\\^A | [**\^A**]{.underline}, [**\^A**]{.underline}TG | | () | group regular expressions | (ATC){3} | [**ATCATCATC**]{.underline} | | | | | | | \[\] | match any one of a set characters | \[AT\] | [**A**]{.underline}CG, C[**T**]{.underline}G, [**A**]{.underline}C[**T**]{.underline}CG | | \[A-Z\] | match [**upper**]{.underline} case letter from A to Z | \[A-Z\] | \>GL000226.1 | | \[a-z\] | math [**lower**]{.underline} case letter form a to z | \[a-z\] | /[**home**]{.underline}/[**user**]{.underline}100 | | \[0-9\] | match digit from 0-9 | \[0-9\] | \>GL[**000226**]{.underline}.[**1**]{.underline} | | \[\^\] | match any character except those in the square bracket after carat \^ | \[\^T\] | [**GCCGAG**]{.underline}, [**\>GL000226.1**]{.underline} | | | | | | | \* | match up [**zero or more times**]{.underline} the preceding character | AT\*CG | ACG, A[**T**]{.underline}CG, A[**TT**]{.underline}CG | | \+ | match up [**one or more times**]{.underline} the preceding character | AT+CG | A[**T**]{.underline}CG, A[**TT**]{.underline}CG, A[**TTT**]{.underline}CG | | ? | match up [**zero or one times**]{.underline} the preceding character | AT?CG | A[**T**]{.underline}CG,ACG | | {N} | preceding character is matched [**exactly N times**]{.underline} | AT{3}CG | A[**TTT**]{.underline}CG | | {N,} | preceding character is matched [**at least N times or more**]{.underline} | AT{3,}CG | A[**TTT**]{.underline}CG, A[**TTTT**]{.underline}CG | | {N,M} | preceding character is matched [**at least N times but not more than M times**]{.underline} | AT{2,3}CG | A[**TT**]{.underline}CG, A[**TTT**]{.underline}CG | ### Text file manipulation #### Filtering `grep` Globally search a Regular Expression and Print: is a command-line utility for searching plain-text data sets for lines that match a specified pattern. ``` bash grep "pattern" file.txt # Search for "pattern" in file.txt grep -i "pattern" file.txt # Case-insensitive search grep -n "pattern" file.txt # Show line numbers grep -v "pattern" file.txt # Invert the sense of matching, to select non-matching lines ``` `cut` is used to extract sections from each line of input: - Extract columns or fields from text files - Specify delimiter for fields - Useful for processing structured data like CSV or TSV files - Works line by line - Column extraction based on delimiter or character position ``` bash cut -f1 file.txt # Extract first column from a tab-delimited file cut -d',' -f1,3 file.csv # Extract first and third columns from a comma-separated file cut -c1-5 file.txt # Extract characters from position 1 to 5 ``` ``` bash # Piping and combining commands grep "pattern" file.txt | cut -f1,3 | sort | uniq ``` #### Sort ```bash sort file.txt # Sort lines in file.txt alphabetically sort -n file.txt # Sort lines numerically sort -r file.txt # Sort lines in reverse order sort -u file.txt # Sort and remove duplicate lines sort -k2 file.txt # Sort by the second column ``` #### Duplicates ```bash uniq file.txt # Remove duplicate lines (requires sorted input) uniq -c file.txt # Count occurrences of each line sort file.txt | uniq -c | sort -nr # Count and sort by frequency ``` #### Merge ```bash paste file1.txt file2.txt > combined.txt # Join files side by side ``` #### Find & replace `sed` is a stream editor used to perform basic text transformations: \* substituction, deletion, insertion, and more \* works line by line, applying patterns and edits ``` bash ## Remove all spaces from a string echo "a b c" | sed 's/ //g' #> "abc" ## Remove all occurences of "foo" with "bar" echo "foo baz foo" | sed 's/foo/bar/g' #> "bar baz bar" ## Delete lines containing a "pattern" pattern from file.txt sed '/pattern/d' file.txt ## Print only lines matching a "pattern" pattern from file.txt sed -n '/pattern/p' file.txt ## Print lines 2 to 5 of a file sed -n '2,5p' file.txt ``` ### Data processing - `awk` used for pattern scanning and processing. \* More powerful than `cut` for data extraction and reporting \* Process text as fileds and records, support arithmetic and logic - `awk` is a powerful text-processing tool and programming language used for manipulating and analyzing data - Search files for lines (or other units of text) that contain certain patterns and perform specified actions on those lines - By default, awk prints every line of data from the specified file - Basic syntax: `awk '<pattern> {<action>}' <file>` - regular expression `awk '/pattern/ {print $1}' file.txt` will print the first column of lines that match the pattern - Build in variables: - `$0`: entire row of input - `$1`, `$2`, `$3`, ...: n-th column of inut - `NR`: the number of the current line (row) being processed. - `NF`: the number of fields (columns) in the current line. - `SNF`: the last field (column) in the current row. - `FS`: Field separator (default is whitespace). - `OFS`: Output field separator (default is whitespace). - `ORS`: Output record separator (default is newline). - `BEGIN` and `END`: Special blocks that allow you to specify actions to be performed before processing any lines and after processing all lines, respectively. - Operators - `> | >=`: greater than or greater than or equal to - `< | <=`: less than or less than or equal to - `==`: equal to - `!=`: not equal to - `&&`: And (reports "true" if both conditions are true) - `||`: Or (reports "true" if either condition, or both, are true) - `!`: Not (negates the condition, reports "true" if the condition is false) ```bash awk '/a/{print $1}' file.txt # Print the first column of lines that contain the letter "a" ## Print the first column of a file awk '{print $1}' file.txt ## Sum values in the first column awk '{sum += $1} END {print sum}' file.txt ## Print lines where the third column is greater than 100 awk '$3 > 100' file.txt ## Print the lines that match a pattern "error" awk '/error/' file.txt ## Print the number of lines in a file awk 'END {print NR}' file.txt ``` - Delimiters - `-F FS`: - `FS`: input field separator (default is whitespace) - `-v OFS`: - `OFS`: output field separator (default is whitespace) ```bash awk -F ',' 'NR==1 || $3>10 {print $1, $2, $3}' file.csv # Print the first three columns of a CSV file where the third column is greater than 10, including the header row awk -F ',' -v OFS='\t' 'NR==1 || $3>10 {print $1, $2, $3}' file.csv ``` - Syntax with `BEGIN` and `END` blocks - `awk 'BEGIN {<action1>} <pattern> {<action2>} END {<action3>}' <file>` - `BEGIN {<actioin1>}`: the `BEGIN` block is executed before any input lines are processed. It is often used to initialize variables, set field separators and setting up the environment. - `<pattern> {<action2>}`: This is the main processing block for each input line. For each line in the input file, if the line matches the specified pattern, the associated action is executed. If no pattern is specified, the action is applied to all lines. - `END {<action3>}`: The `END` block is executed after all input lines have been processed. It is typically used for final calculations, printing summaries, or performing cleanup tasks. - Arithmetic operations - `+`: addition - `-`: subtraction - `*`: multiplication - `/`: division - `^`: exponentiation - `%`: remainder - `++`: increment - `--`: decrement - `+=`: add and assign - `-=`: subtract and assign ### Search anything `find` is a powerful command to search for files and directories based on various criteria: \* Search by name, type, size, modification time, permissions, etc. \* Execute actions on found items (delete, move, etc.) \* Supports complex expressions with logical operators ``` bash ## Find all .txt files in current directory and subdirectories find . -name "*.txt" ## Find files/folders by name find /path/to/search -name "pattern" ## Find files larger than 100MB find /path/to/search -type f -size +100M ## Find files in pattern and list details find ./Clean -name "*Clean.fastq.gz" -type f -exec ls -lh {} \; ``` ### Redirection Redirection is a way to direct the input and output of a command to a file or another command. It allows you to save the output of a command to a file, or to use the output of one command as the input for another command. ``` bash ## Redirect output to a file command > output.txt ## Append output to a file command >> output.txt ``` Standard streams in Linux: \* `stdin` (standard input): The default input stream for a command, 0 \* `stdout` (standard output): The default output stream for a command, 1 \* `stderr` (standard error): The default error output stream for a command, 2