export a recursive directory & file listing to a text file in Linux Bash shell with an SSH command

June 23, 2017 - Reading time: 3 minutes
find is a good utility to obtain (recursively) all the content of a directory. If you want the file (and directory) names with their permissions:
find /home/kparisi -printf "%M %p\n"

You can then use ssh to run this command on the remote server:

ssh kparisi@remote.com 'find /home/kparisi -printf "%M %p\n"'

And finally, if you want to store this in a file on your local server:

ssh kparisi@remote.com 'find /home/kparisi -printf "%M %p\n"' > file 

Alternatively:

tree > fsstruct.txt

for me this format is simpler to read.

It is easy also to use other features of tree:

tree -p > fsstruct.txt

prints file type and permissions before file and it is more accessible when you are reading plain text, which is a file and which is a directory.

tree -h > fsstruct.txt

this prints sizes of files in a human readable format:

tree -ph > fsstruct.txt

Also, it is possible to send output to the file:

tree -ph . -o fsstruct.txt

or create HTML:

tree -Hph . -o tree.htm


ffmpeg - convert flv to mp4 without losing quality

June 15, 2017 - Reading time: ~1 minute
ffmpeg -i filename.flv -c:v libx264 -crf 19 -strict experimental filename.mp4

This command only changes container without reencoding:

ffmpeg -i input.flv -codec copy output.mp4

According to this guide, the proper command to convert FLV to MP4 without any AV recoding is this:
ffmpeg -i input.flv -c copy -copyts output.mp4

It worked perfectly for me in all cases with AVC video and AAC or MP3 audio.


resize an ext4 partition from the command line

May 25, 2017 - Reading time: 6 minutes

You can use fdisk to change your partition table while running.  Refer to Live resizing of an ext4 filesytem on Linux (on The silence of the code blog):

Disclaimer: The following instructions can easily screw your data if you make a mistake.  I was doing this on a VM which I backed up before performing the following actions.  If you lose your data because you didn’t perform a backup don’t come and complain.

...

First: Increase the disk size.

In ESXi this is simple, just increase the size of the virtual disk. Now you have a bigger hard drive but you still need to a) increase the partition size and b) resize the filesystem.

Second: Increase the partition size.

You can use fdisk to change the partition table while running.  The stock Ubuntu install has created 3 partitions: one primary (sda1), one extended (sda2) with a single logical partition (sda5) in it. The extended partition is simply used for swap, so I could easily move it without losing any data.

  1. Delete the primary partition

  2. Delete the extended partition

  3. Create a new primary partition starting at the same sector as the original one just with a bigger size (leave some for swap)

  4. Create a new extended partition with a logical partition in it to hold the swap space

    me@ubuntu:~$ sudo fdisk /dev/sda
    
    Command (m for help): p
    
    Disk /dev/sda: 268.4 GB, 268435456000 bytes
    255 heads, 63 sectors/track, 32635 cylinders, total 524288000 sectors
    Units = sectors of 1 * 512 = 512 bytes
    Sector size (logical/physical): 512 bytes / 512 bytes
    I/O size (minimum/optimal): 512 bytes / 512 bytes
    Disk identifier: 0x000e49fa
    
       Device Boot      Start         End      Blocks   Id  System
    /dev/sda1   *        2048   192940031    96468992   83  Linux
    /dev/sda2       192942078   209713151     8385537    5  Extended
    /dev/sda5       192942080   209713151     8385536   82  Linux swap / Solaris
    
    Command (m for help): d
    Partition number (1-5): 1
    
    Command (m for help): d
    Partition number (1-5): 2
    
    Command (m for help): n
    Partition type:
       p   primary (0 primary, 0 extended, 4 free)
       e   extended
    Select (default p): p
    Partition number (1-4, default 1):
    Using default value 1
    First sector (2048-524287999, default 2048):
    Using default value 2048
    Last sector, +sectors or +size{K,M,G} (2048-524287999, default 524287999): 507516925
    
    Command (m for help): p
    
    Disk /dev/sda: 268.4 GB, 268435456000 bytes
    255 heads, 63 sectors/track, 32635 cylinders, total 524288000 sectors
    Units = sectors of 1 * 512 = 512 bytes
    Sector size (logical/physical): 512 bytes / 512 bytes
    I/O size (minimum/optimal): 512 bytes / 512 bytes
    Disk identifier: 0x000e49fa
    
       Device Boot      Start         End      Blocks   Id  System
    /dev/sda1            2048   507516925   253757439   83  Linux
    
    Command (m for help): n
    Partition type:
       p   primary (1 primary, 0 extended, 3 free)
       e   extended
    Select (default p): e
    Partition number (1-4, default 2): 2
    First sector (507516926-524287999, default 507516926):
    Using default value 507516926
    Last sector, +sectors or +size{K,M,G} (507516926-524287999, default 524287999):
    Using default value 524287999
    
    Command (m for help): p
    
    Disk /dev/sda: 268.4 GB, 268435456000 bytes
    255 heads, 63 sectors/track, 32635 cylinders, total 524288000 sectors
    Units = sectors of 1 * 512 = 512 bytes
    Sector size (logical/physical): 512 bytes / 512 bytes
    I/O size (minimum/optimal): 512 bytes / 512 bytes
    Disk identifier: 0x000e49fa
    
       Device Boot      Start         End      Blocks   Id  System
    /dev/sda1            2048   507516925   253757439   83  Linux
    /dev/sda2       507516926   524287999     8385537    5  Extended
    
    Command (m for help): n
    Partition type:
       p   primary (1 primary, 1 extended, 2 free)
       l   logical (numbered from 5)
    Select (default p): l
    Adding logical partition 5
    First sector (507518974-524287999, default 507518974):
    Using default value 507518974
    Last sector, +sectors or +size{K,M,G} (507518974-524287999, default 524287999):
    Using default value 524287999
    
    Command (m for help): p
    
    Disk /dev/sda: 268.4 GB, 268435456000 bytes
    255 heads, 63 sectors/track, 32635 cylinders, total 524288000 sectors
    Units = sectors of 1 * 512 = 512 bytes
    Sector size (logical/physical): 512 bytes / 512 bytes
    I/O size (minimum/optimal): 512 bytes / 512 bytes
    Disk identifier: 0x000e49fa
    
       Device Boot      Start         End      Blocks   Id  System
    /dev/sda1            2048   507516925   253757439   83  Linux
    /dev/sda2       507516926   524287999     8385537    5  Extended
    /dev/sda5       507518974   524287999     8384513   83  Linux
    
    Command (m for help): t
    Partition number (1-5): 5
    
    Hex code (type L to list codes): 82
    Changed system type of partition 5 to 82 (Linux swap / Solaris)
    
    Command (m for help): p
    
    Disk /dev/sda: 268.4 GB, 268435456000 bytes
    255 heads, 63 sectors/track, 32635 cylinders, total 524288000 sectors
    Units = sectors of 1 * 512 = 512 bytes
    Sector size (logical/physical): 512 bytes / 512 bytes
    I/O size (minimum/optimal): 512 bytes / 512 bytes
    Disk identifier: 0x000e49fa
    
       Device Boot      Start         End      Blocks   Id  System
    /dev/sda1            2048   507516925   253757439   83  Linux
    /dev/sda2       507516926   524287999     8385537    5  Extended
    /dev/sda5       507518974   524287999     8384513   82  Linux swap / Solaris
    
    Command (m for help): w
    The partition table has been altered!
    
    Calling ioctl() to re-read partition table.
    
    WARNING: Re-reading the partition table failed with error 16: Device or resource busy.
    The kernel still uses the old table. The new table will be used at
    the next reboot or after you run partprobe(8) or kpartx(8)
    Syncing disks.
    
    me@ubuntu:~$ sudo reboot 
    

I noticed afterwards that I didn’t set the bootable flag but apparently you don’t really need it.

Third: Enlarge the filesystem.

You can do this with resize2fs online on a mounted partition.

me@ubuntu:~$ df -h
Filesystem      Size  Used Avail Use% Mounted on
/dev/sda1        91G   86G   12M 100% /
udev            3.9G  4.0K  3.9G   1% /dev
tmpfs           1.6G  696K  1.6G   1% /run
none            5.0M     0  5.0M   0% /run/lock
none            3.9G  144K  3.9G   1% /run/shm
none            100M   16K  100M   1% /run/user

me@ubuntu:~$ sudo resize2fs /dev/sda1
resize2fs 1.42.5 (29-Jul-2012)
Filesystem at /dev/sda1 is mounted on /; on-line resizing required
old_desc_blocks = 6, new_desc_blocks = 16
The filesystem on /dev/sda1 is now 63439359 blocks long.

me@ubuntu:~$ df -h
Filesystem      Size  Used Avail Use% Mounted on
/dev/sda1       239G   86G  142G  38% /
udev            3.9G   12K  3.9G   1% /dev
tmpfs           1.6G  696K  1.6G   1% /run
none            5.0M     0  5.0M   0% /run/lock
none            3.9G  152K  3.9G   1% /run/shm
none            100M   36K  100M   1% /run/user

Slight catch: After rebooting the swap space wasn’t active. Turned out you need to run mkswap, adjust /etc/fstab to the new UUID and turn the swap on

me@ubuntu:~$ sudo mkswap /dev/sda5 
Setting up swapspace version 1, size = 8384508 KiB
no label, UUID=141d401a-b49d-4a96-9b85-c130cb0de40a

me@ubuntu:~$ sudo swapon --all --verbose
swapon on /dev/sda5
swapon: /dev/sda5: found swap signature: version 1, page-size 4, same byte order
swapon: /dev/sda5: pagesize=4096, swapsize=8585740288, devsize=8585741312

Edit /etc/fstab to replace the UUID for the old swap partition with the new one from mkswap.


concatenate multiple files but include filename as section headers

April 2, 2017 - Reading time: ~1 minute
tail -n +1 file1.txt file2.txt file3.txt

grep "" *.txt

This should do the trick as well:

find . -type f -print -exec cat {} \;

Means:

find    = linux `find` command finds filenames, see `man find` for more info
.       = in current directory
-type f = only files, not directories
-print  = show found file
-exec   = additionally execute another linux command
cat     = linux `cat` command, see `man cat`, displays file contents
{}      = placeholder for the currently found filename
\;      = tell `find` command that it ends now here

You further can combine searches trough boolean operators like -and or -or, also find -ls.


ffmpeg - changing frame rate

March 5, 2017 - Reading time: ~1 minute
With re-encoding:
ffmpeg -y -i seeing_noaudio.mp4 -vf "setpts=1.25*PTS" -r 24 seeing.mp4

Without re-encoding:

First step - extract video to raw bitstream

ffmpeg -y -i seeing_noaudio.mp4 -c copy -f h264 seeing_noaudio.h264

Remux with new framerate

ffmpeg -y -r 24 -i seeing_noaudio.h264 -c copy seeing.mp4

batch-convert document formats

February 8, 2017 - Reading time: 4 minutes

Install the latest LibreOffice and it's library
On most Ubuntu version LibreOffice is defaulty installed, just upgrade it to the latest version.

  • apt-get install libreoffice

Install unoconv
unoconv is a command line utility that can convert any document format (doc, docx. odt, ods, xls, xlsx) that LibreOffice can import, to any file format (xml, pdf, doc, docx. odt, ods, xls, xlsx) that LibreOffice is capable of exporting.

  • apt-get install unoconv
Installing LibreOffice is more than enough to answer your needs in converting document, but for some reason unoconv do better (despite using the same library) when the output is use by other application, we can say it more acceptable.

Batch Convert Document to csv, pdf, jpg, docx, xlsx, odt, or ods

To batch convert document you have to work under terminal, prepare a document that will be converted in a folder (can be consist of several sub-folders),  then do following :

Synopsis :

  • unoconv [options] to-file from-file

Convert single xls format to pdf

  • unoconv -f pdf some-document.xls

Convert single png format to jpg

  • unoconv -f jpg some-document.png

Batch Convert docx format to pdf

  • unoconv -f pdf *.docx

Batch Convert xlsx format to ods

  • unoconv -f ods *.xlsx

Batch Convert csv format to xlsx

  • unoconv -f xlsx *.csv
Batch Convert csv format to ods
  • unoconv -f ods *.csv

You can change the option, source, and destination format according to your needs. Some document format  which supported by the this application are pdf, odf, odt, ods, xls, xlsx, doc, docx, rtf, ppt, pptx, csv, png, jpg, bmp and svg.