how to clone to a smaller harddisk

November 27, 2018 - Reading time: 5 minutes

You clearly cannot clone a larger partition to a smaller partition (using dd and the like) since there is simply not enough space.

However, if the files that are on the larger partition would also fit on the smaller partition, you could use rsync to copy those files. The exact options to use depend on your particular use case, but to simply copy all the files the following should do:

rsync -av /mount/point/of/large/partition/ /mount/point/of/small/partition

Edit: Once again: You cannot clone a larger partition onto a smaller partition. (But do read on, your problem can be solved yet.)

The reason is simple: your source partition is bigger than your target partition. What do you expect? Should some blocks just be dropped? Which ones? And how should dd know? Of course, you could use dd's bs= and count= options to only copy the first so-and-so-many blocks of your source partition such that it fits onto your target partition, but you will end up with a broken partition. That is certainly not what you want.

So, since you cannot clone a larger partition onto a smaller partition, the only thing you could do is to first reduce the size of your source partition to a size smaller or equal to that of your target partition with something like gparted which is aware of the filesystem specifics, such that you do not lose data. And only then could you use dd to clone the partition. Ideally, the new size of your source partition should be equal to the size of your target partition (and not just smaller or equal), or else you will end up with some unallocated space on your target partition after the cloning.

Please also note that you should not simply copy an MBR of a larger drive onto the MBR of a smaller drive (or vice versa, for that matter). The MBR, which has a size of 512 bytes and is the first section of your hard drive, contains information on the layout of the harddrive:

  • 446 bytes - Bootstrap.
  • 64 bytes - Partition table.
  • 2 bytes - Signature.

(Note that 446+64+2=512.)

If you insist on cloning the MBR, then only clone the first 446 bytes like so:

dd if=/dev/source of=/tmp/mbr.bak bs=512 count=1
dd if=/tmp/mbr.bak of=/dev/target bs=446 count=1

...replacing /dev/source and /dev/target with the device names of the source and target harddrives, e.g., /dev/sda and /dev/sdb, respectively. (More information is available here.)

However, the proper way to do it would be to do a clean Grub reinstall (or whatever you have on your MBR) on the new harddrive.

Summing up, if you want to clone a larger drive onto a smaller drive, proceed as follows:

  1. Lay out a partition table on the target drive with as many partitions as on the source drive. There should be a one-to-one correspondence between the partitions on your source drive and the partitions on your target drive, except that (some of) the partitions on the target drive can be smaller than their corresponding partitions on the source drive. Use a tool such as fdisk or cfdisk for that.

  2. For each partition on the target drive which is smaller than its corresponding partition on the source drive, reduce the size of this corresponding partition on the source drive to match the size of the partition on the target drive. Use a tool such as gparted for that.

  3. For each partition on the source drive, issue the command

    dd if=/dev/sdaX of=/dev/sdbY
    

    ... to clone the partition /dev/sdaX from the source drive to the corresponding partition /dev/sdbY on the target drive (replace the device names appropriately, of course.)

  4. If you insist on also cloning the MBR, use the two dd commands written further above in this post (those with the /tmp/mbr.bak stuff). However, keep in mind that a clean Grub re-install would be better.


eject / safely remove vs umount

April 10, 2018 - Reading time: ~1 minute

If you are using systemd then use udisksctl utility with power-off option:

power-off

Arranges for the drive to be safely removed and powered off. On the OS side this includes ensuring that no process is using the drive, then requesting that in-flight buffers and caches are committed to stable storage.

I would recommend first to unmount all filesystems on that usb. This can be done also with udisksctl, so steps would be:

udisksctl unmount -b /dev/sda1
udisksctl power-off -b /dev/sda

If you are not using systemd then old good udisks should work:

udisks --unmount /dev/sda1
udisks --detach /dev/sda

export a recursive directory & file listing to a text file in Linux Bash shell with an SSH command

June 23, 2017 - Reading time: 3 minutes
find is a good utility to obtain (recursively) all the content of a directory. If you want the file (and directory) names with their permissions:
find /home/kparisi -printf "%M %p\n"

You can then use ssh to run this command on the remote server:

ssh kparisi@remote.com 'find /home/kparisi -printf "%M %p\n"'

And finally, if you want to store this in a file on your local server:

ssh kparisi@remote.com 'find /home/kparisi -printf "%M %p\n"' > file 

Alternatively:

tree > fsstruct.txt

for me this format is simpler to read.

It is easy also to use other features of tree:

tree -p > fsstruct.txt

prints file type and permissions before file and it is more accessible when you are reading plain text, which is a file and which is a directory.

tree -h > fsstruct.txt

this prints sizes of files in a human readable format:

tree -ph > fsstruct.txt

Also, it is possible to send output to the file:

tree -ph . -o fsstruct.txt

or create HTML:

tree -Hph . -o tree.htm


resize an ext4 partition from the command line

May 25, 2017 - Reading time: 6 minutes

You can use fdisk to change your partition table while running.  Refer to Live resizing of an ext4 filesytem on Linux (on The silence of the code blog):

Disclaimer: The following instructions can easily screw your data if you make a mistake.  I was doing this on a VM which I backed up before performing the following actions.  If you lose your data because you didn’t perform a backup don’t come and complain.

...

First: Increase the disk size.

In ESXi this is simple, just increase the size of the virtual disk. Now you have a bigger hard drive but you still need to a) increase the partition size and b) resize the filesystem.

Second: Increase the partition size.

You can use fdisk to change the partition table while running.  The stock Ubuntu install has created 3 partitions: one primary (sda1), one extended (sda2) with a single logical partition (sda5) in it. The extended partition is simply used for swap, so I could easily move it without losing any data.

  1. Delete the primary partition

  2. Delete the extended partition

  3. Create a new primary partition starting at the same sector as the original one just with a bigger size (leave some for swap)

  4. Create a new extended partition with a logical partition in it to hold the swap space

    me@ubuntu:~$ sudo fdisk /dev/sda
    
    Command (m for help): p
    
    Disk /dev/sda: 268.4 GB, 268435456000 bytes
    255 heads, 63 sectors/track, 32635 cylinders, total 524288000 sectors
    Units = sectors of 1 * 512 = 512 bytes
    Sector size (logical/physical): 512 bytes / 512 bytes
    I/O size (minimum/optimal): 512 bytes / 512 bytes
    Disk identifier: 0x000e49fa
    
       Device Boot      Start         End      Blocks   Id  System
    /dev/sda1   *        2048   192940031    96468992   83  Linux
    /dev/sda2       192942078   209713151     8385537    5  Extended
    /dev/sda5       192942080   209713151     8385536   82  Linux swap / Solaris
    
    Command (m for help): d
    Partition number (1-5): 1
    
    Command (m for help): d
    Partition number (1-5): 2
    
    Command (m for help): n
    Partition type:
       p   primary (0 primary, 0 extended, 4 free)
       e   extended
    Select (default p): p
    Partition number (1-4, default 1):
    Using default value 1
    First sector (2048-524287999, default 2048):
    Using default value 2048
    Last sector, +sectors or +size{K,M,G} (2048-524287999, default 524287999): 507516925
    
    Command (m for help): p
    
    Disk /dev/sda: 268.4 GB, 268435456000 bytes
    255 heads, 63 sectors/track, 32635 cylinders, total 524288000 sectors
    Units = sectors of 1 * 512 = 512 bytes
    Sector size (logical/physical): 512 bytes / 512 bytes
    I/O size (minimum/optimal): 512 bytes / 512 bytes
    Disk identifier: 0x000e49fa
    
       Device Boot      Start         End      Blocks   Id  System
    /dev/sda1            2048   507516925   253757439   83  Linux
    
    Command (m for help): n
    Partition type:
       p   primary (1 primary, 0 extended, 3 free)
       e   extended
    Select (default p): e
    Partition number (1-4, default 2): 2
    First sector (507516926-524287999, default 507516926):
    Using default value 507516926
    Last sector, +sectors or +size{K,M,G} (507516926-524287999, default 524287999):
    Using default value 524287999
    
    Command (m for help): p
    
    Disk /dev/sda: 268.4 GB, 268435456000 bytes
    255 heads, 63 sectors/track, 32635 cylinders, total 524288000 sectors
    Units = sectors of 1 * 512 = 512 bytes
    Sector size (logical/physical): 512 bytes / 512 bytes
    I/O size (minimum/optimal): 512 bytes / 512 bytes
    Disk identifier: 0x000e49fa
    
       Device Boot      Start         End      Blocks   Id  System
    /dev/sda1            2048   507516925   253757439   83  Linux
    /dev/sda2       507516926   524287999     8385537    5  Extended
    
    Command (m for help): n
    Partition type:
       p   primary (1 primary, 1 extended, 2 free)
       l   logical (numbered from 5)
    Select (default p): l
    Adding logical partition 5
    First sector (507518974-524287999, default 507518974):
    Using default value 507518974
    Last sector, +sectors or +size{K,M,G} (507518974-524287999, default 524287999):
    Using default value 524287999
    
    Command (m for help): p
    
    Disk /dev/sda: 268.4 GB, 268435456000 bytes
    255 heads, 63 sectors/track, 32635 cylinders, total 524288000 sectors
    Units = sectors of 1 * 512 = 512 bytes
    Sector size (logical/physical): 512 bytes / 512 bytes
    I/O size (minimum/optimal): 512 bytes / 512 bytes
    Disk identifier: 0x000e49fa
    
       Device Boot      Start         End      Blocks   Id  System
    /dev/sda1            2048   507516925   253757439   83  Linux
    /dev/sda2       507516926   524287999     8385537    5  Extended
    /dev/sda5       507518974   524287999     8384513   83  Linux
    
    Command (m for help): t
    Partition number (1-5): 5
    
    Hex code (type L to list codes): 82
    Changed system type of partition 5 to 82 (Linux swap / Solaris)
    
    Command (m for help): p
    
    Disk /dev/sda: 268.4 GB, 268435456000 bytes
    255 heads, 63 sectors/track, 32635 cylinders, total 524288000 sectors
    Units = sectors of 1 * 512 = 512 bytes
    Sector size (logical/physical): 512 bytes / 512 bytes
    I/O size (minimum/optimal): 512 bytes / 512 bytes
    Disk identifier: 0x000e49fa
    
       Device Boot      Start         End      Blocks   Id  System
    /dev/sda1            2048   507516925   253757439   83  Linux
    /dev/sda2       507516926   524287999     8385537    5  Extended
    /dev/sda5       507518974   524287999     8384513   82  Linux swap / Solaris
    
    Command (m for help): w
    The partition table has been altered!
    
    Calling ioctl() to re-read partition table.
    
    WARNING: Re-reading the partition table failed with error 16: Device or resource busy.
    The kernel still uses the old table. The new table will be used at
    the next reboot or after you run partprobe(8) or kpartx(8)
    Syncing disks.
    
    me@ubuntu:~$ sudo reboot 
    

I noticed afterwards that I didn’t set the bootable flag but apparently you don’t really need it.

Third: Enlarge the filesystem.

You can do this with resize2fs online on a mounted partition.

me@ubuntu:~$ df -h
Filesystem      Size  Used Avail Use% Mounted on
/dev/sda1        91G   86G   12M 100% /
udev            3.9G  4.0K  3.9G   1% /dev
tmpfs           1.6G  696K  1.6G   1% /run
none            5.0M     0  5.0M   0% /run/lock
none            3.9G  144K  3.9G   1% /run/shm
none            100M   16K  100M   1% /run/user

me@ubuntu:~$ sudo resize2fs /dev/sda1
resize2fs 1.42.5 (29-Jul-2012)
Filesystem at /dev/sda1 is mounted on /; on-line resizing required
old_desc_blocks = 6, new_desc_blocks = 16
The filesystem on /dev/sda1 is now 63439359 blocks long.

me@ubuntu:~$ df -h
Filesystem      Size  Used Avail Use% Mounted on
/dev/sda1       239G   86G  142G  38% /
udev            3.9G   12K  3.9G   1% /dev
tmpfs           1.6G  696K  1.6G   1% /run
none            5.0M     0  5.0M   0% /run/lock
none            3.9G  152K  3.9G   1% /run/shm
none            100M   36K  100M   1% /run/user

Slight catch: After rebooting the swap space wasn’t active. Turned out you need to run mkswap, adjust /etc/fstab to the new UUID and turn the swap on

me@ubuntu:~$ sudo mkswap /dev/sda5 
Setting up swapspace version 1, size = 8384508 KiB
no label, UUID=141d401a-b49d-4a96-9b85-c130cb0de40a

me@ubuntu:~$ sudo swapon --all --verbose
swapon on /dev/sda5
swapon: /dev/sda5: found swap signature: version 1, page-size 4, same byte order
swapon: /dev/sda5: pagesize=4096, swapsize=8585740288, devsize=8585741312

Edit /etc/fstab to replace the UUID for the old swap partition with the new one from mkswap.


repair a broken Ext4 Superblock

January 18, 2017 - Reading time: 8 minutes

First, figure out what partition we’re dealing with.

sudo fdisk -l

The above will list all the partitions on all the drives in your computer. To recover a lost partition, your going to need Testdisk. Testdisk is included in Parted Magic, and there’s a great guide on their site. For this though, we just need the partition number, such as /dev/sda3 or /dev/hdb1. Now, make sure your superblock is the problem, by starting a filesystem check, replacing xxx with your partition name. Here, you can change ext4 to ext3, or ext2 to suit the filesystem.

1
sudo fsck.ext4 -v /dev/xxx

If your superblock is corrupt, the output will look like this

1
2
3
4
5
6
7
8
9
10
11
fsck /dev/sda5
fsck 1.41.4 (27-Jan-2009)
e2fsck 1.41.4 (27-Jan-2009)
fsck.ext4: Group descriptors look bad... trying backup blocks...
fsck.ext4: Bad magic number in super-block while trying to open /dev/sda5
 
The superblock could not be read or does not describe a correct ext4
filesystem.  If the device is valid and it really contains an ext4
filesystem (and not swap or ufs or something else), then the superblock
is corrupt, and you might try running e2fsck with an alternate superblock:
e2fsck -b 8193 <device>

Now lets find where your superblock backups are kept.

1
sudo mke2fs -n /dev/xxx

Down at the bottom of this output, should be a list of the backups

 

1
2
Superblock backups stored on blocks:
32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632, 2654208

Your almost there. Finally, restore the superblock from the backup, again replacing the x’s with your partition name, and block_number with the first backup superblock.

 

1
sudo e2fsck -b block_number /dev/xxx

mount a disk image from the command line

January 27, 2016 - Reading time: 2 minutes

If it was a hard-drive image with a MBR partition table, I would fdisk the image to find the offset for the partition I need to mount.

fdisk -lu /path/disk.img

Then I would mount it passing the offset.

mount -o loop,offset=xxxx /path/disk.img /mnt/disk.img.partition

The offset value is in bytes, whereas fdisk shows a block count, so you should multiply the value from the "Begin" or "Start" column of the fdisk output by 512 (or whatever the block size is) to obtain the offset to mount at.

On most modern GNU system the mount command can handle that:

mount -o loop file.iso /mnt/dir

to unmount you can just use the umount command

umount /mnt/dir

If your OS doesn't have this option you can create a loop device:

losetup -f # this will print the first available loop device ex:/dev/loop0
losetup /dev/loop0 /path/file.iso #associate loop0 with the specified file
mount /dev/loop0 /mnt/dir #It may be necessary specify the type (-t iso9660)

to umount you can use -d:

umount /mnt/dir
losetup -d /dev/loop0

If the file have partitions, example a HD image, you can use the -P parameter (depending on you OS), it will map the partitions in the file content:

losetup -P /dev/loop0 /path/file.iso # will create /dev/loop0 
ls /dev/loop0p* #the partitions in the format /dev/loop0pX