Defragmenters for Linux

There were a lot of words were told about defragmentation of Linux filesystems. Summing them up, one can say: if fsck tells that fragmentation is below 10% then defrag is not necessary. But a few defragmentators for Linux were written. So let's discuss them.


Search results
A brief look at defragmentation tools for Linux is enough to find two utilites, a defrag and a shake. The defrag utility by Cone Colivas seems to be outdated. But there is a shake that can defragment a selected directory in a file system. There is also a useful script frag.perl located here that can measure fragmentation level of directories:

sudo ./frag.pl /root
3.2258064516129% non contiguous files, 1.04838709677419 average fragments.

Using these tools, shake and frag.perl, we are going to fight with filesystem's fragmentation in Linux.

Setting it up
The setup of shake is slightly complicated process. The sources are located here, and if you are using Ubuntu you can take the binary file here. And here I had experienced some difficulties...

Firstly, latest versions of shake depends from fresh version of cdbs; that is why it is better to use older version. For the shake to be built, following packages are reqiured:

dpkg-checkbuilddeps: Unmet build dependencies: cdbs (>= 0.4.50) cmake (>= 2.4) help2man (>= 1.26) libattr1-dev (>= 2.0.0)

The program had compiled as well as her auxiliary tool unattr. For using shake more efficiently you might to mount a filesystem with user_xattr. To do such you need to edit /etc/fstab such like:

/dev/sda2 /mnt/flash/exthdd ext3 rw,user_xattr,user,auto 0 0

Then rmount: mount -o remount /dev/sda2
This trick allows the shake to works faster. And the last but not least, both shake и frag.perl are required root privileges so use sudo.


An example
I had tested shake on Hardy Heron repository that I downloaded. Checking fragmentation level gives:

./frag.perl /home/beast/flash/exthdd/linux/i386/ubuntu/pool/main 0.795800880460549% non contiguous files, 1.00998984083982 average fragments.

That is not a Windows situationan apocalypse. Then I had started shake:

./shake /home/beast/flash/exthdd/linux/i386/ubuntu/pool/main

The defragmentation took about 15-20 and all MD5SUMs are correct. So we can use shake for more extended experiments.


The extended experiment
For such experiment I had used to defragment Firefox's profile on my laptop: Firefox 2.0.0.4, ReiserFS, Asus M5200AE (P-M 1.75, 512Mb RAM).

There are a lot of files, and even worse: in Firefox profile there is a huge collection of webpages captured by Scrapbook. Estimated fragmentation is:

./frag.perl /home/beast/.mozilla/
3.24491439557299% non contiguous files, 1.12239224753839 average fragments.

I should say that my Firefox is covered with extensions like a Christmas tree, so it starts very sloooooooowly:

time /opt/firefox2/firefox
real 0m40.467s
user 0m5.389s
sys 0m0.407s

time /opt/firefox2/firefox
real 0m41.383s
user 0m5.453s
sys 0m0.403s

Firefox had been started two times with laptop clod-start each time.

Then defragmentation had been performed by shake. It took about 9 minutes. After that I started Firefox again:

time /opt/firefox2/firefox
real 0m39.522s
user 0m5.398s
sys 0m0.374s

time /opt/firefox2/firefox
real 0m39.214s
user 0m5.279s
sys 0m0.370s

There are no miracles, and 39 seconds are not fantastical results.

Conclusion
Sure, I can defrag something else but it is unlikely that results will be different. The analogous results are here and there.
Read more...

Magic SysRq Key

Nothing is perfect in this world, and even Linux sometimes crashes. But even then there is a connection with kernel - Magic SysRq Key.


What is this?
If you look to the keyboard then you see a strange key PrtSc / SysRq. Most people thinks that such key is for taking screenshots, but SysRq appeared on keyboards many years before Redmond behemoth.

Using SysRq key it is possible to force a Linux system to sync disks, remount disks into "read-only" mode, and then reboot computer safely. The key combination is intercepted directly by kernel, hence it is guaranty that Alt+SysRq is workable almost in any situations.


How we can use it?
First, we need to ensure that Magic SysRq key is enabled in kernel config: CONFIG_MAGIC_SYSRQ value must be Y (in most distributions it is so). To use such key, press and hold Alt, then press SysRq, release it and press another key - meanings of most useful keys are listed below

Simultaneously pressed Alt + SysRq +

'H'elp - prints to output all key combinations;

loglevel'0'-'8' - can change verbosity of the output from 0 (only critical messages) to 8 (most verbose mode);

re'B'oot - immediately reboot, like RESET button (no sync, no unmount filesystems);

'C'rashdump - runs kexec to reboot for obtaining errors info;

hol'D's - shows all blockers that blocks devices and files;

power'O'ff - correctly shuts down the system (if tweaked);

'S'ync - tries to sync all mounted filesystems and remount them in ReadOnly mode; you must see "Emergency Remount R/O" in console, and "Emergency Remount Complete" when it is over;

'U'nmount - tries to remount all mounted filesystems to ReadOnly mode;

show'T'asks - shows all tasks that are running now;

sho'W'-blocked-tasks - shows all uninterruptable tasks that are blocked or waiting I/O;

'F'ull - running oom_kill (out-of-memory kill) to kill applications that are eating all of memory;

secure access 'K'ey - to kill all tasks on the current console (don't do it on a console where X server running; in this case you can not see anything);

un'R'aw - takes away mouse and keyboard from X server; it it useful if X is trapped so you can change console and kill unwanted tasks;

t'E'rm - send SIGTERM signal to all processes except init.

K'I'LL - send SIGKILL signal (immediate termination) to all processes except init.

Thus for emergency sync you should press ALT+SysRq + s then ALT+SysRq + u and after that you can reset you computer or press ALT+SysRq+b

Links
Except kernel documentation you may be interested in reading this and this links.
Read more...

SSH for mere mortals

There are a lot of documents about the SSH, but in this post I want to tell about simple things. Let me explain some simple and obvious SSH tricks...


How to install SSH in Debian
From the Etch's release, packages for the SSH client and server are separated. Hence let's install SSH using command:

# aptitude install ssh
or
# aptitude install openssh-server openssh-client

and wait a few moments for configuration of the packages.

On a client side
Now we must edit the configuration files located in the /etc/ssh directory. A client-side config file is ssh-config and a server-side config file is sshd-config. On a client-side let's allow to receive X11Forward; or such purposes, changing:

ForwardX11 yes
ForwardX11Trusted yes

Now the client machine can start the graphical applications on a server. Now let's go to the admin of the far-far-away kingdom server...


On a server-side
We need to edit config file (you have root password from the server, aren't you? :-)) in  /etc/ssh/sshd-config and changing the keys to such ones:

X11Forwarding yes
X11DisplayOffset 10
X11UseLocalhost yes

Here we allowed to start graphical applications remotely and redirect them in the client machine. Restart the SSH daemon:

sudo /etc/init.d/ssh restart

That's practically all.

If there is necessity to restrict access to the machine, one need to change config file /etc/ssh/sshd_config in such way:

AllowUsers hacker@*
AllowUsers *@192.168.1.*

SSH in work
It's all done, now open console and type:

$ ssh remoteusername@ip_address_remotemachine

For example, in my case I write: ssh beast@192.168.1.5

After that, the SSH daemon asks us: this IP address isn't recognized yet, can I trust them? Sure! :-) Next, type the password of remote system, and if the password is correct then you log in into shell of the remote machine. During password typing you see nothing; you have 3 attempts or connection lost.
So, the SSH system will greet us like this:

penta4@penta4rce:~$ ssh beast@192.168.1.5
Password:
Linux notebeast 2.6.15.7 #3 PREEMPT Sun Jul 2 12:51:07 MSD 2006 i686 GNU/Linux

The programs included with the Debian GNU/Linux system are free software;
the exact distribution terms for each program are described in the
individual files in /usr/share/doc/*/copyright.

Debian GNU/Linux comes with ABSOLUTELY NO WARRANTY, to the extent
permitted by applicable law.

Last login: Tue Oct 10 19:23:57 2006 from 192.168.1.1
beast@notebeast:~$

Now we control remote computer: don't entangle machines or you shut down the wrong one! :-)


Remote using of the graphical applications
When you logged in, just type the name of the program with ampersand at the end of command:

$ gimp&

This command starts the GIMP on the remote machine and returns command prompt to you. If you do not enter ampersand then command prompt of the current SSH shell locks and returns only after application's over.

Thus, started remotely application is redirected to your machine and works the same as it started on your machine. Of course, the remotely started application will use files on the remote filesystem.

Conclusion
It is clear that SSH can be useful to mere mortals and not only for gurus. More about SSH one can  read in Goooogle or in this  this or this inks.

Read more...

Long-time remote shooting with Canon EOS 400D

Problem: Shooting with exposure times 30 and more is required and this process must be automated.
Solve: using soldering iron, common chips and bash script in Linux, it is possible to make PC-driven remote control device.


What we have
We have Canon EOS 400D, Debian-powered notebook and necessity of shooting pictures with exposure longer than 30 seconds. There is good scheme proposed by Michael A. Covington here. Anyway, mirroring it here:


Pretty good scheme, but it doesn't work for Canon EOS 400D - shutter will lift up bot not down.

Scheme for Canon EOS 400D
After some fruitless trying, I am with my colleague Alexey Ropyanoi, found out why proposed scheme not work and propose new one:


And it works! Our laboratory Canon EOS 400D begin open and close shutter by computer command.

Necessary electric components
To do the same remote shooting wire, you need 4-wire cable (from audio devices or from telephone cable), 2.5mm jack (or 3/32 inch jack), mentioned in scheme chips, 9-pin COM-port and USB-COM adapter (for using this remote shooting wire on novel computers).

The best USB-COM adapter is on Profilic 2303 chip - it is the most common chip and it works in Linux, like practically all, "out of the box".


Software
For remote control of camera, little program on C is needed. It is setSerialSignal and it source code is placed here. It can be compiled with GCC, which is part of any UNIX-like OS distribution.

gcc -o setSerialSignal setSerialSignal.c

Works on Debian GNU/Linux v4.0 r.0 "Etch", gcc version 4.1.2 20061115 (prerelease) (Debian 4.1.1-21).

This is the code:

/*
* setSerialSignal v0.1 9/13/01
* www.embeddedlinuxinterfacing.com
*
*
* The original location of this source is
* http://www.embeddedlinuxinterfacing.com/chapters/06/setSerialSignal.c
*
* This program is free software; you can redistribute it and/or modify
* it under the terms of the GNU Library General Public License as
* published by the Free Software Foundation; either version 2 of the
* License, or (at your option) any later version.
*
* This program is distributed in the hope that it will be useful, but
* WITHOUT ANY WARRANTY; without even the implied warranty of
* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
* Library General Public License for more details.
*
* You should have received a copy of the GNU Library General Public
* License along with this program; if not, write to the
* Free Software Foundation, Inc.,
* 59 Temple Place, Suite 330, Boston, MA 02111-1307 USA
*/
/* setSerialSignal
* setSerialSignal sets the DTR and RTS serial port control signals.
* This program queries the serial port status then sets or clears
* the DTR or RTS bits based on user supplied command line setting.
*
* setSerialSignal clears the HUPCL bit. With the HUPCL bit set,
* when you close the serial port, the Linux serial port driver
* will drop DTR (assertion level 1, negative RS-232 voltage). By
* clearing the HUPCL bit, the serial port driver leaves the
* assertion level of DTR alone when the port is closed.
*/

/*
gcc -o setSerialSignal setSerialSignal.c
*/


#include 
#include 
#include 

/* we need a termios structure to clear the HUPCL bit */
struct termios tio;

int main(int argc, char *argv[])
{
int fd;
int status;

if (argc != 4)
{
printf("Usage: setSerialSignal port                  DTR RTS\n");
printf("Usage: setSerialSignal /dev/ttyS0|/dev/ttyS1 0|1 0|1\n");
exit( 1 );
}

if ((fd = open(argv[1],O_RDWR)) < 0)
{
printf("Couldn't open %s\n",argv[1]);
exit(1);
}
tcgetattr(fd, &tio);          /* get the termio information */
tio.c_cflag &= ~HUPCL;        /* clear the HUPCL bit */
tcsetattr(fd, TCSANOW, &tio); /* set the termio information */

ioctl(fd, TIOCMGET, &status); /* get the serial port status */

if ( argv[2][0] == '1' )      /* set the DTR line */
status &= ~TIOCM_DTR;
else
status |= TIOCM_DTR;

if ( argv[3][0] == '1' )      /* set the RTS line */
status &= ~TIOCM_RTS;
else
status |= TIOCM_RTS;

ioctl(fd, TIOCMSET, &status); /* set the serial port status */

close(fd);                    /* close the device file */
}

Sending signals
Compiling program and making it executable, and below listed signals which will open and close shutter:

DTR
setSerialSignal /dev/ttyS0 1 0


Clear DTR
setSerialSignal /dev/ttyS0 0 0


RTS
setSerialSignal /dev/ttyS0 0 1


Clear RTS
setSerialSignal /dev/ttyS0 1 1


Shutter opens at DTR and closes at RTS.


Shell script for remote shooting
Next, it is comfortable to use bash script by Eugeni Romas aka BrainBug, but for Canon 400D script was edited and here it is:

#!/bin/bash

for i in `seq $3`; do
{
setSerialSignal /dev/ttyUSB0 0 0 &&
sleep $1 && setSerialSignal /dev/ttyUSB0 0 1 &&
sleep 0.3 && setSerialSignal /dev/ttyUSB0 0 0 &&
sleep $2 && setSerialSignal /dev/ttyUSB0 1 1 && echo "One more image captured!" &&
sleep $4;

}
done

echo "Done!"

Script parameters:
1: shutter opening delay
2: exposure time in seconds
3: amount of shots
4: delay between shots

Example:

make_captures 4 60 30 2

Script is written to work with USB-COM adaptor, and you need to edit it if you have different port.

How it works
Remote shooting wire is ready, inserting USB-COM adapter with wire and next:

• Turn on camera, setting BULB mode, setting aperture size and ISO speed.
• Inserting jack into the camera, another and in COM-USB adapter and then in USB-port.
• Looking at logs: kernel must recognize chip and write something like this:

usb 2-1: new full speed USB device using uhci_hcd and address 2
usb 2-1: configuration #1 chosen from 1 choice
drivers/usb/serial/usb-serial.c: USB Serial support registered for pl2303
pl2303 2-1:1.0: pl2303 converter detected
usb 2-1: pl2303 converter now attached to ttyUSB0
usbcore: registered new interface driver pl2303
drivers/usb/serial/pl2303.c: Prolific PL2303 USB to serial adaptor driver

• Now shoot:

  make_capture 1 5 2 3

Here we make 2 images with 5 second exposure, delay between shots is 3 seconds, delay for shutter lifting 1 second.

Acknowledgements
I would like to express my gratitude to:

• Michael A. Covington for his original article "Building a Cable Release and Serial-Port Cable for the Canon EOS 300D Digital Rebel".
• Eugeni Romas aka BrainBug for link to original post and discussion.
• Anton aka NTRNO for searching key posts at Astrophorum.
• Alexey Ropjanoi, who experimentally found out problem and eliminated it, proposing new shceme for shooting.

And I deeply thankful to my colleagues for Solid State physic department of Moscow Engineer Physics Institute.
Read more...

How to change MAC-address of Ethernet card

Sometimes changing MAC-address of a network card is needed. For example, when it used for authorisation by internet provider. This is relatively simple yet unobvious in a first time procedure, but here are some examples.



Temporary MAC-address change
Go to the terminal and type:

sudo ifconfig eth0 hw ether xx:xx:Xx:xx

and MAC-address change and will stay until reboot. If you wish to change MAC-address permanently, you need to do this:


Permanent change of MAC-address
In Debian, for example, go to  /etc/network/if-pre-up.d/ and create file from root named pre-up.
Let`s write to them strings like this:

#! /bin/sh

ifconfig eth0 hw ether 00:00:00:00

Change zeroes by needed MAC-address. To make changes active immediately, restart of network subsystem is required:

/etc/init.d/networking restart

In first time, following diagnostic messages are expected to appear:

# /etc/init.d/networking restart
Setting up IP spoofing protection: rp_filter.
Reconfiguring network interfaces...SIOCDELRT: No such process
ifup: interface lo already configured
SIOCSIFHWADDR: Device or resource busy
run-parts: /etc/network/if-pre-up.d/pre-up exited with return code 1
done.

No problems, repeat command, and it will execute smoother:

notebeast:/home/beast# /etc/init.d/networking restart
Setting up IP spoofing protection: rp_filter.
Reconfiguring network interfaces...ifup: interface lo already configured
done.

Here we go, the MAC-address is changed permanently.


How to find out MAC-address of network card in Linux
It's very simple, just type:

sudo ifconfig

and you see something like this:

eth0 Link encap:Ethernet HWaddr 00:0A:E4:53:AA:2D
inet addr:10.26.49.77 Bcast:10.26.63.255 Mask:255.255.240.0
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:208554 errors:0 dropped:0 overruns:0 frame:0
TX packets:125071 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:40664531 (38.7 MiB) TX bytes:45919980 (43.7 MiB)
Interrupt:21 Base address:0x4c00

lo Link encap:Local Loopback
inet addr:127.0.0.1 Mask:255.0.0.0
UP LOOPBACK RUNNING MTU:16436 Metric:1
RX packets:18511 errors:0 dropped:0 overruns:0 frame:0
TX packets:18511 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:0
RX bytes:537155 (524.5 KiB) TX bytes:537155 (524.5 KiB)

One's MAC-address is emphased by red font colour.


Yet another tips to change MAC-address

For example, MAC-address in Gentoo can be changed in /etc/conf.d/net

mac_eth0="00:50:8D:63:41:DE"
config_eth0=( "192.168.100.37 netmask 255.255.252.0" )
routes_eth0=(
"default via 192.168.100.1"
)

Or edit /etc/network/interfaces adding one string:

pre-up ifconfig eth0 hw ether 00:00:00:00:00:00

Indeed, there are loads of variants, because it is Linux :-)

iface eth0 inet static
address 192.168.1.1
netmask 255.255.255.0
hwaddress ether xxxxxxxxxxxx

Conclusion
This post can be useful to those who connected to Internet via provider authorising by MAC-address.

Read more...

Yakuake - terminal in a Quake-style

When you use console in X-Window, often and often you desktop are literally wasted with xterms. It is more important, that you productivity is lowered, too. But here is solution - Yakuake.


Gamer`s console
When I was young, I spent pretty much time with CounterStrike and QuakeIII. And I felled in love with this small and elegant console hidden-in-a-top of screen in Quake. Till now, I was missing this feature on my desktop. But surfing in Synaptic recently, I found what I missed so much - Yakuake - and installed it immediately.


Tweaking is necessary
The main thing after installing Yakuake is tweaking. For comfort and productive work, you should customise Yakuake to yourself: keyboard shortcuts, theme, console hide duration and so on.

First step is to define for yourself with size of console: if it`s too small, you can`t see all that is written in console, and if it`s too large, it can overlap all of applications. I think that vertical size of 40% is fine for me - I can see everything in console and it don`t occludes other applications.

You can customize not only height of Yakuake console, but position on screen as well. It can be positioned in left, right on in centre. Of course, you can set position in procents precisely.

But the main thing in Yakuake are hotkeys. The work in console itself disposes to hotkeys and keyboard use, and properly configured hotkeys combinations can increase you productivity with Yakuake dramatically. There are not so much keys to customise, so tweaking don`t take so much time. By default you can call Yakuake by pressing F12 functional key, but you can redefine it to you choice.

Because Yakuake is very dynamic thing, only video can show it real capabilities, so I made short youtube film and here it is:




A little comment: first of all, film shows conventional work in terminal with xterms in X-Window, and they overlap each other. Then Yakuake is ran, and do the same stuff much more effective.

Links:
More about Yakuake can be readed here, there and here.
Read more...