Anil Gangolli

Notes on gigabit ethernet, ubuntu, file transfers, samba

Over time, more of the devices on my home network have been becoming gigabit-capable. In practice this doesn't buy us much. Most of the time, we are all on the 54 mbps wireless net and most of the traffic is Internet browsing at much slower rates. So to date, I hadn't really paid much attention to what I'm actually getting from all of the gigabit hardware.

Recently however, with some video files and backup disk images getting stored on the home server, it has really become worthwhile to connect to the wired net for specific activities. The Mac laptops in the household have gigabit ethernet ports, and so does my development desktop box.

On a recent Samba transfer of a relatively short 200MB home video clip from my wired desktop to the server, I noticed that I was still getting rates consistent with only 100mbps ethernet, nothing near gigabit rates.

Here are some notes on my experiences trying to improve this.

The Setting

The client is running Ubuntu 7.04 (Feisty Fawn) on a homemade desktop box with an ASUS P5WDG2 WS PRO motherboard. The motherboard has two gigabit ethernet devices on it, one based on the Marvell Yukon 88E8001 and one based on the 88E8052 chipset. I'm only really using the first one (eth0 on my box). The drivers are the skge drivers. I'm not sure which version is in the kernel (2.6.20-16-generic) in the 7.04 distribution.

The server is running Fedora Core 6 on another homemade box with an ASUS P4P800-E Deluxe motherboard; it has one Marvell 88E8001 device onboard, and another gigabit device from a Linksys card which is the one on the internal home net. Both use the skge drivers as well.

These are both wired using 7-foot CAT 6 cables to a Netgear GS-105 gigabit switch.

First Problem Found: Autonegotiation on the Ubuntu desktop box

After poking around with ethtool, I find, hey, I'm getting 100mbps rates because eth0 on the Ubuntu box thinks it is running at 100mbps. Check the indicator lights on the switch. Yep. Hmm. Now what?

Unplug, replug, ifconfig eth0 down, ifconfig eth0 up. No luck. For some reason, the Ubuntu box is autonegotiating to 100mbps though both sides are 1000mbps.

My google searches provide no answer but suggest some others have similar issues; I've still found nothing that would suggest how to get the autonegotiation to work. The nearly identical situation with Fedora on the server is having no problems reliably autonegotiating up to the gigabit rate. Eventually, I resort to forcing the Ubuntu box to go gigabit by adding a pre-up line to my /etc/network/interfaces< file before the interface is brought up, as shown in the third line of the following fragment from this file:


auto eth0
iface eth0 inet dhcp
pre-up /usr/sbin/ethtool -s eth0 speed 1000 duplex full autoneg off


Activating Jumbo Frames

OK. Now that I get that working reliably, I do some perf tests with iperf running on both sides, using TCP_NODELAY, and otherwise taking iperf defaults, I get about 537mbps.


% iperf -N -c 192.168.1.2
------------------------------------------------------------
Client connecting to 192.168.1.2, TCP port 5001
TCP window size: 27.6 KByte (default)
------------------------------------------------------------
[ 3] local 192.168.1.102 port 41548 connected with 192.168.1.2 port 5001
[ 3] 0.0-10.0 sec 640 MBytes 537 Mbits/sec


Much better! But I decide to check what I get by activating jumbo frames which my hardware all supports.
With jumbo frame support enabled I get:


% iperf -N -c 192.168.1.2
------------------------------------------------------------
Client connecting to 192.168.1.2, TCP port 5001
TCP window size: 73.6 KByte (default)
------------------------------------------------------------
[ 3] local 192.168.1.102 port 51573 connected with 192.168.1.2 port 5001
[ 3] 0.0-10.1 sec 923 MBytes 764 Mbits/sec


That's more like it. I think this is about the best I am likely to do, and in any case I decide I like 764 much better than 537 , so I add another pre-up line to set the MTU on each
reboot. My /etc/network/interfaces entry for the interface now looks like this:


auto eth0
iface eth0 inet dhcp
pre-up /usr/sbin/ethtool -s eth0 speed 1000 duplex full autoneg off
pre-up /sbin/ifconfig eth0 mtu 9000


Samba, Nautilus still slow

OK. Now that I've got my network speeds up I should be getting great transfer speeds, right?
Not quite.

Back in Nautilus (the GNOME finder GUI) I try dragging the 200MB video clip across, and I find
it still takes about 20 seconds, or 10MB per sec or roughly 100mbps again.

Could disk speed be the bottleneck? I don't think so; I've got fast SATA drives and they shouldn't
be a problem. To verify I try a little ftp test:


ftp> get Garden.dv
local: Garden.dv remote: Garden.dv
200 PORT command successful. Consider using PASV.
150 Opening BINARY mode data connection for Garden.dv (219360000 bytes).
226 File send OK.
219360000 bytes received in 2.23 secs (96182.1 kB/s)


OK! Now that's what I expect, roughly the speed of I'm seeing with iperf. So my desktop transfer problem must be a Samba problem right?

I go check my Samba server config, and consulting various sources on the
net, I find I already have the options set pretty well:


socket options = IPTOS_LOWDELAY TCP_NODELAY SO_RCVBUF=16384 SO_SNDBUF=16384


So, what about straight Samba transfers? Let's try smbget:


% /usr/bin/time -p smbget -w BUSYBUDDHA -a smb://sycamore-int/public/Garden.dv
Using workgroup BUSYBUDDHA, guest user
smb://sycamore-int/public/Garden.dv
Command exited with non-zero status 1
real 4.35
user 0.20
sys 1.01


The nonzero exit status turns out to be spurious. The file has transferred fine; took twice the time of FTP, but still not bad. Close to 500mbps. A similar test with smbclient yields similar results for me. Both of these are far better than what I'm seeing with Nautilus.

After more searching on the net, I decide to try smbfs for comparison


# mount -t smbfs -o username=notshown,password=notshown,sockopt=TCP_NODELAY //sycamore-int/public /media/public


This mounts it on /media/public and then using straight cp I get a 12 second transfer; slower than smbget.
But then I try a second transfer, and I get:


/usr/bin/time -p cp /media/public/Garden.dv .
real 0.95
user 0.02
sys 0.90


Great! Less than a second. Wait. That shouldn't really be possible. Check the md5sum. The file is all there. What is going on? Oh. Some sort of caching. Force the cache to be blown by remounting and you get the 12 second transfer again. So smbfs might perform really well for situations where you get to use the cache, but based on my limited experimentation, not for initial transfers. [Note: I also tried this with mount -t cifs and got the
same kind of behavior.]

Faster than Nautilus which is amazingly achieving only about 100mpbs still. So I'm still looking for improvement.

Summary

In summary, for my particular situation I found:

• Check that your ethernet device negotiated to 1000mbps. I had to force mine.


• If your hardware supports it, use jumbo frames. This got me to a measured 760 mbps using iperf (TCP) which I feel is probably as good as I'm going to get.


• FTP is able to achieve nearly the transfer rates that I see with iperf, and Disk I/O is included.


• I was able to get Samba doing about 500mbps via smbget or smbclient. I suspect this is about as good as I'll get.


• Using smbfs mounts is slower on initial transfers than smbget or smbclient, but fast on cached stuff.


• Samba transfers via Nautilus are still only getting transfer rates close to about 100mbps. I still don't know why Nautilus is so darn slow.

Tagged:  ethernet ftp gigabit linux samba ubuntu

Posted at 01:59PM Jul 22, 2007 in tech  |  Permalink