Showing posts with label Project FrankenNAS. Show all posts
Showing posts with label Project FrankenNAS. Show all posts

Saturday, July 3, 2010

Freenas Transmission Tweaks

Blatantly liberated from http://code.google.com/p/freenas-rss-extension/wiki/MyTweaks.

** FOR THOSE SEARCHING FOR ERRORS RELATED TO "BLOCKLIST_UPDATE: Config directory not set"

The script below searches through FreeNAS' configurtion files to find the torrents 'Config' directory as per the below screenshot. This option is in the WebUI -> Services -> Torrents



You really really shouldnt need to, but if your still haveing huge problems you could try replacing the code:
config_dir=`/usr/local/bin/xml sel -t -m "//bittorrent" -v configdir /cf/conf/config.xml | /usr/local/bin/xml unesc`
With:
config_dir="/path/to/torrent/config/directory/"
Note: Make sure you put the torrent config directory in a location that allows write (i.e. dont put it willy nilly somewhere in the root filesystem - put it in your RAID or on the spare bit of your OS drive)

1) Fix for file permissions when using samba for windozer file sharing.
Why? Well, basically when you log in Anonymously in windows via SMB protocol, it forces your computer to work under FreeNAS's 'ftp' user. Which is fine, untill...
When Tranmission saves a file, it does so under the 'transmission' user. Also fine.
But when you browse to your NAS from windows (being forced to work as the 'ftp' user) and you try and move or whatnot a torrent file (which has been saved as the 'transmission' user) FreeNAS cracks a wobbly and tells you that you dont have permission to do that (as all it sees is one user trying to fiddle with anothers files)

So we can baindaid her up, with the below additions to the 'Auxiliary paramaters' section

This forces the windows client to pretend to be the 'transmission' user, not the 'ftp' user. So it all plays nice!

2) Blocklist updater.
By default, FreeNAS doesnt use a blocklist (in fact, it disables it quite semi-permantly). We can fixxer that up however.

The script below is also taken from the above wiki - I prefer this one because of the location it gets its blocklists from (Bluetack) and the felxibility of lists.

(Big note: If you copy this into notepad and save it onto the NAS, youll encounter one big problem: Windows uses CF+LF to dictate end of line (return). *NIX uses just LF. This confuses the bash interpreter and it will not run. Download and use Notepad2 and when you save the file select 'File->Line encoding-> Unix (LF)' to avoid those problems)

First off: youll have to stop FreeNAS from diabling blocklist. Do so by editing the rc.conf variables via System -> Advanced -> RC.CONF in the WebUI


Either way, before running the file, youll have to log into the NAS via SSH (Enable the SSH service, setup a user in the webUI with 'bash' as its shell, and point Putty to your NAS) and run some commands on the file.


Bonus geek points if you noticed the nethack.alt.org telnet settings!

First, use the SU command (and enter admin password to NAS) to gain 'superuser' privelages, browse to the direcrory using the CD command, and CHMOD +x the file. Then you can run it directly to ensure it works. Like ive done below.

(Note the LS -Gl [this lists the directory in colour mode] - the permissions list to the far left of blocklist.sh is -rwxrwxrwx. The x's mean its eXecutable, which is what we need to run the script (this is what CHMOD +x has done to the file - CHange MODe eXecute)



CODE: blocklist.sh
#! /bin/bash

###
# Select the blacklists you would like to download & use, all available lists
# (except exemptions) are listed below
#LISTS="ads-trackers-and-bad-pr0n bogon dshield edu fornonlancomputers hijacked iana-multicast iana-private iana-reserved level1 level2 Microsoft proxy spider spyware"
#
LISTS
="bogon dshield hijacked iana-multicast iana-private iana-reserved level1 level2 Microsoft"

###
# YOU SHOULD NOT EDIT BELOW THIS LINE UNLESS YOU KNOW WHAT YOU ARE DOING
#
url
="http://www.bluetack.co.uk/config/"
config_dir
=`/usr/local/bin/xml sel -t -m "//bittorrent" -v configdir /cf/conf/config.xml | /usr/local/bin/xml unesc`
updated
=0

if [ "$config_dir" == "" ]; then
logger
-t blocklist_update -- Config directory not set
exit 1
fi

if [ ! -d "$config_dir/gzblocklists" ]; then
mkdir
"$config_dir/gzblocklists"
fi

for list in $LISTS; do
file
="$config_dir/gzblocklists/$list.gz"
lsize
=`stat -qf %z "$file" || echo 0`
rsize
=`fetch -apsw 5 $url$list.gz || exit 1`
if [ "$lsize" -ne "$rsize" ]; then
logger
-t blocklist_update -- Updated $list

# need to download to different folder than $config_dir/blocklists
# transmission seems to have trouble at times starting possibly by trying to parse
# gz'd files
fetch
-qapw 5 -o "$file" $url$list.gz
gunzip
-kqf "$file"
mv
"$config_dir/gzblocklists/$list" "$config_dir/blocklists/"
updated
=1
fi
done

if [ "$updated" -eq "1" ]; then
/etc/rc.d/transmission restart &> /dev/null
else
logger
-t blocklist_updater -- All lists up-to-date
fi

Then its a simple matter of setting up a cron job to run it on a schedule. Below is my settings for midnight every Wednesday.

Note: Your path to the script will differ.

Friday, July 2, 2010

Freenas extras

So after trials in a virtual machine, and three installs on ol' Frankenputer, FrankenNAS lives! Insert demoic laughter and all that.

I was going to do a big walkthrough to installing FreeNAS, but my apathy knows no bounds. Plus, google can tell you how to install it. I think you can install it falling over, actually.

  • Download FreeNAS from http://sourceforge.net/projects/freenas/
  • Burn CD - imgBurn or the like will be a-ok
  • Pop into CD drive, set BIOS to boot from CD and reboot
  • FreeNAS will boot, and run off the CD
  • Youll want to install it to a HDD or CF card (you can get IDE -> CF card adaptors)
  • I went for a Full install + swap. Swap is always fun. Keep in mind that booting the OS off your RAID (if you have a hardware RAID card) is a terrible idea. Grab either the IDE->CF card or a 40/80GB HDD and boot it off that.
  • Once its installed (all what, 15 seconds?) remove the CD and reboot. This time it will be off the HDD
  • Youll end up back at the same menu now. (It runs off the CD exactly the same, however you dont want it to run off the CD permantly - it wont perform all that well)
  • Now press 1. and setup the network (It's likely it guessed all that for you, but configuring it cant hurt) Youll have to select which network device to use (you may well only have 1 so go nuts) Im not sure how its wireless support goes here - I would advise against it anyway.
  • Once thats configured, press 2. Here you should select DHCP (unless you want to manually enter a IP address) and select no for IPv6. It will now give you a funky IP address.
  • FIN
You can now browse to that IP address in your webbrowser! As shown below, it will tell you the IP address you can log in with below.


The steam special is insanity. Oh, and also FreeNAS is shown.

Youll want to set up your hard drives first before hand, obviously.

This post will leave you to play with the webui. Google is your friend here.

Stay tuned for:
  • Transmission (Torrent client) blocklist
  • Transmission RSS auto-downloader
  • Transmission RSS auto-downloader email hack
  • Custom script ive used for email status reports
  • PS3 Media server setup (The pre-installed uPNP server is shite, sorry)
  • Performance Tweaks?

Monday, June 21, 2010

RAID Rundown

A rundown of what I intend to do with FrankeNAS, RAID, and OS differences. Please note its written from a Home media NAS perspective (with some additional services like UPNP, torrents, etc) Not a corporate enviroment.

What is FrankenNAS?
Well, I have a always-on server sitting next to my TV called Frankenputer. Its name came from its origin - it started its life with me as a throw-out desktop from work. It had been sitting covered up by archived paperwork in a small and forgotten shed. I was given it free, and cobbled bits and pieces of it and old hardware I had lying around into a working computer.
Its been running a custom version of XP I made up for a while. Unfortunatly, all ive done is add 1tb external drives to it. These drives have all my digital files, no backup, are tying up a full powerboard, and full, making organizing files a nightmare.
I intend to make a RAID home server, basically a Network Attached Storage (NAS) PC. Hence, FrankenNAS

Kryten says: Frankenstein was the creator - not the monster. It's a common misconception, held by all truly stupid people!

Yes I know Kryters, but I think people will get it.

Porject Goals
  • Numerous Terabyte hard drives linked into one volume under RAID-5
  • Web administration
  • Email notifications
  • Run headless (No screen/keyboard/whatnot)
  • Torrent client with web administration
  • MP3 player with iPhone remote capabilities
  • Reliable (Windows PC kept dropping of the network for no apparent reason daily)
Whats RAID?
Redundant Array of Independent Disks (Originally Redundant Array of Inexpensive Disks, but thats not longer the only option). Its a number of disks set up to work together. Depending on your RAID level config, it can provide a level of redundancy (i.e. one or two drives can fail, and you wont lose your data), possibly performance increases, and a godsend to me, one volume between all the drives.
So in my RAID-5 setup, 3 x 2tb hard drives would show up as a single 3.7tb 'drive', which can suffer a single hard drive failure, and keep chugging along OK.

What are the numbers after RAID for?
Theres different 'levels' of RAID. the most common are 0, 1, 5, and 1+0 (usually written as RAID-10)
RAID 0: Multiple disks as one volume. No redundancy, and therefore not a true RAID.
RAID 1: Contents of one disk is identically mirrored to another: Total capacity is half the total raw storage space.
RAID 5: Using funky XOR binary mathematics, parity, and a little fairy dust, Shares data and parity amongst disks. Total storage capacity is the total storage space of the drives minus one hard drives worth of storage
RAID 1+0 (RAID 10): A combination of RAID 1 and RAID 0 - pairs of RAID 1 (i.e. mirrored) drives in a large, single volume

Why RAID 5
Better storage capacity. For example, a 4 x 2tb RAID 5 has 6tb (minus 7%) storage space and can handle one drive failure. A 4 x 2tb RAID 10 drive has 4tb (minus 7%) storage space and can handle one drive failure per pair. So if the two failures were HDD 1 & 2 of a pair, goodbye data.

My decision: RAID-5, balance between storage space (I'm not made of money) and some redundancy (can lose a single drive OK)

Oh, so RAID = backup! Yay sausages!
No, and if you dump important data on a RAID and lose it all you had it coming.
RAID DOES NOT MEAN BACKUP.
RAID will give you some level of protection vs hardware failure. It will be useless if:
  • PSU dies, taking 2+ HDD's
  • More than 1 HDD dies
  • If using hardware RAID - If a hardware RAID card dies and you cant get a similar replacement, same diff
  • Someone breaks into your house and sledgehammers your box
  • Flood, fire, acts of god, etc
Always have a backup plan.

Why minus 7% on total capacity?
Manufactures rip everyone off with their total storage PR crap. A drive may have 1TB of storage space avaliable, but your Operating System (OS) must create a file system to archive it, which takes storage space. Usually this is approx 7% of the total drive. It would be like getting a 1000 page blank book from a publisher and filled it with information. You can just dump it in, you have to have contents pages, index pages... So your actual information usage will be less than 1000 pages, it may be more like 930 useful pages and 30 pages for administrative stuff (legal, contents, index, etc) - or 7%.

Software RAID and Hardware RAID? WTF?
Theres two main ways of seting up a RAID - Software and Hardware.
Hardware involves a RAID card (PCI usually, i think, but you might be able to get ones for faster slots). The RAID card (If its any good) does the calculations on parity/sharing/whatnot on it, which means the computers main CPU isnt tied up doing all sorts of crap. Hardware is expensive (>AUD$150-200 for a 8 port RAID controller) but is supposed to be the shiznit.
Software RAID is managed through a OS (Linux, BSD, Windoz, etc). This means the CPU is doing the hard work on calculations. On the upside, its cheaper (cheap as free in many cases) and should do OK for a home system if the budget is tight/Female of the house requires new shoes.

My decision: Software RAID (Not made of money, blah blah)

Which OS to use?
FreeNAS
  • Webui
  • Torrents build in
  • Software RAID
  • FreeBSD-based
  • Some argue not as mature as a Linux NAS system
  • UPNP (Fuppes) Built in
  • Basically read for Home server use
  • OS so small it can run on next to nothing hardware and can fit onto a CF card into a IDE->CF flash card. This means no HDD specifically for the OS
  • Can mount + share other OS's drives (NTFS etc). So if a mate comes around, you can plug in their external USB drives and share it.
The problem I had was when I ran the torrent program, it shut my internet down. After a lot of fiddling, I've found that it was my cheap-o router which was the problem. (I think torrents + it are the reason i was having problems in the past - after 24 hours of torrents my internet connection died and it required a restart)
The security log showed it was blocking massive amounts of SYN FLOOD. Basically the amount of data the 'transmission' torrent program was throwing up was triggering its firewall protection. So, considering ive never used its firewall, I disabled it and all is right again (hopefully even better now)

OpenFiler
  • WebUI
  • No torrent built in
  • Software RAID
  • Linux based
  • Arguably more mature codebase
  • OS needs hard drive
  • Cannot mount existing file systems - have to format and copy across
Due to the above.. Meh! Its built toward professional deployment, which is what it is best used for.

Windows
Ill break down the windows option here in installation steps
  1. Throw out CD
  2. Save time, just take a sledgehammer to your HDD's
  3. Same result, less time

My decision: FreeNAS!

Friday, June 18, 2010

FrankenNAS?

Alright, its not the project. The project is ready to be sanded and painted, which will take a bit of time.

In the meantime, I'm thoroughly fed up with my headless server sitting next to my plasma/PS3. Its got 1TB external HDD's all over it, and its starting to look like Medusa's hair with all the cables snaking over it.

Im out of HDD space AGAIN, and Its completly disorganized, requires a whole 8 socket powerboard for itself, and its running... XP. But a customized XP, if that makes it any better.

Ive sucked it up and forked out for 3 x 2TB HDDS (for a start, im anticipating ending up at 8 x 2TB when i have more disposable income) and I plan to be running a RAID 5 array shortly.

So tonight, while im recovering from the damn flu, im playing around with FreeNAS on a Virtual Machine. Looking pretty good currently!