The basic idea of a backup is to have an extra copy of something so that you can replace it if the original file is dumped or damaged.
What should you back up? Anything that won't be replaced by re-installing your system and programs: your word processing files, pictures, emails, music, etc. If you're on a Windows machine, this is pretty much everything in the folder "My Documents", plus whatever folder you keep your stories in if it's not in there (you are keeping your stories all in one place, right? Good. If not, it's time to do a bit of housecleaning -- keeping everything in one place makes files easier to find, back up, and restore).
How often should you back up your computer? As often as necessary. In other words, you should back up a file whenever you make changes to it that you wouldn't want to lose.
Printing out hard copies of a finished short story or novel is a fine idea -- but then you're forced to type or scan the whole thing in again if the only electronic version is lost. Making a backup of a file on the same hard drive -- which some programs do by default -- can help you out if an individual file is badly damaged in a computer crash, but will be no help at all if your hard drive dies.
So, for optimal file safety, you need to back it up onto something other than your hard drive. Floppy disks are better than nothing, but can be damaged by magnetic fields from speakers and TVs and are too small to hold some files. Keychain flash drives are okay, but can be a little flaky and are easy to lose. Burning your files to CDs every so often is better -- CDs are more durable than magnetic media, and once written can be destroyed but not tampered with.
If you've got a file that's particularly important to you, it's best to keep a copy in a secure place that's not your residence. Several authors have lost precious manuscripts and computer files to floods and fires. There are many services (such as Apple's .Mac) which offer online storage for files, but remember my earlier warning about keeping social security numbers etc. away from prying online eyes -- some files you might be better off storing as burned CDs and printouts in a small fireproof safe.
Manual backups can be aggravating, but there are plenty of programs like Retrospect, LaCie Backup, etc. to make things fairly painless.
