Advanced Multimedia on the Linux Command Line

There was a time that Apple macOS was the best platform to handle multimedia (audio, image, video). This might be still true in the GUI space. But Linux presents a much wider range of possibilities when you go to the command line, specially if you want to:

  • Process hundreds or thousands of files at once
  • Same as above, organized in many folders while keeping the folder structure
  • Same as above but with much fine grained options, including lossless processing that most GUI tools won’t give you

The Open Source community has produced state of the art command line tools as ffmpeg, exiftool and others, which I use every day to do non-trivial things, along with Shell advanced scripting. Sure, you can get these tools installed on Mac or Windows, and you can even use almost all these recipes on these platforms, but Linux is the native platform for these tools, and easier to get the environment ready.

These are my personal notes and I encourage you to understand each step of the recipes and adapt to your workflows. It is organized in Audio, Video and Image+Photo sections.

I use Fedora Linux and I mention Fedora package names to be installed. You can easily find same packages on your Ubuntu, Debian, Gentoo etc, and use these same recipes.


Show information (tags, bitrate etc) about a multimedia file

ffprobe file.mp3
ffprobe file.m4v
ffprobe file.mkv

Lossless conversion of all FLAC files into more compatible, but still Open Source, ALAC

ls *flac | while read f; do
	ffmpeg -i "$f" -acodec alac -vn "${f[@]/%flac/m4a}" < /dev/null;

Convert all FLAC files into 192kbps MP3

ls *flac | while read f; do
   ffmpeg -i "$f" -qscale:a 2 -vn "${f[@]/%flac/mp3}" < /dev/null;

Convert all FLAC files into ~256kbps AAC with Fraunhofer AAC encoder

First, make sure you have Negativo17 build of FFMPEG, so run this as root:

dnf config-manager --add-repo=
dnf update ffmpeg

Now encode:

ls *flac | while read f; do
   ffmpeg -i "$f" -vn -c:a libfdk_aac -vbr 5 "${f[@]/%flac/m4a}" < /dev/null;

Same as above but under a complex directory structure

# Create identical directory structure under new "alac" folder
find . -type d | while read d; do
   mkdir -p "alac/$d"

find . -name "*$FMT" | sort | while read f; do
   ffmpeg -i "$f" -acodec alac -vn "alac/${f[@]/%$FMT/m4a}" < /dev/null;

Convert APE+CUE, FLAC+CUE, WAV+CUE album-on-a-file into a one file per track ALAC or MP3

If some of your friends has the horrible tendency to commit this crime and rip CDs as 1 file for entire CD, there is an automation to fix it. APE is the most difficult and this is what I’ll show. FLAC and WAV are shortcuts of this method.

  1. Make a lossless conversion of the APE file into something more manageable, as WAV:
    ffmpeg -i audio-cd.ape audio-cd.wav
  2. Now the magic: use the metadata on the CUE file to split the single file into separate tracks, renaming them accordingly. You’ll need the shnplit command, available in the shntool package on Fedora (to install: yum install shntool):
    shnsplit -t "%n • %p ♫ %t" audio-cd.wav < audio-cd.cue
  3. Now you have a series of nicely named WAV files, one per CD track. Lets convert them into lossless ALAC using one of the above recipes:
    ls *wav | while read f; do
       ffmpeg -i "$f" -acodec alac -vn "${f[@]/%wav/m4a}" < /dev/null;

    This will get you lossless ALAC files converted from the intermediary WAV files. You can also convert them into FLAC or MP3 using one of the other recipes above.

Now the files are ready for your tagger.


Add chapters and soft subtitles from SRT file to M4V/MP4 movie

This is a lossless and fast process, chapters and subtitles are added as tags and streams to the file; audio and video streams are not reencoded.

  1. Make sure your SRT file is UTF-8 encoded:
    bash$ file ISO-8859 text, with CRLF line terminators

    It is not UTF-8 encoded, it is some ISO-8859 variant, which I need to know to correctly convert it. My example uses a Brazilian Portuguese subtitle file, which I know is ISO-8859-15 (latin1) encoded because most latin scripts use this encoding.

  2. Lets convert it to UTF-8:
    bash$ iconv -f latin1 -t utf8 >
    bash$ file UTF-8 Unicode text, with CRLF line terminators
  3. Check chapters file:
    bash$ cat chapters.txt
    CHAPTER01NAME=Chapter 1
    CHAPTER02NAME=Chapter 2
    CHAPTER03NAME=Chapter 3
  4. Now we are ready to add them all to the movie along with setting the movie name and embedding a cover image to ensure the movie looks nice on your media player list of content. Note that this process will write the movie file in place, will not create another file, so make a backup of your movie while you are learning:
    MP4Box -ipod \
           -itags 'track=The Movie Name:cover=cover.jpg' \
           -add '' \
           -chap 'chapters.txt:lang=eng' \

The MP4Box command is part of GPac. has a large collection of subtitles in many languages and you can search its database with the IMDB ID of the movie. And ChapterDB has the same for chapters files.

Decrypt and rip a DVD the loss less way

  1. Make sure you have the RPMFusion and the Negativo17 repos configured
  2. Install libdvdcss and vobcopy
    dnf -y install libdvdcss vobcopy
  3. Mount the DVD and rip it, has to be done as root
    mount /dev/sr0 /mnt/dvd;
    cd /target/folder;
    vobcopy -m /mnt/dvd .

You’ll get a directory tree with decrypted VOB and BUP files. You can generate an ISO file from them or, much more practical, use HandBrake to convert the DVD titles into MP4/M4V (more compatible with wide range of devices) or MKV/WEBM files.

Convert 240fps video into 30fps slow motion, the loss-less way

Modern iPhones can record videos at 240 or 120fps so when you’ll watch them at 30fps they’ll look slow-motion. But regular players will play them at 240 or 120fps, hiding the slo-mo effect.
We’ll need to handle audio and video in different ways. The video FPS fix from 240 to 30 is loss less, the audio stretching is lossy.

# make sure you have the right packages installed
dnf install mkvtoolnix sox gpac faac

# Script by Avi Alkalay
# Freely distributable


# Get original video frame rate
ifps=`ffprobe -v error -select_streams v:0 -show_entries stream=r_frame_rate -of default=noprint_wrappers=1:nokey=1 "$f" < /dev/null  | sed -e 's|/1||'`

# exit if not high frame rate
[[ "$ifps" -ne 120 ]] && [[ "$ifps" -ne 240 ]] && exit

fpsRateInv=`awk "BEGIN {print $ofps/$ifps}"`

# loss less video conversion into 30fps through repackaging into MKV
mkvmerge -d 0 -A -S -T \
	--default-duration 0:${ofps}fps \
	"$f" -o "v$noext.mkv"

# loss less repack from MKV to MP4
ffmpeg -loglevel quiet -i "v$noext.mkv" -vcodec copy "v$noext.mp4"

# extract subtitles, if original movie has it
ffmpeg -loglevel quiet -i "$f" "s$"

# resync subtitles using similar method with mkvmerge
mkvmerge --sync "0:0,${fpsRate}" "s$" -o "s$noext.mkv"

# get simple synced SRT file
rm "s$"
ffmpeg -i "s$noext.mkv" "s$"

# remove undesired formating from subtitles
sed -i -e 's|<font size="8"><font face="Helvetica">\(.*\)</font></font>|\1|' "s$"

# extract audio to WAV format
ffmpeg -loglevel quiet -i "$f" "$noext.wav"

# make audio longer based on ratio of input and output framerates
sox "$noext.wav" "a$noext.wav" speed $fpsRateInv

# lossy stretched audio conversion back into AAC (M4A) 64kbps (because we know the original audio was mono 64kbps)
faac -q 200 -w -s --artist a "a$noext.wav"

# repack stretched audio and video into original file while removing the original audio and video tracks
cp "$f" "${noext}-slow.${ext}"
MP4Box -ipod -rem 1 -rem 2 -rem 3 -add "v$noext.mp4" -add "a$noext.m4a" -add "s$" "${noext}-slow.${ext}"

# remove temporary files 
rm -f "$noext.wav" "a$noext.wav" "v$noext.mkv" "v$noext.mp4" "a$noext.m4a" "s$" "s$noext.mkv"

1 Photo + 1 Song = 1 Movie

If the audio is already AAC-encoded, create an MP4/M4V file:

ffmpeg -loop 1 -framerate 0.2 -i photo.jpg -i song.m4a -shortest -c:v libx264 -tune stillimage -vf scale=960:-1 -c:a copy movie.m4v

The above method will create a very efficient 0.2 frames per second (-framerate 0.2) H.264 video from the photo while simply adding the audio losslessly. Such very-low-frames-per-second video may present sync problems with subtitles on some players. In this case simply remove the -framerate 0.2 parameter to get a regular 25fps video with the cost of a bigger file size.
The -vf scale=960:-1 parameter tells FFMPEG to resize the image to 960px width and calculate the proportional height. Remove it in case you want a video with the same resolution of the photo. A 12 megapixels photo file (around 4032×3024) will get you a near 4K video.
If the audio is MP3, create an MKV file:

ffmpeg -loop 1 -framerate 0.2 -i photo.jpg -i song.mp3 -shortest -c:v libx264 -tune stillimage -vf scale=960:-1 -c:a copy movie.mkv

If audio is not AAC/M4A but you still want an M4V file, convert audio to AAC 192kbps:

ffmpeg -loop 1 -framerate 0.2 -i photo.jpg -i song.mp3 -shortest -c:v libx264 -tune stillimage -vf scale=960:-1 -c:a aac -strict experimental -b:a 192k movie.mkv

See more about FFMPEG photo resizing.

Image and Photo

Move images with no EXIF header to another folder

mkdir noexif;
exiftool -filename -T -if '(not $datetimeoriginal or ($datetimeoriginal eq "0000:00:00 00:00:00"))' *jpg | xargs -i mv "{}" noexif/

Set EXIF photo create time based on file create time

Warning: use this only if image files have correct creation time on filesystem and if they don’t have an EXIF header.

exiftool -overwrite_original '-DateTimeOriginal< ${FileModifyDate}' *CR2 *JPG *jpg

Rotate photos based on EXIF’s Orientation flag, plus make them progressive. Lossless

jhead -autorot -cmd "jpegtran -progressive '&i' > '&o'" -ft *jpg

Rename photos to a more meaningful filename

This process will rename silly, sequential, confusing and meaningless photo file names as they come from your camera into a readable, sorteable and useful format. Example:

IMG_1234.JPG2015.07.24-17.21.33 • Max playing with water【iPhone 6s✚】.jpg

Note that new file name has the date and time it was taken, whats in the photo and the camera model that was used.

  1. First keep the original filename, as it came from the camera, in the OriginalFileName tag:
    exiftool -overwrite_original '-OriginalFileName<${filename}' *CR2 *JPG *jpg
  2. Now rename:
    exiftool '-filename<${DateTimeOriginal} 【${Model}】%.c.%e' -d %Y.%m.%d-%H.%M.%S *CR2 *JPG *jpg
  3. Remove the ‘0’ index if not necessary:
    \ls *JPG *jpg | while read f; do
        nf=`echo "$f" | sed -e 's/0.JPG/.jpg/i'`;
        t=`echo "$f" | sed -e 's/0.JPG/1.jpg/i'`;
        [[ ! -f "$t" ]] && mv "$f" "$nf";
  4. Optional: make lower case extensions:
    \ls *JPG | while read f; do
        nf=`echo "$f" | sed -e 's/JPG/jpg/'`;
        mv "$f" "$nf";
  5. Optional: simplify camera name, for example turn “Canon PowerShot G1 X” into “Canon G1X” and make lower case extension at the same time:
    ls *JPG *jpg | while read f; do
        nf=`echo "$f" | sed -e 's/Canon PowerShot G1 X/Canon G1X/;
          s/iPhone 6s Plus/iPhone 6s✚/;
          s/iPhone 7 Plus/iPhone 7✚/;
          s/Canon PowerShot SD990 IS/Canon SD990 IS/;
        mv "$f" "$nf";

You’ll get file names as 2015.07.24-17.21.33 【Canon 5D Mark II】.jpg. If you took more then 1 photo in the same second, exiftool will automatically add an index before the extension.

Even more semantic photo file names based on Subject tag

\ls *【*】* | while read f; do
	s=`exiftool -T -Subject "$f"`;
	nf=`echo "$f" | sed -e "s/ 【/ • $s 【/; s/\:/∶/g;"`;
	mv "$f" "$nf";

Full rename: a consolidation of some of the previous commands

exiftool '-filename<${DateTimeOriginal} • ${Subject} 【${Model}】%.c.%e' -d %Y.%m.%d-%H.%M.%S *CR2 *JPG *jpg

Set photo “Creator” tag based on camera model

  1. First list all cameras that contributed photos to current directory:
    exiftool -T -Model *jpg | sort -u

    Output is the list of camera models on this photos:

    Canon EOS REBEL T5i
    iPhone 4
    iPhone 4S
    iPhone 5
    iPhone 6
    iPhone 6s Plus
  2. Now set creator on photo files based on what you know about camera owners:
    CRE="John Doe";    exiftool -overwrite_original -creator="$CRE" -by-line="$CRE" -Artist="$CRE" -if '$Model=~/DSC-H100/'            *.jpg
    CRE="Jane Black";  exiftool -overwrite_original -creator="$CRE" -by-line="$CRE" -Artist="$CRE" -if '$Model=~/Canon EOS REBEL T5i/' *.jpg
    CRE="Mary Doe";    exiftool -overwrite_original -creator="$CRE" -by-line="$CRE" -Artist="$CRE" -if '$Model=~/iPhone 5/'            *.jpg
    CRE="Peter Black"; exiftool -overwrite_original -creator="$CRE" -by-line="$CRE" -Artist="$CRE" -if '$Model=~/iPhone 4S/'           *.jpg
    CRE="Avi Alkalay"; exiftool -overwrite_original -creator="$CRE" -by-line="$CRE" -Artist="$CRE" -if '$Model=~/iPhone 6s Plus/'      *.jpg

Recursively search people in photos

If you geometrically mark people faces and their names in your photos using tools as Picasa, you can easily search for the photos which contain “Suzan” or “Marcelo” this way:

exiftool -fast -r -T -Directory -FileName -RegionName -if '$RegionName=~/Suzan|Marcelo/' .

-Directory, -FileName and -RegionName specify the things you want to see in the output. You can remove -RegionName for a cleaner output.
The -r is to search recursively. This is pretty powerful.

Make photos timezone-aware

Your camera will tag your photos only with local time on CreateDate or DateTimeOriginal tags. There is another set of tags called GPSDateStamp and GPSTimeStamp that must contain the UTC time the photos were taken, but your camera won’t help you here. Hopefully you can derive these values if you know the timezone the photos were taken. Here are two examples, one for photos taken in timezone -02:00 (Brazil daylight savings time) and on timezone +09:00 (Japan):

exiftool -overwrite_original '-gpsdatestamp<${CreateDate}-02:00' '-gpstimestamp<${CreateDate}-02:00' *.jpg
exiftool -overwrite_original '-gpsdatestamp<${CreateDate}+09:00' '-gpstimestamp<${CreateDate}+09:00' Japan_Photos_folder

Use exiftool to check results on a modified photo:

exiftool -s -G -time:all -gps:all 2013.10.12-23.45.36-139.jpg
[EXIF]          CreateDate                      : 2013:10:12 23:45:36
[Composite]     GPSDateTime                     : 2013:10:13 01:45:36Z
[EXIF]          GPSDateStamp                    : 2013:10:13
[EXIF]          GPSTimeStamp                    : 01:45:36

This shows that the local time when the photo was taken was 2013:10:12 23:45:36. To use exiftool to set timezone to -02:00 actually means to find the correct UTC time, which can be seen on GPSDateTime as 2013:10:13 01:45:36Z. The difference between these two tags gives us the timezone. So we can read photo time as 2013:10:12 23:45:36-02:00.

Geotag photos based on time and Moves mobile app records

Moves is an amazing app for your smartphone that simply records for yourself (not social and not shared) everywhere you go and all places visited, 24h a day.

  1. Make sure all photos’ CreateDate or DateTimeOriginal tags are correct and precise, achieve this simply by setting correctly the camera clock before taking the pictures.
  2. Login and export your Moves history.
  3. Geotag the photos informing ExifTool the timezone they were taken, -08:00 (Las Vegas) in this example:
    exiftool -overwrite_original -api GeoMaxExtSecs=86400 -geotag ../moves_export/gpx/yearly/storyline/storyline_2015.gpx '-geotime<${CreateDate}-08:00' Folder_with_photos_from_trip_to_Las_Vegas

Some important notes:

  • It is important to put the entire ‘-geotime’ parameter inside simple apostrophe or simple quotation mark (), as I did in the example.
  • The ‘-geotime’ parameter is needed even if image files are timezone-aware (as per previous tutorial).
  • The ‘-api GeoMaxExtSecs=86400’ parameter should not be used unless the photo was taken more than 90 minutes of any detected movement by the GPS.

Concatenate all images together in one big image

  • In 1 column and 8 lines:
    montage -mode concatenate -tile 1x8 *jpg COMPOSED.JPG
  • In 8 columns and 1 line:
    montage -mode concatenate -tile 8x1 *jpg COMPOSED.JPG
  • In a 4×2 matrix:
    montage -mode concatenate -tile 4x2 *jpg COMPOSED.JPG

The montage command is part of the ImageMagick package.

Docker on Bluemix with automated full-stack deploys and delivery pipelines


This document explains working examples on how to use Bluemix platform advanced features such as:

  • Docker on Bluemix, integrated with Bluemix APIs and middleware
  • Full stack automated and unattended deployments with DevOps Services Pipeline, including Docker
  • Full stack automated and unattended deployments with cf command line interface, including Docker

For this, I’ll use the following source code structure:

The source code currently brings to life (as an example), integrated with some Bluemix services and Docker infrastructure, a PHP application (the WordPress popular blogging platform), but it could be any Python, Java, Ruby etc app.

This is how full stack app deployments should be

Before we start: understand Bluemix 3 pillars

I feel it is important to position what Bluemix really is and which of its parts we are going to use. Bluemix is composed of 3 different things:

  1. Bluemix is a hosting environment to run any type of web app or web service. This is the only function provided by the CloudFoundry Open Source project, which is an advanced PaaS that lets you provision and de-provision runtimes (Java, Python, Node etc), libraries and services to be used by your app. These operations can be triggered through the portal or by the cf command from your laptop. IBM has extended this part of Bluemix with functions not currently available on CloudFoundry, notably the capability of executing regular VMs and Docker containers.
  2. Bluemix provides pre-installed libraries, APIs and middleware. IBM is constantly adding functions to the Bluemix marketplace, such as cognitive computing APIs in the Watson family, data processing middleware such as Spark and dashDB, or even IoT and Blockchain-related tools. These are high value components that can add a bit of magic to your app. Many of those are Open Source.
  3. DevOps Services. Accessible from, it provides:
    • Public and private collaborative Git repositories.
    • UI to build, manage and execute the app delivery pipeline, which does everything needed to transform your pure source code into a final running application.
    • The Track & Plan module, based on Rational Team Concert, to let your team mates and clients exchange activities and control project execution.

This tutorial will dive into #1 and some parts of #3, while using some services from #2.

The architecture of our app

Docker on Bluemix with services

When fully provisioned, the entire architecture will look like this. Several Bluemix services (MySQL, Object store) packaged into a CloudFoundry App (bridge app) that serves some Docker containers that in turns do the real work. Credentials to access those services will be automatically provided to the containers as environment variables (VCAP_SERVICES).

Structure of Source Code

The example source code repo contains boilerplate code that is intentionally generic and clean so you can easily fork, add and modify it to fit your needs. Here is what it contains:

bridge-app folder and manifest.yml file
The CloudFoundry manifest.yml that defines app name, dependencies and other characteristics to deploy the app contents under bridge-app.
Each directory contains a Dockerfile and other files to create Docker containers. In this tutorial we’ll use only the phpinfo and wordpress directories, but there are some other useful examples you can use.
.bluemix folder
When this code repository is imported into Bluemix via the “Deploy to Bluemix” button, metadata in here will be used to set up your development environment under DevOps Services.
admin folder
Random shell scripts, specially used for deployments.

Watch the deployment

The easiest way to deploy the app is through DevOps Services:

  1. Click to deploy

    Deploy to Bluemix

  2. Provide a unique name to your copy of the app, also select the target Bluemix space
    Deploy to Bluemix screen
  3. Go to DevOps Services ➡ find your project clone ➡ select Build & Deploy tab and watch
    Full Delivery Pipeline on Bluemix

Under the hood: understand the app deployment in 2 strategies

Conceptually, these are the things you need to do to fully deploy an app with Docker on Bluemix:

  1. Instantiate external services needed by your app, such as databases, APIs etc.
  2. Create a CloudFoundry app to bind those services so you can handle them all as one block.
  3. Create the Docker images your app needs and register them on your Bluemix private Docker Registry (equivalent to the public Docker Hub).
  4. Instantiate your images in executable Docker containers, connecting them to your backend services through the CloudFoundry app.

The idea is to encapsulate all these steps in code so deployments can be done entirely unattended. Its what I call brainless 1-click deployment. There are 2 ways to do that:

  • A regular shell script that extensively uses the cf command. This is the admin/deploy script in our code.
  • An in-code delivery pipeline that can be executed by Bluemix DevOps Services. This is the .bluemix/pipeline.yml file.

From here, we will detail each of these steps both as commands (on the script) and as stages of the pipeline.

  1. Instantiation of external services needed by the app…

    I used the cf marketplace command to find the service names and plans available. ClearDB provides MySQL as a service. And just as an example, I’ll provision an additional Object Storage service. Note the similarities between both methods.

    Deployment Script
    cf create-service \
      cleardb \
      spark \
    cf create-service \
      Object-Storage \
      Free \
    Delivery Pipeline

    When you deploy your app to Bluemix, DevOps Services will read your manifest.yml and automatically provision whatever is under the declared-services block. In our case:

        label: cleardb
        plan: spark
        label: Object-Storage
        plan: Free
  2. Creation of an empty CloudFoundry app to hold together these services

    The manifest.yml file has all the details about our CF app. Name, size, CF build pack to use, dependencies (as the ones instantiated in previous stage). So a plain cf push will use it and do the job. Since this app is just a bridge between our containers and the services, we’ll use minimum resources and the minimum noop-buildpack. After this stage you’ll be able to see the app running on your Bluemix console.

    Deployment Script
    Delivery Pipeline
    Stage named “➊ Deploy CF bridge app” simply calls cf push;
  3. Creation of Docker images

    The heavy lifting here is done by the Dockerfiles. We’ll use base CentOS images with official packages only in an attempt to use best practices. See phpinfo and wordpress Dockerfiles to understand how I improved a basic OS to become what I need.

    The cf ic command is basically a clone of the well known docker command, but pre-configured to use Bluemix Docker infrastructure. There is simple documentation to install the IBM Containers plugin to cf.

    Deployment Script
    cf ic build \
       -t phpinfo_image \
    cf ic build \
       -t wordpress_image \
    Delivery Pipeline

    Stages handling this are “➋ Build phpinfo Container” and “➍ Build wordpress Container”.

    Open these stages and note how image names are set.

    After this stage, you can query your Bluemix private Docker Registry and see the images there. Like this:

    $ cf ic images
    REPOSITORY                                          TAG     IMAGE ID      CREATED     SIZE     latest  69d78b3ce0df  3 days ago  104.2 MB   latest  a801735fae08  3 days ago  117.2 MB

    A Docker image is not yet a container. A Docker container is an image that is being executed.

  4. Run containers integrated with previously created bridge app

    To make our tutorial richer, we’ll run 2 sets of containers:

    1. The phpinfo one, just to see how Bluemix gives us an integrated environment
      Deployment Script
      cf ic run \
         -P \
         --env 'CCS_BIND_APP=bridge-app-name' \
         --name phpinfo_instance \;
      IP=`cf ic ip request | 
          grep "IP address" | 
          sed -e "s/.* \"\(.*\)\" .*/\1/"`;
      cf ic ip bind $IP phpinfo_instance;
      Delivery Pipeline

      Equivalent stage is “➌ Deploy phpinfo Container”.

      Open this stage and note how some environment variables are defined, specially the BIND_TO.

      Bluemix DevOps Services default scripts use these environment variables to correctly deploy the containers.

      The CCS_BIND_APP on the script and BIND_TO on the pipeline are key here. Their mission is to make the bridge-app’s VCAP_SERVICES available to this container as environment variables.

      In CloudFoundry, VCAP_SERVICES is an environment variable containing a JSON document with all credentials needed to actually access the app’s provisioned APIs, middleware and services, such as host names, users and passwords. See an example below.

    2. A container group with 2 highly available, monitored and balanced identical wordpress containers
      Deployment Script
      cf ic group create \
         -P \
         --env 'CCS_BIND_APP=bridge-app-name' \
         --auto \
         --desired 2 \
         --name wordpress_group_instance \
      cf ic route map \
         --hostname some-name-wordpress \
         --domain $DOMAIN \

      The cf ic group create creates a container group and runs them at once.

      The cf ic route map command configures Bluemix load balancer to capture traffic to and route it to the wordpress_group_instance container group.

      Delivery Pipeline

      Equivalent stage is “➎ Deploy wordpress Container Group”.

      Look in this stage’s Environment Properties how I’m configuring container group.

      I had to manually modify the standard deployment script, disabling deploycontainer and enabling deploygroup.

See the results

At this point, WordPress (the app that we deployed) is up and running inside a Docker container, and already using the ClearDB MySQL database provided by Bluemix. Access the URL of your wordpress container group and you will see this:

WordPress on Docker with Bluemix

Bluemix dashboard also shows the components running:

Bluemix dashboard with apps and containers

But the most interesting evidence you can see accessing the phpinfo container URL or IP. Scroll to the environment variables section to see all services credentials available as environment variables from VCAP_SERVICES:

Bluemix VCAP_SERVICES as seen by a Docker container

I use these credentials to configure WordPress while building the Dockerfile, so it can find its database when executing:

RUN yum -y install epel-release;\
	yum -y install wordpress patch;\
	yum clean all;\
	sed -i '\
		         s/.localhost./getenv("VCAP_SERVICES_CLEARDB_0_CREDENTIALS_HOSTNAME")/ ; \
		s/.database_name_here./getenv("VCAP_SERVICES_CLEARDB_0_CREDENTIALS_NAME")/     ; \
		     s/.username_here./getenv("VCAP_SERVICES_CLEARDB_0_CREDENTIALS_USERNAME")/ ; \
		     s/.password_here./getenv("VCAP_SERVICES_CLEARDB_0_CREDENTIALS_PASSWORD")/ ; \
	' /etc/wordpress/wp-config.php;\
	cd /etc/httpd/conf.d; patch < /tmp/wordpress.conf.patch;\
	rm /tmp/wordpress.conf.patch

So I’m using sed, the text-editor-as-a-command, to edit WordPress configuration file (/etc/wordpress/wp-config.php) and change some patterns there into appropriate getenv() calls to grab credentials provided by VCAP_SERVICES.

Dockerfile best practices

The containers folder in the source code presents one folder per image, each is an example of different Dockerfiles. We use only the wordpress and phpinfo ones here. But I’d like to highlight some best practices.

A Dockerfile is a script that defines how a container image should be built. A container image is very similar to a VM image, the difference is more related to the file formats that they are stored. VMs uses QCOW, VMDK etc while Docker uses layered filesystem images. From the application installation perspective, all the rest is almost the same. But only only Docker and its Dockerfile provides a super easy way to describe how to prepare an image focusing mostly only on your application. The only way to automate this process on the old Virtual Machine universe is through techniques such as Red Hat’s kickstart. This automated OS installation aspect of Dockerfiles might seem obscure or unimportant but is actually the core of what makes viable a modern DevOps culture.

  1. Being a build script, it starts from a base parent image, defined by the FROM command. We used a plain official CentOS image as a starting point. You must select very carefully your parent images, in the same way you select the Linux distribution for your company. You should consider who maintains the base image, it should be well maintained.
  2. Avoid creating images manually, as running a base container, issuing commands manually and then committing it. All logic to prepare the image should be scripted in your Dockerfile.
  3. In case complex file editing is required, capture edits in patches and use the patch command in your Dockerfile, as I did on wordpress Dockerfile.
    To create a patch:

    diff -Naur configfile.txt > configfile.patch

    Then see the wordpress Dockerfile to understand how to apply it.

  4. Always that possible, use official distribution packages instead of downloading libraries (.zip or .tar.gz) from the Internet. In the wordpress Dockerfile I enabled the official EPEL repository so I can install WordPress with YUM. Same happens on the Django and NGINX Dockerfiles. Also note how I don’t have to worry about installing PHP and MySQL client libraries – they get installed automatically when YUM installs wordpress package, because PHP and MySQL are dependencies.

When Docker on Bluemix is useful

CloudFoundry (the execution environment behind Bluemix) has its own Open Source container technology called Warden. And CloudFoundry’s Dockerfile-equivalent is called Buildpack. Just to illustrate, here is a WordPress buildpack for CloudFoundry and Bluemix.

To chose to go with Docker in some parts of your application means to give up some native integrations and facilities naturally and automatically provided by Bluemix. With Docker you’ll have to control and manage some more things for yourself. So go with Docker, instead of a buildpack, if:

  • If you need portability, you need to move your runtimes in and out Bluemix/CloudFoundry.
  • If a buildpack you need is less well maintained then the equivalent Linux distribution package. Or you need a reliable and supported source of pre-packaged software in a way just a major Linux distribution can provide.
  • If you are not ready to learn how to use and configure a complex buildpack, like the Python one, when you are already proficient on your favorite distribution’s Python packaging.
  • If you need Apache HTTPD advanced features as mod_rewrite, mod_autoindex or mod_dav.
  • If you simply need more control over your runtimes.

The best balance is to use Bluemix services/APIs/middleware and native buildpacks/runtimes whenever possible, and go with Docker on specific situations. Leveraging the integration that Docker on Bluemix provides.

WordPress on Fedora with RPM, DNF/YUM

WordPress is packaged for Fedora and can be installed as a regular RPM (with DNF/YUM). The benefits of this method are that you don’t need to mess around with configuration files, filesystem permissions and since everything is pre-packaged to work together, additional configurations are minimal. At the end of this 3 minutes tutorial, you’ll get a running WordPress under an SSL-enabled Apache using MariaDB as its backend.

All commands need to be executed as root. Continue reading

Is Open Source Swift a good thing ?

swift-progOn December 3 Apple has open sourced the Swift programming language on The language was first released (not Open Source yet) about the same time as iOS 8 and was created by Apple to make Mac and iOS app development an easier task. Swift is welcome as one more Open Source language and project but is too early to make a lot of noise about it. Here are my arguments: Continue reading

A nova TI do iPhone

Do PC ao Datacenter, como o iPhone mudou tudo o que fazíamos em TI

A fórmula era ambiciosa para 2007: um telefone com inovadora tela multitoque grande, teclado virtual que finalmente funcionava, SMS repensado e apresentado como uma conversa, aplicação de e-mail com interface extremamente efetiva e clara, inúmeros sensores que interagiam com o mundo físico. E, acima de tudo, um browser completo e avançado, que funcionava tão bem quanto o que tínhamos no desktop. Continue reading

WordPress Community is in Pain

I don’t know about you senior bloggers but I’m starting to hate the way the WordPress community has evolved and what it became.

From a warm and advanced blogging software and ecosystem it is now an aberration for poor site makers. Themes are now mostly commercial, focused on institutional/marketing sites and not blogs anymore. WordPress is simply a very poor tool for this purpose. You can see this when several themes are getting much more complex than WordPress per se. Continue reading

SMS sem Ansiedade

SMS, WhatsApp, iMessage, Hangouts mudaram a forma como nos comunicamos.

Só não podemos nos deixar cair na armadilha de achar que a mensagem entrou no cérebro do destinatário quando aparece ✔✔. Evite ansiedade desnecessária pois o destinatário pode estar ocupado, esqueceu de responder ou simplesmente viu mas não leu direito.

De resto essas Apps são adoráveis mesmo.

Publicado também no Facebook.

Como “Cloudificar” seu Datacenter Atual

Os gerentes de infra-estrutura de TI de hoje em dia tem um novo concorrente inesperado: as Clouds Públicas.

Ele investiu nos últimos anos em energia, rede e computadores de alta qualidade para ver seus usuários internos acabarem preferindo as Clouds Públicas por serem mais ágeis, elásticas e com modelo de cobrança mais preciso (paga só se usar). Continue reading

Ode às Redes Sociais e à Livre Circulação de Pensamento

Tirando uns 60% de conteúdo ainda meio supérfluo, redes como Facebook e Twitter são ferramentas sem precedentes na história da humanidade.

Se você consegue enxergar além da piadinha, da foto do bebê e do bichinho, perceberá que tratam-se de verdadeiras usinas de difusão e circulação de pensamento que mantém a mente fascinada, o raciocinio arejado e o coração aberto.

Não menospreze essas ferramentas alegando que prefere relações pessoais cara a cara. É como rejeitar voar só porque a natureza não te deu asas. É como esnobar Paris só porque você é carioca da gema. Já superamos isso, é uma desculpa ingênua, que não cola, que soa mal e não “cool”.

Seja um partícipe na circulação do pensamento. As idéias, a informação, o pensamento, tudo isso quer ser útil, de alto alcance, para transformar. Não exclusivo, não de difícil acesso e nem caro. Esses sistemas de engajamento podem completar e potencializar o melhor de você como qualquer ferramenta quando usada para o bem, só que de uma forma nunca antes vista na história desta humanidade.

Microsoft Windows na plataforma Power com KVM e QEMU

Com o lançamento de KVM para Power se aproximando no horizonte, tem se falado muito sobre rodar o Microsoft Windows em Power.

Só uma rápida retrospectiva, KVM é a tecnologia do Kernel do Linux que permite rodar máquinas virtuais de forma muito eficiente. E o QEMU é o software que emula diversos aspectos de um computador (portas serias, rede, BIOS/firmware, disco etc). O QEMU existia antes do projeto KVM e possibilita rodar, de forma razoavelmente lenta devido a emulação de todos os aspectos do hardware, outro sistema operacional completo dentro dele.

Continue reading

O Twitter vai acabar, Facebook vai prevalecer

Prevejo (e costumo acertar essas coisas) que a médio prazo o Twitter tende a desaparecer. Mesmo com conteúdo melhor — pelo menos das pessoas que eu sigo —, seu concorrente, o Facebook, tem mais funcionalidades e possibilidades, é mais auto-contido e é mais colorido e diverso, o que o torna mais popular também.

Então acho que muitos continuarão migrando para o Facebook e deixando gradativamente de usar o Twitter, infelizmente.

Publicado também no Facebook

OpenShift for Platform as a Service Clouds

OpenShift-LogoAt the Fedora 20 release party another guy stepped up and presented+demonstrated OpenShift, which was the most interesting new feature from Red Hat for me. First of all I had to switch my mindset about cloud from IaaS (infrastructure as a service, where the granularity are virtual machines) to PaaS. I heard the PaaS buzzword before but never took the time to understand what it really means and its implications. Well, I had to do that at that meeting so I can follow the presentation, of course hammering the presenter with questions all the time.
Continue reading

GMail as mail relay for your Linux home server

Since my Fedora Post-installation Configurations article, some things have changed in Fedora 20. For example, for security and economy reasons, Sendmail does not get installed anymore by default. Here are the steps to make your Linux home computer be able to send system e-mails as alerts or from things that run on cron. All commands should be run as user root. This is certified to work on Fedora 21.

Continue reading

Coisas que aprendi e descobri na Latinoware 2013


Inovação e o Filme do Steve Jobs

Jobs movie Ashton KutcherBom mesmo o filme sobre Steve Jobs, para mostrar como é árida a jornada para a Inovação. Como é solitário, como todos ficam te falando que está tudo errado.

Inovar não é repetir essa mesma palavra 800 vezes em PowerPoints. É ter Visão (A Centelha Motivadora), se preocupar com Detalhes e Persistir. Sendo que Visão é o fator menos importante porque ela muda, refina e se adapta durante a jornada. Persistir é deveras mais dispendioso para conseguir atravessar o mar de antiquados que tentarão te impedir.

Essa é pros loucos, pros deslocados, rebeldes, encrenqueiros, pros pinos redondos nos buracos quadrados… pros que vêem as coisas diferente — eles não curtem regras… Você pode citá-los, discordar deles, glorificar ou denegrí-los, mas a única coisa que você não pode fazer é ignorá-los, porque eles mudam as coisas… eles empurrar a humanidade prá frente, e enquanto alguns os vêem como loucos, nós vemos genialidade, porque aqueles que são loucos o suficiente prá achar que podem mudar o mundo, são os que mudam.

Rob Stiltanen & Ken Segall

Install OS X on a Mac computer from an ISO file

For some reason nobody published a simple guide like this. Maybe nobody tryied this way. I just tryied and it works with OS X Mountain Lion on a Mid 2012 MacBook Air.

If you have a Mac computer or laptop and want to install OS X, and all that you have is the operating system installation ISO image, you just need an external USB storage (disk or pen drive) of 5GB minimum size. Those regular 120GB or 1TB external disks will work too.

Just remember that all data on this external storage will be erased, even if the Mac OS X installation ISO is just 4.7GB. So make a backup of your files and after installtion you can re-format the external disk and recover the files on it.

To make the OS X installation ISO image file usable and bootable from the external storage, use the Mac OS terminal app or, on Linux, use the command line. This is the magic command:

dd if="OS X Install DVD.iso" of=/dev/disk1 bs=10m

You might want to change the red part of this command to the disk name that you get when inserted the external storage. Remember to not use things like disk1s1 or, on Linux, sdc1. The highlighted blue part on these examples are the partition name, and you don’t want that. You want to use the whole storage, otherwise it will not boot the computer.

After the command finishes execution, boot the Mac computer with the alt/option key pressed. Several devices will appear on screen for you to choose wich one to boot. Select the one with the USB logo and called “EFI Boot“.

Mac OS X installation app will boot and you can start the process. Remember that the default behavior here is to upgrade the installed system. If you want a clean install, select the Disk Utility app on the menu and make sure you erase and create a new partition on the Mac internal storage.

As a side technical note, this is all possible because ISO images — primarily designed for optical disks — can also be written to regular other storages as pen drives. And Apple has also put the right bits on these ISO images to allow it to boot from non-optical disks too.

Exchange large amounts of files with me

I maintain a large amount of browseable files at that can be accessible also by SFTP on s

You can access thse files with any SFTP (Secure Shell FTP) client on s

There is a visual way to download files from there using FileZilla. Just follow the steps. Continue reading

Unicode ♥ וניקוד ☻ Уникод ♫ يونيكود

Você sabia que há pouco tempo era impossível misturar diversas línguas numa mesma frase de texto sem a ajuda de um editor multilíngue especial? Mais ainda, que havia idiomas cujas letras sequer tinham uma representação digital sendo impossível usá-las em computadores?

Tudo isso virou passado com o advento do Unicode e para entendê-lo vamos relembrar alguns conceitos: Continue reading

Sobre Profetas e Bolas de Cristal

Este é um artigo sobre Business Intelligence, Business Analytics, Big Data e Data Mining.

Há quem diga que os antigos profetas eram pessoas comuns que proferiam simples consequências lógicas baseadas em observação mais profunda de fatos de seu presente e passado. Tudo o que vemos à nossa volta é resultado de alguma ação, tem uma história e um motivo de ser e de existir. Em contrapartida, seguindo um mesmo raciocínio científico, se algo aparentemente “não tem explicação” é porque ninguém se aprofundou suficientemente nos fatos históricos que o causaram. Continue reading

iPhone Call History Database

Either if you are doing forensics or just want better reports about your call patterns, the iPhone Call History database can be very handfull.

If you have a jailbroken iPhone, you can access the database file directly. If you are not, you can still access it offline simply copying the file from an unencrypted iTunes backup to some other folder on you computer to manipulate it. Here are the real files path inside the iPhone and their counterparts on an iTunes backup folder:
Continue reading

O site de Linux da IBM Brasil foi desativado e isso é bom

Antigo site IBM Linux
Antigo site IBM Linux

O site oficial de Linux da IBM Brasil ficava em e lá era nosso QG virtual quando o Linux Impact Team existia e eu fazia parte dele. Nosso time foi formado na época para estabelecer a idéia de que Linux, Software Livre, Padrões Abertos etc são coisas boas, desmitificar algumas crenças confusas, ajudar clientes IBM a usar Linux com nossos produtos etc. Continue reading

Reduzi o custo de minha Banda Larga NET Virtua em 25%

Liguei na NET hoje pelo 5º mês consecutivo para reclamar que estavam cobrando pelo 5º mês consecutivo o NET Fone, mesmo após eu tê-lo cancelado, 5 meses atrás. Minha frase não foi redundante pois faz 5 meses que pedi cancelamento, faz 5 meses que a conta vem errada, mesmo eu ligando todo mês pedindo correção da conta. Continue reading

Como será sua próxima TV

Antena ou “conversor” digital é bobagem para quem tem TV a cabo. Só serve pra quem precisa captar sinal digital do ar e por enquanto só serve para a cidade de São Paulo.

Sobre OLED vs LED vs LCD vs Plasma, é o tipo da coisa que você só sente a diferença de imagem na loja, quando vê a mesma imagem passando em tecnologias diferentes. O importante é você não entrar numa tecnologia que deixaram pra trás, tipo Plasma, e entrar no que é bom em termos de custo/benefício hoje. Meu pai comprou uma LG LCD uns meses atrás com fatores de contraste e luminosidade superbons e preço bacana. E lembre-se que 3 minutos depois que o filme começar, deitado no sofazão, comendo pipoca, o que importa é a emoção e não mais a tecnologia. Este é o fator mais importante. Continue reading

Como comprar e quanto custa um iPhone usado

Furtaram meu iPhone 3GS 32GB na festa de aniversário fechada da minha filha. Fiquei chateado, principalmente porque usava até o último recurso do aparelho de forma muito produtiva.

Decidi comprar outro, desta vez no Brasil mesmo, mas eu tinha um complicador: minha linha é corporativa. Isso significa que minha conta tem tarifas muitíssimo baixas, tanto que no longo prazo não compensaria abandoná-lo e mudar para outro plano que incluísse iPhone. Linha corporativa significa também que não acumulo pontos, coisa que é usada pelos mortais para trocar por aparelhos melhores [e, nas letras miudas do contrato, por mais fidelidade com a operadora].

Então fiquei com as seguintes alternativas: Continue reading