Posts in english.
It took some time to have my honeypot invaded again in a useful way. They were attacking constantly but were using it only to connect to IRC rooms. Useless for me.
But yesterday a sweet cracker invaded it again and left behind about 240000 non-unique combinations of user+password he was using to invade others.
They were all merged into the Compromised Passwords file, now with 142002 unique combinations of user+password that you must never use.
Enjoy.
Although WordPress 2.7 is almost only eye candy, it is great useful eye candy. The user interface is much much more practical, AJAXed, nice and fast.
I recommend an immediate upgrade. It worked flawlessly with my complex homemade theme, my homemade plugins, plus all other plugins I use throughout my blog.
Spam fighting is a difficult job. Many spammers are too smart for Akismet and I was spending too much time cleaning comment spam.
Captcha-based anti-spam methods are ok but they hurt too much the usability and user experience.
Then I finally found Yawasp WordPress plugin that implements an innovative anti-spam method. It makes the comment form field names to be dynamic and to always change so spambots will have a harder time to know what field is what.
Additionally, it creates a hidden field also with a random cryptic name that must be sent empty. Smarter spambots (but not smart enough) usually send it filled because they can’t see the difference between this honeypot field to a real one.
This methods ensure a full shield against spammers without hurting usability. I simply stopped receiving spam since Sunday when I installed it. And Yawasp is reporting this about the 48 hours it is running in my blog:
Yawasp has blocked 888 birdbrained Spambots since its last activation.
♦ 856 Spambots send the default author and/or comment field.
♦ 32 Spambots send the hidden field, but filled it out.
The only drawback of Yawasp is that my WordPress theme PHP code had to be changed to activate Yawasp dynamic filed names hook. But I can deal with that. And Yawasp may also try to change your theme automatically so you don’t have to worry about that.
I recommend Yawasp to any WordPress blog owner.
I love to see crackers attacking my computers. I even wrote a Linux Magazine article about a successful invasion (portuguese).
Since recently, I leave a virtual machine with an easy password ready to be attacked, just to see these crackers’ leftovers, their rootkits and what they’ll do once they brake into my system.
Guess what. Once inside, they use my system to attack more or to create DDoS-capable networks ready to put some website down. They are not looking for my files, nor my other passwords. They are actually looking for my compute power and want to preserve their anonymity through the use of my Internet address. Pretty useless lifestyle.
Those crackers will rarely explore some Apache or Samba vulnerability to get inside. Few of them have brains for this difficult task. They manage to get inside a system simply because some user password is very simple. They use port scanners to find your SSH TCP port open and then try thousands of combinations of user+password until they login. Once inside, they install this same scanning tools and user+password databases to attack other computers on the Internet.
I started to collect this user+password databases in an alphabetically sorted file and realease it here so people may check if some password they are choosing is well known by crackers or not.
Some classical combinations you’ll find in the file are:
root changeme root default root l1nuxb0x root passw0rd root password oracle oracle123 sapdb sapdb
Remember, you will greatelly avoid having your Linux/Unix/BSD/Mac computer being attacked simply following this rules:
PermitRootLogin no
Pretty obvious, but I keep finding Internt-connected systems with very weak and well known passwords.
Pay attention to this screenshot. This is Mac OS running as a VMware virtual machine.
Pretty cool as a proof of technology. But let me tell you, this is quite a useless virtual brick.
The UI is extremely slow. I had to remove the background to improve it a little bit but couldn’t resize to less than 1024×768. Console over network is more than slow, is completely unusable.
Oh, and by the way, there is no networking. The preferences panel recognizes a network interface but it is disabled for configurations, meaning it doesn’t have a driver for VMware virtual network interface. What can be more brick than that ?!
I wanted to have a Mac OS machine to get to know this beautiful platform, maybe develop some apps etc. But I guess I’ll have to wait for improvements for this VMware patches.
Yesterday I met a colleague at IBM Brazil that moved to a world wide position. He is now living in Manhattan, New York.
His everyday routine, as most Information Workers nowadays, is to talk to people, make phone conference calls, e-mails, design and communicate strategies, make some data consolidation and reports, instant messaging, etc.
Since most of this things have become information, and since we have information technology today, he can do his job from any point on earth as long as he has an Internet connection to let the information flow.
He choose NY because he enjoys life jazz and wanted to experiment the Big Apple, not because his job is physically located there. He could choose Montana, Alaska, Manaus, Fernando de Noronha etc to live.
Of course personal contact is sometimes needed. In these occasions, people schedule meetings and get a flight to meet someplace. This is probably cheaper than maintaining an office space for all these information workers, and probably more friendly to the environment than make them drive everyday to a physical location.
I can think about dozens of jobs that could be this way. Lawyers, journalists, writers, architects, web 2.0-related jobs, even also physicians in many situations when they don’t need to examine their patients.
The matter of all these jobs is information and the platform to make it flow is ready — the Internet. Now we just need a sort of cultural shift.
The kind of thing you can find circulating in São Paulo, friday at noon.
I was oweing this for the lovers of M. C. Escher works.
Last July I visited the wonderful Escher Museum in Den Haag, Holland. One of my preferred works is the Print Gallery where a painting gets mixed with the surrounding real world in a precise geometrical way, as you can see bellow (borrowed from aixa.ugr.es/escher/table.html).
It is intrigating what happens in the center of the picture, where the distortion would be very chaotic.
In the museum there was a screen that played the following video with a computer simulation recreating the chaotic center. I recorded it with my own camera. Quality is not perfect, but watchabel.
It was calculated and produced by some university I can’t remember the name. If you know who made it, please let me know to put the credits here.
Check it out.
This is the 12 weeks plus 4 days ultrasound result of our baby.
We don’t know yet if its a boy or a girl. Stay tuned, it will be revealed in the next one.
He is growing fast and we are looking forward for his arrival in february.
[flashvideo filename=http://avi.alkalay.net/articlefiles/2008/08/2008-08-15-Ultrasound_Parts-H264_672x464_650kbps-AAC_64kbps.mp4 overstretch=none width=672 height=487 showstop=true autostart=true volume=100 /]
If you can’t see the video above, please update your Flash Player here. It is a DVD-quality MP4 video compressed by x264.
I just participated in very successful proof of technology with Siemens OpenScape video conferencing products.
We made a point-to-point SIP high definition video conference over a local area network. Another test used H.323 protocol. We also included my SIP-enabled Nokia E61i in the conference over the office WiFi network in a direct multipoint call simply calling my phone’s IP address. Crystal clear sound.
The equipment statistics showed the following:
Video
Audio
Total of aprox. 1850 kbps for excellent and smooth audio and video quality.
In September 1956 IBM launched the 305 RAMAC, the first computer with a Hard Disk Drive (HDD).
Its weight was close to 1 ton and had a capacity of 5MB. So now please do not complain about the lack of storage in your 2GB pen drive.
There is this website called Brand Tags that lets people quickly tag brands that pop up in a web 2.0 style.
Then you can also browse brand names and see their tag cloud.
For example, Linux is associated with “penguin”, “free”, “tux”, “cute”, “free” etc. Windows gets “computer”, “crap”, “crash”, “monopoly”, “sucks” etc. Apple gets “apple”, “computer”, “awesome”, “cool”, “design”, “innovation”, “love”, “trendy”. IBM gets “big big blue”, “boring”, “computer”, “corporate”, “old”, “pc”, “thinkpad”. Nike has a scary association with “child labor”. Oracle has a plain “database”.
In portuguese we use to say that people’s voice is God’s voice.
ISO has a press release about the 4 countries (India, Brazil, South Africa and Venezuela) and their appeal against the OOXML standardization.
The important part of it is this:
According to the ISO/IEC rules, a document which is the subject of an appeal cannot be published as an ISO/IEC International Standard while the appeal is going on. Therefore, the decision to publish or not ISO/IEC DIS 29500 as an ISO/IEC International Standard cannot be taken until the outcome of the appeals is known.
Which means OOXML cannot become a standard until the appeals get resolved and all countries get satisfied with the outcome.
In other words, OOXML is stucked in the limbo. It can’t be described, considered or even sold as an ISO standard.
It was sort of approved but can’t be approved. Well, the final specification text was not published yet anyways. With some luck, to technically fix OOXML can take the same amount of time as of solving the appeal.
Then we’ll go back to first question: does the world need tow standard ways to describe “helvetica, 12pt with red” or “a table with 3 columns and 5 rows” or even “page layout with 1.5cm margins” ?
As a reference, Brazilian appeal was published in Andy’s blog and is an informative yet concise enough text to understand the technical and process issues around OOXML.
Great wine. Recomended. By the way, 2005 is a great year for chilean wines.
The Iron Man movie is all about programmers. Being challenged and have to invent something that will make you get rid of hand work, the adventure of a project, to mold, test and debug, and the joy of having a prototype inspiring the next perfect design.
Just don’t miss it. The metal man movie has a lot of Heavy Metal music and Robert Downey Jr., both fit perfectly.
Oh, and stay until after the final credits to see what are the plans for the next movie.
Isn’t this web app very neat? You must interact to feel it.
A post by Michael DeHaan and others inspired me to say a few words about the importance of ISO standards for developers:
Many developers are clamming that good specifications are more important than standards, specially now that the OOXML process opened to the public the dungeons of ISO processes and how the organization produces standards. This is a clear example on how ISO’s reputation is compromised.
But unfortunately “ISO standards” are what governments tend to use. These are the words they put in their Request For Proposals when they are going to buy things. These “ISO standards” are the words they use to claim how they’ll interoperate and trade across borders. And also how public institutions will interoperate with private institutions. Over time, it defines also how private institutions use technology.
So yes, ISO standards for Information Technology do matter from a developer standpoint because they are crucial in shaping the flow of information in the society in general.
Lets put it this way: technical specifications stand for research and development team while to be an ISO standard equals to have a strong sales force. An IT product needs both to succeed.
So you, developer, can use whatever you want or like. But if you want to interoperate with non-developer folks, you — the smartest guy in this context — will have to use what non-developers use, and they use what has a stronger marketing force as “this is an ISO standard”. This is why ISO standards (good or bad) should be in the focus of our attention.
Since the world already has an International Standards Organization, and since we learned it must be patched, it is our responsability — we, the developers, geeks, sysadmins etc — to be aware of and get involved with what ISO is standardizing right now and help the technical process of standardization to happen solely with technical arguments and not political interests.
Otherwise the future world will still be divided in two groups of developers: those that develop with/for good stuff and those that develop with/for stuff used by non-developers just because a powerful company had the strength to standardize it whatever it takes.
There was an ISO/IEC/JTC1 meeting in Oslo, Norway wednesday and the local community protested against OOXML and demanded more integrity in ISO’s processes. I couldn’t agree more.
This is my first post in the new WordPress 2.5.
Upgrade was as easy as:
$ cd avi.alkalay.net
$ svn sw http://svn.automattic.com/wordpress/tags/2.5/
Everything worked without any changes, including my experimental content-optimized Plasma theme.
The new WordPress has lots of improvements, specially in the administration part and I recommend it.
I am learning a lot from all this standardization process.
I was a member of the brazilian committee and I also analyzed the specification. My country disapproved OOXML and I think this was a decision based on logic lead by the process.
The JTC1 rules page 49 item 9.8 says:
NBs may reply in one of the following ways:
- Approval of the technical content of the DIS as presented (editorial or other comments may be appended);
- Disapproval of the DIS (or DAM) for technical reasons to be stated, with proposals for changes that would make the document acceptable (acceptance of these proposals shall be referred to the NB concerned for confirmation that the vote can be changed to approval);
- Abstention (see 9.1.2).
[Note: Conditional approval should be submitted as a disapproval vote.]
In other words, from my understanding, if there is one or more technical problems, the NB must disapprove the DIS. Many countries found many technical problems in OOXML that are still unresolved even after the BRM.
I also understand that such an important matter as an International Standard for Office Documents can’t be defined by 10 or 30 opinions collected as votes in a committee. Thats why the JTC1 process above talks about technical content, not opinion or vote. What I learned from studying the OOXML specification is that it is not ready for acceptance since many countries have found and reached consensus that the spec has problems, even after the BRM. If the NB-leveraged technical team — formed by people that would vote YES and NO — has produced a list of submitted problems in the spec, this list is by itself the consensus that the spec is still problematic.
I would like to understand why an NB that has produced technical comments voted YES or ABSTAINED. I thought abstention is a position for countries that were not able to create a committee to technically discuss the specification for reasons such as logistics or lack of quorum.
I am learning about all this and I’d like to have more solid arguments to build an opinion about these NB’s maturity to run a well documented process.
Check it out:
From: “***FONDONORMA***” <in*&%@fondonorma.org.ve>
To: <ki*%$#@itscj.ipsj.or.jp>, <br*%$#%@iso.org>, <tak*%$#%@iso.org>, <ga*%$#%@iso.org>, “Maria Teresa Saccucci” <mar*%$#%@fondonorma.org.ve>, “Norma Arias” <norm*%$#%@fondonorma.org.ve>, “***FONDONORMA***” <in*%$#%@fondonorma.org.ve>
Date: Fri, 28 Mar 2008 14:04:26 -0430
Subject: Modification to the vote on DIS 29500 – Venezuela (FONDONORMA)
Toshiko Kimura
Secretary ISO/IEC JTC 1/SC 34Att.: Mr. Keith Brannon, Mr. Maho Takahashi, Ms. Martine Gaillen
Dear Mr. Kimura,
Attached please find a letter from Mrs. María Teresa Saccucci, Standardization Manager of FONDONORMA (Venezuela) through which Venezuela wishes to modify its position on DIS 29500, Information technology – Office Open XML file formats from “Approval with comments” to “Disapproval with comments”.
Best regards,
Leonardo Di Bartolo
Coordinator of International Relations
FONDONORMA
Venezuela
This NO was a result of a very difficult meeting. Consensus was not reached and a lot of confusion happened. Exactly as in the first brazilian NO.
The NB had to decide the final vote based on technical issues still open in OOXML. The problem was the method of deciding and those technical points were not presented. So people only had in their head the Microsoft arguments that I already knew were part half trues and part complete lies.
I participated in the OOXML process in 3 countries and I was able to build an opinion on how normalization happens today in the world, and I’ll show some ideas in a future post.
By the way, this is how Caracas looks like, a city surrounded by huge green mountains that have behind them the Caribbean Sea.
Finally Jomar, one of the brazilian delegate that went to OOXML’s BRM in Geneva has started to tell all the dirty little details of what really happened in that meeting and the surreal modus operandi of how 120 people can discuss 1027 issues in 5 days. Have fun in english and portuguese.
Oh, and talking about dirty playing, check the domain www.DocumentFreedomDay.com but remember that the original one is www.DocumentFreedom.org. The first one really deserves a DDoS attack.
Open Malasia blog has a user friendly post with a map of the voting. See the tables and understand the voting criterias.
It is now official. Brazilian vote was decided by consensus of the entire technical team, including Microsoft crew’s: OOXML does not deserve to be an international ISO standard.
Our first vote, in august, was also NO, due to the same reasons: OOXML is an awful specification.
That outcome was expected because we simply followed the process: technically analyze the OOXML specification, make comments, wait for responses, analyze them and see if all problems were fixed. Is there any single remaining unresolved problem? Vote NO. And in fact there were many many unresolved problems.
If every country followed this simple process, OOXML would receive a NO from 100% of them.
But in some countries, how is the process? Invite a few companies and simply count their votes. The problem here: 10, 20 or 80 votes can never represent what is the best for that country. Only, maybe, if you collect one vote for each citizen.
What I am trying to say is that in this case a decision must be reached by technical consensus, not vote. It is not a matter of will, but a technical issue that can only be reached by rational analysis and deliberation.
In Chile for example, 21 voting companies will define a 15 million people country vote in ISO. How easy is to lobby these 21 companies with Power Point presentations telling complete plain lies ?
Technically speaking, if your country’s vote was YES or ABSTENTION, one of these possibilities happened:
Seems stupid, but these are exactly the 2 possibilities of OOXML getting YES votes in ISO. It is still an awful specification.
By the way, Brazil would vote NO again and again and again even if all OOXML’s technical issues could be resolved. OOXML would still have legal issues and also serious overlap problems with the OpenDocument Format ISO standard.
I was not present in today’s meeting in ABNT because I already knew what would be the result, since the process of analysis and deliberation in Brazil was very strictly followed. Hopefully Jomar will write about it and you can check more details.
Can Open Source Software be more ubiquitous than this ?
A few days ago I was playing extensively with Apple’s iPhone, investigating each sub-menu and little details. There is a section listing legal stuff and software being used with each license. GPL, LGPL, BSD and other Open Source licensed software rule the iPhone.
Some I have noted:
Microsoft started their so called “interoperability” initiative. With a deeper look we find that the main objective of the released specifications is to let developers interact with their products only, something I like to call Intra-operability.
See the difference?
Microsoft technical specifications have serious technical and legal issues, and are being released not to increase interoperability across people, organizations and applications, but to leverage the use of their products. For the standard document format scenario: OOXML is about allowing other people to interact with files that are primarily generated and manipulated with Microsoft Office. It is not about full interoperability, which would enable competition with Microsoft Office, for obvious reason.
For each purpose or scope, better open standards exist and should be adopted and used instead: Java versus .NET, XHTML versus IE-campatible-only DHTML, ODF versus OOXML, etc
My friend Cezar Taurion has also written some words about Intra-operability (in portuguese). Bob Sutor also put together some words about it.
These graphics are available in a friendly Creative Commons license, in this animated ODF file, in case you want to integrate them in your presentations. PDF export also available.
One of the most critical and discussed points of the whole OOXML subject is how the specification lets you include binary proprietary information.
Let me show you how it happens with a piece of an OOXML document, the red-marked text is the problematic part (see for yourself, §6.2.2.14, paper page 4,813, lines 7–13):
<v:shape>
<o:ink i="AMgFHQSWC+YFASAAaAwAAAAAAMA..." annotation="t" contentType="application/x-ms-ink"/>
</v:shape>
So you, as a programmer, please tell me what to do if you are developing an application that must read and generate this kind of document. How can I find documentation about this encoded binary stream? Is this a good practice in XML? Anyway, this kind of (bad) design appears in many parts of the OOXML spec. Want more examples of bad design? Have fun.
Suppose I really want to develop this kind of support in my application, I am a master programmer and reverse-engineered a few examples generated with a copy of MS Office 2007 that I had to buy. Or maybe I just found the specification of this proprietary application/x-ms-ink type and go to develop a library to handle it.
Inevitably, my library will reimplement aspects of some Microsoft library with same functionality, and according to a Software Freedom Law Center report, the Microsoft Open Specification Promise (OSP, basically the not-open-enough license Microsoft lawyers wrote for the usage and implementation of OOXML) cover’s the specification only, not code.
I will be sued for patent infringement. On software rewrite, not on specification usage.
Then the rape begins. Microsoft is claiming that this was solved in the ISO’s BRM meeting. This is the solution, from the ridiculous 12 pages resolutions, page 7:
Resolution 25: The BRM decides to accept the editing instructions contained in http://www.itscj.ipsj.or.jp/sc34/def/BRM/Response_0135_bitfields.doc in place of R 135, replacing “deprecated” by “transitional”, and with the following addition: The Editor shall ensure that all existing attributes defining the bitfields described above shall be “transitional”—so resolved.
Who reads this resolution with high level eyes may think that all binary fields will be removed from the specification. But please, you, as a programmer, tell me how the so called Editor will find all and every single part of a 6000+ pages specification containing bitfields? How he’ll expand all those bitfields in an XML subspecification? Will he invent some? Is he the right person (or team) to do that?
When done and if done, the specification will be something completely new, full of new parts. Will jump from 6063 pages to maybe 7500. Oh, and did I mention it will be something that even MS Office 2007 or 2008 don’t support today? Supporting or not, implemented or not, this new unexistent specification is the “thing” that countries’ National Bodies are voting right now, without even seeing it, without checking if it was corrected, without having time for this because they didn’t receive it for revision.
They won’t have time to review 6000+ pages because the deadlines defined by the ISO’s Fast Track process are over.
So the question is how ISO/IEC and JTC1 let such a big and problematic specification enter the light Fast Track process? My answare: ISO was raped.
I’ve been talking to several people that will define their country’s vote and their mindset is “are you really putting ISO in such a bad position?” Well, yes. You know, in the end ISO is not god. They are a bunch of people that, like you and me, have religions, aspirations, problems, family, go to the bathroom etc. Like you and me, they may be also naive in regard to some subjects, particularly document formats. Then comes Microsoft excellent speakers showing PowerPoint charts that are plain lies (e.g. “OpenOffice.org supports OOXML”) and people believes them.
People have two choices: to question ISO’s reputation in the OOXML case, or question IBM, Sun, Oracle, Red Hat, Free Software Law Center, ODF Alliance and many other institutions’ reputation when they massively scream in chorus that OOXML has serious technical, legal and standardization process issues.
This is a world where organizations like ECMA has completely lost the respect of the technical community. But this is not a big problem, because we have other similar, still reliable bodies as OASIS, W3C, OSF, etc.
This is a world where we will have to work hard to make ISO regain its (currently damaged) reliable status. This is a big problem, because we only have one (and only need one) International Standards Organization.
This is your responsibility. To start this work get involved with your country’s National Body for standardization and promote the creation of a formal letter to ISO about the OOXML process, its problems and how ISO let that happen. INCITS in USA, ABNT in Brazil, INN in Chile etc.
The title was the phrase written in a CD left in a Crown Plaza Hotel room I stood this days.
Its excellent content is a relaxation program with very soft music to listen and do after you go to bed and before you fall asleep, with the aim to make you sleep better.
I found it so effective that I read more through the CD cover. It is produced by a company called SoundSleep with the help of a certain Dr. Michael Breus.
The cover also contained some other “holistic approach to sleep principles” which I also find useful and would like to share:
This is an example of an old Excel bug that was transported to OOXML for “compatibility” reasons.
In new OOXML, there are several non-standard ways to represent dates. The main one is this, that requires that 1900 is a leap year contradicting the Gregorian calendar, used for centuries now.
Around early 1995 I was thinking about to stop eating meat. I thought that would do good for my body and my mind.
Then I was waiting in a room, randomly took a book and randomly open it in the part that finally defined my decision. Later a saw many people becoming vegetarian because of the same part from the same book, so I found it important to share. The author was Sri Yukteswar and the book name is The Holy Science, from 1894.
This is the part of the book that influenced my decision that time (copy and pasted from Sain Louis’ Guide to Vegetarian Life):
What is natural food for man?
First, to select our natural food, our observation should be directed to the formation of the organs that aid in digestion and nutrition, the teeth and digestive canal; to the natural tendency of the organs of sense which guide animals to their food; and to the nourishment of the young.
Observation of teeth
By observation of the teeth we find that in carnivourous animals the incisors are little developed, but the canines are of striking length, smooth and pointed, to seize the prey. The molars also are pointed; these points, however, do not meet, but fit closely side by side to separate the muscular fibers.
In the herbivorous animals the incisors are strikingly developed, the canines are stunted (though occasionally developed into weapons, as in elephants), the molars are broad-topped and furnished with enamel on the sides only.
In the frugivorous all the teeth are of nearly the same height; the canines are little projected, conical, and blunt (obviously not inteded for seizing prey but for exertion of strenth), The molars are broad-topped and furnished at the top with enamel folds to prevent waste caused by their side motion, but not pointed for chewing flesh.
In Omnivorous animals such as bears, on the other hand, the incisors resemble those of the herbivorous, the canines are like those of the carnivorous, and the molars are both pointed and broad-topped to serve a twofold purpose.
Now if we observe the formation of the teeth in man we find that they do not resemble those of the carnivores, neither do they resemble the teeth of the herbivorous or th eomnivorous. They do resemble, eactly, those of the frugivorous animals. The reasonable inference, therefore, is that man is a frugivorous or fruit-eating animal.
Observation of the digestive canal
By observation of the digestive canal we find that the bowels of carnivorous animals are 3 to 5 times the length of their body, measuring from the mouth to the anus; and their stomach is almost spherical. The bowels of the herbivores are 20 to 28 times the length of their body and their stomach is more extended and of compound build. But the bowels of the frugivorous animals are 10 to 12 times the length of their body; their stomach is somewhat broader than that of the carnivorous and has a continuation in the duodenum serving the purpose of a second stomach.
This is exactly the formation we find in human beings, though anatomy says that the human bowels are 3 to 5 times the length of man’s body – making a mistake by measuring the body from the crown to the soles, instead of from mouth to anus. Thus we can again draw the inference that man is, in all probability, a frugivorous animal.
Observation of organs of sense
By observation of the natural tendency of the organs of sense – the guideposts for determining what is nutritious – by which all animals are directed to their food, we find that when the carnivorous animal finds prey, he becomes so much delighted that his eyes begin to sparkle; he boldly seizes the prey and greedily laps the jetting blood. On the contrary, the herbivorous animal refuses even his natural food, leaving it untouched, if it is sprinkled with a little blood. His senses of smell and sight lead him to select grasses and other herbs for his food, which he tastes with delight. Similarly with the frugivorous animals, we find that their senses always direct them to fruits of the trees and field.
In men of all races we find that their senses of smell, sound, and sight never lead them to slaughter animals; on the contrary they cannot bear even the sight of such killings. Slaughterhouses are always recommended to be removed far from the towns; men often pass strict ordinances forbidding the uncovered transportation of flesh meats. Can flesh then be considered the natural food of man, when both his eyes and his nose are so much against it, unless deceived by flavors of spices, salt, and sugar? On the other hand, how delightful do we find the fragrance of fruits, the very sight of which often makes the mouth water! It may also be noticed that various grains and roots possess an agreeable odor and taste, though faint, even when unprepared. Thus again, we are led to infer from these observations that man was intended to be a frugivorous animal.
Observation of the nourishment of the young
By observation of the nourishment of the young we find that milk is undoubtedly the food of the newborn babe. Abundant milk is not supplied in the breasts of the mother if she does not take fruits, grains, and vegetables as her natural food.
Cause of disease. Hence from these observations the only conclusion that can reasonably be drawn is that various grains, fruits, roots, and – for beverage – milk, and pure water openly exposed to air and sun are decidedly the best natural food for man. These, being congenial to the system when taken according to the power of the digestive organs, well chewed and mixed with saliva, are always easily assimilated.
Other foods are unnatural to man and being uncongenial to the system are necessarily foreign to it; when they enter the stomach, they are not properly assimilated. Mixed with the blood, they accumulate in the excretory and other organs not properly adapted to them. When they cannot find their way out, they subside in tissue crevices by the law of gravitation; and being fermented, produce diseases, mental and physical, and ultimately lead to premature death.
By the way, I am not 100% vegetarian today. I still have seafood very rarely, when is very difficult to find veggy options in some fancy restaurants.
This is a numerical photography of what is the Microsoft Office Open XML wannabe specification:
Or graphically:
And you still think OOXML is ready to become an International Standard Format for storing YOUR documents ?
Come on, give me a break !
I decided to take the test to see how much the planet gets stressed with me. This is the result:
I have a lot to improve. Specially in goods (which was a weak test) and transportation.
Some friends asked so the following is how I encode (rip) DVDs.
The ripped video file format is a decision you must make. Currently my format of choice is .MKV or Matroska. I’ll explain why.
It is quite silly to say that an .MP4 movie has better quality than a .AVI or vice-verse (or any other combination of comparisons). OGG, MP4 (MPEG-4 Part 14), MKV (Matroska), AVI, WMV (or ASF) are just containers, envelopes. Video quality depends on what goes inside it.
“Multimedia” has this name because you have multiple types of media: video in multiple angles, multiple audio options including different languages and channels (stereo, 5.1, 6 channels etc), subtitles in several languages, chapter information, menu etc. Think about a DVD. So this is a graphical view of how things are organized inside a 900MB movie file in a modern format as MKV or MP4:
| Header with tags, track names, chapters info, seek positions | Main Video track (MPEG-4 AVC/H.264) | Attachments as JPG images, documents, scripts or text files | |||||||
| Video segment showing another angle (MPEG-4 ASP/Xvid/DivX) | |||||||||
| Audio track: English Dolby Surround 5.1 (AC3) | |||||||||
| Audio track: Director’s comments stereo (MP3) | |||||||||
| Audio track: Portuguese Dolby Surround 5.1 (DTS) | |||||||||
| Subtitle track: Portuguese (Unicode text) | |||||||||
| Subtitle track: Chinese (Unicode text) | |||||||||
| Subtitle track: English (VobSub) | |||||||||
| byte 100K | byte 100M | byte 200M | byte 310M | byte 420M | byte 530M | byte 650M | byte 780M | byte 895M | byte 900M |
A digital multimedia file format must be capable to contain all this different medias and multiplex them in parallel so you won’t have the video in the first 500MB of the file and the audio on the following 500MB (this can’t work for streaming). And this is exactly what modern file formats as MP4 and MKV do: they carry all your movie-related data together.
This is a comparison of all these file formats based on my personal experience with them (a more formal comparison can be found in Wikipedia):
| .MKV | .MP4 | .AVI | |
|---|---|---|---|
| Industry support | Almost none | Good and increasing, specially on Apple platforms, the mobile scene and Nero Digital ecosystem | Treated as legacy popular format |
| Usage on the web | Very popular on HD or high quality DVD rips | Very popular on HD or high quality DVD rips, supported by Flash Player, YouTube, Google Video | Popular amongst low-quality DVD rips |
| Support for advanced video formats and multiple video angles | Yes. MPEG-4 ASP (DivX, Xvid), MPEG-4 AVC (a.k.a. H.264) etc | Yes. Only MPEG-4 systems and a few others | Problematic and No |
| Support for multiple audio tracks (channels, formats, languages and “director’s comments”) | Yes | Yes. Formats are only MP3, AAC and a few others not very popular | Yes |
| Support for tags (artist, title, composer, etc as MP3’s ID3) | Yes | Can be supported by MP4 extensibility but this is not very well standardized across authoring tools (iTunes, GPAC etc) and players (Amarok, Media Player Classic, iPod, Windows Media Player etc) | No |
| Support for attachments with mime-types (used to attach movie posters images or other files) | Yes | No | |
| Support for chapter marks | Yes | No | |
| Support for multiple language embedded soft-subtitles | Yes. VobSub (as extracted from DVDs), plain timed UTF-8 text (SRT, SUB) etc | No | |
| Support for naming tracks with human names as “Director’s comments” or “Portuguese subtitles” etc | Yes | Yes | No |
| Support for carrying menus information (as in DVDs) and interaction | Yes through an XML idiom, but unsupported by most players | Yes through SVG, but unsupported by most players | No |
| The container overhead in bytes in the final file | Very small | Very small | Very big |
| Supported by free and Open Source multiplatform authoring tools | Perfect on Linux, Unix, Windows and Mac | Yes (see GPAC), with some intellectual property issues. Tools need to mature. | Yes |
| Ready for popular web streaming as in Flash player | No | Yes. The popular Flash Player that is installed on everybody’s browser supports playing MP4 files as long as they contain H.264 video and AAC audio tracks. I recommend the LongTail FLV/MP4 Player since it also plays subtitles embedded in MP4 files. | No |
Personally I believe MP4 is the multimedia file format for the future because since it is getting popular, all these non–standardized features will get stabilized. MP4 is an ISO standard and the increasing industry support can be felt on iPods and portable devices, and most notable on home DVD players capable of playing the 700MB MP4 video file burned in a CD.
By the way, remember this:
Meanwhile, MKV wins everything but on the Industry Support category. But this doesn’t really matter, and I’ll explain why. Since MKV is just a container, the large video, audio etc streams can be extracted and repackaged into MP4 and vice-versa in seconds. No transcoding (decoding followed by a lossy encoding into another format) is needed.
So today I store my videos in the most feature rich and well supported by players format: MKV.
OGG or OGM (the container file format) is practically dead in my opinion. They were created as part of the Xiph initiative for a complete open source patent-free multimedia framework, but seems nobody uses it anymore for video. From the same family, Vorbis (the audio codec compared to MP3, a.k.a. .OGG) is very good but also very not popular. Theora (the video codec) is frequently comparable to old MPEG-1 in terms of quality and compression ratio so currently, if you want quality and are not concerned about patents, MPEG-4 AVC is the best choice. FLAC, Xiph’s lossless audio codec, is the winner of the family: very popular, massively used, and recommended.
I use HandBrake, the most practical Open Source (and overall) movie encoder. It runs on Linux, Mac and Windows and uses the same Open Source libraries as ffmpeg, mplayer/mencoder, xine, etc. While these programs are generic video handlers (with thousands of confusing configuration parameters to sustain this generalistic status) HandBrake is optimized only for ripping so it is very easy to use, yet extremely powerful.
#!/bin/bash
##
## This is the script I use to make hifi DVD rips including chapter markers and
## subtitles. It uses Handbrake.
## Contains what I found to be the best quality ripping parameters and
## also let me set simple parameters I need.
##
## Avi Alkalay <avi at unix dot sh>
## http://avi.alkalay.net/2008/03/mpeg4-dvd-rip.html
##
## $Id$
##
#set -vx
HANDBRAKE=${HANDBRAKE:=~/bin/HandBrakeCLI}
#HANDBRAKE=${HANDBRAKE:="/cygdrive/c/Program Files/Handbrake/HandBrakeCLI.exe"}
## Where is the Handrake encoder executable.
## Handbrake is the most practical free, OSS, DVD riper available.
## Download HandBrake for Linux, Mac or Windows at http://HandBrake.fr
INPUT=${INPUT:=/dev/dvd}
## What to process. Can also be a mounted DVD image or simply '/dev/dvd'
TITLE=${TITLE:=L}
## The title number to rip, or empty or "L" to get the longest title
#CHAPTERS=${CHAPTERS:=7}
## Example: 0 or undefined (all chapters), 7 (only chapter 7), 3-6 (chapters 3 to 6)
#VERBOSE=${VERBOSE:="yes"}
## Wether to be verbose while processing.
SIZE=${SIZE:=1200}
## Target file size in MB. The biggest the file size, the best the quality.
## I use to use from 1000MB to 1400MB for astonishing high quality H.264 rips.
OUTPUT=${OUTPUT:="/tmp/output.mkv"}
## Output file. This will also define the file format.
## MKV (Matroska) is currently the best but MP4 is also good.
AUDIO=${AUDIO:="-E ac3 -6 dpl2 -D 1"} # For AC3 passthru (copy).
#AUDIO=${AUDIO:="-E lame -B 160"} # For MP3 reencoding. Good when input is DTS.
## Audio parameters. If input is AC3, use it without transcoding.
## If is DTS, reencode to MP3.
MATRIX=${MATRIX:=`dirname $0`/eqm_avc_hr.cfg}
## x264 matrix to use. The matrix file may increase encoding speed and quality.
## This one is Sharktooth's as found
## at http://forum.doom9.org/showthread.php?t=96298
######### Do not change anything below this line ##############
## Make some calculations regarding title and chapters based on parameters.
SEGMENT=""
if [[ "$TITLE" == "L" || -z "$TITLE" ]]; then
SEGMENT="-L"
else
SEGMENT="-t $TITLE"
fi
[[ -n "$CHAPTERS" && "$CHAPTERS" -ne 0 ]] && SEGMENT+=" -c $CHAPTERS"
[[ "$VERBOSE" != "no" ]] && VERB="-v"
# Define args for the x264 encoder. These are some values I found on the net
# which give excelent results.
X264ARGS="ref=3:mixed-refs:bframes=6:b-pyramid=1:bime=1:b-rdo=1:weightb=1"
X264ARGS+=":analyse=all:8x8dct=1:subme=6:me=umh:merange=24:filter=-2,-2"
X264ARGS+=":ref=6:mixed-refs=1:trellis=1:no-fast-pskip=1"
X264ARGS+=":no-dct-decimate=1:direct=auto"
[[ -n "$MATRIX" ]] && X264ARGS+=":cqm=$MATRIX"
# Encode...
"$HANDBRAKE" $VERB -i "$INPUT" -o "$OUTPUT" \
-S $SIZE \
-m $SEGMENT \
$AUDIO \
-e x264 -2 -T -p \
-x $X264ARGS
# Repackage to optimize file size, to include seek and to include this
# this script as a way to document the rip...
echo $OUTPUT | grep -qi ".mkv"
if [[ $? && -x `which mkvmerge` && -f $OUTPUT ]]; then
mv $OUTPUT $OUTPUT.mkv
mkvmerge -o $OUTPUT $OUTPUT.mkv \
--attachment-name "The ripping script" \
--attachment-description "How this movie was created from original DVD" \
--attachment-mime-type application/x-sh \
--attach-file $0
[[ -f $OUTPUT ]] && rm $OUTPUT.mkv
fi
The script seems long because it is fully documented but it actually only collects some parameters and simply runs the HandBrake encoder like this (passed parameters are in red):
~/bin/HandBrakeCLI -v -i /dev/dvd -o /tmp/output.mkv \ -S 1200 \ -m -L \ -E lame -B 160 \ -e x264 -2 -T -p \ -x ref=3:mixed-refs:bframes=6:b-pyramid=1:bime=1:b-rdo=1:weightb=1:analyse=all:8x8dct=1:subme=6:me=umh:merange=24:filter=-2,-2:ref=6:mixed-refs=1:trellis=1:no-fast-pskip=1:no-dct-decimate=1:direct=auto:cqm=~/src/randomscripts/videotools/eqm_avc_hr.cfg
All the rest is what I found to be the best encoding parameters.
The resulting video file (/tmp/output.mkv in this case) will contain correctly cropped video and audio quality as good as the DVD (it is almost lossless), and chapter breaks at the same positions read from the DVD.
In a Core Duo machine as my laptop running Fedora 8 or Windows XP, a 2 pass H.264 encoding (2 pass improves quality and H.264 is newer standard MPEG-4 technology better than DivX/Xvid) takes about 4 to 5 hours for a regular 2 hours movie, so leave it encoding while you go to sleep. A Pentium 4 machine running Ubuntu takes about 17 hours for the same rip.
I use to rip one chapter from the movie first (use your preferred video player or lsdvd command to find the shortest chapter), check quality, compare to DVD, fine tune, try again and then shoot full DVD ripping.
After encoding I use to repackage the audio/video stream with Matroska‘s mkvmerge (or mmg, its GUI version available on any Linux distribution as “mkvtoolnix” package, and installable for Windows or Mac OS from Matroska’s website) to optimize seeks and to include soft subtitles (that can be turned on and off as on regular DVDs), but I’ll explain that in another HOWTO.
I use to organize my media library in a standard way I invented for myself and which I suggest you to use too.
My movie file names shows everything that the file includes. Some examples:
The advantages of this scheme are:
To play this advanced Matroska media file that contains such a rich set of metainformation and highly compressed digital content you will need an advanced player too. And happens that the best players are the Open Source ones. So use these players:
These are Media Player Classic screenshots demonstrating how to activate the advanced content inside a Matroska file. Players on other platforms have similar capabilities and menus.
Activating embedded subtitles and languages
The player lets you choose the audio language and subtitles. On MPC for example, this is how you turn on and off and choose the language for subtitles.
As you can see, the player found subtitles embedded in the MKV file in English, Hebrew and Portuguese.
If the MKV file contains many audio tracks (as different languages, director’s comments etc) this is how to select it:
And to jump directly to a specific chapter on the movie, if the MKV file contains this kind of information:
Improving audio volume
If you ripped the movie without reencoding the audio, the final file will contain DVD’s original AC3 audio tracks in 6 channels (5+1). This may sound with a lower volume when played in a 2-speaker system as your laptop, iPod, etc because 4 channels are simply not being played. To remediate this the player will have to downsample the audio. In other words, it will remix the 6 channels into 2 stereo channels while playing. The Media Player Classic options should look like this:
A few days ago I saw a post in Google Blog with Google’s position regarding the document format war.
Google supports OpenDocument Format.
The post also provides some short documents with Google’s findings (2 pages) and sample of technical issues unresolved in OOXML (4 pages). This is a fast reading for who wants to understand the main issues of OOXML.
I found Dell’s Digg-like IdeaStorm site today while having a look on Doc Searl’s January 2008 LJ Index.
This is a great example of Web 2.0 in action, together with similar sites as IBM’s Innovation Jam.
It shows how Linux is highly rated by everybody, as a top request amongst these ideas. Check Linux rank as of 2008-2-23 10:00 UTC-3:
The state of Paraná, here in Brazil, is the first to have laws that specifically promotes the use of OpenDocument Format. This is a free translation of such law:
This law was signed by Roberto Requião, the state gorvernor, and Nizan Pereira Almeida, secretary for strategic initiatives.
Other states in Brazil, inspired by Paraná, are creating similar laws.
The folks in the photo, left to right, are: Henrique Menezes, government relations, IBM. Jomar Silva, ODF Alliance chapter Brazil. State congressman Edson Praczyk. Vitorio Furusho, Celepar (a state public company), BrOffice.org users group and Open Source Movement in Paraná. Reginaldo Radel, congressman Prazyk assistent.
Web 2.0 is about people. Its a global party where human beings exchange knowledge, experiences, information and even emotions.
Yesterday I saw the Xanadu movie again with ELO’s All Over the World, a good-vibe song from the 80’s that still has very current lyrics that explain what people are doing in our 2008’s Web 2.0.
Check out the movie excerpt with the song and the lyrics.
| Everybody all around the world Gotta tell you what I just heard There’s gonna be a party all over the world I got a message on the radio All over the world, Everybody walkin’ down the street We’re gonna take a trip across the sea All over the world, |
London, Hamburg, Paris, Rome, Rio, Hong Kong, Tokyo L.A., New York, Amsterdam, Monte Carlo, Shard End and All over the world, Everybody all around the world All over the world, All over the world |
What a cute movie about $$ spent with office suites.
I just listened to an excellent interview with Gary McGraw, a security guru, in the Software Engineering Radio, and I suggest software developers to hear it too.
Some points he touched:
I guess this is my last post of 2007 and I wish everybody a happy new year.