Feed aggregator

Public service announcement

When porting GNU/Linux applications to a new architecture, such as 64-Bit ARM, one gets familiar with the following error message:

checking build system type... x86_64-pc-linux-gnu
checking host system type... Invalid configuration `aarch64-oe-linux': machine `aarch64-oe' not recognized
configure: error: /bin/sh config.sub aarch64-oe-linux failed

This in itself is trivial to fix - run autoreconf or just copy in new versions of config.sub and config.guess. However, when bootstrapping a distribution of 12000+ packages, this becomes quickly tiresome. Thus we have a small request:

If you are an upstream of a software that uses autoconf - Please run autoreconf against autotools-dev 20120210.1 or later, and make a release of your software.

Aarch64 porters will be grateful as updated software trickles down to distributions.

This was the most discussed point during my FOSDEM talk "Porting applications to 64-Bit ARM".

This is my monthly summary of my free software related activities. If you’re among the people who made a donation to support my work (84.25 €, thanks everybody!), then you can learn how I spent your money. Otherwise it’s just an interesting status update on my various projects.

Debian Packaging

In one of my customer projects, I had to use libwebsockets and since it was not packaged for Debian, I filed a “Request For Package” (RFP #697671). I discovered a fork of this library on github and decided to mail the original author and the author of the fork to learn a bit more about the reason of the fork. It turns out that they miscommunicated and that the original author was interested by most of the improvements. The fork still exists but the important fixes and most of the improvements have been merged (and he released a version 1.0 after that!). Furthermore the original author setup a bug tracker to better organize the project and so that the author of the fork can submit patches and be sure that they won’t be forgotten (as it happened in the past). I spend quite some time discussing with both parties but at the end I’m pleased to see that good progress has been made (although nobody stepped up to maintain this package in Debian).

I packaged zim 0.59 (an important bugfix release) and wordpress 3.5.1 (with several security fixes). I updated the dpkg-dev squeeze backports to version 1.16.9~bpo60+1 on request of Daniel Schepler. This backport led me to file #698133 on kgb-client because the bot literally spammed the #debian-dpkg IRC channel for multiple hours by resending old commit notices that got merged in the squeeze-backports branch. BTW, they need help to get this issue fixed.

I updated python-django-registration to fix a compatibility issue with python3-sphinx (see #697721 for details).

Misc Debian Stuff

Serious bug with salt. I filed a grave bug on salt (#697747—) and prepared the upload to fix the issue on request of the maintainer. In the mean time, the maintainer orphaned the package. Franklin G. Mendoza already announced its willingness to take over but this package deserves multiple maintainers since this is a good piece of software that is getting more and more popular.

net-retriever and alternate keyrings. I filed a wishlist bug (#698618) on net-retriever to request a way for derivatives to use another keyring package (i.e. not debian-archive-keyring-udeb) without having to fork net-retriever.

Linux 3.7 on armel/armhf. I helped the kernel maintainers to fix the 3.7 kernel on armel/armhf by reporting on IRC the results of successive failing kernel rebuilds on those architectures (this kernel version is only in experimental).

Carl9170 firmware. I also pinged the kernels maintainers about a missing firmware for the carl9170 driver (already reported in #635840) and Ben Hutchings took care of re-activating its inclusion in upstream’s linux-firmware.git and then uploaded firmware-free 3.2 to Debian. Thanks Ben!

New QA team member. And to finish with the miscellaneous stuff, I helped Holger Levsen to be added to the “qa” group so that he could integrate his awesome work on automated QA checks with Jenkins.

Debian France

Preparation for Solutions Linux. The people organizing the “village of associations” in the Solutions Linux have asked all organizations to apply for a booth if they wanted one. Last year Carl Chenet took care of organizing this and this time we had to find someone else. I made multiple call for volunteers (on the mailing list, on my blog) without much success but I finally managed to convince Tanguy Ortolo to take care of this. Thank you Tanguy!

Get in touch with treasurer who disappeared. During the transition with the former Debian France officers, it has been said that Aurélien Gérôme — another former treasurer of Debian France — had entirely disappeared together with some papers that he never gave to his successor. I didn’t want to give up on this without at least trying to get in touch by myself… so after multiple tries (over IRC, phone, and snail mail), and some weeks without answers, he got back to me, explaining that he’s currently in a foreign country and that he will take care of that next time that he comes in France. \o/

New website in preparation. Replacing the single-page website webpage with a more comprehensive website is an important goal. Alexandre Delanoë provided a basic ikiwiki setup inspired by dsa.debian.org. I cleaned it and integrated it in a git repository on our machine. There’s thus a new test website on http://france.debian.net/test/. Tanguy Ortolo and Fernando Lagrange immeditaly made some small improvements but since then nobody stepped up to further complete the website. I’ll try to do this in February and put the new website in production.

Paypal and handling of members. We installed a paypal plugin in galette so that members can renew their membership online. I asked Christian Bayle to try it out and we found some issues that I reported upstream and that got fixed. But this is only the first step, we want to go much further and automate all the membership handling, from membership renewal mail reminders up to integration in the accounting system. To this end, I filed some new tickets in the Galette tracker and completed some that were already opened: #490, #368 and #394. We requested a quote for those tickets and Debian France is going to fund the work on those tickets so that we have a 100% free software solution for our needs.

Thanks

See you next month for a new summary of my activities.

No comment | Liked this article? Click here. | My blog is Flattr-enabled.

This article looks at 10 modules that integrate with Views and that I think people should know about. From the beginning I will say that the list is not exhaustive of all the cool modules that exist out there for Views. God knows there are many more to come out...

Building reports and charts for a website can be tough. Trying to figure out what report or chart will be useful is kind of like predicting the future. Most of the time a client won't know what report they need until they need it or their boss needs it. As site builders, we need to be quick when a request comes in for a new chart. Luckily, there are tools out there which we can use to build beautiful charts like Google Visualization API and Highcharts.

The Visualization API module allows you to create amazing looking charts and graphs in Drupal 7. The module is intended to be an API for Drupal, and right now has support for Google Visualization API and Highcharts (free for non-commercial use). Custom charting libraries can be added via Ctools plugin. The module also ships with a display plugin for Views, this means you can build "simple" charts by creating a view. Even though, the module is still in alpha, it works pretty well. If you need any more convincing, the Commerce Reporting module uses Visualization API for its charts and graphs.

Episode Number:  101

The Drupal 7 Menu Badges module is a neat little module that allows you to add numbered badges next to menu items on a Drupal 7 website. It integrates nicely with views so you can easily add your own numbered icons to any menu link. You will also need the Link Badges module for this to work.

In this episode you will learn:

• How to add a Menu Badge to a Drupal PrivateMsg link

DDoD Video: 

read more

This was originally posted to the "The influence of subtlety" thread on the Drupalchix group back in 2010, but since that was awhile ago now, and since I'm giving a talk on How to Create Ravenously Passionate Contributors at DrupalCon Sydney (and this experience played a huge part) I figured I'd post this to Drupal Planet as well.

Here are some highlights from my first DrupalCon (Vancouver, 2006).

For context, to those who didn't know me back then, imagine the most introverted, shyest, mousiest little geek you can imagine, whose big ice breaker plan was to put on a Google Summer of Code shirt and tuck herself into a corner in the lounge of the hotel everyone was staying at, hoping desperately that someone would maybe, possibly recognize her and come talk to her, since she was too shy to actually introduce herself to anyone. ;) That was me, 4.5 7 years ago. My interactions at my first DrupalCon had a *huge* impact on the future direction of my life.

• Inclusiveness: Adrian Rossouw, Drupal mad scientist extraordinaire, was talking super excitedly about crazy next-generation Form API stuff and I was sitting there nodding vaguely, kind of deer in the headlights about it. He paused and said basically, "Oh, sorry. Did that not make sense? Here, let me try explaining it a different way, because it's REALLY AWESOME and I want you to be as excited about it as I am." And then proceeded to do so, and I was. :)

  Note that he didn't end the conversation and then go off and talk to someone else more at his level, nor did he ask me if I was in the wrong room, implying I wasn't smart enough to be part of this interesting conversation. Instead he saw me as an equal who just needed to get brought up to speed, and made a conscious effort to include me in the discussion.

• Role models: Though there were other women at that Drupalcon who I bonded with, I remember being particularly drawn to Allie Micka, because she was hardcore. She is a total geek (meant in the most complimentary way!), runs her own hosting company, can rattle off random unix sysadmin facts like nobody's business, and at Drupalcon she was leading or co-leading talks on brain-blowing stuff like relationships API. And the guys respected her. When she talked about why a certain approach was good or bad, they'd sit back and listen, and took her seriously.

  While I wasn't *remotely* at Allie's level (and still am not now, and never will be :)), I saw someone I could relate to, and received confirmation that if someone like that was at home in this community, there was a place for me, too.

  This is why all types of diversity initiatives in our community are really important. We want people from all walks of life, backgrounds, interests, skill levels, etc. to see someone they can relate to kicking ass in our community in an environment of mutual respect.

• Challenges: One evening we were all seated around a big long table in one of the hotel's conference rooms, Moshe at the helm, handing out release-blocker bugs for the folks there to look at so we could get Drupal 4.7 out the door (deja vu, anyone? ;)). I wanted to help, but wasn't sure where to start. Moshe fired me off a couple of bugs to look into. One was way over my head, and I told him so, but he told me to stick with it and do what I could, asking for help if I needed it. I started with a basic code review, but it actually revealed a deeper bug, which would've led to further regressions had it not been cleaned up. It was a little thing, but I felt a profound sense of accomplishment for making some tiny little contribution to helping 4.7 come out faster.

  I hope that we can retain this same "spirit" in code sprints today, even though there are several hundred rather than a dozen or so people around the "table". It's invaluable for communicating the Drupal community's "spirit" of contributing, and getting people hooked. :)

• Encouragement: At lunch with Allie, Earl, and some other folks, we were talking about what sessions we wanted to attend, and I told everyone I was planning on attending the module development tutorial for newbies. I got some chuckles in return, saying "No, you seriously don't need to attend that session. You've been coding modules for 9 months now." It was a silly thing, but it opened my eyes to the fact that "oh, hey, maybe I *do* know something after all." :P~

  I see this come up a lot at the Drupalchix BoFs. A woman will introduce herself as "not a coder" and then go on to detail all of the very-much-code-related things she does: theming, front-end development, putting together code snippets from the repository into something that works, etc. With some encouragement, I think there'd be far less devaluing of skills all around, and more people taking big leaps they hadn't taken before.

The subtlety in these interactions was key. If any one of those instances had gone differently (had, for example, the guys rolled their eyes at Allie when she attempted to explain her approach towards relationship API, or had I been politely asked to leave when I joined the core bug squashing sprint and didn't know anything), I don't know the effect it would've had. But I do know that together, they formed the perfect concoction for getting me totally hooked on the Drupal community and sticking around long-term. :)

I think we can do a lot to off-set this trend by changing some of our default assumptions:

• Assume that the next person you interact with could become the next $prolific_contributor if only they were given the right guidance.
• Assume that people at DrupalCons belong, unless they've told you otherwise, or have explicitly asked for help.
• Assume that someone who starts out their interactions with the community in a blundering fashion could become totally engaged and one of your biggest assets, if they're only shown "the Drupal way".

And most of all, if you see someone mousy and shy, hiding in the corner at Drupalcon who doesn't have anyone to talk to, go talk to 'em. You might just change their life. :)

Tags: drupaldrupalcon

The previous article was introducing basic streaming principles based on Icecast. The issue with this, of course, is lag and overhead of HTTP-based connexions. In this article we introduce RTP-based streaming system, (unfortunately) based on Pulseaudio and multicast.

Why I don't like Pulsaudio

Before I get flamed for attacking Pulseaudio (PA), let's just settle this: I don't like it. I think PA is over-engineered and tries to do too many things at once. I used to (until just now) systematically purge pulseaudio-related packages from my system, mainly because PA has this awful tendency of automatically starting and staying around eternally, which wouldn't be so bad except PA has also the bad habit of hogging the audio device exclusively, which makes regular programs like mplayer, ogg123 fail to simply play audio, unless they go through the PA straight-jacket. In the GNU/Linux audio stack, we already have another system that supports multiple accesses: ALSA itself, so I believe that PA should at least behave nicely to other apps, and it seems I am always having problems with that. (The fact that ALSA itself is over-engineered is also telling of the poor state of the audio stack in GNU/Linux.)

The other beef I have with PA is the Not Invented Here syndrome: instead of extending existing tools like ALSA or Jack, people just figured they could do everything better and start from scratch. So now we just have one more "standard" way of playing audio, good job Lennart

A word on multicast

I took the opportunity of trying out Multicast (it's actually IP multicast, to be more precise), since I wanted to have only minimal configuration to perform on the client side. Plus, since the RTP frames are raw 44khz 16 bit audio, the traffic is actually quite heavy, 1.4mbps, to be more precise (punch in 44kHz * 16bit/Hz * 2 in the excellent Qalculate to see why), so I didn't want bandwidth to explode as more listeners joined in. As a comparison, also note that the Icecast Vorbis stream is around 64 or 128kbps depending on quality.

Multicast allows packets to be sent only once for multiple listeners on the same network. It's actually quite cool, but doesn't work very well over the wider internet, for reasons that are still unclear to me. I presume that the fact there is virtual monopoly over traditional broadcasting systems (e.g. television and radio) means that multicast hasn't really kicked in yet... I hope this will make sense over the Montreal mesh, although the quality would probably need to be dialed down, or the audio streamed encoded instead of be sent as raw samples.

So why PA? VLC/pipe failures

That aside, it seems that to stream through RTP, I have no choice. I have tried to use VLC as a pipe to stream data out of MPD as RTP, but somehow this config snippet just failed:

audio_output { type "pipe" name "RTP" command "cvlc -q - --sout '#rtp{dst=239.72.72.144,port=5004}'" encoder "vorbis" # optional, vorbis or lame format "44100:16:2" }

I don't know why - the multicast port wouldn't open properly and MPD would even hang when restarting. It seems the pipe plugin is somewhat broken. Note that the vlc command above actually works fine to stream audio content out through multicast, which is a good way to test if multicast works in the first place...

Shoving PA down MPD's throat

Instead, I had to go to the clunky PA-way (also know as "my way or the highway"). The first step was to configure the default.pa to enable RTP streaming. I copied over the default /etc/pulse/default.pa file into ~mpd/.pulse/default.pa and uncommented the following lines:

# just to get the PA announced load-module module-zeroconf-publish # we need a null sink, no clue why, but without this glue, it fails load-module module-null-sink sink_name=rtp format=s16be channels=2 rate=44100 sink_properties="device.description='RTP Multicast Sink'" # port is specified otherwise it is random (?!) # loop=1 is to broadcast locally too # other useful setting: destination (defaults to 224.0.0.56) load-module module-rtp-send source=rtp.monitor port=5004 loop=1

See the rtp-send module documentation for more information here.

Then we need MPD to connect to that sink:

audio_output { type "pulse" name "Pulsaudio RTP" sink "rtp" }

Also, now that we got into the PA kitchen sink, we are kind of forced to use it for the local sound card output too, so we'll add this as a sink to:

audio_output { type "pulse" name "Pulseaudio sound card" sink "alsa_output.pci-0000_00_1b.0.analog-stereo" }

(Take a moment to admire the nice sink name. Yours may be different, have fun finding it in the output of pacmd list-sinks, but that works only once PA is started!)

So now you need to restart MPD for those changes to take effect. Note that if the mpd user has already started PA, you're in luck: it won't restart it by itself, and you need to:

service mpd stop killall pulseaudio service mpd start

... yep. Basically because pulseaudio just detaches completely from the parent process when spawned automatically. Lots of fun.

Testing and listening

Of course, you can listen to the stream using... Pulseaudio. I assume this would magically show up on the network thanks to avahi, if you run Pulseaudio on (say) your laptop. Since I still want to stay away from it, I would rather use a straight commandline tool to listen to the stream. According to this page, there aren't that many clients that could listen to such a funky stream. I chose VLC, since its commandline options actually make sense.

Here's the magic mantra to repeat every night before you go to bed:

cvlc rtp://@224.0.0.56:5004

Yes, that's all. Drop the c if you want a GUI.

With mplayer, it's more complicated - you need to specify a bitmask using a hexadecimal string (fun stuff):

mplayer -demuxer rawaudio -rawaudio format=0x20776172 rtp://224.0.0.56:5004

Yes, good old 0x20776172...

Results! We want results!

All this work reduced the lag from 15 seconds to around 1 second. It's still present, and confusing to listen to, but maybe more acceptable. It seems most of the lag is in the vlc client buffering, when I enable debugging, I see:

[0x963c7bc] main input debug: Stream buffering done (1001 ms in 976 ms)

I was able to reduce that delay significantly by using the --rtp-caching option of vlc, as such:

cvlc --rtp-caching=48 rtp://@224.0.0.56:5004

The value is in miliseconds. Anything below 48ms introduces audible artifacts (probably buffer underruns). With that setting, there's still a noticeable delay, especially with isolated sounds that are naturally more noticeable. With more harmonious music, it sounds more like an echo.

I have also tried to fiddle with similar settings with mplayer and simply failed to get any results. --no-cache simply drops samples like crazy. --cache 64 is the lowest I could push it down to, and it still has noticeable delay, which I couldn't measure, but I would assume would be around 400ms at least.

Also note that contrarily to the previous setup I had, based on Icecast, VLC will automatically resume streaming from the RTP stream as it is stopped or started, which is also very nice - as it allows me to disable remote RTP streams easily and harmlessly.

Troubled areas

Speaking of insane nonsense, here are a few pitfalls to avoid.

Take care to disable the ALSA output, if it's not yet commented out, because MPD will freak out with the following error message:

Feb 03 20:25 : output: "Sound card" [alsa] failed to play: Resource temporarily unavailable

I have not been able to find a way to make PA let go of the audio sound card. I would have prefered MPD to open ALSA directly if PA wasn't needed (I need it only for RTP streaming), but it seems that PA assumes you'll want to play stuff locally.

I have tried commenting out the following in default.pa:

load-module module-udev-detect

Seems like it's not enough. Also note that PA is designed with "dumb users" in mind (read: you need a GUI to operate it), so in the context of MPD, it's really annoying to debug things unless you can actually fire up a X11 session as the MPD user (which strikes me as a crazy idea) to configure PA properly with paprefs and all the graphical gizmos.

I just gave up and went with PA all the way, and just added the RTP sink through the commandlines. Other guides can show you screenshots to clikety your way through inane graphical interfaces to do the above, if you are into that kind of stuff.

With PA, you will want to get familiar with the pacmd and pactl commands. Why there are two different commands to basically do the same thing is completely beyond me.

Reference and more reading

I got a lot of information in the Pulseaudio MPD wiki page and the Pulseaudio user documentation. The VLC manual about RTP streaming (server) (client) was also useful for debugging.

The Pulseaudio Network documentation has a set of recipes for using pulseaudio for everything. This Ask Ubuntu question explains how to clikety your way through paprefs to essentially do the above.

I have been using MPD for a good while as my media player, in conjunction with gmpc as a graphical interface (since MPD is just a server). It's really cute, works reasonably well, and is one of the only software that can handle my huge music collection gracefully. The cool thing with the setup is that, because MPD is a network-transparent server, I can control (and listen to!) my music remotely.

Basic HTTP / Icecast streaming

For a while, I have also been streaming my music through Icecast so that I can basically listen to my music from anywhere there is an internet connection with sufficient bandwidth. This configuration is simple enough:

audio_output { type "shout" name "Currently listening to..." host "localhost" port "8000" mount "/radio.ogg" password "hackme" quality "4.0" format "44100:16:2" description "My playlist of the day. High quality (4) OGG/Vorbis stream." }

This will make a stream available at http://radio.example.com:8000/radio.ogg, provided Icecast is configured properly.

That setup is a bit more complicated than necessary, because it needs an icecast server to be installed (which is just apt-get install icecast on Debian, but still). So an alternative, simpler setup, but which may not scale as well to many listeners, is:

audio_output { type "httpd" name "My HTTP Stream" encoder "vorbis" # optional, vorbis or lame port "8000" quality "5.0" # do not define if bitrate is defined bitrate "128" # do not define if quality is defined format "44100:16:1" }

I haven't played around too much with that plugin, so I don't know how well it works - I am using the Icecast plugin instead.

Always on radio

Another cool thing is that I have actually two MPD daemons running at the same time - one is my regular player, but the other is the "shuffle" player. The basic idea is that when the regular player stops, the shuffle starts, and vice versa, so there is always something playing on the stream. The shuffle, by default, doesn't play on the local sound card so that even though the radio stream is up, it doesn't blast random music through my living room speakers all day (and night!).

This trick is accomplished through a hook in the Icecast daemon:

<mount> <mount-name>/radio.ogg</mount-name> <fallback-mount>/random.ogg</fallback-mount> <!-- when I listen to my radio, I turn off the random MPD stream to save CPU --> <on-connect>/home/anarcat/bin/mpd-stop-random</on-connect> <!-- ... and when I stop listening, the other stream just kicks in --> <on-disconnect>/home/anarcat/bin/mpd-start-random</on-disconnect> <!-- i put those here to try to display something on the stats page when there's no client, but it's not working --> <stream-name>Currently listening to...</stream-name> <stream-description>My playlist of the day. High quality (5) OGG/Vorbis stream. If I'm not listening to anything, you'll be redirected to the random playlist.</stream-description> </mount> <!-- final fallback: the random radio, which is started when the above playlist disconnects --> <mount> <mount-name>/random.ogg</mount-name> <stream-name>Random!</stream-name> <stream-description>Random songs from a 33 days playlist</stream-description> </mount>

The mpd-start-random script is rather idiotic:

#!/bin/sh export MPD_HOST=hackme@radio.anarcat.ath.cx export MPD_PORT=6601 mpc play

You can guess what the mpd-stop-random script does (hint: the opposite of play is pause).

So that's the old way of doing things. The one problem I found with this setup is when I want to setup another listener in the same house - the problem is that the Icecast stream has significant delay: a good 14 seconds here, with ogg123 as a client.

Here's the special configuration for the mpd-slave daemon that runs the random playlist:

# same as main player music_directory "/var/lib/mpd/music" playlist_directory "/var/lib/mpd/playlists" # those need to be different db_file "/var/lib/mpd-slave/tag_cache" log_file "/var/log/mpd/mpd-slave.log" pid_file "/var/run/mpd/slave.pid" state_file "/var/lib/mpd-slave/state" # avoids having to update the database by hand all the time auto_update "yes" # disabled by hand after daemon is started audio_output { type "alsa" name "Sound card" } # main icecast stream audio_output { type "shout" name "Random!" host "localhost" port "8000" mount "/random.ogg" password "hackme" quality "4.0" format "44100:16:2" description "Random songs from a 33 days playlist" } # another player for low bandwidth situations, disabled audio_output { type "shout" name "Random! (lowfi)" host "localhost" port "8000" mount "/random.ogg" password "baH4UShi" quality "0.0" format "44100:16:2" description "Random songs from a 33 days playlist" } filesystem_charset "UTF-8" user "mpd" default_permissions "read" port "6601" mixer_type "software" volume_normalization "yes"

The random playlist is done by simply adding the whole library to the playlist. MPD chokes a little the first time you do that, but it's surprisingly fast. I also went through the playlist to remove spoken words and other stuff I don't want to stumble upon, and after I shuffled the playlist.

Final notes

Otherwise, this is a fairly standard MPD setup, although there are some special configuration that vary from the upstream default:

# this needs to be commented out for the server to be accessible from everywhere #bind_to_address "localhost" # setup a password for remote access password "hackme@read,add,control" # make sure the mixer controls don't affect the local hardware mixer mixer_type "software" # we are beyond the ASCII world, but MPD doesn't know filesystem_charset "UTF-8" id3v1_encoding "UTF-8"

This is the first part of a series of articles I intend to write about my MPD configuration, next ones detail the configuration of a dumb player and RTP streaming.

Information overload. Every day my inbox fills up with blog updates and newsletters. Add to that all the tweets and Google+ posts that I skim throughout the day and I occasionally feel lost in an avalanche of information. I'll bet you know the feeling.

But I keep up the pace because I often find things that are informative, funny, inspirational or just make me better at my job. Here are a few of these items I think deserve more than a simple retweet.

The new face of Omega 4

This one was really good if you love Omega theme. As you may have noticed, Omega helps keep the lights on around here, so I was particularly interested in what Kathryn McClintock at Amazee Labs had to say about what's on tap.

While you're giving this one a read, be sure to look at the comments. There is one half way down by fubhy, a major contributor to the Omega project, that will clarify a lot of the questions the post may bring to mind.

Spark update: unified in-place editing

For ages it seems, there has been a part of the Drupal community grousing about the lack of a built-in rich text editor. I was among them. This post by Dries Buytaert talks about some of the cool in-place content editing features that are rolling out with Drupal 8. 

An interesting companion post is Dries' explanation of why they scuttled Aloha Editor and went with CKEditor for the next version of Drupal.

Is Drupal too big?

Tim Millwood's post about the challenges new Drupal developers face building quality websites really got me thinking. I wrote a post for new site builders as a result of my ruminating. The comments for Tim's post are also pretty interesting, particularly if you care about optimizing Drupal performance.

Another really good (and fairly recent) post by Tim is, Top 15 Drupal performance tips, which he wrote for .net magazine.

Aaron Winborn Special Needs Trust

If you're not familiar with Aaron Winborn, you're probably at least familiar with his work. Aaron has contributed modules like Embedded Media Field and Views Slideshow, among others. He's also the author of the book, Drupal Multimedia. In short, he's a major contributor to the Drupal community.

Aaron has developed ALS, a serious degenerative disease. In this post he has appealed for assistance as he and his young family brace themselves for the challenges to come. Please give the post a read and consider giving Aaron and his family whatever support you can.

A good three weeks ago, we introduced the Rcpp Gallery. While this is a joint effort by several of us on the Rcpp team, the backend was conceived and implemented entirely by JJ who also bootstrapped it with same first content, drawing on posts by Hadley, Romain and myself. As the How to contribute page makes plain, this is all backed by GitHub and all logs are public anyway.

So after it was up and working, JJ and I refined the look and feel, and I started to add more content so that would have something by the time the initial announcement came around. A few years I read about an (attributed) secret to Seinfeld's producitivity: "Don't break the chain". Just keep writing, and write every day.

I made my goal of a post every day for just over a month, and created this sequences: <quote> (20 Dec) simulating-pi, (21 Dec) vector-minimum, (22 Dec) gsl-colnorm-example, (23 Dec) fibonacci-sequence, (24 Dec) random-number-generation, (25 Dec) armadillo-sparse-matrix, (26 Dec) timing-rngs, (27 Dec) stl-inner-product, (28 Dec) stl-transform, (29 Dec) stl-transform-for-subsetting, (30 Dec) stl-random-shuffle, (31 Dec) stl-random-sample, (01 Jan) stl-for-each, (02 Jan) armadillo-subsetting, (03 Jan) accessing-environments, (04 Jan) armadillo-eigenvalues, (05 Jan) r-function-from-c++, (06 Jan) using-the-rcpp-timer, (07 Jan) sugar-function-clamp, (08 Jan) using-rcout, (09 Jan) first-steps-with-C++11, (10 Jan) simple-lambda-func-c++11, (11 Jan) eigen-eigenvalues, (12 Jan) getting-attributes-for-xts-example, (13 Jan) intro-to-exceptions, (14 Jan) a-first-boost-example, (15 Jan) a-second-boost-example, (16 Jan) timing-normal-rngs, (17 Jan) creating-xts-from-c++, (18 Jan) gsl-for-eigenvalues, (19 Jan) accessing-xts-api, (20 Jan) custom-as-and-wrap-example, (21 Jan) passing-cpp-function-pointers, </quote>

The Rcpp Gallery continues to grow, we now have 58 posts from 7 different authors. And it is open for business: new contributions are always welcome.

Back at DrupalCon Munich code sprint I started to work on a core issue that is focused on implementing HTML5 multiple uploads support in Drupal. Since I've always been interested in file handling I immediatley found this very attractive. From the other hand I always strugeled when it comes to files in Drupal. There are so many solutions and none of them is really user friendly. I thought that this feature would make Drupal's UX much better. During my travel back home from Munich and the days after I spent some time working on the issue, but then it stuck somwhere in the middle due to my lack of time.

In January I started to work for Examiner.com where I got so-called Drupal day at the end of every sprint. That helped me to focus on this issue once again (and I am very thankful for that). During last 2 or 3 weeks I managed to finish the patch (in terms of features):

• file and managed_file FAPI elements have been updated to accept #multiple option. Once this option is enabled element adds "multiple" argument to it's input tag, which enables multi-file uploads in recent versions of all major browsers. Old browsers should gracefully degrade to single-upload without any problems. 
• File & Image field widgets leverage new #multiple option in it's underlaying form elements to improve experience for multi-value fields. Users will no longer upload galleries on file at a time. Everything can be now done in a single step.
• Existing tests were corrected to pass with newly added features (FAPI element behaviour changed a bit).

I've already been testing, but patch still needs a lot of testing and reviews. More information can be found in the issue queue.

One of the topics that has come up a few times during the development of Movit (my high-performance, high-quality video filter library) is the one of color and color spaces. There's a lot of information out there, but it took me quite a while to put everything together in my own head. Thus, allow me to share a distilled version; I'll try to skip all the detail and the boring parts. Color is an extremely complex topic, though; the more I understand, the more confusing it becomes. Thus, it will probably become quite long.

What is color?

Color is, ultimately, the way our vision reacts to the fact that light comes in many different frequencies. (In a sense, the field of color is actually more a subfield of biology than of physics.) In its most exact form, you can describe this with a frequency spectrum. For instance, here is (from Wikipedia) a typical spectrum of the sky on a clear summer day:

However, human eyes are not spectrometers; there are many colors with different frequency spectra that we perceive as the same. Thus, it's useful to invent some sort of representation that more closely corresponds to how we see color.

Now, almost everybody knows that we represent colors on computers with various amounts of red, green and blue. This is correct, but how do we go from those spectra to RGB values?

XYZ

The first piece of the puzzle comes in the form of the CIE 1931 XYZ color space. It defines three colors, X, Y and Z, that look like this (again Wikipedia; all the images in this post are):

Don't be confused that they are drawn in red, green and blue, because they don't correspond to RGB. (They also don't correspond to the different cones in the eye.)

In any case, almost all modern color theory starts off by saying that describing frequency spectra by various mixtures of X, Y and Z is a good enough starting point. (In particular, this means we discard infrared and ultraviolet.) As a handy bonus, Y corresponds very closely to our perception of overall brightness, so if you set X=Z=0, you can describe a black-and-white picture with only Y. (This is the same Y as you might have seen in YUV or YCbCr. Let me ignore the distinction between Y and Y' for now.)

Actually we tend to go one step further when discussing color, since we don't care about the brightness; we normalize the XYZ coordinates so that x+y+z=1, after which a color is uniquely defined with only its x and y (note that we now write lowercase!) values. (If we also include the original Y value, we have the full description of the same color again, so the xyY color space is equivalent to the XYZ one.)

RGB and spectral colors

So, now we have a way to describe colors in an absolute sense with only three numbers. Now, we already said earlier, usually we do this by using RGB. However, this begs the question: When I say “red”, which color do I mean exactly? What would be its xy coordinates?

The natural answer would probably be some sort of spectral color. We all know the spectral colors from the rainbow; they are the ones that contain a single wavelength of light. (Then we start saying stupid things like “white is not a color” since it is a mixture of many wavelengths, conveniently ignoring that e.g. brown is also a mixture and thus not in the rainbow. I've never heard anyone saying brown is not a color.) If you take all the spectral values, convert them to xy coordinates and draw them in a diagram, you get something like this:

(To be clear, what I'm talking about is the big curve, with markings from 380 nm to 700 nm.)

So now, we can define “red” to be e.g. light at 660 nm, and similar for green and blue. This gives rise to a gamut, the range of colors we can represent by mixing R, G and B in various amounts. For instance, here's the Rec. 2020 (Rec. = Recommendation) color space, used in the upcoming UHDTV standard:

You can see that we've limited ourselves a bit here; some colors (like spectral light at 500 nm) fall outside our gamut and cannot be represented except as some sort of approximation. Still, it's pretty good.

For a full description, we also need a white point that says where we end up when we set R=G=B, but let me skip over the discussion of “what is white” right now. (Hint: Usually it's not “equal amount of all wavelengths”.) There's also usually all sorts of descriptions about ambient lighting and flare in your monitor and whatnot—again, let me skip over them. You can see the white point marked off as “D65” in the diagram above.

A better compromise

You might have guessed by now that we rarely actually use spectral primaries today, and you're right. This has a few very important reasons:

First, it makes for a color space that is very hard to realize in practice. How many things to do you know that can make exactly single-frequency light? Probably only one: Lasers. I'm sure that having a TV built with a ton of lasers would be cool (*pew pew*!), but right now, we're stuck with LCD and LED and such. (You may have noticed that outside a certain point, all the colors in the diagram look the same. Your monitor simply can't show the difference anymore.) You could, of course, argue that we should let it be the monitor's problem to figure out what do to with the colors it can't represent, but proper gamut mapping is very hard, and the subject of much research.

Second, the fact that the primaries are far from each other means that we need many bits to describe transitions between them smoothly. The typical 8 bits of today are not really enough; UHDTV will be done with 10- or 12-bit precision. (Processing should probably have even more; Movit uses full floating-point.)

Third, pure colors are actually quite dim (they contain little energy). When producing a TV, color reproduction is not all you care about; you also care about light level for e.g. white. If we reduce the saturation of our primaries a bit (moving them towards the white point), we make it easier to get a nice and bright output image.

So, here are the primaries of the sRGB color space, which is pretty much universally used on PCs today (and the same primaries as Rec. 709, used for today's HDTV broadcasts):

Quite a bit narrower; in particular, we've lost a lot in the greens. This is why some photographers prefer to work in a wider color space like e.g. Adobe RGB; no need to let your monitor's limitations come between what your camera and printer can do. (Printer gamuts are a whole new story, and they don't really work the same way monitor gamuts do.)

Color spaces and Movit

So, this is why Movit, and really anything processing color data, has to care: To do accurate color processing, you must know what color space you are working in. If you take in RGB pixels from an sRGB device, and then take those exact values and show on an SDTV (which uses a subtly different color space, Rec. 601), your colors will be slightly off. Remember, red is not red. sRGB and SDTV are not so different, but what about sRGB and Rec. 2020? If you take your sRGB data and try to send it on the air for UHDTV, it will look strangely oversaturated.

You could argue that almost everything is sRGB right now anyway, and that the difference between sRGB and Rec. 601 is so small that you can ignore it. Maybe; I prefer not to give people too many reasons to hate my software in the future. :-)

So Movit solves this by moving everything into the same colorspace on input, and processes everything as sRGB internally. (Basically what you do is you convert the color from whatever colorspace to XYZ, and then from XYZ to sRGB. On output, you go the other way.) Lightroom does something similar, only with a huge-gamut colorspace (so big it includes “imaginary colors”, colors that can't actually be represented as spectra) called Pro Photo RGB; I might go that way in the future, but currently, sRGB will do.

this week's report comes live from FOSDEM; which doesn't mean I've been doing any RC bug work here, I've looked at all the mentioned bugs below earlier this week. & as you can see, most was just a bit triaging …

• #698631 – typo3: "typo3: copyright file missing after upgrade (policy 12.5)"
  replace directory with symlink in postinst, upload to DELAYED/5
• #699252 – src:ldiskfsprogs: "ldiskfsprogs: FTBFS: build-dependency not installable: lustre-dev"
  add some info to the bug report, then removed from testing
• #699254 – src:libpdfbox-java: "libpdfbox-java: FTBFS: cp: missing file operand"
  replace removed packages with poppler-data in Build-Depends-Indep
• #699255 – src:ruby-activeresource-2.3: "ruby-activeresource-2.3: FTBFS: test_load_yaml_array(BaseTest) fails"
  tag confirmed
• #699256 – src:cglib: "cglib: FTBFS: 1) testFailOnMemoryLeak(net.sf.cglib.proxy.TestEnhancer)junit.framework.AssertionFailedError: Memory leak caused by Enhancer"
  investigate and tag unreproducible
• #699258 – src:libio-async-loop-glib-perl: "libio-async-loop-glib-perl: FTBFS: test failed"
  confirm and forward upstream
• #699259 – src:gnuradio: "gnuradio: FTBFS: XML parsing error"
  tag unreproducible
• #699261 – src:dhelp: "dhelp: FTBFS: build-dependency not installable: libgettext-ruby-util"
  prepare patches
• #699263 – src:pspp: "pspp: FTBFS: 293. expressions - datediff (evaluate.at:1560): FAILED (evaluate.at:1560)"
  tag confirmed

My magcat module for cataloguing magazines, which compliments my bookcat module (which is a dependency) has been promoted to alpha 2. To get the best use of the module you will need a copy of the MS Windows FNProgramvare's BookCAT product and create your MS Access databases there before importing into MySQL.

Follow the link above to download the latest version.

It was interesting to read recently from Martin F. Krafft a botnet-like configuration management proposal.

Professionally I've used CFEngine, which in version 2.x, supported a bare minimum of primitives, along with a distribution systme to control access to a central server. Using thse minimal primitives you could do almost anything:

• Copy files, and restart services that depend upon them.
• Make minor edits to files. (Appending lines not present, replacing lines you no longer wanted, etc)
• Installing / Removing packages.
• More ..

Now I have my mini cluster (and even before that when I had 3-5 machines) it was time to look around for something for myself.

I didn't like the overhead of puppet, and many of the other systems. Similarly I didn't want to mess around with weird configuration systems. From CFEngine I'd learned that using only a few simple primitives would be sufficient to manage many machines provided you could wrap them in a real language - for control flow, loops, conditionals, etc. What more natural choice was there than perl, the sysadmin army-knife?

To that end slaughter was born:

• Download polices (i.e. rules) to apply from a central machine using nothing more complex than HTTP.
• Entirely client-driven, and scheduled via cron.

Over time it evolved so that HTTP wasn't the only transport. Now you can fetch your policies, and the files you might serve, via git, hg, rsync, http, and more.

Today I've added one final addition, and now it is possible to distribute "modules" alongside policies and files. Modules are nothing more than perl modules, so they can be as portable as you are careful.

I envisage writing a couple of sample modules; for example one allowing you to list available sites in Apache, disable the live ones, enable/disable mod_rewrite, etc.

These modules will be decoupled from the policies, and will thus be shareable.

Anyway , I'm always curious to learn about configuration management systems but I think that even though I've reinvented the wheel I've done so usefully. The DSL that other systems use can be fiddly and annoying - using a real language at the core of the system seems like a good win.

There are systems layered upon SSH, such as fabric, ansible, etc, and that was almost a route I went down - but ultimately I prefer the notion of client-pull to server-push, although it is possible in the future we'll launche a mini-daemon to allow a central host/hosts to initial a run.

With Field collection you could attach a group of fields to a field in any entity in Drupal.

The module is done by fago and tim.plunket, and it adds some really nice flexibility to fields.

Warning: This screencast featurea a short glimpse of a really old persian cat and some beer drinking. You have been warned.

Today we where packing back our holiday decorations into boxes. We also had a music box to put away.

Do you see the defect in the picture? ;) No? Okay ... some years ago my oldest daughter broke it into some pieces and a friend of us (Hi muempf! ;) did glue all together. When he showed us the nice music box, it was recovered very well, but one detail was changed from original. One horseman wasn't reassembled as the other onces.

If you you didn't found the defect, please have a look here.

You had fun with this? Maybe this is another one for you, I took that photo on christmas eve when my youngest daughter had placed her new 'Lisa Plastic':

Keep smiling!

n/a

This is the "too late because FOSDEM" edition. On the plus side, we will be ready to release Sqyeeze next week...

The UDD bugs interface currently knows about the following release critical bugs:

• In Total: 788
  • Affecting wheezy: 224 That's the number we need to get down to zero before the release. They can be split in two big categories:
    • Affecting wheezy and unstable: 132 Those need someone to find a fix, or to finish the work to upload a fix to unstable:
      • 28 bugs are tagged 'patch'. Please help by reviewing the patches, and (if you are a DD) by uploading them.
      • 5 bugs are marked as done, but still affect unstable. This can happen due to missing builds on some architectures, for example. Help investigate!
      • 100 bugs are neither tagged patch, nor marked done. Help make a first step towards resolution!
    • Affecting wheezy only: 92 Those are already fixed in unstable, but the fix still needs to migrate to wheezy. You can help by submitting unblock requests for fixed packages, by investigating why packages do not migrate, or by reviewing submitted unblock requests.
      • 62 bugs are in packages that are unblocked by the release team.
      • 30 bugs are in packages that are not unblocked.

How do we compare to the Squeeze release cycle?

Week Squeeze Wheezy Diff 43 284 (213+71) 468 (332+136) +184 (+119/+65) 44 261 (201+60) 408 (265+143) +147 (+64/+83) 45 261 (205+56) 425 (291+134) +164 (+86/+78) 46 271 (200+71) 401 (258+143) +130 (+58/+72) 47 283 (209+74) 366 (221+145) +83 (+12/+71) 48 256 (177+79) 378 (230+148) +122 (+53/+69) 49 256 (180+76) 360 (216+155) +104 (+36/+79) 50 204 (148+56) 339 (195+144) +135 (+47/+90) 51 178 (124+54) 323 (190+133) +145 (+66/+79) 52 115 (78+37) 289 (190+99) +174 (+112/+62) 1 93 (60+33) 287 (171+116) +194 (+111/+83) 2 82 (46+36) 271 (162+109) +189 (+116/+73) 3 25 (15+10) 249 (165+84) +224 (+150/+74) 4 14 (8+6) 244 (176+68) +230 (+168/+62) 5 2 (0+2) 224 (132+92) +222 (+132/+90) 6 release!

Graphical overview of bug stats thanks to azhag:

My last bitcoin related blog post mentioned that the new bitcoin package for Debian was waiting in NEW. It was accepted by the Debian ftp-masters 2013-01-19, and have been available in unstable since then. It was automatically copied to Ubuntu, and is available in their Raring version too.

But there is a strange problem with the build that block this new version from being available on the i386 and kfreebsd-i386 architectures. For some strange reason, the autobuilders in Debian for these architectures fail to run the test suite on these architectures (BTS #672524). We are so far unable to reproduce it when building it manually, and no-one have been able to propose a fix. If you got an idea what is failing, please let us know via the BTS.

One feature that is annoying me with of the bitcoin client, because I often run low on disk space, is the fact that the client will exit if it run short on space (BTS #696715). So make sure you have enough disk space when you run it. :)

As usual, if you use bitcoin and want to show your support of my activities, please send Bitcoin donations to my address 15oWEoG9dUPovwmUL9KWAnYRtNJEkP1u1b.