Drupal Watchdog: Got Game?

Planet Drupal - Fri, 07/08/2015 - 18:16

This is not an article about gamification. Unless your website is devoted to dragons, adding quests and hit points won’t improve your users’ engagement with your content. (If your website is devoted to dragons, well that’s just awesome.)

Content – be it text, images, video, data, or a combination of mediums – is the reason we build websites in the first place. It’s right there in the acronym: CMS. Drupal is a system – or more accurately, a framework – for managing content. We strongly believe that all website features, layout, and design choices must support the goal of serving your target audiences with the critical information – the content – they need to engage meaningfully with your organization.

Your content is the connection between your organizational goals and your audiences’ motivations. There’s usually a reason a piece of content is added to a website; somebody, at some point, thought it would be useful. Unless that content has meaning to your users, however, it has little value to your organization. Without a strategy guiding the creation and governance of that content, your quest, noble though it may be, is almost doomed to fail.

Fortunately, it’s not hard to start creating a strategy. We believe you can learn the foundations of content strategy for Drupal websites by breaking down what the team at BioWare did in creating Dragon Age: Inquisition.

Goals and Audiences

You have to start with goals.

BioWare’s basic goal is easy to suss out: they want to make money. To support such a massive undertaking – creating Inquisition involved programmers, writers, producers, graphic artists, vocal talent, project management, and more – the end result had to be a game that would appeal to enough people not only to pay for expenses, but to turn a profit. (We have a strong suspicion the team also wanted to turn around the negative reception Dragon Age 2 met by releasing something that would blow more than a few minds.)

Categories: Elsewhere

Neil McGovern: Forty five hours

Planet Debian - Fri, 07/08/2015 - 14:53

As some may know, since October 2013, I’ve been studying to gain my Private Pilot Licence, and I finally achieved this goal. It’s actually taken quite a bit more than 45 hours – a total of around 60, but that does include a day trip to France (Le Touquet) and getting my night rating as well.

This basically means I can fly single engine piston aeroplanes on my own, with passengers, as soon as my paperwork gets processed by the Civil Aviation Authority anyway.

I’ve been flying Cessna 172s from Cambridge Aero Club, where they have four of them, G-SHWK, G-UFCB, G-HERC and G-MEGS, as well as a Extra 200, G-GLOC. It’s a great club, with fantastically maintained planes and great instructors, and Cambridge Airport has a full ATC service as well, so it’s been useful to get that experience, especially as the UK’s airspace is fairly contended with a lot of controlled and military airspace which needs permission to enter.

As for what next, I need to work that out. When you get your licence, it’s often described as a license to learn, so that’s what I intend to do. Apart from popping over to France for lunch every now and again, I’m probably going to have a go at aerobatics and farm strip flying, then probably look at my IMC rating.

So, if it’s a nice day, and you’re around in Cambridge, let me know if you want a trip up in the skies!

Categories: Elsewhere

Mike Gabriel: New plugin for GOsa²: gosa-plugin-mailaddress

Planet Debian - Fri, 07/08/2015 - 14:31

During last week, I hacked a new plugin together for GOsa².

Simply quoting parts from debian/control here to inform you on its functionality:

Package: gosa-plugin-mailaddress
Architecture: all
 gosa (>= 2.7),
Description: Simple plugin to manage user mail addresses in GOsa²
 This plugin is a very light-weighted version of the GOsa² mail plugin.
 Whereas gosa-plugin-mail can be used to manage a complete mail server
 farm, this tiny plugin only provides means to modify the user's mail
 address via a text field.
 This plugin is useful for people that need to maintain users' email
 addresses via GOsa², but do not run their own mailserver(s).
 GOsa² is a combination of system-administrator and end-user web
 interface, designed to handle LDAP based setups.

Mike (aka sunweaver)

Categories: Elsewhere

Mike Gabriel: My FLOSS activities in July 2015

Planet Debian - Fri, 07/08/2015 - 14:28

July 2015 has been mainly dedicated to these five fields of endeavour:

  • Debian Edu rollout at a grammar school (Gymnasium) in Lübeck, Germany
  • GOsa² and Debian Edu testing and fixing
  • Upgrading a Debian Edu squeeze main server to jessie
  • Packaging started to get ILIAS into Debian
  • Work on Debian and Debian LTS
Debian Edu rollout at a grammar school (Gymnasium) in Lübeck, Germany

In spring 2015, we got contacted by the IT coordinator of a grammar school (Gymnasium) in Lübeck, Germany. He asked for some consultancy on the existing school network based on Debian and Linux Mint. The school has been running on Linux all-over for the past 5 years (at least, IIRC).

After several phone calls and a personal meeting, the decision was reached to switch over the educational segment of their school IT completely to Debian Edu / Skolelinux.

GOsa² and Debian Edu testing and fixing

This new customer gave us the opportunity of intensively testing Debian Edu jessie. Diverting from previous rollouts, we dropped LibVirt as virtualization technology and switched over to Ganeti. The Debian Edu machines all run in KVM virtual machines.

Our Diskless Workstations and diskfull workstations have been running on Debian Edu jessie plus MATE desktop environment for a while already. But the main servers at other customers' are still on Debian Edu squeeze.

read more

Categories: Elsewhere

Drupal core announcements: Recording from Aug 7th 2015 Drupal 8 critical issues discussion

Planet Drupal - Fri, 07/08/2015 - 14:00

We met again today to discuss critical issues blocking Drupal 8's release (candidate). (See all prior recordings). Here is the recording of the meeting video and chat from today in the hope that it helps more than just those who were on the meeting:

If you also have significant time to work on critical issues in Drupal 8 and we did not include you, let me know as soon as possible.

The meeting log is as follows (all times are GMT real time at the meeting):

10:10 WimLeers https://www.drupal.org/node/2543332
10:10 Druplicon https://www.drupal.org/node/2543332 => Auto-placeholdering for #lazy_builder without bubbling [#2543332] => 40 comments, 8 IRC mentions

10:11 WimLeers https://www.drupal.org/node/2503963
10:11 Druplicon https://www.drupal.org/node/2503963 => XSS in Quick Edit: entity title is not safely encoded [#2503963] => 27 comments, 2 IRC mentions

10:12 plach https://www.drupal.org/node/2544954
10:12 Druplicon https://www.drupal.org/node/2544954 => SqlContentEntityStorageSchema does not detect column-level schema changes in non-base-fields [#2544954] => 16 comments, 1 IRC mention

10:12 plach https://www.drupal.org/node/2542748
10:12 Druplicon https://www.drupal.org/node/2542748 => Automatic entity updates are not safe to run on update.php by default [#2542748] => 60 comments, 14 IRC mentions

10:14 dawehner https://www.drupal.org/node/2540416
10:14 Druplicon https://www.drupal.org/node/2540416 => Decide whether we need hook_upgrade_N()/upgrade.php front controller [#2540416] => 43 comments, 11 IRC mentions

10:17 webchick https://assoc.drupal.org/d8accelerate

10:17 webchick https://www.drupal.org/node/2485119
10:17 Druplicon https://www.drupal.org/node/2485119 => [meta] The Drupal 8.0.0-rc1 Release Checklist [#2485119] => 32 comments, 11 IRC mentions

10:17 webchick https://www.drupal.org/node/2042447
10:17 Druplicon https://www.drupal.org/node/2042447 => Install a module user interface does not install modules (or themes) [#2042447] => 178 comments, 25 IRC mentions

10:18 webchick https://www.drupal.org/node/2267715
10:18 Druplicon https://www.drupal.org/node/2267715 => [meta] Drupal.org (websites/infra) blockers to a Drupal 8 release [#2267715] => 53 comments, 17 IRC mentions

10:19 alexpott https://www.drupal.org/node/2280965
10:19 Druplicon https://www.drupal.org/node/2280965 => [meta] Remove or document every SafeMarkup::set() call [#2280965] => 109 comments, 24 IRC mentions

10:19 alexpott

10:21 dawehner https://www.drupal.org/node/2503963
10:21 Druplicon https://www.drupal.org/node/2503963 => XSS in Quick Edit: entity title is not safely encoded [#2503963] => 27 comments, 3 IRC mentions

10:23 WimLeers dawehner: catch: alexpott: https://www.drupal.org/node/2547127#comment-10194525
10:23 Druplicon https://www.drupal.org/node/2547127 => [regression] Empty messages container appearing when a Notice occurs [#2547127] => 19 comments, 4 IRC mentions

10:37 alexpott https://www.drupal.org/node/2527360
10:37 Druplicon https://www.drupal.org/node/2527360 => Review all usages of Xss::filter(), Xss::filterAdmin(), SafeMarkup::checkPlain() and SafeMarkup::escape() [#2527360] => 0 comments, 2 IRC mentions

10:38 alexpott https://www.drupal.org/node/2546232
10:38 Druplicon https://www.drupal.org/node/2546232 => Remove SafeMarkup::checkPlain() from UrlHelper [
#2546232] => 3 comments, 1 IRC mention

10:40 dawehner https://www.drupal.org/node/2540416
10:40 Druplicon https://www.drupal.org/node/2540416 => Decide whether we need hook_upgrade_N()/upgrade.php front controller [#2540416] => 43 comments, 12 IRC mentions

10:55 webchick https://www.drupal.org/node/2542748
10:55 Druplicon https://www.drupal.org/node/2542748 => Automatic entity updates are not safe to run on update.php by default [#2542748] => 60 comments, 15 IRC mentions

10:58 alexpott https://www.drupal.org/node/2544932
10:58 dawehner https://www.drupal.org/node/2497243
10:58 Druplicon https://www.drupal.org/node/2544932 => Do not rely on loading php from shared file system [#2544932] => 6 comments, 2 IRC mentions
10:58 Druplicon https://www.drupal.org/node/2497243 => Rebuilding service container results in endless stampede [#2497243] => 208 comments, 48 IRC mentions

11:02 webchick Ok. Time to pass out. ;)

Thanks to @alexpott for the video and log.

Categories: Elsewhere

Ben Hutchings: Debian LTS work, July 2015

Planet Debian - Fri, 07/08/2015 - 13:52

This was my eighth month working on Debian LTS. I was assigned 14.75 hours of work by Freexian's Debian LTS initiative.


I didn't upload any new version of the kernel this month, but I did send all the recent security fixes to Willy Tarreau who maintains the 2.6.32.y branch at kernel.org. I also spent more time working on a fix for bug #770492 aka CVE-2015-1350, which is not yet fixed upstream. I now have a candidate patch for 2.6.32.y/squeeze, and automated tests covering many of the affected filesystems.

Front desk

The LTS 'front desk' role is now assigned on a rota, and I was my first turn in the third week of July. I investigated which new CVEs affected LTS-supported packages in squeeze, recorded this in the secure-testing repository, and mailed the package maintainers to give them a chance to handle the updates.


Groovy had a single issue (CVE-2015-3253) with a simple fix that I could apply to the version in squeeze. Unfortunately the previous version in squeeze had not been properly updated during the squeeze release cycle and could no longer be built from source. I eventually worked out what the build-dependencies should be, uploaded the fix and issued DLA-274-1.


Ruby 1.9.1 also had a single issue (CVE-2014-6438), though the fixes were more complicated and hard to find. (The original bug report for this is still not public in the upstream bug tracker.) I also had to find an earlier upstream change that they depended on. As I've mentioned before, Ruby has an extensive test suite so I could be quite confident in my backported changes. I uploaded and issued DLA-275-1.


The GNU library for Internationalized Domain Names, libidn, required applications to pass only valid UTF-8 strings as domain names. The Jabber instant messaging server turned out not to be validating untrusted domain names, leading to a security issue there (CVE-2015-2059). As there are likely to be other applications with similar bugs, this was resolved by adding UTF-8 validation to libidn.

The fix for this involved importing additional functions from the GNU portability library, gnulib, and there my difficulties began. Confusingly, libidn has two separate sets of functions imported from gnulib, and due to interdependencies it turned out that I would have to update both of these wholesale rather than just importing the new functions that were wanted. This resulted in a 35,000 line patch. Following that I needed to autoreconf the package (and debug that process when it failed), ending up with another 26,000 line patch. Finally, it turned out that the new gnulib code needed gperf to build a header file for Solaris (when building for Linux? huh?). I ended up adding that with another patch instead.

libidn has a decent test suite, so I could at least be confident in the result of my changes. I uploaded and issued DLA-277-1.

Dear upstream developers, please use sane libraries instead of gnulib.

Categories: Elsewhere

Tim Millwood: Composer in contrib: Step 1

Planet Drupal - Fri, 07/08/2015 - 13:38
Drupal 8 has seen a lot of love for Composer. As various posts have mentioned, it’s possible to...
Categories: Elsewhere

InternetDevels: Web developer tools for Drupal 7

Planet Drupal - Fri, 07/08/2015 - 10:18

To create a simple website on Drupal 7, an administrative interface and a simple text editor will be enough, but when it comes to more serious projects, you cannot do without additional Drupal web developer tools and powerful code editors.

Tools used in Drupal web development can be divided into two types:

Read more
Categories: Elsewhere

KnackForge: Drupal 8 Accelerate

Planet Drupal - Fri, 07/08/2015 - 07:48

Categories: Elsewhere

John Goerzen: Detailed Smart Card Cryptographic Token Security Guide

Planet Debian - Fri, 07/08/2015 - 04:24

After my first post about smartcards under Linux, I thought I would share some information I’ve been gathering.

This post is already huge, so I am not going to dive into — much — specific commands, but I am linking to many sources with detailed instructions.

I’ve reviewed several types of cards. For this review, I will focus on the OpenPGP card and the Yubikey NEO, since the Cardomatic Smartcard-HSM is not supported by the gpg version in Jessie.

Both cards are produced by people with strong support for the Free Software ecosystem and have strong cross-platform support with source code.

OpenPGP card: Basics with GnuPG

The OpenPGP card is well-known as one of the first smart cards to work well on Linux. It is a single-application card focused on use with GPG. Generally speaking, by the way, you want GPG2 for use with smartcards.

Basically, this card contains three slots: decryption, signing, and authentication slots. The concept is that the private key portions of the keys used for these items are stored only on the card, can never be extracted from the card, and the cryptographic operations are performed on the card. There is more information in my original post. In a fairly rare move for smartcards, this card supports 4096-byte RSA keys; most are restricted to 2048-byte keys.

The FSF Europe hands these out to people and has a lot of good information about them online, including some HOWTOs. The official GnuPG smart card howto is 10 years old, and although it has some good background, I’d suggest using the FSFE instructions instead.

As you’ll see in a bit, most of this information also pertains to the OpenPGP mode of the Yubikey Neo.

OpenPGP card: Other uses

Of course, this is already pretty great to enhance your GPG security, but there’s a lot more that you can do with this card to add two-factor authentication (2FA) to a lot of other areas. Here are some pointers:

OpenPGP card: remote authentication with ssh

You can store the private part of your ssh key on the card. Traditionally, this was only done by using the ssh agent emulation mode of gnupg-agent. This is still possible, of course.

Now, however, the OpenSC project now supports the OpenPGP card as a PKCS#11 and PKCS#15 card, which means it works natively with ssh-agent as well. Try just ssh-add -s /usr/lib/x86_64-linux-gnu/pkcs11/opensc-pkcs11.so if you’ve put a key in the auth slot with GPG. ssh-add -L will list its fingerprint for insertion into authorized_keys. Very simple!

As an aside: Comments that you need scute for PKCS#11 support are now outdated. I do not recommend scute. It is quite buggy.

OpenPGP card: local authentication with PAM

You can authenticate logins to a local machine by using the card with libpam-poldi — here are some instructions.

Between the use with ssh and the use with PAM, we have now covered 2FA for both local and remote use in Unix environments.

OpenPGP card: use on Windows

Let’s move on to Windows environments. The standard suggestion here seems to be the mysmartlogon OpenPGP mini-driver. It works with some sort of Windows CA system, or the local accounts using EIDAuthenticate. I have not yet tried this.

OpenPGP card: Use with X.509 or Windows Active Directory

You can use the card in X.509 mode via these gpgsm instructions, which apparently also work with Windows Active Directory in some fashion.

You can also use it with web browsers to present a certificate from a client for client authentication. For example, here are OpenSC instructions for Firefox.

OpenPGP card: Use with OpenVPN

Via the PKCS#11 mode, this card should be usable to authenticate a client to OpenVPN. See the official OpenVPN HOWTO or these other instructions for more.

OpenPGP card: a note on PKCS#11 and PKCS#15 support

You’ll want to install the opensc-pkcs11 package, and then give the path /usr/lib/x86_64-linux-gnu/pkcs11/opensc-pkcs11.so whenever something needs the PKCS#11 library. There seem to be some locking/contention issues between GPG2 and OpenSC, however. Usually killing pcscd and scdaemon will resolve this.

I would recommend doing manipulation operations (setting PINs, generating or uploading keys, etc.) via GPG2 only. Use the PKCS#11 tools only to access.

OpenPGP card: further reading

Kernel Concepts also has some nice readers; you can get this card in a small USB form-factor by getting the mini-card and the Gemalto reader.

Yubikey Neo Introduction

The Yubikey Neo is a fascinating device. It is a small USB and NFC device, a little smaller than your average USB drive. It is a multi-application device that actually has six distinct modes:

  • OpenPGP JavaCard Applet (pc/sc-compatible)
  • Personal Identity Verification [PIV] (pc/sc-compatible, PKCS#11-compatible in Windows and OpenSC)
  • Yubico HOTP, via your own auth server or Yubico’s
  • OATH, with its two sub-modes:
    • OATH TOTP, with a mobile or desktop helper app (drop-in for Google Authenticator
  • Challenge-response mode
  • U2F (Universal 2nd Factor) with Chrome

There is a ton to digest with this device.

Yubikey Neo Basics

By default, the Yubikey Neo is locked to only a subset of its features. Using the yubikey-personalization tool (you’ll need the version in stretch; jessie is too old), you can use ykpersonalize -m86 to unlock the full possibilities of the card. Run that command, then unplug and replug the device.

It will present itself as a USB keyboard as well as a PC/SC-compatible card reader. It has a capacitive button, which is used to have it generate keystrokes to input validation information for HOTP or HMAC validation. It has two “slots” that can be configured with HMAC and HOTP; a short button press selects the default slot #1 and a long press selects slot #2.

But before we get into that, let’s step back at some basics.

opensc-tool –list-algorithms claims this card supports RSA with 1024, 2048, and 3072 sizes, and EC with 256 and 384-bit sizes. I haven’t personally verified anything other than RSA-2048 though.

Yubikey Neo: OpenPGP support

In this mode, the card is mostly compatible with the physical OpenPGP card. I say “mostly” because there are a few protocol differences I’ll get into later. It is also limited to 2048-byte keys.

Support for this is built into GnuPG and the GnuPG features described above all work fine.

In this mode, it uses firmware from the Yubico fork of the JavaCard OpenPGP Card applet. There are Yubico-specific tutorials available, but again, most of the general GPG stuff applies.

You can use gnupg-agent to use the card with SSH as before. However, due to some incompatibilities, the OpenPGP applet on this card cannot be used as a PKCS#11 card with either scute or OpenSC. That is not exactly a huge problem, however, as the card has another applet (PIV) that is compatible with OpenSC and so this still provides an avenue for SSH, OpenVPN, Mozilla, etc.

It should be noted that the OpenPGP applet on this card can also be used with NFC on Android with the OpenKeychain app. Together with pass (or its Windows, Mac, or phone ports), this makes a nicely secure system for storing passwords.

Yubikey Neo: PKCS#11 with the PIV applet

There is also support for the PIV standard on the Yubikey Neo. This is supported by default on Linux (via OpenSC) and Windows and provides a PKCS#11-compabible store. It should, therefore, be compatible with ssh-agent, OpenVPN, Active Directory, and all the other OpenPGP card features described above. The only difference is that it uses storage separate from the OpenPGP applet.

You will need one of the Yubico PIV tools to configure the key for it; in Debian, the yubico-piv-tool from stretch does this.

Here are some instructions on using the Yubikey Neo in PIV mode:

A final note: for security, it’s important to change the management key and PINs before deploying the PIV mode.

I couldn’t get this to work with Firefox, but it worked pretty much everywhere else.

Yubikey Neo: HOTP authentication

This is the default mode for your Yubikey; all other modes require enabling with ykpersonalize. In this mode, a 128-bit AES key stored on the Yubikey is used to generate one-time passwords (OTP). (This key was shared in advance with the authentication server.) A typical pattern would be for three prompts: username, password, and Yubikey HOTP. The user clicks in the Yubikey HOTP field, touches the Yubikey, and their one-time token is pasted in.

In the background, the service being authenticated to contacts an authentication server. This authentication server can be either your own (there are several open source implementations in Debian) or the free Yubicloud.

Either way, the server decrypts the encrypted part of the OTP, performs validity checks (making sure that the counter is larger than any counter it’s seen before, etc) and returns success or failure back to the service demanding authentication.

The first few characters of the posted auth contain the unencrypted key ID, and thus it can also be used to provide username if desired.

Yubico has provided quite a few integrations and libraries for this mode. A few highlights:

You can also find some details on the OTP mode. Here’s another writeup.

This mode is simple to implement, but it has a few downsides. One is that it is specific to the Yubico line of products, and thus has a vendor lock-in factor. Another is the dependence on the authentication server; this creates a potential single point of failure and can be undesireable in some circumtances.

Yubikey Neo: OATH and HOTP and TOTP

First, a quick note: OATH and OAuth are not the same. OATH is an authentication protocol, and OAuth is an authorization protocol. Now then…

Like Yubikey HOTP, OATH (both HOTP and TOTP) modes rely on a pre-shared key. (See details in the Yubico article.) Let’s talk about TOTP first. With TOTP, there is a pre-shared secret with each service. Each time you authenticate to that service, your TOTP generator combines the timestamp with the shared secret using a HMAC algorithm and produces a OTP that changes every 30 seconds. Google Authenticator is a common example of this protocol, and this is a drop-in replacement for it. Gandi has a nice description of it that includes links to software-only solutions on various platforms as well.

With the Yubikey, the shared secrets are stored on the card and processed within it. You cannot extract the shared secret from the Yubikey. Of course, if someone obtains physical access to your Yubikey they could use the shared secret stored on it, but there is no way they can steal the shared secret via software, even by compromising your PC or phone.

Since the Yubikey does not have a built-in clock, TOTP operations cannot be completed solely on the card. You can use a PC-based app or the Android application (Play store link) with NFC to store secrets on the device and generate your TOTP codes. Command-line users can also use the yubikey-totp tool in the python-yubico package.

OATH can also use HOTP. With HOTP, an authentication counter is used instead of a clock. This means that HOTP passwords can be generated entirely within the Yubikey. You can use ykpersonalize to configure either slot 1 or 2 for this mode, but one downside is that it can really only be used with one service per slot.

OATH support is all over the place; for instance, there’s libpam-oath from the OATH toolkit for Linux platforms. (Some more instructions on this exist.)

Note: There is another tool from Yubico (not in Debian) that can apparently store multiple TOTP and HOTP codes in the Yubikey, although ykpersonalize and other documentation cannot. It is therefore unclear to me if multiple HOTP codes are supported, and how..

Yubikey Neo: Challenge-Response Mode

This can be useful for doing offline authentication, and is similar to OATH-HOTP in a sense. There is a shared secret to start with, and the service trying to authenticate sends a challenge to the token, which must supply an appropriate response. This makes it only suitable for local authentication, but means it can be done fairly automatically and optionally does not even require a button press.

To muddy the waters a bit, it supports both “Yubikey OTP” and HMAC-SHA1 challenge-response modes. I do not really know the difference. However, it is worth noting that libpam-yubico works with HMAC-SHA1 mode. This makes it suitable, for instance, for logon passwords.

Yubikey Neo: U2F

U2F is a new protocol for web-based apps. Yubico has some information, but since it is only supported in Chrome, it is not of interest to me right now.

Yubikey Neo: Further resources

Yubico has a lot of documentation, and in particular a technical manual that is actually fairly detailed.

Closing comments

Do not think a hardware security token is a panacea. It is best used as part of a multi-factor authentication system; you don’t want a lost token itself to lead to a breach, just as you don’t want a compromised password due to a keylogger to lead to a breach.

These things won’t prevent someone that has compromised your PC from abusing your existing ssh session (or even from establishing new ssh sessions from your PC, once you’ve unlocked the token with the passphrase). What it will do is prevent them from stealing your ssh private key and using it on a different PC. It won’t prevent someone from obtaining a copy of things you decrypt on a PC using the Yubikey, but it will prevent them from decrypting other things that used that private key. Hopefully that makes sense.

One also has to consider the security of the hardware. On that point, I am pretty well satisfied with the Yubikey; large parts of it are open source, and they have put a lot of effort into hardening the hardware. It seems pretty much impervious to non-government actors, which is about the best guarantee a person can get about anything these days.

I hope this guide has been helpful.

Categories: Elsewhere

Eduard Sanou: Reproducible builds on Debian for GSoC 2015, 1st update

Planet Debian - Thu, 06/08/2015 - 20:12

This is the second blog post I’m writing about my experiences contributing to Debian for Google Summer of Code 2015 (check my first post)

Status update First month

It’s been two months and a few days since the GSoC started. During the first month I worked on fixing specific packages, mainly concerning issues with timestamps, which is a very common source of unreproducibility. In many cases, during the build process files are compressed into gzip or zip archives, which store the creation time of files in the metadata. This can lead to unreproducible results when there is timezone variation between builds (easily fixed setting the timezone to UTC before the compression happens). In some cases the compressed files are generated during the build, and thus add build times in the metadata of compressed files (in this case the creation date of the files needs to be normalized somehow).

As explained in my application, I finished exams on the end of June, that’s why I chose to work on small fixes first, so that I could make the most out of my time between studying and finishing university projects and reports. I’m happy with my first month, as I have worked as originally planned. Actually, my estimation of the number of bugs I could submit every week was surpassed in reality!

Second month

Once the university was over, I started dedicating myself fully to the project. This allowed me to start working on toolchain fixes, following my original plan on working with timestamp related issues.

In particular I have been working a lot in implementing a proposal for deterministic timestamps that appeared in the reproducible builds project. The idea is to define an environment variable called SOURCE_DATE_EPOCH that contains a known timestamp in Unix epoch format. With this variable exported, tools that would embed the localtime in their generated or processed files, can use this externally supplied date. This would happen only if SOURCE_DATE_EPOCH is exported, so the behaviour of the tool wouldn’t change if the variable is not set.

The first package I patched to implement this behaviour was gcc. The reason behind this is that there are about 420 unreproducible packages due to using the __DATE__, __TIME__ and __TIMESTAMP__ C macros. My patch changes the behavior of the macros __DATE__ and __TIME__ if SOURCE_DATE_EPOCH is exported. I submitted this patch to the gcc-patches list. Even though there was some interesting discussions in the list, the patch has not been accepted yet. Seeing how the reproducible builds idea is gaining momentum and becoming widespread, I’m positive that at some point the gcc people will be more receptive for such patch.

The second work with SOURCE_DATE_EPOCH was in debhelper; I patched this building tool to export the variable with the latest debian/changelog entry timestamp. With this patch, all the tools that run under dh will be able to use it to embed deterministic timestamps. Unfortunately some parts of the build process of some packages don’t happen under debhelper, so the variable needs to be exported in a different way.

Having submitted the debhelper patch allowed many packages to become reproducible after the tools that embedded timestamps were patched to honour SOURCE_DATE_EPOCH. As of today, the toolchain packages I have patched to do that are: gcc, libxslt, gettext, ghostscript and qt4-x11 (qhelpgenerator).

I have also continued working on fixing individual packages affected by timestamps, random orderings (such as the ones from listing hash keys) and locale depending orderings; I have tagged packages in our infrastructure to note what kind of issue makes them unreproducible; I have updated some parts of the Reproducible Builds Wiki.

Impressions about reproducible builds

The work I did during the first month felt a bit tedious sometimes: it didn’t require much creativity or thinking as most of the fixes where quite mechanical, following a recipe. After I became free from university duties, I started looking into less trivial issues, which require more deep investigation and feel more rewarding once they are fixed. I also worked on toolchain fixes, which need more work and need more care. Fixing toolchain packages feels particularly rewarding because they can cause many packages to become reproducible at once.

There is a very active community in the reproducible builds project! It’s great to see so many people contributing to this project spending so many hours and putting so much effort. I’ve felt very welcome from the beginning and I have gotten kind replies and helpful answers to all the questions and doubts I’ve had, both from my mentor and from the other people in the project.

I want to add that I’m still amazed by the awesome infrastructure set up for the reproducible builds project. The team is using a Jenkins machine to continuously build packages to check for reproducibility, with irc notifications for changes, and also with a really useful web interface to list all the packages organized by issues that allows exploring them individually with all the available information. Also not only the infrastructure is used to build Debian amd64 packages, but also FreeBSD, NetBSD, OpenWRT, coreboot and lately Debian armhf with the help of a few new arm nodes.

Impressions about working on a free software project

This was my first time working on a community driven free software project and I’ve learned so many things.

Something I learned during the first days, which is even more present in such wide project like the reproducible builds, is that contributing is not just writing patches for the features you want; you also need to convince the maintainer of the package to accept the patch! I was a bit surprised at the beginning because even if this is a Debian project, that aims to make changes to the whole distribution, decisions are not absolute for the whole Debian project. After taking decisions within the reproducible builds teams on how to approach things, we need to convince the rest of the Debian developers (mainly maintainers) to follow them, as they are allowed to disagree. So it is required to work together for solutions that makes everyone comfortable, usually with discussions on mailing lists or irc.

There is another fact that I wasn’t expecting before getting involved in this project. The kind of teamwork I have done previously involves having a leader who decides how stuff is done, who takes decisions when needed and oversees the whole project. There seems to be a different philosophy in Debian. Instead of having a leader, everyone tries their best, trying to convince the others that their solution is good, often by showing an implementation of the solution and providing proof that it works, rather than trying to get the solution accepted before coding it. Also, solutions and ideas are valued for their quality rather than by the position of the person submitting them, and there is no hierarchy within the group: all comments and advices are taken equally, valuing their usefulness regardless of who gives them.

Usually there are no votes when deciding things. Members try their best on their approaches, trying to convince others as best as they can. And even if someone disagrees they may end up accepting the solution if they don’t manage to convince the original proposer of doing things differently. The idea is to spend more time working and coding than arguing and deciding on the way to do things. So far I’ve seen this approach to be very efficient :)

I’ve been told by my mentor that for difficult cases there exist a Debian committee that helps mediate on disagreements, but that is only used as a last option, probably when the discussion gets heated up.

Personal experience

Overall I’m very happy to finally having set my foot in the free software community, where I’m able to contribute to the kind of software I have been using for years. The sense of community in Debian is really big, and everyone is invited to participate.

I think that Google is doing an awesome job with the Google Summer of Code, not only because it gives a lot to free software but because it helps students to join the free software world as contributors (which is something that may be difficult to get into when you don’t know how to begin, as it happened to myself for some time). I plan to continue contributing to the free software world, and I’d encourage anyone to find projects to get involved and contribute as well!

Happy hacking to everyone!

Categories: Elsewhere

OSTraining: The Drupal Ecosystem is Consolidating Rapidly (Again)

Planet Drupal - Thu, 06/08/2015 - 19:57

Back in 2012, there was a wave of mergers and acquisitions amongst Drupal agencies.

The biggest merger involved four different companies forming Wunderkraut. However, there was also the Phase2 and Treehouse partnership, plus the merger of DrupalConnect and NorthStudio in Canada. And, although they were smaller deals, Acquia also picked up Cyrve, Growing Venture Solutions, Drupal Scout and Mollom around the same time.

In 2012, acquisitions became so common that they became a community joke.

Fast forward to 2015, and we're seeing a second round of mergers and acquisitions:

Categories: Elsewhere

DrupalCon News: Come for the Con, Stay for the Sprints

Planet Drupal - Thu, 06/08/2015 - 19:14

Each DrupalCon is filled with inspiration, networking, and fun — and while cutting-edge sessions are plentiful, sprints are the heart of the conference. A sprint is a get-together for focused work on a project, and a fantastic opportunity to make contributions to Drupal.

Even if you are not giving a session or leading a BoF ("birds of a feather" event) at DrupalCon, there is plenty of opportunity to give back and make a difference when you attend the sprints. There will be extended sprints taking place between 19-27 September, but the main sprint day is Friday 25 September.

Categories: Elsewhere

Mpumelelo Msimanga: Conditional Formatting of Data In Drupal Views

Planet Drupal - Thu, 06/08/2015 - 15:00
Conditional Formatting of Data In Drupal Views

A few years ago then Google CEO Eric Schmidt was quoted as saying, “There was 5 exabytes of information created between the dawn of civilization through 2003, but that much information is now created every 2 days, and the pace is increasing”. The great volumes of data available for analysis have necessitated the development of tools and methodologies to help users derive information from the vast volumes of data.

Categories: Elsewhere

Christoph Egger: unbreaking tt-rss

Planet Debian - Thu, 06/08/2015 - 13:16

TinyTiny-RSS has some nice failure modes. And upstream support forums aren't really helpfull so when you search for your current problem, chances are that there is one mention of it on the web, in the forum, and the only thing happening there is people making fun of the reporter.

Anyway. This installation has seen lots of error messages from the updater in the last several months:

Warning: Fatal error, unknown preferences key: ALLOW_DUPLICATE_POSTS (owner: 3) in /srv/www/tt-rss.faui2k9.de/root/classes/db/prefs.php on line 108 Warning: Fatal error, unknown preferences key: ALLOW_DUPLICATE_POSTS (owner: 3) in /srv/www/tt-rss.faui2k9.de/root/classes/db/prefs.php on line 108 Warning: Fatal error, unknown preferences key: ALLOW_DUPLICATE_POSTS (owner: 3) in /srv/www/tt-rss.faui2k9.de/root/classes/db/prefs.php on line 108 Warning: Fatal error, unknown preferences key: ALLOW_DUPLICATE_POSTS (owner: 3) in /srv/www/tt-rss.faui2k9.de/root/classes/db/prefs.php on line 108 And, more recently, the android app stopped working with ERROR:JSON Parse failed.. Turns out both things are related.

First thing I noticed was changing preferences in the web panel stopped working until you use the reset to Defaults option and then changed settings. Plugging wireshark in between showed what was going on (Note: API was displayed as enabled in Preferences/Preferences):

HTTP/1.1 200 OK Server: nginx/1.8.0 Date: Thu, 06 Aug 2015 11:00:31 GMT Content-Type: text/json Transfer-Encoding: chunked Connection: keep-alive X-Powered-By: PHP/5.4.43 Content-Language: auto Set-Cookie: [...] Expires: Thu, 19 Nov 1981 08:52:00 GMT Cache-Control: no-store, no-cache, must-revalidate, post-check=0, pre-check=0 Pragma: no-cache Api-Content-Length: 234 ea
Warning: Fatal error, unknown preferences key: ENABLE_API_ACCESS (owner: 2) in /srv/www/tt-rss.faui2k9.de/root/classes/db/prefs.php on line 108
{"seq":0,"status":1,"content":{"error":"API_DISABLED"}} 0

Solution for fixing the Android app (and the logspam on the way as well) seems to be to reset the preferences and then configure tt-rss again (In the webapp, not in the android thing!). Also silences tt-rss update_daemon as well, yay! One last thing: someone out there who wants to explain to me how to fix

Fatal error: Query INSERT INTO ttrss_enclosures (content_url, content_type, title, duration, post_id, width, height) VALUES ('https://2.gravatar.com/avatar/e6d6ceb7764252af8da058e30cd8cb5f?s=96&d=identicon&r=G', '', '', '', '0', 0, 0) failed: ERROR: insert or update on table "ttrss_enclosures" violates foreign key constraint "ttrss_enclosures_post_id_fkey" DETAIL: Key (post_id)=(0) is not present in table "ttrss_entries". in /srv/www/tt-rss.faui2k9.de/root/classes/db/pgsql.php on line 46

Categories: Elsewhere

Paul Johnson: Lightning talks at your Drupal Meetup

Planet Drupal - Thu, 06/08/2015 - 11:16

Last night NWUG held an evening of Lightning Talks at Madlab in Manchester. These are small talks that last up to 10 minutes long or can be run in the Pecha Kucha format with 20 slides in 6 minutes (1 slide every 20 seconds).

For me lightning talks are brilliant. In one evening you get to hear a wide palette of topics, discover from a high level subjects you might like to explore afterwards. They gave the opportunity for first time speakers to presenting in front of a friendly audience. It showed what a diverse and healthy Drupal community we in Manchester. Attracting 25 people, it was also the best turnout we've had in a while.

There were so many stepping up to speak that we ran out of time. There will be more lightning talks at NWDUG in November, so perhaps if you were inspired or curious and would like to join us, come along next time and see what it's all about!

Categories: Elsewhere


Subscribe to jfhovinne aggregator - Elsewhere