Elsewhere

The blank canvas in Photoshop is where many well-intentioned Web redesign projects go to die.

If you want a beautiful magazine ad, then find a talented graphic designer and have them draw up something pretty for you. If you want a rational redesign, you need to start somewhere else.

As defined in the first part of this series, a rational redesign delivers true value to an organization, instead of simply consuming time and money for window dressing.

A little fun this morning. I had a great talk with Jacob Redding yesterday as part of his academic work on open source communities. We talked a bit about the early times of Drupal, including the heady days in Antwerp in 2005 when the ideas for what was to become CCK and then fields-in-core were being explored, debated, and scrawled on whiteboards. I thought I'd share this creation by Steven Wittens. He whipped it up late one night after a long day of architectural discussions. It's been on the wall of my office for years.

John VanDyk, Jonathan Chaffer, and the new fragrance of CCK.

Topic: DrupalDrupal History

A few years ago I was doing research comparing how large software distributions handled shared object libraries, and studied Debian, FreeBSD, and Ubuntu. Extracting data about Debian packages was easy thanks to Peter Palfrader’s snapshot.debian.org service, which provides a machine-usable interface to Debian’s package history. FreeBSD’s data is equally accessible, albeit in a less pleasant format: their ports tree was stored in CVS until July 2012. One could easily rsync a copy of the ports tree’s CVS repository to a local machine to analyze the data. This left you with a local tree full of ,v files, each corresponding to the history of a given file with at that location. I needed to extract all kinds of data from a tree full of these files, such as what revisions contained lines matching a regex, when these revisions were checked in, any tags associated with it, etc. To make things easier, it also helped to know the line numbers of the matching lines. Hence the birth of rcsgrep.

rcsgrep is a Python script that makes use of Paul McGuire’s fabulous pyparsing library. It allows you to search a RCS file (the ,v file format used by RCS and CVS to store revision history) using a Python regex, and the output format is customizable to allow printing only certain kinds of information, such as the revision number, the line number, the matching line, the line’s author, the date it appeared, any tags associated with the line, and (useful when running over a large number of files) the file name. To make machine parsing (using AWK of course) easier, you can also specify the column separator.

For example, I entered the lines “The quick brown”, “fox jumped over”, “the lazy dog. Woof!” into the file abc, checking in the changes after each line. The invocation ./rcsgrep -s ' ' -f rlLda '.*' abc,v, with spaces for column separation, and format options r is for revision, l for line number, L for line contents, d for date, and a for author, outputs:

1.3 1 The quick brown 2013.02.20.14.24.09 ryan 1.3 2 jumped over the 2013.02.20.14.24.09 ryan 1.3 3 lazy dog. Woof! 2013.02.20.14.24.09 ryan 1.2 1 The quick brown 2013.02.20.14.23.48 ryan 1.2 2 jumped over the 2013.02.20.14.23.48 ryan 1.1 1 The quick brown 2013.02.20.14.23.25 ryan

I’m particularly proud about my grep() function in rcsfile.py, which goes through each revision, tracking additions and deletions, but only keeping the lines matching the regex in memory. In any case, rcsgrep is licensed under the ISC license and can be found on github.

Addendum: I learned after the fact that O’Reilly’s “UNIX Power Tools” offers something similar by the same name, except that it is runs several processes, such as co, grep and sed, as opposed to a single Python script.

You did not read that wrong! Yes, Drupal 8 features are frozen, and the massive Drupal 8 Multilingual Initiative is not there to let you even translate a node title or the site name. We made massive amounts of progress with heroic efforts from key contributors, but we are not nearly close to be done yet. Yes, your help is needed! The way the Drupal core release cycle is set up, many things that you might consider features are not classified as such (verified with core maintainers) in the process. Feature freeze means all base features should be in core, however, many things that need to be integrated with these new features are not done yet. Otherwise what would we do for months on still, right? So let's look at the two use cases with this in mind. Start with the bad news!

Translate site name

The biggest feature that will still not be native in Drupal 8 is translation of user provided configuration. We still have lot to work on translating shipped configuration. We've worked hard for months to get a configuration schema system accepted so we can have a programatic understanding of the translatable pieces of configuration. We started work on that in 2012 June at Drupal Dev Days, and it took several forked issues, alternate proposals, personal, IRC and phone meetings to get a universal understanding that a configuration schema system is needed in the first place, with the actual implementation not too far from the original proposals, that landed in core. We would have loved to achieve this milestone sooner but that did not work out.

So the sad news is you will not be able to translate your site name or custom created Views with Drupal 8 core only. Drupal 8 core should be capable to translate shipped configuration though, that is email texts, image styles, default content types, default fields, etc. shipped with Drupal core. The text for these should eventually end up on http://localize.drupal.org for translation. You'll need to use a contributed module to translate user entered configuration though (such as user created Views and your site name). Good news is that the underlying schema system and the configuration override system in core (as well as the soon to be committed configuration context system) supports this use case too, so the missing piece is a user interface to be generated for configuration translation. A very early version of that user interface can be seen in http://drupal.org/sandbox/reyero/1635230.

So as of right now you cannot even translate the shipped email texts, but that is a feature targeted for Drupal 8 core (in combination with http://localize.drupal.org as usual for software translation). Translating configuration created on the user interface and not shipped with modules/themes/profiles will need a contributed module.

Translate node titles

All right, this is a flat out regression, right? Right! Drupal 8 trades node-copy translation that is native to Drupal 7 (using the Content translation module in Drupal 7), where translating a node creates a new copy of the node. In Drupal 8, the module named the same on the user interface (Content translation) works totally differently. It does not create new copies but stores translations in fields under the entity. This has several advantages. For one, it lets us support all kinds of entity types, for example, fields on users, taxonomy terms, comments, etc. We also provide a neat cross-entity configuration screen to set up all your entity types for language.

Yet, node titles remain non-translatable. The basic reason for that is that node titles are not "fields" on nodes, they are good old properties. Properties do not support multilingual storage natively. We introduced a multilingual property system earlier (for the entity_test entity type), and the integration to other entity types did not yet happen. Why? Well, the more sweeping next generation Entity API conversions are still widely underway. Comment entities are done (with performance regressions that are being worked out) and we are hard at work on nodes. There are still files, users, taxonomy terms, etc. ahead.

And then the multilinugal property conversions can happen on top of the new entity system much easier since that system has built-in support for them. That is still not a trivial task and we'll definitely need all hands possible to achieve.

(Side note: once node properties become multilingual, we also need to provide a migration path in core for the legacy content translation module and remove that module from core for good.)

How can I help?

The configuration schema system is easy! We have documentation at http://drupal.org/node/1905070 and a visual configuration browser that applies the schema at http://drupal.org/sandbox/reyero/1635230. There is a huge set of issues at http://www.drupal8multilingual.org/issues/schema to fill in the blanks of schemas where not yet provided or incorrectly written. Tips to help are in the meta issue at http://drupal.org/node/1910624#comment-7088154

The entity API conversions are more involved work. You can participate in the conversion issues at http://drupal.org/node/1818580 - however the process so far has been to attack one type at a time, and nodes are in focus righ now (http://drupal.org/node/1818556). Performance expertise, reroll experience and all kinds of other skills are needed here.

So how can you call Drupal 8 feature frozen? And when will it ever be ready?

Reality is features from the point of the development process are not necessarily the same as user's perceptions. While we wanted to get in solutions sooner (eg. targeted configuration schema and context for Dec 1st 2012), their declared belonging to a later development phase does not help to get reviewers and push them harder for inclusion. We are doing our best to not let this affect the eventual readiness of Drupal 8, but in the state we are in, we are not looking at an integration phase where we can sit back and relax.

The current planned deadline for the integration phase (when all entity properties should be multilingual and all configuration should have schemas) is July 1st 2013. There are three big sprints until then that we plan to use to boost our standing (please come and sign up for these):

• Drupal sprint weekend, March 9th and 10th all around the globe
• Sprints on the weekends both before and after DrupalCon Portland, May 18-19th and 24-26th
• Sprints all week before Drupal Dev Days Dublin, June 24-27th

Sitting around and waiting for later opportunities does not help, so jumping on these issues would be especially helpful. If you have no better idea, start reviewing the configuration schema issues today!

A huge thanks once again for all contributors

I'd like to reiterate once again that all the people on the multilingual intiative worked exceptionally hard (in some cases all the way to burning out) to make Drupal 8 as multilingual as possible for you all. The gaps in implementation are not at all due to people doing poor work, they are due to all the uncertainties involved in working on an Open Source project with set goals but heavy interdependencies and lack of resource commitments possible. As well as all the bliss and baggage of making large scale decisions in an open issue queue format.

See http://www.drupal8multilingual.org/team for more information on our team.

For once, I think we are going a funny DPL compaign thanks to Francesca who brought to us a DPL Game :) So here is my little list of fantastic four:

• Gregor Herrmann
• Lucas Nussbaum
• Russ Allbery
• Enrico Zini (sorry, you've been on my list since the beginning).

(in random order)

Each one of them has super powers (lot of energy, wisdom, impartiality, ability to sort out complex situations) and can do a great DPL!

My fantastic four, please consider running for DPL!

Time from time, while doing your custom Drupal code, you may want to load only one or several specific fields from a defined set of entities. So actually you have three approaches to this:

1. Query for entity/node set and load whole entities to get desired fields data. Works, but not a performant solution.

2. Make a direct sql query and get desired fields out of your database. Works too, is the fasted in terms of performance solution, but not too flexible and portable.

3. Leverage EntityFieldQuery() and field_attach_load(). This approach is not as fast as the second, but way more faster than loading whole nodes, it is flexible and uses field caching mechanism. If you'll decide to change your database backend later in the future, let's say to MongoDB, you'll be able to switch without changing a line in your code, neat!

Read on about Loading Only One Field From An Entity or Node in Drupal 7

Given that they really go together, this week we will look at two different hooks: hook_block_configure() and hook_block_save(). Their purpose is to programatically add extra configuration options to the block configuration form: you know, the one with the region and visibility settings.

I just released umockdev 0.2.

The big new feature of this release is support for evdev ioctls. I. e. you can now record what e. g. X.org is doing to touchpads, touch screens, etc.:

$ umockdev-record /dev/input/event15 > /tmp/touchpad.umockdev # umockdev-record -i /tmp/touchpad.ioctl /dev/input/event15 -- Xorg -logfile /dev/null

and load that back into a testbed with X.org using the dummy driver:

cat <<EOF > xorg-dummy.conf Section "Device" Identifier "test" Driver "dummy" EndSection EOF $ umockdev-run -l /tmp/touchpad.umockdev -i /dev/input/event15=/tmp/touchpad.ioctl -- Xorg -config xorg-dummy.conf -logfile /tmp/X.log :5

Then e. g. DISPLAY=:5 xinput will recognize the simulated device. Note that Xvfb won’t work as that does not use udev for device discovery, but only adds the XTest virtual devices and nothing else, so you need to use the real X.org with the dummy driver to run this as a normal user.

This enables easier debugging of new kinds of input devices, as well as writing tests for handling multiple touchscreens/monitors, integration tests of Wacom devices, and so on.

This release now also works with older automakes and Vala 0.16, so that you can use this from Ubuntu 12.04 LTS. The daily PPA now also has packages for that.

Attention: This version does not work any more with recorded ioctl files from version 0.1.

More detailled list of changes:

• umockdev-run: Fix running of child program to keep stdin.
• preload: Fix resolution of “/dev” and “/sys”
• ioctl_tree: Fix endless loop when the first encountered ioctl was unknown
• preload: Support opening a /dev node multiple times for ioctl emulation (issue #3)
• Fix parallel build (issue #2)
• Support xz compressed ioctl files in umockdev_testbed_load_ioctl().
• Add example umockdev and ioctl files for a gphoto camera and an MTP capable mobile phone.
• Fix building with automake 1.11.3 and Vala 0.16.
• Generalize ioctl recording and emulation for ioctls with simple structs, i. e. no pointer fields. This makes it much easier to add more ioctls in the future.
• Store return values of ioctls in records, as they are not always 0 (like EVIOCGBIT)
• Add support for ioctl ranges (like EVIOCGABS) and ioctls with variable length (like EVIOCGBIT).
• Add all reading evdev ioctls, for recording and mocking input devices like touch pads, touch screens, or keyboards. (issue #1)

Want to switch back-and-forth between directories in your shell script?
The bashism of this week can be of some help, but for most needs, the cd utility is more than enough.

pushd, popd, and the extra built-in dirs are bashisms that allow one to create and manipulate a stack of directory entries. For a simple, temporary, switch of directories the following code is portable as far as POSIX:2001 is concerned:

cd /some/directory
touch some files
unlink others
# etc
cd - >/dev/null
# We are now back at where we were before the first 'cd'

Which is equivalent to the following, also portable, code:

cd /some/directory
touch some files
unlink others
# etc
cd "$OLDPWD"
# We are now back at where we were before the first 'cd'

Multiple switches can also be implemented portably without storing the name of the directories in variables at the expense of using subshells (and their side-effects).

However, if you think you can solve your problem more conveniently by using "pushd" and "popd" don't forget to document the need of those built-ins and to adjust the shebang of your script to that of a shell that implements them, such as bash.

On Planet Debian, Vincent Bernat wrote:

The drawback of this approach is that if you rebuild configure from the released tarball, you don’t have the git tree and the version will be a date. Just don’t do that.

Excuse me‽

This is totally inacceptable. Regenerating files like aclocal.m4 and Makefile.in (for automake), configure (for autoconf), and the likes is one of the absolute duties of a software package. Things will break sooner or later if people do not do that. Additionally, generated files must be remakable from the distfile, so do not break this!

May I suggest, constructively, an alternative? (People – rightfully, I must admit – complain I’m “just” ranting too much.)
When making a release from git, write the “git describe” output into a file. Then, use that file instead of trying to run the git executable if .git/. is not a directory (“test -d .git/.”). Do not call git, because, in packages, it’s either not installed or/and also undesired.

Couldn’t comment on your blog, but felt strongly enough about this I took the effort of writing a full post of my own.

(But thanks for the book recommendation.)

A few weeks ago, a new version of lldpd, a 802.1AB (aka LLDP) implementation for various Unices, has been released.

LLDP is an industry standard protocol designed to supplant proprietary Link-Layer protocols such as EDP or CDP. The goal of LLDP is to provide an inter-vendor compatible mechanism to deliver Link-Layer notifications to adjacent network devices.

In short, LLDP allows you to know exactly on which port is a server (and reciprocally). To illustrate its use, I have made a xkcd-like strip:

If you would like more information about lldpd, please have a look at its new dedicated website. This blog post is an insight of various technical changes that have affected lldpd since its latest major release one year ago. Lots of C stuff ahead!

• Version & changelog
  • Automated version
  • Automated changelog
• Core
  • C99
  • Logging
  • libevent
• Client
  • Serialization
  • Library
  • CLI
    • Parsing & completion
    • Readline
• OS specific support
  • Netlink on Linux
  • BSD support
  • OS X support
  • Upstart and systemd support
  • OS include files

Version & changelog Automated version

In configure.ac, I was previously using a static version number that I had to increase when releasing:

AC_INIT([lldpd], [0.5.7], [bernat@luffy.cx])

Since the information is present in the git tree, this seems a bit redundant (and easy to forget). Taking the version from the git tree is easy:

AC_INIT([lldpd], [m4_esyscmd_s([git describe --tags --always --match [0-9]* 2> /dev/null || date +%F])], [bernat@luffy.cx])

If the head of the git tree is tagged, you get the exact tag (0.7.1 for example). If it is not, you get the nearest one, the number of commits since it and part of the current hash (0.7.1-29-g2909519 for example).

The drawback of this approach is that if you rebuild configure from the released tarball, you don’t have the git tree and the version will be a date. Just don’t do that.

Automated changelog

Generating the changelog from git is a common practice. I had some difficulties to make it right. Here is my attempt (I am using automake):

dist_doc_DATA = README.md NEWS ChangeLog .PHONY: $(distdir)/ChangeLog dist-hook: $(distdir)/ChangeLog $(distdir)/ChangeLog: $(AM_V_GEN)if test -d $(top_srcdir)/.git; then \ prev=$$(git describe --tags --always --match [0-9]* 2> /dev/null) ; \ for tag in $$(git tag | grep -E '^[0-9]+(\.[0-9]+){1,}$$' | sort -rn); do \ if [ x"$$prev" = x ]; then prev=$$tag ; fi ; \ if [ x"$$prev" = x"$$tag" ]; then continue; fi ; \ echo "$$prev [$$(git log $$prev -1 --pretty=format:'%ai')]:" ; \ echo "" ; \ git log --pretty=' - [%h] %s (%an)' $$tag..$$prev ; \ echo "" ; \ prev=$$tag ; \ done > $@ ; \ else \ touch $@ ; \ fi ChangeLog: touch $@

Changelog entries are grouped by version. Since it is a bit verbose, I still maintain a NEWS file with important changes.

Core C99

I have recently read 21st Century C which has some good bits and also handles the ecosystem around C. I have definitively adopted designated initializers in my coding style. Being a GCC extension since a long time, this is not a major compatibility problem.

Without designated initializers:

struct netlink_req req; struct iovec iov; struct sockaddr_nl peer; struct msghdr rtnl_msg; memset(&req, 0, sizeof(req)); memset(&iov, 0, sizeof(iov)); memset(&peer, 0, sizeof(peer)); memset(&rtnl_msg, 0, sizeof(rtnl_msg)); req.hdr.nlmsg_len = NLMSG_LENGTH(sizeof(struct rtgenmsg)); req.hdr.nlmsg_type = RTM_GETLINK; req.hdr.nlmsg_flags = NLM_F_REQUEST | NLM_F_DUMP; req.hdr.nlmsg_seq = 1; req.hdr.nlmsg_pid = getpid(); req.gen.rtgen_family = AF_PACKET; iov.iov_base = &req; iov.iov_len = req.hdr.nlmsg_len; peer.nl_family = AF_NETLINK; rtnl_msg.msg_iov = &iov; rtnl_msg.msg_iovlen = 1; rtnl_msg.msg_name = &peer; rtnl_msg.msg_namelen = sizeof(struct sockaddr_nl);

With designated initializers:

struct netlink_req req = { .hdr = { .nlmsg_len = NLMSG_LENGTH(sizeof(struct rtgenmsg)), .nlmsg_type = RTM_GETLINK, .nlmsg_flags = NLM_F_REQUEST | NLM_F_DUMP, .nlmsg_seq = 1, .nlmsg_pid = getpid() }, .gen = { .rtgen_family = AF_PACKET } }; struct iovec iov = { .iov_base = &req, .iov_len = req.hdr.nlmsg_len }; struct sockaddr_nl peer = { .nl_family = AF_NETLINK }; struct msghdr rtnl_msg = { .msg_iov = &iov, .msg_iovlen = 1, .msg_name = &peer, .msg_namelen = sizeof(struct sockaddr_nl) }; Logging

Logging in lldpd was not extensive. Usually, when receiving a bug report, I asked the reporter to add some additional printf() calls to determine where the problem was. This was clearly suboptimal. Therefore, I have added many log_debug() calls with the ability to filter out some of them. For example, to debug interface discovery, one can run lldpd with lldpd -ddd -D interface.

Moreover, I have added colors when logging to a terminal. This may seem pointless but it is now far easier to spot warning messages from debug ones.

libevent

In lldpd 0.5.7, I was using my own select()-based event loop. It worked but I didn’t want to grow a full-featured event loop inside lldpd. Therefore, I switched to libevent.

The minimal required version of libevent is 2.0.5. A convenient way to check the changes in API is to use Upstream Tracker, a website tracking API and ABI changes for various libraries. This version of libevent is not available in many stable distributions. For example, Debian Squeeze or Ubuntu Lucid only have 1.4.13. I am also trying to keep compatibility with very old distributions, like RHEL 2, which does not have a packaged libevent at all.

For some users, it may be a burden to compile additional libraries. Therefore, I have included libevent source code in lldpd source tree (as a git submodule) and I am only using it if no suitable system libevent is available.

Have a look at m4/libevent.m4 and src/daemon/Makefile.am to see how this is done.

Client Serialization

lldpctl is a client querying lldpd to display discovered neighbors. The communication is done through an Unix socket. Each structure to be serialized over this socket should be described with a string. For example:

#define STRUCT_LLDPD_DOT3_MACPHY "(bbww)" struct lldpd_dot3_macphy { u_int8_t autoneg_support; u_int8_t autoneg_enabled; u_int16_t autoneg_advertised; u_int16_t mau_type; };

I did not want to use stuff like Protocol Buffers because I didn’t want to copy the existing structures to other structures before serialization (and the other way after deserialization).

However, the serializer in lldpd did not allow to handle reference to other structures, lists or circular references. I have written another one which works by annotating a structure with some macros:

struct lldpd_chassis { TAILQ_ENTRY(lldpd_chassis) c_entries; u_int16_t c_index; u_int8_t c_protocol; u_int8_t c_id_subtype; char *c_id; int c_id_len; char *c_name; char *c_descr; u_int16_t c_cap_available; u_int16_t c_cap_enabled; u_int16_t c_ttl; TAILQ_HEAD(, lldpd_mgmt) c_mgmt; }; MARSHAL_BEGIN(lldpd_chassis) MARSHAL_TQE (lldpd_chassis, c_entries) MARSHAL_FSTR (lldpd_chassis, c_id, c_id_len) MARSHAL_STR (lldpd_chassis, c_name) MARSHAL_STR (lldpd_chassis, c_descr) MARSHAL_SUBTQ(lldpd_chassis, lldpd_mgmt, c_mgmt) MARSHAL_END;

Only pointers need to be annotated. The remaining of the structure can be serialized with just memcpy()1. I think there is still room for improvement. It should be possible to add annotations inside the structure and avoid some duplication. Or maybe, using a C parser? Or using the AST output from LLVM?

Library

In lldpd 0.5.7, there are two possible entry points to interact with the daemon:

1. Through SNMP support. Only information available in LLDP-MIB are exported. Therefore, implementation-specific values are not available. Moreover, SNMP support is currently read-only.
2. Through lldpctl. Thanks to a contribution from Andreas Hofmeister, the output can be requested to be formatted as an XML document.

Integration of lldpd into a network stack was therefore limited to one of those two channels. As an exemple, you can have a look at how Vyatta made the integration using the second solution.

To provide a more robust solution, I have added a shared library, liblldpctl, with a stable and well-defined API. lldpctl is now using it. I have followed those directions:

• Consistent naming (all exported symbols are prefixed by lldpctl_). No pollution of the global namespace.
• Consistent return codes (on errors, all functions returning pointers are returning NULL, all functions returning integers are returning -1).
• Reentrant and thread-safe. No global variables.
• One well-documented include file.
• Reduce the use of boilerplate code. Don’t segfault on NULL, accept integer input as string, provide easy iterators, …
• Asynchronous API for input/output. The library delegates reading and writing by calling user-provided functions. Those functions can yield their effects. In this case, the user has to callback the library when data is available for reading or writing. It is therefore possible to integrate the library with any existing event-loop. A thin synchronous layer is provided on top of this API.
• Opaque types with accessor functions.

Accessing bits of information is done through “atoms” which are opaque containers of type lldpctl_atom_t. From an atom, you can extract some properties as integers, strings, buffers or other atoms. The list of ports is an atom. A port in this list is also an atom. The list of VLAN present on this port is an atom, as well as each VLAN in this list. The VLAN name is a NULL-terminated string living in the scope of an atom. Accessing a property is done by a handful of functions, like lldpctl_atom_get_str(), using a specific key. For example, here is how to display the list of VLAN assuming you have one port as an atom:

vlans = lldpctl_atom_get(port, lldpctl_k_port_vlans); lldpctl_atom_foreach(vlans, vlan) { vid = lldpctl_atom_get_int(vlan, lldpctl_k_vlan_id)); name = lldpctl_atom_get_str(vlan, lldpctl_k_vlan_name)); if (vid && name) printf("VLAN %d: %s\n", vid, name); } lldpctl_atom_dec_ref(vlans);

Internally, an atom is typed and reference counted. The size of the API is greatly limited thanks to this concept. There are currently more than one hundred pieces of information that can be retrieved from lldpd.

Ultimately, the library will also enable the full configuration of lldpd. Currently, many aspects can only be configured through command-line flags. The use of the library does not replace lldpctl which will still be available and be the primary client of the library.

CLI

Having a configuration file was requested since a long time. I didn’t want to include a parser in lldpd: I am trying to keep it small. It was already possible to configure lldpd through lldpctl. Locations, network policies and power policies were the three items that could be configured this way. So, the next step was to enable lldpctl to read a configuration file, parse it and send the result to lldpd. As a bonus, why not provide a full CLI accepting the same statements with inline help and completion?

Parsing & completion

Because of completion, it is difficult to use a YACC generated parser. Instead, I define a tree where each node accepts a word. A node is defined with this function:

struct cmd_node *commands_new( struct cmd_node *, const char *, const char *, int(*validate)(struct cmd_env*, void *), int(*execute)(struct lldpctl_conn_t*, struct writer*, struct cmd_env*, void *), void *);

A node is defined by:

• its parent,
• an optional accepted static token,
• an help string,
• an optional validation function and
• an optional function to execute if the current token is accepted.

When walking the tree, we maintain an environment which is both a key-value store and a stack of positions in the tree. The validation function can check the environment to see if we are in the right context (we want to accept the keyword foo only once, for example). The execution function can add the current token as a value in the environment but it can also pop the current position in the tree to resume walk from a previous node.

As an example, see how nodes for configuration of a coordinate-based location are registered:

/* Our root node */ struct cmd_node *configure_medloc_coord = commands_new( configure_medlocation, "coordinate", "MED location coordinate configuration", NULL, NULL, NULL); /* The exit node. The validate function will check if we have both latitude and longitude. */ commands_new(configure_medloc_coord, NEWLINE, "Configure MED location coordinates", cmd_check_env, cmd_medlocation_coordinate, "latitude,longitude"); /* Store latitude. Once stored, we pop two positions to go back to the "root" node. The user can only enter latitude once. */ commands_new( commands_new( configure_medloc_coord, "latitude", "Specify latitude", cmd_check_no_env, NULL, "latitude"), NULL, "Latitude as xx.yyyyN or xx.yyyyS", NULL, cmd_store_env_value_and_pop2, "latitude"); /* Same thing for longitude */ commands_new( commands_new( configure_medloc_coord, "longitude", "Specify longitude", cmd_check_no_env, NULL, "longitude"), NULL, "Longitude as xx.yyyyE or xx.yyyyW", NULL, cmd_store_env_value_and_pop2, "longitude");

The definition of all commands is still a bit verbose but the system is simple enough yet powerful enough to cover all needed cases.

Readline

When faced with a CLI, we usually expect some perks like completion, history handling and help. The most used library to provide such features is the GNU Readline Library. Because this is a GPL library, I have first searched an alternative. There are several of them:

• NetBSD Editline library (libedit).
• Autotool port of the NetBSD Editline library (also libedit).
• Debian version of the Editline library (libeditline).
• linenoise, a small and minimal readline library.
• Many others.

From an API point of view, the first three libraries support the GNU Readline API. They also have a common native API. Moreover, this native API also handles tokenization. Therefore, I have developed the first version of the CLI with this API2.

Unfortunately, I noticed later this library is not very common in the Linux world and is not available in RHEL. Since I have used the native API, it was not possible to fallback to the GNU Readline library. So, let’s switch! Thanks to the appropriate macro from the Autoconf Archive (with small modifications), the compilation and linking differences between the libraries are taken care of.

Because GNU Readline library does not come with a tokenizer, I had to write one myself. The API is also badly documented and it is difficult to know which symbol is available in which version. I have limited myself to:

• readline(), addhistory(),
• rl_insert_text(),
• rl_forced_update_display(),
• rl_bind_key()
• rl_line_buffer and rl_point.

Unfortunately, the various libedit libraries have a noop for rl_bind_key(). Therefore, completion and online help is not available with them. I have noticed that most BSD come with GNU Readline library preinstalled, so it could be considered as a system library. Nonetheless, linking with libedit to avoid licensing issues is possible and help can be obtained by prefixing the command with help.

OS specific support Netlink on Linux

Previously, the list of interfaces was retrieved through getifaddrs(). lldpd is now using directly Netlink on Linux. This is not a big change since the GNU C Library already uses it to implement getifaddrs() and additional information, like VLAN, are still retrieved through ioctl() or sysfs. However, lldpd now gets notified when a change happens and update all interfaces in the next second.

Like many other projects, I have written my own Netlink implementation instead of using libnl, a nice collection of libraries providing everything you need to query the kernel through Netlink, including some advanced bits. Why?

1. The latest version of libnl is still young and its availability in major distributions is scarce. It is not available in Debian Squeeze but will be available in Debian Wheezy. Like libevent, I could circumvent this problem by shipping the library with lldpd and use it when there is not system alternative. But…

2. libnl is licensed under LGPL 2.1. This makes static linking difficult because the license is quite shaddy about static linking being derivative work or not. It is believed that it is authorized under the same provisions as in LGPL 3 which handles the case explicitely. This has been a problem with many projects. For example, OGRE has added an exception for static linking in version 1.6 and switched to MIT license in version 1.7.

I had a short discussion with Thomas Graf about this issue and he seems willing to add a similar exception. This may take some time, but once this is done, I will happily switch to libnl and retrieve more stuff from Netlink.

BSD support

Until version 0.7, lldpd was Linux-only. The rewrite to use Netlink was the occasion to abstract interfaces and to port to other OS. The first port was for Debian GNU/kFreeBSD, then for FreeBSD, OpenBSD and NetBSD. They all share the same source code:

• getifaddrs() to get the list of interfaces,
• bpf(4) to attach to an interface to receive and send packets,
• PF_ROUTE socket to be notified when a change happens.

Each BSD has its own ioctl() to retrieve VLAN, bridging and bonding bits but they are quite similar. The code was usually adapted from ifconfig.c.

The BSD ports have the same functionalities than the Linux port, except for NetBSD which lacks support for LLDP-MED inventory since I didn’t find a simple way to retrieve DMI related information.

They also offer greater security by filtering packets sent. Moreover, OpenBSD allows to lock the filters set on the socket:

/* Install write filter (optional) */ if (ioctl(fd, BIOCSETWF, (caddr_t)&fprog) < 0) { rc = errno; log_info("privsep", "unable to setup write BPF filter for %s", name); goto end; } /* Lock interface */ if (ioctl(fd, BIOCLOCK, (caddr_t)&enable) < 0) { rc = errno; log_info("privsep", "unable to lock BPF interface %s", name); goto end; }

This is a very nice feature. lldpd is using a privileged process to open the raw socket. The socket is then transmitted to an unprivileged process. Without this feature, the unprivileged process can remove the BPF filters. I have ported the ability to lock a socket filter program to Linux. However, I still have to add a write filter.

OS X support

Once FreeBSD was supported, supporting OS X seemed easy. I got sponsored by xcloud.me which provided a virtual Mac server. Making lldpd work with OS X took only two days, including a full hour to guess how to get Apple Xcode without providing a credit card.

To help people installing lldpd on OS X, I have also written a lldpd formula for Homebrew which seems to be the most popular package manager for OS X.

Upstart and systemd support

Many distributions propose upstart and systemd as a replacement or an alternative for the classic SysV init. Like most daemons, lldpd detaches itself from the terminal and run in the background, by forking twice, once it is ready (for lldpd, this just means we have setup the control socket). While both upstart and systemd can accommodate daemons that behave like this, it is recommended to not fork. How to advertise readiness in this case?

With upstart, lldpd will send itself the SIGSTOP signal. upstart will detect this, resume lldpd with SIGCONT and assume it is ready. The code to support upstart is therefore quite simple. Instead of calling daemon(), do this:

const char *upstartjob = getenv("UPSTART_JOB"); if (!(upstartjob && !strcmp(upstartjob, "lldpd"))) return 0; log_debug("main", "running with upstart, don't fork but stop"); raise(SIGSTOP);

The job configuration file looks like this:

# lldpd - LLDP daemon description "LLDP daemon" start on net-device-up IFACE=lo stop on runlevel [06] expect stop respawn script . /etc/default/lldpd exec lldpd $DAEMON_ARGS end script

systemd provides a socket to achieve the same goal. An application is expected to write READY=1 to the socket when it is ready. With the provided library, this is just a matter of calling sd_notify("READY=1\n"). Since sd_notify() has less than 30 lines of code, I have rewritten it to avoid an external dependency. The appropriate unit file is:

[Unit] Description=LLDP daemon Documentation=man:lldpd(8) [Service] Type=notify NotifyAccess=main EnvironmentFile=-/etc/default/lldpd ExecStart=/usr/sbin/lldpd $DAEMON_ARGS Restart=on-failure [Install] WantedBy=multi-user.target OS include files

Linux-specific include files were a major pain in previous versions of lldpd. The problems range from missing header files (like linux/if_bonding.h) to the use of kernel-only types. Those headers have a difficult history. They were first shipped with the C library but were rarely synced and almost always outdated. They were then extracted from kernel version with almost no change and lagged behind the kernel version used by the released distribution3.

Today, the problem is acknowledged and is being solved by both the distributions which extract the headers from the packaged kernel and by kernel developers with a separation of kernel-only headers from user-space API headers. However, we still need to handle legacy.

A good case is linux/ethtool.h:

• It can just be absent.
• It can use u8, u16 types which are kernel-only types. To work around this issue, type munging can be setup.
• It can miss some definition, like SPEED_10000. In this case, you either define the missing bits and find yourself with a long copy of the original header interleaved with #ifdef or conditionally use each symbol. The latest solution is a burden by itself but it also hinders some functionalities that can be available in the running kernel.

The easy solution to all this mess is to just include the appropriate kernel headers into the source tree of the project. Thanks to Google ripping them for its Bionic C library, we know that copying kernel headers into a program does not create a derivative work.

1. Therefore, the use of u_int16_t and u_int8_t types is a left-over of the previous serializer where the size of all members was important. ↩

2. Tokenization is not the only advantage of libedit native API. The API is also cleaner, does not have a global state and has a better documentation. All the implementations are also BSD licensed. ↩

3. For example, in Debian Sarge, the Linux kernel was a 2.6.8 (2004) while the kernel headers were extracted from some pre-2.6 kernel. ↩

You may ask me: why to have page titles for views?

Views in Drupal, technically, are used to display a list of content. Translating to business language, it will be used for category pages. For example, if you build a shopping website in Drupal to sell mobile devices, you will use "views" to construct the page of Apple products (and also Samsung products, LG products, etc ...)

These category pages are very important (second most after your homepage). You will definitely want they appear on top of Google search results for relating keywords. To achieve it, one of the best SEO practices is to set good focused titles for them.

read more

If you have spent any time with Drupal Commerce, then you're aware of the separation between a product and a product display. The product entity is used for the price, title and SKU, whereas the product display is used to display the product on the front-end.

On most Commerce websites, you have to enter in a product two times. First, you have to create the product entity, and then create the product display. I personally love the separation between the product and product display, because of this Commerce can be used as a proper eCommerce framework.

If you want to save time when entering in products, then you should look at the Auto Product Display module for Commerce.

TMGMT

• What exactly is the Translation Management Tool?
  • Started at a Sprint a year ago here in Zurich, initiated and supported by Miro Dietiker, among others.
    Does not introduce new translation functionality but builds on top of the existing content and entity translation modules.
    The focus is on the management of the translation process. Instead manually creating a translation by copy and pasting translations, it allows to create a job that can be sent to a local or external translation service.
    Various translation services are supported, local users, exporting XLIFF files, machine translation and translation agencies like Gengo, Supertext and One Hour Translation.
    The complete workflow of creating a translation is kept under control at any point, including quality control, review processes and saving the translation back to Drupal.
    It provides various management views that give an overview over the translation status of your site and pending translation jobs.
• How does this integrate, or play nicely with i18n?
  • It currently accepts nodes and entities as input. Integration with interface translation and so called i18n objects (menus, terms, views) is initiated but not yet complete. (Help is welcome!)
• What is the current state of development?
  • We are preparing the first beta release, to be released within the next weeks. The focus of this release is node and entity translation, improved user interfaces and the new local translator.
    The development version can be downloaded on the project page and we have a lot of automated tests to make sure that this version is as stable as possible.
    Apart from the core project, there is a translation server distribution coming. It is based on the google summer of code project by Sebastian Siemssen (fubhy). We will release a first version of that distribution together with the beta release.

Use Cases

At any given time, you can take a single node, send it to a translation service like Gengo, review translation and save it back in a single user interface.
Use the provided rules integration to automatically create a machine translation for all languages that your site supports when a news article is created.
Request a translation of all pages from your local translation team, which can use the local translator that provides a separate User interface on the same site.
Do the same for several sites, using a centralized translation server so that your translators only need to access a single site.
Or you can use our directory listing to find the best external translator for your content.

Questions from Twitter

• Brant Wynn @brantwynn
  @ModsUnraveled can #TMGMT be used with Lingotek translation service? #MUP052

Episode Number:  112 Post Topics:  Drupal Drupal 7 Module Development Drupal Planet Javascript

In the last episode we covered creating a simple module to display a javascript confirmation popup before leaving a page. In this episode we take that a step further and add a nice administrative interface to the module.

In this episode you will learn:

• How to implement hook_menu to add an administrative page to your custom Drupal module
• How to build a Drupal administrative form

DDoD Video: 

read more

Download Podcast 99

Jennifer Lampton (jenlampton) of Jeneration Web Development joins Ryan Price and Mike Anello to talk about the most-excellent new theme engine forthcoming in Drupal 8: twig. Jen tells us everything that’s wrong with theming in Drupal 7, and how twig in Drupal 8 will be like riding a pure white unicorn on candy-coated road across an aqua-blue sky so much better. WYSIWYG in core, microcopy, documentation, and some really good picks of the week are also covered in the last ever 2-digit DrupalEasy podcast!

read more

Title"No al voto electrónico", Triple W, W Radio Publication TypeBroadcast Year of Publication2013 AuthorsWolf G, Ruiz O Refereed DesignationNon-Refereed Keywordse-vote Full Text

In yet another episode where we push for the population to be aware of the perils that electronic voting represents, Octavio Ruiz (@tacvbo) and me were invited to (briefly) talk about the topic in W Radio, one of the largest radio networks in Mexico.
The interview was short-ish, but we managed to get several points accross. And, of course, one of the best ways to do so is via a radio show with tens of thousands of listeners. So, we were quite happy to be there!
Here is the audio of the segment we presented in Fernanda Tapia's radio show "Triple W", in W Radio, Mexico.

AttachmentSize Tripe W - Entrevista: no al voto electronico9.97 MB

DRI3000 — Even Better Direct Rendering

This all started with the presentation that Eric Anholt and I did at the 2012 X developers conference, and subsequently wrote about in my DRI-Next posting. That discussion sketched out the goals of changing the existing DRI2-based direct rendering infrastructure.

Last month, I gave a more detailed presentation at Linux.conf.au 2013 (the best free software conference in the world). That presentation was recorded, so you can watch it online. Or, you can read Nathan Willis’ summary at lwn.net. That presentation contained a lot more details about the specific techniques that will be used to implement the new system, in particular it included some initial indications of what kind of performance benefits the overall system might be able to produce.

I sat down today and wrote down an initial protocol definition for two new extensions (because two extensions are always better than one). Together, these are designed to provide complete support for direct rendering APIs like OpenGL and offer a better alternative to DRI2.

The DRI3 extension

Dave Airlie and Eric Anholt refused to let me call either actual extension DRI3000, so the new direct rendering extension is called DRI3. It uses POSIX file descriptor passing to share kernel objects between the X server and the application. DRI3 is a very small extension in three requests:

1. Open. Returns a file descriptor for a direct rendering device along with the name of the driver for a particular API (OpenGL, Video, etc).

2. PixmapFromBuffer. Takes a kernel buffer object (Linux uses DMA-BUF) and creates a pixmap that references it. Any place a Pixmap can be used in the X protocol, you can now talk about a DMA-BUF object. This allows an application to do direct rendering, and then pass a reference to those results directly to the X server.

3. BufferFromPixmap. This takes an existing pixmap and returns a file descriptor for the underlying kernel buffer object. This is needed for the GL Texture from Pixmap extension.

For OpenGL, the plan is to create all of the buffer objects on the client side, then pass the back buffer to the X server for display on the screen. By creating pixmaps, we avoid needing new object types in the X server and can use existing X apis that take pixmaps for these objects.

The Swap extension

Once you’ve got direct rendered content in a Pixmap, you’ll want to display it on the screen. You could simply use CopyArea from the pixmap to a window, but that isn’t synchronzied to the vertical retrace signal. And, the semantics of the CopyArea operation precludes us from swapping the underlying buffers around, making it more expensive than strictly necessary.

The Swap extension fills those needs. Because the DRI3 extension provides an X pixmap reference to the direct rendered content, the Swap extension doesn’t need any new object types for its operation. Instead, it talks strictly about core X objects, using X pixmaps as the source of the new data and X drawables as the destination.

The core of the Swap extension is one request — SwapRegion. This request moves pixels from a pixmap to a drawable. It uses an X fixes Region object to specify the area of the destination being painted, and an offset within the source pixmap to align the two areas.

A bunch of data are included in the reply from the SwapRegion request. First, you get a 64-bit sequence number identifying the swap itself. Then, you get a suggested geometry for the next source pixmap. Using the suggested geometry may result in performance improvements from the techniques described in the LCA talk above.

The last bit of data included in the SwapRegion reply is a list of pixmaps which were used as source operands to earlier SwapRegion requests to the same drawable. Each pixmap is listed along with the 64-bit sequence number associated with an earlier SwapRegion operation which resulted in the contents which the pixmap now contains. Ok, so that sounds really confusing. Some examples are probably necessary.

• If the SwapRegion operation was implemented by copying data out of the source pixmap into the destination drawable, then the idle swap count will be equal to the swap count from this SwapRegion operation.

• If the SwapRegion operation was implemented by swapping the destination contents with the source contents, then the idle swap count will be equal to the previous swap count on the destination drawable.

I’m hoping you’ll be able to tell that in both cases, the idle swap count tries to name the swap sequence at which time the destination drawable contained the contents currently in the pixmap.

Note that even if the SwapRegion is implemented as a Copy operation, the provided source pixmap may not be included in the idle list as the copy may be delayed to meet the synchronization requirements specfied by the client.

Finally, if you want to throttle rendering based upon when frames appear on the screen, Swap offers an event that can be delivered to the drawable after the operation actually takes place.

Because the Swap extension needs to supply all of the OpenGL SwapBuffers semantics (including a multiplicity of OpenGL extensions related to that), I’ve stolen a handful of DRI2 requests to provide the necessary bits for that:

1. SwapGetMSC
2. SwapWaitMSC
3. SwapWaitSBC

These work just like the DRI2 requests of the same names.

Current State of the Extensions

Both of these extensions have an initial protocol specification written down and stored in git:

1. DRI3 protocol

2. Swap protocol

dynalogin 1.0.0 has finally been released. It is available now from the dynalogin web site

Packages for Debian GNU/Linux are already in the experimental catalog. It will be made available in Debian unstable and backports as soon as the freeze is lifted for the wheezy release.

dynalogin is a distributed solution for one-time-password/token based two-factor authentication on Linux/UNIX based operating systems. It supports PAM (pam_dynalogin) and also has native client code for C/C++ and PHP applications. It is free, open source software, so users can ascertain for themselves that it is genuinely secure and reliable for their needs. dynalogin is licensed under GPL terms.

dynalogin works with various soft-tokens, including the dynalogin token for Android and the Google Authenticator for various mobile devices. HOTP and TOTP-based hardware tokens can also be used.

Many thanks to Simon Josefsson and the OATH toolkit project which provides the underlying authentication algorithms for dynalogin

dynalogin can be integrated with the SimpleID OpenID provider to provide a turn-key solution for web-based single-sign-on, intranets and extranets. All the necessary components are conveniently packaged in Debian and Ubuntu.

dynalogin 1.0.0 has finally been released. It is available now from the dynalogin web site

Packages for Debian are already in the experimental catalog. It will be made available in Debian unstable and backports as soon as the freeze is lifted for the wheezy release.

dynalogin is a distributed solution for one-time-password/token based two-factor authentication on Linux/UNIX based operating systems. It is free, open source software, so users can ascertain for themselves that it is genuinely secure and reliable for their needs. dynalogin is licensed under GPL terms.

dynalogin works with various soft-tokens, including the dynalogin token for Android and the Google Authenticator for various mobile devices. HOTP and TOTP-based hardware tokens can also be used.

Many thanks to Simon Josefsson and the OATH toolkit project which provides the underlying authentication algorithms for dynalogin

dynalogin can be integrated with the SimpleID OpenID provider to provide a turn-key solution for web-based single-sign-on, intranets and extranets.