Keith Packard: udevwrap

Planet Debian - Tue, 16/08/2016 - 08:32
Wrapping libudev using LD_PRELOAD

Peter Hutterer and I were chasing down an X server bug which was exposed when running the libinput test suite against the X server with a separate thread for input. This was crashing deep inside libudev, which led us to suspect that libudev was getting run from multiple threads at the same time.

I figured I'd be able to tell by wrapping all of the libudev calls from the server and checking to make sure we weren't ever calling it from both threads at the same time. My first attempt was a simple set of cpp macros, but that failed when I discovered that libwacom was calling libgudev, which was calling libudev.

Instead of recompiling the world with my magic macros, I created a new library which exposes all of the (public) symbols in libudev. Each of these functions does a bit of checking and then simply calls down to the 'real' function.

Finding the real symbols

Here's the snippet which finds the real symbols:

static void *udev_symbol(const char *symbol) { static void *libudev; static pthread_mutex_t find_lock = PTHREAD_MUTEX_INITIALIZER; void *sym; pthread_mutex_lock(&find_lock); if (!libudev) { libudev = dlopen("libudev.so.1.6.4", RTLD_LOCAL | RTLD_NOW); } sym = dlsym(libudev, symbol); pthread_mutex_unlock(&find_lock); return sym; }

Yeah, the libudev version is hard-coded into the source; I didn't want to accidentally load the wrong one. This could probably be improved...

Checking for re-entrancy

As mentioned above, we suspected that the bug was caused when libudev got called from two threads at the same time. So, our checks are pretty simple; we just count the number of calls into any udev function (to handle udev calling itself). If there are other calls in process, we make sure the thread ID for those is the same as the current thread.

static void udev_enter(const char *func) { pthread_mutex_lock(&check_lock); assert (udev_running == 0 || udev_thread == pthread_self()); udev_thread = pthread_self(); udev_func[udev_running] = func; udev_running++; pthread_mutex_unlock(&check_lock); } static void udev_exit(void) { pthread_mutex_lock(&check_lock); udev_running--; if (udev_running == 0) udev_thread = 0; udev_func[udev_running] = 0; pthread_mutex_unlock(&check_lock); } Wrapping functions

Now, the ugly part -- libudev exposes 93 different functions, with a wide variety of parameters and return types. I constructed a hacky macro, calls for which could be constructed pretty easily from the prototypes found in libudev.h, and which would construct our stub function:

#define make_func(type, name, formals, actuals) \ type name formals { \ type ret; \ static void *f; \ if (!f) \ f = udev_symbol(__func__); \ udev_enter(__func__); \ ret = ((typeof (&name)) f) actuals; \ udev_exit(); \ return ret; \ }

There are 93 invocations of this macro (or a variant for void functions) which look much like:

make_func(struct udev *, udev_ref, (struct udev *udev), (udev)) Using udevwrap

To use udevwrap, simply stick the filename of the .so in LD_PRELOAD and run your program normally:

# LD_PRELOAD=/usr/local/lib/libudevwrap.so Xorg Source code

I stuck udevwrap in my git repository:


You can clone it using

$ git git://keithp.com/git/udevwrap
Categories: Elsewhere

Palantir: The Secret Sauce, Ep. 27: A Peek at Drupal GovCon

Planet Drupal - Tue, 16/08/2016 - 04:19
The Secret Sauce, Ep. 27: A Peek at Drupal GovCon The Secret Sauce manley Mon, 08/15/2016 - 21:19 Ken Rickard, Kirsten Burgard, and James King with Allison Manley Aug 16, 2016

We want to make your project a success.

Let's Chat.

Drupal GovCon has been gaining momentum for four years now. Curious about what it has to offer?

On this week’s episode of The Secret Sauce, we are joined by guests Kirsten Burgard and James King, organizers of Drupal GovCon.

iTunes | RSS Feed | Download| Transcript

Subscribe to all of our episodes over on iTunes.

Stay connected with the latest news on web strategy, design, and development.

Sign up for our newsletter. Transcript

Allison Manley [AM]: Hi again everyone, and welcome to The Secret Sauce, a short podcast by Palantir.net, that offers a quick bit of advice to help your business run a little bit better. 

I’m Allison Manley, an Account Manager, and today we have a different Secret Sauce. Ken Rickard, our Director of Professional Services, recently attending Drupal GovCon in Washington DC in July, and got to sit down with two of the organizers: James King from the National Institutes of Health (NIH), and Kirsten Burgard from the US Department of State to chat about this annual Drupal event. So they are going to share what this event is about, and why you may want to check it out next year. 
All right! Take it away Ken . . . 

Ken Rickard [KR]:  This is Ken Rickard. We’re at the Palantir Secret Sauce podcast. This is our broadcast from Drupal GovCon, and we’ve invited two of the organizers to join us today: we’re with Kirsten Burgard and James King. 

Kirsten Burgard [KB]: Yay! Welcome to Drupal GovCon.

KR: Thank you. This is I think my third or fourth . . . it used to be Capital Camp, and it’s now GovCon. This is the second year I’ve been here at the NIH, I know that. So tell me, how did the two of you get involved in this event?

KB: Well, it started . . . this is all really Tim Wood’s fault. He’s at the Department of Commerce. And he thought it would be really great if we all got together and started to do events, mini events where we could share information. The very first event one we did we had thirteen people. Then we decided to hold a larger event as a government-focused one at Commerce, and that was 2012.
KR: Yes, I was there.
KB: Yeah, we killed the wifi before 8:00 in the morning [laughs]. I had never seen that before. We thought we’d get about 200 people. We had 330. And at that event James approached me and said, “Hey, what about NIH hosting it?” And I thought, ‘this is never going to happen. Who at NIH is really going to make this happen?’ And it’s been James for three years now.
James King [JK]: So I did go to the Commerce event, and I had started using Drupal since 2010. When I got hired here in 2009, they gave me a project that hadn’t been started, and they bought this thing called Drupal and had a server, and I knew nothing about Drupal so I was learning on my feet Drupal 6. And I started playing with it and really liked it, and wanted to learn more about it. Found out about the Commerce event and that, like what was there, but saw how cramped it was.
KB: [Laughs]
JK: And wanted to get involved, and also selfishly wanted to be able to get more exposure at NIH on Drupal. I work for the NIH Library . . . the internal research library for NIH . . . and one of the things that we’re trying to do is foster community in different areas. And since I have a technology background, I’m trying to encourage use of technology across NIH. And Drupal being one of the things we’re working on, we were trying to encourage more people to know about and use Drupal.
So it made sense to at least try to have an event at our place. Since then obviously this has continued to grow, and we now have user group meetings as well just for NIH people. And those are growing as well.
KR: Yeah, the GovCon is a little bit of a special event of all the ones that I go to. Most of the ones that I go to are regionally-themed, but this one is industry-themed, or in this case, government service.
KB: Yeah.
KR: Public service themed. So, I mean, what are the goals of the whole idea? What are we trying to accomplish here?
KB: Well when we started doing DrupalGov . . . it’s actually called Drupal For Gov . . . we really just wanted to make it possible for government practitioners in open source communities to get together. Drupal just had the largest influx of folks within government. We also have Linux people, WordPress people, Joomla people. We have a wide cross-section of open source CMS’ mostly, and some back-end things. And our primary goal was to make it possible for government employees to get access to the information they needed. Whether it was training, or collaboration, or even just innovative new thoughts and processes.
Oftentimes in government we’re very stove-piped. We don’t collaborate, we don’t cross sections. We don’t . . . even within my old agency, Department of Veterans Affairs, one section didn’t talk to the next section. It’s very confrontational almost between offices. So to make an organization like ours, which started with 11 people, to an event that now has over 1,000 people attending, is pretty weird [laughs]! It’s just pretty darn weird.
JK: So as a librarian geek or information professional, information architect, I very much embrace the idea of open source, but also government use. The government spends a lot of money with contractors developing themes, developing modules, so forth, at a minimum I wanted to try and bring together the NIH people to be able to share that. To not only share the products, the deliverables, but to share lessons learned, to share the modules they’re using, tips and tricks, to come together on training, and so forth. Drupal GovCon was an easy way to foster that.
Having it here on campus made it very easy for the NIH people to come, but we’re also trying to be a sharing, open environment, so we’re making it as broadly available as possible. That’s why we continue to try to keep it to be free so that any level person, whether they’re a budding sysadmin, or developer, or a UX person, we’ll be able to come and learn.
KB: And we’ll have something for them too. We have sections that aren’t just like the regular “here are the tracks,” but actually sections across the board, and on top of that, additional training too.
KR: Yes, it’s a very interesting lineup. You have a very diverse speaker group, and very diverse attendee group. It’s interesting too, a lot of the Drupal events are weekend events . . . this is during the week. And so it’s almost a professional event. I think that’s fascinating too.
So based on the success we’ve seen from the last few years, I was at the Commerce event, what are you hoping to see next year?
KB: So next year might be a little bit more difficult because like I said, we’re over 1,000 now. Our attendee drop is nowhere where it needs to be on a free event. So typically a free event will have like a 50% drop in attendance over registration. Ours hovers at less than 40%. That makes it much more difficult for us to gauge how many lunches to buy, how many cups of coffee we need . . . we ran out of coffee yesterday morning an hour into coffee service, and we bought 800 cups! We ran out of lunches really close to the very end, so it wasn’t as like a lot of people had to go buy lunches. I believe we ran out of lunches again today, but not badly . . . only a couple.
JK: No, we were pretty close. The auditorium seats 500.
KR: And that’s the biggest space we have available.
KB: Yeah. And what we do through the day is we flux space everything. So we try to tell all the attendees, “Please, refresh your screens. Sessions will move.” And they do. And sometimes speakers forget where they’re supposed to be.
JK: I appreciate and I expect that we’ll continue to have that diversity as highlighted in our keynotes. The first day keynote was challenging attendees to really look at diversity and bias that’s in the industry, and how do we step back and address that. Today was more of a practical on how do we practically move an agency top-level site to Drupal. And tomorrow . . .
KB: Tomorrow it’s all security. We actually have the keynote from Velocity from last year, Laura Bell, who is a security expert from New Zealand. So I’ve had all kinds of fan boys come up to me and say, “oh my god, how did you get Laura Bell?” and I’m like, “I asked.” [laughs] 
KR: That might be the lesson for folks to takeaway from this episode of the podcast is sometimes all you have to do is show up and ask.
KB: Yeah. Show up and ask. It works really well.
KR: All right. Thank you both for joining me.
KB: Thank you. 

AM: That’s the end of this week’s Secret Sauce. For more great tips, check out our website at palantir.net, and check us out on Twitter. Our handle is @palantir. 

Have a great day!

Categories: Elsewhere

Drupal Association News: Changes for the Drupal Association Events Team

Planet Drupal - Tue, 16/08/2016 - 02:22

As Megan mentioned in her blog post, The Association’s mission is to unite the community to build and promote Drupal and DrupalCon remains an important way to achieve this mission. Unfortunately, our DrupalCon team is smaller due to staff reductions and we needed to adjust our work accordingly. To decide on what to eliminate, we took a critical look at what services the Association can continue to provide with minimal impact to the mission critical event elements that help move the Project forward.

DrupalCon has traditionally supported our contribution journey, helping developers and other contributors level up their skill to not only build better sites, but to also learn how to contribute code to core or through modules. We will continue providing the programing elements to support our contributors.

Over the years, these events have attracted some people whose  organizations aren't Drupal service providers. They've been organizations who chose, or considered, Drupal for their own systems. Going forward, we want to focus more on welcoming these newcomers and make sure they connect with content that serves their needs, for example: how to expand their use of Drupal and find new ways to solve their business solutions, find agencies and new technologies to include in their solutions, and connect with their peers. We already began this work through Summits and we are looking into other programs for them in the future.

Yet, while we will sustain our programming and even expand upon it, we do need to eliminate some of our workload. As a team, and with Megan and the board of directors, we looked at all of our work with a few lenses - what can we streamline, what can other remaining staff pick up and run with, what do we need to reduce or eliminate. We tried to prioritize the things that were mission critical to the project, would be noticeable to a community members' event experience, and do as many cuts behind-the-scenes as possible.

Things that we have decided to eliminate:

  • Hotel welcome letter - If you've stayed at one of the partner hotels at a DrupalCon, you may have noticed the welcome letter from the Drupal Association. We will not be producing these letters going forward.
  • Extended Sprints hosted lunches - I know this one may be disappointing, but it's time consuming for staff to manage, particularly on site during the Con and has a direct expense associated with it. Having no formal ticketing or RSVP system made planning the appropriate amount of food (with a myriad of unknown dietary needs) rather challenging and occasionally left attendees grumbling if their needs weren't met.
  • Re-bidding vendors each year - We have cultivated a list of reliable & cost-effective vendors for various production supplies that offer competitive pricing. Rather than seeking additional bids for each Con, and then picking the ones we used last time, we will renew previous contracts when feasible, a good working relationship is established, and the value is less than $25,000 USD. (behind-the-scenes)

Things that we have decided to decrease or reduce:

  • Scholarship reimbursement procedure has changed - In the past we have assisted recipients in making appropriate accommodations - hotel arrangements on our staff block, working with a travel agent to procure flights, etc. Moving forward, recipients will book / arrange for their own travel and accommodations and will submit expenses to the Drupal Association for reimbursement.
  • Scholarship funds will be reduced - We have reduced the grants / scholarship budget from 20,000€ to 10,000€. We made this reduction as a part of our cost savings effort.
  • BOF boards will be pre-printed - We will have a cut-off date and print sessions directly onto signage. You would be surprised how much time it takes on site to get those boards lined up and written out. We will reserve the slots that weren’t booked by the cut-off date as well as additional timeslots for on-site sign-ups.
  • Visa letters will not be mailed - we will no longer print and mail visa letters unless specifically requested to. The PDF that is emailed to the attendee is sufficient for the majority of visa applications. Special requests will be honored.
  • Customer service will be distributed - Whereas in the past we had one primary team member focused on customer service, we are now taking a team approach to this. Submissions to our Contact Us form are now being split amongst staff and on site registration support will now happen from a variety of staff members. (behind the scenes)
  • DrupalCon website theming has shifted - While the Drupal Association engineering team continues to maintain events.Drupal.org, we no longer have staff resources to build new features, nor to implement the custom themes we use to represent the character of our host cities. Instead, we have updated our contracts with our contracted DrupalCon design firms so that they will provide the sub-themes for upcoming events. (behind the scenes)

Our goal throughout this process was to make DrupalCon continue to be as seamless of an experience for you as possible. While we believe we've done a solid first pass at turning the behemoth amount of work into a realistic workload for the remaining team, there may continue to be minor tweaks as we run through our first Con with our new team format.

As we rebound from the re-organization, we look forward to enhancing what DrupalCon provides. We have a host of ideas, and are eager to dig in with this new framework to see what all we can accomplish with the assistance of the community.

If you are interested in getting involved with DrupalCons, here are 7 easy ways you can contribute:

  1. Stay at a partner hotel
    This lowers event costs, which keeps ticket prices lower. Hotels want your business and we are able to negotiate special rates and accommodations for the group with group purchasing power. When we fill up our hotel block - the hotels "comp" hotel room nights for the Association - dropping our production expenses notably. For instance in New Orleans, we earned 122 free hotel room nights.
  2. Thank a sponsor, talk to them on site
    Sponsorships greatly subsidize the costs of DrupalCon. Go out of your way to tell a sponsor thank you, ask them about why sponsoring DrupalCon was important to them. Check them out in the exhibit hall, they've got more going on than just t-shirts… ...although they do make great t-shirts!
  3. Thank the speakers and program team
    They are volunteering a substantial amount of their time to speaking at DrupalCon, or selecting sessions for DrupalCon - it is no small feat. They are driven to contribute to the community and help everyone become super Drupalers by sharing their knowledge.
  4. Buy a ticket
    Rather self-explanatory, but DrupalCon conference, training, and summit ticket sales are vital to the sustainability of future DrupalCon events.
  5. Spread the word, invite someone
    Tell your co-workers, maybe a counterpart at a client or customer, speak at a local meet-up, tweet it out - that DrupalCon is the place to go for Drupal education and training. 92% of people believe recommendations from friends and family, use your influence to impact an event you care about.
  6. Volunteer on site
    In the weeks before the Con we ask for people who are interested in volunteering on site to step up. There are a host of ways, from helping stuff name badges, to watching sessions and counting attendees, to being a sprint mentor. There is a volunteer role to fit all types of interests.
  7. Complete the surveys!
    After the Con we always email out a survey, please complete them with your feedback. We read every response and use it to make decisions for future Cons.

About Rachel

Rachel Friesen (RachFrieee) joined the Drupal Association in April of 2014, and is the Events Manager. She leads the events team that produces DrupalCons across the globe. You probably have seen her running around a DrupalCon talking into a walkie talkie or at the Closing Session.

When not wrangling events, Rachel can be found running and biking around Portland, OR, USA.

Categories: Elsewhere

Talha Paracha: GSoC'16 – Pubkey Encrypt – Week 12 Report

Planet Drupal - Tue, 16/08/2016 - 02:00

This week marks the completion of my Google Summer of Code 2016 project Pubkey Encrypt. So I’ve spent the past 3 months building this module for Drupal 8 in the supervision of mentors Adam Bergstein (@nerdstein) and Colan Schwartz (@colan). You can download Pubkey Encrypt from its official Drupal.org project page. Accordingly, the module provides a way of encrypting data with users’ login-credentials. It is based on ownCloud’s Data Encryption Model and is an attempt of providing an easy-to-use security module for website maintainers.

Categories: Elsewhere

myDropWizard.com: Creating a static copy of a Drupal 6 site

Planet Drupal - Mon, 15/08/2016 - 22:40

As you probably already know, Drupal 6 is end-of-lifed. This creates many risks for Drupal 6 site owners to keep their site up! First and foremost: security updates are difficult to find and get. Because the energy of the Drupal community as a whole is focused on Drupal 8, the resources dedicated to keeping Drupal 6 secure are very limited. So, your site is more vulnerable to be hacked.

Many low-end website owners shrug off the risk of being hacked. Don't dismiss this risk! Your website could be defaced, have private data become public, and jeopardize your business and reputation. Those may be obvious risks. What may be less obvious is that a hacked Drupal 6 website can be used to infiltrate and deface other websites hosted from the same server. These can be websites operated by your organization - or in a shared hosting environment - even other businesses and organizations.

Take steps to protect yourself. myDropWizard offers Long-Term Support for Drupal 6, so, that is one option.

However, it is possible that you may not need support! For example, if your site doesn't get updated at all - or very minimally - it may be an option for you to get rid of Drupal altogether. I have a video tutorial that will walk you through how to do such a cloned backup.

Categories: Elsewhere

Virtuoso Performance: Translation migration and more

Planet Drupal - Mon, 15/08/2016 - 19:46
Translation migration and more

I’m running a bit behind, but I want to make it a practice to blog about each migration project I’m involved with (with each client’s permission, of course). Constructed examples are all well and good, but there’s nothing like real-world scenarios to give the flavor of using the migration framework in practice. Where possible (again with the client’s permission), I will share the full migration implementation on GitHub, so people can see not only code snippets illustrating particular points, but their full context.

This spring I contracted with North Studio, a web and mobile development shop based in British Columbia, to assist with the migration of their customer The Carlyle Group’s multilingual site from Drupal 6 to Drupal 8. As happens more often than not when moving from one major version to another of Drupal, the opportunity was taken to refactor the site, rethinking some of the original site structure as well as taking advantage of the new Drupal 8 architecture. One aspect of restructuring was consolidating 26 content types into 8. At the time I joined the project, some of the new content types, and all of the taxonomy, had already been manually recreated in Drupal 8, and my responsibility was to develop automated migration of five remaining content types (four of them with translations), plus editorial user accounts.

User migration - mapping roles

The user migration makes use of the core d6_user migration almost verbatim - the crucial difference is that, as with content types, roles are being consolidated in the move to Drupal 8. In d6_user, roles are mapped 1-to-1 from Drupal 6:

roles: plugin: migration migration: d6_user_role source: roles

In this project, the desired roles were already setup in the Drupal 8 site, and we used a static map to translate the Drupal 6 numeric role IDs to the corresponding Drupal 8 role configuration IDs, consolidating three different D6 roles into the D8 “manager” role.

roles: plugin: static_map source: roles bypass: true map: 4: manager # content author 3: manager # developer 6: manager # html content author 16: media_room_manager # media room manager 11: translator # translator Files

As with users, the migration is mostly identical to the core d6_file migration. The key difference in the file migration itself is that, because much of the content (including files) has already been moved to D8, we want to be sure that the automated file migration we’re implementing here only pulls the files that are referenced by the specific content types we’re migrating. We do this by creating a source plugin extending the core d6_file source plugin, and overriding prepareRow() to ignore any file ID (fid) which is not in the field tables for those content types (by returning FALSE). Note that we could just as easily have done this by implementing hook_migrate_prepare_row(), or a subscriber to the PREPARE_ROW event provided by migrate_plus, and throwing MigrateSkipRowException. In a later post (for a later project) I will demonstrate the event subscription approach.

The other side to this is populating file fields on nodes. We would be able to do this using the core d6_cck_file process plugin - but that has a hardcoded reference to the core d6_file migration. Therefore, we needed to define our own carlyle_cck_file process plugin, extending d6_cck_file to override create() and reference our own file migration (named, imaginatively enough, ‘file’).

Nodes Conversion of node references to taxonomy references

One aspect of the site refactoring was the conversion of some node types to vocabularies (and thus node references become taxonomy term references). For example, the former “industry” node type became an “industry” vocabulary. This vocabulary was already populated with all necessary terms when this migration project started - the challenge was to convert the node references. There were two elements to this:

  1. A getTargetTitles() method was added to a common source plugin class, CarlyleNode, which from a node reference field name would gather the node ID (nid) references in that source property, look up the title for each referenced node, and set the source property to the list of titles.

  2. The migration configuration file mapped the field using the entity_lookup process plugin (provided in migrate_plus). This lookup enables one to populate a reference field by looking up the incoming source value against a given property of the destination field’s target type, and set the referenced entity ID.

So, given this field mapping:

field_industry: plugin: entity_lookup source: field_industry ignore_case: true

The source property is initially populated by the core migration framework with the referenced node IDs on Drupal 6. In prepareRow(), getTargetTitles() is invoked to convert those nids to node titles (which in the node->vocabulary conversion were used as the term names). Then, in the processing stage, the entity_lookup plugin queries the field_industry target vocabulary for terms with those names, returning the term IDs to populate the term references. The ignore_case setting, of course, makes sure case differences between the manually-created terms in D8 and the incoming values don't prevent us from making the match.


The biggest challenge of this project was handling translated nodes. At the time of this project, migration of translations had not yet been implemented in core. The issue to implement this for Drupal 6 had become an epic, because the mechanism for storing node translations had changed completely - before Drupal 8, each translation was a separate node, referencing the parent node via a ‘tnid’ column in the node table. With Drupal 8, all versions of a piece of content are stored as a single node, with each revision containing all translations of the node as of the time of that revision (and a new translation introducing a new revision). Substantial progress had been made by others (particularly vasi) on that patch at the time I needed the functionality here, but had run into the problem of properly setting the “default” language for the node (the language of the parent node in D6). For this to work, that parent node needs to be migrated before any of its translations are, and we simply could not come up with a clean way to do this within a single migration. The solution I came up with for the Carlyle project, on which the core solution was ultimately based, was to have separate migrations for the parent node and for its translations - by making the translation migration dependent on the parent node migration, we guaranteed things were created in the right order.

I will note that the following notes, as well as the code on GitHub, are not the code exactly as implemented for this project (which was basically the proof-of-concept for the core solution), but reflects the final solution committed to Drupal core as of 8.1.8 and later (and thus what you would use to implement your own translation migrations).

Considering bio nodes, the node_bio migration imports the parent nodes for each logical piece of content - nothing is special here to support translation, it’s a straight-forward node migration. The migration of bio node translations (node_bio_translation) is nearly identical to node_bio with the following differences:

  1. The source plugin is flagged for translations:

      plugin: bio_node
      node_type: bio
      translations: true

    This tells the node source plugin to import only translated nodes - i.e., those which have a tnid which is neither 0 nor identical to nid. When this is omitted, only untranslated nodes (tnid == 0) and parent nodes for translation (tnid == nid) are imported.

  2. The nid is explicitly set to the nid of the D8 version of the original translation (which was migrated by node_bio):

      plugin: migration
      migration: node_bio
      source: tnid

    Note that the node_bio migration does not map nid - we cannot preserve the original nid in this case, because the destination site has manually-created content which would conflict with migrated nids.

  3. The destination plugin is flagged for translations:

      plugin: 'entity:node'
      translations: true

    This triggers some code to make sure that a translation is created on the node if the incoming language does not already exist on the node, and to add language as a destination key field to the map table for this migration.

Community contributions

The following community contributions were made in the course of this project.

Migrate Drupal 6 core node translation to Drupal 8 - as noted above, this project was the proof-of-concept for the separate-translation-migration approach which became part of the solution in core. Several people contributed the code which made this solution a reality.

default_value: null in static map skips empty rows - For this project (specifically in the podcast node migration), we needed to have a lack of a match in a static_map plugin return null, preventing the subsequent entity_lookup plugin from running. There was a core bug preventing this from working properly, which I fixed.

config-devel-import-one does not work with a .yml argument - A question frequently asked in #drupal-migrate is how to get changes to a migration configuration entity in config/install to take effect, without uninstalling and re-enabling your module? Starting with this project I’ve been using the cdi1 command in the config_devel module to reload changes - well, once I fixed this little bug.

Route collision reported when alias matches a different language - The translation migration in this project triggered this pathauto bug. My diagnosis assisted in getting this fixed, and I also contributed some work on the tests.

mikeryan Monday, August 15, 2016 - 12:46pm Tags Planet Drupal Drupal
Categories: Elsewhere

Appnovation Technologies: Drupal 8: Where Features meet Functionality

Planet Drupal - Mon, 15/08/2016 - 19:38

Drupal 8 is here. We know that developers and users alike are full of praise for Drupal 8, but what, if anything, is so desirable about it, so ‘must-have’?

Categories: Elsewhere

Chromatic: Drupal Code Standards: Formatting

Planet Drupal - Mon, 15/08/2016 - 18:56

This is the third post in a series about Drupal coding standards. In this post we’ll talk specifically about standards regarding formatting. This is by no means an exhaustive list of PHP syntax rules, but rather is focused on formatting standards for Drupal.

Other posts in this series:

  1. Code Standards: What Are They?
  2. Code Standards: How Do We Implement Them?
  3. Code Standards: Formatting
  4. Code Standards: Documentation
  5. Code Standards: The t() function
  6. Code Standards: Object Oriented Coding & Drupal 8
  7. Code Standards: Twig in Drupal 8

There is much debate over tabs vs spaces in the programming world, but here in the Drupal community, we use spaces. Two spaces, to be exact.

In our previous post, we talked about setting up your editor to help you out - you can set it to use 2 spaces for indentation. This is really easy in Sublime Text:


No trailing whitespace! There should never be a space at the end of a line. In our previous post, we talked about how you can set up your text editor to remove this for you automatically. Try to avoid extra blank lines throughout your files and functions. Use blank lines sparingly to keep crowded code readable, if necessary.

File endings

Unix file endings - a single blank line at the end of each file. This is another thing most text editors can do for you! Just one line, no more, no less.

Line Length

Lines should be 80 characters long. However, keep in mind that this is primarily for readability. If forcing your code to be broken up over multiple lines makes it less readable, then you should reconsider. This is especially true for conditions, which should never be wrapped onto multiple lines. Comment and documentation text, however, should always be 80 characters or under. Make sure that you have a ruler set up in your editor to show you where you’re going over, and you’ll never have to guess. This is easy in Sublime Text:

If you have an array declaration that’s longer than 80 characters, split it into a multi-line array, like so:

$items['advanced_forum_l'] = array( 'variables' => array( 'text' => NULL, 'path' => NULL, 'options' => array(), 'button_class' => NULL, ), );

Here we see each item is on its own line, and each item is followed by a comma, even the last item. This is Drupal best practice regarding arrays in PHP (other languages, such as Javascript, may differ).

While we’re on the subject of arrays - if you have a super long array (hundreds of items, for example), you could break each item into its own line. That would be very long, but very readable. However, if the programmer who looks at this code next is unlikely to need this information at their fingertips (for example, a list of countries or zip codes that the programmer will not need to reference), consider importing it from a csv file or similar, and keeping it out of your code.


There should always be one space around operators (=, -, +, *, =>, ., etc). Whether you’re doing math, assignments, or concatenating strings - when in doubt, every piece of an expression probably needs to be separated by one space. Just one! You do not need spaces just inside of parentheses.

Here’s an example without spaces, to show how hard it is to read:

if ($a='system'||$b=='system') { return $a=='system'?-1:1; }

And properly formatted:

if ($a == 'system' || $b == 'system') { return $a == 'system' ? -1 : 1; } Function Calls & Declarations

When declaring a function, there should always be a single space after the argument list and before the opening curly brace. The function then begins on the next line, indented with 2 spaces. The closing brace goes on its own line.

A function call always has a set of parentheses, with no spaces on either side of them, whether or not there are parameters. If there are parameters, they should be separated by a comma, followed by a space. This update hook from the Advanced Forum contrib module is a simple example of both a function declaration and function call:

function advanced_forum_update_7200() { if (variable_get('advanced_forum_forum_disabled') == NULL) { variable_set('advanced_forum_forum_disabled', FALSE); } Constants

Take a look the code above - notice the all caps? TRUE, FALSE, and NULL are always capitalized in Drupal code. They are constants, which are always in all caps in Drupal.

Custom constants must be prefixed with the module name. Here’s an example from the CKEditor module:

define('CKEDITOR_FORCE_SIMPLE_TOOLBAR_NAME', 'DrupalBasic'); define('CKEDITOR_ENTERMODE_P', 1); define('CKEDITOR_ENTERMODE_BR', 2); define('CKEDITOR_ENTERMODE_DIV', 3); Control Structures

When using control structures like if, else, elseif, case, switch, foreach, while, do, etc., there should always be a space after the control structure term. Also, there should always be a space before the opening curly brace. The statement is indented on the next line, and the closing brace is on its own line, much like functions.

Inline control structures are not permitted in Drupal, although they are valid PHP. You should not use either of the following structures in Drupal:

if($foo) echo bar();


if($foo) echo bar();

Control structures must always have braces, and the statement(s) must always be on the next line:

if ($foo) { echo bar(); }

Here’s an example using if and foreach from the Advagg contrib module:

// Loop through all files. foreach ($files as $values) { // Insert files into the advagg_files table if it doesn't exist. // Update if needed. if (advagg_insert_update_files($values['files'], $type)) { $write_done = TRUE; } // Insert aggregate into the advagg_aggregates table if it doesn't exist. if (advagg_insert_aggregate($values['files'], $values['aggregate_filenames_hash'])) { $write_done = TRUE; } // Insert aggregate version information into advagg_aggregates_versions. if (advagg_insert_aggregate_version($values['aggregate_filenames_hash'], $values['aggregate_contents_hash'], $root)) { $write_done = TRUE; } } return $write_done;

Here’s another example with statements using if, elseif, and else. Note that in Drupal, the standard is to use elseif as one word, not else if. Both are valid PHP, but the Drupal standards specify it as one word.

if ($type === 'css') { list($contents) = advagg_get_css_aggregate_contents($file_aggregate, $aggregate_settings); } elseif ($type === 'js') { list($contents) = advagg_get_js_aggregate_contents($file_aggregate, $aggregate_settings); } if (!empty($contents)) { $compressed = gzencode($contents, 9, FORCE_GZIP); $files[$type][$filename] = strlen($compressed); } else { $files[$type][$filename] = 0; }

Here’s an example from the Advanced Forum contrib module showing how to format a switch statement. Every case-breaking statement must be followed by a blank line. A case-breaking statement is the last statement that is executed, generally a break or return. If you take a look at the last line of the following example, you’ll see that a closing brace counts as a blank line. This is also a good example of spacing between operators.

switch ($period) { case 'day': $period_arg = 60 * 60 * 24; break; case 'week': $period_arg = 60 * 60 * 24 * 7; break; case 'month': $period_arg = 60 * 60 * 24 * 30; break; case 'quarter': $period_arg = 60 * 60 * 24 * 91; break; case 'year': $period_arg = 60 * 60 * 24 * 365; break; } Alternate control statement syntax for theme templates

For ease of coding and readability, there is an alternate structure to use for control structures inside of theme templates in Drupal 7. Use if (): and endif; instead of braces. Statements must still be on their own line, as must the endif statement. Here’s an example from the Zen subtheme:

<?php if ($comments && $node->type != 'forum'): ?> <h2 class="comments__title title"><?php print t('Comments'); ?></h2> <?php endif; ?> Twig

In Drupal 8, we use the Twig template engine. The Drupal Twig standards can be found here, and are based on the Twig coding standards. We’ll go into more detail in a later post!


For casting, always put a space between the type and the variable, like in this snippet from the Big Menu contrib module:

$p_depth = 'p' . (string) ((int) $depth + 3);

Note that there is a space after (string) and after (int).


Every PHP statement ends with a semicolon. Always!

PHP tags

All PHP files begin with an opening tag: <?php but never, ever use a closing tag! There are many reasons for this, one of which is that whitespace after a closing tag can cause errors, so allowing PHP to close it on its own eliminates those errors.

Also, never use php short tags (<? ?>).

This is a pretty solid overview of the major things that Drupal may do differently than other PHP frameworks or content management systems. If you want to dig into Drupal and PHP syntax, there’s a long rabbit hole waiting for you, but these basics will keep you from making major mistakes, and keep your code readable! You’ll also run into less hiccups when contributing to core code or modules on Drupal.org! Many of these examples were from Drupal 7 - to find out more about Drupal 8 and object-oriented programming, check back for a future blog post in this series!

We hope you learned a lot - if you learned anything new, or want to school us, reach out to us on twitter - @ChromaticHQ

Our next post will cover documentation, which is always important - in Drupal or anywhere else, so stay tuned!

Hero Photo Attribution: Windell Oskay CC

Categories: Elsewhere

myDropWizard.com: YouTube videos stop working with Lightbox2? Here's the fix!

Planet Drupal - Mon, 15/08/2016 - 14:12

This is similar to our last article about how YouTube videos stopped working with Embedded Video Field on Drupal 6 - except this is for Lightbox2 and affects Drupal 6, 7 and 8.

While we still can't seem to find any announcement from Google, it appears that the old YouTube embed code (which is used by the Lightbox2 module) has stopped working.

Using Lightbox2 to play videos is sort of an edge case, and there wasn't an existing fix for it, so we created a patch on this issue.

We encountered this issue with one of our Drupal 6 Long-Term Support customers, so we created a Drupal 6 patch too!

(We created a Drupal 7 patch, and looking at the code the Drupal 8 version appears to be affected also, but since there is no stable release for it, we didn't test it or create a D8 patch.)

To fix: just apply the 6.x-1.x or 7.x-2.x patch!

Categories: Elsewhere

Shirish Agarwal: The road to TOR

Planet Debian - Mon, 15/08/2016 - 12:31

Happy Independence Day to all. I had been looking forward to this day so I can use to share with my brothers and sisters what little I know about TOR . Independence means so many things to many people. For me, it means having freedom, valuing it and using it to benefit not just to ourselves but to people at large. And for that to happen, at least on the web, it has to rise above censorship if we are to get there at all. I am 40 years old, and if I can’t read whatever I want to read without asking the state-military-Corporate trinity than be damned with that. Debconf was instrumental as I was able to understand and share many of the privacy concerns that we all have. This blog post is partly a tribute to being part of a community and being part of Debconf16.

So, in that search for privacy couple of years ago, I came across TOR . TOR stands for ‘The Onion Router’ project. Explaining tor is simple. Let us take the standard way in which we approach the website using a browser or any other means.

a. We type out a site name, say debian.org in the URL/URI bar .
b. Now the first thing the browser would do is look into its DNS Cache to see if the name/URL has been used before. If it is something like debian.org which has been used before and is *fresh* and there is content already it would serve the content from the cache there itself.
c. In case, if it’s not or the content is stale or something, it would generate a DNS lookup through the various routing tables till the DNS IP Address is found and information relayed to the browser.
d. The browser takes the IP Address and opens a TCP connection to the server, you have the handshake happen and after that it’s business as usual.
e. In case if it doesn’t work, you could get errors like ‘Could not connect to server xyz’ or some special errors with error codes.

This is a much simplified version of what happens or goes through normally with most/all of the browsers.

One good way to see how the whole thing happens is to use traceroute and use the whois service.

For e.g. –

[$] traceroute debian.org

and then

[$] whois | grep inetnum
inetnum: -

Just using whois IP Address gives much more. I just shared a short version because I find it interesting that Debian has booked all 255 possible IP Addresses but speculating on that would be probably be a job for a different day.

Now the difference when using TOR are two things –

a. The conversation is encrypted (somewhat like using https but encrypted through the relays)
b. The conversation is relayed over 2-3 relays and it will give a somewhat different identification to the DNS server at the other end.
c. It is only at the end-points that the conversation will be in plain text.

For e.g. the TOR connection I’m using atm is from me – France (relay) – Switzerland (relay) – Germany (relay) – WordPress.com . So wordpress thinks that all the connection is happening via Germany while I’m here in India. It would also tells that I’m running MS-Windows some version and a different browser while I’m from somewhere in India, on Debian, using another browser altogether

There are various motivations for doing that. For myself, I’m just a private person and do not need or want that any other person/s or even the State should be looking over my shoulder as to what I’m doing. And the argument that we need to spy on citizens because Terrorists are there doesn’t hold water over me. There are many ways in which they can pass messages even without tor or web. The Government-Corporate-Military just get more powerful if and when they know what common people think, do, eat etc.

So the question is how does you install tor if you a private sort of person . If you are on a Debian machine, you are one step closer to doing that.

So the first thing that you need to do is install the following –

$ sudo aptitude install ooniprobe python-certifi tor tor-geoipdb torsocks torbrowser-launcher

Once the above is done, then run torbrowser-launcher. This is how it would work out the first time it is run –

[$] torbrowser-launcher

Tor Browser Launcher
By Micah Lee, licensed under MIT
version 0.2.6
Creating GnuPG homedir /home/shirish/.local/share/torbrowser/gnupg_homedir
Downloading and installing Tor Browser for the first time.
Downloading https://dist.torproject.org/torbrowser/update_2/release/Linux_x86_64-gcc3/x/en-US
Latest version: 6.0.3
Downloading https://dist.torproject.org/torbrowser/6.0.3/tor-browser-linux64-6.0.3_en-US.tar.xz.asc
Downloading https://dist.torproject.org/torbrowser/6.0.3/tor-browser-linux64-6.0.3_en-US.tar.xz
Verifying signature
Extracting tor-browser-linux64-6.0.3_en-US.tar.xz
Running /home/shirish/.local/share/torbrowser/tbb/x86_64/tor-browser_en-US/start-tor-browser.desktop
Launching './Browser/start-tor-browser --detach'...

As can be seen above, you basically download the tor browser remotely from the website. Obviously, for this port 80 needs to be opened.

One of the more interesting things is that it tells you where it installs the browser.

/home/shirish/.local/share/torbrowser/tbb/x86_64/tor-browser_en-US/Browser/start-tor-browser and then detaches.

The first time the TOR browser actually runs it looks something similar to this –

Torbrowser picture

Additionally it would give you 4 choices. Depending on your need for safety, security and convenience you make a choice and live with it.

Now the only thing remaining to do is have an alias for your torbrowser. So I made

[$] alias tor


It is suggested that you do not use the same usernames on the onion network.

Also apart from the regular URL addresses such as ‘flossexperiences.wordpress.com’ you will also see sites such as https://www.abc12defgh3ijkl.onion.to (fictional address)

Now there would be others who would want to use the same/similar settings say as there are in their Mozilla Firefox installation.

To do that do the following steps –

a. First close down both Torbrowser and Mozilla Firefox .
b. Open your file browser and go to where your mozilla profile details are. In typical Debian installations it is at


In the next tab, navigate to –


c. Now copy the following files over from your mozilla profile to your tor browser profile and you can resume where you left off.

    logins.json (Firefox 32 and above)
    signons3.txt (if exists)

and the following folders/directories

    chrome (if it exists)
    searchplugins (if it exists)

Once the above is done, fire up your torbrowser with the alias shared. This is usually put it in your .bashrc file or depending on whatever terminal interpreter you use, wherever the config file will be.

Welcome to the world of TOR. Now, after a time if you benefit from tor and would like to give back to the tor community, you should look up tor bridges and relay. As the blog post has become long enough, I would end it now and hopefully we can talk about tor bridges and relay some other day.

Filed under: Miscellenous Tagged: #anonymity, #Debconf16, #debian, #tor, #torbrowser, GNU, Linux, Privacy
Categories: Elsewhere

Jim Birch: Making a multi-column content section with Drupal Paragraphs and Bootstrap

Planet Drupal - Mon, 15/08/2016 - 11:00

The Drupal Paragraphs module allows developers to created chunks of content, allowing admins to have more defined control than they would have in a normal WYSIWYG.  In this post, I will explain how to create a single paragraph bundle that can be used to display a 2, 3, 4, or 6 column layout.  In each of these columns, we will be able to use any Paragraph bundles that we define. 

This post covers how to do it in Drupal 8, but the process for Drupal 7 is very similar, just the markup in the template would be different.

Once you have the Paragraphs module installed, go to Structure > Paragraph types to click the Add paragraphs type button.  Paragraph types are entities, just like Nodes (Content Types).  They don't have Titles or URLs like nodes do, but they are revision-able, and take fields.  If you are new Paragraphs, take some time to create a few Paragraph types that fit your content needs.  I always create a "Simple" type, Paragraph that has a Text (formatted, long) field.  This fills the need for adding text to the page.  I also create an "Image" Paragraph type that has an Entity reference field to the Image Media bundle.

Read more

Categories: Elsewhere

DrupalCon News: A message to our CXOs: Believe In What You Do

Planet Drupal - Mon, 15/08/2016 - 10:00

According to marketing guru Seth Godin, ‘The secret of leadership is simple: do what you believe in. Paint a picture of the future. Go there. People will follow’.

Oh, if only it was that simple!

Categories: Elsewhere

LevelTen Interactive: We Have Started the Journey of the Results Oriented Web Roadshow

Planet Drupal - Mon, 15/08/2016 - 07:00

You might have read a blog a couple of weeks ago that announced we were going on the road on The Results Oriented Web RoadshowWell, guess what? It finally happened! Tom McCracken hopped on an RV to travel around the country to talk Drupal and Content Marketing. Don't believe us? Watch this clip:

...Read more

Categories: Elsewhere

Russ Allbery: Review: Winds of Fate

Planet Debian - Mon, 15/08/2016 - 03:11

Review: Winds of Fate, by Mercedes Lackey

Series: Mage Winds #1 Publisher: DAW Copyright: 1991 Printing: July 1992 ISBN: 0-88677-516-7 Format: Mass market Pages: 460

As a kid working my way through nearly everything in the children's section of the library, I always loved book series, since it meant I could find a lot more of something I liked. But children's book series tended to be linear, with a well-defined order. When I moved into the adult SF section, I encountered a new type of series: one that moves backwards and forwards in time to fill in a broader story.

I mention that here because Winds of Fate, although well into the linked series that make up Valdemar, was one of the first Valdemar books I read. (I think it was the first, but my memory is hazy.) Therefore, in my brain, this is where the story of Valdemar "begins": with Elspeth, a country that has other abilities but has forgotten about magic, a rich world full of various approaches to magic, and very pushy magic horses. Talia's story, and particularly Vanyel's, were always backstory, the events that laid the groundwork for Elspeth's story. (I didn't encounter Tarma and Kethry until somewhat later.)

Read now in context, this is obviously not the case. The Mage Winds trilogy, of which this is the first book, are clearly sequels to the Arrows of the Queen trilogy. Valdemar was victorious in the first round of war with Ancar, but the Heralds have slowly (and with great difficulty) become aware of their weakness against magic and their surprising lack of it. Elspeth has grown into the role of heir, but she's also one of the few who find it easy to talk about and think about magic (perhaps due to her long association with Kerowyn, who came into Valdemar from the outside world in By the Sword). She therefore takes on the mission of finding an Adept who can return to Valdemar, solve the mystery of whatever is keeping magic out of the kingdom, and start training mages for the kingdom again.

Meanwhile, we get the first viewpoint character from the Tayledras: the elf-inspired mages who work to cleanse the Pelagiris forests from magic left over from a long-ago war. They appeared briefly in Vanyel's story, since his aunt was friends with a farther-north tribe of them and Valdemar of the time had contact with mages. Darkwind and his people are far to the south, up against the rim of the Dhorisha crater. Something has gone horribly wrong with Heartstone of the k'Sheyna, his tribe: it cracked when being drained, killing most of the experienced mages including Darkwind's mother, and now it is subtly wrong, twisting and undermining the normal flow of magic inside their Vale. In the aftermath of that catastrophe, Darkwind has forsworn magic and become a scout, putting him sharply at odds with his father. And it's a matter of time before less savory magic users in the area realize how vulnerable k'Sheyna is.

Up to this point in the Valdemar series, Lackey primarily did localized world-building to support the stories and characters she was writing about. Valdemar and its Heralds and Companions have been one of the few shared elements, and only rarely did the external magic-using world encounter them. Here, we get the first extended contact between the fairly naive Heralds and experienced mages who understand how they and their Companions fit into the broader system of magic. We also finally get the origin of the Dhorisha Plains and the Tayledras and Shin'a'in, and a much better sense of the broader history of this world. And Need, which started as Kethry's soul-bonded sword and then became Kerowyn's, joins the story in a much more active way.

The world-building is a definite feature if you like this sort of thing. It doesn't withstand too much thinking about the typical sword and sorcery lack of technology, but for retroactive coherence constructed from originally scattered stories, it's pretty fun. (I suspect part of why I like the Valdemar world-building is that it feels a lot like large shared universe world-building in comics.) And Need is the high point of the story: she brings a much-needed cynical stubbornness to the cast and is my favorite character in this book.

What is not a feature, unfortunately, is the characterization. Darkwind is okay but a largely unremarkable here, more another instance of the troubled but ethical Tayledras type than a clearly defined character. But Elspeth is just infuriating, repeatedly making decisions and taking hard positions that seem entirely unwarranted by the recorded events of the book. This is made worse by how frequently she's shown to be correct in ways that seem like authorial cheating. At times, it feels like she's the heroine by authorial fiat, not because she's doing a particularly good job. I can muster some sympathy for not wanting to follow the plan of the Companions when it became clear they were acting entirely out of character and actively intervening, but she expresses that with petulant, childish insistence rather than reasoned argument. And she suddenly decides Skif is in love with her and treating her like a fragile princess on the basis of absolutely no action that occurs on camera in this book so far as I can tell, and proceeds to treat him like dirt for large sections of the book. That Skif then lives down to this suddenly negative impression doesn't help.

This book also has quite a lot of the U-shaped story arc in which everything gets worse and more horrific and more hopeless for the heroes over the course of the book until it turns into torture, and only then do they manage to claw their way back out. I've come to dislike this way of building tension. It's undeniably effective, but the parts of the story near the bottom of the U are difficult and painful reading. I prefer a bit more evenly-spread hurt/comfort storytelling in my popcorn fantasy reading.

Winds of Fate is, sadly, not a very good book. Most of the characterization is intensely irritating, the writing is a bit uneven, and the middle section of the book is rather painful to read. For me, though, that's balanced by the world-building and the sense of broadened scope, by Need's abrasive decisiveness, and by some really entertaining reactions to the combination of Elspeth, Need, and her Companion walking naive into the broader world. I still have a fond spot in my heart for it, but I'm hoping the remaining books of the trilogy are better.

Rating: 6 out of 10

Categories: Elsewhere

Paul Tagliamonte: Minica - lightweight TLS for everyone!

Planet Debian - Mon, 15/08/2016 - 02:40

A while back, I found myself in need of some TLS certificates set up and issued for a testing environment.

I remembered there was some code for issuing TLS certs in Docker, so I yanked some of that code and made a sensable CLI API over it.

Thus was born minica!

Something as simple as minica tag@domain.tls domain.tld will issue two TLS certs (one with a Client EKU, and one server) issued from a single CA.

Next time you’re in need of a few TLS keys (without having to worry about stuff like revocation or anything), this might be the quickest way out!

Categories: Elsewhere

Russ Allbery: Summer haul

Planet Debian - Mon, 15/08/2016 - 01:08

The reality is that I've just not been reading very much, as you can probably tell from the lack of reviews here. Lots of other things have been occupying my time, including rather too much on-line political reading eating into my off-line book reading. But hope springs eternal, so more books have been acquired in the interim. Since I use these posts to keep myself from buying duplicates, in the absence of a real database that I've not yet written or set up, here they are:

Mishell Baker — Borderline (sff)
Curtis C. Chen — Waypoint Kangaroo (sff)
Mark Forster — Secrets of Productive People (nonfiction)
Yoon Ha Lee — Ninefox Gambit (sff)
Seanan McGuire — Every Heart a Doorway (sff)
Don Norman — The Design of Everyday Things (nonfiction)
Kristina Rizga — Mission High (nonfiction)
John Scalzi — Lock In (sff)

This a pretty random collection of things from authors I know I like, non-fiction that looked really interesting from on-line reviews, the next book for book club reading for work (The Design of Everyday Things, which I've somehow never managed to read), and the first SF novel by an old college friend of mine (Waypoint Kangaroo by Curtis Chen).

Categories: Elsewhere

Reproducible builds folks: Reproducible Builds: week 68 in Stretch cycle

Planet Debian - Mon, 15/08/2016 - 00:38

What happened in the Reproducible Builds effort between Sunday August 7 and Saturday August 13 2016:

GSoC and Outreachy updates Reproducible work in other projects

Thomas Schmitt scdbackup@gmx.net implemented a new -as mkisofs option:

--set_all_file_dates timestring Set mtime, atime, and ctime of all files and directories to the given time. Valid timestring formats are: 'Nov 8 14:51:13 CET 2007', 110814512007.13, 2007110814511300. See also --modification-date= and man xorriso, Examples of input timestrings. This action stays delayed until mkisofs emulation ends. Up to then it can be revoked by --set_all_file_dates with empty timestring. In any case files which get into the ISO after mkisofs emulation ended will not be affected, unless another mkisofs emulation applies --set_all_file_date again.

LEDE developer Jonas Gorski submitted a patch to fix build times in their kernel:

kernel: allow reproducable builds Similar how we fix the file times in the filesystems, fix the build time of the kernel, and make the build number static. This should allow the kernel build to be reproducable when combined with setting the KERNEL\_BUILD\_USER and \_DOMAIN in case of different machines. The reproducability only applies to non-initramfs kernels, those still require additional changes. Signed-off-by: Jonas Gorski <jonas.gorski@gmail.com> Packages reviewed and fixed, and bugs filed

Patches have been submitted by:

Package reviews

28 reviews have been added, 4 have been updated and 7 have been removed in this week, adding to our knowledge about identified issues.

Issue types have been added/updated:

Weekly QA work

FTBFS bugs have been reported by:

  • Chris Lamb (23)
  • shirish शिरीष (1)
diffoscope development strip-nondeterminism development tests.reproducible-builds.org
  • schedule testing/i386 more often than unstable+experimental, in order to see the results of building with build path variation. (h01ger)
  • spectranaut wrote a patch for using sqlalchemy which has yet to be merged.
  • Our build path variation tests on testing/i386 have brought the first results: the build essential package set is now 43% unreproducible compared to "only" 26% on amd64. So another conclusion from this is probably that the build essential maintainers should merge our patches, the other is that build path variation is still a goal far way, which also can be seen "nicely" now in the general suite graph showing the impact of build path variation introduced last week. (h0lger)
  • Chris Lamb wrote a patch to improve the top-level navigation so that we can always get back to "home" of a package.
  • Chris Lamb also wrote a patch to explicitly log when a build was successful instead of it being implicit.

Chris started to ping old bugs with patches and no maintainer reaction so far.

This week's edition was written by Chris Lamb and Holger Levsen and reviewed by a bunch of Reproducible Builds folks on IRC.

Categories: Elsewhere

Steinar H. Gunderson: Linear interpolation, source alignment, and Debian's embedding policy

Planet Debian - Sun, 14/08/2016 - 23:57

At some point back when the dinosaurs roamed the Earth and I was in high school, I borrowed my first digital signal processing book from a friend. I later went on to an engineering education and master's thesis about DSP, but the very basics of DSP never stop to fascinate me. Today, I wanted to write something about one of them and how it affects audio processing in Nageru (and finally, how Debian's policies put me in a bit of a bind on this issue).

DSP texts tend to obscure profound truths with large amounts of maths, so I'll try to present a somewhat less general result that doesn't require going into the mathematical details. That rule is: Adding a signal to weighted, delayed copies of itself is a filtering operation. (It's simple, but ignoring it will have sinister effects, as we'll see later.)

Let's see exactly what that means with a motivating example. Let's say that I have a signal where I want to get rid of (or rather, reduce) high frequencies. The simplest way I can think of is to add every neighboring sample; that is, set yn = xn + xn-1. For each sample, we add the previous sample, ie., the signal as it was one sample ago. (We ignore what happens at the edges; the common convention is to assume signals extend out to infinity with zeros.)

What effect will this have? We can figure it out with some trigonometry, but let's just demonstrate it by plotting instead: We assume 48 kHz sample rate (which means that our one-sample delay is 20.83 µs) and a 22 kHz note (definitely treble!), and plot the signal with one-sample delay (the x axis is sample number):

As you can see, the resulting signal is a new signal of the same frequency (which is always true; linear filtering can never create new frequencies, just boost or dampen existing ones), but with much lower amplitude. The signal and the delayed version of it end up cancelling each other mostly out. Also note that there signal has changed phase; the resulting signal has been a bit delayed compared to the original.

Now let's look at a 50 Hz signal (turn on your bass face). We need to zoom out a bit to see full 50 Hz cycles:

The original signal and the delayed one overlap almost exactly! For a lower frequency, the one-sample delay means almost nothing (since the waveform is varying so slowly), and thus, in this case, the resulting signal is amplified, not dampened. (The signal has changed phase here, too—actually exactly as much in terms of real time—but we don't really see it, because we've zoomed out.)

Real signals are not pure sines, but they can be seen as sums of many sines (another fundamental DSP result), and since filtering is a linear operation, it affects those sines independently. In other words, we now have a very simple filter that will amplify low frequencies and dampen high frequencies (and delay the entire signal a little bit). We can do this for all frequencies from 0 to 24000 Hz; let's ask Octave to do it for us:

(Of course, in a real filter, we'd probably multiply the result with 0.5 to leave the bass untouched instead of boosting it, but it doesn't really change anything. A real filter would have a lot more coefficients, though, and they wouldn't all be the same!)

Let's now turn to a problem that will at first seem different: Combining audio from multiple different time sources. For instance, when mixing video, you could have input from two different cameras or sounds card and would want to combine them (say, a source playing music and then some audience sound from a camera). However, unless you are lucky enough to have a professional-grade setup where everything runs off the same clock (and separate clock source cables run to every device), they won't be in sync; sample clocks are good, but they are not perfect, and they have e.g. some temperature variance. Say we have really good clocks and they only differ by 0.01%; this means that after an hour of streaming, we have 360 ms delay, completely ruining lip sync!

This means we'll need to resample at least one of the sources to match the other; that is, play one of them faster or slower than it came in originally. There are two problems here: How do you determine how much to resample the signals, and how do we resample them?

The former is a difficult problem in its own right; about every algorithm not backed in solid control theory is doomed to fail in one way or another, and when they fail, it's extremely annoying to listen to. Nageru follows a 2012 paper by Fons Adriaensen; GStreamer does… well, something else. It fails pretty badly in a number of cases; see e.g. this 2015 master's thesis that tries to patch it up. However, let's ignore this part of the problem for now and focus on the resampling.

So let's look at the case where we've determined we have a signal and need to play it 0.01% faster (or slower); in a real situation, this number would vary a bit (clocks are not even consistently wrong). This means that at some point, we want to output sample number 3000 and that corresponds to input sample number 3000.3, ie., we need to figure out what's between two input samples. As with so many other things, there's a way to do this that's simple, obvious and wrong, namely linear interpolation.

The basis of linear interpolation is to look at the two neighboring samples and weigh them according to the position we want. If we need sample 3000.3, we calculate y = 0.7 x3000 + 0.3 x3001 (don't switch the two coefficients!), or, if we want to save one multiplication and get better numerical behavior, we can use the equivalent y = x3000 + 0.3 (x3001 - x3000). And if we need sample 5000.5, we take y = 0.5 x5000 + 0.5 x5001. And after a while, we'll be back on integer samples; output sample 10001 corresponds to x10000 exactly.

By now, I guess it should be obvious what's going on: We're creating a filter! Linear interpolation will inevitably result in high frequencies being dampened; and even worse, we are creating a time-varying filter, which means that the amount of dampening will vary over time. This manifests itself as a kind of high-frequency “flutter”, where the amount of flutter depends on the relative resampling frequencies. There's also cubic resampling (which can mean any of several different algorithms), but it only really reduces the problem, it doesn't really solve it.

The proper way of interpolating depends a lot on exactly what you want (e.g., whether you intend to change the rate quickly or not); this paper lays out a bunch of them, and was the paper that originally made me understand why linear interpolation is so bad. Nageru outsources this problem to zita-resampler, again by Fons Adriaensen; it yields extremely high-quality resampling under controlled delay, through a relatively common technique known as polyphase filters.

Unfortunately, doing this kind of calculations takes CPU. Not a lot of CPU, but Nageru runs in rather CPU-challenged environments (ultraportable laptops where the GPU wants most of the TDP, and the CPU has to go down to the lowest frequency), and it is moving in a direction where it needs to resample many more channels (more on the later), so every bit of CPU helps. So I coded up an SSE optimization of the inner loop for a particular common case (stereo signals) and sent it in for upstream inclusion. (It made the code 2.7 times as fast without any structural changes or reducing precision, which is pretty much what you can expect from SSE.)

Unfortunately, after a productive discussion, suddenly upstream went silent. I tried pinging, pinging again, and after half a year pinging again, but to no avail. I filed the patch in Debian's BTS, but the maintainer understandably is reluctant to carry a delta against upstream.

I also can't embed a copy; Debian policy would dictate that I build against the system's zita-resampler. I could work around it by rewriting zita-resampler until it looks nothing like the original, which might be a good idea anyway if I wanted to squeeze out the last drops of speed; there are AVX optimizations to be had in addition to SSE, and the structure as-is isn't ideal for SSE optimizations (although some of the changes I have in mind would have to be offset against increased L1 cache footprint, so careful benchmarking would be needed). But in a sense, it feels like just working around a policy that's there for good reason. So like I said, I'm in a bit of a bind. Maybe I should just buy a faster laptop.

Oh, and how does GStreamer solve this? Well, it doesn't use linear interpolation. It does something even worse—it uses nearest neighbor. Gah.

Categories: Elsewhere

Chris Lamb: try.diffoscope.org CLI client

Planet Debian - Sun, 14/08/2016 - 20:43

One criminally-unknown new UNIX tool is diffoscope, a diff "on steroids" that will not only recursively unpack archives but will transform binary formats into human-readable forms in order to compare them instead of simply showing the raw difference in hexadecimal.

In an attempt to remedy its underuse, in December 2015 I created the try.diffoscope.org service so that I—and hopefully others—could use diffoscope without necessarily installing the multitude of third-party tools that using it can require. It also enables trivial sharing of the HTML reports in bugs or on IRC.

To make this even easier, I've now introduced a command-line client to the web service:

$ apt-get install trydiffoscope [..] Setting up trydiffoscope (57) ... $ trydiffoscope /etc/hosts.allow /etc/hosts.deny --- a/hosts.allow +++ b/hosts.deny │ @@ -1,10 +1,17 @@ │ -# /etc/hosts.allow: list of hosts that are allowed to access the system. │ -# See the manual pages hosts_access(5) and hosts_options(5). │ +# /etc/hosts.deny: list of hosts that are _not_ allowed to access the system. │ +# See the manual pages hosts_access(5) and hosts_options(5).

You can also install it from PyPI with:

$ pip install trydiffoscope

Mirroring the original diffoscope command, you can save the output locally in an even more-readable HTML report format by appending "--html output.html".

In addition, if you specify the --webbrowser (or -w) argument:

$ trydiffoscope -w /etc/hosts.allow /etc/hosts.deny https://try.diffoscope.org/gaauupyapzkb

... this will automatically open your default browser to view the results.

Categories: Elsewhere

Dirk Eddelbuettel: rfoaas 1.0.0

Planet Debian - Sun, 14/08/2016 - 20:16

The big 1.0.0 is here! Following the unsurpassed lead of the FOAAS project, we have arrived at a milestone: Release 1.0.0 is now on CRAN.

The rfoaas package provides an interface for R to the most excellent FOAAS service--which itself provides a modern, scalable and RESTful web service for the frequent need to tell someone to f$#@ off.

Release 1.0.0 brings fourteen (!!) new access points: back(), bm(), gfy(), think(), keep(), single_(), look(), looking(), no(), give(), zero(), pulp(), sake(), and anyway(). All with documentation and testing.

Even more neatly, thanks to a very pull request by Tyler Hunt, we can now issue random FOs via the getRandomFO() function!

As usual, CRANberries provides a diff to the previous CRAN release. Questions, comments etc should go to the GitHub issue tracker. More background information is on the project page as well as on the github repo

This post by Dirk Eddelbuettel originated on his Thinking inside the box blog. Please report excessive re-aggregation in third-party for-profit settings.

Categories: Elsewhere


Subscribe to jfhovinne aggregator - Elsewhere