Elsewhere

Mario Lang: soundCLI works again

Planet Debian - Sun, 12/10/2014 - 15:30

I recently ranted about my frustration with GStreamer in a SoundCloud command-line client written in Ruby.

Well, it turns out that there was quite a bit confusion going on. I still haven't figured out why my initial tries resulted in an error regarding $DISPLAY not being set. But now that I have played a bit with gst-launch-1.0, I can positively confirm that this was very likely not the fault of GStreamer.

THe actual issue is, that ruby-gstreamer is assuming gstreamer-1.0, while soundCLI was still written against the gstreamer-0.10 API.

Since the ruby gst module doesn't have the Gstreamer API version in its name, and since Ruby is a dynamic language that only detects most errors at runtime, this led to all sorts of cascaded errors.

It turns out I only had to correct the use of query_position, query_duration, and get_state, as well as switching from playbin2 to playbin.

soundCLI is now running in the background and playing my SoundCloud stream. A pull request against soundCLI has also been opened.

On a somewhat related note, I found a GCC bug (ICE SIGSEGV) this weekend. My first one. It is related to C++11 bracketed initializers. Given that I have heard GCC 5.0 aims to remove the experimental nature of C++11 (and maybe also C++14), this seems like a good time to hit this one. I guess that means I should finally isolate the C++ regex (runtime) segfault I recently stumbled across.

Categories: Elsewhere

Guido Günther: Testing a NetworkManager VPN plugin password dialog

Planet Debian - Sun, 12/10/2014 - 13:55

Testing the password dialog of a NetworkManager VPN plugin is as simple as:

echo -e 'DATA_KEY=foo\nDATA_VAL=bar\nDONE\nQUIT\n' | ./auth-dialog/nm-iodine-auth-dialog -n test -u $(uuid) -i

The above is for the iodine plugin when run from the built source tree. This allows one to test these dialogs although one didn't see them since ages since GNOME shell uses the external UI mode to query for the password.

This blog is flattr enabled.

Categories: Elsewhere

John Goerzen: First impressions of systemd, and they’re not good

Planet Debian - Sun, 12/10/2014 - 05:18

Well, I finally bit the bullet. My laptop, which runs jessie, got dist-upgraded for the first time in a few months. My brightness keys stopped working, and it no longer would suspend to RAM when the lid was closed, and upon chasing things down from XFCE to policykit, eventually it appears that suddenly major parts of the desktop breaks without systemd in jessie. Sigh.

So apt-get install systemd-sysv (and watch sysvinit-core get uninstalled) and reboot.

Only, my system doesn’t come back up. In fact, over several hours of trying to make it boot with systemd, it failed in numerous spectacular and hilarious (or, would be hilarious if my laptop would boot) ways. I had text obliterating the cryptsetup password prompt almost every time. Sometimes there were two processes trying to read a cryptsetup password at once. Sometimes a process was trying to read that while another one was trying to read an emergency shell password. Many times it tried to write to /var and /tmp before they were mounted, meaning they *wouldn’t* mount because there was stuff there.

I noticed it not doing much with ZFS, complaining of a dependency loop between zfs-mount and $local-fs. I fixed that, but it still wouldn’t boot. In fact, it simply hung after writing something about wall passwords.

I’ve dug into systemd, finding a “unit generator for fstab” (whatever the hack that is, it’s not at all made clear by systemd-fstab-generator(8)).

In some cases, there’s info in journalctl, but if I can’t even get to an emergency mode prompt, the practice of hiding all stdout and stderr output is not all that pleasant.

I remember thinking “what’s all the flaming about?” systemd wasn’t my first choice (I always thought “if it ain’t broke, don’t fix it” about sysvinit), but basically ignored the thousands of messages, thinking whatever happens, jessie will still boot.

Now I’m not so sure. Even if the thing boots out of the box, it seems like the boot process with systemd is colossally fragile.

For now, at least zfs rollback can undo upgrades to 800 packages in about 2 seconds. But I can’t stay at some early jessie checkpoint forever.

Have we made a vast mistake that can’t be undone? (If things like even *brightness keys* now require systemd…)

Categories: Elsewhere

Zero to Drupal: Headless Drupal & AngularJS Hackathon - St. Louis

Planet Drupal - Sun, 12/10/2014 - 01:14
What do you get

When you cross Drupal, AngularJS, and a room full of folks eager to learn more about Drupal 8, api's, rest services, and front-end frameworks? You get St. Louis' first Headless Drupal Hackathon...that's what!

When

Thursday, October 23rd - 6:30p - 9:00p

Where

The Journey - Reber Place 4900 Reber Place St. Louis, MO

Details

For the first time in our beloved history as a Drupal User's Group, we will be hosting our first interactive meetup. Our goal is to collectively build a faux pizza ordering app using Drupal 8 as a backend, and AngularJS as the front end.

For more information, see our Meetup event page for more info.

Hope to see you there!

Tags
Categories: Elsewhere

SitePoint PHP Drupal: DrupalCon Amsterdam 2014 Report

Planet Drupal - Sat, 11/10/2014 - 18:00

As has been the pattern of many recent DrupalCons and Camps, DrupalCon Amsterdam 2014 was all about Drupal 8 and the changes that it’s bringing to the platform and community.

This has coincided with an increase in attendance at Drupal events (2300 in Amsterdam) and an increasing professionalism to DrupalCons. Drupal 8 has pulled us (sometimes forcibly) from out of our comfort zone and into the wider PHP and developer community. This has resulted in more talks covering a variety of non-Drupal topics, which, in my opinion, is a great thing.

The big news of the conference came on day 2, with Drupal 8 finally making it into beta. You can now effectively build basic sites in Drupal 8. In fact, a few brave souls already have, and I intend to do so too with my next site.

Dries Keynote

The regular ‘Driesnote’ was a thought provoking academic discussion on a current hot topic in the Open Source world, sustainability of projects and funding models. He started by stating that there are actually few good Open Source examples we could be following. In Dries’ opinion, the prevalent model of one company funding development is not a good one. Instead he suggested we look to other models, especially the concept of how Open Source software could be treated as a public good, or to coin a British term, ‘The Commons’. He used the example of public roads to show how community desire and amateur implementation can grow. Firstly via business investment (and sometimes privatization) and often resulting in Government control and management. To summarize:

Continue reading %DrupalCon Amsterdam 2014 Report%

Categories: Elsewhere

Dirk Eddelbuettel: RPushbullet 0.1.0 with a lot more awesome

Planet Debian - Sat, 11/10/2014 - 05:29

A new release 0.1.0 of the RPushbullet package (interfacing the neat Pushbullet service) landed on CRAN today.

It brings a number of goodies relative to the first release 0.0.2 of a few months ago:

  • pushing of files is now supported thanks to a nice pull request bu Mike Birdgeneau
  • a default device can be designated in the ~/.rpushbullet.json file or options
  • initialization has been rewritten to use recpients which can be indices, device names or, if missing entirely, the (new) default device
  • alternatively, email is supported as another recipient option in which case the Pushbullet service will send an email to the give address
  • pbGetDevices() now returns a proper S3 object with corresponding print() and summary() methods
  • the documentation regarding package initialization, and setting of key, devices, etc has been expanded
  • more examples has been added to the documentation
  • various minor cleanups, fixes, corrections throughout

There is a whole boat load of more wickedness in the Pushbullet API so if anybody feels compelled to add it, fire off pull requests at GitHub.

More details about the package are at the RPushbullet webpage and the RPushbullet GitHub repo.

Courtesy of CRANberries, there is also a diffstat report for this release.

This post by Dirk Eddelbuettel originated on his Thinking inside the box blog. Please report excessive re-aggregation in third-party for-profit settings.

Categories: Elsewhere

CiviCRM Blog: Improved permissions for activities and cases

Planet Drupal - Fri, 10/10/2014 - 20:39
A little background.

Established in 1972, we are a non-profit human services organization that serves many diverse populations in the Greater Pittsburgh area.

A large part of our focus is on homeless services, but we also do some mental and physical health programming, early childhood development, community integration and host a large food pantry near the University of Pittsburgh campus. 

Because of our disparate clientele and locations, we have been using multiple Access databases to track participant information.  For many years we wanted to move it online so staff in people's homes, could enter updates from the field.  However, the cost to implement was prohibitive to say the least.

Our Drupal and CiviCRM experience

When we rebuilt our website using Drupal, I stumbled upon CiviCRM as an alternate to Constant Contact.  For two years, we have used Civi on the backend of our site to register for events, make online contributions and send out bulk mails.

Civi has helped us raise a lot of dough.

After working in Civi and Drupal, we realized the combination could effectively be used to consolidate our program and human resources data as well as make it available online.

So we set off on the challenge of implementing a database to track all of our program's data, but repeatedly were having to reinvent the wheel to avoid staff seeing information that had nothing to do with their program.  Since the site was not for the public, we wanted to limit all work in Drupal (i.e: creating views and webforms), and instead focus on streamlining the database.  In essence, Drupal became the backend and Civi the front.

However, the Drupal permissions for CiviCRM were too general to control access to Activities and Cases and while the ACLs in Civi do provide more granular security, they don't cover Activities and Cases.

Make it Happen

We began rolling out our shiny new database in July 2014.

During the roll-out, I began investigating the cost of adding security to Activities and Cases and stumbled upon some threads discussing it.

This one that started in 2009 got me going and we haven't looked back since.

With these changes, it becomes dramatically easier for non-profits like us to implement CiviCRM as a total solution.

Thanks to Donald Lobo and Michael McAndrew for their guidance.

Please share your thoughts.

Categories: Elsewhere

Last Call Media: DrupalCamp LA: Manage the Gap

Planet Drupal - Fri, 10/10/2014 - 18:19
DrupalCamp LA: Manage the Gap

Usually when I travel anywhere outside of the Northeast, I tell people I am from Massachusetts or “outside of Boston” and people smile and nod. In rare instances granularity increases in the conversation and I eventually reveal that it’s actually  “a hip little town in the western part of the state called Northampton”.  Well, at DCLA, ~3000 miles from home, before 8:30am, I was 5 for 5 with people from California that not only know where Northampton is, but had visited before and had a favorite hotspot to share. Way to go Noho, you have officially been nationally recognized as geographically relevant.

Everyone was so friendly, and greeted us by name as we walked through the Campus Center. It was incredible that in the land of Hollywood, two folks from Massachusetts, could feel like celebrities. I am not sure it was only because we were the cool kids that had traveled the longest distance, this stardom and warm welcome may have been derived from the early recognition we received for IT mastery.  I basically spent the first morning of the camp traveling from table to table, helping all the sponsors connect to the wifi. I know that doesn’t sound like a big deal to you, but considering the internet prowess in the room, the fact that this marketing manager type found a way to be technically useful and wave a magic connectivity wand to save the people, was pure bliss. People were genuinely super grateful and relieved to have access to the magical interwebs, I realized for techies, having internet was as important to them as my morning coffee is to me.

I learned a ton at the conference, and attended one session in particular that really resonated with me.  James Smith from Image X not only brought his Project Management A-game to the podium, but did so while sporting a sweet mustache.

Development & Profit in Project Management - James Smith, ImageX

The project manager is at the service of the team, not the other way around. James suggested that a daily check in with the team should sound like this:  “Have I met your expectations today? Did you get what you need from me? Did that do what you needed it to do? What do you want or think you need next? When do you need it by?"  

We all know that in the end, a happy team = a happy client. This is very much in line with one piece of our implementation of the Maker's Schedule within our Task-Driven Teamwork model whereby a dynamic hierarchy works, appropriately and effectively, to treat each the client, the project, and team members as the top priority. This DCLA presentation was very fitting: as we are growing our team at every position, we are especially looking for people to join our Project Management team.

If you have excellent communication skills and really advocate for both your clients and team, please consider joining us!

Categories: Elsewhere

IXIS: First time at DrupalCon

Planet Drupal - Fri, 10/10/2014 - 16:17

I'm Andy, a developer at Ixis and having just settled back in after my first DrupalCon I thought I’d wrap up my thoughts after attending the annual European conference for the first time.

Initially - wow - DrupalCon is big! I’ve only been to some smaller PHP conferences so to see over 2000 people in one place was quite something. What struck me was how well it was organised - everything was on time with very few technical hiccups. I found the number of sessions quite overwhelming - there was so much to choose from, so having the videos of the sessions online with in an hour or so after it finished was really helpful. I’m still ploughing through the ones I’m interested in.

read more

Categories: Elsewhere

Drupal core announcements: Drupal 7 core security release on Wednesday, October 15 (and release window for Drupal 6)

Planet Drupal - Fri, 10/10/2014 - 15:48
Start:  2014-10-15 (All day) America/New_York Sprint Organizers:  David_Rothstein

There will be a security release of Drupal 7 core on Wednesday, October 15.

Although we normally only announce security release windows (rather than definite plans for a release), this month we are confident that a release will happen, so please be prepared to update your Drupal 7 sites on Wednesday.

A security release window for Drupal 6 core will be on the same date; this does not mean that a Drupal 6 core security release will necessarily take place on that date, only that you should prepare to look out for one (and be ready to update your Drupal 6 sites in the event that the Drupal security team decides to make a release).

There will be no bug fix release on this date; the next window for a Drupal core bug fix release is Wednesday, November 5.

For more information on Drupal core release windows, see the documentation on release timing and security releases, and the discussion that led to this policy being implemented.

Categories: Elsewhere

Matthias Klumpp: Listaller + Glick: Some new ideas

Planet Debian - Fri, 10/10/2014 - 14:59

As you might know, due to invasive changes in PackageKit, I am currently rewriting the 3rd-party application installer Listaller. Since I am not the only one looking at the 3rd-party app-installation issue (there is a larger effort going on at GNOME, based on Lennarts ideas), it makes sense to redesign some concepts of Listaller.

Currently, dependencies and applications are installed into directories in /opt, and Listaller contains some logic to make applications find dependencies, and to talk to the package manager to install missing things. This has some drawbacks, like the need to install an application before using it, the need for applications to be relocatable, and application-installations being non-atomic.

Glick2

There is/was another 3rd-party app installer approach on the GNOME side, by Alexander Larsson, called Glick2. Glick uses application bundles (do you remember Klik from back in the days?) mounted via FUSE. This allows some neat features, like atomic installations and software upgrades, no need for relocatable apps and no need to install the application.

However, it also has disadvantages. Quoting the introduction document for Glick2:

“Bundling isn’t perfect, there are some well known disadvantages. Increased disk footprint is one, although current storage space size makes this not such a big issues. Another problem is with security (or bugfix) updates in bundled libraries. With bundled libraries its much harder to upgrade a single library, as you need to find and upgrade each app that uses it. Better tooling and upgrader support can lessen the impact of this, but not completely eliminate it.”

This is what Listaller does better, since it was designed to do a large effort to avoid duplication of code.

Also, currently Glick doesn’t have support for updates and software-repositories, which Listaller had.

Combining Listaller and Glick ideas

So, why not combine the ideas of Listaller and Glick? In order to have Glick share resources, the system needs to know which shared resources are available. This is not possible if there is one huge Glick bundle containing all of the application’s dependencies. So I modularized Glick bundles to contain just one software component, which is e.g. GTK+ or Qt, GStreamer or could even be a larger framework (e.g. “GNOME 3.14 Platform”). These components are identified using AppStream XML metadata, which allows them to be installed from the distributor’s software repositories as well, if that is wanted.

If you now want to deploy your application, you first create a Glick bundle for it. Then, in a second step, you bundle your application bundle with it’s dependencies in one larger tarball, which can also be GPG signed and can contain additional metadata.

The resulting “metabundle” will look like this:

 

 

 

 

 

 

 

 

 

This doesn’t look like we share resources yet, right? The dependencies are still bundled with the application requiring them. The trick lies in the “installation” step: While the application above can be executed right away without installing it, there will also be an option to install it. For the user, this will mean that the application shows up in GNOME-Shell’s overview or KDEs Plasma launcher, gets properly registered with mimetypes and is – if installed for all users – available system-wide.

Technically, this will mean that the application’s main bundle is extracted and moved to a special location on the file system, so are the dependency-bundles. If bundles already exist, they will not be installed again, and the new application will simply use the existing software. Since the bundles contain information about their dependencies, the system is able to determine which software is needed and which can simply be deleted from the installation directories.

If the application is started now, the bundles are combined and mounted, so the application can see the libraries it depends on.

Additionally, this concept allows secure updates of applications and shared resources. The bundle metadata contains an URL which points to a bundle repository. If new versions are released, the system’s auto-updater can automatically pick these up and install them – this means e.g. the Qt bundle will receive security updates, even if the developer who shipped it with his/her app didn’t think of updating it.

Conclusion

So far, no productive code exists for this – I just have a proof-of-concept here. But I pretty much like the idea, and I am thinking about going further in that direction, since it allows deploying applications on the Linux desktop as well as deploying software on servers in a way which plays nice with the native package manager, and which does not duplicate much code (less risk of having not-updated libraries with security flaws around).

However, there might be issues I haven’t thought about yet. Also, it makes sense to look at GNOME to see how the whole “3rd-party app deployment” issue develops. In case I go further with Listaller-NEXT, it is highly likely that it will make use of the ideas sketched above (comments and feedback are more than welcome!).

Categories: Elsewhere

Code Karate: Creating a Drupal 7 Entity Reference View

Planet Drupal - Fri, 10/10/2014 - 14:26
Episode Number: 172

The Entity Reference module not only allows you to reference entities within your Drupal site, it also allows you to do so using a custom created entity reference view. This allows you to leverage the power of the views module to control how the entity reference selection field is displayed.

Tags: DrupalEntity ReferenceViewsDrupal 7Site BuildingDrupal Planet
Categories: Elsewhere

Deeson: The value of Drupal mentoring at DrupalCon Amsterdam 2014

Planet Drupal - Fri, 10/10/2014 - 13:09

Last week was my first time as a Drupal Core mentor at a DrupalCon event

Why does Drupal need mentors?

At DrupalCon Amsterdam, Dries mentioned in his key note that Drupal needs more people pitching in and contributing to Drupal Core. 

The Drupal community is growing, but the ratio of contributors to free loaders is getting worse. This is probably because people are struggling to get involved or don't understand which skills are needed.

To help overcome these barriers, DrupalCon events ask for people involved in Drupal Core to act as mentors.

How do you get involved?

First of all, you need to attend a mentor orientation session where you meet other mentors who share their experiences. At my orientation session, seasoned mentor Cathy Theys (YesCT) gave an excellent talk about what mentoring involves and what to expect.

Our introductions to each other were very amusing as we shared our Drupal.org usernames. Most of us felt it necessary to defend our choice of usernames with a full entomology and my personal favourite was eojthebrave (you'll have to ask him!).

What did I do as a mentor?

Last Wednesday I spent two hours triaging the issue queue, which contains issues with possible tasks suitable for novice contributors.

A novice is unlikely to complete a whole issue on their own, but there may be components of the issue, called tasks, that they can do. Tasks could include providing steps to reproduce a problem, feeding back on UX or design, taking screen shots or writing documentation. You do not always need to be a developer to contribute.

On Thursday I manned the mentor desk for a few hours. The DrupalCon events have a mentor desk manned by mentors who can advocate to delegates about contributing to Drupal Core and explain how to get involved. A key priority is to encourage delegates to attend a mentored sprint on the last day of DrupalCon, their first steps into the world of the Drupal Core community.

During the mentored sprints, more than 100 people got together to listen to a community tools presentation which covered the installation of everything you need to contribute, how to use the issue queue and comment and IRC etiquette.

Drupal 8 presentation

As a mentor, I gave a presentation to a small group of new contributors which helped them configure the tools and install Drupal 8 for the first time. For some, this was quite an achievement in itself.

Once set up with the tools, the new contributors broke up into smaller groups and selected issues to work on together with the aid of a mentor.

Critical issues

Some of the contributors I worked with looked at an issue to provide more user-friendly text on the 'Extend' section of the site. This was called 'Modules' on Drupal 7 and is where site administrators searched for modules to enable or disable them.

If the patch the contributors provided is committed, then the text they chose may appear on thousands of websites throughout the world - not bad for a short period of collaborative working!

The contributors also uncovered an issue in Drupal Core which we decided to mark as critical. Some of the new contributors were selected to have their finished patches committed directly into Drupal 8 by Drupal Core maintainer Angie Byron (webchick). Angie showed how the process of patch review and committing a patch to core works.

Après sprint

After a tiring, yet satisfying day, the mentors went to dinner together, where we discussed Drupal, development and PHPStorm.

Give it a try!

Being a mentor was a fantastic experience and I recommend it highly. I met some pretty incredible mentors and new contributors.

The Drupal community is made of a wide and varied skill base and everyone can get involved to make a difference to Drupal Core's quality.

If you've not contributed before, come along to the mentored sprint at the next DrupalCon.

If you can't wait that long, there is the Drupal Ladder which describes the small steps you can take to become a regular and useful contributor to Drupal Core.

Categories: Elsewhere

Janez Urevc: Drupal 8 from my media perspective - update #2

Planet Drupal - Fri, 10/10/2014 - 12:20

It's been a while since my last post and with DrupalCon Amsterdam behind us it seems a good timing for another update.

Categories: Elsewhere

Purencool Website Developer: How to set-up Compass on Ubuntu 14.04 for Omega 4 Development

Planet Drupal - Fri, 10/10/2014 - 11:34

I had some issues installing Compass on Ubuntu 14.04 this is what I did and the errors I came up against that gave me a large headache. I hope others can benefit from what I learnt. This assumes that you have already created a sub-theme in Omega4. I installed the following packages initially:

Tags Drupal Planet Web Development SASS
Categories: Elsewhere

Mario Lang

Planet Debian - Fri, 10/10/2014 - 11:00
GStreamer and the command-line?

I was recently looking for a command-line client for SoundCloud. soundCLI on GitHub appeared to be what I want. But wait, there is a problem with its implementation.

soundCLI uses gstreamer's playbin2 to play audio data. But that apparently requires $DISPLAY to be set.

So no, soundCLI is not a command-line client. It is a command-line client for X11 users.

Ahem.

A bit of research on Stackoverflow and related sites did not tell me how to modify playbin2 usage such that it does not require X11, while it is only playing AUDIO data.

What the HECK is going on here. Are the graphical people trying to silently overtake the world? Is Linux becoming the new Windows? The distinction between CLI and GUI has become more and more blurry in the recent years. I fear for my beloved platform.

If you know how to patch soundCLI to not require X11, please let me know. My current work-around is to replace all gstreamer usage with a simple "system" call to vlc. That works, but it does not give me comment display (since soundCLI doesn't know the playback position anymore) and hangs after every track, requiring me to enter "quit" manually on the VLC prompt. I really would have liked to use mplayer2 for this, but alas, mplayer2 does not support https. Oh well, why would it need to, in this day and age where everyone seems to switch to https by default. Oh well.

Categories: Elsewhere

Drupal Association News: Doei Doei, DrupalCon Amsterdam

Planet Drupal - Fri, 10/10/2014 - 10:49

DrupalCon Amsterdam has wrapped up, and now that we’re over the jet lag, it’s time to look back on one of the most successful DrupalCons to date. DrupalCon Amsterdam was the largest European DrupalCon yet, by far. Just to knock your socks off, here are some numbers:

  • More than 2,300 attendees showed up to 120 sessions and nearly 100 BoFs
  • 115 attendees showed up to the community summit and the business summit
  • 146 training attendees, 400 trivia night attendees, and 400 Friday sprinters made the week a success
  • …and through it all, we ate 1,200 stroopwafels.

The fun extended to more than just the conference — with 211 transit passes and 56 bike rentals, attendees from over 64 countries were able to enjoy all the city of Amsterdam had to offer. What a success!

As with any DrupalCon, DrupalCon Amsterdam wouldn’t have been a success without lots and lots of help from our passionate volunteers. We’d like to take a moment to send out a big THANK YOU to all of our track chairs and summit organizers:

  • Pedro Cambra - Coding and Development
  • Théodore Biadala - Core Conversations
  • Steve Parks - Drupal Business
  • Lewis Nyman and Ruben Teijeiro - Frontend
  • Michael Schmid - Site Building
  • Bastian Widmer - DevOps
  • Cathy Theys, Ruben Teijeiro, Gábor Hojtsy and the Core Mentors - Sprints
  • Adam Hill and Ieva Uzule - Onsite Volunteer Coordinators
  • Social Media - Baris Wanschers
  • Emma Jane Westby - Business Summit
  • Morten Birch and Addison Berry - Community Summit

We also appreciate everything our core mentors did to make DrupalCon Amsterdam a hit — and it’s thanks to lots of hard work from our passionate community members that Drupal 8 is in Beta!

We hope you had as fun and exiting a time in Amsterdam as we did. For those of us who weren’t able to make it, and even for those who were, you can relive the fun on the flickr stream, or catch any number of great sessions on the Drupal Association YouTube channel.  And remember to mark your calendar for DrupalCon Barcelona in September 2015. See you there! Below is Holly Ross' set of slides from the Amsterdam closing session. 

DrupalCon Amsterdam Closing Session from DrupalAssociation

Image credit to Pedro Lozano on Flickr.

Categories: Elsewhere

Martin Pitt: Running autopkgtests in the cloud

Planet Debian - Fri, 10/10/2014 - 09:25

It’s great to see more and more packages in Debian and Ubuntu getting an autopkgtest. We now have some 660, and soon we’ll get another ~ 4000 from Perl and Ruby packages. Both Debian’s and Ubuntu’s autopkgtest runner machines are currently static manually maintained machines which ache under their load. They just don’t scale, and at least Ubuntu’s runners need quite a lot of handholding.

This needs to stop. To quote Tim “The Tool Man” Taylor: We need more power!. This is a perfect scenario to be put into a cloud with ephemeral VMs to run tests in. They scale, there is no privacy problem, and maintenance of the hosts then becomes Somebody Else’s Problem.

I recently brushed up autopkgtest’s ssh runner and the Nova setup script. Previous versions didn’t support “revert” yet, tests that leaked processes caused eternal hangs due to the way ssh works, and image building wasn’t yet supported well. autopkgtest 3.5.5 now gets along with all that and has a dozen other fixes. So let me introduce the Binford 6100 variable horsepower DEP-8 engine python-coated cloud test runner!

While you can run adt-run from your home machine, it’s probably better to do it from an “autopkgtest controller” cloud instance as well. Testing frequently requires copying files and built package trees between testbeds and controller, which can be quite slow from home and causes timeouts. The requirements on the “controller” node are quite low — you either need the autopkgtest 3.5.5 package installed (possibly a backport to Debian Wheezy or Ubuntu 12.04 LTS), or run it from git ($checkout_dir/run-from-checkout), and other than that you only need python-novaclient and the usual $OS_* OpenStack environment variables. This controller can also stay running all the time and easily drive dozens of tests in parallel as all the real testing action is happening in the ephemeral testbed VMs.

The most important preparation step to do for testing in the cloud is quite similar to testing in local VMs with adt-virt-qemu: You need to have suitable VM images. They should be generated every day so that the tests don’t have to spend 15 minutes on dist-upgrading and rebooting, and they should be minimized. They should also be as similar as possible to local VM images that you get with vmdebootstrap or adt-buildvm-ubuntu-cloud, so that test failures can easily be reproduced by developers on their local machines.

To address this, I refactored the entire knowledge how to turn a pristine “default” vmdebootstrap or cloud image into an autopkgtest environment into a single /usr/share/autopkgtest/adt-setup-vm script. adt-buildvm-ubuntu-cloud now uses this, you shold use it with vmdebootstrap --customize (see adt-virt-qemu(1) for details), and it’s also easy to run for building custom cloud images: Essentially, you pick a suitable “pristine” image, nova boot an instance from it, run adt-setup-vm through ssh, then turn this into a new adt specific "daily" image with nova image-create. I wrote a little script create-nova-adt-image.sh to demonstrate and automate this, the only parameter that it gets is the name of the pristine image to base on. This was tested on Canonical's Bootstack cloud, so it might need some adjustments on other clouds.

Thus something like this should be run daily (pick the base images from nova image-list):

$ ./create-nova-adt-image.sh ubuntu-utopic-14.10-beta2-amd64-server-20140923-disk1.img $ ./create-nova-adt-image.sh ubuntu-utopic-14.10-beta2-i386-server-20140923-disk1.img

This will generate adt-utopic-i386 and adt-utopic-amd64.

Now I picked 34 packages that have the "most demanding" tests, in terms of package size (libreoffice), kernel requirements (udisks2, network manager), reboot requirement (systemd), lots of brittle tests (glib2.0, mysql-5.5), or needing Xvfb (shotwell):

$ cat pkglist apport apt aptdaemon apache2 autopilot-gtk autopkgtest binutils chromium-browser cups dbus gem2deb glib-networking glib2.0 gvfs kcalc keystone libnih libreoffice lintian lxc mysql-5.5 network-manager nut ofono-phonesim php5 postgresql-9.4 python3.4 sbuild shotwell systemd-shim ubiquity ubuntu-drivers-common udisks2 upstart

Now I created a shell wrapper around adt-run to work with the parallel tool and to keep the invocation in a single place:

$ cat adt-run-nova #!/bin/sh -e adt-run "$1" -U -o "/tmp/adt-$1" --- ssh -s nova -- \ --flavor m1.small --image adt-utopic-i386 \ --net-id 415a0839-eb05-4e7a-907c-413c657f4bf5

Please see /usr/share/autopkgtest/ssh-setup/nova for details of the arguments. --image is the image name we built above, --flavor should use a suitable memory/disk size from nova flavor-list and --net-id is an "always need this constant to select a non-default network" option that is specific to Canonical Bootstack.

Finally, let' run the packages from above with using ten VMs in parallel:

parallel -j 10 ./adt-run-nova -- $(< pkglist)

After a few iterations of bug fixing there are now only two failures left which are due to flaky tests, the infrastructure now seems to hold up fairly well.

Meanwhile, Vincent Ladeuil is working full steam to integrate this new stuff into the next-gen Ubuntu CI engine, so that we can soon deploy and run all this fully automatically in production.

Happy testing!

Categories: Elsewhere

Ingo Juergensmann: Buildd.Net: update-buildd.net v0.99 released

Planet Debian - Thu, 09/10/2014 - 21:29

Buildd.Net offers a buildd centric view to autobuilder network such as previously Debians autobuilder network or nowadays the autobuilder network of debian-ports.org. The policy of debian-ports.org requires a GPG key for the buildd to sign packages for upload that is valid for 1 year. Buildd admins are usually lazy people. At least they are running a buildd instead of building those packages all manually. Being a lazy buildd admin it might happen that you miss to renew your GPG key, which will render your buildd unable to upload newly built packages.

When participating in Buildd.Net you need to run update-buildd.net, a small script that transmits some statistical data about your package building. I added now a GPG key expiry check to that script that will warn the buildd admin by mail and text on the Buildd.Net arch status page, such as for m68k. So, either your client updates automatically to the new version or you can download the script yourself.

Kategorie: DebianTags: BuilddNetDebianSoftware 
Categories: Elsewhere

Pages

Subscribe to jfhovinne aggregator - Elsewhere