J-P Stacey: Architecting a Drupal 7 module into multiple files

Planet Drupal - Mon, 22/06/2015 - 16:42

In the Drupal modules that I inherit or review, I see a lot of different ways of factoring out into separate files, of what might have begun in the main module file. This can be useful for performance (to a limited extent) and legibility, but depending on how you do it, you might end up ironically spoiling both.

How should you break down your Drupal module files? Well, I'm not here to tell you the perfect file breakdown. Matching the architecture is good, although what "the architecture" means in Drupal 7 isn't clear. Outside of a Drupal 8/Symfony-style architectural model, there's a limit to how much the file breakdown really needs to match the architecture, and a limit to how useful doing so would be.

Read more of "Architecting a Drupal 7 module into multiple files"

Categories: Elsewhere

BlackMesh: Summer Drupal 8 Sprints

Planet Drupal - Mon, 22/06/2015 - 16:03

Sprints are times dedicated to focused work on a project or topic. People in the same location (physical space, IRC, certain issue pages) work together to make progress, remove barriers to completion, and try to get things to "done".

This summer, there are many Drupal 8- focused sprint opportunities before DrupalCon Barcelona.

Some of these are open to everyone, some have mentors or workshops to help new contributors, some have limited space, depending on the event.

Earlier this summer

DrupalCon Los Angeles had very productive extended sprints, and the main sprint on Friday was huge! Since then, many sprints have continued the progress, including: Drupal Camp Spain, DrupalCamp Wroclaw, Moldcamp in Chișinău, Moldova, and Frontend United.

And, we had two sprints (New Hampshire and New Jersey) aided by Drupal 8 Accelerate. If you have more money than time for sprinting or resources for planning or hosting a sprint, and you want to help get Drupal 8 out, giving to D8 Accelerate really helps.

New Hampshire, USA

The Drupal 8 critical meta issue on SafeMarkup was a focus of the New Hampshire, USA sprint.

(photo: @Cottser)

Jersey Shore Sprint, New Jersey, USA

crowdcg, kgoel, cilifen, jcloys (who got a first Drupal Core commit mention), akalata, davidhernandez, Ryan Weal.
(photo: pwolanin, cropped)

Continuing from the New Hampshire sprint, a main focus of the Jersey Shore sprint was the Drupal 8 critical meta issue on SafeMarkup. (I was there too.)

Coming soon! Go to one! June 25-28, 2015
Twin Cities Drupal Camp, Minneapolis, Minnesota, USA
@TCDrupal #TCDrupal

Sprints are concurrent with the training day on Thursday, and concurrent with sessions Friday and Saturday. The dedicated sprint day is on Sunday. We are expecting about 10 people on Thursday and 60 people on Sunday. Sunday will have a workshop for new contributors, Core/Contrib sprints, and a Drupal 8 Manual sprint.

See the sprint page on the tcdrupal.org site for details.

June 25-28, 2015
Drupal North, Toronto, Canada

Sunday is a dedicated Drupal 8 sprint day. There will be a small unofficial sprint on Thursday.

The whole 4 day camp is focused on Drupal 8.

June 28, 2015
Antwerp, Belgium

This is a dedicated one day sprint to help get Drupal 8 out.

See the announcement for details.

July 2-8, 2015
D8 Accelerate critical issue sprint, London

This 7 day sprint will be focused on Drupal 8 critical issues and space is limited.

See the groups.drupal.org post for details.

July 4, 2015
DrupalCamp Bristol, United Kingdom
@DrupalCampBris #dsbristol

Sprints will run concurrent with sessions on Saturday.

July 16-19, 2015
NYC Camp, New York, USA
@NYCCamp_org #NYCCamp

Monday to Wednesday are sprint only days, with Drupal 8 Core and Media for Drupal 8 scheduled. Panopoly is scheduled for Tuesday and Wednesday. Sprints will also be concurrent with trainings, summits, and sessions on Thursday through Sunday.

See the sprint page on the nyccamp.org site and the schedule for details.

July 22-26, 2015
DrupalCamp North, Sunderland, United Kingdom

Wednesday to Friday are dedicated extended sprint days and there will also be a sprint room at the camp concurrent with sessions on Saturday and Sunday.

July 22-26, 2015
GovCon, Bethesda, Maryland, USA
@DrupalGovCon #DrupalGovCon

GovCon conference is July 22-24 with sprints concurrent with sessions, but there will be two dedicated sprint days following on Saturday and Sunday, July 25 and 26 in Washington, DC at the ForumOne offices.

August 6-9, 2015
Drupalaton, Hungary
@Drupalaton #Drupalaton

Sprints will be concurrent with the camp all days.

Last year was super focused and relaxing.

August 12-15, 2015
MidWest Developers Summit (MWDS), Chicago, Illinois, USA

4 days of only sprinting, hosted in the Palantir.net offices.

Details still to be announced. See the groups.drupal.org event page for more details.

September 11-18, 2015
Montréal to Barcelona Sprint, Montréal, Canada

For some people in North America, Montréal could be on the way to Barcelona, where DrupalCon extended sprints start September 19.

See the groups.drupal.org event page for more details.


I will be at Twin Cities, GovCon, and MWDS.

Thanks to @klobutschar for fast changes to http://www.drupical.com to show the sprints even better.

Thanks to @mparker_17, @da_wehner, @ievauzule, @zsofimajor, @royscholten, @adshill, @ryan_weal, @kristen_pol, @emma_maria88, @opdavies, and @davidnarrabilis for helping me find more events.

Drupal PlanetDrupalSprints
Categories: Elsewhere

InternetDevels: DrupalTour siteseeing

Planet Drupal - Mon, 22/06/2015 - 15:38

If you want pizza you can either go to cafe or order delivery, right? So why should Drupal be different? :) We made Drupal delivery possible to any Ukrainian city with DrupalTour!

Read more
Categories: Elsewhere

Niels Thykier: Introducing dak auto-decruft

Planet Debian - Mon, 22/06/2015 - 15:11

Debian now have over 22 000 source packages and 45 500 binary packages.  To counter that, the FTP masters and I have created a dak tool to automatically remove packages from unstable!  This is also much more efficient than only removing them from testing! :)


The primary goal of the auto-decrufter is to remove a regular manual work flow from the FTP masters.  Namely, the removal of the common cases of cruft, such as “Not Built from Source” (NBS) and “Newer Version In Unstable” (NVIU).  With the auto-decrufter in place, such cruft will be automatically removed when there are no reverse dependencies left on any architecture and nothing Build-Depends on it any more.

Despite the implication in the “opening” of this post, this will in fact not substantially reduce the numbers of packages in unstable. :) Nevertheless, it is still very useful for the FTP masters, the release team and packaging Debian contributors.

The reason why the release team benefits greatly from this tool, is that almost every transition generates one piece of “NBS”-cruft.  Said piece of cruft currently must be  removed from unstable before the transition can progress into its final phase.  Until recently that removal has been 100% manual and done by the FTP masters.

The restrictions on auto-decrufter means that we will still need manual decrufts. Notably, the release team will often complete transitions even when some reverse dependencies remain on non-release architectures.  Nevertheless, it is definitely an improvement.


Omelettes and eggs: As an old saying goes “You cannot make an omelette without breaking eggs”.  Less so when the only “test suite” is production.  So here are some of the “broken eggs” caused by implementation of the auto-decrufter:

  • About 30 minutes of “dak rm” (without –no-action) would unconditionally crash.
  • A broken dinstall when “dak auto-decruft” was run without “–dry-run” for the first time.
  • A boolean condition inversion causing removals to remove the “override” for partial removals (and retain it for “full” removals).
    • Side-effect, this broke Britney a couple of times because dak now produced some “unexpected” Packages files for unstable.
  • Not to mention the “single digit bug closure” bug.

Of the 3, the boolean inversion was no doubt the worst.  By the time we had it fixed, at least 50 (unique) binary packages had lost their “override”.  Fortunately, it was possible to locate these issues using a database query and they have now been fixed.

Before I write any more non-trivial patches for dak, I will probably invest some time setting up a basic test framework for dak first.


Filed under: Debian, Release-Team
Categories: Elsewhere

Lunar: Reproducible builds: week 8 in Stretch cycle

Planet Debian - Mon, 22/06/2015 - 14:48

What happened about the reproducible builds effort this week:

Toolchain fixes

Andreas Henriksson has improved Johannes Schauer initial patch for pbuilder adding support for build profiles.

Packages fixed

The following 12 packages became reproducible due to changes in their build dependencies: collabtive, eric, file-rc, form-history-control, freehep-chartableconverter-plugin , jenkins-winstone, junit, librelaxng-datatype-java, libwildmagic, lightbeam, puppet-lint, tabble.

The following packages became reproducible after getting fixed:

Some uploads fixed some reproducibility issues but not all of them:

Patches submitted which have not made their way to the archive yet:

  • #788747 on 0xffff by Dhole: allow embedded timestamp to be set externally and set it to the time of the debian/changelog.
  • #788752 on analog by Dhole: allow embedded timestamp to be set externally and set it to the time of the debian/changelog.
  • #788757 on jacktrip by akira: remove $datetime from the documentation footer.
  • #788868 on apophenia by akira: remove $date from the documentation footer.
  • #788920 on orthanc by akira: set HTML_TIMESTAMP=NO in Doxygen configuration.
  • #788955 on rivet by akira: set HTML_TIMESTAMP=NO in Doxygen configuration.
  • #789040 on liblo by akira: set HTML_TIMESTAMP=NO in Doxygen configuration.
  • #789049 on mpqc by akira: remove $datetime from the documentation footer.
  • #789071 on libxkbcommon by akira: set HTML_TIMESTAMP=NO in Doxygen configuration.
  • #789073 on libxr by akira: remove $datetime from the documentation footer.
  • #789076 on lvtk by akira: set HTML_TIMESTAMP=NO in Doxygen configuration.
  • #789087 on lmdb by akira: pass HTML_TIMESTAMP=NO to Doxygen.
  • #789184 on openigtlink by akira: remove $datetime from the documentation footer.
  • #789264 on openscenegraph by akira: pass HTML_TIMESTAMP=NO to Doxygen.
  • #789308 on trigger-rally-data by Mattia Rizzolo: call dh_fixperms even when overriding dh_fixperms.
  • #789396 on libsidplayfp by akira: set HTML_TIMESTAMP=NO in Doxygen configuration.
  • #789399 on psocksxx by akira: set HTML_TIMESTAMP=NO in Doxygen configuration.
  • #789405 on qdjango by akira: set HTML_TIMESTAMP=NO in Doxygen configuration.
  • #789406 on qof by akira: set HTML_TIMESTAMP=NO in Doxygen configuration.
  • #789428 on qsapecng by akira: pass HTML_TIMESTAMP=NO to Doxygen.

Bugs with the ftbfs usertag are now visible on the bug graphs. This explain the recent spike. (h01ger)

Andreas Beckmann suggested a way to test building packages using the “funny paths” that one can get when they contain the full Debian package version string.

debbindiff development

Lunar started an important refactoring introducing abstactions for containers and files in order to make file type identification more flexible, enabling fuzzy matching, and allowing parallel processing.

Documentation update

Ximin Luo detailed the proposal to standardize environment variables to pass a reference source date to tools that needs one (e.g. documentation generator).

Package reviews

41 obsolete reviews have been removed, 168 added and 36 updated this week.

Some more issues affecting packages failing to build from source have been identified.


Minutes have been posted for Tuesday June 16th meeting.

The next meeting is scheduled Tuesday June 23rd at 17:00 UTC.


Lunar presented the project in French during Pas Sage en Seine in Paris. Video and slides are available.

Categories: Elsewhere

Annertech: Tough at the top (of Google) - Content Strategy, SEO, and Low Bounce Rates

Planet Drupal - Mon, 22/06/2015 - 12:44
Tough at the top (of Google) - Content Strategy, SEO, and Low Bounce Rates

Annertech is #1 on Google for a number of key search phrases and when we're not, we're usually only beaten by the Drupal Ireland page from g.d.o. (groups.drupal.org/ireland). How did we get to the top of Google? How do we stay there? Two words: hard work - but it really revolves around two other words: content strategy. Let's get down to the details.

Categories: Elsewhere

LevelTen Interactive: DrupalCon LA 2015 Video: Mediacurrent Interview

Planet Drupal - Mon, 22/06/2015 - 07:00

In last weeks' DrupalCon interview, we featured Percona.... Read more

Categories: Elsewhere

Midwestern Mac, LLC: DrupalCamp St. Louis 2015 finished, session videos available online!

Planet Drupal - Sun, 21/06/2015 - 20:28

DrupalCamp St. Louis 2015 was held this past weekend, June 20-21, 2015, at SLU LAW in downtown St. Louis. We had nine sessions and a great keynote on Saturday, and a full sprint day on Sunday.

The view coming off the elevators at SLU LAW.

Every session was recorded (slides + audio), and you can view all the sessions online:

The Camp went very well, with almost sixty participants this year! We had a great time, learned a lot together, and enjoyed some great views of downtown St. Louis (check out the picture below!), and we can't wait until next year's DrupalCamp St. Louis (to be announced)!

Categories: Elsewhere

Wim Leers: Eaton & Urbina: structured, intelligent and adaptive content

Planet Drupal - Sun, 21/06/2015 - 20:08

While walking, I started listening to Jeff Eaton’s Insert Content Here podcast episode 25: Noz Urbina Explains Adaptive Content. People must’ve looked strangely at me because I was smiling and nodding — while walking :) Thanks Jeff & Noz!

Jeff Eaton explained how the web world looks at and defines the term WYSIWYG. Turns out that in the semi-structured, non-web world that Noz comes from, WYSIWYG has a totally different interpretation. And they ended renaming it to what it really was: WYSIWOO.

Jeff also asked Noz what “adaptive content” is exactly. Adaptive content is a more specialized/advanced form of structured content, and in fact “structured content”, “intelligent content” and “adaptive content” form a hierarchy:

  • structured content
    • intelligent content
      • adaptive content

In other words, adaptive content is also intelligent and structured; intelligent content is also structured, but not all structured content is also intelligent or adaptive, nor is all intelligent content also adaptive.

Basically, intelligent content better captures the precise semantics (e.g. not a section, but a product description). Adaptive content is about using those semantics, plus additional metadata (“hints”) that content editors specify, to adapt the content to the context it is being viewed in. E.g. different messaging for authenticated versus anonymous users, or different nuances depending on how the visitor ended up on the current page (in other words: personalization).

Noz gave an excellent example of how adaptive content can be put to good use: he described how we he had arrived in Utrecht in the Netherlands after a long flight, “checked in” to Utrecht on Facebook, and then Facebook suggested to him 3 open restaurants, including cuisine type and walking distance relative to his current position. He felt like thanking Facebook for these ads — which obviously is a rare thing, to be grateful for ads!

Finally, a wonderful quote from Noz Urbina that captures the essence of content modeling:

How descriptive do we make it without making it restrictive?

If it isn’t clear by now — go listen to that podcast! It’s well worth the 38 minutes of listening. I only captured a few of the interesting points, to get more people interested and excited.1

What about adaptive & intelligent content in Drupal 8?

First, see my closely related article Drupal 8: best authoring experience for structured content?.

Second, while listening, I thought of many ways that Drupal 8 is well-prepared for intelligent & adaptive content. (Drupal already does structured content by means of Field API and the HTML tag restrictions in the body field.) Implementing intelligent & adaptive will surely require experimentation, and different sites/use cases will prefer different solutions, but:

  • An intelligent_content module for Drupal 8: allow site builders/content strategists to define custom HTML tags (e.g. <product_description>) to capture site-specific semantics. A CKEditor Widget could hugely simplify the authoring experience for creating intelligent content, by showing a specific HTML representation while editing (WYSIWOO!), thanks to HTML (Twig) templates associated with those custom HTML tags.
  • An adaptive_content module for Drupal 8: a text filter that allows any tag to be wrapped in a <adaptive_content> tag, which specifies the context in which the wrapped content should be shown/hidden.
  • The latter leads to cacheability problems, because the same content may be rendered in a multitude of different ways, but thanks to cache contexts in Drupal 8 and the fact that text filters can specify cache contexts means adaptive content that is still cacheable is perfectly possible. (This is in fact exactly what it was intended for!) cache contexts

I think that those two modules would be very interesting, useful additions to the Drupal ecosystem. If you are working on this, please let me know — I would love to help!

  1. That’s right, this is basically voluntary marketing for Jeff Eaton — you’re welcome, Jeff! 

  • Drupal
  • structured content
Categories: Elsewhere

Enrico Zini: debtags-rewrite-python3

Planet Debian - Sun, 21/06/2015 - 18:04
debtags rewritten in python3

In my long quest towards closing #540218, I have uploaded a new libept to experimental. Then I tried to build debtags on a sid+experimental chroot and the result runs but has libc's free() print existential warnings about whatevers.

At a quick glance, there are now things around like a new libapt, gcc 5 with ABI changes, and who knows what else. I figured how much time it'd take me to debug something like that, and I've used that time to rewrite debtags in python3. It took 8 hours, 5 of pleasant programming and the usual tax of another 3 of utter frustration packaging the results. I guess I gained over the risk of spending an unspecified amount of hours of just pure frustration.

So from now on debtags is going to be a pure python3 package, with dependencies on only python3-apt and python3-debian. 700 lines of python instead of several C++ files built on 4 layers of libraries. Hopefully, this is the last of the big headaches I get from hacking on this package. Also, one less package using libept.

Categories: Elsewhere

Blue Drop Shop: Camp Record Beta Test Four: DrupalCamp STL 2015

Planet Drupal - Sun, 21/06/2015 - 17:58

Following a successful MidCamp and with some new ideas how to improve the kit, I was eager to hit the road for more testing. Problem is, I'm a freelancer with a limited budget, and getting to camps comes out of my own pocket. On a lark, I tweeted the following:

Planning a #drupalcamp and need your sessions recorded? Sponsor me & I will record your sessions. Ping me! #drupal /cc @drupalstl @tcdrupal

— Kevin Thull (@kevinjthull) April 8, 2015

To my delight, both Twin Cities and St. Louis camps took me up on my offer. Of course, the stakes are even higher now, because it's no longer my own money on the line.

But I'm also feeling more confident about this solution and improve on the process with each camp. Connecting to non-HDMI-capable laptops remains the biggest challenge overall. I've added in a couple (full) DisplayPort to HDMI converters and even successfully tested a new VGA to HDMI converter that got my ancient Sony VAIO to display on my home flatscreen:

The new VGA to HDMI converter shows promise. My ancient Sony Vaio WinXP laptop just connected! #drupalcamp pic.twitter.com/PXb0kBvsCl

— Kevin Thull (@kevinjthull) June 16, 2015

And at DrupalCamp STL I finally got the 100% success rate that I've been shooting for! Three sessions needed fixing in post, but overall, this camp went very smoothly. A huge bonus was the fact that the two rooms were next to each other, minimizing the distance to cover when trying to coordinate laptop hookups and verify timely starts and stops of the records.

Twin Cities is next week, with a much more challenging schedule: five concurrent sessions across two buildings and multiple floors. My Fitbit will likely hit a new high. That, and I need to finally get down to some documentation and podium signage. It's time to share the knowledge I've gained and get more hands and minds involved.

And now for the learnings from DCSTL:

  • swapping thumb drives throughout the day means recordings can be posted during camp
  • well-timed presenter starts/stops means no trimming, which means more recordings can be posted during camp
  • one room had screen flicker and setting the PVR resolution to 1080 helped (typically, the resolution needs to come down to 720 for this, as well as fixing color shifts)
  • having extra SD cards means bad audio can be fixed during down times, which means more recordings can be posted during camp
  • power strips at the podium shouldn't be assumed, and the powered USB hub and voice recorder both have short plugs
  • never plug the powered usb into the laptop, because that can kill your record if resolution changes or the laptop goes to sleep
  • taping down individual components means less cord chaos throughout the day
  • access to ethernet port with a reasonably large pipe going up will get videos posted faster
Categories: Elsewhere

Steve Kemp: We're all about storing objects

Planet Debian - Sun, 21/06/2015 - 02:00

Recently I've been experimenting with camlistore, which is yet another object storage system.

Camlistore gains immediate points because it is written in Go, and is a project initiated by Brad Fitzpatrick, the creator of Perlbal, memcached, and Livejournal of course.

Camlistore is designed exactly how I'd like to see an object storage-system - each server allows you to:

  • Upload a chunk of data, getting an ID in return.
  • Download a chunk of data, by ID.
  • Iterate over all available IDs.

It should be noted more is possible, there's a pretty web UI for example, but I'm simplifying. Do your own homework :)

With those primitives you can allow a client-library to upload a file once, then in the background a bunch of dumb servers can decide amongst themselves "Hey I have data with ID:33333 - Do you?". If nobody else does they can upload a second copy.

In short this kind of system allows the replication to be decoupled from the storage. The obvious risk is obvious though: if you upload a file the chunks might live on a host that dies 20 minutes later, just before the content was replicated. That risk is minimal, but valid.

There is also the risk that sudden rashes of uploads leave the system consuming all the internal-bandwith constantly comparing chunk-IDs, trying to see if data is replaced that has been copied numerous times in the past, or trying to play "catch-up" if the new-content is larger than the replica-bandwidth. I guess it should possible to detect those conditions, but they're things to be concerned about.

Anyway the biggest downside with camlistore is documentation about rebalancing, replication, or anything other than simple single-server setups. Some people have blogged about it, and I got it working between two nodes, but I didn't feel confident it was as robust as I wanted it to be.

I have a strong belief that Camlistore will become a project of joy and wonder, but it isn't quite there yet. I certainly don't want to stop watching it :)

On to the more personal .. I'm all about the object storage these days. Right now most of my objects are packed in a collection of boxes. On the 6th of next month a shipping container will come pick them up and take them to Finland.

For pretty much 20 days in a row we've been taking things to the skip, or the local charity-shops. I expect that by the time we've relocated the amount of possesions we'll maintain will be at least a fifth of our current levels.

We're working on the general rule of thumb: "If it is possible to replace an item we will not take it". That means chess-sets, mirrors, etc, will not be carried. DVDs, for example, have been slashed brutally such that we're only transferring 40 out of a starting collection of 500+.

Only personal, one-off, unique, or "significant" items will be transported. This includes things like personal photographs, family items, and similar. Clothes? Well I need to take one jacket, but more can be bought. The only place I put my foot down was books. Yes I'm a kindle-user these days, but I spent many years tracking down some rare volumes, and though it would be possible to repeat that effort I just don't want to.

I've also decided that I'm carrying my complete toolbox. Some of the tools I took with me when I left home at 18 have stayed with me for the past 20+ years. I don't need this specific crowbar, or axe, but I'm damned if I'm going to lose them now. So they stay. Object storage - some objects are more important than they should be!

Categories: Elsewhere

Joachim Breitner: Running circle-packing in the Browser, now using GHCJS

Planet Debian - Sat, 20/06/2015 - 22:50

Quite a while ago, I wrote a small Haskell library called circle-packing to pack circles in a tight arrangement. Back then, I used the Haskell to JavaScript compiler fay to create a pretty online demo of that library, and shortly after, I create the identical demo using haste (another Haskell to JavaScript compiler).

The main competitor of these two compilers, and the most promising one, is GHCJS. Back then, it was too annoying to install. But after two years, things have changed, and it only takes a few simple commands to get GHCJS running, so I finally created the circle packing demo in a GHCJS variant.

Quick summary: Cabal integration is very good (like haste, but unline fay), interfacing JavaScript is nice and easy (like fay, but unlike haste), and a quick check seems to indicate that it is faster than either of these two. I should note that I did not update the other two demos, so they represent the state of fay and haste back then, respectively.

With GHCJS now available at my fingertips, maybe I will produce some more Haskell to be run in your browser. For example, I could port FrakView, a GUI program to render, expore and explain iterated function systems, from GTK to HTML.

Categories: Elsewhere

Russell Coker: BTRFS Status June 2015

Planet Debian - Sat, 20/06/2015 - 06:47

The version of btrfs-tools in Debian/Jessie is incapable of creating a filesystem that can be mounted by the kernel in Debian/Wheezy. If you want to use a BTRFS filesystem on Jessie and Wheezy (which isn’t uncommon with removable devices) the only options are to use the Wheezy version of mkfs.btrfs or to use a Jessie kernel on Wheezy. I recently got bitten by this issue when I created a BTRFS filesystem on a removable device with a lot of important data (which is why I wanted metadata duplication and checksums) and had to read it on a server running Wheezy. Fortunately KVM in Wheezy works really well so I created a virtual machine to read the disk. Setting up a new KVM isn’t that difficult, but it’s not something I want to do while a client is anxiously waiting for their data.

BTRFS has been working well for me apart from the Jessie/Wheezy compatability issue (which was an annoyance but didn’t stop me doing what I wanted). I haven’t written a BTRFS status report for a while because everything has been OK and there has been nothing exciting to report.

I regularly get errors from the cron jobs that run a balance supposedly running out of free space. I have the cron jobs due to past problems with BTRFS running out of metadata space. In spite of the jobs often failing the systems keep working so I’m not too worried at the moment. I think this is a bug, but there are many more important bugs.

Linux kernel version 3.19 was the first version to have working support for RAID-5 recovery. This means version 3.19 was the first version to have usable RAID-5 (I think there is no point even having RAID-5 without recovery). It wouldn’t be prudent to trust your important data to a new feature in a filesystem. So at this stage if I needed a very large scratch space then BTRFS RAID-5 might be a viable option but for anything else I wouldn’t use it. BTRFS still has had little performance optimisation, while this doesn’t matter much for SSD and for single-disk filesystems for a RAID-5 of hard drives that would probably hurt a lot. Maybe BTRFS RAID-5 would be good for a scratch array of SSDs. The reports of problems with RAID-5 don’t surprise me at all.

I have a BTRFS RAID-1 filesystem on 2*4TB disks which is giving poor performance on metadata, simple operations like “ls -l” on a directory with ~200 subdirectories takes many seconds to run. I suspect that part of the problem is due to the filesystem being written by cron jobs with files accumulating over more than a year. The “btrfs filesystem” command (see btrfs-filesystem(8)) allows defragmenting files and directory trees, but unfortunately it doesn’t support recursively defragmenting directories but not files. I really wish there was a way to get BTRFS to put all metadata on SSD and all data on hard drives. Sander suggested the following command to defragment directories on the BTRFS mailing list:

find / -xdev -type d -execdir btrfs filesystem defrag -c {} +

Below is the output of “zfs list -t snapshot” on a server I run, it’s often handy to know how much space is used by snapshots, but unfortunately BTRFS has no support for this.

NAME USED AVAIL REFER MOUNTPOINT hetz0/be0-mail@2015-03-10 2.88G – 387G – hetz0/be0-mail@2015-03-11 1.12G – 388G – hetz0/be0-mail@2015-03-12 1.11G – 388G – hetz0/be0-mail@2015-03-13 1.19G – 388G –

Hugo pointed out on the BTRFS mailing list that the following command will give the amount of space used for snapshots. $SNAPSHOT is the name of a snapshot and $LASTGEN is the generation number of the previous snapshot you want to compare with.

btrfs subvolume find-new $SNAPSHOT $LASTGEN | awk '{total = total + $7}END{print total}'

One upside of the BTRFS implementation in this regard is that the above btrfs command without being piped through awk shows you the names of files that are being written and the amounts of data written to them. Through casually examining this output I discovered that the most written files in my home directory were under the “.cache” directory (which wasn’t exactly a surprise).

Now I am configuring workstations with a separate subvolume for ~/.cache for the main user. This means that ~/.cache changes don’t get stored in the hourly snapshots and less disk space is used for snapshots.


My observation is that things are going quite well with BTRFS. It’s more than 6 months since I had a noteworthy problem which is pretty good for a filesystem that’s still under active development. But there are still many systems I run which could benefit from the data integrity features of ZFS and BTRFS that don’t have the resources to run ZFS and need more reliability than I can expect from an unattended BTRFS system.

At this time the only servers I run with BTRFS are located within a reasonable drive from my home (not the servers in Germany and the US) and are easily accessible (not the embedded systems). ZFS is working well for some of the servers in Germany. Eventually I’ll probably run ZFS on all the hosted servers in Germany and the US, I expect that will happen before I’m comfortable running BTRFS on such systems. For the embedded systems I will just take the risk of data loss/corruption for the next few years.

Related posts:

  1. BTRFS Status Dec 2014 My last problem with BTRFS was in August [1]. BTRFS...
  2. BTRFS Status March 2014 I’m currently using BTRFS on most systems that I can...
  3. BTRFS Status July 2014 My last BTRFS status report was in April [1], it...
Categories: Elsewhere


Subscribe to jfhovinne aggregator - Elsewhere