Elsewhere

Roy Scholten: UX meeting recap

Planet Drupal - Tue, 23/08/2016 - 23:58

Two meetings every week since end of march this year. Safe to say we’ve found a consistent rhythm.

And it’s working. It’s become a useful way to check in on current priorities, review patches while screensharing (visuals, transitions, flows!), decide on next steps and generally discuss hard problems without having to type so much.

The agenda for today was
  1. Roadmap – The core roadmap was updated to current state of things. Use this page to get a bird’s eye view on where Drupal core is moving towards.
  2. Ideation process – The first part of going from idea to plan got positive feedback and helpful questions and suggestions. Maybe a few words from actual maintainers and then we can make it so.
  3. Status page – We have a beautiful new design that now needs review and more code to create a complete patch. If you know your core markup and CSS, go have a look!
  4. Block place design update – An issue that iterates the design of an initial commit. We quickly discussed what the feedback should be and I added a comment to that effect right then and there. It’s an example of managing scope, pushing to focus on the actual fix first, and defer new, potentially good ideas to a followup discussion.
  5. Sample content – Kevin shared his initial thoughts and ideas for adding sample content, structure and configuration to core. More questions then answers but that’s just where we’re at for now. You are welcome to add your questions and ideas, please do.

As always, most welcome to join if you’re interested. Find us in the ux channel of the Drupal Slack room, or follow @drupalux on the twitters.

Tags: drupaluxuxmeetingdrupalplanetSub title: Topics du jour, how we work, where you can help, the usual
Categories: Elsewhere

Drupal Blog: Drupal 8.2, now with more outside-in

Planet Drupal - Tue, 23/08/2016 - 21:14

Over the weekend, Drupal 8.2 beta was released. One of the reasons why I'm so excited about this release is that it ships with "more outside-in". In an "outside-in experience", you can click anything on the page, edit its configuration in place without having to navigate to the administration back end, and watch it take effect immediately. This kind of on-the-fly editorial experience could be a game changer for Drupal's usability.

When I last discussed turning Drupal outside-in, we were still in the conceptual stages, with mockups illustrating the concepts. Since then, those designs have gone through multiple rounds of feedback from Drupal's usability team and a round of user testing led by Cheppers. This study identified some issues and provided some insights which were incorporated into subsequent designs.

Two policy changes we introduced in Drupal 8—semantic versioning and experimental modules—have fundamentally changed Drupal's innovation model starting with Drupal 8. I should write a longer blog post about this, but the net result of those two changes is ongoing improvements with an easy upgrade path. In this case, it enabled us to add outside-in experiences to Drupal 8.2 instead of having to wait for Drupal 9. The authoring experience improvements we made in Drupal 8 are well-received, but that doesn't mean we are done. It's exciting that we can move much faster on making Drupal easier to use.

In-place block configuration

As you can see from the image below, Drupal 8.2 adds the ability to trigger "Edit" mode, which currently highlights all blocks on the page. Clicking on one — in this case, the block with the site's name — pops out a new tray or sidebar. A content creator can change the site name directly from the tray, without having to navigate through Drupal's administrative interface to theme settings as they would have to in Drupal 7 and Drupal 8.1.

Making adjustments to menus

In the second image, the pattern is applied to a menu block. You can make adjustments to the menu right from the new tray instead of having to navigate to the back end. Here the content creator changes the order of the menu links (moving "About us" after "Contact") and toggles the "Team" menu item from hidden to visible.

In-context block placement

In Drupal 8.1 and prior, placing a new block on the page required navigating away from your front end into the administrative back end and noting the available regions. Once you discover where to go to add a block, which can in itself be a challenge, you'll have to learn about the different regions, and some trial and error might be required to place a block exactly where you want it to go.

Starting in Drupal 8.2, content creators can now just click "Place block" without navigating to a different page and knowing about available regions ahead of time. Clicking "Place block" will highlight the different possible locations for a block to be placed in.

Next steps

These improvements are currently tagged "experimental". This means that anyone who downloads Drupal 8.2 can test these changes and provide feedback. It also means that we aren't quite satisfied with these changes yet and that you should expect to see this functionality improve between now and 8.2.0's release, and even after the Drupal 8.2.0 release.

As you probably noticed, things still look pretty raw in places; as an example, the forms in the tray are exposing too many visual details. There is more work to do to bring this functionality to the level of the designs. We're focused on improving that, as well as the underlying architecture and accessibility. Once we feel good about how it all works and looks, we'll remove the experimental label.

We deliberately postponed most of the design work to focus on introducing the fundamental concepts and patterns. That was an important first step. We wanted to enable Drupal developers to start experimenting with the outside-in pattern in Drupal 8.2. As part of that, we'll have to determine how this new pattern will apply broadly to Drupal core and the many contributed modules that would leverage it. Our hope is that once the outside-in work is stable and no longer experimental, it will trickle down to every Drupal module. At that point we can all work together, in parallel, on making Drupal much easier to use.

Users have proven time and again in usability studies to be extremely "preview-driven", so the ability to make quick configuration changes right from their front end, without becoming an expert in Drupal's information architecture, could be revolutionary for Drupal.

If you'd like to help get these features to stable release faster, please join us in the outside-in roadmap issue.

Thank you

I'd also like to thank everyone who contributed to these features and reviewed them, including Bojhanyoroypwolaninandrewmacphersongtamaspetycompzsofimajor,SKAUGHTnod_effulgentsiaWim Leerscatchalexpott, and xjm.

And finally, a special thank you to Acquia's outside-in team for driving most of the design and implementation: tkolearywebchicktedbowGábor Hojtsytim.plunkett, and drpal.

Acquia's outside-in team celebrating that the outside-in patch was committed to Drupal 8.2 beta. Go team!

Categories: Elsewhere

Reproducible builds folks: Reproducible Builds: week 69 in Stretch cycle

Planet Debian - Tue, 23/08/2016 - 14:56

What happened in the Reproducible Builds effort between Sunday August 14 and Saturday August 20 2016:

Fasten your seatbelts

Important note: we enabled build path variation for unstable now, so your package(s) might become unreproducible, while previously it was said to be reproducible… given a specific build path it probably still is reproducible but read on for the details below in the tests.reproducible-builds.org section! As said many times: this is still research and we are working to make it reality.

Media coverage

Daniel Stender blogged about python packaging and explained some caveats regarding reproducible builds.

Toolchain developments

Thomas Schmitt uploaded xorriso which now obeys SOURCE_DATE_EPOCH. As stated in its man pages:

ENVIRONMENT [...] SOURCE_DATE_EPOCH belongs to the specs of reproducible-builds.org. It is supposed to be either undefined or to contain a decimal number which tells the seconds since january 1st 1970. If it contains a number, then it is used as time value to set the default of --modification-date=, --gpt_disk_guid, and --set_all_file_dates. Startup files and program options can override the effect of SOURCE_DATE_EPOCH. Packages reviewed and fixed, and bugs filed

The following packages have become reproducible after being fixed:

The following updated packages appear to be reproducible now, for reasons we were not able to figure out. (Relevant changelogs did not mention reproducible builds.)

  • vulkan/1.0.21.0+dfsg1-1 by Timo Aaltonen.

The following 2 packages were not changed, but have become reproducible due to changes in their build-dependencies: tagsoup tclx8.4.

Some uploads have addressed some reproducibility issues, but not all of them:

Patches submitted that have not made their way to the archive yet:

Bug tracker house keeping:

  • Chris Lamb pinged 164 bugs he filed more than 90 days ago which have a patch and had no maintainer reaction.
Reviews of unreproducible packages

55 package reviews have been added, 161 have been updated and 136 have been removed in this week, adding to our knowledge about identified issues.

2 issue types have been updated:

Weekly QA work

FTBFS bugs have been reported by:

  • Chris Lamb (16)
  • Santiago Vila (2)
diffoscope development

Chris Lamb, Holger Levsen and Mattia Rizzolo worked on diffoscope this week.

Improvements were made to SquashFS and JSON comparison, the https://try.diffoscope.org/ web service, documentation, packaging, and general code quality.

diffoscope 57, 58, and 59 were uploaded to unstable by Chris Lamb. Versions 57 and 58 were both broken, so Holger set up a job on jenkins.debian.net to test diffoscope on each git commit. He also wrote a CONTRIBUTING document to help prevent this from happening in future.

From these efforts, we were also able to learn that diffoscope is now reproducible even when built across multiple architectures:

< h01ger> | https://tests.reproducible-builds.org/debian/rb-pkg/unstable/amd64/diffoscope.html shows these packages were built on amd64: < h01ger> | bd21db708fe91c01ba1c9cb35b9d41a7c9b0db2b 62288 diffoscope_59_all.deb < h01ger> | 366200bf2841136a4c8f8c30bdc87057d59a4cdd 20146 trydiffoscope_59_all.deb < h01ger> | and on i386: < h01ger> | bd21db708fe91c01ba1c9cb35b9d41a7c9b0db2b 62288 diffoscope_59_all.deb < h01ger> | 366200bf2841136a4c8f8c30bdc87057d59a4cdd 20146 trydiffoscope_59_all.deb < h01ger> | and on armhf: < h01ger> | bd21db708fe91c01ba1c9cb35b9d41a7c9b0db2b 62288 diffoscope_59_all.deb < h01ger> | 366200bf2841136a4c8f8c30bdc87057d59a4cdd 20146 trydiffoscope_59_all.deb

And those also match the binaries uploaded by Chris in his diffoscope 59 binary upload to ftp.debian.org, yay! Eating our own dogfood and enjoying it!

tests.reproducible-builds.org

Debian related:

  • show percentage of results in the last 24/48h (h01ger)
  • switch python database backend to SQLAlchemy (Valerie)
  • vary build path varitation for unstable and experimental for all architectures. (h01ger)

The last change probably will have an impact you will see: your package might become unreproducible in unstable and this will be shown on tracker.debian.org, while it will still be reproducible in testing.

We've done this, because we think reproducible builds are possible with arbitrary build paths. But: we don't think those are a realistic goal for stretch, where we still recommend to use ´.buildinfo´ to record the build patch and then do rebuilds using that path.

We are doing this, because besides doing theoretical groundwork we also have a practical goal: enable users to independently verify builds. And if they only can do this with a fixed path, so be it. For now

To be clear: for Stretch we recommend that reproducible builds are done in the same build path as the "original" build.

Finally, and just for our future references, when we enabled build path variation on Saturday, August 20th 2016, the numbers for unstable were:

suite all reproducible unreproducible ftbfs depwait not for this arch blacklisted unstable/amd64 24693 21794 (88.2%) 1753 (7.1%) 972 (3.9%) 65 (0.2%) 95 (0.3%) 10 (0.0%) unstable/i386 24693 21182 (85.7%) 2349 (9.5%) 972 (3.9%) 76 (0.3%) 103 (0.4%) 10 (0.0%) unstable/armhf 24693 20889 (84.6%) 2050 (8.3%) 1126 (4.5%) 199 (0.8%) 296 (1.1%) 129 (0.5%) Misc.

Ximin Luo updated our git setup scripts to make it easier for people to write proper descriptions for our repositories.

This week's edition was written by Ximin Luo and Holger Levsen and reviewed by a bunch of Reproducible Builds folks on IRC.

Categories: Elsewhere

myDropWizard.com: Survey Results From "Is Drupal Hard?"

Planet Drupal - Tue, 23/08/2016 - 14:51

A few weeks ago, we had A Survey! Is Drupal Hard?

First of all, thank you for taking the time to answer (even though we had a short-lived technical snafu!). At myDropWizard, we believe in transparency and openness, so I'm going to share the unfiltered data with you - as well as what my thoughts are in interpreting this non-scientific study.

The Data

Some Issues

  1. Many of those answering would be biased towards Drupal
  2. The choices were somewhat arbitrary, and no doubt some were left out
  3. There was an initial bug in the survey keeping some good data out
  4. The choice "It's about what I would expect" is probably better labeled as something more along the lines of "an average web developer should be able to accomplish their needs in a reasonable amount of time"
Analysis

So, there were plenty of flaws in the data, but that won't keep me from drawing a few conclusions - and I'm anxious to hear what others feel as well.

Drupal 7 Still a Favorite

"0" respondents claimed it "impossible" to do complex things while coupled with the lowest number of "Never Used It / No Opinion" at 5.

Categories: Elsewhere

Nuvole: Maintain local development configuration with Configuration Split

Planet Drupal - Tue, 23/08/2016 - 14:21
Answering the most commonly asked question about Configuration Management in Drupal 8; and a preview of our DrupalCon training.

This post is an excerpt from the topics covered by our DrupalCon Dublin training: Drupal 8 Development - Workflows and Tools.

For the last two years we have been giving trainings and presentations at various Drupal events about configuration management and its new workflows in Drupal 8. One of the recurring questions has been:

How do I keep some configuration from being deployed? For example the devel module and its configuration?

Until now we have answered to use drush with the --skip-modules flag and then to gitignore the configuration for devel. But then you can't share the development configuration and the process is error prone when there are more modules and other configuration items that depend on the configuration you gitignore.

For some simpler cases where configuration needs to be different between environments (for example the error verbosity) the configuration override system allows to override the configuration in settings.php. This is a solution for some cases, however, you can not override which modules are enabled or override configuration that doesn't exist.

Enter: Configuration split!

Our new module Configuration split splits the configuration when exporting it with a special Drupal Console command. It can be configured to split out enabled modules and given blacklisted configuration and it will then separate all the configuration that is dependent on the listed modules or configuration. The module's settings also allow to set a folder to which the separated configuration will be exported to.

That way the configuration set which you use to deploy configuration between different environments is a subset of your development configuration and the trusty configuration system of Drupal 8 can be used unharmed. Of course when importing with the special command the split configuration is merged back, allowing you to keep your development configuration in place.

"To split" is a synonym for "to break" and as such "Configuration split" has a dangerous ring to it. This is on purpose because the exported subset is on purpose not what you have on your development site and not what you have locally tested. So you need to compensate that with a workflow that imports and verifies the configuration you are going to deploy. This is better than to import and export individual configuration because Drupal needs the whole set of configuration to do its checks.

How do I use it?

Download and enable the Configuration split module like any other module. Then configure it under admin/config/development/configuration/config_split. Set the split folder to a different folder than your normal config sync folder. If you want a prettier interface, consider using Chosen. Then use the Drupal Console command config_split:export and config_split:import to export and import respectively.

Let's look at the devel example from above. We typically version the configuration outside of the webroot in ../config/sync as seen by Drupal. (In the project root when starting a project with drupal-composer/drupal-project.) So the folder we specify for config_split would be ../config/dev. The module to filter would be Devel and the rest can be left empty. The devel.settings will be detected automatically. Note though that the system.menu.devel does not depend on the devel module and can not be detected automatically, but it is easy to add it to the blacklisted config and it could theoretically be deployed without breaking anything.

The config_split.settings.yml could look something like this:

folder: ../config/dev
module:
  devel: 0
theme: {  }
blacklist:
  - system.menu.devel

Finally the following command will export the configuration to the default sync directory without the devel module enabled and export the devel configuration into the dev directory.
web$ ../vendor/bin/drupal config_split:export

Now if you would import the configuration through the UI or with drush cim the devel module would be un-installed, and you can do that to see the site without its development configuration. However, If you want the development configuration to stay or become active and the devel module installed use the following command:
web$ ../vendor/bin/drupal config_split:import

How does it work internally?

We implemented a StorageWrapper that allows filters to interact with the configuration after it has been read and before it is written to the wrapped FileStorage during the import and export operation. The SplitFilter has a secondary storage and decides where to read from or write to. This is a very similar concept to what drush does with its --skip-modules flag since we will want to easily integrate this with drush in the future.

What comes next?

The module is still in an early development stage and some more additions in the scope of splitting configuration could be added. The subset without the split configuration could be verified after exporting it and we could warn the user if it couldn't be imported. Or for example we currently use only config_split.settings but if the need arises we could support multiple split configurations. Or we could add a "gray list" to ignore some configuration that exists rather than removing it when splitting, essentially making it a configuration override outside of the scope of Drupal's runtime. This could be useful when maintaining several sites that are almost the same but all have their little "special snowflake" configuration which in turn could be synchronized with the normal workflow.

It is important to understand that the configuration system of Drupal has limitations that are there for a good reason. Most of them are to ensure data integrity and robustness and reliability of the synchronisation and so on. In other words measures to protect you from accidentally breaking your site. Using tools like config_split or drush --skip-modules you circumvent some of these security and integrity checks so use them with caution.

Since the module is not required for the regular functioning of a Drupal 8 site (it can even be used to blacklist itself) it can already be used in the development process of your current projects already. Your feedback is welcome, see you in the issue queue.

Tags: Drupal PlanetDrupal 8DrupalConTrainingCode Driven Development
Categories: Elsewhere

Rakesh's DSoC 16 blog: GSoC Final Submission

Planet Drupal - Tue, 23/08/2016 - 10:59
GSoC Final Submission rakesh Tue, 08/23/2016 - 14:29
Categories: Elsewhere

Dries Buytaert: Drupal 8.2, now with more outside-in

Planet Drupal - Tue, 23/08/2016 - 09:10

Over the weekend, Drupal 8.2 beta was released. One of the reasons why I'm so excited about this release is that it ships with "more outside-in". In an "outside-in experience", you can click anything on the page, edit its configuration in place without having to navigate to the administration back end, and watch it take effect immediately. This kind of on-the-fly editorial experience could be a game changer for Drupal's usability.

When I last discussed turning Drupal outside-in, we were still in the conceptual stages, with mockups illustrating the concepts. Since then, those designs have gone through multiple rounds of feedback from Drupal's usability team and a round of user testing led by Cheppers. This study identified some issues and provided some insights which were incorporated into subsequent designs.

Two policy changes we introduced in Drupal 8 — semantic versioning and experimental modules — have fundamentally changed Drupal's innovation model starting with Drupal 8. I should write a longer blog post about this, but the net result of those two changes is ongoing improvements with an easy upgrade path. In this case, it enabled us to add outside-in experiences to Drupal 8.2 instead of having to wait for Drupal 9. The authoring experience improvements we made in Drupal 8 are well-received, but that doesn't mean we are done. It's exciting that we can move much faster on making Drupal easier to use.

In-place block configuration

As you can see from the image below, Drupal 8.2 adds the ability to trigger "Edit" mode, which currently highlights all blocks on the page. Clicking on one — in this case, the block with the site's name — pops out a new tray or sidebar. A content creator can change the site name directly from the tray, without having to navigate through Drupal's administrative interface to theme settings as they would have to in Drupal 7 and Drupal 8.1.

Making adjustments to menus

In the second image, the pattern is applied to a menu block. You can make adjustments to the menu right from the new tray instead of having to navigate to the back end. Here the content creator changes the order of the menu links (moving "About us" after "Contact") and toggles the "Team" menu item from hidden to visible.

In-context block placement

In Drupal 8.1 and prior, placing a new block on the page required navigating away from your front end into the administrative back end and noting the available regions. Once you discover where to go to add a block, which can in itself be a challenge, you'll have to learn about the different regions, and some trial and error might be required to place a block exactly where you want it to go.

Starting in Drupal 8.2, content creators can now just click "Place block" without navigating to a different page and knowing about available regions ahead of time. Clicking "Place block" will highlight the different possible locations for a block to be placed in.

Next steps

These improvements are currently tagged "experimental". This means that anyone who downloads Drupal 8.2 can test these changes and provide feedback. It also means that we aren't quite satisfied with these changes yet and that you should expect to see this functionality improve between now and 8.2.0's release, and even after the Drupal 8.2.0 release.

As you probably noticed, things still look pretty raw in places; as an example, the forms in the tray are exposing too many visual details. There is more work to do to bring this functionality to the level of the designs. We're focused on improving that, as well as the underlying architecture and accessibility. Once we feel good about how it all works and looks, we'll remove the experimental label.

We deliberately postponed most of the design work to focus on introducing the fundamental concepts and patterns. That was an important first step. We wanted to enable Drupal developers to start experimenting with the outside-in pattern in Drupal 8.2. As part of that, we'll have to determine how this new pattern will apply broadly to Drupal core and the many contributed modules that would leverage it. Our hope is that once the outside-in work is stable and no longer experimental, it will trickle down to every Drupal module. At that point we can all work together, in parallel, on making Drupal much easier to use.

Users have proven time and again in usability studies to be extremely "preview-driven", so the ability to make quick configuration changes right from their front end, without becoming an expert in Drupal's information architecture, could be revolutionary for Drupal.

If you'd like to help get these features to stable release faster, please join us in the outside-in roadmap issue.

Thank you

I'd also like to thank everyone who contributed to these features and reviewed them, including Bojhan, yoroy, pwolanin, andrewmacpherson, gtamas, petycomp, zsofimajor, SKAUGHT, nod_, effulgentsia, Wim Leers, catch, alexpott, and xjm.

And finally, a special thank you to Acquia's outside-in team for driving most of the design and implementation: tkoleary, webchick, tedbow, Gábor Hojtsy, tim.plunkett, and drpal.

Acquia's outside-in team celebrating that the outside-in patch was committed to Drupal 8.2 beta. Go team!
Categories: Elsewhere

Sylvain Le Gall: Release of OASIS 0.4.7

Planet Debian - Tue, 23/08/2016 - 01:24

I am happy to announce the release of OASIS v0.4.7.

OASIS is a tool to help OCaml developers to integrate configure, build and install systems in their projects. It should help to create standard entry points in the source code build system, allowing external tools to analyse projects easily.

This tool is freely inspired by Cabal which is the same kind of tool for Haskell.

You can find the new release here and the changelog here. More information about OASIS in general on the OASIS website.

Pull request for inclusion in OPAM is pending: https://github.com/ocaml/opam-repository/pull/7281

Here is a quick summary of the important changes:

  • Drop support for OASISFormat 0.2 and 0.1.
  • New plugin "omake" to support build, doc and install actions.
  • Improve automatic tests (Travis CI and AppVeyor)
  • Trim down the dependencies (removed ocaml-gettext, camlp4, ocaml-data-notation)

Features:

  • findlib_directory (beta): to install libraries in sub-directories of findlib.
  • findlib_extra_files (beta): to install extra files with ocamlfind.
  • source_patterns (alpha): to provide module to source file mapping.

This version contains a lot of changes and is the achievement of a huge amount of work. The addition of OMake as a plugin is a huge progress. The overall work has been targeted at making OASIS more library like. This is still a work in progress but we made some clear improvement by getting rid of various side effect (like the requirement of using "chdir" to handle the "-C", which leads to propage ~ctxt everywhere and design OASISFileSystem).

I would like to thanks again the contributor for this release: Spiros Eliopoulos, Paul Snively, Jeremie Dimino, Christopher Zimmermann, Christophe Troestler, Max Mouratov, Jacques-Pascal Deplaix, Geoff Shannon, Simon Cruanes, Vladimir Brankov, Gabriel Radanne, Evgenii Lepikhin, Petter Urkedal, Gerd Stolpmann and Anton Bachin.

Categories: Elsewhere

Liip: Drupal 8 accessibility features

Planet Drupal - Mon, 22/08/2016 - 22:36

The Drupal accessibility initiative started with some advancements in Drupal 7 to ensure that Drupal core followed the World Wide Web Consortium (W3C) guidelines: WCAG 2.0 (Web Content Accessibility Guidelines) and ATAG 2.0 (Authoring Tool Accessibility Guidelines).
Many elements introduced in Drupal 7 were improved and bugs discovered through intensive testing were addressed and integrated to Drupal 8 core as well. Let’s take a tour of the accessibility in Drupal 8 !

Contrasts improved

Drupal’s accessibility maintainers improved contrasts in core themes so people that suffer from colorblindness are able to visit websites clearly. It is also good when visiting the website under bright sunlight, on mobile for instance.

Color contrasts in Bartik theme in Drupal 7.43 and Drupal 8.

See the related WCAG 2.0 section about contrasts.

Alternative texts for images

The alternative text for images is really useful for blind people who use screen readers. They can understand the meaning of an image through short descriptive phrases. This alternative text is now by default a required field in Drupal 8.

The alternative text for an image is required by default in Drupal 8 content edition.

See the related WCAG 2.0 section about alternative texts.

More semantics

Many accessibility improvements are hard to see as it involves semantics. Drupal 8 uses HTML5 elements in its templates which add more meaning into the code. For instance, assistive technology such as screen readers can now interpret elements like <header>, <footer> or <form>.

Moreover, WAI-ARIA (Web Accessibility Initiative – Accessible Rich Internet Applications) additional markup really improved semantics using:

  • landmarks to identify regions in a page, for instance: role="banner" ;
  • live regions to indicate that an element will be updated, for instance: aria-live="polite";
  • roles to describe the type of widgets presented, for instance: role="alert";
  • properties: attributes that represent a data value associated with the element.

See the related WCAG 2.0 sections about semantics.

Tabbing order

Drupal 8 introduces the TabbingManager javascript feature. It enables to constrain tabbing order on the page and facilitates navigation with keyboards. It is really helpful to guide a non-visual user to the most important elements on the page and minimize confusion with screen readers.

See the related WCAG 2.0 section about keyboard operations.

Forms

Drupal 8 accessibility involves many improvements regarding forms in Drupal 8.

In Drupal 7, all errors were displayed by default on top of the form and fields were highlighted in red. It was not right for colorblind people to understand where the errors were.
In Drupal 8, there is an experimental option to enable form inline errors and an error icon is displayed next to the field. Nevertheless, note that this feature is not enabled by default as there are still some pending issues.

The error message is displayed below the field when the Form Inline Error module is enabled.

See the related WCAG 2.0 sections about error identification.

Regarding the form API, radios and checkboxes are now embedded in fieldsets to meet WCAG compliance. Indeed, grouping related elements will help screen readers to navigate in complex forms. Plus, all fields have a label associated with the right element using the “for” attribute.

Here is an example of HTML code for radio buttons:

Poll status Closed Active

See the related WCAG technical section about fieldsets

Tables and views

As Views UI module is in core now, it became accessible.
The views tables markup is more semantic. Data cells are associated with header cells through “id” and “headers” attributes. It is also possible to add a <caption> element to explain the purpose of the table and a <summary> element to give an overview on how the data is organized and how to navigate the table.
Plus, the “scope” attribute enables to explicitly mark row and column headings.

Here is an example of a table HTML code generated by a view:

Caption for the table Details for the table Description for details Title Content type Premo Quae Vero Article Capto Dolor Article

See the related WCAG section about tabular information.

Hidden elements

Using "display:none;" CSS styling can be problematic as it will hide elements for both visual and non-visual users and consequently, screen readers will not be able to read them.
Drupal 8 accessibility maintainers decided to standardize in the naming convention of HTML5 Boilerplate using different classes to hide elements:

  • “hidden“: hide an element visually and from screen readers;
  • “visually-hidden“: hide an element only visually but available for screen readers;
  • “invisible“: hide an element visually and from screen readers without affecting the layout.
Aural alerts

Users with visual impairment will not be able to see all visual updates of the page such as color changes, animations or texts appended to the content. In order to make these changes apparent in a non-visual way, Drupal provides the Drupal.announce() JavaScript method which creates an “aria-live” element on the page. This way, text appended to the node can then be read by a screen reading user agent.

Here is an example of a code using the aural alert:

Drupal.announce('Please fill in your user name', 'assertive');

The first parameter is a string for the statement, the second is the priority:

  • “polite“: this is the default, polite statements will not interrupt the user agent;
  • “assertive“: assertive statements will interrupt any current speech.

See the related WCAG technical section about how to use live regions to identify errors.

CKEditor WYSIWYG accessibility

Drupal community helped improving CKEditor accessibility.
First of all, the WYSIWYG editor now comes with keyboard shortcuts which are beneficial for both power users and keyboard-only users.
Drupal 8 implements more semantic elements. For instance, the user can create HTML tables with headers, caption and summary elements. <figure> and <figcaption> HTML5 tags are also available to add captions to images.
Moreover, every image added through CKEditor are required by default, as it is on image fields.

CKEditor module also introduces a language toolbar button so that users can select a part of text and specify the language used. Screen readers will be able then to choose the appropriate language for each content.

See the related WCAG technical section about language attributes.

Finally, there is an accessibility checker plugin for CKEditor. It is not in core yet as a CKEditor issue blocks its integration, you can find more information on the related Drupal issue queue. However, you will find a module that implements it currently: ckeditor_a11checker.

All these options will definitely help users to generate accessible contents.

Conclusion

Drupal core maintainers accomplished great enhancements regarding accessibility in Drupal 8. These accessibility features will definitively be beneficial to keyboard-only users, low-vision users and colorblind people but will also be good for the usability and the SEO of your website.
Nevertheless, there is still work to be done to make Drupal 8 core fully accessible and Drupal needs contributors to tackle the remaining issues.

If you want to learn more about Drupal 8 accessibility, you can watch the presentation about “How Drupal 8 makes your website more easily accessible” given by Mike Gifford, one of the main accessibility core maintainer for Drupal.

Categories: Elsewhere

OpenLucius: 14 Cool Drupal 8 modules for site builders | August 2016

Planet Drupal - Mon, 22/08/2016 - 20:44

Here’s what struck me last month about Drupal modules. This month I have chosen to focus on Drupal 8 as this is slowly becoming the go-to version - partly because many required modules have been migrated and grown up.

1. Require Login
Categories: Elsewhere

Luciano Prestes Cavalcanti: AppRecommender - Last GSoC Report

Planet Debian - Mon, 22/08/2016 - 18:54
My work on Google Summer of Code is to create a new strategy on AppRecommender, where this strategy should be able to get a referenced package, or a list of referenced packages, then analyze the packages that the user has already installed and make a recommendation using the referenced packages as a base, for example: if the user runs "$ sudo apt install vim", the AppRecommender uses "vim" as the referenced package, and should recommend packages with relation between "vim" and the other packages that the user has installed. This work is done and added to the official AppRecommender repository.   During the GSoC program, more contributions were done with the AppRecommender project helping the system to improve the recommendations, installation and configurations to help Debian package.   The following link contains my commits on AppRecommender: https://github.com/tassia/AppRecommender/commits/master?author=LucianoPC   During the period destined to students get to know the community of the project, I talked with the Debian community about my project to get feedback and ideas. When talking to the Debian community on the IRC channels, we came up with the idea of using the popularity-contest data to improve the recommendations. I talked with my mentors, who approved the idea, then we increased the project scope to use the popularity-contest data to improve the AppRecommender recommendations.   The popularity-contest has several privacy political terms, then we did a research and published, on the Debian Planet, a post that explains why we need the popularity-contest data to improve the recommendations and how we use this data. This post also contains an explanation about the risks and the measures taken to minimize them.   Then two activities were added to be made. One of them is to create a script to be added on popularity-contest. This script is destined to get the popularity-contest data, which is the users' packages, and generate clusters that group these packages analyzing similar users. The other activity is to add collaborative data into the AppRecommender, where this will download the clusters data and use it to improve the recommendations.   The popularity-contest cluster script was done and reviewed by my mentor, but was not integrated into popularity-contest yet. The usage of clusters data into AppRecommender has been already implemented, but still not added on official repository because it is waiting the cluster cript's acceptance into the popularity-contest. This work is not complete, but I will continue working with AppRecommender and Debian community, and with my mentors' help, I will finish this work.   The following link contains my commits on repository with the popularity-contest cluster script's feature, as well as other scripts that I used to improve my work, but the only script that will be sent to popularity-contest is the create_popcon_clusters.py: https://github.com/TCC-AppRecommender/scripts/commits/master?author=LucianoPC   The following link contains my commits on repository with the AppRecommender collaborative data feature:  https://github.com/tassia/AppRecommender/commits/collaborative_data?author=LucianoPC   Google Drive folder with the patch: https://drive.google.com/drive/folders/0BzmGBlxBo2G3Q3F5YjBpRl9yWUk?usp=sharing
Categories: Elsewhere

Chromatic: Create a Custom Views Sort Plugin with Drupal 8

Planet Drupal - Mon, 22/08/2016 - 18:46

Having recently grokked custom views sorting in Drupal 7, I took a leap to investigate how to create this functionality in Drupal 8. The documentation out there is scarce at best. As a starting point, I decided to replicate Dave Vasilevsky’s task where he makes a custom views sort plugin for ordering upcoming and past events in D7. The requirement was to show future events in chronological order (i.e. show future events nearest the current date ahead of later future events) before showing past events in reverse chronological order (i.e. most recent past events first followed by earlier past events).

Screenshot from Vasilevsky's post

Create content type and view:

First order of business is to create an Event content type with a date field and a new view of events using fields in the display format. Here is a gist with the configuration yaml files for importing an Event content type with a body and field_event_date field and an Events view.

Looking at the results returned from this view as is, all the events are shown in haphazard order:

Now let’s create a custom views sort plugin. Since we need to alter the query for sorting event results by an event date field in a special way, a new sort handler needs to be added to the view by extending the field table for field_event_date. Using hook_views_data_alter() allows us to add this custom sort handler.

Declare the sort in hook_views_data_alter(): /** * Implements hook_views_data_alter(). */ function mymodule_views_data_alter(array &$data) { $data['node__field_event_date']['event'] = array( 'title' => t('Custom event sort'), 'group' => t('Content'), 'help' => t('Sort events by past/future, then distance from now.'), 'sort' => array( 'field' => 'field_event_date_value', 'id' => 'event', ), ); }

In this particular case, the date field that was added to the Event content type has a machine name of field_event_date. This generated a new table in the database called node__field_event_date that holds the actual value of the datetime field of an Event node. This datetime field is a column called field_event_date_value - this is the field on which we need to perform our custom sorting. Since we added the field_event_date as a field to the view, it will be joined to the base node_field_data table that gets queried for matching Event nodes.

In hook_views_data_alter(), we’re altering the query for the node__field_event_date table by extending it with a new event array that adds some basic information like title, group, and help which we’ll see in the views UI when we go to select the custom sort. The new event array more importantly adds a sort key with the field_event_date_value as the field we want to sort on plus the id of the plugin (in this case a new sort plugin class named ‘event’) we will use to add the custom sorting functionality.

Create the plugin to handle the new sorting: <?php namespace Drupal\mymodule\Plugin\views\sort; use Drupal\views\Plugin\views\sort\Date; /** * Basic sort handler for Events. * * @ViewsSort("event") */ class Event extends Date { /** * Called to add the sort to a query. */ public function query() { $this->ensureMyTable(); $date_alias = "UNIX_TIMESTAMP($this->tableAlias.$this->realField)"; // Is this event in the past? $this->query->addOrderBy(NULL, "UNIX_TIMESTAMP() > $date_alias", $this->options['order'], "in_past" ); // How far in the past/future is this event? $this->query->addOrderBy(NULL, "ABS($date_alias - UNIX_TIMESTAMP())", $this->options['order'], "distance_from_now" ); } }

This sort handler basically extends the Date sort plugin which can be found in core (/core/modules/views/src/Plugin/views/sort/Date.php). All that we’re overriding is the query() function to add our custom sorting, effectively running two sorts (future and past) on the results based on the current date.

Since we need to sort differently for future and past events, we convert the datetime formatted value of field_event_date_value into a unix timestamp and run comparisons against the current date.

We can then alter the query by adding custom ORDER BY clauses to the SQL statement. When we go to add the new sort, the new query will look like:

Let’s see all this in action through the views UI. Back in our view, click on Add next to "SORT CRITERIA". Our new custom sort should be visible in the list:

In the next part of the ajax form, since we didn’t override any of the Date form options, we’ll see the order and granularity options. Just leave all the defaults as is and click "Apply".

Now we should see our new custom sort in the UI:

Finally we’ll see our events sorted with future events in chronological order and past events in reverse chronology:

The theory behind how to accomplish this task in Drupal 8 is not that much different from Drupal 7. In both cases, we need to add the custom sort handler in a hook_views_data_alter() and override the query. The difference in D8 is everything uses the plugin system so the syntax for extending the base Date sort class varies from extending the default sort handler in D7.

And there you have it - a custom views sort plugin in D8!

Categories: Elsewhere

Lars Wirzenius: Linux 25 jubilee symposium

Planet Debian - Mon, 22/08/2016 - 17:03

I gave a talk about the early days of Linux at the jubilee symposium arranged by the University of Helsinki CS department. Below is an outline of what I meant to speak about, but the actual talk didn't follow it exactly. You can compare these to the video once it comes online.

  • Linus and I met at uni, the only 2 Swedish speaking new students that year, so we naturally migrated towards each other.
  • After a year away for military service, got back in touch, summer of
    1. .
  • C & Unix course fall of 1990; Minix.
  • Linus didn't think atime updates in real time were plausible, but I showed him; funnily enough, atime updates have been an issue in Linux until fairly recently, since they slow things down (without being particularly useful)
  • Jan 5, 1991 bought his first PC (i386 + i387 + 4 MiB RAM and a small hard disk); he had a Sinclair QL before that.
  • Played Prince of Persia for a couple of months.
  • Then wanted to learn i386 assembly and multitasking.
  • A/B threading demo.
  • Terminal emulation, Usenet access from home.
  • Hard disk driver, mistaking hard disk for a modem.
  • More ambition, announced Linux to the world for the first time
  • first ever Linux installation.
  • Upload to ftp.funet.fi, directory name by Ari Lemmke.
  • Originally not free software, licence changed early 1992.
  • First mailing list was created and introduced me to a flood of email (managed with VAX/VMS MAIL and later mush on Unix).
  • I talked a lot with Linus about design at this time, but never really participated in the kernel work (partly because disagreeing with Linus is a high-stress thing).
  • However, I did write the first sprintf for the kernel, since Linus hadn't learnt about varargs functions in C; he then ruined it and added the comment "Wirzenius wrote this portably..." (add google hit count for wirzenius+fucked).
  • During 1992 Linux grew fast, and distros happened, and a lot of packaging and porting of software; porting was easier because Linus was happy to add/change things in the kernel to accomodate software
  • A lot of new users during 1992 as well.
  • End of 1992 I and a few others founded the Linux Documentation Project to help all the new users, some of who didn't come from a Unix background.
  • In fact, things progressed so fast in 1992 that Linus thought he'd release 1.0 very soon, resulting in a silly sequence of version numbers: 0.12, 0.95, 0.96, 0.96b, 0.96c, 0.96c++2.
  • X server ported to Linux; almost immediate prediction of the year of the Linux desktop never happening unless ALL the graphics cards were supported immediately.
  • Linus was of the opinion that you needed one process (not thread) per window in X; I taught him event driven programming.
  • Bug in network code, resulting in ban on uni network.
  • Pranks in the shared office room.
  • We released 1.0 in an event at the CS dept in March, 1994; this included some talks and a ritual compilation of the release version during the event.
Categories: Elsewhere

Tag1 Consulting: Tag1 Quo and Drupal 6 Long Term Support

Planet Drupal - Mon, 22/08/2016 - 16:42
Tag1 Quo and Drupal 6 Long Term Support Jeremy Mon, 08/22/2016 - 07:42 Or, What We Did This Summer

It’s been an exciting summer, building our first product with Drupal 8. When we originally made the decision to offer Long Term Support for Drupal 6, we were thinking about a few of our clients that were a little behind on their upgrade plans, and had envisioned a mostly manual process.

Categories: Elsewhere

Acquia Developer Center Blog: Progressively Decoupled Drupal Approaches

Planet Drupal - Mon, 22/08/2016 - 16:34

Progressive decoupling, a concept outlined last year in Dries Buytaert’s first post about decoupled Drupal, is a compelling approach to building Drupal's front end where content editors, site assemblers, and front-end developers maintain contiguous experiences. For content editors and site assemblers, progressive decoupling allows for contextualized interfaces, content workflow, site preview, and other features to remain usable and integrated with Drupal as a whole.

Tags: acquia drupal planet
Categories: Elsewhere

Satyam Zode: Google Summer of Code 2016 : Final Report

Planet Debian - Mon, 22/08/2016 - 16:02
Project Title : Improving diffoscope tool and reproducibility of Debian packages

Project details

This project aims to improve diffoscope tool and fix Debian packages which are unreproducible in Reproducible builds testing framework. diffoscope recursively unpack archives of many kinds and transform various binary formats into more human readable form to compare them. As a part of this project I worked on argument completion feature and ignoring .buildinfo feature. This project is a part of Reproducible Builds effort

Mentor and Co-Mentor
  • Jérémy Bobbio (Lunar) : Mentor
  • Reiner Herrmann (deki) : Co-Mentor
  • Holger Levsen (h01ger) : Co-Mentor
  • Mattia Rizzolo (mapreri) : Co-Mentor

Project Discussion

  • Introduction to Reproducible Builds in Debian

    First time I came to know about Reproducible Builds was during Debconf 2015. I started to get involve from the start of March 2016. At the beginning Lunar suggested me to watch the talks given on Reproducible Builds wiki. I read documentation on Reproducible Builds site and started to participate in IRC discussions on #debian-reproducible.

  • Application Review Period

    During proposal discussion period we discussed the areas where work needs to be done. I wrote the proposal and got it reviewed by community on the mailing list. Simultaneously, I worked on bug #818111 and submitted patch for same. That not only helped me to understand the concept of Reproducible Builds but also helped me to setup testing environment required to check the reproducibility of Debian packages.

  • Community Bonding Period

    During community bonding period I studied the codebase of diffoscope and also spent enough amount of time for learning Python3 metaprogramming and other OOP concepts. We also discussed more about hiding differences and options for same. I couldn’t finish my project research work during this period since I had exams in May 2016 and it consumed almost half of community bonding period and week 1 of coding period.

Project Implementation

Challenges and Work Left
  • To understand the main purpose of diffoscope in the context of Reproducible Builds. I had to go through complete Reproducible Builds project. It consumed significant amount of time to understand what Reproducible Builds is, why it’s necessary important for Free software to build reproducibly. Diffoscope is the last tool in Reproducible Builds toolchain. It was a big challenge for me to understand whole process and objective of diffoscope.
  • Work Left:

Future work
  • Based on the research work and implementation done during Coding Period make diffoscope better and enhance ignoring capabilities of diffoscope.
  • Improve the parallel processing feature of diffoscope. This particular problem is hard to understand and implement.
  • Make diffoscope better by solving exsting bugs.

Acknowledgement

I would like to express my deepest gratitude to Lunar for mentoring me throughout Google Summer of Code program and for being cool. Lunar’s deep knowledge regarding diffoscope and Python skills helped me a lot throughout the project and we literally had great discussions. I would also like to thank Debaian community and Google for giving me this opportunity. Special thanks to Reproducible Builds folks for all the guidance!

Categories: Elsewhere

DebConf team: Proposing speakers for DebConf17 (Posted by DebConf17 team)

Planet Debian - Mon, 22/08/2016 - 15:44

As you may already know, next DebConf will be held at Collège de Maisonneuve in Montreal from August 6 to August 12, 2017. We are already thinking about the conference schedule, and the content team is open to suggestions for invited speakers.

Priority will be given to speakers who are not regular DebConf attendees, who are more likely to bring diverse viewpoints to the conference.

Please keep in mind that some speakers may have very busy schedules and need to be booked far in advance. So, we would like to start inviting speakers in the middle of September 2016.

If you would like to suggest a speaker to invite, please follow the procedure described on the Inviting Speakers page of the DebConf wiki.


DebConf17 team

Categories: Elsewhere

BlackMesh: Our Ongoing Commitment to Security: Partnering with Tag1 Consulting

Planet Drupal - Mon, 22/08/2016 - 15:22

As you may already know, the BlackMesh team is committed to ensuring developers can focus on their website goals without worrying about scalability, infrastructure, and – in particular – security. That’s why we’re thrilled to have partnered with leading Drupal security agency Tag1 Consulting to provide our Drupal clients a comprehensive tool for managing site updates.

What?

The new Tag1 Quo is a hosted security dashboard that provides up-to-the-minute snapshots of a client’s security status and potential vulnerabilities. Basically, this innovative tool provides Drupal users consolidated and critical security updates for their websites. Tag1 Quo features include essentials for a website’s success, such as self-service monitoring, security notifications for out-of-date modules, and patch and release delivery.

How?

Tag1 Quo automatically monitors upstream releases and security advisories in collaboration with module maintainers and other Drupal 6 long term support (LTS) providers. The Tag1 team of Drupal security experts review and decide which issues affect the client, carefully backporting those that are applicable to their site. Quo then delivers timely notifications to the client’s inbox. Quo users can choose to either leverage pre-patched releases or quickly apply the patches themselves to bring their website up-to-date and secure against all known vulnerabilities. For extra peace of mind, the Quo dashboard provides at-a-glance visualization of all client websites, highlighting all outstanding updates across them.

Tag1 Quo Dashboard

Why?

Though D6 was phased out earlier this year, Tag1 Consulting continues to provide long-term D6 support to those who need it. As a way to adequately manage the significant maintenance and monitoring of these systems, the Tag1 team developed Tag1 Quo.

If you are currently managing a D6 site, signing up for Tag1 Quo is a no-brainer. It provides affordable long-term support for all of your core and contributed modules and themes, tested and delivered by an approved LTS provider, backed by a team of renowned Drupal experts.

Your Security. Your Options …

Even if you don’t maintain a D6 site, you’ll still want to check it out.

Depending on your needs and budget, Tag1 Quo offers three different plan options. Tag1’s roadmap includes upcoming support for D7 and D8, WordPress 4.6 and 4.7, application programming interface (API) access, upgrade planning, and more.

Signing up for Tag1 Quo means less time spent on maintenance and security – and that means more time focusing on your goals and overall mission. Whenever BlackMesh teams up with companies like Tag1, we’re advancing our commitment to your security. Contact us to learn more about how we can make the Tag1 Quo dashboard solution fit your needs.

 

Like this article? Follow us on Facebook and share your thoughts!

SecurityDrupal
Categories: Elsewhere

Vincent Sanders: Down the rabbit hole

Planet Debian - Mon, 22/08/2016 - 14:24
My descent began with a user reporting a bug and I fear I am still on my way down.

The bug was simple enough, a windows bitmap file caused NetSurf to crash. Pretty quickly this was tracked down to the libnsbmp library attempting to decode the file. As to why we have a heavily used library for bitmaps? I am afraid they are part of every icon file and many websites still have favicons using that format.

Some time with a hex editor and the file format specification soon showed that the image in question was malformed and had a bad offset header entry. So I was faced with two issues, firstly that the decoder crashed when presented with badly encoded data and secondly that it failed to deal with incorrect header data.

This is typical of bug reports from real users, the obvious issues have already been encountered by the developers and unit tests formed to prevent them, what remains is harder to produce. After a debugging session with Valgrind and electric fence I discovered the crash was actually caused by running off the front of an allocated block due to an incorrect bounds check. Fixing the bounds check was simple enough as was working round the bad header value and after adding a unit test for the issue I almost moved on.

Almost...

We already used the bitmap test suite of images to check the library decode which was giving us a good 75% or so line coverage (I long ago added coverage testing to our CI system) but I wondered if there was a test set that might increase the coverage and perhaps exercise some more of the bounds checking code. A bit of searching turned up the american fuzzy lop (AFL) projects synthetic corpora of bmp and ico images.

After checking with the AFL authors that the images were usable in our project I added them to our test corpus and discovered a whole heap of trouble. After fixing more bounds checks and signed issues I finally had a library I was pretty sure was solid with over 85% test coverage.

Then I had the idea of actually running AFL on the library. I had been avoiding this because my previous experimentation with other fuzzing utilities had been utter frustration and very poor return on investment of time. Following the quick start guide looked straightforward enough so I thought I would spend a short amount of time and maybe I would learn a useful tool.

I downloaded the AFL source and built it with a simple make which was an encouraging start. The library was compiled in debug mode with AFL instrumentation simply by changing the compiler and linker environment variables.

$ LD=afl-gcc CC=afl-gcc AFL_HARDEN=1 make VARIANT=debug test
afl-cc 2.32b by <lcamtuf@google.com>
afl-cc 2.32b by <lcamtuf@google.com>
COMPILE: src/libnsbmp.c
afl-cc 2.32b by <lcamtuf@google.com>
afl-as 2.32b by <lcamtuf@google.com>
[+] Instrumented 751 locations (64-bit, hardened mode, ratio 100%).
AR: build-x86_64-linux-gnu-x86_64-linux-gnu-debug-lib-static/libnsbmp.a
COMPILE: test/decode_bmp.c
afl-cc 2.32b by <lcamtuf@google.com>
afl-as 2.32b by <lcamtuf@google.com>
[+] Instrumented 52 locations (64-bit, hardened mode, ratio 100%).
LINK: build-x86_64-linux-gnu-x86_64-linux-gnu-debug-lib-static/test_decode_bmp
afl-cc 2.32b by <lcamtuf@google.com>
COMPILE: test/decode_ico.c
afl-cc 2.32b by <lcamtuf@google.com>
afl-as 2.32b by <lcamtuf@google.com>
[+] Instrumented 65 locations (64-bit, hardened mode, ratio 100%).
LINK: build-x86_64-linux-gnu-x86_64-linux-gnu-debug-lib-static/test_decode_ico
afl-cc 2.32b by <lcamtuf@google.com>
Test bitmap decode
Tests:606 Pass:606 Error:0
Test icon decode
Tests:392 Pass:392 Error:0
TEST: Testing complete
I stuffed the AFL build directory on the end of my PATH, created a directory for the output and ran afl-fuzz

afl-fuzz -i test/bmp -o findings_dir -- ./build-x86_64-linux-gnu-x86_64-linux-gnu-debug-lib-static/test_decode_bmp @@ /dev/null
The result was immediate and not a little worrying, within seconds there were crashes and lots of them! Over the next couple of hours I watched as the unique crash total climbed into the triple digits.

I was forced to abort the run at this point as, despite clear warnings in the AFL documentation of the demands of the tool, my laptop was clearly not cut out to do this kind of work and had become distressingly hot.

AFL has a visualisation tool so you can see what kind of progress it is making which produced a graph that showed just how fast it managed to produce crashes and how much the return plateaus after just a few cycles. Although it was finding a new unique crash every ten minutes or so when aborted.

I dove in to analyse the crashes and it immediately became obvious the main issue was caused when the test tool attempted allocations of absurdly large bitmaps. The browser itself uses a heuristic to determine the maximum image size based on used memory and several other values. I simply applied an upper bound of 48 megabytes per decoded image which fits easily within the fuzzers default heap limit of 50 megabytes.

The main source of "hangs" also came from large allocations so once the test was fixed afl-fuzz was re-run with a timeout parameter set to 100ms. This time after several minutes no crashes and only a single hang were found which came as a great relief, at which point my laptop had a hard shutdown due to thermal event!

Once the laptop cooled down I spooled up a more appropriate system to perform this kind of work a 24way 2.1GHz Xeon system. A Debian Jessie guest vm with 20 processors and 20 gigabytes of memory was created and the build replicated and instrumented.

To fully utilise this system the next test run would utilise AFL in parallel mode. In this mode there is a single "master" running all the deterministic checks and many "secondary" instances performing random tweaks.

If I have one tiny annoyance with AFL, it is that breeding and feeding a herd of rabbits by hand is annoying and something I would like to see a convenience utility for.

The warren was left overnight with 19 instances and by morning had generated crashes again. This time though the crashes actually appeared to be real failures.

$ afl-whatsup sync_dir/
Summary stats
=============

Fuzzers alive : 19
Total run time : 5 days, 12 hours
Total execs : 214 million
Cumulative speed : 8317 execs/sec
Pending paths : 0 faves, 542 total
Pending per fuzzer : 0 faves, 28 total (on average)
Crashes found : 554 locally unique
All the crashing test cases are available and a simple file command immediately showed that all the crashing test files had one thing in common the height of the image was -2147483648 This seemingly odd number is actually meaningful to a programmer, it is the largest negative number which can be stored in a 32bit integer (INT32_MIN) I immediately examined the source code that processes the height in the image header.

if ((width <= 0) || (height == 0))
return BMP_DATA_ERROR;
if (height < 0) {
bmp->reversed = true;
height = -height;
}
The bug is where the height is made a positive number and results in height being set to 0 after the existing check for zero and results in a crash later in execution. A simple fix was applied and test case added removing the crash and any possible future failure due to this.

Another AFL run has been started and after a few hours has yet to find a crash or non false positive hang so it looks like if there are any more crashes to find they are much harder to uncover.

Main lessons learned are:
  • AFL is an easy to use and immensely powerful and effective tool. State of the art has taken a massive step forward.
  • The test harness is part of the test! make sure it does not behave in a poor manner and cause issues itself.
  • Even a library with extensive test coverage and real world users can benefit from this technique. But it remains to be seen how quickly the rate of return will reduce after the initial fixes.
  • Use the right tool for the job! Ensure you head the warnings in the manual as AFL uses a lot of resources including CPU, disc and memory.
I will of course be debugging any new crashes that occur and perhaps turning my sights to all the projects other unit tested libraries. I will also be investigating the generation of our own custom test corpus from AFL to replace the demo set, this will hopefully increase our unit test coverage even further.

Overall this has been my first successful use of a fuzzing tool and a very positive experience. I would wholeheartedly recommend using AFL to find errors and perhaps even integrate as part of a CI system.
Categories: Elsewhere

Michal &#268;iha&#345;: Continuous integration on multiple platforms

Planet Debian - Mon, 22/08/2016 - 12:00

Over the weekend I've played with continuous integration for Gammu to make it run on more platforms. I had to remember many things from the Windows world on the way and the solution is not yet complete, but the basic build is working, the only problematic part are external dependencies.

First of all we already have Linux builds on Travis CI. These cover compilation with both GCC and Clang compilers, hopefully covering most of the possible problems.

Recently I've added OS X builds on Travis CI, what was pretty much painless and worked out of the box.

The next major architecture to support is Windows. Once I've discovered AppVeyor I thought it might be the way to go. The have free plans for open-source projects (though it has only one parallel build compared to four provided by Travis CI).

As our build system is cross platform based on CMake, it should work pretty much out of the box, right? Well almost, tweaking the basics took some time (unfortunately there is no CMake support on AppVeyor, so you have to script it a bit).

The most painful things on the way:

  • finding our correct way to invoke build and testsuite
  • our code was broken on Windows, making the testsuite to fail
  • how to work with power shell (no, I'm not going to like it)
  • how to download and install executable to PATH
  • test output integration with AppVeyor - done using XSLT transformation and uploading test results manually
  • 32-bit / 64-bit mess, CMake happily finds 32-bit libs during the 64-bit build and vice versa, what makes the build fail later when linking - fixed by trying if code can be built with given library
  • 64-bit code crashes in dummy driver, causing testsuite failures (this has to be something Windows specific as the code works fine on 64-bit Linux) - this seems to be caused by too big allocations on stack, moving them to heap will fix this

You can check our current appveyor.yml in case you're going to try something similar. Current build results are on AppVeyor.

As a nice side effect, we now have up to date Windows binaries for Gammu.

Filed under: Debian English Gammu | 0 comments

Categories: Elsewhere

Pages

Subscribe to jfhovinne aggregator - Elsewhere