Tips and tricks for working with REST module in Drupal 8.
The Public Liberties and Human Rights Center was founded in 2008 as a "desk" within Al Jazeera. Today it is staffed by a diverse team which works across different areas of the network. The editorial team operates on all of Al Jazeera's platforms and in all of the network's languages. The journalists produce original stories and content examining human rights issues around the world, complementing the network's hard-hitting news. Their website contains information about initiatives and events organized by the Center as well as many images and videos related to their projects. In 2015, Vardot developed a new website to bring together all the work Al Jazeera does regarding human right through all its platforms to one hub.
Goal of the project
The goal of Vardot was to build an editor-friendly distribution that will also bring visitors a seamless user experience. The mission was to launch a modern multilingual SEO-optimized website that will be integrated with social networks and have the ability to handle a high traffic.
The right CMS for media networks
Vardot already partnered with Al Jazeera to develop other AJ websites such as Sharek, Forum, Stream and Cafe. All these sites were built on Drupal because the client was looking for a CMS that will be able to handle high traffic, different permission levels, a large number of subpages and at the same time be secure and flexible for users. For the AJ Public Liberties project, we used our very own Drupal 7 distribution, Uber Publisher, that in our opinion ideally meets the needs of the Media producers and Online News Publishers.
The idea of this distribution is easy: most of news websites require the same package of features such as the ability to upload and edit text quickly, create roles and permissions for a better security, add taxonomy terms to organize the content in the convenient way, handle high traffic and more. Uber Publisher does just that. It is a combination of modules, configurations, settings and custom development for online publishers.
Uber Publisher as a competitive advantage
Having the “foundation” of the website ready, our team was able to concentrate on the uniqueness of the Liberties website without wasting time for a basic coding. As a result, the work on the new website of Al Jazeera Public Liberties & Human Rights Centre took less than one month.
Furthermore, we were able to to provide Al Jazeera with many different ways to change the display of content listings to meet their needs at any time. Depending on what Al Jazeera content managers wanted to showcase, the site allows them to feature one main article or multiple articles to tell a story and promote the most important news first.
Building a good website in just one month is very impressive, but we are looking forward to be able to work even faster. After updating Uber Publisher and making it more universal for any kind of online publishing company we expect to decrease not only the time of the development, but also the average total cost of ownership of news websites.
Building sites for enterprise companies always means accepting a big challenge, but our team likes to exceed expectations. Working with Al Jazeera before we’ve learned a lot about our client’s assumptions and ways of working, and this time we’ve just used the experience collected before and made a high-quality website in less than one month. Like Al Jazeera Public Liberties & Human Rights Centre contributes a lot to the awareness of humanitarian organizations, Vardot is happy to add some improvements to Drupal and share our experience with you.Tags: Drupal Planet Title: Rights and Liberty on Your Screen: New Website of Al Jazeera Media Network
We developed the WalkHub Help Widget to help site owners guide their visitors to the right documentation content - right inside an application or web site. You can also set up your own widget creation interface easily - in this blogpost, we will guide you through the steps.
I made a simple implementation of text classification with Recurrent CNN.
It uses chainer, a Deep Learning framework.
Recurrent convolutional neural networks for text classification
Siwei Lai, Liheng Xu, Kang Liu, Jun Zhao, Chinese Academy of Sciences, China
One of the major features of Drupal 8 is the Configuration Management Initiative (CMI). I’ll briefly touch on why CMI is so important, but our own Alex Pott has written extensively on the creation, principles and importance of this feature.
Previous to Drupal 8, it was cumbersome to trade configuration between sites and environments. Moving configuration changes from a developmental environment into production meant either making the changes manually through the UI, or utilizing an advanced option like Features. A third option would be full-database swapping, but that can be quite risky and is generally considered poor-form.
The goal of automated testing is confidence: confidence in application stability, and confidence that new features work as intended. Continuous integration as a philosophy is about speeding the rate of change while keeping stability. As the number of contributing programmers increase, the need to have automated testing as a means to prove stability increases.
This post is focused on how the automated testing infrastructure on Drupal.org works, not actually writing tests. Much more detail about how to write tests during Drupal development can be found in community documentation:
DrupalCI essentially runs two categories of tests:
Functional tests (also called blackbox testing) are the most common type of test run on DrupalCI hardware. These tests run assertions that test functionality by installing Drupal with a fresh database and then exercising that installation by inserting data and confirming the assertions complete. Front-end tests and behavior driven tests (BDD) tend to be functional. Upgrade tests are a type of functional tests that run a full installation of Drupal, then run upgrade commands.
Unit tests run assertions that test a unit of code and do not require a database installation. This means they execute very quickly. Because of its architecture, Drupal 8 has much more unit test coverage than Drupal 7.
These test categories can be broken down further into more specific test types.What testing means at the scale of Drupal
Drupal 8, with its 3,000+ core contributors and 7,288 contrib developers (so far), needs testing as a means to comfortably move forward code that everyone can trust to be stable.
Between January and May 2016, 90,364 test runs were triggered in DrupalCI. That is about 18,000 test runs requested per month. Maintainers set whether they want tests to run on demand, with every patch submitted, or nightly. They also determine what environments those tests will run on; there are 6 combinations of PHP and database engines available for maintainers to choose from.
The majority of these test runs are Drupal 8 tests at this point. (19,599 core tests and 47,713 contrib project tests were run during those 5 months.) Each test costs about 12 cents to run on Amazon Web Services. At the time of writing this post, we averaged around $2,000 per month in testing costs for our community. (Thank you supporters!)An overly simple history of automated testing for Drupal
Automated testing first became a thing for Drupal contributed projects during Drupal version 4.5 with the introduction of the SimpleTest module. It was not until Drupal 6 that we started manually building out testbots and running these tests on Drupal.org hardware.
In Drupal 7, SimpleTest was brought into Drupal Core. (More information about what that took can be reviewed in the SimpleTest Roadmap for Drupal 7.)
In Drupal 8, PHPUnit testing was added to Drupal Core. PHPUnit tests are much faster than a full functional test in SimpleTest—though runtest.sh still triggers a combination of these test types in Drupal 8.
The actual implementation of automated testing was much more complicated than this history suggests. The original testbot infrastructure that ran for 7 years on Drupal.org hardware was manually managed by some fiercely dedicated volunteers. The manual nature of that maintenance led to the architecture of DrupalCI, which was meant to make it easier to test locally at first and later focused on autoscaling on powerful hardware that could plow through tests more quickly.DrupalCI's basic structure
In The Drupal.org Complexity, we could see the intricate ways that Drupal's code base interacts with other parts of the system.
We could further break out how systems like DrupalCI are interrelated.
DrupalCI is a combination of data stored on Drupal.org, cron jobs, drush commands, and most importantly a couple of Jenkins installations to manage all the automation.
Jenkins is the open source automation server project that makes most of the system possible. We use it for automating our build process and deploying code to our dev, staging and production environments. It automates just about anything and is used by companies small and large to run continuous integration or continuous deployment for their applications. It's considered a "best practice" solution alongside options like Travis, CircleCI, and Bamboo. They all have slightly different features, but automation is at the core of most of these DevOps tools.
To provide continuously integrated tests, you need to trigger those tests at a moment when the tests will have the greatest value.
The three triggers for running a test job are when a patch is added to an issue comment, when code is committed to a repository or daily on a cron. Maintainers can specify which triggers are associated with which branches of their projects and which environments should run those tests.
For core these settings look something like this:
This detail allows for specific tests to run at specific times per the Drupal.org Testing Policy for DrupalCI.
To make this automation happen, we have an installation of Jenkins (Infrastructure Jenkins below) that is polled by Drupal.org once per minute with testing jobs to be queued.
These jobs live in a database record alongside Drupal.org.
Infrastructure Jenkins speaks to the CI Dispatcher (also Jenkins) where the testing queue regularly passes those jobs to available testbots. CI Dispatcher uses an Amazon Web Services EC2 plugin to spin up new testbots when no existing testbot is available. Each testbot can spin up Docker containers with the specific test images defined by the maintainer. Theses containers pull from DockerHub repositories with official combinations of PHP and database engines that Drupal supports.
After a testbot is running, the CI Dispatcher is in constant communication with the bots. You can even click through to the console on CI Dispatcher and watch the tests run. (It's not very exciting—perhaps we should add sound effects to the failures—but it is very handy.)
Once per minute, Drupal.org polls the CI Dispatcher for test status. It responds with pending, running, failed or passed. Failed and passed tests are then pulled back into Drupal.org for display using the Jenkins JSON API.
Tests can also be run on demand at the patch, commit or branch level using the handy "add test" and "retest" links.Why did we build this ourselves? Why not use [insert testing platform here]
Lot's of people have asked why we don't use TravisCI, CircleCI or some other hosted testing solution. The short answer is that most publicly available testing systems require Github authentication and integration.
Additionally, our testing infrastructure is powerful because of its integration with our issue queues. Read the aforementioned The Drupal.org Complexity for more information.
Another reason to run our own testing is scale. To get through all of the core tests for Drupal 8 in an acceptable amount of time (about 44 mins on average), we run very large testbots. These bots have 2 processors with 8 hardware cores. With hyperthreading, that means we have 32 hardware execution threads—about 88 EC2 compute units. They are not exactly super computers, but they are very performant.
We average nearly 18,000 test runs per month. During our peak usage we spin up as many as 25 testbot machines—though usually we cap at 15 bots to keep costs under control. This helps us plow through our testing needs during sprints at DrupalCons and large camps.
We have explored using an enterprise licensed version of either Github or CircleCI with our own hardware to tackle testing. That same consideration has been given to SauceLabs for front-end testing. Right now, there is not a cost savings to tackle this migration, but that does not rule it out in the future. Testing services continues to evolve.Accelerating Drupal 8
In my first months as CTO, I was told repeatedly that the most important thing for us to work on was testing for Drupal 8. In those early days as I built out the team, we were mostly focused on catching up from the Drupal 7 upgrade and tackling technical debt issues that cropped up. In DrupalCon Austin, I had members of my team learn how to maintain the testbot infrastructure so that we could take over the process of spinning up bots and dealing with spikes in demand.
By early 2015, we had optimized the old testbots as much as they were going to be optimized. We moved them to AWS so we could spin up faster machines and more bots, but there were features that were waiting on the new DrupalCI infrastructure that were blocking key development on Drupal 8.
In March of 2015, we invited all the community developers that had helped with DrupalCI to the Drupal Association offices in Portland and sprinted with them to figure out the remaining implementation needs. The next couple of months involved tweaking DrupalCI's architecture and cutting out any nice to have features to get something launched as soon as possible.
It is no coincidence that from the time of DrupalCI's launch until the release of Drupal 8, progress was rapidly accelerated.
I am immensely proud of the work of all the community members and staff that worked directly with core maintainers to unblock Drupal 8 development and make it faster. This work was critical.
Thank you to jthorson, ricardoamaro, nick_schuch, dasrecht, basic, isntall, drumm, mikey_p, mixologic, hestenet, chx, mile23, alexpott, dawehner, Shyamala, and webchick. You all made DrupalCI. (And huge apologies to all those I'm undoubtedly leaving out.) Also thank you to anyone who chimed in on IRC or in the issue queues to help us track down bugs and improve the service.What's next for testing Drupal
Most of the future state of testing is outlined in the Drupal.org Testing Policy for DrupalCI.
Key issues that we still need to solve are related to concurrent testing improvements and new test types to support. While we have PhantomJS integrated with the test runner, there are optimizations that need to happen.
Testing is not an endpoint. Like much of our work, it is an ongoing effort to continuously improve Drupal by providing a tool that improves how we test, what we test, and when we test.
You want to develop with Drupal 8? We show you some useful sources of lectures, examples and documentations that will ease your work.
Hi geeks! Did the post title get you excited? Great, because Organic Groups (OG) and Message stack are getting closer to being Drupal 8 ready.
I’d like to give an overview about their state, the amazing community effort around them, and also share some of my personal thoughts about contribution to Drupal 8 in general.Organic Groups
For years Organic Groups has been one of the proven solutions for multi-sites functionality, in the form of one code base, one database, and one dashboard to rule them all. After so many years and seeing so many different implementations, such as Harvard’s OpenScholar, OpenAtrium, and many others, I’m even more confident OG is doing many things right.
Most of OG7’s concepts are being migrated to OG8, but obviously this is also a good time to fix some old mistakes. One of the mistakes was treating users and content (i.e. non-user entities) alike. But, well, you know - they are not. Because when we came to re-think of it, membership really makes sense only for users. For example, if the membership state is active, pending or blocked, that should indeed be applied only to users. So we’ve changed it:
GSoC students have officially been coding since 23 May (about 2.5 weeks) and are almost half-way to the mid-summer evaluation (20 - 27 June). Students who haven't completed some meaningful work before that deadline don't receive payment and in such a large program, there is no possibility to give students extensions or let them try and catch up later.
Every project and every student are different, some are still getting to know their environment while others have already done enough to pass the mid-summer evaluation.
I'd like to share a few tips to help students ensure they don't inadvertently fail the mid-summer evaluationKill electronic distractions
As a developer of real-time communications projects, many people will find it ironic or hypocritical that this is at the top of my list.
Switch off the mobile phone or put it in silent mode so it doesn't even vibrate. Research has suggested that physically turning it off and putting it out of sight has significant benefits. Disabling the voicemail service can be an effective way of making sure no time is lost listening to a bunch of messages later. Some people may grumble at first but if they respect you, they'll get into the habit of emailing you and waiting for you to respond when you are not working.
Get out a piece of paper and make a list of all the desktop notifications on your computer, whether they are from incoming emails, social media, automatic updates, security alerts or whatever else. Then figure out how to disable them all one-by-one.
Use email to schedule fixed times for meetings with mentors. Some teams/projects also have fixed daily or weekly times for IRC chat. For a development project like GSoC, it is not necessary or productive to be constantly on call for 3 straight months.Commit every day
Habits are a powerful thing. Successful students have a habit of making at least one commit every day. The "C" in GSoC is for Code and commits are a good way to prove that coding is taking place.
GSoC is not a job, it is like a freelance project. There is no safety-net for students who get sick or have an accident and mentors are not bosses, each student is expected to be their own boss. Although Google has started recommending students work full time, 40 hours per week, it is unlikely any mentors have any way to validate these hours. Mentors can look for a commit log, however, and simply won't be able to pass a student if there isn't code.
There may be one day per week where a student writes a blog or investigates a particularly difficult bug and puts a detailed report in the bug tracker but by the time we reach the second or third week of GSoC, most students are making at least one commit in 3 days out of every 5.Consider working away from home/family/friends
Can you work without anybody interrupting you for at least five or six hours every day?
Do you feel pressure to help with housework, cooking, siblings or other relatives? Even if there is no pressure to do these things, do you find yourself wandering away from the computer to deal with them anyway?
Do family, friends or housemates engage in social activities, games or other things in close proximity to where you work?
All these things can make a difference between passing and failing.
Maybe these things were tolerable during high school or university. GSoC, however, is a stepping stone into professional life and that means making a conscious decision to shut those things out and focus. Some students have the ability to manage these distractions well, but it is not for everybody. Think about how leading sports stars or musicians find a time and space to be "in the zone" when training or rehearsing, this is where great developers need to be too.
Some students find the right space in a public library or campus computer lab. Some students have been working in hacker spaces or at empty desks in local IT companies. These environments can also provide great networking opportunities.Managing another summer job concurrently with GSoC
It is no secret that some GSoC students have another job as well. Sometimes the mentor is aware of it, sometimes it has not been disclosed.
The fact is, some students have passed GSoC while doing a summer job or internship concurrently but some have also failed badly in both GSoC and their summer job. Choosing one or the other is the best way to succeed, get the best results and maximize the quality of learning and community interaction. For students in this situation, now it is not too late to make the decision to withdraw from GSoC or the other job.
If doing a summer job concurrently with GSoC is unavoidable, the chance of success can be greatly increased by doing the GSoC work in the mornings, before starting the other job. Some students have found that they actually finish more quickly and produce better work when GSoC is constrained to a period of 4 or 5 hours each morning and their other job is only in the afternoon. On the other hand, if a student doesn't have the motivation or energy to get up and work on GSoC before the other job then this is a strong sign that it is better to withdraw from GSoC now.
I'll tune in to the post-based conversation being held on Planet Debian: Russell Coker wonders about what's needed to get university graduates with enough skills for a sysadmin job, to which Lucas Nussbaum responds with his viewpoints. They present a very contrasting view of what's needed for students — And for a good reason, I'd say: Lucas is an academician; I don't know for sure about Russell, but he seems to be a down-to-the-earth, dirty-handed, proficient sysadmin working on the field. They both contact newcomers to their fields, and will notice different shortcomings.
I tend to side with Lucas' view. That does not come as a surprise, as I've been working for over 15 years in an university, and in the last few years I started walking from a mostly-operative sysadmin in an academic setting towards becoming an academician that spends most of his time sysadmining. Subtle but important distinction.
I teach at the BSc level at UNAM, and am a Masters student at IPN (respectively, Mexico's largest and second-largest universities). And yes, the lack of sysadmin abilities in both is surprising. But so is a good understanding of programming. And I'm sure that, were I to dig into several different fields, I'd feel the same: Student formation is very basic at each of those fields.
But I see that as natural. Of course, if I were to judge people as geneticists as they graduate from Biology, or were I to judge them as topologists as they graduate from Mathematics, or any other discipline in which I'm not an expert, I'd surely not know where to start — Given I have about 20 years of professional life on my shoulders, I'm quite skewed as to what is basic for a computing professional. And of course, there are severe holes in my formation, in areas I never used. I know next to nothing of electronics, my mathematical basis is quite flaky, and I'm a poor excuse when talking about artificial intelligence.
Where am I going with this? An university degree (BSc in English, would amount to "licenciatura" in Spanish) is not for specialization. It is to have a sufficiently broad panorama of the field, and all of the needed tools to start digging deeper and specializing — either by yourself, working on a given field and learning its details as you go, or going through a postgraduate program (Specialization, Masters, Doctorate).
Even most of my colleagues at the Masters in Engineering in Security and Information Technology lack of a good formation in fields I consider essential. However, what does information security mean? Many among them are working on legal implications of several laws that touch our field. Many other are working on authenticity issues in images, audios and other such media. Many other are trying to come up with mathematical ways to cheapen the enormous burden of crypto operations (say, "shaving" CPU cycles off a very large exponentiation). Others are designing autonomous learning mechanisms to characterize malware. Were I as a computing professional to start talking about their research, I'd surely reveal I know nothing about it and get laughed at. That's because I haven't specialized in those fields.
University education should give a broad universal basis to enter a professional field. It should not focus on teaching tools or specific procedures (although some should surely be presented as examples or case studies). Although I'd surely be happy if my university's graduates were to know everything about administering a Debian system, that would be wrong for a university to aim at; I'd criticize it the same way I currently criticize programs that mix together university formation and industry certification as if they were related.
What happened in the Reproducible Builds effort between May 29th and June 4th 2016:Media coverage
Ed Maste will present Reproducible Builds in FreeBSD at BDSCan 2016 in Ottawa, Canada on June 11th.GSoC and Outreachy updates
- Satyam blogged about his first experiences hacking on diffoscope.
- Ceridwen wrote about her first steps on reprotest, which now has a git repo.
- Paul Gevers uploaded fpc/3.0.0+dfsg-5 with a new helper script fp-fix-timestamps, which helps with reproducibility issues of PPU files in freepascal packages.
- Sascha Steinbiss uploaded a patched version of epydoc to our experimental repository to test a patch for the use_epydoc issue.
The following 53 packages have become reproducible due to changes in their build-dependencies: angband blktrace code-saturne coinor-symphony device-tree-compiler mpich rtslib ruby-bcrypt ruby-bson-ext ruby-byebug ruby-cairo ruby-charlock-holmes ruby-curb ruby-dataobjects-sqlite3 ruby-escape-utils ruby-ferret ruby-ffi ruby-fusefs ruby-github-markdown ruby-god ruby-gsl ruby-hdfeos5 ruby-hiredis ruby-hitimes ruby-hpricot ruby-kgio ruby-lapack ruby-ldap ruby-libvirt ruby-libxml ruby-msgpack ruby-ncurses ruby-nfc ruby-nio4r ruby-nokogiri ruby-odbc ruby-oj ruby-ox ruby-raindrops ruby-rdiscount ruby-redcarpet ruby-redcloth ruby-rinku ruby-rjb ruby-rmagick ruby-rugged ruby-sdl ruby-serialport ruby-sqlite3 ruby-unicode ruby-yajl ruby-zoom thin
The following packages have become reproducible after being fixed:
- aft/2:5.098-3 by Robert Lemmen, original patch by Chris Lamb.
- bbswitch/0.8-4 by Vincent Cheng, original patch by Reiner Herrmann.
- cgal/4.8-4 by Joachim Reichel.
- gnuplot/4.6.6-4 by Anton Gladky.
- jmodeltest/2.1.10+dfsg-2 by Andreas Tille.
- kball/0.0.20041216-9 by Markus Koschany, original patch by Reiner Herrmann.
- netmaze/0.81+jpg0.82-15 by John Goerzen, original patches by Chris Lamb (#778200) and Maria Valentina Marin (#793731).
- pacemaker/1.1.15~rc3-1 by Ferenc Wágner.
- pam/1.1.8-3.3 by Laurent Bigonville, original patch by Juan Picca.
- plast/2.3.1+dfsg-4 by Sascha Steinbiss.
- prospector/0.11.7-7 by Daniel Stender.
- pymia/0.1.8-1 by Gert Wollny.
- python-cobra/0.4.1-2 by Sascha Steinbiss.
- python-latexcodec/1.0.3-3 by Sascha Steinbiss, original patch by Chris Lamb.
- pyx/0.12.1-5 by Stuart Prescott, original patch by Alexis Bienvenüe.
- rdp-readseq/2.0.2-2 by Sascha Steinbiss.
- stevedore/1.14.0-1 by Ondřej Nový.
- wise/2.4.1-18 by Sascha Steinbiss.
Some uploads have addressed some reproducibility issues, but not all of them:
- binutils/2.26-10 by Matthias Klose, original patch by Chris Lamb.
- pyx3/0.14.1-2 by Stuart Prescott, based on original patch by Alexis Bienvenüe.
Uploads with an unknown result because they fail to build:
- h2database/1.4.192-1 by Emmanuel Bourg, which forces a specific locale to generate documentation.
Patches submitted that have not made their way to the archive yet:
- #825764 against docbook-ebnf by Chris Lamb: sort list of globbed files.
- #825857 against python-setuptools by Anton Gladky: sort list of files in native_libs.txt.
- #825968 against epydoc by Sascha Steinbiss: traverse lists in sorted order.
- #826051 against dh-lua by Reiner Herrmann: sort list of Lua versions embedded into control file.
- #826093 against osc by Alexis Bienvenüe: use SOURCE_DATE_EPOCH for manpage date.
- #826158 against texinfo by Alexis Bienvenü: use SOURCE_DATE_EPOCH for dates in makeinfo output.
- #826162 against slime by Alexis Bienvenüe: sort list of contributors locale-independently.
- #826209 against fastqtl by Chris Lamb: normalize permissions and order in tarball.
- #826309 against gnupg2 by intrigeri: don't embed hostname and timestamp into gpgv.exe.
45 reviews have been added, 25 have been updated and 25 have been removed in this week.
- New issues:
12 FTBFS bugs have been reported by Chris Lamb and Niko Tyni.diffoscope development
- diffoscope 53 was been released by Mattia Rizzolo, with:
- diffoscope 54 (released shortly after) to address a regression involving --list-tools, where a syntax error prevented proper listing of all tools.
Mattia uploaded strip-nondeterminism 0.018-1 which improved support for *.epub files.tests.reproducible-builds.org
- Added pages with oldest logs per arch (h01ger)
- 3 new armhf hosts were added (vagrant)
- setup and maintenance jobs added (h01ger)
- 7 new builder jobs for armhf added (h01ger)
- HOME environment variation on tests.r-b.org and prebuilder (mattia), after Johannes Schauer discovered that this variation was not detected.
- 7 new package sets (for a total of 42) were added by h01ger:
- Improved rescheduling of packages in "depwait" state (h01ger)
- Use eatmydata on i386+armhf (h01ger) (doesn't work yet; needs unreleased pbuilder features)
This week's edition was written by Reiner Herrman, Holger Levsen and Chris Lamb and reviewed by a bunch of Reproducible builds folks on IRC.
For developing complex, real-world Docker images, there are a number of tools that can make life easier.
The first thing to realise is that the Dockerfile format is severely limited. At work, we have eventually outgrown it and it has been replaced with a structured YAML document that is processed into a Dockerfile by a tool called dogen. There are several advantages to this, but I'll point out two: firstly, having data about the image available in a structured format makes automatically deriving technical documentation very easy. Secondly, some of the quirks of Dockerfiles, such as the ADD command respecting the environment's umask, are worked around in the dogen tool.
We have a large suite of integration tests that we run against images to make sure that we haven't introduced regressions during their development. The core of this is the Container Testing Framework, which makes use of the Behave system.
Each command that is run in a Dockerfile generates a new docker image layer. In practice, this can mean a real-world image has a great number of layers underneath it. Docker-dot-com have resisted introducing layer squashing into their tools, but with both hard limits for layers in some of the storage backends, and performance issues for most of the rest, this is a very real issue. Marek Goldmann wrote a squashing tool that we use to control the number of intermediate layers that are introduced by our images.
Finally, even with tools like dogen and ctf, we would like to be able to have more sophisticated tools than shell scripts for configuring images, both at image build time and container run time. We want to do this without introducing extra dependencies inside the images which will not otherwise be used for their operation.
Ansible could be a solution for this, but there are practical issues with relying on it for runtime configuration in our situation. For that reason David Becvarik is designing and implementing Container Configuration Tool, or cct, a framework for performing configuration of containers written in Python.