Tips and tricks for working with REST module in Drupal 8.
The Public Liberties and Human Rights Center was founded in 2008 as a "desk" within Al Jazeera. Today it is staffed by a diverse team which works across different areas of the network. The editorial team operates on all of Al Jazeera's platforms and in all of the network's languages. The journalists produce original stories and content examining human rights issues around the world, complementing the network's hard-hitting news. Their website contains information about initiatives and events organized by the Center as well as many images and videos related to their projects. In 2015, Vardot developed a new website to bring together all the work Al Jazeera does regarding human right through all its platforms to one hub.
Goal of the project
The goal of Vardot was to build an editor-friendly distribution that will also bring visitors a seamless user experience. The mission was to launch a modern multilingual SEO-optimized website that will be integrated with social networks and have the ability to handle a high traffic.
The right CMS for media networks
Vardot already partnered with Al Jazeera to develop other AJ websites such as Sharek, Forum, Stream and Cafe. All these sites were built on Drupal because the client was looking for a CMS that will be able to handle high traffic, different permission levels, a large number of subpages and at the same time be secure and flexible for users. For the AJ Public Liberties project, we used our very own Drupal 7 distribution, Uber Publisher, that in our opinion ideally meets the needs of the Media producers and Online News Publishers.
The idea of this distribution is easy: most of news websites require the same package of features such as the ability to upload and edit text quickly, create roles and permissions for a better security, add taxonomy terms to organize the content in the convenient way, handle high traffic and more. Uber Publisher does just that. It is a combination of modules, configurations, settings and custom development for online publishers.
Uber Publisher as a competitive advantage
Having the “foundation” of the website ready, our team was able to concentrate on the uniqueness of the Liberties website without wasting time for a basic coding. As a result, the work on the new website of Al Jazeera Public Liberties & Human Rights Centre took less than one month.
Furthermore, we were able to to provide Al Jazeera with many different ways to change the display of content listings to meet their needs at any time. Depending on what Al Jazeera content managers wanted to showcase, the site allows them to feature one main article or multiple articles to tell a story and promote the most important news first.
Building a good website in just one month is very impressive, but we are looking forward to be able to work even faster. After updating Uber Publisher and making it more universal for any kind of online publishing company we expect to decrease not only the time of the development, but also the average total cost of ownership of news websites.
Building sites for enterprise companies always means accepting a big challenge, but our team likes to exceed expectations. Working with Al Jazeera before we’ve learned a lot about our client’s assumptions and ways of working, and this time we’ve just used the experience collected before and made a high-quality website in less than one month. Like Al Jazeera Public Liberties & Human Rights Centre contributes a lot to the awareness of humanitarian organizations, Vardot is happy to add some improvements to Drupal and share our experience with you.Tags: Drupal Planet Title: Rights and Liberty on Your Screen: New Website of Al Jazeera Media Network
We developed the WalkHub Help Widget to help site owners guide their visitors to the right documentation content - right inside an application or web site. You can also set up your own widget creation interface easily - in this blogpost, we will guide you through the steps.
I made a simple implementation of text classification with Recurrent CNN.
It uses chainer, a Deep Learning framework.
Recurrent convolutional neural networks for text classification
Siwei Lai, Liheng Xu, Kang Liu, Jun Zhao, Chinese Academy of Sciences, China
One of the major features of Drupal 8 is the Configuration Management Initiative (CMI). I’ll briefly touch on why CMI is so important, but our own Alex Pott has written extensively on the creation, principles and importance of this feature.
Previous to Drupal 8, it was cumbersome to trade configuration between sites and environments. Moving configuration changes from a developmental environment into production meant either making the changes manually through the UI, or utilizing an advanced option like Features. A third option would be full-database swapping, but that can be quite risky and is generally considered poor-form.
The goal of automated testing is confidence: confidence in application stability, and confidence that new features work as intended. Continuous integration as a philosophy is about speeding the rate of change while keeping stability. As the number of contributing programmers increase, the need to have automated testing as a means to prove stability increases.
This post is focused on how the automated testing infrastructure on Drupal.org works, not actually writing tests. Much more detail about how to write tests during Drupal development can be found in community documentation:
DrupalCI essentially runs two categories of tests:
Functional tests (also called blackbox testing) are the most common type of test run on DrupalCI hardware. These tests run assertions that test functionality by installing Drupal with a fresh database and then exercising that installation by inserting data and confirming the assertions complete. Front-end tests and behavior driven tests (BDD) tend to be functional. Upgrade tests are a type of functional tests that run a full installation of Drupal, then run upgrade commands.
Unit tests run assertions that test a unit of code and do not require a database installation. This means they execute very quickly. Because of its architecture, Drupal 8 has much more unit test coverage than Drupal 7.
These test categories can be broken down further into more specific test types.What testing means at the scale of Drupal
Drupal 8, with its 3,000+ core contributors and 7,288 contrib developers (so far), needs testing as a means to comfortably move forward code that everyone can trust to be stable.
Between January and May 2016, 90,364 test runs were triggered in DrupalCI. That is about 18,000 test runs requested per month. Maintainers set whether they want tests to run on demand, with every patch submitted, or nightly. They also determine what environments those tests will run on; there are 6 combinations of PHP and database engines available for maintainers to choose from.
The majority of these test runs are Drupal 8 tests at this point. (19,599 core tests and 47,713 contrib project tests were run during those 5 months.) Each test costs about 12 cents to run on Amazon Web Services. At the time of writing this post, we averaged around $2,000 per month in testing costs for our community. (Thank you supporters!)An overly simple history of automated testing for Drupal
Automated testing first became a thing for Drupal contributed projects during Drupal version 4.5 with the introduction of the SimpleTest module. It was not until Drupal 6 that we started manually building out testbots and running these tests on Drupal.org hardware.
In Drupal 7, SimpleTest was brought into Drupal Core. (More information about what that took can be reviewed in the SimpleTest Roadmap for Drupal 7.)
In Drupal 8, PHPUnit testing was added to Drupal Core. PHPUnit tests are much faster than a full functional test in SimpleTest—though runtest.sh still triggers a combination of these test types in Drupal 8.
The actual implementation of automated testing was much more complicated than this history suggests. The original testbot infrastructure that ran for 7 years on Drupal.org hardware was manually managed by some fiercely dedicated volunteers. The manual nature of that maintenance led to the architecture of DrupalCI, which was meant to make it easier to test locally at first and later focused on autoscaling on powerful hardware that could plow through tests more quickly.DrupalCI's basic structure
In The Drupal.org Complexity, we could see the intricate ways that Drupal's code base interacts with other parts of the system.
We could further break out how systems like DrupalCI are interrelated.
DrupalCI is a combination of data stored on Drupal.org, cron jobs, drush commands, and most importantly a couple of Jenkins installations to manage all the automation.
Jenkins is the open source automation server project that makes most of the system possible. We use it for automating our build process and deploying code to our dev, staging and production environments. It automates just about anything and is used by companies small and large to run continuous integration or continuous deployment for their applications. It's considered a "best practice" solution alongside options like Travis, CircleCI, and Bamboo. They all have slightly different features, but automation is at the core of most of these DevOps tools.
To provide continuously integrated tests, you need to trigger those tests at a moment when the tests will have the greatest value.
The three triggers for running a test job are when a patch is added to an issue comment, when code is committed to a repository or daily on a cron. Maintainers can specify which triggers are associated with which branches of their projects and which environments should run those tests.
For core these settings look something like this:
This detail allows for specific tests to run at specific times per the Drupal.org Testing Policy for DrupalCI.
To make this automation happen, we have an installation of Jenkins (Infrastructure Jenkins below) that is polled by Drupal.org once per minute with testing jobs to be queued.
These jobs live in a database record alongside Drupal.org.
Infrastructure Jenkins speaks to the CI Dispatcher (also Jenkins) where the testing queue regularly passes those jobs to available testbots. CI Dispatcher uses an Amazon Web Services EC2 plugin to spin up new testbots when no existing testbot is available. Each testbot can spin up Docker containers with the specific test images defined by the maintainer. Theses containers pull from DockerHub repositories with official combinations of PHP and database engines that Drupal supports.
After a testbot is running, the CI Dispatcher is in constant communication with the bots. You can even click through to the console on CI Dispatcher and watch the tests run. (It's not very exciting—perhaps we should add sound effects to the failures—but it is very handy.)
Once per minute, Drupal.org polls the CI Dispatcher for test status. It responds with pending, running, failed or passed. Failed and passed tests are then pulled back into Drupal.org for display using the Jenkins JSON API.
Tests can also be run on demand at the patch, commit or branch level using the handy "add test" and "retest" links.Why did we build this ourselves? Why not use [insert testing platform here]
Lot's of people have asked why we don't use TravisCI, CircleCI or some other hosted testing solution. The short answer is that most publicly available testing systems require Github authentication and integration.
Additionally, our testing infrastructure is powerful because of its integration with our issue queues. Read the aforementioned The Drupal.org Complexity for more information.
Another reason to run our own testing is scale. To get through all of the core tests for Drupal 8 in an acceptable amount of time (about 44 mins on average), we run very large testbots. These bots have 2 processors with 8 hardware cores. With hyperthreading, that means we have 32 hardware execution threads—about 88 EC2 compute units. They are not exactly super computers, but they are very performant.
We average nearly 18,000 test runs per month. During our peak usage we spin up as many as 25 testbot machines—though usually we cap at 15 bots to keep costs under control. This helps us plow through our testing needs during sprints at DrupalCons and large camps.
We have explored using an enterprise licensed version of either Github or CircleCI with our own hardware to tackle testing. That same consideration has been given to SauceLabs for front-end testing. Right now, there is not a cost savings to tackle this migration, but that does not rule it out in the future. Testing services continues to evolve.Accelerating Drupal 8
In my first months as CTO, I was told repeatedly that the most important thing for us to work on was testing for Drupal 8. In those early days as I built out the team, we were mostly focused on catching up from the Drupal 7 upgrade and tackling technical debt issues that cropped up. In DrupalCon Austin, I had members of my team learn how to maintain the testbot infrastructure so that we could take over the process of spinning up bots and dealing with spikes in demand.
By early 2015, we had optimized the old testbots as much as they were going to be optimized. We moved them to AWS so we could spin up faster machines and more bots, but there were features that were waiting on the new DrupalCI infrastructure that were blocking key development on Drupal 8.
In March of 2015, we invited all the community developers that had helped with DrupalCI to the Drupal Association offices in Portland and sprinted with them to figure out the remaining implementation needs. The next couple of months involved tweaking DrupalCI's architecture and cutting out any nice to have features to get something launched as soon as possible.
It is no coincidence that from the time of DrupalCI's launch until the release of Drupal 8, progress was rapidly accelerated.
I am immensely proud of the work of all the community members and staff that worked directly with core maintainers to unblock Drupal 8 development and make it faster. This work was critical.
Thank you to jthorson, ricardoamaro, nick_schuch, dasrecht, basic, isntall, drumm, mikey_p, mixologic, hestenet, chx, mile23, alexpott, dawehner, Shyamala, and webchick. You all made DrupalCI. (And huge apologies to all those I'm undoubtedly leaving out.) Also thank you to anyone who chimed in on IRC or in the issue queues to help us track down bugs and improve the service.What's next for testing Drupal
Most of the future state of testing is outlined in the Drupal.org Testing Policy for DrupalCI.
Key issues that we still need to solve are related to concurrent testing improvements and new test types to support. While we have PhantomJS integrated with the test runner, there are optimizations that need to happen.
Testing is not an endpoint. Like much of our work, it is an ongoing effort to continuously improve Drupal by providing a tool that improves how we test, what we test, and when we test.
You want to develop with Drupal 8? We show you some useful sources of lectures, examples and documentations that will ease your work.