Planet Drupal

Subscribe to Planet Drupal feed
Drupal.org - aggregated feeds in category Planet Drupal
Updated: 19 min 15 sec ago

Chromatic: Drupal 8 Deployments with Jenkins, GitHub & Slack

Wed, 17/02/2016 - 23:36

We recently launched our first Drupal 8 site--actually it’s this very site that you’re reading! While this wasn’t our first time using or developing for Drupal 8, it was our first full site build and launch on the new platform. As such, it was the first time we needed to handle Drupal 8 code deployments. While I’ve previously covered the benefits of using Jenkins, this post will take you through the steps to create a proper Drupal 8 deployment and how to integrate GitHub and Slack along the way. In other words, you’ll see our current recipe for deploying code automatically, consistently and transparently.

First Things First: Some Assumptions

This post assumes you already have a Jenkins server up and running with the following plugins installed:

If you don’t yet have these things ready to go, getting a Jenkins server setup is well documented here. As is how to install Jenkins plugins. For us, we typically use a Linode instance running Ubuntu LTS for our Jenkins servers. This post also assumes that the external environment you’re trying to deploy to already has Drush for Drupal 8 installed.

Example Deployment Script with Drush

Before we dive into setting up the Jenkins job to facilitate code deployments, I’d like to take a look at what exactly we’re trying to automate or delegate to our good friend Jenkins. At the heart of virtually any of our Drupal deployments (be them Drupal 7, 8 or otherwise) is a simple bash script that executes Drush command in succession. At a macro level, this typically means doing the following, regardless of version:

  1. SSH to the server
  2. Change directory to repository docroot
  3. Pull down latest code on the master branch
  4. Clear Drush cache
  5. Run database updates
  6. Update production configuration
  7. Clear Drupal caches

In Drupal 7, where we relied heavily on Features to deploy configuration, we would typically do something like this:

echo "" echo "Switching to project docroot." cd /var/www/drupal-7-project/docroot echo "" echo "Pulling down latest code." git pull origin master echo "" echo "Clearing drush cache" drush cc drush echo "" echo "Run database updates." drush updb -y echo "" echo "Reverting features modules." drush fra -y echo "" echo "Clearing caches." echo "" drush cc all echo "" echo "Deployment complete."

In Drupal 8, we have the magical unicorn that is the Configuration Management System, so our deployments scripts now look something like this:

If you’re familiar with creating Jenkins jobs already and are just looking for a Drupal 8 deploy script, these next lines are for you.

echo "" echo "Switching to project docroot." cd /var/www/chromatichq.com/docroot echo "" echo "Pulling down the latest code." git pull origin master echo "" echo "Clearing drush caches." drush cache-clear drush echo "" echo "Running database updates." drush updb -y echo "" echo "Importing configuration." drush config-import -y echo "" echo "Clearing caches." drush cr echo "" echo "Deployment complete."

Seriously, configuration management in Drupal 8 is amazing. Hat tip to all of those who worked on it. Bravo.

Another notable difference is that with Drupal 8, clearing caches uses the cache-rebuild Drush command or drush cr for short. drush cc all has been deprecated. R.I.P. little buddy. ⚰

If you have a site that needs to be put into "Maintenance mode" during deployments, you can handle that in Drupal 8 with drush sset system.maintenance_mode 1 to enable and drush sset system.maintenance_mode 0 to disable.

Creating our Jenkins Slave & Job

Now that we’ve covered what it is we want Jenkins to handle automatically for us, let’s quickly run down the punch list of things we want to accomplish with our deployment before we dive into the actual how-to:

  1. Automatically kickoff our deployment script when merges to the master branch occur in GitHub
  2. Run our deployment script from above (deploys latest code, imports config, clears caches, etc.)
  3. Report deployment results back to Slack (success, failure, etc.)
Create Your Jenkins Slave

For Jenkins to orchestrate anything on a remote box, it first needs to know about said box. In Jenkins parlance, this is known as a "node". In our case, since we’re connecting to a remote machine, we’ll use a “Dumb Slave”. Navigate to Manage Jenkins > Manage Nodes > New Node

At Chromatic our naming convention matches whatever we’ve named the machine in Ansible. For the purposes of this article, you can just name this something that makes sense to you. Example:** Drupal-Prod-01**

As part of the creation of this node, you’ll need to specify the Host and the Credentials Jenkins should use to access the box remotely. If you don’t yet have credentials added to Jenkins, you can do so at Jenkins > Credentials > Global credentials (unrestricted). From there things are pretty self-explanatory.

Setup the Basics for our Jenkins Job

Now that we have a way for Jenkins to target a specific server (our slave node) we can start building our deployment job from scratch. Start by navigating to: Jenkins > New Item > Freestyle Project.

From there press "OK" and move on to setting up some basic information about your job, including Project Name, Description and the URL to your GitHub repository. Pro tip: take the time to add as much detail here, especially in the Description field as you can. You’ll thank yourself later when you have loads of jobs.

Configure Slack Notification Settings (optional)

Assuming you’re interested in tying your deployment status messages to Slack and you’ve installed the Slack Notification Plugin, the next step is to tell Jenkins how/where to report to Slack. You do this under the Slack Notifications options area. As far as notifications go, we prefer to use only the "Notify Failure", “Notify Success” and “Notify Back To Normal” options. This is the right mix of useful information without becoming noisy. To allow Jenkins to connect to your Slack channels, you’ll need to follow these steps for adding a Jenkins integration. Then just fill in your Slack domain, the integration token from Slack and the channel you’d like to post to. These settings are hidden under “Advanced…”.

Configure Where this Job Can Run

This is where we instruct Jenkins on which nodes the job is allowed to run. In this case, we’ll limit our job to the slave we created in step one: Drupal-8-Prod-01. This ensures that the job can’t run, even accidentally, on any other nodes that Jenkins knows about. Jenkins allows this to be one node, multiple nodes, or a group.

Configure GitHub Repository Integration

Under "Source Code Management" we’ll specify our version control system, where are repository lives, the credentials used to access the repo and the branches to “listen” to. In our example, the settings look like this:

Here we’re using a jenkins system user on our servers that has read access on our GitHub repositories. You’ll want to configure credentials that make sense for your architecture. Our "Branch Specifier" (*/master) tells Jenkins to look for changes on the master branch or any remote name by using the wildcard, “*”.

Configure Your Build Triggers

This is where the rubber meets the road in terms automation. At Chromatic, we typically opt for smaller, more frequent deployments instead of larger releases where there is a higher probability of regressions. Since we rely heavily on the GitHub pull request model, we often have many merges to master on any given day of development for an active project. So we configure our deployments to coincide with these merges. The following setup (provided via the GitHub Jenkins Plugin) allows us to automate this by selecting "Build when a change is pushed to GitHub".

Setup Your Deployment Script

Here’s where we’ll implement the example deployment script I wrote about earlier in the post. This is the meat and potatoes of our job, or simply put, this is what Jenkins is going to do now that it finally knows how/when to do it.

Under "Build" choose “Add build step” and select “Execute shell”. Depending on your installed plugins, your list of options might vary.

Then add your deployment script to the textarea that Jenkins exposed to you. If you want somewhere to start, here is a gist of my job from above. When you’ve added your script it should look something like this:

Last Step! Enable Slack Notifications

Although earlier in the job we configured our Slack integration, we still need to tell Jenkins to send any/all notifications back to Slack when a build is complete. You do this sort of thing under the "Add post-build action" menu. Select “Slack Notifications” and you’re good to go.

Our deployment job for Drupal 8 is now complete! Click "Save" and you should be able to start testing your deployments. To test the job itself, you can simply press “Build Now” on the following screen OR you can test your GitHub integration by making any change on the master branch (or whichever branch you configured). With the setup I’ve covered here, Jenkins will respond automatically to merges and hot-fix style commits to master. That is to say, when a PR is merged or when someone commits directly to master. Of course no one on your team would ever commit directly to master, would they?!

Wrapping Up

Assuming everything is setup properly, you should now have a robust automatic deployment system for your Drupal 8 project! Having your deployments automated in this way keeps them consistent and adds transparency to your entire team.

Categories: Elsewhere

Chromatic: Creating Links Within Twig Templates Using path() and url()

Wed, 17/02/2016 - 23:36

Drupal 8 comes packed with loads of great new features, APIs and developer tools. There are sweeping changes aplenty. Not the least of which is a brand new templating system called Twig. Twig is the bee's knees and a welcome improvement over the much maligned PHPTemplate. However, many front-end developers who've grown accustomed to the templating ways of old might feel a bit lost as they enter the strange, new world of Drupal 8: Twig and render arrays. Furthermore, the Drupal community is still learning, documentation is still being written, etc.

As we approach the completion of our first official Drupal 8 project here at Chromatic, I thought it would be helpful to start sharing a bit of what I've learned along the way, starting with some examples on linking from within Twig templates.

If you want to just see how the hell you link to things from within Twig templates, skip these next bits on context.

Some Context

In Drupal 7 and versions prior, we used paths to define destinations for our content, APIs, etc. In Drupal 8, these are abstracted into routes. Where in Drupal 7 you would define a custom page via hook_menu(), like so:

<?php function chromatic_conact_menu() { $items['contact'] = array( 'title' => 'Chromatic Contact', 'page callback' => 'contact_page', 'access arguments' => array('access content'), 'type' => MENU_SUGGESTED_ITEM, ); return $items; } ?>

In Drupal 8, you define this in a special routing file that follows this naming convention: module_name.routing.yml. Here's a similar example, where chromatic_contact_contact is our route name and contact is the internal path where it can be accessed.

chromatic_contact_contact: path: 'contact' defaults: _form: '\Drupal\chromatic_contact\Form\ChromaticContactForm' _title: 'Contact Us!' requirements: _permission: 'access content'

Here's how Drupal 8 sets up the route for node type creation:

node.type_add: path: '/admin/structure/types/add' defaults: _entity_form: 'node_type.add' _title: 'Add content type' requirements: _permission: 'administer content types' Got it. Routes are the new hotness in D8. Now why does this matter to me? I'm a front-ender.

Glad you asked. Routes matter because if you want to generate URLs to custom pages from your templates and you want to do it properly, you need to understand routes. The old methods of using paths are "dead in Drupal 8".

The url() and path() functions are how we handle this type of linking in D8. These functions don't expect an internal path like you're probably familiar with in prior versions of Drupal. Instead, they expect proper system routes. The type of thing you now see in module_name.routing.yml.

So what's the difference?
  • path() - Generates a [relative] URL path given a route name and parameters.
  • url() - Generates an absolute URL given a route name and parameters.
Examples

// Link to the default frontpage content listing view: <a href="{{ path('view.frontpage') }}">{{ 'View all content'|t }}</a> // Link to a specific node page: <a href="{{ path('entity.node.canonical', {'node': node.id}) }}">{{ 'Read more'|t }}</a> // Link to a specific user profile page: <a href="{{ path('entity.user.canonical', {'user': user.id}) }}">{{ 'View user profile'|t }}</a> // Link to a view, and throw in some additional query string parameters: <a href="{{ path('view.articles.page_1', {'page': 2}) }}">{{ 'Go to page 2'|t }}</a> // Link to a view and pass in some arguments to the view: <a href="{{ path('view.recent_articles_by_author.page_1', {'arg_0': user.field_display_name.value|drupal_escape }) }}">{{ 'See all the articles written by'|t }} {{ user.field_display_name.value }}</a>

The source code for the path() and url() Twig extensions can be found here within Drupal 8 core: /core/lib/Drupal/Core/Template/TwigExtension.php.

If you're curious about how all of this came to pass, here's the D.O issue on which the work occurred: https://www.drupal.org/node/2073811. It's long. Really long. The tldr; is basically a lot of back and forth over how routes are a barrier to entry for beginners and that there should still be an easy way to use paths. I honestly see both sides, but in the end, routes won out in the name of consistency and compatibility. The original patch contained a urlFromPath() function but it has since been removed. :-/. Beginner or not, you should now understand how to generate URLs within your templates. Now go! Commence Twigging!

One more thing!

The Drupal.org documentation page that explained these Twig functions (and others) was marked as incomplete, so I took this opportunity to finish it up. If you have other examples to share, please do so there!

Categories: Elsewhere

Chromatic: Drupal Camp Chattanooga 2015

Wed, 17/02/2016 - 23:36

A few weekends ago I was fortunate enough to attend my first Drupal Camp ever. What was even more fortunate for me was that it was located near where I grew up in Chattanooga, TN. I’ve been to several leadership/business conferences in my life, but this is the first one that I’ve been to where it felt like everyone was genuinely glad to be there, even on a rainy fall day. In the spirit of open source, it wouldn’t feel right to keep all of the helpful info I learned to myself, so I wanted to take a moment and share some of my key take-aways.

Selling Drupal

Bob Snodgrass from net2Community gave a great seminar on sales. He started his presentation on selling Drupal by stating "Don’t sell Drupal, sell results!" So many times we become entrenched in the technology that we use and sell, and that’s ok!, but we have to remember that clients are more interested in the results we can provide, instead of the medium we use to get there. A good example given was a donut in a donut box. People rarely care about what box it comes in, they just want the donut!

Bob also brought attention to how each team member contributes to "selling", whether they realize it or not. The interactions, work and relationships that we manage on a daily basis help clients decide if they will continue to work with us or recommend us to their work colleagues.

Becoming a PM Ninja

Justin Rhodes from Commerce Guys led a good discussion on increasing efficiencies as a Project Manager. One of the first things we discussed was Parkingson’s Law: tasks will swell or shrink according to the amount of time given to them. If you use this law to your advantage, you can create small window deadlines to create efficiencies in your development process.

When it comes to planning and estimating phases for a project, Justin shared the success he has had doing this by using relativity to help the team give input. For example, rather than trying to nail down hours for each process step right way, you can tell everyone to estimate the size of the task by using shirt sizes (S, M, L, XL) to align everyone by using relative sizes. This can give the resource planner a clear visual of which parts of the project are going to take the longest. Of course, there are no complete replacements for old fashioned hourly estimates. The key to success on estimates is to check the team’s accuracy by comparing estimates against actual spend.

Contributing to Open Source

Our keynote speaker was Mike Anello from Drupal Easy. He gave a great talk on why we should be contributing back to the Drupal community. It wasn’t the typical, do it because it is the right thing to do, lecture. Mike made the case that not only does contributing back to the community grow your network and visibility, but it has a very real probability of increasing your bottom line. It is easy to say that we are going to contribute, but unless you deliberately set aside time, it probably isn’t going to happen. One way to make contributing a priority is to work it into your marketing budget. Switching your perspective on contributing from a way to get karma points to a powerful marketing tool, will change your priorities around where it fits into the company’s strategy.

The last take away I got from Mike’s talk was that even those of us who are nontechnical team members can contribute back to the Drupal community. Reaching out to module owners and asking how you can help, cleaning up ticket queues on modules, confirming bugs and assisting in documentation, are all ways to help move the ball forward (and I’m told they are greatly appreciated). Don’t forget that Drupal events don’t just happen by themselves. There is always a need for folks to help with the planning and coordination of meetups, camps and conventions.

Final Thoughts

Although I was unable to attend the pre and post party, I have no doubt that the enthusiasm of the group spilled into every conversation that was had. Who knows where I will be living next year (the benefits of working for a distributed company), but if I am able to, I’ll be returning to DrupalCamp Chattanooga!

(photo credit goes to @DrupalNooga)

Categories: Elsewhere

Chromatic: TheaterMania: Lessons Learned on Localization

Wed, 17/02/2016 - 23:36

We recently launched a new site for an existing client, TheaterMania. We helped launch and currently maintain and develop The Gold Club, which is a subscription-based discount theater club in New York City. The new site is the same thing, but in London - same language, same codebase, new database, different servers. We only had to migrate users, which were already exported for us, so nothing exceptional there. Shouldn't be a big deal, right? We learned that's not always the case.

Architectural Decisions

One of our first problems, besides the obvious localization issues (currency, date formats, language), was to decide what we were shipping. Were we just building another site? Were we packaging software? There will most likely be more sites in other cities in the future - how far did we want to go in terms of making this a product that we could ship? In the end, we wound up going somewhere in the middle. We had to decide initially if we would use Organic Groups to have one site with multiple "clubs," one Drupal multisite installation, or multiple Drupal installations. The final decision was to combine the latter two choices - we created multisite-style directories so that if we need to take the site in a multi-site direction, we can easily do that. The sites each have a site-specific settings file, full of various configuration variables.

Now that the site has been launched, we're not sure if this list of variables will be developer-friendly moving forward, and have been keeping in mind that we may want a more elegant solution for this. The best part about this setup is that we have one codebase, one master branch, and each site is configured to use the appropriate settings. The most important thing is that this is all very thoroughly documented, both in the code, README files, and the repo wiki.

Currency & Recurly: Easier than Expected

One of the issues I thought would be very problematic was currency, but that wasn't actually an issue. All of the existing transactions are set up in cents - ie, 100 instead of 1.00 for a dollar, and that translates perfectly from dollars to pounds. We use Recurly, an external payment and subscription processor, so we didn't have to worry about any localization issues on that front. Most of the currency abstractions I did were to remove any hard-coded references to the dollar sign, and create functions and variables to get the appropriate currency symbol.

Dealing with Dates; Ugh.

Date formats were something I expected to be easy, but that wound up being more complex. I discovered hook_date_combo_process_alter() to change the display of the date in calendar popup fields. This made what I’d thought was going to be a difficult series of view handlers really simple. We have several fields using the date combo box on both content types and entities, and this function took care of them.

/** * Implements hook_date_combo_process_alter(). * * Changes the date format. */ function gc_display_date_combo_process_alter(&$element, &$form_state, $context) { if (isset($element['#entity']->type)) { switch ($element['#entity']->type) { case 'event': $element['value']['#date_format'] = variable_get('date_format_short'); break; case 'partner': $element['value']['#date_format'] = variable_get('date_format_short'); $element['value2']['#date_format'] = variable_get('date_format_short'); break; case 'promo_offer': $element['value']['#date_format'] = variable_get('date_format_short'); $element['value2']['#date_format'] = variable_get('date_format_short'); break; default: break; } } elseif (isset($element['#entity']->field_name)) { if ($element['value']['#instance']['widget']['type'] == 'date_popup' && $element['#entity']->field_name == 'field_user_csr_notes') { $element['value']['#date_format'] = variable_get('date_format_short'); } } }

I took the dozen or so existing date formats from Drupal, altered some of them to meet our needs, and added a few more. My head also started spinning when testing because I'm so used to M/D/Y formats that D/M/Y formats look really strange after a while, especially because code changes needed to be tested on the US and UK sites, so I had to be really careful when visually testing a page to make sure that a US page was showing 9/1/15 and the UK page was showing 1/9/15. In the future, I’d definitely advocate for a testing suite on a project like this. Overall, making sure all of the dates were changed was somewhat tedious, but not difficult. It required a lot of attention to detail and familiarity with PHP date formats, and vigorous testing by the whole team to make sure nothing had been missed.

Proper Use of t() Early == Wins Later

This project made me extremely grateful for the t() function. Since both sites were in English, we didn't have a need for site-wide translation, but we did need to localize a handful of strings, both for language issues (words like 'personalize' vs 'personalise'), and the general language preference of the stakeholders. It was easy enough to find the strings and list them in locale_custom_strings_en to switch them out. One gotcha we came across that I wasn't familiar with - you cannot use t() in your settings files. The function isn't available at that point in the bootstrapping. You can use get_t(), but we opted to remove the translation strings from any variables and make sure that t() was used when the variable was called. This wasn't something I had run into before, and it caused some problems before we figured it out.

Miscellany

A few tricky miscellaneous problems cropped up, too. There was a geolocation function enabled in Recurly, which was defaulting to the US and we were unable to change the settings - we also didn't realize this when testing in the US, and we scratched our heads when the London team told us the field was defaulting to US until we came across the culprit. We were able to fix it, and put in a patch for the library causing the issue.

I also realized how many various settings default to the US when working on this project - a lot of the location-related work was just abstracting out country defaults. Something to keep in mind if you're working on a project with locations. Don't make more work for developers who live or work on projects outside of the US. Plan for the future! Assume nothing!

Looking Back

I'm really glad that I worked on this project, because it's made me develop with a better eye for abstraction of all kinds, and making sure that it's easy for developers or users to work with my code anywhere. In the future, I’d put more thought into managing our configurations from the start, as well as automating the testing process, both for time-saving and better QA.

If you’ve ever worked on a site with challenges like these, I’d love to hear how you handled them! What are your best practices for managing custom locale strings and other site-specific variables? To what extent do you abstract things like dates and currency when developing a site, even when you don’t know if those will ever change?

Categories: Elsewhere

Chromatic: BADCamp 2015: Transitioning From theme() and Theme Functions to Render Arrays and Templates

Wed, 17/02/2016 - 23:36

I was fortunate to attend and speak at BADCamp for the first time this year. BADCamp is the Bay Area Drupal Camp, held annually in Berkeley, CA. I don't know the official numbers, but I believe over 1,000 were in attendance, giving it the feel of a smaller DrupalCon. The camp was well organized, the sessions were high quality, and I met and got to know quite a few great people.

Anyone who wasn't able to attend can watch my session, 7 to 8: Transitioning From theme() and Theme Functions to Render Arrays and Templates, here:

My slides are also available online. The video and slides include in-depth examples, but for the TL;DW crowd more interested in the key takeaways:

Render Arrays > theme()

The theme() function no longer exists in Drupal 8. Instead, render markup by passing render arrays to the template. Start using render arrays instead of theme() in Drupal 7 right now. Not only will it make the code easier to port to Drupal 8, but there are other advantages.

Using render arrays allows for the data to remain an array until the template renders it into markup. Waiting to render markup allows other modules or themes to intercept and alter the data before it is rendered. This is typically done with preprocess hooks.

Templates > Theme Functions

When Drupal 7 goes to render markup, it does so with either a theme function, such as theme_image(), or a template. Theme functions contain a lot of logic and are a tough way to write and alter markup. Overriding theme functions also involves copying a significant amount of code to make changes.

Instead of writing theme functions, write templates. Keep the logic in preprocess and process functions. This separation will make altering the logic and markup much easier. Other modules or themes can easily use preprocess hooks or override templates as needed.

Drupal 8 has taken this approach. While theme functions can still exist in Drupal 8, all core theme functions were converted to templates.

More information

For more on this topic, check out these resources on drupal.org:

There were also a number of other sessions at BADCamp related to this topic. They included:

Thanks again to the BADCamp organizers. I hope to see everyone again next year!

Categories: Elsewhere

Chromatic: Drupal 8 Configuration Management - Solving the Configuration Conundrum

Wed, 17/02/2016 - 23:36

A common difficulty in web development is keeping configuration consistent between environments. Drupal keeps most configuration settings in the database and it can be a pain to keep databases across environments synchronized. Solving this conundrum was one of Drupal 8's goals and the focus of D8's Configuration Management Initiative (CMI). All site configuration is saved in YAML files (hooray for version control!) and can be shared between site environments.

The Way We Were

Let's take a classic example from before D8, using the beloved Views module. When you build a great new view on your local machine you still need to get it onto the production site. You could repeat what you did on your local machine and build the View by hand on prod, but that's just repeating all the work you have already completed, not to mention, prone to user error. Most developers are too lazy to do that anyways, so they depend on Drupal's Features module. The Features module works by exporting db configuration settings into Drupal module code which can be shared amongst environments. It usually works pretty well, but it turns out the Features module was not built for handling deployment and configuration. These tasks have fallen into its lap because, until now, it was the only game in town.

The Way We're Going

With every Drupal 8 installation, the configuration system creates a configuration directory*, sync, to hold the aforementioned YAML configuration files. The directory is actually created as a sub-directory in sites/default/files, into a folder with a long, random hash as its name, prefaced by config_. (Including a random hash makes it near impossible for bad guys to guess the configuration folder name.)

* At the time of writing the default installation actually created two directories: active and staging, but that was changed to just one, the staging directory which subsequently was changed to sync. The blog post has been updated to reflect this.

In picture form...a folder with a really, really long, random name

So in the above screenshot the configuration directory's name is

config_Z1fHk5YnKXiTdJl9lAfz_dqGmrMTzqT9lNnbUF6z4kwKxglnC8srJZBTcI1dIMSCOmOwvEMZ5g

and the full path of the sync directory is:

sites/default/files/config_Z1fHk5YnKXiTdJl9lAfz_dqGmrMTzqT9lNnbUF6z4kwKxglnC8srJZBTcI1dIMSCOmOwvEMZ5g/sync

So that's where the site configuration files live, but by default, for reasons of speed and security, the actual configuration settings are saved in the database. So all the configuration from those YAML files actually gets imported into the database.

The configuration management system provides a UI for uploading and double-checking changes before applying them to the live site. (Of course, there are Drush commands, drush config-export and drush config-import that allow you to import/export through the command line too.)

How do you Feel About Clones?

Now hold on, you can't upload one site's configuration to any old site; you can only synchronize configuration settings between cloned instances of a site. This restriction was a conscious decision made by the Drupal CMI team. So you need to start with one environment and then clone it. If you don't do this, Drupal simply will not allow configuration settings to be imported.

Steps for cloning a site:
  1. Do a db dump of the original database and import it into an empty database for your new clone environment.

  2. Assuming you are using git, (you are using git, right?) you can run git clone to clone the original site files into the clone environment.

  3. Check folder permissions (Did .htaccess come across? Does sites/default/files have sufficient permissions?).

  4. Change settings.php in cloned site to point to the fresh db from step 1.

Assuming your cloned site is working, you are ready to import settings from one environment to another. In my case, I created a new view called bookshelf that needs to get to prod. Below are screenshots of what the process looks like.

Export and Download from Dev

In this screenshot, I have just exported and downloaded my dev environment's configuration tar.gz file.

Upload into Prod

Having switched to my prod environment, I select the downloaded tar.gz file and upload it.

Successful Upload

After a successful upload you can see that there are changes. Note that my new bookshelf view is there. I can click the View differences button to compare the changes.

Vive la Difference

Having clicked the View differences button, I see the changes that are about to be made.

Import All

Satisfied with what I see, I can click Import all to apply the changes.

Import Progress

A progress bar displays as Drupal imports the settings.

A Site With a View

Voilà, the new view is on my prod site.

Whew, so there were a bunch of steps to get things set up, but once you have your environments in place, it's easy to move settings between them. Of course, in a full-on, professional development environment you would harness the power of Drush and use scripts to do all the dirty work. But that's a tutorial for another time.

Categories: Elsewhere

Chromatic: Programatically Creating and Storing WordPress Migrate Migrations in Drupal

Wed, 17/02/2016 - 23:36

Migrations are never glamorous, but doing them right and verifying their integrity is essential to their success. The WordPress Migrate module gives you an easy turnkey solution to migrating content into Drupal from WordPress. It allows you to create each migration through an interactive admin form, allowing you to configure your migration entirely through the UI. This is great, but it does not make creating or storing the resulting migrations easy to manage across multiple environments, since the migrations are not defined in code like a typical Migrate class. Short of copying database tables or re-entering the configuration through the admin forms, developers are stuck with the migrations stored in a single database and thus it is not easy to move to other environments for testing or further development.

Copying data tables is almost always the wrong solution and manually re-entering all of the migrations would be way too time consuming, so our solution was to create the migrations programmatically. To do this, we hooked into the existing WordPress Migrate codebase and used its logic to build programmatically, what it builds from data input to its admin forms. Then we are able to define all of our migration sources in code and instantly create all of our migrations in a new environment, or recreate them after something fails during development.

As mentioned, this solution relies upon programmatically submitting admin forms, which is often not an ideal scenario. Additionally, there is the almost inevitable request to add additional customizations beyond what Wordpress Migrate supports out of the box. Sometimes this makes WordPress Migrate more of a hinderance than a help. So why not just create a custom Migrate class from the outset and avoid all of these issues? Here are some factors to consider:

  • Writing a custom Migrate class for your WordPress content always sounds more appealing until you run into problems and realize WordPress Migrate already solved those issues.
  • The WordPress Migrate module offers a lot of functionality, including file transfer, author migration, embedded video processing, internal link rewriting, comment migration, etc.
  • You might not need much custom code and just tweaking the WordPress Migrate functionality by extending one of its classes will easily do the trick.
  • You might not have the resources (time, knowledge, etc.) to write a custom Migrate class.
  • Running and testing the migrations on multiple environments might not be in your workflow, although I would argue it should be.
  • You might only have one or two WordPress sites to migrate content from, so manually re-creating them is not an issue.

If after weighing all of the factors, you decide using the WordPress Migrate module is in your best interest and manually recreating the migrations is not an option, then follow along as we walk you through our approach to creating and storing WordPress Migrate migrations programmatically.

Our Solution

First we need to define the list of source blogs. The keys of each item in this array can be added to as needed to override the default values we assign later.

/** * Define the WordPress blogs to be imported. */ function example_wordpress_migrate_wordpress_blogs() { // Any key not set here will default to the values set in the // $blog_default_settings variable in the drush command. $blogs = array( array( 'domain' => 'www.example.com/site-one/', ), array( 'domain' => 'www.example.com/site-two/', ), array( 'domain' => 'www.test.com/', ), ); return $blogs; }

Next we'll create a custom drush command so that we can easily trigger the creation of our migrations from the command line.

/** * Implements hook_drush_command(). */ function example_wordpress_migrate_drush_command() { $items = array(); // Creates WordPress migrations. $items['example-migrate-create-wordpress-migrations'] = array( 'description' => 'Creates the WordPress migrations.', 'aliases' => array('mcwm'), ); return $items; }

Be sure to note the example_migrate_wordpress_password variable below, as you will need to ensure you set that in settings.php before creating the migrations. The WordPress Migrate code needs to be able to login to your site to download the source XML file, and a password is paramount to the success of that operation!

/** * Callback for WordPress migration creation drush command. */ function drush_example_wordpress_migrate_create_wordpress_migrations() { // Reset the file_get_stream_wrappers static cache so the 'wordpress' stream // wrapper created by the wordpress_migrate module is available. $wrappers_storage = &drupal_static('file_get_stream_wrappers', NULL, TRUE); // The wordpress_migrate module's UI is a multi-step form that collects all // configuration needed to migrate a given blog. As this form's steps are // submitted and validated, an export file is downloaded for each blog and its // contents are migrated. There is no easy way to export these settings or use // code to provide that configuration and then trigger a migration, so the best // bet is simulate the submission of those form steps with the needed data. module_load_include('inc', 'migrate_ui', 'migrate_ui.wizard'); // Get a list of blogs to migrate. $blogs = example_migrate_wordpress_blogs(); $blog_default_settings = array( 'source_select' => '1', 'domain' => '', 'username' => 'admin', 'password' => variable_get('example_migrate_wordpress_password', ''), 'wxr_file' => NULL, 'do_migration' => 0, 'default_author' => 'admin', 'page_type' => '', 'blog_post_type' => 'story', 'path_action' => 1, 'tag_field' => '', 'category_field' => '', 'attachment_field' => '', 'text_format' => 'filtered_html', 'text_format_comment' => 'filtered_html', ); // Import each of the blogs. foreach ($blogs as $blog_settings) { // Combine the default settings and the custom per blog settings. $blog_settings = array_merge($blog_default_settings, $blog_settings); // Skip the import if no username or password was found. if (empty($blog_settings['username']) || empty($blog_settings['password'])) { $message = t('The :site-name migration was not created since no username and/or password could be found. Verify that the example_migrate_wordpress_password variable has been set.'); $replacements = array( ":site-name" => $blog_settings['domain'], ); drupal_set_message(t($message, $replacements), 'warning'); continue; } // Set the form state values. $form_state['values'] = $blog_settings; // Store the values so we can use them again since $form_state is // a reference variable. $form_state_values = $form_state['values']; // Build the import form. $form = drupal_get_form('migrate_ui_wizard', 'WordPressMigrateWizard'); $form = migrate_ui_wizard($form, $form_state, 'WordPressMigrateWizard'); // Create a Migrate Wizard object. $form_state['wizard'] = new WordPressMigrateWizard(); // Set the number of steps in the form. $form_steps = 6; // Go through all of the steps. foreach (range(1, $form_steps) as $step) { // Validate the form data. $form_state['wizard']->formValidate($form_state); // Submit the form page. migrate_ui_wizard_next_submit($form, $form_state); // Put any values removed from the array back in for the next step. $form_state['values'] = array_merge($form_state_values, $form_state['values']); } // Submit the form. drupal_form_submit('migrate_ui_wizard', $form_state); // Save the settings into the wizard object. $form_state['wizard']->formSaveSettings(); // Notify the user that the migration was created successfully. $replacements = array( '@site-name' => $blog_settings['domain'], ); $message = t('The @site-name migration was successfully created.', $replacements); drupal_set_message($message, 'success'); } }

With all of this in place, the source WordPress sites and the configuration needed to import them are now fully defined in code along with a custom Drush command to create the required migrations. No longer will each individual site need to be re-entered through the UI introducing opportunities for mistakes and wasted time.

Now when you are in a new environment or after you reset your migrations, you can simply run drush mcwm.

Following its successful completion, the following are done for you:

  • A new Migrate group is created for each individual blog.
  • The actual Migrate classes within each group that migrate, authors, content, terms, and attachments are created and configured as defined in the code.
  • The source WordPress XML file is downloaded for each site and stored in wordpress://.

Then simply run drush ms to verify everything was created successfully, and you are ready to migrate some content!

Now that you have the tools and knowledge to evaluate your unique migration needs, you can make a well informed decision if this approach is right for you. However, we think that more often than not, all of the incredible functionality you get pre-built with the WordPress Migrate module will outweigh the issues that arise from not being able to fully build and store your migrations in code, especially when you add the functionality outlined above that gets you the best of both worlds. So have your cake and eat it too, and define your migrations in code and utilize the WordPress Migrate module while you are at it!

If you decide to go this route, all of the code referenced here is available in this gist. Please note that this was all built for WordPress Migrate 7.x-2.3, so future updates to the module could break this functionality.

Categories: Elsewhere

DrupalEasy: Contact + Contact Storage (contrib) Module as a Drupal 8 Webform/Entityform Replacement

Wed, 17/02/2016 - 23:06

Looking to migrate your Drupal 6 or 7 site to Drupal 8 but can't do it yet because your site it too reliant on the Webform (or Entityform) module? There's actually very elegant solution that is ready to go today. The core Drupal 8 "Contact" module is now a fully-fledged fieldable entity - meaning you can create various "contact form" types with different sets of fields. Like Entityform, you can use any field types provided by Drupal core as well as contributed modules. When a contact form is submitted, the data is emailed to the assigned addresses.

The Contact Storage module provides the functionality missing from the core Contact form - the ability to save user submissions in the database. Furthermore, Contact Storage stores all submissions in a way that makes them automatically compatible with Views!

The Drupal 8 version of DrupalEasy.com uses exactly this technique for all of our forms, including our main contact form, our Drupal Career Online application form, as well as our various training student surveys. 

Categories: Elsewhere

OSTraining: Using Entity Reference Fields in Drupal Views

Wed, 17/02/2016 - 22:29

An OSTraining member came to us with a question. The member had two content types that were linked by an Entity Reference field. They wanted to use Views to automatically show import from one content type to another.

In this tutorial, I'm going to show you how to use Entity Reference fields inside Views.

Categories: Elsewhere

Jason Pamental's RWT.io Blog: Beep Edition comes to Drupal8

Wed, 17/02/2016 - 16:42

I’ve never had particularly lofty ambitions for Beep Edition, the responsive base theme I started a few years ago. Rather than make a new ‘all singing, all dancing’ kind of theme, I wanted this to be my starting point, and include all of my collected experience in responsive design, accessibility, typography and general good practice.

Categories: Elsewhere

Promet Source: On the Road to DrupalCon Asia

Wed, 17/02/2016 - 16:41

Read some more updates from Promet's journeys across the Drupal-verse:

Categories: Elsewhere

Acquia Developer Center Blog: Drupal 8 Module of the Week: Metatag

Wed, 17/02/2016 - 16:19
Jeffrey A. "jam" McGuire

Each day, more Drupal 7 modules are being migrated over to Drupal 8; new modules are also being created for the Drupal community’s latest major release. In this series, the Acquia Developer Center is profiling some of the most prominent, useful modules available for Drupal 8. This week: Metatag.

Tags: acquia drupal planetmetatagmodules
Categories: Elsewhere

Cheppers blog: Big migration guide to Drupal 8 - Part I

Wed, 17/02/2016 - 16:16

Migrations are becoming a crucial part of Drupal 8, especially in connection with the upcoming Drupal 6 EOL. In this first article, we are going to show you how to quickly migrate your site to Drupal 8.

Categories: Elsewhere

myDropWizard.com: You need to update libc on all your Linux servers NOW!

Wed, 17/02/2016 - 13:35

If you don't follow security news, you might not be aware of the libc vulnerability published yesterday. The vulnerability was introduced in 2008 - so, it's likely this affects all Linux servers (and desktops, if you run Linux on the desktop) that you're responsible for.

It's a remote code execution vulnerability, meaning attackers can run arbitrary code on your server. It's in the function used to lookup domain names, so it can be exploited by any code that requests a URL, by giving it a special domain name!

While this isn't a Drupal vulnerability, it could definitely be exploited from Drupal, if you provide a way for users to ask the server to request a URL.

The first example that comes to mind, is if you're using the Feeds module, and allow authenticated users to create a new feed (either in the admin UI, or by creating a special piece of content). All they need to do is enter a URL with a special domain name, and when Feeds tries to download it, they'd be able to execute code on your server!

But there's probably countless other examples...

So, if you didn't update yesterday, then be sure to update today!

Categories: Elsewhere

InternetDevels: How we estimate our Drupal web development projects

Wed, 17/02/2016 - 13:17

In one of our previous blog posts, we shared our
project management experiences. Now it’s time to take a
closer look at the way we estimate Drupal projects.

Read more
Categories: Elsewhere

Lullabot: Javascript aggregation in Drupal 7

Wed, 17/02/2016 - 12:20

Javascript aggregation in Drupal 7

What is it? Why should we care?

Javascript aggregation in Drupal is just what it sounds like: it aggregates Javascript files that are added during a page request into a single file. Modules and themes add Javascript using Drupal’s API, and the Javascript aggregation system takes care of aggregating all of that Javascript into one or more files. Drupal does this in order to cut down on the number of HTTP requests needed to load a page. Fewer HTTP requests is generally better for front-end performance.

In this article we’ll take a look at the API for adding Javascript, paying specific attention to the options affecting aggregation in order to make best use of the system. We’ll also take a look at some common pitfalls to look out for and how you can avoid them using Advanced Aggregation (AdvAgg) module. This article will mostly be looking at Drupal 7, however much of what we’ll cover applies equally to Drupal 8, and we’ll take a look specifically at what’s changed in Drupal 8.

If fewer is better, why multiple aggregates?

Before we dig in, I said that fewer HTTP requests is generally better for front-end performance. You may have noticed that Drupal creates more that one Javascript aggregate for each page request. Why not a single aggregate? The reason Drupal does this is to take advantage of browser caching. If you had a single aggregate with all the Javascript on the page, any difference in the aggregate from page to page would cause the browser to download a different aggregate with a lot of the same code. This situation arises when Javascript is conditionally added to the page, as is the case with many modules and themes. Ideally, we would group Javascript that appears on every page apart from conditionally added Javascript. That way the browser would download the every page Javascript aggregate once and subsequently load it from cache on other page requests on your website. Drupal’s Javascript aggregation attempts to give you the ability to do exactly that, striking a balance between making fewer HTTP requests, and leveraging browser caching.

The Basics

Let’s take a step back and briefly go over how to use Javascript aggregation in Drupal 7. You can turn on Javascript aggregation by going to Site Configuration > Development > Performance in your Drupal admin. Check off "Aggregate Javascript files" and Save. That’s it. Drupal will start aggregating module and theme Javascript from there on out.

Using the API

Once you’ve enabled Javascript aggregation, great! You’re on your way to sweet, sweet performance gains. But you’ve got modules and themes to build, and Javascript to go along with them. How do you make use of Javascript aggregation provided by Drupal? What kind of control do you have over the process?

There are a couple API entry points for adding Javascript to the page. Most API entry points have parameters that let you tell Drupal how to aggregate your Javascript. Let’s look at each in turn.

drupal_add_js

This is a loaded function that is used to add a Javascript file, externally hosted Javascript, a setting, or inline Javascript to the page. When it comes to Javascript aggregation, only Javascript files are aggregated by Drupal, so we’re going to focus on adding files. The other types (inline and external) do have important impacts on Javascript aggregation though, which we’ll get into later.

hook_library/drupal_add_library

Drupal also provides a mechanism to register Javascript/CSS "libraries" associated with a module. This is nice because you can specify the options to pass to drupal_add_js in one place. That gives you the flexibility to call drupal_add_library in multiple places, without having to repeat the same options. The other nice thing is that you can specify dependencies for your library and Drupal will take care of resolving those dependencies and putting them in the right order for you.

I like to register every script in my module as a library, and only add scripts using drupal_add_library (or the corresponding #attached method, see below). This way I’m clearly expressing any dependencies and in the case where my script is added in more than one place, the options I give the aggregation system are all in one place in case I need to make a change. It’s also a nice way to safely deprecate scripts. If all the dependencies are spelled out, you can be assured that removing a script won’t break things.

Instructions for using hook_library and drupal_add_library are well documented. However, one important thing to note is the third parameter to drupal_add_library, every_page, which is used to help optimize aggregation. At registration time in hook_library, you typically don’t know if a library will be used on every page request or not. Especially if you’re registering libraries for other module authors to use. This is why drupal_add_library has an every_page parameter. If you’re adding a library unconditionally on every page, be sure to set every_page parameter to TRUE so that Drupal will group your script into a stable aggregate with other scripts that are added to every page. We’ll discuss every_page in more detail below.

#attached

attached is a render array property that lets you add Javascript, CSS, libraries and arbitrary data to the page. #attached is the preferred approach for adding Javascript in Drupal. In fact, drupal_add_js and drupal_add_library are gone in Drupal 8 in favor of #attached. #attached is nice since it is part of a render array and can be easily altered by other modules and themes. It’s also essential for places where render arrays are cached. With caching in play, functions that build out a render array are bypassed in favor of a cached copy. In that scenario, you need to have your Javascript addition included in #attached on the render array, because a call to drupal_add_js/drupal_add_library in your builder function would otherwise be missed.

Drupal developers often fall back on drupal_add_js for lack of a render array in context. It’s common to add Javascript in places like hook_init, preprocess and process functions, where there is no render array available to add your Javascript via #attached. In these scenarios, consider whether your Javascript is more appropriately added via a hook_page_build, or hook_node_view, where there is a render array available to add your script via #attached. Not only will you begin to align yourself with the Drupal 8 way of doing things, but you open yourself up to being able to use the render_cache module, which can gain you some performance. See the documentation for drupal_process_attached for more information.

Controlling aggregation

Let’s take a close look at the parameters affecting Javascript aggregation. Drupal will create an aggregate for each unique combination of the scope, group, and every_page parameters, so it’s important to understand what each of these mean.

scope

The possible values for scope are ‘header’ and ‘footer’. A scope of ‘header’ will output the script in the

tag, a scope of ‘footer’ will output the script just before the closing tag.

group

The ‘group’ option takes any integer as a value, although you should stick to the constants JS_LIBRARY, JS_DEFAULT and JS_THEME to avoid excessive numbers of aggregates and follow convention.

every_page

The ‘every_page’ option expects a boolean. This is a way to tell the aggregation system that your script is guaranteed to be included on every page. This flag is commonly overlooked, but is very important. Any scripts that are added on every page should be included in a stable aggregate group, one that is the same from page to page, such that the browser can make good use of caching. The every_page parameter is used for exactly that purpose. Miss out on using every_page and your script is apt to be lumped into a volatile aggregate that changes from page to page, forcing the browser to download your code anew on each page request.

What can go wrong

There are a few problems to watch out for when it comes to Javascript aggregation. Let’s look at some in turn.

Potential number of aggregates

The astute reader will have noticed that there is a potential for a lot of aggregates to be created. When you combine scope, group and every_page options, the number of aggregates per page can get out of control if you’re not careful. With two values for scope, three for groups and two for every_page, there is potential to climb to twelve aggregates in a page. Twelve seems a little out of control. Sure, we want to leverage browser caching, but this many HTTP requests is concerning. If you’re getting high into the number of aggregates it’s likely that modules and themes adding Javascript are not using the API as best as they could.

Misuse of Groups

As was mentioned, JS_LIBRARY, JS_DEFAULT and JS_THEME are default constants that ship with Drupal to group Javascript. When adding Javascript, modules and themes should stick to these groups. As soon as you deviate, you introduce a new aggregate. Often when people deviate from the default groups, it’s to ensure ordering. Maybe you need your script to be the very first script on the page, so you set your group to JS_LIBRARY - 100. Instead, use the ‘weight’ option to manage ordering. Set your group to JS_LIBRARY and set the weight very low to ensure your first in the JS_LIBRARY aggregate. Another issue I see is themes adding script explicitly under the JS_THEME group. This is logical, after all, it is Javascript added by the theme! However, it’s better to make use of hook_library, declare your dependencies and let the group default to JS_DEFAULT. The order you need is preserved by declaring your dependencies, and you avoid creating a new aggregate unessesarily.

Inline and External Scripts

Extra care should be employed when adding external and inline scripts. drupal_add_js and friends preserve order really well, such that they track the order the function is called for all the script added to the page and respect that. That’s all well and good. However, if an inline or external script is added between file scripts, it can split the aggregate. For example if you add 3 scripts: 2 file scripts and 1 external, all with the same scope, group, and every_page values, you might think that Drupal would aggregate the 2 file scripts and output 2 script tags total. However, if drupal_add_js gets called for the file, then external, then file, you end up getting two aggregates generated with the external script in between for a total of 3 script tags. This is probably not what you want. In this case it’s best to specify a high or low weight value for the external script so it sits at the top or bottom of the aggregate and doesn’t end up splitting it. The same goes for inline JS.

Scripts added in a different order

I alluded to this in the last one, but certain situations can arise where scripts get added to an aggregate in a different order from one request to the next. This can be for any number of reasons, but because Drupal is tracking the call order to drupal_add_js, you end up with a different order for the same scripts in a group and thus a different aggregate. Sadly the same code will sit in two aggregates on the server, with slightly different source order, but otherwise they could have been exactly equal, produced a single aggregate and thus cached by the browser from page to page. The solution in this case is to use weight values to ensure the same order within an aggregate from page to page. It’s not ideal, because you don’t want to have to set weights on every hook_library / drupal_add_js. I’d recommend handling it on a case by case basis.

Best Practices

Clearly, there is a lot that can go wrong, or at least end up taking you to a place that is sub-optimal. Considering all that we’ve covered, I’ve come up with a list of best practices to follow:

Always use the API, never ‘shoehorn’ scripts into the page

A common problem is running into modules and themes that sidestep the API to add their Javascript to the page. Common methods include using drupal_add_html_head to add script to the header, using hook_page_alter and appending script markup to $variables[‘page_bottom’], or otherwise outputting a raw

Use hook_library

hook_library is a great tool. Use it for any and all Javascript you output (inline, external, file). It centralizes all of the options for each script so you don’t have to repeat them, and best of all, you can declare dependencies for your Javascript.

Use every_page when appropriate

The every_page option signals to the aggregation system that your script will be added to all pages on the site. If your script qualifies, make sure your setting every_page = TRUE. This puts your script into a "stable" aggregate that is cached and reused often.

AdvAgg

Advanced CSS/JS Aggregation (AdvAgg) is a contributed module that replaces Drupal’s built in aggregation system with it’s own, making many improvements and adding additional features along the way. The way that you add scripts is the same, but behind the scenes, how the groups are generated and aggregated into their respective aggregates is different. AdvAgg also attempts to overcome many of the scenarios where core aggregation can go wrong that I listed above. I won’t cover all of what AdvAgg has to offer, it has an impressive scope that would warrant it’s own article or more. Instead I’ll touch on how it can solve some of the problems we listed above, as well as some other neat features.

Out of the box the core AdvAgg module supplies a handful of transparent backend improvements. One of those is stampede protection. After a code release with JS/CSS changes, there is a potential that multiple requests for the same page will all trigger the calculation and writing of the same new aggregates, duplicating work. On high traffic sites, this can lead to deadlocks which can be detrimental to performance. AdvAgg implements locking so that only the first process will perform the work of calculating and writing aggregates. Advagg also employs smarter caching strategies so the work of calculating and writing aggregates is done as infrequently as possible, and only when there is a change to the source files. These are nice improvements, but there is great power to behold in AdvAgg’s submodules.

AdvAgg Compress Javascript

AdvAgg comes with two submodules to enhance JS and CSS compression. They each provide a pluggable way to have a compressor library act on the each aggregate before it’s saved to disk. There are a few options for JS compression, JSqueeze is a great option that will work out of the box. However, if you have the flexibility on your server to install the JSMIN C extension, it’s slightly more performant.

AdvAgg Modifier

AdvAgg Modifier is another sub module that ships with AdvAgg, and here is where we get into solving some of the problems we listed earlier. Let’s explore some of the more interesting options that are made available by AdvAgg Modifier:

Optimize Javascript Ordering

There are a couple of options under this fieldset that seek to solve some of the problems laid out above. First up, "Move Javascript added by drupal_add_html_head() into drupal_add_js()" does just what it says, correcting the errant use of the API by module/theme authors. Doing this allows the Javascript in question to participate in Javascript aggregation and potentially removes an HTTP request if it was a file that was being added. Next “Move all external scripts to the top of the execution order” and “Move all inline scripts to the bottom of the execution order” are both an attempt at resolving the issue of splitting the aggregate. This is more or less assuming that external files are more library-esque and inline Javascript is meant to be run after all other Javascript has run. That may or may not be the case depending on what you’re doing and you should definitely use caution. Such as that is, if it works for you, it could be a simple way to avoid splitting the aggregate without having to do manual work of adjusting weights.

Move JS to the footer

As we discussed, you typically add Javascript to one of two scopes, header or footer. A common performance optimization is to move all the Javascript to the footer. AdvAgg gives you a couple choices in how you do that ranging from some to all Javascript moved to the footer. This can easily break your site however, and you definitely need to test and use caution here. The reason of course is that there could be other script that aren’t declaring their dependencies properly, and/or are ‘shoehorning’ their Javascript into the page that are depending on a dependency existing in the header. These will need to be fixed on a case by case basis (submit patches to contrib modules!). If you still have Javascript that really does need to be in the header, AdvAgg extends the options for drupal_add_js and friends to include a boolean scope_lock that when set to TRUE, will prevent AdvAgg from moving the script to the footer. There always seems to be some stubborn ads or analytics provider that insists on presence in the header, and is backed by the business that forces you to make at least one of these exceptions. Another related option is "Put a wrapper around inline JS if it was added in the content section incorrectly". This is an attempt to resolve the problem of inline Javascript in the body depending on things being in the header. AdvAgg will wrap any inline Javascript in a timer that waits on jQuery and Drupal to be defined before running the inline code. It uses a regular expression to search for the scripts, which while neat, feels a little brittle. There is an alternate option to search for the scripts using xpath instead of regex, which could be more reliable. If you try it, do so with care, test first.

Drupal 8

Not a lot is different when it comes to Javascript aggregation in D8, but there have been a few notable changes. First, the aggregation system has been refactored under the hood to be pluggable. You can now cleanly swap out and/or add to the aggregation system, whereas in Drupal 7 it was a difficult and messy affair to do so. Modules like AdvAgg should have a relatively easier time implementing their added functionality. Other improvements to the aggregation system that didn’t quite make the D8 release will now have a clearer path to implementation for the same reason. In terms of its function and the way in which Javascript is grouped for aggregation, everything is as it was in Drupal 7.

The second change centres around adding Javascript to the page. drupal_add_js and drupal_add_library have been removed in Drupal 8 in favor of #attached. The primary motivation is to improve cacheability. With Javascript consistently added using #attached, it’s possible to do better render array caching. The classic example being that if you were using drupal_add_js in your form builder function, your Javascript would be missed if the cached render array was used and the builder function skipped. Caching aside, it’s nice that there is a single consistent way to add Javascript. Also related to this change, is hook_library definitions have gone the way of the dodo. In their place, *.libraries.yml files are now used to define your Javascript and CSS assets in libraries, continuing the Drupal 8 theme of replacing info hooks with YML files.

A third somewhat related change that I’ll point out is that Drupal core no longer automatically adds jQuery, Drupal.js and Drupal.settings (drupalSettings in Drupal 8). jQuery still ships with core, but if you need it, you have to declare it as a dependency in your *.libraries.yml file. There have also been steps taken to reduce Drupal core’s reliance on jQuery. This isn’t directly related to Javascript aggregation per se, but it’s a big change nonetheless that you’ll have to keep in mind if you’re used to writing Javascript in Drupal 7 and below. It’s a welcome improvement with potential for some nice front end performance gains. For more information and specific implementation details, check out the guide for adding Javascript and CSS from a theme and adding Javascript and CSS from a module.

HTTP 2.0

Any article about aggregating Javascript assets these days deserves a disclaimer regarding HTTP 2.0. For the uninitiated, HTTP 2.0 is the new version of the HTTP protocol that while maintaining all the semantics of HTTP 1.1, significantly changes the implementation details. The most noted change in a nutshell is that instead of opening a bunch of TCP connections to download all the resources you need from a single origin, HTTP 2.0 opens a single connection per origin and multiplexes resources on that connection to achieve parallelism. There is a lot of promise here because HTTP 2.0 enables you to minimize the number of TCP connections your browser has to open, while still downloading assets in parallel and individually caching assets. That means that you could be better off not aggregating your assets at all, allowing the browser to maximize caching efficiency without incurring the cost of multiple open TCP connections. In practice, the jury is still out on whether no aggregation at all is a good idea. It turns out that compression doesn’t do as well on a collection of individual small files as it does on aggregated large files. It’s likely that some level of aggregation will continue to be used in practice. However as hosts more widely support HTTP 2.0 and old browsers fall off, this will become a larger area of interest.

Conclusion

Whew, we’ve come to the end at last. My purpose was to go over all of the nuances of Javascript aggregation in Drupal. It’s an important topic when it comes to front end performance, and one that deserves careful consideration when you're trying to eek every last bit of performance out of your site. With Drupal 8 and HTTP 2.0 now a reality, it’ll be interesting to watch as things evolve for the better.

Categories: Elsewhere

Modules Unraveled: 156 Using BigPipe to Achieve Incredible Site Speed in Drupal 8 with Wim Leers - Modules Unraveled Podcast

Wed, 17/02/2016 - 10:00
Published: Wed, 02/17/16Download this episodeBig Pipe
  • What is Big Pipe?
  • This isn’t a Drupal specific thing right? Where did the idea of BigPipe come from?
  • How does it work?
    • Builds on Drupal 8’s caching system, specifically cacheability metadata (tags, contexts, max-age)
    • Rendering system (bubbling, placeholders)
    • page cache / dynamic page cache
    • BigPipe is built ON TOP OF ALL OF THE ABOVE
  • Does it work for anonymous and authenticated users?
  • Is this compatible with reverse proxies like Varnish?
  • Does BigPipe affect the need for something like redis or memchache?
  • How does BigPipe relate to authcache?
Use Cases
  • How can we start using it? What’s the install process?
  • What do we need to do to configure it for our site?
  • Is BigPipe ready to be used?
  • Is there anything like this for Drupal 7?
Questions from Twitter
  • Daniel Noyola @danielnv18
    What can I do to make my site compatible with BigPipe? O what shouldn't do?
  • Daniel Noyola @danielnv18
    Is it compatible with contrib modules like panels or display suite?
  • Ryan Gibson @ryanissamson
    I may have missed it, when BigPipe is included in D8.1, will it be enabled by default?
  • TheodorosPloumis @theoploumis
    Is the bigpipe caching working with the RESTful responses?
Episode Links: Wim on drupal.orgWim on TwitterBigPipe ModuleBigPipe DemoFacebook Engineering blog post about BigPipeRendervizWimleers.comTags: Drupal 8Big PipePerformanceplanet-drupal
Categories: Elsewhere

Vardot: 5 Security Modules That Every Drupal Website Must Have

Wed, 17/02/2016 - 08:57
Resources

With intruders, thieves, hackers and spammers ready to perpetuate threat to most businesses online, virtual security is definitely the top concern in today's web that a firm should focus. While there are several hundred CMSs out there to provide a digital platform, Drupal is considered one of the top secure platforms that can seamlessly address the firm's security needs.

Do you know what makes Drupal a ‘credible content management system’? Backed with some phenomenal modules and themes, this great CMS is heartily manned by a dedicated community of developers. Unlike other content management systems, Drupal offers the most consistent security updates, keeping the chances of risks as minimal as possible. However, if you are already using Drupal, then it is necessary to keep your website extra secure with the gifted grace of ‘security modules’ available for Drupal. Here we have listed the 5 best security modules that every Drupal website must include - have a look!

 

1. Password Policy

Usage: For now, Password Policy is used by more than 20k websites. The module has also been downloaded more than 260k times.

According to Verizon’s Data Breach Investigation Report statistics for the year 2013, 4 out of every 5 data breaches occur due to exploited or stolen credentials.

Thus, defining the password policies of your website and enforcing restrictions on user passwords is important. Make it possible with the help of ‘Password Policy’ module that brings a defined set of constraints implemented right before a website accepts the password changes of a user. With each constraint, the module has some pre-defined parameters that have to be ‘satisfied’ to validate the conditions. Password Policy has recently updated a brand new version for Drupal 8, introducing the constraints as plugins. Apart from this, the module comes with a password expiration feature where a user is either forced to change the password or it is optionally blocked once the old password expires.

Functionality: The module sets a credible policy for passwords, manning control through constraints while the password is generated. With Password Policy module, you can also control the structure of password in terms of length, capitalization, unique character, numbers, reuse of password, and so on.

 

2. Security Kit

Usage: Currently, about 11k sites are using Security Kit as a reliable easy-to-use module. The module has reported more than 143k downloads.

Do you know at Annual Website Security Statistics report 2012, Whitehat has identified cross-site scripting and cross-site request forgery as two most prevalent vulnerabilities?

The module promises security under four major categories- cross-site scripting, cross-site request forgery, clickjacking, and SSL/TLS. Cross site scripting safeguards content policies, handling browser reporting and violation management. Through cross-site request forgery, you can control the Origin HTTP request header. Similarly, Clickjacking works on the implementation of X-Frame-Options HTTP response header and Javascript (coupled with CSS and Noscript protection and customized text to disable the JavaScript message). Last but not the least- SSL/TLS handles the implementation of HTTP Strict Transport Security (HSTS) response header. With this, you can simply prevent the role of middleman and possibly control the potential attacks by eavesdropping.

Functionality: Security Kit brings the best security-hardening options that you can use on your website! Security Kit module enables safekeeping by lowering the risks of exploitation of different web application vulnerabilities.

 

3. Security Review

Usage: About 34k websites are using this module, which has been downloaded over 270k times.

Your site might have some silly easy-to-make mistakes that could be a bit embarrassing at times. Don’t worry and avoid the humiliation of making silly mistakes in your Drupal website with the help of Security Review module. How does it work? Well, Security Review module automates the testing process of a website in the simplest manner. It is a quick and easy to implement module. With Security Review Module, you can check the safe file system permission of your website that limits the execution of any arbitrary code. The module protects a site against dangerous XSS tags available in the text formats. It enables safety in reporting errors, avoids unnecessary information disclosure, promises security of private cum personal files, enables easy and safe uploading of extensions, and checks the large database for errorless usage. Moreover, you can have one Drupal admin as permission to protect your website against any form of malice or misconfiguration.

Functionality: Security Review will not add automated changes to your website, unless and until the checklist and its resources are not manually secured. The module is readily available for the latest Drupal 8 interface.

 

4. Username Enumeration Prevention

Usage: The module has been used by more than 7k sites and has been downloaded by almost 29k Drupal users.

Though Drupal is quite a secure interface yet the chance of exploitation is evident in the CMS, especially for Drupal 6. As Drupal 6 doesn’t come with brute force prevention functionality, it makes the website vulnerable for easy hacking through just a username. It means a hacker can quite easily attack a site through a username, followed by brute force hack. The best security practice is to keep the usernames protected. And this is exactly what Username Enumeration Prevention does - with this module, you can make your usernames hard to find for the hacker. The technique ‘username enumeration’ can enable an attacker to find usernames by using the ‘forgot password’ form. By simply adding the username that doesn’t exist in real, the attacker will get Drupal’s response. An attacker can thereby follow a simple process i.e. to keep trying different usernames on the ‘forgot password’ form until he find a valid username. With the enabling of this module, the attacker will be redirected back to the login form, while the error message will also replace the preview message. It works through replacing the message that appears to the user when requesting a new password to something more ambiguous instead of indicating that the user exists or not, which allows the attacker to figure the username through the status message that appears.

Functionality: Username Enumeration Prevention is an important way to secure the usernames of your website. The module simply disables the status message that could let the attacker know that a username exists through the "Request new password" function.

 

5. Generate Password

Usage: Almost 6k websites are currently using the Generate Password module. The module has almost 32k downloads.

Generate Password module works with a simple trick. It makes the password field optional or rather hidden on the new user registration page, so that when the password is not set during the registration process, the system generated a generic password. As an admin, you can optionally choose to display this password by the time it is created in the registration process. How does it work? For the configuration settings, just visit the account settings of the user page and change the behavior of the password generation.

Functionality: Customize the structure of password generation as in the length of the password, generated password entropy, nature of characters (capitalization, special character, and numbers), password algorithm, and password display as per your requirements.

 

Use these 5 Drupal security modules for your website to keep all those unwanted hackers and attackers away. So while you might wish to keep your website secure with all these modules, we would also recommend you to ensure the perfect dose of necessities with Varbase, an enhanced Drupal distribution built, packaged and maintened by Vardot, that users some of the most important and widely used Drupal modules, features, and configurations. What is the best part about Varbase? Well, the little ‘bundle of Drupal joys’ has these 5 modules bundled and configured in the "Security" feature. Varbase is highly recommended for certain reasons - it speeds up the Drupal developmental process, provides the most sorted/standardized configurations, offers one stop solutions for diverse Drupal development needs and ensures ease in implementation. The Drupal distribution is also quite frequently updated by the developers. Varbase is successfully used by almost 70 sites and has registered more than 10k downloads.

As said, the security of your business is highly important, so don’t forget to monitor your website regularly. We hope this article will resolve some of your security concerns related to the Drupal development. Do you have any others? Share them with us in comments.

Tags:  Drupal Planet Drupal Modules Security Title:  5 Security Modules That Every Drupal Website Must Have
Categories: Elsewhere

Dries Buytaert: When traffic skyrockets your site shouldn't go down

Wed, 17/02/2016 - 05:06

This week's Grammy Awards is one of the best examples of the high traffic events websites that Acquia is so well known for. This marks the fourth time we hosted the Grammys' website. We saw close to 5 million unique visitors requesting nearly 20 million pages on the day of the awards and the day after. From television's Emmys to Superbowl advertisers' sites, Acquia has earned its reputation for keeping their Drupal sites humming during the most crushing peaks of traffic.

These "super spikes" aren't always fun. For the developers building these sites to the producers updating each site during the event, nothing compares to the sinking feeling when a site fails when it is needed the most. During the recent Superbowl, one half-time performer lost her website (not on Drupal), giving fans the dreaded 503 Service Unavailable error message. According to CMSWire: "Her website was down well past midnight for those who wanted to try and score tickets for her tour, announced just after her halftime show performance". Yet for Bruno Mars' fans, his Acquia-based Drupal site kept rolling even as millions flooded his site during the half-time performance.

For the Grammys, we can plan ahead and expand their infrastructure prior to the event. This is easy thanks to Acquia Cloud's elastic platform capacity. Our technical account managers and support teams work with the producers at the Grammys to make sure the right infrastructure and configuration is in place. Specifically, we simulate award night traffic as best we can, and use load testing to prepare the infrastructure accordingly. If needed, we add additional server capacity during the event itself. Just prior to the event, Acquia takes a 360 degree look at the site to ensure that all of the stakeholders are aligned, whether internal to Acquia or external at a partner. We have technical staff on site during the event, and remote teams that provide around the clock coverage before and after the event.

Few people know what goes on behind the scenes during these super spikes, but the biggest source of pride is that our work is often invisible; our job well done means that our customer's best day, didn't turn into their worst day.

Categories: Elsewhere

OSTraining: How to Change the Main Language of a Drupal site

Wed, 17/02/2016 - 04:36

One OSTraining had set up a site in English and wanted to move everything over to their native language.

In this tutorial, I'll show you how to completely change the main language of a Drupal 7 site.

I'm going to use the example of an English-language site that needs to be changed to Spanish.

First, go to "Modules" and enable the Locale module:

Categories: Elsewhere

Pages