Elsewhere

Are you a project manager working for a company adopting Drupal? Are you new to managing Drupal projects? This course is the right one for you!

This course follows the life cycle of a Drupal project from start to finish and back again and is based on our full day Drupal for Project Manager’s course.

Mentorship consulting is one of the many services that Lullabot provides, and is something we’re known for in the Drupal community. When we work with potential clients to describe what we can do for them, it can sometimes be very difficult to explain how a consulting relationship works. This is especially true if they have never participated in consulting engagements with us or another agency.

In my professional life I am sometimes required to provide technical support to one of our salesmen. I find this an interesting change in pace though sometimes challenging.

Occasionally I fail to clearly convey the solution we are trying to sell because of my tendency to focus on detail the customer probably does not need to understand but I think is the interesting part of the problem.

Conversely sometimes the sales people gloss over important technology choices which have a deeper impact on the overall solution. I was recently in such a situation where as part of a larger project the subject of internationalisation (you can see why it gets abbreviated to i18n) was raised.

I had little direct personal experience with handling this within a project workflow so could not give any guidance but the salesman recommended the Transifex service as he had seen it used before, indicated integration was simple and we moved onto the next topic.

Unfortunately previous experience tells me that sometime in the near future someone is going to ask me hard technical questions about i18n and possibly how to integrate Transifex into their workflow (or at least give a good estimate on the work required).
LearningBeing an engineer I have few coping strategies available for situations when I do not know how something works. The approach I best know how to employ is to give myself a practical crash course and write up what I learned...so I did.

I proceeded to do all the usual things you do when approaching something unfamiliar (wikipedia, google, colleagues etc.) and got a basic understanding of internationalisation and localisation and how they fit together.

This enabled me to understand that the Transifex workflow proposed only covered the translation part of the problem and that, as Aldrich observed in my title quote, there is an awful lot more to translation than I suspected.
PlatformsMy research indicated that there are numerous translation platforms available for both open source and commercial projects and Transifex is one of many solutions.

Although the specific platform used was Transifex most of these observations apply to all these other platforms. The main lesson though is that all platforms are special snowflakes and once a project invests effort and time into one platform it will result in the dreaded lock in. The effort to move to another platform afterwards is at least as great as the initial implementation.

It became apparent to me that all of these services, regardless of their type, boil down to a very simple data structure. They appear to be a trivial table of Key:Language:Value wrapped in a selection of tools to perform format conversions and interfaces to manipulate the data.

There may be facilities to attach additional metadata to the table such as groupings for specific sets of keys (often referred to as resources) or translator hints to provide context but the fundamental operation is common.

The pseudo workflow is:

• Import a set of keys
• Provide a resource grouping for the keys.
• Import any existing translations for these keys.
• Use the services platform to provide additional translations
• Export the resources in the desired languages.

The first three steps are almost always performed together by the uploading of a resource file containing an initial set of translations in the "default" language and  due to the world being the way it is this is almost always english (some services are so poorly tested with other defaults they fail if this is not the case!)
The platforms I looked at generally follow this pattern with a greater or lesser degree of freedom in what the keys are, how the groupings into resources are made and the languages that can be used. The most common issue with these platforms (especially open source ones) is that the input convertors will only accept a very limited number of formats and often restricted to just GNU gettext PO files. This means that to use those platforms a project would have to be able to convert any internal resources into gettext translatable format. 
The prevalence of the PO format pushes assumptions into almost every platform I examined, mainly that a resource is for a single language translation and that the Key (msgid in gettext terms) is the untranslated default language string in the C locale.
The Transifex service does at least allow for the Key values to be arbitrary although the resources are separated by language.

Even assuming a project uses gettext PO files and UTF-8 character encoding (and please can we kill every other character encoding and move the whole world to UTF-8) the tools to integrate the import/export into the project must be written.

A project must decide some pretty important policies, including:

• Will they use a single service to provide all their translations.
• Will they allow updates to the files in their revision control system and how those will be integrated.
• Will there be a verification step and if so who and how will that be performed. Especially important is the question of a reviewer understanding the translated language being integrated and how that is controlled.
• Will the project be paying for translations
• Will the project allow machine translations, if not can they be used as an initial hint (sometimes useful if the translators are weak in the "default" language

These are project policy decisions and, as I discovered, just as difficult to answer as the technical challenges.

Armed with my basic understanding it was time to move on and see how the transifex platform could be integrated into a real project workflow.
ImplementingProof of conceptMy first exercise was to take a trivial command line tool, use xgettext to generate a PO file and add the relevant libintl calls to produce gettext internationalised tool.

A transifex project was created and the english po file uploaded as the initial resource. A french language was added and the online editor used to provide translations for some strings. The PO resource file for french was exported and the tool executed with LANGUAGE=fr and the french translation seen.

This proved the trivial workflow was straightforward to implement. It also provided insight into the need to automate the process as the manual website operation would soon become exceptionally tedious and error prone.
Something more usefulTo get a better understanding of a real world workflow I needed a project that:

• Already internationalised but had limited language localisation 
• Did not directly use gettext 
• Had a code base I understood
• Could be modified reasonably easily.
• Might find the result useful rather than it being a purely academic exercise.

I selected the NetSurf web browser as it best fit this list.

Internally NetSurf keeps all the translated messages in a simple associative array this is serialised to an equally straightforward file named FatMessages. The file is UTF-8 encoded with keys separated from values by a colon. The Key is constrained to be ASCII characters with no colons and is structured as language.toolkit.identifier and is unique on identifier part alone.

This file is processed at build time into a simple identifier:value dictionary for each language and toolkit.

Transifex can import several resource formats similar to this, after experimenting with YAML and Android Resource format I immediately discovered a problem, the services import and export routines were somewhat buggy.

These routines coped ok with simple use cases but having more complex characters such as angle brackets and quotation marks in the translated strings would completely defeat the escaping mechanisms employed by both these formats (through entity escaping in android resource format XML is problematic anyway)

Finally the Java property file format was used (with UTF-8 encoding) which while having bugs in the import and export escaping these could at least be worked around. The existing tool that was used to process the FatMessages file was rewritten to cope with generating different output formats and a second tool to merge the java property format resources.

To create these two tools I enlisted the assistance of my colleague Vivek Dasmohapatra as his Perl language skills exceeded my own. He eventually managed to overcome the format translation issues and produce correct input and output.

I used the Transifex platforms free open source product, created a new project and configured it for free machine translation from the Microsoft service, all of which is pretty clearly documented by Transifex.

Once this was done the messages file was split up tinto resources for the supported languages and uploaded to the transifex system.

I manually marked all the uploaded translations as "verified" and then added a few machine translations to a couple of languages. I also created spanish as a new language and machine translated most of the keys.

The resources for each language were then downloaded and merged and the resulting FatMessages file checked for differences and verified only the changes I expected appeared.

I quickly determined that manually downloading the language resources every time was not going to work with any form of automation, so I wrote a perl script to retrieve the resources automatically (might be useful for other projects too).

Once these tools were written and integrated into the build system I could finally make an evaluation as to how successful this exercise had been.
ConclusionsThe main things I learned from this investigation were:

• Internationalisation has a number of complex areas
• Localisation to a specific locale is more than a mechanical process.
• The majority of platforms and services are oriented around textural language translation
• There is a concentration on the gettext mode of operation in many platforms
• Integration to any of these platforms requires both workflow and technical changes.
• At best tools to integrate existing resources into the selected platform need to be created
• Many project will require format conversion tools, necessitating additional developer time to create.
• The social issues within an open source project may require compromise on the workflow.
• The external platform may offer little benefits beyond a pretty user interface.
• External platforms introduce an external dependency unless the project is prepared and able to run its own platform instance.
• Real people are still required to do the translations and verify them.

Overall I think the final observation has to be that integrating translation services is not a straightforward operation and each project has unique challenges and requirements which reduce the existing platforms to compromise solutions.

Twig is Drupal's best theme engine! Tomorrow I will speak at the DrupalJam about 'Twig in Drupal 8'. I will explain the pros and cons of Twig for Drupal themers, show examples of new Twig templates, explain a summary of the Twig syntax and of course do a small demo of working with Twig templates. You'll find my slides here.

Tags:  theming Twig Render Array

My team has been looking into the Firefox add-on Selenium IDE as a quick and simple way to create automated tests for our Drupal sites. Selenium IDE out of the box does not support conditionals, making it hard to account for unexpected behaviour. For instance, it's easy to make a test of the basic Drupal login functionality, but if the user is already logged in when running that test (a common scenario when working on a site), the test will fail. The solution to this is called Sideflow.

Sideflow was created by Darren DeRidder and currently extends Selenium IDE with these commands: goto/gotoLabel, label, gotoIf, while, endWhile, and push. I'm only going to use gotoIf and label in this example, but you can read about the others in the Sideflow GitHub repo and in the announcement on Darren's blog (where you can also find a lot of general Selenium IDE tips). Also see his blog posts Selenium IDE Sideflow Update 1 and Selenium IDE Sideflow Update 2. You might wonder why I don't just start the test with deleteAllVisibleCookies (a tip I got from 6 Ways to Make The Most of Selenium IDE), which in effect would log out the user before running the rest of the test. Well, it doesn't work with Drupal's session cookie since it's set to HttpOnly, meaning it can't be controlled by Javascript. A test that does work, however, is this: open | /userstoreElementPresent | //input[@id='edit-name'] | userIsLoggedOutgotoIf | ${userIsLoggedOut} == false | userIsLoggedInwaitForElementPresent | //input[@id='edit-name'] |type | //input[@id='edit-name'] | adminwaitForElementPresent | //input[@id='edit-pass'] |type | //input[@id='edit-pass'] | adminwaitForElementPresent | //input[@id='edit-submit'] |clickAndWait | //input[@id='edit-submit'] |label | userIsLoggedIn storeElementPresent looks for an element, in this case the username field shown when the user is logged out, and stores a Boolean value (true or false) in the variable userIsLoggedOut. If that element isn't present we can make a qualified guess that the user is already logged in, and just skip down to the label userIsLoggedIn, ending the test.  Green every time!

With Drupalcon now only a few days away, preparations are beginning to ramp up (or maybe starting to die down, depending on how much prep work your company has already done). Drupalcon is *the* Drupal event of the year—and with more than 3,000 attendees and 50+ expert-led sessions, there’s a lot to think about before boarding that plane to Portland.

So you're launching a new website or replacing an old one and want to make sure everything goes smoothly? This guide will give you a run-down of everything you can check to avoid common pitfalls!

Site status

You should always start by checking the status report (http://example.com/admin/reports/status). This page shows you all of the basic requirements for your Drupal site to run correctly.

Any issues will be highlighted in red and typically have a link to a configuration page or the documentation to help you resolve the problem.

Scheduled tasks

Drupal 7 will run scheduled tasks (known as cron jobs) out of the box, but only when users are visiting your site.

This is great for small sites which don't need much housekeeping, but if your site is a bit bigger or if you don't have visitors 24/7, you should set up a cron job to run periodically.

You can also look at a module such as Ultimate cron which gives you fine grained control over when each scheduled task will run.

Web services

Many web services such as Mollom or Google analytics need a domain name specific API key to use.

If you use any of these services on your website, you should ensure that you've registered your real domain name with the service and you've updated Drupal with your new API key.

Broken links

It can be easy when copying and pasting to accidentally link directly to a file or image on a development site.

These links can often stop working or perhaps worse, may direct users away from your live site and onto the development site instead.

You can use a module such as Link checker to ensure this doesn't happen, and it is good practice to password protect your development website, so that users (or more likely Google!) cannot stumble accross it.

Site optimisation

The site performance page (http://example.com/admin/config/development/performance) will allow you to configure a number of options to help optimise your Drupal site. This includes page caching and optimising CSS and JavaScript files.

See our series on High Performance in Drupal for some expert tips - High performance in Drupal Part 1: Give your site a boost and High performance in Drupal Part 2: Lightning fast code.

Development modules

Development modules such as devel can often reduce your website's performance, so it's worth turning them off on your live site. You can still keep them running in your development environment if needed.

User accounts

Many of us are guilty of using a common or simple password to make life easier when building a website.

Once the site is live, it's worth taking the time to update any administrative accounts with secure passwords.

It is also worth removing any unnecessary development accounts and content. Just in case.

Error messages

Being able to see debug messages and errors are handy when creating a site, but may scare off users once the site is live.

Ensure that errors and warnings are hidden by visiting the Logging and errors config page.

Site information

The site information config page holds all of the most common site information, such as the website name and email address.

It's worth double checking that all of this information is correct. It could be quite embarrassing if your first newsletter arrives from dev@example.com.

 

Some of these pitfalls can be avoided from the get go, if you follow a few simple principles. Check out Rick Donohoe's blog article Drupal site building 101 for some handy hints and tips on this!

Moving Drupal website clients to cloud hosting has been great as they're able to get high performance, scalable capacity at a pretty reasonable rate. However, we have discovered when clients offer an email sign-up, the emails that are generated from the cloud-hosted Drupal website are often rejected as spam. For those clients who have chosen the Rackspace cloud, here is a step-by-step solution to the problem.... Read more

This post discusses the differences between Drupal and WordPress, and also takes a quick look at a couple of other web frameworks. What are the benefits with each platform, and how do you know which one to choose?

I'm a millennial, but even I remember the experience of calling the telephone operator and getting a live human to look up the number of a business or place a collect call. We have the digital means to complete lots of tasks like that today, but that doesn't mean all of our methods are equally effective for everyone.

"Drupal 8 will be the most accessible version of Drupal yet," declare J. Renée Beach and Wim Leers in their Drupalcon Portland session description.

They're both part of the Spark team, an initiative to improve the authoring experience in Drupal for everyone.

Spark is more well known for things like in-place editing and a mobile friendly toolbar, which you can see at right. But from the beginning, improving the experience for everyone has been a big priority, and one of the most exciting developments is a new aural interface.

That's right, Drupal is getting a switchboard operator:

OK, so that doesn't look terribly exciting all on its own. But trust me, when you watch the videos of people interacting with Drupal 8 and having menus and selections read as they go, it's pretty cool.

When I spoke with J. Renée about Drupal 8 and the nature of working on accessibility, the passion for this work really shown through. I'm really looking forward to their session with Wim, "Drupal Speaks: Aural user interfaces, new Drupal 8 accessibility features," on Wednesday at 10:45 AM. Hope to see you there!

IB: What are we missing when we talk about accessibility right now?

JRB: I want developers to understand that accessibility is fundamental to user interface development. We tend to talk about accessibility like we talk about gender. Both have coded values. When we speak of being gendered, we are often talking about being non-male. Male is a kind of genderless base state. So is it with accessibility. When we speak of making something accessible, we tend to refer to making an interface for blind users or for users with physical capabilities that make keyboard and mouse use difficult, as examples. Visual is a kind of accessible base state.

We risk "othering" folks for whom accessibility is an issue because as developers, in general, non-visual accessibility has not been a primary concern. I know what is is like to be othered. In some ways, highlighting otherness can be an effective way to bring focus to a problem. Eventually though, we need to resolve those issues and close the loop on the otherness. We can be other and also be equal. Now is the time for front end developers to start thinking about accessibility as a multi-modal effort. We no longer have the excuse that the tools and technologies available to us do not support efficient workflows for non-visual UI development.

IB: Where is Drupal 8 going to do better?

JRB: Most importantly, we have more individual core contributors this cycle who truly believe in addressing accessibility issues. And they are all smart, wonderful people which makes working with them a pleasure!

For example, take this issue about requirement warnings during installation. For a sighted user, a warning during installation is immediately apparent. The missing requirement is made distinct with color contrast. For a blind user, they must traverse every cell in the table to discover a missing requirement. Would we ever impose such a burden on a sight user through the UI? No, not without grumbles in the issue queues at least. With more contributors invested in improving these types of non-visual details, we are polishing all the rough edges — the ones we see and the ones we don't.

IB: How important is context in aural interfaces?

JRB: Context is important to all interfaces. As front end developers we build templates that expose context in a predictable, consumable way. As a practice we have established and then refined patterns of visual expression over the past 30-plus years.

Metaphors grounded visual pointer displays on a virtual desktop. We talk of visual affordances in rounded, gradient-embellished, reflective buttons. Skeumorphic designs bring our understanding of the physical world to bear on pixels and bits.

Where are the metaphors in aural interface design? I know of none. To me, these interfaces are flat. The metal is bare underneath them.

Perhaps non-visual interfaces have one less level of abstraction to traverse. Maybe there's no need to translate language into symbol and then back into language. But that little bit of designer in me, that memory of a linguist I almost was, remembers being thunderstruck with insight reading Jackendoff's unfurling of metaphor after I had just so recently fallen smitten with the strict generative grammar of early Chomsky. Jackendoff gives us a way of understanding language that starts at basic physical dichotomies — up/down and near/far — and from there offers us a model of communication. He gives us pattern. (Early) Chomsky gave us metal. So much that we humans do starts with structure that softens with time to fit our curvy, winding nature.

I want to believe that the aural interfaces we have today still just the awkward first attempts to build an abstract audio interface pattern language. That non-visual interface design is still working through its structuralist phase. We are still learning how to pack context into denser forms through non-visual expressions.

IB: Will the Drupal 8 improvements have things to offer module developers?

JRB: In Drupal 8, we are building tools that manage a couple of the trickier components of accessibility in a browser. These are:

1. Outputting audio updates
2. Managing tabbing in constrained workflows

Module developers will be able to pass a string to a method called "announce" on the Drupal object and have that string read by a screen reader.

Another method on the Drupal object called "tabbingManager" will constrain tabbable elements on the page. A developer will select those elements, either through JavaScript methods or jQuery, and pass them to the tabbingManager. Tabbing is then constrained to those elements until the constraint is superseded or released. I know that must not be completely clear, but that's why we're presenting a session about aural user interfaces and how we can use these new tools to build them!

Top image: Public domain. Drupal images from the drupal.org issue queue and the session slides.

Join Rootwork on Twitter, Facebook and SlideShare.

Learn about Rootwork's services for nonprofits and social change.

We are pleased to announce that Commerce Kickstart has won the Walkthrough documentation prize. The prize, which was determined by votes from Walkthrough.it backers, will use the Commerce Kickstart Drupal distribution to showcase the capabilities of Walkthrough.it.

Commerce Kickstart is the quickest way to get up and running with Drupal Commerce. The distribution provides everything to create a fully-featured demo store out of the box, complete with theme, catalog, and custom back office interface.

The Linux init systems are a bit in flux at the moment. That is, they're in flux in Debian; outside Debian, most other distributions have stepped away from sysvinit and towards something else (systemd, openrc, or upstart). I've not been a proponent of any switch, though I understand the reasoning, and it probably makes sense for us to switch at some point. But yesterday, the fact that this customer's system was running sysvinit and not systemd or upstart saved me quite a bit.

There's a server. It has one quadcore processor. For reasons that I won't go into here, the customer wants an extra quadcore processor to be added to the system.

After having done so, I power on the system... only to see it power itself off at some point during boot. I did notice some kernel messages fly by just moments before the system would power itself off, but it was impossible for me to read them. So what did I do?

• Boot the system with init=/bin/bash,
• After having booted the system, go to /etc/rcS.d and manually run each and every one of the scripts there in turn. When the system powers off, I know what the problem is.
• Disable the init script that causes the problem, and boot the system normally.

That last bit is, obviously, a bit of an ugly workaround; the better way to fix this issue would have been to debug what the actual issue was, and implement a proper fix. However, I didn't have time for that (the fact that there was need for a second quadcore chip explains how much this system is in use), and the workaround was acceptable for the customer. It is not the first time that this ability to single-step the init system has saved me. The fact that sysvinit is so simplistic is what makes this possible, and I consider that one of its most important features.

Recently, I came into contact with a distribution that uses systemd as its init system (in casu, Arch Linux). I had made a mistake in configuration; I had installed and enabled a graphical login system, but had no xterm or similar available, and had done something else wrong through which I couldn't get a regular shell on the console anymore, either. To fix this, I tried doing something like the above (running with init=/bin/bash and single-stepping the init system), but found that doing so with systemd is nigh impossible. In the end, I knew what exactly the problem was and could disable automatically starting the login manager through removing a symlink, but it brought home the issue that debugging a similar issue when running systemd rather than sysvinit might be a lot harder to do.

We'll see what the future brings.

Last year at DebConf12 and the Paris mini-DebConf I mentioned some of the sophisticated techniques that the likes of Microsoft and Facebook are using to monitor their customers.

So when Skype was busted spying on the content of chat messages, it was no surprise for many people in the Debian community.

People are already rushing to find alternatives like XMPP and Jitsi. Debian 7 has been released just in time, with powerful features like TURN support that finally allow users to make free calls and chats with seamless NAT traversal. Sadly, Debian's built-in VoIP/RTC client, Empathy, only uses Google's TURN servers and not native Debian servers, but hopefully a solution will come soon, but it is easy enough to install Jitsi instead and configure it to use any of the free TURN server software on Debian.

It should be emphasized that Skype does not just spy on URLs in chat - it has simply been possible to detect this form of spying by detecting when the URL is accessed. Microsoft has taken out various patents for secretive monitoring of Internet phone calls and the analysis of speech patterns to detect both the content and emotions during a conversation. This allows them to get a very thorough analysis of the state of mind of every user at almost every moment and fine-tune the type of advertising and branding that is delivered to that person through conventional means and also through biased `news' reporting and other means.

Field as Block is a lightweight module that allows you to display a field as a block. The same results can be achieved by using Panels, Display Suite or custom code but this module offers a lightweight alternative.

CCK Blocks offers similar functionality, however on the project page they recommend that you use Field as Block for new projects. It looks like CCK Blocks will be deprecated in favour of Field as Block. For more details read issue #1920636 (comment #4).

Today Google announced PHP support for Google App Engine! I have been one of the lucky folks who had early access and so of course I worked on getting Drupal up and running on GAE. There are a few things that still need to be worked out which I will continue to discuss with the app engine team, but I have a working Drupal setup which I will detail below. Note that much of this may also apply to other PHP frameworks.

Getting up and running

I will cover the steps specific to getting Drupal 7 (notes for Drupal 6 along with branches in repository) up and running on App Engine and not how to use the SDK and development flow which is detailed in the documentation. For an example (minimal profile from core) of Drupal running on Google App Engine see boombatower-drupal.appspot.com.

Sign up to be whitelisted for PHP runtime

Currently, the PHP runtime requires you to sign up specifically for access. Assuming you have access you should be able to follow along with the steps below. Otherwise, the following steps will give you a feel for what it takes to get Drupal running on GAE.

Create an app

Create app by visiting appengine.google.com and clicking Create Application, see the documentation for more details.

Create a Cloud SQL Instance

Follow the documentation for setting up a Cloud SQL Instance. Be sure to give your application access to the instance.

Once the instance has been created select the SQL Prompt tab and create a database for your Drupal site as follows.

CREATE DATABASE drupal;

Download Drupal

There are a few tweaks that need to be made to get Drupal to run properly on GAE which are explained below, but for the purposes of this walk-through one can simply download my branch containing all the changes from github.

git clone --branch 7.x-appengine https://github.com/boombatower/drupal-appengine.git   # or for Drupal 6 git clone --branch 6.x-appengine https://github.com/boombatower/drupal-appengine.git

or download as a zip or for Drupal 6 download as a zip.

Configure Drupal database settings

Since GAE does not allow the filesystem to be writeable one must configure the database settings ahead of time.

Copy default.settings.php as settings.php and add the following bellow <?php $databases = array(); ?> around line 213.

<?php
$databases = array();
$databases['default']['default'] = array(
  'driver' => 'mysql',
  'database' => 'drupal', // The database created above (example used 'drupal').
  'username' => 'root',
  'password' => '',
  // Setting the 'host' key will use a TCP connection which is not supported by GAE.
  // The name of the instance created above (ex. boombatower-drupal:drupal).
  'unix_socket' => '/cloudsql/[INSTANCE]',
//  'unix_socket' => '/cloudsql/boombatower-drupal:drupal',
  'prefix' => '',
);
?>

For Drupal 6 around line 91.

<?php
$db_url = 'mysql://root:@cloudsql__boombatower-drupal___drupal/drupal';
?> Push to App Engine

Update the application name in the app.yaml file to the one you created above and upload by following the documentation.

# See https://developers.google.com/appengine/docs/php/config/appconfig.   application: drupal # <-- change this to your application version: 1 runtime: php api_version: 1 threadsafe: true   handlers: # Default handler for requests (wrapper which will forward to index.php). - url: / script: wrapper.php   # Handle static requests. - url: /(.*\.(ico$|jpg$|png$|gif$|htm$|html$|css$|js$)) # Location from which to serve static files. static_files: \1 # Upload static files for static serving. upload: (.*\.(ico$|jpg$|png$|gif$|htm$|html$|css$|js$)) # Ensures that a copy of the static files is left for Drupal during runtime. application_readable: true   # Catch all unhandled requests and pass to wrapper.php which will simulate # mod_rewrite by forwarding the requests to index.php?q=... - url: /(.+) script: wrapper.php appcfg.py update drupal/ Install

Visit your-app.appspot.com/install.php and follow the installation steps just as you would normally except that the database information will already be filled in. Go ahead and ignore the mbstring warning and note that the GAE team is looking into supporting mbstring.

Explanation of changes

If you are interested in what changes/additions were made and the reasons for them continue reading, otherwise you should have a working Drupal install ready to explore! There are a few basic things that do not work perfectly out of the box on GAE. The changes can be seen by diffing the 7.x-appengine branch against the 7.x branch in my repository.

File directory during installation

The Drupal installer requires that the files directory be writeable, but GAE does not allow for local write access thus the requirement must be bypassed in order for the installation to complete.

Author: boombatower <boombatower@google.com> Date: Wed May 15 15:49:03 2013 -0700   Hack to trick Drupal into ignoring that file directory is not writable.   diff --git a/modules/system/system.install b/modules/system/system.install index 1b037b8..9931aad 100644 --- a/modules/system/system.install +++ b/modules/system/system.install @@ -333,6 +333,8 @@ function system_requirements($phase) { } $is_writable = is_writable($directory); $is_directory = is_dir($directory); + // Force Drupal to think the directories are writable during installation. + $is_writable = $is_directory = TRUE; if (!$is_writable || !$is_directory) { $description = ''; $requirements['file system']['value'] = $t('Not writable'); Clean URLs

In order to take advantage of clean urls, of which most sites take advantage, mod_rewrite is required for Apache environments. Since GAE does not use Apache it does not support mod_rewrite and thus another solution is needed. The app.yaml can configure handlers which allow for wildcard matching which means multiple paths can easily be routed to a single script. Taking that one step further we can alter the <?php $_GET['q']?> variable just as mod_rewrite would so that Drupal functions properly. Rather than modify core this can be done via a wrapper script as show below (this should work well for other PHP applications).

<?php
/**
 * @file
 * Provide mod_rewrite like functionality and correct $_SERVER['SCRIPT_NAME'].
 *
 * Pass through requests for root php files and forward all other requests to
 * index.php with $_GET['q'] equal to path. In terms of how the requests will
 * seem please see the following examples.
 *
 * - /install.php: install.php
 * - /update.php?op=info: update.php?op=info
 * - /foo/bar: index.php?q=/foo/bar
 * - /: index.php?q=/
 */

$path = parse_url($_SERVER['REQUEST_URI'], PHP_URL_PATH);

// Provide mod_rewrite like functionality. If a php file in the root directory
// is explicitely requested then load the file, otherwise load index.php and
// set get variable 'q' to $_SERVER['REQUEST_URI'].
if (dirname($path) == '/' && pathinfo($path, PATHINFO_EXTENSION) == 'php') {
  $file = pathinfo($path, PATHINFO_BASENAME);
}
else {
  $file = 'index.php';

  // Provide mod_rewrite like functionality by using the path which excludes
  // any other part of the request query (ie. ignores ?foo=bar).
  $_GET['q'] = $path;
}

// Override the script name to simulate the behavior without wrapper.php.
// Ensure that $_SERVER['SCRIPT_NAME'] always begins with a / to be consistent
// with HTTP request and the value that is normally provided (not what GAE
// currently provides).
$_SERVER['SCRIPT_NAME'] = '/' . $file;
require $file;
?> PHP $_SERVER['SCRIPT_NAME'] variable

The <?php $_SERVER['SCRIPT_NAME'] ?> implementation differs from Apache mod_php implementation which can cause issues with a variety of PHP applications. The variable matches the HTTP spec and not the filesystem when called through Apache.

For example a script named foo.php contains the following.

<?php
var_dump($_SERVER['SCRIPT_NAME']);
?>

When executed from command line here are the results.

$ php foo.php string(7) "foo.php"   $ php ./foo.php string(9) "./foo.php"

When invoked through Apache like http://example.com/foo.php.

string(8) "/foo.php"

The documentation does not talk about this behavior (although many comments demonstrated the expected Apache behavior), but it is definitely depended on.

The difference causes Drupal to format invalid URLs.

example.com.foo.css (instead of ...com/foo.css) example.comsubdir/foo.css (instead of ...com/subdir/foo.css)

Drupal derives the URL from <?php dirname() ?> of <?php $_SERVER['SCRIPT_NAME'] ?> which will return . if no slashes or just / for something like /index.php.

The wrapper script above solves this by ensuring that the SCRIPT_NAME variable alway starts with a leading slash.

HTTP requests

GAE does not yet support support outbound sockets for PHP (although supported for Python and Java) and if/when it does the preferred way will continue to be streams due to automatic caching of outbound requests using urlfetch. I have included a small change to provide basic HTTP requests through drupal_http_request(). A proper solution would be to override the drupal_http_request_function variable and provide a fully functional alternative using streams. Drupal 8 has converted drupal_http_request() to use Guzzle which supports streams. Making a similar conversion for Drupal 7 seems like the cleanest way forward rather than reinventing the change.

php.ini

GAE disables a number of functions for security reasons, but only softly disables some functions which may then be enabled. Drupal provides access to phpinfo() from admin/reports/status and uses output buffering, both of which are disabled by default. The included php.ini enables both functions in addition to getmypid which is used by drupal_random_bytes().

# See https://developers.google.com/appengine/docs/php/config/php_ini.   # Required for ob_*() calls which you can find by grepping. # grep -nR '\sob_.*()' . output_buffering = "1"   # See https://developers.google.com/appengine/docs/php/runtime#Functions-That-Must-Be-Manually-Enabled # phpinfo: Provided on admin/reports/status under PHP -> "more information". # getmypid: Used by drupal_random_bytes(), but not required. google_app_engine.enable_functions = "getmypid, phpinfo" Future

I plan to continue working with the GAE team to ensure that support for Drupal can be provided in a clean and simple manner. Once current discussions have been resolved I hope to provide more formal documentation and support for Drupal.

File handling

I worked on file support, but there were a number of upcoming changes that would make things much cleaner so I decided to wait. GAE provides a stream wrapper for Google Cloud Storage which makes using the service very simple. Assuming you have completed the prerequisites files on GCS may be accessed using standard PHP file handling functions as shown in the documentation.

<?php
$file = 'gs://my_bucket/hello.txt';
file_put_contents($file, 'hello world');

$contents = file_get_contents($file);
var_dump($contents); // prints: hello world
?>

Unfortunately, the wrapper does not currently support directories nor does file_exists() work properly. Keep in mind that the filesystem is flat so a file may be written to any path without explicitly creating the directory. Meaning one can write to gs://bucket/foo/bar.txt without creating the directory foo. With that being the case it is possible to get some hacky support by simply disabling all the directory code in Drupal, but not really usable. It should be possible to hack support in through the stream wrapper since directories are simply specially name files, but the app engine team has indicated they will look into the matter so hopefully this will be solved cleanly.

Assuming the stream wrappers are fixed up then support can be added in much the same way as that Amazon S3 support is added except that no additional library will be needed.

Additionally, the documentation also notes the following.

Direct file uploads to your POST handler, without using the App Engine upload agent, are not supported and will fail.

In order to support file uploads the form must be submitted to the url provided by CloudStorageTools::createUploadUrl() and the forwarded result handled by Drupal. A benefit of proxying requests through uploader service is that uploaded files may be up to 100TB in size.

Other

There are a number of additional services provided as part of GAE of which Drupal could take advantage.

• Task Queue API
• Google account authentication
• Memcache seems to work out of the box with the memcache module
• <?php $SERVER['HTTP_X_APPENGINE*'] ?> variables (like geo information) Large number of services in API Console

Closing

Hopefully this will be useful in getting folks up and running quickly on GAE with Drupal and understanding the caveats of the environment. Obviously there is a lot more to cover and I look forward to seeing what others publish on the matter.

Tags:

• drupal
• drupal 6
• drupal 7
• google
• google app engine

Review: Asimov's Science Fiction, July 2011

Editor: Sheila Williams Issue: Volume 35, No. 7 ISSN: 1065-2698 Pages: 112

Williams's editorial is a mildly interesting piece about story titles. Silverberg's column is a more interesting (and rather convincing) rebuttal of the joke that fiction authors are "professional liars," combined with an examination of a fake and fantastic 14th travelogue that (at least in Silverberg's telling) was widely believed at the time. The precis of Silverberg's argument is that lying requires an intent to deceive, which is a property of deceptive memoir writers but not of fiction authors.

Di Filippo's review column, as usual, is devoted almost entirely to esoterica, although I was moderately interested to hear of Stableford's continued work on translating early French SF. None of it seems compelling enough to go buy, but good translations of early works seem like a good thing to have in the world.

"Day 29" by Chris Beckett: The conceit of this novelette is an interstellar travel system akin to a transporter that allows near-instantaneous travel between worlds. The drawback is that all memories from somewhere between 40 and 29 days before transit up until transit are wiped. The progatonist is a data analyst who is about to travel, and therefore by agency rule is required to stop doing work on day 40 before transmission since he can't be held legally liable for anything he has no recollection of doing. (I would like to say that I find this implausible, since one could always keep records, but it's exactly the sort of ass-covering regulation that a human resources department would come up with.)

The premise is quite interesting: what do you do during that period that you're going to forget? Beckett wisely mixes Stephen's current waiting period on the colony world with his diary of his original waiting period on Earth the first time he went through the transmission process, and the latter adds greatly to the reader's appreciation of the weirdness of the forgotten interval.

Unfortunately, this is a story more about psychological exploration than about plot, and Stephen just isn't very interesting. The telepathic but possibly nonsentient aliens add weirdness but not much else, and the ending of the story provided little sense of closure or conclusion for me. A good idea, but not the execution I wanted. (5)

"Pug" by Theodora Goss: Since I grew up with a pug, I have a soft spot for a story featuring one; sadly, though, this story has insufficient pug in it. This is a quiet fantasy (Asimov's calls it SF, presumably on the basis of parallel worlds and a hypothesized scientific explanation, but it reads like fantasy to me) featuring Victorian girls, including one with a bad heart. They discover a hidden door to other versions of their world and do some minor exploration. There's little or nothing in the way of plot; the story is more of an attempt to capture a mood. It's mildly diverting, but I wish it had gone somewhere more substantial. (5)

"Dunyon" by Kristine Kathryn Rusch: A Rusch story is often the highlight of an issue, and this is no exception. The protagonist is the owner of a bar in a space station that's become a combination of a refugee camp and a slum. War and chaos have created desperate people, most of whom are attempting to find some way to resources and get out of the bottom of society. The story is about a rumor: a mythical system named Dunyon that's safe and far away. And it's about how people react to that rumor. There's nothing particularly surprising about the direction the story goes (it's fairly short), but Rusch is always a good storyteller. (7)

"The Music of the Sphere" by Norman Spinrad: I've had mixed feelings about Spinrad's fiction (and some of his essays), but I liked this story, despite its implausibility. It's set in the near future, featuring an expert in cetaceans and dolphin perception and a composer obsessed with both loud music and classical musical style. Just from that description, you can probably predict much of the story, but I thought it had some neat ideas about dolphins, whales, and alternate perception and aesthetics. (Note: neat, not necessarily biologically plausible.) Enjoyable. (6)

"Bring on the Rain" by Josh Roseman: In a change of pace from the rest of the issue, this is a post-apocalyptic story of caravans of wheeled ships traversing a scorched and ruined landscape in search of weather systems and rain. The feel is of an inverted Waterworld, but with more emphasis on military tactics and cooperating fleets. The transposition of fleet maneuvers to huge ground vehicles adds some extra fun. The plot has little to do with the background and is a fairly stock military adventure scenario, but it's reasonably well-told. The story feels like an excerpt from a larger military-SF-inspired adventure, but the length keeps the quantity of tactics and maneuvering below the threshold where I would get bored. (6)

"Twelvers" by Leah Cypess: This is a sharp and occasionally mean story of adolescent cruelty and alienation. Darla is a "twelver," a child who was carried an extra three months in the womb using newly-invented medical technology because of a belief in the advantages this would bring in later life. Unfortunately for all those who used this technique, what it also brought was a preternatural calm and an unusual reaction to emotions. Darla finds it almost impossible to get upset at anything, and that, of course, prompts the cruelty and abuse of other children. Most of the story is a description of that abuse, leading up to Darla stumbling into a nasty solution to her immediate problem. It's all very believable (well, apart from the motivating biology), but I didn't enjoy reading about it, and I'm certainly not convinced that the ending will lead to anything good. (5)

"The Messenger" by Bruce McAllister: This is a very short time travel story, where time travel is used to try to unwind old family pain. This world follows the unalterable history model: no changes to the past are possible, and anything you do in the past has already happened. The mechanics are mostly avoided. Instead, McAllister concentrates on his mother, his father, and their complex relationship. I would have needed a bit more background on the characters to care enough about them for the story to be fully effective, but while the heartstring-pulling is kind of obvious, it's still a solid story. (6)

"The Copenhagen Interpretation" by Paul Cornell: This is the most ingenious of the stories in this issue. It's set in a future world that extends what seemed to me to be pre-World-War-I great power politics, although there may be a hint of the Cold War. Great nations have reached a careful balance of power, and spies and secret services work to sustain that balance. The progatonist is one of those agents, making use of advanced technology like space folds in the service of a cause that he doesn't entirely believe in. Cornell mixes in mental conditioning, artificial people, space travel, and even aliens (maybe) in a taut thriller plot that, for me, gained a great deal from the unexplained strangeness of its background. If you like diving into the deep end and following a fast-moving plot against a background of strangeness, this is the sort of SF you'll enjoy. (7)

Rating: 6 out of 10

Come meet Drupal.org team in person at DrupalCon Portland!

We’re hard at work upgrading Drupal.org to Drupal 7. DrupalCon is a perfect opportunity for you to find out what is going on with the upgrade, give us feedback on the new issue page layout and, of course, help us in the issue queue.

Where to find us:

• Weekend before DrupalCon - We are taking part in the extended sprints, come and help us close some issues
• Tuesday, May 21, 4:30pm - We’re having a BoF, Drupal.org improvements and D7 upgrade (Room B112)
• Wednesday, May 22 at 6pm - I’m presenting D7 upgrade report at the Drupal Association board meeting
• Thursday, May 23, 10:15am-1:15pm - I’ll be at the Drupal Association booth in the Exhibit Hall’s Community Village
• Friday, May 24 - We'll be closing more issues at the Contribution Sprint

Come and learn how you can help out with development, site building, or QA!

Here are the highlights from a few of the conversations I had with attendees of the 2013 Drupal Camp Alpe-Adria, held in April in Ljubljana, Slovenia. The camp was a wild success and attracted a large, international crowd. I'll post a couple more interviews I did at this event in coming weeks.

alpe-adria-voices.mp3

Good news: we finally tracked down the intermittant redirect looping bug so that I could fix it! Bad news: it was also a security vulnerability. Thankfully, it was fairly specific: you had to be using FastCGI for the login page and you also had to be using the $REMUSER_REDIRECT option. But in those situations, WebAuth versions from 4.4.1 through 4.5.2 could potentially leak authentication state from one user to another.

The full scenario is somewhat tedious to explain, but the short version is that, in 4.4.1, I switched over to using a single persistent CGI::Application object instead of re-creating it for each request. This takes better advantage of FastCGI. However, CGI::Application doesn't reset header properties between requests, and while we mostly did that internally, there was one specific case around REMOTE_USER redirects where we didn't.

For more details, including a patch for those who don't want to upgrade, see the security advisory.

WebAuth 4.5.3 has been released with only this fix relative to 4.5.2. You can get the latest release from the official WebAuth distribution site or from my WebAuth distribution pages.

There are currently 99 new Drupal contributors awaiting review of their first project. This is a great place to contribute to the community and learn about interesting upcoming projects, for example...

Module: Subscriptions by Reference What does it do?

The Subscriptions module allows you to subscribe directly to nodes, but what if you want to subscribe to one node referenced by another? Previously, you could do that with a moderate amount of custom code, maybe with the help of a blog post explaining the process. Now, the Subscriptions by Reference module promises to make that even easier. Just tell it which content types and field you want to use in the subscription and it does the rest.

Look Useful? Review it!

If you would like to see this module readily available on Drupal.org, you should review it and help make that happen.

Review It

Pro Tip: If you've never reviewed a project application before, you can find instructions for reviewers on Drupal.org and the Code Review group is happy to help more people get involved.