Elsewhere

Modules Unraveled: 115 Drupal Core Gittip Team with Jennifer Hodgdon, Bojhan Somers Alex Pott and Cathy Theys - Modules Unraveled Podcast

Planet Drupal - mer, 20/08/2014 - 07:00
Published: Wed, 08/20/14Download this episodeGitTip
  • What is GitTip? How does it work?

  • What is a GitTip team?

Drupal Core GitTip Team
  • How did the Drupal Core team come about? What prompted it’s genesis?

  • Who is the organizer of the Drupal Core team, and who is benefiting from it?
    19 members, Alex and Cathy are administering the group, a couple are on vacation.
    16 others are taking money.

  • On the GitTip page it says your goal is $5,000 US/week. What would that cover?
    Cathy: This week is the first week that we will not be able to fund the modest goal of giving people $64/week. The past few weeks we have been paying out $700. We have now eaten all our balance and have only $350 coming in this week.
    The $5k goal is what a guess at funding 6 people about ¼ time.

  • What have you all been working on lately as a result of this funding?
    Cathy: tips are for work already done, so… I'm not sure. Maybe it motivates future work, or planning to be able to do future work? Jen? Bojhan?
    What has this funding enabled you to do?

Episode Links: GitTip Team PageGitTip MembersDrupal Core Gittip Team FAQBojhan on drupal.orgBojhan on TwitterJennifer on drupal.orgJennifer on TwitterJennifer’s company siteCathy on drupal.orgCathy on TwitterAlex on drupal.orgAlex on TwitterCore Conversation at AmsterdamDrupal.org Working GroupsDries call for many separate companies to hire core developersJennifer’s case for core developers on staff at small shopsAlex Pott hired by Chapter ThreeCathy hired by BlackMeshGittip issue to change how team funds are splitTags: Drupal 8Drupal Coreplanet-drupal
Catégories: Elsewhere

Gunnar Wolf: Bigger than the cloud

Planet Debian - mer, 20/08/2014 - 06:10

Summer is cool in Mexico City.

It is cool because, unlike Spring, this is our rainy season — And rains are very predictable. Almost every day we wake up with a gorgeous, clean, blue sky.

Cool, nice temperature, around 15°C. The sun slowly evaporates the rain throughout the morning; when I go out for lunch, the sky is no longer so blue, giving way to a seemingly dirty white/grayish tint. No, it's not our world-famous pollution: It's just yesterday's rain.

Rain starts falling usually between 4 and 7 PM. Sometimes it starts as a light rain, sometimes it starts with all of its thunder, all of its might. But anyway, almost every night, there is a moment of awe, of not believing how much rain we are getting today.

It slowly fades away during the late night. And when I wake up, early next morning, everything is wet and still smells fresh.

Yes, I love our summer, even though it makes shy away from my much enjoyed cycling to work and school. And I love taking some minutes off work, look through the window of my office (located ~70m over the level of our mostly flat city) and watching how different parts of the city have sun or rain; learning to estimate the distance to the clouds, adding it to the direction and guessing which of my friends have which weather.

But I didn't realize our city had so clearly defined micro-climates... (would they really be *micro*-climates?) In fact, it even goes against my knowledge of Mexico City's logic — I always thought Coyoacán, towards the South of the city, got more rain than the Center and North because we are near the mountains, and the dominant air currents go Southwards, "clumping" the clouds by us.

But no, or at least, not this year. Regina (still in the far South — Far because she's too far away from me and I'm too egocentric; she returns home after DebConf) often asks me about the weather, as our friends working nearer the center of the city. According to the photos they post on their $social_media_of_the_day accounts, rains are really heavier there.

Today I heard on the radio accounts of yesterday's chaos after the rain. This evening, at ESIME-Culhuacán, I saw one of the reported fallen trees (of course, I am not sure if it's from yesterday's rain). And the media pushes galleries of images of a city covered in hail... While in Copilco we only had a regular rain, I'd even say a mild one.

This city is bigger than any cloud you can throw at it.

AttachmentSize IMG_20140819_155052.jpg929.69 KB
Catégories: Elsewhere

PreviousNext: Drupal 8 Now: PHPUnit tests in Drupal 7

Planet Drupal - mer, 20/08/2014 - 06:10

Drupal 8 comes with built-in support for PHP Unit for unit-testing, the industry standard for unit-tests.

But that doesn't mean you can't use PHP Unit for your testing and CI in Drupal 7, if you structure your code well.

Read on to find out what you need to do to use PHP Unit in Drupal 7.

Catégories: Elsewhere

Holger Levsen: 20140819-lts-july-2014

Planet Debian - mer, 20/08/2014 - 04:34
Debian LTS - impressions and thoughts from my first month involvement About LTS - we want feedback and more companies supporting it financially

Squeeze LTS, and even more Debian LTS, is a pretty young project, started in May 2014, so it's still a bit unclear where exactly we'll be going One purpose of this post is to spread some information about the initiative and invite you to tell us what you think about it or what your needs are.

LTS stands for "Long Term Support" and the goal of the project is to extend the security support for Squeeze (aka the current oldstable distribution) by two years. If it weren't for Squeeze LTS, the security support for it would have been stopped in May 2014 (=one year after the release of the current stable distribution), which for many is a too short timespan after it's release in February 2011. It's an experiment, we hope that there will be a similar Wheezy LTS initiative in future, but the future is unwritten and things will change based on our experiences and your needs.

If you have feedback on the direction LTS should take (or anything else LTS related), please comment on the lts mailing list. For immediate feedback there is also the #debian-lts IRC channel.

Another quite pragmatic way to express your needs is to read more about how to financially contribute to LTS and then doing exactly that - and unsurprisingly we are prioritizing the updates based on the needs expressed from our paying customers.

My LTS work in July 2014

So, "somehow" I started working for money on Debian LTS in July, which means there were 10h I got paid, and probably another 10h where I did LTS related work unpaid. I used those to release four updates for squeeze-lts (linux-2.6, file, munin and reportbug) fixing 22 CVEs in total.

The unpaid work was mostly spent on unsuccessfully working on security updates and on adding support for LTS to the security team tracker, which I improved but couldn't fully address and which I haven't properly shared / committed yet... but at least I have a local instance of the tracker now, which - for LTS - is more useful than the .debian.org one. Hopefully during DebConf14 we'll manage to fix the tracker for good.

Without success I've looked at libtasn1-3 (where the first fixes applied easily but then the code had changed too much from what was in squeeze compared to the available patches that I gave up) and libxstream-java (which is at version 1.3, while patches exist for upstream branches 1.4 and 2.x, but those need newer java to build and maybe if I'll spend two more hours I'll can get it build and then I'll have to find a useful test case, which looked quite difficult on a brief look.. so maybe I give up on libxstream-java too.... OTOH, if you use it and can do some testing, please do tell me.

Working on all these updates turned out to be more team work than expected and a lot of work involving code (which I did expect), and often code which I'd normally not look at... similarily with tools: one has to deal with tools one doesnt like, eg I had to install cdbs... And as I usually like challenges, this has actually been a lot of fun! Though it's also pretty cool to use common best practices, easy and understandable workflows. I love README.Source (or better yet, when it's not needed). And while git is of course really really great, it's still very desirable if your package builds twice (=many times) in a row, without resetting it via git.

Some more observations

The first 16 updates (until July 19th) didn't have a DLA ID, until I suggested to introduce them and insisted until we agreed. So now we agreed to put the DLA ID in the subject of the announcement mails and there is also some tool support for generating the templates/mails, but enforcing proper subjects is not done, as silent bounces are useless (and non silent ones can be abused too easily). I'm not really happy about this, but who is happy with the way email works today? And I agree, it's not the end of the world if an LTS announcement is done without a proper ID, but it looks unprofessional and could be avoided, but we have more important work to do. But one day we should automate this better.

Another detail I'm not entirely happy is the policy/current decision that "almost everything is fine to upload if deemed sensible by the uploader" (which is everyone in the Debian upload keyring(s)). In discussions before actually having the archive some people suggested the desire to upload new upstream versions too (eg newer kernels, iceweasel or other software to keep running a squeeze desktop in the modern world were discussed). I sincerely hope for most intrusive new upstream versions squeeze-(sloppy-)backports is used instead, and squeeze-lts rather not. Luckily so far all uploads were (IMHO) sensible and so, right now, I will just say that I hope it will stay this way. And it's true, one also has to install these upgrades in the first place. But people do that blindly all the time...

So by design/desire currently there is no gatekeeping mechanism whatsover (no NEW, no proposed updates), except that only some selected "few" can upload. What is uploaded (and signed correctly), gets pushed to archive, buildds and the mirrors, and hopefully maybe someone will send an announcement. So far this has worked remarkedly well - and it's also the way the Debian Security team works, as I'm told. Looking at this from a process quality / automatisation perspective, all this manual and errorprone labour seems very strange to me.

And then there is another thing: as already mentioned, the people working paid hours for this, are prioritizing their work based on customer requests. So we did two updates (python-scipy and file), which are not fixed in wheezy yet. I think this is unfortunate and while I could probably prepare the wheezy DSA for file, I don't really want to join the Security Team... or maybe I want/should join the Security Team and be a seldomly active member (eg fixing file in wheezy now....)

A note related to this: out of those 37 uploads done until today, 16 were done by those two people being paid, while the other 21 uploads were done by 10 volunteers (or at least not paid by Debian LTS). It will be interesting to see how this statistics evolves over time.

Last, but not least, there is also this can of worms (aka: the discussion) about paying people to work on Debian... I do agree it's work we didnt find volunteers for and I also see how the (financial side of the) setup is done outside of Debian (and well too, btw!), but... then we also use Debian ressources like buildds, the archive itself and official mailing lists. Also I'm currently biased in this discussion, as I directly (and happily) profit from it. I'm mentioning this here, because I believe it's important we discuss this and come to both good and practical conclusions. FWIW, we have also discussed this on the list, feel free to search the archives for it.

To bring this post to an end: for those attending DebConf14 (directly or thanks to some ninjas), there will be an event about LTS in Portland, though I'm not sure yet what I will have to talk about what hasn't been already covered here But this probably means that will be a good opportunity for you to do lots of talking instead! I'm curious what you will have to say!

Thanks for reading this far. I think I can promise that my next LTS report will be shorter

Catégories: Elsewhere

Dirk Eddelbuettel: RcppArmadillo 0.4.400.0

Planet Debian - mer, 20/08/2014 - 04:00
After two pre-releases in the last few days, Conrad finalised a new Armadillo version 4.400 today. I had kept up with the pre-releases, tested twice against all eighty (!!) CRAN dependents of RcppArmadillo and have hence uploaded RcppArmadillo 0.4.400.0 to CRAN and into Debian.

This release brings a number of new upstream features which are detailed below. As included is s bugfix for sparse matrix creation at the RcppArmadillo end which was found by the ASAN tests at CRAN --- which are similar to the sanitizers tests I recently blogged. I was able to develop and test the fix in the very docker r-devel-san images I had written about which was nice. Special thanks also to Ryan Curtin for help with the fix.

Changes in RcppArmadillo version 0.4.400.0 (2014-08-19)
  • Upgraded to Armadillo release Version 4.400 (Winter Shark Alley)

    • added gmm_diag class for statistical modelling using Gaussian Mixture Models; includes multi-threaded implementation of k-means and Expectation-Maximisation for parameter estimation

    • added clamp() for clamping values to be between lower and upper limits

    • expanded batch insertion constructors for sparse matrices to add values at repeated locations

    • faster handling of subvectors by dot()

    • faster handling of aliasing by submatrix views

  • Corrected a bug (found by the g++ Address Sanitizer) in sparse matrix initialization where space for a sentinel was allocated, but the sentinel was not set; with extra thanks to Ryan Curtin for help

  • Added a few unit tests for sparse matrices

Courtesy of CRANberries, there is also a diffstat report for the most recent release. As always, more detailed information is on the RcppArmadillo page. Questions, comments etc should go to the rcpp-devel mailing list off the R-Forge page.

This post by Dirk Eddelbuettel originated on his Thinking inside the box blog. Please report excessive re-aggregation in third-party for-profit settings.

Catégories: Elsewhere

Joey Hess: using a debian package as the remote for a local config repo

Planet Debian - mer, 20/08/2014 - 00:55

Today I did something interesting with the Debian packaging for propellor, which seems like it could be a useful technique for other Debian packages as well.

Propellor is configured by a directory, which is maintained as a local git repository. In propellor's case, it's ~/.propellor/. This contains a lot of haskell files, in fact the entire source code of propellor! That's really unusual, but I think this can be generalized to any package whose configuration is maintained in its own git repository on the user's system. For now on, I'll refer to this as the config repo.

The config repo is set up the first time a user runs propellor. But, until now, I didn't provide an easy way to update the config repo when the propellor package was updated. Nothing would break, but the old version would be used until the user updated it themselves somehow (probably by pulling from a git repository over the network, bypassing apt's signature validation).

So, what I wanted was a way to update the config repo, merging in any changes from the new version of the Debian package, while preserving the user's local modifications. Ideally, the user could just run git merge upstream/master, where the upstream repo was included in the Debian package.

But, that can't work! The Debian package can't reasonably include the full git repository of propellor with all its history. So, any git repository included in the Debian binary package would need to be a synthetic one, that only contains probably one commit that is not connected to anything else. Which means that if the config repo was cloned from that repo in version 1.0, then when version 1.1 came around, git would see no common parent when merging 1.1 into the config repo, and the merge would fail horribly.

To solve this, let's assume that the config repo's master branch has a parent commit that can be identified, somehow, as coming from a past version of the Debian package. It doesn't matter which version, although the last one merged with will be best. (The easy way to do this is to set refs/heads/upstream/master to point to it when creating the config repo.)

Once we have that parent commit, we have three things:

  1. The current content of the config repo.
  2. The content from some old version of the Debian package.
  3. The new content of the Debian package.

Now git can be used to merge #3 onto #2, with -Xtheirs, so the result is a git commit with parents of #3 and #2, and content of #3. (This can be done using a temporary clone of the config repo to avoid touching its contents.)

Such a git commit can be merged into the config repo, without any conflicts other than those the user might have caused with their own edits.

So, propellor will tell the user when updates are available, and they can simply run git merge upstream/master to get them. The resulting history looks like this:

* Merge remote-tracking branch 'upstream/master' |\ | * merging upstream version | |\ | | * upstream version * | user change |/ * upstream version

So, generalizing this, if a package has a lot of config files, and creates a git repository containing them when the user uses it (or automatically when it's installed), this method can be used to provide an easily mergable branch that tracks the files as distributed with the package.

It would perhaps not be hard to get from here to a full git-backed version of ucf. Note that the Debian binary package doesn't have to ship a git repisitory, it can just as easily ship the current version of the config files somewhere in /usr, and check them into a new empty repository as part of the generation of the upstream/master branch.

Catégories: Elsewhere

Phase2: Profiling Drupal Performance with PHPStorm and Xdebug

Planet Drupal - mar, 19/08/2014 - 22:34

Profiling is about measuring the performance of PHP code, at least when we are talking about Drupal and Xdebug. You might need to profile your site or app if you work at a firm where performance is highly scrutinized, or if you are having problems getting a migration to complete. Whatever the reason, if you have been tasked with analyzing the performance of your Drupal codebase, profiling is one great way of doing so. Note that Xdebug’s profiler does not track memory usage. If you want to know more about memory performance tracking you should check out Xdebug’s execution trace features.

Alright then lets get started! 

Whoa there cowboy! First you need to know that the act of profiling your code is itself taking resources to accomplish. The more work your code does, the more information that the profiler stores; file sizes for these logs can get very big very quickly. You have been warned. To get going with profiling Drupal in PHPStorm and Xdebug you need:

To setup your environment, edit your php.ini file and add the following lines:

xdebug.profiler_output_dir=/tmp/profiler/ xdebug.profiler_enable=on xdebug.profiler_trigger=on xdebug.profiler_append=on

Depending on what you are testing and how, you may want to adjust the settings for your site. For instance, if you are using Drush to run a migration, you can’t start the profiler on-demand, and that affects the profiler_trigger setting. For my dev site I used the php.ini config you see above and simply added a URL parameter “XDEBUG_PROFILE=on” to my site’s url; this starts Xdebug profiling from the browser.

To give you an idea of what is possible, lets profile the work required to view a simple Drupal node. To profile the node view I visited http://profiler.loc/node/48581?XDEBUG_PROFILE=on in my browser. I didn’t see any flashing lights or hear bells and whistles, but I should have a binary file that PHPStorm can inspect, located in the path I setup in my php.ini profiler_output_dir directive.

Finally lets look at all of our hard work! In PHPStorm navigate to Tools->Analyze Xdebug Profile Snapshot. Browse to your profiler output directory and you should see at least one cachgrind.out.%p file (%p refers to the process id the script used). Open the file with the largest process id appended to the end of the filename.

We are then greeted with a new tab showing the results of the profiler.

The output shows us the functions called, how many times they were called, and the amount of execution time each function took. Additionally, you can see the hierarchy of all function calls and follow potential bottlenecks down to their roots.

There you have it! Go wild and profile all the things! Just kidding, don’t do that.

Catégories: Elsewhere

Phase2: Profiling Drupal Performance with PHPStorm and Xdebug

Planet Drupal - mar, 19/08/2014 - 22:00

Profiling is about measuring the performance of PHP code, at least when we are talking about Drupal and Xdebug. You might need to profile your site or app if you work at a firm where performance is highly scrutinized, or if you are having problems getting a migration to complete. Whatever the reason, if you have been tasked with analyzing the performance of your Drupal codebase, profiling is one great way of doing so. Note that Xdebug’s profiler does not track memory usage. If you want to know more about memory performance tracking you should check out Xdebug’s execution trace features.

Alright then lets get started! 

Whoa there cowboy! First you need to know that the act of profiling your code is itself taking resources to accomplish. The more work your code does, the more information that the profiler stores; file sizes for these logs can get very big very quickly. You have been warned. To get going with profiling Drupal in PHPStorm and Xdebug you need:

  • PHPStorm
  • PHP with the Xdebug extension
  • A website running on Drupal.

To setup your environment, edit your php.ini file and add the following lines:

xdebug.profiler_output_dir=/tmp/profiler/ xdebug.profiler_enable=on xdebug.profiler_trigger=on xdebug.profiler_append=on

Depending on what you are testing and how, you may want to adjust the settings for your site. For instance, if you are using Drush to run a migration, you can’t start the profiler on-demand, and that affects the profiler_trigger setting. For my dev site I used the php.ini config you see above and simply added a URL parameter “XDEBUG_PROFILE=on” to my site’s url; this starts Xdebug profiling from the browser.

To give you an idea of what is possible, lets profile the work required to view a simple Drupal node. To profile the node view I visited http://profiler.loc/node/48581?XDEBUG_PROFILE=on in my browser. I didn’t see any flashing lights or hear bells and whistles, but I should have a binary file that PHPStorm can inspect, located in the path I setup in my php.ini profiler_output_dir directive.

Finally lets look at all of our hard work! In PHPStorm navigate to Tools->Analyze Xdebug Profile Snapshot. Browse to your profiler output directory and you should see at least one cachgrind.out.%p file (%p refers to the process id the script used). Open the file with the largest process id appended to the end of the filename.

We are then greeted with a new tab showing the results of the profiler.

 

The output shows us the functions called, how many times they were called, and the amount of execution time each function took. Additionally, you can see the hierarchy of all function calls and follow potential bottlenecks down to their roots.

There you have it! Go wild and profile all the things! Just kidding, don’t do that.

Catégories: Elsewhere

DrupalCon Amsterdam: Schedules, BOFS, & Training in Amsterdam

Planet Drupal - mar, 19/08/2014 - 20:20

The schedule for DrupalCon Amsterdam is live, which means that you can start planning out every detail of your Amsterdam experience. You can start the hard work of choosing the sessions, BOFs, and social events you want to attend, and build your own schedule right on the DrupalCon Amsterdam site.

BOF scheduling is live

Speaking of BOFs, you don’t have to wait until DrupalCon Amsterdam to claim yours: you can start using the online booking feature today to schedule your BOFs. Be sure you do it soon, though— BOF rooms go fast!

Register for training before 5 September

Lastly, we need more people to register to attend training at DrupalCon Amsterdam. Show us you're interested in these topics! Book before 5 September to make sure the course you want to attend runs. We know it's difficult to make a decision this early out, but classes which do not meet the minimum to run will be cancelled!

The training options are all fantastic and a great opportunity to learn more about Drupal, so register today.

See you in Amsterdam!

Catégories: Elsewhere

Julien Danjou: Tracking OpenStack contributions in GitHub

Planet Debian - mar, 19/08/2014 - 19:00

I've switched my Git repositories to GitHub recently, and started to watch my contributions statistics, which were very low considering I spend my days hacking on open source software, especially OpenStack.

OpenStack hosts its Git repositories on its own infrastructure at git.openstack.org, but also mirrors them on GitHub. Logically, I was expecting GitHub to track my commits there too, as I'm using the same email address everywhere.

It turns out that it was not the case, and the help page about that on GitHub describes the rule in place to compute statistics. Indeed, according to GitHub, I had no relations to the OpenStack repositories, as I never forked them nor opened a pull request on them (OpenStack uses Gerrit).

Starring a repository is enough to build a relationship between a user and a repository, so this is was the only thing needed to inform GitHub that I have contributed to those repositories. Considering OpenStack has hundreds of repositories, I decided to star them all by using a small Python script.

And voilà, my statistics are now including all my contributions to OpenStack!

Catégories: Elsewhere

Drupalize.Me: Upgrading Drush to work with Drupal 8

Planet Drupal - mar, 19/08/2014 - 16:00

When I first started learning Drupal, I remember the process of enabling and disabling modules on the Modules page and it took for-ev-er. My laptop was in serious danger of getting hurled across the room, due to my frustration. Then I discovered drush, and downloading and enabling modules was now performed with ease instead of pain and suffering. Of course there's a lot more you can do with drush than just download and enable modules, this is just one example.

I've been using Drush 6.x on my local machine for quite some time now. Poking around Drupal 8's UI and seeing what's new, I haven't missed drush too much...until it was time to test drive a new contrib module for Drupal 8. When I typed into my Terminal window drush dl page_manager, I got quite the error message:

Drush 6.x only works with Drupal 6 or 7. If I wanted to use Drush on my Drupal 8 site, I would need to upgrade to Drush 7.x.

Catégories: Elsewhere

Drupalize.Me: Upgrading Drush to work with Drupal 8

Planet Drupal - mar, 19/08/2014 - 16:00

When I first started learning Drupal, I remember the process of enabling and disabling modules on the Modules page and it took for-ev-er. My laptop was in serious danger of getting hurled across the room, due to my frustration. Then I discovered drush, and downloading and enabling modules was now performed with ease instead of pain and suffering. Of course there's a lot more you can do with drush than just download and enable modules, this is just one example.

I've been using Drush 6.x on my local machine for quite some time now. Poking around Drupal 8's UI and seeing what's new, I haven't missed drush too much...until it was time to test drive a new contrib module for Drupal 8. When I typed into my Terminal window drush dl page_manager, I got quite the error message:

Drush 6.x only works with Drupal 6 or 7. If I wanted to use Drush on my Drupal 8 site, I would need to upgrade to Drush 7.x.

Catégories: Elsewhere

ComputerMinds.co.uk: Language lessons: What are you translating?

Planet Drupal - mar, 19/08/2014 - 14:00
Content (node-level) translation or entity (field-level) translation?

It seems an obvious question to ask, but what are you translating?

The tools exist to translate just about anything in Drupal 7*, but in many different ways, so you need to know exactly what you're translating. Language is 'a first-class citizen', in the sense that any piece of text is inherently written by someone on some language, which Drupal 7 is built to recognise. Sometimes you want to translate each & every individual piece of text (e.g. at the sentence or paragraph level). Other times you want to translate a whole page or section that is made up of multiple pieces of text.

Catégories: Elsewhere

Joachim's blog: Getting Module Builder ready for Drupal 8

Planet Drupal - mar, 19/08/2014 - 10:12

I've just made a commit to Module Builder that adds unit tests. This is a big deal, because having these frees me up to start making the big changes that are needed for supporting Drupal 8's new structures: routes, plugins, forms, and so on.

The biggest challenge is going to be the interface. Currently, you give Module Builder just a module name and a list of hook names, and it does the necessary. On the command line it's nice and simple:

drush mb mymodule install schema node_insert form_alter views_data_alter

The first parameter is the module name, and everything that follows is a hook name. Now we add to the mix requests such as a form called MyModuleCakeToppingForm, or an entity type plugin, or a route bake_my_cake and its page controller. How to elegantly specify all that over the command line, without making it horribly unwieldy and impossible to remember how to use?

It's also going to be an interesting exercise in reading my own documentation and seeing how much sense it makes after something like 7 months away from the code.

From what I recall, Module Builder uses a hierarchy of component generators to build your module. Taking our example above, the first thing that happens is that the Module generator class kicks in. 'So, you want a module, do you?' it asks, 'You'll need some of these.' And it begins to assemble a list of further generators, for the components it needs: an info file, and the hooks generator. The hooks generator does the actual job of examining your list of requested hooks, and decides based on that that you need three code files: a .module, a .install, and a .views.inc. So by now we have a tree of generators like this:

- Module -- Info file -- Hooks --- Code file: .module --- Code file: .install --- Code file: .views.inc

This is not a class hierarchy; this is a tree of objects where each generator has a list of the generators beneath it, and is responsible for collecting data from them. Once we have the tree, we iteratively have each generator assemble the data it wants to contribute, starting with the Module generator at the top.

The original plan when I wrote this system was to make the smallest granularity be a file. The leaves of the generator tree would assemble the text for their file's contents, and the Module generator would collect the files up and return them to the caller for output (either in the UI, or to write them directly).

However, while the original intention of this system was that it could be generalised to base components other than modules (so profiles and themes, which are both supported to some extend but lack the UI, see above!), it's also proven to be extendable downwards to smaller components, and to be worthwhile to do so.

Enter the Form generator. Once we have a generic Function generator (and its child class the HookImplementation), we can create a Form generator. Given a form machine name, 'foo_form', it simply knows to add three copies of the Function generator: 'foo_form', 'foo_form_validate', 'foo_form_submit', along with the correct parameters and some boiler plate code.

And we can specialize this further: the AdminSettingsForm simply extends the Form generator, and adds a menu item component, which itself ensures hook_menu() is requested.

At this point it starts to get a bit complicated, as we have components that request other components that are in totally different parts of the component tree. That's the point at which I think I was when I realized I needed tests so that I can refactor and clean up the messy bits of this, and enhance and extend it, without breaking what's already there.

So that's the current state of Module Builder: not yet ready for Drupal 8, but has lots of potential. At this point, I'd really welcome input on the Drush interface, as that's the big quandary. And any input on new Drupal 8 component generators would be great too; there are a few open issues in the queue. And finally, Module Builder is a complex beast; should anyone looking at the code find it baffling and impenetrable, do please file a documentation issue to highlight the problem and request clarification.

Catégories: Elsewhere

Jelmer Vernooij: Using Propellor for configuration management

Planet Debian - lun, 18/08/2014 - 23:15

For a while, I've been wanting to set up configuration management for my home network. With half a dozen servers, a VPS and a workstation it is not big, but large enough to make it annoying to manually log into each machine for network-wide changes.

Most of the servers I have are low-end ARM machines, each responsible for a couple of tasks. Most of my machines run Debian or something derived from Debian. Oh, and I'm a member of the declarative school of configuration management.

Propellor

Propellor caught my eye earlier this year. Unlike some other configuration management tools, it doesn't come with its own custom language but it is written in Haskell, which I am already familiar with. It's also fairly simple, declarative, and seems to do most of the handful of things that I need.

Propellor is essentially a Haskell application that you customize for your site. It works very similar to e.g. xmonad, where you write a bit of Haskell code for configuration which uses the upstream library code. When you run the application it takes your code and builds a binary from your code and the upstream libraries.

Each host on which Propellor is used keeps a clone of the site-local Propellor git repository in /usr/local/propellor. Every time propellor runs (either because of a manual "spin", or from a cronjob it can set up for you), it fetches updates from the main site-local git repository, compiles the Haskell application and runs it.

Setup

Propellor was surprisingly easy to set up. Running propellor creates a clone of the upstream repository under ~/.propellor with a README file and some example configuration. I copied config-simple.hs to config.hs, updated it to reflect one of my hosts and within a few minutes I had a basic working propellor setup.

You can use ./propellor <host> to trigger a run on a remote host.

At the moment I have propellor working for some basic things - having certain Debian packages installed, a specific network configuration, mail setup, basic Kerberos configuration and certain SSH options set. This took surprisingly little time to set up, and it's been great being able to take full advantage of Haskell.

Propellor comes with convenience functions for dealing with some commonly used packages, such as Apt, SSH and Postfix. For a lot of the other packages, you'll have to roll your own for now. I've written some extra code to make Propellor deal with Kerberos keytabs and Dovecot, which I hope to submit upstream.

I don't have a lot of experience with other Free Software configuration management tools such as Puppet and Chef, but for my use case Propellor works very well.

The main disadvantage of propellor for me so far is that it needs to build itself on each machine it runs on. This is fine for my workstation and high-end servers, but it is somewhat more problematic on e.g. my Raspberry Pi's. Compilation takes a while, and the Haskell compiler and libraries it needs amount to 500Mb worth of disk space on the tiny root partition.

In order to work with Propellor, some Haskell knowledge is required. The Haskell in the configuration file is reasonably easy to understand if you keep it simple, but once the compiler spits out error messages then I suspect you'll have a hard time without any Haskell knowledge.

Propellor relies on having a central repository with the configuration that it can pull from as root. Unlike Joey, I am wary of publishing the configuration of my home network and I don't have a highly available local git server setup.

Catégories: Elsewhere

Daniel Pocock: Is WebRTC private?

Planet Drupal - lun, 18/08/2014 - 21:55

With the exciting developments at rtc.debian.org, many people are starting to look more closely at browser-based real-time communications.

Some have dared to ask: does it solve the privacy problems of existing solutions?

Privacy is a relative term

Perfect privacy and its technical manifestations are hard to define. I had a go at it in a blog on the Gold Standard for free communications technology on 5 June 2013. By pure co-incidence, a few hours later, the first Snowden leaks appeared and this particular human right was suddenly thrust into the spotlight.

WebRTC and ICE privacy risk

WebRTC does not give you perfect privacy.

At least one astute observer at my session at Paris mini-DebConf 2014 questioned the privacy of Interactive Connectivity Establishment (ICE, RFC 5245).

In its most basic form, ICE scans all the local IP addresses on your machine and NAT gateway and sends them to the person calling you so that their phone can find the optimal path to contact you. This clearly has privacy implications as a caller can work out which ISP you are connected to and some rough details of your network topology at any given moment in time.

What WebRTC does bring to the table

Some of this can be mitigated though: an ICE implementation can be tuned so that it only advertises the IP address of a dedicated relay host. If you can afford a little latency, your privacy is safe again. This privacy protecting initiative could be made by a browser vendor such as Mozilla or it can be done in JavaScript by a softphone such as JSCommunicator.

Many individuals are now using a proprietary softphone to talk to family and friends around the world. The softphone in question has properties like a virus, siphoning away your private information. This proprietary softphone is also an insidious threat to open source and free operating systems on the desktop. WebRTC is a positive step back from the brink. It gives people a choice.

WebRTC is a particularly relevant choice for business. Can you imagine going to a business and asking them to make all their email communication through hotmail? When a business starts using a particular proprietary softphone, how is it any different? WebRTC offers a solution that is actually easier for the user and can be secured back to the business network using TLS.

WebRTC is based on open standards, particularly HTML5. Leading implementations, such as the SIP over WebSocket support in reSIProcate, JSCommunicator and the DruCall module for Drupal are fully open source. Not only is it great to be free, it is possible to extend and customize any of these components.

What is missing

There are some things that are not quite there yet and require a serious effort from the browser vendors. At the top of the list for privacy:

  • ZRTP support - browsers currently support DTLS-SRTP, which is based on X.509. ZRTP is more like PGP, a democratic and distributed peer-to-peer privacy solution without needing to trust some central certificate authority.
  • TLS with PGP - the TLS protocol used to secure the WebSocket signalling channel is also based on X.509 with the risk of a central certificate authority. There is increasing chatter about the need for TLS to use PGP instead of X.509 and WebRTC would be a big winner if this were to eventuate and be combined with ZRTP.

You may think "I'll believe it when I see it". Each of these features, including WebRTC itself, is a piece of the puzzle and even solving one piece at a time brings people further out of danger from the proprietary mess the world lives with today.

Catégories: Elsewhere

Daniel Pocock: Is WebRTC private?

Planet Debian - lun, 18/08/2014 - 21:55

With the exciting developments at rtc.debian.org, many people are starting to look more closely at browser-based real-time communications.

Some have dared to ask: does it solve the privacy problems of existing solutions?

Privacy is a relative term

Perfect privacy and its technical manifestations are hard to define. I had a go at it in a blog on the Gold Standard for free communications technology on 5 June 2013. By pure co-incidence, a few hours later, the first Snowden leaks appeared and this particular human right was suddenly thrust into the spotlight.

WebRTC and ICE privacy risk

WebRTC does not give you perfect privacy.

At least one astute observer at my session at Paris mini-DebConf 2014 questioned the privacy of Interactive Connectivity Establishment (ICE, RFC 5245).

In its most basic form, ICE scans all the local IP addresses on your machine and NAT gateway and sends them to the person calling you so that their phone can find the optimal path to contact you. This clearly has privacy implications as a caller can work out which ISP you are connected to and some rough details of your network topology at any given moment in time.

What WebRTC does bring to the table

Some of this can be mitigated though: an ICE implementation can be tuned so that it only advertises the IP address of a dedicated relay host. If you can afford a little latency, your privacy is safe again. This privacy protecting initiative could be made by a browser vendor such as Mozilla or it can be done in JavaScript by a softphone such as JSCommunicator.

Many individuals are now using a proprietary softphone to talk to family and friends around the world. The softphone in question has properties like a virus, siphoning away your private information. This proprietary softphone is also an insidious threat to open source and free operating systems on the desktop. WebRTC is a positive step back from the brink. It gives people a choice.

WebRTC is a particularly relevant choice for business. Can you imagine going to a business and asking them to make all their email communication through hotmail? When a business starts using a particular proprietary softphone, how is it any different? WebRTC offers a solution that is actually easier for the user and can be secured back to the business network using TLS.

WebRTC is based on open standards, particularly HTML5. Leading implementations, such as the SIP over WebSocket support in reSIProcate, JSCommunicator and the DruCall module for Drupal are fully open source. Not only is it great to be free, it is possible to extend and customize any of these components.

What is missing

There are some things that are not quite there yet and require a serious effort from the browser vendors. At the top of the list for privacy:

  • ZRTP support - browsers currently support DTLS-SRTP, which is based on X.509. ZRTP is more like PGP, a democratic and distributed peer-to-peer privacy solution without needing to trust some central certificate authority.
  • TLS with PGP - the TLS protocol used to secure the WebSocket signalling channel is also based on X.509 with the risk of a central certificate authority. There is increasing chatter about the need for TLS to use PGP instead of X.509 and WebRTC would be a big winner if this were to eventuate and be combined with ZRTP.

You may think "I'll believe it when I see it". Each of these features, including WebRTC itself, is a piece of the puzzle and even solving one piece at a time brings people further out of danger from the proprietary mess the world lives with today.

Catégories: Elsewhere

Drupal.org Featured Case Studies: Viraland

Planet Drupal - lun, 18/08/2014 - 20:27
Completed Drupal site or project URL: http://www.viraland.gr/

Viraland is a Greek online community where you can find the latest viral messages from around the world. Users participate in the community by posting new content and sharing it through social channels. Strange, meaningful and popular messages that go viral are posted daily in Viraland, reflecting the interests of its users.

Key modules/theme/distribution used: ViewsPanelsZenMediaJanrain Social LoginRulesInternationalizationField PermissionsTeam members: highvrahos
Catégories: Elsewhere

SitePoint PHP Drupal: Fine Tuning Drupal Themes with Patterns, Arg and Types

Planet Drupal - lun, 18/08/2014 - 18:00

In this article, we’ll discuss how you can leverage various Drupal API functions to achieve more fine grained theming. We’ll cover template preprocessing and alter hooks using path patterns, types and args(). We’ll use the arg() function which returns parts of a current Drupal URL path and some pattern matching for instances when you want […]

Continue reading %Fine Tuning Drupal Themes with Patterns, Arg and Types%

Catégories: Elsewhere

Julien Danjou: OpenStack Ceilometer and the Gnocchi experiment

Planet Debian - lun, 18/08/2014 - 17:00

A little more than 2 years ago, the Ceilometer project was launched inside the OpenStack ecosystem. Its main objective was to measure OpenStack cloud platforms in order to provide data and mechanisms for functionalities such as billing, alarming or capacity planning.

In this article, I would like to relate what I've been doing with other Ceilometer developers in the last 5 months. I've lowered my involvement in Ceilometer itself directly to concentrate on solving one of its biggest issue at the source, and I think it's largely time to take a break and talk about it.

Ceilometer early design

For the last years, Ceilometer didn't change in its core architecture. Without diving too much in all its parts, one of the early design decision was to build the metering around a data structure we called samples. A sample is generated each time Ceilometer measures something. It is composed of a few fields, such as the the resource id that is metered, the user and project id owning that resources, the meter name, the measured value, a timestamp and a few free-form metadata. Each time Ceilometer measures something, one of its components (an agent, a pollster…) constructs and emits a sample headed for the storage component that we call the collector.

This collector is responsible for storing the samples into a database. The Ceilometer collector uses a pluggable storage system, meaning that you can pick any database system you prefer. Our original implementation has been based on MongoDB from the beginning, but we then added a SQL driver, and people contributed things such as HBase or DB2 support.

The REST API exposed by Ceilometer allows to execute various reading requests on this data store. It can returns you the list of resources that have been measured for a particular project, or compute some statistics on metrics. Allowing such a large panel of possibilities and having such a flexible data structure allows to do a lot of different things with Ceilometer, as you can almost query the data in any mean you want.

The scalability issue

We soon started to encounter scalability issues in many of the read requests made via the REST API. A lot of the requests requires the data storage to do full scans of all the stored samples. Indeed, the fact that the API allows you to filter on any fields and also on the free-form metadata (meaning non indexed key/values tuples) has a terrible cost in terms of performance (as pointed before, the metadata are attached to each sample generated by Ceilometer and is stored as is). That basically means that the sample data structure is stored in most drivers in just one table or collection, in order to be able to scan them at once, and there's no good "perfect" sharding solution, making data storage scalability painful.

It turns out that the Ceilometer REST API is unable to handle most of the requests in a timely manner as most operations are O(n) where n is the number of samples recorded (see big O notation if you're unfamiliar with it). That number of samples can grow very rapidly in an environment of thousands of metered nodes and with a data retention of several weeks. There is a few optimizations to make things smoother in general cases fortunately, but as soon as you run specific queries, the API gets barely usable.

During this last year, as the Ceilometer PTL, I discovered these issues first hand since a lot of people were feeding me back with this kind of testimony. We engaged several blueprints to improve the situation, but it was soon clear to me that this was not going to be enough anyway.

Thinking outside the box

Unfortunately, the PTL job doesn't leave him enough time to work on the actual code nor to play with anything new. I was coping with most of the project bureaucracy and I wasn't able to work on any good solution to tackle the issue at its root. Still, I had a few ideas that I wanted to try and as soon as I stepped down from the PTL role, I stopped working on Ceilometer itself to try something new and to think a bit outside the box.

When one takes a look at what have been brought recently in Ceilometer, they can see the idea that Ceilometer actually needs to handle 2 types of data: events and metrics.

Events are data generated when something happens: an instance start, a volume is attached, or an HTTP request is sent to an REST API server. These are events that Ceilometer needs to collect and store. Most OpenStack components are able to send such events using the notification system built into oslo.messaging.

Metrics is what Ceilometer needs to store but that is not necessarily tied to an event. Think about an instance CPU usage, a router network bandwidth usage, the number of images that Glance is storing for you, etc… These are not events, since nothing is happening. These are facts, states we need to meter.

Computing statistics for billing or capacity planning requires both of these data sources, but they should be distinct. Based on that assumption, and the fact that Ceilometer was getting support for storing events, I started to focus on getting the metric part right.

I had been a system administrator for a decade before jumping into OpenStack development, so I know a thing or two on how monitoring is done in this area, and what kind of technology operators rely on. I also know that there's still no silver bullet – this made it a good challenge.

The first thing that came to my mind was to use some kind of time-series database, and export its access via a REST API – as we do in all OpenStack services. This should cover the metric storage pretty well.

Cooking Gnocchi A cloud of gnocchis!

At the end of April 2014, this led met to start a new project code-named Gnocchi. For the record, the name was picked after confusing so many times the OpenStack Marconi project, reading OpenStack Macaroni instead. At least one OpenStack project should have a "pasta" name, right?

The point of having a new project and not send patches on Ceilometer, was that first I had no clue if it was going to make something that would be any better, and second, being able to iterate more rapidly without being strongly coupled with the release process.

The first prototype started around the following idea: what you want is to meter things. That means storing a list of tuples of (timestamp, value) for it. I've named these things "entities", as no assumption are made on what they are. An entity can represent the temperature in a room or the CPU usage of an instance. The service shouldn't care and should be agnostic in this regard.

One feature that we discussed for several OpenStack summits in the Ceilometer sessions, was the idea of doing aggregation. Meaning, aggregating samples over a period of time to only store a smaller amount of them. These are things that time-series format such as the RRDtool have been doing for a long time on the fly, and I decided it was a good trail to follow.

I assumed that this was going to be a requirement when storing metrics into Gnocchi. The user would need to provide what kind of archiving it would need: 1 second precision over a day, 1 hour precision over a year, or even both.

The first driver written to achieve that and store those metrics inside Gnocchi was based on whisper. Whisper is the file format used to store metrics for the Graphite project. For the actual storage, the driver uses Swift, which has the advantages to be part of OpenStack and scalable.

Storing metrics for each entities in a different whisper file and putting them in Swift turned out to have a fantastic algorithm complexity: it was O(1). Indeed, the complexity needed to store and retrieve metrics doesn't depends on the number of metrics you have nor on the number of things you are metering. Which is already a huge win compared to the current Ceilometer collector design.

However, it turned out that whisper has a few limitations that I was unable to circumvent in any manner. I needed to patch it to remove a lot of its assumption about manipulating file, or that everything is relative to now (time.time()). I've started to hack on that in my own fork, but… then everything broke. The whisper project code base is, well, not the state of the art, and have 0 unit test. I was starring at a huge effort to transform whisper into the time-series format I wanted, without being sure I wasn't going to break everything (remember, no test coverage).

I decided to take a break and look into alternatives, and stumbled upon Pandas, a data manipulation and statistics library for Python. Turns out that Pandas support time-series natively, and that it could do a lot of the smart computation needed in Gnocchi. I built a new file format leveraging Pandas for computing the time-series and named it carbonara (a wink to both the Carbon project and pasta, how clever!). The code is quite small (a third of whisper's, 200 SLOC vs 600 SLOC), does not have many of the whisper limitations and… it has test coverage. These Carbonara files are then, in the same fashion, stored into Swift containers.

Anyway, Gnocchi storage driver system is designed in the same spirit that the rest of OpenStack and Ceilometer storage driver system. It's a plug-in system with an API, so anyone can write their own driver. Eoghan Glynn has already started to write a InfluxDB driver, working closely with the upstream developer of that database. Dina Belova started to write an OpenTSDB driver. This helps to make sure the API is designed directly in the right way.

Handling resources

Measuring individual entities is great and needed, but you also need to link them with resources. When measuring the temperature and the number of a people in a room, it is useful to link these 2 separate entities to a resource, in that case the room, and give a name to these relations, so one is able to identify what attribute of the resource is actually measured. It is also important to provide the possibility to store attributes on these resources, such as their owners, the time they started and ended their existence, etc.

Relationship of entities and resources

Once this list of resource is collected, the next step is to list and filter them, based on any criteria. One might want to retrieve the list of resources created last week or the list of instances hosted on a particular node right now.

Resources also need to be specialized. Some resources have attributes that must be stored in order for filtering to be useful. Think about an instance name or a router network.

All of these requirements led to to the design of what's called the indexer. The indexer is responsible for indexing entities, resources, and link them together. The initial implementation is based on SQLAlchemy and should be pretty efficient. It's easy enough to index the most requested attributes (columns), and they are also correctly typed.

We plan to establish a model for all known OpenStack resources (instances, volumes, networks, …) to store and index them into the Gnocchi indexer in order to request them in an efficient way from one place. The generic resource class can be used to handle generic resources that are not tied to OpenStack. It'd be up to the users to store extra attributes.

Dropping the free form metadata we used to have in Ceilometer makes sure that querying the indexer is going to be efficient and scalable.

The indexer classes and their relations REST API

All of this is exported via a REST API that was partially designed and documented in the Gnocchi specification in the Ceilometer repository; though the spec is not up-to-date yet. We plan to auto-generate the documentation from the code as we are currently doing in Ceilometer.

The REST API is pretty easy to use, and you can use it to manipulate entities and resources, and request the information back.

Macroscopic view of the Gnocchi architecture Roadmap & Ceilometer integration

All of this plan has been exposed and discussed with the Ceilometer team during the last OpenStack summit in Atlanta in May 2014, for the Juno release. I led a session about this entire concept, and convinced the team that using Gnocchi for our metric storage would be a good approach to solve the Ceilometer collector scalability issue.

It was decided to conduct this project experiment in parallel of the current Ceilometer collector for the time being, and see where that would lead the project to.

Early benchmarks

Some engineers from Mirantis did a few benchmarks around Ceilometer and also against an early version of Gnocchi, and Dina Belova presented them to us during the mid-cycle sprint we organized in Paris in early July.

The following graph sums up pretty well the current Ceilometer performance issue. The more you feed it with metrics, the more slow it becomes.

For Gnocchi, while the numbers themselves are not fantastic, what is interesting is that all the graphs below show that the performances are stable without correlation with the number of resources, entities or measures. This proves that, indeed, most of the code is built around a complexity of O(1), and not O(n) anymore.

Next steps Clément drawing the logo

While the Juno cycle is being wrapped-up for most projects, including Ceilometer, Gnocchi development is still ongoing. Fortunately, the composite architecture of Ceilometer allows a lot of its features to be replaced by some other code dynamically. That, for example, enables Gnocchi to provides a Ceilometer dispatcher plugin for its collector, without having to ship the actual code in Ceilometer itself. That should help the development of Gnocchi to not be slowed down by the release process for now.

The Ceilometer team aims to provide Gnocchi as a sort of technology preview with the Juno release, allowing it to be deployed along and plugged with Ceilometer. We'll discuss how to integrate it in the project in a more permanent and strong manner probably during the OpenStack Summit for Kilo that will take place next November in Paris.

Catégories: Elsewhere

Pages

Subscribe to jfhovinne agrégateur - Elsewhere