A lecture at LCA 2011 which included some inappropriate slides was followed by long discussions on mailing lists. In February 2011 I wrote a blog post debunking some of the bogus arguments in two lists . One of the noteworthy incidents in the mailing list discussion concerned Ted Ts’o (an influential member of the Linux community) debating the definition of rape. My main point on that issue in Feb 2011 was that it’s insensitive to needlessly debate the statistics.
Recently Valerie Aurora wrote about another aspect of this on The Ada Initiative blog  and on her personal blog. Some of her significant points are that conference harassment doesn’t end when the conference ends (it can continue on mailing lists etc), that good people shouldn’t do nothing when bad things happen, and that free speech doesn’t mean freedom from consequences or the freedom to use private resources (such as conference mailing lists) without restriction.
Craig Sanders wrote a very misguided post about the Ted Ts’o situation . One of the many things wrong with his post is his statement “I’m particularly disgusted by the men who intervene way too early – without an explicit invitation or request for help or a clear need such as an immediate threat of violence – in womens’ issues“.
I believe that as a general rule when any group of people are involved in causing a problem they should be involved in fixing it. So when we have problems that are broadly based around men treating women badly the prime responsibility should be upon men to fix them. It seems very clear that no matter what scope is chosen for fixing the problems (whether it be lobbying for new legislation, sociological research, blogging, or directly discussing issues with people to change their attitudes) women are doing considerably more than half the work. I believe that this is an indication that overall men are failing.Asking for Help
I don’t believe that members of minority groups should have to ask for help. Asking isn’t easy, having someone spontaneously offer help because it’s the right thing to do can be a lot easier to accept psychologically than having to beg for help. There is a book named “Women Don’t Ask” which has a page on the geek feminism Wiki . I think the fact that so many women relate to a book named “Women Don’t Ask” is an indication that we shouldn’t expect women to ask directly, particularly in times of stress. The Wiki page notes a criticism of the book that some specific requests are framed as “complaining”, so I think we should consider a “complaint” from a woman as a direct request to do something.
The geek feminism blog has an article titled “How To Exclude Women Without Really Trying” which covers many aspects of one incident . Near the end of the article is a direct call for men to be involved in dealing with such problems. The geek feminism Wiki has a page on “Allies” which includes “Even a blog post helps” . It seems clear from public web sites run by women that women really want men to be involved.
Finally when I get blog comments and private email from women who thank me for my posts I take it as an implied request to do more of the same.
One thing that we really don’t want is to have men wait and do nothing until there is an immediate threat of violence. There are two massive problems with that plan, one is that being saved from a violent situation isn’t a fun experience, the other is that an immediate threat of violence is most likely to happen when there is no-one around to intervene.Men Don’t Listen to Women
Rebecca Solnit wrote an article about being ignored by men titled “Men Explain Things to Me” . When discussing women’s issues the term “Mansplaining” is often used for that sort of thing, the geek feminism Wiki has some background . It seems obvious that the men who have the greatest need to be taught some things related to women’s issues are the ones who are least likely to listen to women. This implies that other men have to teach them.
Craig says that women need “space to discover and practice their own strength and their own voices“. I think that the best way to achieve that goal is to listen when women speak. Of course that doesn’t preclude speaking as well, just listen first, listen carefully, and listen more than you speak.
Craig claims that when men like me and Matthew Garrett comment on such issues we are making “women’s spaces more comfortable, more palatable, for men“. From all the discussion on this it seems quite obvious that what would make things more comfortable for men would be for the issue to never be discussed at all. It seems to me that two of the ways of making such discussions uncomfortable for most men are to discuss sexual assault and to discuss what should be done when you have a friend who treats women in a way that you don’t like. Matthew has covered both of those so it seems that he’s doing a good job of making men uncomfortable – I think that this is a good thing, a discussion that is “comfortable and palatable” for the people in power is not going to be any good for the people who aren’t in power.The Voting Aspect
It seems to me that when certain issues are discussed we have a social process that is some form of vote. If one person complains then they are portrayed as crazy. When other people agree with the complaint then their comments are marginalised to try and preserve the narrative of one crazy person. It seems that in the case of the discussion about Rape Apology and LCA2011 most men who comment regard it as one person (either Valeria Aurora or Matthew Garrett) causing a dispute. There is even some commentary which references my blog post about Rape Apology  but somehow manages to ignore me when it comes to counting more than one person agreeing with Valerie. For reference David Zanetti was the first person to use the term “apologist for rapists” in connection with the LCA 2011 discussion . So we have a count of at least three men already.
These same patterns always happen so making a comment in support makes a difference. It doesn’t have to be insightful, long, or well written, merely “I agree” and a link to a web page will help. Note that a blog post is much better than a comment in this regard, comments are much like conversation while a blog post is a stronger commitment to a position.
I don’t believe that the majority is necessarily correct. But an opinion which is supported by too small a minority isn’t going to be considered much by most people.The Cost of Commenting
The Internet is a hostile environment, when you comment on a contentious issue there will be people who demonstrate their disagreement in uncivilised and even criminal ways. S. E. Smith wrote an informative post for Tiger Beatdown about the terrorism that feminist bloggers face . I believe that men face fewer threats than women when they write about such things and the threats are less credible. I don’t believe that any of the men who have threatened me have the ability to carry out their threats but I expect that many women who receive such threats will consider them to be credible.
The difference in the frequency and nature of the terrorism (and there is no other word for what S. E. Smith describes) experienced by men and women gives a vastly different cost to commenting. So when men fail to address issues related to the behavior of other men that isn’t helping women in any way. It’s imposing a significant cost on women for covering issues which could be addressed by men for minimal cost.
It’s interesting to note that there are men who consider themselves to be brave because they write things which will cause women to criticise them or even accuse them of misogyny. I think that the women who write about such issues even though they will receive threats of significant violence are the brave ones.Not Being Patronising
Craig raises the issue of not being patronising, which is of course very important. I think that the first thing to do to avoid being perceived as patronising in a blog post is to cite adequate references. I’ve spent a lot of time reading what women have written about such issues and cited the articles that seem most useful in describing the issues. I’m sure that some women will disagree with my choice of references and some will disagree with some of my conclusions, but I think that most women will appreciate that I read what women write (it seems that most men don’t).
It seems to me that a significant part of feminism is about women not having men tell them what to do. So when men offer advice on how to go about feminist advocacy it’s likely to be taken badly. It’s not just that women don’t want advice from men, but that advice from men is usually wrong. There are patterns in communication which mean that the effective strategies for women communicating with men are different from the effective strategies for men communicating with men (see my previous section on men not listening to women). Also there’s a common trend of men offering simplistic advice on how to solve problems, one thing to keep in mind is that any problem which affects many people and is easy to solve has probably been solved a long time ago.
Often when social issues are discussed there is some background in the life experience of the people involved. For example Rookie Mag has an article about the street harassment women face which includes many disturbing anecdotes (some of which concern primary school students) . Obviously anyone who has lived through that sort of thing (which means most women) will instinctively understand some issues related to threatening sexual behavior that I can’t easily understand even when I spend some time considering the matter. So there will be things which don’t immediately appear to be serious problems to me but which are interpreted very differently by women. The non-patronising approach to such things is to accept the concerns women express as legitimate, to try to understand them, and not to argue about it. For example the issue that Valerie recently raised wasn’t something that seemed significant when I first read the email in question, but I carefully considered it when I saw her posts explaining the issue and what she wrote makes sense to me.
I don’t think it’s possible for a man to make a useful comment on any issue related to the treatment of women without consulting multiple women first. I suggest a pre-requisite for any man who wants to write any sort of long article about the treatment of women is to have conversations with multiple women who have relevant knowledge. I’ve had some long discussions with more than a few women who are involved with the FOSS community. This has given me a reasonable understanding of some of the issues (I won’t claim to be any sort of expert). I think that if you just go and imagine things about a group of people who have a significantly different life-experience then you will be wrong in many ways and often offensively wrong. Just reading isn’t enough, you need to have conversations with multiple people so that they can point out the things you don’t understand.
This isn’t any sort of comprehensive list of ways to avoid being patronising, but it’s a few things which seem like common mistakes.
Anne Onne wrote a detailed post advising men who want to comment on feminist blogs etc , most of it applies to any situation where men comment on women’s issues.
-  http://etbe.coker.com.au/2011/02/10/lca2011-harassment/
-  http://adainitiative.org/2012/10/open-source-software-open-to-all/
-  http://blog.taz.net.au/2012/11/18/done/
-  http://geekfeminism.wikia.com/wiki/Women_Don’t_Ask
-  http://geekfeminism.org/2012/09/17/how-to-exclude-women-without-really-trying/
-  http://geekfeminism.wikia.com/wiki/Allies
-  http://www.alternet.org/story/82222/men_explain_things_to_me
-  http://geekfeminism.wikia.com/wiki/Splaining
-  http://etbe.coker.com.au/2012/11/08/rape-apology-lca2011/
-  http://www.codon.org.uk/~mjg59/ted_mail/0013.html
-  http://tigerbeatdown.com/2011/10/11/on-blogging-threats-and-silence/
-  http://rookiemag.com/2012/05/it-happens-all-the-time/
-  http://www.thefword.org.uk/features/2008/06/men_feminism_ne
- A Lack of Understanding of Nuclear Issues Ben Fowler writes about the issues related to nuclear power...
I have been keeping very busy on this blog the past few days with some very exciting updates! Everything is now in place for my Google Summer of Code project! Elektra now has the ability to merge KeySets, ucf has been patched to allow custom merge commands, we included a new elektra-merge script to use in conjunction with the new ucf command, and we wrote a great tutorial on how to use Elektra to merge configuration files in any package using ucf. There are few things I have missed updating you all on or are still wrapping up.
First of all, I would like to address the Debian packages. While Elektra includes everything needed for my Google Summer of Code Project, it must be built from source right now. Unfortunately, with the rapid development Elektra has seen the past few months, we did not pay enough attention to the Debian packages and they became dusty and riddles with bugs. Fortunately, we have a solution and his name is Pino Toscano. Pino has, very graciously, agreed to help us fix our Debian packages back into shape. If you wish to see the current progress of the packages, you can check his repo. Pino has already made fantastic progress creating the Debian packages. I will post here when packages are all fixed up and the latest versions of Elektra are into the Debian repo.
Some great news that we just received is that Elektra 0.8.7 has just been accepted into Debian unstable! This is huge progress for our team and means that users can now download our packages directly from Debian’s unstable repo and test out these new features. Obviously, the normal caveats apply for any packages in unstable but this is still an amazing piece of news and I would like to thank Pino again for all the support he has provided.
Another worthy piece of news that I unfortunately haven’t had time to cover is that thanks to Felix Berlakovich Elektra has a new plug-in called ini. In case you couldn’t guess, this new plug-in is used to mount ini files and the good news is that it is a vast improvement on our older simpleini plug-in. While it is still a work in progress, this new plug-in is much more powerful than simpleini. The new plug-in supports section and comments and works with many more ini files than the old plug-in. You may have noticed that I used this new plug-in to mount smb.conf in my technical demo of Samba using Elektra from configuration merging. Since smb.conf follows the same syntax as an ini file this new plug-in works great for mounting this file into the Elektra Key Database.
I hope my blogs so far have been very informative and helpful.
Ian S. Donnelly
Updated to version 3.2.2 of Eigen
Rcpp::as() now supports the conversion from R vector to “row array”, i.e., Eigen::Array<T, 1, Eigen::Dynamic>
Rcpp::as() now supports the conversion from dgRMatrix (row oriented sparse matrices, defined in Matrix package) to Eigen::MappedSparseMatrix<T, Eigen::RowMajor>
Conversion from R matrix to Eigen::MatrixXd and Eigen::ArrayXXd using Rcpp::as() no longer gives compilation errors
Is your Content Management System (CMS) really working for you? In this eBook, Solutions Architect Paul McKibben takes a lighthearted look at common issues you might find in an aging CMS:
We had a pretty good turnout here at Pantheon HQ for a Headless Drupal themed SF Drupal User’s Group:August 19, 2014
The excitement is clear. So what's next?Put Your Code Where Your Mouth Is
In any place where there's much excitement, there also tends to be a lot of discussion about what could or should be done. My suggestion is that we focus on creating real-world implementations with various JS frameworks and other API consuming clients.
To that end, I'm putting my code where my mouth is. Here's a working demo site:
I've placed the rough demo code online at GitHub, so you can set up your own. Feel free to experiment with different implementations, or help out with the listed TODOs. Pull requests are welcome!Next Steps
My ultimate vision is that we have a commonly accepted goal for demonstration implementations, and we find people who want to work on them for a number of popular JS MVC frameworks. It would also be interesting to have people do straight API implementations in raw Python or Ruby, or even just using Curl.
The end result would be something similar to TodoMVC, which helps people evaluate and embrace different JS frameworks. Having a repository of implementations would definitely speed up the process of attracting front-end developers to think about Drupal as a back-end for future projects.
At the same time, by focusing on practical/working implementations, we can much more effectively provide input and guidance to the continued core (and contrib) development. There's no need for Drupal engineers to build Headless in a vacuum.
Stay tuned for more updates as we see the outcomes of the community response. Based on what I've seen online and at the SF DUG meetup, I think some really exciting energy will come out of this.Blog Categories: Engineering Tweet
Drupal has included basic site search functionality since its first public release. Search administration was added in Drupal 2.0.0 in 2001, and search quality, relevance, and customization was improved dramatically throughout the Drupal 4.x series, especially in Drupal 4.7.0. Drupal's built-in search provides decent database-backed search, but offers a minimal set of features, and slows down dramatically as the size of a Drupal site grows beyond thousands of nodes.
In the mid-2000s, when most custom search solutions were relatively niche products, and the Google Search Appliance dominated the field of large-scale custom search, Yonik Seeley started working on Solr for CNet Networks. Solr was designed to work with Lucene, and offered fast indexing, extremely fast search, and as time went on, other helpful features like distributed search and geospatial search. Once the project was open-sourced and released under the Apache Software Foundation's umbrella in 2006, the search engine became one of the most popular engines for customized and more performant site search.
As an aside, I am writing this series of blog posts from the perspective of a Drupal developer who has worked with large-scale, highly customized Solr search for Mercy (example), and with a variety of small-to-medium sites who are using Hosted Apache Solr, a service I've been running as part of Midwestern Mac since early 2011.Timeline of Apache Solr and Drupal Solr Integration
If you can't view the timeline, please click through and read this article on Midwestern Mac's website directly.A brief history of Apache Solr Search and Search API Solr
Only two years after Apache Solr was released, the first module that integrated Solr with Drupal, Apache Solr Search, was created. Originally, the module was written for Drupal 5.x, but it has been actively maintained for many years and was ported to Drupal 6 and 7, with some relatively major rewrites and modifications to keep the module up to date, easy to use, and integrated with all of Apache Solr's new features over time. As Solr gained popularity, many Drupal sites started switching from using core search or the Views module to using Apache Solr.
As part of a client project we recently had the requirement of displaying an image field's default image in the node add form, before the user had uploaded a picture. This seemed simple enough to implement but thinking about it I realized this was quite a generic feature request and that such a module might prove useful for others. Rather by accident I even noticed that Drupal.org itself employs the same feature on user profile pages.
Thus, I hereby present the Image widget default image module, which does just that: If you have provided a default image for an image field and the user has not yet uploaded an image for that field, it displays that default image as a preview. Not many bells or whistle, but it works. (I can claim that, because it comes with tests. ) If it does not work for you or if you have any questions please leave a comment or open an issue on Drupal.org.Weitere Bilder:
In part 2 of the multiple views series you will learn how to add the jQuery needed to switch between multiple classes. By having the ability to use multiple classes, we will (in part 3) be able to use CSS to change the look and feel of the same view.
Over 30 people attended this wildly successful training at DrupalCon Austin. Now is your chance to attend this training at DrupalCon Amsterdam!
In this course, we cut past the evangelism that exists around Agile, and instead focus on real-world practical training that you can put into action.
The course is delivered using the Agile Scrum techniques it teaches. At the start, delegates see the backlog of requirements that the product owner (a trainer) has developed for the course, and the prioritization of those requirements. The course then progresses in one-hour periods of work called sprints, working through training modules from the top of the backlog.
Part way through the morning, delegates are ready to take over as the product owners. They will take responsibility for specifying the requirements for the course, based on the needs and interests of the delegates in the room, re-prioritise them, and even add completely new requirements. Our trainers will respond to these changes by creating new training modules on the fly based on real project experience, to provide the highest possible value to the delegates.
Through this approach we demonstrate and explain the processes of Agile many times, and we also demonstrate their value, and delegates leave with a real insight into how they could apply agile, and handle some of the challenges they may have faced.
The trainers are highly experienced Agile coaches, who mentor teams at Wunderkraut (known as WunderRoot in the UK), as well as consulting with large clients about ensuring successful delivery of their projects.Meet the Trainers from Wunderkraut
This training will be held on Monday, 29 September from 09:00-17:00 at the Amsterdam RAI during DrupalCon Amsterdam. The cost of attending this training is €400 and includes training materials, meals and coffee breaks. A DrupalCon ticket is not required to register to attend this event.
Our training courses are designed to be small enough to provide attendees plenty of one-on-one time with the instructor, but large enough that they are a good use of the instructor's time. Each training course must meet its minimum sign-up number by 5 September in order for the course to take place. You can help to ensure your training course takes place by registering before this date and asking friends and colleagues to attend.
Recently I've been getting annoyed with the Debian Administration website; too often it would be slower than it should be considering the resources behind it.
As a brief recap I have six nodes:
- 1 x MySQL Database - The only MySQL database I personally manage these days.
- 4 x Web Nodes.
- 1 x Misc server.
The misc server is designed to display events. There is a node.js listener which receives UDP messages and stores them in a rotating buffer. The messages might contain things like "User bob logged in", "Slaughter ran", etc. It's a neat hack which gives a good feeling of what is going on cluster-wide.
I need to rationalize that code - but there's a very simple predecessor posted on github for the curious.
Anyway enough diversions, the database is tuned, and "small". The misc server is almost entirely irrelevent, non-public, and not explicitly advertised.
So what do the web nodes run? Well they run a lot. Potentially.
Each web node has four services configured:
- Apache 2.x - All nodes.
- uCarp - All nodes.
- Pound - Master node.
- Varnish - Master node.
Apache runs the main site, listening on *:8080.
One of the nodes will be special and will claim a virtual IP provided via ucarp. The virtual IP is actually the end-point visitors hit, meaning we have:Master HostOther hosts
Pound is configured to listen on the virtual IP and perform SSL termination. That means that incoming requests get proxied from "vip:443 -> vip:80". Varnish listens on "vip:80" and proxies to the back-end apache instances.
The end result should be high availability. In the typical case all four servers are alive, and all is well.
If one server dies, and it is not the master, then it will simply be dropped as a valid back-end. If a single server dies and it is the master then a new one will appear, thanks to the magic of ucarp, and the remaining three will be used as expected.
I'm sure there is a pathological case when all four hosts die, and at that point the site will be down, but that's something that should be atypical.
Yes, I am prone to over-engineering. The site doesn't have any availability requirements that justify this setup, but it is good to experiment and learn things.
So, with this setup in mind, with incoming requests (on average) being divided at random onto one of four hosts, why is the damn thing so slow?
We'll come back to that in the next post.
(Good news though; I fixed it ;)
Yesterday, I spent most of the day finishing up the multiarch work I'd been doing on introducing multiarch to the eID middleware, and did another release of the Linux builds. As such, it's now possible to install 32-bit versions of the eID middleware on a 64-bit Linux distribution. For more details, please see the announcement.
Learning how to do multiarch (or biarch, as the case may be) for three different distribution families has been a, well, learning experience. Being a Debian Developer, figuring out the technical details for doing this on Debian and its derivatives wasn't all that hard. You just make sure the libraries are installed to the multiarch-safe directories (i.e., /usr/lib/<gnu arch triplet>), you add some Multi-Arch: foreign or Multi-Arch: same headers where appropriate, and you're done. Of course the devil is in the details (define "where appropriate"), but all in all it's not that difficult and fairly deterministic.
The Fedora (and derivatives, like RHEL) approach to biarch is that 64-bit distributions install into /usr/lib64 and 32-bit distributions install into /usr/lib. This goes for any architecture family, not just the x86 family; the same method works on ppc and ppc64. However, since fedora doesn't do powerpc anymore, that part is a detail of little relevance.
Once that's done, yum has some heuristics whereby it will prefer native-architecture versions of binaries when asked, and may install both the native-architecture and foreign-architecture version of a particular library package at the same time. Since RPM already has support for installing multiple versions of the same package on the same system (a feature that was originally created, AIUI, to support the installation of multiple kernel versions), that's really all there is to it. It feels a bit fiddly and somewhat fragile, since there isn't really a spec and some parts seem fairly undefined, but all in all it seems to work well enough in practice.
The openSUSE approach is vastly different to the other two. Rather than installing the foreign-architecture packages natively, as in the Debian and Fedora approaches, openSUSE wants you to take the native foo.ix86.rpm package and convert that to a foo-32bit.x86_64.rpm package. The conversion process filters out non-unique files (only allows files to remain in the package if they are in library directories, IIUC), and copes with the lack of license files in /usr/share/doc by adding a dependency header on the native package. While the approach works, it feels like unnecessary extra work and bandwidth to me, and obviously also wouldn't scale beyond biarch.
It also isn't documented very well; when I went to openSUSE IRC channels and started asking questions, the reply was something along the lines of "hand this configuration file to your OBS instance". When I told them I wasn't actually using OBS and had no plans of migrating to it (because my current setup is complex enough as it is, and replacing it would be far too much work for too little gain), it suddenly got eerily quiet.
Eventually I found out that the part of OBS which does the actual build is a separate codebase, and integrating just that part into my existing build system was not that hard to do, even though it doesn't come with a specfile or RPM package and wants to install files into /usr/bin and /usr/lib. With all that and some more weirdness I've found in the past few months that I've been building packages for openSUSE I now have... Ideas(TM) about how openSUSE does things. That's for another time, though.
(disclaimer: there's a reason why I'm posting this on my personal blog and not on an official website... don't take this as an official statement of any sort!)
I still consider myself a newbie teacher. I'm just starting my fourth semester. And yes, I really enjoy it.
Now, how did I come to teaching? Well, my training has been mostly on stages for different conferences. More technical, more social, whatever — I have been giving ~10 talks a year for ~15 years, and I must have learnt something from that.
Some good things, some bad habits.
When giving presentations, a most usual technique is to prepare a set of slides to follow/support the ideas. And yes, that's what I did for my classes: Since my first semester, I prepared a nice set of slides, thematically split in 17 files, with ~30 to ~110 pages each (yes, huge variation). Given the course spans 32 classes (72 hours, 2¼ hours per class), each slide lasts for about two classes.
But, yes, this tends to make the class much less dynamic, much more scripted, rigid, and... Boring. From my feedback, I understand the students don't think I am a bad teacher, but still, I want to improve!
So, today I was to give the introduction to memory management. Easy topic, with few diagrams and numbers, mostly talking about the intuitive parts of a set of functions. I started scribbling and shortening the main points on a piece of paper (yes, the one on the picture). I am sure I can get down to more reduction — But this does feel like an improvement!
The class was quite successful. I didn't present the 100% of the material (which is one of the reasons I cling to my presentations — I don't want to skip important material), and at some point I do feel I was a bit going in circles. However, Operating Systems is a very intuitive subject, and getting the students to sketch by themselves the answers that describe the working of real operating systems was a very pleasant experience!
Of course, when I use my slides I do try to make it as interactive and collaborative as possible. But it is often unfeasible when I'm following a script. Today I was able to go around with the group's questions, find my way back to the outline I prepared.
I don't think I'll completely abandon my slides, specially for some subjects which include many diagrams or pictures. But I'll try to have this alternative closer to my mind.
Today I wanted to talk a bit about our technical demo. We patched a version of Samba to use our elektra-merge script in order to handle it’s configuration file smb.conf. Using the steps from my previous tutorial, we patched Samba to use this new technique of config merging. This patched version of Samba mounts it’s configuration to system/samba/smb in the Elektra Key Database. Then, during package upgrades, it uses the new --threeway-merge-command command with elektra-merge as the specified command. The result is automatic handling of smb.conf that is conffile-like (thanks ucf!) and the ability to have a powerful, automatic, three-way merge solution on package upgrades.
The main thing I would like to discuss is how this project is an improvement upon the current implementation of three-way merges in ucf. Before this project, ucf could attempt three-way merges on files it controlled using the diff3 tool. The main limitation of tools like diff3 are that they are line based and don’t inherently understand the files they are dealing with. Elektra on the other hand allows for a powerful system of backends which use plug-ins to understand configuration files. Elektra doesn’t store configuration data on a line-by-line basis, but in a more abstract way that is tailored to each configuration file using backends. smb.conf is a great example of this because it uses the syntax of an ini file so Elektra can mount it in a way that is intuitive to an ini file. Since data is stored in a format of key=data within ini files, Elektra stores this data in a similar way. For each key in smb.conf store a Key in Elektra with the value of that key store in a string. Then, during a merge, we can compare Keys in each version of smb.conf and easily see which ones changed and how they need to be merged into the result. On the other hand, diff3 has no concept of ini files or keys, it just compares the different versions line by line which results in many more conflicts than using elektra-merge. Moreover, a traditional weakness of diff is moving lines or data around. While diff3 does a very good job at handling this, it’s not perfect. In Elektra, Keys are named in an intelligent way based on their backend, so for smb.conf the line workgroup = HOME would always be saved under system/samba/smb/workgroup. It doesn’t matter if the lines are changed between versions because Elektra just has to check for the Key and its value.
My favorite example is a shortcoming in the diff3 algorithm (at least in my opinion). If something is changed to the same value in ours and theirs, but they differ from base, diff3 reports a conflict. On the other hand elektra-merge can easily handle this problem. A simple example of this would be changing the max log size value in Samba. Here is that line in each version of smb.conf:
max log size = 1000
max log size = 2000
max log size = 2000
Obviously, in the merged version, result, one would expect this line to be:
max log size = 2000
Let’s check the result from elektra-merge:
max log size = 2000
Great! How about diff3:
max log size = 1000
max log size = 2000
Whoops! As I mentioned the diff3 algorithm can’t handle this type of change, it just results in a conflict. Note that smb.conf.base is just representative of the file used as base and that smb.conf.theirs is representative of the file used as theirs. The file names were changed for the sake of clarity.
There are many other examples of the benefits to storing configuration data in a database of Keys that can better conform to actual data as opposed to storing configuration parameters in files where they can only be compared on a line to line basis. With the help of storage plug-ins, Elektra can ‘understand’ the configurations stored in it’s Key Database. Since we store the data in a way that makes sense for configuration data, we can easily merge actual configuration data as opposed to just lines in a file. A good example of this is in-line comments. Many of our storage plug-ins understand the difference between comments and actual configuration data. So if a configuration file has an inline comment like so:
max log size = 10000 ; Controls the size of the log file (in KiB)
we can compare the actual Keys, value pairs between versions max log size = 10000 and deal with the comments separately.
As a result, if we have a base:
max log size = 1000 ; Size in KiB
max log size = 10000 ; Size in KiB
max log size = 1000 ; Controls the size of the log file (in KiB)
The result using elektra-merge would be:
max log size = 10000 ; Controls the size of the log file (in KiB)
Obviously, this line would cause a conflict on any line-based merge algorithm such as diff3 or git. It is worth noting that the ability of elektra-merge is directly related to the quality of the storage plug-ins that Elektra relies on. elektra-merge only looks at the name, value, and any metadata affiliated with each key. As a result, using the line plug-in would result in a merge only as powerful as any other line-based merge. Yet by using the ini plug-in on an ini file we get a much more advanced merge like the one described above.
As you can tell, this new method offers clear advantages to the traditional method of using diff3 to handle configuration merges. Also, I hope this demo shows how easy it is to implement these great features into your Debian packages. The community can only benefit if maintainers take advantage of these great features. I am glad to say that my Google Summer of Code Project has been a success, even if we had to do a little change of plans. The ucf integration ended up working great and is really easy for maintainers to implement. Hope you enjoyed this demo and better understand the power of using Elektra.
Ian S. Donnelly
This year’s Twin Cities DrupalCamp had no shortage of new faces, quality sessions, trainings, and after parties. Most of my time was spent in frontend sessions and talking with folks. Being that I live in Minneapolis, this camp is especially rewarding from a hometown Drupal represent kind of perspective. Below are some of my favorite sessions and camp highlights.Community Drupal
As you can see it comes in a nice packaging with a world-compatible power adapter. It uses a Ingenic JZ4780 SoC with a dual core MIPS32 CPU running at 1.2GHz with a PowerVR SGX540 GPU. The board is fitted with 1GB of RAM, 8GB of NOR flash, HDMI output, USB 2.0 ports, Ethernet + Wi-Fi + BlueTooth, SD card slot, IR receiver, expansion headers and more. The schematics are available. The Linux kernel and the U-Boot bootloader sources are also available.
Powering this board with a USB keyboard, a USB mouse and a HDMI display, it boots off the internal flash on a Debian Wheezy up to the XFCE environment. Besides the kernel, the Wi-Fi + Bluetooth firmware, and very few configuration changes, it runs a vanilla Debian. Unfortunately I haven’t found time to play more with it yet, but it looks already quite promising.
The board has not been formally announced yet, so I do not know when it will become available, nor the price, but if you are interested I’ll bring it to DebConf14. Don’t hesitate to ask me if you want to look at it or play with it.
More and more often I am asked, when putting together a design Drupal for a website, what is the importance of designing above the fold and whether or not that today’s users will scroll to read content.
In my experience as an organization leader one of the most important tools in my toolbox has always been my personal values. It's been my experience that even when data points in one direction, and best practices say you should approach the problem in this way, it's always been my values that help me make the best decisions as an individual. And the truth is, there are very rarely any decisions that are 100% clear cut. In almost every decision, some amount of judegment is required.
That's why I believe so strongly in defining values for the organizations I work for. When everyone is working from a shared sense of values, we're making decisions - even big giant judgement calls - from the same perspective. To that end, we spent some time this year working with the board and staff to develop a values statement for the Drupal Association.
We started in a board retreat, brainstorming the implicit (though not documented) values of the Association and the larger Drupal Community. The board ranked their favorites, and then we created a committee of board and staff to draft some language. Those initial values statements were vetted by both the entire Association staff and the full board, and then additional edits were made. Here's the result of that process:
The Drupal Association shares the values of our community, our staff, and open source projects:
- TEAMWORK: We add value to the Drupal community by helping each other solve problems to create quality human and digital experiences.
- COMMUNICATION: We value communication. We seek community participation. We are open and transparent.
- ACTION: We act decisively and proactively, embracing what we learn from both our successes and our mistakes.
- RESPECT: We respect and value inclusivity in our global community and strive to recognize, understand, and respond to its needs.
- FUN: We create environments that embrace humor resulting in fun, positive, supportive and safe interactions.
To be clear, these are the values we're defining for our staff. We're not trying to impose these values on the larger community. However, we do hope they reflect the values you feel are important in the larger Drupal community as well. We also want to recognize that writing down the words is one thing, and living up to them is something else. We intend to live these values in all our work.
Now it's your turn. The values are set by the board and staff, but we want to make sure we know what you think.
Flickr photo: Howard Lake
This is a preview of Nuvole's training at DrupalCon Amsterdam: An Effective Development Workflow in Drupal 8.
One of the new key features of Drupal 8 is the possibility to deal with configuration in code. Since configuration is now in text files, we want to put it under version control in Git to enjoy the many advantages this brings: comparing configuration states, keeping a history of configuration changes and even moving configuration between sites.Setup
We will assume that you have a development version of Drupal 8, git and drush available on your system.
You can set up your Drupal git repository in several ways. One of them is outlined in Building a Drupal site with Git on drupal.org. The document is written for Drupal 7, but can easily be adapted for Drupal 8.
Another, probably simpler method is to simply download a Drupal 8 (alpha) release and initialise a new repository with it.
In either case you should copy example.gitignore to .gitignore and adapt it to your needs to prevent settings.php and the files directory from being versioned.
The next step is to make sure that a configuration directory is versionable. By default Drupal 8 will place the staging directory under sites/default/files and it is considered a good practice to not version that location, but an alternative location can easily be specified in settings.php:<?php
$config_directories['staging'] = 'config/staging';
It is also possible and even advisable to specify a directory outside of the web root of course. In that case you would put the parent directory of your web root where drupal is under version control and use ../config/staging. We will later see that it is also possible to add more directories and keys to the $config_directories variable.
Because the configuration management of Drupal 8 only works between different instances of the same site, the different instances of the site need to be cloned. Cloning a Drupal 8 site is done the same way as cloning a Drupal 7 site. Just dump the database of the site to clone and import it in the other environment.Development
After cloning your site you can go ahead and start configuring your site.
Once the part of the configuration you were working on is done the whole configuration of the site needs to be exported.
The current contents of your export directory (config/staging) will be deleted. (y/n): y
Configuration successfully exported to config/staging.
Next, you need to merge the work of other developers. In some cases it may be enough to simply use git pull, otherwise the configuration has to be merged after it has been committed:
Add all configuration to git and commit it.
Use git pull (or git fetch and git merge) and resolve any conflicts if necessary.
Git can merge changes in text files quite well, but git does not know about Drupal and its yaml format for configuration. It is, therefore, important to verify that the merged configuration makes sense and is valid. In most cases it will probably not be an issue and just work, but it is always better to be vigilant and be on the safe side. So, after merging, you should always run:local$ drush config-import staging
If the import went smooth you can push the configuration to the remote repository. Otherwise the configuration needs to be fixed first.Deployment
The simplest case is when the configuration on the production site has not been changed. There is an interesting Configuration Read-only mode module that can enforce this.
If the configuration did not change deploying the new configuration is simply:remote$ git pull
remote$ drush config-import staging
If the configuration changes on the production site, it is best to frequently export the live configuration into a dedicated directory.
Add a new config directory in settings.php:
$config_directories['to_dev'] = 'config/to_dev';
?> remote$ drush config-export to_dev -y
Add, commit and push it to the production branch so that the developers can deal with it and integrate the changes into the configuration which will be deployed next. Exporting the configuration into a dedicated directory rather than the staging directory avoids the danger that merge conflicts happen on the production site. The deployment to the production site should be kept hassle free, so it should always be safe to pull from git and import the configuration without the risk of a conflict.Important notes
It is important to first export the configuration changes and then pull changes from collaborators because the exporting action wipes the directory and re-populates it with the active configuration. Since everything is in git, you can recover from such a mistake without much difficulty but why make your life complicated.
Import the configuration before pushing it to the remote repository. Broken configuration breaks the site, be a nice co-worker.
Git doesn't solve everything! Imagine Alice and Bob start with the same site, it has one content type and among others an "attachment" field. Alice deletes the attachment field, exports the configuration and pushes it to git. In the meantime, Bob creates a new content type and adds the attachment field to it. Bob exports his configuration, merges Alice's configuration changes without a problem (the changes are separate files) and imports the merged configuration. The attentive reader sees where this leads. The commit of Alice deletes the field storage for the attachment field, but Bob added a field instance which depends on the field storage.
The exported configuration now contains a field instance that can't be imported.
At the time of writing, drush will signal a successful import but doesn't actually import it while the UI is more helpful and complains that the attachment field instance was not imported due to the missing field storage.
We’ve been busy setting up a Virtual Machine (VM) image to be used by participants of a MOOC that’s opening in early september on Relational Databases at Telecom SudParis.
We’ve chosen to use Vagrant and VirtualBox which are used to build, distribute and run the box, providing scriptability (reproducibility) and making it portable on most operating systems.
The VM itself contains a Debian (jessie) minimal system which runs (in the background) PostgreSQL, Apache + mod_php, phpPgAdmin, and a few applications of our own to play with example databases already populated in PostgreSQL.
As the MOOC’s language will be french, we expect the box to be used mostly on machines with azerty keyboards. This and other context elements led us to add some customizations (locale, APT mirror) in provisioning scripts run during the box creation.
At the moment, we generate 2 variants of the box, one for 32 bits kernel (i686) and one for 64 bits kernel (amd64) which (once compressed) represent betw. 300 and 350 Mb.
The resulting boxes are uploaded to a self-hosting site, and distributed through vagrantcloud. Once the VM are created in VirtualBox, the typical VMDK drives file is around 1.3Gb.
We use our own Debian base boxes containing a minimal Debian jessie/testing, instead of relying on someone else’s, and recreate them using (the development branch version of) bootsrap-vz. This ensure we can put more trust in the content as it’s a native Debian package installation without MITM intervention.
The VM are meant to be run headless for the moment, keeping their size to the minimum, even though we also provide a script to install and configure a desktop environment based on XFCE4.
The applications are either used through vagrant ssh, for instance for SQL command-line in psql, or in the Web browser, for our own Web based SQL exerciser, or phpPgAdmin (see a demo screencast (in french, w/ english subtitles)), which can then be used even off-line by the participants, which also means this requires no servers availability for our IT staff.
The MOOC includes a section on PHP + SQL programming, whose exercises can be performed using a shared sub-folder of /vagrant/ which allows editing on the host with the favourite native editor/IDE, while running PHP inside the VM’s Apache + mod_php.
The sources of our environment are available as free software, if you’re interested to replicate a similar environment for another project.
As we’re still polishing the environment before the MOOC opening (on september 10th), I’m not mentioning the box URLs but they shouldn’t be too hard to find if you’re investigating (refering to the fusionforge project’s web site).
We don’t know yet how suitable this environment will be for learning SQL and database design and programming, and if Vagrant will bring more difficulties than benefits. Still we hope that the participants will find this practical, allowing them to work on the lab / exercises whenever and wherever they chose, removing the pain of installing and configuring a RDBMS on their machines, or the need to be connected to a cloud or to our overloaded servers. Of course, one limitation will be the requirements on the host machines, that will need to be reasonably modern, in order to run a virtualized Linux system. Another is access to high bandwidth for downloading the boxes, but this is kind of a requirement already for downloading/watching the videos of the MOOC classes
Big thanks go to our intern Stéphane Germain, who joined us this summer to work on this virtualized environment.
Part 1 of a 2-part conversation with Angie Byron in front of the cameras at NYC Camp 2014. In this part of our conversation we go over some of the inspiring and thought-provoking ideas we encountered there, and then jump to some of the benefits to users of the technical improvements built into Drupal 8.