Agrégateur de flux
Drupalpress, Drupal in the Health Sciences Library at UVA: two new drupal distros – one for voting, one for 3d printing e-commerce
Two new drupal distributions available on github
** https://github.com/alibama/rapid-prototyping-ecommerce-drupal – this is the code behind http://rpl.mae.virginia.edu/ it’s an e-commerce solution for 3d printing… A lot of this is implemented in rules and other well-standardized code thanks to Joe Pontani - a talented developer here in Virginia. Joe integrated several third party tools, and set up the UVa payment gateway through Nelnet.
Both sites are getting updates over the next few months – the Charlottesville Council website also has a drupalgap implementation on it – absolutely awesome toolset…
18F API compliance is another feature I’m pretty stoked about… I got most of that done with the oauth2 server, views datasource, services and a couple of great notification features done with rules + views i’ll get that feature out asap = it’s really convenient – matching a profile2 taxonomy field onto content taxonomy fields for notifications with new content.
any questions – please drop a line in the comments below
It is always fun to write new stuff, and be able to show off that shiny new piece of code that just come out of your brilliance and/or restless effort. But the world does not spin based just on shiny things; for free software to continue making the world work, we also need the dusty, and maybe and little rusty, things that keep our systems together. Someone needs to make sure the rust does not take over, and that these venerable but useful pieces of code keep it together as the ecosystem around them evolves. As you know, Someone is probably the busiest person there is, so often you will have to take Someone’s job for yourself.
rmail is a Ruby library able to parse, modify, and generate MIME mail messages. While handling transitions of Ruby interpreters in Debian, it was one of the packages we always had to fix for new Ruby versions, to the point where the Debian package has accumulated quite a few patches. The situation became ridiculous.
Since doing this type of port is always painful, I decided instead to do something about the sorry state in which rmail was on the upstream side.
The reasons why it was not properly maintained upstream does not matter: people lose interest, move on to other projects, are not active users anymore; that is normal in free software projects, and instead of blaming upstream maintainers in any way we need to thank them for writing us free software in the first place, and step up to fix the stuff we use.
I got in touch with the people listed as owner for the package on rubygems.org, and got owner permission, which means I can now publish new versions myself.
With that, I cloned the repository where the original author had imported the latest code uploaded to rubygems and had started to receive contributions, but that repository was inactive for more than one year. It had already got some contributions from the sup developers which never made it in a new rmail release, so the sup people started using their own fork called “rmail-sup”.
Already in my repository, I have imported all the patches that still made sense from the Debian repository, did a bunch of updates, mainly to modernize the build system, and did a 1.1.0 release to rubygems.org. This release is pretty much compatible with 1.0.0, but since I did not test it with Ruby versions older than than one in my work laptop (2.1.5), I bumped the minor version number as warning to prospective users still on older Ruby versions.
In this release, the test suite passes 100% clean, what always gives my mind a lot of comfort:$ rake /usr/bin/ruby2.1 -I"lib:." -I"/usr/lib/ruby/vendor_ruby" "/usr/lib/ruby/vendor_ruby/rake/rake_test_loader.rb" "test/test*.rb" Loaded suite /usr/lib/ruby/vendor_ruby/rake/rake_test_loader Started ............................................................................... ............................................................................... ........ Finished in 2.096916712 seconds. 166 tests, 24213 assertions, 0 failures, 0 errors, 0 pendings, 0 omissions, 0 notifications 100% passed 79.16 tests/s, 11546.95 assertions/s
And in the new release I have just uploaded to the Debian experimental suite (1.1.0-1), I was able to drop all of the patches and just use the upstream source as is.
So that’s it: if you use rmail for anything, consider testing version 1.1.0-1 from Debian experimental, or 1.1.0 from rubygems.org if you into that, and report any bugs to the [github repository](https://github.com/terceiro/rmail). My only commitment for now is keep it working, but if you want to add new features I will definitively review and merge them.
I have been working on refactoring stuff for the past two or three weeks. At first, I found it much more exciting and interesting than merely fixing minor bugs. So I worked on most parts of the codebase, and made lots of changes.
It took me much more time and energy to do all this stuff. But sadly, the more energy results in a more broken codebase. It seems all these efforts are a waste since the code is no longer be able to be merged. It was very frustrating.Get back on track
Well, I guess such trials and failures are just inevitable on the way towards an experienced developer. Despite all the divergences and dismay I have gone through, my internship must be back on track.
I am now more realistic. My first and foremost task is to GET THINGS DONE by focusing on small and doable changes. Although it seems the improvement is LESS, but it means MORE to have a not-so-perfect finished-product (in my view), rather than a to-be-perfect messy.Journey continues.
I haven't blogged for a long time, but I've decided that I'm going to try to write again, at least about technical stuff.
My plan was to blog about the projects I've been working on lately, the main one being the setup of the latest version of Kolab with the systems we already have at work, but I'll do that on the next days.
Today I'm just going to make a list of the tools I use on a daily basis and my plans to start using additional ones in the near future.Shells, Terminals and Text Editors
For text editing I've been using Vim for a long time (even on Mobile devices) and while I'm aware I don't know half of the things it can do, what I know is good enough for my day to day needs.
In the past I also used Emacs as a programming editor and my main tool to write HTML, SGML and XML, but since I haven't really needed an IDE for a long time and I mainly use Lightweight Markup Languages I haven't used it for a long time (I briefly tried to use Org mode, but for some reason I ended up leaving it).Documentation formats and tools
Since a long time ago I've been an advocate of Lightweight Markup Languages; I started to use LaTeX and Lout, then moved to SGML/XML formats (LinuxDoc and DocBook) and finally moved to plain text based formats.
I started using Wiki formats (parsewiki) and soon moved to reStructuredText; I also use other markup languages like Markdown (for this blog, aka ikiwiki) and tried MultiMarkdown to replace reStructuredText for general use, but as I never liked Markdown syntax I didn't liked an extended version of it.
While I've been using ReStructuredText for a long time, I recently found Asciidoctor and the Asciidoc format and I guess I'll be using it instead of rst whenever I can (I still need to try the slide backends and conversions to ODT, but if that works I guess I'll write all my new documents using Asciidoc).Programming languages
I'm not a developer, but I read and patch a lot of free software code written on a lot of different programming languages (I wouldn't be able to write whole programs on most of them, but thanks to Stack Overflow I'm usually able to fix what I need).
For a long time I've been a Subversion user, at least for my own projects, but seems that everything has moved to git now and I finally started to use it (I even opened a github account) and plan to move all my personal subversion repositories at home and at work to git, including the move of all my debian packages from svn-buildpackage to git-buildpackage.Further Reading
With the previous plans in mind, I've started reading a couple of interesting books:
- Learn You a Haskell by Miran Lipovača (http://learnyouahaskell.com/)
- Pro Git written by Scott Chacon and Ben Straub (http://git-scm.com/book/en/v2)
Now I just need to get enough time to finish reading them ... ;)
Required or not required? To validate or not to validate? That is the question. So you've setup (the site builder's way, no custom forms) your required fields and custom validations for Node types, just to get this feedback from the customer:
That field we defined as mm..... as required (something trivial and not really critical such as an image file) is actually not always required. Users X and Y should be able to bypass that restriction.More articles...
- Drupal on IIS or Apache
- Bypassing Form Validations and Required Fields in Drupal: the BFV module.
- Node Comment and Forum working together to boost user participation
- Installing Drupal on Windows and SQL Server
- Setting up Code Syntax Higlighting with Drupal
- Getting #2,000 requests per second without varnish
- Distinct options in a views exposed filter: The Views Selective Filters Module
I just had the best Valentine's Day ever.
Ever worked at a company (or on a codebase, or whatever) where it seemed like, no matter what the question was, the answer was written down somewhere you could easily find it? Most people haven’t, sadly, but they do exist, and I can assure you that it is an absolute pleasure.
On the other hand, practically everyone has experienced completely undocumented systems and processes, where knowledge is shared by word-of-mouth, or lost every time someone quits.
Why are there so many more undocumented systems than documented ones out there, and how can we cause more well-documented systems to exist? The answer isn’t “people are lazy”, and the solution is simple – though not easy.Why Johnny Doesn’t Read
When someone needs to know something, they might go look for some documentation, or they might ask someone else or just guess wildly. The behaviour “look for documentation” is often reinforced negatively, by the result “documentation doesn’t exist”.
At the same time, the behaviours “ask someone” and “guess wildly” are positively reinforced, by the results “I get my question answered” and/or “at least I can get on with my work”. Over time, people optimise their behaviour by skipping the “look for documentation” step, and just go straight to asking other people (or guessing wildly).Why Johnny Doesn’t Write
When someone writes documentation, they’re hoping that people will read it and not have to ask them questions in order to be productive and do the right thing. Hence, the behaviour “write documentation” is negatively reinforced by the results “I still get asked questions”, and “nobody does things the right way around here, dammit!”
Worse, though, is that there is very little positive reinforcement for the author: when someone does read the docs, and thus doesn’t ask a question, the author almost certainly doesn’t know they dodged a bullet. Similarly, when someone does things the right way, it’s unlikely that anyone will notice. It’s only the mistakes that catch the attention.
Given that the experience of writing documentation tends to skew towards the negative, it’s not surprising that eventually, the time spent writing documentation is reallocated to other, more utility-producing activities.Death Spiral
The combination of these two situations is self-reinforcing. While a suitably motivated reader might start by strictly looking for documentation, or an author initially be enthused to always fully documenting their work, over time the “reflex” will be for readers to just go ask someone, because “there’s never any documentation!”, and for authors to not write documentation because “nobody bothers to read what I write anyway!”.
It is important to recognise that this iterative feedback loop is the “natural state” of the reader/author ecosystem, resulting in something akin to thermodynamic entropy. To avoid the system descending into chaos, energy needs to be constantly applied to keep the system in order.The Solution
Effective methods for avoiding the vicious circle can be derived from the things that cause it. Change the forces that apply themselves to readers and authors, and they will behave differently.
On the reader’s side, the most effective way to encourage people to read documentation is for it to consistently exist. This means that those in control of a project or system mustn’t consider something “done” until the documentation is in a good state. Patches shouldn’t be landed, and releases shouldn’t be made, unless the documentation is altered to match the functional changes being made. Yes, this requires discipline, which is just a form of energy application to prevent entropic decay.
Writing documentation should be an explicit and well-understood part of somebody’s job description. Whoever is responsible for documentation needs to be given the time to do it properly. Writing well takes time and mental energy, and that time needs to be factored into the plans. Never forget that skimping on documentation, like short-changing QA or customer support, is a false economy that will cost more in the long term than it saves in the short term.
Even if the documentation exists, though, some people are going to tend towards asking people rather than consulting the documentation. This isn’t a moral failing on their part, but only happens when they believe that asking someone is more beneficial to them than going to the documentation. To change the behaviour, you need to change the belief.
You could change the belief by increasing the “cost” of asking. You could fire (or hellban) anyone who ever asks a question that is answered in the documentation. But you shouldn’t. You could yell “RTFM!” at everyone who asks a question. Thankfully that’s one acronym that’s falling out of favour.
Alternately, you can reduce the “cost” of getting the answer from the documentation. Possibly the largest single productivity boost for programmers, for example, has been the existence of Google. Whatever your problem, there’s a pretty good chance that a search or two will find a solution. For your private documentation, you probably don’t have the power of Google available, but decent full-text search systems are available. Use them.
Finally, authors would benefit from more positive reinforcement. If you find good documentation, let the author know! It requires a lot of effort (comparatively) to look up an author’s contact details and send them a nice e-mail. The “like” button is a more low-energy way of achieving a similar outcome – you click the button, and the author gets a warm, fuzzy feeling. If your internal documentation system doesn’t have some way to “close the loop” and let readers easily give authors a bit of kudos, fix it so it does.
Heck, even if authors just know that a page they wrote was loaded N times in the past week, that’s better than the current situation, in which deafening silence persists, punctuated by the occasional plaintive cry of “Hey, do you know how to…?”.
Do you have any other ideas for how to encourage readers to read, and for authors to write?
From time to time, people ask me, with a bit of a disbelieving look on their face, “Tell me again why you chose to move to Kansas?” I can explain something about how people really care about their neighbors out here, how connections through time to a place are strong, how the people are hard-working, achieve great things, and would rather not talk about their achievements too much. But none of this really conveys it.
This week, as I got word that my great uncle Willis Goerzen passed away, it occured to me that the reason I live in Kansas is simple: people like Willis.
Willis was a man that, through and through, simply cared. For everyone. He had hugs ready anytime. When I used to see him in church every Sunday, I’d usually hear his loud voice saying, “Well John!” Then a hug, then, “How are you doing?” When I was going through a tough time in life, hugs from Willis and Thelma were deeply meaningful. I could see how deeply he cared in his moist eyes, the way he sought me out to offer words of comfort, reassurance, compassion, and strength.
Willis didn’t just defy the stereotypes on men having to hide their emotions; he also did so by being just gut-honest. Americans often ask, in sort of a greeting, “How are you?” and usually get an answer like “fine”. If I asked Willis “How are you?”, I might hear “great!” or “it’s hard” or “pretty terrible.” In a place where old-fashioned stoicism is still so common, this was so refreshing. Willis and I could have deep, heart-to-heart conversations or friendly ones.
Willis also loved to work. He worked on a farm, in construction, and then for many years doing plumbing and heating work. When he retired, he just kept on doing it. Not for the money, but because he wanted to. I remember calling him up one time about 10 years ago, asking if he was interested in helping me with a heating project. His response: “I’ll hitch up the horses and be right there!” (Of course, he had no horses anymore.) When I had a project to renovate what had been my grandpa’s farmhouse (that was Willis’s brother), he did all the plumbing work. He told me, “John, it’s great to be retired. I can still do what I love to do, but since I’m so cheap, I don’t have to be fast. My old knees can move at their own speed.” He did everything so precisely, built it so sturdy, that I used to joke that if a tornado struck the house, the house would be a pile of rubble but the ductwork would still be fine.
One of his biggest frustrations about ill health was being unable to work, and in fact he had a project going before cancer started to get the best of him. He was quite distraught that, for the first time in his life, he didn’t properly finish a job.
Willis installed a three-zone system (using automated dampers to send heat or cool from a single furnace/AC into only the parts of the house where it was needed) for me. He had never done that before. The night Willis and his friend Bob came over to finish the setup was one to remember. The two guys, both in their 70s, were figuring it all out, and their excitement was catching. By the time the evening was over, I certainly was more excited about thermostats than I ever had been in my life.
I heard a story about him once – he was removing some sort of noxious substance from someone’s house. I forget what it was — whatever it was, it had pretty bad long-term health effects. His comment: “Look, I’m old. It’s not going to be this that does me in.” And he was right.
In his last few years, Willis started up a project that only Willis would dream up. He invited people to bring him all their old and broken down appliances and metal junk – air conditioners, dehumidifiers, you name it. He carefully took them apart, stripped them down, and took the metals into a metal salvage yard. He then donated all the money he got to a charity that helped the poor, and it was nearly $5000.
Willis had a sense of humor about him that he somehow deployed at those perfect moments when you least expected it. Back in 2006, before I had moved into the house that had been grandpa’s, there was a fire there. I lost two barns (one was the big old red one with lots of character) and a chicken house. When I got out there to see what had happened, Willis was already there. It was quite the disappointment for me. Willis asked me if grandpa’s old manure spreader was still in the chicken house. (Cattle manure is sometimes used as a fertilizer.) This old manure spreader was horse-drawn. I told him it was, and so it had burned up. So Willis put his arm around me, and said, “John, do you know what we always used to call a manure spreader?” “Nope.” “Shit-slinger!” That was so surprising I couldn’t help but break out laughing. Willis was the only person that got me to laugh that day.
In his last few years, Willis battled several health ailments. When he was in a nursing home for a while due to complications from knee surgery, I’d drop by to visit. And lately as he was declining, I tried to drop in at his house to visit with Willis and Thelma as much as possible. Willis was always so appreciative of those visits. He always tried to get in a hug if he could, even if Thelma and I had to hold on to him when he stood up. He would say sometimes, “John, you are so good to come here and visit with me.” And he’d add, “I love you.” As did I.
Sometimes when Willis was felling down about not being able to work more, or not finish a project, I told him how he was an inspiration to me, and to many others. And I reminded him that I visited with him because I wanted do, and being able to do that meant as much to me as it did to him. I’m not sure if he ever could quite believe how deeply true that was, because his humble nature was a part of who he was.
My last visit earlier last week was mostly with Thelma. Willis was not able to be very alert, but I held his hand and made sure to tell him that I love and care for him that time. I’m not sure if he was able to hear, but I am sure that he didn’t need to. Willis left behind a community of hundreds of people that love him and had their lives touched by his kind and inspirational presence.
Something that inspired me recently to write about DUG, are the efforts of MediaCurrent. Media Current has recently been pushing forward a series of postings talking about how they are giving back and being a lot more open about use of time to give back (which is awesome).
In terms of wireless, Cisco primarily makes hardware (access points), but they also have a relatively wide range of associated support software: In particular, WLC (Wireless Controller, the thing that all your Cisco APs talk to for centralized configuration/authentication/load balancing/etc.), PI (Prime Infrastructure, logging/management/inventory), and MSE (Mobility Services Engine, physical positioning management).
You can buy all of these as hardware appliances, but they're also lately available as virtual appliances—after all, they're just Linux machines with software on. (You will need a Cisco support contract to download them; then you will get a free 30-day trial if you don't have a license.) But of course, since this is ENTERPRISE and it is NETWORKING, it flat-out assumes that your virtualization solution of choice is VMware ESXi. Not VMware Player, not VirtualBox, not KVM, not Hyper-V. Of course you already have an ESXi box with ~24 GB spare RAM, no?
Well, I didn't, and I wanted to try all of these three anyways. First of all, I should add that this is quite obviously not supported by Cisco. You will not get any support, and you will not be able to buy a license against such VMs; it's only for learning and evaluation. So here's the hackery needed to get it to run under KVM/QEMU:
vWLC is easy; supposedly it's even sort-of supported. Just untar the .vmdk image, convert the inner image with qemu-img to qcow2, and run. Tada.
MSE is harder. You can install it (assuming you give it exactly 8192 MB of RAM; no more, no less), but half-way through the process, it will freeze in strange ways. What's going on under-the-hood is that it tries to get the number of CPUs, and this fails in the default CPU map KVM presents through SMBIOS. The magic incantation you need to add is
-smbios type=1,product="VMware Virtual Platform" -smbios type=4,sock_pfx="Proc"
And add -cpu host for good measure.
PI, on the other hand, took quite a while. I will not go into details, but it turns out what you need is to modify the emulated BIOS from SeaBIOS to simulate the ESXi Option ROM:
( cat pc-bios/bios-256k.bin ; dd if=/dev/zero bs=655511 count=1 ; printf "VMware-56 4d 45 81 db 1a 63 8d-a9 45 65 c1 af f3 a1 a1\0" ; dd if=/dev/zero bs=130866 count=1 ) > mybios.bin
and then start with -bios mybios.bin.
Now, after all of this is done and installation is done, you can even get a root shell from the appliance, upgrade the kernel (find a more modern CentOS kernel from somewhere), modify the initrd script to load virtio_blk and virtio_pci, and then use virtio disk instead of IDE. (virtio networking works out-of-the-box.)
... is the new hype these days. Everyone seems to want to be part of it; even Microsoft wants to allow Docker to run on its platform. How they visualise that is slightly beyond me, seen as how Docker is mostly a case of "run a bunch of LXC instances", which by their definition can't happen on Windows. Presumably they'll just run a lot more VMs, then, which is a possible workaround. Or maybe Docker for Windows will be the same in concept, but not in implementation. I guess the future will tell.
As I understand the premise, the idea of Docker is that getting software to run on "all" distributions is a Hard Problem[TM], so in a Docker thing you just define that this particular stuff is meant to run on top of this and this and that environment, and Docker then compartmentalises everything for you. It should make things easier to maintain, and that's a good thing.
I'm not a fan. If the problem that Docker tries to fix is "making software run on all platforms is hard", then Docker's "solution" is "I give up, it's not possible". That's sad. Sure, having a platform which manages your virtualisation for you, without having to manually create virtual machines (or having to write software to do so) is great. And sure, compartmentalising software so that every application runs in its own space can help towards security, manageability, and a whole bunch of other advantages.
But having an environment which says "if you want to run this applicaiton, I'll set up a chroot with distribution X for you; if you want to run this other application, I'll set up a chroot with distribution Y for you; and if you want to run yet this other application yere, I'll start doing a chroot with distribution Z for you" will, in the end, get you a situation where, if there's another bug in libc6 or libssl, you now have a nightmare trying to track down all the different versions in all the docker instances to make sure they're all fixed. And while it may work perfectly well on the open Internet, if you're on a corporate network with a paranoid firewall and proxy, downloading packages from public mirrors is harder than just creating a local mirror instead. Which you now have to do not only for your local distribution of choice, but also for the distributions of choice of all the developers of the software you're trying to use. Which may result in more work than just trying to massage the software in question to actually bloody well work, dammit.
I'm sure Docker has a solution for some or all of the problems it introduces, and I'm not saying it doesn't work in practice. I'm sure it does fix some part of the "Making software run on all platforms is hard" problem, and so I might even end up using it at some point. But from an aesthetical point of view, I don't think Docker is a good system.
I'm not very fond of giving up.
Angie Byron: Webchick's "plain Drupal English" Guide to the Remaining Drupal 8 Critical Issues: DrupalCon Bogotá Edition
(Apologies for the atrocious state of the HTML that follows; this content is originally from this Google Doc.)
Webchick's "plain Drupal English" Guide to the Remaining Drupal 8 Critical Issues: DrupalCon Bogotá Edition
DrupalCon Bogotá just finished up, and critical issue-wise we've managed to stay in the 50s for a few days (down from a high of 150 last summer!), so now seems like as good a time as any to write down what's left to ship Drupal 8!
This post will attempt to document all of the remaining 55 criticals (as of this writing), and attempt to offer a somewhat "plain English" (or at least "Drupal English" ;)) description of each, loosely categorized into larger areas in which we could really use extra help. There are over 2,600 contributors to Drupal 8 at this time, please join us!
(Note: These descriptions might not be 100% accurate; this is my best approximation based on the issue summary and last few comments of each issue. If I got the description of your pet issue wrong, please update your issue summary. ;))Table of contents Quick vocabulary lesson
Within this list, there are numerous "markers" used to signify that some of the issues in this list are more important to fix ASAP. These are:
- D8 upgrade path: An issue tagged D8 upgrade path (currently, 13) means it blocks a beta-to-beta upgrade path for Drupal 8, generally because they materially impact the data schema or they impact security. Once we resolve all of these blockers, early adopters will no longer need to reinstall Drupal between beta releases, but can just run the update.php script as normal. This is currently our biggest priority.
- Blocker: An issue tagged blocker (currently, 5) means it blocks other issues from being worked on. This is currently our second-biggest priority (or 0th priority in the case an issue blocks a D8 upgrade path issue :D). I've noted these as "sub-bullets" of the issues that are blocking them.
- Postponed: Issues that are marked postponed (currently, 9) are either currently blocked by one of the "Blocker" issues, or we've deliberately chosen to leave off until later.
- >30 days: These patches have a patch more than 30 days old, and/or were last meaningfully commented on >30 days ago. If you're looking for a place to start, re-rolling these is always helpful!
- No patch: This issue doesn't have a patch yet. Oh the humanity! Want to give it a shot?
Other weird core issue nomenclature:
- "meta" means a discussion/planning issue, with the actual patch action happening in related/child issues.
- "PP-3" means "this issue is postponed on 3 other issues" (PP-1 means 1 other issue; you get the drift).
Sections roughly organized from "scariest" to "least scary" in terms of how likely they are to make Drupal 8 take a longer time to come out.Security
Because Drupal 8 hasn't shipped yet, it's not following Drupal's standard Security Advisory policy, so there are still outstanding, public security issues (13 as of this writing). We need to resolve most of these prior to providing a Drupal 8 beta-to-beta upgrade path, as this is the time when we signal to early adopters that it's an OK time to start cautiously building real sites on Drupal 8.
Skills needed: VariousSecurity Parity with Drupal 7
This class of security issue is to ensure that when Drupal 8 ships, it won't have any regressions security-wise relative to Drupal 7.
- Check every Drupal 7 contrib SA that may affect Drupal 8 core modules (D8 upgrade path) In order to ship Drupal 8, we need to ensure that there are no outstanding security advisories for contributed modules that were pushed into Drupal 8 core. nickwaring89 has started a fantabulous spreadsheet for tracking this.
- Port SA-CONTRIB-2013-096 to D8 (D8 upgrade path) Here's one such issue for Entity Reference module. SA-CONTRIB-2013-096 addressed a relatively esoteric remote access bypass bug, and the patch needs to be forward-ported to Drupal 8.
- Port SA-CONTRIB-2015-039 to D8 (D8 upgrade path) SA-CONTRIB-2015-039 addressed two issues in Views module, a redirect and default permissions for disabled views. The first was fixed in D8, but access checks are still missing from a few views for the second.
- SA-CORE-2014-002 forward port only checks internal cache (D8 upgrade path) Oopsie. Missed a spot. :P SA-CORE-2014-002 was a moderately critical Form API issue, where anonymous users' form entries on cached forms could potentially leak to other anonymous users. It was partially fixed, but not for reverse-proxies.
- Entity/field access and node grants not taken into account with core cache contexts We need to figure out and document what the API looks like for field/entity access modules that interact with the new render cache in Drupal 8.
Because of various intricate dependencies, the authentication part of Drupal 8 isn't yet converted to object-oriented code, and prevents us from further optimizing bootstrap. This set of issues fixes various problems with this part of the code, and ensures these important security APIs are complete and ready to ship.
- [meta] Finalize Session and User Authentication API (Blocker) The main tracking issue for work in this area.
- Remove dependency of current_user on request and authentication manager Aims to solve a circular dependency when implementing alternative authentication schemes, and move authentication to only happening once per request, closing a potential security hole.
- Session for an authenticated user can only be set by Cookie AuthenticationProvider (>30 days, No patch) Currently, alternative authentication providers, such as HTTP basic authentication, do not play nicely with the default login form, because Cookie trumps all.
- [meta] Security audit the Authentication component (Postponed) Since the Authentication component is new to Drupal 8, this issue proposes performing a security audit on it once it's complete, and prior to a release candidate.
- REST user updates bypass tightened user account change validation (D8 upgrade path) Since Drupal 7, when you edit your user account, you have to provide the existing password when you want to change the password or e-mail. This security feature is currently by-passed by REST user updates as you can change the password or e-mail without providing the password.
- External caches mix up response formats on URLs where content negotiation is in use (>30 days) Drupal 8's request processing system is currently based on content negotiation (which allows you to serve multiple versions of a document at the same URI based on what headers are sent e.g. Accept: text/html or Accept: application/json). This is generally considered the "right way" to do REST. However, various external caches and CDNs have trouble with this mechanism, and can mix them up and can send random formats back. The issue proposes changing from content negotiation to separate, distinct paths such as /node/1.json.
These issues affect new security improvements we want to make over and above what Drupal 7 does.
- [meta] Document or remove every SafeMarkup::set() call One of the big security improvements in Drupal 8 is the introduction of Twig's autoescape feature, which ensures that all output to the browser is escaped by default. However, this is quite a big change that requires all of the code that was previously escaping content to stop doing that, else it gets double-escaped (so you start seeing < and " and whatnot in the UI). We originally introduced the ability to manually mark markup safe with SafeMarkup::set(), but the recommended approach is actually to use Twig everywhere, so this issue is to ensure that all remaining instances of the manual way are fixed, or at least documented to explain why they're using the non-recommended method.
- Passing in #markup to drupal_render is problematic (>30 days) Another issue in the Twig autoescape space, we need to ensure that markup set by the "#markup" in e.g. form definitions is properly escaped.
- Limit PDO MySQL to executing single statements if PHP supports it Remember SA-CORE-2014-005? Yeah, so do we. ;) This issue is to make sure that if another SQL injection vulnerability is ever found again, the damage it can do is more limited by eliminating the ability for MySQL to execute multiple queries per PDO statement.
Tied with security, 13 of the remaining issues are tagged Performance. While it may seem odd/scary to have this be a big chunk of the work left, it's a common practice to avoid premature optimization, and instead focus on optimization once all of the foundations are in place.
Skills needed: Profiling, caching, optimization, render APIProfiling
Here are a sub-set of issues where we need performance profiling to determine what gives us the biggest bang for our effort.
- [Meta] Make drupal install and run within reasonable php memory limits so we can reset the memory requirements to lower levels Due to a variety of issues, including the YAML parsing slowness mentioned above, Drupal 8 currently requires 64M of memory to install, which will only go up as contrib modules are added. The goal is to reduce that significantly, more towards Drupal 7's numbers. The issue contains a number of profiling results and sub-issues that help.
- Profile to determine which services should be lazy Drupal 8 exposes a number of Services (which contain re-usable functionality and allow for pluggability/replacement). Normally, all services that are dependencies of other services are loaded on page load. However, we recently introduced the ability to mark individual services as "lazy"—meaning, to only load them on-demand. This issue is to determine which services are currently loading on every request, yet unneeded for most, so we can mark them as such.
- Profile/rationalise cache tags Drupal 8's caching API introduces the notion of cache tags, allowing for much more focused and targeted cache clears for much better performance. This issue involves investigating our usage of cache tags in D8 and seeing how they could be optimized/improved.
- [meta] Resolve known performance regressions in Drupal 8 This is the main tracking issue in this space. During the 8.x cycle we've introduced several known performance regressions compared to Drupal 7 (sometimes to make progress on features/functionality, other times because we introduced changes that we hoped would buy us better scalability down the line), which we need to resolve before release so that Drupal 8 isn't slower than Drupal 7. The performance team meets weekly and tracks their progress in a detailed spreadsheet.
- YAML parsing is very slow, cache it with APCu in Drupal\Core\Config\FileStorage::read (Blocker) Installation in Drupal 8 is not as quick as it otherwise would be due to the slowness of parsing YAML files, sometimes more than once. This issue proposes to add a caching layer to speed things up, and also help eliminate noise found in profiling.
- Convert menu CSRF tokens to use #post_render_cache (Blocker, >30 days) Drupal employs robust Cross-Site Request Forgery protection which involves appending a user-specific token on forms and links. However, this is both a bit overkill (in most systems there is just a single CSRF token per request) and also prevents caching of CSRF-protected forms/links.
- [PP-1] Cache localized, access filtered, URL resolved, (and rendered?) menu trees (Postponed, >30 days) An impressive performance improvement for the new D8 toolbar, as well as menu blocks.
- Add cache wrapper to the UrlGenerator In Drupal 8, the url() function has been replaced by the UrlGenerator class instead. This issue is proposing to add caching to make it able to not re-do work once it's already generated a given URL on the page.
- Optimize the route rebuilding process to rebuild on write Rebuilding the list of routes is expensive, and can result in race conditions (this also affects Drupal 7). This issue proposes to move menu rebuilding to write-only requests, which are expected to be expensive anyway.
- Cache-enabled forms generate cached form data for every user on every request (No patch) There's currently a bug exposed by Views—Views exposed filter form causes enormous form state cache entries—but also visible in other forms that employ caching, which results in the form cache ballooning out of control. Needs to be fixed.
- BlockContentBlock ignores cache contexts required by the block_content entity This is a bug fix (critical because there could be access control implications if a custom block has access-controlled fields on it) that ensures that a block and its associated block content both share the same list of cache contexts (e.g. language, roles, etc.).
Tracked under the Entity Field API tag (currently 6 issues).
Skills needed: Entity/Field API, Form API, Schema API
- Schema for newly defined entity types is never created (D8 upgrade path) When you first install a module that defines an entity type (for example, Comment), its database tables are correctly generated. However, if an entity definition is later added by a developer to an already-installed module, the related database schema won't get created, nor will it be detected in update.php as an out-of-date update to run.
- FileFormatterBase should extend EntityReferenceFormatterBase (D8 upgrade path) Entity Reference fields define a EntityReferenceFormatterBase class, which contains logic about which entities to display in the lookup, including non-existing entities and autocreated entities. File field's FileFormatterBase class currently duplicates that logic, except it misses some parts, including access checking, which makes this a security issue. The issue proposes to simply make File field's base class a sub-class of Entity Reference's, removing the need of "sort of but not quite the same" code around key infrastructure.
- FieldTypePluginManager cannot instantiate FieldType plugins, good thing TypedDataManager can instantiate just about anything Currently, you get a fatal error if you attempt to use Drupal 8's Plugin API to create a new instance of a field type. The current code in core is avoiding this problem by going roundabout via the Typed Data API instead. This issue's critical because these are two of the most central APIs in Drupal 8, and they should work as expected.
- [META] Untie content entity validation from form validation Despite all the work to modernize Drupal 8 into a first-class REST server, there still remain places where validation is within form validation functions, rather as part of the proper entity validation API, which means REST requests (or other types of workflows that bypass form submissions) are missing validation routines. This meta issue tracks progress of moving the logic to its proper place.
- Entity forms skip validation of fields that are edited without widgets (>30 days) If a field can be edited with a form element that is not a Field API widget, we do not validate its value at the field-level (i.e., check it against the field's constraints). Fixing this issue requires ensuring that all entity forms only use widgets for editing field values.
- Entity forms skip validation of fields that are not in the EntityFormDisplay (No patch, >30 days) Drupal 8 has a new feature called "form modes" (basically analogous to "view modes" in Drupal 7, except allowing you to set up multiple forms for a given entity instead). Currently, we're only validating fields that are displayed on a given form mode, even though those fields might have validation constraints on other fields that are not displayed. Critical because it could present a security issue.
Views issues are generally tracked with the VDC tag. There are currently 6 criticals at this point which touch on Views (some already covered in earlier sections).
- Views base fields need to use same rendering as Field UI fields, for formatting, access checking, and translation consistency (D8 upgrade path, Blocker) This is a critical blocker to multilingual functionality; right now, Views mixes up languages when a node title (base field) and body (field UI field) are in the same view. However, it's also the cause of various other inconsistencies, like the inability to select formatters and formatter options on base fields. This issue proposes treating base entity fields the same as Field UI fields in Views.
- [PP-1] Base entity fields using 'standard' plugin added via EntityViewsData to not respect field level access (D8 upgrade path, Postponed) Basically, a subset of the same problem. It's postponed because the above issue may end up solving it.
- Views should set cache tags on its render arrays, and bubble the output's cache tags to the cache items written to the Views output cache This one is critical because it could result in Views showing stale content due to not correctly associating the cache tags of content displayed inside a view with the view itself.
- Views exposed filter form causes enormous form state cache entries Because serialized views are ginormous, the size of the form cache grows exponentially on repeated load of a view with an exposed filter. This issue aims to reduce what is cached by views to stop this from happening.
The configuration system is remarkably close to being shippable! Only 4 critical issues left. We're now working on finalizing the niggly bits around edge cases that involve configuration that depends on other configuration.
Skills needed: Configuration system, Entity Field API, Views
- [meta-3] CMI path to release: The main tracking issue for CMI-related issues.
- Don't install a module when its default configuration has unmet dependencies (D8 upgrade path) Seems like a good idea. :P Basically handles the situation where a module provides some default configuration (say, a default View), which references a dependency on some other module (say, an Entity Reference field). You want to ensure that the module's default configuration can't be installed unless all the various dependencies it needs are there.
- Determine which config entities can be fixed and which will be deleted when a dependency is removed (Blocker) When we uninstall a module we list which other configuration will be "affected" by the uninstallation. This issue proposes to add new functionality to the configuration system to work out what is going to happen when a specified dependency (or set of dependencies in the case of multiple modules) is going to be removed.
- [PP-1] Delete dependent config entities that don't implement onDependencyRemoval() when a config entity is deleted (D8 upgrade path) In the case where dependent configuration is part of the main configuration (for example, fields on a node type) we want to ensure clean-up is done when the main configuration is deleted.
This subset of issues are things that are part of core currently, and we would really like to keep, but are willing to make some hard choices in the event they are among the last remaining criticals blocking release. The "postponed" among this list means "postponed until we're down to only a handful of criticals left." If these issues end up remaining in the list, we will move their functionality to contrib, and hope to add it back to core in a later point release if it gets fixed up.
Skills required: Various, but mainly low-level infrastructure and non-MySQL database skills.
- [meta] Drupal.org (websites/infra) blockers to a Drupal 8 release (Blocker) This issue contains a "grab bag" of Drupal.org blockers that prevent an optimal Drupal 8 release, including things like semantic versioning support, testing support for multiple PHP/database versions, and support for Composer-based installations. If this issue is one of the last remaining criticals, we might choose to ship Drupal 8 anyway, and jettison one or more features in the process, such as…
- [Meta] Make Drupal 8 work with PostgreSQL The meta/planning issue for fixing PostgreSQL (both in terms of functionality and in terms of failing tests). bzrudi71 is predominantly leading the charge here and making steady progress, but more hands would be greatly appreciated.
- PostgreSQL constraints do not get renamed by db_rename_table() One of the sub-issues of the above, critical because it causes failing tests.
- [policy, no patch] Move PostgreSQL driver support into contrib (Postponed) If efforts to fix PostgreSQL fails or we don't get testbot support for PostgreSQL in time, it's off to contrib-land (where, sadly, it is even less likely to survive regressions).
- [meta] Database tests fail on SQLite (>30 days) Same deal as PostgreSQL but for SQLite. Unlike PostgreSQL though, this one doesn't have anyone leading the charge at this time, and it's also a lot harder to punt this to contrib, since we use it for various things such as testbot. Help wanted!
- Remove the UI for installing/updating modules from update module if it is not fixed in time for release (Postponed) One major security improvement of Drupal 7 was providing the ability to install/update modules and themes directly from the browser. However, the feature has atrophied in Drupal 8 due to lack of test coverage and lack of active use/maintenance, and now the functionality is broken. And while the feature's very useful, it's not useful enough to further delay Drupal 8's release if it's one of the last critical issues left. joelpittet is making a valiant effort to try and save this feature in Install a module user interface does not install modules (or themes), but the issue would definitely benefit from other helping hands, particularly for extra testing/patch reviews.
These are all basic things we need to keep on top of between now and release, to ensure that when we're down to only a handful of criticals, we're ready to ship a release candidate. The good news is, these are also all generally really easy patches to make, and often also to test.
Skills needed: Basic patch rolling / reviewing / testing skills. (good for newbies!)
- [meta] Ensure vendor (PHP) libraries are on latest stable release Basically, exactly what it says. :) Making sure that all of the external libraries referenced in core's composer.json file are up to the latest stable releases.
- Upgrade validator integration for Symfony versions 2.5+ This one is called out specially because doing this brings us inline with Symfony 3, which is important for future-proofing Drupal 8.
- [META-12] Review 'revisit before release candidate' tag There are a number of issues that for one reason or another (for example, because we made a decision in order to unblock progress but weren't completely sure if it'd be the right one N months/years later when D8 shipped) we've tagged revisit before release candidate. We need to make sure this list is down to zero in order to ship Drupal 8.
- [meta] Provide a beta to beta upgrade path (D8 upgrade path, Postponed) A policy issue that documents what holds up a beta-to-beta upgrade path, and what happens after we ship an "upgrade path beta." Postponed until all other critical D8 upgrade path issues are fixed.
- [policy, no patch] 8.0.0 release candidates, release, patch versions, 8.1.x A policy discussion about what happens once we reach zero critical issues. Needs to be figured out before that happens. :)
I couldn't figure out a nice heading for these, so here's the rest.
- Remove _system_path from $request->attributes Symfony provides a $request object, which has an "attributes" property for the purpose of storing various contextual bits. But the problem with $request->attributes->get('_MAGIC_KEY') is that the values are undocumented, there's no IDE autocompletion, and it's not clear which are internal vs. public properties, so we have an issue at [meta] Stop using $request->attributes->get(MAGIC_KEY) as a public API. to try and stop doing that.
However, _system_path in particular is used a ton, since it's very common to want to know the path of the current request. The patch exposes a "CurrentPath" service instead, which eliminates all of those issues.
- Potential data loss: concurrent node edits leak through preview Because the temp store that Drupal 8's new node preview system employs uses an entity's ID as the key, rather than something uniquely identifiable to a user, if two users are editing the same node and hit preview at the same time, one of them is going to lose data due to a race condition.
- Ajax file uploads fail on IE 9 Pretty much exactly what it says on the tin. :P
Well, not so thrilling, but at least a conclusion. :)
- Anywhere you see a blocker issue, attack it with fire. Those are holding other criticals up.
- The biggest area of focus right now is D8 upgrade path blockers. Many of them are security issues.
- Another big area is Performance, both fixing existing regressions, and profiling to determine where our biggest wins are.
- Views and Entity Field API are tied in third place for number of remaining criticals. Let's have a race, shall we? ;)
- The configuration system is looking pretty good, but still has a handful of sticky issues left.
- There are a series of important features we'll lose if they're not fixed up in time.
- If you're looking for something somewhat easy/mundane, help yourself to one of the general house-keeping issues.
- Don't forget about the other miscellaneous issues I was too tired to categorize.
Sorry this post was so long (and probably has its share of inaccuracies) but I hope it will be helpful to some. It's basically what I needed to get back up to speed after taking a few months off of Drupal 8, so figured I'd document my way to understanding.
Now, let's get 'er done! :DTags: drupal 8drupaldrupal core diaries
If one of your Behat scenarios kicks off a batch job (e.g., a Feeds import), and you want to wait for that batch job to finish before moving on to the next step, add this step definition in your FeatureContext.php file:
Long story short, we put in a bid to host Debconf 16 in Cape Town, and we got it!
Back at Debconf 12 (Nicaragua), many people asked me when we’re hosting a Debconf in South Africa. I just laughed and said “Who knows, maybe some day”. During the conference I talked to Stefano Rivera (tumbleweed) who said that many people asked him too. We came to the conclusion that we’d both really really want to do it but just didn’t have enough time at that stage. I wanted to get to a point where I could take 6 months off for it and suggested that we prepare a bid for 2019. Stefano thought that this was quite funny, I think at some point we managed to get that estimate down to 2017-2018.
That date crept back even more with great people like Allison Randal and Bernelle Verster joining our team, along with other locals Graham Inggs, Raoul Snyman, Adrianna Pińska, Nigel Kukard, Simon Cross, Marc Welz, Neill Muller, Jan Groenewald, and our international mentors such as Nattie Mayer-Hutchings, Martin Krafft and Hannes von Haugwitz. Now, we’re having that Debconf next year. It’s almost hard to believe, not sure how I’ll sleep tonight, we’ve waited so long for this and we’ve got a mountain of work ahead of us, but we’ve got a strong team and I think Debconf 2016 attendees are in for a treat!
Since I happened to live close to Montréal back in 2012, I supported the idea of a Debconf bid for Montréal first, and then for Cape Town afterwards. Little did I know then that the two cities would be the only two cities bidding against each other 3 years later. I think both cities are superb locations to host a Debconf, and I’m supporting Montréal’s bid for 2017.
Want to get involved? We have a mailing list and IRC channel: #debconf16-capetown on oftc. Thanks again for all the great support from everyone involved so far!
Here's to a happy, successful, and overall quite awesome DebConf16 in Cape Town, South Africa.
As a very welcome surprise, the Montreal team is already planning a mini-DC and already have a strong bid for DC17.
Update: Well, that was quick...
The UDD bugs interface currently knows about the following release critical bugs:
- In Total:
192 bugs affecting
- Affecting Jessie:
147 (key packages:
110) That's the number we need to get down to zero
before the release. They can be split in two big categories:
- Affecting Jessie and unstable:
106 (key packages:
82) Those need someone to find a fix, or to finish the
work to upload a fix to unstable:
- 25 bugs are tagged 'patch'. (key packages: 23) Please help by reviewing the patches, and (if you are a DD) by uploading them.
- 4 bugs are marked as done, but still affect unstable. (key packages: 0) This can happen due to missing builds on some architectures, for example. Help investigate!
- 77 bugs are neither tagged patch, nor marked done. (key packages: 59) Help make a first step towards resolution!
- Affecting Jessie only: 41 (key packages: 28) Those are already fixed in unstable, but the fix still needs to migrate to Jessie. You can help by submitting unblock requests for fixed packages, by investigating why packages do not migrate, or by reviewing submitted unblock requests.
- Affecting Jessie and unstable: 106 (key packages: 82) Those need someone to find a fix, or to finish the work to upload a fix to unstable:
- Affecting Jessie: 147 (key packages: 110) That's the number we need to get down to zero before the release. They can be split in two big categories:
How do we compare to the Squeeze and Wheezy release cycles?Week Squeeze Wheezy Jessie 43 284 (213+71) 468 (332+136) 319 (240+79) 44 261 (201+60) 408 (265+143) 274 (224+50) 45 261 (205+56) 425 (291+134) 295 (229+66) 46 271 (200+71) 401 (258+143) 427 (313+114) 47 283 (209+74) 366 (221+145) 342 (260+82) 48 256 (177+79) 378 (230+148) 274 (189+85) 49 256 (180+76) 360 (216+155) 226 (147+79) 50 204 (148+56) 339 (195+144) ??? 51 178 (124+54) 323 (190+133) 189 (134+55) 52 115 (78+37) 289 (190+99) 147 (112+35) 1 93 (60+33) 287 (171+116) 140 (104+36) 2 82 (46+36) 271 (162+109) 157 (124+33) 3 25 (15+10) 249 (165+84) 172 (128+44) 4 14 (8+6) 244 (176+68) 187 (132+55) 5 2 (0+2) 224 (132+92) 175 (124+51) 6 release! 212 (129+83) 161 (109+52) 7 release+1 194 (128+66) 147 (106+41) 8 release+2 206 (144+62) 9 release+3 174 (105+69) 10 release+4 120 (72+48) 11 release+5 115 (74+41) 12 release+6 93 (47+46) 13 release+7 50 (24+26) 14 release+8 51 (32+19) 15 release+9 39 (32+7) 16 release+10 20 (12+8) 17 release+11 24 (19+5) 18 release+12 2 (2+0)
We're always on the lookout for great sites built with Drupal Commerce, our truly flexible software that's changing the face of eCommerce one site at a time.
Perhaps the biggest strength of Drupal Commerce is it's flexibility, and that's clearly at work on the Novus Bio web site, a niche eCommerce site that's servicing a unique need in BioTech. Novus Biologicals features a commerce suite with a multitude of products available internationally for buyers of many different languages. Not to mention they are selling "cells", How cool is that?
To see Drupal Commerce sites we've Spotlighted in previous weeks view the Other Spotlight Sites