I run a cluster for the Debian Administration website, and the code is starting to show its age. Unfortunately the code is not so modern, and has evolved a lot of baggage.
Given the relatively clean separation between the logical components I'm interested in trying something new. In brief the current codebase allows:
- Posting of articles, blog-entries, and polls.
- The manipulation of the same.
- User-account management.
It crossed my mind the other night that it might make sense to break this code down into a number of mini-servers - a server to handle all article-related things, a server to handle all poll-related things, etc.
If we have a JSON endpoint that will allow:
- GET /article/32
- POST /article/ [create]
- GET /articles/offset/number [get the most recent]
Then we could have a very thin shim/server on top of that whihc would present the public API. Of course the internal HTTP overhead might make this unworkable, but it is an interesting approach to the problem, and would allow the backend storage to be migrated in the future without too much difficulty.
At the moment I've coded up two trivial servers, one for getting user-data (to allow login requests to succeed), and one for getting article data.
There is a tiny presentation server written to use those back-end servers and it seems like an approach that might work. Of course deployment might be a pain..
It is still an experiment rather than a plan, but it could work out: http://github.com/skx/snooze/.
Those reading this journal may have noticed that my rate of posting has dropped a bit in the past few years, and quite a lot in the past year. One of the major reasons for this was work, which had been getting more bureaucratic, more stressful, less trusting, and more fearful. After this got drastically worse in the past six months, I finally decided enough was enough and took advantage of a good opportunity to do something different.
I will be joining Dropbox's site reliability engineering team in a week and a half (which means that I'll be working on their servers, not on the product itself). It will take a few months to settle in, but hopefully this will mean a significant improvement to my stress levels and a lot of interesting projects to work on.
I'm taking advantage of this change to inventory the various things I'm currently committed to and let go of some projects to make more space in my life. There are also a variety of software projects that I was maintaining as part of my job at Stanford, and I will be orphaning many of those packages. I'll make another journal post about that a bit later.
For Debian folks, I am going to be at Debconf, and hope to meet many of you there. (It's going to sort of be my break between jobs.) In the long run, I'm hoping this move will let me increase my Debian involvement.
In the long run, I expect most of my free software work, my reviews, and the various services I run to continue as before, or even improve as my stress drops. But I've been at Stanford for a very long time, so this is quite the leap into the unknown, and it's going to take a while before I'm sure what new pattern my life will fall into.
Normally I'm disgusted by fangirling of jwz, but it seems that he finally wrote something I like.
If you are a user of Ganglia, Nagios, RRDtool or R or just an enthusiastic C or Python developer, you may be able to use and provide feedback for the students while benefitting from the cool new features they have been working on.Student Technology Comments Chandrika Parimoo Python, Nagios and some Syslog Chandrika generalized some of my ganglia-nagios-bridge code into the PyNag library. I then used it as the basis for syslog-nagios-bridge. Chandrika has also done some work on improving the ganglia-nagios-bridge configuration file format. Oliver Hamm C Oliver has been working on metrics about Ganglia infrastructure. If you have a large and dynamic Ganglia cloud, this is for you. Plamen Dimitrov R, RRDtool Plamen has been building an R plugin for inspecting RRD files from Ganglia or any other type of RRD. Rana NVIDIA, C Rana has been working on improvements to Ganglia monitoring of NVIDIA GPUs, especially in HPC clusters Zhi An Java, JMX Zhi An has been extending the JMXetric and gmetric4j projects to provide more convenient monitoring of Java server processes.
If you have any feedback or questions, please feel free to discuss on the Ganglia-general mailing list and CC the student and their mentor.
Since the people behind and maintaining the plugins <= 1.5 were forced to rename the software project into Monitoring Plugins there was some work behind the scenes and much QA work necessary to release the software in a proper state. This happened 4 weeks ago with the release of the version 2.0 of the Monitoring Plugins.
With one day of delay the package was uploaded into unstable, but did hit the Debian NEW queue due the changed package name(s). Now we (and maybe you) are waiting to get them reviewed by ftp-master. This will hopefully happen before the jessie freeze.
It's funny how quick things happen - Really it is!
Just a week ago I posted I am a Follower & Thinker describing some of my experiences from Open Source. Then, just days later, someone had left me a message in an open Drupal chatroom. What happened after is the result of a chain of interesting - but more or less isolated - events and situations.Quick background
I'm currently spending some of my time working on an open initiative called Baksteg. I have quite a bit of experience with Drupal and for me it is more than good enough to build the site I have in mind with. Just recently I begun doing some real prototyping of ideas too. Most of the testing have gone very well, but some not so and for those I have started to seek the online community more - poking for help to find solutions or alternative ways.Online communities are everywhere
www.drupal.org is a fantastic resource to start with. Many problems can quickly be solved doing a quick search there, or it's related sites such as groups.drupal.org and api.drupal.org. Then, when you struggle to find useful help by just searching, you have already started to find other channels to communicate on. In fact you find people using, working with, Drupal everywhere these days. That even includes the same social networks everyone else uses, such as Twitter, Facebook and LinkedIn. Personally I prefer Twitter as it fills my needs and interests good enough, both with Drupal and other ones.
Over the last few months I have worked on tuning my own use of Twitter. I wanted it less as a *megaphone* and more a communication tool to have meaningful, while at the same time a bit entertaining, conversations on. It has worked out really well, while also - as an welcomed bonus - helped me much better appreciate what others actually share out there. My two main feeds are now @tsvenson (personal, mainly in English) and @Baksteg (mainly in Swedish). They both play important roles for my daily needs, which - totally coincidently - works really well with how I now see open source collaboration works out to ;)Kinda one of the original Social Networks, not that long time ago
However this story happened in the Swedish Drupal-channel on IRC - a social network for geeks and nerds since long ago. It was from kristofferw_ (Kristoffer Wiklund) asking me about this entity ID *problem* I was having troubles with. Some days earlier I had posted a description of it in the Swedish Drupal Group. While that post resulted in some nice advices and ideas, they all turned out to be dead ends. Still, it gave me opportunity to play around with some other modules that later will be used.
- Practice is always good I'm told ;)
Kristoffer was one of them who had helped there. We also go further back, including several Drupal events around. Thus we already know each other, at least when it comes to Drupal stuff. I explained that testing been good, but the work had to, for good reasons, be *pushed* forward. That's when Kristoffer offered to help and write a simple module, if nothing else just to get some coding experience poking around.
As things often end up then, when passionate nerds and geeks find an interesting problem or challenge, brainstorming starts and ideas flows back and forth.
IRC is an important channel in the Drupal community, but it is not because of it's fancy features. It's its simplicity that makes it into such a useful tool to use as a communication hub. A hub that connects us to worldwide chatrooms that can run in the background only to be brought forward when needed or when we have a moment to spare. Or just as inspiration...
Just following the discussions is itself educational and often spurns new ideas. There are many different specialized chatrooms too, one simply called #Drupal-support, most often filled with several hundreds of users, helping each other. It is in these chatrooms many of the toughest challenges with improving the project is ironed out.A new project is born
It is also here many new projects are born, small as larger ones. As Kristoffer and I began to talk, we quickly found a much more interesting approach. This one had much better potential and many more use cases as well as better flexibility and UX benefits too - So we created a sandbox project on drupal.org. It is now used so that we can experiment further in a better collaborative, open and efficient way.
If this module turns out the way we think it will, then we can take the next step and apply for it to become a Full project. This is a form of quality assurance process that includes us behind the project too, but to get there we need to pass gateways. These are not put in place to stop us though, quite the opposite. Many members in the community voluntary spend their own time to help others to pass. The whole guided process is filled with tools, tips and personal advices about how to make the project work as good as possible, not just for one self but for others too.
Once a full project, access to new features to organize and administrate is granted. Includes proper name space and better ways to manage versions and releases. These are features rarely needed when just poking around and testing ideas in the sandbox.
What I have also learned is what an amazing way to improve my own skills this is. Not just coding related, but also the way to collaborate and how great knowledge transfer can work too. All while at the same time get to know new interesting people and getting exposed to new cultures and ideas.
For me the Drupal community encapsulates all this, and more. Then, taking its size and success into count, it is a pretty remarkable achievement that shows how things can be done quite differently.
Add this to the mix:
- Drupal is used on millions of sites
- Drupal probably already generates a multi billion dollar ecosystem around it
Still there is room for practically anyone, like myself, to feel welcomed and included.
Even if it just starts out as a learning experience!What this module does
At first glance it looks to do little more than adding an extra step, when creating new content, while hiding parts of the form from displaying. That's right, that is basically what it does and one of its main purposes.
Some might now say - Hey, you just going to end up with tons of garbage content! - and they would be right too. Sure, this is going to make it quicker to create a lot of content - Yes, it can certainly be used for that!
But it also makes some other quite interesting new things possible - This is why:
- Entities in Drupal have unique ID-numbers, which can be used for all sorts of interesting things.
- A new entity doesn't get its own ID until after it has been saved the first time.
Therein we find my initial problem! There where no smooth way to get around the fact: Users filling in those forms must remember and manually save once before adding content to certain fields. Worse, it would be tricky to notice, when forgotten, as in most cases the data would evaluate, just with something much different than the entity ID needed.
As I played around, with several other modules to see if it was possible to circumvent this somehow, I always hit the same brick-wall. Problem was: Every new idea that showed promise resulted in a solution with tons of complexity, not just in one place but several. Gladly, that complexity was what Kristoffer and I could avoid with just a little bit collaborative brainstorming!
What this module now does is simple:
- Hijacks entity create (just for content types yet)
- Allows to limit to content create form down to only display the Title field
- Creates the new entity
- Immediately reopens it in normal edit
Content type settings:
Adding a new node:
Thanks to this, I can now avoid displaying any fields that needs the entity ID, minimizing the risk of mistakes. At the same time it also means this new - pre create content - step can be used to only show the bare essential fields while opening up to new interesting possibilities. Any rarely used, and other optional, fields can be dealt with better as they come into play.
This new, less complex, solution will also be much easier to improve upon. It has actually already allowed me to visualize and identify a whole bunch of interesting uses and UX-benefits. It will improve for many roles, not the least site builders and content editors.
So, for me, this is no garbage creator. Instead I see how a small improvement can open up many new creative ways to work and collaborate on content. However that depends, after all it is still just a sandbox project on drupal.org a kind of - Open Source playground - for nerds and geeks like myself.
Not everything is as limited as what is usually read on the tin. For me, this is a few good examples of how things in Open Source can work out just nicely.
Note: What *specifics* Kristoffer gets out of this I know little of. Those specifics are his to share and actually not that important for me - As long as I sense we both get something of value out of the collaboration.
As you may have noticed I wrote a new plug-in for Elektra called “line“. I used it for a lot of examples in my tutorial, How-To: Write a Plug-In. The line plug-in is a very simple storage plug-in for Elektra. This plug-in stores files into the Elektra Key Database creating a new Key for each line and setting the string value of the Key to the string value of the line of that file. So if we have a file called “hello.txt”:
And we mount it to kdb like so: kdb mount ~/line.txt system/hello_line line. The output of kdb ls system/hello_line the output would be:
With the getString values of #1 and #2 being Hello and World! respectively. If this seems like a very simple plug-in, that’s because it is. Obviously, this plug-in isn’t a great showcase for the robustness of Elektra, any data structure could store a file line by line relatively easy, so why did we add a line plug-in at all?
The answer is that we included a line plug-in to allow any line-based file to use functions of Elektra, particularly the new Merge function. My Google Summer of Code project is to allow for automatic three-way merge of Debian conffiles during package upgrades as opposed to the current prompt and manual merging a user must do if a conffile is edited. Using Elektra and the new mergecode we can mount a conffile with the best plug-in for it, (the ini plug-in for Samba’s smb.conf for instance) and that allows for a very powerful merging ability with a lot more success than a simple diff merge. However, there are a lot of conffiles that don’t use any particular standard (such as ini, xml, or JSON) to store data. That is where the line plug-in comes in. We can still mount these files using the line plug-in and attempt a merge. Of course it is much more likely to have conflicts, and this type of merge is still susceptible to many of the same flaws as regular file merges (such as not being able to detect when a line has been moved), but it simple cases, the merge may succeed which would reduce the overall number of times a user would be prompted during an upgrade.
Basically, I wrote a line plug-in for Elektra as a fallback for conffile merges when we can’t mount the conffile in any more meaningful way. While merges using KeySets that were mounted using line are more likely to fail than other, more specialized plug-ins, there are cases that these merges will succeed and the user will not have to deal with a confusing prompt. The whole point of my Google Summer of Code project is to make upgrading packages and dealing with conffiles much smoother and easier than it is now by including a three-way mere and this line plug-in will help with this goal.
Ian S. Donnelly
Since our last update, May 2014, the #d8rules initiative was able to complete funding for Milestone 1 and has made major development progress with two unstable releases and loads of commits on our GitHub repository.
Thanks to 142 great supports, we were able to raise $ 15k on Drupalfund. In addition to that, Technocrat stepped in with buying our second largest sponsor package "Reusable component providers" which helps us cover 100 hours of Rules 8.x development. All in all, we are excited to say that Milestone 1 is now fully funded thanks to the crowd funding and 10 companies sponsoring #d8rules.
If you are interested in more background on the #d8rules Drupalfund campaign, make sure to Virginia's blog post with all the details.#d8rules Rulers & invoices info
As promised, everybody who has donated $65+ on the Drupalfund will get one of the fancy Rulers provided by Nico from Ausgetrock.net! We just finished production & packaging and will send them out by the next week!Development update
Development has mainly focused on working with Drupal's context system. Rules actions and conditions need parameters (for example a node object), which is represented as plugin context as used in Drupal core. They also need to provide variables (for example an entity load action will provide the loaded node object), which is implemented as provided plugin context in Rules itself. Provided context might be interesting for other Drupal 8 modules as well, so this might be moved out of Rules either into Drupal core or a Ctools-like project in D8 contributed space.
Currently we try to figure out all parts of the Rules core engine, how context is passed around and how data selectors are applied (example: node:uid:entity:mail:value to access the email adress of a node author). We implemented prototypes of those system parts, but the API is still experimental and we expect to change and improve things here a lot.
We are currently working towards milestone 1 of our roadmap and we are doing monthly unstable releases to keep you up to date with development progress.Drupalaton training & sprints: Porting Actions
After a great training/sprint session at DrupalCamp Alpe-Adria Portoroz we finished porting most of the conditions. This weekend, at Drupalaton we just delivered our second training session to get contributors up to speed with Rules in Drupal 8. Topics delivered include our git pull request workflow and the new and shiny things about Drupal 8 including the plug-in system, CMI and many more.Follow [META] Port all actions to 8.x to find out about the current status of actions being ported. Our getting started google doc provides steps in addition to the documentation available on fago's Rules GitHub repository. Upcoming sprint: DrupalCon Amsterdam
- See the sprints information & make sure to fill in at the #d8rules section in the sign-up sheet.
- Drupal 8 Contrib Module Update
Thank you all for contributing.
Josef / dasjo on behalf of the #d8rules team
Drupal 8 is not far off from being released, and you may have heard some chatter about the differences in how you create custom modules. The reality is that, while there are some differences, it's not really that hard to wrap your head around them. This article provides a gentle introduction to creating forms in Drupal 8, and highlights the differences and similarities to how you would do this in previous versions of the platform.
Topics: Tools, Technique, Education
Lately I've been very interested in declarative programming. Last week Smashing Magazine published my article on Declarative Programming and the Web, and last weekend I gave a talk at DrupalCamp Colorado on "Footless Drupal" in which I talked about how Drupal 8 is using declarative programming for configuration; how the Config in Code (CINC) module is aiming to, among other things, backport that to Drupal 7; and how it's already possible to use custom configuration workflows outside the default Drupal interfaces.
If you're not already familiar with declarative programming and/or the Drupal 8 configuration API, I recommend reading my declarative programming article in Smashing (of course I do), as well as Drupal's configuration API documentation before continuing. All caught up? Great, let's talk about some next steps for expanding declarative programming in Drupal.Custom Configuration Tools
First, we need more tools for managing configuration, with interfaces designed around more specific workflows. We're using a wide variety of approaches to building Drupal sites, and with a standard configuration format, there's no reason we shouldn't have the same variety of configuration interfaces. Do you define your content types in spreadsheets? Me too. I made an interface for that. Do you test your menus as static HTML? Great, let's make an interface that converts HTML menus to Drupal menu config. Do 90% of your views show all content of a given type? Let's auto-generate those Views configs.
While there's still a lot changing in Drupal 8, and some of that will likely impact configuration structures, those structures are stable enough and simple enough that we should be building our own tools around them today. And as CINC gets closer to D8 config API parity, we can use more and more of the same configuration in D7 as well. These tools can be written entirely outside Drupal, in a completely different language if you prefer, which then exports to YAML. Or you can take advantage of the information available within Drupal, i.e. current site configuration and content, and build modules with new interfaces. The world is our new Drupal configuration playground. Let's get playing.Declarative Forms
But there's also no reason declarative programming in Drupal needs to stop at configuration. Drupal has a wide variety of non-configuration concepts that could benefit from declarative approaches. One area I've been thinking a lot about lately is forms. Drupal's form API is already mostly declarative, but it's built around PHP arrays, and we interact with form arrays with imperative code. We could probably learn a lot by comparing Drupal forms to existing standards for declarative forms, like XForms. But simply taking a form array and formatting it as YAML would be a good start. (Or we could do it as JSON, as Amitai Burstein has suggested).
I'm sure declarative forms in Drupal would end up being a large and complex project, but it would come with large benefits as well. Many form alters apply universally to a given form, so they might as well be editing the original form definition directly. That wouldn't make sense in community module code, but moving the form definition outside code would allow such form alters to happen directly in the configuration. The same way we can now do \Drupal::config('node.type.page')->set('name', 'Basic page')->save(), we could do form alters with something like \Drupal::form('formid')->set('mytextfield.#title', 'My Text Field')->save(). Or with a YAML workflow, simply editing a YAML file would edit the form. That's already a huge benefit, as it would open up many form alters to a much wider group of implementers. Remember when we edited forms directly in HTML? A YAML form interface could give us that same ease of editing while still maintaining the power of Drupal's form API.
And just like with declarative configuration, declarative forms would open up a wider world of use cases. Forms created in Webform or the field API could be easily reused in custom modules. And the same forms could be built or used entirely outside Drupal, opening the door for more form-building tools focused on specific workflows.
Hopefully this is enough to get more people excited about taking advantage of the parts of Drupal that already have declarative interfaces, and also pushing declarative programming even further in Drupal. I'm continuing to work on these ideas and will likely have more examples to share soon, but I'd really like to see more people playing here. If you're already working or interested in working on declarative programming in Drupal, let's talk.
(Yes, I am on Debian's trademark team and no, I have no idea what that means. Yet.)
This week the podcast gets a little bit meta by discussing how to find Drupal news. Addison Berry is joined by three different community news sources: Mike Anello of DrupalEasy podcast, Bob Kepford of The Weekly Drop newsletter, and Chris Weber from Drupal and Coffee in the Morning hangout and the Google+ Drupal community. We discuss why following the news is important, and how we manage to keep up with it.
Occasionally, one gets in touch with kind of ‘foreign’ technologies and needs to get stuff working anyways.
Recently, I had to do some various hacking with and around .bat files. Bat files are a kind of script files for Microsoft Windows.
Calling external commands
Imagine need to call some other command, let’s say git diff. So from a cmd thing, you would write
similar to writing shell scripts on unixes. But there is a catch. If the thing you want to call is another bat-script, just calling it ensures it ‘replaces’ the current script and never returns. So you need
call git diff
if the command you want to run is a bat file and you want to return to your script.
Calling an external helper next to your script
If you for some reason needs to call some external helper placed next to your script, there is a helpful thing to do that as well. Imagine your helper is called helper.bat
is the very self-explanatory way of doing that.
Stopping execution of your script
If you somehow encounter some condition in your script that requires you to stop your script, there is a command ‘exit’ handy. It even takes a argument for what error code there is.
stops your script with return code 2. But it also have the nice added feature that if you do it in a script you run by hand in a terminal, it also exits the terminal.
Luckily there is also a fix for that:
exit /b 2
and it doesn’t exit your interactive terminal, and it sets the %ERRORLEVEL% variable to the exit code.
Fortunately, the fun doesn’t stop here.
If the script is run non-interactively, exit /b doesn’t set the exit code for for example perl’s system() call. You need to use exit without /b for that. So now you need two scripts. one for “interactive” use that calls exit /b and a similar one using exit for use by other apps/scripts.
Or, we can combine some of our knowledge and add a extra layer of indirection.
- write your script for interactive use (with exit /b) and let’s call it script.bat
- create a simple wrapper script
- call the wrapper for non-interactive use
and then success.
Oh. and on a unrelated note. Windows can’t schedule tasks for users that aren’t logged in and don’t have a password set. The response “Access Denied” is the only clue given.
Will you be coming to DrupalCon Amsterdam?
DrupalCon is where all the big magic happens. From sprints that improve the project to training that brings in new talent, attending DrupalCon is the best way to get involved, get connected, and give back.
Take a break from your summer holiday and secure your place at DrupalCon before ticket prices rise on 8 August (that's tonight!) at 23:59 Amsterdam local time. Save the €50 you'd pay on a late ticket and make sure you buy your tickets now to take advantage of the regular rate. You can even buy for your company, with our easy prepaid tickets.On the fence about attending?
If you're having trouble deciding whether to attend, we suggest you check out the schedule of sessions, BoFs, parties, and more, see who's going to be in Amsterdam, what you can learn, and how you can help the project while you're there. We've even got Cory Doctorow keynoting the event on Wednesday! What's not to love about that?
If you're having trouble convincing your boss to send you, check out these resources we've put together to help you get your manager's buy-in on the conference.
We hope to see you in Amsterdam!