Feed aggregator

Jonathan Carter: Debconf 2016 to be hosted in Cape Town

Planet Debian - Fri, 13/02/2015 - 22:35

Long story short, we put in a bid to host Debconf 16 in Cape Town, and we got it!

Back at Debconf 12 (Nicaragua), many people asked me when we’re hosting a Debconf in South Africa. I just laughed and said “Who knows, maybe some day”. During the conference I talked to Stefano Rivera (tumbleweed) who said that many people asked him too. We came to the conclusion that we’d both really really want to do it but just didn’t have enough time at that stage. I wanted to get to a point where I could take 6 months off for it and suggested that we prepare a bid for 2019. Stefano thought that this was quite funny, I think at some point we managed to get that estimate down to 2017-2018.

That date crept back even more with great people like Allison Randal and Bernelle Verster joining our team, along with other locals Graham Inggs, Raoul SnymanAdrianna PińskaNigel Kukard, Simon CrossMarc Welz, Neill Muller, Jan Groenewald, and our international mentors such as Nattie Mayer-Hutchings, Martin Krafft and Hannes von Haugwitz. Now, we’re having that Debconf next year. It’s almost hard to believe, not sure how I’ll sleep tonight, we’ve waited so long for this and we’ve got a mountain of work ahead of us, but we’ve got a strong team and I think Debconf 2016 attendees are in for a treat!

Since I happened to live close to Montréal back in 2012, I supported the idea of a Debconf bid for Montréal first, and then for Cape Town afterwards. Little did I know then that the two cities would be the only two cities bidding against each other 3 years later. I think both cities are superb locations to host a Debconf, and I’m supporting Montréal’s bid for 2017.

Want to get involved? We have a mailing list and IRC channel: #debconf16-capetown on oftc. Thanks again for all the great support from everyone involved so far!

Categories: Elsewhere

Richard Hartmann: DC16.za

Planet Debian - Fri, 13/02/2015 - 21:59

Here's to a happy, successful, and overall quite awesome DebConf16 in Cape Town, South Africa.

As a very welcome surprise, the Montreal team is already planning a mini-DC and already have a strong bid for DC17.

Update: Well, that was quick...

Categories: Elsewhere

Richard Hartmann: Release Critical Bug report for Week 07

Planet Debian - Fri, 13/02/2015 - 21:16

The UDD bugs interface currently knows about the following release critical bugs:

  • In Total: 1071 (Including 192 bugs affecting key packages)
    • Affecting Jessie: 147 (key packages: 110) That's the number we need to get down to zero before the release. They can be split in two big categories:
      • Affecting Jessie and unstable: 106 (key packages: 82) Those need someone to find a fix, or to finish the work to upload a fix to unstable:
        • 25 bugs are tagged 'patch'. (key packages: 23) Please help by reviewing the patches, and (if you are a DD) by uploading them.
        • 4 bugs are marked as done, but still affect unstable. (key packages: 0) This can happen due to missing builds on some architectures, for example. Help investigate!
        • 77 bugs are neither tagged patch, nor marked done. (key packages: 59) Help make a first step towards resolution!
      • Affecting Jessie only: 41 (key packages: 28) Those are already fixed in unstable, but the fix still needs to migrate to Jessie. You can help by submitting unblock requests for fixed packages, by investigating why packages do not migrate, or by reviewing submitted unblock requests.
        • 11 bugs are in packages that are unblocked by the release team. (key packages: 6)
        • 30 bugs are in packages that are not unblocked. (key packages: 22)

How do we compare to the Squeeze and Wheezy release cycles?

Week Squeeze Wheezy Jessie 43 284 (213+71) 468 (332+136) 319 (240+79) 44 261 (201+60) 408 (265+143) 274 (224+50) 45 261 (205+56) 425 (291+134) 295 (229+66) 46 271 (200+71) 401 (258+143) 427 (313+114) 47 283 (209+74) 366 (221+145) 342 (260+82) 48 256 (177+79) 378 (230+148) 274 (189+85) 49 256 (180+76) 360 (216+155) 226 (147+79) 50 204 (148+56) 339 (195+144) ??? 51 178 (124+54) 323 (190+133) 189 (134+55) 52 115 (78+37) 289 (190+99) 147 (112+35) 1 93 (60+33) 287 (171+116) 140 (104+36) 2 82 (46+36) 271 (162+109) 157 (124+33) 3 25 (15+10) 249 (165+84) 172 (128+44) 4 14 (8+6) 244 (176+68) 187 (132+55) 5 2 (0+2) 224 (132+92) 175 (124+51) 6 release! 212 (129+83) 161 (109+52) 7 release+1 194 (128+66) 147 (106+41) 8 release+2 206 (144+62) 9 release+3 174 (105+69) 10 release+4 120 (72+48) 11 release+5 115 (74+41) 12 release+6 93 (47+46) 13 release+7 50 (24+26) 14 release+8 51 (32+19) 15 release+9 39 (32+7) 16 release+10 20 (12+8) 17 release+11 24 (19+5) 18 release+12 2 (2+0)

Graphical overview of bug stats thanks to azhag:

Categories: Elsewhere

DrupalDare: Nginx, Memcache, Drupal page cache #1

Planet Drupal - Fri, 13/02/2015 - 20:09
Reverse proxy caching is something that is almost a must have for any popular site today. For Drupalers Varnish is by far the most used reverse proxy since it's easy to use and works really well/stable. Nginx has been succesful as a reverse proxy with Drupal as well, but it has mainly been been used with the file system and modules like Boost. But there exists a Memcache module that can speak directly to Nginx as well. In this article I will do some benchmarking on how to save the data when applying memcache to Nginx.
Categories: Elsewhere

Commerce Guys: Drupal Commerce Site Spotlight: Novusbio.com

Planet Drupal - Fri, 13/02/2015 - 19:06

We're always on the lookout for great sites built with Drupal Commerce, our truly flexible software that's changing the face of eCommerce one site at a time.

Perhaps the biggest strength of Drupal Commerce is it's flexibility, and that's clearly at work on the Novus Bio web site, a niche eCommerce site that's servicing a unique need in BioTech. Novus Biologicals features a commerce suite with a multitude of products available internationally for buyers of many different languages. Not to mention they are selling "cells", How cool is that?

 

To see Drupal Commerce sites we've Spotlighted in previous weeks view the Other Spotlight Sites

Categories: Elsewhere

Lullabot: Front-End Fundamentals, a Book Written by Bots

Planet Drupal - Fri, 13/02/2015 - 16:22

One of the coolest things about Lullabots is their desire to teach and share their knowledge. They do this in many formats: podcasts, articles, presentations, and even writing books. Joe Fender and Carwin Young decided there was an absolute need to write a book that brings all aspects of Front-End tools, frameworks, concepts, and procedures into one place — Front-End Fundamentals.

Categories: Elsewhere

Olivier Berger: Testing the RuneStone interactive Python courses server in docker

Planet Debian - Fri, 13/02/2015 - 15:36

I’ve been working on setting up a Docker container environment allowing to test the RuneStone Interactive server.

RuneStone Interactive allows the publication of courses containing interactive Python examples, and while most of the content is static (the Python examples are run innside a Python interpreter implemented in JavaScript, hence locally in the JS VM of the Web browser), the tool also offers an environment allowing to monitor the progress of learners in a course, which is dynamic and is queried by the browser over AJAX APIs.

That’s the part which I wanted to be able to operate for test purposes. As it is a web2py application, it’s not exactly obvious to gather all dependencies and run locally. Well, in fact it is, but I want to understand the architecture of the tool to be able to understand the deployment constraints, so making a docker image will help in this purpose.

The result is the following :

Now, it’s easier to test the writing of a new course (yet another container above the latter one), and directly test for real.

Categories: Elsewhere

InternetDevels: InternetDevels+Drupal = Love

Planet Drupal - Fri, 13/02/2015 - 14:07

We are serious about Drupal. Our relationship lasts for already 7 years by now. Today is St. Valentine’s Day — a good day to express our love to Drupal. Drupal united us and allowed making new friends, so it IS awesome and incredibly cool without any doubt! So here’s few reasons we love it (just listen to it, sounds like an ode to a real loved one):

Read more
Categories: Elsewhere

Iztok Smolic: 4 essential tips on implementing best practices

Planet Drupal - Fri, 13/02/2015 - 13:27

Drupal community talks a lot about best practices. When I talk about best practices I mean code driven development, code reviews, SCRUM, automated tests… I immediately realised that introducing new ways of working is not going to be easy. So I figured, why not asking one of the smart people how to start. Amitai (CTO of Gizra) was very kind to have […]

The post 4 essential tips on implementing best practices appeared first on Iztok.

Categories: Elsewhere

Daniel Leidert: Motion picture capturing: Debian + motion + Logitech C910 - part II

Planet Debian - Fri, 13/02/2015 - 12:40

In my recent attempt to setup a motion detection camera I was disappointed, that my camera, which should be able to record with 30 fps in 720p mode only reached 10 fps using the software motion. Now I got a bit further. This seems to be an issue with the format used by motion. I've check the output of v4l2-ctl ...

$ v4l2-ctl -d /dev/video1 --list-formats-ext
[..]
ioctl: VIDIOC_ENUM_FMT
Index : 0
Type : Video Capture
Pixel Format: 'YUYV'
Name : YUV 4:2:2 (YUYV)
[..]
Size: Discrete 1280x720
Interval: Discrete 0.100s (10.000 fps)

Interval: Discrete 0.133s (7.500 fps)
Interval: Discrete 0.200s (5.000 fps)
[..]

Index : 1
Type : Video Capture
Pixel Format: 'MJPG' (compressed)
Name : MJPEG
[..]
Size: Discrete 1280x720
Interval: Discrete 0.033s (30.000 fps)

Interval: Discrete 0.042s (24.000 fps)
Interval: Discrete 0.050s (20.000 fps)
Interval: Discrete 0.067s (15.000 fps)
Interval: Discrete 0.100s (10.000 fps)
Interval: Discrete 0.133s (7.500 fps)
Interval: Discrete 0.200s (5.000 fps)
[..]

... and motion:

$ motion
[..]
[1] [NTC] [VID] v4l2_set_pix_format: Config palette index 17 (YU12) doesn't work.
[1] [NTC] [VID] v4l2_set_pix_format: Supported palettes:
[1] [NTC] [VID] v4l2_set_pix_format: (0) YUYV (YUV 4:2:2 (YUYV))
[1] [NTC] [VID] v4l2_set_pix_format: 0 - YUV 4:2:2 (YUYV) (compressed : 0) (0x56595559)
[1] [NTC] [VID] v4l2_set_pix_format: (1) MJPG (MJPEG)
[1] [NTC] [VID] v4l2_set_pix_format: 1 - MJPEG (compressed : 1) (0x47504a4d)

[1] [NTC] [VID] v4l2_set_pix_format Selected palette YUYV
[1] [NTC] [VID] v4l2_do_set_pix_format: Testing palette YUYV (1280x720)
[1] [NTC] [VID] v4l2_do_set_pix_format: Using palette YUYV (1280x720) bytesperlines 2560 sizeimage 1843200 colorspace 00000008
[..]

Ok, so both formats YUYV and MJPG are supported and recognized and I can choose both via the v4l2palette configuration variable, citing motion.conf:

# v4l2_palette allows to choose preferable palette to be use by motion
# to capture from those supported by your videodevice. (default: 17)
# E.g. if your videodevice supports both V4L2_PIX_FMT_SBGGR8 and
# V4L2_PIX_FMT_MJPEG then motion will by default use V4L2_PIX_FMT_MJPEG.
# Setting v4l2_palette to 2 forces motion to use V4L2_PIX_FMT_SBGGR8
# instead.
#
# Values :
# V4L2_PIX_FMT_SN9C10X : 0 'S910'
# V4L2_PIX_FMT_SBGGR16 : 1 'BYR2'
# V4L2_PIX_FMT_SBGGR8 : 2 'BA81'
# V4L2_PIX_FMT_SPCA561 : 3 'S561'
# V4L2_PIX_FMT_SGBRG8 : 4 'GBRG'
# V4L2_PIX_FMT_SGRBG8 : 5 'GRBG'
# V4L2_PIX_FMT_PAC207 : 6 'P207'
# V4L2_PIX_FMT_PJPG : 7 'PJPG'
# V4L2_PIX_FMT_MJPEG : 8 'MJPEG'
# V4L2_PIX_FMT_JPEG : 9 'JPEG'
# V4L2_PIX_FMT_RGB24 : 10 'RGB3'
# V4L2_PIX_FMT_SPCA501 : 11 'S501'
# V4L2_PIX_FMT_SPCA505 : 12 'S505'
# V4L2_PIX_FMT_SPCA508 : 13 'S508'
# V4L2_PIX_FMT_UYVY : 14 'UYVY'
# V4L2_PIX_FMT_YUYV : 15 'YUYV'
# V4L2_PIX_FMT_YUV422P : 16 '422P'
# V4L2_PIX_FMT_YUV420 : 17 'YU12'
#
v4l2_palette 17

Now motion uses YUYV as default mode as shown by its output. So it seems that all I have to do is to choose MJPEG in my motion.conf:

v4l2_palette 8

Testing again ...

$ motion
[..]
[1] [NTC] [VID] vid_v4lx_start: Using V4L2
[1] [NTC] [ALL] image_ring_resize: Resizing pre_capture buffer to 1 items
[1] [NTC] [VID] v4l2_set_control: setting control "Brightness" to 25 (ret 0 )
Corrupt JPEG data: 5 extraneous bytes before marker 0xd6
[1] [CRT] [VID] mjpegtoyuv420p: Corrupt image ... continue
[1] [NTC] [VID] v4l2_set_control: setting control "Brightness" to 14 (ret 0 )
Corrupt JPEG data: 1 extraneous bytes before marker 0xd5
[1] [CRT] [VID] mjpegtoyuv420p: Corrupt image ... continue
[1] [NTC] [VID] v4l2_set_control: setting control "Brightness" to 36 (ret 0 )
Corrupt JPEG data: 3 extraneous bytes before marker 0xd2
[1] [CRT] [VID] mjpegtoyuv420p: Corrupt image ... continue
[1] [NTC] [VID] v4l2_set_control: setting control "Brightness" to 58 (ret 0 )
Corrupt JPEG data: 1 extraneous bytes before marker 0xd7
[1] [CRT] [VID] mjpegtoyuv420p: Corrupt image ... continue
[1] [NTC] [VID] v4l2_set_control: setting control "Brightness" to 80 (ret 0 )
Corrupt JPEG data: 4 extraneous bytes before marker 0xd7
[1] [CRT] [VID] mjpegtoyuv420p: Corrupt image ... continue
[1] [ERR] [ALL] motion_init: Error capturing first image
[1] [NTC] [ALL] image_ring_resize: Resizing pre_capture buffer to 16 items
Corrupt JPEG data: 4 extraneous bytes before marker 0xd1
[1] [CRT] [VID] mjpegtoyuv420p: Corrupt image ... continue
Corrupt JPEG data: 11 extraneous bytes before marker 0xd1
[1] [CRT] [VID] mjpegtoyuv420p: Corrupt image ... continue
Corrupt JPEG data: 3 extraneous bytes before marker 0xd4
[1] [CRT] [VID] mjpegtoyuv420p: Corrupt image ... continue
Corrupt JPEG data: 7 extraneous bytes before marker 0xd1
[..]

... and another issue is turning up :( The output above goes on and on and on and there is no video capturing. So accordingly to $searchengine the above happens to a lot of people. I just found one often suggested fix: pre-load v4l2convert.so from libv4l-0:

$ LD_PRELOAD=/usr/lib/x86_64-linux-gnu/libv4l/v4l2convert.so motion

But the problem persists and I'm out of ideas :( So atm it lokks like I cannot use the MJPEG format and don't get 30 fps at 1280x720 pixels. During writing I then discovered a solution by good old trial-and-error: Leaving the v4l2_palette variable at its default value 17 (YU12) and pre-loading v4l2convert.so makes use of YU12 and the framerate at least raises to 24 fps:

$ LD_PRELOAD=/usr/lib/x86_64-linux-gnu/libv4lg/v4l2convert.so motion
[..]
[1] [NTC] [VID] v4l2_do_set_pix_format: Testing palette YU12 (1280x720)
[1] [NTC] [VID] v4l2_do_set_pix_format: Using palette YU12 (1280x720) bytesperlines 1280 sizeimage 1382400 colorspace 00000008
[..]
[1] [NTC] [EVT] event_new_video FPS 24
[..]

Finally! :) The results are nice. It would maybe even be a good idea to limit the framerate a bit, to e.g. 20. So that is a tested configuration for the Logitech C910 running at a resolution of 1280x720 pixels:

v4l2_palette 17
width 1280
height 720
framerate 20
minimum_frame_time 0
pre_capture 10 # 0,5 seconds pre-recording
post_capture 50 # 2,5 seconds after-recording
auto_brightness on
ffmpeg_variable_bitrate 2 # best quality

Now all this made me curious, which framerate is possible at a resolution of 1920x1080 pixels now and how the results look like. Although I get 24 fps too, the resulting movie suffers of jumps every few frames. So here I got pretty good results with a more conservative setting. By increasing framerate - tested up to 15 fps with good results - pre_capture needed to be decreased accordingly to values between 1..3 to minimize jumps:

v4l2_palette 17
width 1920
height 1080
framerate 12
minimum_frame_time 0
pre_capture 6 # 0,5 seconds pre-recording
post_capture 30 # 2,5 seconds after-recording
auto_brightness on
ffmpeg_variable_bitrate 2 # best quality

Both configurations lead to satisfying results. Of course the latter will easily fill your hardrive :)

TODO

I guess, the results can be optimzed further by playing around with ffmpeg_bps and ffmpeg_variable_bitrate. Maybe then it is possible to record without jumps at higher framerates too(?). I also didn't test the various norm settings (PAL, NTSC, etc.).

Categories: Elsewhere

OpenLucius: Dependency injection in Drupal 8, an introduction.

Planet Drupal - Fri, 13/02/2015 - 10:45
Introduction

So, like a bunch of other Drupal people, we're also experimenting with Drupal 8; for our Drupal distro OpenLucius. Us being 'less is more'-developers, one aspect we really like is dependency injection.

Categories: Elsewhere

Steve McIntyre: Linaro VLANd v0.2

Planet Debian - Fri, 13/02/2015 - 07:53

I've been working on this for too long without really talking about it, so let's fix that now!

VLANd is a simple (hah!) python program intended to make it easy to manage port-based VLAN setups across multiple switches in a network. It is designed to be vendor-agnostic, with a clean pluggable driver API to allow for a wide range of different switches to be controlled together.

There's more information in the README file. I've just released v0.2, with a lot of changes included since the last release:

  • Massive numbers of bugfixes and code cleanups
  • Improve how we talk to the Cisco switches - disable paging on long output
  • Switch from "print" to "logging.foo" for messages, and add logfile support
  • Improved test suite coverage, and added core test scripts for the lab environment

I've demonstrated this code today in Hong Kong at the Linaro Connect event, and now I'm going on vacation for 4 weeks. Australia here I come! :-)

Categories: Elsewhere

Neil Williams: OpenTAC mailing list

Planet Debian - Fri, 13/02/2015 - 03:10

After the OpenTAC session at Linaro Connect, we do now have a mailing list to support any and all discussions related to OpenTAC. Thanks to Daniel Silverstone for the list.

List archive: http://listmaster.pepperfish.net/pipermail/opentac-vero-apparatus.com

More information on OpenTAC: http://wiki.vero-apparatus.com/OpenTAC

Categories: Elsewhere

Jimmy Berry: The woes of the testbot

Planet Drupal - Fri, 13/02/2015 - 01:43

For those not familiar with me, a little research should make it clear that I am the person behind the testbot deployed in 2008 that has revolutionized Drupal core development, stability, etc. and that has been running tens of thousands of assertions with each patch submitted against core and many contributed modules for 6 years.

My intimate involvement with the testbot came to a rather abrupt and unintended end several years ago due to a number of factors (which only a select few members of this community are clearly aware). After several potholes, detours, and bumps in the road, it became clear to me the impossibility of maintaining and enhancing the testbot under the policies and constraints imposed upon me.

Five years ago we finished writing an entirely new testing system, designed to overcome the technical obstacles of the current testbot and to introduce new features that would enable an enormous improvement in resource utilization that could then be used for new and more frequent QA.

Five years ago we submitted a proposal to the Drupal Association and key members of the community for taking the testbot to the next level, built atop the new testing system. This proposal was ignored by the Association and never evaluated by the community. The latter is quite puzzling to me given:

  • the importance of the testbot
  • the pride this open source community has in openly evaluating and debating literally everything (a healthy sentiment especially in the software development world)
  • I had already freely dedicated years of my life to the project.

The remainder of this read will:

  • list some of the items included in our proposal that were dismissed with prejudice five years ago, but since have been adopted and implemented
  • compare the technical merits of the new system (ReviewDriven) with the current testbot and a recent proposal regarding "modernizing" the testbot
  • provide an indication of where the community will be in five years if it does nothing or attempts to implement the recent proposal.

This read will not cover the rude and in some cases seemingly unethical behavior that led to the original proposal being overlooked. Nor will this cover the roller coaster of events that led up to the proposal. The intent is to focus on a technical comparison and to draw attention to the obvious disparity between the systems.

About Face

Things mentioned in our proposal that have subsequently been adopted include:

  • paying for development primarily benefiting drupal.org instead of clinging to the obvious falacy of "open source it and they will come"
  • paying for machine time (for workers) as EC2 is regularly utilized
  • utilizing proprietary SaaS solutions (Mollom on groups.drupal.org)
  • automatically spinning up more servers to handle load (e.g. during code sprints) which has been included in the "modernize" proposal
Comparison

The following is a rough, high-level comparison of the three systems that makes clear the superior choice. Obviously, this comparison does not cover everything.

table#testbot-comparison td { border: 1px solid white; } table#testbot-comparison td:nth-child(2), table#testbot-comparison td:nth-child(3), table#testbot-comparison td:nth-child(4) { width: 33% } table#testbot-comparison tr:nth-child(1), table#testbot-comparison td:nth-child(1) { font-weight: bold; font-size: 120%; } table#testbot-comparison td:nth-child(2) { background-color: #FFCC00; } table#testbot-comparison td:nth-child(3) { background-color: #D46A6A; color: white; } table#testbot-comparison td:nth-child(4) { background-color: #55AA55; color: white; }

Baseline Backwards modernization True step forward System Current qa.drupal.org "Modernize" Proposal ReviewDriven Status It's been running for over 6 years Does not exist Existed 5 years ago at ReviewDriven.com Complexity Custom PHP code and Drupal Does not make use of contrib code Mish mash of languages and environments: ruby, python, bash, java, php, several custom config formats, etc.

Will butcher a variety of systems from their intended purpose and attempt to have them all communicate

Adds a number of extra levels of communication and points of failure Minimal custom PHP code and Drupal

Uses commonly understood contrib code like Views Maintainability Learning curve but all PHP Languages and tools not common to Drupal site building or maintenance

Vast array of systems to learn and the unique ways in which they are hacked Less code to maintain and all familiar to Drupal contributors Speed Known; gets slower as test suite grows due to serial execution Still serial execution and probably slower than current as each separate system will add additional communication delay An order of magnitude faster thanks to concurrent execution

Limited by the slowest test case

*See below Extensibility (Plugins) Moderately easy, does not utilize contrib code so requires knowledge of current system Several components, one on each system used

New plugins will have to be able to pass data or tweak any of the layers involved which means writing a plugin may involve a variety of languages and systems and thus include a much wider breadth of required knowledge Much easier as it heavily uses commons systems like Views

Plugin development is almost entirely common to Drupal development:
define storage: Fields
define display: Views
define execution: CTools function on worker

And all PHP Security Runs as same user as web process Many more surfaces for attack and that require proper configuration Daemon to monitor and shutdown job process, lends itself to Docker style with added security 3rd party integration Basic RSS feeds and restricted XML-RPC client API Unknown Full Services module integration for public, versioned, read API and write for authorized clients Stability When not disturbed, has run well for years, primary causes of instability include ill-advised changes to the code base

Temporary and environment reset problems easily solved by using Docker containers with current code base Unknown but multiple systems imply more points of failure Same number of components as current system

Services versioning which allows components to be updated independently

Far less code as majority depends on very common and heavily used Drupal modules which are stable

2-part daemon (master can react to misbehaving jobs)

Docker image could be added with minimal effort as system (which predates Docker) is designed with same goals as Docker Resource utilization Entire test suite runs on single box and cannot utilize multiple machines for single patch Multiple servers with unshared memory resources due to variety of language environments

Same serial execution of test cases per patch which does not optimally utilize resources An order of magnitude better due to concurrent execution across multiple machines

Completely dynamic hardware; takes full advantage of available machines.

*See below Human interaction Manually spin up boxes; reduce load by turning on additional machines Intended to include automatic EC2 spin up, but does not yet exist; more points of failure due to multiple systems Additional resources are automatically turned on and utilized Test itself Tests could be run on development setup, but not within the production testbot Unknown Yes, due to change in worker design.

A testbot inside a testbot! Recursion! API Does the trick, but custom XML-RPC methods Unknown Highly flexible input configuration is similar to other systems built later like travis-ci

All entity edits are done using Services module which follows best practices 3rd party code Able to test security.drupal.org patches on public instance Unknown, but not a stated goal Supports importing VCS credentials which allows testing of private code bases and thus supports the business aspect to provide as a service and to be self sustaining

Results and configuration permissioned per user to allow for drupal.org results to be public on the same instance as private results Implemented plugins Simpletest, coder None exist Simpletest, coder, code coverage, patch conflict detection, reroll of patch, backport patch to previous branch Interface Well known; designed to deal with display of several 100K distinct test results; lacks revision history; display uses combination of custom code and Views Unknown as being built from scratch and not begun

Jenkins can not support this interface (in Jenkins terminology multiple 100K jobs) so will have to be written from scratch (as proposal confirms and was reason for avoiding Jenkins in past)

Jenkins was designed for small instances within businesses or projects, not a large central interface like qa.drupal.org Hierarchical results navigation from project, branch, issue, patch

Context around failed assertion (like diff -u)

Minimizes clutter, focuses on results of greatest interest (e.g. failed assertions); entirely built using Views so highly customizable

Simplified to help highlight pertinent information (even icons to quickly extract status)

Capable of displaying partial results as they are concurrently streamed in from the various workers Speed and Resource Utilization

Arguably one of the most important advantages of the ReviewDriven system is concurrency. Interestingly, after seeing inside Google I can say this approach is far more similar to the system Google has in place than Jenkins or anything else.

Systems like Jenkins and especially travis-ci, which for the purpose of being generic and simpler, do not attempt to understand the workload being performed. For example Travis simply asks for commands to execute inside a VM and presents the output log as the result. Contrast that with the Drupal testbot which knows the tests being run and what they are being run against. Why is this useful? Concurrency.

Instead of running all the test cases for a single patch on one machine, the test cases for a patch may be split out into separate chunks. Each chunk is processed on a different machine and the results are returned to the system. Because the system understands the results it can reassemble the chunked results in a useful way. Instead of an endlessly growing wait time as more tests are added and instead of having nine machines sitting idle while one machine runs the entire test suite all ten can be used on every patch. The wait time effectively becomes the time required to run the slowest test case. Instead of waiting 45 minutes one would only wait perhaps 1 minute. The difference becomes more exaggerated over time as more tests are added.

In addition to the enormous improvement in turnaround time which enables the development workflow to process much faster you can now find new ways to use those machine resources. Like testing contrib projects against core commits, or compatibility tests between contrib modules, or retesting all patches on commit to related project, or checking what other patches a patch will break (to name a few). Can you even imagine? A Drupal sprint where the queue builds up an order of magnitude more slowly and runs through the queue 40x faster?

Now imagine having additional resources automatically started when the need arises. No need to imagine...it works (and did so 5 years ago). Dynamic spinning up of EC2 resources which could obviously be applied to other services that provide an API.

This single advantage and the world of possibility it makes available should be enough to justify the system, but there are plenty more items to consider which were all implemented and will not be present in the proposed initiative solution.

Five Years Later

Five years after the original proposal, Drupal is left with a testbot that has languished and received no feature development. Contrast that with Drupal having continued to lead the way in automated testing with a system that shares many of the successful facets of travis-ci (which was developed later) and is superior in other aspects.

As was evident five years ago the testbot cannot be supported in the way much of Drupal development is funded since the testbot is not a site building component placed in a production site. This fact drove the development of a business model that could support the testbot and has proven to be accurate since the current efforts continue to be plagued by under-resourcing. One could argue the situation is even more dire since Drupal got a "freebie" so to speak with me donating nearly full-time for a couple of years versus the two spare time contributors that exist now.

On top of lack of resources the current initiative, whose stated goal is to "modernize" the testbot, is needlessly recreating the entire system instead of just adding Docker to the existing system. None of the other components being used can be described as "modern" since most pre-date the current system. Overall, this appears to be nothing more than code churn.

Assuming the code churn is completed some time far in the future; a migration plan is created, developed, and performed; and everything goes swimmingly, Drupal will have exactly what it has now. Perhaps some of the plugins already built in the ReviewDriven system will be ported and provide a few small improvements, but nothing overarching or worth the decade it took to get there. In fact the system will needlessly require a much rarer skill set, far more interactions between disparate components, and complexity to be understood just to be maintained.

Contrast that with an existing system that can run the entire test suite against a patch across a multitude of machines, seamlessly stitch the results together, and post back the result in under a minute. Contrast that with having that system in place five years ago. Contrast that with the whole slew of improvements that could have also been completed in the four years hence by a passionate, full-time team. Contrast that with at the very least deploying that system today. Does this not bother anyone else?

Contrast that with Drupal being the envy of the open source world, having deployed a solution superior to travis-ci and years earlier.

Please post feedback on drupal.org issue.

Tags:
Categories: Elsewhere

DrupalCon News: A place for Drupal.org at DrupalCon

Planet Drupal - Fri, 13/02/2015 - 00:19

DrupalCon Los Angeles will be the first Con where Drupal.org, home of Drupal and the Drupal community, has its very own track.

The track will feature presentations from the Drupal Association Engineering Team, where they share long and short term plans for website development, demo new and upcoming features, and gather community feedback.

A limited amount of spots are available for sessions submitted from the community. That’s where you come in.

Categories: Elsewhere

Mediacurrent: Efficient Drupal Development with Tmux and Tmuxinator

Planet Drupal - Thu, 12/02/2015 - 23:41

Have you ever wished you could just type one command and load up all of the things you need to work on for a project? Wouldn’t it be nice to have your terminal set up with the correct Drush alias, tailing the watchdog, with access to your servers just a couple keystrokes away? Sounds nice, right?

Categories: Elsewhere

Victor Kane: Setting up a Reusable and DurableDrupal Lean Process Factory - Presentation 2/11/2015 at DrupalCon Latin America 2015

Planet Drupal - Thu, 12/02/2015 - 22:52

[para español ver más abajo]

The purpose of the presentation was to describe how to use reusable tools and processes, tailored and in constant evolution, in order to finally defeat waterfall and guarantee delivered value in the development of websites and web applications.

The following main topics were covered in depth:

  • Kanban (not Scrum)
  • Project Inception and Vision
  • Team Kickoff
  • Development Workflow with Everything in Code
  • DevOps, Server Provisioning and Deployment
  • User Validation

Links to resources:

This is a huge amount of material, based on both my successful and unsuccessful experiences, and I earnestly hope it will help other web centered knowledge workers. If you have questions, please ask them on twitter @victorkane with hashtag #DurableDrupalLean. There were quite a few other fascinating and very good presentations on the subject of Process and DevOps, overlapping my own substantially and it should be very worthwhile to share them here: I greatly appreciate having had the opportunity to present at this incredibly important, fun and well-organized DrupalCon. See you all in Los Angeles and Rio de Janeiro!

read more

Categories: Elsewhere

Richard Hartmann: A Dance with Dragons

Planet Debian - Thu, 12/02/2015 - 22:51

Yesterday, I went to the Federal Office for Information Security (BSI) on an invitation to their "expert round-table on SDN".

While the initial mix of industry attendees was of.. varied technical knowledge.. I was pleasantly surprised by the level of preparation by the BSI. None of them were networkers, but they did have a clear agenda and a pretty good idea of what they wanted to know.

During the first round-table, they went through

  • This is our idea of what we think SDN is
  • Is SDN a fad or here to stay?
  • What does the industry think about SDN?
  • What are the current, future, and potential benefits of SDN?
  • What are the current, future, and potential risks of SDN?
  • How can SDN improve the security of critical infrastructure?
  • How can you ensure that the whole stack from hardware through data plane to control plane can be trusted?
  • How can critical parts of the SDN stack be developed in, or strongly influenced from, players in Germany or at least Europe?

Yes, some of those questions are rather basic and/or generic, but that was on purpose. The mix of clear expectations and open-ended questions was quite effective at getting at what they wanted to know.

During lunch, we touched on the more general topic of how to reach and interact with technical audiences, with regards to both networks and software. The obvious answer for initial contact in regards to networks was DENOG; which they didn't know about.

With software, the answer is not quite as simple. My suggestion was to engage in a positive way and thus build trust over time. Their clear advantage is that, contrary to most other services, their raison d'être is purely defensive and non-military so they can focus on audits, support of key pieces of software, and, most important of all, talk about their results. No idea if they will actually pursue this, but here's to hoping; we could all use more government players on the good side.

Categories: Elsewhere

Morpht: How to Use Custom Markers for OpenLayers

Planet Drupal - Thu, 12/02/2015 - 22:00

OpenLayers module is a popular solution for mapping in Drupal. The biggest benefit is the ability to use different map providers, complete Feature support and, last but not least, the simplicity of creating custom markers.

Categories: Elsewhere

DrupalCon News: Apply for a DrupalCon Grant or Scholarship

Planet Drupal - Thu, 12/02/2015 - 21:11

In 2014 we received over 200 DrupalCon grant and scholarship applications. Thanks to our generous sponsor contributions, we were able to get over 60 individuals to DrupalCon Austin and Amsterdam. This year, we hope to award even more!

If you need help getting to DrupalCon Los Angeles, and are an active Drupal contributor or community leader, we're here to help you make YOUR dreams of attending DrupalCon a reality. Apply for a Grant or Scholarship!

Categories: Elsewhere

Pages

Subscribe to jfhovinne aggregator