Planet Debian

Subscribe to Planet Debian feed
Planet Debian - http://planet.debian.org/
Updated: 19 min 45 sec ago

Russ Allbery: C TAP Harness 4.0

Sun, 08/05/2016 - 01:07

When I originally wrote my test framework for C, I used SOURCE and BUILD as the preprocessor symbols and environment variables that pointed to the source and build directories of the software being tested. Subsequent discussion and thought convinced me that I should have used some sort of prefix on those variables to distinguish from other uses.

This release starts the process of changing to C_TAP_SOURCE and C_TAP_BUILD instead. You now have to use the new names when setting preprocessor directives when building the test harness. For now, the test harness will set all four environment variables, but test code should switch to the new environment variables, since I'll drop the old SOURCE and BUILD variables in a later release.

I also fixed a missing va_end() call in is_double(), thanks to a report from Julien ÉLIE.

You can get the latest release from the C TAP Harness distribution page.

Categories: Elsewhere

Elena 'valhalla' Grandi: Pyra preorders

Sat, 07/05/2016 - 14:30
Pyra preorders

If you've met me at a conference you may have noticed that instead of a laptop I was using a handeld which looks like a laptop scaled down to nintendo DS size, the https://en.wikipedia.org/wiki/Openpandora.

I've used it as my main computing device while travelling for a few years, even for work (as a programmer)so happily that when EvilDragon announced at FOSDEM (link points to youtube video) https://www.youtube.com/watch?v=4T-w1KqrVsM that he was working on a successor device I started saving money for it even before I knew many details about the specs, other that they would have been way better than the Pandora ones (which is getting painful to use a browser on, because of its 256MB RAM).

http://social.gl-como.it/photos/valhalla/image/5a9fdfadf33e40e9e3517b18ca7cba68

Now this successor device is almost ready, they have opened the preorders https://www.dragonbox.de/en/45-pyra, and they have already reached the absolute minimum number of orders for mass production and are almost there for a more reasonable number of 1000 devices, so if you want a chance to get one of the first batch devices now it's time to visit their store.

A few highlights, from my point of view, include:

* It will run Debian with just a custom kernel/bootloader (and a few configuration only packages): most of the kernel mods are being submitted upstream, so maybe one day there won't even be a need for this kernel (but e.g. with Pandora upstream didn't accept the custom way they managed the keyboard; on the Pyra the keyboard is managed in a more standard way, but there may be other similar issues).

* It has been designed with modularity in mind: the CPU board is socketed on the main board and in the future upgrades may require just replacing the CPU board. I haven't read the details on the actual licensing, but it seems that the hardware design will be open enough that 3rd party boards may also be a possibility.

* just like on Pandora: real keyboard. hardware analog volume wheel. Huge user-replaceable battery (I don't think that there are any independent reviews of the pyra battery yet, but the one on the Pandora is still able to go through a day of FOSDEM — i.e. alternating often between on with wifi and suspendend — and only go down to 50% or so charge). Stylus (and 3d-printed quill) friendly touchscreen. Long term support from the producer.

* The 4G version has been designed in such a way that the GSM modem can be actually turned off (just like on the https://neo900.org/)

There are of course a few bad parts:

* PowerVR. The good news is that there is a risk that no 3d drivers will be available at all, and this means that the Pyra has been tested and considered good enough with just (FOSS) software acceleration.

* The price: yes, it is expensive. I'm happy I've saved money in advance for it, otherwise I wouldn't have been able to afford it. Some of it is a problem of small production, some is actual product quality. If you consider that it can take the place of both a laptop (and small ones are getting quite expensive, now that netbooks have disappeared) and a smartphone (I don't do lots of voice calls) it will start going down from "oh so **** high" to "high, but not unreasonably so"

Disclaimer: I have preordered one, so I am interested in the success of the project because it will mean better software and better support for the device.

Edit: forgot the link to the press kit the images comes from http://cloud.openpandora.org/index.php/s/a0Q0TXV8gh5NLAC?path=%2F, which also includes more infos on specs etc.
Categories: Elsewhere

Craig Small: Displaying Linux Memory

Sat, 07/05/2016 - 14:00

Memory management is hard, but RAM management may be even harder.

Most people know the vague overall concept of how memory usage is displayed within Linux. You have your total memory which is everything inside the box; then there is used and free which is what the system is or is not using respectively. Some people might know that not all used is used and some of it actually is free.  It can be very confusing to understand, even for a someone who maintains procps (the package that contains top and free, two programs that display memory usage).

So, how does the memory display work?

What free shows

The free program is part of the procps package. It’s central goal is to give a quick overview of how much memory is used where. A typical output (e.g. what I saw when I typed “free -h”) could look like this:

total used free shared buff/cache available Mem: 15G 3.7G 641M 222M 11G 11G Swap: 15G 194M 15G

I’ve used the -h option for human-readable output here for the sake of brevity and because I hate typing long lists of long numbers.

People who have good memories (or old computers) may notice there is a missing “-/+ buffers/cache” line. This was intentionally removed in mid-2014 because as the memory management of Linux got more and more complicated, these lines became less relevant. These used to help with the “not used used memory” problem mentioned in the introduction but progress caught up with it.

To explain what free is showing, you need to understand some of the underlying statistics that it works with. This isn’t a lesson on how Linux its memory (the honest short answer is, I don’t fully know) but just enough hopefully to understand what free is doing. Let’s start with the two simple columns first; total and free.

Total Memory

This is what memory you have available to Linux. It is almost, but not quite, the amount of memory you put into a physical host or the amount of memory you allocate for a virtual one. Some memory you just can’t have; either due to early reservations or devices shadowing the memory area. Unless you start mucking around with those settings or the virtual host, this number stays the same.

Free Memory

Memory that nobody at all is using. They haven’t reserved it, haven’t stashed it away for future use or even just, you know, actually using it.  People often obsess about this statistic but its probably the most useless one to use for anything directly. I have even considered removing this column, or replacing it with available (see later what that is) because of the confusion this statistic causes.

The reason for its uselessness is that Linux has memory management where it allocates memory it doesn’t use. This decrements the free counter but it is not truly “used”. If you application needs that memory, it can be given back.

A very important statistic to know for running a system is how much memory have I got left before I either run out or I start to serious swap stuff to swap drives. Despite its name, this statistic will not tell you that and will probably mislead you.

My advice is unless you really understand the Linux memory statistics, ignore this one.

Who’s Using What

Now we come to the components that are using (if that is the right word) the memory within a system.

Shared Memory

Shared memory is often thought of only in the context of processes (and makes working out how much memory a process uses tricky – but that’s another story) but the kernel has this as well. The shared column lists this, which is a direct report from the Shmem field in the meminfo file.

Slabs

For things used a lot within the kernel, it is inefficient to keep going to get small bits of memory here and there all the time. The kernel has this concept of slabs where it creates small caches for objects or in-kernel data strucutures that slabinfo(5) states  “[such as] buffer heads, inodes and dentries”. So basically kernel stuff for the kernel to do kernelly things with.

Slab memory comes in two flavours. There is reclaimable and unreclaimable. This is important because unreclaimable cannot be “handed back” if your system starts to run out of memory. Funny enough, not all reclaimable is, well, reclaimable. A good estimate is you’ll only get 50% back, top and free ignore this inconvenient truth and assume it can be 100%. All of the reclaimable slab memory is considered part of the Cached statistic. Unreclaimable is memory that is part of Used.

Page Cache and Cached

Page caches are used to read and write to storage, such as a disk drive. These are the things that get written out when you use sync and make the second read of the same file much faster. An interesting quirk is that tmpfs  is part of the page cache. So the Cached column may increase if you have a few of these.

The Cached column may seem like it should only have Page Cache, but the Reclaimable part of the Slab is added to this value. For some older versions of some programs, they will have no or all Slab counted in Cached. Both of these versions are incorrect.

Cached makes up part of the buff/cache column with the standard options for free or has a column to itself for the wide option.

Buffers

The second component to the buff/cache column (or separate with the wide option) is kernel buffers. These are the low-level I/O buffers inside the kernel. Generally they are small compared to the other components and can basically ignored or just considered part of the Cached, which is the default for free.

Used

Unlike most of the previous statistics that are either directly pulled out of the meminfo file or have some simple addition, the Used column is calculated and completely dependent on the other values. As such it is not telling the whole story here but it is reasonably OK estimate of used memory.

Used component is what you have left of your Total memory once you have removed:

  • Free memory – because free is not used!
  • The Cached value – recall this is made up of the Page Cache plus the Reclaimable part of Slab
  • The buffers

Notice that the unreclaimable part of slab is not in this calculation, which means it is part of the used memory. Also note this seems a bit of a hack because as the memory management gets more complicated, the estimates used become less and less real.

Available

In early 2014, the kernel developers took pity on us toolset developers and gave us a much cleaner, simpler way to work out some of these values (or at least I’d like to think that’s why they did it). The available statistic is the right way to work out how much memory you have left. The commit message explains the gory details about it, but the great thing is that if they change their mind or add some new memory feature the available value should be changed as well. We don’t have to worry about should all of slab be in Cached and are they part of Used or not, we have just a number directly out of meminfo.

What does this mean for free?

Poor old free is now at least 24 years old and it is based upon BSD and SunOS predecessors that go back way before then. People expect that their system tools don’t change by default and show the same thing over and over. On the other side, Linux memory management has changed dramatically over those years. Maybe we’re all just sheep (see I had to mention sheep or RAMs somewhere in this) and like things to remain the same always.

Probably if free was written now; it would only need the total, available and used columns with used merely being total minus available. Possibly with some other columns for the wide option.

The code itself (found in libprocps) is not very hard to maintain so its not like this change will same some time but for me I’m unsure if free is giving the right and useful result for people that use it.

 

Categories: Elsewhere

Dirk Eddelbuettel: BH 1.60.0-2

Sat, 07/05/2016 - 05:41

A new minor release of BH is now on CRAN. BH provides a large part of the Boost C++ libraries as a set of template headers for use by R, possibly with Rcpp as well as other packages.

This release uses the same Boost 1.60.0 version of Boost as the last release, but adds three more library: bimap, flyweight and icl.

A brief summary of changes from the NEWS file is below.

Changes in version 1.60.0-2 (2016-05-06)
  • Added Boost bimap via GH pull request #24 by Jim Hester.

  • Added Boost icl via GH pull request #27 by Jay Hesselbert.

  • Added Boost flyweight as requested in GH ticket #26.

Courtesy of CRANberries, there is also a diffstat report for the most recent release.

Comments and suggestions are welcome via the mailing list or the issue tracker at the GitHubGitHub repo.

This post by Dirk Eddelbuettel originated on his Thinking inside the box blog. Please report excessive re-aggregation in third-party for-profit settings.

Categories: Elsewhere

Scarlett Clark: Kubuntu: Debian: KDE: Outreachy! Yay! Upcoming changes

Fri, 06/05/2016 - 20:30

I was approved to participate in Outreachy!
https://www.gnome.org/outreachy/

I will be working on the Debian reproducible builds.
https://wiki.debian.org/ReproducibleBuilds

Some of you in the Debian community might know me from working with the debian-qt-kde team
as we (Kubuntu) collaborated in packaging for some time.

Will this affect my current volunteer work for my current projects? Absolutely.
I am currently putting in 10-18 hours a day in volunteer work. Obviously, I cannot give up sleep.
I love all of my projects dearly, but after some recent events, I have found that I absolutely must
resurrect my career. AKA I need to make money to be able to stand on my own two feet without relying on
someone else. I do appreciate the donations, but they do not even dent my student loans :-).
So with all of that said, this will be a positive move in my career path.

I have not sorted out how my current projects contributions will change, but I can tell you
Debian reproducible builds will be number one priority come May 20th. This will last for 3 months.
Then my focus will be on getting a paying job.

Please understand, I need this. One needs money to survive, and working for donations is not working.
I feel the Outreachy work will improve my chances at obtaining a paid job in the open source field.
I have been chasing my “dream job” for some time now, and it is time to go get it.

I will post again, when I have sorted out how I will split up my time.
Thank you my friends.

Categories: Elsewhere

Russ Allbery: Review: The Language of Power

Fri, 06/05/2016 - 05:15

Review: The Language of Power, by Rosemary Kirstein

Series: Steerswomen #4 Publisher: Rosemary Kirstein Copyright: 2004, 2014 Printing: April 2014 ISBN: 0-9913546-3-X Format: Kindle Pages: 400

This is the fourth book in the Steerswomen series and definitely not the place to start. It's also a difficult series to review without spoilers, so I won't be able to provide too many details about the plot.

I will say that this is a reunion and a return of sorts to themes from earlier in the series, rather than a direct follow-up to the revelations at the end of The Lost Steersman. Rowan is back in the Inner Lands, continuing to investigate the affairs of wizards. In particular, she's digging into the past of the town of Donner, following up on the report of an earlier steerswoman and investigating a now-dead wizard who seemed to act far different from a typical wizard. And Bel is back at her side again, watching her back.

The first half of The Language of Power goes over somewhat familiar ground. Similar to both The Steerswoman and The Lost Steersman, Rowan is poking around in a city, getting to know unfamiliar people, being a steerswoman, and winning people over with her unique charm. But that's a theme I don't mind seeing repeated, since Rowan is one of my favorite protagonists from any series I've read. She's both ethical and respectful in a way that doesn't feel artificial or constructed. She thinks oddly and dives into sudden fascinations, and she does rely on the conventions for interacting with steerswomen, but the more time one spends with her, the better one likes her. This is true of both the reader and the town inhabitants, and Kirstein is brilliant at writing the gradual getting-to-know and growing-respect process.

Events in The Language of Power slowly build up to another confrontation with wizards, and this one is full of major revelations about the world. Some of the ambiguity of earlier books is firmly resolved, we find out a lot more about how wizards view themselves and their abilities, and tons of new questions are raised. It's not a conclusion in any way, which is a bit unfortunate given that the next book is still being written (twelve years later, although thankfully it's being actively worked on as I write this). But we get the first clear look at the substratum of the world that Kirstein is building in this series.

This sounds satisfying, and to some extent it is, but any regular SFF reader will have guessed at many of the revelations here. I was pretty sure the world was following one of two possible patterns partway through the second book, certain which it was during the third book, and was nodding right along with the revelations in this book. I'm trying to avoid spoilers, but if you read a lot of SFF, chances are you've read about something akin to this background before. That takes a bit of the thrill out of the revelations, unfortunately.

What adds the excitement and thrill back in are Rowan's reactions. It's very difficult to write a character who comes from an entirely different perspective than either the author or the reader, and Kirstein does an amazing job. Not perfect, quite, at least for me: there were a few points where I thought Rowan was more baffled or more upset than it felt like she should have been. But they are few and far between, and it's quite possible my expectations are the ones that wouldn't ring true if written into the story. It's just such a delight to see Rowan analyzing the world, incorporating new revelations into her growing world model, and figuring out how to take the most moral action at any point. I would happily read another dozen books of this (but I wish they were all already written).

If you've read the previous three books, definitely pick up this one as well. For me, it narrowly misses being the best book of the series (I think that's still The Outskirter's Secret because of the difficulty of the perspective change Kirstein pulls off), but it's a close competition. And the final reveal at the very end of this book points to upcoming adventures that I can hardly wait to read.

Rating: 8 out of 10

Categories: Elsewhere

Dirk Eddelbuettel: RcppArmadillo 0.6.700.6.0

Fri, 06/05/2016 - 04:18

A second Armadillo release 6.700.6 came out in the 6.700 series, and we uploaded RcppArmadillo 0.6.700.6.0 to CRAN and Debian. This followed the usual thorough reverse-dependecy checking of by now 220 packages using.

This release is a little unusual in that it contains both upstream bugfixes in the same series (see below) but also two nice bug fixes from the RcppArmadillo side. Both were squashed by George G. Vega Yon via two focused pull request. The first ensures that we can now use ARMA_64BIT_WORD (provided C++11 is turned on too) allowing for much bigger Armadillo objects. And the second plugs a small leak in the sparse matrix converter I had added a while back. Nice work, all told!

Armadillo is a powerful and expressive C++ template library for linear algebra aiming towards a good balance between speed and ease of use with a syntax deliberately close to a Matlab.

Changes in this release are as follows:

Changes in RcppArmadillo version 0.6.700.6.0 (2016-05-05)
  • Upgraded to Armadillo 6.700.6 (Catabolic Amalgamator Deluxe)

    • fix for handling empty matrices by kron()

    • fix for clang warning in advanced matrix constructors

    • fix for false deprecated warning in trunc_log() and trunc_exp()

    • fix for gcc-6.1 warning about misleading indentation

    • corrected documentation for the solve() function

  • Added support for int64_t (ARMA_64BIT_WORD) when required during compilation time. (PR #90 by George G. Vega Yon, fixing #88)

  • Fixed bug in SpMat exporter (PR #91 by George G. Vega Yon, fixing #89 and #72)

Courtesy of CRANberries, there is also a diffstat report for this release. As always, more detailed information is on the RcppArmadillo page. Questions, comments etc should go to the rcpp-devel mailing list off the R-Forge page.

This post by Dirk Eddelbuettel originated on his Thinking inside the box blog. Please report excessive re-aggregation in third-party for-profit settings.

Categories: Elsewhere

Sean Whitton: dh_make_elpa & dh_elpa_test

Thu, 05/05/2016 - 20:08

I recently completed and released some work on Debian’s tooling for packaging Emacs Lisp addons to GNU Emacs. Emacs grew a package manager called package.el a few years ago, and last year David Bremner wrote the dh_elpa tool to simplify packaging addons for Debian by leveraging package.el features. Packaging a series of addons for Debian left me with a wishlist of features for dh_elpa and I was recently able to implement them.

Debian tooling generally uses Perl, a language I didn’t know before starting on this project. I was fortunate enough to receive a free review copy of Perl 5 by Example when I attended a meeting of the Bay Area Linux Users Group while I was visiting San Francisco a few months ago. I accepted the book with the intent of doing this work.

dh_make_elpa

dh_make_elpa (at present available from Debian experimental) is a Perl script to convert a git repository cloned from the upstream of an Emacs Lisp addon to a rudimentary Debian package. It performs a lot of guesswork, and its simple heuristics are something I hope to improve on. Since I am new to object-oriented program design in Perl and I wanted to leverage object-oriented Debian tooling library code, I took the structure of my project from dh_make_perl. In this manner I found it easy and pleasant to write a maintainable script.

dh_elpa_test

A lot of Emacs Lisp addon packages use a program called Cask to manage the Emacs Lisp dependencies needed to run their test suites. That meant that dh_auto_test often fails to run Emacs Lisp addon package test suites. Since the Debian packaging toolchain already has advanced dependency management, it’s undesirable to involve Cask in the package build pipeline if it can be avoided. I had been copying and pasting the code needed to make the tests run in our environment to the debian/rules files of each package whose test suite I wanted to run.

dh_elpa_test tries to detect Emacs Lisp addon package test suites and run them with the workarounds needed in our environment. This avoids boilerplate in debian/rules. dh_elpa_test also disables dh_auto_test to avoid a inadvertent Cask invocation.

Future & acknowledgements

My hope for this work was to make it easier and faster to package Emacs Lisp addon packages for Debian, for my own sake and for anyone new who is interested in joining the pkg-emacsen team. In the future, I want to have dh_elpa_test generate an autopkgtest definition so that a Testsuite: pkg-emacsen line in debian/control is enough to have an Emacs Lisp addon package test suite run on Debian CI.

I’m very grateful to David Bremner for reviewing and supporting this work, and also for supporting my Emacs Lisp addon packaging work more generally.

Categories: Elsewhere

Petter Reinholdtsen: The Pyra - handheld computer with Debian preinstalled

Wed, 04/05/2016 - 10:00
A friend of mine made me aware of The Pyra, a handheld computer which will be delivered with Debian preinstalled. I would love to get one of those for my birthday. :)

The machine is a complete ARM-based PC with micro HDMI, SATA, USB plugs and many others connectors, and include a full keyboard and a 5" LCD touch screen. The 6000mAh battery is claimed to provide a whole day of battery life time, but I have not seen any independent tests confirming this. The vendor is still collecting preorders, and the last I heard last night was that 22 more orders were needed before production started.

As far as I know, this is the first handheld preinstalled with Debian. Please let me know if you know of any others. Is it the first computer being sold with Debian preinstalled?

Categories: Elsewhere

Debian Java Packaging Team: What's new since Jessie?

Wed, 04/05/2016 - 09:55

Jessie was released one year ago now and the Java Team has been busy preparing the next release. Here is a quick summary of the current state of the Java packages:

  • A total of 136 packages have been added, 63 removed, 213 upgraded to a new upstream release, and 145 updated. We are now maintaining 892 packages (+12.34%).
  • OpenJDK 8 is now the default Java runtime in testing/unstable. OpenJDK 7 has been removed, as well as several packages that couldn't be upgraded to work with OpenJDK 8 (avian, eclipse).
  • OpenJDK 9 is available in experimental. As a reminder, it won't be part of the next release; OpenJDK 8 will be the only Java runtime supported for Stretch.
  • Netbeans didn't make it into Jessie, but it is now back and up to date.
  • The main build tools are close to their latest upstream releases, especially Maven and Gradle which were seriously lagging behind.
  • Scala has been upgraded to the version 2.11. We are looking for Scala experts to maintain the package and its dependencies.
  • Freemind has been removed due to lack of maintenance, Freeplane is recommended instead.
  • The reproducibility rate has greatly improved, climbing from 50% to 75% in the past year.
  • Backports are continuously provided for the key packages and applications: OpenJDK 8, OpenJFX, Ant, Maven, Gradle, Tomcat 7 & 8, Jetty 8 & 9, jEdit.
  • The transition to Maven 3 has been completed, and packages are no longer built with Maven 2.
  • We replaced several obsolete libraries and transitioned them to their latest versions - for example, asm2, commons-net1 and commons-net2. Groovy 1.x was replaced with Groovy 2, and we upgraded BND, an important tool to develop with OSGi, and more than thirty of its reverse-dependencies from the 1.x series to version 2.4.1.
  • New packaging tools have been created to work with Gradle (gradle-debian-helper) and Ivy (ivy-debian-helper).
Outlook, goals and request for help
  • We have several difficult transitions ahead: BND 3, Tomcat 7 to 8, Jetty 8 to 9, ASM 5, and of course Java 9. Any help would be welcome.
  • Eclipse is severely outdated and currently not part of testing. We would like to update this important piece of software and its corresponding modules to the latest upstream release, but we need more active people who want to maintain them. If you care about the Eclipse ecosystem, please get in touch with us.
  • We still are in the midst of removing old libraries like asm3, commons-httpclient and the servlet 2.5 API, which is part of the Tomcat 6 source package.
  • Want to see Azureus/Vuze in Stretch again? Packaging is almost complete but we are looking for someone who can clarify remaining licensing issues with upstream and wants to maintain the software for the foreseeable future.
  • Do you have more ideas and want to get involved with the Java Team? Just send your suggestions to debian-java@lists.debian.org or chat with us on IRC at irc.debian.org, #debian-java.
Java and Friends
  • The Java Team is not the only team that maintains Java software in Debian. DebianMed, DebianScience and the Android Tools Maintainers rely heavily on Java. By helping the Java Team and working together, you can improve the Java ecosystem and further the efforts of multiple other fields of endeavor all at once.
Package updates

The packages listed below detail the changes in jessie-backports and testing. Libraries and Debian specific tools have been excluded.

Packages added to jessie-backports:

Packages removed from testing:

Packages added to testing:

Packages upgraded in testing:

Categories: Elsewhere

Junichi Uekawa: Fighting with my emacs configuration.

Wed, 04/05/2016 - 02:04
Fighting with my emacs configuration. I'm trying to get a nice emacs terminal, and trying to set up 256 color mode in screen. This is so hard.

Categories: Elsewhere

Martín Ferrari: New sources for contributors.debian.org

Wed, 04/05/2016 - 01:59

Many people might not be aware of it, but since a couple of years ago, we have an excellent tool for tracking and recognising contributors to the Debian Project: Debian Contributors

Debian is a big project, and there are many people working that do not have great visibility, specially if they are not DDs or DMs. We are all volunteers, so it is very important that everybody gets credited for their work. No matter how small or unimportant they might think their work is, we need to recognise it!

One great feature of the system is that anybody can sign up to provide a new data source. If you have a way to create a list of people that is helping in your project, you can give them credit!

If you open the Contributors main page, you will get a list of all the groups with recent activity, and the people credited for their work. The data sources page gives information about each data source and who administers it.

For example, my Contributors page shows the many ways in which the system recognises me, all the way back to 2004! That includes commits to different projects, bug reports, and package uploads.

I have been maintaining a few of the data sources that track commits to Git and Subversion repositories:

The last two are a bit problematic, as they group together all commits to the respective VCS repositories without distinguishing to which sub-projects the contributions were made.

The Go and Perl groups' contributions are already extracted from that big pile of data, but it would be much nicer if each substantial packaging team had their own data source. Sadly, my time is limited, so this is were you come into the picture!

If you are a member of a team, and want to help with this effort, adopt a new data source. You can be providing commit logs, but it is not limited to that; think of translators, event volunteers, BSP attendants, etc.

The initial work is very small, and there is almost no maintenance. There is information on how to contribute here and here, but I would be more than happy to guide you if you contact me.

Comment

Categories: Elsewhere

Neil Williams: Moving to Pelican

Wed, 04/05/2016 - 00:15

Prompted by Tollef, moving to Hugo, I investigated a replacement blog engine. The former site used Wordpress which is just overhead - my blog doesn't need to be generated on every view, it doesn't need the security implications of yet another website login and admin interface either.

The blog is static, so I've been looking at static generators. I didn't like the look of Hugo and wanted something where the syntax was familiar - so either Jinja2 or ReST.

So, I've chosen Pelican with the code living in a private git repo, naturally. I wanted a generator that was supported in Jessie. I first tried nikola but it turns out that nikola in jessie has syntax changes. I looked at creating backports but then there is a new upstream release which adds a python module not yet in Debian, so that would be an extra amount of work.

Hopefully, this won't flood planet - I've gone through the RSS content to update timestamps but the URLs have changed.

Categories: Elsewhere

Carl Chenet: Feed2tweet, your RSS feed to Twitter Python self-hosted app

Tue, 03/05/2016 - 18:15

Feed2tweet is a self-hosted Python app to send you RSS feed to Twitter.

Feed2tweet is in production for Le Journal du hacker, a French Hacker News-style FOSS website and LinuxJobs.fr, the job board of the French-speaking FOSS community.

Feed2tweet 0.3 now only runs with Python 3.  It also fixes a nasty bug with RSS feeds modifying the RSS entry orders. Have a look at the Feed2tweet 0.3 changelog:

Using Feed2tweet? Send us bug reports/feature requests/push requests/comments about it!


Categories: Elsewhere

Jamie McClelland: Monitoring Deflect

Tue, 03/05/2016 - 15:41

May First/People Link has several members that are targets of politically motivated denial of service attacks (mostly groups that support reproductive justice for women and palestinian rights). To fight off the attacks, we work closely with Deflect - a non-governmental organization based in Canada that fights against this kind of censorship.

When a site is down, it's not always easy to understand why. Deflect runs as many as 5 edge servers, any of them could be down. And, of course, the origin server could also be down.

I tried using a commericial/free as in beer service for monitoring up time, but when it reported the site being down, I had no idea which part was down.

So... httping to the rescue. Unfortunately, it depends on --divert-connect which is only available in Debian Stretch. I run the script via a cron job and output the results to a log file.

#!/bin/bash # Test all given edges domain="$1" origin="$2" proto=http if [ -n "$3" ]; then proto="$3" fi if [ -z "$domain" ]; then printf "Please pass the domain as first argument.\n" exit 1 fi if ! ping -c 1 4.2.2.1 >/dev/null; then # printf "We are offline. Not running.\n" exit 1 fi ips=$(dig +short "$domain") if [ "$?" -ne "0" ]; then # printf "DNS lookup failure. Not running.\n" exit 1 fi if [ -n "$origin" ]; then ips="$ips $origin" fi l= if [ "$proto" = "https" ]; then l=-l fi for ip in $ips; do date=$(date +%Y.%m.%d-%H:%M) for i in 1 2 3; do out=$(httping $l -m -t 5 -c 1 --divert-connect "$ip" "$proto://$domain") [ -z "$out" ] && out=1 printf "%s %s %s\n" "$date" "$ip" "$out" done done
Categories: Elsewhere

Raphaël Hertzog: My Free Software Activities in April 2016

Tue, 03/05/2016 - 09:13

My monthly report covers a large part of what I have been doing in the free software world. I write it for my donators (thanks to them!) but also for the wider Debian community because it can give ideas to newcomers and it’s one of the best ways to find volunteers to work with me on projects that matter to me.

Debian LTS

I handled a new LTS sponsor that wanted to see wheezy keep supporting armel and armhf. This was not part of our initial plans (set during last Debconf) and I thus mailed all teams that were impacted if we were to collectively decide that it was OK to support those architectures. While I was hoping to get a clear answer rather quickly, it turns out that we never managed to get an answer to the question from all parties. Instead the discussion drifted on the more general topic of how we handle sponsorship/funding in the LTS project.

Fortunately, the buildd maintainers said they were OK with this and the ftpmasters had no objections, and they both implicitly enacted the decision: Ansgar Burchardt kept the armel/armhf architectures in the wheezy/updates suite when he handled the switch to the LTS team, and Aurélien Jarno also configured wanna-build to keep building armel/armhf for the suite. The DSA team did not confirm that this change was not interfering with one of their plans to decommission some hardware. Build daemons are a shared resource anyway and a single server is likely to handle builds for multiple releases.

DebConf 16

This month I registered for DebConf 16 and submitted multiple talk/BoF proposals:

  • Kali Linux’s Experience of a Debian Derivative Based on Testing (Talk)
  • 2 Years of Work of Paid Contributors in the Debian LTS Project (Talk)
  • Using Debian Money to Fund Debian Projects (BoF)

I want to share the setup we use in Kali as it can be useful for other derivatives and also for Debian itself to help smooth the relationship with derivatives.

I also want to open again the debate on the usage of money within Debian. It’s a hard topic but we should really strive to take some official position on what’s possible and what’s not possible. With Debian LTS and its sponsorship we have seen that we can use money to some extent without hurting the Debian project as a whole. Can this be transposed to other teams or projects? What are the limits? Can we define a framework and clear rules? I expect the discussion to be very interesting in the BoF. Mehdi Dogguy has agreed to handle this BoF with me.

Packaging

Django. I uploaded 1.8.12 to jessie-backports and 1.9.5 to unstable. I filed two upstream bugs (26473 and 26474) for two problems spotted by lintian.

Unfortunately, when I wanted to upload it to unstable, the test suite did not ran. I pinned this down to a sqlite regression. Chris Lamb filed #820225 and I contacted the SQLite and Django upstream developers by email to point them to this issue. I helped the SQLite upstream author (Richard Hipp) to reproduce the issue and he was quick to provide a patch which landed in 3.12.1.

Later in the month I made another upload to fix an upgrade bug (#821789).

GNOME 3.20. As for each new version, I updated gnome-shell-timer to ensure it works with the new GNOME. This time I spent a bit more time to fix a regression (805347) that dates back to a while and that would never be fixed otherwise since the upstream author orphaned this extension (as he no longer uses GNOME).

I have also been bitten by display problems where accented characters would be displayed below the character that follows. With the help of members of the GNOME team, we found out that this was a problem specific to the cantarell font and was only triggered with Harfbuzz 1.2. This is tracked in Debian with #822682 on harfbuzz and #822762 in fonts-cantarell. There’s a new upstream release (with the fix) ready to be packaged but unfortunately it is blocked by the lack of a recent fontforge in Debian. I thus mailed debian-mentors in the hope to find volunteers to help the pkg-fonts team to package a newer version…

Misc Debian/Kali work

Distro Tracker. I started to mentor Vladimir Likic who contacted me because he wants to contribute to Distro Tracker. I helped him to setup his development environment and we fixed a few issues in the process.

Bug reports. I filed many bug reports, most of them due to my work on Kali:

  • #820288: a request to keep the wordpress package installable in older releases (due to renaming of many php packages)
  • #820660: request support of by-hash indices in reprepro
  • #820867: possibility to apply overrides on already installed packages in reprepro
  • #821070: jessie to stretch upgrade problem with samba-vfs-modules
  • #822157: python-future hides and breaks python-configparser
  • #822669: dh_installinit inserts useless autoscript for System V init script when package doesn’t contain any
  • #822670: dh-systemd should be merged into debhelper, we have systemd by default and debhelper should have proper support for it by default

I also investigated #819958 that was affecting testing since it has been reported to Kali as well. And I made an NMU of dh-make-golang to fix #819472 that I reported earlier.

Thanks

See you next month for a new summary of my activities.

No comment | Liked this article? Click here. | My blog is Flattr-enabled.

Categories: Elsewhere

Russ Allbery: Review: The Effective Engineer

Tue, 03/05/2016 - 05:59

Review: The Effective Engineer, by Edmond Lau

Publisher: Effective Bookshelf Copyright: 2015 ISBN: 0-9961281-0-7 Format: Trade paperback Pages: 222

Silicon Valley start-up tech companies have a standard way of thinking about work. Large chunks of this come from Google, which pioneered a wide variety of new, or at least not-yet-mainstream, ways of organizing and thinking about work. The rest accreted through experience with fast-paced start-ups, engineer-focused companies, web delivery of products, and rabid turnover and high job mobility within a hothouse of fairly similar companies. A key part of this mindset is the firm belief that this atmosphere has created a better way to work, at least for software engineers (and systems administrators, although heaven forbid that one call them that any more): more effective, more efficient, more focused on what really matters.

I think this is at least partly true, at least from the perspective of a software engineer. This Silicon Valley work structure focuses on data gathering, data-based decision-making, introspection, analysis, and continuous improvement, all of which I think are defensibly pointed in the right direction (if rarely as rigorous as one might want to believe). It absorbs bits and pieces of work organization techniques that are almost certainly improvements for the type of work software engineers do: Agile, Lean, continuous deployment, and fast iteration times.

In other cases, though, I'm less convinced that this Silicon Valley consensus is objectively better as opposed to simply different; interviewing, for instance, is a puzzle that I don't think anyone has figured out, and the remarkable consensus in Silicon Valley on how to interview (basically, "like Google except for the bits we thought were obnoxious") feels more like a social fad than a sign of getting it right. But every industry has its culture of good ideas, bad ideas, fads, and fashion, and it's quite valuable to know that culture if you want to work in that industry.

The Effective Engineer is a self-published book by Edmund Lau, a Silicon Valley software engineer who also drifted (as is so common in Silicon Valley) into mentoring, organizing, and speaking to other software engineers. Its purpose, per the subtitle, is to tell you "how to leverage your efforts in software engineering to make a disproportionate and meaningful impact." While that's not exactly wrong, and the book contains some useful and valuable tips, I'd tend to give it a slightly different subtitle: "a primer on how a Silicon Valley software engineer is expected to think about their work." This is a bit more practical, a bit less confident, and a bit less convinced of its own correctness than Lau might want to present his work, but it's just as valuable of a purpose if you want to work in the industry. (And is a bit more honest about its applicability outside of that industry.)

What this book does extremely well is present, in a condensed, straightforward, and fast-moving form, most of the highlights of how start-ups and web-scale companies approach software engineering and the SWE role in companies (SWE, meaning software engineer, is another bit of Google terminology that's now nearly universal). If you've already worked in or around this industry for a while, you've probably picked up a lot of this via osmosis: prioritize based on impact and be unapologetic about letting other things drop, have a growth mindset, reprioritize regularly, increase your iteration speed, measure everything constantly, check your assumptions against data, derisk your estimates, use code review and automated testing (but not too much), automate operations, and invest heavily in hiring and onboarding. (The preceding list is a chapter list for this book.) If you're working at one of these sorts of companies, you're probably currently somewhere between nodding and rolling your eyes because no one at work will shut up about these topics. But if you've not worked inside one of these companies, even if you've done software engineering elsewhere, this is a great book to read to prepare yourself. You're going to hear about these ideas constantly, and, if it achieves nothing else at all, The Effective Engineer will give you a firm enough grounding in the lingo and mindset that you can have intelligent conversations with people who assume this is the only way to think about software engineering.

By this point, you might be detecting a certain cynicism in this review. It's not entirely fair: a lot of these ideas are clearly good ones, and Lau does a good job of describing them quickly and coherently. It's a good job for what it is. But there are a couple of things that limited its appeal for me.

First, it's definitely a primer. I read it after having worked at a web-scale start-up for a year and a half. There wasn't much in it that seemed particularly new, and it's somewhat superficial. The whole middle section in particular (build tools for yourself, measure everything, be data-driven) are topics for which the devil is often in the details. Lau gives you the terminology and the expected benefits, but putting any one of these techniques into practice could be a book (or several) by itself. Don't expect to come away from The Effective Engineer with much of a concrete plan for how to do these things in your day-to-day software development projects. But it's a good reminder to be thinking about, say, how to embed metrics and data-gathering hooks into the software you write. This is the nature of a primer; no 222-page book can get into much depth about the fractal complexity of doing good, fast, scalable software development.

Second, there's a fundamental question raised by a book like this: effective at what? Lau tackles that in the first chapter with his focus on impact and leverage, and it's good advice as far as it goes. (Regular readers of my book reviews know that I love this sort of time management and prioritization discussion.) But measuring impact is a hard problem that requires a prioritization framework, and this is not really the book for this. The Effective Engineer is written primarily for software developers at start-ups, leaves the whole venture-capital start-up process as unquestioned background material, and accepts without comment the standard measures of value in that world: fast-deployed products, hypergrowth, racing competitors for perceived innovation, and finding ways to extract money. That's as deep into the question of impact as Lau gets: increases in company revenue.

There's nothing wrong with this for the kind of book Lau intended to write, and it's not his fault that I find it unsatisfying. But don't expect The Effective Engineer to ask any hard questions about whether that's a meaningful definition of impact, or to talk much about less objective goals: quality of implementation, craftsmanship, giving back to a broader community via free software contributions, impact on the world in ways that can't be measured in market share, or anything else that is unlikely to lead to objective impact for company profits. At best he leaves a bit of wiggle room around using the concept of impact with different goals.

If you're a new graduate who wants to work at Silicon-Valley-style start-ups, this is a great orientation, and likewise if you're coming from a different area of software development into that world. If you're not working in that industry, The Effective Engineer may still be moderately interesting, but it's not written for that audience and has little or nothing to say of the challenges of other types of businesses. But if you've already worked in the industry for a while, or if you're more interested in deeper discussions of goals and subjective values, you may not get much out of this.

Rating: 7 out of 10

Categories: Elsewhere

Reproducible builds folks: Reproducible builds: week 53 in Stretch cycle

Mon, 02/05/2016 - 21:49

What happened in the Reproducible Builds effort between April 24th and 30th 2016.

Media coverage

Reproducible builds were mentioned explicitly in two talks at the Mini-DebConf in Vienna:

  • Martin Michlmayr had a talk in which he presented an overview about innovations and changes in Debian in the last years. Martin expressed his disappointment that there was no talk from us in Vienna (we'll fix this at DebConf16 in Cape Town) and described the reproducible builds work as "a real innovation". His talk is very much worth seeing, whatever your current perspective, it might change your view on Debian.
  • Ben Hutchings explains how Secure Boot will use signed kernels via separate signature packages and how this was designed with reproducible builds in mind.

Aspiration together with the OTF CommunityLab released their report about the Reproducible Builds summit in December 2015 in Athens.

Toolchain fixes

Now that the GCC development window has been opened again, the SOURCE_DATE_EPOCH patch by Dhole and Matthias Klose to address the issue timestamps_from_cpp_macros (__DATE__ / __TIME__) has been applied upstream and will be released with GCC 7.

Following that Matthias Klose also has uploaded gcc-5/5.3.1-17 and gcc-6/6.1.1-1 to unstable with a backport of that SOURCE_DATE_EPOCH patch.

Emmanuel Bourg uploaded maven/3.3.9-4, which uses SOURCE_DATE_EPOCH for the maven.build.timestamp.

(SOURCE_DATE_EPOCH specification)

Other upstream changes

Alexis Bienvenüe submitted a patch to Sphinx which extends SOURCE_DATE_EPOCH support for copyright years in generated documentation.

Packages fixed

The following 12 packages have become reproducible due to changes in their build dependencies: hhvm jcsp libfann libflexdock-java libjcommon-java libswingx1-java mobile-atlas-creator not-yet-commons-ssl plexus-utils squareness svnclientadapter

The following packages have became reproducible after being fixed:

Some uploads have fixed some reproducibility issues, but not all of them:

Patches submitted that have not made their way to the archive yet:

  • #822566 against stk by Alexis Bienvenüe: sort lists of object files for reproducible linking order.
  • #822948 against shotwell by Alexis Bienvenüe: normalize tarball permissions and use locale/timezone-independent modification time.
  • #822963 against htop by Alexis Bienvenüe: use SOURCE_DATE_EPOCH for embedded copyright year, which has before already been applied in git and upstream.
Package reviews

95 reviews have been added, 15 have been updated and 129 have been removed in this week.

22 FTBFS bugs have been reported by Chris Lamb and Martin Michlmayr.

diffoscope development
  • diffoscope 52~bpo8+1 has been uploaded to jessie-backports by Mattia Rizzolo, where it is currently waiting for NEW-approval.
  • Support for the deb(5) format (uncompressed data.tar/control.tar, control.tar.xz) (Closes: #818414) has been completed by Reiner Herrmann in git.
strip-nondeterminism development
  • Support for EPUB documents has been added (to the development version in git) by Holger Levsen, to address the timestamps_in_epub issue.
tests.reproducible-builds.org Misc.

Amongst the 29 interns who will work on Debian through GSoC and Outreachy there are four who will be contributing to Reproducible Builds for Debian and Free Software. We are very glad to welcome ceridwen, Satyam Zode, Scarlett Clark and Valerie Young and look forward to working together with them the coming months (and maybe beyond)!

This week's edition was written by Reiner Herrmann and Holger Levsen and reviewed by a bunch of Reproducible builds folks on IRC.

Categories: Elsewhere

Vincent Bernat: Pragmatic Debian packaging

Mon, 02/05/2016 - 21:25

While the creation of Debian packages is abundantly documented, most tutorials are targeted to packages implementing the Debian policy. Moreover, Debian packaging has a reputation of being unnecessarily difficult1 and many people prefer to use less constrained tools2 like fpm or CheckInstall.

However, I would like to show how building Debian packages with the official tools can become straightforward if you bend some rules:

  1. No source package will be generated. Packages will be built directly from a checkout of a VCS repository.

  2. Additional dependencies can be downloaded during build. Packaging individually each dependency is a painstaking work, notably when you have to deal with some fast-paced ecosystems like Java, Javascript and Go.

  3. The produced packages may bundle dependencies. This is likely to raise some concerns about security and long-term maintenance, but this is a common trade-off in many ecosystems, notably Java, Javascript and Go.

Pragmatic packages 101§

In the Debian archive, you have two kinds of packages: the source packages and the binary packages. Each binary package is built from a source package. You need a name for each package.

As stated in the introduction, we won’t generate a source package but we will work with its unpacked form which is any source tree containing a debian/ directory. In our examples, we will start with a source tree containing only a debian/ directory but you are free to include this debian/ directory into an existing project.

As an example, we will package memcached, a distributed memory cache. There are four files to create:

  • debian/compat,
  • debian/changelog,
  • debian/control, and
  • debian/rules.

The first one is easy. Just put 9 in it:

echo 9 > debian/compat

The second one has the following content:

memcached (0-0) UNRELEASED; urgency=medium * Fake entry -- Happy Packager <happy@example.com> Tue, 19 Apr 2016 22:27:05 +0200

The only important information is the name of the source package, memcached, on the first line. Everything else can be left as is as it won’t influence the generated binary packages.

The control file§

debian/control describes the metadata of both the source package and the generated binary packages. We have to write a block for each of them.

Source: memcached Maintainer: Vincent Bernat <bernat@debian.org> Package: memcached Architecture: any Description: high-performance memory object caching system

The source package is called memcached. We have to use the same name as in debian/changelog.

We generate only one binary package: memcached. In the remaining of the example, when you see memcached, this is the name of a binary package. The Architecture field should be set to either any or all. Use all exclusively if the package contains only arch-independent files. In doubt, just stick to any.

The Description field contains a short description of the binary package.

The build recipe§

The last mandatory file is debian/rules. It’s the recipe of the package. We need to retrieve memcached, build it and install its file tree in debian/memcached/. It looks like this:

#!/usr/bin/make -f DISTRIBUTION = $(shell lsb_release -sr) VERSION = 1.4.25 PACKAGEVERSION = $(VERSION)-0~$(DISTRIBUTION)0 TARBALL = memcached-$(VERSION).tar.gz URL = http://www.memcached.org/files/$(TARBALL) %: dh $@ override_dh_auto_clean: override_dh_auto_test: override_dh_auto_build: override_dh_auto_install: wget -N --progress=dot:mega $(URL) tar --strip-components=1 -xf $(TARBALL) ./configure --prefix=/usr make make install DESTDIR=debian/memcached override_dh_gencontrol: dh_gencontrol -- -v$(PACKAGEVERSION)

The empty targets override_dh_auto_clean, override_dh_auto_test and override_dh_auto_build keep debhelper from being too smart. The override_dh_gencontrol target sets the package version3 without updating debian/changelog. If you ignore the slight boilerplate, the recipe is quite similar to what you would have done with fpm:

DISTRIBUTION=$(lsb_release -sr) VERSION=1.4.25 PACKAGEVERSION=${VERSION}-0~${DISTRIBUTION}0 TARBALL=memcached-${VERSION}.tar.gz URL=http://www.memcached.org/files/${TARBALL} wget -N --progress=dot:mega ${URL} tar --strip-components=1 -xf ${TARBALL} ./configure --prefix=/usr make make install DESTDIR=/tmp/installdir # Build the final package fpm -s dir -t deb \ -n memcached \ -v ${PACKAGEVERSION} \ -C /tmp/installdir \ --description "high-performance memory object caching system"

You can review the whole package tree on GitHub and build it with dpkg-buildpackage -us -uc -b.

Pragmatic packages 102§

At this point, we can iterate and add several improvements to our memcached package. None of those are mandatory but they are usually worth the additional effort.

Build dependencies§

Our initial build recipe only work when several packages are installed, like wget and libevent-dev. They are not present on all Debian systems. You can easily express that you need them by adding a Build-Depends section for the source package in debian/control:

Source: memcached Build-Depends: debhelper (>= 9), wget, ca-certificates, lsb-release, libevent-dev

Always specify the debhelper (>= 9) dependency as we heavily rely on it. We don’t require make or a C compiler because it is assumed that the build-essential meta-package is installed and it pulls those. dpkg-buildpackage will complain if the dependencies are not met. If you want to install those packages from your CI system, you can use the following command4:

mk-build-deps \ -t 'apt-get -o Debug::pkgProblemResolver=yes --no-install-recommends -qqy' \ -i -r debian/control

You may also want to investigate pbuilder or sbuild, two tools to build Debian packages in a clean isolated environment.

Runtime dependencies§

If the resulting package is installed on a freshly installed machine, it won’t work because it will be missing libevent, a required library for memcached. You can express the dependencies needed by each binary package by adding a Depends field. Moreover, for dynamic libraries, you can automatically get the right dependencies by using some substitution variables:

Package: memcached Depends: ${misc:Depends}, ${shlibs:Depends}

The resulting package will contain the following information:

$ dpkg -I ../memcached_1.4.25-0\~unstable0_amd64.deb | grep Depends Depends: libc6 (>= 2.17), libevent-2.0-5 (>= 2.0.10-stable) Integration with init system§

Most packaged daemons come with some integration with the init system. This integration ensures the daemon will be started on boot and restarted on upgrade. For Debian-based distributions, there are several init systems available. The most prominent ones are:

  • System-V init is the historical init system. More modern inits are able to reuse scripts written for this init, so this is a safe common denominator for packaged daemons.
  • Upstart is the less-historical init system for Ubuntu (used in Ubuntu 14.10 and previous releases).
  • systemd is the default init system for Debian since Jessie and for Ubuntu since 15.04.

Writing a correct script for the System-V init is error-prone. Therefore, I usually prefer to provide a native configuration file for the default init system of the targeted distribution (Upstart and systemd).

System-V§

If you want to provide a System-V init script, have a look at /etc/init.d/skeleton on the most ancient distribution you want to target and adapt it5. Put the result in debian/memcached.init. It will be installed at the right place, invoked on install, upgrade and removal. On Debian-based systems, many init scripts allow user customizations by providing a /etc/default/memcached file. You can ship one by putting its content in debian/memcached.default.

Upstart§

Providing an Upstart job is similar: put it in debian/memcached.upstart. For example:

description "memcached daemon" start on runlevel [2345] stop on runlevel [!2345] respawn respawn limit 5 60 expect daemon script . /etc/default/memcached exec memcached -d -u $USER -p $PORT -m $CACHESIZE -c $MAXCONN $OPTIONS end script

When writing an Upstart job, the most important directive is expect. Be sure to get it right. Here, we use expect daemon and memcached is started with the -d flag.

systemd§

Providing a systemd unit is a bit more complex. The content of the file should go in debian/memcached.service. For example:

[Unit] Description=memcached daemon After=network.target [Service] Type=forking EnvironmentFile=/etc/default/memcached ExecStart=/usr/bin/memcached -d -u $USER -p $PORT -m $CACHESIZE -c $MAXCONN $OPTIONS Restart=on-failure [Install] WantedBy=multi-user.target

We reuse /etc/default/memcached even if it is not considered a good practice with systemd6. Like for Upstart, the directive Type is quite important. We used forking as memcached is started with the -d flag.

You also need to add a build-dependency to dh-systemd in debian/control:

Source: memcached Build-Depends: debhelper (>= 9), wget, ca-certificates, lsb-release, libevent-dev, dh-systemd

And you need to modify the default rule in debian/rules:

%: dh $@ --with systemd

The extra complexity is a bit unfortunate but systemd integration is not part of debhelper7. Without those additional modifications, the unit will get installed but you won’t get a proper integration and the service won’t be enabled on install or boot.

Dedicated user§

Many daemons don’t need to run as root and it is a good practice to ship a dedicated user. In the case of memcached, we can provide a _memcached user8.

Add a debian/memcached.postinst file with the following content:

#!/bin/sh set -e case "$1" in configure) adduser --system --disabled-password --disabled-login --home /var/empty \ --no-create-home --quiet --force-badname --group _memcached ;; esac #DEBHELPER# exit 0

There is no cleanup of the user when the package is removed for two reasons:

  1. Less stuff to write.
  2. The user could still own some files.

The utility adduser will do the right thing whatever the requested user already exists or not. You need to add it as a dependency in debian/control:

Package: memcached Depends: ${misc:Depends}, ${shlibs:Depends}, adduser

The #DEBHELPER# marker is important as it will be replaced by some code to handle the service configuration files (or some other stuff).

You can review the whole package tree on GitHub and build it with dpkg-buildpackage -us -uc -b.

Pragmatic packages 103§

It is possible to leverage debhelper to reduce the recipe size and to make it more declarative. This section is quite optional and it requires understanding a bit more how a Debian package is built. Feel free to skip it.

The big picture§

There are four steps to build a regular Debian package:

  1. debian/rules clean should clean the source tree to make it pristine.

  2. debian/rules build should trigger the build. For an autoconf-based software, like memcached, this step should execute something like ./configure && make.

  3. debian/rules install should install the file tree of each binary package. For an autoconf-based software, this step should execute make install DESTDIR=debian/memcached.

  4. debian/rules binary will pack the different file trees into binary packages.

You don’t directly write each of those targets. Instead, you let dh, a component of debhelper, do most of the work. The following debian/rules file should do almost everything correctly with many source packages:

#!/usr/bin/make -f %: dh $@

For each of the four targets described above, you can run dh with --no-act to see what it would do. For example:

$ dh build --no-act dh_testdir dh_update_autotools_config dh_auto_configure dh_auto_build dh_auto_test

Each of those helpers has a manual page. Helpers starting with dh_auto_ are a bit “magic”. For example, dh_auto_configure will try to automatically configure a package prior to building: it will detect the build system and invoke ./configure, cmake or Makefile.PL.

If one of the helpers do not do the “right” thing, you can replace it by using an override target:

override_dh_auto_configure: ./configure --with-some-grog

Those helpers are also configurable, so you can just alter a bit their behaviour by invoking them with additional options:

override_dh_auto_configure: dh_auto_configure -- --with-some-grog

This way, ./configure will be called with your custom flag but also with a lot of default flags like --prefix=/usr for better integration.

In the initial memcached example, we overrode all those “magic” targets. dh_auto_clean, dh_auto_configure and dh_auto_build are converted to no-ops to avoid any unexpected behaviour. dh_auto_install is hijacked to do all the build process. Additionally, we modified the behavior of the dh_gencontrol helper by forcing the version number instead of using the one from debian/changelog.

Automatic builds§

As memcached is an autoconf-enabled package, dh knows how to build it: ./configure && make && make install. Therefore, we can let it handle most of the work with this debian/rules file:

#!/usr/bin/make -f DISTRIBUTION = $(shell lsb_release -sr) VERSION = 1.4.25 PACKAGEVERSION = $(VERSION)-0~$(DISTRIBUTION)0 TARBALL = memcached-$(VERSION).tar.gz URL = http://www.memcached.org/files/$(TARBALL) %: dh $@ --with systemd override_dh_auto_clean: wget -N --progress=dot:mega $(URL) tar --strip-components=1 -xf $(TARBALL) override_dh_auto_test: # Don't run the whitespace test rm t/whitespace.t dh_auto_test override_dh_gencontrol: dh_gencontrol -- -v$(PACKAGEVERSION)

The dh_auto_clean target is hijacked to download and setup the source tree9. We don’t override the dh_auto_configure step, so dh will execute the ./configure script with the appropriate options. We don’t override the dh_auto_build step either: dh will execute make. dh_auto_test is invoked after the build and it will run the memcached test suite. We need to override it because one of the test is complaining about odd whitespaces in the debian/ directory. We suppress this rogue test and let dh_auto_test executes the test suite. dh_auto_install is not overriden either, so dh will execute some variant of make install.

To get a better sense of the difference, here is a diff:

--- memcached-intermediate/debian/rules 2016-04-30 14:02:37.425593362 +0200 +++ memcached/debian/rules 2016-05-01 14:55:15.815063835 +0200 @@ -12,10 +12,9 @@ override_dh_auto_clean: -override_dh_auto_test: -override_dh_auto_build: -override_dh_auto_install: wget -N --progress=dot:mega $(URL) tar --strip-components=1 -xf $(TARBALL) - ./configure --prefix=/usr - make - make install DESTDIR=debian/memcached + +override_dh_auto_test: + # Don't run the whitespace test + rm t/whitespace.t + dh_auto_test

It is up to you to decide if dh can do some work for you, but you could try to start from a minimal debian/rules and only override some targets.

Install additional files§

While make install installed the essential files for memcached, you may want to put additional files in the binary package. You could use cp in your build recipe, but you can also declare them:

  • files listed in debian/memcached.docs will be copied to /usr/share/doc/memcached by dh_installdocs,
  • files listed in debian/memcached.examples will be copied to /usr/share/doc/memcached/examples by dh_installexamples,
  • files listed in debian/memcached.manpages will be copied to the appropriate subdirectory of /usr/share/man by dh_installman,

Here is an example using wildcards for debian/memcached.docs:

doc/*.txt

If you need to copy some files to an arbitrary location, you can list them along with their destination directories in debian/memcached.install and dh_install will take care of the copy. Here is an example:

scripts/memcached-tool usr/bin

Using those files make the build process more declarative. It is a matter of taste and you are free to use cp in debian/rules instead. You can review the whole package tree on GitHub.

Other examples§

The GitHub repository contains some additional examples. They all follow the same scheme:

  • dh_auto_clean is hijacked to download and setup the source tree
  • dh_gencontrol is modified to use a computed version

Notably, you’ll find daemons in Java, Go, Python and Node.js. The goal of those examples is to demonstrate that using Debian tools to build Debian packages can be straightforward. Hope this helps.

  1. People may remember the time before debhelper 7.0.50 (circa 2009) where debian/rules was a daunting beast. However, nowaday, the boilerplate is quite reduced. 

  2. The complexity is not the only reason. Those alternative tools enable the creation of RPM packages, something that Debian tools obviously don’t. 

  3. There are many ways to version a package. Again, if you want to be pragmatic, the proposed solution should be good enough for Ubuntu. On Debian, it doesn’t cover upgrade from one distribution version to another, but we assume that nowadays, systems get reinstalled instead of being upgraded. 

  4. You also need to install devscripts and equivs package. 

  5. It’s also possible to use a script provided by upstream. However, there is no such thing as an init script that works on all distributions. Compare the proposed with the skeleton, check if it is using start-stop-daemon and if it sources /lib/lsb/init-functions before considering it. If it seems to fit, you can install it yourself in debian/memcached/etc/init.d/. debhelper will ensure its proper integration. 

  6. Instead, a user wanting to customize the options is expected to edit the unit with systemctl edit. 

  7. See #822670 

  8. The Debian Policy doesn’t provide any hint for the naming convention of those system users. A common usage is to prefix the daemon name with an underscore (like _memcached). Another common usage is to use Debian- as a prefix. The main drawback of the latest solution is that the name is likely to be replaced by the UID in ps and top because of its length. 

  9. We could call dh_auto_clean at the end of the target to let it invoke make clean. However, it is assumed that a fresh checkout is used before each build. 

Categories: Elsewhere

Michal &#268;iha&#345;: Weekly phpMyAdmin contributions 2016-W17

Mon, 02/05/2016 - 06:00

Last week was quite split into many smaller tasks - working on our libraries (both SQL parser and motranslator got new releases with bug fixes), fixing bugs for upcoming 4.6.1 and working on documentation.

From the libraries side, probably most visible is release of motranslator 1.0, just to claim it's now stable enough. Let's see if somebody else will pick it up as well or it will stay only for our use.

Most time was however spent on our documentation. We've agreed to move wiki from our server to GitHub wiki and reduce content available on the wiki. So far it's really mixture of user documentation, notes and developer documentation. The final shape should be that wiki will contain only developer documentation and all end user documentation will go to our documentation. So far I've gone through about half of user docs pages, deleted duplicated ones and moved content to our documentation. It is most visible on the user guide which now contains way more information and hopefully it will get more complete in near future.

Handled issues:

Filed under: English phpMyAdmin | 0 comments

Categories: Elsewhere

Pages