Erich Schubert: Big data predictions for 2015

Planet Debian - Tue, 13/01/2015 - 16:01
My big data predictions for 2015:
  1. Big data will continue to fail to deliver for most companies.
    This has several reasons, including in particular: 1: lack of data to analyze that actually benefits from big data tools and approaches (and which is not better analyzed with traditional tools). 2: lack of talent, and failure to attract analytics talent. 3: stuck in old IT, and too inflexible to allow using modern tools (if you want to use big data, you will need a flexible "in-house development" type of IT that can install tools, try them, abandon them, without going up and down the management chains) 4: too much marketing. As long as big data is being run by the marketing department, not by developers, it will fail.
  2. Project consolidation: we have seen hundreds of big data software projects the last years. Plenty of them on Apache, too. But the current state is a mess, there is massive redundancy, and lots and lots of projects are more-or-less abandoned. Cloudera ML, for example, is dead: superseded by Oryx and Oryx 2. More projects will be abandoned, because we have way too many (including much too many NoSQL databases, that fail to outperform SQL solutions like PostgreSQL). As is, we have dozens of competing NoSQL databases, dozens of competing ML tools, dozens of everything.
  3. Hype: the hype will continue, but eventually (when there is too much negative press on the term "big data" due to failed projects and inflated expectations) move on to other terms. The same is also happening to "data science", so I guess the next will be "big analytics", "big intelligence" or something like that.
  4. Less openness: we have seen lots of open-source projects. However, many decided to go with Apache-style licensing - always ready to close down their sharing, and no longer share their development. In 2015, we'll see this happen more often, as companies try to make money off their reputation. At some point, copyleft licenses like GPL may return to popularity due to this.
Categories: Elsewhere

Drupalize.Me: Changes in the Form API in Drupal 8

Planet Drupal - Tue, 13/01/2015 - 15:13

In my previous post, I documented the first of my Adventures in Porting a D7 Form Module to Drupal 8. In that article, I documented how I used the Drupal Module Upgrader to convert my Drupal 7 module, Form Fun, to Drupal 8 and what I learned along the way about how Routes and Controllers replaced hook_menu, and what I gleaned from change records about other API changes. This article is a continuation of that post, so you might want to pop over and give it a read so that you're up to speed with what we're doing here.

Categories: Elsewhere

DrupalOnWindows: Setting up Code Syntax Higlighting with Drupal

Planet Drupal - Tue, 13/01/2015 - 07:00

While setting up (still in progres..) this website we found the need to enable easy Code Highlighting to be done through our WYSIWYG editor of choice: CKeditor. What looked like a 30 minute task, ended up in a more than 3 hour adventure. Let's see what happened.

Language English
Categories: Elsewhere

Dirk Eddelbuettel: RcppGSL 0.2.3

Planet Debian - Tue, 13/01/2015 - 03:33

A new version of RcppGSL is now on CRAN today. This package provides an interface from R to the GNU GSL using our Rcpp.

Similar to the recent RcppClassic release, this update was triggered by the CRAN maintainers desire to keep the Makefile free of GNU make extensions. At the same time we simplified the configure file a little, and refreshed the look and feel of the vignette.

No user-facing changes were made.

The NEWS file entries follows below:

Changes in version 0.2.3 (2015-01-10)
  • The src/Makevars.in was pruned of GNU make features at the request of the CRAN Maintainers

  • configure.ac and configure were updated, and shortened

  • The RcppGSL-intro.Rnw vignette was updated for its look and feel.

Courtesy of CRANberries, a summary of changes to the most recent release is available.

More information is on the RcppGSL page. Questions, comments etc should go to the rcpp-devel mailing list off the R-Forge page.

This post by Dirk Eddelbuettel originated on his Thinking inside the box blog. Please report excessive re-aggregation in third-party for-profit settings.

Categories: Elsewhere

Daniel Pocock: Lumicall: big steps forward, help needed

Planet Debian - Mon, 12/01/2015 - 21:02

I've recently made more updates to Lumicall, the free, open source and secure alternative to Viber and Skype.

Here are some of the highlights:

  • The dialing popup is now optional, so if you want to call your friends with Lumicall / SIP but they don't want to see the popup when making calls themselves, you can disable the popup on their phone.
  • The dialer popup now shows results asynchronously so you can dial more quickly
  • SIP SIMPLE messaging is now supported, Lumicall is now taking on WhatsApp
  • Various bugs in the preferences/settings have been fixed and adding SIP accounts is now easier
  • Dialing with a TURN relay is now much more reliable
  • It is now possible to redial SIP calls in the call history without seeing the nasty Android bug / popup telling you that you don't have Internet calling configured
F-Droid not updated yet

F-Droid is not yet carrying the latest version. The F-Droid team may need assistance as they appear to be reviewing a lot of the third-party dependencies used by apps they distribute to make sure the full stack is 100% free software. If people want to continue having the option to get Lumicall and other free software through F-Droid instead of Google Play then helping the F-Droid community is the number one way you can help Lumicall.

Other ways you can help, even without coding

You don't have to be a developer to help with Lumicall.

Taking on Viber, Skype and now WhatsApp as well may not sound easy. It isn't. Your help could make the difference though.

Here are some of the things that need assistance:

  • Helping to get it on Wikipedia, they keep deleting the page while happily hosting pages about similar products like Sipdroid and CSipSimple
  • Helping get the latest dependencies and Lumicall version into F-Droid
  • UI design ideas
  • Web site assistance
  • Documentation and screenshots, e.g. for use with Asterisk and FreeSWITCH and various SIP proxies
  • Translation
  • Messaging: to be the default SMS app on an Android device, Lumicall would need a full messaging UI and the ability to replace all functions of the default SMS app. Can anybody identify any other free software that does this and is modular enough to share the relevant code with Lumicall?
  • ZRTP: can you help improve the ZRTP stack used by Lumicall?
Try SIP SIMPLE messaging

When composing a message to a Lumicall user, the SIP address is written sip:number @sip5060.net. E.g. if the number is +442071234567 then the SIP address to use when calling or composing a message is sip:+442071234567 @sip5060.net

Debian Developers should be able to interact with Lumicall users from rtc.debian.org using the SIP over WebSocket messaging.

Getting the latest Lumicall

It is available from:

Categories: Elsewhere

Petter Reinholdtsen: Norwegian Bokmål subtitles for the FSF video User Liberation

Planet Debian - Mon, 12/01/2015 - 21:00

A few days ago, the Free Software Foundation announced a new video explaining Free software in simple terms. The video named User Liberation is 3 minutes long, and I recommend showing it to everyone you know as a way to explain what Free Software is all about. Unfortunately several of the people I know do not understand English and Spanish, so it did not make sense to show it to them.

But today I was told that English subtitles were available and set out to provide Norwegian Bokmål subtitles based on these. The result has been sent to FSF and made available in a git repository provided by Github. Please let me know if you find errors or have improvements to the subtitles.

Categories: Elsewhere

Ritesh Raj Sarraf: What firmware

Planet Debian - Mon, 12/01/2015 - 20:04

Dear Lazy Web,

I have  an HP Envy J104TS laptop. Recently I saw an interesting message in the kernel log.

[99360.969652] [Firmware Bug]: battery: (dis)charge rate invalid.

Does anybody know what firmware is it referring to here ? I don't think the current set of firmwares shiped by linux are involved in battery related information. Is it the BIOS ?

[95474.561491] IPv6: ADDRCONF(NETDEV_UP): wlan0: link is not ready [95474.803578] r8169 0000:0f:00.0 eth0: link down [95474.803627] IPv6: ADDRCONF(NETDEV_UP): eth0: link is not ready [95474.933797] IPv6: ADDRCONF(NETDEV_UP): wlan0: link is not ready [95477.111599] wlan0: authenticate with 00:03:7f:fe:00:02 [95477.127389] wlan0: send auth to 00:03:7f:fe:00:02 (try 1/3) [95477.129315] wlan0: authenticated [95477.131352] wlan0: associate with 00:03:7f:fe:00:02 (try 1/3) [95477.133605] wlan0: RX AssocResp from 00:03:7f:fe:00:02 (capab=0x411 status=0 aid=3) [95477.133686] wlan0: associated [95477.133703] IPv6: ADDRCONF(NETDEV_CHANGE): wlan0: link becomes ready [99125.123138] device vethNLAABF entered promiscuous mode [99125.123377] IPv6: ADDRCONF(NETDEV_UP): vethNLAABF: link is not ready [99125.188568] IPv6: ADDRCONF(NETDEV_CHANGE): vethNLAABF: link becomes ready [99125.188615] lxcbr0: port 1(vethNLAABF) entered forwarding state [99125.188631] lxcbr0: port 1(vethNLAABF) entered forwarding state [99140.239301] lxcbr0: port 1(vethNLAABF) entered forwarding state [99360.969652] [Firmware Bug]: battery: (dis)charge rate invalid. [99361.729291] NMI watchdog: enabled on all CPUs, permanently consumes one hw-PMU counter. [99361.864205] EXT4-fs (dm-2): re-mounted. Opts: errors=remount-ro,data=writeback,commit=0 [99361.905210] EXT4-fs (sda6): re-mounted. Opts: data=ordered,commit=0 [102236.267648] lxcbr0: port 1(vethNLAABF) entered disabled state [102236.556452] lxcbr0: port 1(vethNLAABF) entered disabled state [102236.557476] device vethNLAABF left promiscuous mode [102236.557483] lxcbr0: port 1(vethNLAABF) entered disabled state Categories: Keywords: 
Categories: Elsewhere

Russell Coker: Systemd Notes

Planet Debian - Mon, 12/01/2015 - 19:07

A few months ago I gave a lecture about systemd for the Linux Users of Victoria. Here are some of my notes reformatted as a blog post:

Scripts in /etc/init.d can still be used, they work the same way as they do under sysvinit for the user. You type the same commands to start and stop daemons.

To get a result similar to changing runlevel use the “systemctl isolate” command. Runlevels were never really supported in Debian (unlike Red Hat where they were used for starting and stopping the X server) so for Debian users there’s no change here.

The command systemctl with no params shows a list of loaded services and highlights failed units.

The command “journalctl -u UNIT-PATTERN” shows journal entries for the unit(s) in question. The pattern uses wildcards not regexs.

The systemd journal includes the stdout and stderr of all daemons. This solves the problem of daemons that don’t log all errors to syslog and leave the sysadmin wondering why they don’t work.

The command “systemctl status UNIT” gives the status and last log entries for the unit in question.

A program can use ioctl(fd, TIOCSTI, …) to push characters into a tty buffer. If the sysadmin runs an untrusted program with the same controlling tty then it can cause the sysadmin shell to run hostile commands. The system call setsid() to create a new terminal session is one solution but managing which daemons can be started with it is difficult. The way that systemd manages start/stop of all daemons solves this. I am glad to be rid of the run_init program we used to use on SE Linux systems to deal with this.

Systemd has a mechanism to ask for passwords for SSL keys and encrypted filesystems etc. There have been problems with that in the past but I think they are all fixed now. While there is some difficulty during development the end result of having one consistent way of managing this will be better than having multiple daemons doing it in different ways.

The commands “systemctl enable” and “systemctl disable” enable/disable daemon start at boot which is easier than the SysVinit alternative of update-rc.d in Debian.

Systemd has built in seat management, which is not more complex than consolekit which it replaces. Consolekit was installed automatically without controversy so I don’t think there should be controversy about systemd replacing consolekit.

Systemd improves performance by parallel start and autofs style fsck.

The command systemd-cgtop shows resource use for cgroups it creates.

The command “systemd-analyze blame” shows what delayed the boot process and
systemd-analyze critical-chain” shows the critical path in boot delays.

Sysremd also has security features such as service private /tmp and restricting service access to directory trees.


For basic use things just work, you don’t need to learn anything new to use systemd.

It provides significant benefits for boot speed and potentially security.

It doesn’t seem more complex than other alternative solutions to the same problems.




Related posts:

  1. systemd – a Replacement for init etc The systemd projecct is an interesting concept for replacing init...
  2. Some Notes on DRBD DRBD is a system for replicating a block device across...
  3. licence for lecture notes While attending LCA it occurred to me that the lecture...
Categories: Elsewhere

Liran Tal's Enginx: Drupal Performance Tip – be humble on hook_init()

Planet Drupal - Mon, 12/01/2015 - 17:06
This entry is part 5 of 5 in the series Drupal Performance Tips

In the spirit of the computer video game Doom and its skill levels, we’ll review a few ways you can improve  your Drupal speed performance     and optimize for better results and server response time. These tips that we’ll cover may be at times specific to Drupal 6 versions, although     you can always learn the best practices from these examples and apply them on your own code base.

Doom skill levels: (easiest first)

1. I’m too young to die

2. Hey, not too rough

3. Hurt me plenty

4. Ultra-violence

5. Nightmare!

  This post is rated “I’m too young too die” difficulty level.


Drupal is known for its plethora of hooks, and their use is abundant through-out any Drupal modules to plug into the way that Drupal works. That’s fine, though once you’ve decided you’re moving on with Drupal as your live web application/website and you’re using modules from the eco-system, that is when you need to spend some more time reviewing modules a little bit closer than just their download counts or issues on drupal.org

hook_init() runs on every page load. Imagine you’re having a few modules implementing this hook, then you already have impact on your server response time performance for every page access in Drupal. Maybe those modules have a very slight overhead there, maybe that’s part of what they do, and that’s fine, but it may at times benefit you to review and investigate if the code there, that maybe your team added too, is better being re-factored to some other place and not on every page load.

There is another perspective for it of course, maybe things do need to take place on every page load, but their implementation in the code might be faulty. Imagine you’re doing some expensive IO on every page load, like calling an API, or querying a heavy table. Maybe you can re-factor to cache this information?


(adsbygoogle = window.adsbygoogle || []).push({});

The post Drupal Performance Tip – be humble on hook_init() appeared first on Liran Tal's Enginx.

Categories: Elsewhere

Evolving Web: Parallelized web scraping using RollingCurl

Planet Drupal - Mon, 12/01/2015 - 16:35

The web is full of information! Your web sites probably already use many APIs for maps, Twitter, IP geolocation, and more. But what about data that's on the web, but doesn't have a readily available API?

read more
Categories: Elsewhere

Yuriy Gerasimov: Centralize your logs with logstash (getting started guide)

Planet Drupal - Mon, 12/01/2015 - 13:51

Logstash is a great tool to centralize logs in your environment. For example we have several drupal webheads that write logs into syslog. It would be really nice to see those logs somewhere centrally to find out about your system's health status and debug potential problems.

In this article I would like to show how easy to start using logstash for local development.

First of all in order to run logstash you need to follow instructions http://logstash.net/docs/1.4.2/tutorials/getting-started-with-logstash.

Logstash has following concepts:

  • inputs -- where we grab logs from. This can be files on local files system, records of database table, redis and many more.
  • codecs -- way you can serialize/unserialize you data. Think about it as json decode when you get records or running json encode when you are saving log message.
  • filters -- instruments to filter particular log records we want to process. Example -- syslog has many records but we want to extract only drupal related.
  • outputs -- where we are passing our processed log records. It can be a file (multiple different formats), stdout or what is most interesting elastic search

Tricky part comes when you need to install Elastic Search to store your logs and Kibana to view them. There is very nice shortcut for development purposes -- to use already built docker image for that.

I have found very handy to use https://registry.hub.docker.com/u/sebp/elk/ image.

So you need docker to be installed (http://docs.docker.com/installation/ubuntulinux/). Then you import docker image and run it.

sudo docker pull sebp/elk sudo docker run -p 5601:5601 -p 9200:9200 -p 5000:5000 -it --name elk sebp/elk

Now we have docker image working plus it has port forwarding to our localhost.

In order to send your logstash logs to elastic search you need to use elasticsearch output. Here is logstash configuration file example that can be run for testing.

input { stdin { } } output { stdout { codec => rubydebug } elasticsearch { host => "localhost" port => "9200" protocol => "http" } }

Now when you run logstash and enter couple of messages they will be fed to elasticsearch. Now you can open http://localhost:5601/ to see kibana in action.

Next step would be to set up your own rules of extracting drupal (or any other type) logs and pushing them to elastic search. But this is very individual task that is out of the scope of this guide.

Tags: drupaldrupal planetlogstash
Categories: Elsewhere

Chris Lamb: Giant's Causeway puzzle

Planet Debian - Mon, 12/01/2015 - 11:34

Over Christmas I found the above puzzle at my parent's house. I don't know if it has a name but it seemed apt to name it after The Giant's Causeway.

The idea is that you swap and/or rotate the outer tiles until the colours at the edges match. The center tile is fixed, or at least I assume it is as the wooden edges are deliberately less rounded.

Of course, solving it manually would be boring so here's a dumb brute force solution. I actually admire how inefficient and stupid it is...

import itertools class Tile(list): def __getitem__(self, x): return super(Tile, self).__getitem__(x % len(self)) def rotate(self, x): return Tile(self[x:] + self[:x]) COLOURS = ('YELLOW', 'WHITE', 'BLACK', 'RED', 'GREEN', 'BLUE') YELLOW, WHITE, BLACK, RED, GREEN, BLUE = range(len(COLOURS)) TILES = ( Tile((WHITE, YELLOW, RED, BLUE, BLACK, GREEN)), Tile((RED, BLUE, BLACK, YELLOW, GREEN, WHITE)), Tile((WHITE, BLACK, YELLOW, GREEN, BLUE, RED)), Tile((WHITE, BLUE, GREEN, YELLOW, BLACK, RED)), Tile((GREEN, BLUE, BLACK, YELLOW, RED, WHITE)), Tile((RED, YELLOW, GREEN, BLACK, BLUE, WHITE)), ) CENTER = Tile((WHITE, BLACK, RED, GREEN, BLUE, YELLOW)) def pairwise(it): a, b = itertools.tee(it) next(b, None) return itertools.izip(a, b) def validate(xs): for idx, x in enumerate(xs): if x[idx + 3] != CENTER[idx]: raise ValueError("Tile does not match center") for idx, (a, b) in enumerate(pairwise(xs)): if a[idx + 2] != b[idx + 5]: raise ValueError("Tile does not match previous anticlockwise tile") return xs def find_solution(): # For all possible tile permutations.. for xs in itertools.permutations(TILES): # ... try all possible rotations for ys in itertools.product(range(len(COLOURS)), repeat=len(COLOURS)): try: return validate([x.rotate(y) for x, y in zip(xs, ys)]) except ValueError: pass raise ValueError("Could not find a solution.") for x in find_solution(): print ', '.join(COLOURS[y] for y in x)

This prints (after 5 minutes!):


UPDATE: Seems like this puzzle is called "Circus Seven". Now to dust off my Prolog...

Categories: Elsewhere

Dirk Eddelbuettel: rfoaas 0.1.1

Planet Debian - Mon, 12/01/2015 - 00:24

A brand new and shiny version of rfoaas is now on CRAN. The rfoaas package provides an interface for R to the most excellent FOAAS service--which provides a modern, scalable and RESTful web service for the frequent need to tell someone to f$#@ off.

There are two (internal) changes of merit in this version. First off, as FOAAS was refactored upstream, we are now forced to supply an accept: text/plain http request header. Which, sadly enough, is not something the url() function in R can do---so we brought in more cavalry and now depend on the httr package by Hadley, and use its GET() method. A second change is that we added a (simple but effective enough) regression test which simply calls all foaas entry points available throug rfoaas, and compares this to the anticipated result. To run it, you need to set an environment variable RunFOAASTests=yes as eg out Travis script does. Finally, we aligned the version number with upstream to signal that we cover all available entry points of that version.

As usual, CRANberries provides a diff to the previous release. Questions, comments etc should go to the GitHub issue tracker.

This post by Dirk Eddelbuettel originated on his Thinking inside the box blog. Please report excessive re-aggregation in third-party for-profit settings.

Categories: Elsewhere

Enrico Zini: halmerweg

Planet Debian - Sun, 11/01/2015 - 22:20
<3 Bremen

Greedy people exploit our cultural differences to justify fighting wars to seize wealth for themselves, when sharing and enjoying cultural diversity makes us far richer than if we possess material wealth!

From a recursive murales in Bremen Halmerweg.

Categories: Elsewhere

Diego Escalante Urrelo: Practical contentment and art

Planet Debian - Sun, 11/01/2015 - 21:55

I was browsing Eric Kim’s street photography blog, when I stumbled across one article about the well-known “Gear Acquisition Syndrome” that plagues enthusiast photographers.

Gear Acquisition Syndrome is the obsession that enthusiasts tend to have for new gear, new gadgets, new stuff, as an excuse for not developing their skills or taking action. You can recognize it by phrases like: If only I had that new gear, I would be better at what I do… I can’t do that project until I buy this gear… My super expensive camera is not good for that, I need an even more expensive one… (etc).

The article, “How to Be Grateful For What You Have” is a funny and well argued look on why photographers, or artists in general, can get lost in what they don’t have, instead of what they have.

The article is a recommended read in itself and delivers the point whether you are into photography or not. That said, inside the article I found three nice links worth noting:

The Tiny Collective
A group of photographers that use only their phones as their camera. It’s inspiring to see a specific, curated, selection of images made with phones. It definitely makes you wonder why would you need anything but a phone to make great images (answer: phones work great, even old ones).

A Guide to Practical Contentment
One of the most simple and effective messages about contentment that I have read. It’s not about “not wanting things” or settling for “whatever”. It’s all about taking a step back and realizing that life is not about absolutes. If you are not a billionaire, that doesn’t mean you have failed in life. Perhaps you are just a millionaire, and that’s fine :).

(…) if you start in this place of fixing what’s wrong with you, you keep looking for what else is wrong with you, what else you need to improve. So maybe now feel like you don’t have enough muscles, or six pack abs, or you think your calves don’t look good, or if it’s not about your body, you’ll find something else.

So it’s this never-ending cycle for your entire life. You never reach it. If you start with a place of wanting to improve yourself and feeling stuck, even if you’re constantly successful and improving, you’re always looking for happiness from external sources. You don’t find the happiness from within, so you look to other things.

If you’re externally looking for happiness, it’s easy to get too into food, or shopping, or partying, or overwork, to try to be happy.

If instead, you can find contentment within and not need external sources of happiness, then you’ll have a reliable source of happiness.

So, instead of looking at sources of external happiness, why don’t you look into sources of internal happiness? It’s one of the hardest things to learn how to do, but I’m personally slowly getting there. It’s life changing.

Bonus points: quotes on simplification and minimalism
For extra credit, here are some interesting quotes on the above topics, note that as every internet quote page it might be filled with false quotes. I happen to like the one attributed to Donald Horban, who apparently doesn’t exist outside quote pages in the internet. Maybe that’s the biggest minimalism? Not existing?

“We don’t need to increase our goods nearly as much as we need to scale down our wants. Not wanting something is as good as possessing it.”

Anyway, take these with a grain of salt, and just read them as interesting phrases. Just take it easy :).

Categories: Elsewhere

Paul Rowell: Drupal fields; improving select lists

Planet Drupal - Sun, 11/01/2015 - 21:39

Everyone knows what a select list is, what it looks like and how it works. But that doesn't mean it can't get better, here a few modules that can be used to improve the experience for users when selecting items.

Categories: Elsewhere

Drupal Camp NJ 2015: Drupal 8 core critical issues sprint to coincide with camp

Planet Drupal - Sun, 11/01/2015 - 17:53

The Central NJ Drupal Meetup has received one of the first grants from the new Drupal Association Drupal 8 Accelerate program. The Sprint will take place from January 29 to Febrary 1. For more details see https://groups.drupal.org/node/453848 

Categories: Elsewhere


Subscribe to jfhovinne aggregator - Elsewhere