Elsewhere

Dirk Eddelbuettel: Like peanut butter and jelly: x13binary and seasonal

Planet Debian - Tue, 02/02/2016 - 13:58

This post was written by Dirk Eddelbuettel and Christoph Sax and will be posted on both author's respective blogs.

The seasonal package by Christoph Sax brings a very featureful and expressive interface for working with seasonal data to the R environment. It uses the standard tool of the trade: X-13ARIMA-SEATS. This powerful program is provided by the statisticians of the US Census Bureau based on their earlier work (named X-11 and X-12-ARIMA) as well as the TRAMO/SEATS program by the Bank of Spain. X-13ARIMA-SEATS is probably the best known tool for de-seasonalization of timeseries, and used by statistical offices around the world.

Sadly, it also has a steep learning curve. One interacts with a basic command-line tool which users have to download, install and properly reference (by environment variables or related means). Each model specification has to be prepared in a special 'spec' file that uses its own, cumbersome syntax.

As seasonal provides all the required functionality to use X-13ARIMA-SEATS from R --- see the very nice seasonal demo site --- it still required the user to manually deal with the X-13ARIMA-SEATS installation.

So we decided to do something about this. A pair of GitHub repositories provide both the underlying binary in a per-operating system form (see x13prebuilt) as well as a ready-to- use R package (see x13binary) which uses the former to provide binaries for R. And the latter is now on CRAN as package x13binary ready to be used on Windows, OS-X or Linux. And the seasonal package (in version 1.2.0 -- now on CRAN -- or later) automatically makes use of it. Installing seasaonal and x13binary in R is now as easy as:

install.packages("seasonal")

which opens the door for effortless deployment of powerful deasonalization. By default, the principal function of the package employs a number of automated techniques that work well in most circumstances. For example, the following code produces a seasonal adjustment of the latest data of US retail sales (by the Census Bureau) downloaded from Quandl:

library(seasonal) library(Quandl) ## not needed for seasonal but has some niceties for Quandl data rs <- Quandl(code="USCENSUS/BI_MARTS_44000_SM", type="ts")/1e3 m1 <- seas(rs) plot(m1, main = "Retail Trade: U.S. Total Sales", ylab = "USD (in Billions)")

This tests for log-transformation, performs an automated ARIMA model search, applies outlier detection, tests and adjusts for trading day and easter effects, and invokes the SEATS method to perform seasonal adjustment. And this is how the adjusted series looks like:

Of course, you can access all available options of X-13ARIMA-SEATS as well. Here is an example where we adjust the latest data for Chinese exports (as tallied by the US FED), taking into account the different effects of Chinese New Year before, during and after the holiday:

xp <- Quandl(code="FRED/VALEXPCNM052N", type="ts")/1e9 m2 <- seas(window(xp, start = 2000), xreg = cbind(genhol(cny, start = -7, end = -1, center = "calendar"), genhol(cny, start = 0, end = 7, center = "calendar"), genhol(cny, start = 8, end = 21, center = "calendar") ), regression.aictest = c("td", "user"), regression.usertype = "holiday") plot(m2, main = "Goods, Value of Exports for China", ylab = "USD (in Billions)")

which generates the following chart demonstrating a recent flattening in export activity measured in USD.

We hope this simple examples illustrates both how powerful a tool X-13ARIMA-SEATS is, but also just how easy it is to use X-13ARIMA-SEATS from R now that we provide the x13binary package automating its installation.

This post by Dirk Eddelbuettel originated on his Thinking inside the box blog. Please report excessive re-aggregation in third-party for-profit settings.

Categories: Elsewhere

Deeson: Warden: Monitoring the Secruity of a Web Estate

Planet Drupal - Tue, 02/02/2016 - 13:30

Warden is a solution for in-house development teams and agencies who need to keep track of the status of many Drupal websites, hosted on a variety of different platforms.

Warden gives you a central dashboard which lists all your Drupal websites and highlights any which have issues, for example needing secuity updates.

Hosting companies, like Acquia and Pantheon, have their own reporting tools but these only work if you host on their platforms. If you have an estate of websites which run on multiple platforms you need a tool which can report on them all.

The Warden application is composed of two parts, a Warden module which you need to install on each of your websites and the central Warden dashboard you will need to host on a web server. The Warden dashboard is an application written in Symfony and is freely available on github.

At present only a Drupal integration exists but work is underway to produce a pluggable system which will allow new modules to be created for Wordpress and pure Symfony sites. Others may then wish to contribute additions for their own needs, for example by providing different kinds of reports for the sites.

Warden Dashboard

After correctly configuring the Warden Symfony application you will be presented with the Warden Dashboard. This lists all the sites in your estate with high level details of each. Sites requiring a security update are highlighted as red, sites with module updates which are not security are yellow and sites with no problems are white.

Drupal modules listing screen

The Drupal plugin for the Warden application provides a modules listing screen. This lists all Drupal modules installed across all you estate and allows you to see which Drupal websites have and do not have a particular module installed. This helps when you need to know how many sites need to be updated as a result of a module change or knowing how many of your Drupal sites might be missing a best practice module.

Security

The Warden application uses OpenSSL to encyrpt data which is sent between it and the Drupal website. The PHP OpenSSL Cryptography extension is required for both Warden and the Drupal sites it will take data from. You can also IP restrict which servers can request data from your Drupal websites in the module configuration.

In normal operation the Warden dashboard will poll the sites periodically to request the sites data be refreshed. You can alternatively configure it so that the sites push the data to the Warden dashboard. In either configuration, the site will only send data to the configured dashboard and not to the site making the request for data.

It is also recommended that you use a signed SSL certificate on your Drupal websites and your Warden dashboard.

Where to get Warden

You can download the Warden central applications from GitHub here: https://github.com/teamdeeson/warden 

The Drupal module is available on drupal.org here: https://www.drupal.org/project/warden

What next?

We welcome contributions to the Drupal module or the Symfony application codebase, let us know what you think! 

If you are intersted in integrating Warden into other web tools then you'll need a copy of the PHP API which is available here: https://github.com/teamdeeson/wardenapi  

Categories: Elsewhere

Norbert Preining: Gaming: The Talos Principle – Road to Gehenna

Planet Debian - Tue, 02/02/2016 - 12:30

After finishing the Talos Principle I immediately started to play the extension Road to Gehenna, but was derailed near completion by the incredible Portal Stories: Mel. Now that I finally managed to escape from the test chambers my attention returned to the Road to Gehenna. As with the pair Portal 2 and Portal Stories: Mel, the challenges are going up considerably from the original Talos Principle to the Road to Gehenna. Checking the hours of game play it took me about 24h through all the riddles in Road to Gehenna, but I have to admit, I had some riddles where I needed to cheat.

The Road to Gehenna does not bring much new game play elements, but loads of new riddles. And the best of all, playable on Linux! And as with the original game, the graphics are really well done, while still be playable on my Vaio Pro laptop with Intel integrated graphic card – a plus that is rare in the world of computer games where everyone is expected to have a high-end nVidia or Radeon card. Ok, there is not much action going on where quick graphic computations are necessary, still the impression of the game is great.

The riddles contain the well known elements (connectors, boxes, jammer, etc), but the settings are often spectacular, sometimes very small and narrow, just a few moves if done in the right order, sometimes like wide open fields with lots of space to explore. Transportation between various islands suspended in the air is with vents, giving you a lot of nice flight time!

If one searches a lot, or uses a bit of cheating, one can find good old friends from the Portal series, burried in the sand in one of the world. This is not the only easter egg hidden in the game, there are actually a lot, some of which I have not seen but only read about afterwards. Guess I need to replay the whole game.

Coming back to the riddles, I really believe that the makers have been ingenious in using the few items at hand to create challenging and surprising riddles. As it is so often, many of the riddles look completely impossible at first glance, and often even after staring at them for tens and tens of minutes. Until (and if) one has the the a-ha effect and understands the trick. This often still needs a lot of handwork and trial-error rounds, but all in all the game is well balanced. What is a bit a pain – similar to the original game – are collecting the stars to reach the hidden world and free the admin. There the developers overdid it in my opinion, with some rather absurd and complicated stars.

The end of the game, ascension of the messengers, is rather unspectacular. A short discussion on who remains and then a big closing scene with the messenger being beamed up a la Starship Enterprise, and a closing black screen. But well, the fun was with the riddles.

All in all an extension that is well worth the investment if one enjoyed the original Talos, and is looking for rather challenging riddles. Now that I have finished all the Portal and Talos titles, I am hard thinking of what is next … looking into Braid …

Enjoy!

Categories: Elsewhere

Michal &#268;iha&#345;: Weekly phpMyAdmin contributions 2016-W04

Planet Debian - Tue, 02/02/2016 - 12:00

As I've already mentioned in separate blog post we mostly had some security issues fun in past weeks, but besides that some other work has been done as well.

I've still focused on code cleanups and identified several pieces of code which are no longer needed (given our required PHP version). Another issue related to security updates was to set testing of 4.0 branch using PHP 5.2 as this is what we've messed up in the security release (what is quite bad as this is only branch supporting PHP 5.2).

In addition to this, I've updated phpMyAdmin packages in both Debian and Ubuntu PPA.

All handled issues:

Filed under: Debian English phpMyAdmin | 0 comments

Categories: Elsewhere

Russell Coker: Compatibility and a Linux Community Server

Planet Debian - Tue, 02/02/2016 - 06:44

Compatibility/interoperability is a good thing. It’s generally good for systems on the Internet to be capable of communicating with as many systems as possible. Unfortunately it’s not always possible as new features sometimes break compatibility with older systems. Sometimes you have systems that are simply broken, for example all the systems with firewalls that block ICMP so that connections hang when the packet size gets too big. Sometimes to take advantage of new features you have to potentially trigger issues with broken systems.

I recently added support for IPv6 to the Linux Users of Victoria server. I think that adding IPv6 support is a good thing due to the lack of IPv4 addresses even though there are hardly any systems that are unable to access IPv4. One of the benefits of this for club members is that it’s a platform they can use for testing IPv6 connectivity with a friendly sysadmin to help them diagnose problems. I recently notified a member by email that the callback that their mail server used as an anti-spam measure didn’t work with IPv6 and was causing mail to be incorrectly rejected. It’s obviously a benefit for that user to have the problem with a small local server than with something like Gmail.

In spite of the fact that at least one user had problems and others potentially had problems I think it’s clear that adding IPv6 support was the correct thing to do.

SSL Issues

Ben wrote a good post about SSL security [1] which links to a test suite for SSL servers [2]. I tested the LUV web site and got A-.

This blog post describes how to setup PFS (Perfect Forward Secrecy) [3], after following it’s advice I got a score of B!

From the comments on this blog post about RC4 etc [4] it seems that the only way to have PFS and not be vulnerable to other issues is to require TLS 1.2.

So the issue is what systems can’t use TLS 1.2.

TLS 1.2 Support in Browsers

This Wikipedia page has information on SSL support in various web browsers [5]. If we require TLS 1.2 we break support of the following browsers:

The default Android browser before Android 5.0. Admittedly that browser always sucked badly and probably has lots of other security issues and there are alternate browsers. One problem is that many people who install better browsers on Android devices (such as Chrome) will still have their OS configured to use the default browser for URLs opened by other programs (EG email and IM).

Chrome versions before 30 didn’t support it. But version 30 was released in 2013 and Google does a good job of forcing upgrades. A Debian/Wheezy system I run is now displaying warnings from the google-chrome package saying that Wheezy is too old and won’t be supported for long!

Firefox before version 27 didn’t support it (the Wikipedia page is unclear about versions 27-31). 27 was released in 2014. Debian/Wheezy has version 38, Debian/Squeeze has Iceweasel 3.5.16 which doesn’t support it. I think it is reasonable to assume that anyone who’s still using Squeeze is using it for a server given it’s age and the fact that LTS is based on packages related to being a server.

IE version 11 supports it and runs on Windows 7+ (all supported versions of Windows). IE 10 doesn’t support it and runs on Windows 7 and Windows 8. Are the free upgrades from Windows 7 to Windows 10 going to solve this problem? Do we want to support Windows 7 systems that haven’t been upgraded to the latest IE? Do we want to support versions of Windows that MS doesn’t support?

Windows mobile doesn’t have enough users to care about.

Opera supports it from version 17. This is noteworthy because Opera used to be good for devices running older versions of Android that aren’t supported by Chrome.

Safari supported it from iOS version 5, I think that’s a solved problem given the way Apple makes it easy for users to upgrade and strongly encourages them to do so.

Log Analysis

For many servers the correct thing to do before even discussing the issue is to look at the logs and see how many people use the various browsers. One problem with that approach on a Linux community site is that the people who visit the site most often will be more likely to use recent Linux browsers but older Windows systems will be more common among people visiting the site for the first time. Another issue is that there isn’t an easy way of determining who is a serious user, unlike for example a shopping site where one could search for log entries about sales.

I did a quick search of the Apache logs and found many entries about browsers that purport to be IE6 and other versions of IE before 11. But most of those log entries were from other countries, while some people from other countries visit the club web site it’s not very common. Most access from outside Australia would be from bots, and the bots probably fake their user agent.

Should We Do It?

Is breaking support for Debian/Squeeze, the built in Android browser on Android <5.0, and Windows 7 and 8 systems that haven’t upgraded IE as a web browsing platform a reasonable trade-off for implementing the best SSL security features?

For the LUV server as a stand-alone issue the answer would be no as the only really secret data there is accessed via ssh. For a general web infrastructure issue it seems that the answer might be yes.

I think that it benefits the community to allow members to test against server configurations that will become more popular in the future. After implementing changes in the server I can advise club members (and general community members) about how to configure their servers for similar results.

Does this outweigh the problems caused by some potential users of ancient systems?

I’m blogging about this because I think that the issues of configuration of community servers have a greater scope than my local LUG. I welcome comments about these issues, as well as about the SSL compatibility issues.

Related posts:

  1. Name Server IP and a Dead Server About 24 hours ago I rebooted the system that runs...
  2. Server Costs vs Virtual Server Costs The Claim I have seen it claimed that renting a...
  3. My Blog Server was Cracked On the 1st of August I noticed that the server...
Categories: Elsewhere

Pages

Subscribe to jfhovinne aggregator - Elsewhere