Feed aggregator

Yuriy Gerasimov: Automated deployments to Acquia. Cloud API

Planet Drupal - Fri, 08/07/2016 - 06:45

When you set up your Continuous Integration you really would like to set your deployments automatically. If you use Acquia hosting for your website it does make a lot of sense to use all environments in your workflow. But how you can automate deployments to these environments without touching UI (copying database, files, deploying code)?

The answer is in Cloud API

You can call them either with drush command or curl request. We will touch the drush commands approach in this post.

I personally was heavily involved in workflow called CIBox that uses separate from Acquia github repo.

I used 'master' branch to deploy to DEV environment. But both STAGING and PRODUCTION environments are tag based.

DEV environment deployment

First step of deployment for me was to sync the repositories.

cd /var/git/acquia git pull github master git push origin master # Sleep for 30 seconds. We expect Acquia to update the code. sleep 30

Little note: all these commands are run on CI server for me. So you will find plenty of ssh and scp to Acquia servers later.

Repository /var/git/acquia is a clone from hosting repo but with set up remote to our own private repo. If you use hosting repo as primary you probably won't need this step.

In Acquia UI I have set up DEV environment to follow master branch. So code gets deployed automatically.

In my CI set up, I keep copy of project's database on CI server to use it in all builds. So I deploy this db to DEV environment as next step. Workflow diagram looks like this.

Code snippet

# Copy db to DEV server scp /var/www/backup/DBNAME.sql.gz PROJECT.dev@staging-XXXX.prod.hosting.acquia.com:/home/PROJECT/proddump.sql.gz # Deploy db on DEV server ssh -t PROJECT.dev@staging-XXXX.prod.hosting.acquia.com 'rm -rf /home/PROJECT/DBNAME.sql && gunzip /home/PROJECT/proddump.sql.gz && cd /var/www/html/PROJECT.dev/docroot && drush sql-drop -y && `drush sql-connect` < /home/PROJECT/proddump.sql'

Remember in order to run this operation you need to set up your ssh keys so jenkins user (I use Jenkins as CI tool) can go to Acquia servers without password being requested.

And the last step is run all the updates, registry rebuilds and whatever your project requires.

# Run registry rebuild and clear caches ssh -t PROJECT.dev@staging-XXXX.prod.hosting.acquia.com 'cd /var/www/html/PROJECT.dev/docroot && drush php-eval "registry_rebuild();" && drush cc all -y'   # Run hook updates ssh -t PROJECT.dev@staging-XXXX.prod.hosting.acquia.com 'cd /var/www/html/PROJECT.dev/docroot && drush -y updb'

These commands actually can be set up with drush aliases. But I used terminal approach as already using it for deploying database. Just consistency reasons.

Another part I skip here is copying files over. We don't do that. Instead we enable stage_file_proxy on DEV and STAGE environments and point them to PROD so files got copied over upon request. This saves plenty of space.

STAGE environment deployment

As staging environment uses tags we need to change our code deployment part.

In order to use Cloud API you need to set up special private key in Acquia UI. Please review https://docs.acquia.com/cloud/api/auth for more details.

After setting up the key, ssh to DEV server and run command 'drush ac-api-login' and provide your email and key to it. In this way you will set up all your credentials and be able to run Cloud API drush commands.

And now, we can deploy the code.

ssh -t PROJECT.dev@staging-XXXX.prod.hosting.acquia.com 'cd /var/www/html/PROJECT.dev/docroot && drush @PROJECT.dev ac-code-deploy test --verbose'   # Sleep for 30 seconds. We expect Acquia to update the code. sleep 30

This will deploy the code from DEV environment to STAGE and adds the tag automatically. Basically it mimics the operation of dragging the code bar in Acquia UI.

All other steps (database deployment and cache clear) are the same as with DEV environment.

PROD environment deployments

Production deployment is going to be the same as STAGE with only difference we need to ssh to STAGE server to deploy the code. And of course we do not to deploy database anywhere.

I am sure in your projects you might need to add some more steps. Maybe reindexing solr, or clearing varnish caches. All these can be done with drush commands.

How do you do deployments? Please share your experience in comments.

Tags: drupal 7drupal planet
Categories: Elsewhere

Steve Kemp: I've been moving and updating websites.

Planet Debian - Fri, 08/07/2016 - 05:30

I've spent the past days updating several of my websites to be "responsive". Mostly that means I open the site in firefox then press Ctrl-alt-m to switch to mobile-view. Once I have the mobile-view I then fix the site to look good in small small space.

Because my general design skills are poor I've been fixing most sites by moving to bootstrap, and ensuring that I don't use headers/footers that are fixed-position.

Beyond the fixes to appearances I've also started rationalizing the domains, migrating content across to new homes. I've got a provisional theme setup at steve.fi, and I've moved my blog over there too.

The plan for blog-migration went well:

  • Setup a redirect to from https://blog.steve.org.uk to https://blog.steve.fi/
  • Replace the old feed with a CGI script which outputs one post a day, telling visitors to update their feed.
    • This just generates one post, but the UUID of the post has the current date in it. That means it will always be fresh, and always be visible.
  • Updated the template/layout on the new site to use bootstrap.

The plan was originally to setup a HTTP-redirect, but I realized that this would mean I'd need to keep the redirect in-place forever, as visitors would have no incentive to fix their links, or update their feeds.

By adding the fake-RSS-feed, pointing to the new location, I am able to assume that eventually people will update, and I can drop the dns record for blog.steve.org.uk entirely - Already google seems to have updated its spidering and searching shows the new domain already.

Categories: Elsewhere

Appnovation Technologies: Drupal 8 and the User Experience Upgrade

Planet Drupal - Fri, 08/07/2016 - 02:27

Drupal and user experience can sometimes be at odds.

Categories: Elsewhere

Lior Kaplan: First uses of the PHP 5.4 security backports

Planet Debian - Fri, 08/07/2016 - 02:07

I recently checked the Debian PHP 5.4 changelog and found out this message (5.4.45-0+deb7u3 and 5.4.45-0+deb7u4):

* most patches taken from https://github.com/kaplanlior/php-src
Thanks a lot to Lior Kaplan for providing them.

I was very pleased to see my work being used, and I hope this would save other time while providing PHP long term support.

Also, while others do similar work (e.g. Remi from RedHat), it seems I’m the only when that make this over GIT and with full references (e.g. commit info, CVE info and bug number).

Comments and suggestions are always welcome… either mail or even better – a pull request.

Filed under: Debian GNU/Linux, PHP
Categories: Elsewhere

Matthew Garrett: Bluetooth LED bulbs

Planet Debian - Fri, 08/07/2016 - 00:38
The best known smart bulb setups (such as the Philips Hue and the Belkin Wemo) are based on Zigbee, a low-energy, low-bandwidth protocol that operates on various unlicensed radio bands. The problem with Zigbee is that basically no home routers or mobile devices have a Zigbee radio, so to communicate with them you need an additional device (usually called a hub or bridge) that can speak Zigbee and also hook up to your existing home network. Requests are sent to the hub (either directly if you're on the same network, or via some external control server if you're on a different network) and it sends appropriate Zigbee commands to the bulbs.

But requiring an additional device adds some expense. People have attempted to solve this in a couple of ways. The first is building direct network connectivity into the bulbs, in the form of adding an 802.11 controller. Go through some sort of setup process[1], the bulb joins your network and you can communicate with it happily. Unfortunately adding wifi costs more than adding Zigbee, both in terms of money and power - wifi bulbs consume noticeably more power when "off" than Zigbee ones.

There's a middle ground. There's a large number of bulbs available from Amazon advertising themselves as Bluetooth, which is true but slightly misleading. They're actually implementing Bluetooth Low Energy, which is part of the Bluetooth 4.0 spec. Implementing this requires both OS and hardware support, so older systems are unable to communicate. Android 4.3 devices tend to have all the necessary features, and modern desktop Linux is also fine as long as you have a Bluetooth 4.0 controller.

Bluetooth is intended as a low power communications protocol. Bluetooth Low Energy (or BLE) is even lower than that, running in a similar power range to Zigbee. Most semi-modern phones can speak it, so it seems like a pretty good choice. Obviously you lose the ability to access the device remotely, but given the track record on this sort of thing that's arguably a benefit. There's a couple of other downsides - the range is worse than Zigbee (but probably still acceptable for any reasonably sized house or apartment), and only one device can be connected to a given BLE server at any one time. That means that if you have the control app open while you're near a bulb, nobody else can control that bulb until you disconnect.

The quality of the bulbs varies a great deal. Some of them are pure RGB bulbs and incapable of producing a convincing white at a reasonable intensity[2]. Some have additional white LEDs but don't support running them at the same time as the colour LEDs, so you have the choice between colour or a fixed (and usually more intense) white. Some allow running the white LEDs at the same time as the RGB ones, which means you can vary the colour temperature of the "white" output.

But while the quality of the bulbs varies, the quality of the apps doesn't really. They're typically all dreadful, competing on features like changing bulb colour in time to music rather than on providing a pleasant user experience. And the whole "Only one person can control the lights at a time" thing doesn't really work so well if you actually live with anyone else. I was dissatisfied.

I'd met Mike Ryan at Kiwicon a couple of years back after watching him demonstrate hacking a BLE skateboard. He offered a couple of good hints for reverse engineering these devices, the first being that Android already does almost everything you need. Hidden in the developer settings is an option marked "Enable Bluetooth HCI snoop log". Turn that on and all Bluetooth traffic (including BLE) is dumped into /sdcard/btsnoop_hci.log. Turn that on, start the app, make some changes, retrieve the file and check it out using Wireshark. Easy.

Conveniently, BLE is very straightforward when it comes to network protocol. The only thing you have is GATT, the Generic Attribute Protocol. Using this you can read and write multiple characteristics. Each packet is limited to a maximum of 20 bytes. Most implementations use a single characteristic for light control, so it's then just a matter of staring at the dumped packets until something jumps out at you. A pretty typical implementation is something like:


where r, g and b are each just a single byte representing the corresponding red, green or blue intensity. 0x56 presumably indicates a "Set the light to these values" command, 0xaa indicates end of command and 0xf0 indicates that it's a request to set the colour LEDs. Sending 0x0f instead results in the previous byte (0x00 in this example) being interpreted as the intensity of the white LEDs. Unfortunately the bulb I tested that speaks this protocol didn't allow you to drive the white LEDs at the same time as anything else - setting the selection byte to 0xff didn't result in both sets of intensities being interpreted at once. Boo.

You can test this out fairly easily using the gatttool app. Run hcitool lescan to look for the device (remember that it won't show up if anything else is connected to it at the time), then do gatttool -b deviceid -I to get an interactive shell. Type connect to initiate a connection, and once connected send commands by doing char-write-cmd handle value using the handle obtained from your hci dump.

I did this successfully for various bulbs, but annoyingly hit a problem with one from Tikteck. The leading byte of each packet was clearly a counter, but the rest of the packet appeared to be garbage. For reasons best known to themselves, they've implemented application-level encryption on top of BLE. This was a shame, because they were easily the best of the bulbs I'd used - the white LEDs work in conjunction with the colour ones once you're sufficiently close to white, giving you good intensity and letting you modify the colour temperature. That gave me incentive, but figuring out the protocol took quite some time. Earlier this week, I finally cracked it. I've put a Python implementation on Github. The idea is to tie it into Ulfire running on a central machine with a Bluetooth controller, making it possible for me to control the lights from multiple different apps simultaneously and also integrating with my Echo.

I'd write something about the encryption, but I honestly don't know. Large parts of this make no sense to me whatsoever. I haven't even had any gin in the past two weeks. If anybody can explain how anything that's being done there makes any sense at all[3] that would be appreciated.

[1] typically via the bulb pretending to be an access point, but also these days through a terrifying hack involving spewing UDP multicast packets of varying lengths in order to broadcast the password to associated but unauthenticated devices and good god the future is terrifying

[2] For a given power input, blue LEDs produce more light than other colours. To get white with RGB LEDs you either need to have more red and green LEDs than blue ones (which costs more), or you need to reduce the intensity of the blue ones (which means your headline intensity is lower). Neither is appealing, so most of these bulbs will just give you a blue "white" if you ask for full red, green and blue

[3] Especially the bit where we calculate something from the username and password and then encrypt that using some random numbers as the key, then send 50% of the random numbers and 50% of the encrypted output to the device, because I can't even

Categories: Elsewhere

Lior Kaplan: Anonymous CVE requests

Planet Debian - Thu, 07/07/2016 - 19:21

A year ago I’ve blogged about people requesting CVE without letting upstream know. On the other hand, per requests from Debian, I’m working on improving PHP upstream CVE request process. For the last few release this means I ask the security list members which issues they think should have CVE and ask for them in parallel to the release being made (usually in the space between the release being tagged publicly and is actually announced).

In the last week, I’ve encountered a case where a few CVE were assigned to old PHP issues without any public notice. The fixes for these issues have been published a year ago (August 2015). And I find out about these assignment through warning published by the distributions (mostly Debian, which I’m close to).

Sometimes things fall between the chairs, and it’s perfectly OK to ask for CVE to make sure security issues do get attention even if time has passed. But after the issues (and fixes) are public, I don’t see a reason to do so without making the request itself public as well. And even if the request wasn’t public, at least notify upstream so this info can be added to the right places. Most of these bug were found out when I started to add sequential number into the CVE search after getting an a notice from Debian for two of the issues.

  • CVE-2015-8873 for PHP #69793
  • CVE-2015-8874 for PHP #66387
  • CVE-2015-8876 for PHP #70121
  • CVE-2015-8877 for PHP #70064
  • CVE-2015-8878 for PHP #70002
  • CVE-2015-8879 for PHP #69975
  • CVE-2015-8880 for PHP aa8cac57 (Dec 2015)

And while working on processing these issues for PHP, I also notice they weren’t updated for libGD where appropriate (including recent issues).

I hope this blog post will reach the anonymous people behind these CVE requests, and also the people assigning them. Without transparency and keeping things in synchronization, the idea of having a centralized location for security warning is not going to accomplish its goals.

Filed under: Debian GNU/Linux, PHP
Categories: Elsewhere

Michal &#268;iha&#345;: uTidylib 0.3

Planet Debian - Thu, 07/07/2016 - 18:00

Several years ago I've complained about uTidylib not being maintained upstream. Since that time I've occasionally pushed some fixes to my GitHub repository with uTidylib code, but without any clear intentions to take it over.

Time has gone and there was still no progress and I started to consider becoming upstream maintainer as well. I quickly got approval from Cory Dodt, who was the original author of this code, unfortunately he is not owner of the PyPI entry and the claim request seems to have no response (if you know how to get in touch with "cntrlr" or how to take over PyPI module please let me know).

Anyway the amount of patches in my repository is big enough to warrant new release. Additionally Debian bug report about supporting new HTML tidy library came in and that made me push towards releasing 0.3 version of the uTidylib.

As you might guess, the amount of changes against original uTidylib is quite huge, to name the most important ones:

Anyway as I can not update PyPI entry, the downloads are currently available only on my website: https://cihar.com/software/utidylib/

Filed under: Debian English uTidylib | 0 comments

Categories: Elsewhere


Subscribe to jfhovinne aggregator