Elsewhere

Tim Retout: Cowbuilder and Tor

Planet Debian - Tue, 22/07/2014 - 23:31

You've installed apt-transport-tor to help prevent targeted attacks on your system. Great! Now you want to build Debian packages using cowbuilder, and you notice these are still using plain HTTP.

If you're willing to fetch the first few packages without using apt-transport-tor, this is as easy as:

  • Add 'EXTRAPACKAGES="apt-transport-tor"' to your pbuilderrc.
  • Run 'cowbuilder --update'
  • Set 'MIRRORSITE=tor+http://http.debian.net/debian' in pbuilderrc.
  • Run 'cowbuilder --update' again.

Now any future builds should fetch build-dependencies over Tor.

Unfortunately, creating a base.cow from scratch is more problematic. Neither 'debootstrap' nor 'cdebootstrap' actually rely on apt acquire methods to download files - they look at the URL scheme themselves to work out where to fetch from. I think it's a design point that they shouldn't need apt, anyway, so that you can debootstrap on non-Debian systems. I don't have a good solution beyond using some other means to route these requests over Tor.

Categories: Elsewhere

Neil Williams: Validating ARMMP device tree blobs

Planet Debian - Tue, 22/07/2014 - 23:18

I’ve done various bits with ARMMP and LAVA on this blog already, usually waiting until I’ve got all the issues ironed out before writing it up. However, this time I’m just going to do a dump of where it’s at, how it works and what can be done.

I’m aware that LAVA can seem mysterious at first, the package description has improved enormously recently, thanks to exposure in Debian: LAVA is a continuous integration system for deploying operating systems onto physical and virtual hardware for running tests. Tests can be simple boot testing, bootloader testing and system level testing, although extra hardware may be required for some system tests. Results are tracked over time and data can be exported for further analysis.

The LAVA documentation has a glossary of terms like result bundle and all the documentation is also available in the lava-server-doc package.

The goal is to validate the dtbs built for the Debian ARMMP kernel. One of the most accessible ways to get the ARMMP kernel onto a board for testing is tftp using the Debian daily DI builds. Actually using the DI initrd can come later, once I’ve got a complete preseed config so that the entire install can be automated. (There are some things to sort out in LAVA too before a full install can be deployed and booted but those are at an early stage.) It’s enough at first to download the vmlinuz which is common to all ARMMP deployments, supply the relevant dtb, partner those with a minimal initrd and see if the board boots.

The first change comes when this process is compared to how boards are commonly tested in LAVA – with a zImage or uImage and all/most of the modules already built in. Packaged kernels won’t necessarily raise a network interface or see the filesystem without modules, so the first step is to extend a minimal initramfs to include the armmp modules.

apt install pax u-boot-tools

The minimal initramfs I selected is one often used within LAVA:

wget http://images.armcloud.us/lava/common/linaro-image-minimal-initramfs-genericarmv7a.cpio.gz.u-boot

It has a u-boot header added, as most devices using this would be using u-boot and this makes it easier to debug boot failures as the initramfs doesn’t need to have the header added, it can simply be downloaded to a local directory and passed to the board as a tftp location. To modify it, the u-boot header needs to be removed. Rather than assuming the size, the u-boot tools can (indirectly) show the size:

$ ls -l linaro-image-minimal-initramfs-genericarmv7a.cpio.gz.u-boot -rw-r--r-- 1 neil neil 5179571 Nov 26 2013 linaro-image-minimal-initramfs-genericarmv7a.cpio.gz.u-boot $ mkimage -l linaro-image-minimal-initramfs-genericarmv7a.cpio.gz.u-boot Image Name: linaro-image-minimal-initramfs-g Created: Tue Nov 26 22:30:49 2013 Image Type: ARM Linux RAMDisk Image (gzip compressed) Data Size: 5179507 Bytes = 5058.11 kB = 4.94 MB Load Address: 00000000 Entry Point: 00000000

Referencing http://www.omappedia.com/wiki/Development_With_Ubuntu, the header size is the file size minus the data size listed by mkimage.

5179571 - 5179507 == 64

So, create a second file without the header:

dd if=linaro-image-minimal-initramfs-genericarmv7a.cpio.gz.u-boot of=linaro-image-minimal-initramfs-genericarmv7a.cpio.gz skip=64 bs=1

decompress it

gunzip linaro-image-minimal-initramfs-genericarmv7a.cpio.gz

Now for the additions

dget http://ftp.uk.debian.org/debian/pool/main/l/linux/linux-image-3.14-1-armmp_3.14.12-1_armhf.deb

(Yes, this process will need to be repeated when this package is rebuilt, so I’ll want to script this at some point.)

dpkg -x linux-image-3.14-1-armmp_3.14.12-1_armhf.deb kernel-dir cd kernel-dir

Pulling in the modules we need for most needs, comes thanks to a script written by the Xen folks. The set is basically disk, net, filesystems and LVM.

find lib -type d -o -type f -name modules.\* -o -type f -name \*.ko \( -path \*/kernel/lib/\* -o -path \*/kernel/crypto/\* -o -path \*/kernel/fs/mbcache.ko -o -path \*/kernel/fs/ext\* -o -path \*/kernel/fs/jbd\* -o -path \*/kernel/drivers/net/\* -o -path \*/kernel/drivers/ata/\* -o -path \*/kernel/drivers/scsi/\* -o -path \*/kernel/drivers/md/\* \) | pax -x sv4cpio -s '%lib%/lib%' -d -w >../cpio gzip -9f cpio

original Xen script (GPL-3+)

I found it a bit confusing that i is used for extract by cpio, but that’s how it is. Extract the minimal initramfs to a new directory:

sudo cpio -id < ../linaro-image-minimal-initramfs-genericarmv7a.cpio

Extract the new cpio into the same location. (Yes, I could do this the other way around and pipe the output of find into the already extracted location but that's for when I get a script to do this):

sudo cpio --no-absolute-filenames -id < ../ramfs/cpio

CPIO Manual

Use newc format, the new (SVR4) portable format, which supports file systems having more than 65536 i-nodes. (4294967295 bytes)
(41M)

find . | cpio -H newc -o > ../armmp-image.cpio

... and add the u-boot header back:

mkimage -A arm -T ramdisk -C none -d armmp-image.cpio.gz debian-armmp-initrd.cpio.gz.u-boot Now what?

Now send the combination to LAVA and test it.

Results bundle for a local LAVA test job using this technique. (18k)

submission JSON - uses file:// references, so would need modification before being submitted to LAVA elsewhere.

complete log of the test job (72k)

Those familiar with LAVA will spot that I haven't optimised this job, it boots the ARMMP kernel into a minimal initramfs and then expects to find apt and other tools. Actual tests providing useful results would use available tools, add more tools or specify a richer rootfs.

The tests themselves are very quick (the job described above took 3 minutes to run) and don't need to be run particularly often, just once per board type per upload of the ARMMP kernel. LAVA can easily run those jobs in parallel and submission can be automated using authentication tokens and the lava-tool CLI. lava-tool can be installed without lava-server, so can be used in hooks for automated submissions.

Extensions

That's just one DTB and one board. I have a range of boards available locally:

* iMX6Q Wandboard (used for this test)
* iMX.53 Quick Start Board (needs updated u-boot)
* Beaglebone Black
* Cubie2
* CubieTruck
* arndale (no dtb?)
* pandaboard

Other devices available could involve ARMv7 devices hosted at www.armv7.com and validation.linaro.org - as part of a thank you to the Debian community for providing the OS which is (now) running all of the LAVA infrastructure.

That doesn't cover all of the current DTBs (and includes many devices which have no DTBs) so there is plenty of scope for others to get involved.

Hopefully, the above will help get people started with a suitable kernel+dtb+initrd and I'd encourage anyone interested to install lava-server and have a go at running test jobs based on those so that we start to build data about as many of the variants as possible.

(If anyone in DI fancies producing a suitable initrd with modules alongside the DI initrd for armhf builds, or if anyone comes up with a patch for DI to do that, it would help enormously.)

This will at least help Debian answer the question of what the Debian ARMMP package can actually support.

For help on LAVA, do read through the documentation and then come to us at #linaro-lava or the linaro-validation mailing list or file bugs in Debian: reportbug lava-server.

, so you can ask me.

I'm giving one talk on the LAVA software and there will be a BoF on validation and CI in Debian.

Categories: Elsewhere

Greater Los Angeles Drupal (GLAD): Drupal Migrate using xml 0 to 35

Planet Drupal - Tue, 22/07/2014 - 20:55

Using Drupal Migrate is a great way to move you content into Drupal. Unfortunately the documentation for xml import can be obscure. This comes about when those that developed the module try to communicate how they did what they did to someone that did not do the work. Things that seem obvious to them are not to someone else.

I have spent some time recently importing content using xml. In no way am I an expert that is speeding down the fast lane, something more in the cruising around town at a comfortable 35 mph.

To use Drupal Migrate you need to define your own class. A class is php code that is used in Object Oriented Programming that defines your data and defines how you can manipulate your data. Most of the actual migration work is done with the classes provide by the migrate module, you simply have to define the details of your migration.

Constructor - The constructor modifies the migration modules classes to define your specific data. I was able to follow the SourceList method, this provides one xml (file or feed) that contains the ID number for all the content you want to import, and a second (file or feed) that contains the content. The wine example migrate has this but understanding what it really wants is more difficult to understand.

Below is my class file explained:
=====================
<?php

/**
* @file
* Vision Article migration.
*/

/**
* Vision Article migration class.
*/
class VisionArticleMigration extends XMLMigration {
public function __construct() {
parent::__construct();
$this->description = t('XML feed of Ektron Articles.');

---------------
So far pretty easy. You need to name your class, extend from the proper migration. and give it an extension.

-----------------

// There isn't a consistent way to automatically identify appropriate
// "fields" from an XML feed, so we pass an explicit list of source fields.
$fields = array(
'id' => t('ID'),
'lang_type' => t('Language'),
'type' => t('Type'),
'image' => t('Image'),
'authors' => t('Authors'),
'article_category' => t('Article Category'),
'article_series_title' => t('Article Series Title'),
'article_part_no' => t('Article Series Part Number'),
'article_title' => t('Article Title'),
'article_date' => t('Article Date'),
'article_display_date' => t('Article Display Date'),
'article_dropheader' => t('Article Dropheader'),
'article_body' => t('Article Body'),
'article_author_name' => t('Article Author Name'),
'article_author_url' => t('Article Author Email Address'),
'article_authors' => t('Article Additional Authors'),
'article_postscript' => t('Article Postscript'),
'article_link_text' => t('Article Link text'),
'article_link' => t('Article Link'),
'article_image' => t('Article Image'),
'article_image_folder' => t('Article Image Folder'),
'article_image_alt' => t('Article Image Alt'),
'article_image_title' => t('Article Image Title'),
'article_image_caption' => t('Article Image Caption'),
'article_image_credit' => t('Article Image Credit'),
'article_sidebar_element' => t('Article Side Bar Content'),
'article_sidebar_element_margin' => t('Article Margin between Sidebar Content'),
'article_archived_html_content' => t('Article HTML Content from old system'),
'article_video_id' => t('Article ID of Associated Video Article'),
'metadata_title' => t('Metadata Title'),
'metadata_description' => t('Metadata Description'),
'metadata_keywords' => t('Metadata Keywords'),
'metadata_google_sitemap_priority' => t('Metadata Google Sitemap Priority'),
'metadata_google_sitemap_change_frequency' => t('Metadata Google Sitemap Change Freequency'),
'metadata_collection_number' => t('Metadata Collection Number'),
'title' => t('Title'),
'teaser' => t('Teaser'),
'alias' => t('Alias from old system'),
'taxonomy' => t('Taxonomy'),
'created_date' => t('Date Created')
);

-------------------
So what doe this mean?
You will need a field name below. It has nothing to do with your xml file, you will need a field for each thing you want to import. Such as article_image_alt is the alt text for the image. Later you will define the xpath to load this variable. This will start to come together below, just remember each unique piece of information needs a variable.

---------------------

// The source ID here is the one retrieved from the XML listing URL, and
// used to identify the specific item's URL.
$this->map = new MigrateSQLMap($this->machineName,
array(
'ID' => array(
'type' => 'int',
'unsigned' => TRUE,
'not null' => TRUE,
'description' => 'Source ID',
)
),
MigrateDestinationNode::getKeySchema()
);

---------------------
This has to do with setting up the migration table in the database. This has to do with the input database, the Source ID is the field in the input file that has the pointer to the data record. My source file looks like:

567
1054

So we need a table with a field for the id which an integer.

-----------------------

// Source list URL.
$list_url = 'http://www.vision.org/visionmedia/generateexportlist.aspx';
// Each ID retrieved from the list URL will be plugged into :id in the
// item URL to fetch the specific objects.
// @todo: Add langtype for importing translated content.
$item_url = 'http://www.vision.org/visionmedia/generatecontentxml.aspx?id=:id';

// We use the MigrateSourceList class for any source where we obtain the
// list of IDs to process separately from the data for each item. The
// listing and item are represented by separate classes, so for example we
// could replace the XML listing with a file directory listing, or the XML
// item with a JSON item.
$this->source = new MigrateSourceList(new MigrateListXML($list_url),
new MigrateItemXML($item_url), $fields);

$this->destination = new MigrateDestinationNode('vision_article');

-----------------

Now we are setting up the magic. We setup a list url that contains the ID's of all the content to import, then another one that uses this ID to fetch the details for this ID. Then you tell Migrate to use the MigrateListXML to find the items to import with MigrateItemXML. Then finally in the MigrateDestinationNode to tell Migrate which content type to use. This means we need a separate migration class for each content type to import. I have been creating each class in it's own inc file and adding this to the files section in the info file.

-----------------

// TIP: Note that for XML sources, in addition to the source field passed to
// addFieldMapping (the name under which it will be saved in the data row
// passed through the migration process) we specify the Xpath used to retrieve
// the value from the XML.
$this->addFieldMapping('created', 'created_date')
->xpath('/content/CreateDate');

------------------
Now we map the source field with the destination field. Created is the field name in the content type (vision_article), created_date is from our fields section above. Remember I said we needed a definiation for each part of the content we want to import. The xpath then points to the data in the xml feed. So this says take the content of the /contnet/CreateDate in the xml file and load this into the source variable created_date, then store this in the created field in a new vision_article content item. I say this in this way because if you do like me and cut and paste and forget to change the source varable, the source varable will contain the bottom data from xpath.

------------------

$this->addFieldMapping('field_category', 'article_category')
->defaultValue(1)
->xpath('/content/html/root/article/Category');

-------------------

You can set a default value in case the xml does not contain any data

----------

$this->addFieldMapping('field_series_title', 'article_series_title')
->xpath('/content/html/root/article/ArticleSeriesTitle');
$this->addFieldMapping('field_part_number', 'article_part_no')
->xpath('/content/html/root/article/ArticlePartNo');
$this->addFieldMapping('field_h1_title', 'article_title')
->arguments(array('format' => 'filtered_html'))
->xpath('/content/html/root/article/Title');
$this->addFieldMapping('field_display_date', 'article_display_date')
->xpath('/content/html/root/article/DisplayDate');
$this->addFieldMapping('field_drophead', 'article_dropheader')
->arguments(array('format' => 'filtered_html'))
->xpath('/content/Taxonomy');

-------------

Another field argument, the default content type is plain text, so if your content contains HTML you need to set the correct format here.

---------------

$this->addFieldMapping('body', 'article_body')
->arguments(array('format' => 'filtered_html'))
->xpath('/content/html/root/article/Body');
$this->addFieldMapping('body:summary', 'teaser')
->arguments(array('format' => 'filtered_html'))
->xpath('/content/Teaser');

-----------

Note you can set the teaser as a part of the body. One of the drush migrate commands make is easy to discover the additional parts of your content field, drush mfd (Migrate Field Destinations). This will display all the destination fields and their options.

------------

$this->addFieldMapping('field_author', 'article_author_email')
->xpath('/content/html/root/article/AuthorURL');
$this->addFieldMapping('field_author:title', 'article_author_name')
->xpath('/content/html/root/article/AuthorName');
$this->addFieldMapping('field_ext_reference_title', 'article_postscript')
->arguments(array('format' => 'filtered_html'))
->xpath('/content/html/root/article/Postscript');

---------
see explanation below
--------
$this->addFieldMapping('field_article_image:file_replace')
->defaultValue(MigrateFile::FILE_EXISTS_REUSE); //FILE_EXISTS_REUSE is in the MigrateFile class
$this->addFieldMapping('field_article_images', 'article_image')
->xpath('/content/html/root/article/Image/File/img/file_name');
$this->addFieldMapping('field_article_images:source_dir', 'article_image_folder')
->xpath('/content/html/root/article/Image/File/img/file_path');
$this->addFieldMapping('field_article_images:alt', 'article_image_alt')
->xpath('/content/html/root/article/Image/File/img/@alt');
$this->addFieldMapping('field_article_images:title', 'article_image_title')
->xpath('/content/html/root/article/Image/File/img/@alt');

--------------

This section gets tricky. You are importing an Image or other file. The default migration for a file is MigrateFileUrl. You can migrate all your files ahead of time or as I am doing do it inline. The main components for this is the main field, which is the file name, and the source_dir for the path to this image. Drual 7 has a database table for the files is uses with the url to the file. MigrateFile then uploads this file to the public folder and creates an entry into the files_,amaged table to indicate the url. What I did was copy all the images to a public location on S3 storage so I did not want Migrate to create a new file but use the existing file. Thus the file_replace setting to the constant MigrateFile::FILE_EXISTS_REUSE. This tells migrate to use the existing file and make an entry in the file_managed table for this file.

Later in the PrepareRow method I will show how we separate this and add it to the xml.

------------

$this->addFieldMapping('field_archive', 'article_archived_html_content')
->xpath('/content/archive_html');
$this->addFieldMapping('field_ektron_id', 'id')
->xpath('/content/ID');
$this->addFieldMapping('field_ektron_alias', 'alias')
->xpath('/content/html/Alias');
$this->addFieldMapping('field_sidebar', 'article_sidebar_element')
->arguments(array('format' => 'filtered_html'))
->xpath('/content/html/root/article/SidebarElement/SidebarElementInformation');
$this->addFieldMapping('field_slider_image:file_replace')
->defaultValue(MigrateFile::FILE_EXISTS_REUSE); //FILE_EXISTS_REUSE is in the MigrateFile class
$this->addFieldMapping('field_slider_image', 'image')
->xpath('/content/Image/file_name');
$this->addFieldMapping('field_slider_image:source_dir', 'image_folder')
->xpath('/content/Image/file_path');
$this->addFieldMapping('field_slider_image:alt', 'image_alt')
->xpath('/content/Title');
$this->addFieldMapping('field_slider_image:title', 'image_title')
->xpath('/content/Title');
$this->addFieldMapping('title', 'title')
->xpath('/content/Title');
$this->addFieldMapping('title_field', 'title')
->xpath('/content/Title');

// Declare unmapped source fields.
$unmapped_sources = array(
'article_author_url',
'article_authors',
'article_sidebar_element_margin',
'article_video_id',
'metadata_title',
'metadata_description',
'metadata_keywords',
'metadata_google_sitemap_priority',
'metadata_google_sitemap_change_frequency',
'metadata_collection_number',

);

-------------

If you are not using a source field, best practices state that you declare it in the unmapped sources

------------

$this->addUnmigratedSources($unmapped_sources);

// Declare unmapped destination fields.
$unmapped_destinations = array(
'revision_uid',
'changed',
'status',
'promote',
'sticky',
'revision',
'log',
'language',
'tnid',
'is_new',
'body:language',
);

----------------------

If you are not using a destination field best practices state that you declare in the unmaped destinations array. Note if you later use this field you need to remove it from the unused array.

---------------------

$this->addUnmigratedDestinations($unmapped_destinations);

if (module_exists('path')) {
$this->addFieldMapping('path')
->issueGroup(t('DNM'));
if (module_exists('pathauto')) {
$this->addFieldMapping('pathauto')
->issueGroup(t('DNM'));
}
}
if (module_exists('statistics')) {
$this->addUnmigratedDestinations(array('totalcount', 'daycount', 'timestamp'));
}
}

------------

The rest of the constructor is from the example. Did not cause me a problem so did not worry about it.

------------
/**
* {@inheritdoc}
*/

---------------

Now we can add our own magic. We can effect the data from the content item before it is saved in to the content item.

-----------------

public function prepareRow($row) {
if (parent::prepareRow($row) === FALSE) {
return FALSE;
}
$ctype = (string)$row->xml->Type;
//set variable for return code
$ret = FALSE;
//dpm($row);

------------

You will see these scattered through the prepareRow function. These are the devel command to print to the screen for debuging. They should be commented out but you can see the process I went through to debug my particular prepareRow. Also note this is a great use of the Migrate UI, these print statment only help you in the web interface, if you use Drush you will not see these diagnostic prints.

---------------

if ($ctype == '12'){

---------------

This is specific to my migrate. The following code is only applicable to a content type of 12. The other content types have a different data structure. If prepareRow returns False the row will be skipped.

------------------

// Map the article_postscript source field to the new destination fields.
//if((string)$row->xml->root->article->Title == ''){
// $row->xml->root->article->Title = $row->xml->root->Title;
//}
$postscript = $row->xml->html->root->article->Postscript->asXML();
$postscript = str_replace('','',$postscript);
$postscript = str_replace('','',$postscript);
$row->xml->html->root->article->Postscript = $postscript;

-------------------

Again this is something unique to my migrate. The content structure is contained in xml so the HTML is recognized by SimpleXML as xml. So the asXML() function returns a string containing the xml of the node. Now I can save this string to the node and it becomes a string node and is back to straight html. So I need to do this for all the nodes that contain html. Most of the time you will be able to pass the html string as a node and will not have to do this transform.

-------------------

//converts html nodes to string so they will load.
$body = $row->xml->html->root->article->Body->asXML();
$body = str_replace('','',$body);
$body = str_replace('','',$body);
$row->xml->html->root->article->Body = $body;
$title = $row->xml->html->root->article->Title->asXML();
$title = str_replace('','',$title);
$title = str_replace('','',$title);
$row->xml->html->root->article->Title = $title;
$drophead = $row->xml->html->root->article->Dropheader->asXML();
$drophead = str_replace('','',$drophead);
$drophead = str_replace('','',$drophead);
//If Dropheader is empty
$drophead = str_replace('','',$drophead);
$row->xml->html->root->article->Dropheader = $drophead;
//Array to allow conversion of Category text to IS
$cat_tax = array(
'Science and Environment' => 1,
'History' => 2,
'Social Issues' => 3,
'Family and Relationships' => 4,
'Life and Health' => 5,
'Religion and Spirituality' => 6,
'Biography' => 7,
'Ethics and Morality'=> 8,
'Society and Culture' => 9,
'Current Events and Politics' => 10,
'Philosophy and Ideas' => 11,
'Personal Development' => 12,
'Reviews' => 13,
'From the Publisher' => 14,
'Interviews' => 17,
);
//Convert additional taxonomies to tags
//$tax_id_in = (string)$row->xml->Taxonomy;
//$tax_id_array = explode(',',$tax_id_in);
//$tax_in_array = array();
//foreach($tax_id_array as $tax){
// If(is_null($cat_tax[tax]))
// $tax_in_array[] = $cat_tax[$tax];
//}
//$new_tax = implode(',',$tax_in_array);
//dpm($new_tax);
//dpm($row);
//$row->xml->Taxomomy = $new_tax;
// Change category text to ID
$category = (string)$row->xml->html->root->article->Category;
//Specify unknown category if we do not recognize the category
//This allows the migrate and allow us to fix later.
$tax_cat = $cat_tax[trim($category)];
//dpm($category);
if(is_null($tax_cat)) {$tax_cat = 18;}
//dpm($tax_cat);
$row->xml->html->root->article->Category = $tax_cat;

-------------

The category field in the source is a text field. The categories are a entity reference to a taxonomy field, which requires an id rather than text. I manually setup the categories ahead of time so I created an array that has the text as the key and the is as the content. Then you can use this to quickly look up the id for the text in he category field. Then we can replace the text in Category with the id. This works, another way to do this is migrate the categories first then use this migration to translate this for you. This is a feature built into migrate. The explanation of this will come later.

----------------

//modify the image file node.
//dpm((string)$row->xml->ID);
if((string)$row->xml->html->root->article->Image->File->asXML() != ''){
//dpm((string)$row->xml->html->root->article->Image->File->asXML());
$src = (string)$row->xml->html->root->article->Image->File->img->attributes()->src;
$src_new = str_replace('/visionmedia/uploadedImages/','http://assets.vision.org/uploadedimages/',$src);
$row->xml->html->root->article->Image->File->img->attributes()->src = $src_new;
$file_name = basename($src_new);
$file_path = rtrim(str_replace($file_name,'', $src_new), '/');;
$row->xml->html->root->article->Image->File->img->addChild('file_name',$file_name);
$row->xml->html->root->article->Image->File->img->addChild('file_path',$file_path);
}

--------------

There is alot of stuff here. Remember for the MigrateFile you need to present the file name and source directory. The Image/File node contains an img tag. So we need to get the scr attribute and extract the file name and source directory. So why the if? Migrate will import a null node as null, but this is php code running on the row. If you try to get the src attribute on a null node it will throw an error. So the if statement checks to see if the File node is empty (only contains /File) and skips this tranformation, Migrate will simply import a null or empty field.

The src is the relative path to the website, so the first thing we do is change this to full url to the s3 content storage. The path is basically the same except in the uploadedimages the i in the database is uppercase. This was a Windows server so it did not make a difference but the s3 url is case sensitive. We then use base name to extract the file name and use this to remove the file from the path for the file path and create a new child in the xml row to store these. I did not point this out but this is the xpath use in the field mapping above.

--------------

$email = (string)$row->xml->html->root->article->AuthorURL;
if (!empty($email)){
$email = 'mailto:'.$email;
$row->xml->html->root->article->AuthorURL = $email;
}

-------------

The author url is the email to the author of the article. We turn this into a mailto link so that it will generate a link to send the author an email.

---------------

$archive_html = (string)$row->xml->html->asXML();
$row->xml->addChild('archive_html',$archive_html);
$sidebar_element = (string)$row->xml->html->root->article->SidebarElement->SidebarElementInformation->asXML();
$row->xml->html->root->article->SidebarElement->SidebarElementInformation = $sidebar_element;
$slider_src = (string)$row->xml->Image;
$slider_src_new = str_replace('/visionmedia/uploadedImages/','http://assets.vision.org/uploadedimages/',$slider_src);
$row->xml->Image = $slider_src_new;
$slider_file_name = basename($slider_src_new);
$slider_file_path = rtrim(str_replace($slider_file_name,'', $slider_src_new), '/');;
$row->xml->Image->addChild('file_name',$slider_file_name);
$row->xml->Image->addChild('file_path',$slider_file_path);
//dpm($row);

---------------

The rest is repitition of the above techniques. Note that we return TRUE if we want to process the row and false if we do not want to process the row.

-----------------

$ret=TRUE;
//dpm($src);
}
//Need to add processing for other Article Content types especially 0 (HTML content)
//dpm($row);
return $ret;
}

}

----------

This is the class I use for one of the imports. I told you that I would show the use of another migrate in the field mappings. Below is a snippet of code from the issues migration. The issue contains entity reference to vision_articles that were imported from above.

-------------

$this->addFieldMapping('field_articles', 'article_id')
->sourceMigration('VisionArticle')
->xpath('/item/articles/article/ID');

--------------

So this says use the VisionArticle (I will show you were to find this next), it knows to look up the source ID and relate it to the DestinationID and store this in the field_articles field.

---------------

Migrate has been around for a while. Initially they said that the class would automaticall be registed and you could manually register them if needed. Then they changed to say that they will not manually register and you should register your classes. So you should have as part of your migration module the following that will register your classes. Note the name of the array element is the name used above.

----------------

function vision_migrate_migrate_api() {
$api = array(
'api' => 2,
// Give the group a human readable title.
'groups' => array(
'vision' => array(
'title' => t('Vision'),
),
),
'migrations' => array(
'VisionArticle' => array('class_name' => 'VisionArticleMigration'),
'VisionIssue' => array('class_name' => 'VisionIssueMigration'),
'VisionVideoArticle' => array('class_name' => 'VisionVideoArticleMigration'),
'VisionFrontpage' => array('class_name' => 'VisionFrontpageMigration'),
),
);

return $api;
}

----------------

I hope this makes things a little easier to understand. You will need some basic module building skills, knowing the file names and things like that, but this should help you through the more obscure parts of creating your migration class.

Tags: Planet Drupal
Categories: Elsewhere

Drupal Association News: Why we moved Drupal.org to a CDN

Planet Drupal - Tue, 22/07/2014 - 19:54

As of a little after 19:00 UTC on 2 July 2014, Drupal.org is now delivering as many sites as possible via our EdgeCast CDN.

Why a CDN?

We are primarily concerned with the network level security that a CDN will provide Drupal.org.

The CDN enables us to restrict access to our origin servers and disallow directly connecting to origin web nodes (which is currently possible). The two big advantages are:

  1. Accelerate cacheable content (static assets, static pages, etc).
  2. Allow us to easily manage network access and have a very large network in front of ours to absorb some levels of attacks.

Here are some examples of how the CDN helps Drupal.org:

  • We were having issues with a .js file on Drupal.org. The network was having routing issues to Europe and people were complaining about Drupal.org stalling on page loads. There was basically nothing we could do but wait for the route to get better. This should never be a problem again with EdgeCast's global network.
  • We constantly have reports of updates.drupal.org because blacklisted because it serves a ton of traffic coming in and out of a small number of IP addresses. This should also not happen again because the traffic is distributed through EdgeCast's network.
  • A few months ago we were under consistent attack from a group of IPs that was sub-HTTP and was saturating the origin network's bandwidth. We now have EdgeCast's large network in front of us that can 'take the beating'.
updates.drupal.org

By enabling EdgeCast's raw logs, rsync, and caching features, we were able to offload roughly 25 Mbps of traffic from our origin servers to EdgeCast. This change resulted in a drastic drop in origin network traffic, which freed up resources for Drupal.org. The use of rsync and the raw log features of EdgeCast enabled us to continue using our current project usage statistics tools. We do this by syncing the access logs from EdgeCast to Drupal.org’s utility server that processes project usage statistics.

Drupal.org

Minutes after switching www.drupal.org to use the CDN, there were multiple reports of faster page load times from Europe and North America.

A quick check from France / webpagetest.org:
Pre-CDN results: first page load=4.387s. repeat view=2.155s
Post-CDN results: first page load=3.779s, repeat view=1.285s

Why was the www.drupal.org rename required?

Our CDN uses a combination of Anycast IP addresses and DNS trickery. Each region (Asia, North America, Europe, etc.) has an Anycast IP address associated with it. For example cs73.wac.edgecastcdn.net might resolve to 72.21.91.99 in North America, and 117.18.237.99 in Japan.

Since 72.21.91.99, 117.18.237.99, etc. are Anycast IPs, generally their routes are as short as possible, and the IP will route to whatever POP is closest. This improves network performance globally.

Why can't drupal.org be a CNAME?

The DNS trickery above works by using a CNAME DNS record. Drupal.org must be an A record because the root domain cannot be a CNAME. MX records and any other records are not allowed by the RFC on CNAME records. To work around this DNS limitation, Drupal.org URLs are now redirected to www.drupal.org.

 

 

Related issues
https://www.drupal.org/node/2087411
https://www.drupal.org/node/2238131

Categories: Elsewhere

Stanford Web Services Blog: Cherry Picking - Small Git lesson

Planet Drupal - Tue, 22/07/2014 - 18:56

Small commits allow for big wins.

Something that I have been using a lot lately is GIT's cherry pick command. I find the command very usefull and it saves me bunches of time. Here is a quick lesson on what it does and an example use case.

What is GIT cherry-pick? man page

Git cherry pick allows you to merge a single commit from one branch into another.  To use the cherry pick command follow these steps:

Categories: Elsewhere

2bits: Improve Your Drupal Site Performance While Reducing Your Hosting Costs

Planet Drupal - Tue, 22/07/2014 - 17:00
We were recently apporached by a non-profit site that runs on Drupal. Major Complains Their major complaint was that the "content on the site does not show up". The other main complain is that the site is very slow. Diagnosis First ... In order to troubleshoot the disappearing content, we created a copy of the site in our lab, and proceeded to test it, to see if we can replicate the issues.

read more

Categories: Elsewhere

Drupalize.Me: Drupal 8 Has All the Hotness, but So Can Drupal 7

Planet Drupal - Tue, 22/07/2014 - 15:30

Drupal 8 is moving along at a steady pace, but not as quickly as we all had hoped. One great advantage this has is it gives developers time to backport lots of the features Drupal 8 has in core as modules for Drupal 7. My inspiration and blatant rip-off for this blog came from the presentation fellow Lullabot Dave Reid did at Drupalcon Austin about how to Future-Proof Your Drupal 7 Site. Dave’s presentation was more about what you can do to make your Drupal 7 “ready” where this article is more about showing off Drupal 8 “hotness” that we can use in production today.

Categories: Elsewhere

Drupal Easy: DrupalEasy Podcast 135: Deltron 3030 (Ronan Dowling, Backup and Migrate 3.0)

Planet Drupal - Tue, 22/07/2014 - 15:09
Download Podcast 135

Ronan Dowling (ronan), lead developer at Gorton Studios joins Ted and Mike to talk about all the new features in Backup and Migrate 3.0 including file and code backup and a improved plugin architecture. We also get up-to-speed with Drupal 8 development, review some Drupal-y statistics, make our picks of the week, and ask Ronan 5-ish questions.

read more

Categories: Elsewhere

Acquia: Enforcing Drupal Coding Standards During the Software Versioning Process

Planet Drupal - Tue, 22/07/2014 - 14:18

Cross-posted with permission from Genuine Interactive

Les is a web applications engineer at Genuine Interactive. He is a frequent Drupal community contributor. Genuine’s PHP team works on projects in a range of industries from CPG, B2B, financial services, and more.

Categories: Elsewhere

Blair Wadman: Create your first Drupal admin interface

Planet Drupal - Tue, 22/07/2014 - 12:34

One of the key features of a Drupal module is an admin interface. An admin interface enables you to make a module's settings configurable by a site editor or administrator so they can change them on the fly.

Tags: Drupal Module DevelopmentPlanet Drupal
Categories: Elsewhere

Russell Coker: Public Lectures About FOSS

Planet Debian - Tue, 22/07/2014 - 10:22
Eventbrite

I’ve recently started using the Eventbrite Web site [1] and the associated Eventbrite Android app [2] to discover public events in my area. Both the web site and the Android app lack features for searching (I’d like to save alerts for my accounts and have my phone notify me when new events are added to their database) but it is basically functional. The main issue is content, Eventbrite has a lot of good events in their database (I’ve got tickets for 6 free events in the next month). I assume that Eventbrite also has many people attending their events, otherwise the events wouldn’t be promoted there.

At this time I haven’t compared Eventbrite to any similar services, Eventbrite events have taken up much of my available time for the next 6 weeks (I appreciate the button on the app to add an entry to my calendar) so I don’t have much incentive to find other web sites that list events. I would appreciate comments from users of competing event registration systems and may write a post in future comparing different systems. Also I have only checked for events in Melbourne, Australia as I don’t have any personal interest in events in other places. For the topic of this post Eventbrite is good enough, it meets all requirements for Melbourne and I’m sure that if it isn’t useful in other cities then there are competing services.

I think that we need to have free FOSS events announced through Eventbrite. We regularly have experts in various fields related to FOSS visiting Melbourne who give a talk for the Linux Users of Victoria (and sometimes other technical groups). This is a good thing but I think we could do better. Most people in Melbourne probably won’t attend a LUG meeting and if they did they probably wouldn’t find it a welcoming experience.

Also I recommend that anyone who is looking for educational things to do in Melbourne visit the Eventbrite web site and/or install the Android app.

Accessible Events

I recently attended an Eventbrite event where a professor described the work of his research team, it was a really good talk that made the topic of his research accessible to random members of the public like me. Then when it came to question time the questions were mostly opinion pieces disguised as questions which used a lot of industry specific jargon and probably lost the interest of most people in the audience who wasn’t from the university department that hosted the lecture. I spent the last 15 minutes in that lecture hall reading Wikipedia and resisted the temptation to load an Android game.

Based on this lecture (and many other lectures I’ve seen) I get the impression that when the speaker or the MC addresses a member of the audience by name (EG “John Smith has a question”) then it’s strongly correlated with a low quality question. See my previous post about the Length of Conference Questions for more on this topic [3].

It seems to me that when running a lecture everyone involved has to agree about whether it’s a public lecture (IE one that is for any random people) as opposed to a society meeting (which while free for anyone to attend in the case of a LUG is for people with specific background knowledge). For a society meeting (for want of a better term) it’s OK to assume a minimum level of knowledge that rules out some people. If 5% of the audience of a LUG don’t understand a lecture that doesn’t necessarily mean it’s a bad lecture, sometimes it’s not possible to give a lecture that is easily understood by those with the least knowledge that also teaches the most experienced members of the audience.

For a public lecture the speaker has to give a talk for people with little background knowledge. Then the speaker and/or the MC have to discourage or reject questions that are for a higher level of knowledge.

As an example of how this might work consider the case of an introductory lecture about how an OS kernel works. When one of the experienced Linux kernel programmers visits Melbourne we could have an Eventbrite event organised for a lecture introducing the basic concepts of an OS kernel (with Linux as an example). At such a lecture any questions about more technical topics (such as specific issues related to compilers, drivers, etc) could be met with “we are having a meeting for more technical people at the Linux Users of Victoria meeting tomorrow night” or “we are having coffee at a nearby cafe afterwards and you can ask technical questions there”.

Planning Eventbrite Events

When experts in various areas of FOSS visit Melbourne they often offer a talk for LUV. For any such experts who read this post please note that most lectures at LUV meetings are by locals who can reschedule, so if you are only in town for a short time we can give you an opportunity to speak at short notice.

I would like to arrange to have some of those people give a talk aimed at a less experienced audience which we can promote through Eventbrite. The venue for LUV talks (Melbourne University 7PM on the first Tuesday of the month) might not work for all speakers so we need to find a sponsor for another venue.

I will contact Linux companies that are active in Melbourne and ask whether they would be prepared to sponsor the venue for such a talk. The fallback option would be to have such a lecture at a LUV meeting.

I will talk to some of the organisers of science and technology events advertised on Eventbrite and ask why they chose the times that they did. Maybe they have some insight into which times are best for getting an audience. Also I will probably get some idea of the best times by just attending many events and observing the attendance. I think that the aim of an Eventbrite event is to attract delegates who wouldn’t attend other meetings, so it is a priority to choose a suitable time and place.

Finally please note that while I am a member of the LUV committee I’m not representing LUV in this post. My aim is that community feedback on this post will help me plan such events. I will discuss this with the LUV committee after I get some comments here.

Please comment if you would like to give such a public lecture, attend such a lecture, or if you just have any general ideas.

Related posts:

  1. Sex and Lectures about Computers I previously wrote about the appropriate references to porn in...
  2. Phone Based Lectures Early this month at a LUV meeting I gave a...
  3. Car vs Public Transport to Save Money I’ve just been considering when it’s best to drive and...
Categories: Elsewhere

Martin Pitt: autopkgtest 3.2: CLI cleanup, shell command tests, click improvements

Planet Debian - Tue, 22/07/2014 - 08:16

Yesterday’s autopkgtest 3.2 release brings several changes and improvements that developers should be aware of.

Cleanup of CLI options, and config files

Previous adt-run versions had rather complex, confusing, and rarely (if ever?) used options for filtering binaries and building sources without testing them. All of those (--instantiate, --sources-tests, --sources-no-tests, --built-binaries-filter, --binaries-forbuilds, and --binaries-fortests) now went away. Now there is only -B/--no-built-binaries left, which disables building/using binaries for the subsequent unbuilt tree or dsc arguments (by default they get built and their binaries used for tests), and I added its opposite --built-binaries for completeness (although you most probably never need this).

The --help output now is a lot easier to read, both due to above cleanup, and also because it now shows several paragraphs for each group of related options, and sorts them in descending importance. The manpage got updated accordingly.

Another new feature is that you can now put arbitrary parts of the command line into a file (thanks to porting to Python’s argparse), with one option/argument per line. So you could e. g. create config files for options and runners which you use often:

$ cat adt_sid --output-dir=/tmp/out -s --- schroot sid $ adt-run libpng @adt_sid Shell command tests

If your test only contains a shell command or two, or you want to re-use an existing upstream test executable and just need to wrap it with some command like dbus-launch or env, you can use the new Test-Command: field instead of Tests: to specify the shell command directly:

Test-Command: xvfb-run -a src/tests/run Depends: @, xvfb, [...]

This avoids having to write lots of tiny wrappers in debian/tests/. This was already possible for click manifests, this release now also brings this for deb packages.

Click improvements

It is now very easy to define an autopilot test with extra package dependencies or restrictions, without having to specify the full command, using the new autopilot_module test definition. See /usr/share/doc/autopkgtest/README.click-tests.html for details.

If your test fails and you just want to run your test with additional dependencies or changed restrictions, you can now avoid having to rebuild the .click by pointing --override-control (which previously only worked for deb packages) to the locally modified manifest. You can also (ab)use this to e. g. add the autopilot -v option to autopilot_module.

Unpacking of test dependencies was made more efficient by not downloading Python 2 module packages (which cannot be handled in “unpack into temp dir” mode anyway).

Finally, I made the adb setup script more robust and also faster.

As usual, every change in control formats, CLI etc. have been documented in the manpages and the various READMEs. Enjoy!

Categories: Elsewhere

PreviousNext: Using Drupal 8 Condition Plugins API

Planet Drupal - Tue, 22/07/2014 - 08:03

Although Drupal 8 has had a Conditions Plugin API for a several months, it wasn't until during DrupalCon Austin sprint we managed to get blocks to use the Conditions Plugin API for block visibility.

The great thing about Condition Plugins, is they are re-usable chunks of code, and many contrib projects will be able to take advantage of them (Page Manager, Panels, Rules anyone?)

In this post, I show how you can create an example Page Message module that uses a RequestPath condition plugin to show a message on a configured page.

Categories: Elsewhere

DrupalCon Amsterdam: Come to the Devops Track at DrupalCon Amsterdam

Planet Drupal - Tue, 22/07/2014 - 08:00

So you've finished building a beautiful Drupal website. That means your work is done, right?

Not even close! Building the site is only the beginning: every website needs to be deployed, hosted, monitored, maintained, upgraded, security patched, scaled, and more— and if you start thinking about those things only after finishing your site, you’re bound to run into trouble.

Fortunately, DrupalCon Amsterdam is here to help! We’ll be running a #devops track that will bring devs and ops closer together. We’ll be discussing ways to achieve easier deployments, as well as how to ensure better stability, scalability and security for your big, beautiful Drupal website.

We've got a bunch of awesome speakers with experience in all of the above topics, as well as:

  • managing large sites,
  • doing continuous delivery of applications,
  • automated testing to improve quality
  • ... and many more topics that you should think about when building that beautiful website that can't afford to go down.

    The DrupalCon Amsterdam DevOps track will feature a broad range of talks covering the various technologies used in devops— and we expect it will be a nice counterpart to the traditional Drupal-centric tracks. These DevOps sessions will give you a perfect opportunity to peek into new technologies and talk with the best people working on those solutions.

    Whether you are putting together a small internal application or a large, popular, internet-facing site, your job does not end at the last commit. So join us in learning how to release stronger and better software faster. We’re all in this together, so let’s share the love and learn from each other!

    Categories: Elsewhere

    MJ Ray: Three systems

    Planet Debian - Tue, 22/07/2014 - 05:59

    There are three basic systems:

    The first is slick and easy to use, but fiddly to set up correctly and if you want to do something that its makers don’t want you to, it’s rather difficult. If it breaks, then fixing it is also fiddly, if not impossible and requiring complete reinitialisation.

    The second system is an older approach, tried and tested, but fell out of fashion with the rise of the first and very rarely comes preinstalled on new machines. Many recent installations can be switched to and from the first system at the flick of a switch if wanted. It needs a bit more thought to operate but not much and it’s still pretty obvious and intuitive. You can do all sorts of customisations and it’s usually safe to mix and match parts. It’s debatable whether it is more efficient than the first or not.

    The third system is a similar approach to the other two, but simplified in some ways and all the ugly parts are hidden away inside neat packaging. These days you can maintain and customise it yourself without much more difficulty than the other systems, but the basic hardware still attracts a price premium. In theory, it’s less efficient than the other types, but in practice it’s easier to maintain so doesn’t lose much efficiency. Some support companies for the other types won’t touch it while others will only work with it.

    So that’s the three types of bicycle gears: indexed, friction and hub. What did you think it was?

    Categories: Elsewhere

    Drupal Watchdog: Outed in Austin

    Planet Drupal - Tue, 22/07/2014 - 03:26

    Today, my cover was blown.

    If you came across this photo on your Facebook account or Twitter feed and you’ve been wondering – or furiously texting friends who were in Austin last month – “Is that really...?” or “Was he...” and “Did you meet him?”

    The answer is:

    No. Howard Stern did not attend DrupalCon as a Drupal Watchdog secret agent. (Although we did hear a rumor that he was in Austin at the time, taping an episode of America’s Got Talent, and that he just might put in a surprise appearance at the Convention Center.)

    In fact, I can say with some certainty – having been a guest on his show – that Howard wouldn’t know Drupal from poopal.

    Truly, that was I in the photo: Ronnie Ray, your fearless Watchdog blogster.

    So now you know. But kindly keep this information on the QT.

    Although if you or your company want a Howardish presence at DrupalCon Amsterdam, feel free to send checks, cash, a NYC-Amsterdam round-trip airline ticket (and Howard only flies first class, alas) to R. Brawer, c/o this magazine.

    Images: 
    Categories: Elsewhere

    Andrew Pollock: [debian] Day 174: Kindergarten, startup stuff, tennis

    Planet Debian - Tue, 22/07/2014 - 03:23

    I picked up Zoe from Sarah this morning and dropped her at Kindergarten. Traffic seemed particularly bad this morning, or I'm just out of practice.

    I spent the day powering through the last two parts of the registration block of my real estate licence training. I've got one more piece of assessment to do, and then it should be done. The rest is all dead-tree written stuff that I have to mail off to get marked.

    Zoe's doing tennis this term as her extra-curricular activity, and it's on a Tuesday afternoon after Kindergarten at the tennis court next door.

    I'm not sure what proportion of the class is continuing on from previous terms, and so how far behind the eight ball Zoe will be, but she seemed to do okay today, and she seemed to enjoy it. Megan's in the class too, and that didn't seem to result in too much cross-distraction.

    After that, we came home and just pottered around for a bit and then Zoe watched some TV until Sarah came to pick her up.

    Categories: Elsewhere

    FiveRDesign: Drupal HowTo: Turn Off The "Read More" In Commerce Product Displays

    Planet Drupal - Tue, 22/07/2014 - 02:00
    A post describing how to overcome the obscureness of how to turn off the read more option on the body field of Drupal Commerce product displays.
    Categories: Elsewhere

    PreviousNext: Easy improvements for Drupal content editors using contrib

    Planet Drupal - Tue, 22/07/2014 - 01:00

    Get started with some quick and easy ways to improve the content editor experience, using contributed modules.

    Categories: Elsewhere

    Pages

    Subscribe to jfhovinne aggregator - Elsewhere