I like dragons and my wife likes dragons. I have been waiting to print this one for a while but it was a 32 hour print!
Only 100 grams of PLA but lots of outer wall distance meaning very slow details and tens of thousands of retractions. Printed on my Ender 3.
Leaving prints running overnight is still something I don’t like to do. I worry about what happens if it fails while I’m asleep. It has protection though. Thermal runaway protection and a smoke/heat detector combo would alert me to any major problems.
It’s still a bit of a scary prospect, I don’t want to put those failsafes to the test.
That being said any time I do print something that’s over 12 hours I like to print awesome things and this dragon turned out great. It doesn’t knock very well but that’s beside the point.
Designed by Shira – the files can be found on Thingiverse by searching. Dragon Door Knocker. Thing number 1458545.
Printed in white, coated with primer, painted with silver enamel and then a final wash of tinned black paint made it turn out great.
The finishing touch was gold eyes – printed in a shiny golden PLA with sparked flecks through it. It was a great way to finish it off.
I don’t often talk about my personal life. My life outside of work – the offline world. I’d like to change that a bit and share some more things about me with the people that I consider friends.
Sharing things about myself is hard for me. I can talk all day about the projects that I am involved with but talking about me… what I like… how I’m doing… yeah that’s pretty hard for me.
I’m going to ease in here with this post. I’ll start by sharing some things about myself and what I get up to.
My favourite colour is orange. Sandwiches and mashed potato share the top spot of best foods on the planet in my view*. Heavy metal, rock and blues is my musical preference.
A mashed potato sandwich. I tried it once. It was not good. You can make it good if you know how and maybe one day I will share that secret.
William Patton – lover of sandwiches and mash.
My Family
Most of my day (and indeed almost every day) is spent here in my home. At this desk. But it is far from lonely.
The hustle and bustle of family life is always happening and that is something that never stops. I have a wife and 2 children and we live in a small town in Scotland.
Half of Scotland it is standard towns and cities like you would expect the other half is small towns and villages considered as Highlands & Islands. I live in the Highlands & Islands bit. It’s still mostly as you would expect – except there is lots of green hills and blue waters.
My wife is amazing. Without her I would be lost. She looks after us all better than we can look after ourselves.
My kids like to play games, watch cartoons and generally get themselves involved in anything that causes a little bit of mischief and things that cause a lot of mess.
My mother is nearby, a brother and my mother-in-law. We all get along and look out for each other. When one of us needs something the others help.
My Work
An average work day for me is quite boring. I work from my home. I wake up, work, eat, sleep and repeat. Most of my day is spent here at my desk with a text editor on one monitor and a browser window on the other.
Daily client work, too much time spent replying to emails and slack/skype messages, a bit of time on marketing/outreach and a few hours working on whatever projects I have going.
Work is a 7 days a week thing for me, the internet has no weekends and neither do I.
WordPress
A nice little keepsake from my 3D Printing.
The bulk of all of my work revolves around WordPress. I create custom themes, plugins and sites for clients. I modify existing sites. I help manage servers and the software that is used to run a WordPress site.
I spend a lot of my time with people from the Theme Review Team, chatting and handling things over there. I am team lead and also the Themes component maintainer so I try to spend as much time there as I can.
I’ve made a lot of friends in the WordPress community. Many of them are also involved with Themes to some degree but I’ve also met some really great people in other areas. Plugins, Forums Support, Accessibility, Privacy to name just a few.
My Hobbies
Computer programming and consumer electronics/tech. Those are my hobbies. Pretty much the same kind of things as I do for work.
When I’m not working I’m still at my computer. Did I mention I spend way too much time here at my desk?
WordPress
I really do love WordPress. I work with it nearly every day and I also spend a lot of my hobby time with it as well.
Writing code for it is great but so is spending time chatting about it with other people. Mostly I chat with them as I write code :).
In the WordPress community there is a huge spread of people with all kinds of skills and interests. Talking about technical things with them, code things or otherwise, is something I could do – and have actually done on more than one occation – all day.
3D Printing
I also have became a huge fan of 3D printing. Printing models, functional items, little mechanisms, All kinds of things.
I have enjoyed doing this immensely. Watching something be printed in plastic (and some really cool types like wood fill or metal fill) and appear slowly in front of you is astonishing.
I am a fan of the game Zelda. I printed this castle from one of the games. The video turned out cool, it’s a time-lapse lasting about 40 seconds.
Print time was a little over 28 hours. It’s kinda slow.
Home Automation
I got myself a single Arudino Uno to go with my 3D Printer. I needed it to flash a bootloader. After that I connected it to a photoresitor and some LEDs. They turned on themselves when it was dark. That was cool.
It was my first automation and then I started to think about what other things I could do. Temperature sensors + room fans were next.
Then a temperature sensor outside my home with a hole in the wall for the wire… It was getting serious. Permanent.
It was not a far jump from those first small projects onto the idea of home automation or some kind of smart home. I have been making that jump.
Such is Life
This is my life, pretty much all of it. It seems like it’s not really a lot and to be fair I could quite easily embellish each area with more specifics. In my family I have members with additional care needs. In my work I spend time on projects that are not remotely related to WordPress.
In truth I need mundane and consistency. I do my best to minimise unexpected changes. Knowing some of what the day entails helps me to handle the other parts of daily life which I struggle with.
Some people like excitement, adrenaline rushes, spontaneous events. They like to travel and have new experiences as much as they can. I do not enjoy similar things.
My life may be boring. My daily tasks mundane and repetitious, That is perfect for me. Not everyone likes this kind of life, I do and I wouldn’t trade it for anything.
Covering days 16 through 20 and culminating in 1 month and 1/5th of the way through 100 days.
This week I had spent a substantial portion of my time working on a project I've supported for last 5+ years – the WP Bootstrap Navwalker.
I've used the walker since version 2 of Bootstrap and been a contributer all of BS3 lifespan. With Bootstrap 4 finally in a stable release (after around 3 years in development!) and I have been working through a substantial rework of the entire navwalker.
I've focused on a few things through the rebuild.
Simplifying the codebase and improving readability.
Better link modifier (disabled, header, divider) and icon handling.
Handling long standing issues that were difficult to fix or required a large rework in the previous iteration.
Simplifying The Codebase And Improving Readability
To simplify things I have focused on a few things.
Too many paths!
Reducing the number of possible paths in the start_lvl function has been the first focus. This function has grown to contain lots of conditionals that result in different processing or outputs. At one point in Scrutinizer I seen it say that this function had over 20000 paths!
Lots of paths result in code that becomes hard to follow so to help improve that I decided to externalize some parts of the code into their own functions.
Prime candidates were some decision logic that didn't need to be cluttering up the main thread. A single call with a well named function does a better job of explaining what is happening than the large conditional code blocks – even through those codeblocks are well commented.
Better Link Modifier and Icon Handling
Bootstrap navigation has always required the use of specific classes on the container, the link and the wrapper elements for the dropdown elements to function. WordPress provides a way to set classes for individual nav items in the Menu Editor UI.
In previous versions the Title Attribute input was used to handle special classes for modifiers and icons.
The complexity here is that most of the time Walkers add the classes to the wrapper element – in this walker that is an <li> – however some of the time you need those classes to get applied directly to the link.
Icon classes need to be placed separately from the wrapper and the link. They are added to their own <i> element with aria-hidden="true" so that screen readers don't try read an icon character.
$icon_classes = array(); // This array would contain valid icon classes.
// Join any icon classes plucked from $classes into a string.
$icon_class_string = join( ' ', $icon_classes );
/**
* Initiate empty icon var, then if we have a string containing any
* icon classes form the icon markup with an <i> element. This is
* output inside of the item before the $title (the link text).
*/
$icon_html = '';
if ( ! empty( $icon_class_string ) ) {
// append an <i> with the icon classes to what is output before links.
$icon_html = '<i class="' . esc_attr( $icon_class_string ) . '" aria-hidden="true"></i> ';
}
To make this happen I needed to loop through arrays of classnames and strip specific classes from the main array and save them in a new array for later use. The $icon_classes array is filled with that function.
I decided to handle link modifiers and icon classes separately as the link modifiers are used for setting a typeflag for decision making later in the execution.
/**
* Find any custom linkmod or icon classes and store in their holder
* arrays then remove them from the main classes array.
*
* Supported linkmods: .disabled, .dropdown-header, .dropdown-divider
* Supported iconsets: Font Awesome 4/5, Glypicons
*
* NOTE: This accepts the linkmod and icon arrays by reference.
*
* @since 4.0.0
*
* @param array $classes an array of classes currently assigned to the item.
* @param array $link_classes an array to hold linkmod classes. Passed by reference.
* @param array $icon_classes an array to hold icon classes. Passed by reference.
* @param integer $depth an integer holding current depth level.
*
* @return array $classes a maybe modified array of classnames.
*/
function seporate_linkmods_and_icons_from_classes( $classes, &$linkmod_classes, &$icon_classes, $depth ) {
// Loop through $classes array to find linkmod or icon classes.
foreach ( $classes as $key => $class ) {
// If any special classes are found, store the class in it's
// holder array and and unset the item from $classes.
if ( preg_match( '/disabled/', $class ) ) {
// Test for .disabled.
$linkmod_classes[] = $class;
unset( $classes[ $key ] );
} elseif ( preg_match( '/dropdown-header|dropdown-divider/', $class ) && $depth > 0 ) {
// Test for .dropdown-header or .dropdown-divider and a
// depth greater than 0 - IE inside a dropdown.
$linkmod_classes[] = $class;
unset( $classes[ $key ] );
} elseif ( preg_match( '/fa-(\S*)?|fas(\s?)|fa(\s?)/', $class ) ) {
// Font Awesome.
$icon_classes[] = $class;
unset( $classes[ $key ] );
} elseif ( preg_match( '/glyphicons-(\S*)?|glyphicons(\s?)/', $class ) ) {
// Glyphicons.
$icon_classes[] = $class;
unset( $classes[ $key ] );
}
}
return $classes;
}
Unit Testing My Code
When tests are successful and all is green 🙂
While I was going through the code and dealing with long standing issues I also took care of an issue which was opened not long ago on the early dev version of the v4 walker. The issue explained that the fallback method was broken and outputting nothing when it should have been outputting something.
The fix was easy – I had mistakenly used assign instead of append so swapping the operator on all of the markup to be output made it work.
Once I'd fixed it I realised that the project has Unit Tests that can be run locally and are run by Travis but all they do is check that the file, class and methods exist. Those tests could be a lot more useful.
If there was a test in place that made sure the fallback output when it was supposed to and didn't when it wasn't I would have caught this mistake early and fixed it quick. Instead it escaped me till someone else pointed it out.
Testing That Functions Produce The Output They Should
The project has PHPUnit already available so all I had to do was add some tests that could check the output.
This fallback function should produce 2 different types of output. For a logged out user it should output nothing. If a user is logged in with edit_theme_options capability it should output markup with a link. There's also a flag that determines if it echos or returns.
I wrote some tests that do a few things:
Test that logged out users get empty strings and that both 'echo' and 'return' are the same.
Test that logged in users get html that looks to be valid and that 'echo' and 'return' are both the same.
When testing output of a function there are a few methods that PHPUnit has available. Those methods are useful but somewhat limited so many times – for output that may be complicated or subject to slight changes over time – you will want to check for traits of the valid output rather than the exact output to save you reworking tests all the time. PHPUnit already runs tests inside of a content buffer so all you need to do is get your output with ob_get_contents;.
One thing that got me stuck for a few minutes was how exactly could I get logged in output when this is run in CLI mode where no user is set. Turns out it's quite easy. Make a new user with the capability we need – I made an admin that does have all core capabilities – and set them to the current user.
// make an admin user and set it to be the current user.
$user_id = $this->factory->user->create( array( 'role' => 'administrator' ) );
wp_set_current_user( $user_id );
The full pair of test ended up coming out like this:
/**
* Test Fallback method output for logged out users.
*
* Expects that for logged out users both echo and return requests should
* produce empty strings.
*/
function test_fallback_function_output_loggedout() {
// default is to echo reults, buffer.
ob_start();
WP_Bootstrap_Navwalker::fallback( $this->valid_sample_fallback_args );
$fallback_output_echo = ob_get_clean();
// empty string expected when not logged in.
$this->assertEmpty(
$fallback_output_echo,
'Fallback output for logged out user created a non-empty string in echo mode.'
);
// set 'echo' to false and request the markup returned.
$fallback_output_return = WP_Bootstrap_Navwalker::fallback( array_merge( $this->valid_sample_fallback_args, array(
'echo' => false,
) ) );
// return and echo should result in the same values (both empty).
$this->assertEquals(
$fallback_output_echo,
$fallback_output_return,
'Fallback output for logged out user created a non-empty string in return mode.'
);
}
/**
* Test Fallback method output for logged in users.
*
* Expects strings to be produced with html markup and that they match when
* requesting either a return or defaulting to echo.
*/
function test_fallback_function_output_loggedin() {
// make an admin user and set it to be the current user.
$user_id = $this->factory->user->create( array( 'role' => 'administrator' ) );
wp_set_current_user( $user_id );
// default is to echo results, buffer.
ob_start();
WP_Bootstrap_Navwalker::fallback( $this->valid_sample_fallback_args );
$fallback_output_echo = ob_get_clean();
// rudimentary content test - confirm it opens a div with 2 expected
// values and ends by closing a div.
$match = ( preg_match('/^(<div id="a_container_id" class="a_container_class">)(.*?)(<\/div>)$/', $fallback_output_echo ) ) ? true : false;
$this->assertTrue(
$match,
'Fallback method seems to create unexpected html for logged in users in echo mode.'
);
// set 'echo' to false and request the markup returned.
$fallback_output_return = WP_Bootstrap_Navwalker::fallback( array_merge( $this->valid_sample_fallback_args, array(
'echo' => false,
) ) );
// return and echo should both produce the same strings.
$this->assertEquals(
$fallback_output_echo,
$fallback_output_return,
'Fallback method seems to create unexpected html for logged in users in return mode.'
);
}
Testing Private Methods of a Class
When I was writing other tests I needed to call private methods and test their output. Private methods are not intended to be called from outside of the function so to make that happen we need to use a technique called Reflection.
We use a reflector to duplicate the class and methods we want and then set the methods to be publicly accessible from the reflector.
$wp_bootstrap_navwalker = $this->walker;
// since we're working with private methods we need to use a reflector.
$reflector = new ReflectionClass( 'WP_Bootstrap_Navwalker' );
// get a reflected method for the opener function and set to public.
$method = $reflector->getMethod( 'linkmod_element_open' );
$method->setAccessible( true );
Once you have the reflected method setup and accessible you can then use invokeArgs on the $method – passing the real class object followed by an array containing the args to pass to it.
// test openers for headers and dividers.
$header = $method->invokeArgs( $wp_bootstrap_navwalker, array( $this->valid_linkmod_typeflags[0], 'stringOfAttributes' ) );
$this->assertNotEmpty( $header, 'Got empty string for opener of ' . $this->valid_linkmod_typeflags[0] );
I'm perfectly capable of reading and writing JS code with plenty of real world experience working with it. It's not one of my strongest skills though so I took this as an opportunity to improve and learn some more things.
Javascript and ES6
When people say modern Javascript what they often mean is ES6 Javascript. The 6th edition was finalized in 2015 and is called ECMAScript 2015.
Sadly even though it's been around for some time browser implementation was slow to start and there is no backporting being done in many cases for older browser versions.
Modern browsers support ES6 code fairly well. Chrome, Opera, Safari, Firefox and even Edge have good levels of support – older browser versions do not and there's essentially no support for it on any version of IE.
Developers want to take advantage of the newer features that ES6 offers. They write ES6 code then compile it to ES5 compatible code – systems like babel make that easy. I wanted to take advantage of the new paradgrims too..
New Methods and Features in ES6 I Was Excited About
Deciding to go full ES6 in this project allowed me to take advantage of some features that I had previously only made minimal or no use of. There are 3 things I discovered in ES6 that made writing JavaScript a vastly more pleasant experience.
Native Classes to build with.
Variable and Function scoping inside blocks with let.
Default value assignments for function declarations.
I'm excited to work with those features all the time. Classes especially.
In ES5 you could use variables with objects, custom prototypes and anonymous functions to get similar Classes-like behaviors. To me that method always seemed to create very convoluted and hard to read code so native Classes are very welcome.
There is a couple of other very nice additions that I'm looking forward to using in future and in code refactors.
Rest and Spread Operators – ...variable.
Template Literals with raw string access.
Constants – Immutable Variables
String Searching.
Doing Things the ES6 Way
There are a few things in this project that I done differently to in the past and in ES6 they are much cleaner.
Merging Objects
While working I wanted to merge some objects. Previously I'd done that with a custom function that iterates through all prototypes of an object and merged them into another object. ES6 has a method for doing just that in a single line.
let defaults = {the defaults};
/**
* Merge 2 (or several) objects together.
*
* This merges myObject into the object containing the defaults.
* and returns the updated destination object only.
*/
this.myObject = Object.assign( defaults, this.myObject );
Better Sub-String Methods
A lot of work takes place with strings and string manipulation. In my latest project I had the pleasure of searching for values inside strings using much cleaner methods.
let myString = 'a string';
// there's a `startsWith` method now and it includes an index spot. This returns true because we start at index 4. There is also an `endsWith` that behaves similarly.
if ( myString.startsWith( 'ring', 4 ) ) {
console.log( 'found "ring" at end of ' + myString );
}
// my favorite is `includes`. Previously I would have used indexOf or a regex for complex strings for this.
if ( myString.includes( 'tri' ) {
console.log( 'Found "tri" inside of . + myString );
}
This is part 2 of the 2nd week of my 100 Days of Code Challenge.
I'm somewhat of a plain styles kind of person. Color conveys a meaning that I don't often care for when it comes to things I create for myself. A white background with black for me is perfectly ok in most cases. When I'm particularly adventurous I may opt for a dark mode.
This week I decided that I'd add some color to some of my scripts. Since I was working on the backup script I decided to put some color into the info that the script output. It outputs simple pieces of text with echo just like this:
echo -e 'files archived'
echo -e 'starting db dump'
Different terminals use different colorization methods and I've colorized text a few times by Googling to find out how to make it happen. This time I done the same lol.
When I looked up the codes this time I got to a page with a very thorough breakdown of what can be done and how. It had this note at the top.
In Bash, the <Esc> character can be obtained with the following syntaxes:
\e
\033
\x1B
Seeing the list of escape characters reminded me of a time previously when I had added color in a python script. It output color in my terminal and in a terminal running on a Windows 10 machine. At the time I thought nothing about it.
Suddenly realizing that the color codes must be somewhat standardized I decided to look more closely. I discovered the codes come from a system called ANSI/VT100. A lot of terminals support it in some way or another. Windows added support for it in version 10. Prior to windows 10 there was no support.
Escape Color Sequences
Colorizing terminal output is done by using an escape sequence followed by a color code and then closed. Reset follows the same method so the codes end up looking something like these.
\e[1m # bold
\e[7m # inverted
\e[0m # reset
# OR with different escape sequences that bash understands.
\033[1m # bold
\x1B[7m # inverted
\e[0m # reset
I have no idea exactly what the escape sequences that windows terminal supports however with my python script these were the colors I had used with the \033 escape sequence.
# http://stackoverflow.com/a/287944/2375493
class tcolors:
HEADER = '\033[95m'
BLUE = '\033[94m'
GREEN = '\033[92m'
WARNING = '\033[93m'
FAIL = '\033[91m'
ENDC = '\033[0m'
BOLD = '\033[1m'
UNDERLINE = '\033[4m'
You can see a large list of color codes and support for them on this page.
Color Escaped Text in a Bash Script
I decided to invert the colors of some of the info messages to make them stand out from the other messages that may scroll past. For a success messge I also turned the text green as well as doing the invert.
Those echo commands end up starting with the color code, outputting the text and then end with a reset code.
echo -e '\e[7m\e[92mfiles archived\e[0m'
echo -e '\e[7mstarting DB dump\e[0m'
Of course helper text and colorization is only useful in a manual run of this script. Since it's primary purpose is to be run as a cron job this info is going to be put to /dev/null. But I wanted to learn how it's done properly so that's what I did.
As part of my 100 Days of Code Challenge this is something I worked on in week 2.
The start of this week with the sniffles as my wife calls it. I had the cold and I spent a lot of time reading on my phone, not a lot of time working unfortunately.
One of the things I did work on in that downtime was improving a backups setup that I had been using.
I wrote a script to run through each of the sites I wanted backed up and while writing it I decided I'd put a little color in it for a change to make it easier to see the status of the execution.
I run a lot of WordPress sites using docker and multi-container setups. It's mainly based on similar ideas I wrote about when I first put WordPress Containers into production.
What Needs Backup
There are 2 things we need to back up for WordPress sites. There's the site files and there's the database. Both of those are run inside of different containers.
The content of these sites isn't updated often – many of them are essentially static at this point. Slow weekly/monthly backups are enough to secure what they store.
All of the sites and their databases get included in different system level backups at different times of the month. It's not ideal but it's been enough so far.
In a more tailored system the file and database backups would occur more regularly – and be more targeted towards each individual site.
Backup WordPress Docker Container Files
The container that runs the WordPress files holds them inside of a directory on the host that is mounted inside of the the container. Technically – and in the default home directory backups I run as part of a different system – they can be backed up from the host by backing up those directories.
I wanted to perform the backup without relying on the host system. Instead I came up with this one liner to fire up a different container that's using the ubuntu image and archive the files through it for the backup. It's a little complex but I'll explain each bit below.
docker run --rm --volumes-from pattonwebzinfo_wordpress_1 -v $(pwd)/backup:/backup/ ubuntu tar -cvf /backup/pattonwebz_info.tar /var/www/
docker run --rm – Run a container that will be removed when it exits.
--volumes-from pattonwebz_info_wordpress_1 – Mount the volumes FROM another container INTO this container too.
-v $(pwd)/backup:/backup/ create a volume that's found at [current working directory]/backup on the host and make it accessible inside the container from /backup/
ubuntu – the container we're using is a base ubuntu instance which will have the command we need available.
tar -cvf /backup/pattonwebz_info.tar /var/www/ – using tar command we backup the /var/www/ directory (which is a directory mounted from the wordpress container) and store the archive inside of this containers mounted volume – backup.
Backup a Mysql Database Inside of A Docker Container
To backup the database I considered spinning up a different mysql container and mounting the directories inside of it in a read-only mode to prevent from 2 systems accessing the same set of files.
I worried that file locking might be a problem with this and it seemed like a custom image may be needed – or for custom configs to be written in advance and mounted into the db backup container.
There was also authentication issues to deal with. I don't want to be bundling credentials into scripts if I don't need to.
I opted to take a different approach and use docker exec to execute a mysqldump command inside of the container and redirect the output to a file inside of the host filesystem. Inside of [current working directory]/backup/wp_database.sql– this puts it right next to the sites files archive.
Script it up to make it cycle through all the containers I need to backup and then trigger that script via cron. Targeted scheduled backups for WordPress docker containers.
For the first week of my 100 Days of Code challenge I didn't go too far out of my way to work with code. It's part of my job, I do it day-to-day.
I spent some time looking for a code bug to discover it was simple typesetting issue. I learned, once again, the value of explicit typesetting and the pitfalls of allowing an operation to do it's own type juggling.
When you're in need of specific operations or types to be the outcome it's best to strictly define those types instead of allowing the system to do it for you.
Another thing that I was amazed by is how much can be done, in a far more efficient manner, when you spend time only in the command line and semi-automate tasks.
There are times when it's far more time effective and less error prone to do that over manually processing data and verifying the results of your work.
Juggling & Defensive Typecasting
Be defensive when it comes to casting variables to the types you expect when data comes from somewhere you're not in control of. Sometimes it's what you expect – sometimes it's not.
William Patton
Wait… What type is this??
Strict typesetting is a good idea when working with variable types in a loosely declared language because sometimes types are juggled in a way that you don't expect and when you've got a type you're not expecting your results may not be what you're expecting as well.
Example of this happening with strings and numbers in JavaScript:
// Data comes from an object.
var data = {
'id': '1452',
}
// Try increment the ID number by 5.
let new_id = data.id + 5;
/**
* Instead of incrementing the starting data was a string
* and appended our number to the end like a string concat.
*/
console.log( new_id ); // new_id = 14525
/**
* What we really wanted to happen was to increment the number
* by 5.
*/
let new_id = parseInt( data.id ) + 5; // parsed the `id` and made sure it was int.
console.log( new_id ); // new_id = 1457
There is also a different operator specifically made for doing type conversions from string to number in javascript and is written in a more easily flowing manner. Both methods create the same output when successful.
var data = {
'id': '1452',
}
let new_id = data.id;
new_id = + 5;
console.log( new_id ); // new_id = 1457
Be defensive when it comes to casting variables to the types you expect when data comes from somewhere you're not in control of. Sometimes it's a number – sometimes it's not.
— William Robert James Patton (@Will_Patton_88) January 1, 2018
Command Line Automation
Working in command line and creating automation scripts beats manual tagging and sorting for large datasets.
Every. Single. Time.
William Patton
With a directory of .html files to have articles exported and a database of thousands of posts already imported: Find what files were not already converted and imported to the site as articles.
The list of content that was intended to include came from thousands of .html files – in various directories arranged mostly by year.
Using mysql queries to extract data about articles then filter it so it had only a value easily compared against the files needing import – that value will be a match with the filename that's to be imported.
With the data use BASH with some loops and a direct comparison logic to find matches.
Use git diff reporting to determine what posts were missing from the list of imported content.
BASH Scripts That Loop and Compare
At the heart of things were bash scripts that mostly looped within a loop.
Iterate through lines of a file while looping through lines of another file then output the results to different file.
It's a blunt object approach. A lot of overheads, not at all efficient in operations.
Regardless of inefficiencies it didn't take long to run through. It was simple and incredibly effective to highlight the differences between items present and items expected.
Working in command line and creating automation scripts beats manual tagging and sorting for large datasets.
Every. Single. Time.
— William Robert James Patton (@Will_Patton_88) January 3, 2018
I called the script it match.sh then ran it passing 3 .txt files. One filled with the files that are expected to be in the list, another filled with the list of articles present for this category and the 3rd file to store the results.
#!/bin/bash
echo "Matching $1 inside of $2 and saving to $3"
echo "Do the above values look correct?"
sleep 5s
# loop through all the lines in the first file passed.
while IFS='' read -r line || [[ -n "$line" ]]; do
# loop through all the lines in the second file passed.
while IFS='' read -r linein || [[ -n "$linein" ]]; do
# if line from f2 matches the line from f1...
if [[ $linein == *$line* ]]; then
# echo line 1 (filename) to a file.
echo "$line" >> ./$3
# we found a match, break from loop 2.
break
fi
# loop 2 is passed contents of file passed as arg2.
done < "$2"
# loop 1 is passed the contents of file passed as arg1.
done < "$1"
The next step was a repeat of this except to compare the list of expected items against the list of matches and save that to new file too.
The final step was to use git diff to compare the differences between the matched items and the original list of items expected.
One thing that helped here to improve processing speed by several minutes per run was to split the large groups of data into smaller, more manageable, sets or categories and then handle comparing them in them in batches. Since my compare method was so inefficient this helped a lot.
I've been mulling over this idea for a long time. 100 days of code. The original idea is essentially spend 1 hour a day coding, document the progress and share on code platforms or on social.
On the surface that sounds easy as I spend almost every single day with code in some form or another. Generally it's an hour or more.
The fact is that it's difficult to have 100 consecutive days to work on an idea without interruptions. Even if it's just 1 hour a day – sometimes you might just not have it. There's work and family life to prioritize.
30 days ok, 60 maybe. 100 full days in a row working directly with code and no deviations: That's more of a challenge than I can commit to.
An Outline of a More Forgiving 100 Days of Code Challenge
It's not just about writing code – there is plenty of value to be found in working with code it in other ways.
It is not 100 consecutive days and you never need to commit 7 days a week.
Improving documentation for code or reading and reviewing someone else's code are both high value tasks in some cases.
It's 100 working days, no minimum per day.
Code review or other improvements are included and highly encouraged.
Documentation isn't necessary everyday – once a week is a good target.
Daily tracking isn't required – track ideas or projects over the space of a week or more.
The 100 days could start from any point. I've decided to start January 1st.
Between January 1st and June 1st there are a little over 100 working days.
That's 5 days in a week – or every day minus weekends and major holidays. Spread over around 5 months of time.
I will be documenting some things I work on each week, write some example code and schedule posts to publish throughout the span of the challenge.
Documentation Format Suggestions
Sharing of knowledge in the programming and development world is done in a few formats. Most obviously the sharing of code. That's a great way let someone see the work that's been done or methods that were used.
Another common way to share is via articles, blog posts and books. This is useful for sharing overall process, detailing setup steps or explaining some thought processes behind what's been done.
Anything that can be shared in a blogpost. Text, screenshots or examples could be included here.
Links to code sharing platforms like a GitHub repo or a Codepen.
Social shares like tweets.
Consider helping someone to solve a coding problem as being a valid exercise as well. Working through an issue tracker like in GitHub repo or similar counts. StackOverflow is a great place to offer assistance and to share what you know with someone in need of help.
Important Notes
With an emphasis on working with code there's some things I think will be relatively important to keep in mind throughout the 100 days. It's about working with code not necessarily writing code all the time.
Looking at and learning from some code is a valid exercise.
Adding inline docblocks or improving existing comments is valid.
Participating in an open source project of any kind will be valid.
Remember if it's working with code then it can be considered part of the challenge. Some other things to note throughout the challenge:
Sharing of example code is always good, sharing of full source when possible is encouraged too.
Creation of actual code documentation – either external documentation or inline code doc block addition or improvements is a very big part of writing code that is for sharing.
I've done sysadmin type work in addition to general web development for a long time. I've built many a server stack and helped solve a lot of performance issues.
I've heard the term microcaching a lot recently. I had never heard of it for http web caches until then. If that is a new term for this process then it's a term for something people have been doing for a long time. If I just somehow missed that it was called that then ooops :p
What is Microcaching?
I don't know if there is a technical definition for it specifically but essentially it's the process of caching things for unusually short lifetimes. It helps a ton on sites with specific dynamic pages but also are highly trafficked.
The Semi-Unusual Cases Recently Where Microcaching Was A Big Win
Twice in the last 6 months I've been contacted by clients who were in similar situations. Steadily growing and been asked by their hosting companies to upgrade because they were hitting limits or impacting other users.
When I was contacted they had upgraded in the past as well from shared to VPS hosting, one of the sites had even employed someone previously to tweak caching plugins and apache configs.
The usual big win caching items were all enabled and tweaked, in-memory caches were available on the local host. The problem was that most of those were done for only logged out users.
Logged in users often need fresher content – or the rules of caching are more complicated than is viable to offer through a plugin interface – so only minor improvements could be made by the site through the plugin.
A reverse proxy with some specially tailored caching rules was an incredible fix in areas that the plugins couldn't handle.
Even though both sites had previously been looked into by their hosts and even at one point a performance specialist somehow they missed something…
Logs showed that both sites were getting 50-75% of all their traffic on a single page type from logged in users. Looking at the pages there were obvious reasons for the high number of requests to them. Speaking with the site owners though I discovered they didn't need to be generated all the time.
Group Social Freshness
There was a social network where the main group channel was the culprit for most of the server load.
The page was long, with many short messages each requiring it's own query and loop to get the data and output it to the page, new messages were posted every 5 seconds during peak times and there was no client-side loading. It was an expansive operation compared to other pages on the site.
It turns out that the group channel showed the same content regardless of who viewed it so long as they were logged in.
Short term caching of this page made a huge difference. ~85% cache hit rate on a highly dynamic page that changes roughly every 5 seconds dropped the server load to less than half what it was before.
Globally cached for all users who visit it with freshness lifetime of 5 seconds, stale pages allowed but only for up to 30 seconds.
Online Store Confirmation/Delivery Details
The other site was an online store, it had a slightly different problem. The page it was getting most hits on was unique to each user and each order. It was the each customer which each customer visited directly after ordering. They would sit on the page and refresh it over and over again until status changed from pending to success.
Since this site was items with customization options available through drop shipping it took between 5-10 minutes for the order to be fully confirmed.
Refreshing between order and confirmation many times created dozens and dozens of needlessly generated pages. To make it worse the pages were also making Ajax calls to check for updated shipping info on each request. So all requests essentially = 2 requests.
Orders took between 300-600 seconds to confirm because of the drop shipping and no amount of refreshing is going to change what is returned within that window.
There was at least 50 times the requests to that page than there were unique sales.
Unique cached item for each user with 60 second freshness, allow stale for up to 90 seconds.
Takeaway: Win-Win
Microcaching is a thing. It brings big wins on highly dynamic pages with lots of requests per second.
Ultimately in these 2 specific cases a single rule was able to drop server load by more than half and prevent the need for these companies to upgrade to more expensive hosting plans.