AppleScript & interactive shell-script sessions

I thought I should share the brief research I finished yesterday.

Being AppleScript (as the main language to drive apple-events in the mac) the mother of the automation since MacOS 7 days, I was still missing the path to automate interactive shell scripting sessions, such as ssh, telnet, mail, etc., without the need of scripting the Terminal app, which is ugly.

Yesterday I found info about “expect” (google for it and click the first search result if you use english as your main language). It’s kind of a pseudo-terminal which you can invoke just like bash or sh, and it accepts a serie of special commands, being “expect” the killer one.

I’m working in a top-secret project with gnuchess (an app which acts as interface for UCI chess engines => well, the app which in fact was used to determine how UCI chess engines should communicate with UIs) and I needed grabbing input from it. But gnuchess must be launched in interactive mode, etc. And AS’s “do shell script” doesn’t work like that.

So, I wrote my first “expect” script, which works like this:

(“sample.sh” invoked from AS «do shell script “cd xxx; ./sample.sh ‘file.pgn'”»)

#!/usr/bin/expect
# this is how you read passed arguments (index-based)
set pgnName [lindex $argv 0]

cd /path/to/gnuchess-6.0.2/src/;
# start listening
spawn ./gnuchess

expect “White (1) : “
send “pgnload $pgnName\n”

expect “White (1) : “
send “book on\n”

expect “White (1) : “
send “hard\n”

expect “White (1) : “
send “go\n”

expect “) : “
send “quit\n”

close

Basically:

  • spawn: launches and starts listening a process.
  • expect: listens the process to output the given string (such as “White (1) : “).
  • send: upon that string being sent by the process, enter the given string (such as “book on”).
  • close: that is.

I invite you to fire the Terminal and type “man expect”. You will find more options and ways to write more compact code.

Green fields in front of me 😉

This one and the new features in Adobe AIR will keep me busy for the next two years (appart from that damn mobile devices and the html5 headache everyone loves).

Be good.

MySQL remove duplicates (large database + fast)

In the end, this was for me the fastest method to remove dups from the big db I’m working on:

  1. Duplicate the structure of the database (ie, “db”) to a new database (ie, “db2”, setting the fields you don’t want to be dups to UNIQUE.
  2. Copy db to db2 using INSERT IGNORE:

INSERT IGNORE INTO
db2
(`xxx`,`yyy`,`zzz`)
SELECT `xxx`,`yyy`,`zzz`
FROM `db`

This reduced the amount of time to minutes, while any other method could take many hours, appart from creating large temporary files which could fill the disk holding the temp dir and finally abort the process.

If you don’t need in the end to have those UNIQUE fields (the indexes takes many HD), you should re-dump db2 to a new db without that UNIQUE fields => again some more minutes, but still worth against 12 hours of CPU/RAM intensive working (in a highly mysql responsive profile) with other kind of filtering methods, such as DISTINCT or GROUP BY.

Recover MySQL from files

This is how I recovered my databases from a HD crash using Mac OS X. These included both MyISAM and InnoDB tables.

  1. Get the files, typically from a backup or booting into your machine/server in recovery mode, usually located @ /var/lib/mysql
  2. Read (if you own this file) mysql_upgrade_info, so you see which mysql version created the files.
  3. Install the same version in your machine. If needed, uninstall whatever exists (if you don’t need it, of course):
brew remove mysql
  1. Install the closest mysql version to that you found in mysql_upgrade_info (or whatever you guess is the version). See the available formulas and do it:
brew search mysql
brew install mysql@5.5
  1. I installed myself version 5.5, which is the one I needed.
  2. Now locate the place where the local files reside, typically @ /usr/local/var/mysql
  3. Zip this folder as a backup measure and remove all of its contents.
  4. Move inside all the files and folders you got in step 1.
  5. Launch mysql:
/usr/local/opt/mysql@5.5/bin/mysql.server start
  1. Connect and mysqldump whatever you need.

Hints to manipulate Apple’s Photos programmatically

Today we have many tools to handle data in the cloud, but there is no tool in the market able to do many things you can think of, like having a centralized photo DB in your computer coming from different devices (ie, merging your iPhone photos with those of your wife’s).

So, some hints are coming… (just hints, no final code, just read and adapt to your needs/circumstances). These are some useful locations:

/Users/<USER>/Pictures/Photos Library.photoslibrary/database/photos.db

And these for shared albums:

/Users/<USER>/Library/Containers/com.apple.cloudphotosd/Data/Library/Application Support/com.apple.cloudphotosd/services/com.apple.photo.icloud.sharedstreams/coremediastream-state/38332866+6004658345420041391/Model.sqlite

/Users/<USER>/Library/Containers/com.apple.cloudphotosd/Data/Library/Application Support/com.apple.cloudphotosd/services/com.apple.photo.icloud.sharedstreams/com.apple.photo.icloud.sharedstreams-38332866.cloudphotoservicelibrary/database/photos.db

In the surrounding folders there are the original pictures and resampled versions (thumbnails and so on).

The databases are SQLite and you can manipulate them easilly. There are libraries in almost any programming language and you have a built-in sqlite3 command-line tool you can invoke from any scripting language, such as AppleScript, like this:

do shell script ¬
    "sqlite3 FILE.DB 'SELECT * FROM table;'"

This would work to get out binary data:

do shell script ¬
    "sqlite3 FILE.DB 'SELECT quote(blobfield) FROM table;'" & ¬
    " | cut -d \"'\" -f2 | xxd -r -p > /tmp/out.gz'"

Photos stores many things in binary plists, so the previous code will come in handy to extract that binary fields, which you can later uncompress and parse as regular XML data.

If you can’t read the DB files, don’t panic. Most probably they are locked (opened by another process). You can check it like this:

lsof | grep -e 'FILE.DB'

Most probably there are some sync services running and thus locking the databases. You can quit temporarily this process, do your stuff, and it will be relaunched after that. Ie, if the “offending” process is photolibraryd, you can:

kill -3 `ps xww | grep photolibraryd | grep -v grep`; sqlite3 FILE.DB 'SELECT * FROM table;'

More tips tomorrow!

HTML5 today: 1 code

After many years out of Flash and inside HTML5, I can dictate: nightmare.

It’s a world of possibilities but, regarding developers, a totally *ucking nightmare.

100% of customers think they need support for desktop computers and mobiles devices. This means a minimum of Chrome, IE, Firefox and Safari, in different environments (desktop, tablet, mobile) and OS’s (Win, Mac, iOS, Android, including Win 7 thru 10, Mac 10 thru 12, iOS 8 thru 11…), and usually as well the customer’s frontend app, which uses to be a Cordova-based app (also available for iOS, Android and desktop Win and Mac).

Regarding audio and video, you stick with mp3/ogg and mp4/ogv. Regarding programming, you rely mostly in general-use libraries such as jQuery. For the rest of functions, you create your own libraries, aimed to support around 40-50 different browsers/versions.

In the end, you spend in development a similar amount of time than you spent previously in Flash (mostly because Flash is still available for impossible tasks out of Flash, such as RAD —standing for Rapid Animation Development, instead of Application—, now with HTML5 output thanks to CreateJS).

But some times you spend around 100 thru 200% of the time with debugging tasks. The last time I had a bug in the customer’s iOS app. The code worked fine in the other 40-50 browsers, but not here. I had to re-make at home the app and simulate the customer’s code in order to replicate and debug the issue. This converted a simple 2-hours job into a complex 4-days job.

My advice to all my customers all this time is the following: force your users to use your own app. This way you develop and mantain 1 code.

Otherwise, take care of your problems. No! Your supported browser’s problems! The transition from IE10 to IE11 caused an issue which affected to almost 1000+ interactives to one of my customers. This customer had the luck to have my company as the responsible of such interactives, as we managed to resolve the issue quickly. However, this issue caused a bunch of negative comments in the different app stores. This is because my customer lets the browsers decide which work and which doesn’t, how and when.

Flash movies worked the same all around the browsers and operative systems: 1 code, 1 behavior.

This is just the first round. I wanna tell you more about this!

ASCII to compressed ASCII

Here is a funny experiment with javascript. It takes normal text as input and returns encoded ASCII text (from 0x20 thru 0xFF) with a 30% reduction in length, which can be decoded later to the original, with some exceptions/rules. It preserves only alphanumeric characters (except for some of them which are transliterated specially, like 1 = I or Z = 2).

This could be funny as a workaround in some situations, like circumventing the classical Twitter limit of 140 characters (would need the decoder on the other side, of course).

https://jsfiddle.net/julifos/ehuwaae2/

A few websites

A few websites I developed or helped to develop in the past few years:

http://www.julifos.com/

http://www.pescadosweb.com/

realmadridkidsplanet.com/ (dead)

http://www.chesswithoutborders.com/http://www.ajedrezsinfronteras.com/

http://pgrtraducciontecnica.com/ (dead)

http://yogamanam.com/

http://www.qilania.com/

http://www.damainvisible.com/ (dead)

And hundreds of web applications I can’t show because they reside in private areas 😉

I hope to resume my activities in this blog, as I have many things to share…

HTML5 6, Swiffy: fscommand & setFlashVars, back and forth communication

Plenty of new tools are arising in order to make a quick-move from the “old” Flash to the “new” HTML5, as HTML5 can’t still be taken into consideration when you need to create loads of contents. Simply put, there are no IDEs which would allow rapid-application-development. If you want graphic libraries, you must do it yourself. Coding? The same. Sound? The same.

Sure, there are lot of pre-written libraries, but they can’t be usable in a real pro environment. I can’t charge my interactives 1 or 2MB of javascript libraries (minimified, of course), only to have very limited possibilities, given that for me is a must-have compatibility with many browsers and platforms. Scalable graphics, toons animation with A/V support, etc. Many of my old customers are calling now companies in third-party countries, which are employing hordes of pseudo-programmers only to make what we could do before using a very compact and tiny team. Maybe cheaper. I don’t like the results in many situations and I don’t think my customers like ’em. Cheap and quick is never a real solution if you wanna be a top-notch in your field. Functionality and design are the answers. Before that could be done in a more or less cheap and fast way, as we had tools. Now STILL it can’t be done.

Adobe Edge should be in a future a possible answer.

Some of the answers we have now (Edge not being a realiable solution today we can take into account) are “Flash Professional Toolkit for CreateJS” (formerly called Wallaby), which didn’t meet the requirements the last time I tried it.

I’ve tested today Google’s Swiffy. The last time it wasn’t either ready. And it isn’t! But I found a use today, so I thought I should comment.

It works pretty fine for plain animations (cartoons) in the main timeline, and supports a very basic (but still usable) subset of AS2 (more limited AS3). Enough for clicks and so on… And I found the way to inter-communicate with the animation, so I can rely in Swiffy for the animation and cover from outside the rest (such as sync audio or whatever is needed). The cost is pretty high, though (164Kb), but at least I’m not forced to write my own animation engine (which is why I wasted this entire day to test Swiffy).

There are things you must considerate, such as the fact that some times you can’t play two things at the same time (ie, the main timeline and a mc). But if you learn the rules, it isn’t difficult to create quick, realiable and fashionable animations.

This is how you do it:

AS2 (place this on the root of your Flash movie):

// "custom methods" broadcasted to JS
playSound = function(id){fscommand('playSound',id)}
_stop = function() {fscommand('stopped');stop();}

// orders coming from JS (such as FlashVars)
this.onEnterFrame = function(){
    if(this.command){
        var c = this.command;
        if (c=='play'){
            play();
        } else if (c=='stop'){
            stop();
        }
    }
    this.command = null;
}

I used AS2, because it offers more methods and classes, including fscommand. You have as well getURL (and navigateToURL in AS3), but it’s much more rough than fscommand, as it addresses directly a named js function instead of passing it to the browser, such as “javascript:function()”.

You can see in this code two parts:

  1. Send info to javascript using fscommand.
  2. Receive info from javascript using the old-school flashvars method.

This is how you receive info in the javascript-side (well, Swiffy is nothing more than JS, but we’ll make the illusion, as if it was a different thing):

function swiffycontainer_DoFSCommand(cmd,args){
    if(cmd=='stopped'){
        alert('movie stopped')
    } else if(cmd=='playSound'){
        // do whatever, ie play sound id "args"
    }
}

Hear this. The function receiving the fscommand must be called DIVNAME + “_DoFSCommand”. DIVNAME is the name of the div containing the swiffy object, such as (fix the DIV tags, as this damm CODE tag in WP doesn’t handle it):

DIV id="swiffycontainer" style="width: 808px; height: 424px">
CLODEDIV

That div is used later (code auto-inserted by Swiffy) as follow:

var stage = new swiffy.Stage(document.getElementById('swiffycontainer'),swiffyobject);

stage.start();

So, we need the name of the div, in this example “swiffycontainer”, and append “_DoFSCommand”. You will receive one or two arguments in that function (depending on your desires), being the first one mandatory (as you’re commanding something) and the second one optional (you can pass here any options if you need to).

And, finally, this is how you send commands or info to the Swiffy object:

stage.setFlashVars('command=play');

You’re setting FlashVars as key=value. And you must address stage, which is the var you defined before to hold the swiffy object (well, Swiffy did it for you, but can change that name).

This sample one-liner will set the variable “command” to the string “play”. You will read that from the enterFrame handler you wrote before, and make something with it.

The part I don’t like is using SVG. Doesn’t seem very good in performance in complex animations, compared to their opponents Flash & canvas. That’s browser’s fault, as they have the SVG specs since ancient times (!).

In my preliminary tests, seems that the output is supported in much more browsers than those stated in the Swiffy website, but that may depend on the features you need (maybe some filters won’t work in older FF versions, or something like that). But works in a wide range of browsers/versions for the basic features I tested.

Cheers.

ACK Media in collaboration with Pescados

We are proud to announce that ACK Media (Amar Chitra Katha) will collaborate with Pescados in the production of Qilania.

They will use their proved expertise to bring to the net this unique virtual world for childs called Qilania.

Appart from being the leading company in India regarding entertainment and education for young people, they have published already a MMORPG called The Legend of Katha and a special website for children, called TinkleOnline.

We are sure we will make a very strong team 😉