Bypassing Cloudflare

I use to automate many tasks in the web (such as triggering news and sorting them according to my interests) and some times I hit with Cloudflare’s DDoS protection, I mean client-side, I guess they will have some real DDoS protection server-side, as the client-side is very easy to break.
They use to change their algorithms from time to time, but the basics are always the same:

1. curl the protected page and you will get an invisible form which auto-sends itself after making some client-side calculations. Currently this form is called challenge-form and has three hidden fields called jschl_vc, pass and jschl_answer.
2. Compute the javascript you find in the curl-ed page and send the form (the results of the calculations will be populated to the jschl_answer field). You can emulate these calculations in your favorite language (Python, AppleScript, PHP, whatever) or have a JS engine to execute the scripts. I’ve used node for this, which is available as a command-line tool you can install and execute easilly.
3. If everything is OK, you will have now two cookies: __cfduid and cf_clearance. Using that cookies, you can now surf freely the website.

The details may change from time to time, as Cloudflare updates their methods, but it’s been very similar across the years. Just take a look in your regular browser/developer tools and find the magic under the hoods.

Advertisement

Mac: sqlite3 command-line one-liner load extension

You are using the sqlite3 command-line binary and you need an extension, for example the Levenshtein algorithm which will allow you to find in a database text similar to other texts, let’s say you would like to consider “Jules Verne” and “Julio Verne” the same author.

You learn that you can load the spellfix extension which will allow you to use a function called editdist3, which is this algorithm (also called edit-distance, meaning the amount of changes needed in a string to look like another string).

Mac OS X’s built-in sqlite3 binary doesn’t allow you to load extensions (maybe because of security issues), so after some time googling, you do the following:

1. Install an alternate copy of sqlite3 through brew (this is a must-have utility if you are some kind of programmer):

   $ brew install sqlite

   This will install a new copy of the sqlite3 binary with the ability to load extensions enabled by default. You will keep your system’s sqlite3 unalthered, so you are safe, as you will be the only one using the “unsafe” sqlite3, installed in
   /usr/local/opt/sqlite3/bin/sqlite3.

2. Compile yourself the spellfix extension. You will download sqlite’s source code. I got mine from here: https://www.sqlite.org/src/, clicking here or here =>
   
   You may get the tarball if you feel better.
   Unzip the file and look for spellfix.c inside sqlite/ext/misc. Right? Now let’s compile it following official sqlite’s instructions:

   $ cd /path/to/sqlite/ext/misc

   $ gcc -g -fPIC -dynamiclib spellfix.c -o spellfix.dylib

   If you are compiling a different extension, just substitute the words in red (spellfix.c and spellfix.dylib) with the right ones. If you receive this kind of message:

   -bash: gcc: command not found

   Then you dont’ have installed Apple’s command-line tools for Xcode. Go and get them @ Apple’s developer website.

3. Okay. This was the easy part. Now you are ready to load and run your brand-new copy of sqlite3 with loading-extensions enabled, and your brand-new compiled extension. It doesn’t exist a man page for sqlite3 and you don’t realize the how-to if you happen to run sqlite3 -help in a Terminal window… Using the trial and error method for minutes or hours, depending on your skills or how spirited you are, you will find the syntax. This is the one-liner, as I will be running this code from within another utility which doesn’t let me interact with sqlite3. And this is the reason of this entire post:

   cd /path/to/dir/with/spellfix/; /usr/local/opt/sqlite3/bin/sqlite3 /path/to/database.db 'SELECT load_extension("spellfix.dylib")' 'SELECT title FROM books WHERE editdist3(author, "Jules Verne") < 450'

I colored some things:

/path/to/dir/with/spellfix/ You will run your command from within the directory containing spellfix.dylib. Most probably you could specify the full path to spellfix.dylib later in the load_extension command, but this could lead to issues depending on your OS (ie, Win), so I prefer it this way.

Now follows the full path to the new sqlite3 binary you installed, as the system’s built-in sqlite3 won’t do the job.

/path/to/database.db This is the full path to the database. This means that we will be running the load_extension command from within SQL, not from the binary itself. I didn’t find a way. I tried using the -cmd switch mixed with the dot-command syntax (.load extension) but didn’t work.

load_extension(“spellfix.dylib”) Now we run two separate SQL statements, quoted and space-separated. In the first one we load the extension. In the second one, our statements (ie, searching for something).

editdist3(author, “Verne, Jules”) And here is how we are using the function which is available only after loading the spellfix extension.

BTW, here is the link to the spellfix documentation, just in case you are interested in its capabilities => https://www.sqlite.org/spellfix1.html

Insolvable and slow recaptchas fixed (jDownloader Mac)

Launching jDownloader captcha’s solver in a Chrome incognito window (easilly adaptable to other browser?). Look in Advanced prefs for “BrowseCaptchaSolver: Browser Commandline” and enter the following =>

[ "/Applications/Google Chrome.app/Contents/MacOS/Google Chrome", "-incognito", "%s" ]

This will fix the issue of insolvable and slow recaptchas (maybe some kind of conflict between your google id and the process of solving captchas).

The same should work without jDownloader: just solve your captcha in a incognito window.

Backup remote mysql database to local machine

I told you here a way to backup remote MySQL databases using a mix of AppleScript and shell script (ssh).

But that method won’t work with certain table types, which don’t store the data in files, such as InnoDB. So, here is a one-liner using mysqldump to backup that kind of databases…

mysqldump --opt --compress --host='website.com' --user='USER' --password='PWD' --all-databases | mysql -uUSER -pPWD -h127.0.0.1

Just substitute where needed USER and PWD. This shell script requires MySQL running in both machines and privileges to access the data in both sides.

IMHO, it runs much slower than the other method (making a copy of the remote db files) so, if you can use it, go for it! Otherwise, this method is safe for your data (safe if you don’t lose power or internet connection in the process 😉 ).

See “man mysqldump” for more options (such as extracting only certain databases or tables, as this script will backup everything!).

Modify tiles walkability in OpenSpace (on-the-fly)

This post applies to OpenSpace 1.x, but may apply also to 2.x (I didn’t study it yet!).

One major issue for OS1 owners is controlling avatar’s movements all the time long. Full control over the avatar. All the methods and properties related to avatar movement and re-creation of the tile-map according to new settings are private, closed. You can’t access and modify them unless you purchase a source license.

But here we go with a custom solution, which may be helpful for you in some situations as it was for us. As there is no available method to prevent avatar movement (ie, the user clicks in a tile and the avatar walks there in despite of what you say), the only solution is preventing user clicks!

Simple, not smart, but working solution.

And how in hell you do that? Also simple: place a transparent-clickable movie-clip in the tile, so it catches user clicks and nulls them. Just iterate over OpenSpace.getChildAt(0)’s children (the tiles) and do the job.

And now a few hints, as the situations are very different depending on your needs…

You can’t moveMyAvatar() or teleportMyAvatar() to the current tile –> this way you can’t prevent avatar movements! So, before user clicks, you must evaluate and decide if the avatar can walk (click, if you use the suggested workaround).

If you can’t decide prior to user’s election if he can walk/click, then you must evaluate after he clicks and starts walking. In that situation, take into consideration the previous paragraph. As you can’t moveMyAvatar() or teleportMyAvatar() to the current avatar’s coordinates (as coming from getCurrentTile()), move or teleport the avatar to a different tile.

There is a variety of events you can use to evaluate your conditions (ENTER_TILE, TILE_SELECTED, etc.), so most probably you will find some solution to your headache 😉

EDIT: Appart from the exposed method, still valid, would be nice to take a look to the following approach: dispatch a first moveMyAvatar() to a fake tile, so OpenSpace registers as current-tile the next one in the pathfinding, the issue a new moveMyAvatar() to the current (real) one.

Flash bitmap filters (ie: DisplacementMapFilter, perlinNoise) and CPU usage

I was looking for a way to animate water in qilania (our virtual world) and I found some quick code using the DisplacementMapFilter and perlinNoise, which seems very realistic for our traditional way to animate stuff (more like cartoons), but is a quick and acceptable hack which can actually fit with our design and will save some time to the illustrator in chief.

But seems that kind of scripted animation (should say more “effect” than animation) consumes so much CPU, specially in the SmartFox/OpenSpace environment which uses to be 30fps by default.

After some researching and looking for alternative methods, we decided the fractal noise was looking so nice after all, but we needed some way to save some CPU and RAM for the end user, specially looking towards old machines and architectures. And we found a way to decrease it. This is a very simple tip: decreasing the frame rate. We did it this way (follows pseudo-code):

this.addEventListener(Event.ENTER_FRAME, go);
var nvar:int = 0;
function go(evt:Event):void {
	nvar++;
	if(nvar%3==0){
		// perlinNoise stuff...
	}
}

This way we reduced the filter to 1/3 (that is 10fps for a 30fps project), which saves more than 50% of the CPU at execution time.

It’s still so much, but minimum enough and, after all, we don’t have water in all our maps!

Hope this simple idea can help someone in a future: the simple, the better 😉

Run command-line from JSFL

This is a somewhat old topic born with the original FLfile library by Guy Watson: the undocumented runCommandLine JSFL command, which still exists in FLfile. Previously we had FLappleScript, but it stopped working some time before runCommandLine was available for us in the Mac (IIRC).

If you are new to this, you may be shocked when you type and run this in a new “Flash JavaScript File” document:

FLfile.runCommandLine("echo 'ok' > /tmp/sample.txt;open /tmp/sample.txt");
// works in Mac, use your windows' shell knowledge if you use Win

Or this one:

FLfile.runCommandLine("osascript -e 'tell app \"iTunes\" to playpause'");

These are stupid examples, but I have a couple of interesting projects using a mix of various technologies, including JSFL and Flash, AppleScript, the shell and Illustrator, etc., which is a very promiscuous mode of scripting, but very funny and inedited for many people.

This command only returns (I think) the exit status (1 or 0), but still there is a bunch of ways to get your data (such as writing it to tmp files and reading them from JSFL using FLfile.read())

WindowSWF, mix ActionScript and JSFL for automation and productivity

It’s a long title for a post which could, in fact, be the only message without extra ellaboration. But I’ll ellaborate a bit, because I think this is a underrated feature I love in Flash, which allows to the end-user a full experience of automation and high detail customization of the Flash IDE. This post is for those of you who don’t know about it.

You can create a UI in Flash/ActionScript, link it with JSFL (Javascript Flash, a javascript-based language to drive the Flash IDE) and create your own utilities panel with customized solutions for your daily work. Such as this:

I use every day the buttons in this panel. The first button in the top-left corner, for example, will convert selected items in the stage to named movie clips. It’s a silly task, but some times, before I worked out this solution, I was forced to do it manually hundreds of times in a single day (!) –> select, F8, insert name for symbol, insert name for instance in the stage.

My most impressive automation task was one responsible of arranging, making certains changes and publishing thousands of FLA files in a row, which reduced various workdays to 20 or 30 minutes I could spend in a delicious capuccino while the machine was working by itself.

This isn’t a guide, so I’ll describe briefly the mechanism:

1. You have some need (usually automation of repetitive tasks in the Flash IDE).
2. You write a JSFL command in order to replicate what you use to do manually –> most of times *any* task can be duplicated via JSFL. If you don’t believe it, create a new document, make a rectangle, group it, then go to the menu Window > Other Panels > History. Everything listed in there is pure JSFL recorded which you can copy to the clipboard. More complex tasks will require heavy tweaks, though 😉
3. You create a FLA file with a button (or buttons, progress bars or whatever) linking to the JSFL command. Usually, JSFL commands are stored in the “Commands” folder of Flash’ main configuration dir and can be accesed as well from the Commands menu. You can even assign them a shortcut. But sometimes you run out of shortcuts or you can’t remember more shortcuts (!). This is the time when you need to create your own toolbar with that bunch of customized daily tasks. You call the JSFL command from ActionScript as follow:

myButton.onRelease=function(){
	MMExecute('fl.runScript(fl.configURI + "Commands/selec-to-movies.jsfl");');
}

This example is using AS 2 syntax, but that isn’t a problem, very simple to port it to AS 3. Basically, you call the ActionScript method of the same name, which executes JSFL code. This method is *only* available to “WindowSWF” panels, XMLtoUI dialogs (created on the fly usually by JSFL commands, that is another topic) or UI components. Obviously, no sense out of the Adobe Flash IDE.
4. Move the resulting SWF file to the WindowSWF folder of the config folder (next to Commands, Components, etc.).
5. Relaunch Flash. Now you can access your brand new panel from the menu Window > Other Panels, and it will behave just like any other utilities panel (more or less, you will learn some tiny details when you do it real).

And that’s all. Even if you don’t write your own JSFL solutions, you can still create a WindowSWF panel and link it to others’ JSFL commands you can find in Adobe’s Exchange web site.

Flash movie overlapping other layers fix

Some times, you need something to overlap (that is, display over) a Flash movie, and seems Flash doesn’t like that. By default. But you can use the parameter “wmode” to change this float-above-all behaviour (which, BTW, doesn’t show the same in all the browsers). The default “wmode”, if not specified, is window. We want it to be opaque or transparent (if transparent, the flash movie won’t have a solid background and you will see whatever is below the flash movie). Of course, the layer which will overlap the Flash movie, must be in a z-index higher than the own Flash movie.

I know this is easy, but there are lots of controversial in the web regarding this topic, so I thought I should clarify it, and also keep this info in this blog for future self-consumption.

This article explains it all in detail and includes examples for all “wmode”s: http://www.communitymx.com/content/article.cfm?cid=e5141

AppleScript to convert HTML to PDF via Smile

This has been allways a tricky thing (convert a web page to PDF, preserving the more info possible, including text and clickable links), but since Smile 3.2 is a one-liner I thought I should share (specially when I didn’t find examples for the pageloaded event). This will save a “random” web page to your desktop, in a file called “file.pdf”. If you wanna see it in action, just remove “visible:false”:

script callback
	on pageloaded w
		save w in "~/Desktop/file.pdf"
		beep 2
		close w
	end pageloaded
end script

set webPage to "http://www.qilania.com/"

make new web window with properties ¬
	{path name:webPage, script:callback, visible:false}

It won’t preserve properly the formatting of certain web pages or embedded plug-ins contents, so if you need the screenshot for aesthetic purposes, Smile also provides a command called take screenshot, which will make a image of the rendered page (use it instead of “save w” as “take screenshot of w in …”). You may first resize the window to your favorite size, as the screenshot will only contain the visible area of the web window, but that is left as an exercice for the reader 😉

Very useful code for many tasks (web related), specially if Smile is your main script editor.