Bypassing Cloudflare

I use to automate many tasks in the web (such as triggering news and sorting them according to my interests) and some times I hit with Cloudflare’s DDoS protection, I mean client-side, I guess they will have some real DDoS protection server-side, as the client-side is very easy to break.
They use to change their algorithms from time to time, but the basics are always the same:

  1. curl the protected page and you will get an invisible form which auto-sends itself after making some client-side calculations. Currently this form is called challenge-form and has three hidden fields called jschl_vc, pass and jschl_answer.
  2. Compute the javascript you find in the curl-ed page and send the form (the results of the calculations will be populated to the jschl_answer field). You can emulate these calculations in your favorite language (Python, AppleScript, PHP, whatever) or have a JS engine to execute the scripts. I’ve used node for this, which is available as a command-line tool you can install and execute easilly.
  3. If everything is OK, you will have now two cookies: __cfduid and cf_clearance. Using that cookies, you can now surf freely the website.

The details may change from time to time, as Cloudflare updates their methods, but it’s been very similar across the years. Just take a look in your regular browser/developer tools and find the magic under the hoods.

Advertisements

Mac: sqlite3 command-line one-liner load extension

You are using the sqlite3 command-line binary and you need an extension, for example the Levenshtein algorithm which will allow you to find in a database text similar to other texts, let’s say you would like to consider “Jules Verne” and “Julio Verne” the same author.

You learn that you can load the spellfix extension which will allow you to use a function called editdist3, which is this algorithm (also called edit-distance, meaning the amount of changes needed in a string to look like another string).

Mac OS X’s built-in sqlite3 binary doesn’t allow you to load extensions (maybe because of security issues), so after some time googling, you do the following:

  1. Install an alternate copy of sqlite3 through brew (this is a must-have utility if you are some kind of programmer):
    $ brew install sqlite

    This will install a new copy of the sqlite3 binary with the ability to load extensions enabled by default. You will keep your system’s sqlite3 unalthered, so you are safe, as you will be the only one using the “unsafe” sqlite3, installed in
    /usr/local/opt/sqlite3/bin/sqlite3.

  2. Compile yourself the spellfix extension. You will download sqlite’s source code. I got mine from here: https://www.sqlite.org/src/, clicking here or here =>
    Untitled-1
    You may get the tarball if you feel better.
    Unzip the file and look for spellfix.c inside sqlite/ext/misc. Right? Now let’s compile it following official sqlite’s instructions:

    $ cd /path/to/sqlite/ext/misc
    $ gcc -g -fPIC -dynamiclib spellfix.c -o spellfix.dylib

    If you are compiling a different extension, just substitute the words in red (spellfix.c and spellfix.dylib) with the right ones. If you receive this kind of message:

    -bash: gcc: command not found

    Then you dont’ have installed Apple’s command-line tools for Xcode. Go and get them @ Apple’s developer website.

  3. Okay. This was the easy part. Now you are ready to load and run your brand-new copy of sqlite3 with loading-extensions enabled, and your brand-new compiled extension. It doesn’t exist a man page for sqlite3 and you don’t realize the how-to if you happen to run sqlite3 -help in a Terminal window… Using the trial and error method for minutes or hours, depending on your skills or how spirited you are, you will find the syntax. This is the one-liner, as I will be running this code from within another utility which doesn’t let me interact with sqlite3. And this is the reason of this entire post:
    cd /path/to/dir/with/spellfix/; /usr/local/opt/sqlite3/bin/sqlite3 /path/to/database.db 'SELECT load_extension("spellfix.dylib")' 'SELECT title FROM books WHERE editdist3(author, "Jules Verne") < 450'

I colored some things:

/path/to/dir/with/spellfix/ You will run your command from within the directory containing spellfix.dylib. Most probably you could specify the full path to spellfix.dylib later in the load_extension command, but this could lead to issues depending on your OS (ie, Win), so I prefer it this way.

Now follows the full path to the new sqlite3 binary you installed, as the system’s built-in sqlite3 won’t do the job.

/path/to/database.db This is the full path to the database. This means that we will be running the load_extension command from within SQL, not from the binary itself. I didn’t find a way. I tried using the -cmd switch mixed with the dot-command syntax (.load extension) but didn’t work.

load_extension(“spellfix.dylib”) Now we run two separate SQL statements, quoted and space-separated. In the first one we load the extension. In the second one, our statements (ie, searching for something).

editdist3(author, “Verne, Jules”) And here is how we are using the function which is available only after loading the spellfix extension.

BTW, here is the link to the spellfix documentation, just in case you are interested in its capabilities => https://www.sqlite.org/spellfix1.html

Insolvable and slow recaptchas fixed (jDownloader Mac)

Launching jDownloader captcha’s solver in a Chrome incognito window (easilly adaptable to other browser?). Look in Advanced prefs for “BrowseCaptchaSolver: Browser Commandline” and enter the following =>

[ "/Applications/Google Chrome.app/Contents/MacOS/Google Chrome", "-incognito", "%s" ]

This will fix the issue of insolvable and slow recaptchas (maybe some kind of conflict between your google id and the process of solving captchas).

The same should work without jDownloader: just solve your captcha in a incognito window.

Backup remote mysql database to local machine

I told you here a way to backup remote MySQL databases using a mix of AppleScript and shell script (ssh).

But that method won’t work with certain table types, which don’t store the data in files, such as InnoDB. So, here is a one-liner using mysqldump to backup that kind of databases…

mysqldump --opt --compress --host='website.com' --user='USER' --password='PWD' --all-databases | mysql -uUSER -pPWD -h127.0.0.1

Just substitute where needed USER and PWD. This shell script requires MySQL running in both machines and privileges to access the data in both sides.

IMHO, it runs much slower than the other method (making a copy of the remote db files) so, if you can use it, go for it! Otherwise, this method is safe for your data (safe if you don’t lose power or internet connection in the process 😉 ).

See “man mysqldump” for more options (such as extracting only certain databases or tables, as this script will backup everything!).

Modify tiles walkability in OpenSpace (on-the-fly)

This post applies to OpenSpace 1.x, but may apply also to 2.x (I didn’t study it yet!).

One major issue for OS1 owners is controlling avatar’s movements all the time long. Full control over the avatar. All the methods and properties related to avatar movement and re-creation of the tile-map according to new settings are private, closed. You can’t access and modify them unless you purchase a source license.

But here we go with a custom solution, which may be helpful for you in some situations as it was for us. As there is no available method to prevent avatar movement (ie, the user clicks in a tile and the avatar walks there in despite of what you say), the only solution is preventing user clicks!

Simple, not smart, but working solution.

And how in hell you do that? Also simple: place a transparent-clickable movie-clip in the tile, so it catches user clicks and nulls them. Just iterate over OpenSpace.getChildAt(0)’s children (the tiles) and do the job.

And now a few hints, as the situations are very different depending on your needs…

You can’t moveMyAvatar() or teleportMyAvatar() to the current tile –> this way you can’t prevent avatar movements! So, before user clicks, you must evaluate and decide if the avatar can walk (click, if you use the suggested workaround).

If you can’t decide prior to user’s election if he can walk/click, then you must evaluate after he clicks and starts walking. In that situation, take into consideration the previous paragraph. As you can’t moveMyAvatar() or teleportMyAvatar() to the current avatar’s coordinates (as coming from getCurrentTile()), move or teleport the avatar to a different tile.

There is a variety of events you can use to evaluate your conditions (ENTER_TILE, TILE_SELECTED, etc.), so most probably you will find some solution to your headache 😉

EDIT: Appart from the exposed method, still valid, would be nice to take a look to the following approach: dispatch a first moveMyAvatar() to a fake tile, so OpenSpace registers as current-tile the next one in the pathfinding, the issue a new moveMyAvatar() to the current (real) one.

Flash bitmap filters (ie: DisplacementMapFilter, perlinNoise) and CPU usage

I was looking for a way to animate water in qilania (our virtual world) and I found some quick code using the DisplacementMapFilter and perlinNoise, which seems very realistic for our traditional way to animate stuff (more like cartoons), but is a quick and acceptable hack which can actually fit with our design and will save some time to the illustrator in chief.

But seems that kind of scripted animation (should say more “effect” than animation) consumes so much CPU, specially in the SmartFox/OpenSpace environment which uses to be 30fps by default.

After some researching and looking for alternative methods, we decided the fractal noise was looking so nice after all, but we needed some way to save some CPU and RAM for the end user, specially looking towards old machines and architectures. And we found a way to decrease it. This is a very simple tip: decreasing the frame rate. We did it this way (follows pseudo-code):

this.addEventListener(Event.ENTER_FRAME, go);
var nvar:int = 0;
function go(evt:Event):void {
	nvar++;
	if(nvar%3==0){
		// perlinNoise stuff...
	}
}

This way we reduced the filter to 1/3 (that is 10fps for a 30fps project), which saves more than 50% of the CPU at execution time.

It’s still so much, but minimum enough and, after all, we don’t have water in all our maps!

Hope this simple idea can help someone in a future: the simple, the better 😉

Run command-line from JSFL

This is a somewhat old topic born with the original FLfile library by Guy Watson: the undocumented runCommandLine JSFL command, which still exists in FLfile. Previously we had FLappleScript, but it stopped working some time before runCommandLine was available for us in the Mac (IIRC).

If you are new to this, you may be shocked when you type and run this in a new “Flash JavaScript File” document:

FLfile.runCommandLine("echo 'ok' > /tmp/sample.txt;open /tmp/sample.txt");
// works in Mac, use your windows' shell knowledge if you use Win

Or this one:

FLfile.runCommandLine("osascript -e 'tell app \"iTunes\" to playpause'");

These are stupid examples, but I have a couple of interesting projects using a mix of various technologies, including JSFL and Flash, AppleScript, the shell and Illustrator, etc., which is a very promiscuous mode of scripting, but very funny and inedited for many people.

This command only returns (I think) the exit status (1 or 0), but still there is a bunch of ways to get your data (such as writing it to tmp files and reading them from JSFL using FLfile.read())