lkubuntu

A listing of random software, tips, tweaks, hacks, and tutorials I made for Ubuntu

Category Archives: Uncategorized

The Perfect C Array Library

I love C. And I loathe C++.

But there’s one thing I like about C++: The fact that I don’t have to write my own dynamic array libraries each time I try to start a project.

Of course, there are many libraries that exist for working with arrays in C. Glib, Eina, DynArray, etc. But I wanted something as easy to use as C++’s std::vector, with the performance and memory usage of std::vector.

By the way, I am not talking about algorithmic performance. I’m writing this assuming the algorithms are identical (i.e. I’m writing purely about implementation differences).

There is a few problems with the performance and memory usage of the aforementioned libraries, the major one being that the element size is stored as a structure member. Which means an extra 4-8 bytes per array, and constantly having to read a variable (which means many missed optimization opportunities). While this may not sound too bad (and in the grand scheme of things, probably isn’t), it is undeniably less efficient than C++.

This isn’t the only problem, there are other missed optimization opportunities in the function (vs macro)-based variants, for example, calling functions for tiny operations, calling memcpy for types that fit within registers, etc.

All of this might seem like splitting hairs, and it probably is. But knowing that C++ can be faster, more memory efficient, and less bothersome to code in than C is not a thought I like very much. So I wanted to try to level the playing field.

It took me a rather long amount of sporadic work for me to create my very own “Perfect C Array Library”, that, I thought, fulfilled my requirements.

First, let’s look at some example code using it:

array(int) myarray = array_init();
array_push(myarray, 5);
array_push(myarray, 6);

for (int i = 0; i < myarray.length; i++) {
    printf("%i\n", myarray.data[i]);
}

array_free(myarray);

Alright, it might be a tiny bit less pretty than C++. But hey, this is good enough for me.

In terms of performance and memory issues, I fixed the issues I wrote above. So in theory, it should be just as fast as C++, right?

Turns out I missed one issue. Cache Misses. In my mind, if everything was written as a macro, it would, in theory, be faster than functions. I was wrong. Large portions of code inlined can result in cache misses, which will quite negatively impact the performance of the function.

So, as far as I can see, it is impossible to write a set of array functions for C that will be as fast and easy to use as C++’s std::vector. But please correct me if I’m wrong!

With that being said, this implementation is the most efficient I’ve been able to write so far, so let me show you the idea behind it:

#define array(type)  \
  struct {           \
      type* data;    \
      size_t length; \
  }

#define array_init() \
  {                  \
      .data = NULL;  \
      .length = 0;   \
  }

#define array_free(array) \
  do {                    \
      free(array.data);   \
      array.data = NULL;  \
      array.length = 0;   \
  } while (0)

#define array_push(array, element)                \
  do {                                            \
      array.data = realloc(array.data,            \
                           sizeof(*array.data) *  \
                             (array.length + 1)); \
      array.data[array.length] = element;         \
      array.length++;                             \
  } while (0)

The magic is in sizeof(*array.data). For some reason I never knew this was legal in C, but it does exactly what it says it does: it returns the size of type. Which eliminates the need to store this in the struct.

The code above is vastly oversimplified to demonstrate the idea. It’s very incomplete, algorithmically slow, and unsafe. But the idea is there.

To summarize, I am not aware of any way to write a completely zero-compromise array library in C, but the code above shows the closest I’ve come to that.

 

P.S. There is one problem I am aware of with this method:

array(int) myarray;
array(int) myarray1 = myarray; /* 'error: invalid initializer' */

There are 2 ways to get around this:

memcpy(&myarray1, &myarray, sizeof(myarray));
/* or */
myarray1 = *((typeof(myarray1)*)&myarray); /* requires GNU C */

Both of which should, under a decent optimization level, result in the same assembly.

Advertisements

How to fix lockdown error -5 on usbmuxd

When I plug my iPhone in (iOS 9), usbmuxd (and hence libimobiledevice-related software) runs just fine, but when I plug my older iPad in (iOS 5), I get this error:

[18:51:51.199][3] Connected to v2.0 device 1 on location 0x20009 with serial number ("random" string of numbers)
[18:51:51.867][1] preflight_worker_handle_device_add: ERROR StartSession failed on device (same "random" string of numbers), lockdown error -5

I spent quite a long time trying to fix this issue. I tracked down this file, that contains this line:

LOCKDOWN_E_SSL_ERROR = -5

So it’s an SSL-related issue, and it happens with an older device, and not a newer one … figured it might be the certificates, and, since I haven’t connected my iPad to the internet for a while, decided to give that a shot. Same issue. But hey, it was fun being able to use Cydia! (at the time of writing, there is no public jailbreak for iOS >=9.2)

After a while of playing with certificates on my host computer (with the same luck), I decided to try compiling both libimobiledevice and usbmuxd with debug flags.

Right off the bat, I got my first (and only) compiler error: undefined reference to 'SSLv3_method' (or something like that).

Turns out that OpenSSL disables SSLv3 by default, for security reasons. After changing SSLv3_method to SSLv23_method (as per this patch) and enabling debug flags, I found that, indeed, the problem relied exactly where the compiler error was:

19:11:34 idevice.c:722 idevice_connection_enable_ssl(): ERROR in SSL_do_handshake: SSL_ERROR_SSL

Thankfully, libimobiledevice contains support for GnuTLS, which, also thankfully, still supports SSLv3! Although there were a few compiler errors because I’m guessing nobody really used the GnuTLS backend for some time, once it was built, I finally managed to make usbmuxd (and libimobiledevice-using software) recognize my iPad!

EDIT: libimobiledevice fixed the compiler errors upstream.

So, if you encounter this error, here’s how you fix it:

git clone https://github.com/libimobiledevice/libimobiledevice.git
cd libimobiledevice
./autogen.sh
./configure.sh --prefix=/usr --disable-openssl
make
sudo make install

I hope this can help someone else with this issue! If you have any problems, feel free to leave a comment and I’ll try my best to help!

openlux 0.2.1 + roadmap

A few months ago, I wrote openlux as an open-source alternative to f.lux, similar to redshift, but different in goal and execution.

openlux 0.2.1 isn’t a particularly exciting release, but it fixes problems with graphics cards that support 10-bit displays .. something that openlux claimed to support, but, as it turns out, didn’t. After nvidia’s new driver update, I found that out the hard way (the screen went really weird, and at one point, totally black … not fun haha).

I also changed the logo slightly, I think it looks much nicer now! Here it is:

openlux.svg

Now about the roadmap, I’m planning on adding these features later on:

  • Two new backends, DRM and RandR, to support both direct tty (DRM), and more graphics drivers (RandR) that don’t properly support XF86VM (what openlux currently uses as its main backend)
  • A GUI for automating openlux via cron (and also for using it directly)
  • A GUI for iOS that will support both scheduling and using it directly (similar to the feature above, but iOS doesn’t support cron)
  • iOS 9 support (apparently iOS 9 works in a different way for this … sadly, my iPhone isn’t jailbroken yet, so I can’t test it yet)

 

Explaining Javascript Closures Simply

Please note that this article might not be entirely accurate, it is simply meant as a very simple explanation that will get you in the right path to understand them. If you wish for a technically correct answer, see this question on Stack Overflow.

Javascript Closures tend to confuse people who aren’t used to Javascript (don’t feel bad if you’re in this category, Javascript is a weird language, I am certainly included in this category as well).

However, they’re really simple. But many guides tend to make them seem more complicated than they really are. Stack Overflow contains a wonderful array of answers that ended up leaving me more confused than when I started.

So, I’ll give the answer that I believe would have made it clearest to me: A closure in Javascript inherits the stack of the function that declared it. The stack is an object, in the sense that it only goes away when there are no more references.

If you understand what that means, then great! Hopefully this helped! If not, I’ll explain :)

First, a closure in javascript is basically a ‘lambda’, a sort of ‘anonymous function’, and if these terms confuse you (don’t feel stupid if they do, I think we all suck with terms), here’s what it looks like:

setTimeout(/* This is a closure! --> */ function() {
    alert("Hello world!");
} /* <-- End of the closure */, 1000);

You’ve probably seen these everywhere in Javascript.

Now, the “stack” (in Javascript) is basically the local variables in a function. For example:

function hello() {
    var first = "Hello";
    var space = " ";
    var second = "world!"

    var message = first + space + second;

    /* first, space, second, and message are all part of the "stack" */
}

Technically, every function in Javascript is a closure, so therefore, every function inherits the stack of the function that declared it, not the stack of the function that calls it.

In other words:

function hello() {
    var message = "Hello world!";

    return function() {
        alert(message);
    };
}

var message = "Farewell planet";

var closure = hello();
closure(); /* This will output "Hello world!", because it inherits hello's stack */

But doesn’t the stack end once hello() has finished running? Nope, the stack is like an object (it might actually be one, I’m not sure), in the sense that the object will live so long as there is a reference to it. The closure has a reference to it (every object has a reference to their stack), and therefore, the stack is still alive.

What if a closure makes a new variable .. will that change be reflected in the stack of the old function? Nope, every function makes a new stack, that inherits the parent stack (the stack of the parent function that declared this function). For example:

function hello() {
    var first = "Hello";
    var second = "world!";

    [
        function() {
            first = "Goodbye";      // Modifies 'first' in the parent stack

            var second = "planet!"; // Notice the "var" .. this makes a new variable
                                    // in the _current_ stack.
                                    //
                                    // Any future reference to 'second' will refer
                                    // to the variable in the _current_ stack, not
                                    // the variable in the parent stack.

            second = "galaxy!";

            var message = "Hi";     // This was declared in the current stack, message
                                    // will not be declared in the parent stack.
        },
        function() {
            console.log(first);     // "Goodbye"
            console.log(second);    // "world!"
            console.log(message);   // message isn't declared
        }
    ].forEach(function(x) {x()});   // Runs these functions, one after the other
}

That’s about it!

I hope this helped you to understand how closures worked in Javascript. If you’re still confused, feel free to leave a comment!

I’m not an expert on this, and certainly, this guide isn’t meant as an expert technical analysis of how closures work, it’s mainly just meant as a (hopefully) simple bird’s-eye view of how  they work. As I mentioned a few times before, you can find a more detailed explanation at Stack Overflow.

Download videos online with Inject2Download

and I swear this blog hasn’t been hijacked . Apologies for the clickbait title, although, in all honesty, I’m not sure how else to word it … suggestions? :)

Most video download scripts I’ve seen generally tend to rely on some download server that does magic behind the scenes, which may or may not work, among other issues (ads, loading times, etc.). I also don’t like the fact that I can’t really know what’s going on behind the scenes (one of the reasons I use free software).

Thing is, for many websites that embed videos (that don’t host content with their own proprietary players, i.e. not youtube, vimeo, wistia, dailymotion, etc.), they have the direct video URL somewhere in the javascript (plainly readable or obfuscated).

Since most websites use the same player engines to play their videos (jwplayer, flowplayer, or video.js), all that is needed to do is to inject code into those engines when the page is loaded that will capture the video URL and somehow share the URL with the user.

I originally started writing this using Chrome, until I found out that Chrome actually doesn’t support injecting code directly after a library is loaded (which Firefox does), so I ended up making this only Firefox-compatible. I tried to sort of “race” the code so as to try to run the code as soon as possible after the libraries were loaded, but it only worked a fraction of the time, and plus it slowed down the webpage heavily.

Inject2Download is a user script, so you’ll need Greasemonkey to run it. If there’s enough interest, I’ll make it a proper extension later :)

Download and install it here: https://greasyfork.org/en/scripts/18671-inject2download

Github: https://github.com/AnonymousMeerkat/inject2download

What you’ll (hopefully) notice is that when you go to a website that hosts a video player, a little box will pop up at the top-left corner of the page (you might have to scroll up to see it), containing one or more URLs.

Some websites host ads on the player, and it’s sometimes (although rarely, thankfully) a bit difficult for the script to know which is an ad, and which is a legitimate video, so just use common sense and avoid URLs with “ads” or other suspicious text as part of them :)

If you have any issues with this, please feel free to let me know, either on here or via github, I’d be happy to help!

 

trivial-require: Closure-friendly Browserify(ish)

… and that title is about as accurate as saying that D is a superset of C, but the most accurate one I could think of that could fit in a relatively small space of text.

Before I explain further, I’ll explain my use-case scenario. I’m writing a web app that is meant to be used on my iPhone. 3G can be used up really quickly by browsing websites, and I plan on using the app a lot, so I need to make sure that the website uses as little bandwidth as possible.

Google’s Closure Compiler is very good at minifying Javascript code (the best I know of), and, under normal circumstances, it works just fine. Thing is, I’m writing the server in node.js, and I want to be able to share the same codebase with both the server and the client. With a tool like Browserify, this becomes very easy.

Problem is, Closure and Browserify don’t match very well. Sure, you can use Closure on a Browserified piece of code, but it doesn’t optimize nearly as well as it could. Few functions are properly inlined or evaluated, many variable and property names stay intact, and there’s a lot of needless code around it.

Use case scenario done. Now on to trivial-require (this line is for the TL;DR people :P)

I wrote trivial-require as a very quick and dirty hack for the project I was working on. It might not be ideal for everyone, but hopefully some people will be able to find it useful as well :)

trivial-require sort of functions like Browserify, in the sense that it will turn node.js code into browser code, and, if the winds are in your favour, it might work.

Okay, that might have been a bit of an exaggeration. This following line will sum up almost exactly what it does (bolded for the TL;DR folk, again)

trivial-require will literally include the contents of the require()d file at the spot where it is require()d. module.exports is entirely disregarded

No extra code is added around it. Literally the only difference between this and C’s #include directive is that this only will include a file once. In other words, var module = require('file'); var module2 = require('file'); will result in file.js being included once (both lines are removed in the output file).

This works in a very different way from the way that browserify and node.js work. But, it is possible to write code that works both with node.js and trivial-require.

Before I go to this though, I’ll explain how to install/run it:

Installing:

sudo npm install -g trivial-require

Running:

trivial-require script.node.js > script.browser.js

Now that that’s done, let’s go to the guidelines on using it:

Do not use ambiguous variable names. It might be overridden by a future module.

// Wrong
var Logger = require('./Logger')("ModuleName");

// Correct
var Logger_ModuleName = require('./Logger')("ModuleName");

Do not use module.exports as a means for writing a function (or other). Any line containing module.exports is deleted.

// Wrong
module.exports = function() {
    console.log("Hello World");
};

// Correct
function HelloWorld() {
    console.log("Hello World");
};

module.exports = HelloWorld;

Use the same module name when require()ing a file. Each file is literally included, and both require() and module.exports lines are deleted.

//// Wrong

// HelloWorld.js
function HelloWorld() = {
    console.log("Hello World");
};

module.exports = HelloWorld;

// index.js
var GreetPlanet = require('./HelloWorld');


//// Correct

// index.js
var HelloWorld = require('./HelloWorld');

require() in the global scope. Caching might hurt you later.

// Wrong
function my_function() {
    var HelloWorld = require('./HelloWorld');
    HelloWorld();
}

// Correct
var HelloWorld = require('./HelloWorld');

function my_function() {
    HelloWorld();
}

require()s containing non-relative pathnames will be removed. Use your own modules instead

// Wrong
var utf8 = require('utf8');

var encoded = utf8.encode("Hello World");

// Correct
var is_node = false;
if (typeof window === "undefined")
    is_node = true;

var encoded;

if (is_node) {
    var utf8 = require('utf8');
    encoded = utf8.encode("Hello World");
} else {
    encoded = unescape(encodeURIComponent("Hello World"));
}

And that’s about it! I hope that trivial-require might be useful for you, and that this guide is clear enough :) If you need help with anything, feel free to leave a comment, and I’ll try to help!

Some random wallpapers I made a while back

I haven’t made a post in a rather long time, but I have been busy on a few different projects (which I will release soon). For now, I would like share some wallpapers I made in GIMP about a year or two ago. I have released them under a couple different places, but they were mostly either segregated, private, or I forgot the link to it, so now I will finally publish them to a rather more central place :P

Anyways, here they are:

cool_glow_circle

Made using the "flame fractal"

Made using the “flame fractal”

cool_glow_flame_2 cool_glow_flame_3 cool_glow_flame_4 cool_glow_flame_prev cool_glow_hurricane

My personal favourite ^_^

My personal favourite ^_^

Thought this looked a bit like the fluid from The Matrix

Thought this looked a bit like the fluid from The Matrix

Had some fun with the motion blur :P

Had some fun with the motion blur :P

cool_spiral_blue cool_spiral
As you can see, some were variations on others, a few were made with the flame fractal, and most were made using “whirl and pinch” plus a lot of blend modes for layers.

Hope some of you may enjoy them :)

I’m not 22

For a long time, I’ve been saying that I was 21/22 (1/1/1990) for internet security reasons, and then it was because I had told others that I was 21/22. I think that it would be about time for me to release my age publicly, as I feel that it would do more harm than good if I continued claiming that I’m 22 (soon to be 23 :P).

Read more of this post

purgeconfig – A safer way to reset configuration files

Many people try purging packages and reinstalling them to reset them, but it sometimes removes other packages with it, so it becomes a headache (especially when you are trying to reset the X11 configuration files, removing a few hundred packages with it).

To solve these problems, I made a script called purgeconfig that will artificially reset the configuration files (manually removing the configuration files, marking the packages as purged, then reinstalling the packages).

WARNING: Do NOT run Apt/Aptitude/DPKG/Synaptic/Software Center while running purgeconfig.

Download and install the script with this command:
wget http://purgeconfig.sf.net/purgeconfig.sh && chmod +x purgeconfig.sh && sudo cp purgeconfig.sh /usr/bin/purgeconfig

Usage is simple:
sudo purgeconfig package1 package2 package3 etc...

You can specify the architecture:
sudo purgeconfig package1:i386 etc...

 

If you have any questions/comments, feel free to leave a comment.

How to compile rt2870sta successfully

Many people have issues on compiling the rt2870sta driver, since it is no longer updated. The current Linux headers are much too new for the drivers. Luckily, the solution is very simple.
NOTE: This tutorial is only for USB Wireless Sticks.

Open up a Terminal window, and type in it:

lsusb

Look for a line a bit like this:

Bus 001 Device 004: ID 1234:5555 BRAND_OF_STICK

Remember the parts in bold.
Now type in a Terminal window:

wget http://dl.dropbox.com/u/21016209/DRIVER.tar.lzma.bz2
bunzip2 DRIVER.tar.lzma.bz2
unlzma DRIVER.tar.lzma
tar -xf DRIVER.tar
cd 2010_0709_RT2870_Linux_STA_v2.4.0.1
gedit common/rtusb_dev_id.c

A text editor should pop up. Add a new line (before the #endif // RT2870 // line), press TAB, and type:

{USB_DEVICE(0x1234,0x5555)},

And replace the parts in bold by the ID you saw in the lsusb command.
Save and close the file.
Then type in the Terminal window:

sudo make
sudo make install
cat << EOF | sudo tee -a /etc/modprobe.d/blacklist.conf
blacklist rt2x00usb
blacklist rt2x00lib
blacklist rt2800usb
EOF
echo 'rt2870sta' | sudo tee -a /etc/modules

Reboot.
If you had any problems, feel free to comment below.

If this helped you, please link to this post and/or leave a comment!