Yet Another Software Blog

The worm attack on pre 2.8.3 wordpress versions started a blame game between wordpress users and wordpress developers and core community.  While the users claim that although wordpress is great piece of blogging software, it is not the center of their lives, and it is ridiculous to expect them to check every day if there is a new version available, and dedicate instantly the time to perform an upgrade. The developers on the other hand, claim that it is the user’s responsibility to keep their wordpress updated whenever it is needed.

As I am a great believer in personal responsibility, I sympathize with the developer’s position – If a user can not spare the time to maintain their wordpress, they should consider opening a blog at wordpress.com, or other service of the same kind.  But then I think about how wordpress is being marketed, with its 5 minutes install, central repository of plugins and themes with a one-click install, and other very user friendly features. But surely this set of features attracts users which are less technically savvy and if they complain about the upgrade process it probably means that it is relatively complex, especially when comparing to other features of wordpress.

My understanding is that the main complaint is that the upgrade process might fail due to a plugin or theme which are not compatible with the new version. It is actually a serious problem when the development team wants to push out a new version every three months, which practically means that users have to spend time on upgrading and testing every three months. For non technical bloggers this might be a real pain.

What wordpress need is an upgrade path that is by definition more stable then upgrading to a new release. Maybe there should be support for older version for a year, in which a new minor update will be released for any security related problem. This way the users will be assured that the security upgrade will not break their plugins and themes, and they will be able to plan in advance when to upgrade to the latest and greatest release, when it is comfortable for them, and not when some wordpress dude says that you have to upgrade right now.

Even if it happens, there would be users which will not upgrade, but you can not force everyone to do the right thing ….

Plugins should add content to the RSS feed but only if that is there main function, otherwise they probably should not. Why not? because the main functionality of the RSS feed is to notify a subscriber about new content, while giving him a glimpse of what the content is about.

Most of the plugins which add something to the content displayed on the web page, such as the social bookmarks icons, do not add any new content to the post itself, and therefor they should not be adding anything to the RSS feed.

A typical social bookmarking plugin code looks like

function handle_content($content) {

 $content = $content.'<div><a href="url1"><img src="src1" /></a><a href="url2"><img src="src2" /></a></div>
';
 return $content;
}

Which looks good on the web page, but in the RSS the result will depend on the number of words in the original post. If the original post contained less than 55 words, the extra html which was added by the plugin will be sent as part of the RSS.

The result might be (due to some cleaning that WordPress does)

Original content

<a href="url1"><img src="src1" /></a>
<a href="url2"><img src="src2" /></a>

Which adds two links which are not part of the content. Actually it can be even worse with the extra content being split up and the result is something that resembles junk. That is why the plugin code should look like

function handle_content($content) {

 if  (is_feed()) // do nothing if we are generating a feed
   return $content;

 $content = $content.'<div><a href="url1"><img src="src1" /></a><a href="url2"><img src="src2" /></a></div>
';
 return $content;
}

This way the changes made by the plugin are not seen in the RSS feed. But there always should be an exception, and if the RSS has the full post maybe it does make sense to add the same content to the feed and the post. Add a check of the value of the rss_use_excerpt option, which indicates whether the RSS is full, and the code becomes

function handle_content($content) {

 if  (is_feed() && (get_option('rss_use_excerpt') == 1)) // do nothing if we are generating a feed of excerpts
   return $content;

 $content = $content.'<div><a href="url1"><img src="src1" alt="" /></a><a href="url2"><img src="src2" alt="" /></a></div>
';
 return $content;
}

But what should be done with shortcodes?

I have spent the first eight years of my software development career developing an embedded software for telecom devices. At that point of my life the term real time indicated for me a system that has to reliably and consistently perform a task in a very small a mount of time. For example switching from a faulty controller of a fiber to the backup should be done in less the 50 milli-seconds.

Later I wrote network monitoring software, and discovered the term “soft real time”. Soft real time is about the human perception of time and its granularity. User interface should respond to input in less than 5 second, if your software takes a second to respond that is good, but if it takes half a second, it is most likely that no one will notice the difference. If a router fails it is important to notify the on duty system administrator in no more than a minute, but since it will take him several seconds to just open is cellphone and look at the message, there is no much point to do extra work just to cut the notification time to one second.

Now people are talking about real time updates on the web. To have a meaningful discussion about it, and the proper ways to implement it, someone has to define what is the expected notification time and the longest time which can pass from the origination of a message until people are notified about it. Without a proper definition no one can objectively asses the pros and cons of methods to implement it.

Why my site is “rocking” when it is being loaded? seems to be a question not being asked enough. Even in digg.com, the capital of nerddom, the site rocks after it content was loaded to show an infomercial at the top of the page.

It is annoying because you actually start reading, or at least scan the page, when the text start to show. When the page “rocks” it disorients you, and you have to search for the location of the text you were reading.

It used to be caused by images embedded in the HTML code. When the browser reads gets to an IMG tag when parsing the HTML, it has no way to know what are its height and width, until after the image was fetched from the server, and therefor how much visual space it should reserve for it.
This impacts the browsers decision where to place the other items of the page, and when finally the image is loaded the browser finds its height and width and has to recalculate the position of the other elements on the page (a process also known as reflow).

The cure for this is very simple – in most cases you know in advance what is the expected dimension of the image when it is an integral part of the design, or you can use some library to calculate if it is stored on your server, therefor you can set the width&height attributes of the IMG element or specify it in a CSS style rules. This leaves as a problematic issue only the situations in which you embed an image which is located on other server not controlled by you, but this probably happens very rarely for most of the sites.

AJAX scripts are another cause for “rocking” . The main reason for AJAX “rocking” is post loading of dynamic information, usually done when the site serves static HTML pages, probably because of performance related reasons, but needs to show some dynamic content, and therefor loads it after the page was loaded.

As with the images, you simply have to specify the height and width in advance – since the target area for the data received from the AJAX operation is usually a DIV, just specify its height and width.

Note: There is a fundamental difference between rocking when the page loads as result of AJAX, and “rocking” when page changes as result of AJAX. When the page “rocks” after a user initiated an action, the user is idle while waiting for the page to change, and therefor the “rocking” is not as annoying (especially if you conceder that the alternative is full page load).

I was testing some wordpress optimization ideas which lead me into writing some profiling code which somehow lead me into installing alex rabe’s wordpress memory usage plugin. What especially caught my eye was the test that demonstrated that wordpress with german translation requires 6.5 Megs more then the english wordpress version. I have repeated his test with the hebrew translation and got a difference of about 4 Megs (the difference between the languages can be explained by the fact that german has notoriously long words or the hebrew translation is somewhat sloppy).

This just confirmed something that I have thought for quite a long time but hadn’t figured out how to quantify it. The problem with the wordpress translation process is that you translate the whole of wordpress and generate a single translation file which essential contains pairs of strings and their translation. Then you place your translation in predefined place and wordpress loads the whole translation pairs to the memory when the first string is being translated.

The problem with this approach is that as time passes by, the admin interface accumulates much more translatable strings then the front end functions. Of the ~3200 translatable string in wordpress I guess that no more the 400 are used when the actual HTML for the blog posts is generated. In other words, about 80% of the memory consumed by the translation will never be used for the work wordpress does for the 99% of the incoming trafic.

As if to demonstrate the problem, for version 2.8 the wordpress developers had split the translation file and created a unique translation file for the cities of the world, used when configuring time zone information, which is loaded only in one location in the admin interface and by that saved 400 string from loading at any other page of the site. But then, those strings where actually loaded when you reached that page, and with some help from inefficient PHP code, have caused an “out of memory” php error. The end result was that part of the page did not get displayed and since the error was not displayed, it was really hard to even understand what is the problem, and harder to find its cause.

The whole issue is not being helped by the fact that there is no way to use the PHP gettext module from wordpress without hacking the core. I don’t know how much it will be more efficient but since it is written in the C language it can theoretically use less memory and have a faster response time then the PHP implementation.

Ideal translation implementation will have a single translation file for a single uri, but this is currently far from being practical as it is hard to know all the possible execution paths just by looking and the code, and therefor hard to determine which translations will be required for each uri. Maybe some run-time adaptive algorithm which will compute the translation file while wordpress is executing will give the best approximation to the ideal.

Meanwhile, in the real world, the translation should be separated at least in to two parts – the front end, and the admin. This should be done either by moving all of the admin related string to reside under the wp-admin directory while the front end will be found only under wp-includes directory. Alternatively the wordpress release process should generate two POT files for the release by employing some internal knowledge on which files/strings should be used only at the admin, aknowledge which most of wordpress tranlators lacks.

While this will not necessarily solve the problem of the translation “nuking” the admin interface, it will at least reduce the memory consumption and improve CPU performance at the front end

Note: while caching plugins like super cache can be used to practically ignore this problem, not every shared hosting has the facilities to support it, and it has its own drawbacks which will prevent it from being a bullet proof solution (otherwise its place is in the wordpress core).

Update: Looks like I am not the only one who is bothered by the inefficiencies in the way the translation works. Johan Eenfeldt had created a plugin which provides caching to the parsed translation file. This might not solve my memory related problems (I will have to use the file cache option), but at least it is a performance gain.

There is an urban legend about bill gates of microsoft saying in the old DOS days that 640KB of RAM should be enough for everybody. It seems like someone in the mozilla organization is thinking along the same lines of thought, and while you actually have an option to change the amount of disk space which can be used by caching, the practical limits are not a lot higher then the default of 50MB as firefox, as the real limiting factor on the usage of the cache is an obscure technical decision to limit the number of files in the cache to 8192, a limit which is reached very easily today with all the small javascripts and images you get from every site.

The relevant bug contains a very small patch which probably solves the problems, but for sure normal users will not be able to set up the build environment required to build firefox, and even most of the geeks will have trouble doing so. In any case no one really wants to fork firefox just because of this bug….

So what can be done until the high console of mozilla will decide to solve this bug? Install a caching proxy!. Unfortunatly I could not find a nice small one in 5 minutes search on the net. All I have found were ad blocking non-caching proxies which are not interesting in this context

Although I think that I understand the possible value that twitter may have, I actually find that I don’t posses the hyperactivity traits needed to have a never ending real time communication, and actually maybe I’m not very communicative at all which results in me wanting to control the time and duration of the communication when I decide that I have to do it.

I used to be very negative about twitter*, but as time flows by even I find that there are some useful aspects for the service, the problem is that it is so over hyped and misused that even for people like me that are not interested in the commotion it is hard to ignore it.

The latest annoying twitter sightings in the WordPress world is in blog comments. For some reason people think that it is cool that people are twitting about their posts, and they are probably right. The problem is that comment are good for discussion and sharing information and sometimes they have a content which is actually more important/interesting then the actual post. Then you got trackbacks, which some WordPress themes relegate to the end of the comment section, and while they are not adding anything to the discussion they let the casual visitor from a search engine to explore other opinions on the topic.

Now comes the twittings as part of the comments. Like the trackbacks they do not add anything real to the discussion, but unlike the trackbacks they are relevant for no more than a day (at least I couldn’t find a reasonable way to follow a discussion on twitter which is more they a day old), so while at the first day from the posting they might add some value to the reader, for the casual visitor from a search engine they will be just a visual and bandwidth dead beef.

Conclusion: even if you enjoy the ego boost which comes from being mentioned by other people, do me (and maybe other people as well) a favor and make your comments section easier to read by cleaning up the twitts from time to time.

*One of my clients (web/social ad company) had a link to the twitter of one of its client which was kind of design firm. I asked once what is the purpose of the twitts? in what situation will it be helpful for a reader to have a real time announcement of something related to design? The answer I got was that maybe I’m wandering in the streets of tel aviv with nothing to do and the twitt will give me an idea. I was far from being impressed with this answer, but at least they have done it as part of their overall service and didn’t charge outrageous sums of money just to open a twitter profile. Wow, it seems like it was in the distant past, but this naive approach for advertising in twitter was articulated in 2008

I was very excited when the awesome bar was introduced in firefox 3.0, I thought that it will change the way I use my browser – no more automatic googling for sites in which I have been rarely before, but rather first doing a local search in the browser history which at least should be faster.

I was right. The knowledge of the existence of the awesomebar have actually changed the way I use the browser. I no more attempt to guess the initial character in the name of the site but rather some part of it, or an outstanding word in the title. On the other hand, the awesome bar caused me a lot of misery every time I have misspelled something which had no match in the history DB as it would practically freeze the input to the address bar while it was conducting its useless search.

FF 3.5.2 have changed things to the better, as even when there was no match the perceived responsiveness have improved*, but now mozilla guys have publicized an hack which improves the performance of the awesome bar by essentially optimizing its DB. This shows a really sloppy coding practizes by mozilla. If you are going to use a DB embedded in your product, you better have someone how understands the pitfalls of using such a DB. The obvious pitfall in this case is that from time to time the DB has to be optimized in order to have fast responses from the DB, which means in the embedded DB case, that the embedding software have to do the cleanup automatically.

I have read some excuses about the optimization taking long time in which the GUI is frozen, but those are mere excuses as the optimization can be moved to the background as most of the time the browser is just seating idle, or to the startup or shutdown parts of the code. In the worst case FF should at least try to use some heuristic to calculate if an optimization is required and prompt the user that it is about to run one. I find it hard to believe that anyone will complain if this will happen once in two months as upgrades to new dot releases are being pushed in a higher frequency (and maybe it should be combined?) and I haven’t heard any complains about that.

* As a side note, I think that with FF 3.5 mozilla got away without getting the amount of criticism that a commercial company would get for releasing an uncomplete product. 3.5.2 solved a major usability bugs of the awesome bar and the startup time, bugs which were surly known at the time 3.5 was developed but waited another two “dot” releases to be at least partially fixed.

Not having any inclanation to look into the PHP and APACHE source code I will have to qualify the statement in the title that it is coming from an observation and should not be treated as a consistent fact.

To make a long story short, I have moved from a shared hosting environment into an unmanaged VPS, which have a relativity small amount of RAM memory – 256M. One of the things that I am running on this VPS is a gregarious based RSS aggregator (pity that this project is virtually dead), the rest were my hebrew wordpress based blog, this wordpress mu based installation, and a mail server.

Once in several days the swap usage of the server will sky rocket and I will have to manually reboot the VPS. Though I have tried to find a common reason to this phenomenon, I could not fins one and I assumed That I am just over using my memory.

I have tried to do various change to the apache parameter controling the number of instances and the time it takes before they restart themselves, but altough there was an improvment, the server will still freeze from time to time.

One day the aggregator stopped giving any output at all. I have made some debugging and found out that what was killing it was an output compression done by calling ob_start(‘ob_gzhandler’) since when I have removed the call everything started working fine.

Then I have spent some time thinking about the “why” since while the prices of bandwidth have fallen through the years it is still nice to save if it is not too difficult. Finally it struck me that the only way the ob_gzhandler handler can work is by allocating memory which is in some way proportional to the amount of output generated, and for a memory starved machine and large output this can be a real killer since, as every software developer should know, there is not a single software on earth which can gracefully deal with shortage of memory resources.

Anyway, since most major web servers include now the capability of compressing the output, maybe it is better to leave it to them instead of implementing it in software, as they mught have a better dedicated code then PHP, and when they perform the compression they have probably cleaned the memory used by the PHP code, so they have more resources.

Wait a moment!, shouldn’t this line of reasoning be applicable to the usage of ob_start in any case? IMHO the answer is yes, and ob_start should be used for the shortest time, and the shortest output you can reasonably manage to get, unless you have idle memory and you don’t want it to feel neglected.

OK, that was a major annoyance!. I found a nice mysql backup script, changed all the constant which define the DB names, user, password and backup path, run it from the terminal and everything looked great. The following day I have checked the backup and there was no automatic backup done!

I have checked that all the paths in the script are absolute, and the script is executable and it has enough permissions to run, even changed the location of backup files, but still nothing happened.

For two months I have run the script manually when I happened to log into the server, but to day I woke up determined to stop this nonsense and try to find the root cause and solution to this problem.

After some googling and reading deep into some forum discussions I have discovered that the “run-parts” script which is the real engine behind executing the script in cron.daily don’t know how to handle files with extensions , and I used a .sh extensions for the backup file since I like the name of the file to express as much as possible its content.

This was a major PITA. You expect the extension part to be significant on windows but not on Unix, and this is not documented in any of the major documentation sites of Linux and Ubuntu …. Well, at least now there is a blog post about this…….