OpenGL, GLIntercept, wglGetPixelFormat and GL_INVALID_OPERATION

March 21st, 2010

I was recently working on implementing an OpenGL layer to our rendering engine and was using GLIntercept to find out what was going on “under the bonnet”.  The first problem I encountered was that some of my asserts for GL_NO_ERROR == glGetError() were firing, in seemingly benign locations. After a whole lot of Google detective work, I found what I believe to be the conclusive diagnosis for this issue. It seems to be an artefact of a particular setup combination – NVidia drivers (I have a GeForce 8400M GT in a Dell Vostro laptop) and Vista.

The post below describes how the NVidia driver is unusual in that it fires off another thread which calls wglGetPixelFormat whenever SwapBuffers is called, and if you are running GLIntercept, this multithreaded access causes a false positive to occur. I think the reason my asserts were firing is that the gl error state persisted beyond the SwapBuffers and fired whenever the next gl command was issued.

Anyway, I hope this find is of use to someone, it’s taken me a couple of days to work it out!

I found my rendering problem and have posted another message requesting help with that (dirtying color arrays when vbo are used). I also tracked down the problem in GLIntercept. I have hacked up a fix for that but a proper one will require a far more comprehensive look at the architecture of GLIntercept.

The problem stems from the fact that the underlying driver is making calls back into OpenGL on another thread when some OpenGL functions are called. and that GLIntercept intercepts calls to glGetError and returns a cached error value in some circumstances.

This leads to the following.

1. Main thread calls SwapBuffers (windows GDI function) which eventually calls wglSwapBuffers which is intercepted by GLIntercept.
2. GLIntercept end up calling the real glGetError to check the result of the wglSwapBuffers.
3. The nvidia drivers implementation of glGetError for some reason tells (or maybe creates) another thread to do some work and waits for it to complete.
4. The other thread calls wglGetPixelFormat which is intercepted by GLIntercept, which again calls the real glGetError, luckily this does not seem to bother with another OpenGL call, however the cached error code is now messed up.
5. The worker thread releases the main thread which returns
6. The main thread calls glGetError (I have the OSG check every frame option set) and gets a bad cached value from GLIntercept.

My fix is very hacky and just stops GLintercept calling glGetError if it is already in the process of doing so. This gets rid of most of the spurious errors.

I dont plan to go much further. GLIntercept also crashes on program exit it the runtime heap checks. But the logs seem OK. Let moe know if you want the modified code.

I found my rendering problem and have posted another message requesting help with that (dirtying color arrays when vbo are used). I also tracked down the problem in GLIntercept. I have hacked up a fix for that but a proper one will require a far more comprehensive look at the architecture of GLIntercept.

The problem stems from the fact that the underlying driver is making calls back into OpenGL on another thread when some OpenGL functions are called. and that GLIntercept intercepts calls to glGetError and returns a cached error value in some circumstances.

This leads to the following.

1. Main thread calls SwapBuffers (windows GDI function) which eventually calls wglSwapBuffers which is intercepted by GLIntercept.
2. GLIntercept end up calling the real glGetError to check the result of the wglSwapBuffers.
3. The nvidia drivers implementation of glGetError for some reason tells (or maybe creates) another thread to do some work and waits for it to complete.
4. The other thread calls wglGetPixelFormat which is intercepted by GLIntercept, which again calls the real glGetError, luckily this does not seem to bother with another OpenGL call, however the cached error code is now messed up.
5. The worker thread releases the main thread which returns
6. The main thread calls glGetError (I have the OSG check every frame option set) and gets a bad cached value from GLIntercept.

My fix is very hacky and just stops GLintercept calling glGetError if it is already in the process of doing so. This gets rid of most of the spurious errors.

I dont plan to go much further. GLIntercept also crashes on program exit it the runtime heap checks. But the logs seem OK. Let moe know if you want the modified code.

If your Vista thumbnails aren’t updating properly…

August 10th, 2009

I’ve been doing some more scanning, and stumbled upon yet another annoying bug in Vista. After rotating or cropping one or more images in a folder, I noticed that occasionally the thumbnail view wasn’t updating the image properly, and no amount of refreshing or browsing in/out of the folder would fix it. Unlike XP there’s no thumbs.db file to delete, but I found that if you run Disk Cleanup, you can select the thumbnail cache for deletion and although heavy handed – it certainly fixed the problem for me. Hope that tip helps someone else out there.

Vista & lossy JPEG rotation

August 31st, 2008

I’m really mad at Microsoft – they’ve wasted days of work for me. I’ve been batch scanning in old family slides using my Minolta slide scanner and just discovered that the Photo viewer built into Vista does not give a warning (like XP does) that any image with dimensions indivisible by 16 will be re-compressed when you rotate it. All of my scans are fairly arbitrary dimensions (since the cropping tends to vary slightly from picture to picture), and so I’ve been happily rotating away to find that most of my scans have not only been re-JPEG’d – but at a quality level low enough to introduce artefacts.

I only got suspicious when I noticed that one image had shrunk from 5Mb (I output all my JPEGs at 95% quality) to 1.5Mb. Further research revealed that Microsoft had indeed elected to drop the warning citing usability reasons, but quite frankly I am now in the camp of disgruntled users who will never use that tool again.  To Microsoft: these are my digital files – they cannot be permanently degraded without my permission.

For anyone in a similar position I can highly recommend JpegCrop – a simple and effective tool that rotates and crops JPEG’s using lossless techniques.

Netgear ReadyNAS Duo & Vista performance

August 30th, 2008

readynas-duo-240x240.jpg

So after a lot of watching and waiting I finally found the ideal storage device for our home network. I knew exactly what I wanted, it just hadn’t been available upto now. I wanted to get a single box with at least 500Gb storage, it also had to be small, reliable, well featured, low power, with printer server, backup and data redundancy features. We have quite a few computers in the house now, and with the Xbox just waiting to stream movies for us we just needed somewhere to have as a central store for all our movies, music and photos.

Given that hard drives have a finite lifespan, number one priority is data security, and this NAS has two hard drives that mirror our precious data on both drives so that in the event that one drive fails, the other drive will be used until the first can be replaced. What if both drives fail or the unit blows up I hear you say? Well, this NAS let’s you run regular backups onto other network destinations or USB drives too, so you get perferct peace of mind. I had read that some other (Maxtor I think) NAS devices with two drives are sealed units, which sort of defeats the object – with the ReadyNAS you just click a button to release the harddrive in it’s tray.

Anyway I’m pleased to report that this one is truly plug’n'play- it just works. Found it straight away as a windows share, the default shares and user permissions are sensible, the web-based control panel is clear and easy to use, and the unit has many other features that sound potentially useful (iTunes, BitTorrent services etc.)

Performance was also good on my wired XP machine, I’m pretty sure I got 100Mps data throughput. The only problem I had was really really slow access from a Vista laptop, and I mean slow – browsing the folders took forever – and ok I am on a 54Mpbs wifi network but the other XP wifi laptop didn’t seem to suffer as much. I checked out the ReadyNAS forums and found a couple of similar complaints but no hard and fast solutions. So for anyone else having similar problems, I did manage to fix it n the end by doing two things – I upgraded my laptop to Vista Service Pack 1, and changed the MTU to 1492 instead of the default 1500. Performance seems much better now, the only side effect is that I get no thumbnails when browsing for some reason.

So all in all, this is a great little NAS.

Konica Minolta Dimage Scan Elite 5400

January 25th, 2008

Konica Minolta DImage Scan Elite 5400I just acquired one of these scanners from ebay, which is easier said than done since they are always popular auctions. Remarkable really, since these kind of 35mm scanners are largely discontinued by the manufacturers. I have a Nikon CoolScan V already but what attracted me to this particular scanner was that it allows upto 4 slides to be scanned at once (the Nikon supports only one slide at once). Apart from that they have the same Digital ICE 4 technology, which I feel is essential for scanning negatives and slides – the difference in quality after hardware dust, grain and colour processing is amazing.

Anyway, I’m gonna give it a workout now, but for anyone else who is looking, the latest v1.1.5 driver can be downloaded from here. In case you didn’t know already, this scanner doesn’t seem to be compatible with Windows Vista unfortunately. However Silverfast and VueScan seem to support it ok, and if you’re like me, the following websites with scanning tips about how to use VueScan and Silverfast will come in pretty handy.

Porting MSVC C++ templates to GCC: typename is the missing link

December 20th, 2007

I’m in the middle of porting some game engine code to a GCC based build system, and got potholed badly. So far, this code has only ever been compiled by MSVC, and the loving touch of GCC’s standards compliance has been most revealing.

Consider this bit of template code (which compiles fine on MSVC):

    template<class RESOURCE>
    struct ResourceArray : public std::vector<RESOURCE>
    {
        std::vector<RESOURCE>::iterator Find(RESOURCE pResource)
        {
            std::vector<RESOURCE>::iterator it;
            for (it=begin(); it!=end(); ++it)
                if ((*it) == pResource)
                    return it;
            return it;
        }
    };

The first attempt to get GCC to compile this generates some very misleading errors, and my initial interpretation was there was either a namespace or a precompiled header issue.

../Code/include/GeResource.h(56): error: expected ';' before 'Find'
../Code/include/GeResource.h(64): error: expected `;' before '}' token

Having determined that it wasn’t namespace or pch related, I looked at trying to reduce the code and isolate the problem. The first step was to introduce some typedefines for those iterators, something I’ve incedentally started to do by habit anyway (all that std::vector<TYPE>::iterator stuff gets a bit verbose doesn’t it?).

    template<class RESOURCE>
    struct ResourceArray : public std::vector<RESOURCE>
    {
        typedef std::vector<RESOURCE>::iterator    ResourceIterator;

        ResourceIterator Find(RESOURCE pResource)
        {
            ResourceIterator it;
            for (it=begin(); it!=end(); ++it)
                if ((*it) == pResource)
                    return it;
            return it;
        }
    };

This still didn’t compile, but produced much more useful error messages, in particular an explicit fault with the iterator declaration.

../Code/include/GeResource.h(56): warning: ISO C++ forbids declaration of 'iterator' with no type
../Code/include/GeResource.h(56): error: cannot declare member 'std::vector<_Ty, std::allocator<_Elem> >::iterator' within 'GeLib::ResourceArray<RESOURCE>'
../Code/include/GeResource.h(56): error: expected ';' before 'ResourceIterator'
../Code/include/GeResource.h(58): error: 'ResourceIterator' does not name a type

Doing a bit of hunting on this error message was most fruitful. Indeed GCC is pointing out that my code is ambiguous and does not meet the recent C++ standards. Declaring types based on template parameters needs to done in an explicit fashion using the typename keyword.

A quick refactor of the code, using the typename specification as well as a bit of explicit scoping, gives us the clean compile we desire!

    template<class RESOURCE>
    struct ResourceArray : public std::vector<RESOURCE>
    {
        typedef             std::vector<RESOURCE>              ResourceVector;
        typedef typename    std::vector<RESOURCE>::iterator    ResourceIterator;

        ResourceIterator Find(RESOURCE pResource)
        {
            ResourceIterator it;
            for (it=ResourceVector::begin(); it!=ResourceVector::end(); ++it)
                if ((*it) == pResource)
                    return it;
            return it;
        }
    };

Feel good about defragmenting your hard drive

December 15th, 2007

My latest toy is JKDefrag. You know how after having a computer for a while the hard drive seems to get busier and slower during bootup? Well, it’s probably caused by fragmentation of your hard drive. The remedy built into Windows XP is the Microsoft defragment utility, which displays a barcode-like visualization of how fragmented your drive is, before it proceeds to spend a lot of time shuffling the bars around. I’ve often wondered how effective this tool really is, since you can run it twice in succession and it will find work to do on both runs.

JKDefrag is a great example of a tool that does one thing very well. Not only does it have an intelligent approach to the defragmentation and optimal organization of the files on your drive, you also get a convincing visualization of the de-fragmentation process – what starts out as a mess of green and yellow pixels finishes up as neat stripes of colours. In terms of improved performance, I don’t really know exactly how beneficial defragmentation is, but the brochure sure sounds nice!

Google Picasa: Are we there yet?

November 14th, 2007

PicasaBrian Auer offered an interesting article about his experiences with Picasa and what seemingly obvious improvements Google still need to make to it’s now-veteran, but still rather attractive photo organization software.

I’ve fallen in and out of love with Picasa a few times over the past couple of years. The last time we broke up was because I learned that it had been storing all of my carefully crafted labels and captions in a database file hidden away deep in some user specific documents and settings type folder. This was bad news for three reasons:

  1. Difficult to backup. My photos and my picasa photo database were hopelessly decoupled.
  2. Difficult to contribute. All of my attempts to label and categorize years of family photos were tied to a single user account on the computer so other family members would not be able to share the fruits of this labour.
  3. Difficult to share. Any image files shared with friends/family via a CD or email would not contain my hilariously witty captions and labels

After giving Picasa Web Albums a try I thought I’d re-visit Picasa 2, since the two are closely related, and see if could grow to love it again, and I can say that I’m almost there.

Gone is the problematic and bespoke labelling system in favour of something simpler – you now just have captions and ‘keywords’ (aka tags) for photos. Also, it appears that Picasa now follows the golden rule and stores the keywords and captions (metadata) in the jpeg file itself rather than in a separate ini file or whatever. At last, my photo archive is now truly portable, and it seems that my gripes above have been addressed. Now if only I could find some time to start sorting out those photos…..

Sharing Digital Photos should be easy

November 7th, 2007

Recently I signed up for Flickr, and I love it. For me as a hobbyist photographer, Flickr is perfect – it has the community, and the interface is geared around portfolios, comments and tags.

However when it comes to sharing family photos, Flickr isn’t quite so ideal. The main flaw is that I will require privacy by default for such photos, but Flickr only gives me that if I get all of my guests to also sign up to Flickr (aka Yahoo!) so that they can then view my photos by invitation. Not ideal.

The second preference for me is that I’d like to invite some members of my family & friends to upload their pictures as well, so that we all have a single place to view, contribute and collaborate on the mammoth task of scanning and tagging the thousands of family photos we’ve have been accumulated over the years.

Finally, I’m interested in the idea of a ‘archive’ approach, since quite a lot of these photos are being painstakingly scanned from slides, negatives and prints it would be good to know they are always available to everyone for download. This requirement means I need long term confidence in the service, so I reckon the bigger the name, the better.

My search for such a solution uncovered a number of options, but I was drawn to Picasa Web Albums. Picasa is one of Google’s babies, and is effectively Google’s answer to online photo albums. It doesn’t have the whistles and bells that Flickr has, but in the great Google tradition you get a minimalistic yet functional interface, support for tags, captions, and albums (sets) and it let’s you download whole albums at a time.

Google give you 1Gb (1024Mb) for free with no upload limits, whereas Flickr offer unlimited storage but restrict uploads with free accounts. For $20 per year, Google will let you rent 10Gb (probably space enough for 5000 six mega pixel images) , but Flickr lets you upload as much as you like.

To get it going, you only need a Google account, for which the signup is simple and impersonal, and crucially it doesn’t indirectly sign you up for email and other services you may not want. This suits me as I can create an account for the family to share, but for now I already have a Google account so I’m giving it a test run, and so far, so good!

Content is king, but rambling is more likely. Welcome to my first blog!

October 11th, 2007

Welcome dear reader, a warm greeting to you and your clan.

Well at long last, I’ve found a spare moment to put something on podvilla that can hopefully structure my more lucid streams of consciousness and channel them somehow to this old domain that I registered all those years ago, (probably using a 14.4 modem it was that long ago!)

That said, it’s long been my ambition to contribute something meaningful to the internet, and so far I don’t think that a few photos of crap cars, hamsters or forum posts have got me even close to giving something back to the community from which I take so much (although I did once undo some vandalism on the wikipedia page for ‘Worms’, which actually gave me a warm sense of patriotism for a whole minute or so.)

The problem for me has never been ideas for website content, it’s been a quest to find the right delivery mechanism for it (I know I know, he’s blaming the tools). But I’ve done meticulous ‘research’ into so-called CMS over the years, you name it, I’ve played with it. I even learned Perl just so I could install Twiki. Then when cgi-bin got unfashionable I learned about PHP and MySQL and tried phpForums, phpWikis, phpNukes, phpWebsites etc. etc. Somehow it seemed that actually installing these tools was more fun than using them.

Now, here we are in the bright new age of Web 2.0 and I’ve found WordPress. So now I’m wondering with where to start with all of the various brainwaves and amusing jpeg’s I’ve been squirrelling away over the years. Stay tuned, dear reader, and we’ll soon find out. :)