In the process of moving to USA all my data is scattered around. It has been almost 2 years now but still I need to re-organize most of the data which are saved in various random places for redundancy. Having a couple of HDD failures didn’t help either.

All of the pictures are organized now; I had to write few Python scripts which check the EXIF data to find duplicates though. Well, it’s another story, now I’ll talk about the SVN installation.

Few weeks ago I started to gather my personal codebase part of which was in CVS repository. I wanted to port it to Subversion. For that I installed the SVN server and setup a repository on our Windows7 computer which we use as a home server.

Note to Self 1: This is the web page that helped me to learn the main steps of the process.

Note to Self 2: Also the cvs2svn script was a life saver, which preserved all of the history of the projects. Although they say that cvs2svn does not officially support CVSNT repositories, in my case the conversion went pretty smooth.

The next step is digging out more of my old sources and adding them to the repository.

 

Here is the Python script that I use to launch Visual Studio for working on Brazil code locally. It sets the proper search paths -including 3ds max SDK folders- for the compiler and linker for the chosen target platform and 3ds max version, then launches the correct visual studio.

import os
import msvcrt
import platform
import subprocess
from winsound import *

#===============================================================================
# Machine dependent Values
#===============================================================================
MaxBins={
	3:"c:\\graphic\\3d\\3dsmax3\\",
	4:"c:\\graphic\\3d\\3dsmax4\\",
	5:"c:\\graphic\\3d\\3dsmax5\\",
	6:"c:\\graphic\\3d\\3dsmax6\\",
	7:"c:\\graphic\\3d\\3dsmax7\\",
	8:"c:\\graphic\\3d\\3dsmax8\\",
	9:"c:\\graphic\\3d\\3dsmax9\\",
	10:"c:\\graphic\\3d\\3dsmax2008\\",
	}

MaxSDKs={
	3:"C:\\dev\\SDK\\Maxsdk3\\",
	4:"C:\\dev\\SDK\\Maxsdk4\\",
	5:"C:\\dev\\SDK\\Maxsdk5\\",
	6:"C:\\dev\\SDK\\Maxsdk6\\",
	7:"C:\\dev\\SDK\\Maxsdk7\\",
	8:"C:\\dev\\SDK\\Maxsdk8\\",
	9:"C:\\dev\\SDK\\Maxsdk9\\",
	10:"C:\\dev\\SDK\\Maxsdk10\\",
}

VCExes = {
	6:"C:\\dev\\vs6\\Common\\MSDev98\\Bin\\MSDEV.EXE",
	7:"C:\\dev\\vs7\\Common7\\IDE\\devenv.exe",
	8:"C:\\dev\\vs8\\Common7\\IDE\\devenv.exe",
	}

SetVarBatchesVC8x64={  #host OS dependent batches for target platform
	"x86"  :"C:\\dev\\vs8\\VC\\bin\\x86_amd64\\vcvarsx86_amd64.bat",
	"AMD64":"C:\\dev\\vs8\\VC\\bin\\amd64\\vcvarsamd64.bat",
	}

SetVarBatches = {
	"x86":{
		6:"C:\\dev\\vs6\\VC98\\Bin\\vcvars32.bat",
		7:"C:\\dev\\vs7\\vc7\\Bin\\vcvars32.bat",
		8:"C:\\dev\\vs8\\vc\\Bin\\vcvars32.bat"
		},
	"x64":{
		8:SetVarBatchesVC8x64[os.environ["PROCESSOR_ARCHITECTURE"]]
		}
	}

BrazilProjectPaths = {
	1: "D:\\_works\\dev\\SplutterFishCVS\\B1Max\\",
	2: "D:\\_works\\dev\\SplutterFishCVS\\Daedalus_01\\"
	}

BrazilTypeDefs = {
	"P": "BRAZIL_PRO_EDITION",
	"B": "BRAZIL_BASIC_EDITION",
	"R": "BRAZIL_RIO_EDITION",
	"E": "BRAZIL_EDU_EDITION",
	"N": ""
	}	

#===============================================================================
# Generic Values
#===============================================================================
MaxSDKIncSub = "\\include"
MaxSDKLibSubs= {"x86":"\\lib", "x64":"\\x64\\lib"}

#===============================================================================
# Functions
#===============================================================================

#_______________________________________________________________________________
def choose_max():
	while True:
		print "\nMAX version        [ 3 | 4 | 5 | 6 | 7 | 8 | 9 | 1(0) | Q ]    : ",
		choice = msvcrt.getche().capitalize()
		if choice=="Q": exit()
		if choice in ["3","4","5","6","7","8","9"]:	return choice
		if choice=="0": return "10"
		print " incorrect input!",
		MessageBeep(MB_ICONHAND) 

#_______________________________________________________________________________
def choose_platform():
	while True:
		print "\nPlatform Type      [ (3) x86 | (6) x64 | (Q)uit ]              : ",
		choice = msvcrt.getche().capitalize()
		if choice == "Q": exit()
		if choice == "3": return "x86"
		if choice == "6": return "x64"
		print " incorrect input!",
		MessageBeep(MB_ICONHAND) 

#_______________________________________________________________________________
def choose_brazil_ver():
	while True:
		print "\nBrazil r/s version [ 1 | 2 | N | Q ]                           : ",
		choice = msvcrt.getche().capitalize()
		if choice=="Q":	exit()
		if choice in ["1","2","N"]:	return choice
		print " incorrect input!",
		MessageBeep(MB_ICONHAND) 

#_______________________________________________________________________________
def choose_brazil_type():
	while True:
		print "\nBuild Type         [ (P)ro | (B)asic | (R)io | (E)du | (Q)uit ]: ",
		choice = msvcrt.getche().capitalize()
		if choice=="Q":	exit()
		if choice in ["P","B","R","E"]: return choice
		print " incorrect input!",
		MessageBeep(MB_ICONHAND) 

#_______________________________________________________________________________
def choose_compiler():
	while True:
		print "\nCompiler Type      [ (M)icrosoft | (I)ntel | (Q)uit ]          : ",
		choice = msvcrt.getche().capitalize()
		if choice=="Q":	exit()
		if choice in ["M","I"]:	return choice
		print " incorrect input!",
		MessageBeep(MB_ICONHAND) 	

#_______________________________________________________________________________
def choose_asm_generation():
	while True:
		print "\nGenerate ASM code  [ (Y)es | (N)o | (Q)uit ]                   : ",
		choice = msvcrt.getche().capitalize()
		if choice=="Q":	exit()
		if choice in ["Y","N"]:	return choice
		print " incorrect input!",
		MessageBeep(MB_ICONHAND) 

#_______________________________________________________________________________
def choose_pdb_generation():
	while True:
		print "\nGenerate PDB files [ (Y)es | (N)o | (Q)uit ]                   : ",
		choice = msvcrt.getche().capitalize()
		if choice=="Q":	exit()
		if choice in ["Y","N"]:	return choice
		print " incorrect input!",
		MessageBeep(MB_ICONHAND) 

#_______________________________________________________________________________
def do_menu():
	global max_ver, platform, brazil_ver, brazil_type, compiler_type, gen_asm, gen_pdb
	# ask max version
	max_ver = int(choose_max())

	# ask target platform
	if max_ver==9:	platform = choose_platform()
	else:			platform = "x86"

	# ask Brazil version
	brazil_ver = choose_brazil_ver()

	# ask Brazil type
	if brazil_ver != "N":
		brazil_ver  = int(brazil_ver)
		brazil_type = choose_brazil_type()
	else:
		brazil_ver	= 0
		brazil_type = None

	# ask Compiler
	compiler_type = choose_compiler()

	#ask ASM and PDB generation
	gen_asm = choose_asm_generation()
	gen_pdb = choose_pdb_generation()

#_______________________________________________________________________________
def run_msvc():
	# determine vc version
	if 	 max_ver in range(3, 6):vc_ver = 6
	elif max_ver in range(6, 9):vc_ver = 7
	elif max_ver in range(9,11):vc_ver = 8
	else:
		print "unsupported max version"
		exit();

	# determine the include and lib paths
	max_sdk_inc = MaxSDKs[max_ver] + MaxSDKIncSub
	max_sdk_lib = MaxSDKs[max_ver] + MaxSDKLibSubs[platform]

	# prepare the common command strings
	call_bat_cmd= "call " + SetVarBatches[platform][vc_ver]
	vc_exe_cmd 	= VCExes[vc_ver] + " /USEENV"

	# prepare the environment variables
	lib_env	= MaxSDKs[max_ver]+MaxSDKLibSubs[platform]
	inc_env	= MaxSDKs[max_ver]+MaxSDKIncSub
	cl_env  = ""
	link_env= ""

	# do the version specific adjustments to the commands
	if compiler_type=="I":
		cl_env +=" /DUSING_INTEL_COMPILER"
	if vc_ver==6:
		vc_exe_cmd +=" /Y3" # show build time (undocumented switch)
	if vc_ver==8:
		cl_env +=" /D_CRT_SECURE_NO_DEPRECATE" 	

	# set brazil type
	if brazil_type<> None:
		cl_env += " /D" + BrazilTypeDefs[brazil_type]

	# Turn on ASM List Generation
	if gen_asm=="Y":
		cl_env	+=" /FAs"

	# Turn on PDB Generation
	if gen_pdb=="Y":
		cl_env	+=" /Zi"
		link_env+=" /DEBUG" 

	# Display the variables
	print "\n\n"
	print "========================================================================="
	print "call_bat_cmd=" , call_bat_cmd
	print "vc_exe_cmd  =" , vc_exe_cmd
	print "lib_env     =" , lib_env
	print "inc_env     =" , inc_env
	print "cl_env      =" , cl_env
	print "link_env    =" , link_env
	print "=========================================================================\n"

	# set the environment variables of the sub process
	env_vars = os.environ
	env_vars.update({
		"LIB"		: lib_env,
		"INCLUDE"	: inc_env,
		"CL"		: cl_env,
		"LINK"		: link_env,
		})

	# have sub cmd interpreter execute the commands that needs to be in the same process tree
	print "running the compiler..."
	command_str = "cmd /s /c "
	command_str+= call_bat_cmd
	command_str+= " && " + "call ListDevEnvVars.bat"
	command_str+= " && " + vc_exe_cmd
	subprocess.call(command_str, env=env_vars )

#_______________________________________________________________________________
def incremental_backup():
	print "daily backup ..."
	subprocess.call("daily_backup.bat")

#===============================================================================
# MAIN
#===============================================================================

do_menu()
run_msvc()
incremental_backup()

#exit
print "press any key to exit...\n"
MessageBeep(MB_ICONASTERISK)
msvcrt.getch()
 

Approximately 1 month ago I downloaded a .gz file which supposed to be an iso file of an software installation DVD from the software company’s own site. I’ll not disclose the company name in case this is a weird security method (which should not since I’ve legit access to that file and they should inform how to open that file.) Anyhow to be on the safe side I’ll not tell which company and which product. That’s not important indeed.

After downloading 1.5GB of data with my 1024kbit ADSL line, I failed to open the file with WinRAR which is able to open gz files normally. I downloaded the gzip utility thinking that the file may be a in a new gz format that WinRAR is not aware of… Nope! that didn’t work either. The 3rd thing I tried was renaming the file from xxx.iso.gz to xxx.iso hoping that the file is uncompressed somehow on the way (I read that’s possible, client de-compressing the gz file during download). That approach failed to.

So there was only one thing to do, re-downloading the file as it’s obviously broken. I did a second download (many more hours). And guess what? The new file is broken too!!! GRRR! I did a binary comparison on the first and second files and they were identical (WinHex is a very powerful life-saver tool BTW). So the file is either broken on the server or it’s being spoiled -in a consistent way- by something on the way or on my end. Since I’ve already spent so much time, I changed the plan and downloaded a public-trial version of the software and that was enough for me.

That was 1 month ago. Then today…

Today one of my co-workers downloaded the new (next) version of that software from that legit location again. This time it’s a 2.3GB download and guess what? Yup: she could not open the gz file. I was curious and I started downloading it. Meanwhile I started poking the old file, that I failed to open last month. I kept that file in case I can open it someday. I’m glad that I did.

First I opened the file with WinHex. I did a spectral analysis on the file (counting bytes to see which characters are occurring in which frequency).
spectrum of gz file
Almost all the characters were occurring with the same frequency and that was a good indicator that the file contains compressed data (very basic check for high entropy). For a comparison here is the spectrum of uncompressed iso file (with obviously low entropy):
spectrum of iso file
Since the file looks like compressed file, then I decided to check if the file matches the gz file format. I googled the web and found this gz file format descriptor text.

The first few bytes of my gz file was like this:

3C 8B 08 08 2B 3C FC 46 00 03

I started cross-checking.

offset:0      2 bytes  magic header  0x1f, 0x8b (\037 \213)

my file has “3C 8B” instead of “1F 8B”. Weird. This didn’t match but since one byte is matching I went on…

offset:2      1 byte   compression method
0: store (copied)
1: compress
2: pack
3: lzh
4..7: reserved
8: deflate

My file has 08 which is logical (deflate)

 offset:3      1 byte   flags
bit 0 set: file probably ascii text
bit 1 set: continuation of multi-part gzip file, part number present
bit 2 set: extra field present
bit 3 set: original file name present
bit 4 set: file comment present
bit 5 set: file is encrypted, encryption header present
bit 6,7: reserved

I have 08 which means “original file name present” and yeah I see the iso file name after few bytes. Hmm.

offset:4      4 bytes  file modification time in Unix format

I have 2B 3C FC 46 which is 0x46FC3C2B which makes 1190935595 in decimal. And that converts to “Fri Sep 28 2007 02:26:35 GMT+0300 (GTB Daylight Time)” which makes sense.

offset:8      1 byte   extra flags (depend on compression method)

I have 00 which is meaningful too. And lastly

offset:9      1 byte   OS type

I got 03 there and that means Unix which makes complete sense too. And after this point I have null-terminated string having the iso file name.

So it seems that only the first byte of the file is wrong and besides that the rest of the file is indeed a gzip file. I modified that first byte with WinHex to 1F so that my file starts with bytes

1F 8B 08 08 2B 3C FC 46 00 03

instead of

3C 8B 08 08 2B 3C FC 46 00 03 

… and YAY! WinRAR can now open and uncompress that file happily. I did a brief search on the net to see if there is a variant of gzip file with 3C 8B header magic instead of 1F 8B but I failed to find such info. I still wonder how that happened both to me and my colleagues. But anyhow we have a solution now. Since my co-worker didn’t have a WinHex license, I sketched this Python code which changes the first byte of the given file to 0x1f

f=open('C:\\foo.gz', 'r+b')
f.seek(0)
f.write("\x1F")
f.close()

If anybody know anything about those weird gz files, I’d like to hear.

 

At Splutterfish we decided that Brazil r/s should support earlier 3ds max versions too. We support max 4 and above (Earlier max versions lack some of the core functionality that Brazil relies on). Unfortunately, because of the relatively poor design of the 3ds max API, max plugins need to be compiled with the same compiler that the max is compiled with. This is mainly due to the heap mismatches between the run time libraries (RTL) of the different versions of compilers. I’ll briefly explain why this compiler dependency occurs.

In some part of the 3ds max API design, the plugin needs to allocate memory and pass it to host application (3ds), which frees that memory itself afterwards. This unbalanced allocation/deallocation location becomes a problem if the host and the plugin are compiled with different compilers as the memory is allocated by one RTL and deallocated by another. This problem is usually/normally dealt with adding a DeleteThis() method to classes so that each object knows how to free itself, so every object is created by a factory in the plugin module and freed by itself which is also in the plugin module. For non-object memory blocks similar create and delete functions can be added. This makes sure that all allocation and de-allocation is done on the plugin module thus uses the same RTL. In fact 3ds max API generally makes use of this DeleteThis() approach. The problem is it’s not used everywhere! Don’t ask me why, I have no idea! Therefore the heap mismatches become a problem.

With 3ds max 6, they started to export the 3ds max RTL’s memory allocation and de-allocation methods in the max API libraries. This gives the plugins the opportunity to use max’s allocation and de-allocation routines with some tricky overrides. So if the plugin uses those routines instead of the ones in it’s own RTL, then all allocation and de-allocation will be done by the RTL of the host application thus the problem is solved. It’s not a clean solution though, it’s more like a hack in my opinion. Some data is still being allocated on one side (plugin) and being deleted on the other side (host application). This is a poor design and Autodesk’s solution of “forcing both sides to use the memory routines from the same RTL” is not correcting the design problem, but just patches it. Please note that plugin will use memory routines from the host application’s RTL but will continue to use other routines of it’s own RTL. So the plugin will use two RTL’s; RTL1 for memory operations, RTL2 for other operations. Personally I’d prefer them fixing the design problems in the API by removing the tightly coupled allocation/de-allocation. Probably they decided to avoid the changes which would force them rewrite and retest some part of the 3ds, but took this “clever” route. What I’ve learned through my professional software development experience is that “local smart” solutions will be pain in the ass in the long term. You should save your intelligence to come up with a good design instead. Autodesk developers are not very happy about this solution either (they should not indeed) and they too recommend using the same compiler as the max for the plugins. Which is wise in my opinion too and it’s what we’d do regardless their recommendation.

This brings us another (relatively minor) problem: We need to be able run different compilers (different versions of Visual Studio) on the same computer. This is, of course, possible but you need to be careful about it, especially about the path clashes. 32-bit and 64-bit target (and host) platforms makes the things a bit more complicated. In the next post I’ll talk about the system that I came up with to use VS 6, VS .NET 2002 (7.0) and VS .NET 2005 (8.0) on the same computer along with 8 different versions of max SDKs, starting from max3 up to max 10 (aka max 2008).

 

Well, I need to start blogging from somewhere and in my opinion it doesn’t need to be a typical “welcome yo!” message. So this is about what I’m working at the moment. Maybe later I can add a proper first(!) post.

Currently I’m debugging a minidump file for a remote crash and since the crash was not on my computer, I needed to tell visual studio where to find corresponding binaries in my computer (if you are not familiar with postmortem debugging, here you can find a very nicely prepared article). I know that it’s the MODPATH argument that points to local module path.

But it didn’t work… After digging around a bit, I found a little information here:

In previous versions of Visual Studio, the MODPATH argument was used to specify additional module search paths. Visual Studio 2005 looks for modules in the specified symbol search paths and does not use MODPATH.

Grrr! :)

© 2012 Notes to Shelf Suffusion theme by Sayontan Sinha