Regarding modern Tau hand to hand proficiency:
It's very clear in the original Tau background that their lesser close combat stats are entirely due to training and experience factor. This is why the Farsight Enclaves, who have much more extensive training and experience in hand to hand combat, have Fire Warriors with close combat stats equal to Guardsmen in 3rd edition (and Battlesuit Shas'Vre and Commanders with close combat stats that come close to a Space Marine!) for the low low cost of 5 points per model.
Thus, it can be assumed that whatever physiological deficiencies Tau might have in the context of hand to hand combat, they are either so minor their effect is negligible or are offset by physiological attributes that provide other advantages (like having a nice hard set of hooves to kick into things like shins and other more sensitive areas).
It should also be noted in the context of 'modern' Tau that my own Kakapo Canon(tm) holds that the Farsight Enclaves have in their isolation developed their own set of martial arts and hand to hand combat techniques independent of anything in the Tau Empire proper.
Regarding Ancient (pre-Ethereal) Tau:
When discussing Tau civilisation before the advent of the Ethereals (and even after), one must always remember this: Tau are not humans
. They may have some similarities, but they are ultimately an entirely distinct alien species, with an entirely distinct alien civilisation, originating from an entirely distinct alien history on an alien world.
This is important, because Tau warfare appears to have evolved along a very different path to how it did in most human societies. Going by official GW design notes, pre-Ethereal Tau still fought using mostly ranged combat:
Jess Goodwin, White Dwarf issue 400 (AU), pg 82, 'Battlesuits: The Beginning' wrote:Tau combat doctrine turns [the significance of hand to hand combat] firmly on its head. Instead the Tau conform to ancient principles that liken their foe to the prey on a hunt and focus entirely on slaying it with ranged weapons. In their ancient history that will have been with bows, and we imagined situations where the use of a bow would become and almost ceremonial affair, with warriors spending years honing the perfect way to draw and loose an arrow.
Going by this, it seems the principle weapon of choice for the proto-Fire Caste Plains Tau in battle was the bow and arrow. This actually makes a lot of sense when you stop to think about it - after all the Fire Tribe plains Tau were, before anything else, hunters
, so it's only logical that when they needed to go to war and fight battles they would simply bring their hunting bows (which would presumably be readily available to them and something they had years of practice using, since being a nomadic hunter-gatherer society means they would be one of their main ways of getting food) and fight with them.
As a result, it's likely that any hand-to-hand combat methods used by the Fire Tribe Tau would have been largely if not entirely secondary to archery (and later musketry once firearms started proliferating across T'au). As an aside, this has some implications on the kind of weaponry used - Kakapo Canon(tm) includes Ancient Fire Tribe weapons such as the 'hand-spear', a small single-handed stabbing weapon with a pointed tip and colourful set of rear feathering to distract opponents and fashioned from surplus arrows as well as various flexible fighting staffs based on a combination of bow limbs an a riser (I.E. a bow without the bowstring), both created as logical assumptions about what kinds of weapons might be produced by a society whose main focus in weapon construction and practice is the bow an arrow. Spears would also be likely to find use by Fire Tribe warriors, as they also have use in hunting, but again would be of secondary importance.
Thus, as I see it, there are two potential ways to approach Ancient Fire Tribe martial arts. The first would be as a largely defensive practice intended to facilitate further ranged combat. In this case most techniques would revolve around getting the enemy away or pushing them back far enough to shoot them with an arrow (or later on a gun), and would likely use the bow and arrow as a central part with an emphasis on quick close range archery. A Fire Tribe warrior fighting in close combat this way would probably look something like Legolas in the climax of The Fellowship of The Ring
); hand-to-hand fighting yes, but largely as a means to push away an opponent back to arms length and still secondary to close-range archery/shooting. Further support for this school of thought can also be seen in Dawn of War: Dark Crusade
and its sister game Dawn of War: Soulstorm
, where the close combat animations of Tau Fire Warriors suggest that 'push the enemy back to arms' length so you can shoot them in the face' is exactly what they try to do in close combat situations.
The alternative idea is to consider the Fire Tribe Tau's roots as hunters. Much of the Fire Tribe (and by extension the Fire Caste) art of war and combat is derived from hunting, and it's likely that any traditional martial arts practised by them would be no exception. Therefore it's likely that at least a few traditional Fire Tribe martial arts would be based on techniques used to subdue or take down large dangerous animals, and were likely first used for that before being re-purposed on enemy warriors. If you wish to pursue this path look to techniques for capturing wild boar or cattle for inspiration. Such martial arts would also likely include more non-lethal techniques, making them useful for capturing prisoners and high-value targets that need to be taken alive.
It is even possible that both schools of thought were practised by Fire Tribe Tau, forming two separate classes or schools of martial arts the way Mont'ka and Kau'yon are two separate combat doctrines.
It should be noted that there are two limitations in this world-building exercise. The first is that it is primarily focused on the Fire Tribe of plains-dwelling Tau that would later become the Fire Caste. All of the different tribes of Tau on Ancient T'au would have fought in conflicts before The Ethereals appeared, and thus it's likely that they would have developed their own different styles of martial arts, some of which may have been taken up by the Fire Caste following the end of the Mont'au. The Earth Tribe of Tau in particular warrant further consideration, given their position as the main rivals of the Fire Tribe and their entirely different cultural development. With their greater emphasis on defence of static locations (like the ramparts of their city-fortresses), it's likely that the Earth Tribe developed distinct martial arts more in line with what you might find in a human civilisation, and would presumably have a more extensive arsenal of hand-to-hand weapons for repelling attackers trying to storm their fortifications (polearms seem especially useful to me here, given their potential uses in pushing attackers off siege ladders and impaling multiple enemies packed together inside a siege tower), since their more static methods of warfare would leave them in more hand-to-hand combat situations - if a warband of nomadic Fire Tribe Tau are attacked, they can simply pack up and move away before riddling you with arrows, but if an Earth Tribe Tau settlement is attacked, the defenders a kind of stuck in one spot unless they want to abandon their fortifications.
The other factor to consider is that the above has been primarily focused on the Tau Empire proper. As I mentioned earlier (and include as Kakapo Canon(tm)), the Enclave Tau are almost certain to have developed their own hand-to-hand combat techniques.