The best way to grip a rattlesnake by the neck is generally not to do it at all. Tubing is much safer and more humane, resulting in less risk to the handler and less physical stress and risk of injury to the snake. Most procedures including oral examination can be carried out while the snake's head and neck are safely held inside a short, clear plastic tube. For some procedures (particularly venom extraction) a direct head grip may be the most appropriate.
Note how the thumb and first knuckle are tight to the back of the mandible and there are no fingers within fang range. This is a firm but gentle restraint The grip should not be so tight that it compresses the animal's glottis and interferes with breathing. A good way to determine whether your grip is too tight is to check on a healthy, hydrated animal how fast the neck skin rebounds into a normal shape after you have released it. Also check carefully for swelling in this area over the next few days.
If you are not confident of your ability to maintain a gentle grip that is safe for both you and the snake, all of your procedures should be performed with the animal still in the tube.
A pit viper's neck is an extremely fragile structure and can be damaged very easily. Adult elapids and colubrids can be gripped with somewhat less delicacy as they have stronger supporting muscle structures, but care should still be taken not to hurt the animal or interfere with its breathing. If your grip is painful or uncomfortable for the snake, you can expect much more struggling. This is dangerous to both the snake and the handler, so try to make sure that your grip is as comfortable and gentle as possible while still being firm enough for effective restraint.
Here's the same grip from a different camera perspective. For many procedures it provides better support to keep the animal down on a padded surface rather than picking it up. This way there is less risk of injury to the spine or to the vulnerable occipital condyle if the animal struggles violently.
For the snake's safety as well as your own, the lower body of the snake should be gently but firmly restrained during any procedures. A second handler may assist. The snake's lower body may be confined in a tube or a pillowcase for ease of control if your procedures involve only the cranial anatomy.