There are no technical limits on the size of a bomb but the curvature of the earth sets a limitation on the area that can be devastated by a groundburst. Airbursts are far more damaging and if detonated say about a kilometre above the ground could flatten entire cities.

I don't think enough volatiles exist within reach to blow up the entire planet. At best you might cause shockwaves sufficient to create tectonic movement and trigger a supervolcano erruption. That would be enough to end human civilisation.

A more effective weapon would be a space based inertia weapon. The first country to achieve permanent foothold in deep space will own the entire solar system. You need only give a near earth asteroid a slight nudge in the right direction to take out an entire continent.